| ctjctj | noname01x3, your virtual box has lost a disk drive. In my world that would be somebody messing up big time and I'd be screaming at them. Since that drive is mounted on /data I would assume it is a database or something like that so even if we get the server back up and running it is unlikely to be functioning correctly. To get your system to boot into something other than emergency mode (single user mode in old timer | 00:03 |
|---|---|---|
| ctjctj | speak) you need to edit /etc/fstab (nano /etc/fstab) and put a # in front of the line that has /dev/sdc1 /data in it. Write that file out and reboot with "shutdown -r +1". Your server should then boot but it will be unlikely to function. Get on the phone with your boss and let him know that whomever is in charge of the virtualization seems to have taken a disk drive away and that you are expecting sever data loss. | 00:03 |
| === JasonMc92 is now known as CodeMouse92 | ||
| === CodeMouse92 is now known as CodeMouse92__ | ||
| Noname01x2 | hi | 00:47 |
| Noname01x2 | hey | 00:52 |
| tsimonq2 | ho | 01:08 |
| tarpman | hum | 01:08 |
| Noname01x2 | hey guys | 01:10 |
| tsimonq2 | Hiya Noname01x2, you need anything in particular? :) | 01:11 |
| Noname01x2 | ctjctj, are u here? | 01:13 |
| tsimonq2 | Noname01x2: What did you need from him? Maybe someone else can lend a hand too. :) | 01:15 |
| tarpman | tsimonq2: scroll up. tl;dr - VM with a disk removed, systemd is complaining (understandably) | 01:19 |
| tsimonq2 | tarpman: Hm ok thanks | 01:22 |
| Noname01x2 | im back | 01:27 |
| Noname01x2 | ctjctj, hey | 01:27 |
| Noname01x2 | ctjctj, hellooo | 01:27 |
| sarnold | he may be gone for the night, who knows | 01:34 |
| sarnold | best to just ask questions :) | 01:34 |
| tsimonq2 | ^ | 01:35 |
| Noname01x2 | ok | 01:39 |
| Noname01x2 | im at home now soo uhhggg | 01:39 |
| Noname01x2 | sarnold, what am i to do with this stupid. | 01:46 |
| Noname01x2 | stupid server | 01:46 |
| sarnold | Noname01x2: where are you stuck? | 01:46 |
| Noname01x2 | emergency mode | 01:48 |
| sarnold | still? last I saw you were trying to bring back a missing disk | 01:49 |
| === giraffe is now known as Guest52692 | ||
| Noname01x2 | disk was not mounted maybe | 01:50 |
| === JanC is now known as Guest69133 | ||
| === JanC_ is now known as JanC | ||
| noname01 | im back | 02:09 |
| noname01 | on my laptop now | 02:09 |
| mwhahaha | coreycb, jamespage: tests are passing, only outstanding issue is that designate mdns is not happy. we're disabling deploying that for now, but just fyi http://logs.openstack.org/48/422248/15/check/gate-puppet-openstack-integration-4-scenario003-tempest-ubuntu-xenial-nv/684b4df/logs/syslog.txt.gz#_Jan_24_23_48_19 | 02:20 |
| coreycb | mwhahaha, thanks for the update. i'll take a look at designate in the morning. | 02:23 |
| noname01 | i get kicked way too much | 02:43 |
| === giraffe is now known as Guest5935 | ||
| === Darkman802_ is now known as Darkman802 | ||
| JemalMoha | Help ! How to Disable ETRN and VRFY commands. | 06:26 |
| cpaelzer | JemalMoha: depends on your program - search engine gave me plenty of hits for "Disable The VRFY, EXPN, and ETRN commands in SMTP" | 08:32 |
| === rvba` is now known as rvba | ||
| zul | coreycb: im updating clients today | 14:10 |
| coreycb | zul, ok | 14:10 |
| mdow814 | Hey everyone, is this the correct chat for conjure up issues? | 14:19 |
| noname01x3 | yeah | 14:25 |
| zul | coreycb:im so confused.....there is something called python-cinderclient-ext now | 14:27 |
| coreycb | zul, what's that for? | 14:36 |
| zul | i have no idea | 14:36 |
| zul | http://git.openstack.org/cgit/openstack/python-brick-cinderclient-ext/tree/README.rst | 14:37 |
| adrian_1908 | If I want to forward email sent to name@mydomain.com (ubuntu server) to another address, is "postfix" the tool of choice, or is there a simple service designed for just that? | 14:49 |
| rbasak | Forwarding is a pain because you end up with spam being forwarded, then being unable to reject it at your boundary, so you have to create backscatter or drop them on the floor. | 15:02 |
| rbasak | Unless I'm behind on the times, postfix in particular is not great at handling this kind of thing well. | 15:03 |
| adrian_1908 | ah damn, I didn't even think about that. | 15:03 |
| rbasak | Better to have the final destination receive the email directly so it can do the spam filtering at the same time. I appreciate that's not always possible depending on what requirements you have for the final receiving service. | 15:05 |
| rbasak | cpaelzer: did you just accidentally undo a bunch of blueprint changes by starting from an old version? | 15:06 |
| adrian_1908 | rbasak: yes, I might actually consider that instead. Thanks for the insight. | 15:08 |
| cpaelzer | rbasak: I had about 40 minutes between pressing edit and finally getting to do the update | 15:15 |
| cpaelzer | rbasak: is there a log so that we can recover? | 15:16 |
| cpaelzer | maybe some changes fell into that window? | 15:16 |
| cpaelzer | rbasak: If the content is only refresh on a full page refresh it might even have lost more as I had the tab open from yesterday | 15:16 |
| cpaelzer | rbasak: I better not press edit again until I heard from you how we coordinate the recovery | 15:18 |
| cpaelzer | rbasak: please ping me, query/hangout as you want | 15:18 |
| jgrimm | cpaelzer, i have the diff of your changes if you need them | 15:22 |
| cpaelzer | jgrimm: yeah please send me a mail - I know what I wanted to change and can sort out the rest | 15:22 |
| jgrimm | cpaelzer, it was probably only the re-ordering that i did yesterday | 15:23 |
| jgrimm | cpaelzer, ack | 15:23 |
| cpaelzer | jgrimm: I subscribed now to do that without help in any case it happens again | 15:23 |
| jgrimm | cpaelzer, no worries. sent | 15:23 |
| cpaelzer | thx jgrimm | 15:24 |
| cpaelzer | rbasak: I'd clean up now - please stop me if you are already on it | 15:24 |
| coreycb | mwhahaha, designate-mdns needs python-monasca-statsd. we have that as a suggested dependency so i'm going to move that to a required dependency. | 15:29 |
| mwhahaha | coreycb: oh i had previously dealt with that by adding in a noop to bypass the statsd thing | 15:30 |
| coreycb | mwhahaha, oh interesting | 15:30 |
| mwhahaha | coreycb: https://review.openstack.org/#/c/389800/ | 15:30 |
| mwhahaha | coreycb: cause rdo didn't want to package statsd so maybe i need to check our config | 15:31 |
| mwhahaha | coreycb: do you know if that change and https://review.openstack.org/#/c/393829/ is in the package? | 15:31 |
| coreycb | mwhahaha, yeah it looks like it is in there | 15:32 |
| cpaelzer | jgrimm: rbasak: I made two changes now - the first is a revert of the former accident (thnkas for the diff jgrimm) and the second is the change I actually wanted to make | 15:33 |
| jgrimm | \o/ | 15:33 |
| rbasak | cpaelzer: sorry, I've only just seen this. I think it's resolved now? | 15:34 |
| cpaelzer | rbasak: yes | 15:36 |
| coreycb | mwhahaha, is this commit missing any code by any chance? https://github.com/openstack/designate/commit/224e279a39b4f7dd12fe0ed0747f22d647a691ca | 15:48 |
| noname01x3 | hey guys ok I have more information about a problem i was having with my ubuntu server. | 15:54 |
| noname01x3 | if anyone here is interested in helping me out. let me know. thanks. | 15:55 |
| mwhahaha | coreycb: no | 15:55 |
| mwhahaha | coreycb: that was the one that fixed the noop stuff so that should work. i didn't think the mdns error was related to the stats stuff but i can look closer later today | 15:56 |
| coreycb | mwhahaha, ok i'm just confused as to why the metrics.py bits didn't land | 15:58 |
| coreycb | mwhahaha, yeah adding statsd fixes the mdns error | 15:59 |
| mwhahaha | coreycb: so you have to config it not to use statsd and use noop byd efault | 15:59 |
| coreycb | mwhahaha, ah ok i see | 15:59 |
| mwhahaha | because in their infinite wisdom, designate merged that in on by default | 15:59 |
| coreycb | :) | 15:59 |
| coreycb | mwhahaha, ok lemme know how it goes | 15:59 |
| mwhahaha | coreycb: oh so we handled the missing statsd with an oslo thing | 16:02 |
| mwhahaha | coreycb: https://github.com/openstack/designate/blob/master/designate/metrics.py#L46-L47 | 16:02 |
| mwhahaha | so that should fall back to the noop if statsd is not present | 16:02 |
| mwhahaha | if you guys have python-monasca-statd as a dep that'd fix it (and probably be better for the end user) | 16:03 |
| coreycb | mwhahaha, thanks. i'm going to leave it as a Suggests dependency and update the default config to fall back to noop since it really is optional | 16:05 |
| mwhahaha | coreycb: so i was wrong that it's not a config option, as it should just fall back in the code. if i get some time i'll dig into it more but it didn't seem like any error i had seen related to this previously but who knows | 16:06 |
| coreycb | mwhahaha, i just noticed that, it defaults to disabled | 16:06 |
| mwhahaha | designate makes me sad :( | 16:06 |
| coreycb | mwhahaha, heh | 16:06 |
| coreycb | mwhahaha, based on the traceback with TypeError: 'NoneType' object is not callable, i think the failure is at the @metrics.timed('mdns.xfr.zone_sync') decorator in designate/mdns/xfr.py | 16:14 |
| noname01x3 | sudo apt-get install boot-repair | 16:14 |
| coreycb | metrics = None | 16:14 |
| coreycb | mwhahaha, which comes from designate/metrics.py | 16:14 |
| mwhahaha | coreycb: hmm ok weird | 16:14 |
| coreycb | mwhahaha, and i confirmed that installing python-monasca-statsd fixed it | 16:14 |
| mwhahaha | coreycb: based on that, https://github.com/openstack/designate/commit/224e279a39b4f7dd12fe0ed0747f22d647a691ca should have been the fix for that | 16:19 |
| mwhahaha | coreycb: metrics = None isn't the issue cause it gets set on line 115 | 16:19 |
| mwhahaha | coreycb: my fix should have fixed the nonetype error because without it, https://github.com/openstack/designate/blob/master/designate/metrics.py#L107-L109 blows up | 16:19 |
| mwhahaha | but i'll dig into later | 16:20 |
| noname01x3 | my server is down.... | 16:33 |
| zul | coreycb: hey....i think we need a MIR for monasca-statsd | 16:39 |
| coreycb | zul, i'm adding it back to suggests | 16:39 |
| zul | coreycb: ok | 16:39 |
| zul | coreycb: http://pastebin.ubuntu.com/23864831/ | 18:19 |
| coreycb | zul, does that test designate-mdns? | 18:20 |
| zul | coreycb: yes | 18:21 |
| coreycb | zul, ok i'm guessing you have monasca-statsd installed | 18:22 |
| zul | coreycb:possibly...i didnt check | 18:22 |
| zul | coreycb: yes | 18:28 |
| coreycb | zul, ok so the test should currently fail w/out statsd | 18:28 |
| zul | ok...so we should probably have it depend on statsd then no? | 18:29 |
| coreycb | zul, no | 18:29 |
| zul | or have the tests install statsd | 18:29 |
| coreycb | zul, since it's optional it'd be good to run the tests without it installed | 18:29 |
| zul | ok | 18:29 |
| faraway | While monitoring the processes (/proc/stat) on my server I observe regular peeks of process launches and I would like to figure out what processes are launched, can someone recommend to tool to log which processes are launched? | 18:38 |
| axisys | interface file only take route command? can I use ip route instead? | 18:38 |
| axisys | interfaces* | 18:38 |
| zul | coreycb: i was thinnking we should do barbican and manila as well | 18:41 |
| coreycb | zul, that would be great | 18:41 |
| maswan | faraway: process accounting | 18:44 |
| rbasak | faraway: atop | 18:45 |
| rbasak | (it uses process accounting) | 18:45 |
| sarnold | faraway: execsnoop | 18:45 |
| sarnold | (it uses kernel tracing) | 18:45 |
| sarnold | granted, execsnoop isn't new processes, but it does show new execs, which in many cases is what you're more interested in anyway | 18:46 |
| faraway | I’ll take a look at those. I’m not exactly sure what is causeing that peeks, but I guess it could be the statistic tool because. The problem is the processes seem to be only active for a short time so the don’t appear in htop or top. I’ll take a look at your suggestions | 18:48 |
| faraway | […]because it it appears in regular intervals and only for a short period. | 18:49 |
| zul | coreycb: ugh...this is slightly less than ideal http://pastebin.ubuntu.com/23865072/ | 19:17 |
| faraway | sarnold, maswan, rbasak many thanks for your help. I was able to find which commands where causing the peeks. | 19:21 |
| coreycb | zul, what's that from? | 19:21 |
| zul | coreycb: aodh dep8 tests | 19:21 |
| zul | coreycb: im onit | 19:21 |
| sarnold | faraway: nice :) | 19:22 |
| === miczac_ is now known as miczac | ||
| Amgine | How to start mysqld with --skip-grant-tables --skip-networking ? when I use # /usr/sbin/mysqld the socket does not exist, so the attempt to reset the root password fails. | 19:53 |
| sarnold | Amgine: are you trying to make a permanent change or a temporary change? | 19:54 |
| Amgine | Permanent, since I cannot recover the previous mysql root pass. | 19:54 |
| tomreyn | Amgine: does the mysql server still up fine without changes? | 20:06 |
| tomreyn | if it does, you could start it then use "mysql --defaults-file=/etc/mysql/debian.cnf" to gain privileged access, then use sql to set a new root password. | 20:07 |
| Amgine | tomreyn: how would that work? let me check that .cnf | 20:10 |
| Amgine | does debian-sys-maint have full privs? | 20:11 |
| tomreyn | Amgine: this configuration file contains credentials of a management user which is automatically setup by debian derivates so that it can carry out maintenance tasks. since it needs elevated privileges to carry out several of these maintenance tasks it does get some of those by default. it shoudl be sufficient to reset the root password. | 20:11 |
| tomreyn | the *mysql* root user's password, that is | 20:12 |
| coreycb | zul, i'm going to add an epoch to stevedore and oslo.context and downgrade the to align with upper-constraints | 20:17 |
| coreycb | s/the/them | 20:17 |
| Amgine | w00t, that worked. Thank you tomreyn!! | 20:18 |
| zul | coreycb: ok | 20:18 |
| noname01x3 | sarnold, hey dude | 20:22 |
| noname01x3 | sarnold, now I can bug you again!!! yes! | 20:22 |
| sarnold | hey noname01x3 :) | 20:23 |
| noname01x3 | sarnold, Ok. so my boss says he ran an update and thats when this problem started where it boots into emergency mode. | 20:28 |
| noname01x3 | sarnold, it was working fine before and after one of the physical hard drives were removed. | 20:29 |
| sarnold | noname01x3: one problem at a time | 20:29 |
| tarpman | noname01x3: clearly your first mistake was to let a boss anywhere near the servers ;) | 20:29 |
| sarnold | noname01x3: why was the hard drifve removed? was it just yanked out via the management console or did he also remove the fstab entries and all references to it on the server? | 20:30 |
| noname01x3 | tarpman, exactly | 20:30 |
| noname01x3 | tarpman it took so long to even get him to admit that he upated it. | 20:30 |
| noname01x3 | sarnold, hold on... my boss is saying after he removed the hard drive... he didnt make any coresponding changes on the server - other than running an update because he saw the small file size and thought it would be no big deal to update. | 20:42 |
| noname01x3 | arnold, so he removed the drive. was able to run the server, with no problem. then ran an update. and still there was no problem... until he shut down and booted up, and it booted into emergency mode. | 20:44 |
| noname01x3 | sarnold, we are planning to have a vendor that has access to that server, come in and redo the server from scratch. this will cost us a lot of money and there will be down time. we have until friday to fix this. | 20:45 |
| sarnold | noname01x3: so, where are you stuck? have you removed the filesystem from the /etc/fstab yet? | 20:49 |
| noname01x3 | sarnold, no i have not. this is my first time trying to fix a linux server. | 20:50 |
| noname01x3 | sarnold, I would like to try this. please instruct me. | 20:50 |
| sarnold | noname01x3: once you're in rescue mode again, edit the /etc/fstab with nano or whatever editor it is you like, and add a # to the line that had your /data filesystem | 20:51 |
| noname01x3 | sarnold, ok i will try that now | 20:52 |
| noname01x3 | sarnold, it already has the # | 20:58 |
| noname01x3 | sarnold, can you tell me what me /ect/fstab is telling me? | 20:59 |
| noname01x3 | i remember pastebin | 20:59 |
| noname01x3 | Http://paste.ubuntu.com/23865652 | 21:00 |
| sarnold | noname01x3: /etc/fstab tells the system which filesystems to mount | 21:03 |
| sarnold | noname01x3: note that this shows the /data filesystem is still not commented out | 21:04 |
| noname01x3 | sarnold, I think i made a boo boo. | 21:05 |
| noname01x3 | I have a blank screen | 21:05 |
| noname01x3 | I saw that the # was missing on one of the rows so I added it. | 21:06 |
| noname01x3 | OMG WAIT ITS NOW LOADING THE OS | 21:06 |
| noname01x3 | AHSHHSHAH | 21:06 |
| sarnold | The # is a comment character | 21:06 |
| noname01x3 | you did it | 21:06 |
| sarnold | it means "don't read the rest of this line" | 21:06 |
| noname01x3 | holyyy | 21:06 |
| noname01x3 | ddude | 21:06 |
| noname01x3 | its pick and purple | 21:06 |
| noname01x3 | pink | 21:06 |
| sarnold | most configuration files have a comment character and most of them pick #, but sometimes it's ; or ! | 21:06 |
| noname01x3 | sarnold, you have saved us! holy crap | 21:06 |
| noname01x3 | i can use my mouse now | 21:07 |
| noname01x3 | wow you fixed it! | 21:07 |
| noname01x3 | thank you so much dude | 21:07 |
| noname01x3 | holy lmao | 21:08 |
| sarnold | keep looking through the logs to make sure you get everything back to normal | 21:08 |
| noname01x3 | omg i dont even want to touch it now | 21:08 |
| noname01x3 | how do i go into logs again, however? | 21:09 |
| noname01x3 | i mean not from the terminal | 21:09 |
| noname01x3 | wait wait let me call me boss to give the good news. | 21:10 |
| noname01x3 | lol this is funny to me. so the boot instructions had a comment as the instruction. haha | 21:15 |
| sarnold | noname01x3: depends what you're looking for; dmesg for the kernel logs, journalctl for 'all the logs', /var/log/ has a huge pile of more specific logs, most things wind up in /var/log/syslog | 21:16 |
| noname01x3 | sarnold, oh wow so its much more complicated. i get it though. | 21:16 |
| noname01x3 | sarnold, I'm learning a lot and surprisingly I'm able to remember this stuff. | 21:17 |
| sarnold | good good :D | 21:18 |
| noname01x3 | sarnold, I would like to stay in touch. can I have your email address? Mine is NoName01x1@outlook.com | 21:21 |
| sarnold | noname01x3: seth.arnold@gmail.com | 21:22 |
| noname01x3 | thx have a good evening. tell ctjctj that my issue has been resolved. | 21:23 |
| noname01x3 | I owe you guys one. | 21:24 |
| sarnold | have fun noname01x3 :) | 21:25 |
| * genii harvests the emails and goes to make coffee | 21:26 | |
| genii | ;) | 21:26 |
| sarnold | genii: google saves me from all but like three spam a year. i don't know how they do it but it's impressive. | 21:27 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!