ctjctjnoname01x3, your virtual box has lost a disk drive.  In my world that would be somebody messing up big time and I'd be screaming at them.  Since that drive is mounted on /data I would assume it is a database or something like that so even if we get the server back up and running it is unlikely to be functioning correctly.  To get your system to boot into something other than emergency mode (single user mode in old timer00:03
ctjctjspeak) you need to edit /etc/fstab (nano /etc/fstab) and put a # in front of the line that has /dev/sdc1 /data in it.  Write that file out and reboot with "shutdown -r +1".  Your server should then boot but it will be unlikely to function.  Get on the phone with your boss and let him know that whomever is in charge of the virtualization seems to have taken a disk drive away and that you are expecting sever data loss.00:03
=== JasonMc92 is now known as CodeMouse92
=== CodeMouse92 is now known as CodeMouse92__
Noname01x2hey guys01:10
tsimonq2Hiya Noname01x2, you need anything in particular? :)01:11
Noname01x2ctjctj, are u here?01:13
tsimonq2Noname01x2: What did you need from him? Maybe someone else can lend a hand too. :)01:15
tarpmantsimonq2: scroll up. tl;dr - VM with a disk removed, systemd is complaining (understandably)01:19
tsimonq2tarpman: Hm ok thanks01:22
Noname01x2im back01:27
Noname01x2ctjctj, hey01:27
Noname01x2ctjctj, hellooo01:27
sarnoldhe may be gone for the night, who knows01:34
sarnoldbest to just ask questions :)01:34
Noname01x2im at home now soo uhhggg01:39
Noname01x2sarnold, what am i to do with this stupid.01:46
Noname01x2stupid server01:46
sarnoldNoname01x2: where are you stuck?01:46
Noname01x2emergency mode01:48
sarnoldstill? last I saw you were trying to bring back a  missing disk01:49
=== giraffe is now known as Guest52692
Noname01x2disk was not mounted maybe01:50
=== JanC is now known as Guest69133
=== JanC_ is now known as JanC
noname01im back02:09
noname01on my laptop now02:09
mwhahahacoreycb, jamespage: tests are passing, only outstanding issue is that designate mdns is not happy. we're disabling deploying that for now, but just fyi http://logs.openstack.org/48/422248/15/check/gate-puppet-openstack-integration-4-scenario003-tempest-ubuntu-xenial-nv/684b4df/logs/syslog.txt.gz#_Jan_24_23_48_1902:20
coreycbmwhahaha, thanks for the update.  i'll take a look at designate in the morning.02:23
noname01i get kicked way too much02:43
=== giraffe is now known as Guest5935
=== Darkman802_ is now known as Darkman802
JemalMohaHelp ! How to Disable ETRN and VRFY commands.06:26
cpaelzerJemalMoha: depends on your program - search engine gave me plenty of hits for "Disable The VRFY, EXPN, and ETRN commands in SMTP"08:32
=== rvba` is now known as rvba
zulcoreycb: im updating clients today14:10
coreycbzul, ok14:10
mdow814Hey everyone, is this the correct chat for conjure up issues?14:19
zulcoreycb:im so confused.....there is something called python-cinderclient-ext now14:27
coreycbzul, what's that for?14:36
zuli have no idea14:36
adrian_1908If I want to forward email sent to name@mydomain.com (ubuntu server) to another address, is "postfix" the tool of choice, or is there a simple service designed for just that?14:49
rbasakForwarding is a pain because you end up with spam being forwarded, then being unable to reject it at your boundary, so you have to create backscatter or drop them on the floor.15:02
rbasakUnless I'm behind on the times, postfix in particular is not great at handling this kind of thing well.15:03
adrian_1908ah damn, I didn't even think about that.15:03
rbasakBetter to have the final destination receive the email directly so it can do the spam filtering at the same time. I appreciate that's not always possible depending on what requirements you have for the final receiving service.15:05
rbasakcpaelzer: did you just accidentally undo a bunch of blueprint changes by starting from an old version?15:06
adrian_1908rbasak: yes, I might actually consider that instead. Thanks for the insight.15:08
cpaelzerrbasak: I had about 40 minutes between pressing edit and finally getting to do the update15:15
cpaelzerrbasak: is there a log so that we can recover?15:16
cpaelzermaybe some changes fell into that window?15:16
cpaelzerrbasak: If the content is only refresh on a full page refresh it might even have lost more as I had the tab open from yesterday15:16
cpaelzerrbasak: I better not press edit again until I heard from you how we coordinate the recovery15:18
cpaelzerrbasak: please ping me, query/hangout as you want15:18
jgrimmcpaelzer, i have the diff of your changes if you need them15:22
cpaelzerjgrimm: yeah please send me a mail - I know what I wanted to change and can sort out the rest15:22
jgrimmcpaelzer, it was probably only the re-ordering that i did yesterday15:23
jgrimmcpaelzer, ack15:23
cpaelzerjgrimm: I subscribed now to do that without help in any case it happens again15:23
jgrimmcpaelzer, no worries. sent15:23
cpaelzerthx jgrimm15:24
cpaelzerrbasak: I'd clean up now - please stop me if you are already on it15:24
coreycbmwhahaha, designate-mdns needs python-monasca-statsd.  we have that as a suggested dependency so i'm going to move that to a required dependency.15:29
mwhahahacoreycb: oh i had previously dealt with that by adding in a noop to bypass the statsd thing15:30
coreycbmwhahaha, oh interesting15:30
mwhahahacoreycb: https://review.openstack.org/#/c/389800/15:30
mwhahahacoreycb: cause rdo didn't want to package statsd so maybe i need to check our config15:31
mwhahahacoreycb: do you know if that change and https://review.openstack.org/#/c/393829/ is in the package?15:31
coreycbmwhahaha, yeah it looks like it is in there15:32
cpaelzerjgrimm: rbasak: I made two changes now - the first is a revert of the former accident (thnkas for the diff jgrimm) and the second is the change I actually wanted to make15:33
rbasakcpaelzer: sorry, I've only just seen this. I think it's resolved now?15:34
cpaelzerrbasak: yes15:36
coreycbmwhahaha, is this commit missing any code by any chance?  https://github.com/openstack/designate/commit/224e279a39b4f7dd12fe0ed0747f22d647a691ca15:48
noname01x3hey guys ok I have more information about a problem i was having with my ubuntu server.15:54
noname01x3if anyone here is interested in helping me out. let me know. thanks.15:55
mwhahahacoreycb: no15:55
mwhahahacoreycb: that was the one that fixed the noop stuff so that should work. i didn't think the mdns error was related to the stats stuff but i can look closer later today15:56
coreycbmwhahaha, ok i'm just confused as to why the metrics.py bits didn't land15:58
coreycbmwhahaha, yeah adding statsd fixes the mdns error15:59
mwhahahacoreycb: so you have to config it not to use statsd and use noop byd efault15:59
coreycbmwhahaha, ah ok i see15:59
mwhahahabecause in their infinite wisdom, designate merged that in on by default15:59
coreycbmwhahaha, ok lemme know how it goes15:59
mwhahahacoreycb: oh so we handled the missing statsd with an oslo thing16:02
mwhahahacoreycb: https://github.com/openstack/designate/blob/master/designate/metrics.py#L46-L4716:02
mwhahahaso that should fall back to the noop if statsd is not present16:02
mwhahahaif you guys have python-monasca-statd as a dep that'd fix it (and probably be better for the end user)16:03
coreycbmwhahaha, thanks.  i'm going to leave it as a Suggests dependency and update the default config to fall back to noop since it really is optional16:05
mwhahahacoreycb: so i was wrong that it's not a config option, as it should just fall back in the code. if i get some time i'll dig into it more but it didn't seem like any error i had seen related to this previously but who knows16:06
coreycbmwhahaha, i just noticed that, it defaults to disabled16:06
mwhahahadesignate makes me sad :(16:06
coreycbmwhahaha, heh16:06
coreycbmwhahaha, based on the traceback with TypeError: 'NoneType' object is not callable, i think the failure is at the @metrics.timed('mdns.xfr.zone_sync') decorator in designate/mdns/xfr.py16:14
noname01x3sudo apt-get install boot-repair16:14
coreycbmetrics = None16:14
coreycbmwhahaha, which comes from designate/metrics.py16:14
mwhahahacoreycb: hmm ok weird16:14
coreycbmwhahaha, and i confirmed that installing python-monasca-statsd fixed it16:14
mwhahahacoreycb: based on that, https://github.com/openstack/designate/commit/224e279a39b4f7dd12fe0ed0747f22d647a691ca should have been the fix for that16:19
mwhahahacoreycb: metrics = None isn't the issue cause it gets set on line 11516:19
mwhahahacoreycb: my fix should have fixed the nonetype error because without it, https://github.com/openstack/designate/blob/master/designate/metrics.py#L107-L109 blows up16:19
mwhahahabut i'll dig into later16:20
noname01x3my server is down....16:33
zulcoreycb: hey....i think we need a MIR for monasca-statsd16:39
coreycbzul, i'm adding it back to suggests16:39
zulcoreycb: ok16:39
zulcoreycb:  http://pastebin.ubuntu.com/23864831/18:19
coreycbzul, does that test designate-mdns?18:20
zulcoreycb: yes18:21
coreycbzul, ok i'm guessing you have monasca-statsd installed18:22
zulcoreycb:possibly...i didnt check18:22
zulcoreycb: yes18:28
coreycbzul, ok so the test should currently fail w/out statsd18:28
zulok...so we should probably have it depend on statsd then no?18:29
coreycbzul, no18:29
zulor have the tests install statsd18:29
coreycbzul, since it's optional it'd be good to run the tests without it installed18:29
farawayWhile monitoring the processes (/proc/stat)  on my server I observe regular peeks of process launches and I would like to figure out what processes are launched, can someone recommend to tool to log which processes are launched?18:38
axisysinterface file only take route command? can I use ip route instead?18:38
zulcoreycb: i was thinnking we should do barbican and manila as well18:41
coreycbzul, that would be great18:41
maswanfaraway: process accounting18:44
rbasakfaraway: atop18:45
rbasak(it uses process accounting)18:45
sarnoldfaraway: execsnoop18:45
sarnold(it uses kernel tracing)18:45
sarnoldgranted, execsnoop isn't new processes, but it does show new execs, which in many cases is what you're more interested in anyway18:46
farawayI’ll take a look at those. I’m not exactly sure what is causeing that peeks, but I guess it could be the statistic tool because. The problem is the processes seem to be only active for a short time so the don’t appear in htop or top. I’ll take a look at your suggestions18:48
faraway[…]because it it appears in regular intervals and only for a short period.18:49
zulcoreycb: ugh...this is slightly less than ideal http://pastebin.ubuntu.com/23865072/19:17
farawaysarnold, maswan, rbasak many thanks for your help. I was able to find which commands where causing the peeks.19:21
coreycbzul, what's that from?19:21
zulcoreycb: aodh dep8 tests19:21
zulcoreycb: im onit19:21
sarnoldfaraway: nice :)19:22
=== miczac_ is now known as miczac
AmgineHow to start mysqld with --skip-grant-tables --skip-networking ? when I use # /usr/sbin/mysqld the socket does not exist, so the attempt to reset the root password fails.19:53
sarnoldAmgine: are you trying to make a permanent change or a temporary change?19:54
AmginePermanent, since I cannot recover the previous mysql root pass.19:54
tomreynAmgine: does the mysql server still up fine without changes?20:06
tomreynif it does, you could start it then use "mysql --defaults-file=/etc/mysql/debian.cnf" to gain privileged access, then use sql to set a new root password.20:07
Amginetomreyn: how would that work? let me check that .cnf20:10
Amginedoes debian-sys-maint have full privs?20:11
tomreynAmgine: this configuration file contains credentials of a management user which is automatically setup by debian derivates so that it can carry out maintenance tasks. since it needs elevated privileges to carry out several of these maintenance tasks it does get some of those by default. it shoudl be sufficient to reset the root password.20:11
tomreynthe *mysql* root user's password, that is20:12
coreycbzul, i'm going to add an epoch to stevedore and oslo.context and downgrade the to align with upper-constraints20:17
Amginew00t, that worked. Thank you tomreyn!!20:18
zulcoreycb: ok20:18
noname01x3sarnold, hey dude20:22
noname01x3sarnold, now I can bug you again!!! yes!20:22
sarnoldhey noname01x3 :)20:23
noname01x3sarnold, Ok. so my boss says he ran an update and thats when this problem started where it boots into emergency mode.20:28
noname01x3sarnold, it was working fine before and after one of the physical hard drives were removed.20:29
sarnoldnoname01x3: one problem at a time20:29
tarpmannoname01x3: clearly your first mistake was to let a boss anywhere near the servers ;)20:29
sarnoldnoname01x3: why was the hard drifve removed? was it just yanked out via the management console or did he also remove the fstab entries and all references to it on the server?20:30
noname01x3tarpman, exactly20:30
noname01x3tarpman it took so long to even get him to admit that he upated it.20:30
noname01x3sarnold, hold on... my boss is saying after he removed the hard drive... he didnt make any coresponding changes on the server - other than running an update because he saw the small file size and thought it would be no big deal to update.20:42
noname01x3arnold, so he removed the drive. was able to run the server, with no problem. then ran an update. and still there was no problem... until he shut down and booted up, and it booted into emergency mode.20:44
noname01x3sarnold, we are planning to have a vendor that has access to that server, come in and redo the server from scratch. this will cost us a lot of money and there will be down time. we have until friday to fix this.20:45
sarnoldnoname01x3: so, where are you stuck? have you removed the filesystem from the /etc/fstab yet?20:49
noname01x3sarnold, no i have not. this is my first time trying to fix a linux server.20:50
noname01x3sarnold, I would like to try this. please instruct me.20:50
sarnoldnoname01x3: once you're in rescue mode again, edit the /etc/fstab with nano or whatever editor it is you like, and add a # to the line that had your /data filesystem20:51
noname01x3sarnold, ok i will try that now20:52
noname01x3sarnold, it already has the #20:58
noname01x3sarnold, can you tell me what me /ect/fstab is telling me?20:59
noname01x3i remember pastebin20:59
sarnoldnoname01x3: /etc/fstab tells the system which filesystems to mount21:03
sarnoldnoname01x3: note that this shows the /data filesystem is still not commented out21:04
noname01x3sarnold, I think i made a boo boo.21:05
noname01x3I have a blank screen21:05
noname01x3I saw that the # was missing on one of the rows so I added it.21:06
sarnoldThe # is a comment character21:06
noname01x3you did it21:06
sarnoldit means "don't read the rest of this line"21:06
noname01x3its pick and purple21:06
sarnoldmost configuration files have a comment character and most of them pick #, but sometimes it's ; or !21:06
noname01x3sarnold, you have saved us! holy crap21:06
noname01x3i can use my mouse now21:07
noname01x3wow you fixed it!21:07
noname01x3thank you so much dude21:07
noname01x3holy lmao21:08
sarnoldkeep looking through the logs to make sure you get everything back to normal21:08
noname01x3omg i dont even want to touch it now21:08
noname01x3how do i go into logs again, however?21:09
noname01x3i mean not from the terminal21:09
noname01x3wait wait let me call me boss to give the good news.21:10
noname01x3lol this is funny to me. so the boot instructions had a comment as the instruction. haha21:15
sarnoldnoname01x3: depends what you're looking for; dmesg for the kernel logs, journalctl for 'all the logs', /var/log/ has a huge pile of more specific logs, most things wind up in /var/log/syslog21:16
noname01x3sarnold, oh wow so its much more complicated. i get it though.21:16
noname01x3sarnold, I'm learning a lot and surprisingly I'm able to remember this stuff.21:17
sarnoldgood good :D21:18
noname01x3sarnold, I would like to stay in touch. can I have your email address? Mine is NoName01x1@outlook.com21:21
sarnoldnoname01x3: seth.arnold@gmail.com21:22
noname01x3thx have a good evening. tell ctjctj that my issue has been resolved.21:23
noname01x3I owe you guys one.21:24
sarnoldhave fun noname01x3 :)21:25
* genii harvests the emails and goes to make coffee21:26
sarnoldgenii: google saves me from all but like three spam a year. i don't know how they do it but it's impressive.21:27

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!