[01:26] <Radar> We had a little over an hour of downtime this morning. What happened was that MySQL was killed due to out of memory, and then apparmor denied MySQL from starting up again. How can we investigate why this issue occurred?
[01:26] <Radar> Particularly why apparmor blocked MySQL
[01:28] <sarnold> Radar: dmesg | grep DEN ought to show you the denials; you can add the right privileges to allow them to the /etc/apparmor.d/usr.sbin.mysqld file -- and if they aren't specific to your site, you could file a bug so we include them for everyone else
[01:28] <Radar> sarnold: This hasn't been an issue before today. From what I can see in the usr.sbin.mysqld file, the permissions are correct.
[01:28] <Radar> I can show you the file if you'd like.
[01:28] <sarnold> sure
[01:29] <Radar> https://gist.github.com/radar/991f13af7859cdb69646
[01:30] <Radar> the /var/lib/mysql path is symlinked to /srv/mysql
[01:32] <Radar> I don't know what the "k" means here: https://gist.github.com/radar/991f13af7859cdb69646#file-mysqld-L33
[01:32] <Radar> Looks like that it has been configured for access to those directories.
[01:33] <sarnold> Radar: aha, probably that symlink...
[01:33] <Radar> It's been symlinked for ages.
[01:34] <Radar> We have an identically configured server where this is not an issue.
[01:34] <sarnold> Radar: add a rule like "alias /var/lib/mysql/ -> /srv/mysql/" to /etc/apparmor.d/tunables/alias
[01:35] <Radar> argh, our other server doesn't use a symlink!
[01:35] <jjohansen> Radar, sarnold: if you suspect any of apparmor's trusted helpers might be involved then greping /var/log/syslog is better, as they are logging there
[01:35] <jjohansen> eg. dbus
[01:35] <Radar> My apologies, I thought they were configured the same.
[01:35] <sarnold> jjohansen: gah thanks for the reminder. I'm still not used to this new world. :)
[01:36] <sarnold> Radar: ahhhh :)
[01:36] <sarnold> Radar: once you add the alias rule, you'll want to run apparmor_parser --replace /etc/apparmor.d/usr.sbin.mysqld
[01:37] <Radar> Thanks for the tip sarnold :)
[01:37] <sarnold> probably /etc/init.d/apparmor reload would do the same job...
[01:37] <sarnold> Radar: the 'k' means the process can issue fctnl locks on the file; some days I wonder if it's worth actually mediating those, but we do. :)
[01:47] <patdk-lap> I do
[08:07] <lordievader> Good morning.
[08:55] <solo1> how can i configure mdadm in a server raid1 ... it was with only 1 hd and i want to get raid1 using it ... i ve no idea how can i set ... 'due i found a lot of tutorial saying i must decide it when both are void or blank during server installation .... pls help
[08:57] <lordievader> solo1: https://www.howtoforge.com/how-to-set-up-software-raid1-on-a-running-system-incl-grub2-configuration-debian-squeeze-p2
[08:59] <solo1> to lordievader: tnx dude
[09:13] <solo1> to lordievader i got only 2 hdd and the guide speaks about 3
[09:17] <lordievader> solo1: Change it to your situation.
[10:03] <histo> solo1: how are you going to use 1 hd and do raid1
[10:05] <lordievader> histo: "i got only 2 hdd" Seems to me like he has two.
[10:05] <histo> lordievader: didn't get that far.  His first comment was stated weird.
[11:44] <IronDev> Guten Tag
[12:36] <smoser> strikov, did you make any headway on bug 1371634
[12:37] <strikov> smoser: i was able to create kvm instance with scsi multipath
[12:37] <smoser> oh nice.
[12:37] <strikov> smoser: playing with it now
[12:37] <strikov> smoser: funny thing is that even qemu-emulated multipath contains spaces in disk name
[12:37] <smoser> that is funny.
[12:38] <smoser> maybe user_friendly_names is just what we need ?
[12:38] <strikov> smoser: i suspect that issue might be in a different place
[12:38] <smoser> i reproduced the issue using kpartx on a regular device
[12:38] <smoser> see the bug
[12:38] <strikov> smoser: yes, that's the first thought I have, thats quite strange that it is not default then
[12:38] <smoser> i think that maybe kpartx to do that needs to replace ' ' with '\x20' or what not. maaybe the kernel juts expectst that and kpartx isnt doing it.
[12:39] <strikov> smoser: i think that (a) kpartx indeed has a bug but (b) upstream didn't fix it because you usually don't get such an input
[12:39] <strikov> smoser: example
[12:39] <smoser> maybe
[12:39] <strikov> smoser: http://pastebin.ubuntu.com/11564916/
[12:39] <strikov> smoser: tool which gets disk serial has a way to eliminate spaces
[12:40] <strikov> smoser: why not use it this way to generate udev name w/o spaces?
[12:40] <smoser> http://paste.ubuntu.com/11564919/
[12:40] <smoser> thta is from powerKVM
[12:41] <smoser> on a similar system, and yeah, they don thave the funny names there.
[12:41] <smoser> cmdline there has: $ cat /proc/cmdline
[12:41] <smoser> root=UUID=6ef1254c-50ac-421a-bb4a-25619189b327 ro console=tty0 console=hvc0 crashkernel=1024M rd.lvm.lv=ibmpkvm_vg_data/ibmpkvm_lv_data rd.lvm.lv=ibmpkvm_vg_log/ibmpkvm_lv_log rd.lvm.lv=ibmpkvm_vg_root/ibmpkvm_lv_root rd.lvm.lv=ibmpkvm_vg_swap/ibmpkvm_lv_swap
[12:41] <smoser> so thats where naems come from
[12:41] <strikov> smoser: you don't see names because kpartx failed I think
[12:42] <smoser> you think it failed on powerKVM ?
[12:43] <smoser> strikov, see : /usr/share/doc/multipath-tools/examples/multipath.conf.synthetic:#
[12:43] <smoser> getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
[12:44] <strikov> smoser: okay, so i reproed inside vm: http://pastebin.ubuntu.com/11564961/
[12:45] <strikov> smoser: yeah, i just can't understand why default config doesn't work if disk names *usually* contain spaces
[12:45] <strikov> smoser: i thought that we met some corner case
[12:46] <strikov> smoser: but that's not the case if vm disks looks the same
[12:46] <strikov> smoser: i'm trying to understand where dmsetup get this name
[12:51] <smoser> strikov, i think it gets name from rules/60-persistent-storage.rules
[12:51] <smoser> i think
[12:51] <smoser> hm... well, at least that calls scsi_id
[12:52] <strikov> smoser: i think that dmsetup doesn't use scsi_id and that's the problem
[12:53] <smoser> strikov maybe you're right. /lib/udev/rules.d/55-dm.rules
[12:53] <smoser> uses SYMLINK+=mapper/..... DM_NAME
[12:54] <strikov> smoser: interesting: http://pastebin.ubuntu.com/11565129/
[12:54] <smoser> hm. i dont know
[12:55] <smoser> right.
[12:55] <smoser> thats why my hack worked
[12:55] <smoser> (when i replaced spaces with the \x20
[12:55] <smoser> not sure what is creating either of those though
[12:57] <strikov> smoser: http://pastebin.ubuntu.com/11565218/
[12:57] <strikov> that's how udev gets the name, and name seems to be ok
[12:59] <smoser> hm.
[12:59] <strikov> smoser: alternative route (see udev rule) returns incorrect name though: http://pastebin.ubuntu.com/11565314/
[12:59] <smoser> so something is creating both
[12:59] <smoser>   /dev/mapper/1IBM     IPR-0   5EC2A900000000A0
[12:59] <smoser> and
[13:00] <smoser>   /dev/mapper/1IBM\x20\x20\x20\x20\x20IPR-0\x20\x20\x205EC2A900000000A0
[13:00] <strikov> smoser: I think that path with \x20 comes from udev which gets it from kernel
[13:00] <strikov> smoser: but i have no idea who creates first path with spaces
[13:01] <smoser> ah. and the second is the symlink to ../dm-X . and the first is an actual block device (with same major minor as ../dm-X)
[13:03] <smoser> https://lists.ubuntu.com/archives/foundations-bugs/2015-April/233983.html
[13:03] <smoser> awesome.
[13:03] <smoser> someone solved this for us 3 weeks ago
[13:03] <smoser> :)
[13:03] <smoser> https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1432062
[13:07] <strikov> smoser: well, that's awesome
[13:09] <strikov> smoser: let me try it inside a vm
[13:11] <strikov> smoser: problem here is that we need to backport it everywhere i think
[13:12] <strikov> smoser: or generate config with user_friendly_names if that really helps
[13:13] <smoser> strikov, right. we would probably want to backport to trusty.
[13:14] <smoser> but the bug reporter says that the install (d-i probably) installs 'user_friendly_names'
[13:14] <smoser> so that would seem also acceptable
[13:14] <smoser> but. would like to generally fix this also.
[13:15] <strikov> smoser: http://pastebin.ubuntu.com/11565504/
[13:15] <strikov> smoser: user friendly fixes the issue
[13:15] <strikov> smoser: maybe we can generate this trivial .conf in curtin?
[13:16] <strikov> smoser: with user friendly it uses mpathN instead of disk name which fixes the issue
[13:32] <smoser> strikov, sweet. i'm testing user friendly names rightnow, and will update that bug.
[13:32] <smoser> reboot in progress.
[13:38] <smoser> strikov, worked!
[13:38] <smoser> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371634/comments/17
[13:40] <smoser> strikov, ... but this doesnt give you warm fuzzies
[13:40] <smoser> http://paste.ubuntu.com/11565939/
[13:41] <smoser> smb, ^
[13:42] <smoser> so.. installed a system with curtin
[13:42] <smoser> followed https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371634/comments/17
[13:42] <smoser> reboot
[13:42] <smoser> scary messages about ext4 in mdesg
[13:44] <strikov> smoser: did it boot, i mean do you have something working?
[13:44] <smoser> yeah. it worked
[13:44] <smoser> but then scary ext4 messages :)
[13:45] <smoser> those scary messages appear on both systems that i tried it on
[13:45] <strikov> smoser: do they block everything or you can continue doing something in bash
[13:46] <strikov> smoser: trying to repro in the vm; my multipath disk is not used as / so i could miss it
[13:48] <smoser> https://bugzilla.kernel.org/show_bug.cgi?id=42723
[13:49] <smoser> strikov, it all "just works" with those steps
[13:49] <smoser> (other than the fact that you get filesystem errors :)
[13:50] <smoser> strikov, can you write somewhere how you fixed ?
[13:50] <smoser> er... how you made kvm do multipath
[13:51] <strikov> smoser: http://pastebin.ubuntu.com/11566036/
[13:52] <strikov> smoser: (1) that's important to put same serial numbers for both disks and (2) scsi-virtio controller needs to be added
[13:52] <strikov> smoser: that's basically it
[13:53] <bitfury> good morning all
[13:54] <strikov> smoser: i used cloud image as mail disk and userdata was generated with cloud-localds
[13:55] <smoser> strikov, would be nice to make 'launch' do that :)
[13:55] <smoser> do/support
[13:58] <strikov> smoser: yeah, i just didn't look into xkvm stuff so not sure if it's simple to do it with it or not
[14:00] <strikov> smoser: i just run bonnie++ fs benchmark on mounted multipath partition, don't see any ext4 issues
[14:02] <smoser> strikov, well i fyou grab the kvm command that should give you most of what you need to know.
[14:02] <strikov> smoser: hm, do you think that installing multipath-tools-boot (not multipath-tools) is enough for curtin
[14:02] <smoser> -tools-boot depends on -tools
[14:02] <smoser> so yeah
[14:03] <smoser> so with xkvm... if you added '--multipath-disk=disk1.img' and it did that magic, that'd be cool.
[14:05] <smoser> strikov, but maybe you shouldnt spend time on that.
[14:05] <smoser> but work on the other curtin things. thats probably better idea
[14:06] <strikov> smoser: well, we need implement multipath in curtin :) i'm working on mp now
[14:07] <rbasak_> Hi
[14:08] <rbasak_> I'm on 3G.
[14:08] <rbasak_> Sorry for the lack of notice. I wasn't expecting four electricians to turn up in my kitchen all at once.
[14:08] <rbasak_> They've had to turn off the entire house power because it seems the kitchen didn't have a supply of its own and was just leeching from other bits of the house. So I have no working sockets right now :-(
[14:09] <rbasak_> Oh, wrong channel.
[14:09] <rbasak_> gaughen: ^^
[14:09] <gaughen> :-)
[14:16] <smoser> strikov, thats true.
[14:16] <smoser> multipath in curtin and a way to test it via 'launch' would be nice.
[14:17] <strikov> smoser: do you see any simple way to try modified curtin with your machine w/ ,p?
[14:17] <strikov> *w/ mp
[14:19] <strikov> smoser: are you sure that 'sudo update-initramfs -u -k all' is really needed?
[14:20] <strikov> smoser: i suspect that multpath-tools-boot should do that internally
[14:20] <smoser> strikov, but i write the file after i install the packkage
[14:22] <smoser> that was just to make sure our version of the file was installed.
[14:23] <strikov> smoser: oh, i see your point, i forgot that multipath.conf needs to be included into initramfs
[14:59] <smoser> strikov, its possible that system has some bad disk i think
[15:01] <smoser> http://paste.ubuntu.com/11567312/
[15:03] <strikov> smoser: yeah, i suspect that's not multipath-related
[15:03] <strikov> smoser: we may want to ask #hs about it
[15:29] <strikov> smoser: http://bazaar.launchpad.net/~strikov/curtin/multipath-2/revision/210
[15:44] <squisher> rbasak, heh, good luck with the electricians! Ping on the bcache pkg-ing... I think a glance should be good enough, then we should just upload it and if there are issues people will hopefully complain :D
[15:57] <Voyage> How much downtime is expected if I have hosting setup right but I transfer my domain registrar from one registrar  to another
[15:58] <squisher> Voyage, totally OT, but you shouldn't expect any downtime if things go right
[15:59] <Voyage> OT?
[16:12] <filipoi> Hi there, I have a problem with setting up nginx in lxc. It doesnt want to serve any files. After fresh install when I check my domain in a browser I got default page – ok. But when trying to change root directory I've got blank page with "ERR_CONNECTION_REFUSED" message. I thougt that I didnt properly fowarded port 80 to lxc – but no I could run python simple HTTP server on port 80 and everything
[16:12] <filipoi> works correctly. Also I thought that maybe it has something to do with permission on my root dir – but I think no – I change nginx user to root user and nothing changed – still I cannot serve directory that I pointed in my server block.
[16:12] <filipoi> Any ideas?
[16:13] <filipoi> I'm a begginer so I probably made some stupid mistake but I dont know where...
[16:22] <strikov> filipoi: do you see anything suspicious in nginx's logs?
[16:24] <filipoi> strikov: in error.log Ive got only one line
[16:24] <filipoi>  [alert] 10198#0: unlink() "/run/nginx.pid" failed (2: No such file or directory)
[16:24] <teward> there's your problem
[16:24] <teward> nginx can't find the pid :P
[16:24] <filipoi> aha
[16:24] <teward> (which is bad)
[16:24] <filipoi> I guess ;)
[16:25] <filipoi> how can I fix it?
[16:25] <teward> filipoi: "Connection Refused" is the system's way of saying that you either got a "REJECT" packet or the connection attempt was blocked because no service was listening.
[16:25] <teward> run `pidof nginx` first to see if it's still 'running'
[16:26] <teward> if it is, `sudo killall nginx` then try and reload the nginx service
[16:26] <filipoi> yaeh the output is " 11032 11031 11030 11029 11028 "
[16:26] <teward> filipoi: start by terminating the existing nginx processes, first with `sudo service nginx stop`
[16:26] <teward> if thta fails kill them with the prior command
[16:26] <teward> `service nginx start` then
[16:27] <teward> filipoi: stupid question but you DID refresh the configuration right, when you edited the config?
[16:27] <filipoi> tewardtewar: I think so
[16:27] <filipoi> I mean I restarted nginx
[16:27] <teward> well killing nginx and reopening it will refresh the config
[16:27] <filipoi> and reloaded
[16:27] <teward> filipoi: i think you restarted it but it failed to load given the 'failed to find pid' issue and "Connection Refused"
[16:28] <filipoi> oh I see
[16:28] <filipoi> but why it failed?
[16:28] <teward> no idea, it'd require a lot more debugging than I cna throw at it right now
[16:28] <teward> also not sure why you're running it in a container, it sounds like it's a hassle
[16:29] <filipoi> heh somebody told me that it is a good way ;)
[16:29] <filipoi> anyway it is just for learning pupropses
[16:29] <filipoi> I dont have any real website to serve
[16:30] <squisher> filipoi, if you're learning, then it's probably a good idea to get familiar with the software itself first, and only then run it in a container afterwards
[16:30] <teward> ^ that
[16:30] <teward> me, I just VM everything, but i have a large VM cluster for that purpose
[16:30] <filipoi> yeah – but I had nginx running on the host before
[16:31] <filipoi> now I wanted to try soemthing different
[16:34] <filipoi> ok I gotta go – thanks for your time.
[16:35] <teward> well i've had bad luck with lxc containers anyways, hence why I invested in a beefy VM host server xD
[16:35] <teward> and no problem
[16:37] <squisher> If I was to experiment with containers it'd probably be docker
[16:37] <teward> what squisher said xD
[16:38] <teward> although i prefer VMs since i have many vlans for different network access levels xD
[19:00] <Kully3xf> what's up guys/gals. Trying to write a loop that will inject a shell script over ssh taking in IP's from a list
[19:07] <bekks> So do it.
[19:28] <bitfury> that's strange, trying to sync clocks against NIST NTP servers but iptables is blocking it even though I have a proper rule for allowing returning traffic (established, related)
[19:29] <bitfury> http://is.gd/Wgweju
[19:29] <patdk-wk> that only works if your using conntrack
[19:30] <bitfury> paste of iptable rules, ntpq -p returns timed out, nothing received
[19:30] <bitfury> :\
[19:30] <patdk-wk> I wouldn't use nist ever though
[19:30] <patdk-wk> those timeservers are so overloaded, and they bandwidth limit
[19:30] <bitfury> what's onntrack?
[19:30] <patdk-wk> just use pool.ntp.org
[19:30] <bitfury> c
[19:30] <Kully3xf> what does ssh -tt do
[19:31] <bitfury> patdk-wk: we're required to use NIST (compliance of some sort)
[19:31] <patdk-wk> ok
[19:31] <patdk-wk> test with pool.ntp.org to VERIFY it's not a nist issue
[19:31] <bitfury> lets see
[19:33] <patdk-wk> what is your ip?
[19:33] <bitfury> hmm
[19:33] <bitfury> same with pool.ntp.org
[19:34] <bitfury> what ip
[19:34] <patdk-wk> that your traffic is coming from
[19:35] <Sprocks> i seem to be having a problem mounting a cifs share in fstab, when i use the same info that works on another ubuntu i get mount error(95): Operation not supported Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) but it doesnt say what is wrong. If i use sudo mount with the same info it mounts fine
[19:35] <sarnold> Sprocks: is there anything in dmesg?
[19:36] <Sprocks> how do i check?
[19:36] <bitfury> patdk-wk: not sure I would like to disclose it here
[19:36] <patdk-wk> hmm?
[19:36] <patdk-wk> your public ip is so private your not allowed to use it on the internet?
[19:36] <sarnold> Sprocks: dmesg | tail
[19:36] <patdk-wk> sorry, but heh, not sure how to help anymore
[19:37] <Sprocks> sarnold: cifs_mount failed w/return code = -95
[19:37] <bitfury> patdk-wk: not about being private, that's like asking someone I just met for their home address
[19:38] <patdk-wk> no it's not
[19:38] <bitfury> I think it is
[19:38] <Sprocks> sarnold: also says Server requires packet signing to be enabled in /proc/fs/cifs/SecurityFlags
[19:38] <quantic> well, it's not
[19:38] <patdk-wk> ya, cause if I know your ip address, I know where you are and can kill you in your sleep
[19:39] <patdk-wk> networks use ip addresses
[19:39] <patdk-wk> without ip addresses diagnostics of network issues are impossible
[19:39] <patdk-wk> sorry,  Ican't help you
[19:39] <bitfury> ok thanks
[19:40] <sarnold> bitfury: turn off iptables entirely and test?
[19:40] <bitfury> hmm lets give that a shot
[19:41] <sarnold> Sprocks: hmm never heard of that; maybe try to modprobe cifs manually?
[19:43] <Sprocks> sarnold: to turn off iptable i use ufw disable right?
[19:43] <bitfury> sarnold: that did it..
[19:43] <sarnold> Sprocks: I think so
[19:44] <bitfury> but my first rule should allow return traffic though..
[19:44] <sarnold> bitfury: hah, sorry, now you have to figure out the rules :)
[19:44] <bitfury> heh, yep
[19:45] <Sprocks> sarnold: like i said is i use mount -t cifs it works but info in /etc/fstab doesnt
[19:46] <Sprocks> sarnold: i used this guide to get it to work on the one machine https://wiki.ubuntu.com/MountWindowsSharesPermanently#Mount_password_protected_network_folders
[19:49] <sarnold> Sprocks: interesting; that guide recommends putting the credentials in your home directory. that seems odd. try putting the credentials in /etc somewhere, set file owner and group to root, and then adjust /etc/fstab for the new name..
[19:50] <Sprocks> sarnold: i thought that was wierd also but it worked great on the one machine with an identical setup as far as im aware
[19:54] <Sprocks> sarnold: moved it to /etc and still same result "operation not supported"
[19:56] <sarnold> Sprocks: oh well, that was a long shot :/ I've got to run; pastebin your fstab and maybe someonme will figurfe it out before Ig et back
[20:07] <Sprocks> ok so to continue the testing process i removed the uid arg and now i get mount error(79): Can not access a needed shared library
[20:35] <Sprocks> im getting mount error(13): Permission denied when trying to mount a cifs share in fstab is someone able to help?
[20:44] <Sprocks> ok i fixed it now, there was a hidden character in my credentials file somehow. sarnold the issue i had before seemed to be the sec=ntlm part but since ntlm is default i guess it doesnt matter that its not there
[21:53] <sarnold> Sprocks: hah, that's a crazy path to get there.. nice debugging :)
[23:15] <trippeh> Hmmm. I cant control systemd-timesyncd without dbus?
[23:16] <patdk-lap> you can't use systemd without dbus
[23:16] <trippeh> patdk-lap: pretty sure I am.
[23:16] <patdk-lap> ok, your limiting your systemd usage :)
[23:16] <trippeh> I'm running ubuntu core, it has systemd and no dbus running.
[23:17] <patdk-lap> hmm
[23:17] <patdk-lap> I thought dbus was a goal of systemd
[23:17] <trippeh> udev is active tho, of course ;-)
[23:18] <trippeh> seems most parts of systemd works fine without dbus, at least the stuff needed for servers.
[23:18] <trippeh> except controlling timesyncd ;)
[23:19] <trippeh> I really need to take a long and hard think about if I want to have dbus on my servers.
[23:19] <ObrienDave> decisions, decisions
[23:23] <trippeh> I have until 16.04 LTS to decide ;)
[23:24] <sarnold> please file bugs :)
[23:24] <trippeh> maybe I dont really need timesyncd *control*. would be nice for debugging the thing tho.
[23:26] <sarnold> it's hard to imagine a timesync thing not needing debugging or control once in a while, and I completely understand the deisre to keep dbus off if you can..
[23:30] <trippeh> easy way out is just to run ntpd, but the deb/ubu packaged version is too old for the client-only mode
[23:30] <trippeh> which means I have to firewall the thing, and using conntrack and stuff (thanks, UDP)
[23:39] <trippeh> I suppose I could do it statelessly if using just a few known ntp servers. not perfect, but what is
[23:57] <Scott_S> Hey there! Does anyone here have any idea why Postfix would be rejecting port 25 connections from outside of localhost? inet_interfaces is set to all, but it still won't accept the connection.
[23:58] <sarnold> Scott_S: is there anything in the postfix logs? has that port been filtered by your firewall? has that port been filtered by your ISP?
[23:58] <Scott_S> To verify, is that mail.log and mail.err? Sorry, I don't have any amazing amount of knowledge about POstfix.
[23:59] <sarnold> me neither :) both those are good places to start, anyway..
[23:59] <Scott_S> Neither of them has anything in them, unfortunately.
[23:59] <sarnold> dang. check postconf output for what looks like log files..
[23:59] <trippeh> Port 25 is often filtered if you are testing from a residential connection