/srv/irclogs.ubuntu.com/2015/06/04/#ubuntu-server.txt

=== markthomas is now known as markthomas|away
=== ideopathic_ is now known as ideopathic
RadarWe had a little over an hour of downtime this morning. What happened was that MySQL was killed due to out of memory, and then apparmor denied MySQL from starting up again. How can we investigate why this issue occurred?01:26
RadarParticularly why apparmor blocked MySQL01:26
sarnoldRadar: dmesg | grep DEN ought to show you the denials; you can add the right privileges to allow them to the /etc/apparmor.d/usr.sbin.mysqld file -- and if they aren't specific to your site, you could file a bug so we include them for everyone else01:28
Radarsarnold: This hasn't been an issue before today. From what I can see in the usr.sbin.mysqld file, the permissions are correct.01:28
RadarI can show you the file if you'd like.01:28
sarnoldsure01:28
Radarhttps://gist.github.com/radar/991f13af7859cdb6964601:29
Radarthe /var/lib/mysql path is symlinked to /srv/mysql01:30
RadarI don't know what the "k" means here: https://gist.github.com/radar/991f13af7859cdb69646#file-mysqld-L3301:32
RadarLooks like that it has been configured for access to those directories.01:32
sarnoldRadar: aha, probably that symlink...01:33
RadarIt's been symlinked for ages.01:33
RadarWe have an identically configured server where this is not an issue.01:34
sarnoldRadar: add a rule like "alias /var/lib/mysql/ -> /srv/mysql/" to /etc/apparmor.d/tunables/alias01:34
Radarargh, our other server doesn't use a symlink!01:35
jjohansenRadar, sarnold: if you suspect any of apparmor's trusted helpers might be involved then greping /var/log/syslog is better, as they are logging there01:35
jjohanseneg. dbus01:35
RadarMy apologies, I thought they were configured the same.01:35
sarnoldjjohansen: gah thanks for the reminder. I'm still not used to this new world. :)01:35
sarnoldRadar: ahhhh :)01:36
sarnoldRadar: once you add the alias rule, you'll want to run apparmor_parser --replace /etc/apparmor.d/usr.sbin.mysqld01:36
RadarThanks for the tip sarnold :)01:37
sarnoldprobably /etc/init.d/apparmor reload would do the same job...01:37
sarnoldRadar: the 'k' means the process can issue fctnl locks on the file; some days I wonder if it's worth actually mediating those, but we do. :)01:37
patdk-lapI do01:47
=== NomadJim_ is now known as NomadJim
=== Lcawte|Away is now known as Lcawte
=== cipi is now known as CiPi
lordievaderGood morning.08:07
solo1how can i configure mdadm in a server raid1 ... it was with only 1 hd and i want to get raid1 using it ... i ve no idea how can i set ... 'due i found a lot of tutorial saying i must decide it when both are void or blank during server installation .... pls help08:55
lordievadersolo1: https://www.howtoforge.com/how-to-set-up-software-raid1-on-a-running-system-incl-grub2-configuration-debian-squeeze-p208:57
solo1to lordievader: tnx dude08:59
solo1to lordievader i got only 2 hdd and the guide speaks about 309:13
lordievadersolo1: Change it to your situation.09:17
histosolo1: how are you going to use 1 hd and do raid110:03
lordievaderhisto: "i got only 2 hdd" Seems to me like he has two.10:05
histolordievader: didn't get that far.  His first comment was stated weird.10:05
=== markthomas|away is now known as markthomas
=== masACC is now known as maswan
IronDevGuten Tag11:44
smoserstrikov, did you make any headway on bug 137163412:36
strikovsmoser: i was able to create kvm instance with scsi multipath12:37
smoseroh nice.12:37
strikovsmoser: playing with it now12:37
strikovsmoser: funny thing is that even qemu-emulated multipath contains spaces in disk name12:37
smoserthat is funny.12:37
smosermaybe user_friendly_names is just what we need ?12:38
strikovsmoser: i suspect that issue might be in a different place12:38
smoseri reproduced the issue using kpartx on a regular device12:38
smosersee the bug12:38
strikovsmoser: yes, that's the first thought I have, thats quite strange that it is not default then12:38
smoseri think that maybe kpartx to do that needs to replace ' ' with '\x20' or what not. maaybe the kernel juts expectst that and kpartx isnt doing it.12:38
strikovsmoser: i think that (a) kpartx indeed has a bug but (b) upstream didn't fix it because you usually don't get such an input12:39
strikovsmoser: example12:39
smosermaybe12:39
strikovsmoser: http://pastebin.ubuntu.com/11564916/12:39
strikovsmoser: tool which gets disk serial has a way to eliminate spaces12:39
strikovsmoser: why not use it this way to generate udev name w/o spaces?12:40
smoserhttp://paste.ubuntu.com/11564919/12:40
smoserthta is from powerKVM12:40
smoseron a similar system, and yeah, they don thave the funny names there.12:41
smosercmdline there has: $ cat /proc/cmdline12:41
smoserroot=UUID=6ef1254c-50ac-421a-bb4a-25619189b327 ro console=tty0 console=hvc0 crashkernel=1024M rd.lvm.lv=ibmpkvm_vg_data/ibmpkvm_lv_data rd.lvm.lv=ibmpkvm_vg_log/ibmpkvm_lv_log rd.lvm.lv=ibmpkvm_vg_root/ibmpkvm_lv_root rd.lvm.lv=ibmpkvm_vg_swap/ibmpkvm_lv_swap12:41
smoserso thats where naems come from12:41
strikovsmoser: you don't see names because kpartx failed I think12:41
smoseryou think it failed on powerKVM ?12:42
smoserstrikov, see : /usr/share/doc/multipath-tools/examples/multipath.conf.synthetic:#12:43
smosergetuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"12:43
strikovsmoser: okay, so i reproed inside vm: http://pastebin.ubuntu.com/11564961/12:44
strikovsmoser: yeah, i just can't understand why default config doesn't work if disk names *usually* contain spaces12:45
strikovsmoser: i thought that we met some corner case12:45
strikovsmoser: but that's not the case if vm disks looks the same12:46
strikovsmoser: i'm trying to understand where dmsetup get this name12:46
smoserstrikov, i think it gets name from rules/60-persistent-storage.rules12:51
smoseri think12:51
smoserhm... well, at least that calls scsi_id12:51
strikovsmoser: i think that dmsetup doesn't use scsi_id and that's the problem12:52
smoserstrikov maybe you're right. /lib/udev/rules.d/55-dm.rules12:53
smoseruses SYMLINK+=mapper/..... DM_NAME12:53
strikovsmoser: interesting: http://pastebin.ubuntu.com/11565129/12:54
smoserhm. i dont know12:54
smoserright.12:55
smoserthats why my hack worked12:55
smoser(when i replaced spaces with the \x2012:55
smosernot sure what is creating either of those though12:55
strikovsmoser: http://pastebin.ubuntu.com/11565218/12:57
strikovthat's how udev gets the name, and name seems to be ok12:57
smoserhm.12:59
strikovsmoser: alternative route (see udev rule) returns incorrect name though: http://pastebin.ubuntu.com/11565314/12:59
smoserso something is creating both12:59
smoser  /dev/mapper/1IBM     IPR-0   5EC2A900000000A012:59
smoserand12:59
smoser  /dev/mapper/1IBM\x20\x20\x20\x20\x20IPR-0\x20\x20\x205EC2A900000000A013:00
strikovsmoser: I think that path with \x20 comes from udev which gets it from kernel13:00
strikovsmoser: but i have no idea who creates first path with spaces13:00
smoserah. and the second is the symlink to ../dm-X . and the first is an actual block device (with same major minor as ../dm-X)13:01
smoserhttps://lists.ubuntu.com/archives/foundations-bugs/2015-April/233983.html13:03
smoserawesome.13:03
smosersomeone solved this for us 3 weeks ago13:03
smoser:)13:03
smoserhttps://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/143206213:03
strikovsmoser: well, that's awesome13:07
strikovsmoser: let me try it inside a vm13:09
=== maxb_ is now known as maxb
strikovsmoser: problem here is that we need to backport it everywhere i think13:11
strikovsmoser: or generate config with user_friendly_names if that really helps13:12
smoserstrikov, right. we would probably want to backport to trusty.13:13
smoserbut the bug reporter says that the install (d-i probably) installs 'user_friendly_names'13:14
smoserso that would seem also acceptable13:14
smoserbut. would like to generally fix this also.13:14
strikovsmoser: http://pastebin.ubuntu.com/11565504/13:15
strikovsmoser: user friendly fixes the issue13:15
strikovsmoser: maybe we can generate this trivial .conf in curtin?13:15
strikovsmoser: with user friendly it uses mpathN instead of disk name which fixes the issue13:16
smoserstrikov, sweet. i'm testing user friendly names rightnow, and will update that bug.13:32
smoserreboot in progress.13:32
smoserstrikov, worked!13:38
smoserhttps://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371634/comments/1713:38
smoserstrikov, ... but this doesnt give you warm fuzzies13:40
smoserhttp://paste.ubuntu.com/11565939/13:40
smosersmb, ^13:41
smoserso.. installed a system with curtin13:42
smoserfollowed https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1371634/comments/1713:42
smoserreboot13:42
smoserscary messages about ext4 in mdesg13:42
strikovsmoser: did it boot, i mean do you have something working?13:44
smoseryeah. it worked13:44
smoserbut then scary ext4 messages :)13:44
smoserthose scary messages appear on both systems that i tried it on13:45
strikovsmoser: do they block everything or you can continue doing something in bash13:45
strikovsmoser: trying to repro in the vm; my multipath disk is not used as / so i could miss it13:46
smoserhttps://bugzilla.kernel.org/show_bug.cgi?id=4272313:48
smoserstrikov, it all "just works" with those steps13:49
smoser(other than the fact that you get filesystem errors :)13:49
smoserstrikov, can you write somewhere how you fixed ?13:50
smoserer... how you made kvm do multipath13:50
strikovsmoser: http://pastebin.ubuntu.com/11566036/13:51
strikovsmoser: (1) that's important to put same serial numbers for both disks and (2) scsi-virtio controller needs to be added13:52
strikovsmoser: that's basically it13:52
bitfurygood morning all13:53
strikovsmoser: i used cloud image as mail disk and userdata was generated with cloud-localds13:54
smoserstrikov, would be nice to make 'launch' do that :)13:55
smoserdo/support13:55
strikovsmoser: yeah, i just didn't look into xkvm stuff so not sure if it's simple to do it with it or not13:58
strikovsmoser: i just run bonnie++ fs benchmark on mounted multipath partition, don't see any ext4 issues14:00
smoserstrikov, well i fyou grab the kvm command that should give you most of what you need to know.14:02
strikovsmoser: hm, do you think that installing multipath-tools-boot (not multipath-tools) is enough for curtin14:02
smoser-tools-boot depends on -tools14:02
smoserso yeah14:02
smoserso with xkvm... if you added '--multipath-disk=disk1.img' and it did that magic, that'd be cool.14:03
smoserstrikov, but maybe you shouldnt spend time on that.14:05
smoserbut work on the other curtin things. thats probably better idea14:05
strikovsmoser: well, we need implement multipath in curtin :) i'm working on mp now14:06
rbasak_Hi14:07
rbasak_I'm on 3G.14:08
rbasak_Sorry for the lack of notice. I wasn't expecting four electricians to turn up in my kitchen all at once.14:08
rbasak_They've had to turn off the entire house power because it seems the kitchen didn't have a supply of its own and was just leeching from other bits of the house. So I have no working sockets right now :-(14:08
rbasak_Oh, wrong channel.14:09
rbasak_gaughen: ^^14:09
gaughen:-)14:09
smoserstrikov, thats true.14:16
smosermultipath in curtin and a way to test it via 'launch' would be nice.14:16
strikovsmoser: do you see any simple way to try modified curtin with your machine w/ ,p?14:17
strikov*w/ mp14:17
strikovsmoser: are you sure that 'sudo update-initramfs -u -k all' is really needed?14:19
strikovsmoser: i suspect that multpath-tools-boot should do that internally14:20
smoserstrikov, but i write the file after i install the packkage14:20
smoserthat was just to make sure our version of the file was installed.14:22
strikovsmoser: oh, i see your point, i forgot that multipath.conf needs to be included into initramfs14:23
=== DenBeiren is now known as zz_DenBeiren
=== ashleyd is now known as ashd
smoserstrikov, its possible that system has some bad disk i think14:59
smoserhttp://paste.ubuntu.com/11567312/15:01
strikovsmoser: yeah, i suspect that's not multipath-related15:03
strikovsmoser: we may want to ask #hs about it15:03
strikovsmoser: http://bazaar.launchpad.net/~strikov/curtin/multipath-2/revision/21015:29
squisherrbasak, heh, good luck with the electricians! Ping on the bcache pkg-ing... I think a glance should be good enough, then we should just upload it and if there are issues people will hopefully complain :D15:44
VoyageHow much downtime is expected if I have hosting setup right but I transfer my domain registrar from one registrar  to another15:57
squisherVoyage, totally OT, but you shouldn't expect any downtime if things go right15:58
VoyageOT?15:59
=== Lcawte is now known as Lcawte|Away
filipoiHi there, I have a problem with setting up nginx in lxc. It doesnt want to serve any files. After fresh install when I check my domain in a browser I got default page – ok. But when trying to change root directory I've got blank page with "ERR_CONNECTION_REFUSED" message. I thougt that I didnt properly fowarded port 80 to lxc – but no I could run python simple HTTP server on port 80 and everything16:12
filipoiworks correctly. Also I thought that maybe it has something to do with permission on my root dir – but I think no – I change nginx user to root user and nothing changed – still I cannot serve directory that I pointed in my server block.16:12
filipoiAny ideas?16:12
filipoiI'm a begginer so I probably made some stupid mistake but I dont know where...16:13
strikovfilipoi: do you see anything suspicious in nginx's logs?16:22
=== Lcawte|Away is now known as Lcawte
filipoistrikov: in error.log Ive got only one line16:24
filipoi [alert] 10198#0: unlink() "/run/nginx.pid" failed (2: No such file or directory)16:24
tewardthere's your problem16:24
tewardnginx can't find the pid :P16:24
filipoiaha16:24
teward(which is bad)16:24
filipoiI guess ;)16:24
filipoihow can I fix it?16:25
tewardfilipoi: "Connection Refused" is the system's way of saying that you either got a "REJECT" packet or the connection attempt was blocked because no service was listening.16:25
tewardrun `pidof nginx` first to see if it's still 'running'16:25
tewardif it is, `sudo killall nginx` then try and reload the nginx service16:26
filipoiyaeh the output is " 11032 11031 11030 11029 11028 "16:26
tewardfilipoi: start by terminating the existing nginx processes, first with `sudo service nginx stop`16:26
tewardif thta fails kill them with the prior command16:26
teward`service nginx start` then16:26
tewardfilipoi: stupid question but you DID refresh the configuration right, when you edited the config?16:27
filipoitewardtewar: I think so16:27
filipoiI mean I restarted nginx16:27
tewardwell killing nginx and reopening it will refresh the config16:27
filipoiand reloaded16:27
tewardfilipoi: i think you restarted it but it failed to load given the 'failed to find pid' issue and "Connection Refused"16:27
filipoioh I see16:28
filipoibut why it failed?16:28
tewardno idea, it'd require a lot more debugging than I cna throw at it right now16:28
tewardalso not sure why you're running it in a container, it sounds like it's a hassle16:28
filipoiheh somebody told me that it is a good way ;)16:29
filipoianyway it is just for learning pupropses16:29
filipoiI dont have any real website to serve16:29
squisherfilipoi, if you're learning, then it's probably a good idea to get familiar with the software itself first, and only then run it in a container afterwards16:30
teward^ that16:30
tewardme, I just VM everything, but i have a large VM cluster for that purpose16:30
filipoiyeah – but I had nginx running on the host before16:30
filipoinow I wanted to try soemthing different16:31
filipoiok I gotta go – thanks for your time.16:34
tewardwell i've had bad luck with lxc containers anyways, hence why I invested in a beefy VM host server xD16:35
tewardand no problem16:35
squisherIf I was to experiment with containers it'd probably be docker16:37
tewardwhat squisher said xD16:37
tewardalthough i prefer VMs since i have many vlans for different network access levels xD16:38
=== Voyage_ is now known as Voyage
=== NomadJim_ is now known as NomadJim
=== markthomas is now known as markthomas|away
Kully3xfwhat's up guys/gals. Trying to write a loop that will inject a shell script over ssh taking in IP's from a list19:00
=== markthomas|away is now known as markthomas
bekksSo do it.19:07
bitfurythat's strange, trying to sync clocks against NIST NTP servers but iptables is blocking it even though I have a proper rule for allowing returning traffic (established, related)19:28
bitfuryhttp://is.gd/Wgweju19:29
patdk-wkthat only works if your using conntrack19:29
bitfurypaste of iptable rules, ntpq -p returns timed out, nothing received19:30
bitfury:\19:30
patdk-wkI wouldn't use nist ever though19:30
patdk-wkthose timeservers are so overloaded, and they bandwidth limit19:30
bitfurywhat's onntrack?19:30
patdk-wkjust use pool.ntp.org19:30
bitfuryc19:30
Kully3xfwhat does ssh -tt do19:30
bitfurypatdk-wk: we're required to use NIST (compliance of some sort)19:31
patdk-wkok19:31
patdk-wktest with pool.ntp.org to VERIFY it's not a nist issue19:31
bitfurylets see19:31
patdk-wkwhat is your ip?19:33
bitfuryhmm19:33
bitfurysame with pool.ntp.org19:33
bitfurywhat ip19:34
patdk-wkthat your traffic is coming from19:34
Sprocksi seem to be having a problem mounting a cifs share in fstab, when i use the same info that works on another ubuntu i get mount error(95): Operation not supported Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) but it doesnt say what is wrong. If i use sudo mount with the same info it mounts fine19:35
sarnoldSprocks: is there anything in dmesg?19:35
Sprockshow do i check?19:36
bitfurypatdk-wk: not sure I would like to disclose it here19:36
patdk-wkhmm?19:36
patdk-wkyour public ip is so private your not allowed to use it on the internet?19:36
sarnoldSprocks: dmesg | tail19:36
patdk-wksorry, but heh, not sure how to help anymore19:36
Sprockssarnold: cifs_mount failed w/return code = -9519:37
bitfurypatdk-wk: not about being private, that's like asking someone I just met for their home address19:37
patdk-wkno it's not19:38
bitfuryI think it is19:38
Sprockssarnold: also says Server requires packet signing to be enabled in /proc/fs/cifs/SecurityFlags19:38
quanticwell, it's not19:38
patdk-wkya, cause if I know your ip address, I know where you are and can kill you in your sleep19:38
patdk-wknetworks use ip addresses19:39
patdk-wkwithout ip addresses diagnostics of network issues are impossible19:39
patdk-wksorry,  Ican't help you19:39
bitfuryok thanks19:39
sarnoldbitfury: turn off iptables entirely and test?19:40
bitfuryhmm lets give that a shot19:40
sarnoldSprocks: hmm never heard of that; maybe try to modprobe cifs manually?19:41
Sprockssarnold: to turn off iptable i use ufw disable right?19:43
bitfurysarnold: that did it..19:43
sarnoldSprocks: I think so19:43
bitfurybut my first rule should allow return traffic though..19:44
sarnoldbitfury: hah, sorry, now you have to figure out the rules :)19:44
bitfuryheh, yep19:44
Sprockssarnold: like i said is i use mount -t cifs it works but info in /etc/fstab doesnt19:45
Sprockssarnold: i used this guide to get it to work on the one machine https://wiki.ubuntu.com/MountWindowsSharesPermanently#Mount_password_protected_network_folders19:46
sarnoldSprocks: interesting; that guide recommends putting the credentials in your home directory. that seems odd. try putting the credentials in /etc somewhere, set file owner and group to root, and then adjust /etc/fstab for the new name..19:49
Sprockssarnold: i thought that was wierd also but it worked great on the one machine with an identical setup as far as im aware19:50
Sprockssarnold: moved it to /etc and still same result "operation not supported"19:54
sarnoldSprocks: oh well, that was a long shot :/ I've got to run; pastebin your fstab and maybe someonme will figurfe it out before Ig et back19:56
Sprocksok so to continue the testing process i removed the uid arg and now i get mount error(79): Can not access a needed shared library20:07
Sprocksim getting mount error(13): Permission denied when trying to mount a cifs share in fstab is someone able to help?20:35
Sprocksok i fixed it now, there was a hidden character in my credentials file somehow. sarnold the issue i had before seemed to be the sec=ntlm part but since ntlm is default i guess it doesnt matter that its not there20:44
sarnoldSprocks: hah, that's a crazy path to get there.. nice debugging :)21:53
=== zz_DenBeiren is now known as DenBeiren
trippehHmmm. I cant control systemd-timesyncd without dbus?23:15
patdk-lapyou can't use systemd without dbus23:16
trippehpatdk-lap: pretty sure I am.23:16
patdk-lapok, your limiting your systemd usage :)23:16
trippehI'm running ubuntu core, it has systemd and no dbus running.23:16
patdk-laphmm23:17
patdk-lapI thought dbus was a goal of systemd23:17
trippehudev is active tho, of course ;-)23:17
trippehseems most parts of systemd works fine without dbus, at least the stuff needed for servers.23:18
trippehexcept controlling timesyncd ;)23:18
trippehI really need to take a long and hard think about if I want to have dbus on my servers.23:19
ObrienDavedecisions, decisions23:19
trippehI have until 16.04 LTS to decide ;)23:23
sarnoldplease file bugs :)23:24
trippehmaybe I dont really need timesyncd *control*. would be nice for debugging the thing tho.23:24
sarnoldit's hard to imagine a timesync thing not needing debugging or control once in a while, and I completely understand the deisre to keep dbus off if you can..23:26
trippeheasy way out is just to run ntpd, but the deb/ubu packaged version is too old for the client-only mode23:30
trippehwhich means I have to firewall the thing, and using conntrack and stuff (thanks, UDP)23:30
trippehI suppose I could do it statelessly if using just a few known ntp servers. not perfect, but what is23:39
Scott_SHey there! Does anyone here have any idea why Postfix would be rejecting port 25 connections from outside of localhost? inet_interfaces is set to all, but it still won't accept the connection.23:57
sarnoldScott_S: is there anything in the postfix logs? has that port been filtered by your firewall? has that port been filtered by your ISP?23:58
Scott_STo verify, is that mail.log and mail.err? Sorry, I don't have any amazing amount of knowledge about POstfix.23:58
sarnoldme neither :) both those are good places to start, anyway..23:59
Scott_SNeither of them has anything in them, unfortunately.23:59
sarnolddang. check postconf output for what looks like log files..23:59
trippehPort 25 is often filtered if you are testing from a residential connection23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!