[00:05] <rabbitdew> Does anyone know of a light-weight utility that would listen on a TCP port, so I could continually check connectivity from another server? My first thought was netcat but I don't like how the daemon reads anything from stdin and can't get it to sotp
[00:37] <rbasak> rabbitdew: inetd has built-in "discard" and "echo" servers. Are they any use to you?
[00:38] <cncr04s> use the ping command?
[00:38] <rbasak> Also "daytime" and "time".
[00:39] <rbasak> And they're all "standard" TCP protocols :-)
[01:12] <rabbitdew> rbasak: Wow, I've never heard of those, thanks for giving my something to look at. cncr04s: I'm trying to look out for what I believe is a network problem but not long ago one of the servers were totally borked and ping still worked for some reason
[01:17] <cncr04s> idk, just ssh into the server, if it fails, well, it fails.
[02:36] <DammitJim> so, I am definitely doing something wrong when I install Ubuntu 16.04.1 server and I am defining the logical volumes (instead of just the root default)
[02:36] <DammitJim> is there a good document I can follow to do this?
[02:38] <Ben64> really depends on what you're wanting
[02:42] <DammitJim> well, I went through the process of creating the volume group and then the logical volumes, but after an apt-get upgrade and a reboot, the system wouldn't find the volume group
[03:19] <DammitJim> can any of you test this?
[04:39] <DammitJim> ok, weirdest thing
[04:39] <DammitJim> when I create a logical volume for var during a ubuntu 16 install
[04:40] <DammitJim> after a do an apt-get upgrade and reboot, the system hangs during the shutdown
[04:40] <DammitJim> if I don't explicitly create a var logical volume, it doesn't have that problem!
[04:40] <DammitJim> could this be related to unattended upgrades being shut down?
[04:41] <PryMar56>  /var/run symlinked to /run
[04:42] <PryMar56>  the pid files are not found and nothing shuts down
[04:42] <PryMar56> chances are /var/run is full of orphan PID's
[04:58] <DammitJim> ugh
[05:03] <PryMar56> ls -al /var/run   /var/run -> ../run
[05:04] <DammitJim> what's that?
[08:57] <lordievader> Good morning.
[09:23] <nugget_> does anyone know the difference between "libnss-ldap" and "libnss-ldapd" and which is preferred?
[09:59] <nugget_> also i have read this:  https://wiki.debian.org/LDAP/NSS
[09:59] <nugget_> which does seem relevant
[10:00] <nugget_> but doesn't really explain the differences between the two in great detail
[13:47] <zul> coreycb: os-vif is in dep wait right now
[13:48] <coreycb> zul, ok
[14:55] <cpaelzer> stgraber: currently cleaning old bugs - wanted to ping on https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1549524
[14:55] <cpaelzer> stgraber: likely close-by-out-of-date, but I'd want tyou to decide
[16:23] <DammitJim> are you guys aware of any ubuntu 16.04.1 server hanging on shutdown for defined logical volume for /var ?
[16:23] <DammitJim> it's the strangest thing and I don't know how to troubleshoot it
[16:24] <lordievader> DammitJim: Ah, I wanted to ask you what you did after creating the lv.
[16:24] <lordievader> Since just creating the lv should change nothing.
[16:25] <DammitJim> lordievader, I actually just finished upgrading a server from 14.04.5 to 16.04.1 and the same behavior shows
[16:25] <DammitJim> the 14.04.5 server has been running for over a year!
[16:26] <DammitJim> what do you think I should do to provide more information and pin point what the root cause is?
[16:26] <DammitJim> logs don't say anything (I wanna say logging seems to be already disabled at this stage of the shutdown) but maybbe I'm wrong
[16:26] <beisner> coreycb, python-oslo.privsep promoted to uca newton-updates re: https://bugs.launchpad.net/bugs/1616240
[16:26] <lordievader> DammitJim: Please answer my question ;)
[16:27] <DammitJim> oh, what do you mean after I create the lv?
[16:27] <beisner> coreycb, and swift promoted to uca mitaka-proposed re: https://bugs.launchpad.net/swift/+bug/1642538
[16:27] <DammitJim> it let it finish creating the LVs and configure the hard drive
[16:27] <lordievader> DammitJim: Like I said, just creating a lv changes nothing.
[16:27] <DammitJim> mind you, I am doing this during install of Ubuntu 16
[16:27] <lordievader> DammitJim: Ah, I thought you had a running system already.
[16:27] <DammitJim> it finishes the installation and the server will reboot just fine many times
[16:28] <DammitJim> but then I do: apt-get update && sudo apt-get upgrade
[16:28] <DammitJim> and the thing will hang from that point on
[16:28] <lordievader> How is /var mounted then?
[16:28] <lordievader> What options?
[16:29] <DammitJim> lordievader, is that something I look up in fstab?
[16:30] <DammitJim> just like usr and home: /dev/mapper/template16--vg-usr /usr ext4 defaults 0 2
[16:52] <pmatulis> DammitJim, better to look at output to 'mount' command
[16:58] <DammitJim> where?
[17:10] <lordievader> Like pmatulis says, take the mount output.
[17:10] <pmatulis> DammitJim, command 'mount'
[17:10] <pmatulis> type 'mount' in a terminal
[19:01] <DammitJim> ok, running mount spits this out for var: /dev/mapper/template16--vg-var on /var type ext4 (rw,relatime,data=ordered)
[19:21] <lordievader> Seems rather normal.
[20:09] <DammitJim> I know!
[20:09] <DammitJim> I wonder if there is something else that is accessing the volume or whenever in the shutdown sequence, it unmounts it and now something else is looking for that resource?
[20:09] <DammitJim> I saw something about unattended-updates running
[20:14] <lordievader> In what state is the apt process when it hangs?
[20:14] <Seveas> a sad state
[20:18] <DammitJim> lordievader, how do I find that out?
[20:20] <lordievader> top, ps, something else.
[20:20] <DammitJim> lordievader, this hang happens when shutting down
[20:21] <lordievader> Or is the entire box frozen?
[20:21] <DammitJim> you know, when you see your screen starting to print: STOP this, STOP that, SHUTDOWN this, that
[20:21] <DammitJim> it's not frozen
[20:21] <DammitJim> it just hangs for like 8 minutes
[20:21] <lordievader> You were talking about apt hanging earlier...
[20:22] <DammitJim> hhmmm... I might have, but I don't recall
[20:26] <pmatulis> DammitJim, to summarize, you created a logical volume, formatted, and mounted it. then when you shut down the system hangs?
[20:30] <DammitJim> pmatulis, I installed Ubuntu 16 (new VM). During the installation, I selected to use LVM, then deleted the root logical volume (since it was using up ALL the storage), and then created the logical volumes for root, home, usr, and var. Finished the install and the system would reboot no problem
[20:30] <DammitJim> Then I performed an apt-get upgrade and every time I shut down the system, it hangs for like 7/8 minutes
[20:31] <lordievader> On what service does it hang?
[20:33] <DammitJim> I don't know... it just says: Stopped LVM2 metadata daemon
[20:33] <pmatulis> DammitJim, did you add encryption anywhere?
[20:33] <DammitJim> so, whatever is after that (which is not getting printed)
[20:33] <DammitJim> no, no encryption
[20:34] <lordievader> Sounds a bit like something is in use and systemd is waiting for the process to die and release the resource. Ultimately giving up.
[20:35] <pmatulis> DammitJim, does 'sudo vgscan' complete quickly?
[20:35] <DammitJim> yes
[20:35] <DammitJim> I also shortened the timeout in
[20:36] <DammitJim> /etc/systemd/system.conf
[20:36] <pmatulis> DammitJim, after the problem was detected?
[20:37] <DammitJim> right
[20:37] <DammitJim> but it doesn't help
[20:37] <pmatulis> DammitJim, did you do *any* other futzing around? configuration changes? any at all?
[20:37] <DammitJim> no
[20:38] <DammitJim> clean install
[20:38] <DammitJim> you can try it
[20:38] <DammitJim> and like I said, the only difference I saw was that if I don't add the /var logical volume, it doesn't do this
[20:38] <pmatulis> DammitJim, provide output (pastebin) to 'df -h'
[20:39] <lordievader> Just speculating, would some process want to write to /var after lvm stopping (some log or something).
[20:41] <DammitJim> right!
[20:41] <DammitJim> but how do we get eyes into that situation?
[20:41] <pmatulis> DammitJim, still waiting for output. obviously you looked at your logs right?
[20:42] <DammitJim> yeah, but I can send thm
[20:42] <DammitJim> I don't see anything peculiar in there, but I'm not that smart
[20:43] <DammitJim> hold on... let me do df -h on a window I can copy from
[20:43] <pmatulis> DammitJim, also, does it pause at bootup?
[20:43] <DammitJim> there is a short pause on a line that talks about mounting /usr
[20:43] <DammitJim> what is the preferred pastie on this channel?
[20:44] <lordievader> !paste | DammitJim
[20:44] <pmatulis> nice one
[20:44] <DammitJim> http://paste.ubuntu.com/23788640/
[20:45] <DammitJim> and btw, the same behavior is there when I upgraded a 14.04.5 system with logical volumes for var,usr,root, and home
[20:48] <pmatulis> DammitJim, not that it matters but this layout reminds me of ~13 years ago working with slackware. slicing up storage like this was all the rage. it's not normally done anymore afaict. i wonder if testing is not catching a buglet because of that
[20:49] <DammitJim> could be
[20:49] <DammitJim> are you saying I shouldn't be using logical volumes?
[20:49] <pmatulis> LVM is fine. it's just the number of them
[20:50] <lordievader> DammitJim: Do you have a requirement for a separate /var?
[20:52] <DammitJim> I do
[20:52] <lordievader> Because?
[20:52] <DammitJim> there is variable data that goes on that partition
[20:52] <lordievader> Why cant that be shared on / ?
[20:53] <DammitJim> and we would hate for /var to get full and halt the system if it is on /
[20:53] <DammitJim> you know, one of the premises as to why we wanted separate partitions a Looooong time ago
[20:54] <lordievader> You could isolate the process/directory which fills quickly and put that on a separate volume.
[20:54] <lordievader> If that allows the server to reboot ;)
[21:16] <DammitJim> I know and I appreciate the feedback, but that's all a workaround
[21:16] <DammitJim> there is something fundamental here that I'd like to address
[21:17] <DammitJim> I have over a hundred servers that I would have to change for that var thing
[21:17] <DammitJim> when I upgrade from 14 to 16
[21:17] <DammitJim> it's one more thing to worry about
[21:18] <lordievader> True, more research to the cause is needed, I suppose.
[21:19] <nacc> DammitJim: if you want, i'd file a bug (perhaps against lvm for now) with very clear reproducible steps?
[22:32] <coreycb> jamespage, b2 is ready to promote when you get a chance.  we're at 9 tempest failures.
[22:33] <coreycb> jamespage, I'd like to hold off on promoting horizon though, as upstream b2 is not in good shape.  i just tried out a snapshot from today and it seems better so far. i may upload that tomorrow.