[09:01] <cpaelzer> jamespage: no backporting by cloud-archive of new DPDK further back than Xenial right?
[09:10] <jamespage> cpaelzer: correct
[09:10] <jamespage> any earlier pockets are closed now
[09:10] <cpaelzer> good, thanks jamespage
[09:10] <cpaelzer> there was a fix to recent dpdk to build on older kernels <=3.10 I think
[09:10] <cpaelzer> no need to rush it in then
[12:59] <frickler> jamespage: coreycb: ICYMI nova 15.0.1 is out with high impact fixes for ocata, pls update
[14:04] <jamespage> frickler: on my list for as soon as it appeared
[14:04] <jamespage> frickler: working that next
[15:28] <fullstop> any good tools to understand why I'm having poor network performance on a server?
[15:29] <fullstop> There are four NICs, and throughput is choppy, even on a NIC which is unused for anything else, so I'm guessing that it's something with the scheduler.
[15:29] <fullstop> The server is a little busy, load avg of ~16
[15:29] <fullstop> but there are 24 cores
[17:14] <exegesis> Hi there. I'm having trouble executing a script at startup.
[17:14] <exegesis> I have added it to cron as a @reboot job, not working.
[17:14] <exegesis> Can anyone give me a light on how to add it to rc.local?
[17:29] <erlon> stgraber: Hey Stephane, Im hitting this bug (https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1576341) do you know what is the status? I basically cant run iscsid inside the container (failed to mlockall, exiting...). I added the lxc.aa_profile=unconfined, as suggested in the bug, restarted the container, but no success, I still get the same error
[17:30] <erlon> stgraber: any idea?
[17:33] <nacc> erlon: did you verify the running container is unconfined?
[17:34] <nacc> erlon: what version of open-iscsi?
[17:34] <erlon> nacc: the last, 2.0.873
[17:34] <erlon> nacc: hmm, one point worth to mention, this container is running inside a KVM machine
[17:35] <erlon> nacc: not sure if that can be a problem
[17:35] <nacc> erlon: specific version -- is this i ubuntu 16.04?
[17:35] <erlon> open-iscsi                           2.0.873+git0.3b4b4500-14ubuntu3.3 amd64
[17:35] <erlon> nacc: yes
[17:36] <nacc> erlon: right it doesn't have the fix, afaict
[17:36] <nacc> erlon: only zesty does
[17:36] <nacc> erlon: (well, the open-iscsi fix)
[17:36] <nacc> erlon: you can try making the same change to the service file locally
[17:36] <erlon> nacc: reALLY?
[17:37] <erlon> opps
[17:37] <erlon> caps
[17:37] <nacc> erlon: i see no tasks for xenial and the last comment is for the zesty version (per changelog and rmadison)
[17:38] <erlon> nacc: you mean configuring the iscsid.service  in the container?
[17:38] <nacc> erlon: let me see if i can find the change verbatim to show you
[17:38] <erlon> nacc: ok
[17:40] <erlon> https://www.irccloud.com/pastebin/kAgDBIav/
[17:40] <nacc> erlon: http://paste.ubuntu.com/24183873/
[17:41] <nacc> erlon: interesting -- i don't know enough about why mlockall is failing then, it would seem if it was unconfined, it shouldn't have issues with it
[17:42] <erlon> nacc: as it seems that fix just blocks iscsi from running inside confined containers, righ?
[17:43] <erlon> nacc: ConditionVirtualization=!container, 'if this is a container, don't run'
[17:44] <nacc> erlon: i think it prevents it from running in containers period?
[17:44] <erlon> nacc: yes,
[17:45] <nacc> hallyn: --^
[17:45] <nacc> hallyn: should that change be SRU'd?
[17:46] <erlon> nacc: but checking here, I have run iscsid inside a container running in a baremetal machine and is not getting the same problem
[17:46] <nacc> erlon: same version of the container (16.04)?
[17:47] <erlon> nacc: I believe yes, checking
[17:48] <erlon> nacc: very same version
[17:48] <erlon>  iscsid -f --version
[17:48] <erlon> iscsid version 2.0-873
[17:49] <nacc> erlon: the iscsid version isn't very helpful
[17:49] <nacc> need the package versions
[17:49] <nacc> `apt policy open-iscsi`
[17:49] <erlon> dpkg -l | grep iscsi
[17:49] <erlon> ii  open-iscsi                       2.0.873+git0.3b4b4500-14ubuntu3.3  amd64        iSCSI initiator tools
[17:49] <erlon> https://www.irccloud.com/pastebin/4VZlxypy/
[17:50] <erlon> https://www.irccloud.com/pastebin/BYzncru8/
[17:51] <erlon> nacc: ^this last one inside the virtualized container
[17:51] <nacc> erlon: ack
[17:52] <nacc> erlon: well, i'm a bit stumped myself. It does seem like hallyn's fix in 17.04 is to prevent open-iscsi.service from running in containers at all. But I'm not sure why that'
[17:52] <nacc> that's acceptable -- i guess you would need to do it manually
[17:52] <patdk-wk> I'm having a horrible time on 16.04 with a cups server
[17:52] <erlon> nacc: hmm, but I can't run it  even manualy
[17:52] <patdk-wk> when run with cupsd -f, it works fine
[17:53] <erlon> nacc:
[17:53] <nacc> erlon: right, that's probably why it was disabled
[17:53] <erlon> root@juju-5efd81-1-lxd-1:/home/ubuntu# iscsid -f
[17:53] <erlon> iscsid: failed to mlockall, exiting...
[17:53] <patdk-wk> but when run as a service, or manually, using cupsd -l
[17:53] <patdk-wk> it timesout after 60seconds and exits
[17:53] <nacc> erlon: as you don't have CAP_IPC_LOCK actually
[17:53] <erlon> nacc: Im mean calling the binary directly
[17:54] <nacc> erlon: right, the systemd service just calls the binary
[17:54] <nacc> erlon: so rather than have it fail all the time in containers, hallyn disabled it
[17:54] <erlon> nacc: but inside the baremetal container it runs fine
[17:54] <jamespage> frickler: just published the 15.0.1 update for nova through to zesty; that will auto-backport in the next hour and then I'll promote and test to the UCA (that might be AM tomorrow)
[17:54] <nacc> erlon: are the two hosts (baremetal and KVM) the same Ubuntu?
[17:55] <jamespage> if you want it early then it will be in ppa:ubuntu-cloud-archive/ocata-staging
[17:55] <erlon> nacc: I believe yes, just double checking
[17:56] <drab> patdk-wk: had the same issue, went back to 14.04, couldn't figure it out
[17:56] <nacc> erlon: and same version of lxd in both?
[17:56] <patdk-wk> it's odd
[17:56] <nacc> erlon: and kernel (since there are hwe now)
[17:56] <patdk-wk> only happening on this one server, the other one it works fine
[17:56] <patdk-wk> but think that has something to do with avahi stuff keeping it alive from hitting the 60sec timeout
[17:57] <drab> patdk-wk: yeah, I think so, because onew ork around was to add a "ping" cronjob
[17:58] <patdk-wk> ya, as long as I keep hitting the web interface to configure and do setup, it keeps running
[17:58] <patdk-wk> as soon as I'm done, it's dead
[17:58] <erlon> nacc: HOST->"Ubuntu 16.04.2 LTS" ->  KVM -> ="Ubuntu 16.04.1 LTS", -> LXC -> Ubuntu 16.04.2 LTS
[17:58] <drab> I did some tracking in systemd thinking it was a socket problem, and found some stuff was indeed missing, but couldn't figure out what needed to be fixed
[17:58] <drab> patdk-wk: but imho the prob is there, with ssytemd and the "on-demand" stuff
[17:58] <nacc> erlon: so the KVM and baremetal systems are different?
[17:59] <nacc> erlon: rather than using codenames, it's probably better to pastebin things like `uname -a`, `apt policy lxd`
[17:59] <drab> hey DammitJim
[17:59] <drab> DammitJim: how did that work out?
[18:00] <patdk-wk> epoll_wait(4, [], 65536, 1000)          = 0
[18:00] <patdk-wk> epoll_wait(4, [], 65536, 60000)         = 0
[18:00] <patdk-wk> epoll_ctl(4, EPOLL_CTL_DEL, 9, 0x7ffe5adefb70) = 0
[18:00] <patdk-wk> epoll_ctl(4, EPOLL_CTL_DEL, 10, 0x7ffe5adefb70) = 0
[18:00] <patdk-wk> close(9)                                = 0
[18:00] <patdk-wk> close(10)                               = 0
[18:00] <erlon> nacc: hold on, just fcorrenting KVM I mean the lxc running
[18:00] <patdk-wk> right after the return from that 60000 epoll_wait, cupsd start to shutdown
[18:00] <patdk-wk> socket ping should keep it alive :( but annoying
[18:00] <drab> yeah
[18:01] <erlon> nacc: so yes, the LXC container that iscsid can work (16.04.1 ) is different from the LXC running under KVM (16.04.2)
[18:01] <patdk-wk> doesn't happen on my laptop, but that has the full gui installed on it, not just a cups server
[18:02] <erlon> nacc: just does not makes sense the the later version should work, not the 16.04.1
[18:03] <patdk-wk> ● cups.service - CUPS Scheduler
[18:03] <patdk-wk>    Loaded: loaded (/lib/systemd/system/cups.service; enabled; vendor preset: enabled)
[18:03] <patdk-wk>    Active: inactive (dead) since Wed 2017-03-15 13:32:05 EDT; 31min ago
[18:03] <patdk-wk>      Docs: man:cupsd(8)
[18:03] <patdk-wk>   Process: 13103 ExecStart=/usr/sbin/cupsd -l (code=exited, status=0/SUCCESS)
[18:03] <patdk-wk>  Main PID: 13103 (code=exited, status=0/SUCCESS)
[18:04] <ducasse> is there a difference between the zfs.ko that comes with the kernel and the zfs-dkms package? both are the same version, afaict.
[18:04] <axisys> I increase the virtual hard disk.. how do I realize the new size in the VM? pvs still shows old size
[18:04] <patdk-wk> ducasse, yes
[18:04] <axisys> I increased the size of the virtual disk..
[18:06] <DammitJim> drab, I told one of the other managers and he said we'll talk about it
[18:06] <DammitJim> thanks for following up
[18:09] <axisys> ah.. it took a while for pvresize to take in effect.. now I see the large size
[18:10] <DammitJim> axisys, good luck with resizing
[18:10] <axisys> DammitJim: done
[18:10] <DammitJim> I still have to document how I do that to standardize it in our company
[18:10] <DammitJim> awesome
[18:11] <ducasse> patdk-wk: i assume i should use the zfs.ko in the kernel, as the zfs metapackage doesn't drag in zfs-dkms?
[18:11] <patdk-wk> either is fine
[18:11] <nacc> for ubuntu kernels you shouldn't need zfs-dkms, iiuc
[18:11] <patdk-wk> dkms is for when you use a kernel that does not have it
[18:11] <axisys> my /dev/sdb was a PV .. so once I changed the size of the disk in vmware, I had to run pvresize.. and it took a little type to reflect
[18:11] <ducasse> patdk-wk: ok, thanks
[18:11] <patdk-wk> or if if you want to use a different version than the one in the kernel
[18:12] <ducasse> i see. perfect :)
[18:12] <frickler> jamespage: great, thx
[18:27] <nacc> erlon: so i'm not sure where we stand -- it's sort of hard to tell which version is which (as 16.04.1 and 16.04.2 are the same version of lxc and open-iscsi, the difference is in the kernel/x stacks potentially)
[20:26] <patdk-wk> drab, this is not really the *right* fix
[20:27] <patdk-wk> but editing the systemd cups.service unit file to change the -l (run on demand) to -f, has fixed the issue
[20:33] <drab> patdk-wk: oh, good point, better than the ping or stick with 14.04 I guess
[20:33] <drab> thanks
[20:33] <patdk-wk> it seems like -F works too
[20:33] <patdk-wk> testing that, as it seems like a better solution
[20:34] <patdk-wk> atleast till the real reason is solved, this on demand stuff for a server heh
[20:34] <drab> while on the topic, I don't know your setup, but you wouldn't happen to have quite a few diff systems there and a reasonable way to centralized printing?
[20:34] <drab> I wrote to the cups ML, but no response for some reason
[20:34] <drab> the client.conf thing they say is deprecated and "you must run a local cups connecting to the remote one"
[20:35] <drab> I get that in principle, but in practice it's a mess ime
[20:35] <patdk-wk> that is how I do it
[20:35] <drab> ok, maybe I'm doing something wrong, here's the big problem for me
[20:35] <patdk-wk> I setup a cups printer, that advertizes airprint and google print
[20:35] <patdk-wk> setup cups-browsed
[20:36] <drab> printer goes down, queue is still up because cups doesn't actually check the printer
[20:36] <drab> user goes to machineA, tries to print, doesn't work, tries again, doesn't work, a third time, doesn't... now machineA has 3 jobs in the local queue
[20:36] <drab> then use thinks "I nkow, there's a problem with machine A" and goes to machine B and does the same thing
[20:37] <drab> now I have two machines with 3 jobs each of the same thing in the queue
[20:37] <drab> I go to fix the printer and bam, 6 copies of the same thing gets printed
[20:37] <patdk-wk> setup your cups server, with cups-browsed, and BrowseLocalProtocols cups
[20:37] <patdk-wk> then on the client machines install both again, but with cups-browsed setting of BrowseRemoteProtocols cups
[20:37] <patdk-wk> it will automatically add the cups servers printers locally
[20:37] <drab> with client.conf that doesn't happen because jobs only exist on the remote cups, which I can easily monitor and see the dup'ed jobs and remove them before I restart the printer
[20:38] <drab> the problem isn't finding/configuring the printer
[20:38] <drab> it's job management
[20:38] <drab> specfically stuck jobs
[20:38] <drab> when something is wrong with the printer, local queues ime are a nightmare
[20:38] <drab> becaue I have no clue on which machine theya re and purging them is a pain
[20:38] <patdk-wk> yes, but the job won't be on the local machines then
[20:40] <drab> mmmh, I had that situation quite a bit, which is why we switched to client.conf
[20:40] <drab> but maybe there was something else wrong
[20:41] <patdk-wk> dunno, I never have to worry about queues ever myself
[20:41] <patdk-wk> I mainly use that feature for my laptop and macs
[20:41] <patdk-wk> they show up on the lan, the printer just appears and is usable
[20:41] <patdk-wk> no local config setup or anything needed
[20:41] <patdk-wk> I can't imagine it would get stuck in a local queue when doing that
[20:42] <patdk-wk> as it should always be able to go to the remote queue
[20:42] <patdk-wk> or the printer would *vanish* from the local machine
[20:43] <drab> I see
[20:43] <drab> eer, above I said "printer has a problem", the problem case is "remote cups has a problem"
[20:43] <drab> which is when the job is cached locally
[20:44] <drab> in the other case you're right, the job would be gone from the local queue even if not printed
[20:44] <drab> I'll take a look at the browserd thing you mentioned, thanks
[20:44] <drab> if the printer did vanish that'd stop the problem altogether
[21:13] <patdk-wk> ok, not sure -F works :(
[21:19] <patdk-wk> ok, found the real reason
[21:19] <patdk-wk> it's cause I have no shared printers configured
[21:21] <patdk-wk> adding this as in the bug report I found solves the issue
[21:21] <patdk-wk> ListenStream=631
[21:21] <patdk-wk> bug #1598300
[21:33] <patdk-wk> ya, that looks like it fixed up
[21:33] <patdk-wk> upgrading to the cups -proposed package
[21:33] <patdk-wk> that is damned annoying
[21:40] <drab> patdk-wk: good catch. the upstream patch seemed unresolved tho, ie they don't accept the ListenStream=631
[21:40] <patdk-wk> different issue, kindof
[21:41] <patdk-wk> default in ubunt is to have the webinterface enabled
[21:41] <patdk-wk> then cups should never exit
[21:41] <patdk-wk> the patch fixes that
[21:41] <patdk-wk> so then it becomes a non-issue