[16:41] <adam_g> smoser: ping
[16:45] <smoser> adam_g, here...
[16:51] <adam_g> smoser: https://bugs.launchpad.net/software-properties/+bug/829109 breaks the apt CC module because cloud-init/cfg are running with a tty attached. any thoughts? im looking at it now
[16:51] <uvirtbot`> Launchpad bug 829109 in software-properties "add-apt-repository - New confirmation dialog breaks existing scripts" [Medium,Fix released]
[16:52] <smoser> adam_g, will look at it in 15 minutes
[16:53] <adam_g> smoser: if you're jammed up don't worry about it
[16:54] <smoser> no, just patch piloting for another 15 minute sor so
[16:54] <adam_g> k
[17:20] <adam_g> smoser: /win 5
[17:21] <adam_g> smoser: errr, fix incoming
[18:22] <smoser> hallyn, any progress?
[18:23] <smoser> adam_g, so do you ahve a redirect to /dev/null or '-y' add for cloud-init ? or you want me to do that
[18:24] <smoser> i guess i wonder why there is a console there anyway.. maybe we should generally kill that in cloud-init
[18:24] <smoser> as there is no reason that there should be a console
[18:26] <hallyn> smoser: not sure yet how/why, but i think the root cause is that /var/lib/nova is a symlink to /mnt/var-dirs/nova
[18:26] <hallyn> ah, yes
[18:26] <hallyn> i getr it
[18:27] <hallyn> it's a symlink.  that symlink doesn't correctly dereference while we're pivot-rooting
[18:27] <hallyn> or at least, not to /.oldroot/...
[18:28] <hallyn> maybe just using the right path in xml will work?  lemme try
[18:30] <smoser> really?
[18:30] <smoser> holly carp
[18:30] <smoser> so that pivot root issue would be actually kernel ?
[18:30] <hallyn> no,
[18:31] <hallyn> issue is that libvirt_lxc is trying to mount --move from /.oldroot/var/lib/nova/.../dev/pts, but a part of that is a symlink which does not dereference
[18:32] <hallyn> biam
[18:39] <smoser> hallyn, so why did that not fail with reasonable debug output ?
[18:39] <smoser>   virReportSystemError(errno, "%s", _("Failed to mount /dev/pts in container"));
[18:39] <hallyn> well in the end (IF i'm right) that's pretty reasonable output :)
[18:42] <smoser> its *very* reasonable output
[18:42] <smoser> but i did not see it in any log
[18:42] <adam_g> smoser: sorry, back
[18:42] <hallyn> oh
[18:43] <hallyn> yeah
[18:43] <smoser> does it segfault or something?
[18:43] <smoser> adam_g, i see your merge proposal
[18:43] <hallyn> anyway, it's definately a bug, but we'd have to (and this will take time) come up with a reproducible case and work with upstream for a good fix
[18:43] <hallyn> but, i'm still not 100% working
[18:43] <adam_g> adam_g: sent a merge proposal that just redirects input to /dev/null otherwise it'll have a tty. thats the better way to do it, as '-y' would only work with the recent software-properties versions
[18:43] <smoser> adam_g, do you happen to have an idea on how we would say "stdout is not a terminal" globally in cloud-init ?
[18:45] <smoser> hallyn, that is easy to do if you're right. (the test case)
[18:45] <smoser> if thats the only issue.
[18:45] <adam_g> smoser: we want to do that for stdin, and its handled per subprocess via the subp() function in cloudinit/util.py
[18:45] <smoser> and i'm *really* sorry that we wasted time on this... i never would have expected that symlink i was doing to cuase such a pita
[18:46] <smoser> right... stdin, not stdout. my bad.
[18:46] <RoAkSoAx> adam_g: im here :)
[18:48] <hallyn> smoser: well, the thing i'm hanging on now (after not using the symlink) is a failed (-EBUSY) umount of .oldroot.  which should NOT be hanging.  it's not busy
[18:48] <hallyn> we may need to make it do lazy umount if umount fails.  not sure how danpb will feel about that
[18:48] <RoAkSoAx> smoser: isn't there a way (in user-data) to tell add-apt-repository to use -y?
[18:48] <hallyn> or, if it's just masking over real problems
[18:49] <hallyn> oh wait, maybe bc i didn't change console file
[18:49] <smoser> RoAkSoAx, i could think of a hack
[18:50] <RoAkSoAx> smoser: what do you have in mind?
[18:51] <smoser> boothook that did: f=/usr/bin/add-apt-repository; [ -f "$f.real" ] || mv $f $f.real && echo '#!/bin/sh\nexec $0.real -y "$@"\n' && chmod 755 $f
[18:55] <hallyn> smoser: yes!  i'd messed things up enough that i needed a reboot, but in the end that fixes it
[18:55] <RoAkSoAx> smoser: cool thanks
[18:56] <smoser> hallyn, right. reboot fixes it.
[18:56] <smoser> well, sort of.
[18:56] <smoser> it would then break the nbd mount
[18:57] <smoser> which will break that directory from being an lxc container
[18:57] <hallyn> smoser: well id' really been mucking with my mount tables
[18:57] <hallyn> now it works, so long as i edit the xml to use the non-symlink path
[18:57] <hallyn> smoser: you're using an older lxcguest in that rootfs?  it has console.conf...
[18:57] <hallyn> guess you don't really care
[18:58] <smoser> hallyn, well, where'd you et the image from ?
[18:58] <hallyn> smoser: can you proceed this way (just sed -i 's./var/lib./mnt/nova-dirs.' on any xmls you create)
[18:58] <hallyn> smoser: i follwed your instructions to the letter.  (without paying much attention :)
[18:59] <smoser> yeah
[18:59] <smoser> that will get the most recent daily
[18:59] <smoser> hallyn, which is built either either yesterday or today depending on when you got it
[19:00] <smoser> (the right serial number should be in 'ls ~/images')
[19:00] <hallyn> oh. hmm.  actually console.conf has the right contents.  So wtf is it running a getty?
[19:02] <hallyn> all right ignore me i guess i was on crack.  it works fine now
[19:29] <smoser> what works fine, hallyn?
[19:36] <hallyn> smoser: starting those containers
[19:36] <hallyn> smoser: once you sed -i 's./var/lib./mnt/nova-dirs.' on the xml
[19:38] <smoser> hm..
[19:38] <smoser> so what do you thikn is failing ?
[19:39] <smoser> and why don't we get a debug message that says what is failing?
[19:39] <smoser> is it segfaulting somewhere ?
[19:40] <hallyn> you have a long path /a/b/c/d/e/f/g, and c along the way is a symlink which is not valid when looking at /.oldroot/a/b/c/d/e/f/g
[19:40] <hallyn> so i'm not sure whether upstream will say it is a bug in libvirt, or just abuse
[19:40] <hallyn> the latter doesn't really seem reasonable, but i'm not sure what they can do to cleanly fix it
[19:41] <hallyn> anyway we just need to report it upstream, and you need to just specify paths not containing symlinks in the meantime in .xml for libvirt-lxc
[19:45] <smoser> hallyn, ok. i'll try a work around in nova
[19:45] <smoser> that should be reasonable there.
[19:45] <smoser> basically we need a os.path.realpath() in libvirt-lxc
[19:46] <hallyn> no that's not enough
[19:46] <smoser> oh, really?
[19:46] <hallyn> oh, you mean for your workaround?
[19:46] <hallyn> yes, that'll do :)
[19:46] <hallyn> i think
[19:47] <smoser> i can easily do the os.path.realpath() in nova
[19:47] <hallyn> ok
[19:47] <smoser> i'm less C capable, but if that is sufficient, doing it in libvirt-lxc would be more beneficial
[19:47] <hallyn> if you do that in a python shell right now on /var/lib/libvirt/nova, what do you get?
[19:49] <hallyn> yeah that works
[19:49] <hallyn> it just has to be done while writing the xml
[19:50] <smoser> well, it doesn' thave to be done then. it could be done while consuming the xml.
[19:56] <hallyn> true
[21:22] <hallyn> smoser: oddly, when i try to reproduce the symlink failure with a busybox lxc container, it works just fine
[21:23] <hallyn> ah, i think i see
[21:34] <hallyn> nope.
[21:35] <hallyn> smoser: apart from an hour or two tonight and a bit of checking over the week, i'll be out this week.  if you manage to reproduce it with a trivial busybox container pls shoot me an email.  i have, so far, failed
[21:35] <hallyn> bbl
[22:17] <Kyle__> Is it normal for your nodes to not reconnect to the controller after a reboot?
[22:52] <Kyle__> Is there anything stored in eucalyptus's conf files that is mac-address specific?
[23:03] <obino> Kyle__: if you mean about the NC host, then no
[23:03] <Kyle__> obino: OK.  And on the CC?
[23:04] <obino> Eucalyptus do not uses MAC. The only are involved when using STATIC mode
[23:05] <Kyle__> obino: OK.  Thanks.  Switched NICs out, having a bit of weirdness now
[23:11] <obino> NC not reporting?