[11:16] <bac> good morning frankban
[11:54] <frankban> morning bac
[12:01] <gary_poster> not sure it is needed after that upgrade, but going to restart anyway...
[12:09] <gary_poster> bac benji frankban call in 1 or 2 (gmb is sick)
[12:09]  * bac hordeing
[12:31] <gary_poster> For future reference for myself, in case I have to tell someone about this for help later...http://pastebin.ubuntu.com/905522/
[12:31] <gary_poster> off to restart and choose the new kernel; wish me luck
[12:55] <gary_poster> -1 point to gary for not following instructions properly; +0.5 for figuring it out before asking anyone about it.  Similar complaints for correct kernel as opposed to incorrect kernel http://pastebin.ubuntu.com/905554/
[12:56] <gary_poster> ...and I need to restart again
[13:17] <bac> hey gary you want to collaborate on an email to francis re: the switchpoint conference?
[13:19] <gary_poster> bac, sure
[13:19] <gary_poster> bac, I feel like I should stare at the website more
[13:19] <bac> my points for wanting to go:  1) learn more about how open source is being using by NGOs, especially for health care in africa, 2) see how ubuntu is being used in that mix and look for opportunities for canonical, ubuntu, and ubuntu-ngo, 3) profit
[13:20] <bac> it is worth noting the main sponsor (intrahealth) has been a long time user of ubuntu, bzr, and launchpad.  check out their suite of LP projects:  https://launchpad.net/ihris-suite
[13:20] <gary_poster> bac, sounds like you should write the email, and I should affix my name to it ;-)
[13:20] <bac> :)
[13:21] <bac> i've been in touch with those folks before and even did an in-house training session for LP and bzr many moons ago
[13:27] <gary_poster> bac, the only possible addition I had was whether anything technically interesting was going to be discussed.  I saw a lot of seemingly big titles from big named tech companies, but I was suspicious that the message was going to be more high-level than technical.  I'm pretty sure that's the case, despite the one seemingly technical talk title I saw of "Building for Scale".
[13:28] <bac> gary_poster: i don't even see that session.  where did you find it?
[13:29] <gary_poster> bac, seriously, if you are ok with it, are you ok with writing the message, then I'll read and discuss as appropriate, and then forward?  session: fifth down in afternoon segment at http://www.switchpointideas.com/schedule
[13:29] <bac> gary_poster: i'll send you a draft
[13:30] <gary_poster> cool bac, thanks
[13:30] <bac> gary_poster: you did note it is kind of pricey?
[13:30] <gary_poster> frankban, I'm in https://talkgadget.google.com/hangouts/extras/canonical.com/goldenhordeoneonone .  Head on over when ready; no rush
[13:30] <gary_poster> heh. another title: "Hybrid Ideation"
[13:32] <bac> i don't even know what that means
[13:34] <gary_poster> me either
[13:35] <bac> gary_poster: off topic, but are getting/got a new ipad?  you'd mentioned in the past you mentioned the screen resolution being a limiting factor for you.  i'm curious to see how much clearer the text is.
[13:35] <gary_poster> benji, frankban saw your error on a non-parallel ec2 run.  That opens us up even better to a disabling story.  In fact, I might encourage frankban to disable, and then you can follow along and try to fix in a timebox.  http://pastebin.ubuntu.com/905569/
[13:36] <benji> gary_poster: sounds good
[14:40] <benji> frankban: I'm really confused about something in setuplxc, do you have a minute to help me?
[14:40] <frankban> sure benji
[14:43] <benji> frankban: heh, as such things happen, I believe I've figured out the answer just by formulating the question
[14:43] <benji> frankban: thank you for all your help ;)
[14:43] <bac> frankban: i'm having problems with setuplxc too.  i get this:
[14:43] <bac> 'ubuntu' template installed
[14:43] <bac> 'lptests' created
[14:43] <bac> ssh: Could not resolve hostname lptests: Name or service not known
[14:43] <bac> with the last message repeated lots
[14:44] <frankban> bac: and then it exits?
[14:44] <bac> frankban: no
[14:44] <bac> frankban: it is stuck, emitting that message every so often.  about 20 now
[14:44] <bac> if i kill it, do i have to start over?
[14:45] <frankban> it should exit after 30 retries
[14:46] <frankban> bac: no, 60
[14:46] <bac> er, ok
[14:47] <frankban> bac:  and you don't have to start over, but you are experiencing a known issue involving precise and dns resolution
[14:47] <bac> ok
[14:47] <frankban> bac: the script is not able to ssh inside the container
[14:48] <bac> yeah, i figured as much.  but i don't know how much more work it needs to do
[14:48] <frankban> bac, can I see the contents of your resolv.conf?
[14:48] <bac> nameserver 10.0.3.1
[14:48] <bac> # Generated by NetworkManager
[14:48] <bac> domain spoetzl.net
[14:48] <bac> search spoetzl.net
[14:48] <bac> nameserver 127.0.0.1
[14:49] <frankban> hm...
[14:49] <frankban> actually you have the right nameserver, bac are you able to ssh lptests?
[14:49] <bac> the localhost line is odd.  it should not be there.  it should be my local DNS at 192.168.x.x
[14:50] <bac> no, manually ssh'ing shows the same name resolution problem
[14:50] <frankban> bac, but the conteiner is running, right?
[14:51] <bac> i assume so.  how would i tell?
[14:51] <bac> since i cannot ssh to it
[14:52] <frankban> sudo lxc-info -n lptests
[14:52] <bac> no, it is stopped
[14:52] <bac> hmm
[14:55] <frankban> bac: what about the script
[14:55] <frankban> still running?
[14:56] <bac> yes
[14:58] <frankban> bac: could you check the lxc starts correctly?
[14:58] <frankban> sudo lxc-start -n lptests
[14:59] <bac> the script just ended with
[14:59] <bac> Error running command: ssh -t -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /home/bac/.ssh/id_rsa bac@lptests -- true
[14:59] <benji> has anyone seen this error from setuplxc? AppArmor parser error for /etc/apparmor.d/usr.bin.lxc-start in /etc/apparmor.d/usr.bin.lxc-start at line 5: syntax error, unexpected TOK_END_OF_RULE, expecting TOK_ID
[14:59] <benji> bac: I'm working on that one as we speak
[14:59] <bac> frankban:
[14:59] <bac> lxc-start: failed to attach 'vethttuY7k' to the bridge 'lxcbr0' : No such device
[14:59] <bac> lxc-start: failed to create netdev
[14:59] <bac> lxc-start: failed to create the network
[14:59] <bac> lxc-start: failed to spawn 'lptests'
[14:59] <bac> lxc-start: No such file or directory - failed to remove cgroup '/sys/fs/cgroup/cpuset//lxc/lptests'
[15:03] <frankban> bac: do you have the 'lxcbr0' device?
[15:03] <frankban> benji: that's new, something has changed in apparmor?
[15:04] <bac> frankban: no, no /dev/lxc*
[15:04] <benji> frankban: I can only assume so.  I don't know if this is related to the log-in issue or not (that seems to have to do with the binding (-b) option to lxc-create)
[15:04] <frankban> bac, what virtual device you have running ifconfig?
[15:05] <benji> I think we should have been more strict with our checking of errors when running subprocesses, I don't know when that error started or if it was coincident with the ssh issue
[15:06] <bac> frankban: i have virbr0
[15:06] <frankban> yes benji, I agree, setuplxc reports errors and exits only during ssh operations. lpsetup is different
[15:06] <frankban> bac: aha
[15:07] <frankban> and I assume you have an upgraded precise host...
[15:10] <bac> frankban: yes, i'm on the slashing edge
[15:12] <frankban> bac: do you have dnsmasq installed and running in your host?
[15:16] <frankban> bac: if so, maybe you are encountering this bug: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/928524
[15:16] <_mup_> Bug #928524: lxcbr0 fails to come up when dnsmasq is installed <dnsmasq (Ubuntu):Confirmed> <libvirt (Ubuntu):Confirmed> <lxc (Ubuntu):Confirmed> < https://launchpad.net/bugs/928524 >
[15:18] <benji> it's wstarting to look like -b is broken
[15:18] <bac> frankban: i do.  but a manually-installed lxc had been working for me for a very long time
[15:20] <frankban> bac: and you were following those instructions? https://dev.launchpad.net/Running/LXC
[15:21] <bac> frankban: yes
[15:21] <gary_poster> benji, which -b?
[15:21] <frankban> bac: ok, so you were using virbr0
[15:21] <benji> gary_poster: lxc-create's
[15:22] <gary_poster> benji, interesting.  If you want me to try and dupe, just ask.
[15:22] <frankban> bac: you can try to use that now, change lxc.network.link in /var/lib/lxc/lptests/config and try to start the container again
[15:23] <benji> gary_poster: since bac has the same problem with his lxc, I suspect it's endogenous
[15:26] <bac> frankban: yes, it will now start, then goes to a login prompt but doesn't recognize ubuntu/ubuntu
[15:26] <bac> ssh lptests does not work either as before
[15:27] <gary_poster> I'm going to look, and ask serge if I see something similar
[15:27] <bac> benji: you win vocabulary bingo today
[15:27] <frankban> bac: to be able to use the lxc name you have to change the first line of resolv.conf to point to the ip address of virbr0
[15:27] <bac> frankban: ah, ok
[15:27] <benji> ha!
[15:28] <benji> I've been listening to too many economist podcasts.
[15:28] <bac> frankban: great, that worked
[15:29] <bac> frankban: so, has my adventure exposed any real problems or just known issues with dnsmasq/virbr0, etc?
[15:29] <bac> benji: which podcasts?
[15:30] <bac> economic podcasts are the new ambien
[15:30] <benji> heh
[15:30] <benji> econtalk.org is very good
[15:30] <frankban> bac: I didn't know that issue, so, if not fixed, it can be a problem when using lpsetup to set up a development environment. It should not be a problem for the testing environment
[15:33] <frankban> gary_poster: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/928524 : IMHO this means that we need to check the interface is up before doing anything in lpsetup, it should not be a problem for the testing environment.
[15:33] <_mup_> Bug #928524: lxcbr0 fails to come up when dnsmasq is installed <dnsmasq (Ubuntu):Confirmed> <libvirt (Ubuntu):Confirmed> <lxc (Ubuntu):Confirmed> < https://launchpad.net/bugs/928524 >
[15:36] <frankban> bac: I don't know if your lxc is in a consistent state now, but, if you want, you can try to resulm setuplxc using -a initialize_lxc
[15:36] <frankban> s/resulm/resume
[15:36] <bac> frankban: ok
[15:37] <frankban> bac: you should add that option to the many you used before
[15:39] <bac> frankban: it looks like -a must come last or it tries to interpret the directory as an action
[15:39] <gary_poster> I thought benji meant indigenous and didn't look it up (until now) :-P
[15:40] <benji> yes, I imagine bugs as tribesmen running around in the forest of code
[15:40] <bac> gary_poster: me too,but i gave it the benefit of the doubt and learned something
[15:41] <gary_poster> frankban, agreed that we should check.  We don't need it in setuplxc?  I thought we would
[15:43] <frankban> gary_poster: can't we (in setuplxc, for testing purposes) assume that dnsmasq is not installed?
[15:43] <gary_poster> frankban, if we already connect exclusively via IP.  I didn't think that was the case
[15:46] <frankban> gary_poster: no, we connect using the name, but I thought only dnsmasq-base is required, e.g. in my host I have only that installed and I can see both interfaces (lxcbr0 and virbr0)
[15:48] <frankban> gary_poster: but I will agree if you say: let's do the check to prevent future lxc/dnsmasq/whatever bugs
[15:49] <frankban> gary_poster: we can check if lxcbr0 exists, then fallback to virbr0, and then, if none is found, exit. unfortunately we can do that only after the installation of lxc and libvirt-bin
[15:50] <gary_poster> frankban, ok cool.  so, it would be nice to have but is not urgent, if I understand you correctly.  I'll make a low priority bug; maybe we'll be ready to use lpsetup first
[15:50]  * bac -> long lunch/ride
[15:52] <frankban> gary_poster: yes, and I will create a card for lpsetup
[15:56] <gary_poster> cool frankban
[16:03] <gary_poster> benji, the problem is that -b sets up an fstab that has "$homedir $homedir none bind 0 0" instead of "$homedir /var/lib/lxc/$container/rootfs/$homedir none bind 0 0".  Have you already seen this?  Have you filed a bug, or shall I?  Would you like me to pursue with hallyn, or shall I?
[16:04] <benji> gary_poster: I was coming to the same conclusion.  Feel free to file a bug.  I think I'll step away for lunch now.
[16:06] <gary_poster> cool benji
[16:55] <benji> ok, I'm back
[16:56] <benji> since Gary is at lunch, I assume he's not working on the binding bug, so I'll look at fixing that
[17:08] <frankban> gary_poster: forwhen you're back, could you re-review https://code.launchpad.net/~frankban/launchpad/bug-609986-add-zope.testing-3.9.4-p5/+merge/99536? thanks
[17:24] <gary_poster> sure frankban.  binding bug: you mean, implement hallyn's hix?
[17:24] <gary_poster> well, his workaround?
[17:24] <gary_poster> because I'm sure he is going to fix it
[17:24] <gary_poster> in the lxc stuff
[17:26] <gary_poster> frankban, re-reviewed
[17:27] <benji> gary_poster: I wasn't aware of hallyn's workaround; was that in the bug or in IRC?
[17:27] <gary_poster> benji, https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/968371
[17:27] <_mup_> Bug #968371: lxc-create -b is broken <lxc (Ubuntu):Confirmed> < https://launchpad.net/bugs/968371 >
[17:27] <frankban> thanks gary_poster
[17:27] <gary_poster> welcome
[17:27] <gary_poster> so benji, I'm not sure what to do in terms of automation
[17:28] <gary_poster> because the only thing we can do is come behind and fix the broken fstabs
[17:28] <gary_poster> which should be fixed soon anyway
[17:28] <gary_poster> so implementing that (in bash, no less) seems like a waste
[17:28] <benji> or we could patch lxc-start-ephemeral, but if it's going to be fixed "soon", I would just wait
[17:29] <gary_poster> well, we would need to fix both lxc-start-ephemeral and the ubuntu template
[17:29] <benji> I do think hacking on lxc-start-ephemeral and making sure that everything is working end-to-end is important though, we've had several regressions lately
[17:29] <gary_poster> the ubuntu template appears to be broken, and lxc-start-ephemeral is therefore also broken
[17:29] <benji> in that case we could just wait and do the end-to-end test once the bug is fixed
[17:30] <gary_poster> several?  I'm forgetting.  what beyond this one?
[17:30] <gary_poster> unless lately mean "the past month or so"
[17:30] <gary_poster> in which case I agree
[17:30] <gary_poster> but no others this week, right?
[17:31] <benji> we also had to add --use-http switch
[17:31] <benji> (that was this week)
[17:32] <gary_poster> gotcha
[17:32] <benji> and I removed the code to add the buildbot user to a group because it keeps hooks/install from being re-runnable (but that may not have ever caused a first-run error)
[17:32] <gary_poster> end-to-end test: mmm...well, right.  I mean, you could make a proposal for a change to lxc-start-ephemeral, but you'd want to make sure that hallyn isn't already working on it
[17:32] <benji> and the bug that neccesitated adding it to a group has been fixed
[17:33] <gary_poster> but you couldn't have an end-to-end until the template was fixed
[17:33] <gary_poster> yeah, I don't think the buildbot user to a group thing was a regression
[17:33] <benji> gary_poster: I don't understand that last bit.  I propose we wait until the binding bug is fixed and then we do a juju deploy of a master and slave and make sure the tests run
[17:35] <gary_poster> benji, then I think we should talk. :-)
[17:35] <gary_poster> goldenhorde?
[17:35] <benji> sure
[17:36]  * gary_poster is there
[17:54] <bac> gary_poster, benji: after applying frankban's virbr0 work-around i now have an lxc but it is not mounting my home directory.  if i ssh to it there is nothing there.  on the host, /home/bac/lxc/devel has a LP tree.  any ideas on how to salvage this setup?
[17:54] <benji> bac: yep, that's a known bug, hallyn is working on it/will work on is soon
[17:55] <benji> if he isn't going to get to is very soon we may take it up since it's hurting us so bad
[17:55] <bac> looks like i picked the wrong week to try setuplxc for the first time.
[17:55] <gary_poster> :-/
[17:55] <gary_poster> bac, there's an easy workaround
[17:55] <gary_poster> bac, in your container's fstab...
[17:56] <gary_poster> (/var/lxc/$containername/fstab)
[17:56] <gary_poster> ...modify the line that starts with "/home/bac /home/bac" to start with "/home/bar home/bac"
[17:56] <gary_poster> that is, delete that initial slash in the second entry
[17:57] <bac> and change bac to bar?  :)
[17:58] <bac> ok, then stop and start the container, i guess
[17:58] <gary_poster> bar, not so much :-)
[17:58] <gary_poster> yeah
[17:59] <bac> cool, thanks
[18:02] <gary_poster> welcome.  bug 968371 fwiw bac
[18:02] <_mup_> Bug #968371: lxc-create -b is broken <lxc (Ubuntu):Confirmed> < https://launchpad.net/bugs/968371 >
[18:07] <bac> hmm, perhaps i should've run lxc-stop before monkeying around with the fiddly bits.  now it is hanging
[19:30] <bac> yo gary_poster
[19:32] <gary_poster> bac, hey.  https://talkgadget.google.com/hangouts/extras/canonical.com/goldenhordeoneonone
[20:28] <bac> gary_poster: script now works.  i guess i shit-can the unit tests since there is nowhere for them to live
[20:28] <gary_poster> bac :-(
[20:28] <gary_poster> bac, there will be a place for them in lpsetup
[20:28] <bac> ok, i'll keep a local copy
[20:28] <gary_poster> that can have standalone scripts
[20:29] <gary_poster> maybe send it to yellow, bac?  frankban might want it
[20:29] <bac> ok
[21:23] <gary_poster> Go error handling: :-/ unconvinced and skeptical, but willing to give it a try.  Not something I'd be attracted to myself without the company advocating it.
[21:24] <gary_poster> Well, maybe that's too strong...
[21:24] <gary_poster> but I'm a bit skeptical so far.
[21:24] <gary_poster> ttyl, all