[10:08] <bac>  hi gmb
[10:20] <bac> gmb, the latest drop of juju deprecated some settings in environment.yaml in favor of usng --constraints.  it makes the rewriting o the environments file done by your run_lp_tests script unecessary.  i've made changes to it and to charmhelpers to account for juju changes but haven't completed the changes or pushed them anywhere yet.
[10:21] <bac> i'll do that after breakfast.  just wanted to give you a heads up so you don't duplicate effort.
[10:45] <gmb> bac, Awesome, I was just looking at that.
[10:45] <gmb> I'll let it lie, then.
[10:45] <bac> gmb: great.  i'll get to it post-oatmeal
[12:10] <gary_poster> bac benji frankban gmb, call asap
[12:10] <gmb> Booting FF now.
[12:10]  * bac enhordes
[12:33]  * frankban lunches
[12:33] <bac> i have pushed run_lp_tests branch to lp:~yellow/+junk/run_lp_tests
[12:33] <bac> that allows me to move the card, so the lane is unblocked now
[12:34]  * gmb moves bug 974586 card, blocks the lane again.
[12:34] <_mup_> Bug #974586: bugtracker test logging failures <paralleltest> <Launchpad itself:Triaged> < https://launchpad.net/bugs/974586 >
[12:41]  * bac should read warthogs more.  didn't know poolie was leaving.  :(
[12:55]  * gmb -> lunch
[12:56] <bac> how does one set up PYTHONPATH so that it'll load the code being tested instead of the system version?  i would've expect this approach to work, but the local version (/home/bac/add-charm-helpers/helpers/python) is stuck in the middle and isn't getting loaded.  http://paste.ubuntu.com/924798/plain/
[12:57] <benji> gary_poster: I'm back for the time being teathered to my phone.  I doubt I
[12:58] <benji> can do video confrencing, but other things seem to be ok.
[13:00] <bac> so eggs in /usr/local/lib always get inserted before PYTHONPATH?
[13:00] <gary_poster> cool benji.  replying to jml now.  his branch looks good
[13:01] <gary_poster> bac, that's a policy; maybe a deb one
[13:01] <bac> ergh
[13:02] <benji> gary_poster: will you point me at his branch?
[13:02] <bac> gary_poster: do you know a work-around?  gmb added this to the makefile but it is ineffective:
[13:02] <bac> PYTHONPATH=helpers/python python helpers/python/charmhelpers/tests/test_charmhelpers.py
[13:03] <gary_poster> benji, sorry forgot you did not have IRC history.  #launchpad-dev now fwiw.   https://code.launchpad.net/~jml/testtools/forward-current-tags/+merge/101538
[13:04] <benji> thanks; I'm following along
[13:12] <frankban> bac: if you only need to get rid of the system site packages to test an app locally, I think virtualenv is an easy tool to get that kind of isolation
[13:14] <bac> frankban: but the package in question is the one being tested.  i think requiring people to use virtualenv is a bit heavy handed.  i can achieve the same results by hacking sys.path in the test.  ugly.
[13:16] <frankban> bac: oh, I thought  it was only for your local environment
[13:17] <gary_poster> bac, sorry, was talking on launchpad-dev.  reviewing backlog...
[13:17] <bac> np
[13:18] <gary_poster> bac, system Pythons suck for isolation, sorry.  sys.path hack might be best bet.  -S will leave a clean sys.path initially, but will probably not be what you want
[13:19] <bac> gary_poster: sys.path hack isn't so bad.  mainly i was just surprised to see my PYTHONPATH stuck in the middle.  :(
[13:19] <gary_poster> bac, it's pretty common to make a requirement that virtualenv --whatever-that-option-is be required in these kinds of circumstances I think
[13:19] <gary_poster> yeah
[13:19] <gary_poster> reminds me of dark days trying to figure that stuff out :-P
[13:26] <bac> i've pushed a new branch of charm-tools to lp:~yellow/charm-tools/trunk and have created a merge proposal.
[13:26] <bac> until it is packaged we should try to work off of the ~yellow version
[13:27] <gary_poster> gmb, you up for reviewing lpsetup branch from frankban?
[13:38] <gary_poster> bac, actually, did you see than Sean actually committed code with his fix?
[13:39] <bac> gary_poster: i did not.
[13:39] <gary_poster> I'm tempted to use it, rather than ours, and see if it works
[13:39] <bac> that's good.
[13:39] <gary_poster> bac, but a review from you would be appreciated by him and me
[13:41] <bac> gary_poster: ok.  i'll add comments to the bug since he didn't use a MP
[13:41] <gary_poster> bac, makes sense
[13:42] <bac> i'm a bit partial, but i think my approach was cleaner.  :)
[13:43] <gary_poster> bac, heh, you are probably right :-)
[13:46] <bac> gary_poster: so he has committed code but not made a release.  are you still going to make an interim egg and package it in LP?
[13:51] <gary_poster> bac, yes, that's what I was thinking.
[13:53] <gmb> gary_poster, frankban: Just finishing lunch etc., but I'll be happy to take that review in the next 20 minutes or so.
[13:53] <gary_poster> thx gmb
[13:53] <frankban> thanks gmb
[13:56] <frankban> gary_poster: when you have a minute, I'd like to discuss about lxc and dns servers
[13:58] <gary_poster> ack frankban.  should be available in 3
[13:58] <frankban> thanks gary_poster
[14:00] <gary_poster> frankban, in https://talkgadget.google.com/hangouts/_/extras/canonical.com/goldenhorde & ready whenever you are
[14:10] <gmb> frankban, Is there a diff for your lpsetup changes somwhere?
[14:11] <frankban> gmb: https://code.launchpad.net/~frankban/lpsetup/create-scripts/+merge/101536
[14:11] <gmb> Ta
[14:25] <bac> gmb, frankban: please see my comment on frankban's MP.
[14:26] <gmb> bac, I'd already made the same note :)
[14:26] <bac> okey dokey
[15:03] <frankban> thanks bac, I've just replied
[15:11] <bac> frankban: we need to agree as a group whether our related work will use the LP style guide or not.  IIRC "launchpad related"  projects have continued to use it.  from a ease-of-programming perspective that is what i would advocate, meaning i'd prefer to have one set of guidelines and use it across projects where practical.  the exception to that rule is for projects that will be community-supported and only have a tangential relationship to LP.
[15:14] <gary_poster> bac, frankban, given that lpsetup has PEP 8 everywhere, I'd suggest keeping PEP8 for this MP, with bac raising the issue on the dev list.  Depending on that resolution, we can have a future style-only branch.
[15:14] <gary_poster> for now, let's go with consistency within lpsetup
[15:15] <gary_poster> well, and for the future too; we can switch all at once
[15:17] <bac> frankban, gary_poster: i hadn't reviewed the whole branch and didn't realize that was the case.  our style guide also says "stay consistent with existing code" or something like that.
[15:17] <gary_poster> right bac, cool.  still I agree with your assertion that we should collectively agree
[15:18] <gary_poster> about this kind of policy
[15:18] <gary_poster> bac, if we have not already agreed...?
[15:18] <bac> with open("worms") as can:
[15:18] <gary_poster> heh
[15:18] <bac>    colour = random.choice(palette)
[15:18] <bac>    paint()
[15:19] <frankban> :-)
[15:19] <gmb> There should be a law against people phoning me whilst I'm reviewing.
[15:20] <gary_poster> bac, :-)
[15:30] <gmb> frankban, I'm really sorry, I've been looking at this branch for ages and I can't find anything I don't like.
[15:30] <gmb> r=me
[15:30] <gmb> Apologies.
[15:31] <frankban> gmb: what a shame :-) however thanks for trying
[15:31] <gmb> :)
[15:35] <benji> gary_poster: my connectivity looks like it might be back, but it's intermittent; I'm going to take an early lunch and hope that it's stable by the time I'm done
[15:48] <gary_poster> benji ack good luck
[16:20] <benji> gary_poster: I take it that I should re-implement our worker tagging test result on top of lp:~jml/testtools/forward-current-tags.  Correct?
[16:21] <gary_poster> benji, yes, that's my suggestion.  Please feel free to argue otherwise, as usual
[16:21] <benji> gary_poster: does that mean that it's usual for me to argue?  ;P
[16:22] <gary_poster> benji, :-P no
[16:22] <benji> heh
[16:22] <benji> that looks like a good approach; I'll see where it takes me
[16:25] <gary_poster> cool thx
[16:55] <gmb> gary_poster, It's just been pointed out to me that the first Monday of UDS is a UK national holiday. Is it okay with you if I take the coming Monday (16th) in lieu of it, since I'm going to be swimming the nerd sea on that day?
[16:56] <gary_poster> gmb, sure, though what is the nerd sea? :-)
[16:56] <gmb> gary_poster, The UDS lobby, of course :)
[16:56] <gmb> (I hasten to add, it was Jono Bacon's name for it, not mine :))
[16:57] <gmb> gary_poster, Thank you. I'll add it in CanonicalAdmin now.
[16:57] <gary_poster> lol gmb, cool
[17:00] <gmb> Ahaha.
[17:00]  * gmb accidentally builds LP tree on Precise.
[17:00] <gmb> Hilarity ensues.
[17:00] <gary_poster> in order to work around an ec2 test problem with lightweight checkouts, I have to specify --pqm-public-location.  Because I'm trying to have fun with sed, I end up with the general command...
[17:00] <gary_poster> ec2 test --pqm-public-location=`bzr info | sed -e '/push branch/!d' -e '/push branch/s/\s*push branch: //'`
[17:01] <gary_poster> evilness is fun
[17:01] <gary_poster> so is making a noun into a noun
[17:01] <gmb> Wow.
[17:01] <gary_poster> evilnessitude might be even better
[17:02] <gmb> gary_poster, That's disturbingly like something I would do :)
[17:02] <gary_poster> heh
[17:02] <gary_poster> ok, test started, sed played with...
[17:02] <gary_poster> time for some lunch
[17:03]  * gmb knocks off for the evening. Night all.
[17:05] <benji> night, gmb
[17:27] <gary_poster> night gmb.  approved Monday.
[17:50] <bac> gary_poster, with an 8 cpu ec2 machine, should we expect a slave to come up within an hour?
[17:51] <gary_poster> bac, give or take.  two hours is usually too long.  You should be able to look at the juju log on the slave instance to see the output of setuplxc
[18:19] <gary_poster> bac, I self-approved this, but in case you are curious: https://code.launchpad.net/~gary/launchpad/trivial/+merge/101619
[18:19] <gary_poster> tests have been running for a couple of hours now.
[18:20] <gary_poster> bac, I'd like to get a parallel test run or two or three to see how we are doing.  Did you get one running?
[18:21] <bac> memcache change looks good.  thanks for the thorough documenting
[18:21] <bac> gary_poster: i just started one a little bit ago and am waiting for the slave to come up
[18:22] <bac> i think gmb's script is working now it was just too aggressive in only giving the slave 1 hour to come up
[18:22] <gary_poster> bac, memcache: cool, thanks.  waiting for slave: great.  Please lemme know when tests have run: I'd like you to collect the output of testr failing after a test run.
[18:22] <bac> gary_poster: will do
[18:23] <gary_poster> thanks
[18:23] <bac> sadly i don't think my run got your memcache fix
[18:26] <gary_poster> no, it won't
[18:57] <bac> gary_poster: my attempt failed, at least the automatic part.
[18:57] <gary_poster> bac, :-/ how so?
[18:57] <bac> while waiting for the slave to come up, a call to 'juju status' got
[18:58] <bac> CalledProcessError: Command '['juju', 'status']' returned non-zero exit status 1
[18:58] <bac> 2012-04-11 14:47:17,862 INFO Connecting to environment...
[18:58] <bac> 2012-04-11 14:47:19,807 INFO Connected to environment.
[18:58] <bac> [('SSL routines', 'SSL3_READ_BYTES', 'ssl handshake failure')]
[18:58] <bac> 2012-04-11 14:47:22,699 ERROR [('SSL routines', 'SSL3_READ_BYTES', 'ssl handshake failure')]
[18:58] <bac> this is not uncommon
[18:58] <bac> but, i can manually wait for the slave to come up and force the build by hand
[19:00] <bac> it has been up 66 minutes so perhaps it'll be happy soon
[19:02] <gary_poster> hm
[19:02] <gary_poster> ok
[19:14] <bac> ok, setuplxc is done.  took about 80 minutes
[19:19] <bac> gary_poster: you ever see juju status report 'relation-errors'?
[19:19] <bac>         relation-errors:
[19:19] <bac>           buildbot:
[19:19] <bac>           - buildbot-slave
[19:20] <gary_poster> bac, rarely, yes
[19:20] <bac> well it is my lucky day
[19:20] <gary_poster> trying to think of something helpful....
[19:20] <gary_poster> bac, usually the logs are the place I go
[19:21] <bac> yeah, i see some stuff in charm.log
[19:22] <bac> nothing too informative, though
[19:23] <gary_poster> bac, also try master's log
[19:23] <gary_poster> /var/lib/buildbot/masters/master/twistd.log
[19:25] <bac> hmm, at :06 the bb master received SIGTERM and shut itself down
[19:25] <bac> i don't think the slave was even up yet
[19:27] <bac> i was able to restart the master by hand
[19:32] <bac> and manually started a slave.  go juju go.
[19:38] <bac> gary_poster: my hand starting stuff did work.  it looks like the slaves started as the wrong user.
[19:38] <bac>  /usr/bin/xvfb-run: 181: /var/lib/buildbot/slaves/slave/lucid-devel/build/bin/test: Permission denied
[19:39] <bac> which confuses me as the log shows:
[19:39] <bac> sudo /usr/local/bin/launchpad-lxc-test -t stories/gpg
[19:50] <gary_poster> bac, the permission denied with xvfb-run usually indicates a bad command passed to xvfb-run
[19:50] <gary_poster> caused by crazy bash escaping
[19:50] <gary_poster> we've had careful escaping to try and prevent this
[19:50] <bac> gary_poster: such as where?
[19:51] <gary_poster> so, the trigger for the problem is that we are trying to pass an argument with spaces to xvfb-run's -a argument IIRC
[19:51] <bac> gary_poster: i'm using stock yellow charms.  are you suggesting it is an error in one of the hooks?
[19:52] <gary_poster> bac, no, the problem is arguably in ssh, but we are not fixing that today. :-P .  Today's breakage of the workaround we implemented is in one of these:
[19:52] <gary_poster> launchpad-lxc-test
[19:52] <gary_poster> or lxc-start-ephemeral
[19:52] <gary_poster> I don't think there would be another
[19:54] <gary_poster> bac, to see a related behavior, change lxc-start-ephemeral to begin with set -ex (IIRC)
[19:55] <gary_poster> that will spit out the commands lxc-start-ephemeral sends
[19:55] <gary_poster> bac, could you give me the url of your master
[20:00] <bac> gary_poster: looking
[20:01] <bac> gary_poster: ec2-23-21-33-127.compute-1.amazonaws.com:8010
[20:17] <gary_poster> bac, ack.  Maybe we should pair on getting a test run manually tomorrow.  If we run into any problems, you can get a feel for where the bodies are buried
[20:17] <bac> gary_poster: that's a good idea
[20:17] <bac> if this instance is no good to you i'll tear it down now
[20:21] <gary_poster> bac yeah tear down
[20:21] <gary_poster> sorry for frustrations (and slow turn around; other stuff going on too)
[20:22] <bac> ok
[20:48] <bac> gary_poster: sean says he'll push a new py-memcached tonight
[20:48] <gary_poster> bac, heh, ok cool
[20:49] <bac> twas the right decision
[20:49] <gary_poster> cool bac
[20:49] <bac> besides we haven't seen it yet
[20:53] <gary_poster> :-) true
[20:54] <gary_poster> I also haven't seen my testresults yet...