[11:23] <benji> I think I have termbeamer installable from a PPA now.  Testing would be appreciated.
[11:43] <bac> benji: cool
[11:44] <benji> bac: sudo apt-add-repository ppa:benji/termbeamer; sudo apt-get install termbeamer if you want to try it out
[11:44] <bac> ah, you forgot sudo apt-get update!  :)
[11:45] <gary_poster> hey bac, did you get what you needed from that ec2 instance?  I have some worker lists from it if you haven't that I could save.  I want to restart the instance and see if the changes to my charm work
[11:45] <bac> gary_poster: please save.  i need more data to chase a theory
[11:45] <gary_poster> cool
[11:46] <frankban> gary_poster: http://ec2-23-21-26-159.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/1/steps/shell_8/logs/summary the first failure seems a new one, the second should be 996729
[11:46] <frankban> gary_poster: and thanks for the review
[11:46] <frankban> s
[11:47] <gary_poster> frankban, the first one is also known, and Brad has a simple egg I was supposed to test to address it (simply increasing a timeout)
[11:47] <gary_poster> That said, frankban, I hadn't filed a bug because it was tied to some previous work Brad had done
[11:47] <gary_poster> that was probably a mistake
[11:47] <gary_poster> so if you file a bug, Brad or I can take it
[11:48] <frankban> gary_poster: oh, ok
[11:48] <benji> bac: I did forget the update.  Once it's installed try "tb --help"
[11:48] <frankban> gary_poster: do you need the worker list?
[11:48] <gary_poster> reviews: you're welcome of course--thanks for that work
[11:48] <gary_poster> frankban, nah thanks
[11:50] <frankban> gary_poster: https://bugs.launchpad.net/launchpad/+bug/1002820
[11:50] <_mup_> Bug #1002820: lp.testing.layers.RabbitMQLayer:setUp times out rarely/intermittently in parallel tests <paralleltest> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1002820 >
[11:51] <frankban> adding it to the board
[11:51] <gary_poster> perfect thanks frankban
[11:51]  * benji rushes to make coffee before the call.
[11:54] <bac> benji: what happens if i have two-level auth turned on for my gmail account?
[11:54] <bac> benji: and i do.  :)
[11:56] <benji> bac: me too, you have to create an application password: support.google.com/accounts/bin/answer.py?answer=185833
[11:57] <bac> great
[11:59] <bac> benji: hey that appears to work.  i did get a traceback: http://paste.ubuntu.com/1000684/
[12:01] <benji> bac: that's caused by not having my fixed python-vte package, which I made a PPA dependency but I guess that doesn't work the way I think it works; add the benji/vte-fixes ppa and do the update/upgrade dance and you should be good to go
[12:02] <benji> the best part of being home is that I can reliably make a decent cup of coffee
[12:02] <gary_poster> benji, which home is this now? :-)
[12:02] <benji> gary_poster: heh, VA
[12:03] <gary_poster> how/when did you get there?
[12:03] <gary_poster> I guess when is the interesting part, assuming a teleporter wasn't involved
[12:03] <bac> benji: you don't travel with your own stash of coffee?
[12:04] <benji> how: Dodge Caravan, when: Sunday at about 8:30 pm local time
[12:04] <gary_poster> ah!  I thought you were still in TN yesterday
[12:04] <bac> benji: we now have a clean launch.  just like SpaceX.
[12:05] <benji> bac: heh, maybe I should from now on; although my dad has one of those keurig one-cup-at-a-time things, they're pretty good
[12:05] <gary_poster> frankban, bac, you are the designated default maintainers for our buildbot-slave and buildbot-master charms, respectively, because you had the most commits.  Would you object if I try to see if we can make yellow the maintainer for both of those?
[12:06] <bac> nope
[12:06] <benji> bac: maybe after the call we can do a tb test run
[12:07] <bac> i'll prepare the iron lung
[12:07] <bac> or was that polio?
[12:07] <frankban> gary_poster: no objections
[12:07] <bac> hey you've got a name for your follow on project
[12:08] <benji> lol
[12:08] <gary_poster> lol
[12:08] <gary_poster> thanks bac & frankban
[12:08] <gary_poster> bac benji frankban gmb call in 2
[12:08] <gary_poster> I'll get us a room, as it were
[12:08] <gmb> k
[12:09] <benji> (I think it was polio.  People with TB just died.)
[12:10] <gary_poster> https://talkgadget.google.com/hangouts/_/ca56b39f7ec51a3a743513e476a5512a15a0253e
[12:11] <gary_poster> gmb yoohoo
[12:11] <gmb> gary_poster, Firefox in quitting hilarity.
[12:11] <gmb> Bear with me.
[12:12] <gary_poster> :-P
[12:12] <gary_poster> cool
[12:26] <benji> bac: want to try termbeamer?
[12:26] <bac> oh sure
[12:26] <bac> i am brad.crittenden@gmail
[12:27] <bac> benji: what is your xmpp?
[12:27] <benji> bac: you've been invited to chat (approving the other person is a one-time first step that we have to do)
[12:27] <benji> benji.york@gmail
[12:28] <bac> so i just did --xmpp-receive
[12:28] <bac> and nothing is happening
[12:29] <bac> tb --xmpp-user=brad.crittenden@gmail.com --xmpp-password=XXX --xmpp-receive
[12:29] <bac> is that right benji?
[12:29] <benji> bac: did you go to gmail and approve my request to chat?
[12:29] <benji> bac: yep, it looks good
[12:29] <gmb> Wups. Better remove that sweary fake test from the LP tree before I commit...
[12:30] <bac> benji: go where?  should i receive an email?
[12:30] <benji> bac: the gmail UI should prompt you
[12:31] <bac> er, cc.booty2?
[12:31] <bac> ok, i authed that.  still see nothing
[12:32]  * bac -> tea
[12:33] <benji> bac: I invited the wrong you to chat, look for another auth message
[12:34] <gary_poster> bac, workers 0-7 of r15280 on parallel buildbot: http://pastebin.ubuntu.com/1000718/ http://pastebin.ubuntu.com/1000719/ http://pastebin.ubuntu.com/1000720/ http://pastebin.ubuntu.com/1000722/ http://pastebin.ubuntu.com/1000724/ http://pastebin.ubuntu.com/1000727/ http://pastebin.ubuntu.com/1000728/ http://pastebin.ubuntu.com/1000729/
[12:35] <gary_poster> argh
[12:35] <gary_poster> we don't have 15280 on lpbuildbot >:-(
[12:35]  * gmb lunches
[12:38] <gary_poster> I'll make a new set
[12:40] <bac> thanks gary_poster
[12:40] <gary_poster> welcome
[12:40] <bac> benji: done
[12:40] <bac> benji: i can see you
[12:40] <benji> yay!
[12:41] <benji> bac: watch this...
[12:41] <bac> ok
[12:41] <bac> what/
[12:41] <benji> did your window get smaller when I resied mine?
[12:41] <bac> no
[12:41] <benji> darn
[12:41] <bac> oh, i made my window big
[12:42] <benji> bac: oh! I forgot to get you to use the --gui-client switch
[12:42] <benji> restart with that
[12:42] <bac> done
[12:43] <bac> yes, smallerized
[12:43] <benji> :)
[12:43] <benji> cool
[12:43] <bac> yes, very neat
[12:43] <bac> can i do anything or just watch?
[12:43] <gary_poster> watch the blinken lights
[12:43] <benji> bac: right now just watch, the very next thing I want to add is client to server communication
[12:44] <bac> that will be cooler but this is very neat indeed
[12:44] <benji> bac: did you see the flashing crosshair?
[12:44] <bac> yes
[12:45] <benji> that's the first client-to-server thing I want to add, so we can both point things out
[12:46] <benji> bac: do these colors look sane?
[12:46] <bac> yes.
[12:47] <bac> the italics and underline2 didn't do anything
[12:47] <benji> bac: yeah, most terminals don't support that (including the terminal widget I'm using)
[12:48] <bac> benji: also note i'm running this in an ssh session from X on OS X to precise and it still works.
[12:48] <benji> it does support strikethough, but I don't bother sending that data, strikethrough seems rare
[12:48] <benji> very cool
[12:48] <benji> bac: do you want to try being the sender?
[12:49] <bac> ok, sure
[12:49] <bac> benji: do i use --gui-client?
[12:49] <benji> bac: nope
[12:49] <benji> the server is gui-only so there's no switch for that
[12:50] <benji> bac: I see you
[12:50] <benji> bac: heh; try your console editor of choice
[12:50] <benji> you and your white background
[12:51] <bac> how do i point?
[12:51] <benji> control-click
[12:51] <benji> bac: looks good
[12:52] <bac> oh, you can see the crosshair?
[12:52] <benji> yep
[12:52] <bac> i don't see anything when i control click
[12:52] <benji> I need to make the server show it too for feedback
[12:52]  * benji adds to the TODO
[12:53] <bac> well that's a smashing success
[12:54] <benji> cool
[12:55] <gary_poster> bac db-devel r11611.  parallel tests report 16718 tests, lpbuildbot reports 16761 tests (https://lpbuildbot.canonical.com/builders/lucid_db_lp/builds/1979).  parallel buildbot test worker lists follow, 0-7: http://pastebin.ubuntu.com/1000745/ http://pastebin.ubuntu.com/1000746/ http://pastebin.ubuntu.com/1000747/ http://pastebin.ubuntu.com/1000749/ http://pastebin.ubuntu.com/1000750/ http://pastebin.ubuntu.com/10
[12:55] <gary_poster> 00753/ http://pastebin.ubuntu.com/1000756/ http://pastebin.ubuntu.com/1000757/ .  If I add the line numbers from those eight lists, I get 16718, so it seems to represent what the parallel test buildbot is counting.  Also frankban, those test lists are perfectly round-robin in counts (every list is either 2089 or 2090) which is good...except odd because it was not a first test run. :-/  Anyway, bac, that looks lik
[12:55] <gary_poster> e something you can use.
[12:55] <gary_poster> I'm going to shut down the instance and restart now,
[12:55] <gary_poster> .
[12:57] <bac> gary_poster: great.  that blows my theory, though.  had parallel shown more tests then i had a possible explanation
[13:03] <gary_poster> :-/ k
[13:05] <gary_poster> oh, bac, did you see my idea on presenting about parallel testing to that pythoncarolina thing in CH?  Not sure if it fits your timeline
[13:15] <bac> gary_poster: no
[13:15]  * gary_poster looks for it
[13:15] <bac> gary_poster: found it
[13:16] <bac> that sounds interesting
[13:16] <gary_poster> cool
[13:16] <bac> gary_poster: if i'm not around you can just wing it.  :)
[13:16] <gary_poster> :-)
[13:17] <bac> gary_poster: so i ran into a guy at RDU wearing an Open Stack t-shirt.  he said "yeah, i work for this company called 'Canonical'"...  odd
[13:21] <gary_poster> bac, weird.  https://directory.canonical.com/list/state/NC/ ... bfox or bladernr?
[13:22] <bac> gary_poster: no, i know those guys.  some new guy named adam stokes who doesn't have his address in the directory correctly
[13:22] <gary_poster> huh
[13:22] <gary_poster> cool,  I guess
[13:22] <bac> and he pronounced it caneenical
[13:25] <benji> heh
[13:26] <benji> that's the little-used Knights Who Say Caneeenical pronunciation
[13:27] <frankban> wow, DEBUG = True in canonicaladmin: https://directory.canonical.com/NOOOO
[13:27] <benji> frankban: that is hilarious
[13:27] <frankban> gary_poster: I've moved the card to done-done, AFAICT testrepository is doing the right thing
[13:28] <bac> frankban: well, at least that is only on the directory not canonicaladmin
[13:28] <frankban> bac: yes
[13:29] <benji> https://directory.canonical.com/orgchart/ takes a while to respond :)
[13:33] <benji> frankban: we should report that to whomever is responsible (I don't know who that is)
[13:36] <frankban> benji: indeed (me either)
[13:43] <bac> benji, frankban: probably stuartm
[14:16] <gary_poster> frankban, awesome thanks
[14:20] <bac> gary_poster: we have some tests (mainly doctests i think) that get run multiple times.  lpbuildbot is reporting them individually where our parallel test output only lists them once.  summary: http://paste.ubuntu.com/1000909/
[14:20] <gary_poster> bac, should we be paranoid that these tests are not actually being run multiple times as they should be?
[14:21] <gary_poster> concerned
[14:21] <bac> gary_poster: i'm looking at the data collection code now.  i suspect it is a hash indexed by test name which throws away multiples
[14:22] <gary_poster> bac, that would make sense, except that we have seen some duplicates, like that api test from, um, some bug we had
[14:22] <gary_poster> benji, I see you are working on the rabbitmq bug
[14:23] <gary_poster> did you get the egg from bac already?
[14:23] <gary_poster> back in a sec
[14:24] <benji> gary_poster: I feel like I should know what you're talking about because it was discussed this morning.
[14:24] <bac> benji: last week i produced a rabbit-fixture egg that set absurdly high timeouts
[14:24] <benji> and those timeouts were still being reached?
[14:25] <bac> benji: no, gary never used it
[14:25] <bac> benji: and now i can't find it.  :(
[14:26] <bac> benji: found
[14:27] <benji> what is the intended use of that egg?
[14:28] <bac> benji: to run a novel branch of launchpad that uses that egg under parallel testing to see if the timeouts go away
[14:28] <gary_poster> bac, benji, this is intermittent.
[14:29] <benji> bac: are the timeouts frequent enough that doing that would tell us anything?
[14:29] <gary_poster> therefore, I suggest that we may need to actually land use of that egg
[14:29] <bac> benji: it is rabbitfixture 0.3.3-bac-exp and is now in download-cache/dist
[14:29] <gary_poster> and see if it goes away.
[14:29] <gary_poster> We've seen it twice in about 30 or 40 runs, I'd guess
[14:29] <benji> Meta: I think there is room for process improvment here.  I feel like these things should have been written down in the bug.
[14:30] <benji> yeah, either reproducing it or dramatically upping the timeouts seem like the only two rational approaches
[14:31] <benji> bac: did you try to reproduce the timeout?
[14:33] <gary_poster> benji, process improvement: Gary should have made a bug.  It was kind of a follow-on to existing work, so I didn't. Lesson: always make a bug.  It is not written in the bug because it is a placeholder that I asked Francesco to make. Ideally I would have at least added a "talk to me or Brad" to the bug.
[14:34] <gary_poster> We did discuss it in this morning's meeting, but again, Gary should have written it down.  Lesson: always write stuff down in the proper place.
[14:34] <gary_poster> I know those lessons, just didn't do 'em.
[14:34] <gary_poster> Lesson: kick Gary.
[14:35] <gary_poster> benji, I am pretty sure bac did not try to repro this timeout.  He had reproed an earlier one to determine that there was a bug in the egg.
[14:35] <gary_poster> Someone had already fixed the code
[14:35] <bac> and if we are to make an egg that is referenced by launchpad in production then it needs to be a full release of rabbitfixture and the code pushed
[14:35] <gary_poster> but not released it
[14:35] <bac> gary_poster: yes, but that 'fix' was not sufficient
[14:35] <bac> benji: no, i have not tried to reproduce with the new experimental egg
[14:36] <gary_poster> simply because the timeout was not enough for what we need, as we believe
[14:36] <bac> yes.  it would be the proper fix for a non-parallel environment
[14:36] <benji> so I should make a "full release" of rabbitfixture and add that to LP, right?
[14:37] <gary_poster> Yeah, that's my suggestion.  I'm not sure what Brad means by a full release
[14:37] <gary_poster> bac, do you think we should merge this?  I was thinking we ought to have a fork, and maybe file a bug to remove the fork and merge to trunk once we get 40 or so clean runs
[14:37] <bac> gary, i mean 0.3.4 instead of 0.3.3.-exp
[14:38] <gary_poster> gotcha, then I understood...
[14:38] <bac> gary_poster: a fork is fine
[14:38] <gary_poster> ok
[14:38] <gary_poster> benji ^^
[14:38] <bac> gary_poster: just don't do it via egg drops
[14:38] <gary_poster> gotcha bac, agreed
[14:38] <bac> per your previous edict
[14:38] <gary_poster> i'm all about my edicts
[14:38] <bac> i prefer hot'n'sour over egg drop
[14:38] <benji> revision: I will fork rabbitfixture, make an egg, put that in the dependencies, and land an LP branch that uses that version
[14:39] <gary_poster> go thou! go forth! nay, stop that villainy!
[14:39] <bac> benji: do you want my code?
[14:39] <benji> bac: sure (any reason we don't just use your egg as-is?)
[14:39] <gary_poster> because it is an egg drop
[14:39] <gary_poster> not a branch
[14:39] <benji> ah
[14:39] <benji> I missed that bit.
[14:40] <frankban> aaargh gary_poster, I've written buildout instead of buildbot in the dependencies dir (slave charm) can I r=you the fix? btw, we should call the next project "buildnot"
[14:41] <benji> what PPAs are to debs X are to eggs (we need an X)
[14:41] <bac> benji: see lp:~/rabbitfixture/timeout-exp
[14:41] <benji> "buildnot"!
[14:41] <bac> benji: review the changes as you may not like them.  it is a mix of just upping the timeout and trying to parameterize it
[14:42] <bac> buildsnot
[14:42] <benji> bac: ok, thanks
[14:42] <benji> pfft
[14:54] <gary_poster> frankban, sorry, of course
[14:55] <gary_poster> this error looks really familiar... http://ec2-184-72-171-77.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/0/steps/shell_8/logs/summary
[14:55] <gary_poster> anyone recognize it?  Looking for it...
[14:56] <benji> gary_poster: that looks similar to one Brad and I fixed
[14:56] <gary_poster> bug 993510
[14:56] <_mup_> Bug #993510: lp.services.job.tests.test_runner.TestJobRunner.test_runJobHandleErrors_oops_generated_user_notify_fails fails intermittently/rarely in parallel tests <paralleltest> <Launchpad itself:Triaged> < https://launchpad.net/bugs/993510 >
[14:56] <gary_poster> bug 992692
[14:56] <_mup_> Bug #992692: lp.services.mail.tests.test_incoming.TestIncoming.test_invalid_to_addresses fails intermittenty/rarely in parallel tests <paralleltest> <qa-untestable> <Launchpad itself:Fix Released by benji> < https://launchpad.net/bugs/992692 >
[14:57] <bac> gary_poster: do you have the full, unfiltered subunit output for the db-devel r 11611 run?
[14:57] <benji> gary_poster: 992692 is the one I was thinking of
[14:57] <gary_poster> bac, no, sorry.  If that's necessary then I have another machine running tests now, but you'll have to start the analysis over again
[14:57] <gary_poster> :-(
[14:57] <bac> gary_poster: it's all scripted
[14:58] <bac> so it is no thing
[14:58] <bac> also, to save you work i prefer the full subunit output not the ones divided by worker
[14:58] <gary_poster> ok cool bac, got it
[14:58] <bac> gary_poster: eta for completion?
[15:30] <gary_poster> bac, 5 min.
[15:31] <gary_poster> We are getting these somewhat frequently
[15:31] <gary_poster> http://ec2-184-72-171-77.compute-1.amazonaws.com:8010/builders/lucid_db_lp/builds/0/steps/shell_6/logs/stdio
[15:31] <gary_poster> Don't quite understand what is going on
[15:31] <gary_poster> I think it happens when we switch between devel and db-devel
[15:31] <gary_poster> but could be wrong
[15:33] <gary_poster> bac, maybe you can just download this  http://ec2-184-72-171-77.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/0/steps/shell_8/logs/subunit/text
[15:33] <bac> ok
[16:10] <frankban> gmb, am I right if I say that you can create a co-located branch from a normal one (residing in a shared repo) with "bzr switch -b foo"?
[16:13] <gmb> frankban: I have no idea, I've never tried that :). Sounds sane, though.
[16:15] <frankban> gmb: oh, because you started using colo-branches with the plugin I guess
[16:15] <gary_poster> frankban, try it :-)
[16:15] <gmb> yep
[16:15] <frankban> ok
[16:26] <gary_poster> it's kind of sad that I really need to use Google to search Launchpad's buugs
[17:13] <bac> yay, i have water again!
[17:21] <gary_poster> yay!
[17:21] <gary_poster> water good
[17:22] <bac> there is a big capacitor that is used at start up to give the pump a good jolt.  it was nasty and the wires fell out when touched.
[17:22] <bac> i heart my redneck plumber
[17:23] <bac> gary_poster:  do you know what revno was used for the run referenced above?
[17:23] <gary_poster> heh
[17:24] <gary_poster> yeah I can get it
[17:24] <gary_poster> bac, 15282 (see http://ec2-184-72-171-77.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/0/)
[17:24] <bac> got it
[17:29] <gary_poster> frankban, was colocation as easy as it sounded?
[17:29] <frankban> gary_poster: yes
[17:29] <gary_poster> frankban, cool!  I'll have to try.  Sounds like the advantages of lightweight checkouts but simpler to use
[17:32] <frankban> gary_poster: yes it's cool, and actually I think each colocated branch is internally a lightweight checkout of the current branch
[17:32] <gary_poster> makes sense
[18:04] <bac> gary_poster: i think the duplicate tests are not being run the required number of times.  each only shows up once in the unified subunit output from the latest run.  the test data i had from yesterday (for r 15139) shows multiple lines for the repeated tests.
[18:05] <gary_poster> bac, oh, so maybe a regression with the newer version of the testr and friends packages.  Thank you for investigating.
[18:06] <bac> gary_poster: yeah, i think so.  what was the date of the 15139 run?
[18:06] <bac> nm
[18:06] <gary_poster> bac, to call this finished, then, we should file a bug.  We can throw it at something like testr and see if it sticks (if Robert will investigate) but +1 if you are willing and interested to take it farther
[18:06] <bac> april 23
[18:07] <gary_poster> right, exactly
[18:07] <bac> ok, i didn't realize it was that old
[18:25] <bac> gary_poster: dumb question, how can i do the equivalent of 'bin/test -t faqcollection.txt' with testr?
[18:29] <bac> ok, so this appears to be what i was after:
[18:29] <bac> bin/test --subunit -t faqcollection.txt |testr load
[18:29] <bac> duh
[18:31] <gary_poster> good to know
[19:09] <gary_poster> bad stats today :-/
[20:12] <bac> gary_poster: have time for a chat?
[20:13] <gary_poster> bac, supposed to be talking to flacoste, but not, so sure
[20:13] <bac> you want to make a hangout
[20:13] <gary_poster> yeah ok
[20:13] <bac> since whatever i did yesterday failed
[20:14] <gary_poster> bac https://talkgadget.google.com/hangouts/_/49af6cb2474cf14af2cb37afc56f73c7c5fc120a
[20:56] <bac> gary_poster: i've been doing some standalone (i.e. not buildbot) tests with testr --parallel
[20:56] <bac> if i run "testr run --parallel -- -t faqcollection.txt" then faqcollection.txt appears once in the test partitions but gets run three times as seen in e.g. .testrepository/1
[20:57] <bac> looking at the code i don't see any use of sets or other attempts to make the test ids unique
[20:58] <gary_poster> bac, dunno
[20:59] <gary_poster> bac, also, when you run them non-parallel, are they in different layers?
[20:59] <bac> so i remain a bit confused.  the subunit output from the test you showed me today only had the suspect tests listed once
[20:59] <bac> no
[20:59] <gary_poster> or more generally
[21:00] <gary_poster> how are they different?  are we sure we are running the, differently even when we see them multiple times across workers?
[21:00] <bac> but, i think i may have a different version of testrepository installed locally due to the funky ppa naming
[21:00] <gary_poster> oh
[21:00] <bac> let me try to get the same version and see what happens
[21:00] <gary_poster> maybe
[21:00] <gary_poster> ok
[21:01] <gary_poster> "the," was supposed to be "them"
[21:01] <gary_poster> bac I have to run
[21:01] <bac> ok, have fun.  wear a helmet if you go out
[21:01] <gary_poster> will be back later fwiw
[21:01] <gary_poster> heh thx
[21:01] <bac> hail hurts