[11:23] I think I have termbeamer installable from a PPA now. Testing would be appreciated. [11:43] benji: cool [11:44] bac: sudo apt-add-repository ppa:benji/termbeamer; sudo apt-get install termbeamer if you want to try it out [11:44] ah, you forgot sudo apt-get update! :) [11:45] hey bac, did you get what you needed from that ec2 instance? I have some worker lists from it if you haven't that I could save. I want to restart the instance and see if the changes to my charm work [11:45] gary_poster: please save. i need more data to chase a theory [11:45] cool [11:46] gary_poster: http://ec2-23-21-26-159.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/1/steps/shell_8/logs/summary the first failure seems a new one, the second should be 996729 [11:46] gary_poster: and thanks for the review [11:46] s [11:47] frankban, the first one is also known, and Brad has a simple egg I was supposed to test to address it (simply increasing a timeout) [11:47] That said, frankban, I hadn't filed a bug because it was tied to some previous work Brad had done [11:47] that was probably a mistake [11:47] so if you file a bug, Brad or I can take it [11:48] gary_poster: oh, ok [11:48] bac: I did forget the update. Once it's installed try "tb --help" [11:48] gary_poster: do you need the worker list? [11:48] reviews: you're welcome of course--thanks for that work [11:48] frankban, nah thanks [11:50] gary_poster: https://bugs.launchpad.net/launchpad/+bug/1002820 [11:50] <_mup_> Bug #1002820: lp.testing.layers.RabbitMQLayer:setUp times out rarely/intermittently in parallel tests < https://launchpad.net/bugs/1002820 > [11:51] adding it to the board [11:51] perfect thanks frankban [11:51] * benji rushes to make coffee before the call. [11:54] benji: what happens if i have two-level auth turned on for my gmail account? [11:54] benji: and i do. :) [11:56] bac: me too, you have to create an application password: support.google.com/accounts/bin/answer.py?answer=185833 [11:57] great [11:59] benji: hey that appears to work. i did get a traceback: http://paste.ubuntu.com/1000684/ [12:01] bac: that's caused by not having my fixed python-vte package, which I made a PPA dependency but I guess that doesn't work the way I think it works; add the benji/vte-fixes ppa and do the update/upgrade dance and you should be good to go [12:02] the best part of being home is that I can reliably make a decent cup of coffee [12:02] benji, which home is this now? :-) [12:02] gary_poster: heh, VA [12:03] how/when did you get there? [12:03] I guess when is the interesting part, assuming a teleporter wasn't involved [12:03] benji: you don't travel with your own stash of coffee? [12:04] how: Dodge Caravan, when: Sunday at about 8:30 pm local time [12:04] ah! I thought you were still in TN yesterday [12:04] benji: we now have a clean launch. just like SpaceX. [12:05] bac: heh, maybe I should from now on; although my dad has one of those keurig one-cup-at-a-time things, they're pretty good [12:05] frankban, bac, you are the designated default maintainers for our buildbot-slave and buildbot-master charms, respectively, because you had the most commits. Would you object if I try to see if we can make yellow the maintainer for both of those? [12:06] nope [12:06] bac: maybe after the call we can do a tb test run [12:07] i'll prepare the iron lung [12:07] or was that polio? [12:07] gary_poster: no objections [12:07] hey you've got a name for your follow on project [12:08] lol [12:08] lol [12:08] thanks bac & frankban [12:08] bac benji frankban gmb call in 2 [12:08] I'll get us a room, as it were [12:08] k [12:09] (I think it was polio. People with TB just died.) [12:10] https://talkgadget.google.com/hangouts/_/ca56b39f7ec51a3a743513e476a5512a15a0253e [12:11] gmb yoohoo [12:11] gary_poster, Firefox in quitting hilarity. [12:11] Bear with me. [12:12] :-P [12:12] cool [12:26] bac: want to try termbeamer? [12:26] oh sure [12:26] i am brad.crittenden@gmail [12:27] benji: what is your xmpp? [12:27] bac: you've been invited to chat (approving the other person is a one-time first step that we have to do) [12:27] benji.york@gmail [12:28] so i just did --xmpp-receive [12:28] and nothing is happening [12:29] tb --xmpp-user=brad.crittenden@gmail.com --xmpp-password=XXX --xmpp-receive [12:29] is that right benji? [12:29] bac: did you go to gmail and approve my request to chat? [12:29] bac: yep, it looks good [12:29] Wups. Better remove that sweary fake test from the LP tree before I commit... [12:30] benji: go where? should i receive an email? [12:30] bac: the gmail UI should prompt you [12:31] er, cc.booty2? [12:31] ok, i authed that. still see nothing [12:32] * bac -> tea [12:33] bac: I invited the wrong you to chat, look for another auth message [12:34] bac, workers 0-7 of r15280 on parallel buildbot: http://pastebin.ubuntu.com/1000718/ http://pastebin.ubuntu.com/1000719/ http://pastebin.ubuntu.com/1000720/ http://pastebin.ubuntu.com/1000722/ http://pastebin.ubuntu.com/1000724/ http://pastebin.ubuntu.com/1000727/ http://pastebin.ubuntu.com/1000728/ http://pastebin.ubuntu.com/1000729/ [12:35] argh [12:35] we don't have 15280 on lpbuildbot >:-( [12:35] * gmb lunches [12:38] I'll make a new set [12:40] thanks gary_poster [12:40] welcome [12:40] benji: done [12:40] benji: i can see you [12:40] yay! [12:41] bac: watch this... [12:41] ok [12:41] what/ [12:41] did your window get smaller when I resied mine? [12:41] no [12:41] darn [12:41] oh, i made my window big [12:42] bac: oh! I forgot to get you to use the --gui-client switch [12:42] restart with that [12:42] done [12:43] yes, smallerized [12:43] :) [12:43] cool [12:43] yes, very neat [12:43] can i do anything or just watch? [12:43] watch the blinken lights [12:43] bac: right now just watch, the very next thing I want to add is client to server communication [12:44] that will be cooler but this is very neat indeed [12:44] bac: did you see the flashing crosshair? [12:44] yes [12:45] that's the first client-to-server thing I want to add, so we can both point things out [12:46] bac: do these colors look sane? [12:46] yes. [12:47] the italics and underline2 didn't do anything [12:47] bac: yeah, most terminals don't support that (including the terminal widget I'm using) [12:48] benji: also note i'm running this in an ssh session from X on OS X to precise and it still works. [12:48] it does support strikethough, but I don't bother sending that data, strikethrough seems rare [12:48] very cool [12:48] bac: do you want to try being the sender? [12:49] ok, sure [12:49] benji: do i use --gui-client? [12:49] bac: nope [12:49] the server is gui-only so there's no switch for that [12:50] bac: I see you [12:50] bac: heh; try your console editor of choice [12:50] you and your white background [12:51] how do i point? [12:51] control-click [12:51] bac: looks good [12:52] oh, you can see the crosshair? [12:52] yep [12:52] i don't see anything when i control click [12:52] I need to make the server show it too for feedback [12:52] * benji adds to the TODO [12:53] well that's a smashing success [12:54] cool [12:55] bac db-devel r11611. parallel tests report 16718 tests, lpbuildbot reports 16761 tests (https://lpbuildbot.canonical.com/builders/lucid_db_lp/builds/1979). parallel buildbot test worker lists follow, 0-7: http://pastebin.ubuntu.com/1000745/ http://pastebin.ubuntu.com/1000746/ http://pastebin.ubuntu.com/1000747/ http://pastebin.ubuntu.com/1000749/ http://pastebin.ubuntu.com/1000750/ http://pastebin.ubuntu.com/10 [12:55] 00753/ http://pastebin.ubuntu.com/1000756/ http://pastebin.ubuntu.com/1000757/ . If I add the line numbers from those eight lists, I get 16718, so it seems to represent what the parallel test buildbot is counting. Also frankban, those test lists are perfectly round-robin in counts (every list is either 2089 or 2090) which is good...except odd because it was not a first test run. :-/ Anyway, bac, that looks lik [12:55] e something you can use. [12:55] I'm going to shut down the instance and restart now, [12:55] . [12:57] gary_poster: great. that blows my theory, though. had parallel shown more tests then i had a possible explanation [13:03] :-/ k [13:05] oh, bac, did you see my idea on presenting about parallel testing to that pythoncarolina thing in CH? Not sure if it fits your timeline [13:15] gary_poster: no [13:15] * gary_poster looks for it [13:15] gary_poster: found it [13:16] that sounds interesting [13:16] cool [13:16] gary_poster: if i'm not around you can just wing it. :) [13:16] :-) [13:17] gary_poster: so i ran into a guy at RDU wearing an Open Stack t-shirt. he said "yeah, i work for this company called 'Canonical'"... odd [13:21] bac, weird. https://directory.canonical.com/list/state/NC/ ... bfox or bladernr? [13:22] gary_poster: no, i know those guys. some new guy named adam stokes who doesn't have his address in the directory correctly [13:22] huh [13:22] cool, I guess [13:22] and he pronounced it caneenical [13:25] heh [13:26] that's the little-used Knights Who Say Caneeenical pronunciation [13:27] wow, DEBUG = True in canonicaladmin: https://directory.canonical.com/NOOOO [13:27] frankban: that is hilarious [13:27] gary_poster: I've moved the card to done-done, AFAICT testrepository is doing the right thing [13:28] frankban: well, at least that is only on the directory not canonicaladmin [13:28] bac: yes [13:29] https://directory.canonical.com/orgchart/ takes a while to respond :) [13:33] frankban: we should report that to whomever is responsible (I don't know who that is) [13:36] benji: indeed (me either) [13:43] benji, frankban: probably stuartm [14:16] frankban, awesome thanks [14:20] gary_poster: we have some tests (mainly doctests i think) that get run multiple times. lpbuildbot is reporting them individually where our parallel test output only lists them once. summary: http://paste.ubuntu.com/1000909/ [14:20] bac, should we be paranoid that these tests are not actually being run multiple times as they should be? [14:21] concerned [14:21] gary_poster: i'm looking at the data collection code now. i suspect it is a hash indexed by test name which throws away multiples [14:22] bac, that would make sense, except that we have seen some duplicates, like that api test from, um, some bug we had [14:22] benji, I see you are working on the rabbitmq bug [14:23] did you get the egg from bac already? [14:23] back in a sec [14:24] gary_poster: I feel like I should know what you're talking about because it was discussed this morning. [14:24] benji: last week i produced a rabbit-fixture egg that set absurdly high timeouts [14:24] and those timeouts were still being reached? [14:25] benji: no, gary never used it [14:25] benji: and now i can't find it. :( [14:26] benji: found [14:27] what is the intended use of that egg? [14:28] benji: to run a novel branch of launchpad that uses that egg under parallel testing to see if the timeouts go away [14:28] bac, benji, this is intermittent. [14:29] bac: are the timeouts frequent enough that doing that would tell us anything? [14:29] therefore, I suggest that we may need to actually land use of that egg [14:29] benji: it is rabbitfixture 0.3.3-bac-exp and is now in download-cache/dist [14:29] and see if it goes away. [14:29] We've seen it twice in about 30 or 40 runs, I'd guess [14:29] Meta: I think there is room for process improvment here. I feel like these things should have been written down in the bug. [14:30] yeah, either reproducing it or dramatically upping the timeouts seem like the only two rational approaches [14:31] bac: did you try to reproduce the timeout? [14:33] benji, process improvement: Gary should have made a bug. It was kind of a follow-on to existing work, so I didn't. Lesson: always make a bug. It is not written in the bug because it is a placeholder that I asked Francesco to make. Ideally I would have at least added a "talk to me or Brad" to the bug. [14:34] We did discuss it in this morning's meeting, but again, Gary should have written it down. Lesson: always write stuff down in the proper place. [14:34] I know those lessons, just didn't do 'em. [14:34] Lesson: kick Gary. [14:35] benji, I am pretty sure bac did not try to repro this timeout. He had reproed an earlier one to determine that there was a bug in the egg. [14:35] Someone had already fixed the code [14:35] and if we are to make an egg that is referenced by launchpad in production then it needs to be a full release of rabbitfixture and the code pushed [14:35] but not released it [14:35] gary_poster: yes, but that 'fix' was not sufficient [14:35] benji: no, i have not tried to reproduce with the new experimental egg [14:36] simply because the timeout was not enough for what we need, as we believe [14:36] yes. it would be the proper fix for a non-parallel environment [14:36] so I should make a "full release" of rabbitfixture and add that to LP, right? [14:37] Yeah, that's my suggestion. I'm not sure what Brad means by a full release [14:37] bac, do you think we should merge this? I was thinking we ought to have a fork, and maybe file a bug to remove the fork and merge to trunk once we get 40 or so clean runs [14:37] gary, i mean 0.3.4 instead of 0.3.3.-exp [14:38] gotcha, then I understood... [14:38] gary_poster: a fork is fine [14:38] ok [14:38] benji ^^ [14:38] gary_poster: just don't do it via egg drops [14:38] gotcha bac, agreed [14:38] per your previous edict [14:38] i'm all about my edicts [14:38] i prefer hot'n'sour over egg drop [14:38] revision: I will fork rabbitfixture, make an egg, put that in the dependencies, and land an LP branch that uses that version [14:39] go thou! go forth! nay, stop that villainy! [14:39] benji: do you want my code? [14:39] bac: sure (any reason we don't just use your egg as-is?) [14:39] because it is an egg drop [14:39] not a branch [14:39] ah [14:39] I missed that bit. [14:40] aaargh gary_poster, I've written buildout instead of buildbot in the dependencies dir (slave charm) can I r=you the fix? btw, we should call the next project "buildnot" [14:41] what PPAs are to debs X are to eggs (we need an X) [14:41] benji: see lp:~/rabbitfixture/timeout-exp [14:41] "buildnot"! [14:41] benji: review the changes as you may not like them. it is a mix of just upping the timeout and trying to parameterize it [14:42] buildsnot [14:42] bac: ok, thanks [14:42] pfft [14:54] frankban, sorry, of course [14:55] this error looks really familiar... http://ec2-184-72-171-77.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/0/steps/shell_8/logs/summary [14:55] anyone recognize it? Looking for it... [14:56] gary_poster: that looks similar to one Brad and I fixed [14:56] bug 993510 [14:56] <_mup_> Bug #993510: lp.services.job.tests.test_runner.TestJobRunner.test_runJobHandleErrors_oops_generated_user_notify_fails fails intermittently/rarely in parallel tests < https://launchpad.net/bugs/993510 > [14:56] bug 992692 [14:56] <_mup_> Bug #992692: lp.services.mail.tests.test_incoming.TestIncoming.test_invalid_to_addresses fails intermittenty/rarely in parallel tests

< https://launchpad.net/bugs/992692 > [14:57] gary_poster: do you have the full, unfiltered subunit output for the db-devel r 11611 run? [14:57] gary_poster: 992692 is the one I was thinking of [14:57] bac, no, sorry. If that's necessary then I have another machine running tests now, but you'll have to start the analysis over again [14:57] :-( [14:57] gary_poster: it's all scripted [14:58] so it is no thing [14:58] also, to save you work i prefer the full subunit output not the ones divided by worker [14:58] ok cool bac, got it [14:58] gary_poster: eta for completion? [15:30] bac, 5 min. [15:31] We are getting these somewhat frequently [15:31] http://ec2-184-72-171-77.compute-1.amazonaws.com:8010/builders/lucid_db_lp/builds/0/steps/shell_6/logs/stdio [15:31] Don't quite understand what is going on [15:31] I think it happens when we switch between devel and db-devel [15:31] but could be wrong [15:33] bac, maybe you can just download this http://ec2-184-72-171-77.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/0/steps/shell_8/logs/subunit/text [15:33] ok [16:10] gmb, am I right if I say that you can create a co-located branch from a normal one (residing in a shared repo) with "bzr switch -b foo"? [16:13] frankban: I have no idea, I've never tried that :). Sounds sane, though. [16:15] gmb: oh, because you started using colo-branches with the plugin I guess [16:15] frankban, try it :-) [16:15] yep [16:15] ok [16:26] it's kind of sad that I really need to use Google to search Launchpad's buugs [17:13] yay, i have water again! [17:21] yay! [17:21] water good [17:22] there is a big capacitor that is used at start up to give the pump a good jolt. it was nasty and the wires fell out when touched. [17:22] i heart my redneck plumber [17:23] gary_poster: do you know what revno was used for the run referenced above? [17:23] heh [17:24] yeah I can get it [17:24] bac, 15282 (see http://ec2-184-72-171-77.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/0/) [17:24] got it [17:29] frankban, was colocation as easy as it sounded? [17:29] gary_poster: yes [17:29] frankban, cool! I'll have to try. Sounds like the advantages of lightweight checkouts but simpler to use [17:32] gary_poster: yes it's cool, and actually I think each colocated branch is internally a lightweight checkout of the current branch [17:32] makes sense [18:04] gary_poster: i think the duplicate tests are not being run the required number of times. each only shows up once in the unified subunit output from the latest run. the test data i had from yesterday (for r 15139) shows multiple lines for the repeated tests. [18:05] bac, oh, so maybe a regression with the newer version of the testr and friends packages. Thank you for investigating. [18:06] gary_poster: yeah, i think so. what was the date of the 15139 run? [18:06] nm [18:06] bac, to call this finished, then, we should file a bug. We can throw it at something like testr and see if it sticks (if Robert will investigate) but +1 if you are willing and interested to take it farther [18:06] april 23 [18:07] right, exactly [18:07] ok, i didn't realize it was that old [18:25] gary_poster: dumb question, how can i do the equivalent of 'bin/test -t faqcollection.txt' with testr? [18:29] ok, so this appears to be what i was after: [18:29] bin/test --subunit -t faqcollection.txt |testr load [18:29] duh [18:31] good to know [19:09] bad stats today :-/ [20:12] gary_poster: have time for a chat? [20:13] bac, supposed to be talking to flacoste, but not, so sure [20:13] you want to make a hangout [20:13] yeah ok [20:13] since whatever i did yesterday failed [20:14] bac https://talkgadget.google.com/hangouts/_/49af6cb2474cf14af2cb37afc56f73c7c5fc120a [20:56] gary_poster: i've been doing some standalone (i.e. not buildbot) tests with testr --parallel [20:56] if i run "testr run --parallel -- -t faqcollection.txt" then faqcollection.txt appears once in the test partitions but gets run three times as seen in e.g. .testrepository/1 [20:57] looking at the code i don't see any use of sets or other attempts to make the test ids unique [20:58] bac, dunno [20:59] bac, also, when you run them non-parallel, are they in different layers? [20:59] so i remain a bit confused. the subunit output from the test you showed me today only had the suspect tests listed once [20:59] no [20:59] or more generally [21:00] how are they different? are we sure we are running the, differently even when we see them multiple times across workers? [21:00] but, i think i may have a different version of testrepository installed locally due to the funky ppa naming [21:00] oh [21:00] let me try to get the same version and see what happens [21:00] maybe [21:00] ok [21:01] "the," was supposed to be "them" [21:01] bac I have to run [21:01] ok, have fun. wear a helmet if you go out [21:01] will be back later fwiw [21:01] heh thx [21:01] hail hurts