[11:49] bac benji frankban (hi gmb!) I'm going to run vote. I think I will be back in time for 8:10. [11:49] If not it will be soon after [11:49] k [11:49] ok gary_poster [11:50] i hope gmb isn't reading irc atm [11:56] why does "download as text" on paste.ubuntu.com require authentication? [12:06] they say there is no bad press: https://plus.google.com/102921374554385564572/posts/PdiEdDE5LeK (re: stevenk's comment) [12:08] heh, lovely [12:09] bac benji frankban call in one or two (asap_ [12:09] ) [12:32] oh, frankban and gary_poster: I meant to mention, I saw a branch that used a negative exit status, that's somewhat odd, and almost certainly transmuted into an unsigned value (see http://www.gnu.org/software/libc/manual/html_node/Exit-Status.html) [12:33] benji: that was probably my branch [12:34] ah, sorry [12:34] I would have responded to the MP but I was reading it on my phone and didn't feel like typing out a response on it. [12:35] benji, ah, I was not aware that it was a problem. Thanks for the heads up. [12:35] i'll change it to 1 instead of -1 [12:35] +, ah, 1 [12:35] :) [12:36] gary_poster: short line at the polls? [12:39] bac, very. I was #81. The only amendment sign there was "against" but our neighborhood in general leans that way, so...not particularly indicative of anything except that I'm glad my neighborhood leans that way. [12:39] #81 for the day, that is. [12:48] gary_poster: UnicodeDecodeError during my test run: http://ec2-107-20-111-209.compute-1.amazonaws.com:8010/waterfall [12:49] it doesn't seem to be related to lpsetup, but to lpbuildbot. [12:53] frankban, weird :-/ ummm...maybe the stdout stream gives us a clue. looking [12:55] frankban, "" is in stdio... [12:55] gary_poster: it seems to be a failure of lib/lp/blueprints/stories/blueprints/xx-non-ascii-imagemap.txt [12:55] yes [12:57] frankban, so I'm not sure what the right solution is, but obviously it is that error_text is...utf8 I assume...and we need to be able to handle it in format_error (and log) [12:57] frankban, could you make a card for that please? [12:57] gary_poster: sure, In the summary there is also an error that I remember: cmapx" not recognized. Use one of: dia hpgl mif mp pcl pic vtx [12:58] maybe some workaround is missing in lpsetup [12:58] frankban, see bug 959352 [12:58] <_mup_> Bug #959352: Ephemeral containers have "/rootfs" prefix in /proc/self/maps entries

< https://launchpad.net/bugs/959352 > [12:59] gary_poster: cool, workaround found [12:59] :-) cool [13:00] gary_poster: does launchpad-database-...-9.1 work? [13:00] frankban, yes! good thinking [13:00] and port is fine automatically [13:01] gary_poster: cool, I asked because I have a lpsetup branch ready with that fix, and now I am going to add the fix for bug 959352 too. [13:01] <_mup_> Bug #959352: Ephemeral containers have "/rootfs" prefix in /proc/self/maps entries

< https://launchpad.net/bugs/959352 > [13:01] cool frankban sounds good [13:14] frankban, what you are working on is kind of a "real" card. I already had a meta card for that task. I moved the meta task over into the active lane and I dragged your card to active. We also technically have 90% of test runs green (better than that even) so we can officially think about these stretch goal cards. I'll think about them later today, but for now... [13:15] bac, benji, frankban, frankban has started a stretch goal card, and we actually can officially do that. This means that we are now going to aim for one bug card at a time. people should either be working on that, or on the stretch goal feature...or on the IS interactio, I guess. :-/ [13:17] gary_poster: ok. I will send a summary email for the meta task later. [13:17] gary_poster: I can't quite parse the above. What is the "that" in "people should either be working on that"? [13:18] benji, the stretch goal card (the active meta card), via a "meta-task subtask" [13:18] so, the next time you are looking for a card, consider the stretch goal. frankban, the email would be great. we also need subtasks put in the "subtask" lane for people to grab [13:19] ah, I didn't realize there was a meta card involved [13:19] if you'd like me to help you identify those with you, I'd be happy to. [13:19] benji, yeah, I decided that those stretch goals seemed pretty much like meta cards. [13:20] frankban, "if you'd like me to help you identify those with you, I'd be happy to" was addressed to you, and "those" are the subtasks for the kanban board :-) [13:20] gary_poster: I think I have already identified the main sub tasks, and I have branches ready for those. As I mentioned, I was actually running parallel tests in a buildbot environment created by lpsetup. [13:20] I've just barely started on my card (bug 994694); shall I put it back and work on the meta/streach card instead (which would put us back under the WIP limit which we somehow exceeded) [13:20] <_mup_> Bug #994694: GeneratorExit exception throughout paralleltest subunit output < https://launchpad.net/bugs/994694 > [13:21] benji, not sure yet. keep trucking, and I'll chat with frankban about what needs to be done. [13:21] frankban, right. so you've pretty much done everything already? :-) [13:21] benji, "keep trucking" == "keep working on that bug" ;-) [13:22] * gary_poster has been on the lookout for his use of vague idioms lately [13:24] gary_poster: probably. To be sure I need to 1. fix lpsetup with postgres 0.1 and the rootfs workaround and 2. complete a test run [13:25] gary_poster invents "Mud flap management" [13:25] postgres 9.1! 0.1 is quite deprecated [13:26] frankban, :-) ack. So at most, at your EoD if you are not done and there are tasks that others could do, send out an email or talk to me about it and we'll see if anyone can take over for you. [13:26] frankban, otherwise, I'll get out of your way. Thanks! [13:26] benji, [13:26] In other news, 12 green runs in a row! [13:30] We're "done" with fixing test bugs when we reach 95% passing runs, right? Over how many runs is that? [13:34] benji: 19, no? [13:35] benji: i mean, i thought it was 95% green runs...so 1 failure in 20 full runs [13:35] yes, benji, what bac says is how I interpret it [13:35] "done" means that we could deploy with that [13:35] we should keep running at that point and fix what we find [13:36] but it means that stage is "good enough" [13:36] and i assume we can ignore any non-green runs that also fail on the real buildbot causing testfix mode. we shouldn't be penalized for those. [13:36] right, bac [13:37] though there are so few commits in the LP tree that it hasn't been an issue. OTOH maybe the UDS session will change that. :-) [13:44] bac were you able to dupe your bug? [13:44] (curiosity( [13:44] ) [13:44] gary_poster: no. [13:44] that jibes with my experience then. [13:47] gary_poster: so should we just keep an eye on this bug and see if it happens again? [13:48] bac...you got me. I thought you were very brave grabbing it, because I wasn't sure what to do with it. Mm, lemme look at it again. [13:49] bac, you could ask abentley if he has any bright ideas as to what to investigate, or if he's going to be replacing this code with celery tomorrow so nevermind, or something. [13:51] bac, no other bright ideas other than running the test 10 times in a loop and seeing if you can dupe it then. I ran it three or four times in a loop and was unable to, after I didn't get any isolation results. [13:51] if neither of those lead anywhere bac, then, yeah, maybe dropping it for later would make sense. [13:54] ok, i'll hammer it and see [13:54] could anybody please review https://code.launchpad.net/~frankban/lpsetup/postgres-9.1/+merge/105070 ? That's the branch I'd like to package and use for the next test run [13:54] bac, we have our first failure in 13 runs, and it's an old friend: http://ec2-23-22-54-190.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/1/steps/shell_8/logs/summary . We haven't seen these in quite some time, which suggests that our previous efforts improved things. I wonder if the timeout needs to just be crazy or something. [13:54] frankban: what is the ETA for the death of setuplxc? [13:54] frankban: your MP is approved [13:54] (i'd already done it) [13:54] er [13:57] bac: oh, thanks, and yes, I will change the exit code. [13:58] * gary_poster started data center test run @ 13:05 Z (note for later) [13:59] bac: ETA for the death of setuplxc -> we have a meta card to replace it, so I think we could stop supporting it (at least for non-blocking bugs). but maybe gary_poster can better answer your question [14:00] frankban, well, when will you merge a branch to our charm that switches to lpsetup? [14:00] Assuming that you do that after everything works, that [14:01] is the moment I'm +1 on deleting setuplxc and forgetting about it [14:01] but yeah, actually: [14:01] we should only make changes to it for blocking bugs, I agree with that, bac, frankban [14:02] even before the switch [14:02] (which I expect to happen this week and maybe even tomorrow if we are lucky, from what frankban says) [14:02] gary_poster: ok, then i won't bother with the exit_code -1 problem [14:02] ok bac, cool [14:02] gary_poster: it is fixed in lpsetup [14:02] cool bac, I saw that in the MP, thanks [14:36] bac, do you think I should reopen bug 974617 or file a new bug. It is the same old problem, IMO. I think I see one more way to have a race condition, and a correlating approach to address it. [14:36] <_mup_> Bug #974617: test_operationalerror_view_integration fails intermittently in parallel tests

< https://launchpad.net/bugs/974617 > [14:37] gary_poster: same bug [14:37] cool, agree, wasn't sure, thanks [15:03] gary_poster: i'm running a set of 20 tests for bug 994777 and so far i have seen one failure [15:03] <_mup_> Bug #994777: lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout_short fails rarely/intermittently in parallel tests < https://launchpad.net/bugs/994777 > [15:04] bac, wow [15:04] i have a set of 84 tests in my layer [15:05] bac, I'm not sure I understand. (1) 20 tests or 84 tests? (2) neither of us were able to dupe before; what are you doing differently? [15:05] 20 full runs of 84 individual tests in my --load-list [15:07] gary_poster: i've done nothing differently but run the test repeatedly as you suggested earlier. [15:07] oh I see [15:08] bac, did you also try running the test by itself 20 times, out of curiosity? [15:08] gary_poster: no but that's a good idea [15:15] bac, benji, I'm trying to copy the testing cabal's new version of the packages we need, and I'm getting this error: "testrepository 0.0.5+bzr153~ppa11~precise1 in precise (version older than the testrepository 0.0.5++z~ppa6~precise1 in precise published in precise)". I'm strongly inclined to delete the ++z packages because the naming is just wrong for the present and the future AFAICT. Do either of you disagree [15:15] or have other advice? [15:16] gary_poster: it may be difficult to kill the ++z versions because of all the caching and other storage of version info that goes on, but I don't know enough about it to be sure it won't work in our narrow case [15:17] gary_poster: i agree it is a mess that needs to be sorted out. i don't have a plan. [15:17] benji, it will be an annoyance for our own machines, if we installed them, but I think we can cope with it. It shouldn't be a problem for the juju machines since they are torn down and rebuilt. [15:18] I plan to cope with the fallout locally first [15:18] so I can hopefully save others the pain [15:18] ok, I'll give it a whirl, I think [15:18] thanks [15:18] gary_poster: sounds like a plan [15:18] cool [15:18] gary_poster: the PPA machinery may interfere. i *think* it remembers the versioning of previously published packages and enforces that the versioning move forward. [15:18] but it is worth trying [15:18] bac, even if it is deleted? yeah, ok, giving it a whirl [15:21] the PPA machinery let me delete and copy as I wished [15:21] it is building now [15:21] after that I will try locally [15:47] gary_poster: it doesn't throw up until publish time. stupid. [15:47] bac :-( [15:47] gary_poster: i am running 20 tests with just the failing twisted test. so far i have seen one failure [15:49] bac, "it doesn't throw up until publish time": I thought you meant https://launchpad.net/~yellow/+archive/ppa/+packages but that looks ok. you must mean your bug, but I'm not sure what publish time is for that job thing [15:50] sorry gary_poster. i'm not communicating very well today. [15:50] the PPA may build fine but then version issues are only caught after the build during the publish phase. [15:51] s'ok, I'm throwing myself at some buildbot issue and feeling a bit numb myself [15:51] ah ok [15:51] * bac -> sandwhich [15:51] but we're still fine in this particular case, right bac, given what you see on that link? [15:51] s'ok, go have a sandwich (which wich?) [15:52] oh, yes, that does look nice [15:52] great [15:52] i'm not sure what i'll order at Sandwhich [15:53] probably a sandwich [15:53] heh [15:54] benji, if you want to be distracted a bit, take a glance at http://ec2-23-22-54-190.compute-1.amazonaws.com:8010/waterfall , top run [15:54] the subunit stream has all tests [15:55] even a failure [15:55] http://pastebin.ubuntu.com/975994/ [15:55] but worker and stats are not complete [15:55] * benji looks [15:55] must run to lunch/babysitting [17:13] frankban, approved https://code.launchpad.net/~frankban/charms/oneiric/buildbot-slave/use-lpsetup/+merge/105086 with small comments [17:15] also approved https://code.launchpad.net/~frankban/lpbuildbot/use-lpsetup/+merge/105085 [17:16] thanks gary_poster: I will change the charm to be backward compatible. For your other question: in lpsetup --testing is an alias for --use-urandom --create-scripts --install-subunit [17:16] frankban, heh, cool! [17:17] another clean run \o/ [17:17] yay! :-) [17:18] frankban, might as well add your runs to https://dev.launchpad.net/ParallelTests/ResultsLog if you are willing and have the time. no problem if not. [17:19] gary_poster: the first one already added, I will add the second and third runs (started now) later. need to go, have a nice evening! [17:20] cool frankban thanks. you too! [18:21] gary_poster: since you reviewed it, I assume you know what's going on with frankban's lpsetup branch; can I move that card from review to landing? Also, I'm looking for something to do, so I figured I should take the next step from his email and remove setuplxc from LP. [18:21] benji, hey. move card: +1. remove setuplxc: not till everything has landed please. what next: give me a sec please, trying to keep something in my head [18:22] ok, thing in my head was used :-) [18:22] um, next... [18:22] * benji envisions parisitic brain worms trying to escape. [18:23] :-) [18:24] benji, 974617 will be easy if you agree with my assessment. The UnicodeDecodeError in Francesco's card should be a bug against lpbuildbot just so we can track it, and then that is probably easy. [18:24] I'm in the middle of filing two other bugs [18:24] and they might not be as easy [18:24] * benji looks at 974617. [18:24] so maybe by the time I've filed the pertinent bits you'll be done with those and can look at one of these [18:27] gary_poster: I like your suggestion for 974617. I'll start on it now. [18:29] cool benji [19:33] gary_poster: I think this is new: http://ec2-23-20-70-98.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/2/steps/shell_8/logs/summary [19:33] * gary_poster looks [19:34] it is frankban. I can file the bug if you give me a few minutes before you shut the instance down [19:34] or you can file it--if you do, please include a pastebin of the worker-1 output [19:35] I'm filing now [19:35] thanks gary_poster, i'll pick the number and update the result log [19:36] thank you [19:37] frankban, bug 996720 [19:37] <_mup_> Bug #996720: lib/lp/soyuz/javascript/tests/test_lp_dynamic_dom_updater.html fails rarely/intermittently in parallel tests < https://launchpad.net/bugs/996720 > [19:41] done [19:58] benji, you are out until next Tuesday, right? [19:58] gary_poster: right [19:58] starting tomorrow I mean [19:58] ok cool, enjoy and safe travels [19:59] benji, source of weird buildbot/subunit output, if you are interested: 996729 [19:59] bug 996729 [19:59] <_mup_> Bug #996729: zope.testing --subunit allows bad output on stdout, which can break subunit processing < https://launchpad.net/bugs/996729 > [19:59] I am interested; looking. [20:00] gary_poster: that's a good one [20:00] yeah, I'm afraid it will be even more interesting to fix, because I expect test fallout [20:00] given the desired solution [20:01] mmm [20:01] I do think your solution is a good one though. [20:01] cool [20:23] gary_poster: regarding https://bugs.launchpad.net/launchpad/+bug/994777 I find it is reproducible at about the same rate (5%) whether it is run in the original grouping or when that test is run individually. [20:23] <_mup_> Bug #994777: lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout_short fails rarely/intermittently in parallel tests < https://launchpad.net/bugs/994777 > [20:23] i think it has nothing to do with paralleltest and should be classified as a generic spurious test. [20:56] bac, sorry was on call with francis [20:56] bac, can we/should we just disable it then? Do you want to timebox it or toss it to aaron for ideas? [20:56] * gary_poster back in a sec [21:01] * gary_poster back [21:09] gary_poster: https://code.launchpad.net/~benji/launchpad/bug-974617/+merge/105130 [21:10] diff updating benji :-) [21:10] there it is... [21:11] ooh, nice benji. [21:12] approved benji [21:12] I'll land tomorrow if you need me to [21:14] gary_poster: yeah, it would probably be best if you land it, that way you get the emails, etc. [21:15] cool benji. have fun! :-) [21:16] gary_poster: thanks, see you later