[12:09] bac, just you and me today [12:09] oic [12:09] ok [12:09] call as soon as you show up [12:10] no rush ;-) [13:56] back in a sec [14:51] Whenever I see "Force a build" on buildbot for some strange reason I always think "Bust a move" [16:30] bac, going to go get some lunch. If you have something even partially working with the --load-test thing, I'd love to try it, or pair with you or something. It will greatly help with triaging the bugs that we get more and more of, I hope. [17:10] gary_poster: my --load-list (basic) is working now and i just completed my first test run [17:45] yay bac. I would love to use it for help with diagnosis, whenever you have a branch I can mess with. [17:46] ok, i'll push something soon [17:46] cool thanks [18:05] gary_poster: pushed to lp:~bac/launchpad/ordered-load-list [18:06] gary_poster: i was able to trim the test failures produced by the branch.txt ordering bug down to the 110 on the DatabaseFunctionalLayer and the failure is shown [18:06] ordered-load-list: awesome, thanks bac. writing doc for tomorrow's meeting then will return to running diagnostics for the bugs. [18:06] branch.txt: that's one of the bugs we have on the board? [18:07] No need to watch, but if you are interested, the doc is https://docs.google.com/a/canonical.com/document/d/1_BtI6VDHbHL5oFp8gPX8b1X9sKfK1uj1pnjFbF1a9n4/edit [18:07] gary_poster: that's the vocabularly bug i fixed by sorting the branch names. [18:07] this branch, pre-fix, shows the bug cropping up with a small number of tests [18:08] by using the new --load-list [18:08] bac, oh, interesting. So, it is an ordering issue of some sort, and it's reproducable now. excellent! [18:09] I suppose it does indicate a test isolation thing to investigate. If you want to, I'd be fine with it, but I'd prefer for it to be after we dig into some of the other bugs [18:46] ok. it has just grabbed my curiousity [18:46] http://pastebin.ubuntu.com/960718/ [18:46] if these five tests are run, with branch.txt last, the failure occurs. eliminating any one of them causes the failure to not be seen. [18:52] wow [18:53] bac, if you want to be stumped, I'd prefer that you be stumped about, say, bug 992184 :-) [18:53] <_mup_> Bug #992184: lib/lp/services/database/doc/textsearching.txt fails intermittently/rarely on parallel tests < https://launchpad.net/bugs/992184 > [18:53] ok [18:54] as the most stumpy of the stumps I've seen lately [18:56] that's beautiful [19:02] lol [19:53] gary_poster: when you have time could you review https://code.launchpad.net/~bac/launchpad/ordered-load-list/+merge/104287 ? [19:53] bac, will do it now. My call is in 7. [19:53] gary_poster: it'll only take you 6 [19:53] :-) k [19:56] bac, how odd that the old code used a list for the tests. A set, which it created along the way, would have been much more efficient for the "in" questions, and the order appears irrelevant to me. [19:57] gary_poster: line 16 shows the old deleted code did use a set [19:57] bac, sorted will create a list [19:58] line 36 of the diff would have been faster with a set [19:58] gary_poster: ah, right. both the use of the set and the sorted destroy the original ordering [19:58] right [19:59] gary_poster: and, of course, the test suites get populated not by the order of the tests list but the order they are seen in iterate_tests(suite) [20:00] so, basically, we can say original order was not important. :) [20:00] :-) right [20:01] my point was simply that you were concerned about efficiency of the new approach, and the old approach was not as efficient as it could have been, even given trivial optimizations/consideration. [20:02] oh, flacoste does not appear to be around. maybe he is at that pre-UDS meeting. In any case, no rush there apparently. :-) [20:04] bac, I question raising an exception (lines 55-56) but I'm considering. If I am trying to rerun tests on a new branch/revision, should it really throw up because a test is no longer around? If it really should, it would be nicer to throw up with *all* the missing tests. Let me see what the old code did... [20:05] yeah the old code used it as a filter [20:05] it did not puke [20:05] I'm inclined to do the same [20:05] though could be argued otherwise, maybe [20:06] bac, if you are worried about efficiency, I think you could have a more efficient initial data structure. [20:07] rather than layers -> testnames -> suites [20:07] you could simply have testnames -> (layer, suite) [20:08] then you would no longer need find_layer [20:08] you could simply do a single dict lookup [20:08] that would probably be way way faster [20:10] also afaict we discard ordered_layers and do nothing with it [20:10] removing it would be a good idea [20:10] bac ^^ [20:10] looking [20:13] yep, ordered_layers is a leftover from a previous attempt [20:13] i like the improvements you sugggest, gary_poster [20:13] cool [20:13] i didn't do a time comparison but it isn't noticably slow, as is. [20:14] with your change it may be faster than the original... [20:14] I think it would probably still be simpler/less code in the way I suggest, at least [20:15] what do you think about the exception, bac? treat the list as an ordered filter and silently ignore missing tests, or abort if things are not as we expect? [20:15] If someone cares, they will presumably be looking at the tests that are run [20:15] I dunno [20:15] could be argued either way [20:15] though I still incline to no exception [20:17] i think maintaining the previous behavior is the tie breaker [20:21] cool [20:21] bac, I added one other note in the MP when I was summarazing our discussion: [20:21] - We should include a comment as to why the code is maintaining order, so that future code navigators will have a clue as to intent. [20:25] ok [20:32] gary_poster: so, here are the timings for the three versions: http://paste.ubuntu.com/960961/ [20:33] with your changes, it is minimally faster and has few LOC [20:33] [20:33] bac, :-) cool [20:33] do you think it reads more nicely? [20:34] yes, and the reader won't get creeped out by that awful looping find_layer [20:34] :-) cool [20:34] interesting that the initial version was so ,uch slower [20:34] m [20:36] bac, btw, please don't lose your ideas from this morning to include an option to only run tests in the layer of a given target test. [20:37] I thought that would be interesting to experiment with, and possibly a real time saver [20:38] gary_poster: no, i haven't forgotten. but, i discovered it is quite easy to discover using --load-list / --list-tests in conjuction so that you can manually trim the input list [20:39] but making that automatic would be swell [20:39] bac, oh, what is that trick? I don't understand yet [20:39] bin/test --load-list in.txt --list-test > out.txt [20:40] but what does that buy you? [20:40] out.txt will then show the layer breaks [20:40] oh! [20:40] cool [20:40] remember that trick for Friday sharing time please bac :-) [20:40] yeah, so my itch got less itchy when i found that [20:40] yeah [20:40] +1 [20:41] thanks for the reminder... [20:41] it's hell being clever and forgetful. you don't get to show off. [20:51] gary_poster: your review was just a comment, not an approve. i've pushed the changes if you'd like to vote. [20:52] bac, cool, going [20:54] bac, all approved. thank you! [20:55] thx