| StevenK | wgrant: http://pastebin.ubuntu.com/1229519/ | 00:01 |
|---|---|---|
| wgrant | StevenK: That looks most excellent. Perhaps try some tests that exercise the DB? | 00:01 |
| StevenK | Hmmm | 00:01 |
| StevenK | -t bugs ? | 00:01 |
| wgrant | Though DatabaseLayer setup worked, so it's a good sign | 00:02 |
| StevenK | And it can run while I nom | 00:02 |
| wgrant | -t bugs is huge | 00:02 |
| wgrant | Try archiveuploader | 00:02 |
| wgrant | Relatively quick and terrible | 00:02 |
| === matsubara is now known as matsubara-afk | ||
| StevenK | wgrant: Right, that looks pretty excellent too | 00:37 |
| wgrant | Right. | 00:39 |
| wgrant | So, jenkins :) | 00:39 |
| StevenK | wgrant: Yes | 00:41 |
| wgrant | StevenK: Should be pretty easy, I guess | 00:41 |
| wgrant | No need for the complex build rules any more | 00:41 |
| wgrant | Just a .testr.conf, lp-setup-lxc-build, testr run --parallel | 00:41 |
| wgrant | Or so | 00:41 |
| StevenK | Do I need the testr from testing-cabal? | 00:42 |
| wgrant | StevenK: You might need new testrepository, testtools and subunit | 00:48 |
| wgrant | Might as well grab them all from there | 00:48 |
| StevenK | Right | 00:48 |
| StevenK | Just installing java and fixing ssh access | 00:48 |
| wallyworld | lol | 00:52 |
| StevenK | The Jenkins slave is java, no choice in the matter | 00:53 |
| wallyworld | i know, just trolling | 00:53 |
| StevenK | Slave successfully connected and online | 00:55 |
| StevenK | wgrant: So, .testr.conf ? | 00:55 |
| StevenK | This is different from the .testr.conf already in the LP tree? | 00:55 |
| wgrant | StevenK: Yeah. Grab it from testr.conf in lp:~launchpad/lpbuildbot/public | 00:56 |
| wgrant | It has a list command and uses lp-setup-lxc-test rather than bin/test | 00:56 |
| StevenK | So I should move it into the tree as the first step? | 00:57 |
| wgrant | Just clobber the .testr.conf that's in trunk locally for now | 00:57 |
| wgrant | buildbot copies it in at the start of each build | 00:58 |
| StevenK | I've added a build step | 00:59 |
| LPCIBot | Project devel build #1091: STILL FAILING in 1 min 3 sec: https://lpci.wedontsleep.org/job/devel/1091/ | 01:01 |
| wgrant | Heh | 01:01 |
| StevenK | sudo lp-setup-lxc-build lptests /home/jenkins/.ssh/id_rsa | 01:02 |
| StevenK | sudo: lp-setup-lxc-build: command not found | 01:02 |
| StevenK | :-( | 01:02 |
| wgrant | Is it in PATH? | 01:02 |
| StevenK | It's in /home/jenkins/bin, but it looks like that doesn't work | 01:03 |
| wgrant | ~/bin isn't in PATH by default | 01:03 |
| StevenK | Well, yes | 01:03 |
| LPCIBot | Project devel build #1092: STILL FAILING in 8.9 sec: https://lpci.wedontsleep.org/job/devel/1092/ | 01:04 |
| LPCIBot | Project devel build #1093: STILL FAILING in 28 sec: https://lpci.wedontsleep.org/job/devel/1093/ | 01:06 |
| StevenK | Haha | 01:06 |
| StevenK | Need testr init first | 01:06 |
| wgrant | Ooh | 01:06 |
| wgrant | It nearly worked | 01:06 |
| StevenK | Are you spying? :-) | 01:06 |
| wgrant | Yes | 01:06 |
| wgrant | How many threads does the instance have? | 01:07 |
| * StevenK firewalls wgrant off | 01:07 | |
| StevenK | 4 processors, apparently | 01:07 |
| wgrant | You'll want testr run --parallel --concurrency=SOMETHING --subunit --full-results | 01:07 |
| wgrant | And you'll need to set a shared TEMP, which probably means it needs to be in the code dir | 01:07 |
| wgrant | buildbot creates temp/ in the tree at the start of each build | 01:08 |
| StevenK | Oooh | 01:10 |
| StevenK | This is looking okay | 01:10 |
| wgrant | Indeed | 01:10 |
| wgrant | concurrency=8 is probably going to die | 01:10 |
| wgrant | But we'll see | 01:10 |
| StevenK | Die how? | 01:11 |
| StevenK | We have 34GiB of RAM | 01:11 |
| wgrant | It'll thrash | 01:11 |
| wgrant | Ah, so at least RAM won't be a problem | 01:11 |
| StevenK | Oh, yeah, java. | 01:11 |
| wgrant | Also, your build failed | 01:11 |
| StevenK | So we have 4GiB of RAM available | 01:11 |
| wgrant | lazr.restful missing from download-cache? | 01:11 |
| wgrant | buildbot's happy with it, so I assume you just failed to update | 01:11 |
| LPCIBot | Project devel build #1094: STILL FAILING in 2 min 19 sec: https://lpci.wedontsleep.org/job/devel/1094/ | 01:11 |
| wgrant | Yeah | 01:12 |
| wgrant | But it otherwise worked | 01:12 |
| StevenK | I guess I need to run bzr pull in the download-cache | 01:12 |
| wgrant | You'll want to add bzr up download-cache, and probably update-sourcecode, steps | 01:12 |
| wgrant | But you can just do a one-off for now, I suppose | 01:12 |
| StevenK | I don't need db setup? | 01:13 |
| wgrant | lp-setup-lxc-build should do that | 01:13 |
| StevenK | I thought so | 01:13 |
| StevenK | Here's a script I prepared earlier | 01:14 |
| LPCIBot | Project devel build #1095: STILL FAILING in 10 sec: https://lpci.wedontsleep.org/job/devel/1095/ | 01:14 |
| wgrant | Evidently not :) | 01:14 |
| wgrant | StevenK: scripts/init_testr.py from the lpbuildbot branch may be handy | 01:14 |
| wgrant | StevenK: It preserves .testrepository between runs, which makes balancing the tests between runners more effective | 01:15 |
| wgrant | And prevents the issue that killed this build :) | 01:15 |
| StevenK | Hmmm | 01:16 |
| StevenK | Right | 01:16 |
| StevenK | So far I'm blowing it away, but that can be fixed | 01:17 |
| wgrant | This is looking better | 01:18 |
| StevenK | Indeed | 01:18 |
| StevenK | wallyworld: Are you watching too, or just ignoring us? | 01:19 |
| wallyworld | huh? | 01:19 |
| wgrant | Doing proper work, hopefully :P | 01:19 |
| StevenK | This is proper work! | 01:20 |
| wallyworld | watching what? | 01:20 |
| StevenK | Also known as buildbot and buildbot-poll die in a large fire | 01:20 |
| StevenK | Jenkins | 01:20 |
| wgrant | StevenK: Check in temp/ | 01:20 |
| wgrant | Are there 8 files with lots of tests listed? | 01:21 |
| StevenK | Yes | 01:21 |
| wgrant | It's starting up 8 new containers, so it looks good | 01:21 |
| wgrant | Great | 01:21 |
| wgrant | I think we're good :) | 01:21 |
| wgrant | Yay | 01:21 |
| StevenK | We are | 01:21 |
| wgrant | And it's even tagged properly | 01:22 |
| StevenK | jenkins@domU-12-31-39-17-3C-7F:~/launchpad/lp-branches/workspace/devel$ uptime | 01:22 |
| wgrant | I must work that out on prod buildbot with gnuoy next week | 01:22 |
| StevenK | 01:22:17 up 3:08, 1 user, load average: 6.70, 2.75, 1.14 | 01:22 |
| wgrant | :) | 01:22 |
| wallyworld | wgrant: i recall reading somewhere why the test count is inflated now. can you remind me? | 01:22 |
| wgrant | wallyworld: The Zope testrunner layer setup/teardowns appear as tests | 01:23 |
| wgrant | eg 'test: lp.testing.layers.BaseLayer:tearDown' | 01:23 |
| wallyworld | really? | 01:23 |
| StevenK | I wonder how it will take with 8 | 01:23 |
| wallyworld | surely we can filter that out | 01:23 |
| wgrant | If you grep them out then the count is accurate | 01:23 |
| wgrant | StevenK: Didn't jenkins previously show live test stats? | 01:24 |
| StevenK | It will show progress, not stats | 01:24 |
| StevenK | But it takes like 3 or 4 builds to prime | 01:25 |
| wgrant | :( | 01:25 |
| wgrant | Yeah | 01:25 |
| wgrant | Doesn't subunit2junitxml let it show live counts? | 01:25 |
| StevenK | ENOIDEA | 01:25 |
| wallyworld | wgrant: StevenK: if you want a break from jenkins, a quickie.... https://code.launchpad.net/~wallyworld/launchpad/embargoed-sharing-policy-ui-1055617/+merge/126585 | 01:27 |
| StevenK | 8 threads is probably what, 2 hours? | 01:27 |
| StevenK | wallyworld: It's building | 01:27 |
| StevenK | So now wgrant and I should do a critical or something | 01:28 |
| wallyworld | do a review first :-) | 01:28 |
| wgrant | I'm waiting for a local testr run too | 01:28 |
| wallyworld | oh bollocks. legit bb failure | 01:29 |
| * wallyworld fixes | 01:29 | |
| StevenK | Line 53 can be sucked back onto 47? | 01:29 |
| StevenK | Which probably makes the branch neutral | 01:29 |
| wallyworld | yeah | 01:29 |
| StevenK | jenkins@domU-12-31-39-17-3C-7F:~/launchpad/lp-branches/workspace/devel$ uptime | 01:30 |
| StevenK | 01:30:10 up 3:16, 1 user, load average: 11.62, 9.77, 5.15 | 01:30 |
| wgrant | StevenK: Well, you did ask it to run 8 multiprocess test suites on 4 cores :) | 01:30 |
| StevenK | I should have gone bigger? | 01:31 |
| wgrant | Doesn't matter too much | 01:31 |
| StevenK | Pft, we have 20GiB of RAM free | 01:31 |
| wgrant | I'm not sure what yellow used for scalability testing, but we don't really need that here | 01:32 |
| wgrant | We don't care about scaling, just working | 01:32 |
| StevenK | It seems to work | 01:33 |
| wgrant | Indeed | 01:33 |
| wgrant | StevenK: You should probably steal the remaining build steps from master.cfg in the lpbuildbot branch | 01:34 |
| wgrant | With scripts/init_testr and co | 01:34 |
| StevenK | Aww, can't we just move on and get this on prase? :-P | 01:34 |
| wgrant | Also shake Julian until MAAS Tarmac and Jenkins configs fall out! | 01:34 |
| StevenK | I think other projects have used it as well | 01:35 |
| StevenK | openstack is one I can remember. Until they switched to git, at least | 01:35 |
| wallyworld | StevenK: you ok to +1 that branch you looked at? | 01:52 |
| StevenK | wallyworld: Sorry, done. | 01:53 |
| wallyworld | np thank you | 01:53 |
| wallyworld | i was doing the testfix anyway till now | 01:54 |
| lifeless | wgrant: pqm 0.6 up on LP, and in trunk. | 02:24 |
| lifeless | wgrant: YMMV, may the farce be with U | 02:25 |
| wgrant | Thanks | 02:26 |
| wgrant | Of course I might just dump PQM and use Tarmac instead :) | 02:26 |
| lifeless | fine by me | 02:26 |
| lifeless | long as you're not running tests under it | 02:26 |
| lifeless | tarmac isn't even faintly paranoid enough about *that* | 02:27 |
| StevenK | Tarmac+Jenkins ? | 02:27 |
| wgrant | Right, Tarmac + Jenkins is the sensible solution | 02:27 |
| lifeless | though I don't know why you'd have tarmac in the picture for that. | 02:27 |
| wgrant | Other projects (eg. maas) have you submitting to Tarmac, Tarmac asks Jenkins to build it, then Tarmac commits if Jenkins succeeds | 02:28 |
| lifeless | you may not know this, but PQM can get its queue from LP | 02:29 |
| lifeless | or could before the code team destroy the API :) | 02:30 |
| lifeless | anyhow | 02:30 |
| lifeless | there is something funky with that layering | 02:30 |
| lifeless | I need to reset my whiteboard to make a case, i think | 02:30 |
| wallyworld | do we know why TestArchiveWebservice.test_getAllPermissions_constant_query_count sometimes fails? | 02:33 |
| wgrant | wallyworld: It might depend on whether it's the first browser test in that process | 02:34 |
| wgrant | lifeless: Yeah, but Tarmac is at least less crufty and abandoned than PQM is today... | 02:35 |
| bigjools | tarmac + jenkins is working extremely well for maas so far | 02:39 |
| StevenK | bigjools: Are the configs available so we can duplicate it? | 02:40 |
| StevenK | wallyworld: You fail. Again. | 02:50 |
| wallyworld | huh? | 02:50 |
| wallyworld | how? | 02:51 |
| StevenK | Oh, the failure wasn't your fault. | 02:51 |
| wallyworld | nope | 02:51 |
| StevenK | Carry on | 02:51 |
| wgrant | Hm | 02:54 |
| wgrant | That subunit stream looks a little broken | 02:54 |
| StevenK | wgrant: I wonder if bug 1020439 is limited to those few branches | 02:54 |
| _mup_ | Bug #1020439: AttributeError: 'NoneType' object has no attribute 'items' in +preview-diff <code-review> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1020439 > | 02:54 |
| wgrant | StevenK: Yes | 02:54 |
| StevenK | Oh, they have no merge_diff ? | 02:54 |
| wgrant | So assign to you and close | 02:54 |
| wgrant | Oh oops, wrong bug | 02:54 |
| * wgrant -> lunch | 02:55 | |
| StevenK | Haha | 02:55 |
| StevenK | Helpful much | 02:55 |
| wgrant | It may be the same, but I can't remember | 02:55 |
| wgrant | And am on my way out | 02:55 |
| StevenK | wgrant: It still happens | 02:55 |
| StevenK | wget a URL from the OOPS gives 00 | 02:55 |
| StevenK | *500 | 02:55 |
| StevenK | The MP exists, and has a diff | 02:58 |
| LPCIBot | Project devel build #1096: STILL FAILING in 1 hr 46 min: https://lpci.wedontsleep.org/job/devel/1096/ | 03:01 |
| StevenK | Wha? | 03:02 |
| * StevenK prods Jenkins | 03:02 | |
| StevenK | Hm, it misreports the failure, too :-( | 03:04 |
| * wallyworld has an appointment with the tax man :-( | 03:39 | |
| StevenK | wallyworld: To pay him? | 03:39 |
| wallyworld | no, to figure out some of my shit for last year | 03:41 |
| wallyworld | bbiab | 03:41 |
| nigelb | Don't you pay him for that? :) | 03:56 |
| lifeless | We don't pay him for his shit. | 03:57 |
| lifeless | That would just be wrong wrong wrong | 03:57 |
| nigelb | Heh | 03:58 |
| StevenK | wgrant: Halp? | 04:48 |
| StevenK | wgrant: bug 1020439 is triggered when previewdiff.diff.diffstat = None and I'm trying to figure out how to not make lazr.restful eat a bullet | 04:49 |
| _mup_ | Bug #1020439: AttributeError: 'NoneType' object has no attribute 'items' in +preview-diff <code-review> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1020439 > | 04:49 |
| lifeless | StevenK: does lazr.restful want diff to never be None ? | 04:52 |
| StevenK | lifeless: Where is that defined? | 04:52 |
| lifeless | the schema for the interface | 04:53 |
| lifeless | I'm wagging here, of course | 04:53 |
| LPCIBot | Project devel build #1097: STILL FAILING in 1 hr 46 min: https://lpci.wedontsleep.org/job/devel/1097/ | 04:53 |
| StevenK | Jenkins, you make me very sad. | 04:53 |
| StevenK | lifeless: readonly=True, I'm not sure what the other defaults are | 04:53 |
| lifeless | me neither | 04:54 |
| StevenK | They are not set to required=True | 04:54 |
| StevenK | So I guess they're False | 04:54 |
| StevenK | _StringException: lost connection during test 'lp/registry/javascript/tests/test_milestone_creation' | 04:56 |
| StevenK | Hm | 04:56 |
| lifeless | StevenK: so - marshaller | 04:56 |
| lifeless | StevenK: look at the marshallers available, I'll bet that its one with a bug in it | 04:57 |
| StevenK | <class 'lazr.restful.marshallers.DictFieldMarshaller'> | 04:58 |
| lifeless | StevenK: so a few lines up | 05:00 |
| lifeless | marshaller = getMultiAdapter((field, self.request), IFieldMarshaller) | 05:00 |
| lifeless | that suggests that that is going off the rails, to me at least | 05:00 |
| StevenK | lifeless: How? | 05:01 |
| StevenK | The field is declared as an exported dict | 05:01 |
| StevenK | And it's attempting to unmarshall the contents | 05:01 |
| StevenK | It just doesn't deal with None at all | 05:01 |
| lifeless | right | 05:01 |
| StevenK | Right, if value is None: return {} works nicely | 05:03 |
| StevenK | HAH | 05:03 |
| StevenK | def _get_diffstat(self): | 05:03 |
| StevenK | if self._diffstat is None: | 05:03 |
| StevenK | return None | 05:03 |
| StevenK | GIGO | 05:04 |
| lifeless | EWTF | 05:04 |
| lifeless | how is that different to | 05:04 |
| lifeless | return self._diffstat ? | 05:04 |
| StevenK | Below that it returns a dict | 05:04 |
| lifeless | ah | 05:04 |
| StevenK | Fantastic, I don't need to touch lazr.restful | 05:08 |
| StevenK | lifeless: So, can you look at the last Jenkins build? | 05:12 |
| StevenK | lifeless: I have this inkling there is something screwy with the subunit stream | 05:12 |
| lifeless | url me up | 05:13 |
| StevenK | lifeless: https://lpci.wedontsleep.org/job/devel/lastBuild/ | 05:13 |
| lifeless | ERROR: cannot verify lpci.wedontsleep.org's certificate, issued by `/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA': | 05:18 |
| lifeless | Issued certificate has expired. | 05:18 |
| StevenK | Yes | 05:21 |
| StevenK | We should fix that by moving to prase | 05:21 |
| * StevenK bangs that drum a little harder | 05:21 | |
| lifeless | StevenK: ok, so tricks: | 05:23 |
| lifeless | cat consoleText| subunit-stats | less | 05:23 |
| lifeless | page down till you see | 05:23 |
| lifeless | Starting up the container... | 05:23 |
| lifeless | after that you should see nothing as subunit will be eating it | 05:23 |
| lifeless | when you start seeing something, its because subunit thinks a test prior to it has not finished | 05:24 |
| lifeless | and is in pass-through mode on everything other than a terminal for that test. | 05:24 |
| lifeless | so | 05:24 |
| lifeless | test: lib/lp/soyuz/stories/ppa/xx-private-ppas.txt | 05:24 |
| lifeless | is the first output | 05:24 |
| lifeless | use that to seek in the raw stream to debug | 05:24 |
| lifeless | less consoleText | 05:24 |
| lifeless | / lib/lp/soyuz/stories/ppa/xx-private-ppas.txt | 05:24 |
| lifeless | and work back: | 05:24 |
| lifeless | test: Could not communicate with subprocess | 05:25 |
| lifeless | time: 2012-09-27 04:39:15.455646Z | 05:25 |
| lifeless | test: lib/lp/soyuz/stories/ppa/xx-private-ppas.txt | 05:25 |
| lifeless | ^ thats invalid yo | 05:25 |
| StevenK | Could not communicate with subprocess is zope.testing garbage, isn't it? | 05:26 |
| lifeless | yes, its odd that it says 'test: Could not communicate with subprocess' | 05:26 |
| lifeless | theres no immediate evidence of interleaving going on | 05:27 |
| lifeless | it *could* be a race condition with gc or something | 05:27 |
| lifeless | or just bad code for handling that situation | 05:27 |
| lifeless | e.g. | 05:27 |
| lifeless | there is a reporter thingy in zope.testing | 05:27 |
| lifeless | it has a special class for subunit | 05:27 |
| lifeless | that class may have a bug | 05:27 |
| lifeless | where it reports this situation via the literal 'test: Could not communicate with subprocess\n' | 05:28 |
| lifeless | which would be a bug. | 05:28 |
| StevenK | Then why doesn't buildbot see that :-( | 05:28 |
| lifeless | it may not be encountering 'Could not communicate with subprocess' | 05:28 |
| lifeless | or it may be one of the things gary keeps posting a bug link to | 05:28 |
| lifeless | StevenK: s def error_with_banner(self, message): | 05:29 |
| lifeless | self._emit_fake_test(message, self.TAG_ERROR_WITH_BANNER) | 05:30 |
| StevenK | Haha | 05:30 |
| StevenK | That does seem a little bit like crack, yes. | 05:31 |
| lifeless | which calls startTest(test), _emit_tag(tag), addSuccess(test) | 05:31 |
| lifeless | so, that should be ok, but.. | 05:31 |
| lifeless | and in fact.. | 05:32 |
| lifeless | test: Could not communicate with subprocess | 05:32 |
| lifeless | is line 108890 | 05:32 |
| lifeless | successful: Could not communicate with subprocess | 05:32 |
| lifeless | is 108996 | 05:32 |
| StevenK | subunit doesn't like tests with whitespace? | 05:33 |
| lifeless | so there is an interleaving problem | 05:33 |
| lifeless | subunit doesn't care about that | 05:33 |
| lifeless | open consoleText with vim | 05:33 |
| lifeless | something wrote, to the same fd, a tonne of other test info between the three lines of that self._emit_fake_test function | 05:34 |
| StevenK | wallyworld, wgrant: https://code.launchpad.net/~stevenk/launchpad/preview-diff-none-type/+merge/126601 | 05:34 |
| StevenK | lifeless: Yeah, the line numbers are staggeringly different | 05:35 |
| lifeless | StevenK: part of the issue here is that you're getting the demultiplexed stream out of testr | 05:36 |
| StevenK | testr run --parallel --subunit --full-results --concurrency=8 | subunit2junitxml -f -o test-results.xml | 05:36 |
| lifeless | yeah | 05:37 |
| lifeless | testr is running 8 streams on separate fd's and demultiplexing them. | 05:37 |
| lifeless | its possibly a bug in your testtools version | 05:37 |
| lifeless | what testtools and subunit versions are you running ? | 05:37 |
| StevenK | ii python-testtools 0.9.14+bzr267~ppa37~precise1 Extensions to the Python unittest library | 05:38 |
| StevenK | ii subunit 0.0.8+bzr176~ppa126~precise1 command line tools for processing Subunit streams | 05:38 |
| StevenK | Both from testing-cabal | 05:38 |
| lifeless | hmmm | 05:39 |
| lifeless | I have to run, cynthia time | 05:39 |
| lifeless | suggestion | 05:39 |
| lifeless | drop the subunit2junitxml -f -o test-results.xml for now | 05:39 |
| nigelb | 54 | 05:39 |
| lifeless | and the --full-results and the --subunit | 05:39 |
| nigelb | argh, sorry! | 05:39 |
| lifeless | testr will write its output to .testrepository, the jenkins fs explorer thingy will let us pull the stream down and see what it gets | 05:40 |
| lifeless | I will return in ~2 hours | 05:40 |
| StevenK | Which is how long the build will take | 05:40 |
| wgrant | StevenK: Did the build look successful other than the dead runner and corrupt stream? | 06:18 |
| StevenK | wgrant: Mostly | 06:19 |
| StevenK | I'm doing what lifeless' suggested, so we'll see | 06:19 |
| StevenK | But since testr likes hiding things, we have no output at all | 06:20 |
| wgrant | StevenK: Tail the stream | 06:20 |
| wgrant | It's in ~/.testrepository | 06:20 |
| wgrant | It'll be tmpFOO | 06:20 |
| StevenK | Right | 06:21 |
| StevenK | wgrant: Can haz review? | 06:21 |
| wgrant | Er | 06:21 |
| wgrant | Not ~/, but the tree | 06:21 |
| StevenK | Yes, I found it | 06:22 |
| StevenK | -rw------- 1 jenkins jenkins 3.4M Sep 27 06:22 .testrepository/tmpTDk8xw | 06:22 |
| wgrant | StevenK: Nothing else relies on diffstat being None sometimes? | 06:23 |
| StevenK | Hm, maybe | 06:24 |
| StevenK | wgrant: Good catch. http://pastebin.ubuntu.com/1229847/ | 06:28 |
| wgrant | Well | 06:28 |
| wgrant | This is interesting | 06:28 |
| wgrant | I think our bugtracker may be sentient | 06:29 |
| wgrant | https://bugs.launchpad.net/launchpad?field.searchtext=quote_like | 06:29 |
| wgrant | Look at the first result | 06:29 |
| wgrant | First thought was "these results are terrible". Open first result, "Cannot search for identifier containing underscores... well I can search for it but the results aren't any good." | 06:30 |
| StevenK | Haha | 06:30 |
| StevenK | wgrant: I can't see any other code that relies on diffstat being None | 06:31 |
| wgrant | Did you consider making lazr.restful deal with it? | 06:31 |
| StevenK | I did, and then decided it was a case of GIGO | 06:32 |
| wgrant | Well, we allow object references to be nullable | 06:32 |
| wgrant | Why not dicts? | 06:32 |
| StevenK | Like you say, it could be fixed in lazr.restful's marshaller. But I decided that fixing LP was a little easier | 06:34 |
| StevenK | And it didn't involve working out how to run its tests and then spinning a new release, uploading to pypi and download-cache and then making an LP branch anyway | 06:35 |
| StevenK | If you strongly object then I can do that on Tuesday | 06:37 |
| wgrant | StevenK: It's not that hard; wallyworld did it yesterday | 06:39 |
| wallyworld__ | and finished the job today | 06:40 |
| StevenK | I know it's not hard. It's easier to just change LP. | 06:40 |
| StevenK | wallyworld__: I don't think lp:lazr.restful contains your changes | 07:06 |
| wallyworld__ | hmm. i'm sure i merged, let me check | 07:06 |
| StevenK | Hmm, yeah, revision 200 | 07:07 |
| wallyworld__ | ah, merged restfulclient | 07:07 |
| wallyworld__ | forgot restful | 07:07 |
| wallyworld__ | let me do it now | 07:07 |
| StevenK | The version.txt in the tree and pypi did not match | 07:08 |
| StevenK | And I didn't want to unfix your carefully fixed bug | 07:08 |
| wallyworld__ | StevenK: done, sorry | 07:09 |
| StevenK | https://code.launchpad.net/~stevenk/lazr.restful/dict-unmarshaller-none/+merge/126615 | 07:15 |
| wgrant | StevenK: Why isn't None None? | 07:16 |
| wgrant | None shouldn't be {} | 07:16 |
| wgrant | It should be None | 07:16 |
| wgrant | probably | 07:16 |
| StevenK | Well, it's better than what we have currently, which is AttributeError | 07:16 |
| wgrant | Exposing an unstable interface in lazr.restful is not better than AttributeError :) | 07:17 |
| wallyworld__ | wgrant: StevenK: i had to reboot again after another unity crash and now lp.dev won't load in ff, but loads in chrome. "The connection was interrupted". any idea? | 07:19 |
| StevenK | Stop stabbing zope while loading it in Firefox? | 07:20 |
| wallyworld__ | i have no knife | 07:20 |
| wgrant | wallyworld__: Tried restarting Apache? That sometimes indicates it's SSL listening on a non-SSL port or vice-versa | 07:20 |
| wallyworld__ | wgrant: yeah, tried that | 07:20 |
| StevenK | wgrant: Diff updated | 07:21 |
| wgrant | StevenK: Have you tried this with lazr.restfulclient? | 07:22 |
| * StevenK de-prams his own toys | 07:22 | |
| StevenK | wgrant: No, I haven't | 07:23 |
| LPCIBot | Project devel build #1098: STILL FAILING in 1 hr 45 min: https://lpci.wedontsleep.org/job/devel/1098/ | 07:27 |
| StevenK | That isn't fun. | 07:28 |
| adeuring | good morning | 07:52 |
| === rvba is now known as Guest78352 | ||
| === almaisan-away is now known as al-maisan | ||
| === _rvba is now known as rvba | ||
| === al-maisan is now known as almaisan-away | ||
| === gary_poster|away is now known as gary_poster | ||
| === matsubara-afk is now known as matsubara | ||
| jtv | Say... I don't have a working launchpad setup at the moment. Would anyone be able to test & land this branch for me? https://code.launchpad.net/~jtv/launchpad/format-relative-imports/+merge/124878 | 13:17 |
| === Guest32361 is now known as slank | ||
| === slank is now known as Guest2828 | ||
| stub | jtv: sending via ec2 | 13:53 |
| jtv | ? | 13:53 |
| jtv | Oh, branch. Thanks! | 13:53 |
| stub | responsive r us | 13:53 |
| tumbleweed | stub: talking of responsiveness, have a moment to eyeball https://code.launchpad.net/~stefanor/launchpad/edit-packagesets/+merge/124555 ? | 14:04 |
| stub | tumbleweed: done | 14:12 |
| tumbleweed | stub: thanks | 14:13 |
| === jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: jcsackett | Firefighting: - | Critical bugs: ~300 | ||
| === slank is now known as Guest55818 | ||
| adeuring | jcsackett: thanks for your review! | 16:29 |
| * adeuring nearly forgot the MP | 16:29 | |
| jcsackett | adeuring: you're welcome. | 16:30 |
| === matsubara is now known as matsubara-lunch | ||
| === matsubara-lunch is now known as matsubara | ||
| lifeless | sinzui: https://bugs.launchpad.net/launchpad/+bug/736010 is affecting openstack | 18:28 |
| _mup_ | Bug #736010: Milestone:+index timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/736010 > | 18:28 |
| lifeless | sinzui: wondering if one of your timeout demolishing machines could cast an eye over it. | 18:29 |
| abentley | deryck: I'm getting a test failure on stable. Both test and code are old. Can you reproduce? http://pastebin.ubuntu.com/1230926/ | 18:30 |
| deryck | abentley, let me see.... | 18:32 |
| lifeless | sinzui: oh, I see ttx is in #launchpad talking abou tit | 18:35 |
| sinzui | lifeless, We have discussed that bug a lot. We think a lot of work is needed t fix this issue. This is a design problem, not a a bad query problem | 18:42 |
| deryck | abentley, I don't get that failure. | 18:44 |
| lifeless | sinzui: Why do you say its a design problem ? | 18:44 |
| lifeless | sinzui: I see 2000ms of time between query 66 and 67 in https://oops.canonical.com/oops/?oopsid=OOPS-c3db3abe60cbad08100ecb38fcac389a#statementlog | 18:45 |
| abentley | deryck: Wacky. It looks legit to me, but it also doesn't seem to happen on ec2. | 18:45 |
| deryck | WEIRD | 18:46 |
| deryck | sorry to shout, fat fingers :) | 18:46 |
| deryck | I wonder what's different for you that you see it? | 18:46 |
| sinzui | lifeless, the page only works if you can display all milestones and blueprints. It does not scale. We can improve the queries, but it still will not scale for large projects. I think a proper fix is to load just the crucial/first items, then use ajax to complete the listings, then do the analysis. | 18:47 |
| abentley | deryck: Maybe python version. Are you on quantal 2.7.3? | 18:47 |
| lifeless | sinzui: I am confused; the OOPS doesn't show queries as a problem at all. | 18:47 |
| lifeless | sinzui: Are we talking about the same issue ? | 18:47 |
| deryck | abentley, ah, no. I still have upgraded actually. :( | 18:48 |
| deryck | abentley, but I am on 2.7.3 | 18:48 |
| sinzui | yes we are, This bug has been around a long time and making queries faster does not fix the issue | 18:48 |
| lifeless | sinzui: ok, so we're slightly in sync :) - I see bad python code / tal code. | 18:48 |
| sinzui | You can fix that query, but it want users the believe you have fixed the issue for there hundreds of items, we need to think about the solution differently | 18:49 |
| sinzui | YES | 18:49 |
| lifeless | sinzui: I agree that doing latency hiding and incremental loading would be better. | 18:49 |
| sinzui | Late evaulation that because the page relies on bug and blueprint rendering rules | 18:49 |
| lifeless | sinzui: but is there not a low hanging just-be-more-efficient step to get Ubuntu and openstack unstuck ? | 18:49 |
| lifeless | sinzui: its not late evaluation, only 68 queries on the page before it timed out . | 18:49 |
| sinzui | I think we just want to pull the data and have the browser do the analysis...such as I do in the burndown charts I wrote | 18:50 |
| lifeless | sinzui: those charts are sweet ;) | 18:50 |
| sinzui | I don't believe the issue is low hanging. I have landed more than a dozen speed ups in 2 years. so have you and William | 18:51 |
| sinzui | I asks jcsackett to not leap into that bug given the tremendous failure rate | 18:51 |
| lifeless | hmm | 18:52 |
| lifeless | loaded for me | 18:52 |
| jcsackett | lifeless, sinzui: timeout fixed. | 18:52 |
| lifeless | 69 queries in 6.83 seconds | 18:52 |
| lifeless | jcsackett: how ? | 18:52 |
| jcsackett | as i said, was increasing the wrong hard_timeout rule. | 18:52 |
| * jcsackett misread the flags. | 18:52 | |
| jcsackett | the flag was for ProjectMilestone; we just now fixed milestone | 18:52 |
| jcsackett | well "fixed" ;-p | 18:52 |
| sinzui | lifeless, wgrant, jcsackett, and wallyworld__can discuss the effort to make that page properly fast for all project sizes in a few hours | 18:53 |
| lifeless | sinzui: my curiosity is piqued | 18:54 |
| lifeless | sinzui: I agree on the long term | 18:54 |
| lifeless | but I'm wondering why the short term is so poor | 18:54 |
| sinzui | certainty. | 18:54 |
| sinzui | 1 day of effort to fix one bug is high certainty to deliver value | 18:55 |
| sinzui | 5 days to fix 5 bugs is less certain, but equal value | 18:55 |
| sinzui | 10 days to fix 1 bug is even less certain to deliver less value. | 18:56 |
| sinzui | lifeless, we are a "machine" because we are taking the most certain bugs first. I asked jcsackett to not work on that one yet because there is more risk involved. | 18:57 |
| * sinzui hoped to fix another escalated bug before having to do feature analysis to fix criticals | 18:57 | |
| lifeless | sinzui: all cool | 18:57 |
| lifeless | sinzui: I meant, what about the code makes this one so poor | 18:58 |
| lifeless | sinzui: not about your team or your choices ;) | 18:58 |
| sinzui | we often choose to make pages do less, or batch. milestones do need to show all items. The tales calls are an embarrassment. I choose to reuse code to keep presentation consistent, but the adaptions are costly. | 19:00 |
| sinzui | lifeless, the analysis is costly in Python I think. I know I can do it in JS now. | 19:02 |
| lifeless | sinzui: I'm getting a profile | 19:05 |
| lifeless | https://qastaging.launchpad.net/++profile++show/ubuntu/+milestone/ubuntu-10.10/+index# | 19:06 |
| lifeless | just sneaked in: 3378720 function calls (3296251 primitive calls) in 19.845 CPU seconds | 19:06 |
| lifeless | sinzui: there are some low hanging fruit that might make a lot of things better (or might be profile artifacts) | 19:12 |
| lifeless | 5134 0.810 0.000 0.979 0.000 enumcol.py:39(__init__) | 19:12 |
| lifeless | ^ 4% of total runtime, I believe wgrant has complained about enumcol performance before | 19:12 |
| sinzui | yes | 19:12 |
| lifeless | a few more like that | 19:13 |
| sinzui | this page might break now that there are private bluerprints too. milestones do the private bugs query then cache the permissions, but I did not see such as change land for blueprints | 19:14 |
| lifeless | yay | 19:15 |
| lifeless | sinzui: the ubuntu timing out page is only 140k in size | 19:19 |
| abentley | jcsackett: I have a sequence of four branches that need review. Are you up for it? | 20:24 |
| jcsackett | abentley: if said sequence starts with new-tests, i'm already looking at it. | 20:24 |
| abentley | jcsackett: Excellent. Continues with https://code.launchpad.net/~abentley/launchpad/storm-sprint-queries/+merge/126770 https://code.launchpad.net/~abentley/launchpad/specification-cleanup/+merge/126789 https://code.launchpad.net/~abentley/launchpad/hide-sprint-blueprints/+merge/126792 | 20:25 |
| jcsackett | abentley: cool. i can get 'em all. | 20:27 |
| sinzui | jcsackett, do you have time to review https://code.launchpad.net/~sinzui/launchpad/email-authentication-errors/+merge/126801 | 21:05 |
| jcsackett | sinzui: sure, you're in the queue. | 21:06 |
| jcsackett | still looking through abentley's series. | 21:06 |
| sinzui | abentley, rick_h_. was the privacy banner or information portlets updated in the last few days | 21:12 |
| sinzui | JS is broken on pages that show the privacy banne like bugs and archives | 21:12 |
| abentley | sinzui: Yes, rick_h_ changed the banner code yesterday IIRC. | 21:13 |
| sinzui | fab, maybe the diff will help me find the problem | 21:13 |
| abentley | sinzui: (rick_h is away) | 21:14 |
| sinzui | okay, diff it will be | 21:14 |
| jcsackett | abentley: i'm through your series of branches. r=me on all but one which has a few questions. | 21:35 |
| abentley | jcsackett: I've replied. | 21:35 |
| abentley | jcsackett: Thanks! | 21:35 |
| jcsackett | abentley: missed both things you pointed out. given those, r=me on that one as well. | 21:38 |
| * jcsackett moves on to sinzui's. | 21:38 | |
| === jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: ~300 | ||
| sinzui | wallyworld__, If your waking: read this to learn what I have learned so far https://bugs.launchpad.net/launchpad/+bug/1057714 | 21:39 |
| _mup_ | Bug #1057714: javascript is broken on pages that show the privacy banner <bugs> <javascript> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1057714 > | 21:39 |
| wallyworld__ | sinzui: haven't read that bug but i think it may be a dupe of mine from yesterday, for the mp ypu approved | 21:40 |
| wallyworld__ | mine is bug 1057248 | 21:40 |
| _mup_ | Bug #1057248: javascript error on bugtask page <javascript> <regression> <Launchpad itself:In Progress by wallyworld> < https://launchpad.net/bugs/1057248 > | 21:40 |
| sinzui | wallyworld__, does this look familiar: http://pastebin.ubuntu.com/1231240/ | 21:40 |
| sinzui | This is the change, but I do not know why info_type is not defined in the privacy module | 21:41 |
| wallyworld__ | sinzui: yes. i changed privacy.js and information_type.js | 21:41 |
| sinzui | okay, | 21:41 |
| wallyworld__ | there were vrious issues - 'throw' was misspelt, lots of lint | 21:41 |
| sinzui | ah@ | 21:42 |
| wallyworld__ | plus the info_type issue | 21:42 |
| sinzui | ! | 21:42 |
| sinzui | okay | 21:42 |
| wallyworld__ | landing my branch now | 21:42 |
| lifeless | wgrant: enumcol would benefit from a cache I suspect | 21:42 |
| lifeless | wgrant: one of the rare times I'll say that | 21:42 |
| wallyworld__ | we can deploy this morning | 21:42 |
| sinzui | we can run the yui layer locally, then lp-land | 21:42 |
| wallyworld__ | sinzui: yes, doing that now | 21:42 |
| sinzui | sorry wallyworld__I remember the diff, but I did not read all the details int he bug. I suck | 21:44 |
| jcsackett | sinzui: r=me on your branch. | 21:45 |
| sinzui | wallyworld__, I was rushing to review branches because I had a day trip to look at houses | 21:45 |
| sinzui | thank you jcsackett | 21:45 |
| wallyworld__ | np | 21:52 |
| cjwatson | wgrant: We've flushed nearly all the queue now following beta-2, and it all seemed pretty snappy; no timeouts in 80+ accepts. At this point I'm confident with going ahead and removing the queue script. Is there any remaining review you feel the need to do on https://code.launchpad.net/~cjwatson/launchpad/remove-queue-tool/+merge/114464 ? | 22:54 |
| wgrant | cjwatson: Let me look | 23:01 |
| wgrant | lifeless: Hmm | 23:03 |
| wgrant | lifeless: I might try to profile it a bit harder now that someone other than me thinks it might be a problem. | 23:03 |
| rick_h_ | wallyworld__: man, so sorry. Just thought I added unused stuff. Tests passed, etc. I shouldn't have tried to rush that before I left | 23:05 |
| wgrant | cjwatson: r=me. Are you also going to remove the feature flag check? | 23:06 |
| wallyworld__ | rick_h_: no problem. shit happens :-) | 23:07 |
| rick_h_ | wallyworld__: I owe you one. broke it apart and tried to stage it in nice simple bits that didn't get too involved and boom | 23:08 |
| wallyworld__ | rick_h_: we may need to look at the events - seems there are 2 sets (show/hide banner, info type public/private) that do the same thing. may be able to be refactored to just one set | 23:09 |
| rick_h_ | wallyworld__: well so I can explain that | 23:10 |
| wallyworld__ | i didn't get into it too far | 23:10 |
| cjwatson | wgrant: That's going through EC2 at the moment (https://code.launchpad.net/~cjwatson/launchpad/pabj-remove-feature-flag/+merge/126090) | 23:10 |
| rick_h_ | because the filbug.js uses the privacy banner directly with a custom message, you need to be able to talk to the privacy banner with a custom message | 23:10 |
| wgrant | cjwatson: Ah, great | 23:10 |
| wgrant | I have a fair bit of MP mail; haven't quite caught up this morning yet | 23:10 |
| cjwatson | Yep. Thanks. I look forward to my giant LoC credit | 23:11 |
| rick_h_ | wallyworld__: but the normal use case is that someone changes the information_type widget, which fires the information_type:changed, and files information_type:is_public/private and the banner will auto show/hide based on that event | 23:11 |
| cjwatson | And I haven't even removed delayed copies yet ... | 23:11 |
| rick_h_ | wallyworld__: so there's really two use cases and almost should be a different banner, since privacy and security are sharing code tbh | 23:11 |
| lifeless | wgrant: there is a kcachegrind file for you | 23:11 |
| wallyworld__ | rick_h_: ok, thanks for the explanation | 23:11 |
| wallyworld__ | my main aim was to add the missing tests and get the fix out | 23:11 |
| rick_h_ | wallyworld__: and in the next branch the banners will render themselves vs the html being in the .pt file, so the events will be used as ways to say "need a banner html setup here" | 23:12 |
| wallyworld__ | cool :-) | 23:12 |
| rick_h_ | wallyworld__: right, understand. and it's not 100% tested because it's part 1 of 4 and in general this was 'new unused' code since nothing is firing the events yet | 23:12 |
| wallyworld__ | rick_h_: one thing. i had to move the info_type namespace variable from the top of the module or else it was undefined | 23:13 |
| wallyworld__ | i don't know why | 23:13 |
| rick_h_ | wallyworld__: so like I said appreciate it. Been offline most of the day on vaca so bummed to log in and see I did wrong for you guys | 23:13 |
| rick_h_ | wallyworld__: hmmm, let me look | 23:13 |
| wgrant | lifeless: But it's from qastaging, so it's not completely useful | 23:14 |
| wgrant | Not useless, though | 23:14 |
| lifeless | not useless | 23:14 |
| rick_h_ | wallyworld__: ok that's strange. I'll check it out when I get to the follow up branch when I get back | 23:15 |
| wgrant | rick_h_: Missing context here, but we only recently moved the banners *into* the HTML, to avoid them popping into the page later on once the JS loads. | 23:15 |
| wallyworld__ | rick_h_: thanks. i didn't get into the root cause. i'd love to know why | 23:15 |
| rick_h_ | wallyworld__: nothing looks odd, but running on 3hrs sleep after driving 600+ miles overnight | 23:15 |
| wallyworld__ | rick_h_: where did you drive from/to? | 23:16 |
| rick_h_ | wallyworld__: https://maps.google.com/maps?saddr=clarkston,+mi&daddr=Charlottesville,+VA&hl=en&sll=37.6,-95.665&sspn=38.593229,86.396484&geocode=FRAWjAIdYh8H-ymTC47xX5ckiDGKYHxzNWKthw%3BFfpHRAIdeopS-ymPpFDqLYaziTH8dIvDlvCGkA&oq=charlottesvi&mra=ls&t=m&z=6 | 23:16 |
| rick_h_ | doh | 23:16 |
| wallyworld__ | looks like a nice drive | 23:17 |
| rick_h_ | wgrant: yea, I'll hopefully be able to take care of that still. The big thing is we need banners on more places and it takes a bunch of code to wire it up it seems. Trying to split it into contained parts | 23:17 |
| rick_h_ | wgrant: for instance trying to not have the privacy banner need to know about the information type, LP.cache locations, etc. | 23:18 |
| wgrant | Indeed. | 23:18 |
| rick_h_ | wgrant: but thanks for the heads up. I didn't realize it was moved due to a flash effect so good to keep an eye on | 23:18 |
| rick_h_ | wallyworld__: yea, well with a 2yr old those overnight drives still the best so he sleeps through most of it. Thanksfully we also had the lion king movie soundtrack for backup lol | 23:19 |
| wallyworld__ | ah, those were the days. long gone for me now | 23:19 |
| wallyworld__ | rick_h_: wgrant still listens to the Lion King for his beddy byes time | 23:19 |
| wgrant | Heh | 23:19 |
| rick_h_ | hah | 23:20 |
| lifeless | wgrant: the other thing is there is a huge amount of content being re-rendered | 23:21 |
| lifeless | wgrant: a simple static result cache for rendering of person priority importance status might help a great deal. | 23:22 |
| wgrant | We can possibly just also make that not terrible | 23:22 |
| lifeless | or do both | 23:22 |
| lifeless | evaluating the same template with the same parameters in the same request to the same bytes seems non horrible to me | 23:23 |
| wgrant | Says he who removed template fragment caching :P | 23:23 |
| wgrant | (yes, yes, it was the wrong way to cache, but blah) | 23:23 |
| wgrant | :) | 23:23 |
| lifeless | I'm not against caches /per se/ | 23:24 |
| wgrant | wallyworld__: qas is updated, if you want to check out the JS stuff | 23:57 |
| wgrant | Not sure why it's [testfix] | 23:57 |
| wallyworld__ | finally, been waiting | 23:57 |
| wallyworld__ | wgrant: because db bb was borked and i needed to get it landed | 23:57 |
| wallyworld__ | we should fix that | 23:57 |
| wgrant | wallyworld__: In that situation you should force the build and wait 5 minutes | 23:58 |
| wgrant | [testfix] skips all QA | 23:58 |
| wallyworld__ | ok | 23:58 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!