[00:02] there shouldn't be [00:02] its a gpg config dir [00:16] gpg might make a symlink as a lock, but it looks like we're not actually running gpg in this test [00:16] i've been running it for a while and it has not failed [00:32] lifeless: how about https://code.launchpad.net/~mbp/launchpad/882324-mtime-test/+merge/80521 then [00:36] commented. Three times. [00:40] you'd do it that way because it will automatically give a clear message? [00:41] are you saying that you want me to keep the 'floor()' call, or do you want to have another attempt? ;-) [00:44] dropping the floor is fine [00:44] yes, it will automatically give a clearer message, and it eliminates another use of a poor primitive (assertTrue) === Ursinha is now known as Ursinha-afk [00:51] great, thanks [00:51] good idea in fact [00:53] is there a link from the ppa to the recipe? [00:53] i guess it's only off the branch [01:00] poolie: If an upload was created by a recipe, the package will link back to it [01:01] poolie: For example, https://launchpad.net/~launchpad/+archive/ppa/+packages and then expand any of the launchpad-dependancies [01:02] ok thanks [01:45] it would be nice to link back recipes that are autobuilding separately from successful builds [01:48] lifeless: Huh? [01:54] auto builds build into specific ppas [01:54] if they haven't successfully built, there is no link from the ppa [01:54] the package has to build for the link to show up [01:54] Ah, daily builds? [01:55] yes [02:02] https://code.launchpad.net/~lifeless/python-oops-amqp/0.0.3/+merge/80524 [02:10] lifeless: If the *recipe* built, but the packages resulting didn't, the link should still be there. === lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 266 [02:11] StevenK: indeed, I'm saying that waiting for that much success is still less clear than we could be [02:13] StevenK: can has review, as you are here? [02:13] or is it wgrants turn as wictum [02:13] * lifeless consults the oracle [02:13] thursday. Its me! [02:13] -> bah [02:14] StevenK: wgrant: ^ [02:14] Haha [02:14] * wgrant blames wallyworld_. === lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: me | Critical bugtasks: 266 [02:14] huh? [02:20] poolie: https://staging.launchpad.net/~wgrant/+archive/experimental/+recipebuild/87804 [02:21] bzr: ERROR: Cannot commit directly to a stacked branch in pre-2a formats. See https://bugs.launchpad.net/bzr/+bug/375013 for details. [02:21] <_mup_> Bug #375013: Cannot commit directly to a stacked branch

< https://launchpad.net/bugs/375013 > [02:21] Is that new? [02:21] I haven't used this recipe in more than a year, but it used to work... [02:22] looks like fallout from fixing commit to stacked branches in 2a; possibly due to a previously undocumented failure more [02:22] mode [02:22] bah https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-2126AT10 [02:24] lifeless: BranchRevision timeout... [02:32] wallyworld_: Can you update the kanban board? I think the cards in Review and Landing can move [02:32] * wallyworld_ looks [02:34] StevenK: the one in landing, i'm fixing tests. makeBug, makeBugTask need a bit of work to make them compatible with the new disclosure rules [02:34] wgrant: looks like it wasn't safe for 1.9 either according to comments inline [02:40] :( [02:41] e.g. that the only reason things worked on 1.9 was it being a no-change commit [02:41] a change commit wouldn't be able to delta correctly in all cases in 1.9 [02:43] I need a review - any volunteerS? [02:56] StevenK: can I beg a review off you? [02:57] lifeless: I suppose. [02:57] https://code.launchpad.net/~lifeless/python-oops-amqp/0.0.3/+merge/80524 [03:00] lifeless: r=me [03:00] thanks [03:08] (back) [03:09] wgrant: pre-2a formats are kinda deprecated now [03:09] i don't think that failure is very conclusive either way for qa [03:11] wgrant: is this about recipes using stacking and falling over when there is a pre-2a branch involved? [03:11] jelmer: Well, I was trying to QA the new bzr-builder, and my test recipe happened to fall into that category. [03:12] wgrant: that's not a new regression FWIW [03:12] k [03:12] Does someone have a less archaic recipe that they can reasonably test on staging? [03:13] wgrant: python-fastimport is one of my usual test recipes [03:13] I can have a look at adding that on staging, but not before dinner... [03:14] Ah, you're still in the US? [03:15] StevenK: Yay, p-d-r back down to 8 minutes. [03:15] Still sucks, but a bit better. [03:15] wgrant: yeah [03:16] o/ jelmer [03:16] wgrant: i can try mine on bzr [03:16] both should be modern formats [03:16] poolie: That works. [03:23] ok doing it now [03:23] Thanks. [03:29] ok https://code.staging.launchpad.net/~mbp/+recipe/bzr-daily exists [03:29] queued for build [03:37] building [03:42] wgrant: 8 minutes sounds pretty good to me. [03:43] StevenK: It is LP good. [03:43] It is not good. [03:43] wgrant: Pretty good compared to the 1.5 hour transactions we were having [03:44] It's considering less than 1500 items. There's no reason for it to take more than 30 seconds, and that's being generous. [03:44] what's p-d-r? [03:44] process-death-row [03:44] Deletes superseded publications [03:45] failed: https://staging.launchpadlibrarian.net/80692680/buildlog.txt.gz [03:45] It's the pool cleaner. [03:45] i guess because of dpkg-deb: error: control directory has bad permissions 700 (must be >=0755 and <=0775) [03:45] poolie: sudo breakage [03:45] lamooooont [03:45] lamont: ^^ [03:45] which control directory is it talking about? the ./debian in the build area? [03:46] probably ./DEBIAN. [03:54] wgrant: that yuixhr thing is still booming [03:57] lifeless: :/ [03:57] I found a think in my OopsHandler change [03:57] (is not None) s/not // to fix [03:57] thinko [03:58] * StevenK kicks Person:+index for being stupid [03:58] anyhow, that *might* be it, if yuixhr is logging warnings or something [04:03] oh ugh [04:03] has anyone seen a test fail with "error: pid file was not removed"? [04:03] so I just realised, if we want different vhosts for txlongpoll, I'll need separate config for oops-amqp [04:04] poolie: nope [04:04] wgrant: do you happen to know an ETA on txlongpoll being unbroken ? [04:05] i see muharem hit it in 2009 [04:05] just bad luck i guess [04:05] wgrant: So, where should I start digging for this +index madness? [04:05] StevenK: queries ? [04:05] lifeless: Hm? [04:06] StevenK: what sort of madness [04:06] lifeless: create_initialized_view(, '+index') doesn't work [04:08] StevenK: it does too [04:08] see TestPersonIndexView [04:08] Does that render the view? [04:09] oh, rendering doesn't work? [04:09] Doesn't seem to. [04:10] thats odd [04:10] uhm, there are additional parameters [04:10] well https://bugs.launchpad.net/launchpad/+bug/882370 [04:10] <_mup_> Bug #882370: spurious failure in pidfile.txt: "pid file was not removed" < https://launchpad.net/bugs/882370 > [04:11] poolie: again, what makes this spurious? Do you know that the thing actually stopped and didn't get lost? [04:11] wgrant: didn't you work on ddebs for ppas a bit ages ago? [04:11] what does "actually stopped and didn't get lost" mean? [04:11] poolie: we have a recurring issue with test helpers going awol and not shutting down [04:12] mwhudson: I did. [04:12] wgrant: did that ever get anywhere? [04:12] a pid file not being deleted is often also a symptom [04:12] mwhudson: For PPAs? It's believe to be finished, with a small fix Julian made last week. [04:12] wgrant: also if i say "ddebs" and "derived distros" in the same sentence, will the result be hollow laughter? [04:12] wgrant: oh cool [04:12] mwhudson: Not fully tested, and can't be at the moment, because DF is screwed. [04:12] Ah. [04:12] Primary archives are another matter. [04:13] They have an implementation constraint that I consider to be false. [04:13] perhaps we have different meanings of 'spurious' [04:13] A very onerous implementation constraint. [04:13] o [04:13] wgrant: i'm looking at https://blueprints.launchpad.net/linaro-ubuntu/+spec/linaro-platforms-lc4.11-ubuntu-leb-dbg-pkgs-support you see [04:13] these tests failed when i didn't (afaik) touch anything related to that test [04:13] It is said that ddebs must live in a separate archive. [04:13] and they're not consistently failing [04:13] poolie: that still fits the symptoms we've experienced. [04:13] mwhudson: Well, the Soyuz team can probably fix that oh wait. [04:13] wgrant: ok [04:13] Now now [04:13] are you saying this is not a bug, or are you saying it's a dupe of a known bug? [04:14] poolie: that said, the test helper has timing code all over it [04:14] mwhudson: So, PPA ddebs are probably a couple of months away. [04:14] poolie: I was investigating to assess that, yes. [04:14] mwhudson: Primary archive ddebs are an undefined period of time and convincement of IS and Julian away. [04:14] wgrant: thanks [04:15] wgrant: is there a LEP or anything for PPA ddebs? [04:15] so on the face of it, it seems spurious that this test would fail based off this change [04:15] imbw [04:15] or anything i can link to, for that matter [04:15] poolie: oh, I don't think your change provoked it [04:15] i can't see any known bug about this test [04:15] are you saying you didn't want me to file a bug? [04:15] Bug #747558 [04:15] <_mup_> Bug #747558: PPAs should create backtracable packages

< https://launchpad.net/bugs/747558 > [04:15] poolie: the question was whether it was related to the test reliability issues we've been having on and off, or a different low-frequency issue all of its own. [04:15] poolie: I think its all of its own and I've commented in the bug [04:15] poolie: filing the bug was the right thing to do [04:16] any test that has the line 'time.sleep(0.1)' is prima facie a critical bug ;) [04:17] poolie: tests that break in buildbot and cause 4 to 8 hour team stalls are, and for all that its low frequency this may well fit that [04:18] yep [04:18] this bug and the previous one caused ~5h stalls for me [04:18] if it had happened in bb it could have been worse of course [04:18] lifeless: So, initializing the view gives the output as "" [04:18] wgrant: i just read your explanation of how ddebs.ubuntu.com works [04:18] lifeless: i wonder if AWS is having some kind of underlying issue that is provoking more timing bugs [04:18] StevenK: No, *rendering* the view does. [04:18] lifeless: wgrant thinks that there is METAL madness involved. [04:18] mwhudson: Sorry, I should have warned you. [04:18] StevenK: sinzui's theory is possibly more relevant for this view. [04:18] eg something in the virtualization layer is making the machine clock irregular [04:19] wgrant: yeah, i may not sleep properly tonight [04:19] mwhudson: :) [04:19] two failures in a row seems suspicuous [04:19] poolie: that would be epic win wouldn't it ;) [04:19] or, perhaps they're just bogging down sometimes [04:20] eg this test waits only 2s for the thing to complete [04:20] or possibly even less [04:20] it's pretty plausible you could get >2s lag on a very loaded host [04:20] I need a rabbit sanity check [04:21] is it *possible* for rabbit to buffer messages in an exchange and send them to a queue that is subsequently bound to it ? [04:21] wgrant: My brain has melted, what was his theory? [04:21] lifeless: I imagine that such a race would be possible in rare circumstances. [04:22] wgrant: I have one nonreproducable failure. [04:22] StevenK: XRDS [04:22] wgrant: test_rosteta_branches_script_oops gets 7 oopses included some with storm DisconnectionError and similar in it [04:22] wgrant: my only theory is that having the exchange with -no- queue at all (e.g. tests in a too-low-layer) actually buffers [04:23] wgrant: eXtensible Resource Descriptor Sequence? Srsly? [04:27] StevenK: Correct. [04:27] ew doctests [04:29] wgrant: So where do I start? [04:29] StevenK: Work out what's returning "" [04:30] http://www.rabbitmq.com/tutorials/tutorial-three-python.html says 'The messages will be lost if no queue is bound to the exchange yet' [04:30] so that rules out that scenario [04:30] It doesn't. [04:30] wgrant: how so ? stale appserver helper s? [04:30] wgrant: So create_initialized_view() ; view() should be enough in the usual case, right? [04:30] 1) It's a tutorial. [04:30] wgrant: on the rabbitmq site [04:30] 2) It probably just means "You can't rely on messages not being lost" [04:31] StevenK: Yes. [04:31] I find reading what is written usually works. [04:31] lifeless: i realize it's intermittent but it offends me [04:31] lifeless: Not in a tutorial, and not from rabbitmq. [04:31] shall i just bump it up to waiting 20s? [04:31] poolie: I take it you also cannot see a cause [04:31] i've looked for races and i don't see any [04:32] sounds like a (painful) workaround [04:32] the fact that it's been there for >2 years but hit only intermittently suggests it timing related [04:32] agreed [04:32] especially as i had another perhaps timing related thing today [04:32] i was wondering about making it also report if the process still exists [04:33] that might help with diagnostics [05:01] wgrant: I think lib/canonical/launchpad/webapp/publisher.py, line 296 is to blam. [05:01] *blame. [05:02] "Oh, Person:+index is a redirect, return u''" [05:02] StevenK: you may need to pass in a layer (not test layer) [05:02] StevenK: pdb will tell you... [05:02] But I doubt it. [05:02] StevenK: this is one of the extra params IIRC. Or something. [05:03] wgrant: I discovered it because of pdf [05:03] *pdb [05:03] I've been tracing for a while [05:03] Hm, or not. [05:03] return self.render() [05:04] It's probably feeds or XRDS interacting badly, but step through in pdb and find out. [05:06] lifeless: https://code.launchpad.net/~mbp/launchpad/882370-pidfile/+merge/80536 [05:06] :/ [05:06] not the most satisfying series of patches [05:06] i'm going to run some errands === almaisan-away is now known as al-maisan [05:13] wgrant: I'm in the middle of zope.tales, so it must be doing useful stuff [05:16] Indeed. [05:16] Which suggests that it may in fact be METAL. [05:18] if current_url != expected_url: [05:18] self.request.response.redirect(expected_url) [05:18] return '' [05:19] lol [05:19] I found that within 5 seconds of you. [05:19] In lib/lp/services/openid/browser/openiddiscovery.py [05:19] Yes. [05:19] Was just switching over to paste that same thing. [05:19] Haha [05:20] Although I was 3 seconds later than you :( [05:21] I'm on my laptop due to my desktop being busy with this evil program called dd, so I was a little slower than pasting [05:21] Heh [05:21] So, that would do it. [05:21] Indeed. [05:23] Current URL: http://127.0.0.1/ [05:23] Expected URL: http://launchpad.dev/~team-name-150898 [05:23] Strangfe [05:23] s/f// [05:24] Right, the fake request (LaunchpadTestRequest, probably) used by create_initialized_view doesn't have a sensible URL. [05:24] Because most things are sensible enough to not require one. [05:24] Right, I was just thinking that LTR is probably implicated. [05:25] Can't we create a request and toss that into create_initialized_view()? [05:26] I have this feeling I've seen that done. *Where* ... [05:29] You could also change create_initialized_view to set the URL to canonical_url(obj, view_name) [05:30] create_initialized_view takes a server_url param [05:30] lifeless: thanks [05:31] wgrant: I'm not going to move the lp_sitecustomise line; I worry its order dependent because its doing monkey patching. [05:31] wgrant: or am I wrong ? [05:31] wgrant: ah, I'm wrong. [05:32] lifeless: I thought that as well, but it's in the sort of file where that would surely not be well-placed. [05:35] RARGH [05:35] Current URL: http://launchpad.dev/~team-name-333852/ [05:35] Expected URL: http://launchpad.dev/~team-name-333852 [05:37] They both call canonical_url, why does one have a slash and the other doesn't ... :-( [05:37] The / will be because you asked for the +index view. [05:37] I suspect the other one just asks for the object. [05:39] if not server_url: [05:39] server_url = canonical_url(context) [05:39] expected_url = canonical_url(self.context) [05:41] If both contexts are the same, the URLs should be equal [05:41] wgrant: I've just pushed rev 14150 if you want to do an incremental review. I've added autosync to the attachOopses() call, which will hopefully result in these naughty oopses being sucked up earlier. [05:42] wgrant: I've added de-duplication of oops ids to accomodate that [05:44] lifeless: I considered asking for that, but it sounds like it could be slow. [05:44] Although I guess it shouldn't be. [06:12] wgrant: if it passes ec2, I should have some timing data [06:12] wgrant: so we can assess, and we'll know [06:12] lifeless: To an extent. [06:12] But yes. [06:13] Wow. [06:13] The security declarations for Bug are... special. [06:14] Notably, there are some attributes listed that I've never heard of. [06:14] And they are in no order whatsoever. === al-maisan is now known as almaisan-away [07:11] Any reviewers in the house? https://code.launchpad.net/~jtv/launchpad/pre-876594/+merge/80537 [07:13] "me" is new. [07:13] hah, ok, lifeless. [07:15] Thanks for figuring that out, Nigel. :) [07:15] lifeless: could I trouble you for a review? It's this one: https://code.launchpad.net/~jtv/launchpad/pre-876594/+merge/80537 [07:15] Not very exciting stuff, I'm afraid. === lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 266 [07:15] sure [07:16] You had me there for a moment. [07:16] heh [07:16] well, its 8pm, well past EOD [07:16] even for extended-batteries-lifeless [07:16] I'd forgotten to update the topic [07:27] jtv: done [07:34] thanks lifeless === rvba changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rvba* (allenap) | Critical bugtasks: 266 [07:37] morning rvba [07:37] Hey jtv. [07:38] rvba: I didn't make very much progress on your query, except to find a sort of rock-bottom reason why it's slow. [07:38] See email. [07:38] Saw your email, with many more details, your reach the same conclusion I did: rows=336922, that's a huge number. [07:39] You may be wondering why I said it's not worth counting if the owner owns as little 1% of the branches. [07:40] So I suppose the comment that lifeless put on the bug itself (that the is a way to use StormRangeFactory (tweaking involved)) is the only way to go. [07:40] jtv: Yes I do. [07:40] It's because with 1% of the branches, you could still end up touching a large portion of the table pages. [07:40] Worst case, you need to touch 1 branch per page. [07:41] If you can skip the count, great. [07:41] Well, using StormRangeFactory would avoid the count, I suspect the perf of the page would be dramatically improved. [07:41] Probably, yes. [07:42] You'd also be rid of the OFFSET. [07:42] Which could matter if we just grew a few hundred thousand branches recently. [07:43] private branches will be getting overhauled by disclosure [07:43] I suggest talking to them about the schema changes planned - the schema drives performance here in fairly significant ways [07:45] The 300K branches were private on dogfood but public on staging. They're only a problem when they're public, but I assumed they'd be public in production. [07:45] The private-branches part isn't much of a performance problem, apart from a lot of repetitive querying. [07:45] All the explains I've done are on production. [07:46] So this isn't tied to branch privacy. [07:46] jtv: I did a previous optimisation pass in june or so, because its a problem :) [07:46] jtv: but if you've newer data than I, great. [07:47] Not for what rvba is looking at. In other words, you probably solved the problem as far as private branches are concerned and we're left with public branches as the bottleneck. [07:47] The small improvement I did with counting private branch (simply caching the results) will help a little. [07:48] But like jtv said, the main problem is counting the public branches. [07:48] It got a bit faster when I omitted the "transitively_private is false" condition from the public-branch count, [07:48] but still not as fast as counting public and private branches separately & adding up the numbers afterwards. [07:50] rvba: just a style thing - I'd hesitate to use a cached property on GenericBranchCollection [07:50] rvba: I suspect it could easily lead to incorrect behaviour [07:50] rvba: its really a fancy query builder; any caching should be done on the resulting query [07:51] lifeless: my goal was to cache the results of a subquery that is repeated more than 10 times. [07:52] rvba: why is it repeated 10 times? [07:52] rvba: also are you aware that passing in long IN clauses can trigger inappropriate sequential scans ? [07:52] rvba: [so if you expect thousands of entries in an IN clause you may need to move to a temp table, insert, analyze, then query] [07:53] rvba: I certainly agree with jtv's suggestion that a union may be better here [07:53] Right, I still need to make sure that the worse that can happen with the IN query is a few thousands. [07:54] I've not used a with query because I wanted to improve lots of queries at once. [07:54] as a data point, the last time I touched this code I fixed one page and broke another [07:54] qa on this will be tricky [07:55] ah, ubuntu-branches, \o/ [07:55] And I confess I'm not sure why I have so many different queries using this subquery, I must say that this code is a little bit spaghetti like, but anyway, this was just a first attempt I made before jtv & I realised this was really not where the improvement should be done. [07:55] k [07:56] so the plan issue is the hashed SubPlan 1, I think [07:56] I think the potential problems of this IN query optimisation are probably not worth the improvement it brings … I shall see if stormrangefactory can be applied. [07:56] select count(*) from branch where owner=2866082; [07:56] count [07:56] -------- [07:56] 338016 [07:56] (1 row) [07:56] Time: 284.695 ms [07:57] 343ms with the lifecycle status [07:57] i've only touched the surface of if so far, but this stormrangefactory is a very, very nice piece of work I must say. [07:57] this is consistent with my previous memory [07:57] That's a lot faster than on the test servers. [07:57] the public branch is not the issue [07:58] That's a typical 1s query that the page is doing: [07:58] 710ms with the transitively_private check added, which is odd. [07:58] https://pastebin.canonical.com/54885/ [07:58] The related explain: https://pastebin.canonical.com/54886/ [07:59] I suspect if you add the lifecycle_status check it will take 1sec. [08:00] 743ms with owner + lifecycle_status + transitively_priate [08:00] lifeless: the slowdown from the privacy check may be something to do with access patterns within the page. Do you know off the top of your head where a tuple's visibility-related info is stored? Maybe it's a compact blob at the beginning of the page. [08:00] I said typical because like I said earlier, many similar count queries are generated by this page (https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-2124AO73) [08:00] jtv: you thinking a repack or something ? [08:01] Wasn't, but always a nice thought. :) [08:01] Oh, I think I know. [08:01] jtv: I don't know that bit of the layout, no. [08:01] checking private (the old column) isn't visibly faster [08:02] good morning [08:02] Morning adeuring! [08:02] 480k branches in total [08:02] hi rvba! [08:02] so this is getting 66% or so of pages [08:02] lifeless: IIRC there's a shortcut for determining that all tuples in a page are visible. We won't make full use of that without index-only scans, but it probably still means that the liveness check has better locality than actual tuple data check. [08:02] hi adeuring [08:03] hi jtv [08:03] Once you start looking at the actual data, you do need to look at the tuple's data. [08:03] Until then, just visibility metadata. [08:03] anyhow, given we're getting 2/3rds of the data, the seq scan is appropriate [08:04] Absolutely. [08:04] Even for a few % of the data it would be appropriate. [08:04] Why aren't we ever discussing clustering? [08:05]