=== wgrant changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 238 - 0:[######=_]:256 [00:36] sinzui: Did you typo your MUA's display name? [00:36] You've lost your capital "c" [00:40] sinzui: Nice presentation. [00:57] wgrant: ? [00:57] i filed https://bugs.launchpad.net/launchpad/+bug/812597 [00:57] <_mup_> Bug #812597: gina imports from debian arrive too slowly < https://launchpad.net/bugs/812597 > [00:59] poolie: It would only be 7 hours if Debian decided to shift their publishing schedule to right after gina runs. [00:59] They still only publish every 6 hours, AFAIK. [01:00] i see [01:00] and we're pretty sure we're in sync with that? [01:00] It requires checking [01:00] The current schedule was devised to run just after they published. [01:00] But they like to change. [01:00] Without telling anybody. [01:13] Doesn't Debian have a push-mirror feature for some mirrors that runs right after publication? Could one host a mirror, and then use post-mirroring hooks to call gina at the right times? [01:14] wgrant: The amount of QA to do makes me sad. [01:58] hmm... is there any way to test emails on qa staging? [01:59] yes [01:59] for in you need losa to run the script by hand [01:59] for out you also need to run the relevant script by hand [02:00] but then you can access the shared mailbox to see what was sent out [02:02] sinzui: Anything to watch out for in oneiric atm? [02:02] dev/run, ecryptfs were fragile recently, may not be totally sorted yet [02:02] wgrant: what presentation ? [02:03] lifeless: sinzui's velocity presentation from the Prague epic. [02:03] ah yes [02:03] +1 [02:03] I use LUKS rather than ecryptfs, because it's flaky and slow. [02:04] lifeless: Ok, I would like to get a notification of a tag change from a bug. What scripts do I need to run? Is there a wiki page for this? [03:12] StevenK: QA! [03:12] wgrant: I have a better idea. LUNCH! [03:12] I was just doing a pre-lunch QA check myself. [03:12] I shall update mawson now. And then break for lunch. [03:13] Thanks. [03:13] Huh. [03:13] Julian used one of Job's YAGNI columns. [03:13] Indeed [03:14] which one ? [03:14] requester [03:39] Hm. [03:39] Has someone decided to show the battery indicator at all times in oneiric? :( [03:39] * wgrant blames thumper. [03:39] heh [03:39] wgrant: not me though [03:39] but at least I have a battery indicator now [03:39] Bah. [03:49] thumper: global menus fixed yet? [03:49] nah mate [03:50] * wallyworld sad [03:55] wallyworld: Hm, we have a picker regression. [03:55] ? [03:55] https://qastaging.launchpad.net/launchpad/+filebug, the "Assign to" picker has quotes around the text. [03:55] Not there on prod. [03:56] Yippie, build fixed! [03:56] Project db-devel build #731: FIXED in 5 hr 31 min: https://lpci.wedontsleep.org/job/db-devel/731/ [03:56] Normal bug assignee pickers are OK. [03:56] https://code.launchpad.net/~wallyworld/launchpad/picker-displays-null-812253/+merge/68318 should fix it [03:56] wgrant: How do you suggest I QA r13455? [03:57] StevenK: Reject a delayed copy. [03:57] wallyworld: Ah, great. [03:57] That means I need to create one [03:57] StevenK: Easy enough to do. [03:57] And delayed copies make me cry. [03:57] Yes [03:58] wgrant: So copy something from the soyuz-team P3A to somewhere else? [03:58] StevenK: Something that's not already public, yes. [03:58] * StevenK digs [03:59] Why is syndaemon enabled by default :/ [03:59] How stupid. [04:08] Delayed copy of launchpad-dependencies - 0.39 (source) [04:09] Good enough === almaisan-away is now known as al-maisan [04:11] wgrant: But I can't find that delayed copy using queue? [04:14] Yippie, build fixed! [04:14] Project devel build #898: FIXED in 5 hr 34 min: https://lpci.wedontsleep.org/job/devel/898/ [04:17] StevenK: You'll have to get the ID manually, I guess. [04:17] Hopefully that will work. [04:22] wgrant: scripts/ftpmaster-tools/queue reject -s hardy -Q accepted 2752292 worked fine [04:22] Running: "info 2752292" [04:22] Item 2752292 is in queue REJECTED [04:28] Great. [04:28] qa-ok, I s'pose. [04:28] Already done [04:31] * StevenK kicks bigjools for the fun that is r13459 [04:31] StevenK: It is flagged. [04:32] Then what's left? [04:35] lifeless: LXC works fairly well in oneiric. [04:35] With most of those bugs fixed. [04:35] StevenK: Check that the methods don't work, I guess. [04:39] wgrant: For r13459? [04:40] Yes. [04:41] And lib/lp/soyuz/interfaces/archive.py contains the generic exception ForbiddenByFeatureFlag. [04:42] Fail. [04:42] Did somebody break the JS build again? [04:42] Oh? [04:43] Ah, no, I just killed buildout in a bad place. [04:46] wgrant: I'm hoping my sudo bug is fixed [04:48] lifeless: Not sure. [04:52] StevenK: Erm. [04:53] wgrant: Hm? [04:53] StevenK: [DOGFOOD] [PPA stevenk] [ubuntu/hardy] launchpad-dependencies 0.39 (Rejected) [04:53] There was a blamer. [04:53] Or SPR creator, possibly. [04:53] flacoste: [04:53] That is the creator [04:54] wgrant: Neither copyPackage{,s} appears in the API on dogfood for devel :-( [04:56] StevenK: You'll probably need to force it to rebuild the WADL [04:56] StevenK: But this is to be tested on qastaging. [04:56] Oh, easy fixed [05:00] wgrant: On qas, I get a 403 calling IArchive.copyPackage() [05:01] Good. [05:02] Ah yes, the exception is 403 [05:02] Shall I qa-ok it? [05:08] StevenK: Might as well. [05:17] GAVIN! [05:18] (The 3 branches linked to bug 809786) [05:18] <_mup_> Bug #809786: Need to filter by package-set on DistroSeries:+localpackagediffs

< https://launchpad.net/bugs/809786 > [05:18] 518 lines, 4720 lines, and 2878 === al-maisan is now known as almaisan-away [06:12] Curious. [06:19] StevenK: Hm? [06:20] wgrant: I finally got +localpackagediffs to load, but I can't tell what Gavin changed/fixed [06:20] StevenK: It was pure refactoring that's landed so far, AFAICT. [06:20] All 6000 lines of it [06:24] wgrant: So qa-untestable? [06:25] Probably. [06:25] wgrant: So the only thing left is Danilo's revision of doom === jam1 is now known as jam [07:48] good morning [07:50] morning all [07:50] I'm trying to understand an LP api timing. I could have sworn I was doing the same request last week, and it was returning in 100-200ms. [07:50] hi jam, hi adeuring! [07:50] But this week it is more like 700-1000ms [07:50] hey jtv [07:50] is there any way to debug these sort of things? [07:50] hi jtv [07:51] jam: What sort of request? [07:51] Can we use ++profile++ on API requests? [07:51] jam: 100-200ms is rather quick for LP. [07:51] jtv: Possibly... ++oops++ doesn't work, since it only executes when the template renders. [07:51] wgrant: getPublishedSources [07:52] jam: Let's see. [07:52] Everything should be faster this week :( [07:52] That's on Archive? [07:52] Could be distroseries. [07:52] Or distribution. [07:52] I guess. [07:52] StevenK: Thanks for looking at the changes to +localpackagediffs. As long as it loads, it's fine. The changes so far are almost all moves and shakes. [07:53] allenap: GAVIN! *Scary* branches! RARGH! [07:53] allenap: But yes, it at least loads [07:53] wgrant: https://api.launchpad.net/1.0/debian/+archive/primary?distro_series=%2Fdebian%2Fsid&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=1 [07:54] Archive [07:54] Yuck. "clauses.append("SourcePackageName.name LIKE '%%%%' || %s || '%%%%'" % quote_like(name))" [07:54] But this is exact_match. [07:54] So that should be skipped. [07:54] (it also shouldn't exist at all, but such is Soyuz) [07:54] Do we actually know we're looking at the archive code, actually? [07:55] Yes. [07:55] debian/+archive/primary [07:56] jam: Render time is just about identical compared to 1.5 weeks ago. [07:56] jam: Do you have any hard data to support the slowdown? [07:57] The only recent direct change in that piece of code was in June, where it looks like multi-name matching was introduced. But that should barely touch the exact-match code path. [07:58] Unless the thing gets passed some kind of placeholder that acts like a string but isn't of type str or unicode, but I don't really know whether that's possible or whether it would work if it was. [08:01] AND SourcePackageName.name = E'bzr' AND [08:01] Good morning [08:01] From using that URL locally. [08:01] morning al [08:01] l [08:03] wgrant: I know I did timing tests last week, and had some perf tests. [08:03] but no, I don't have very hard data. [08:05] wgrant: take that back, I have my .bzr.log [08:05] from today: 6.778 LatestPublication.get_latest_version took 1.524s [08:05] Oh, this is for the checking if the branch is up to date thing? [08:05] from last week: 7.974 LatestPublication.get_latest_version took 0.214s [08:05] wgrant: yep [08:05] Have you moved to Australia lately? [08:05] so the ones I was testing before is "ubuntu/natty/bzr" [08:06] Oh. [08:06] That's a bit different :) [08:06] but that is still slow [08:06] 700ms orso [08:06] :( [08:06] jtv: Storm contains a Like operator that might spell that better if you are coding nearby. [08:06] just tested now 939ms [08:06] stub: I'm not, but that's part of what made me nauseous. [08:07] It's even easier than Like; IIRC you can use the regular str methods such as startswith. [08:07] There's one for "contains" as well. [08:07] I recently used .contains() [08:07] wgrant: with bzr.dev tip, you can run this script: http://paste.ubuntu.com/647087/ [08:07] even better [08:07] wgrant@carob:~$ time curl -H 'Authorization: there is none' 'https://api.launchpad.net/1.0/debian/+archive/primary?distro_series=%2Fdebian%2Fsid&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=1' -s -o /dev/null [08:07] real 0m0.253s [08:08] (python latest_version.py ubuntu natty bzr) [08:08] Someone went to the trouble of coding up a hybrid storm/sqlobject method so that they could write up that horrible LIKE clause. [08:08] jtv: Can you point out the method so we can have it shot? [08:09] jam: So, it takes around 250ms from the DC. [08:09] StevenK: Archive.getPublishedSources [08:09] jam: For natty it's down around 150ms, as expected. [08:09] Since there's a lot less data. [08:09] StevenK: please don't shoot that one [08:09] (this isn't exactly optimised) [08:10] It deserves it. [08:10] wgrant: I'm seeing 600ms in "_ssl.sslwrap" is that the SSL handshake ? [08:10] jam: I assume so. [08:10] so that is very different from last week, I believe [08:11] Sure your connection latency doesn't suck? [08:11] wgrant: I'm in Netherlands [08:11] ping says 36ms [08:11] How long does the curl I listed above take? [08:12] 321ms I believe [08:12] And then try a couple of times without the Authorization header. [08:12] (the presence of even an invalid one gets it past Squid) [08:13] wgrant: but that is because curl fails immediately with "SSL FAILED" [08:13] morning bigjools! [08:13] Helpful. [08:13] hello jtv [08:13] jam: want to meet up later this week? [08:13] but I can pass -s, I guess [08:13] I still don't get any output to tell if it is working [08:14] jam: Well, probably better to make it work rather than ignoring the error... [08:14] :) [08:14] I wasn't sure what -s did [08:15] apparently it is --quiet not "--ignore-certificate" [08:15] wgrant: with -k instead of -s, I get 892ms [08:15] I get no cert error locally or on carob. [08:15] wgrant: launchpad isn't in my cacerts [08:15] -k works fine for testing [08:15] Mine are stock Natty. [08:17] so, still the same time for a successful request, about 900ms, which could be 600ms in ssl handshaking [08:17] which is definitely slower than last week [08:17] I don't know why [08:18] is there a graph/monitoring of ssl stuff? [08:20] Internally it's fine. [08:23] wgrant: right, so from devpad, I see 148ms [08:24] 900ms from 150ms away [08:24] So 650ms overhead, which is about four ways... [08:24] wgrant: I'm 38ms away by "ping" [08:24] so 17 round tirps [08:24] trips === almaisan-away is now known as al-maisan [08:26] Odd. [08:26] Still, fine in and next to the DC. [08:26] So I am tempted to blame your machine/connection. [08:27] .nl hasn't imposed a Great Firewall lately? :) [08:33] :( [08:33] zope.app.testing fixed their Referer default. [08:34] So 350 or so tests fail with OffsiteFormPost errors. [08:47] vila: can you try the query from France? [08:47] otm [08:47] time curl -k 'https://api.launchpad.net/1.0/debian/+archive/primary?distro_series=%2Fdebian%2Fsid&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=1' -o test.txt [08:48] jam: real 0m0.532s [08:48] (from Paris) [08:49] rvba: Is that after a couple of tries? [08:49] real [0m0.362s - 0m0.532s] [08:49] wgrant: that's the range I get [08:50] After a couple of tries: real 0m0.344s [08:50] That's a bit better. [08:51] wgrant: of course *right* now it just dropped to 300ms, but now curl varies from 300-953ms [08:52] and it still is taking 700+ from python [08:53] which still doesn't get close to the 150ms from last week [08:55] Was this on the same machine in the same environment? [08:55] yes [08:55] actually, the only difference offhand is that now I'm on a wired connection to my router, while last week I was on wireless [08:56] I guess it was 200ms last week, which is close to 300. [08:56] the variability is surprising [08:56] and last week was python, not curl [08:56] I get 50ms from squid. [08:57] Which sucks, but it's always within 1ms. [08:57] what do you mean by "from squid" ? [08:57] We have a proxy in front of the appservers [09:00] StevenK: sure, but is he saying bypassing the squid proxy, the appservers respond in 50ms [09:00] or from the squid machine [09:00] or he gets 50ms making a request to squid without an appserver [09:00] No, he is saying that squid replies in 50 [09:00] 50ms to get the copy that squid has cached. [09:00] ie. after requesting a few times without Authorization [09:03] wgrant: my requests also were without auth, but you're saying from where to squid? [09:03] carob to squid [09:03] Via the usual HTTPS frontends [09:04] wgrant@carob:~$ time curl 'https://api.launchpad.net/1.0/ubuntu/+archive/primary?distro_series=%2Fubuntu%2Fnatty&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=1' -s -o /dev/null [09:04] real 0m0.050s [09:04] (Debian is, unsurprisingly, identical) [09:05] sure. Though that just dropped as well [09:05] it was 150ms for me when I first reported devpad numbers [09:05] even running a couple of timse [09:05] times [09:06] wgrant: so I still think there is something that is slowing queries down, that magically disappeared in the last hour [09:08] eg, we're missing some monitoring somewhere that would tell us something is going on for people actually getting a much slower launchpad [09:09] Even in front of squid? [09:09] wgrant: I ran the same query, and it was 150ms from devpad to api.launchpad.net [09:10] Hmm. [09:10] I think I managed to provoke squid to skipping just now, and I got 400ms [09:10] (that was to debian, though) [09:11] but I'm playing with setting Authorization, and I'm getting 50ms [09:11] jam: first try: 1.355s real, then 0.249s, 0.199s, 0.194s, 0.189s , 0.198s [09:11] so it sounds like squid doesn't actually pay attention to it. [09:11] vila: right, so those sound like cached and uncached responses. [09:11] further tries stabilize at 0.200s [09:11] and arguably we should be really counting the uncached values as our user-experience. [09:11] we might need to put this behind a flag [09:13] vila: I couldn't figure out what you meant by OTM, on-the-mumble? [09:13] jam: yeah :) [09:13] jam: Apache is configured to send anything with Authorization or Cookie straight to haproxy, not squid, AFAIK. [09:13] wgrant: tell that to my curl results :) [09:14] though I could be typing something wrong [09:14] ah, Authentication not Authorization [09:14] my bad [09:15] right, that does bypass it [09:15] that's 479ms stabilizing to 230ms on carob bypassing squid requesting debian urls [09:15] which is hopefully worst-case in DC times [09:16] I clicked "Build now" on https://code.launchpad.net/~testing-cabal/+recipe/testtools-daily and get what looks to be a JS alert saying "Server error, please contact an administrator" [09:16] what's going on? [09:21] wgrant: ok, this is weird. curl is going to 91.189.89.224, but python is going to 91.189.89.225 [09:21] and *that* seems to be the difference in timings [09:22] it is "banana" vs "nutmeg" [09:22] Yes, they are two frontends [09:22] StevenK: any idea why banana would be 300ms slower than nutmeg? [09:24] Both 50ms cached... [09:25] And both 200-250ms uncached. [09:28] stub: hi [09:29] lifeless: hi [09:29] stub: I thought I'd touch base on 'kick connections out and keep em out' implementation choices [09:34] anyone got any ideas on where to look to investigate why an api method I added is not available over the wire to dogfood? It's in dogfood's WADL but launchpadlib says "'Entry' object has no attribute ...." === gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugs: 238 - 0:[######=_]:256 [09:35] wgrant: how do you get curl to target a server? [09:35] can you just do the IP address, or are they vhosts? [09:36] (I get ACCESS DENIED going to the IP address.) [09:36] jam: They're vhosts on special IP addresses. [09:36] time curl -H 'Authorization: blah' -H 'Host: api.launchpad.net' 'https://91.189.89.225/1.0/debian/+archive/primary?distro_series=%2Fdebian%2Fsid&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=1' -k -o /dev/null [09:39] wgrant: thanks. So bypassing SQUID going to ubuntu/natty I get a reliable 470-530ms to either server (nutmeg or banana) [09:41] and without Authorization, I get 330-360 from curl [09:41] so something about python is provoking something [09:46] lifeless: I think we are shutting down pgbouncer, which kills all the connections with 'connection terminated unexpectedly' exceptions and future connection attempts will timeout as nothing is listening. The 'suspend' and 'pause' don't terminate the active connections and ignore the 'fail if unable to login in this many seconds' parameter. [09:46] stub: ok [09:47] stub: So we'll need the test fixture to be able to shut down and bring back the one config; I'll add that into a 0.2 tomorrow if noone else gets to it [09:47] lifeless: I don't know if you'll get chance, until your tomorrow, but I've replied to your review of https://code.launchpad.net/~gmb/launchpad/bug-491886/+merge/68174. [09:48] lifeless: We want a pgbouncer test fixture? [09:48] stub: we have one [09:48] stub: the project I put together on Monday [09:48] stub: so we can test this well - we don't need the whole test suite to use pgbouncer, just the tests for interrupt & reconnect detection [09:50] Right. We used to have some disconnection/reconnection tests that used a simple twisted proxy, but this moved to Storm. We don't need to use pgbouncer if we don't want - as far as PG clients are concerned, they just have a socket. [09:50] stub: its pretty quick to bring up, and the code is written [09:51] stub: I suspect we only need 3-4 tests per daemon [09:53] lifeless: any chance you can help me debug a webservice problem on dogfood please? [09:54] I know it's rather late for you [09:54] sure [09:54] sec [09:54] gmb: reviewed [09:54] gmb: generally, I'm happy for any other reviewer to do a follow up once you consider (and action or reject) my suggestions [09:55] lifeless: Thanks, and duly noted for the future. [09:55] gmb: I don't think you need to consider the review pinned to me (unless I specifcally say I want that) [09:55] Okiedoke. [09:55] gmb: I think the try:except: looks a lot simpler :) [09:55] bigjools: whats up ? [09:56] lifeless: Agreed. I was trying to avoid the transaction.abort() but simplicity beats I-don't-want-to-have-to-deal-with-transactions. [09:56] lifeless: I added IArchive.copyPackage to the API. It's on DF's wadl/+apidoc but launchpadlib can't see it :/ [09:56] bigjools: delete your lp lib cached [09:56] bigjools: Remove your cached WADL? [09:56] GAH [09:56] this old chestnut [09:57] why why why [09:57] if it works, file a bug [09:57] I would -so- like to nuke the use of WADL, but I'm old school. [09:57] anything with XML in it reeks of shit [09:58] It lends itself so well to ridicule, though. "qas is WADLing" [09:58] gmb: oh, and the linty stuff I don't care about really, I mentioned it for completeness [09:59] yeah that worked. thanks lifeless [09:59] bigjools: file a bug ? [09:59] yeah doing so [10:00] \o/ [10:00] but on where [10:00] launchpadlib [10:00] its the thing choosing to use wadllib etc; and its the thing we need a versioned dep on to ensure a newer wadllib if thats how we fix it [10:03] could somebody help me land https://code.launchpad.net/~cjwatson/launchpad/multiarch-translations-schema? [10:04] lifeless: bug 812799 [10:04] <_mup_> Bug #812799: Local cache not updated when it should be < https://launchpad.net/bugs/812799 > [10:04] bigjools: triage it ! :) [10:04] [I suggest high] [10:04] lifeless: jeez, gimme time to fart first [10:04] bigjools: well, you're old and all, I don't know I have *that* much time to wait :P [10:05] bigjools: [seriously, when you linked it I figured that meant you'd done all you intended to on it] [10:05] lifeless: my arthritic fingers at my old age don't work as quick as that :) [10:08] lifeless: pgbouncer might help test suite race conditions tearing down/creating databases [10:08] awesome, feature flags are case-sensitive [10:08] mmm... probably not. its more pg tripping over its own feet and waiting for its internals to catch up rather than inbound connections [10:08] stub: if it could force only one connection at a time to template1 [10:09] lifeless: it could do that... [10:09] stub: anyhow, my patch for that seem to work, and we're looking at lxc for parallel in the short-medium term anyhow, now. [10:09] because there are so many similar glitches in our stack at the moment [10:09] [e.g. launchpadlib test cache dir] === al-maisan is now known as almaisan-away [10:16] cjwatson: sure. [10:22] * wgrant curses Python [10:23] Don't make backwards incompatible changes in 2.7 without a non-monkeypatch way to restore the old behaviour. [10:23] wgrant: Hm? [10:23] The default continuation whitespace character in generated email headers changed from tab to space. [10:24] And this is set in an _method of a class. [10:24] So it's not really overridable. [10:24] Yeah, python-devel seems to hate backwards compatibility [10:24] Hmm. [10:24] Ah, mercurial indeed fix it by monkeypatching. [10:24] It turns out I'm not able to land branches any more, because I'm not allowed to edit commit messages on merge proposals for Launchpad. [10:25] Yeah, the emeritus stuff isn't really set up yet. [10:25] Most people are still in ~launchpad. [10:25] Want me to land it instead? [10:26] wgrant: sure. here's the line I'm using: [10:26]