[00:18] wgrant: https://bugs.edge.launchpad.net/soyuz/+bug/669717 [00:18] <_mup_> Bug #669717: archive:+index timeout [00:18] wgrant: just to keep you on your toes ;) [00:27] holy f*ck: 5156 OOPS-1765XMLP299 MailingListApplication:MailingListAPIView [00:27] thats a query count [00:27] lifeless: yeah [00:28] lifeless: I think that is the one I fixed [00:28] * thumper is a sad bunny [00:28] it seems that the storm insert query has non-deterministic column ordering [00:28] s/insert // [00:28] so two subsequent test runs with LP_DEBUG_SQL_EXTRA clash all the time [00:29] lifeless: do you know a fix? [00:29] do you mean column or row ? [00:29] also, ECONTEXT [00:30] thumper: https://bugs.edge.launchpad.net/launchpad-registry/+bug/666580 - I've put the pageid in there - it helps me a lot. [00:30] <_mup_> Bug #666580: MailingListApplication:MailingListAPIView (getMessageDispositions ) mailing list xmlrpc api call makes excessive queries

[00:32] lifeless: %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s to you too. [00:32] But wow. [00:33] thumper: did you mail matt? [00:40] lifeless, for shortening the release timeline, I was wondering if we could lock monday, QA tuesday, unlock wednesday, assuming staging updated ok. Timeline goes from one week to three days. [00:42] if people finish their QA on Friday, then you can lock and unlock on monday [00:42] mars: I think there is not enough slack [00:42] mars: the critical issue is 'what if the db conversion is a problem' - it takes many hours to do a respin of that test [00:43] yeah, but if it went OK, and the QA is up-to-date, then you are looking at a much shorter release [00:44] "if the stars are all aligned just so" - thankfully there are only three to worry about :) [00:44] right [00:44] but we plan for how to handle failure [00:44] not how to handle success ;) [00:44] mars: I think we still need to start friday [00:44] but we could certainly unlock earlier [00:44] mars: I thought my proposal permitted that [00:45] absolutely, it does [00:45] lifeless: Can't you just freeze db-devel on Friday? [00:45] with that extra detail, "Must start Friday, can unlock ASAP", then you can rewrite the release docs [00:45] lifeless: devel doesn't have to be frozen to do DB restore testing. [00:45] wgrant: we don't need to freeze db-devel at all [00:46] wgrant: its not db restore testing [00:46] wgrant: its upgrade-script application [00:46] That's what I meant, sorry. [00:46] wgrant: which we can start doing on qastaging [00:46] (and should) [00:46] Hmm. [00:46] What's the point? [00:46] convergence [00:47] simplicity [00:47] ponies! [00:47] I would say 'speed', but ponies, well... [00:47] wgrant: qastaging is sitting there without any db patches, so we don't need to do a complete restore to test the upgrade [00:47] wgrant: which is /much/ faster [00:47] lifeless: True. [00:48] wgrant: and it updates its code every 30 minutes [00:48] But this still means we have a week where production doesn't update. [00:48] so if we have to fine tune the release, we can do so more rapidly. [00:48] wgrant: perhaps we should change buildbot to only merge QA-ok revisions from stable to db-devel. [00:49] wgrant: the week with prod not updating saddens me, but not as much as a month without updates [00:50] Right, but it doesn't mean we shouldn't try to think of a way to minimise that interval. [00:50] agreed [00:50] Particularly now that there are no CPs. [00:51] I don't see a huge benefit in doing the DB upgrade testing on qastaging rather than staging. And doing it on qastaging means we have to make devel undeployable several days earlier than it would be otherwise. [00:55] mmm [00:55] wgrant: I disagree [00:56] Oh? [00:56] wgrant: friday avo, sat, sun are all undeployable anyway [00:57] True. [00:58] I think if we get 3 clean deploys in a row without needing the extra time, we could move it up to monday. [00:58] lifeless: yes I emailed matt [00:58] also if we have a completely clean development-dbstable report leading up to the report [00:59] I guess that's a good idea. [01:03] wgrant: You still have misgivings? [01:04] StevenK: I think he's looking for a larger win [01:04] which is good [01:06] StevenK: I'd like there to be as little freeze of landings and production deployments as possible. So I will have misgivings forever, unless we reduce both to zero :) [01:08] But this is better than what we have now, so it'll do. [01:09] lifeless: I've got a problem with feature flags [01:09] lifeless: it is breaking tests [01:09] lifeless: and I think I know why [01:10] ok [01:10] 'sup? [01:10] lifeless: there is code that checks the timeout values and uses a feature flag [01:10] this causes a query, and prior to that a flush [01:10] which can cause partially constructed object to be written to the db [01:10] say what? [01:10] causing integrity constraint violations [01:11] that check is done before any domain code [01:11] its in publication [01:11] File "/home/tim/src/launchpad/incremental-diff-job/lib/lp/services/features/rulesource.py", line 90, in getAllRulesAsTuples [01:11] .find(FeatureFlag) [01:11] File "/home/tim/src/launchpad/devel/eggs/storm-0.18-py2.6-linux-x86_64.egg/storm/store.py", line 210, in find [01:11] self.flush() [01:11] those are the bits causing test explosions [01:11] perhaps for general publication [01:11] but not for tests [01:12] so, there are several things we could do [01:12] firstly, outside of a publication stack or similar context, timeouts are meaningless [01:12] so we could remove that [01:12] secondly, the feature values are cached (including absence) [01:13] so evaluating it (e.g. by get_request_timeout()) will cache that outside your code [01:14] how does that interact with the feature flag context manager for tests? [01:14] thirdly test requests have a Null scope provider - they could sensible /also/ install a Null rules provider [01:14] which wouldn't call into the db [01:14] I don't grok that last oen [01:14] one [01:14] see LaunchpadTestRequest.__init__ [01:17] lifeless: it has a NullFeatureController [01:18] thumper: ok [01:18] thumper: so in requests using LTR, no db access should happen at all. [01:18] but it is [01:18] thumper: are you using maris' context manager? [01:18] the code I'm looking at is yes [01:18] self.useContext(feature_flags()) [01:18] set_feature_flag(...) [01:19] although ... [01:20] it is the set_feature_flag code that triggers the flush and the bomb... [01:20] * thumper is now confused [01:20] because the diff should be fully constructed there... [01:22] hmm, that would be nicer as a Fixture I think. [01:22] something for another time. [01:30] * thumper relocates [01:46] I'm gonna miss edge [01:46] why? [01:47] lifeless: the thing about recipes that I spammed in the other channel [02:31] * thumper is very confused [02:31] self.useContext(feature_flags()) [02:31] set_feature_flag(u'code.incremental_diffs.enabled', u'enabled') [02:31] self.assertTrue(getFeatureFlag('code.incremental_diffs.enabled')) [02:31] fails [02:31] in a test [02:31] but only if run after a different test [02:33] thumper: probably has something to do with that "self." thing you have going on. Anytimg I rely on myself for something I crash. [02:33] Oh dear [02:33] MTecknology: :-) [02:33] so this may be related to the cross-test thing gmb/deryck mentioned [02:33] lifeless: almost certainly [02:42] lifeless: email to devel is on my todo list - i just wanted to get the mp set up first [02:43] wallyworld_: sure [02:43] wallyworld_: Belts and braces ;) [02:44] lifeless: save me looking it up, what's the losa mail list? i didn't know there was one [02:44] lifeless: I'm pretty sure it is a cache invalidation problem :) [02:44] thumper: at what layer? we shouldn't have the same featurecontroller reference should we? [02:45] thumper: I'd guess at the request<->thread-local lifetimes being different. [02:45] thumper: perhaps a function to ensure when one is altered, the other is too [02:47] * thumper is poking some more [02:49] lifeless: don't worry, found it [02:50] wallyworld_: oh sorry, yes. [02:50] lifeless: np. i was just letting you know. it wasn't mean to be snarky :-) [02:50] I know :) [02:51] man [02:51] I hope noone is pissed about all these bug closes :) [02:52] lifeless: closed as in fixed- or closed as in won't fix? [02:52] both [02:57] heh [02:57] set_feature_flag has a store.flush [02:58] no... [02:58] set_feature_flag adds a feature to the db [02:58] when the feature is being flushed [02:58] it is doing a db query [02:58] which queries the feature flags [02:58] which sets the _rules cache [02:58] which means the newly set feature isn't in the cache [02:59] thumper, yeah, deryck and I fought with that last week. [02:59] It caused much head scratching. [02:59] I'm not yet sure why sometimes it hits this and other times it doesn't [03:00] * thumper thinks it is a flush ordering problem [03:00] I think it is easily solved though [03:00] transaction.commit() ? [03:00] no [03:01] thumper: No fair sending wallyworld_ my way with a 1,500 line MP [03:01] I know where you live ... [03:01] StevenK: I didn't know it was that big [03:01] sorry [03:01] StevenK: sorry :-( most of it is unit tests :-) [03:02] thumper: looks like devel is still in test fix mode? my pqm-submit was rejected again for that reason [03:02] wallyworld_: I'm working through it, but the current way to get MPs > 800 lines reviewed is to 1. Not, or 2. Bribe a reviewer [03:02] wallyworld_: Consult Ye Olde Buildbot [03:03] We shouldn't be [03:03] StevenK: what can thumper offer you by way of inducement? :-) [03:03] It isn't thumper's MP ... *cough* *hint* [03:03] lifeless: http://pastebin.ubuntu.com/524222/ [03:04] StevenK: yeah, but he was the one who *made* me throw you the hospital pass :-) [03:04] thumper: does that work for you? [03:04] It's all about choice [03:04] said it would be good for your soul :-) [03:04] lifeless: yep [03:05] thumper: something smells here [03:05] thumper: I think features.per_thread.features is stale perhaps ? [03:05] thumper: if so, thats critical to fix. [03:05] lifeless: yes... in that the features has cached the rules before we add one [03:05] wallyworld_: If thumper said that with a straight face, then I take the comment about his poker face back [03:06] thumper: if its cached the rules from another test, or even another *request*, then there is an isolation bug that this qualifies as a workaround for. [03:06] StevenK: i can offer you paul's eternal gratitutde as he *really* wants to get merge queues done before he takes his sabatical from the code team [03:06] lifeless: more of a problem with tests than real life [03:06] lifeless: let me plug in headphones [03:06] wallyworld_: I thought I already had that [03:06] lifeless: then skype may help here [03:06] ok [03:07] StevenK: hmmm. i'm running out of possible bribes. [03:13] wallyworld_: Hah, serves you right :-P [03:16] thumper: so - two lines [03:16] da.set_permit_timeout_from_features(False) [03:16] ... [03:17] da.set_permit_timeout_from_features(True) [03:23] webapp/testing/helpers.py [03:23] lifeless: http://pastebin.ubuntu.com/524230/ [03:23] AdapterIsolator = FunctionFixture(set_request_started, lambda x:clear_request_started()) [03:23] in setUp [03:23] http://pastebin.ubuntu.com/524231/ [03:23] self.useFixture(AdapterIsolator) [03:26] --- lib/lp/testing/__init__.py 2010-10-26 15:47:24 +0000 [03:26] +++ lib/lp/testing/__init__.py 2010-11-02 03:26:01 +0000 [03:26] @@ -495,6 +495,9 @@ [03:26] self.oopses = [] [03:26] self.useFixture(ZopeEventHandlerFixture(self._recordOops)) [03:26] self.addCleanup(self.attachOopses) [03:26] + from canonical.launchpad.webapp import adapter [03:26] + self.useFixture(fixtures.FunctionFixture(adapter.set_request_started, [03:26] + lambda _:clear_request_started()) [03:31] thumper: set_permit_timeout_from_features(False) [03:31] thats all [03:35] 11:33 < lifeless> https://dev.launchpad.net/PolicyAndProcess/OptionalReviews [03:35] 11:33 < lifeless> Activities [03:35] 11:33 < lifeless> Submit the branch to create an MP (our toolchains can look at this and it provides a location for a post landing review if the branch has that done to it). Self review with review type 'unreviewed'. Land via the normal [03:35] landing process. [03:42] https://code.launchpad.net/~thumper/launchpad/fix-features/+merge/39819 === mwhudson_ is now known as mwhudson [04:00] lifeless: bzr lp-land checks to make sure you don't review your own code :) [04:01] lifeless: do we land it rs=? [04:01] thumper: change lp-land ? [04:01] maybe it asked a slave [04:01] * thumper checks [04:02] lp-land doesn't deal with rs= [04:02] Or MP-less lands [04:07] the code seems to indicate it should be fine :( [04:08] but something weird is happening [04:09] me gives up and pqm-submits [04:12] thumper: please file a bug on foundations [04:12] lifeless: on the thread local leaking? [04:12] on lp-land not accepting your MP [04:12] thumper: I bet its the review type though [04:12] thumper: try changing the type to nothing/code [04:12] ah [04:12] yeah, you are right [04:13] which is fucked [04:13] * thumper leaves to cook [04:18] At 5pm? [04:18] StevenK: I have kids [04:23] StevenK: hey [04:23] you know soyuz stuff [04:23] help: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html [04:25] I will look after shopping [04:31] heh [04:32] thanks [04:33] Project devel build (173): STILL FAILING in 3 hr 52 min: https://hudson.wedontsleep.org/job/devel/173/ [04:39] wow Archive:EntryResource:getBuildSummariesForSourceIds is slow [04:39] still timing out at 20 seconds [04:40] 1/2 sec per SELECT * FROM ((SELECT BinaryPackageBuild.distro_arch_series, BinaryPackageBuild.id, BinaryPackageBuild.package_build, [04:47] lifeless: Do we have graphs of page performance vs time? [04:47] lifeless: ie. do we know if it's a pg 8.4 regression, or a BFJ refactor regression, or is it just generally crap? [04:47] its crap now [04:48] the api wasn't timing out before, so 8.4 regression [04:48] I think [04:48] no we don't have charts per-query [04:49] wgrant: yeah - https://bugs.edge.launchpad.net/soyuz/+bug/662523 [04:49] <_mup_> Bug #662523: Archive:EntryResource:getBuildSummariesForSourceIds times out === almaisan-away is now known as al-maisan [07:18] Hi henninge [07:19] Hey jtv! ;) [07:19] Feeling better? [07:38] Grr why don't we have checksums on the Ubuntu iso download page? [07:39] interesting failure on launchpad prod branch in buildbot - http://pastebin.ubuntu.com/524300/ [07:39] jtv: I suspect that you don't want to know. [07:39] (the design team probably said they were user-hostile, or something) [07:40] But they're easy enough to find on the mirrors. [07:40] night all [07:40] Night lifeless. [07:40] wgrant: well I do want to bloody know. I now have supposedly identical ISOs with different checksums and it should be easy to figure out which, if any, is right. [07:40] Night lifeless [07:40] jtv: http://releases.ubuntu.com/10.10/MD5SUMS [07:40] Thanks [07:41] Nice security.py speedup, btw. [07:41] It's, er, quite effective. [07:41] Thanks. [07:41] I didn't bother with the remaining largest time waster inside the script, since it only took 4 seconds. [07:41] Script startup however does seem to take quite a while. [07:42] Most of the remaining 'make schema' time seems to come from build. [07:43] So that's our next target. [07:44] The WADL isn't very fast either [07:44] Then lifeless can delete ZCML. [07:44] And we can build a tree in seconds! [07:44] StevenK: That's in build. [07:44] compile has slow buildout. [07:44] build has compile and WADL. [07:45] wgrant: I spent the trip from ORD to LAX exporting blueprints, I got to know very well how long it takes to generate. [07:45] StevenK: I tried to profile the WADL generator. [07:45] But it took too long. [07:45] Haha [07:45] * StevenK attempts to come up with a clean joke, fails [07:47] Hmm. Apparently a horse race happened today [07:48] I've been deliberately avoiding finding out who won. [07:49] I didn't even realise until Sarah loaded smh.com.au while I was walking past [07:51] Hah. [07:51] It's hard not to realise down here; it's a public holiday. [07:52] Yeah, I knew that [07:53] Why a sporting event gets a public holiday I will never know. [07:54] Is it the cricket? [07:55] Less boring than that. [07:55] Paint drying? [07:56] Indeed. [07:57] Something less boring than cricket? Interesting. [07:59] I wouldn't go _that_ far… [08:00] jtv: Do you suggest that something is more boring than cricket? [08:00] Project devel build (174): STILL FAILING in 3 hr 27 min: https://hudson.wedontsleep.org/job/devel/174/ [08:00] Launchpad Patch Queue Manager: [r=jtv][ui=none][bug=667554] Don't send email for work in progress [08:00] merge proposals when requesting reviews or modifying the proposal. [08:01] wgrant: read what nigelb said more closely! [08:01] Ah ha.. [08:01] My dad was dragged into a cricket match once. Didn't even find it boring, to his surprise. [08:01] A decade later he happened to open a cricket mag (at the dentist's or something) and guess what? They were still talking about that famous exciting match. [08:02] lol [08:02] jtv: heh, you caught that :D [08:02] nigelb: red-handed [08:02] heh [08:03] I watched IPL to any extent only for one season. Lost interest. [08:03] IPL = Initial Program Load? Still talking about profiling? [08:03] No, was talking about criket :) [08:04] What's IPL stand for? [08:04] Indian Premier League. The 'big thing' in 20-20. meh. [08:04] nigelb: thank you for TLA #23662 [08:04] <_mup_> Bug #23662: [network-admin] 'connection settings' should not be greyed out when not active (in network settings)

[08:05] No mup, not that. [08:05] http://xs4all.nl/~jtv/gtf/ [08:05] jtv: lol [08:05] nigelb: you can now carry the GTF Contributor Program (GCP) logo on your website or home page. [08:06] \o/ [08:06] Well, this day did bring one achievement. [08:06] Quite. [08:06] jtv: steven@liquified:~$ host hugeurl.wiggy.net [08:06] Host hugeurl.wiggy.net not found: 3(NXDOMAIN) [08:06] :-( [08:07] liquified, no wonder. [08:07] I guess wichert must have shut it down. It was about a decade ago. [08:07] The idea was that tinyurl etc. are nice, but small URLs don't look impressive. [08:07] So he created a URL stretcher. [08:07] Indeed [08:07] Yes, and Wichert is the kind of guy to do it. :-) [08:07] One of the encoding schemes available was: look for three-letter combinations that are in the GTF, and expand them. [08:08] Haven't seen him in ages. You? [08:08] Neither [08:08] :( [08:08] Not for 5 years or so [08:08] One wonders how he is. [08:09] He fell into the black hole that previous DPLs get sucked into [08:09] Does that apply to former colleagues and university friends? [08:09] jtv: Ah, you were at XS with him? [08:10] No, cistron [08:10] And the first FOSDEM—then still called OSDEM. [08:10] I arrived late, having escaped from a lady's bedroom window that morning in a different country. [08:11] Now there's something that sounds like an interesting story [08:11] I think it was our mutual friend Ray who started the vicious and baseless rumour that I had escaped from a lady's bathroom window. [08:11] If you ever hear that version, don't believe it! [08:11] Perhaps it was her ensuite window [08:12] Her wha? [08:12] An ensuite is a small bathroom directly accessible from a bedroom [08:12] Ah. [08:12] No, it was definitely bedroom. [08:15] But thanks for teaching me that word. [08:16] Heh [08:18] jtv: Wait, do we get to hear more of that exploit? [08:18] nigelb: what do _you_ think? [08:18] We must. [08:19] We demand. [08:19] Get me drunk and we'll talk. [08:19] dammit, if I had the money, I'd fly to Australia just to get you drunk ;) [08:19] If you don't have the money to fly to Australia, what makes you think you have the money to get me drunk? [08:20] Good point. [08:20] I was planning on getting you drunk enough for you to hand me you purse and hards :p [08:20] My secret is safe for now. [08:20] Good plan? ;D [08:20] Until January. [08:20] What's in Jan? Epic? [08:20] Ja. [08:21] Too bad its a closed event :( [08:21] Face it. You're not looking for an open event. You're looking for an open bar. [08:21] (See "money" above) [08:22] heh [08:22] True, that. [08:24] jtv: Love your homepage. "Nor do I speak for my employer; he is quite old enough to speak for himself." [08:25] That's served me well for a fair number of employers. :) [08:26] What if they're a she? [08:27] As now, I suppose, they are. [08:27] I didn't know about that particular bit of the English language at the time of writing. Will rectify. [08:42] good morning [09:13] Everybody: land you branches! Testfix coming up again ... :-( [09:14] I've been seeing the same failures in Hudson [09:16] Hello [11:03] Morning, all. [11:06] Yippie, build fixed! [11:06] Project db-devel build (111): FIXED in 3 hr 55 min: https://hudson.wedontsleep.org/job/db-devel/111/ [11:13] gmb, hi. Did you see thumper's email about test isolation failure and his fix to set_feature_flag? [11:14] deryck: Yes; I've already merged my branch but I'll try splitting the tests up again and see if it works now. [11:14] gmb, ok, cool. I expect that will fix the problem we were seeing, too. If using the testing helpers. mars fixture would need to flush too. [11:14]