[00:09] thumper: should i ask sk to review those 2 daily recipe build list page branches? [01:04] bugclient now fails with [01:04] AttributeError: 'Entry' object has no attribute 'markAsDuplicate' [01:04] did this recently change in the api? [01:05] what api version are you using? [01:06] ?11958 [01:10] lifeless, api/1.0> why not? [01:11] james_w: ? [01:11] bug 11958 [01:11] <_mup_> Bug #11958: Unable to show hidden files [01:11] oh, that was a typo [01:11] pasted to the wrong window [01:12] lifeless, salgado-afk: please don't put the blueprints in api/1.0 [01:12] james_w: right. Because 1.0 is supported for many more years. [01:12] but jml has the issue tracker thing in train for the next year. [01:13] why not stop the default being to add features to old versions then? [01:13] there's a bug open on that [01:13] IIRC [01:13] ok [01:13] if there isn't, there should be - this has been discussed. [01:14] we shouldn't 'support' things we haven't finished. [01:14] we should do best effort etc [01:14] but we need to allow room for mistakes. [01:15] hello james_w [01:21] hi poolie [01:26] poolie: hi, so what api version are you using? [01:26] i don't know [01:27] poolie: I ask, because its possible some things that were accidentally exposed have been shuffled, but we're meant to be very conservative about removing stuff.... trying to estimate whether to dig deeper or not. [01:27] i seem to be just taking the default? [01:28] [01:28] <_mup_> Bug #415936: Merge into new branch produces strange log [01:28] so 1.0, i guess [01:29] hmm, you're using edge too :) [01:29] could you switch to LPNET_SERVICE_ROOT :) [01:30] heh, it's a hangover from that once being the only place to get it [01:30] much better to switch things on/off on the server [01:30] sure [01:30] we can't sadly [01:30] lp API clients start with a POST [01:30] i mean in future [01:30] and blow up if thats redirected or otherwise handled by non-edge. [01:31] sure [01:31] uhm, so 1.0 [01:31] lets see [01:31] we could change lplib to make EDGE_SERVICE_ROOT == LPNET_SERVICE_ROOT [01:31] but that might just cause confusion [01:31] we're going to do something like that [01:31] perhaps with a deprecation warning [01:31] hm that name doesn't exist [01:32] ok, i have it [01:33] ah cool [01:33] what was it ? [01:34] ok, so markAsDuplicate is in beta [01:34] not in 1.0 or devel [01:35] I can check the changelog but my guess is that you've moved from a launchpadlib that used beta by default to one that uses 1.0 by default [01:35] the current idiom is to use duplicate_of = xxx [01:35] hm [01:35] something like that [01:35] you can pass beta as the api version you want [01:35] though, this is on my maverick machine, running the packaged lplib [01:35] or update your code [01:36] i doubt that changed in the last few weeks? [01:36] that would be unusual. [01:36] could you file a bug? Probably a mistake then. [01:36] Yippie, build fixed! [01:36] Project devel build (238): FIXED in 3 hr 45 min: https://hudson.wedontsleep.org/job/devel/238/ [01:36] * Launchpad Patch Queue Manager: [r=abentley][ui=none][no-qa] New "images" command for bin/ec2 to [01:36] display all current test images. [01:36] * Launchpad Patch Queue Manager: [r=flacoste][ui=none][bug=677305] Downgrade bzr to 2.2.0 [01:38] hm, i see i was repeating myself and defaulting to edge in two places [01:38] you might find https://dev.launchpad.net/PolicyAndProcess/OptionalReviews interesting [01:39] i've followed some of the mail about it [01:40] ok, so even on 1.0 non-edge, it still fails [01:40] markAsDuplicate isn't in 1.0 [01:40] "stable interfaces in python are hard" [01:40] perhaps it should be ;) - thus a bug is needed. [01:40] i'm pretty sure this was working a week or two ago [01:41] maybe a bit more than this, but within the last couple of months [01:41] i will file [01:44] the shorter Optional Reviews report: "data, bitches!" [01:44] i think that's great [01:44] to be devil's advocate [01:45] i think the previous experience is not so much showing everything needs review [01:45] but rather that people will use these to route around a broken review process [01:45] cf john's mail [01:45] but perhaps things have now changed [01:45] s//probably [01:50] well [01:51] I think that route around is better than stockpiling [01:52] wallyworld_: yes, fling them StevenK's way [01:52] wallyworld_: I'm just about to head and collect the girls from school [01:52] thumper: ok [01:52] wallyworld_: we should have a chat after that [01:52] thumper: i'll be here [01:54] lifeless: i think in the first version of this code i did assign to 'duplicate_of', but.. [01:54] that didn't work [01:54] i can't remember if the change was not saved, or if it gave an error [02:01] wgrant: yo [02:01] poolie: you need to call obj.lp_save() [02:02] i know that, and i think it wasn't enough [02:02] imbw [02:02] anyhow, bug 680339 [02:02] <_mup_> Bug #680339: 'Entry' object has no attribute 'markAsDuplicate' [02:03] thank you [02:16] thumper: back in 15 mins - have to drop the kid to his McJob [02:17] wallyworld_: ok [02:20] thumper, wallyworld_, we're going to do a bzr 2.2.2 release at the end of this week [02:20] that should address the problems you hit in 2.2.1 [02:21] poolie: ok [02:21] feedback welcome, as always [02:22] lifeless: Hi. [02:23] What's broken? [02:24] wgrant: cesium - removing cowboys [02:25] wondering if you know the rev that landed the fix [02:25] wgrant: how was your exam? [02:25] lifeless: Exam was rather better than expected. [02:25] cesium... what was the cowboy? The builder disabling thing? [02:25] wgrant: all done? wooo! [02:26] Indeed. [02:30] lifeless: Unless there was more than just logging and not disabling builders, devel r11938 looks to be the one. [02:35] thumper: call now? [02:37] wgrant: there was [02:37] disabling the failure checking [02:38] http://pastebin.com/Kr44BkbD [02:39] lifeless: 11938 should render that pointless. [02:39] Though I'd, er, check with someone else. [02:41] wgrant: it looks complete to me [02:43] the only gap is that first hunk [02:44] Yeah, that's what I meant. [02:44] I think. [03:57] jamesh: do you remember how to tell zope to use a different thread count? [03:57] lifeless: there is an option in the .conf file [03:57] I think it might even be called thread_count [03:59] this is the ZConfig .conf file [03:59] is that 'launchpad.conf' ? [04:05] yes [04:10] lifeless: looking at one of the ancient launchpad trees on my system, one of the launchpad.conf files under configs/ has "threads 16" at the top level [04:10] thanks! [04:10] also, terrible idea :) [04:11] I remembered that we had bumped the count up for the demo.launchpad.net instance [04:11] so checked that config [04:12] that instance wasn't under heavy load, but it was easy for one user to block everyone else with the default thread count [04:18] yeah, [04:18] we're in the process of (with measurement) dropping down to 1 thread per appserver [04:32] lifeless: interesting! i don't recall you posting about it [04:32] (not that you have to, i suppose) [04:32] in various perf tuesday mails [04:32] on the grounds that, if they're not wasting time, you wouldn't get more than one python thread to run anyhow? [04:32] we're seeing things that thread starvation is the best explanation for [04:32] and yes [04:32] that too [04:33] ok, i do see a mention in passing [04:35] jamesh: yes, thats it [04:35] jamesh: thanks for finding it [04:36] poolie: we could reasonable expect total time/database time threads to be a reasonable figure [04:36] but its simpler to let the OS manage things at that point [04:37] + if we do decide to debug something only one user will be impacted [04:37] poolie: I did some more analysis in a ubuntu-one thread [04:39] agree about letting the OS do it [04:40] istm the main drawback is there are things that python knows are shareable in memory, that the OS doesn't [04:40] like modules [04:40] and for lp they tend to be large [04:40] but maybe this is not a sufficiently important factor [04:42] its a couple hundred MB [04:42] 1.6GB on a fully loaded machine. [04:42] it could be better [04:43] but we should get a win regardless, or so says the theory. [04:46] yay [04:46] https://launchpad.net/~lifeless/+commentedbugs working [04:47] timed out for me.. [04:47] hahrugle [04:48] in 'select count from bugtask....' [04:48] but it's a lovely sounding url :) [04:48] what revno [04:48] 11952 [04:48] damn [04:48] At least 42 queries/external actions issued in 2.16 seconds [04:48] r11952 [04:50] poolie: its working consistently for me [04:50] poolie: whats the OOPS id ? [04:52] https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1788K350 [04:52] works for me now [04:53] is this a new feature, or just a newly-faster feature? [04:53] worked this time [04:53] top OOPS since the monthly rollout [04:53] ah i see [04:53] maybe it was 'want an affecting bugs list' i was thinking of [05:59] wgrant: btw. I'm bemused and entertained at your dedication. Finish final exam, hop on IRC. ISTR that post my final exam at Uni, a group of us went to the Brekky Creek Hotel in Brisvegas for a Steak lunch and afternoon of entertaining the regulars in the beer garden with (badly) Monty Python songs sung. [06:01] spm: Well, a few of us finished early, went to pub, consumed beer, returned to the exam venue to see everyone else, then went back to my office. [06:01] And then IRC, yes :P [06:02] bwhahaha [06:16] spm: hey [06:16] sorry, ECHAN [06:42] spm, ah, the brekky creek [06:42] is it a known problem that all api calls against staging are giving 500 errors? [06:43] and indeed the web ui too [06:44] Hayfever or cold :-( Suspect today is another sick day. [06:44] yeah, staging is borked atm. been fixing the borked prod rollout first tho [06:45] heh, that's probably a good choice [06:45] :) thanks spm === almaisan-away is now known as al-maisan [08:41] good morning [08:41] morning [08:42] how was the exam wgrant? [08:54] is there someone around that can qa https://bugs.launchpad.net/rosetta/+bug/669831 ? [08:54] <_mup_> Bug #669831: obsolete translations exported to the branch

[09:09] Morning [09:12] lifeless: I've landed all code to cover the cowboys on cesium, please roll it out (or I can) [09:12] bigjools: its done [09:12] lifeless: ah great, you checked or Picarded it? [09:12] spm deployed [09:12] and noone has screamed yet [09:13] I figured I'd chat with you briefly :) [09:13] heh [09:13] * bigjools looks at log [09:14] lifeless: I thought it was in the nodowntime set? [09:15] bigjools: it was taken out when it blew up [09:15] when things get cowboyed we remove them from nodowntime. [09:16] so it's back in? I thought I said I was going to approve that first ... [09:16] we may be crossing wires [09:16] back at uds it wasn't in, and we discussed what it would take to be in [09:16] that you were cc'd on a discussion and gave your blessing. [09:17] then the big branch blew builders away [09:17] so it was cowboy-fixed and removed [to stop deploys uncowboying it] [09:17] we cross checked that you'd landed fixes [and they were to be deployed] today, and so uncowboyed it (which implies putting it back in nodowntime] [09:18] bigjools: if you had wanted a further cross-check with you, I'm really sorry - I didn't realise. [09:19] lifeless: np, I talked with Tom not you [09:19] as it happens it's fine to roll, but that's because I had landed everything [09:19] right, we looked first ;) [09:19] in future, if you want a cross-check, could you note that on LPS against the cowboy, or the DeploymentException ? [09:20] yes but one of the cowboys is not landed, but deliberately. How did you reconcile that? :) [09:20] bigjools: knowledge. [09:20] :) [09:20] I had discussed the fault with you. [09:20] so I knew you put it in while you diagnosed the root cause. [09:20] we need to figure out wtf the builders are taking >30 seconds to accept connections [09:21] I also included that cowboy as a ready to go patch in a mail to losas [09:21] so if there is a submarine present there [09:21] its at-hand to recowboy [09:22] one of the side effects of that cowboy is that jobs will "stick" on genuinely failed builders [09:22] I have to keep checking the log [09:30] lifeless: so, did we talk about figuring out the massive slave connection delays? [09:31] not as such [09:36] hey jtv! ;) [09:36] hi henninge! [09:36] jtv: let's be chatty ;) [09:37] henninge: I don't think this connection will support voice chat. [09:38] jtv: oic [09:38] you are also not on #translations [09:38] just a mo' [09:40] jtv: hi [09:40] hi [09:40] https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html [09:41] uh-oh [09:41] are you able to qa 669831 on [qa]staging ? [09:41] henninge: hang on, lifeless wants something :) [09:41] no panic [09:41] lifeless: very slow connection, so will be a bit slow to respond [09:41] but 11960 would remove another cowboy in the datacentre [09:42] jtv: like I say [09:42] no panic [09:42] jtv, lifeless: I can do that [09:42] awesome [09:42] ok, gnight all [09:42] g'night [09:42] lifeless: good night [09:42] * jtv runs around a bit and screams for a while, just because he was told not to panic [09:42] it's the principle of the thing [09:53] have you considered working on Soyuz? [10:06] bigjools: Better than expected. [10:07] wgrant: excellent, but shouldn't you be out celebrating now [10:07] Did that earlier :P [10:07] But yes, probably. [10:08] you party animal [10:08] did you get a chance to look at the expiration query? [10:08] Heh. [10:08] Will look now. [10:11] bigjools: Where's the latest version of that query? [10:11] the one I pasted :) [10:11] k [10:12] http://pastebin.ubuntu.com/535167/ [10:12] Yup. [10:21] bigjools: Are the LFC, DS and DAS joins in the EXCEPT completely useless, or am I stupid and blind? [10:22] one sec [10:24] bigjools: http://pastebin.ubuntu.com/535491/ [10:25] Should be equivalent, except with the retention condition fixed. [10:25] (now it will exclude if the file is unremoved or Obsolete, rather than Published or Obsolete) [10:25] Which should make p-d-r less sad. [10:27] Hmm. [10:29] bigjools: Did the domination thingy help dogfood at all? I guess it pales in comparison with the file list generation :/ [10:29] wgrant: yes, file lists take ~4 hours [10:29] for just maverick release pocket :/ [10:29] something has regressed a lot [10:29] it used to take ~30 mins [10:29] I will poke it in the eye in a few weeks. [10:30] DF is slow, but ... [10:30] Heh. [10:43] wgrant: sql looks good, I am trying it on DF [10:47] bigjools: So, about those people who feel like they need to use obsolete PPAs... [10:47] OEM [10:48] How did I guess :( [10:49] now, I am struggling to understand wtf builder sometimes take in excess of 180 seconds to accept a connection from the manager [10:50] * bigjools -> caffeine [10:51] bigjools: Load graphs from non-virt builders pls. [10:51] It might at least give us some idea of if that's the issue. === matsubara-afk is now known as matsubara [11:14] jml: remember my dirty reactor failure in my b-m tests? [11:16] I added debug output on the Deferreds and it's caused by distribution_mirrorprober. There's a test isolation error... :/ [11:17] The reactor is shared‽ [11:17] in tests [11:17] My interrobang stands. [11:17] * jml briefly steps in from the sick room [11:18] man flu? [11:18] wgrant: yeah, there's exactly one reactor. [11:18] wgrant: it's arguably the biggest flaw in Twisted [11:18] jml: This sounds... like Zope. [11:19] it would be kinda hard to have more than one [11:19] bigjools: I know about those tests. my testtools-experiment branch fixes those delayed calls. [11:19] bigjools: but it's blocked on landing by some weird-ass 500 from the librarian [11:19] jml: yay. So, what can I do about it in my branch? [11:20] bigjools: those tests need to be completely rewritten... let me find you my workaround [11:20] and why are they leaking to my test? [11:20] bigjools: because the distributionmirror_prober tests aren't using trial [11:20] ah ... [11:20] bigjools: so the calls are going on to the reactor, and when your tests are cleaned up ... bang [11:21] that's kinda bad from a test isolation PoV :/ [11:21] wgrant: [11:21] total_files | space_saved [11:21] -------------+-------------- [11:21] 184899 | 307432516434 [11:22] bigjools: yes. the problem is the mutable global state that is the reactor [11:22] indeed [11:22] bigjools: got any suggestions on how to make it better? [11:22] bigjools: Mm, not too implausible. [11:22] jml: I'd make Trial clean the reactor when starting a test? [11:22] bigjools: it can't do that. there might be in-process Twisted-using fixtures [11:23] bigjools: http://pastebin.ubuntu.com/535507/ <- should work around the problem [11:23] jml: at the very start of the test there should be no fixtures yet though? [11:23] bigjools: not if they are shared between tests [11:23] aieeee [11:24] jml: I'll poke your workaround in, thanks [11:24] wgrant: incidentally, it doesn't sound like Zope to me. [11:24] jml: Opaque global state. [11:24] The Zope Way. [11:24] jml: if it's complaining about them when the test ends, how can a fixture's Deferreds never get in the way then? [11:25] if it's shared between tests that is [11:25] bigjools: good point. in Trial, there's some historical crap back from when we thought setUpClass would be a good idea [11:26] (it's a terrible idea, we got rid of it, Guido added it back to Python again, and so the circle of crap continues) [11:26] :/ [11:26] so if someone uses setUpClass with a Deferred, your tests will always fail [11:26] bigjools: in testtools, I guess we could clean the reactor before tests [11:27] bigjools: no, Trial postpones checking of those things until tearDownClass runs. [11:27] bigjools: *that's* the historical crap. [11:27] oy [11:28] as I said, clearing it out in testtools would help [11:29] yeah [11:29] although it wouldn't help that much. [11:29] "some test before this one was bonkers" [11:29] it's still an improvement over the current situation [11:30] I won't ask why the mirrorprober tests are not using Trial then.... :) [11:30] I have NFI [11:30] if I have my way, they'll be using testtools before the year is out. [11:30] \o/ [11:31] hmm I need to book some holiday [11:31] anyway, all this excitement is threatening my delicate constitution [11:31] jml: Berocca [11:31] bigjools: it's a cold, not a hangover. :P [11:31] jml: :) it still works [11:31] the fizzy stuff [11:32] get better soon anyway, go get some rest [11:32] * jml watches The Wire instead. [11:32] a friend of mine swears his PS3 is medicinal [11:32] heh === al-maisan is now known as almaisan-away [12:04] Morning, all. [12:23] wgrant: so, I figured out the problem with the log parser [12:25] bigjools: !! [12:25] What is it? [12:26] wgrant: it reads in gzip files in their entirety :/ [12:26] see lp/services/apachelogparser/base.py [12:26] get_fd_and_file_size() [12:26] le heavy sigh [12:27] bigjools: Hahaha. [12:28] I thought gzip stored the uncompressed size in the header... [12:28] Yes, it's in the footer. [12:29] Not sure how we can access that from Python, though... [12:29] but does any of the python module read that [12:33] It's limited to 4 bytes, so it probably doesn't. [12:33] But let's see. [12:33] http://stackoverflow.com/questions/1704458/get-uncompressed-size-of-a-gz-file-in-python [12:34] It looks like we probably have no choice but to read in chunks. [12:36] did you notice the last answer on that page :) [12:36] Suggesting len(fd.read())? [12:37] Oh. [12:37] Hahahah. [12:37] I rarely look at the author... [12:38] I am tempted to grab the last 4 bytes [12:39] But 2**32... [12:39] I guess we can make it explode if it ever tries to write a bytes_read greater than that. [12:39] Since something is pretty broken if we have a 4GiB log file, I guess. [12:40]