[00:08] do we still use Pygpgme ? [00:09] wgrant: https://bugs.launchpad.net/launchpad/+bug/184835 [00:09] <_mup_> Bug #184835: Add status '9' (which equates to WONTFIX) to the Roundup external bug tracker

< https://launchpad.net/bugs/184835 > [00:09] wgrant: is that fixed incidentally? [00:09] lifeless: No, but it's no longer an OOPS, so it's Low. [00:13] wgrant: what about https://bugs.launchpad.net/launchpad/+bug/125068 ? [00:13] <_mup_> Bug #125068: Bugzilla bug watch updater crashes on POSTs that return a HTTP error

< https://launchpad.net/bugs/125068 > [00:13] lifeless: Some of those are fixed, others are not. [00:13] I wonder if leonardr's work has fixed https://bugs.launchpad.net/launchpad/+bug/106338 ? [00:13] <_mup_> Bug #106338: Editing a bug targeted to a release crashes if you directly edit the untargeted task < https://launchpad.net/bugs/106338 > [00:13] I need to check waht still shows up. [00:14] he says it hasn't fixed it [00:14] but I wonder if we're lucky [00:14] Indeed. [00:14] He tested it. [00:14] And said it wasn't fixed. [00:25] wgrant: hey - https://bugs.launchpad.net/launchpad-buildd/+bug/497772 - would love to get bm crashes oopsing [00:25] <_mup_> Bug #497772: exceptions.AttributeError: 'DebianBuildManager' object has no attribute '_subprocess' < https://launchpad.net/bugs/497772 > [00:35] lifeless: So, roughly, "fuck no" [00:36] Generating OOPSes from user-controlled code == bad [00:37] It's also really hard, since there is no access to the slaves except through lp-buildd, and that's what's broken here. [00:39] wgrant: uhm, thats not what I was saying [00:39] wgrant: if the build manager blows up, it should write an oops. [00:39] ditto lp-buildd [00:39] Where should lp-buildd write an OOPS to? [00:39] if the *build* fails, thats not lp-buildd or the build manager blowing up [00:40] wgrant: hand it off to the bm [00:40] And how do we know a user isn't going to run their own lp-buildd and generate millions of OOPSes? [00:40] wgrant: can they talk to the build master? [00:40] lifeless: I can easily hijack a buildd and get my own lp-buildd to run there. [00:40] wgrant: on the host os? [00:41] lifeless: Outside the chroot, but inside the VM. [00:41] wgrant: isn't that on a different IP ? [00:41] lifeless: lp-buildd runs inside the VM. [00:41] We don't talk to the host at all. [00:41] *blink* [00:41] We don't know that the host exists. [00:41] ok [00:41] uhm [00:41] We just know that there is a command that we can run to reset the VM. [00:41] (because we will need to use alternative virt solutions in the future *cough* ARM *cough*, where there is no Linux on the host) [00:42] you have a good scary story there [00:42] but [00:43] if we get an oops from lp-buildd, its time to kill the build [00:43] that prevents millions of oopses from one instance [00:43] What does killing the build mean? [00:43] Killing the build does not kill lp-buildd. [00:44] wgrant: AIUI we restart the buildd per build because of precisely the trust issues you allude to [00:44] wgrant: so I mean that [00:44] lifeless: At the moment we only restart it when the next build starts. But that is itself a security issue, so we should probably fix that. [01:02] There seem to be three different ways an externalbugtracker can indicate that the importance is unable to be converted :( [01:03] \o/ [01:04] lifeless: Bug #422960 is cheating. [01:04] <_mup_> Bug #422960: appear to be failing to record oops for all +translate HTTP 503 errors

< https://launchpad.net/bugs/422960 > [01:04] wgrant: I disagree :) [01:06] wgrant: it falls into the bucket of a bunch of operational improvements that we should do, IMO [01:07] Maybe. [01:08] Almost OOPS report time. [01:08] yeah [01:08] almost 11 second timeout time [01:08] Exactly. [01:09] What did we start at? [01:09] 16? [01:09] 20 [01:09] 17 on edge [01:09] step 1 was making both 17 [01:10] I was going to say 17, but I thought that seemed a bit odd. [01:10] since then we've ratched as we fixed things [01:10] edge was lower 'to catch slow pages' - but has a different user population, so not very effective at that [01:11] And edge still isn't dead. :-( [01:11] its in the queue [01:12] But at least it no longer has a shorter queue. [01:15] lifeless: Your gardening has nearly been successful. [01:15] But not quite :( [01:15] I'd like to rewrite my Firefox URL suggestions, since most of those tend to be edge URLs [01:17] wgrant: thats just the first 75 [01:17] StevenK: sqlite3 [01:17] StevenK: update ... [01:17] where ... [01:20] lifeless: Er, which of the 4,000,000 sqlite databases? :-) [01:22] places [01:23] thumper: got a sec on mumble? [01:23] sure [01:27] sqlite> select count(*) from moz_places where url like '%edge%'; [01:27] 163 [01:27] :-( [01:34] gnarrrrgh [01:35] If I "accidentally" break bug searching and manage to globally corrupt the history so the old code is irretrievable, does writing an implementation does not UTTERLY SUCK become critical? [01:35] s/implementation/implementation that/ [01:37] How do you plan to globally corrupt the history? :-) [01:37] That's one part of the plan that is not yet entirely defined. [01:38] wgrant: To quote Penny Arcade: Stay Good! [01:40] lifeless: Is bug #558585 valid? [01:40] <_mup_> Bug #558585: librarian outages cause OOPSes on pages showing BranchMergeProposal diffs

< https://launchpad.net/bugs/558585 > [01:40] That's like complaining that Launchpad doesn't work when the DB is down. [01:40] Except that mizuho is shit and a SPOF. [01:42] Next p-s-c run will hit SPR 98000 [01:43] wgrant: its arguable [01:43] I'm not sure if we should say 'yes, yes it does' [01:44] or 'we should degrade gracefully' [01:48] StevenK: Isn't the much faster than we thought? [01:48] Or are lots of early SPRs missing? [01:49] Lots of early SPRs are missing changelogs [01:49] I mean missing in general. [01:49] I don't know what evil was done before the first import. [01:49] Ah. The first SPR id run across was 59510 [01:49] Right. [01:49] wgrant: are you still hacking on https://bugs.launchpad.net/launchpad/+bug/575450? [01:49] <_mup_> Bug #575450: Archive:+copy-packages nearly unusable due to timeouts < https://launchpad.net/bugs/575450 > [01:50] lifeless: The next piece of work in that is bug #733071. [01:50] <_mup_> Bug #733071: Archive:+copy-packages slow when potential conflicts detected < https://launchpad.net/bugs/733071 > [01:50] But there is also work needed to speed up the UI, but I haven't analysed that yet. [01:50] wgrant: there is a stall condition here [01:50] Yes. [01:51] wgrant: if you split out a bug like this, please assign yourself to all components, or only leaf components [01:51] wgrant: you see why ? [01:51] Yeah, sorry. [01:51] no worries [01:51] bigjools pinged me about this morning when discussing the drop to 11 seconds [01:53] hes concerned about usability during release [01:53] That's +queue. [01:53] Not +copy-packages [01:54] wgrant: If my one-liner of evilness is right, we've processed 38k SPRs since the 15th. [01:54] StevenK: Sounds reasonably plausible. [01:54] Pity there is another 500+ k [01:55] wgrant: bah, emybrain [01:55] Speaking of 11 seconds. [01:56] 424 Time Outs [01:56] :( [01:57] You didn't want 424? [01:57] We had 252 yesterday. [01:57] So, no. [01:57] wgrant: The BUILDMASTER oops concerns me [01:57] Since the same thing happened yesterday too [01:57] StevenK: That normally just means the source has been superseded. [01:57] It often happens. [01:58] I thought jelmer had fixed that, but maybe not complete. [02:02] I guess EB and ED are also appservers? [02:02] EB and ED are edge appservers. [02:02] Ahh [02:02] They need more fire. [02:03] We need to get rid of potassium and gandwana. [02:03] Now. [02:03] The +copy-packages GET timeouts are all high non-SQL on gandwana and potassium. [02:04] lifeless: When are the new appservers happening? [02:05] wgrant: probably not till mthaddon is back [02:05] :( [02:05] elmo was checking with charlies/nafallo about the memory [02:06] Is the session queue normally sitting at 0? [02:06] ie. would it help to deprioritise gandwana/potassium instances? [02:06] cve:+index spiked [02:06] jcsackett: has a fix for that [02:06] He does. [02:06] is it landed? [02:06] No. [02:06] >< [02:06] Kanban says it's in review. [02:06] Checking it. [02:06] dropping the timeout [02:07] Excellent. [02:33] Wow, apparently windmill is fixed [02:34] rm? [02:39] have we done anyting to improve https://launchpad.net/ubuntu/+source/kde4libs/+publishinghistory ? [02:47] StevenK: did they use the great "rm -rf" debugger? [03:06] StevenK: Windmill is not fixed, but wallyworld fixed the broken test. [03:07] lifeless: My SPR.copyright fix will have improved that. [03:07] It still times out occasionally, though. [03:13] https://code.launchpad.net/~thumper/launchpad/blueprint-linked-bug-tasks/+merge/53734 anyone? [03:14] I have three other reviews which are the start of the pipeline [03:14] but they are trivial [03:14] and I'm tempted to self review [03:14] but if someone wants to look at them [03:14] here they are: [03:14] https://code.launchpad.net/~thumper/launchpad/bugtask-repr/+merge/53731 [03:14] https://code.launchpad.net/~thumper/launchpad/bugtask-tales-addition/+merge/53732 [03:14] https://code.launchpad.net/~thumper/launchpad/add-publishing-for-factory-distro-sourcepackage-bug-tasks/+merge/53733 [03:15] In bugtask-tales-addition you seem to be trying to reimplement structuredstring. [03:16] wgrant: what do you mean? [03:16] _make_title escapes stuff. [03:17] wgrant: I implemented it in the same way that the main link is generated [03:17] wgrant: I didn't think too hard about the implementation [03:17] Yes, and most of our code is buggy :( [03:17] is that bit buggy? [03:18] It's not buggy, but it's not a pattern that we want to perpetuate. [03:18] If you are calling cgi.escape, you are probably doing it wrong. [03:18] where is structuredstring? [03:19] canonical.launchpad.webapp.menu, or somewhere stupid like that. [03:19] Yeah, canonical.launchpad.webapp.menu.structured is the constructor. [03:22] hmm... [03:22] wgrant: Hm, do you know the diff? [03:22] StevenK: Which diff? [03:23] wgrant: wallyworld's that fixed the failing windmill test [03:24] StevenK: r12593 [03:25] wgrant: can you think of a better place than lp.services.utils for structured? [03:26] lp.app.utils [03:27] StevenK: it isn't really LP specific [03:27] I think lp.app is probably more appropriate. [03:27] But I'm not really sure. [03:27] SpamapS: ping [03:27] https://bugs.launchpad.net/launchpad/+bug/627974 - the url I got from you, did you get that from launchpadlib, or were you url hacking ? [03:27] <_mup_> Bug #627974: MaloneApplication:CollectionResource is slow/times out < https://launchpad.net/bugs/627974 > [03:29] wgrant: lp.app.browser.string ? [03:30] or lp.app.browser.structured [03:30] thumper: string, I think. [03:30] Mm. [03:30] I don't know. [03:31] it needs an interface module too [03:31] I'd not be happy putting it in lp.app.interfaces.launchapd [03:31] or launchpad even [03:31] I'd be happy to have the interface and implementation in a single file in lp.services.string [03:31] Indeed not. [03:32] and include escape and structured [03:32] But lp.services is not webapp-specific, so lp.services.string is wrong. [03:32] lp.services.escaping, maybe. [03:32] l.c.l.w.string then [03:32] But I'd prefer lp.app somewhere. [03:33] lifeless: we are trying to remove things from c.l.w [03:33] wgrant: lp.app with two files? [03:34] thumper: I know, but that seems pointless to me, the angst about it is dwarfed by the number production and deployment issues spent fixing moves [03:34] I can't bring myself to care about labels [03:35] the arguments for moving are that the old structure impeded splitting the tree, but the analysis for splitting the tree was wrong [03:35] * lifeless shrugs [03:36] perhaps we should just move canonical.launchpad.webapp to lp.app.webapp [03:36] :) [03:36] thumper: I am not sure if bugtask-repr is safe. [03:36] wgrant: why? [03:37] thumper: It potentially exposes the bug ID and target, if you manage to get hold of a BugTask through the webservice. [03:38] wgrant: the webservice handles launchpad.View [03:38] wgrant: and a bugtask view permissions filter to the bugs [03:38] wgrant: so not a valid concern I believe [03:38] thumper: The webservice handles launchpad.View for collections, sometimes. [03:38] having that repr certainly helped my debugging [03:43] 8 of todays top 10 OOPSes are fixed. Yay. [03:43] The next few are codehosting. [03:43] Blue squad sounds like a good one to take those :P [03:47] wgrant: I'm having a such a brain fade -- can you remind how to get the latest version of an SPR in a distroseries. [03:48] StevenK: That doesn't really make sense. But getCurrentSourceReleases might do what you want. [03:48] Depending on what you actually mean. [03:48] It is ambiguous in a couple of ways. [03:49] wgrant: So, Julian would like to delete the DSD is the derived series and parent series have the same version published. This makes sense to me, but I'm having a brain fade looking for the method to help me. [03:50] That's deleting the comment history. [03:50] Bad idea to do that so early. [03:50] You can possibly justify doing that once a series is done. [03:50] But not before. [03:51] Why? If something has been synced, the comments are now irrelevant/ [03:51] s/\//./ [03:52] StevenK: I want to know why it was synced. [03:52] Then you can look up the bug. [03:52] There wasn't a bug, because we are no longer in the dark ages. [03:52] The sync button has been implemented. [03:53] Good point. [03:58] how do we avoid the ValidPersonCache queries? [03:58] can we precache it? [04:01] thumper: list(getUtility(IPersonSet).getPrecachedPersonsFromIds(ids, need_validity=True)) [04:02] wgrant: I have the people already from a different join [04:04] although looking at this method, I may change it [04:06] thumper: are you filtering on person ? [04:07] lifeless: I'm looking at two timeouts [04:07] for spritns [04:07] Sprint:+huge-vocabulary? [04:07] one on +index seems to be getting valid persons [04:07] and +attendees-csv is just shite [04:07] ok [04:07] bug #'s ? [04:07] the attendees-csv could be trivially fixed [04:07] except for the ircnicknames [04:08] bug 735996 [04:08] <_mup_> Bug #735996: Sprint:+index timeouts < https://launchpad.net/bugs/735996 > [04:08] there isn't a bug for +attendees-cvs db counts [04:09] ircnicknames is a SQLMultipleJoin [04:09] so not sure how to precache the names for all the people :-( [04:09] You'll have to make it a cachedproperty returning a list. [04:10] wgrant: but that doesn't mean we can get them all at once [04:10] thumper: It does... [04:10] how [04:10] store.find(IRCNick, IRCNick.personID.is_in(personIDs) [04:10] grep around for pre_iter_hook [04:11] hmm... [04:11] I'll look around [04:28] jtv: perhaps you could comment on https://bugs.launchpad.net/launchpad/+bug/708875 [04:28] <_mup_> Bug #708875: AssertionError: More than 6 plural forms were submitted < https://launchpad.net/bugs/708875 > [04:28] lifeless: should be an unexpectedformdata if it happens from the web UI. [04:29] Seems a far-out guess that this is coming from the web UI, but I'm pretty sure I made the importer check for this condition way back when. [04:30] I think the check is wrong. [04:30] I looked at it yesterday and ran away. [04:30] There are only forms 0-5 [04:30] Which is 6. [04:30] Not more than 6. [04:36] The check doesn't even make sense to me. It's an "else" attached to a "for" loop AFAICS [04:36] Right. [04:36] That does make sense. [04:36] But it's not right here. [04:37] (the else block is executed if the loop runs to completion, without a break) [04:37] Gawd [04:37] Pretty much. [04:37] So yes, that explains the off-by-one [04:37] which is the absolute opposite of what *everyone* reads it as [04:37] The loop breaks when it hits the first plural form that is supported but is not in the form. [04:37] And when you submit exactly the maximum number of forms, it never does. [04:37] FFS. [04:38] Exactly. [04:38] But I thought this must have come up often. [04:38] Does it not? [04:39] wgrant: still working on bug 714414 ? [04:39] <_mup_> Bug #714414: unstack debian sid branches < https://launchpad.net/bugs/714414 > [04:39] Probably, yes, but plural forms are relatively rare, and only Arabic uses 6. [04:39] jtv: Right, but wouldn't it occur on every plural arabic translation, then? [04:40] Or is the last one uncommon? [04:40] Every complete plural arabic translation made in the UI, yes. [04:40] Given the debate around ubuntu-l10n-ar, I guess we don't have many arabic translators. [04:40] lifeless: Not really. [04:41] An upstream translation. or one uploaded by a translation team member, wouldn't have this problem. [04:41] I'm expounding on the bug. [04:44] wgrant: unassign? [04:46] is https://bugs.launchpad.net/launchpad-buildd/+bug/719162 deployed? [04:46] <_mup_> Bug #719162: [Natty] check-implicit-pointer-functions fails on natty, resulting possibly broken packages < https://launchpad.net/bugs/719162 > [04:48] lamont was looking at that earlier this week. We need an amd64 builder on staging before we can QA it. [04:48] So no, it's not yet deployed. [04:52] stub: hey [04:53] can you see how long the very big query in bug 722787 takes now on prod? [04:53] <_mup_> Bug #722787: Product:+filebug timeouts < https://launchpad.net/bugs/722787 > [04:53] I suspect the index repack fixed it [04:59] * stub tries to find a real query [05:01] wgrant: and are you doing 729064 ? [05:08] lifeless: Yes. [05:08] I can't even cut and paste the query without making terminal explode :-P [05:08] https://lp-oops.canonical.com/oops.py/?oopsid=1875O1575 has bigger problems than just timing out [05:08] stub: :P [05:09] love our ui [05:09] There are currently no open bugs. [05:09] https://bugs.launchpad.net/launchpad-project/+bugs?field.importance=Critical&start=225 [05:11] lifeless: So the insane query in that oops runs in 3 seconds atm. on one of the slaves. [05:12] stub: great,thanks [05:13] * lifeless closes [05:13] 224 critical bugs [05:13] tada [05:13] including 6 5-digit ones [05:14] stub: going to cook dinner; catchup after that? [05:14] sure [05:23] hi [05:24] re bug 719271, are there multiple threads per scanner process? [05:24] <_mup_> Bug #719271: Branch scanner jobs failing in bzrlib < https://launchpad.net/bugs/719271 > [06:20] stub: ping [06:24] lifeless: pong [06:24] skype? [06:28] https://staging.launchpad.net/successful-updates.txt is odd. [06:30] stub: https://dev.launchpad.net/LEP/ReliableDBDeploys [06:37] Can we move to Python 3 please? [06:37] How do you suggest we move Zope and Storm to Python 3? [06:37] Shh [06:38] Oh, you don't want problems, you want solutions? [06:39] I want non-awful Unicode. [06:39] I also want lucid_db_lp to fail sensibly. [06:39] Because its current failure is not sensible. [07:16] lifeless: tasque [07:30] wgrant: happier with the bug count now ? [07:35] lifeless: Mostly. [07:47] wgrant: there's a question about getting increased space for a ppa. who do i assign that to? [07:51] wallyworld: A LOSA, or poke a commercial admin [07:52] that doesn't sound right. we haven't been doing space increases for ages. can you guys already do that? [07:52] yourselves as in. ? [07:53] spm: if we can do it, i don't know how :-) [07:53] it's an admin option off the ppa itself. or was. [07:53] It needs a commercial admin or a LOSA, and there is approximately one active commercial admin. [07:54] i also have a question about a confirmation code being rejected when someone tried to create an account. assignee? [07:54] wallyworld: Ask them to file a ticket at the SSO support site. [07:54] that needs a level of sanity filtering first [07:54] Or reassign the question to canonical-identity-provider [07:55] wgrant: thanks. will reassign to cip. what's the sso support site? [07:56] https://forms.canonical.com/lp-login-support/ [07:56] thanks [08:01]