[00:24] lifeless, for my education, is there a second category of oopses for things like 404s [00:25] like "probably not our fault" [00:25] ? [00:27] I think at the moment that we do some second-pass filtering in the oops web UI [00:27] Thats wasted overhead though [00:27] OOPS are fairly expensive to create [00:27] so I plan to remove that [00:27] btw hearty +1 to stopping launchpad-bugs backscatter [00:27] and i'm sure jam & co would agree too [00:27] the rule of thumb for oopses is: [00:28] - if its logged we should do something about it [00:28] - (corollary) if its something we cannot do anything about, we should not log it [00:33] sinzui: thoughts on https://bugs.launchpad.net/launchpad/+bug/744888/comments/4 ? [00:33] <_mup_> Bug #744888: Muting bug 1 times out

< https://launchpad.net/bugs/744888 > [00:35] lifeless: I thought Snapshot was taught to always ignore collection fields [00:36] sinzui: Thus I'm asking you :) [00:36] ah, but bugs and questions want to snapshot for the last message I think. Maybe it was not done [00:37] a collection snapshot could just record the tip message (for some apporpriate order) [00:37] and the delta is 'messages newer than X' [00:37] [e.g. see my batch navigator work] [00:39] lifeless, i wonder if there needs to be a category like in bzr where we log a traceback but do not show it to the user [00:39] ie we don't think it's a problem, but if it turns out that they think it's a problem, they still have the data [00:39] poolie: we have that [we use it for nearly-timedout-renders] [00:40] poolie: but I don't think an OOPS is appropriate for the case we're discussing [00:40] oopses gather a lot of data to permit diagnostics and correction [00:40] sure [00:41] this case is pretty clear - the nonce /has/ been used. Whatever went wrong was in the request before the one having trouble [00:41] i agree with avoiding the waste and being clear about what's actually a problem [00:41] thumper, thanks for fixing bug 377519 [00:41] <_mup_> Bug #377519: Stacked on location breaks if the stacked upon branch is renamed

< https://launchpad.net/bugs/377519 > [00:42] if that's going to be rolled out gradually, i would like to add a note on the bug saying that people won't immediately be able to verify it as fixed [00:42] or, they won't immediately see the problem go away [00:42] is that correct/ok? [00:42] anyhow, it was a one off [00:42] poolie: it isn't fully fixed [00:43] i was just curious [00:43] poolie: I forgot to put the --incr flag on the landing [00:43] yes feel free to add a comment [00:43] poolie: it won't be fixed fixed until we do something for existing branches [00:43] I think communicating expectations to our users is a great idea [00:43] :-) [00:43] I'd edit the summary at the top to describe whats going on [00:43] i really liked how you did that the other day [00:43] i forget which bug it was [00:44] one reason to do this is that bug search only searches the subject and summary, not the comments. [00:44] this was a performance thing resulting from the way the fti was assembled way back; I haven't had time to review and examine how to enable searching in commenets [00:45] without messing up performance [00:45] wallyworld_: BTW, Unity distracted me from saying this on the call, but I have no problem with Windmill being back. [00:45] wallyworld_: I want it to be back on by default, but it's simply too expensive if it fails. [00:46] wgrant: np. i agree [00:46] wallyworld_: So I will not hesitate to disable it upon the first spurious failure in buildbot. [00:50] sinzui: ok so we faded out there [00:50] sinzui: is my suggestion a) broken, b) sensible, c) we need to modify how snapshotting deals with collections ? [00:51] sinzui: or d) none of the above [00:52] lifeless: sorry. I very distracted. IAre we certain that is needed? was it cargo culted from old code? [00:53] as for bug and questions looking at messages, I think that is lazy design. I believe there are two emitters of the modifications and they know if a message was appended [00:53] sinzui: bug one has several thousand messages [00:54] sinzui: I'm certain we either need to not snapshot it, or snapshot it much more efficiently [00:54] sinzui: *and* I'm certain that what is snapshotted internally shouldn't match the API attributes for this object either, as the API has different needs vs in-appserver deltas. [00:55] So the publisher emitting the event should be clear what message was added instead of letting the subscriber guess. I do not think snapshot should be working with deap data [00:55] deep [00:56] wallyworld_: From about line 450: http://bazaar.launchpad.net/~huwshimi/launchpad/private-objects-298152/view/head:/lib/lp/bugs/javascript/bugtask_index.js [00:56] sinzui: ok, cool. [00:56] so we doNotSnapshot it, see what breaks, fix the signal to contain the new message if needed [00:57] yep [01:05] wgrant: speaking of bugs.. 744849 [01:06] Bug #744849 [01:06] <_mup_> Bug #744849: populate-archive.py unusably slow < https://launchpad.net/bugs/744849 > [01:06] Yeah. [01:08] man, that was the easiest critical fix ever. [01:11] Oh? [01:11] search for -de in a person picker [01:11] Um, what the failure? http://paste.ubuntu.com/590501/ [01:12] huwshimi: Aha, that has shown up on Jenkins before. [01:12] huwshimi: Spurious, but I'd never seen it elsewhere. Thanks. [01:12] huh [01:12] temp_dir - replace that with fixtures.TempDir [01:12] more robust implementation [01:14] Should I just lp-land it? [01:14] huwshimi: If that was the only failure. [01:15] wgrant: It was, assuming the rest of the tests ran ok [01:16] huwshimi: How many tests did it report? [01:18] wgrant: 13988 [01:22] huwshimi: That sounds right. [01:22] Land it. [01:26] wgrant: Thanks maate [01:26] *-a [01:29] hmm [01:29] is this thing on? [01:30] irc is [01:34] <_thumper_> :( [01:34] <_thumper_> quassel still having problems with the sqlite file [02:06] lifeless: i got bit by the temp_dir issue too. unfortunately the version of fixtures we're on atm does not have TempDir. you see any problem in this change until we can update fixtures and use it? http://pastebin.ubuntu.com/590522/ [02:10] bac: bin/py -m pydoc fixtures shows TempDir as being available [02:10] bac: so I think its usable now [02:11] bac: why do you say it isn't ? [02:11] er, did that just happen? let me update [02:11] bac: no, TempDir is ancient [02:11] lifeless: b/c i looked in the egg and didn't see it [02:12] bac: :!grep TempDir -r eggs/fixtures-0.3.5-py2.6.egg/ [02:12] eggs/fixtures-0.3.5-py2.6.egg/EGG-INFO/PKG-INFO: TempDir [02:12] ... [02:12] lifeless: argh, bitten by emacs file completion... [02:13] bac: I don't mind if you fix temp_dir, but deleting code seems worth a little extra time [02:13] it does need spelling very slightly differnetly (see the pydoc for TempDir) [03:45] wgrant: Should I disable the windmill job on Jenkins? [03:45] StevenK: Probably. But don't delete it... I suspect we might need it back soon. [03:45] Hah [03:46] wgrant: Oh, sure, I was just going to disable it. [03:46] wgrant: Your hate of Windmill is showing. :-) [03:46] I don't hate it. [03:47] I just don't trust that it isn't go to break again. [03:47] And breakage is really really bad. [03:47] Loathing? [03:47] So, the tests that we currently implement through Windmill are necessary. [03:47] They catch real breakage. [03:47] But they are slow and unreliable. [03:53] RARGH, the package cloner is creating builds in the *parent* [03:53] Bad package cloner! [03:54] Heh. [04:01] wgrant: I'm really glad you hold that position now; its (I think) much more useful than the fire n brimstone one you had a few months back. [04:02] lifeless: I have held this position for several months. [04:02] Oh, duh. The first createMissingBuilds call is *me* [04:02] Perhaps the breakage had greater emphasis back then, but the breakage was also itself greater. [04:04] wgrant: Well, 6 months back then :) - anyhow, I like the current attitude; I didn't mean to imply you had a particularly bad one in the past. [04:09] wgrant: how did publish-ftpmaster work out on dogfood yesterday? [04:16] wgrant: sorry, bad news.. [04:16] wgrant: that bug still freaks out in prod [04:18] lifeless: is there a relatively simple way to see if a recent oops contains a particular string? [04:18] yup, grep [04:18] they are all in the lpnet logs dir on asuksa [04:19] a wildcard match will let you pass the directories you need to grep [04:19] grep -i string */2011-04-04 -r [04:19] etc [04:19] bbiab, shopping run for lynne [04:20] are they linked on devpad anywhere? [04:26] oh [04:26] I meant devpad [04:26] asuka is staging [04:26] <- 6am starts bad [04:27] lifeless: Bah. Worked for me at least once :/ [04:27] jtv: I filed a few bugs. [04:27] wgrant: oh… anything serious? [04:27] jtv: I enabled the run-parts stuff. [04:28] jtv: Only one significant non-run-parts bug, IIRC. [04:28] wgrant: do you have a list somewhere? I take it these are all things that need resolving before we can use the new script. [04:29] http://bugs.launchpad.net/launchpad/+bugs?field.bug_reporter=wgrant&orderby=-datecreated [04:30] I think #752279 is a somewhat bigger issue. [04:30] <_mup_> Bug #752279: publish-distro.d scripts expect partner to have dists.new

< https://launchpad.net/bugs/752279 > [04:30] It's not clear that the rsync is working. [04:30] wow that's taking long to load [04:30] lifeless: :/ it just render in 8.1s for me on prod. [04:31] +ed [04:31] Waiting for the OOPS to sync :( [04:32] jtv: All of those bugs except 752279 have workarounds in the DF tree. [04:33] For 752279 there might be a debug print left in the script, since the logging doesn't seem to be working. [04:33] I carefully published partner as well as lucid's non-release pockets yesterday, so we have an archive of reasonable size to work with. [04:33] Just needs a bit of SQL to dirty them. [04:34] wgrant: partner is supposed to do the same dists.new dance as the primary archive. If that's not working, I'd look for a path problem first. [04:34] jtv: Yeah, I realised that after I filed the bug. [04:35] jtv: The script currently crashes in a location that cannot be determined because the atexit crashes. [04:35] There is no atexit. [04:35] Hm. [04:35] Unless it's in the LaunchpadScript's lock-cleanup code or something, but that shouldn't matter. [04:36] If you give me a moment to retake my desktop from the dead grasp of Unity, I will get you the traceback. [04:37] jtv: It's in the finally. [04:37] http://librarian.dogfood.launchpad.net/57722542/7gpgHGXJT50D5VFRhqsoFWlJEla.txt [04:38] 13:35:47 < jtv> Unless it's in the LaunchpadScript's lock-cleanup code or something, but that shouldn't matter. [04:38] 13:36:09 < wgrant> If you give me a moment to retake my desktop from the dead grasp of Unity, I will get you the traceback. [04:38] 13:37:34 < wgrant> jtv: It's in the finally. [04:38] 13:37:37 < wgrant> http://librarian.dogfood.launchpad.net/57722542/7gpgHGXJT50D5VFRhqsoFWlJEla.txt [04:38] wgrant: thanks [04:42] wgrant: I was sort of expecting problems there. [04:42] Yeah. [04:42] Though I don't really see what "directory not empty" means in this case. [04:42] jtv: I suspect it's renaming a directory on top of an existing one. [04:43] mv works fine there; it just moves *into* it. [04:43] rename won't do that. [04:43] That's exactly the problem I was half-expecting. :) [04:43] Basically, it uncovers a bug in the old script AFAICS [04:43] Yeah. [04:43] It won't fail. [04:43] But it will do something completely wrong. [04:44] Question is, is it okay to delete the directory that's in the way? [04:44] Mumble. [04:44] Coffee. [04:45] Please forgive me for pressing Ctrl+Alt, oh Unity. [04:45] No, it will have none of that. [04:46] lifeless: Uhhh [04:46] lifeless: Your OOPS. [04:46] SQL time: 1381 ms [04:46] Non-sql time: 11600 ms [04:47] wgrant: I'm on mumble but don't see you. [04:48] jtv: Oh, that was me mumbling to indicate that I'm not sure. [04:48] Not saying that we should talk about it... although maybe we should. [04:48] lifeless: wampee, too. This looks fairly unideal. [04:49] wgrant: you think contention ? [04:49] I wish we had the actual oops id in the filename instead of some other random shit [04:49] thumper: next iteration will have a search engine [04:49] having to open the file to find the oops id is a PITA [04:49] thumper: and no filenames [04:50] lifeless: I don't know, but this is a pretty serious issue that warrants immediate investigation. [04:50] I forget K's threading config, but it sounds big. [04:50] wgrant: check lp-prod-configs [04:50] I am. [04:51] wgrant: 4 [04:51] lpnet11 [04:51] Yeah. [04:51] Is that 2 concurrent requests in hap? [05:02] 6 [05:02] or perhaps 4 [05:02] check the html dump mthaddon gave me earlier [05:02] Yes, but it had spaces in the name. [05:03] And I can't be bothered typing all those backslashes. [05:03] \\\ [05:03] have mine [05:09] thumper: what app did you use to get your recent screen capture? [05:10] record my desktop [05:10] thanks [05:10] or something like that [05:19] jtv: :( [05:19] Huh. [05:28] \o/ another unicode bug bites the dust [05:28] :) [05:34] * StevenK wishes Storm supported INSERT INTO [05:36] StevenK: add it [05:36] I can give you some tips if you like [05:37] ok, trying for the shops again... bbiab [05:37] Writing SQL via format string is beginning to wear thin [05:40] And trying to add another column to copy sucks [05:45] StevenK: How're you doing it? [05:45] The way I did it in PublishingSet.publishBinaries isn't *that* unmaintainable. [05:45] But it's certainly still not pretty. [05:45] wgrant: I don't think I need it. [05:46] The test case is making my primary archives with requires_virtualized true [05:46] Which is probably wrong [05:47] wgrant: Do you agree? [05:48] StevenK: It's valid either way. [05:48] I don't know what your problem is, though. [05:48] Virt primary archives work fine most places I've tried, though. [05:49] wgrant: This is for IDS, and IDS doesn't copy DAS.supports_virtualized [05:50] StevenK: Ahh. [05:50] That's a bit awkward. [05:50] So if the archive is requires_virtualized, IDS can't create builds [05:50] But, er, why is IDS creating builds? [05:50] I guess for failed ones from the previous series. [05:50] Right [05:50] If IDS has a "rebuild copied sources" option, I will cry. [05:51] ... it does [05:51] What the fuck? [05:51] How is that going to work? [05:51] Somebody didn't think this through :P [05:51] Read the code? It works [05:52] I guess if you create the archive disabled and add external deps with a pre-bootstrapped archive you might get somewhere. [05:52] But, er, creating an empty archive with lots of builds isn't going to get you very far at all :) [05:53] wgrant: I seem to recall Julian was against the idea of copying supports_virtualized, but I can't recall the reason [05:54] StevenK: We don't normally want to support PPAs for a while after the series is initialised. [05:54] wgrant: The idea is for "Scorched Earth rebuilds" with different compiler flags and such, but yes, it will need magic to happen [05:54] Exactly [05:54] So I think I'll change the test harness to make sure Archive.requires_virtualized == False [05:55] Evil magic with sources.list.d is lamont's plan, I believe. [06:16] https://code.launchpad.net/~thumper/launchpad/fix-unicode-path-oops/+merge/56688 [06:16] for those that love unicode === almaisan-away is now known as al-maisan [06:24] jtv: is today a loliday too ? [06:24] lifeless: not that I know, too [06:25] I mean, no [06:25] why? [06:25] thanks [06:25] I have a call scheduled with stub on thursdays [06:25] if hes on leave I can naff off [06:25] and you are lower latency than canonicaladmin [06:31] It's always good for a human to hear that their unique usefulness lies in a slight latency advantage over computers for simple menial tasks. [06:33] Keeps us in our place. [06:46] lifeless: There are some unofficial tags that need a cleanup/removal as well. For example 'post-3-ui-cleanup' should probably just be converted to UI now. Is there any reason I shouldn't just fix that one myself now? [06:46] oh Facepalm: we have a cache of messages on bug, and we don't use it. --fail-- [06:46] huwshimi: JFDI [06:46] lifeless: Sure :) [06:46] jtv: us humans are uppity creatures :) [06:47] yes… can't wait for skynet to take over [06:51] wgrant: I'm fixing the $ARCHIVEROOTS escaping problem. I think the right way to do it would be a double-quoted, escaped string containing double-quoted, escaped strings: "\"directory1\" \"directory2-with-weird-\$\" \"directory\`3\`\"" It may affect quoting in the plugin scripts, though I think it'll be cleaner overall. [06:52] lifeless: Is bug #438540 made invalid due to the substring search change made recently? Or was that relating to something else? [06:52] <_mup_> Bug #438540: assigned to search doesn't do partial matches

< https://launchpad.net/bugs/438540 > [06:52] looking [06:53] jtv: Shell escaping makes me even more sad than JS escaping in classic HTML. [06:53] And that, I hope, is very sad indeed. [06:53] The thing about JavaScript escaping is the false promise: [06:54] huwshimi: no, different search - thats the person picker [06:54] you think you can escape JavaScript, but that's not what it means. [06:54] huwshimi: 'person *to assign to* [06:54] huwshimi: refresh the bug, I've clarified it [06:54] lifeless: Ah right. Yeah I didn't read the description properly [07:07] lifeless: Is our AJAX timeout set to 30 seconds? [07:08] huwshimi: Server-side it's the same as everything else. [07:08] 11s. [07:09] huwshimi: if you see a longer duration one of the following is true: [07:09] - hit max concurrent http connections in your browser [07:10] - poor network e.g. ssl reneg, lost packets, whatever [07:10] - the request stopped doing all db activity before 11 seconds and just played with itself for the rest of the time [07:10] huwshimi: the first one is fairly common, I think. [07:10] [not an exhaustive list, but roughly in terms of frequency/probability I think] [07:11] Interesting. Whenever I get an AJAX failure it always dies at 30 seconds. I wonder if that's being set by YUI? If so we should change that to match our 11 second timeout, right? [07:12] no [07:12] not unless we can guarantee: [07:12] - never have more than N (unknown) concurrent ajax requests to the same host (even from other browser windows) [07:12] - guarantee network conditions won't go quirky [07:13] huwshimi: 30 seconds is the default network SYN attempt timeout IIRC [07:13] huwshimi: so I think you have issues, perhaps the same as smpillaz, with your network path to LP [07:14] huwshimi: do you find bzr push very slow sometimes (> 15 seconds to get started) [07:14] lifeless: I haven't noticed that [07:15] ok [07:15] well, we need to find and fix this [07:15] but I don't think dropping an in browser timeout is sensible given the legitimate reasons it might queue in your browser. [07:15] lifeless: If I have Launchpad open in a bunch of tabs is that where I might hit the max concurrent connections? [07:15] yes [07:16] or even on one tab [07:16] bugtask:+index seems to do 8 separate ajax requests [07:16] which will probably exceed your concurrency limit by some-N (I haven't checked chromium, but it could be as much as 6) === lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews | qastaging down for a schema test, back soon [07:21] lifeless: So what I was doing was opening a bunch of bugs in separate tabs and then editing each one. I was timing about about 40% of the time (rough guess). What I just tried was opening one bug in a new tab and editing it, closing the tab and starting again. I had 3 tabs open max. The first two were fine, the third timed out. === al-maisan is now known as almaisan-away [07:24] Error: Couldn't find a distribution for 'zope.publisher==3.12.0'. [07:25] Is that caused by the aggressive cleanups people have been doing? It's on a fresh db-devel branch, after rocketfuel-get/rocketfuel-setup/bzr pull/make clean [07:25] wgrant: Can you explain to me a bit more about bug #433342? I'm not sure where you're talking about. [07:26] <_mup_> Bug #433342: 1em extra space on top of every page

< https://launchpad.net/bugs/433342 > [07:26] jtv: I see a zope.publisher-3.12.0.zip in download-cache. [07:27] I don't. Only zope.publisher-3.12.1.tar.gz. [07:27] wgrant: Do you know anything about hello-partner in maverick's NEW queue on mawson? [07:27] StevenK: No, I only published partner carefully, I didn't upload anything. [07:27] I blame jtv and bigjools. [07:27] wgrant: It's 16 weeks old! [07:28] Oh. [07:28] That could be me, then. [07:28] * jtv has an alibi [07:28] I did do something around then. [07:28] huwshimi: It seems to be fixed now. [07:28] wgrant: Sweet. [07:29] Changed-By: William Grant [07:30] * StevenK glares at wgrant [07:30] StevenK: Yeah, I didn't realise it was still there, so I thought it was more recent. [07:30] Description: hello-partner - The classic greeting -- proprietary edition [07:30] 16 weeks old... really? [07:30] Does it say "Hello, 640K is enough for anybody!" ? [07:30] What's the date on it? [07:31] Date: Sat, 18 Dec 2010 16:55:52 +1100 [07:31] That's from the .changes [07:31] Ah, so the week I started. [07:31] * StevenK rejects it [07:43] https://code.launchpad.net/~lamont/launchpad/lp-buildd-78/+merge/56700 needs a review. [07:43] Already deployed on the machines that matter. [07:44] It redefines revolting, but it works! [07:45] That is *hideous* [07:46] Yse. [07:46] wgrant: DOne [07:47] Thankyou. I believe IS now owes us new eyes. [07:52] IS or lamont personally? [07:52] Either will do. [07:53] and what's so bad about that? looks downright elegant to me. nicely formatted over a few lines and everything! [07:53] damn. my nose just grew about 5 inches [07:54] spm: I'm sure Little Red Riding Hood's "grandmother" had a good one to explain that away, but I forget what it was. [07:55] heh [08:01] WindMIIIIIIIIIIIL [08:01] L [08:01] wgrant: It just failed? [08:02] * wgrant disabled that test. [08:02] s/disabled/disables/ [08:02] StevenK: lucid_db_lp [08:02] GAAAAAAAAAAAAVIN! [08:02] * StevenK found a bug in +initseries [08:02] Oh? [08:03] If you don't select any packagesets it feeds packagesets=[""] to DistributionJob [08:04] Bah, qas is down. [08:04] Bad lifeless. [08:06] wgrant: bad schema 2004 [08:13] I hope PQM will accept a devel testfix for a db-devel failure. [08:14] yes [08:25] stub: hi [08:25] lifeless: yo [08:25] we're slated for a call I think [08:26] you coffeed up etc etc? [08:26] Yup. Coffeed up. Hungry but breakfast can wait :) [08:26] k, let me plug mic in [09:05] good morning [09:06] hi adeuring [09:07] hi jtv! [09:07] morning [09:08] O hai [09:08] hi bigjools [09:08] the decision to wear shorts this morning feels like a bad one right now [09:09] Haha [09:09] I'm not going to ask. === almaisan-away is now known as al-maisan [09:20] Morning all ;) [09:20] danilos: Hi! [09:20] Good morning [09:20] hi Henning, how's it going? [09:21] danilos: great, thanks ;) [09:21] danilos: This one is especially for you: ;-) [09:21]