[00:16] Thanks StevenK. [00:17] You are not yet imprisoned? [00:17] What has StevenK done now? [00:17] jelmer: On the call today I said I have a rental inspection, so if I'm not around for the rest of the day it's because I've been arrested for killing my landlord. [00:21] * jelmer tries to imagine what a rental inspection is [00:21] like this: http://www.youtube.com/watch?v=aCiYmCVikjo [00:22] heh [00:23] jelmer: Where the landlord comes over to inspect the place you're renting to make sure you're taking care of it [00:23] StevenK: It looks like the FileBugAPI XML-RPC call has been used around once in the last three months. [00:23] StevenK: I don't think my landlord is legally allowed to do that [00:23] Haha, around once [00:23] wgrant: coincidentally i was wondering this morning if there are or should be postmortems for rollbacks [00:24] wgrant: Zero is around once? [00:25] wgrant: It seems cronscripts/create-debwatches.py uses debsync? [00:25] StevenK: while you're here.. can I add a branch to your queue? [00:25] jelmer: I just reviewed one of yours [00:26] jelmer: Or do you mean a different branch? [00:26] StevenK: ah, race condition.. thanks for the review :) [00:27] And create-debwatches isn't actually run [00:27] Orsum [00:30] No, chromium, you may not have 7GB of RAM. [00:31] Bwaha [00:31] StevenK: Right, I'm pretty sure that create-debwatches and update-debwatches have never been used. [00:32] They were written at the end of 2004. [00:32] And haven't really been touched since. [00:34] StevenK: Heh, look at r7675.771.2 [00:34] One of mine: "Port create-debwatches.py away from IDS.publishedBinaryPackages(). The script didn't work anyway, so I'm just going to pretend that I haven't broken it more." [00:34] Haha [00:35] So, IMO you should delete it and update-debwatches.py, and probably trherefore debsync. [00:35] steven@liquified:~/launchpad/lp-branches/kill-cruft% bzr di | diffstat | tail -n 1 [00:35] 7 files changed, 1890 deletions(-) [00:35] What's removed? [00:36] wgrant: http://pastebin.ubuntu.com/657575/ [00:38] StevenK: You've confirmed we have a bugzilla exporter? [00:38] This script was clearly written for the Ubuntu migration and probably hasn't been touched since, but best to check. [00:40] wgrant: scripts/migrate-bugzilla-initialcontacts.py should probably die a firey death too [00:41] Probably, yes. [00:43] wgrant: You don't mean things like lp.bugs.externalbugtracker.bugzilla? [00:44] No. [00:44] That's the checkwatches backend. [00:44] The bugzilla exporter won't be in our tree. [00:44] Ah [00:45] Meh, delete it anyway. There probably is one. [00:45] Haha [00:46] As a serious question, do we actually care? [00:47] No. [00:56] about what ? [00:57] The ancient bugzilla import script. [00:57] ubuntu bugzilla to lp? [00:57] does it even work now ? [00:58] Very probably not. [00:58] Hence its imminent demise. [01:09] lifeless: The XML-RPC FileBugAPI (https://help.launchpad.net/MaloneXMLRPC) has been used four times this year. [01:09] Can we kill it? [01:09] we what now? [01:11] Hm? === Ursinha is now known as Ursinha-afk [01:17] Can haz review? [01:32] wgrant: how much extra work does a rollback cost? [01:32] 10m, an hour, a day? [01:33] i mean across losas and developers [01:34] poolie: you might like the mail I just sent [01:34] poolie: which puts some structure on this for the pre-deploy case. [01:34] poolie: LOSA time: should only be a few minutes, but I don't know how long they spend watching it. [01:34] ah ok [01:34] poolie: things that have been deployed, and then rolledback incur the pre-deploy pipeline overheads *as well* [01:34] i was just writing up a few thoughts on this change [01:34] Well, if we catch it before deployment, no LOSA time at all. [01:36] poolie: which change ? [01:37] mail-scope [01:37] poolie: so it never got deployed [01:37] poolie: no cost to losa [01:37] it got deployed to qastaging [01:37] no? [01:37] or is that not called 'deployed'? [01:38] an unqualified 'deployed' means production only. [01:38] That's not "deployed" [01:39] poolie: the mail scope change falls directly under the performance tuesday mail I just sent. [01:39] Blah. [01:39] what is that called then, "deployed to qastaging"? [01:39] Or just "on qastaging" [01:41] wallyworld: The Subscribe spinner on your Answers work is a bit odd. [01:41] i reused what was there already [01:41] we can iterate if required [01:41] It doesn't appear until a few seconds after the click :( [01:42] i call the method to display the spinner inthe onclick [01:42] Huh. [01:42] well i thought i did [01:42] i will have to check [01:43] lifeless: yeah it totally does [01:43] It seems to make an XHR request, show the spinner, then make another. [01:44] lifeless: "20 minutes DEPLOY -> deployment -> fails or" [01:44] Is that the 75 minutes or so to qastaging? [01:44] wgrant: has it degraded? [01:44] i am trying to develop more sensitivity to risky landings by looking back at things that failed [01:44] wgrant: but yeah, thats what I was refering to. [01:44] or, that could have been risky but worked [01:45] poolie: I think that we have a skewed environment because of two things: [01:45] - long pipeline [01:45] - single pipeline [01:45] yes, that's clearly true [01:45] as we get more services with independent pipelines, and a faster pipeline [01:46] then we can iterate more in the direction responsive companies are talking about [01:46] wgrant: looks like you are right. it was supposed to be displayed before the first xhr call, but instead happens after. i'll raise a bug and fix. thanks [01:46] lifeless: It only checks every half hour, and takes nearly an hour. [01:47] I don't think applying their basic conclusions to us today will work at all well, though knowing their analysis methods / model for the problem is useful. [01:47] wallyworld: Thanks! If possible, it mihgt be nice to also have it change the "(+) Subscribe" into "(o) Subscribing" like Bugs used to do, and hopefully will again soon. [01:47] wgrant: please send an errata [01:47] wgrant: I would appreciate that [01:47] wgrant: i thought about that approach but initially just wanted to reuse as much as possble what was there already [01:48] lifeless: might you have time to +1 that loggerhead mp? [01:48] https://code.launchpad.net/~xaav/loggerhead/export-tarball/+merge/66408 [01:48] wallyworld: I haven't re-read it yet [01:48] np [01:48] wallyworld: this week has been -very- bitsy. Its a high pri for me. [01:48] that's cool. sorry for nagging :-) [01:49] wallyworld: Good plan. [01:49] nothing to apologise for [01:49] I'm holding up someones contribution, which sucks. [01:49] lifeless: sent; would appreciate your thoughts on it some time [01:50] lifeless: You were looking over my kill-cruft MP to review it, or out of interest? [01:50] * wallyworld puts on earplugs - electrician installing new solar inverter - *very noisy* [01:50] StevenK: s/interest/horror/ [01:50] StevenK: :) [01:50] StevenK: I think you should JFDI [01:50] oh no, the inverter's upside down! [01:50] poolie: I wanted to touch base with you anyhow; would you like a chat now ? [01:50] sure [01:51] * wallyworld slaps forehead at poolie's bad pun [01:51] pots? [01:51] I'm on the laptop, so can do sip or skype or whatever [01:51] sip [01:52] 'currently unavailable' [02:11] lifeless: So, I think we should just drop the XML-RPC method. It's been pretty much deprecated since the API was introduced, seems to be used once every couple of months, and whoever is using it can spend the 5 minutes rewriting their script, if indeed someone is actually using it. [02:12] TBH, I agree with lifeless. Let's FF it first [02:16] And soyuz-upload.txt is broken [02:16] Can we delete that yet? [02:16] Wha? [02:16] It passed ec2.. [02:17] http://pastebin.ubuntu.com/657616/ [02:17] Yeah. [02:17] Maybe the race was real. === jtv-afk is now known as jtv === jtv changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK, jtv | Critical bugs: 248 - 0:[#######=]:256 [02:37] StevenK: I think I will just rip the poppy section out of soyuz-upload.txt [02:37] Since test_poppy.py tests it all already. [02:38] Excellent [02:55] In fact, this removes calls to poppy from outside lp.poppy. [02:55] Handy. [02:55] \o/ [02:56] Do you need a review? [02:56] It's only test changes, so I will self-review it. [03:00] hi wallyworld — just letting you know I'm on the mentoring request. [03:00] jtv: thanks! [03:01] I've been doing disappointingly little of that lately, so high time I caught up. [03:20] wgrant: in your lxc's, have you been configuring up postfix ? [03:21] wgrant: or something like nullmailer ? [03:31] wgrant: the subscription portlet's existing behaviour was to display progress after the initial xhr call. i can't fix that for "subscribe someone else" or for when "subscribe me" displays an entry in the list but can fix it for answers as it is currently used [03:37] I wonder if 122 / 3 Person:EntryResource:searchTasks is a regression [03:58] StevenK: want a very small review? https://code.launchpad.net/~wallyworld/launchpad/subscriber-portlet-spinner/+merge/70247 [04:00] wallyworld: r=me, but I'm not very comfortable with my JS-foo [04:00] StevenK: thanks. the change was just moving a few lines around, so should be ok [04:09] erm [04:09] https://launchpad.net/ubuntu/+source/apt/+index [04:09] click on the expanders [04:09] they all seem to go 'failed to fetch package details' [04:09] and retry reloads the whole page [04:10] is this known ? [04:10] lifeless: I haven't. [04:10] lifeless: Just whatever rocketfuel-setup installs, which is IIRC nothing. [04:10] wgrant: recommends are on by default [04:11] wgrant: so it pulls in postfix [04:11] Ah, no, I disable recommends. [04:11] Halves the downloads or so. [04:11] wgrant: right; see my mail to -dev :P [04:11] I always add -o Apt::Install-Recommends=no, mostly for speed and clutter reasons. [04:12] But have never had the heart to do it in devel. [04:12] * wgrant reads. [04:12] Hah [04:12] Erm. [04:12] Isn't that top OOPS meant to be fixed? [04:13] But maybe the fix wasn't a rollback and the original rev wasn't marked qa-bad :( [04:13] 145.73s OOPS-2040CCW4 http://www.urbanvelo.org/ [04:13] ? [04:13] 943 LocationError: (, 'getCacheJSON') [04:13] I suspect not a rollback, not marked bad. [04:13] In my mail about pipelines I've covered this sort of change FWIW [04:14] I'd rather we never do such things. [04:14] https://launchpad.net/ubuntu/+source/apt/+sourcepub/1635507/+listing-archive-extra [04:14] is 404ing [04:14] (from https://launchpad.net/ubuntu/+source/apt/+index) [04:15] wgrant: right any bells? I think you refactored stuff in this area? [04:16] Possibly the expander refactoring a couple of weeks ago. [04:16] I think that should be /ubuntu/+archive/primary, not /ubuntu/+source/apt. [04:17] /** [04:17] * If a page wants to use this outside of an archive context then it [04:17] * can define LP.cache['archive_context'], which should be a [04:17] * full or relative URL to the context archive page required. [04:17] */ [04:18] Suspicious [04:18] And indeed, that's set correctly on DSP:+index... [04:18] So why is it ignored. [04:20] heh, you've got the bit between teeth huh. [04:20] should I file a bug for this, or do you want to ? [04:21] Bit between teeth? [04:22] http://www.usingenglish.com/reference/idioms/bit+between+your+teeth.html [04:22] Ah [04:22] Build Source Stamp: [branch bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel] 13589 [04:22] WGRANT! [04:22] if a horse grabs onto the bit, the bridle can no longer control the angle of its head, because the teeth can take a -lot- of pressure vs the back of the jaw [04:22] Blamelist: William Grant [04:23] StevenK: The testfix landed hours ago! [04:23] Ah, it's soyuz-upload again [04:23] Again? [04:23] This is the build you complained about while it was in progress! [04:24] That was ec2 [04:24] That failure was from buildbot [04:24] Oh. [04:24] I presumed you saw the failure on buildbot. [04:25] Just then, from it's bleating mail [04:26] wgrant: You've killed poppy ftp, can we nail https://bugs.launchpad.net/launchpad/+bug/29645 shut? [04:26] <_mup_> Bug #29645: Weird requirement to poll the ftp.sock during the upload session

< https://launchpad.net/bugs/29645 > [04:27] StevenK: I plan to go through all poppyish bugs soon, yeah. [04:27] Cool [04:36] lifeless: Ah! [04:36] lifeless: It's a fresh regression. [04:36] Aaron's thing. [04:36] It now clobbers LP.cache instead of just adding new items to it. [04:36] wgrant: urgh [04:36] Checking to see what else is affected... [04:36] wgrant: I'll let you file it [04:36] I guess we need to revert prod ? [04:37] Mmm. QA-wise we're in an OK state. [04:38] we don't get js exceptions to count oopses [04:38] We should be able to deploy this fix in 10 hours. [04:38] So, maybe. [04:38] I guess we should. [04:38] so the fallout is unknown [04:38] We have to go back all the way, though. [04:38] yes [04:38] It was my deployment, not the US one :( [04:38] do you happen to know the old good evno ? [04:39] The one before 13548... which was 13543 [04:40] This was reported 22 hours ago, but I didn't notice it :/ [04:42] lifeless: This should be the only fallout of this type. [04:42] There are a couple of other templates which manipulate LP.cache, but they do it on domready and are working fine. [04:43] wgrant: how was it reported ? [04:45] 16:58:00 < diwic> "Failed to fetch package details. Retry" <- is this a known launchpad bug? [04:45] In #launchpad. [04:47] ah, didn't miss a bug report. [04:47] thanks [04:47] Bug #820174, anyway [04:47] <_mup_> Bug #820174: Expanders on DistributionSourcePackage:+index broken by LP.cache changes < https://launchpad.net/bugs/820174 > [04:50] StevenK: Just failed for me in ec2 too :( [04:51] Yet I have an email here, "Test results: murder-poppy => devel: SUCCESS" :( [04:51] Even with your change in devel? [04:51] No. [04:51] I know that's not racy :) [04:52] Strange [04:52] Oh? [04:52] wallyworld: I'm afraid I reverted to my usual nit-picky self on that branch. [04:52] wgrant: That it failed, but you get a success message [04:53] Ah, no, you misunderstand. [04:53] The branch passed ec2. The test did not fail. [04:53] It landed. [04:53] Then broke buildbot and every subsequent ec2 run. [04:54] StevenK: are you ready to exact sweet, sweet revenge for all the horrible things I've done to you? [04:54] Heh [04:54] jtv: i'm looking now, thanks [04:54] jtv: No, because I'm not yet in the same country as you ... [04:54] StevenK: you clearly have not learned to appreciate the full potential of the internet. [04:54] I can't kick people in the face over the Internet. [04:55] lifeless: Would you prefer a full revert, or https://code.launchpad.net/~wgrant/launchpad/fix-dsp-index/+merge/70249? [04:55] Since it can skip ec2. [04:56] wtf [04:56] robertc@lucid-test-lp:~/source/launchpad/lp-branches/working$ ls /etc/postgresql/ [04:56] 8.2 [04:56] Erm. [04:56] Fail [04:56] wgrant: That MP looks good to me [04:57] wgrant: is the other template thing fixed too already ? [04:57] Wasn't 8.2 removed just after hardy? [04:57] wgrant: no freaking idea [04:57] lifeless: Which? [04:57] The top OOPS from yesterday? [04:57] lsb_release -a [04:57] No LSB modules are available. [04:57] Distributor ID: Ubuntu [04:57] Description: Ubuntu 10.04 LTS [04:57] wgrant: yea [04:57] postgresql-8.2 | 8.2.7-1 | hardy/universe | source, amd64, i386 [04:57] StevenK: I still say you're not appreciating (1) the full potential of the internet or (2) the awesome violent scale of my physical reaction to face-kicking attempts. Hence my offer of safe retribution across the internet. [04:58] lifeless: Was fixed in r13560 [04:58] lifeless: So the fix is on prod already. [04:58] jtv: you certainly are very detailed. i'll check in with curtis tomorrow. thanks again. [04:58] wallyworld: s/details/pedantic/ [04:58] lol [04:58] jtv: How? [04:58] wallyworld: it's the risk of over-escaping date formatting that has me worried. [04:58] wgrant: ok, I'm fine with rollforward if we're confident of no other sites. [04:59] StevenK: by reviewing one of my branches! https://code.launchpad.net/~jtv/launchpad/bug-819674/+merge/70135 [04:59] Ah, that branch. [04:59] wgrant: we will want the original bug to be bad-commit flagged and this new one flagged rollback too though. [04:59] wgrant: I think. [04:59] lifeless: Yup. [04:59] jtv: yes, i think that's a fair question to ask [04:59] I am not comfortable doing so, since I don't like the idea of the publisher being SIGINT'd [05:00] i suspect it won't be an issue but still..... [05:01] StevenK/lifeless: Can I have a +1 on the MP? [05:01] wgrant: r=me [05:01] Thankyou sir. [05:02] StevenK: fair enough. It all ends up being nice & safe though. [05:02] I don't trust it either [05:02] How can you be *certain* that the publisher is safe to Ctrl-C [05:02] It's a can of worms [05:03] As long as the backup switch happens only around publish-distro and not process-accepted, it's not too bad,. [05:03] The worst it can do is delay. [05:04] Bleh, and my kill-cruft branch has failed due to soyuz-uploads [05:04] :( [05:05] RETRIBUTION [05:05] Feels good, doesn't it? [05:07] Anyway, if wgrant says it's not too bad, it's probably not too bad. :) I mapped out and simplified the whole directory dance when I re-did publish-ftpmaster, and now nothing risky gets done in the "real" dists directory until all the hard work is done. [05:08] jtv: I was talking about retribution for wgrant [05:08] Oh. [05:08] Just don't revert whatever process-accepted did, and you won't die. [05:11] Hmm. [05:11] So, confirmed, nothing else touches LP.cache in bad ways. [05:11] The rest of that rev may be good. [05:12] Then land your fix [05:12] IMO [05:12] It landed a while ago. [05:12] Then speed up buildbot [05:12] * wgrant points at lifeless and LXC. [05:17] looks like I'm ready to try parallel testing again, having beaten oneiric lxc into submission [05:18] ! \o/ [05:18] lifeless: I see the restart problem has been diagnosed. [05:18] wgrant: there are some terrible hacks under the hood, something about O makes them more visible. [05:19] wgrant: would you be able to review my branch perhaps, since StevenK is too chicken? [05:20] Maybe. [05:25] Maybe thanks. === jtv is now known as jtv-eat [05:27] jtv-eat: So this is just improving tests? [05:27] There seems to be no functional change. [05:32] wheee [05:33] localpackagediffs fail - generated two OOPSes :) [05:33] https://launchpad.net/ubuntu/oneiric/+localpackagediffs?field.name_filter=apt&field.package_type=ignored&field.package_type-empty-marker=1 [05:33] and hit refresh [05:33] Module lp.services.propertycache, line 116, in __get__ [05:33] value = self.populate(instance) [05:33] Module lp.registry.browser.distroseries, line 1081, in cached_differences [05:33] status = package_type_dsd_status[self.specified_package_type] [05:33] KeyError: u'ignored'
[05:35] Hmmm. [05:35] (apt is currently generating a diff) [05:35] That page does have a thread-safety issue, but it's behind a flag and shouldn't cause this. [05:35] It seems to only affect some appservers. [05:36] It renders every so often. [05:50] Project devel build #941: FAILURE in 6 hr 8 min: https://lpci.wedontsleep.org/job/devel/941/ [05:50] Bah, there is an import violation that requires a storm fix [05:59] Excellent. [05:59] All remaining QA is European. [06:00] allenap and danilos, right? [06:01] And rvba. === jtv-eat is now known as jtv [06:20] wgrant: yup, I just didn't realize it until I'd written the tests. [06:20] That explains my confusement. [06:20] And then the tests turned out still to be useful. [06:20] Damn, I thought I mentioned this in the cover letter. [06:21] jtv: Do you know about assertRaisesWithContent [06:21] ? [06:21] Yes [06:21] Seems like you could use that in two of those tests. [06:21] Probably makes more sense to use that. I'll fix it. [06:21] It cuts out about 5 lines from each. [06:23] Yes… it doesn't really matter that it's the same exception object, I guess. [06:24] No. But if you use getUniqueString you can be pretty sure. [06:24] Anyway, apart from that it looks fine. [06:24] More publisher tests are always good. [06:25] Thanks. [06:25] The change is pushing. [06:25] You did sort of mention that it was only test changes in the cover letter, but I sort of assumed that couldn't be the case so ignored it. [06:28] wgrant: meanwhile, I'm just about through the testing I can do for the new publish-ftpmaster, except I'm not sure it expedites security updates. Maybe there's more you can think of. [06:28] jtv: Not sure that it does, or suspicious that it might not? [06:29] Suspicious that it might not—except we didn't actually get around to publishing a security update, and I don't really know what to look for. [06:32] Then again, that may not be a blocker. Can't drag this out forever. [06:33] Well, you could just get two fresh uploads, copy one into security and one not, and check the mtime of the security pool and dists files are before the non-security pool and dists files. [06:35] Thanks.. [06:41] It's easy enough to check, and I can do it if you want. [06:44] wgrant: I'd be lying if I'd say I wouldn't want that — if only because you're so much more likely to spot any glaring mistakes than I am. :) [06:44] In other words, yes please! [06:45] (Please imagine better grammar when reading the above) [06:45] * wgrant enters the molasses. [06:49] StevenK: :( [06:49] StevenK: It wasn't entirely file deletions. [06:49] I am disappoint [06:49] Haha [06:53] jtv: You seem to have a diff in 10-sign-releases. [06:53] #!/bin/sh -e [06:53] [06:53] +exit 0 # XXX: DEBUG [06:53] + [06:56] wgrant: that's just because the gpg won't work on dogfood. [06:56] Without it, test runs will break. [06:56] Ah. [06:57] Hmm. Why is --post-rsync an option>? [06:58] I am loving the sane timestamps in this log. [07:04]