[03:09] <StevenK> wgrant: Hmm, I thought we already had a script to clean up failed translations imports? Ref bug 576822
[03:09] <_mup_> Bug #576822: Add an option to clean up after a failed run of copy-translations-from-parent.py <lp-translations> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/576822 >
[03:12] <wgrant> StevenK: I think there was a one-off script for precise
[03:34] <StevenK> 450282.22s  OOPS-674539797ca55814a0c0894007b34498  Unknown
[03:34] <StevenK> BLINK
[03:35] <StevenK> 5 days, 5 hours
[03:35] <lifeless> thats some request
[03:36] <wgrant> oooooh
[03:36] <wgrant> checkwatches unhung
[03:36] <wgrant> That explains a lot of the other OOPSes
[03:37] <StevenK> Right
[03:37] <StevenK> I just loaded that OOPS
[03:37] <wgrant> So
[03:37] <wgrant> Next step
[03:38] <wgrant> Work out what bugwatch 57103 is
[03:38] <wgrant> bugzilla.filesystems.org #632
[03:39] <wgrant> The host responds, but doesn't look much like a bugzilla at all really
[03:39] <lifeless> honeypot?
[03:39] <wgrant> Ah, on https there's a bugzilla with a bad cert
[03:40] <StevenK> But 5 days 5 hours? Surely we should timeout much quicker
[03:40] <wgrant> Certainly
[03:40] <wgrant> But should and do are different
[03:40] <wgrant> Particularly when it comes to checkwatches
[03:40] <StevenK> Indeed
[03:41] <wgrant> That's the only bugwatch from that tracker
[03:45] <StevenK> wgrant: You've commented on bug 758258 about a migration
[03:45] <_mup_> Bug #758258: buildfarmjob schema is inefficient for reporting <critical-analysis> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/758258 >
[03:46] <wgrant> That's a reasonably big effort still, so it remains suspended until we run out of low-hanging fruit
[03:46]  * StevenK has been looking to no avail for low-hanging fruit
[03:47] <wgrant> There's a lot of fairly easy timeouts and OOPSes left
[03:48] <wgrant> 2012-10-17 10:39:23 INFO    Updating 1 watches for 1 bugs on https://bugzilla.filesystems.org
[03:48] <wgrant> 2012-10-22 15:44:03 INFO    An exception was raised when updating https://bugzilla.filesystems.org/ (OOPS-674539797ca55814a0c0894007b34498)
[03:55] <wgrant> 2012-09-24 07:11:24 INFO    Updating 1 watches for 1 bugs on https://bugzilla.filesystems.org
[03:56] <wgrant> That same bugtracker caused the earlier hang
[04:04] <wgrant> 2012-10-23 04:03:58 INFO    Updating 1 watches for 1 bugs on https://bugzilla.filesystems.org
[04:04] <wgrant> and now we wait..
[04:04] <wgrant> :(
[04:04] <wgrant> 2012-10-23 04:04:10 INFO    Error updating https://bugzilla.filesystems.org/: Failed to parse XML description for https://bugzilla.filesystems.org bugs [u'632']: mismatched tag: line 101, column 4
[04:04] <wgrant> It worked
[04:04] <StevenK> What did you changed?
[04:04] <wgrant> This was locally
[04:05] <StevenK> Sure, but did you change anything
[04:05] <wgrant> No
[04:05] <StevenK> Ah
[04:06] <StevenK> Maybe bugzilla.filesystems.org is blocking loganberry?
[04:08] <wgrant> It even works via squid.internal
[04:08] <StevenK> Bleh, are there no webservice tests for DSDs?
[04:09] <wgrant> Not even under registry?
[04:09] <StevenK> Not that I can see
[04:09] <StevenK> I suspect it's a lazr.restful bug anyway
[04:33] <StevenK> wgrant: I am disappoint. You didn't rip out contrib.glock in disgust while I was on holidays.
[04:39] <wgrant> Indeed.
[04:43] <StevenK> wallyworld_, wgrant: https://code.launchpad.net/~stevenk/lazr.restful/lazr.restful-choice-marshaller-none/+merge/130932
[04:47] <wallyworld_> StevenK: looks ok, but i don't know enough about the semantics of the code to be able to offer a considered opinion
[04:49] <wgrant> StevenK: Is this circumstance actually valid?
[04:50] <StevenK> wgrant: Bug 924291
[04:50] <_mup_> Bug #924291: AttributeError: 'NoneType' object has no attribute 'title' on package differences page <derivation> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/924291 >
[04:51] <wgrant> StevenK: It can happen, but is it valid?
[04:51] <StevenK> parent_package_diff_status is a Choice of PackageDiffStatus, but the property returns None if the DSD's parent_package_diff is None
[04:52] <wgrant> I thought it would usually escape earlier if the value was None
[04:53] <StevenK> wgrant: Like I say in the description, unmarshall_to_closeup will return the entire vocab in any case.
[04:53] <StevenK> I can change LP to just return PackageDiffStatus.PENDING and never-mind this lazr.restful branch
[04:53] <wgrant> Ah, I see now reading the code that your argument makes sense
[04:55] <wgrant> r=me
[04:57] <StevenK> wgrant: Thanks
[06:00] <wgrant> lifeless: Was lazr.postgresql intended for just schema migration stuff? There's some long lock/transaction tools that I need to find somewhere to put
[06:07] <lifeless> wgrant: it was intended to pull out everything postgresql specific that doesn't rightfully belong in storm.
[06:07] <lifeless> wgrant: or in LP itself.
[06:07] <wgrant> Right
[06:07] <wgrant> Thanks
[06:07] <lifeless> and isn't pgbouncer specific
[06:07] <wgrant> That's what I thought, but wanted to check
[06:07] <lifeless> :)
[09:15] <cjwatson> wgrant: So, early this month you fixed a packagediff generation bug that happened when we deleted things from -proposed a bit too early and then tried to copy the orphaned binary resulting from a build
[09:15] <cjwatson> wgrant: packagediff generation is pretty late in the copy process; does that mean that this kind of situation should now completely work without us having to do weird copy dances?
[09:16] <cjwatson> Archive.copyPackage and PCJ.attemptCopy don't seem to care about the publication status of the origin SPPH
[09:17] <cjwatson> So they ought to be able to copy from a deleted SPPH that's had some binaries added to it later ... I think
[09:19] <cjwatson> wgrant: Also, did you forget to close bugs following the recent NDT, possibly?  I'd have expected e.g. bug 1068616 to be closed
[09:19] <_mup_> Bug #1068616: Archive.copyPackages hardcodes RELEASE pocket <package-copies> <qa-ok> <releases> <Launchpad itself:Fix Committed by cjwatson> < https://launchpad.net/bugs/1068616 >
[09:21] <wgrant> cjwatson: Um, I was probably accidentally on the critical list, so missed non-criticals
[09:21] <wgrant> As for the orphaned binary copy, yes, I believe it should all work now
[09:21] <wgrant> But there's a catch
[09:21] <cjwatson> Also bug 1014641 (critical)
[09:21] <wgrant> There's no way to specify the source pocket today
[09:21] <_mup_> Bug #1014641: +vouchers times out <bad-commit-16172> <qa-ok> <salesforce> <timeout> <Launchpad itself:Fix Committed by wallyworld> < https://launchpad.net/bugs/1014641 >
[09:21] <wgrant> cjwatson: That's not deployed yet
[09:21] <cjwatson> Oh, sorry, that's still on the report
[09:21] <cjwatson> Confusing, it was in the LPS dump
[09:22] <wgrant> cjwatson: Ah, yeah, it was landed and then reverted, and then landed again
[09:22] <cjwatson> Oh, the ambiguous pocket thing - right, that sounds pretty easy to fix
[09:22] <wgrant> The first two bits are deployed
[09:22] <wgrant> Indeed
[09:22] <wgrant> But reviving orphaned binaries should work fine now, if you copy back to -proposed first
[09:22] <wgrant> Or if you fix the API to allow an explicit pocket
[09:22] <cjwatson> Right, so I'm conscious that there are some situations where britney is going to create this kind of thing
[09:23] <cjwatson> e.g. 1.0-1 ftbfs on powerpc, 1.0-2 initially does as well, britney sees it's no worse than before and copies, then somebody retries 1.0-2/powerpc and it works
[09:23] <wgrant> Note that we haven't tested it, but packagediff generation is so close to the end of the process that it surely has to
[09:23] <wgrant> work, that is
[09:24] <cjwatson> I was contemplating beefing up test_incremental_binary_copies a bit
[09:24] <wgrant> That might be handy
[09:24] <cjwatson> It tests about 80% of this
[09:24] <wgrant> While you're there it may also be worth sorting out the random override issue
[09:24] <wgrant> Since that's a matter of a .order_by(BPPH.id) and a test
[09:24] <cjwatson> Oh god this is a horrible ancestry nightmare isn't it?
[09:24] <cjwatson> No?
[09:24] <wgrant> I don't think so
[09:25] <wgrant> It's not a total nightmare
[09:25] <cjwatson> Last night I found vague allusions to a lot of this stuff in IRC logs from earlier in the month, but there wasn't enough to track them down to things I could fix on the spot
[09:25] <wgrant> I don't think it'll do the override dance for an intra-archive copy, will it?
[09:25] <wgrant> It'll just use whatever getBuiltBinaries returns
[09:26] <wgrant> The problem with that method is that it doesn't guarantee which publication will be returned if there are multiple BPPHs for a BPR
[09:26] <wgrant> So, if you try to copy an upload that's had its overrides changed, you're going to get a random assortment of overrides
[09:26] <cjwatson> .order_by doesn't sound like a complete fix, then
[09:27] <cjwatson> There won't be a BPPH until the build happens; and we probably want to pick up the overrides from some other architecture that's already built
[09:27] <wgrant> getBuiltBinaries iterates through the entire set of binary publications, and IIRC uses the first one it finds for each BPR
[09:27] <wgrant> So an order_by should fix it
[09:27] <wgrant> Hmm?
[09:27] <wgrant> The build will upload and do normal ancestry calculation, which is correct.
[09:27] <wgrant> It's only copies that use getBuiltBinaries' arbitrary chooser
[09:27] <cjwatson> Otherwise you change 1.0-2 to component: main for all the architectures that already exist, and then when 1.0-2/powerpc appears it'll flip back to whatever 1.0-1/powerpc was?
[09:28] <wgrant> Well, yes, but that's an ancient bug
[09:28] <cjwatson> Oh, right, I understand now
[09:28] <wgrant> From early 2006
[09:28] <wgrant> Which we've lived with for nearly 7 years :)
[09:28] <wgrant> https://bugs.launchpad.net/launchpad/+bug/180218
[09:28] <_mup_> Bug #180218: override mismatch race needs to be fixed <boobytrap> <lp-soyuz> <soyuz-publish> <soyuz-upload> <Launchpad itself:Triaged> < https://launchpad.net/bugs/180218 >
[09:28] <wgrant> There was an older one, once upon a time
[09:28] <cjwatson> Yeah, not fixing that this week
[09:28] <wgrant> But that's the current one about the situation you describe
[09:28] <wgrant> Mine is perhaps more insidious and easier to fix
[09:29] <cjwatson> Wrong overrides are a relatively minor problem in the development series; we have a report for them
[09:29] <cjwatson> (i.e. override mismatches between architectures)
[09:29] <wgrant> Sure
[09:29] <cjwatson> And it's empty most of the time
[09:32] <cjwatson> But yeah, I'm a small handful of LP commits away from having britney allegedly fully working, so just trying to think of ways it might unintentionally explodify the archive in advance ...
[09:33] <cjwatson> I've been able to run it over nearly-trivial cases in raring-proposed in production and have it copy stuff successfully
[09:33] <wgrant> As long as we have robust binary revival copies it should all be recoverable
[10:25] <cjwatson> wgrant: could you have a look at https://code.launchpad.net/~cjwatson/launchpad/db-redirect-release-uploads/+merge/130857 ?
[13:43] <wallyworld_> sinzui: lines 236-241 of the diff add the extra Related Projects link to the person index page
[13:45] <sinzui> i suck. sorry wallyworld_
[13:46] <wallyworld_> np, merely an oversight
[13:46] <wallyworld_> easy to miss
[13:46] <wallyworld_> the rest we can discuss tomorrow, you raise good points, i wasn't sure what to remove and what to keep
[13:47]  * wallyworld_ off to bed
[13:48] <sinzui> wallyworld_, oversight? I knew exactly where the change had to be, and I read the diff, and I looked over the diff for the very file you did change, then told you you needed to make the change that is clearly in the diff
[13:48] <wallyworld_> don't worry, i'm not :-)
[15:33] <cjwatson> wgrant: Can you see why debootstrap/1.0.42ubuntu1 and apparmor/2.8.0-0ubuntu6 didn't get announcement mails when they were accepted into raring-proposed?  I didn't think PROPOSED was special here.  I think I used the API to accept the former; not sure how the latter was done.
[16:13] <czajkowski> cr3: your ml request should get done today
[16:25] <cr3> czajkowski: sweet, thanks so much!
[16:26] <czajkowski> cr3: well we're still checking it but updates will be on the question
[17:45] <cjwatson> wgrant: So, this random overrides thing - I'm having a hard time seeing how to trigger it, since getBuiltBinaries requires the BPPHs it returns to match the distroseries and pocket of the SPPH.  How could there ever be more than one per BPN/architecturetag?
[17:47] <cjwatson> Oh, changeOverride would create a new BPPH, wouldn't it, duh
[21:54] <wgrant> cjwatson: It doesn't require that they're published, and changeOverride creates a new BPPH, indeed
[21:56] <wgrant> cjwatson: Did you find out what happened to the debootstrap and apparmor announcements?
[21:56] <cjwatson> No
[21:56] <wgrant> k, will investigate
[21:56] <cjwatson> Couldn't figure out where useful logs would be
[21:59] <wgrant> You tell funny jokes
[22:51] <wallyworld_> wgrant: StevenK: sinzui: http://askubuntu.com/questions/202677/nvidia-driver-doesnt-work-in-12-10
[23:25] <sinzui> wgrant, I forgot to ask you to take a look at Colin's db branch in reviews
[23:31] <wgrant> sinzui: It's already on my list after he pinged me about it last night :)