[00:14] <penguin42> I seem to be able to trigger a repeatable oops by attempting to change the package on bug 239542 to linux
[00:17] <wgrant> penguin42: What's the OOPS ID?
[00:18] <penguin42> OOPS-9944a90ab5ba3bb77a3c7f4a17c39512
[00:19] <penguin42> wgrant: That's the latest one, I triggered it a few times, 1st when I put in a comment at the same time, then narrowed it down to letting me do that, but not letting me change the package
[00:21] <wgrant> penguin42: Thanks, this is a recent regression
[00:21] <wgrant> Let's see
[00:22] <penguin42> wgrant: Thanks for looking; not too important for me - I was just randomly triaging old bugs
[00:22] <wgrant> There's no workaround right now.
[00:23] <penguin42> not a problem - thanks for looking
[00:24] <penguin42> on a different query; do you know what launchpad.net lets the google search see - it seems to see open bugs but it's rare to find a closed bug in a google search on site:launchpad.net - is that true?
[00:24] <wgrant> Google can see all bugs
[00:24] <penguin42> hmm ok, thanks
[05:26] <tjaalton> hey, getting OOPS-9879a35cd0349e25109da48cdb15dc8b when trying to edit bug 966744
[05:26] <tjaalton> same thing ~8h ago
[07:42] <FourDollars> Hi, totem 3.0.1-0ubuntu21 is missing from lp:ubuntu/precise/totem .
[07:45] <maxb> !info totem precise
[07:47] <FourDollars> maxb: Try `bzr branch lp:ubuntu/precise/totem`.
[08:27] <maxb> FourDollars: Strictly speaking the importer isn't within the remit of the Launchpad team - actually I'm not sure whose remit it is actually in these days, I'm just a community volunteer who happens to be significantly involved with it. Anyway, it seems like a previous import attempt died uncleanly. The importer would have got around to trying again eventually anyway, but I've requeued it as a priority now.
[08:28] <FourDollars> maxb: OK. Thank you.
[08:29] <FourDollars> maxb: Is there any link I can refer to?
[08:34] <maxb> FourDollars: http://package-import.ubuntu.com/ lists failures
[08:34] <FourDollars> maxb: Thanks a lot.
[08:34] <maxb> Except that won't help you right now, because the importer has just run and decided that totem was up to date
[08:37] <maxb> Something weird is going on
[08:39] <czajkowski> maxb: still?
[08:39] <czajkowski> :/
[08:39] <maxb> apparently
[08:39] <FourDollars> That is bad. :(
[08:39] <maxb> the importer seems confused about what is and isn't imported
[08:40] <FourDollars> The latest version on lp:ubuntu/precise/totem is not correct.
[08:42] <FourDollars> The latest version on lp:ubuntu/precise/totem is 3.3.4-0ubuntu1~precise1, but it should be 3.0.1-0ubuntu21.
[08:42] <maxb> what on earth?!
[08:43] <FourDollars> It might be why the importer failed.
[08:44] <wgrant> 'Deleted on 2012-01-10 by Colin Watson
[08:44] <wgrant> requested by jbicha; 3.3.4-0ubuntu1~precise1 uploaded to Ubuntu by mistake, but fortunately dep-waited on all architectures; removing so that we can try to upload an older source package'
[08:44] <wgrant> That's going to require manual recovery, I guess.
[08:45] <maxb> yup, just found that little gem in the publishing history
[08:47] <maxb> So, I need to nuke the {precise,quantal}{release,proposed} branches back to 3.0.1-0ubuntu12 and redo from there
[08:52] <maxb> Erm... where's launchpad gone?
[08:52]  * pmjdebruijn noticed that too
[08:52] <pmjdebruijn> seems to timeout
[08:52] <maxb> networking glitch?
[08:52] <maxb> back now
[08:52] <pmjdebruijn> oh indeed
[08:57] <maxb> FourDollars: ok, surgery completed, lets give the importer 10 minutes or so and see what happens
[08:57] <FourDollars> maxb: Awesome!
[09:20] <maxb> FourDollars: A couple of false starts in which I had to correct extra things, but a new import job has just started, and I think this one will be OK
[09:20] <FourDollars> maxb: OK.
[10:08] <maxb> FourDollars: totem all fixed now
[10:08] <FourDollars> maxb: Thank you very much. :D
[10:11] <FourDollars> maxb: lp:ubuntu/precise/totem is fixed, but lp:ubuntu/precise-proposed/totem is not yet.
[10:14] <dholbach> hiya
[10:14] <dholbach> https://code.launchpad.net/~ubuntu-branches/ubuntu/quantal/checkbox/quantal has ben "updating this branch" for ~2 days now - is this a known problem? :)
[10:15] <czajkowski> :/
[10:16] <czajkowski> wgrant: more strangness happening
[10:18] <wgrant> czajkowski: Could you convince Blue to investigate? I'm not really here atm
[10:19] <FourDollars> maxb: The history of lp:ubuntu/precise-proposed/totem is not sync to lp:ubuntu/precise/totem.
[10:19] <czajkowski> jelmer: mgz this is still going on since yesterday
[10:19] <czajkowski> mgz: were you looking into this issue already ?
[10:21] <mgz> czajkowski: may or may not be related, will have a look
[10:21] <czajkowski> please as this is kinda going on now for a few days and we could do with getting to the bottom of it
[10:23] <mgz> seems like we need a code change just to find out what the problem is, unfortunately
[10:27] <mgz> all I see in the logs is a success:
[10:27] <mgz> [2012-08-21 15:44:13,192: INFO/PoolWorker-1] Running <SCAN_BRANCH branch job (4344425) for ~ubuntu-branches/ubuntu/quantal/checkbox/quantal> (ID 14641498) in status Waiting
[10:28] <mgz> ...
[10:28] <mgz> [2012-08-21 15:44:13,452: INFO/PoolWorker-1] Adding 1 new revisions.
[10:28] <mgz> [2012-08-21 15:44:14,087: INFO/PoolWorker-2] Retrieving ancestry from database.
[10:28] <mgz> [2012-08-21 15:44:14,130: INFO/MainProcess] Task lp.services.job.celeryjob.CeleryRunJobIgnoreResult[BranchScanJob_14641493_f77b9db7-e8cd-484b-a568-8963bc25cd0d] succeeded in 10.7140600681s: None
[10:28] <mgz> the upstream project oopsed though:
[10:28] <mgz> [2012-08-21 15:42:43,212: INFO/PoolWorker-2] Running <SCAN_BRANCH branch job (4344413) for ~checkbox-dev/checkbox/trunk> (ID 14641476) in status Waiting
[10:28] <mgz> ...
[10:28] <wgrant> mgz: Different branches
[10:28] <wgrant> 20:28:11 < mgz> [2012-08-21 15:44:13,452: INFO/PoolWorker-1] Adding 1 new revisions.
[10:28] <wgrant> 20:28:11 < mgz> [2012-08-21 15:44:14,087: INFO/PoolWorker-2] Retrieving ancestry from database.
[10:29] <wgrant> Note the differing worker number
[10:29] <wgrant> And the job ID
[10:29] <mgz> arg, the log is not sequential?
[10:29] <wgrant> 14641498 vs 14641493
[10:29] <wgrant> mgz: There are several workers
[10:29] <wgrant> To avoid starvation
[10:29] <mgz> then I've been misreading it all along
[10:29] <mgz> there can be workers and still have a sane log...
[10:29] <wgrant> You'd have to have several separate logs
[10:30] <mgz> or do per-job rather than per-line logging
[10:30] <wgrant> Which makes them useless
[10:30] <mgz> so, upstream:
[10:30] <mgz> [2012-08-21 15:42:46,483: INFO/PoolWorker-2] Job resulted in OOPS: OOPS-678a3ab26de51d9afedf052cad9532cf
[10:30] <wgrant> Since jobs can run for minutes
[10:31] <mgz> ubuntu branch:
[10:31] <mgz> [2012-08-21 15:44:14,507: INFO/PoolWorker-1] Job resulted in OOPS: OOPS-654d9bdf86466f489d9d050751981e0a
[10:32] <mgz> which is all the same thing
[10:32] <FourDollars> Could anyone help to rebase lp:ubuntu/precise-proposed/totem on lp:ubuntu/precise/totem ?
[10:33] <mgz> dholbach: subscribe to bug 1039638
[10:33] <dholbach> ah, thanks
[10:33] <wgrant> mgz: Have you identified when these OOPSes started?
[10:35] <mgz> only have logs back to the 19th, which have oopses, but those aren't in the oops db
[10:36] <wgrant> I believe it started late on the 13th
[10:37] <mgz> are the older logs around somewhere, or do you just keep copies locally for occassions like this? :)
[10:38] <wgrant> We probably do have older copies of the logs available due to excellent timing, but I haven't looked yet
[10:38] <wgrant> My estimate of late on the 13th comes from the daily OOPS reports
[10:38] <mgz> ah, those are actually useful for something? :)
[10:39] <wgrant> Yes, you should be reading the first few daily and fixing them :)
[10:39] <wgrant> Ah, no, we don't have logs from further back
[10:40] <maxb> FourDollars: the importer has admittedly structured the history a bit weirdly, but it is not wrong
[10:41] <FourDollars> maxb: So... is it OK?
[10:41] <mgz> so, it would be helpful if these older oopses were actually in existence...
[10:41] <wgrant> They expire after a week
[10:41] <mgz> because the error is generic and in reports from way back, but could be from all over the codebase
[10:41] <wgrant> If they're not referenced
[10:42] <mgz> and the summary email doesn't count as a reference...
[10:42] <mgz> headdeskish
[10:42] <wgrant> No, since then we'd be keeping thousands of OOPSes just because they happened
[10:42] <wgrant> daily
[10:44] <mgz> I bet they compress pretty well, and could fit quite a few on a fifty quid 2TB drive
[10:45] <wgrant> The threshold used to be higher, I'm not sure why it's so low now
[10:45] <wgrant> I'm also not sure why our logrotate settings are so terrible
[10:46] <mgz> so, I said I'd finish some other code today
[10:46] <wgrant> There's also that critical branch-rewrite issue which needs to be fixed before Monday
[10:46] <mgz> but when I get that done I'll look at resolving the transaction thing
[10:47] <mgz> what's happening on Monday, apart from my day off?
[10:47] <czajkowski> bank holiday
[10:47] <maxb> FourDollars: I think it's fine to leave as is
[10:48] <wgrant> We've had fastdowntimes blocked for more than a week already, we'll hopefully do the next on Monday or Tuesday
[10:48] <mgz> ah, hadn't gathered that was the cause
[10:48] <FourDollars> maxb: I have a commit based on lp:ubuntu/precise/totem. How do I move to lp:ubuntu/precise-proposed/totem ? Is `bzr merge lp:ubuntu/precise-proposed/totem` or something else?
[10:49] <mgz> so, none of us three are currrently working on that, but we probably want aaron awake anyway as it's likely fallout from his changes
[10:50] <maxb> FourDollars: Based on which incarnation of lp:ubuntu/precise/totem? Before or after I did stuff to it?
[10:50] <FourDollars> maxb: After you did stuff to it.
[10:53] <tjaalton> still getting the oops with bug 966744
[10:53] <maxb> FourDollars: Yes, bzr merge will work fine
[10:53] <tjaalton> OOPS-c6c07586e94dc5cf49d1eb2b3a2e72fa
[10:55] <FourDollars> maxb: Thank you.
[15:47] <joey> sinzui: ouch that hurt. I needed that for my security scans
[15:47] <sinzui> osrry
[15:47] <sinzui> sporry
[15:47] <joey> lol
[15:47]  * sinzui gives up
[15:47] <sinzui> You will be back in just a minute
[15:47] <joey> sinzui: before you do, let me pm you