=== Guest28960 is now known as NCommander
=== NCommander is now known as Guest30460
=== mlankhor1t is now known as mlankhorst
=== tkamppeter__ is now known as tkamppeter
=== maclin__ is now known as maclin
[13:15] <sergiusens> can someone please look at https://bugs.launchpad.net/ubuntu/+bug/1290351
[13:15] <ubot2`> Launchpad bug 1290351 in Ubuntu "[FFe] Sync golang-go-xdg 0~bzr20140219-1 (universe) from Debian unstable (main)" [Wishlist,New]
[13:20] <Laney> sergiusens: you don't need to immediately ping ...
[13:21] <sergiusens> Laney, right; sorry 'bout that
[13:22] <Laney> I'll spend some time on the queue later on today
[13:22] <Laney> (not to say that someone else can't process things)
[14:09] <ogra_> stgraber, so i had some massive probs today to promote images ...
[14:09] <stgraber> ogra_: ah, what kind?
[14:10] <ogra_> stgraber, it took endlessly long .... and it seems it deleted the tarball for some arches during the process
[14:13] <stgraber> ogra_: hmm, I really need to figure out a better way to deal with that case...
[14:13] <stgraber> ogra_: what happened is that as you didn't promote an image in quite a while, the 194 image in trusty-proposed was expired a while back including all deltas it had
[14:14] <stgraber> so when you did the copy earlier, there wasn't any of the 194 => 226 pre-generated deltas which make the copy very quick and they all had to be regenerated during the copy
[14:15] <stgraber> trusty-proposed stores the last 20 images, 226-194 = 32, so that indeed seems to be the problem
[14:16] <ogra_> stgraber, sorry, doorbell
[14:16] <stgraber> (back when we set those values, it seemed unimaginable that it'd take us more than 20 tries before we get a promotable image, guess we were wrong...)
[14:16] <ogra_> stgraber, http://paste.ubuntu.com/7067819/
[14:17] <ogra_> thats my full terminal log from trying to promote all images
[14:20] <stgraber> hmm, still looks like import-images running cleanup_tree under our feet, though the lock should prevent that from happening... I guess I'll have to do some tests of the locking mechanism, something seems wrong there
[14:23] <ogra_> yeah, i could see the tarballs actually show up and vanish again on system-image.ubuntu-com in the browser
[14:23] <ogra_> i show down the cronjob in the end
[14:24] <ogra_> stgraber, i messed up the flo release by missing -k to copy-images, can we fix that ?
[14:25] <stgraber> ogra_: yeah, give me a minute I'll fix it
[14:25] <ogra_> no hurry :)
[14:30] <stgraber> ogra_: should be good now
[14:31] <stgraber> ogra_: http://paste.ubuntu.com/7067901/
[14:33] <ogra_> stgraber, gracias !
=== apachelogger_ is now known as apachelogger
[16:21] <bdmurray> Can I get the release of ubuntu-release-upgrader in saucy-proposed fast tracked to -updates?
=== PaulW2U is now known as G4MBY
[16:24] <infinity> bdmurray: Seems likely.
[16:27] <infinity> bdmurray: Done.
[16:27] <bdmurray> infinity: thanks!
[16:28] <infinity> bdmurray: Is the Ubuntu task on #1289604 valid?
[16:28] <infinity> bdmurray: If not, maybe that should be marked Invalid instead of In Progress...
[16:29] <infinity> bdmurray: Or, if this needs fixing in trusty, please do. :P
[16:30] <bdmurray> infinity: I did fix it but I'm not sure it really needs it, since we can convert the release upgrader to use python3 when development on U starts.
[16:31] <infinity> bdmurray: But isn't this about new release-upgraders needing to run on old systems?
[16:31] <infinity> bdmurray: And the trusty one needs to run on precise...
[16:31] <infinity> Or, oh.  I guess the trusty tarball will work on precise, but doesn't need the dep.
[16:32] <bdmurray> right because the tarball doesn't install deps
[16:32] <infinity> (But then, yeah, it needs python2.7 because it's not python3, because it needs to run on precise...)
[16:32] <infinity> bdmurray: So, is this *only* about the tarball contents, and not at all about any tools that might be run on a trusty system?
[16:33] <bdmurray> No, this is about python-apport not being installed on saucy and the special traceback handler in the upgrader not working because it can't import python-apport code.
[16:34] <infinity> bdmurray: Right, but will that be invoked at py3 on a trusty system then?
[16:34] <infinity> s/at/as/
[16:35] <infinity> I guess the release-upgrader deps on trusty certainly imply that.
[16:35] <bdmurray> When a Trusty system tries to upgrade to U, then it could and should be invoked as python3 provided we change the U tarball creator.
[16:35] <bdmurray> So that's why I think the Trusty change to depend on python-apport is unnecessary.
[16:36] <infinity> Right, unnecessary, and bloaty.
[16:36] <bdmurray> Okay, I'll remove that then.
[16:36] <infinity> I had a buildd install a bunch of python2 stuff on upgrade the other day, I guess that's why?
[16:37] <bdmurray> If the other day was Saturday or Sunday, probably.
[16:37] <infinity> It was yesterday, yeah.
[16:37] <infinity> Brought in python-apport, launchpadlib, etc.
[16:37] <bdmurray> yep
[16:38] <infinity> So, yeah, if you want to maybe prep a branch in advance of "the first U upload of release-upgrader" that flips all the right switches to py3 tarballs, and keep that rebased until T releases, then we can stuff it in as soon as we open, and everyone wins.
[16:38] <infinity> Since some strange people always pester us until do-release-upgrade works, even when no packages have changed yet. :P
[16:39] <bdmurray> heh, did you you see my email about the pgp signature for the release upgrade tarball?
[16:39] <infinity> I did, and it confused me.
[16:39] <infinity> That should be atomic, from a mirror's POV, why do you care if the times are mismatched?
[16:40] <infinity> (ie: it hits disk, some other publisher stuff happens, then it's signed, but at no point is that intermediate step mirrored to the world)
[16:40] <bdmurray> There is a period of time where there is an upgrader and no signature, then upgrades fail.
[16:40] <infinity> If the timestamp mismatch is an issue, we can fudge it with touch, but I don't see why it would be.
[16:40] <infinity> bdmurray: That's the part I don't believe.
[16:41] <infinity> bdmurray: This all happens in dists.in-progress during the publisher, and is atomicly copied to dists before we push mirrors.  Unless we have a bug where we're signing in the NEXT publisher run, but that seems unlikely.
[16:41] <bdmurray> infinity: I'm sorry you don't believe that upgrades fail or that there is a lag?
[16:42] <infinity> bdmurray: I don't believe that upgrades fail because of the lag, because the lag shouldn't be visible to anyone outside the datacentre.
[16:43] <bdmurray> infinity: okay, I was trying to upgrade on Friday and it failed because the .gpg file was missing. If I make this change to the upgrader and upload it soon, you should see it.
[16:43] <bdmurray> It being the issue.
[16:44] <infinity> bdmurray: I'm not discounting that upgrades fail, nor am I implying there's no lag between writing to disk and signing, I'm saying that I'd like some evidence that you actually see the disparity on a mirror, except if you literally catch them mid-rsync, and they don't do atomic dists mirroring.
[16:44] <infinity> (And people who don't do atomic dists mirroring are people we can't fix)
[16:45] <cjwatson> Also, the signing is done strictly before the mirror push
[16:45] <infinity> cjwatson: Yeah, I covered that.
[16:45] <cjwatson> Oh yes
[16:54] <bdmurray> I'm still verifying but I don't think mirrors are used to get the release upgrader.
[16:55] <cjwatson> opaque proxies are possible though
[16:57] <infinity> bdmurray: Everything is a mirror, but I assume you mean "no mirrors other than archive/ports".
[16:59] <bdmurray> infinity: yes, it just uses archive.ubuntu.com
[16:59] <infinity> bdmurray: Right, so those mirrors should be completely "safe" from a mirrors-not-doing-stupid-things perspective.
[16:59] <infinity> bdmurray: They should never get one half of dists without the other, etc, unless something goes wildly wrong.
[17:00] <bdmurray> infinity: well, I'll do this upload and ping you if I see the disparity.
[17:21] <bdmurray> infinity: if you look here you'll see it http://archive.ubuntu.com/ubuntu/dists/trusty-proposed/main/dist-upgrader-all/current/
[17:22] <infinity> Hrm.
[17:22] <infinity> Maybe that's being published early in the security-only run, but not signed until the full run.
[17:22] <ogra_> stgraber, can we bump the amount of kept images on s-i to 50 (for -proposed) ? if we get into situations like today and people try to bi-sect back to the last promoted image 20 is not enough
[17:24] <infinity> Yup, that's what's happening.  Erk.
[17:24] <ogra_> (we do two images per day, keeping 20 means 10 days only)
[17:24] <cjwatson> I guess it's published by process-accepted at the start
[17:24] <cjwatson> Because custom uploads
[17:24] <infinity> cjwatson: Yeahp, that's what the log implies.
[17:25] <infinity> cjwatson: Maybe we need to build a signing run into the security-only pass.
[17:25] <cjwatson> Seems odd for security-only to run finalize but not publish-distro though
[17:25] <stgraber> ogra_: done. I was vaguely concerned by the disk usage but we're only at around 30GB now out of the 160GB we have on the target server, so that should be fine.
[17:25] <ogra_> yeah
[17:26] <infinity> cjwatson: Maybe it only runs publish-distro if *-security was actually dirty.  Which it usually isn'y.
[17:26] <ogra_> stgraber, theoretically having some function that simply keeps all proposed builds between two promoted images would be the best
[17:26] <cjwatson> infinity: I'd just got there, yes
[17:26] <cjwatson> So it needs to run publish-distro either way, I guess
[17:27] <stgraber> ogra_: yeah, that'd be the proper fix, I'd have to add some more channel relationship knowledge into the expiring code, should be doable though
[17:27] <infinity> cjwatson: Or, more sanely, we don't need the security-only run at all if *-security isn't dirty (do we have a way to check that at the beginning and just not run?)
[17:27] <ogra_> right, just sounds like a bunch of code
=== Ursinha is now known as Ursinha-afk
[17:31] <cjwatson> infinity: It might be least invasive to have the various publish methods return a bool to say whether they did anything
[17:32] <cjwatson> um, except that sometimes it'll do nothing else depending on options - not sure I have time/brainspace to work this out right now
[17:33] <cjwatson> lib/lp/archivepublisher/scripts/publish_ftpmaster.py anyway, it's not horribly complicated
[17:42] <infinity> cjwatson: I think I'll toss to cprov and wgrant.
[17:43]  * cjwatson nods
=== Ursinha-afk is now known as Ursinha
[18:03] <bdmurray> slangasek, infinity: could one of you review that?
[18:17] <infinity> bdmurray: Is that byte-for-byte the same patch I just released to saucy?
[18:18] <bdmurray> infinity: aside of the fact that it is the apport hook for update-manager due to the package split yeah.
[18:18] <infinity> Oh, with s/ubuntu-release-upgrader/update-manager/
[18:19] <infinity> bdmurray: Let me double-check the sameness of it, and wave it through.
[18:20] <infinity> bdmurray: No "import re", was it already there in the precise version?
[18:21] <bdmurray> infinity: yeah to get some aptdaemon log stuff
[18:22] <infinity> Ahh, indeed it is.
[18:23] <infinity> bdmurray: Accepted.
[18:23]  * infinity goes to hunt for... Whatever meal this is.
[18:25] <slangasek> I hope it's not fourthmeal
[18:27] <infinity> slangasek: I run all my meals in OF.
[18:28] <slangasek> twitch
=== jackson is now known as Guest57400