[00:01] <lamont> slangasek: they do.
[00:01] <slangasek> yeah
[00:01] <lamont> seems the mirrors get busy around release time
[00:01] <lamont> so we quit using them. :-(
[00:01] <slangasek> so I need some help unwinding this mess quickly
[00:02] <slangasek> but first it seems I need to do another upload of initscripts to fix this bug, which I didn't even notice because it's in a binary package I wasn't looking at :/
[00:02] <lamont> sigh
[00:02] <lamont> so you want the world on manual, for starters?
[00:02] <slangasek> lamont: prolly
[00:02] <lamont> set-builder --all --manual
[00:02]  * lamont hugs the api
[00:03] <lamont> afk for about 5.  OTOH, you're fortunate I was idling
[00:06] <lamont> back
[00:07]  * slangasek frowns.  Not sure how this worked *before*; no /usr/share/sysvinit/inittab in the old package either
[00:08] <lamont> interesting
[00:08] <lamont> also, have we fixed the libc6-breaks-debootstrap-with-no-/var/run bug yet?
[00:08] <slangasek> that's exactly what I was in the middle of fixing
[00:09] <slangasek> aha, got it - something changed that caused debian/postinst to be picked up into the sysvinit-utils package where it hadn't been before
[00:09] <slangasek> it'll be the fact that upstream switched to debconf, sysvinit used to be the first package in debian/control, and we nuke that \o/
[00:12] <slangasek> lamont: sysvinit 2.88dsf-13.10ubuntu3 is the package we need to get build to unbreak the chroots; and sysvinit-utils needs to be put on hold in each of them to get it built
[00:13] <lamont> slangasek: all 4 architectures, eh?
[00:14] <lamont> do we want to have this discussion in #u-d, or drag infinity here?
[00:14] <slangasek> either way :)
[00:14] <slangasek> yes, all 4 archs
[00:19] <lamont> slangasek: I'm understanding you to say you want the chroots with the only change being that sysvinit-utils is on hold, yes?
[00:19] <slangasek> lamont: yes
[00:20] <slangasek> I trust that the apt config will dtrt with other packages not upgradable due to unsatisfiable versioned dependencies
[00:20] <slangasek> (the build-dependencies will all be installable)
[00:22]  * lamont taps his foot at annonaceae
[00:27] <lamont> slangasek: I expect that tomorrow I'll freshen the oneiric chroots, once you've fixed debootstrapping
[00:28] <lamont> ditto for natty, without the prereq
[00:28] <slangasek> sounds good :)
[00:35] <lamont> did you already score sysvinit through the roof?  (as in are you ready for a build?
[00:35] <lamont> tarballs are uploading now
[00:37]  * lamont goes on a scorathon
[00:37] <lamont> kicking builds off
[00:38] <infinity> lamont: Okay.
[00:39] <lamont> 4 builds running, world on manual again (roseapple allspice adare araceae)
[00:39] <infinity> lamont: If you're doing the hacked-up chroots, I can return the status quo when you're off.
[00:39] <lamont> switching the world is one command
[00:39] <lamont> set-builder --all --auto :)
[00:39] <infinity> Erm.
[00:39] <infinity> I meant the chroots, not the buildds.
[00:39] <infinity> But sure.
[00:39] <lamont> I want to wait and make sure the builds finish before I restore the chroot tarball
[00:40] <lamont> Uploading build on roseapple
[00:40] <lamont> \o/
[00:40] <wgrant> (#launchpad-ops would appreciate notification about stuff like this, FWIW)
[00:40] <lamont> wgrant: sorry
[00:41] <infinity> wgrant: Bah. ;)
[00:41] <lamont> there
[00:41] <wgrant> Do we have an ETA?
[00:41] <wgrant> We have non-Platform users :)
[00:41] <infinity> Not until we crank a manual publisher run.
[00:41] <lamont> wgrant: once the freshly built packages are published
[00:41] <wgrant> We really need a better way to do this.
[00:42] <lamont> chroots are now correct (old)
[00:42] <lamont> wgrant: a way to tell lp-buildd to not dist-upgrade the chroot would be (1) a start, and (2) violate a few holies
[00:42] <infinity> slangasek: You want to turn the publisher handle, or shall I?
[00:43] <infinity> lamont: You could mitigate the ickiness of that by pushing chroots that are fully up-to-date except for the package you want on hold, but I agree, it's just a different form of ugly.
[00:43] <lamont> wgrant: all of the virtual builders are back to auto
[00:43] <lamont> infinity: it helps if the release is debootstrappable
[00:43] <wgrant> lamont: Thanks!
[00:43] <lamont> since I started rolling chroot tarballs from whole cloth for repeatability
[00:44] <infinity> lamont: Only because you insist on building your chroots from scratch every time. ;)
[00:44] <lamont> infinity: there was lots of cruft in them when I shot them in the head
[00:44] <infinity> lamont: (I agree it's a decent sanity check, but not quired every day either)
[00:44] <infinity> required*
[00:44] <lamont> wgrant: of course, anyone building oneiric is going to find that their builds fail until the publisher run happens
[00:44] <wgrant> Yes, but it's oneiric.
[00:44] <infinity> lamont: Yeah, I used to regularly deborphan and hunt stale conffiles, etc.  I assume that ended about a day after I left.
[00:45] <lamont> infinity: ./make-chroot.sh -d oneiric --lp  <-- far more trivial than fetching and unpacking the tarball
[00:45] <infinity> (Except when it doesn't work) ;)
[00:45] <lamont> that's a distro problem. :-p
[00:45] <infinity> Also, I think vorlon fell into a black hole.
[00:45] <infinity> I'm going to crank the publisher.
[00:46] <lamont> the current hack of the moment (used to bootstrap libjboss-buildmagic-java) was to /STAGE2=/""/\/var\/run\/agentx/
[00:46] <lamont> crank away
[00:47]  * slangasek climbs back out of the black hole
[00:47] <slangasek> infinity: armel is still building?
[00:47] <slangasek> doesn't that make it premature to crank?
[00:48] <infinity> Lamont led me to believe it was done. :P
[00:48] <infinity> Good thing the publisher's sad.
[00:48] <lamont> dude.
[00:48] <infinity> Did someone kill it mid-run?
[00:48] <lamont> you said you were gonna crank it, I assumed you had looked
[00:48] <infinity> *giggle*
[00:48] <infinity> Either way.  No harm done.
[00:48] <infinity> But yeah, did someone kill the previous publisher run by hand?
[00:48] <infinity> Cause stale locks concern me.
[00:49] <lamont> not me.
[00:49] <infinity> -r--r--r-- 1 lp_publish lp_publish 1 Jul 13 21:09 /srv/launchpad.net/ubuntu-archive/cron.daily.lock
[00:49] <lamont> and arm is in installdocs
[00:49] <lamont> and now it's stripping
[00:49] <infinity> Yeah, I'm watching now. :P
[00:49] <slangasek> infinity: yes, I did
[00:50] <slangasek> to prevent publishing the precise version of sysvinit that broke the chroots (though for a different reason)
[00:50] <infinity> lamont: Hahahaha.  Nine 5s.  Nice.
[00:50] <lamont> it's a historical value
[00:50] <infinity> lamont: Except that it's only eight. :(
[00:50] <lamont> only 8 5s
[00:50] <slangasek> infinity: lock removed
[00:50] <lamont> it's more "pound on the 5 key"
[00:51] <lamont> and yet, the troll refuses to die
[00:51] <infinity> And uploading...
[00:52] <infinity> And publishing.
[00:55] <slangasek> yay
[00:55] <infinity> lamont: So, where does your shiny chroot tool live?
[00:56] <lamont> https://code.launchpad.net/~lamont/launchpad-buildd/chroot-scripts
[00:57] <lamont> see also https://code.launchpad.net/launchpad-buildd
[00:57] <lamont> the wanna-build tree being the dump of history from the forkpoint for wanna-build, for wgrant's happiness
[00:58] <lamont> how much longer on the publisher?
[00:58] <infinity> "Until it's done".
[00:58] <lamont> heh
[00:58] <infinity> vorlon's hasty killing of the earlier run means it had some backlog to chew through. :P
[00:58] <lamont> heh
[00:58] <slangasek> it was not hasty
[00:58] <slangasek> it was premeditated
[00:59] <lamont> murder-one
[00:59] <infinity> And, apparently, it takes 20 seconds to process every effin' kde-l10n-uzbekistnification_all.deb
[00:59] <slangasek> the upload that /led/ to the killing was what was hasty
[00:59] <infinity> Dearest Soyuz.  WTF, over.  No love, Me.
[01:00] <slangasek> ?
[01:00] <infinity> slangasek: I just don't recall it being this slow. :P
[01:00] <lamont> slangasek: I think he's remembering some of the rough edges
[01:00] <wgrant> infinity: We are running at DB capacity.
[01:00] <wgrant> infinity: Because our master is getting a disk upgrade.
[01:00] <infinity> This is sort of like dating a girlfriend you dumped a decade ago.
[01:00] <wgrant> And apparently we can't extend arrays live.
[01:01] <lamont> infinity: I wasn't going to go there
[01:01] <lamont> infinity: and the girlfriend hasn't changed, eh?
[01:01] <wgrant> Also, Soyuz is terrible, but the DB isn't helping.
[01:02] <infinity> lamont: Not so much, no.  Neither the real one, nor the current metaphor.  But I'm smart enough to stay away from the former.
[01:02] <lamont> infinity: smart man
[01:02] <lamont> I was going with the metaphor only
[01:02] <infinity> lamont: Sadly, that intelligence didn't extend to staying away from Soyuz. ;)
[01:03] <lamont> infinity: after all the devs left, someone has to tend to it.
[01:03] <lamont> s/left/got dragged off to other things/
[01:03] <infinity> There we go.  Moving on to on-disk mangling finally.
[01:03] <lamont> well.
[01:04] <lamont> it's more soyuz got stuffed into the greater whole, and lots it's dedicated crew of minions/wranglers
[01:04] <infinity> Indeed.
[01:04] <wgrant> :(
[01:04] <infinity> And no more complimentary cowboy hats with every commit.
[01:04] <infinity> Sad times.
[01:04] <lamont> heh
[01:04] <lamont> say when master is current and I'll light the sky
[01:08] <infinity> It's thinking about it.
[01:08]  * lamont waits for proc 8939 to exit
[01:08] <lamont> I'm guessing that's all it needs
[01:08] <lamont> well, 7035 actually
[01:08] <infinity> When dists.new disappears, it's good.
[01:09] <infinity> Well, or the world has exploded.
[01:09] <infinity> In which case, we have nothing to worry about.
[01:09] <lamont> fsvo "we"
[01:10] <infinity> Loving all these apt-ftparchive warnings about malformed overrides.  Are we abusing something intentionally, or do we actually have a bug, I wonder...
[01:16] <wgrant> (those files are generated by scripts maintained by Platform, FWIW, not us)
[01:17] <infinity> I know.
[01:18] <infinity> Singed release files...
[01:18] <infinity> Almost there.
[01:19] <lamont> and dists.new is gone
[01:19]  * lamont lights things
[01:21] <lamont> wgrant: fwiw, if we had a way to put a particular DAS on hold, and then override that hold on a build-record basis, we wouldn't have to do the things we do
[01:21] <wgrant> Yeah.
[01:21] <wgrant> No Soyuz team to do that, though.
[01:22] <infinity> I have Celso's home number.
[01:22] <infinity> I'll just call him every hour, on the hour, until he agrees to work for me.
[01:22] <lamont> heh
[01:22] <wgrant> A maintenance squad might be able to do it eventually, but not until http://webnumbr.com/launchpad-critical-bugs reaches 0.
[01:22] <wgrant> As you can see, it's going well.
[01:22] <persia> Why does it need a Soyuz team?  can't we just file a bug and get someone to whine at the stakeholders meeting about it?
[01:23] <lamont> persia: I will not push it for stakeholders
[01:23] <infinity> persia: It's not really nearly as critical as a lot of other features we need last year.
[01:23] <lamont> there are bigger fish to fry
[01:23] <persia> Oh, of course.  I'm not saying it's *most* important.
[01:23] <persia> I just wonder if it needs a dedicated "Soyuz" team for anything to happen.
[01:23] <lamont> it doesn't, actually
[01:24] <wgrant> persia: Without a team that cares, it needs to be escalated or it will never happen.
[01:24] <wgrant> That's how things are now :(
[01:24] <persia> wgrant, Are there teams that care for anything at this point?
[01:24] <wgrant> No.
[01:25] <persia> Right.  Then it *does* need whining, but it's in our interest to have folk whine about other things first.
[01:25] <wgrant> Right. The bandwidth available by whining is smaller and different from the bandwidth that was available last year.
[01:26] <persia> Ah.  Now I understand.  Yes, that is bad.
[01:28] <infinity> (Gave back everything in chroot-wait, BTW)
[01:28] <ScottK> Well, it's not like LP is anything more than some web app thing.
[01:28] <slangasek> and publisher back on auto
[01:29] <slangasek> lamont, infinity: thanks much :)
[01:29] <infinity> ScottK: It's pretty much just phpBB with a different skin.
[01:29] <slangasek> I'm going to be afk much of the evening; if anyone finds a critical bug in sysvinit that I *didn't* catch in the nick of time, feel free to ring me :P
[01:30] <ScottK> All you soyuz elitists are just claiming you need a special team and stuff.  Soyuz makes web pages just like the rest of LP.
[01:33] <infinity> slangasek: Oh, should this sysvinit upload actually fix debootstrap, BTW?
[01:33] <slangasek> infinity: yes
[01:33] <persia> It doesn't seem to have done
[01:33] <infinity> slangasek: \o/
[01:34] <slangasek> persia: oh?
[01:34]  * persia started debootstrap *after* things were lit
[01:34] <persia> Same error I got before "W: Failure trying to run: chroot /var/lib/schroot/chroots/oneiric-armel dpkg --force-depends --install /var/cache/apt/archives/libc6_2.13-9ubuntu2_armel.deb"
[01:34] <infinity> persia: Not from ftpmaster.internal, you didn't.
[01:34] <infinity> persia: Mirrors are being pulsed now. :P
[01:34] <slangasek> were you debootstrapping after the packages were published to the world, though :)
[01:34] <persia> Aha!
[01:34]  * slangasek nods
[01:34]  * persia waits for mirror pulse
[01:35] <infinity> Which reminds me, we need to move the mirror pulse in cron.daily to happen right after dsync-flist...
[01:35] <infinity> Delaying it until after cron.germinate and such seems silly.
[01:35] <wgrant> Speaking of dsync, can we remove it?
[01:35] <infinity> wgrant: Why?
[01:35] <infinity> wgrant: I was planning to pick up maintenance and get it building on modern OSes, would that make you happy?
[01:36] <wgrant> infinity: It's slow and shouldn't save anything significant, should it?
[01:37] <infinity> It only sometimes saves some space on ftpmaster, I'll admit.  If it's being run on cesium, it probably saves a ton.
[01:37] <infinity> It's not that slow, though.
[01:37] <wgrant> By cesium you mean germanium?
[01:37] <wgrant> cesium should have no persistent data.
[01:37] <infinity> Err, yes.
[01:37] <infinity> Sorry.  It's been a while. :P
[01:38] <wgrant> We could possibly run it over PPA pools, but we don't.
[01:38] <wgrant> germanium is dead enough as it is.
[01:38] <infinity> Either way, it took all of 56s on the primary archive.
[01:38] <infinity> It's not exactly slow.
[01:38] <infinity> In other words, it took less time than processing two arch:all BPRs. :P
[01:39] <infinity> *cough*
[01:39]  * lamont afk, laters
[01:40]  * persia tries again with 2.88dsf-13.10ubuntu3 this time
[01:58] <infinity> slangasek: libc6 is still a sad panda, trying to touch /var/run/init.upgraded
[01:59] <infinity> slangasek: So, still no debootstrap.
[02:01] <persia> Worked for me
[02:01] <infinity> slangasek: Oh, wait, I might be pulling a persia here, my local mirror's still pulsing. :P
[02:01]  * infinity taps his foot.
[02:01] <persia> (just got "I: Base system installed successfully." about the same time you posted)
[02:02] <infinity> Normally, being 30 minutes behind ftpmaster doesn't bug me in the least.  Sometimes, it's really annoying...
[02:02] <persia> 30 minutes?  I really need to configure push-mirroring at some point.
[02:03] <infinity> I need to weasel my way back into the push-primary rotation again.
[04:54] <slangasek> heh
[04:54] <slangasek> infinity: bootstrapping looking happier now? :)
[04:59] <infinity> slangasek: Indeed.
[10:55]  * cjwatson continues to attempt to get lucid images to build :-/
[17:12] <cjwatson> jibel: All the cronned lucid builds (Ubuntu desktop, Ubuntu alternate, Ubuntu server, Ubuntu DVD, Kubuntu desktop, Kubuntu alternate) should be up to date now
[17:12] <cjwatson> jibel: I can't vouch for them working
[17:13] <cjwatson> well, ia64 and sparc didn't build; I don't know whether we care
[17:13] <cjwatson> does anyone know if the 2.6.32-33 ABI is intended for 10.04.3?
[18:22] <jibel> cjwatson, desktop i386 is ok. I'll sync the other images this evening and try them tomorrow.
[21:59] <skaet> cjwatson, kernel we're hoping to use for 10.04.3 is 2.6.32-33.70.    Based on tracking bug 807175,  looks like its gone through cert and testing, so should be good to move to updates.  However not sure I understand where the ABI ref is coming in from, and the concern.  Can you elaborate?
[21:59] <ubot4> Launchpad bug 807175 in linux (Ubuntu Lucid) (and 10 other projects) "linux: 2.6.32-33.70 -proposed tracker (affects: 4) (dups: 4) (heat: 44)" [Medium,In progress] https://launchpad.net/bugs/807175
[22:03] <infinity> skaet: If lucid's meant to update from 32.62 to 33.70, that'll be an ABI bump (and hence new package names) from -32- to -33-, which will require a d-i upload, and some fiddling, that's all.
[22:04] <infinity> skaet: So, if that's the plan, we should get it into -security,-updates and get with the mangling before testing images that aren't remotely valid. ;)
[22:04] <infinity> skaet: s/meant to update from 32.62 to 33.70/meant to update from 32.62 to 33.70 for the point release/
[22:06] <skaet> infinity,  need to check with security team, but its ready to go to updates based on cert/QA results in the bug.
[22:12] <infinity> I think Colin's question wasn't so much "is it ready?" as "do you want it on the point release images?" :)
[22:14] <infinity> And I think the above constitutes a "yes", so yeah, just need to shove it through and then rebuild d-i, and then images.
[22:15] <skaet> infinity,  as long as security doesn't throw up a red flag in the final stages of the process.    Yes please.