[00:13]  * mwhudson lunches
[00:48]  * NCommander is amazed at how long the LP test suite takes to run
[00:49] <mwhudson> NCommander: welcome to our world
[00:51] <NCommander> mwhudson: how long does a fullrun take?
[00:51] <mwhudson> NCommander: about 4 hours, i think
[00:51] <NCommander> mwhudson: (I'm seeing some codehosting failures which I'm concerend about, since I didn't touch that bit of the code)
[00:51] <NCommander> mwhudson: this has been going for at least 6-7
[00:51] <mwhudson> NCommander: what hardware?
[00:51] <mwhudson> that's pretty slow though
[00:52] <NCommander> mwhudson: 2.2Ghz dual core intel dual core processor
[00:52] <NCommander> 8 GiB of RAM running on an SSD
[00:52] <NCommander> Got a few weird failures
[00:52] <NCommander> mwhudson: I'm running lucid versus karmic though, which may explain some of the failures
[00:53] <mwhudson> ah possibly
[00:53] <NCommander> mwhudson: I know its possible to run the test suit eon EC2; can you shed some light into that?
[00:54] <mwhudson> NCommander: have you seen https://dev.launchpad.net/EC2Test ?
[00:55] <NCommander> mwhudson: how much does EC2 cost for something like this?
[00:55] <mwhudson> a few $ i guess
[00:55] <mwhudson> it uses a c1.xlarge instance for 4-5 hours
[00:56] <NCommander> mwhudson: might be worth it although I can't expense it as I'm not on the LP team :-/
[00:56] <mwhudson> NCommander: i can run it for you if you like
[00:56]  * mwhudson is currently trying to find stuff to delete so i can upgrade to lucid without scaring myself
[00:56] <NCommander> mwhudson: that may be worthwhile, although I'm concerned I may have actually broken part of LP (first db changes I ever made)
[00:57] <mwhudson> NCommander: i don't see why that means you shouldn't run it under ec2 though?
[00:57] <NCommander> mwhudson: oh, no, that was just a side comment, not a blocker :-)
[00:57] <mwhudson> :)
[00:57] <mwhudson> NCommander: where is the branch on lp?
[00:58] <NCommander> mwhudson: its at ~mcasadevall/launchpad/raw_source_changelog, but I'm thinking I should upload this last round of changes as a new branch
[00:58] <NCommander> I rewrote the functionality to use librarian rather than storing in teh DB
[00:58] <mwhudson> NCommander: well, ok, i'm not going anywhere, so if you want me to test a branch, just ping me
[00:58]  * NCommander also feeds a second code review may be a good idea
[00:58] <NCommander> mwhudson: give me a sec
[01:02] <NCommander> mwhudson: lp:~mcasadevall/launchpad/raw_source_changelog_librarian
[01:03] <mwhudson> NCommander: what email address do you want results sent to?
[01:03] <mwhudson> (/msg if you like...)
[01:27] <NCommander> mwhudson: mcasadevall@ubuntu.com
[01:28] <NCommander> mwhudson: (sorry for the delay in answering)
[01:28] <mwhudson> NCommander: ok, starting up ec2 test
[01:33] <NCommander> mwhudson: thanks. I'm thinking I should probably turn my old laptop into running UEC and then run ec2test on it :-)
[01:33] <mwhudson> heh heh
[01:34] <mwhudson> not if you want results quickly i suspect
[01:34] <NCommander> mwhudson: the old laptop is 2.2Ghz 4GiB of RAM
[01:34] <NCommander> its not that slow
[01:35] <mwhudson> oh ok
[01:35] <mwhudson> no :)
[01:39] <NCommander> mwhudson: Total: 25575 tests, 16 failures, 19 errors in 510 minutes 32.100 seconds.
[01:39] <NCommander> irony, it just finished
[01:40] <NCommander> I tee'ed the output, but I don't even know if the failures are due to lucid, or due to something in my environment :-/
[01:42] <mwhudson> NCommander: well the results from ec2 will tell you some of that
[01:42] <NCommander> mwhudson: thanks :-/
[02:17] <wgrant> NCommander: The only known Lucid failure is in apt-ftparchive.
[02:17] <wgrant> NCommander: Lucid's adds SHA1 and SHA256 hashes in addition to pre-Lucid's MD5s.
[02:17] <wgrant> Any other failures are probably real problems.
[02:18] <NCommander> wgrant: I think my postgresql server dumped out towards the end, I got a bunch of *strange* errors with write errors :-/
[02:19] <wgrant> Heh.
[02:19] <wgrant> Does anybody happen to have an idea of where the time is taken in the primary archive publisher?
[02:24] <NCommander> wgrant: well, at least then I know which tests to check
[02:25] <wgrant> NCommander: Right.
[04:16] <lifeless> wgrant: not an accurate one; its a problem I'd like to look at. If you choose to look at it I'd be very happy to be a sounding board.
[04:21] <wgrant> lifeless: From what I can tell, publish-distro.py takes about 20-25 minutes currently.
[04:21] <wgrant> I've cut two or so minutes out of that locally by optimising domination.
[04:22] <wgrant> And I think most of the rest mus tbe in the apt-ftparchive integration. But I'd like to know for sure.
[04:23] <wgrant> (apt-ftparchive integration is the one bit I can't sanely simulate locally, since it requires lots and lots of real packages, not just fake contentless ones)
[04:24] <persia> wgrant: You could simulate by mirroring karmic and "publishing" bits of current lucid.
[04:25] <wgrant> persia: But that requires a local copy of the archive, which has a sane Internet connection as a prereq.
[04:27] <persia> The pacific ocean clearly needs a denser net of high-capacity optical fibre
[04:28] <wgrant> But anyway, I have just reduced native index generation time for largely-unchanged Ubuntu-sized binary pockets to 2-3 seconds.
[04:28] <spm> or electrical flows that exceed the speed of light
[04:29] <wgrant> It would be very interesting to know how much longer than 20 seconds the apt-ftparchive integration code takes to run.
[04:29] <mwhudson> spm: yeee-haw!
[04:29] <wgrant> Can anybody see cron.publish-ftpmaster logs?
[04:29] <persia> Is there some test I can run locally that doesn't require me to setup/install launchpad first?
[04:29] <spm> mwhudson: indeed
[04:30] <wgrant> persia: No.
[04:30] <wgrant> But the times are all in the logs that presumably somebody can see.
[04:30] <mwhudson> dammit, now i want to watch that movie again, and i only watched it last week or so
[04:30] <wgrant> Which movie?
[04:31] <mwhudson> dr strangelove
[04:31] <wgrant> Ah.
[04:31] <mwhudson> spm posted http://davideubank.files.wordpress.com/2008/09/slim-pickens_riding-the-bomb_enh-lores-720p.jpg with the comment "the life of a losa" or something like that
[04:32] <wgrant> Ahhh.
[04:32] <spm> it' an accurate description.
[04:34] <wgrant> spm: Does cron.publish-ftpmaster log anywhere that you can check quickly? I am interested in the time taken by index generation (starting with '* Step C', ending with '* Step D')
[04:34] <spm> looks...
[04:37] <spm> urg. we don't rotate the poppy log. /me hads a MUST FIX.
[04:41] <spm> wgrant: http://paste.ubuntu.com/399107/
[04:41] <wgrant> Aha.
[04:42] <wgrant> So we can cut 17 minutes of the publisher run-time to probably well under one.
[04:42] <wgrant> Thanks spm.
[04:42] <spm> heh; if you can cut the time down? we'll thank you! :-)
[04:46] <persia> wgrant: This isn't all part of some master plan to release 48 times a day, is it?
[04:46]  * NCommander waves to persia 
[04:46] <spm> persia: ??? there's more minutes than 48 in a single day! :-P
[04:48] <persia> spm: Sure, but one has to run the entire publisher for each release, so it would need lots more optimisation to release every minute.
[04:48] <wgrant> persia: To at least make it possible to drive it that fast if necessary, yes
[04:49] <persia> Actually, with the "if necessary" bit, that's a good thing.  I'm just not sure I'd like it that fast by default: sometimes folk catch stuff within a run, which gets hard if it gets smaller.
[04:49] <wgrant> (this maaaay not be unrelated to Saturday's events)
[04:49] <persia> heh
[04:50] <NCommander> persia: well, given a fast enough machine, it should be possible to publish once a minute
[04:51] <wgrant> NCommander: s/fast enough machine/less pathetic publishing code/
[04:51] <persia> NCommander: You'd also need a faster network, faster mirrors, etc.
[04:51] <NCommander> persia: or a longer minute
[04:51] <persia> feh.
[04:51] <NCommander> I mean, if you redefine a minute ...
[04:51] <NCommander> Microsoft redefined stable after all, it could be done :-)
[04:52] <wgrant> My timeline has a 19 minute gap :(
[04:52] <persia> My minute is 551,557,906,200 vibrations of cesium-133 at STP.
[04:53] <NCommander> My minute is 0.06 kiloseconds
[04:54] <NCommander> My second however is define as 0.0864 standard seconds. What sorta second am I?
[04:54] <persia> wgrant: Which bit is missing?
[04:54] <spm> a 2nd rater ?
[04:55] <spm> NCommander: you really left yourself open to that one :-)
[04:55] <NCommander> er
[04:55] <NCommander> *0.864 seconds
[04:55] <NCommander> spm: its a decimal second :-)
[04:56] <wgrant> persia: There is a 19 minute hole containing dsync's link-dupes run, cron.germinate and the cocoplum->syncproxy sync. The last probably takes three minutes.
[04:56] <wgrant> So one of the first two is taking an awful long time.
[04:56] <persia> link-dupes
[04:57] <wgrant> I'd suspect so, yes.
[04:57] <wgrant> I'm not quite sure why it's run.
[04:57] <persia> It takes > 2 hours to run link-dupes against my mirror set.
[04:57] <wgrant> Do you know?
[04:57] <persia> Massively saves space.
[04:57] <wgrant> Even though we symlink matching files anyway?
[04:57] <persia> Erm.
[04:58] <persia> Then no, doesn't do anything at all.
[04:58] <persia> I'd rather see matching files hardlinked, because it mirrors better, but that's just preference.
[04:58] <persia> So even though it has no effect, it still takes forever to run fdupes.
[04:58] <wgrant> I mean, when we need a file published in multiple locations, we symlink the non-master copy as any good archive software should.
[04:59] <wgrant> This is well before the archive is split into archive/ports/security, so I don't see why there would be any dupes.
[04:59] <persia> If it's pre-mirror-split, there should be no dupes.
[04:59] <wgrant> Right.
[04:59] <persia> Or rather, only known multi-publication.
[05:01] <wgrant> I'm guessing you use link-dupes because you maintain mirrors of both ports and non-ports (or portions thereof)?
[05:02] <persia> Actually, I use my own hacked fdupes-based script, and also mirror debian, but yes.
[05:05] <wgrant> So if there's little point running it, and we fix NMAF to support extra-overrides and migrate the primary archive to use it, *and* my two optimisation branches actually work properly, we can cut more than half an hour off the 40-minute-ish publisher.
[05:06] <persia> Which could shorten the time-to-fix for something like Saturday's incident down to ~90 minutes with manual publisher runs.
[05:06] <wgrant> s/90/30
[05:06] <mwhudson> what happened on saturday?
[05:06] <persia> wgrant: consider build times too.
[05:06] <wgrant> Upload CDBS, wait for it to build in a few minutes, publish. Upload two broken packages, built in 10 minutes, publish.
[05:07] <wgrant> mwhudson: Lucid broke right beta 1.
[05:07] <persia> mwhudson: some updates were published to lucid that broke gnome-panel autostart on login.
[05:07] <wgrant> gnome-panel and nautilus broke so they wouldn't autostart.
[05:07] <wgrant> Leaving all the willing testers with no desktop at all.
[05:07] <spm> wooo
[05:08] <persia> (or at least everyone who logged out and logged in after the updates)
[05:08] <mwhudson> i see....
[05:08] <wgrant> persia: Which lots of people did, after they couldn't open folders any more :/
[05:08] <spm> speaking as someone who's 7 mins from downloading for the u/g. is it all fixed on latest updates?
[05:08] <persia> Oh, yeah.  non-logout/non-reboot is the significant minority.
[05:08] <mwhudson> just out of totally idle curiosity, what versions of things contain the fixes?
[05:08] <mwhudson> spm: zing!
[05:08] <persia> It's fixed.
[05:08] <wgrant> mwhudson: -0ubuntu2 of both of those is broken.
[05:09] <wgrant> -0ubuntu3 is safe.
[05:09] <wgrant> It's been fixed since Saturday night.
[05:09] <wgrant> It just took a lot longer than it could have if the publisher didn't take an hour.
[05:09] <mwhudson> mmm
[05:09] <mwhudson> i have -0ubuntu1 of those
[05:09] <mwhudson> i guess my mirror aint so fresh
[05:09] <spm> having my main workstation cactus is a pain; but not unsurvivable. karmic forced me onto my laptop for a week or 2; totally broke my workstation's startup.
[05:10] <persia> Getting the publisher down to ~10 minutes makes it something people will be more willing to babysit too.
[05:10] <wgrant> Right.
[05:12] <wgrant> Plus it'd be less of a pain around releases, when you see release managers tossing up whether to run it now or not.
[05:13] <persia> Absolutely.
[05:13] <mwhudson> ha, ha
[05:13] <mwhudson> 255 updates
[05:55] <NCommander> mwhudson: thanks for ec2ing my branch. TIme to fix some failures
[05:55] <NCommander> mwhudson: you got less failrues and errors than I do it
[05:58] <NCommander> what can cause this: ProgrammingError: permission denied for relation sourcepackagerelease
[05:58] <mwhudson> NCommander: have you encountered database/schema/security.cfg yet?
[06:00] <NCommander> mwhudson: nope
[06:01] <mwhudson> NCommander: in that case "ProgrammingError: permission denied for relation sourcepackagerelease" means you need to make friends with this particular file
[06:01] <mwhudson> NCommander: different bits of launchpad connect to the database as different database users
[06:01] <mwhudson> and the different users have permissions to access the various tables in different ways
[06:03] <NCommander> mwhudson: ugh. I have no idea how to fix this, since there is a SELECT permisison already in place
[06:04] <mwhudson> NCommander: for which user though?
[06:05] <NCommander> mwhudson: librariangc. Looking at the code, its trying to DELETE, but I'm kinda confused on how objects leave librarian
[06:07] <NCommander> mwhudson: it seems to me its trying to delete rows, but I'm not understanding why librariangc isn't happy
[06:08] <NCommander> mwhudson: oh wait ...
[06:09] <NCommander> mwhudson: alright, I think I figured out why the librariangc tests are failing :-)
[06:12] <mwhudson> NCommander: grats :)
[06:18] <NCommander> mwhudson: I assume I need to re-make schema after editing security.cf
[06:18] <NCommander> *cfg
[06:19] <mwhudson> NCommander: i think you can just run python database/schema/security.py launchpad_ftest_template
[06:19] <mwhudson> or something
[06:19] <mwhudson> but i usually just run make schema, yeah
[06:19] <NCommander> mwhudson: thanks, I'm still somewhat new at all this LP hackery. I'll probably ask you to re-run ec2test for me since I got a lot mor efailures on 127.0.0.1 than ec2 did :-/
[06:20]  * NCommander will probably setup EC2 sometime in the future but can't ATM :-/
[06:20] <StevenK> NCommander: It's pretty easy, it took me about 10 minutes
[06:21] <NCommander> StevenK: how much did it cost per run?
[06:21] <StevenK> Roughly $3.50
[06:22] <StevenK> But they don't charge until the end of the month
[06:22] <NCommander> StevenK: that can add up fast -/
[06:22] <mwhudson> i spent a lot on ec2 in february
[06:22] <mwhudson> luckily i can expense it...
[06:32] <NCommander> mwhudson: woo, librarian fixed
[06:32]  * NCommander now works on registery failures
[06:33] <mwhudson> NCommander: i'm going to be going afk soon, but europe should be getting up soon ish :)
[06:47] <NCommander> mwhudson: hrm, is it possible for a test to be failing in db-devel?
[06:47]  * NCommander can't see how this failure is possible
[06:47] <NCommander> er
[06:47] <NCommander> or how is impossible
[06:48] <StevenK> NCommander: Stuff doesn't land if the tests fail
[06:48] <mwhudson> StevenK: not true any more, sadly
[06:48] <NCommander> mwhudson: that was what I was afraid of
[06:48] <StevenK> Awww
[06:48] <mwhudson> NCommander: which tests fail for you?
[06:48] <NCommander> mwhudson: lp.registry.tests.test_distroseries.TestDistroSeriesPackaging
[06:48] <mwhudson> oh
[06:49] <mwhudson> that doesn't seem to be failing in buildbot
[06:49] <NCommander> mwhudson: NameError: global name 'gpg_key' is not defined
[06:49] <NCommander> Looking at the file in question though, there is only one refernece to gpg_key in the whole thing
[06:49] <mwhudson> !
[06:49] <mwhudson> is that in devel or db-devel?
[06:50] <NCommander> mwhudson: db-devel, but hold on
[06:50]  * NCommander might have done something stupid
[06:51] <wgrant> Damn. Turns out my 20 second index generation run had broken bz2 compression. Fixing that blows it out to more than a minute :(
[06:51] <StevenK> wgrant: A minute versus 24, though?
[06:51] <mwhudson> NCommander: i don't see gpg_key in that file
[06:51] <mwhudson> (on either branch)
[06:51] <wgrant> 17-18, but yes.
[06:52] <wgrant> The rest of the 24 is being cut away in other places.
[06:52] <NCommander> mwhudson: I dunno, I think I went a little merge happy at some point, and lack of sleep is not helping
[06:53] <mwhudson> lack of sleep often doesn't help with hacking
[06:57] <NCommander> mwhudson: no, but my girlfriend went ona  cleaning spree
[06:57] <NCommander> and my bed currently isn't where I left it
[06:58] <mwhudson> NCommander: haha
[06:58] <wgrant> Are we week 2 or 3 at the moment?
[06:58] <mwhudson> wgrant: just starting week 3
[06:58] <wgrant> Aha.
[06:59]  * wgrant had better get his last PPA download stats branch in order, then.
[07:00]  * NCommander broke something bad
[07:00] <NCommander> very bad
[07:25] <NCommander> anyone around who can run the ec2test script OR run the test suite locally without getting unrelated failures
[07:25]  * NCommander has it fixed
[08:57] <wgrant> bigjools: Morning.
[08:57] <bigjools> g'day
[08:58] <wgrant> What are the blockers for NMAFing primary? Just extra overrides?
[08:59] <bigjools> yep, barring any hidden bugs
[08:59] <bigjools> and possibly performance
[08:59] <wgrant> Performance is definitely an issue.
[08:59] <wgrant> But with a bit of caching I can publish all indices for a larger-than-Ubuntu archive in 20 seconds.
[09:00] <wgrant> (one series thereof, that is, like most normal runs)
[09:02] <bigjools> wgrant: using NMAF?
[09:02] <wgrant> bigjools: Yeah.
[09:03] <bigjools> it would be interesting to do like for like comparisons
[09:03] <mrevell> Hello
[09:03] <bigjools> hey mrevell
[09:03] <bigjools> wgrant: did you see Bug 543162 BTW?
[09:03] <mup> Bug #543162: launchpad-buildd exception: Try to register an already registered process. <Launchpad Auto Build System:New> <https://launchpad.net/bugs/543162>
[09:03] <wgrant> bigjools: I was going to, but a-f requires real sources and binaries, whereas NMAF doesn't care if they're garbage.
[09:04] <bigjools> wgrant: dogfood FTW :)
[09:04] <wgrant> bigjools: I have, yeah. lamont poked me about it at the time to check if there was anything obvious.
[09:04] <bigjools> it's an odd one - I can't work out if twisted blew up or the slave code
[09:05] <wgrant> bigjools: Is dogfood a good performance indicator?
[09:05] <bigjools> wgrant: not against anything else, but relative to itself of course it is
[09:05] <noodles775> bigjools: if it's related to tearing down twisted code, that behaviour changed recently.
[09:05] <wgrant> I mean, a-f is going to hit the DB in different ways to NMAF.
[09:05] <bigjools> wgrant: indeed
[09:06] <bigjools> noodles775: was that in lucid though?
[09:06] <wgrant> I tried to use memcache, but it was far too slow :(
[09:06] <noodles775> bigjools: nm, not on cesium.
[09:07] <bigjools> ironic :)
[09:07] <wgrant> noodles775: It's the buildds themselves, which are in fact pre-Wellington.
[09:08] <noodles775> wgrant: Right (just reading the bug).
[09:11] <wgrant> It's only been seen on armel, but it happened on two of them (at a reasonably inconvenient, almost simultaneous time)
[09:13] <wgrant> bigjools: Also, do you know the purpose of the dsync link-dupes call in cron.publish-ftpmaster? There shouldn't be any duplicates...
[09:14] <bigjools> no idea, it's always been there
[09:14] <wgrant> (and it probably takes forever and then a little while longer (or just 15 minutes)
[09:18] <wgrant> bigjools: Who's likely to know?
[09:19] <wgrant> 'cause slow publishing makes me sad and breaks lots of users' systems.
[09:22] <bigjools> wgrant: probably people who have left :/
[09:23] <bigjools> I can check with IS
[09:23] <wgrant> It's from dak, so elmo probably does.
[09:50] <deryck> Morning, all.
[09:53] <jml> good morning Launchpadders
[10:25] <wgrant> What do I need to do to get the PPA download stats script running on production?
[10:25] <bigjools> wgrant: is it a cron job?
[10:26] <bigjools> where is it?
[10:27] <wgrant> bigjools: It's a cron job. cronscripts/parse-ppa-apache-access-logs.py
[10:27] <bigjools> ok
[10:27] <bigjools> wgrant: we need to run it on dogfood first
[10:27] <wgrant> Right.
[10:27] <wgrant> Or staging.
[10:28] <wgrant> And it needs production configs.
[10:28] <bigjools> not staging
[10:28] <wgrant> It doesn't need a real archive. Just logs to be put somewhere.
[10:28] <bigjools> I'll work on the configs and test it on DF
[10:28] <wgrant> Thanks.
[10:29] <bigjools> hmm busy day
[10:34] <wgrant> bigjools: Ew, history deletion.
[10:35] <bigjools> wgrant: ?
[10:35] <wgrant> bigjools: Deleting archives and EVERYTHING with them.
[10:35] <bigjools> wgrant: yep
[10:35] <bigjools> like deleting a branch
[10:35] <wgrant> Yes, both are repulsive.
[10:36] <bigjools> deletion is deletion, not some half-assed attempt
[10:36] <bigjools> if you want history, keep the f***ing thing
[10:36] <bigjools> or it can be disabled
[10:36] <wgrant> If you want malicious or misled users to be able to inappropriately destroy things irrevocably, let them actually delete thousands of records in one click.
[10:37] <wgrant> If you want users to be able to remove the only records of their malicious activities on the buildds...
[10:39] <wgrant> Also, how do binary copies work?
[10:39] <wgrant> If I delete an archive that has had binaries copied from it, the builds are more than a little awkward.
[10:40] <bigjools> the binaries won't be deleted
[10:40] <wgrant> What happens to the builds?
[10:40] <wgrant> We cannot delete them.
[10:40] <bigjools> neither those
[10:40] <wgrant> So we lie about their archive?
[10:41] <bigjools> no, we change the code to say their original archive was deleted
[10:41] <bigjools> did you read the bug?
[10:42] <wgrant> It didn't cover builds in particular. I guess stuff might not break if the archive disappears.
[10:43] <bigjools> that's what the test suite is for
[11:46] <elmo> ok, seriously, this sucks
[11:46] <elmo> I've turned referer back on, and I still can't add attachments to bugs
[11:47] <wgrant> What's the error? Still 'Unexpected Form Data' with a 500?
[11:47] <elmo> oh, lala
[11:47] <elmo> sorry, never mind
[11:48] <elmo> it's my fascist firefox apparmor profile
[11:48] <elmo> \o/
[11:48] <wgrant> Heh.
[12:02] <kfogel> adeuring: PQM is behaving like it has not seen my submissions (http://paste.ubuntu.com/399248/).  Am I missing something?  Can it have built so fast that I missed it? :-)
[12:03] <kfogel> adeuring: (oh, I made a cut-and-paste error in that paste -- the submissions actually do match their directories, don't worry)
[12:03] <adeuring> kfogel: ;)
[12:04] <kfogel> adeuring: https://pqm.launchpad.net/ shows nothing though
[12:05] <kfogel> adeuring: bbiab
[12:10] <bigjools> wgrant: ok I have dogfood updated, what configs does that cronscript need? it looks generic
[12:11] <wgrant> bigjools:
[12:11] <wgrant> [ppa_apache_log_parser]
[12:11] <wgrant> logs_root: /srv/ppa.launchpad.net-logs
[12:11] <wgrant> Paths will vary.
[12:11] <bigjools> wgrant: what about where it puts results?
[12:12] <wgrant> bigjools: The DB?
[12:12] <bigjools> ah nm the config is ppa specific
[12:12] <bigjools> how does it know to read that config?
[12:14] <wgrant> bigjools: It's used in cronscripts/parse-ppa-apache-access-logs.py
[12:14]  * bigjools slaps face
[12:14] <bigjools> I was looking at the wrong script, sorry
[12:14] <wgrant> Heh.
[12:14] <wgrant> I thought something might have been a bit odd.
[12:14] <bigjools> yeah :)
[12:22] <bigjools> wgrant: so does this script depend on only ppa logs being in the logs_root ?
[12:22] <wgrant> bigjools: It'll ignore them if they don't match a PPA binary for any reason.
[12:22] <wgrant> So it shouldn't matter.
[12:23] <bigjools> mmkay
[12:24] <bigjools> wgrant: well it does if one of the files is not readable :/
[12:24] <wgrant> Yeah.
[12:24] <wgrant> Not my code :P
[12:24]  * bigjools watches the buck flying overhead
[12:25] <wgrant> It will also do strange things if you run both the librarian and PPA log parsers over the same directory, but that's unlikely to happen.
[12:33] <adeuring> kfogel: did you perhaps receive a mail from pqm with an error message?
[12:33] <bigjools> wgrant: it doesn't log much does it :)
[12:33] <bigjools> and 2.1g resident :/
[12:33]  * bigjools gets coffee while waiting for it
[12:39] <maxb> Can someone fix the grammar fail in the PPA description?  https://edge.launchpad.net/~launchpad/+archive/ppa
[12:40] <wgrant> bigjools: Did you see the warning in the docstring (lovingly copied from the librarian one)
[12:43] <wgrant> (I didn't think there'd be many logs on dogfood -- sorry)
[12:47] <bigjools> maxb: done
[12:47] <bigjools> thanks
[12:48] <bigjools> wgrant: I fiddled the log so that it only appears as a symlink in another dir
[12:50] <wgrant> m_o_d: Did you run rocketfuel-setup?
[12:50] <m_o_d> wgrant: yes
[12:50] <wgrant> m_o_d: Did you see it installing lots of packages?
[12:51] <m_o_d> wgrant: yes, my error is when i run make schema
[12:51] <wgrant> m_o_d: I know. But it's complaining about something that rocketfuel-setup would have instaled.
[12:51] <wgrant> Do you have python-tickcount installed?
[12:51] <jml> bigjools, any recommendations for coffee from hasbean?
[12:51] <m_o_d> yes, 0.1-0ubuntu10launchpad1
[12:52] <wgrant> m_o_d: Which version of Ubuntu are you using?
[12:52] <bigjools> jml: oh yes :)
[12:53] <m_o_d> wgrant: lucid
[12:53] <bigjools> jml: http://www.hasbean.co.uk/products/Guatemala-Finca-Puerta-Verde-Antigua-2009%252d2010.html
[12:53] <bigjools> you had that last week
[12:53] <bigjools> well, on Friday when the other stuff ran out
[12:55] <wgrant> m_o_d: Try to apt-get upgrade. Does that propose anything python-related?
[12:55] <jml> bigjools, thanks.
[12:55] <bigjools> jml: also check out the Bolivia Machacamarca
[12:56] <bigjools> jml: that one is simply awesome
[12:56] <wgrant> bigjools: Is the script still doing stuff?
[12:56] <bigjools> wgrant: no it finished some time ago, I got sidetracked
[12:57] <bigjools> binarypackagereleasedownloadcount has got 4736 rows now
[12:57] <wgrant> Excellent.
[12:57] <jml> bigjools, ta. I'll order the Guatemala one and see how I go.
[12:58] <wgrant> bigjools: Do you have my latest branch?
[12:58] <bigjools> wgrant: I'm using db-devel
[12:58] <wgrant> Ah, the webapp is not up.
[12:58] <jml> bigjools, I'll have to do a taste test.
[12:58] <bigjools> jml: most stuff from central America is great
[12:59] <bigjools> wgrant: noodles775 is playing with iut
[13:04] <kfogel> adeuring: yeah -- testfix/rc
[13:04] <kfogel> adeuring: :-(
[13:09] <barry> bigjools, or maybe anybody else running on lucid+py2.6.  have you seen this error?
[13:09] <barry> http://paste.ubuntu.com/399288/
[13:10] <bigjools> barry: no not seen that
[13:10] <wgrant> maxb: Arrrrgh it's back. ^^
[13:10] <bigjools> barry: but then I'm not really running 2.6 :)
[13:10] <wgrant> barry: That one has been plaguing me on and off since I started using Lucid a month in.
[13:10] <wgrant> maxb has seen it too.
[13:10] <barry> wgrant, bigjools, maxb arg! :)
[13:10] <maxb> *blink*
[13:11] <maxb> UserWarning: Unbuilt egg for setuptools [unknown version] (/usr/lib/python2.6/dist-packages)   <--- that's a new one
[13:11] <barry> wgrant, maxb, bigjools i really don't want to install python2.5 if i can help it.
[13:11] <bigjools> barry: it's part of lp-dev-deps isn't it?
[13:11] <wgrant> barry: You have to. Launchpad doesn't run on 2.6.
[13:11] <barry> maxb: i do have python-setuptools installed, which should provide distribute
[13:12] <bigjools> wot wgrant said
[13:12] <maxb> barry: You want to install python2.5. Unless you want to dedicate your entire time to massive porting project for the forseeable future, you want to install python2.5
[13:12] <wgrant> (this is a bug that really needs to be fixed soon)
[13:12] <bigjools> and I thought barry was fixing it actually :)
[13:12] <barry> dang.
[13:12] <barry> it's gotta run on 2.6 pretty soon, though, right?
[13:13] <kfogel> adeuring, wgrant: have a sec to chat about https://bugs.edge.launchpad.net/malone/+bug/78565/comments/16  (ironically, you'll have to navigate from there to the bug for context, becuase the bug isn't fixed yet).
[13:13] <mup> Bug #78565: no direct link from bug comment page to corresponding bug <breadcrumbs> <confusing-ui> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/78565>
[13:13] <wgrant> kfogel: The breadcrumb infrastructure is buggy, and the bug breadcrumbs are buggy.
[13:13] <kfogel> wgrant: because when the final breadcrumb does not match the page being actually displayed, the breadcrumb is still unlinked?
[13:13] <kfogel> wgrant: or some other bugginess?
[13:14] <wgrant> That is, the breadcrumb infrastructure should linkify (and probably whinge) if the target of the final crumb is not the current context.
[13:14] <wgrant> And bug comments should have breadcrumbs of their own.
[13:14] <wgrant> Ah, that comment.
[13:14] <kfogel> wgrant: mmmm.  The latter alone would fix this bug, of course.
[13:14] <wgrant> Right.
[13:14] <wgrant> I fixed a few Soyuz cases of this a couple of weeks ago.
[13:15] <kfogel> wgrant: oh hey, let me find those branches, since I can probably save some time by looking at that diff.
[13:15] <kfogel> wgrant: or, do you want to just do this one too?  Either way is fine with me.
[13:15] <wgrant> kfogel: devel r10480
[13:15] <kfogel> thx
[13:15] <kfogel> wgrant: ^^
[13:16] <wgrant> I should sleep and do assignments, not add breadcrumbs.
[13:16] <kfogel> wgrant: heh, np.
[13:17]  * maxb lunches
[13:17]  * wgrant really should get around to starting the page titles, breadcrumbs and root contexts discussion on the ML at some point.
[13:19] <leonardr> i had to manually install 4 packages (3 dev packages + python-geoip) to get make to work in lucid. is this a problem? would it help other people if i mentioned these packages on the wiki?
[13:20] <kfogel> leonardr: 'make' in general, or just for lp?
[13:21] <leonardr> kfogel: just for lp, sorry
[13:22] <leonardr> i had to install those packages to start doing my work again
[13:36] <maxb> leonardr: Well, let's figure out whether they ought to be in lp-(dev-)deps, and add them if so
[13:37] <maxb> What were they all?
[13:37] <leonardr> maxb: recreating the list
[13:37] <leonardr> python-geoip
[13:37] <leonardr> libapr1-dev
[13:38] <leonardr> python-subversion
[13:38] <leonardr> python-libsvn
[13:38] <leonardr> libgpgme11-dev
[13:38] <leonardr> python-all-dev
[13:38] <leonardr> i think that's it--i still can't start launchpad but it seems to be a postgres problem
[13:38] <leonardr> i can build everything
[13:39] <maxb> python-libsvn ?!
[13:40] <maxb> leonardr: I conclude that launchpad-dependencies was removed by the upgrade to lucid
[13:40] <leonardr> maxb: ok, i see what happened
[13:40] <jml> a thing that stumped me for a while yesterday was forgetting to re-enable the launchpad sources.list lines
[13:41] <leonardr> i re-enabled the sources.list lines
[13:41] <leonardr> and i ran update/upgrade
[13:41] <leonardr> but nothing says you have to have launchpad-dependencies just because it's available
[13:41] <leonardr> so it never got installed
[13:42] <maxb> So it wouldn't be a bad idea to remind people on the wiki that the Ubuntu upgrader will disable 3rd-party sources.list entries and potentially remove launchpad-developer-dependencies
[13:42] <bigjools> the upgrade has always remove lp-deps, it's a royal PITA
[13:43] <maxb> I like to do something slightly cheeky: When the upgrader pops up the box saying "I've disabled 3rd-party sources.list lines", I re-enable them before I click OK :-)
[13:44] <maxb> (only for PPAs I *know* to be correctly populated for lucid, obviously)
[13:44] <leonardr> i knew from last time the repo would be disabled but i didn't make the connection that i would have to reinstall the package
[13:56] <kfogel> noodles775: you're looking into the pqm stoppage, right?
[14:22] <bigjools> kfogel: he's at lunch, but yes he was
[14:22] <kfogel> bigjools: thx
[14:49] <noodles775> kfogel: Yes, I've got a branch on ec2 test to land for the failure earlier this morning, but buildbot has completed successfully since then.
[14:51] <kfogel> noodles775: argh
[14:51] <kfogel> noodles775: but the fix in your branch is still needed?
[14:51] <noodles775> kfogel: it seems to be timing dependent, so yes.
[14:52] <kfogel> noodles775: well, I guess this just means you can land your branch without 'testfix' :-).
[15:28] <jml> wgrant, still around?
[15:29] <jml> abentley, any movement on https://code.edge.launchpad.net/~abentley/launchpad/multiple-series-recipe/+merge/21257 ?
[15:34] <intellectronica> jml: is there no way we could figure out which packages to chart programatically?
[15:35] <jml> intellectronica, I couldn't think of anything, but I wasn't thinking very hard.
[15:35] <intellectronica> jml: my worry is that this will rot, and when we add new packages we'll forget to add them to the list
[15:35] <jml> intellectronica, we could, for example, specially mark the __init__ of packages that we care about
[15:35] <jml> intellectronica, yeah, it's a good worry.
[15:36] <jml> intellectronica, luckily, it's really obvious when you've missed a package that's been added or removed
[15:36] <jml> intellectronica, removed packages show up as disconnected islands
[15:37] <intellectronica> jml: r=me on the code. it would be nice to have a solution for this, so it's worth thinking of, and maybe filing a bug, but i can't think of a solution right now either
[15:37] <jml> intellectronica, every single file of an added package appears on the graph unless it's included in the cluster list
[15:37] <intellectronica> yeah, i understand
[15:38] <jml> intellectronica, thanks.
[15:38] <jml> intellectronica, do you reckon it's ok to land the code before I update launchpad-developer-dependencies?
[15:39] <intellectronica> jml: i think it is. it won't block anyone - the worst that can happen is that people won't be able to generate the graph
[15:39] <jml> intellectronica, cool. I'll land it now then.
[15:45] <abentley> jml, no.  mars says bjorn is still away.
[15:46] <jml> abentley, oh, if it's just waiting on BjornT then it's ok to land and then get a post-merge review, iirc.
[15:46] <jml> abentley, in your last comment you said there was stuff to discuss with the code team.
[15:48] <abentley> jml, I discussed it, we decided to leave it the way it was.  Stub did not require changing it.
[15:48] <jml> abentley, cool.
[15:48] <jml> noodles775, https://code.edge.launchpad.net/~michael.nelson/launchpad/496862-ppa-installable-binaries/+merge/21623 looks good to land. hint hint.
[15:54] <gmb> jml: Quick question: How do you feel about us adding a next_check column to BugWatch? This would allow us to have a garbo-hourly process that decides when a watch should next be checked based on how much it's error in the past; that saves us complex queries in BugTracker.getBugWatchesNeedingUpdate() and means that we keep that logic out of checkwatches.
[15:54] <gmb> jml: https://bugs.edge.launchpad.net/malone/+bug/544238 for reference
[15:54] <mup> Bug #544238: Add a next_check column to BugWatch <story-bug-watch-error-tracking> <story-reliable-bug-syncing> <Launchpad Bugs:In Progress by gmb> <https://launchpad.net/bugs/544238>
[15:54] <jml> gmb, the quick answer is that it sounds similar to the way the branch puller works and how job tables frequently work
[15:55] <gmb> Right
[15:55] <jml> gmb, so, I guess, let's do more of that and avoid gratuitous inconsistencies
[15:56] <gmb> jml: Cool. We've kinda figured out that the best way to make checkwatches play nice is to take some of its logical toys away, so I think this is a good first step. I'll knock a patch together.
[16:00] <noodles775> jml: I'd love to, but I don't know that it's ready. Did you seen the screenshot I put on the bug (linked from the MP)?
[16:02] <jml> noodles775, no I didn't.
[16:07] <gmb> jml: I've requested a review from you and Stuart for said DB patch. https://code.edge.launchpad.net/~gmb/launchpad/bugwatch-next_check-bug-544238/+merge/21869
[17:10] <persia`> How painful would it be to convince Soyuz to allow some architecture: all packages to build on non-i386?
[17:10] <bigjools> very
[17:10] <bigjools> I was looking at a bug about that yesterday
[17:10] <bigjools> an old bug :)
[17:11] <persia`> The one that openbios-sparc and openhackware have FTBFS since warty?
[17:12] <persia`> So, I know I can construct a package that can trick itself into generating an arch:all binary when told to only build arch:any packages.
[17:12] <persia`> Would it be easier just to allow Soyuz to accept the uploads, whilst still keeping the standard sbuild arguments?
[17:14] <bigjools> persia`: I'm not sure I understand
[17:15] <persia`> So, there's two issues at hand.
[17:15] <persia`> 1) when Soyuz dispatches to sbuild, it only passes -A for i386
[17:15] <persia`> 2) when Soyuz accepts an upload, it doesn't like extra packages appearing in the .changes file.
[17:16] <bigjools> why are they issues?
[17:16] <persia`> So, if we manage to force the package to build the arch:all package even without sbuild's hint, it fails to upload.
[17:17] <persia`> Oh.  Some stuff that's useful on any architecture is impossible to build on i386.
[17:17] <bigjools> that's rather special
[17:17] <bigjools> and hurts my brain to think about how that can happen
[17:18] <persia`> openbios-sparc is the classic example.
[17:18] <persia`> It can only be built on sparc, but is required to run sparc emulation for all architectures.
[17:18] <bigjools> on inspection of that name, it doesn't seem arch independent :)
[17:19] <persia`> Why not?
[17:19] <persia`> It's just a bios blob.
[17:20] <persia`> So, as data, it's useful for any data consumer, on any architecture.
[17:20] <persia`> And without it, qemu is broken.
[17:20] <bigjools> so basically it seems that soyuz is simply not designed to deal with this case
[17:20] <persia`> Yes.
[17:20] <bigjools> which makes me sad
[17:20] <persia`> But since there's only a few packages in the archive (3 or 4) that are affected, it's a low priority bug.
[17:20] <persia`> If it was easy, and just not done, I'd go find someone to do it.
[17:21] <persia`> If it's really hard, then I'm not going to be able to get some random person intersted in fixing it.
[17:21] <bigjools> I guess it would be nice to have a way of overriding the nominated arch-indep builder in the package
[17:21] <persia`> That would be an ideal way to fix it, yes.
[17:21] <bigjools> which is a lot easier than making soyuz deal with arch:all on any arch
[17:21] <persia`> We could insert X-Build-Affinity: (or similar) in the .dsc
[17:21] <bigjools> yep
[17:22] <bigjools> would be pretty easy to do I think, barring any dodgy bits of code I've forgotten about
[17:22] <maxb> Part of the problem is that there's no standard way in the deb formats for this sort of thing (leading to minging hacks like Packages-arch-specific)
[17:22] <persia`> maxb: hence the X, but that's easy to insert.
[17:23] <bigjools> p-a-s makes me want to stick needles in my eyes
[17:23] <persia`> Why do we have P-a-s again?
[17:23] <persia`> Isn't that why we have Architecture: in debian/control ?
[17:23] <maxb> Because Packages / Sources files don't contain enough info to compute whether a given source will build a given binary on all architectures or not
[17:23] <persia`> P-a-s just makes pain for porters, in my experience.
[17:24] <persia`> maxb: So?  Some stuff FTBFS.
[17:24] <maxb> huh?
[17:24] <persia`> And we go fiddle with http://qa.ubuntuwire.com/ftbfs until it builds.
[17:24] <maxb> Perhaps I should say that they don't contain enough info to compute whether a given source *ought* to build .....#
[17:26] <persia`> maxb: How doesn't the Architecture: field in the .dsc do that?
[17:27]  * maxb gets example
[17:28] <persia`> The vast majority of my experience with P-a-s has been complaining that it was too restrictive, so I may be biased on this point :)
[17:28] <maxb> So, take gdb, whose .dsc says:
[17:28] <maxb> Binary: gdb, gdb64, gdbserver, libgdb-dev, gdb-source
[17:28] <maxb> Architecture: any
[17:28] <persia`> OK.
[17:28] <maxb> But, then you look into debian/control inside the unpacked source, and you find:
[17:28] <maxb> Package: gdb64
[17:28] <maxb> Architecture: i386 powerpc s390 sparc
[17:29] <maxb> That's the information which is lost, and then kludged back in via P-a-s so that quinn-diff can figure out whether binaries are missing because they're supposed to be, or because they haven't built yet
[17:30] <persia`> So you don't actually trust the .changes files for this.
[17:31] <persia`> Which explains why it wasn't possible to trick openbios-sparc to generate arch:all in the arch:any run.
[17:31] <maxb> What's the .changes got to do with this?
[17:32] <persia`> The .changes generated from a build is *supposed* to list all the packages that the build generates.
[17:33] <persia`> So, if you trusted the changes, you wouldn't need to run quinn-diff to find out if stuff was missing.
[17:34] <maxb> I'm not necessarily sure LP uses quinn-diff at any point (in fact I doubt it), I'm more describing the original purpose of P-a-s in Debian
[17:35] <persia`> Which actually makes a lot more sense, since the archive system doesn't have trusted slave buildds.
[17:35] <maxb> P-a-s is really all about figuring out whether you need to build something, rather than processing the results of a build
[17:36] <persia`> So, in Soyuz, were we only have source-only uploads, I think it's safe to dispatch the source to the set of architectures defined in the .dsc, and accept the results it generates without P-a-s.
[17:37] <persia`> Because we know we have to build on at least that set of architectures.
[17:37] <persia`> The gdb example wouldn't generate gdb64 on amd64, armel, or ia64, but that ought be fine.
[17:39] <maxb> Another aspect of P-a-s is figuring out whether a source needs a build on a particular architecture at all - i.e. perhaps it will generate zero binary packages on some archs
[17:39] <persia`> That should never be the case if the .dsc has "any".
[17:39] <persia`> (barring build failures)
[17:40] <maxb> I know I've seen real cases of this
[17:41] <maxb> I agree P-a-s is a total hack, fwiw, I just don't think we're in a position to get rid of it without a fair bit more work
[17:43] <persia`> If you find a case where some architecture listed in the .dsc really produces 0 binaries intentionally, I'd love to know the package.
[17:44] <persia`> But for now, I'll leave that alone, and just try to find someone to do X-Build-Affinity: (unless someone gives me a better name)
[17:50] <bigjools> persia`: X-Arch-Indep-Build-Arch would be a lot more obvious
[17:51] <persia`> bigjools: Changed in my notes.  Thanks.
[17:51] <bigjools> thanks
[17:59] <jml> g'night all
[18:01] <mrevell> Goodly evelode everyone
[18:01] <mrevell> Damn, where's the close button in Lucid? :)
[18:35] <kfogel> intellectronica: ec2 builds known to be failing right now?  https://pastebin.canonical.com/29489/
[20:08] <thumper> morning
[20:32] <wgrant> jml: Hi.
[20:34] <wgrant> jml: Ah, I see the email. That branch got stuck waiting for Soyuz to answer, I think.
[20:35] <wgrant> Can somebody please land https://code.edge.launchpad.net/~wgrant/launchpad/export-detailed-binary-download-stats/+merge/21828? The ec2test failed with the two buildfarm spurious failures that have been showing up recently.
[21:05] <jml> wgrant: sure, I'll do it.
[21:06] <wgrant> jml: Thanks.
[21:07] <bdmurray> Is there anybody would could run a simple production db query for me?
[21:12] <jml> wgrant: can you do me a favour and repeat the URL? I need to use my other computer to land it.
[21:12] <wgrant> jml: https://code.edge.launchpad.net/~wgrant/launchpad/export-detailed-binary-download-stats/+merge/21828?
[21:12] <wgrant> This is why we don't use OS X.
[21:13] <jml> wgrant, :)
[21:14] <jml> wgrant, even if I bought a computer with Ubuntu pre-installed, I wouldn't have got Launchpad set up on it today
[21:16] <wgrant> kfogel: Which were those failures? TestMinTimeToNextBuilder and buildd-slave.txt?
[21:21] <wgrant> elmo: Do you know about the dsync link-dupes call in cron.publish-ftpmaster? Why would there be dupes? The archive is not yet split at that point.
[21:30] <jml> wgrant: it should be running through ec2 now.
[21:30] <wgrant> jml: It'll probably just fail spuriously again...
[21:30] <jml> wgrant: perhaps.
[21:36] <kfogel> wgrant: https://pastebin.canonical.com/29489/, but now I have a run that's been going okay for a couple of hours at least
[21:36] <wgrant> kfogel: ENOTCANONICAL
[21:37] <kfogel> wgrant: oops, sorry
[21:37] <kfogel> wgrant: one sec
[21:38] <kfogel> wgrant: http://paste.ubuntu.com/399555/, and it looked like a file existence error
[21:39] <kfogel> wgrant: I'm treating it as spurious
[21:39] <wgrant> kfogel: That's not one I've seen before.
[21:39] <wgrant> But yes, I find it unlikely that you broke that.
[21:40] <kfogel> wgrant: by the way, https://code.edge.launchpad.net/~kfogel/launchpad/78565-bug-comment-link-to-bug
[21:41] <wgrant> kfogel: Ah, good.
[21:41] <wgrant> I wish we had consistent rules for page titles, labels, breadcrumbs and root contexts.
[21:42] <kfogel> wgrant: the "comment" vs "message" thing really bugs me :-).  And that's not even in the UI.
[21:42] <kfogel> wgrant: I think everyone would agree that breadcrumbs should behave this way -- it's a bug in every case where they don't.
[21:42] <wgrant> kfogel: Well, Messages are used internally for others things as well.
[21:42] <cody-somerville> kfogel, Thats not spurious. That means your sourcecode tree is out of date.
[21:43] <cody-somerville> err... maybe spoke too soon.
[21:43] <kfogel> cody-somerville: oh, really?  I though I did an update this morning...
[21:43] <wgrant> That's the opposite error to what one would normally get.
[21:44] <wgrant> Plus ec2test updates download-cache and sourcecode before each build.
[21:44] <kfogel> cody-somerville: but in any case, it's definitely up-to-date now, and the ec2 run is working.  So even if you're not right, you're not wrong :-).
[21:45] <cody-somerville> Should file a bug on it. Ya know, the whole stop the line stuff.
[21:46] <cody-somerville> This is the system speaking to us and we're just not understanding it.
[21:46] <cody-somerville> <insert our LEAN type comments here>
[22:04] <kfogel> cody-somerville: I think it may be what noodles775 was already working on though
[22:30] <lifeless> wgrant: thats a while :P
[22:33] <wgrant> lifeless: Hm?
[22:33] <lifeless> 15:21 < wgrant> lifeless: From what I can tell, publish-distro.py takes about 20-25 minutes currently.
[22:34] <wgrant> lifeless: Yeah, and apt-ftparchive is more than 17 minutes of that.
[22:34] <wgrant> Fortunately 16 minutes of that can go away.
[22:37] <wgrant> publish-distro.py itself runs for 20 minutes, normally. 2 minutes of that is inefficient domination (which I have down to <2s), 17 minutes of that is inefficient index generation (which I have down to <60s, or <20s if we ignore bzip2 indices, which we sadly can't), and the remaining <60s is other stuff.
[22:45] <maxb> domination?
[22:49] <lifeless> wgrant: cool
[23:11] <wgrant> maxb: Domination has two related meanings in Soyuz: 1) Marking packages Superseded and 2) Marking Superseded, Deleted or Obsolete packages for removal when they have no references.