[04:28] <wgrant> WTF is ~itiseasy-org, and why has it uploaded several hundred packages overnight?
[04:28] <wgrant> Like we needed more in the queue...
[06:08] <lifeless> wgrant: dunno
[06:21] <wgrant> lifeless: Could you land https://code.edge.launchpad.net/~wgrant/launchpad/replace-archiveuploader-doctests-0/+merge/30850? You approved it a week ago.
[06:27] <lifeless> kicking it off
[06:28] <lifeless> wgrant: you didn't find anyone to do it for a week ?
[06:29] <wgrant> lifeless: No, I just forgot about it.
[06:29] <lifeless> :)
[06:30] <wgrant> Thanks.
[06:31] <wgrant> Do we have working feature flag support yet?
[06:31] <wgrant> I saw some stuff land.
[06:31] <wgrant> And I'd like to disable something with it, if it does indeed exist...
[06:31] <lifeless> db-devel only
[06:32] <lifeless> could probably port the interface to devel
[06:32] <lifeless> with some care
[06:32] <wgrant> No need.
[06:32] <lifeless> and then your thing would be disabled by it but able to be controlled on staging & next release
[06:32] <wgrant> There's only a few days left until the freeze.
[06:33] <lifeless> last freeze ever!
[06:33] <wgrant> Oh?
[06:33] <wgrant> We're moving to the new merge workflow for 10.09?
[06:33] <wgrant> Well, what was 10.09.
[06:33] <lifeless> I think all the bits are in place
[06:34] <lifeless> if not then very very close
[06:34] <wgrant> Wow.
[06:34] <wgrant> That was swift.
[06:34]  * wgrant whines about tree builds taking far too long.
[06:35] <wgrant> 4 minutes and counting...
[06:35] <lifeless> wgrant: using ?
[06:35] <lifeless> wgrant: well the basic bits are:
[06:36] <wgrant> lifeless: The initial make in a new branch.
[06:36] <lifeless>  - a daily staging [trivialish]
[06:36] <lifeless>  - a way to disable stuff [feature flags, rough but usable]
[06:36] <lifeless>  - new QA toolchain/automation [done, I believe]
[06:37] <wgrant> Right, I wan't aware that the last bit was even started.
[06:37] <wgrant> So the new system will take all the branches in the queue, merge them, test them as a whole, then commit them?
[06:39] <lifeless> no
[06:39] <lifeless> thats totally different
[06:39] <wgrant> Oh.
[06:39] <wgrant> So just the new QA workflow, not the new merge workflow?
[06:39] <lifeless> thats an optimisation for landing stuff
[06:39] <wgrant> Ah.
[06:39] <wgrant> I se.
[06:39] <lifeless> https://dev.launchpad.net/LEP/ReleaseFeaturesWhenTheyAreDone
[06:40] <lifeless> https://dev.launchpad.net/MergeWorkflowDraft specifically
[06:40] <lifeless> https://dev.launchpad.net/MergeWorkflowDraft?action=AttachFile&do=view&target=merge-workflow-draft-2.jpeg is the thing we drew
[06:40] <lifeless> or maris drew, I should say
[06:41] <wgrant> Hm. that looks a lot simpler than the last draft I saw.
[06:41] <lifeless> if the last thing you was before the epic, then yes
[06:41] <wgrant> Right.
[06:41] <lifeless> We revisited the entire proposal there
[06:43] <lifeless> we also need a patch to make 'edge' behaviour depend on the vhost, not on the confi
[06:43] <lifeless> config
[06:43] <lifeless> I don't think anyone has done that yet, would be nice to do.
[06:45] <lifeless> though the meaning of edge is going to erode pretty quickly
[06:48] <wgrant> Yes.
[06:49] <wgrant> I also suppose that the automation of service restarts should make DB upgrades a whole lot quicker.
[06:49] <lifeless> oh yeah thats the other key thing, but its necessary for ratcheting up the frequency, not for switching at all
[06:50] <wgrant> Yep.
[11:13] <lifeless> wgrant: nuts
[15:47] <wgrant> lifeless: Um, devel is broken on 2.5?
[15:47] <wgrant> Ew.
[15:50] <wgrant> I don't see how that happens, unless people aren't ec2ing branches before landing...
[16:01] <jelmer> hey wgrant
[16:03] <wgrant> Morning jelmer.
[16:03] <jelmer> wgrant, it's broken how?
[16:03] <wgrant> jelmer: test_sourcepackagerecipe.py doesn't appear to import with_statement.
[16:04] <jelmer> ah
[16:04] <jelmer> wgrant, not everybody uses ec2, some people run make test locally
[22:17] <wgrant> Can someone please EC2 https://code.edge.launchpad.net/~wgrant/launchpad/replace-archiveuploader-doctests-0/+merge/30850? It was caught in the devel breakage last night.
[22:17] <wgrant> So, the build farm has currently been unusable for five days.
[22:17] <wgrant> I am failing to see how this is not a critical service issue.
[22:18] <wgrant> One of Launchpad's most popular features is broken.
[22:22] <lifeless> wgrant: kicked it off again
[22:22] <lifeless> wgrant: uhm, I think its an important issue.
[22:22] <wgrant> It happens time and time again.
[22:23] <lifeless> wgrant: builds > builders for extended periods, + starvation events ?
[22:24] <wgrant> lifeless: The sustained reduction of the build farm is an issue.
[22:24] <wgrant> As well as prioritisation during extended starvation events.
[22:24] <wgrant> But new builds will be waiting for nearly four days.
[22:25] <lifeless> so, the whole scoring thing is what drives starvation
[22:25] <lifeless> its implementation not policy, and because its complex few people can actually see clearly to define a policy
[22:25] <wgrant> Scoring doesn't drive starvation. It just means that the dailies build before the realtime packages.
[22:25] <wgrant> So it exacerbates the effects.
[22:26] <lifeless> I disagree, but this is an assessment of the impact of the design choice; we both agree that the behaviour is undesirable
[22:26] <wgrant> Starvation is going to occur whatever the ordering of the builds.
[22:26] <wgrant> It's just going to be less noticable if realtime stuff builds first.
[22:27] <lifeless> by realtime do you mean 'manually uploaded single packages' ?
[22:27] <wgrant> Yes.
[22:27] <wgrant> Things that aren't going to be unnoticed if they don't build for a few days.
[22:27] <wgrant> ie. not dailies, and not rebuilds.
[22:27] <lifeless> sure
[22:28] <lifeless> so short term, we can futz with the scoring algorithms
[22:28] <lifeless> builder shortages, and inefficient use of builders, exacerbate the situation
[22:28] <wgrant> Longer-term: where did all those builders go for two or three days?
[22:28] <lifeless> QA
[22:28] <lifeless> only a fraction of the PPA buildds are dedicated to LP
[22:28] <wgrant> Odd that it takes so long.
[22:28] <wgrant> I know, yes.
[22:29] <wgrant> But normally they're only gone for a day at most.
[22:29] <wgrant> Except when they're taken for mirrors.
[22:29] <lifeless> the handover isn't flawless
[22:29] <lifeless> so one possibility is a failed handover and they end up in limbo
[22:30] <lifeless> a system wide failure in the handover would explain a batch not coming back
[22:31] <lifeless> anyhow, with bigjools fix to the sequences nearly here
[22:31] <lifeless> your branch is testing
[22:31] <wgrant> Thanks.
[22:31] <lifeless> s/sequences/sequencer/ we should get much more juice from the buildds, and be in a good position to talk about expanding the farm with less transient resources
[22:32] <wgrant> Right.
[22:32] <wgrant> Although buildd-manager isn't toooo bad with this few builders.
[23:26] <james_w> if someone is bored: https://code.edge.launchpad.net/~james-w/launchpad/buildout-doc/+merge/31465
[23:27] <james_w> wgrant: has a scheduler based on quotas for people/archives been discussed?
[23:28] <wgrant> james_w: Discussed? No.
[23:28] <wgrant> It gets mentioned occasionally.
[23:28] <wgrant> But nothing comes of it.
[23:29] <james_w> what do we need to do to get a new scheduling algorithm?
[23:30] <wgrant> 1) Adjust findBuildCandidate
[23:30] <james_w> is it just a case of trying something, or does there need to be a discussion about what direction to go in?
[23:30] <wgrant> 2) Adjust dispatch time estimation
[23:30] <wgrant> I don't know.
[23:30] <wgrant> 1) is easy.
[23:30] <wgrant> 2) is very difficult.
[23:30] <wgrant> So it's not as easy as it was two years ago :(
[23:31] <wgrant> Although, if the current trend continues we can just replace the estimation with 'give up'.
[23:32] <james_w> p.s. I have an ArchiveCollection running through ec2 to catch any issues before review, and I'm currently taking a hatchet to some soyuz tests to remove sampledata uses
[23:32] <james_w> yay for a bored weekend
[23:32] <wgrant> Ooh, excellent.
[23:33] <wgrant> I've been meaning to see how many tests I can get to not use sampledata simply by tweaking STP.
[23:35] <james_w> yeah, I'd never seen it before, but I'm mainly stopping things from using that
[23:35] <james_w> the factory is just about as good for the uses I have seen so far
[23:36] <wgrant> Oh.
[23:36] <wgrant> STP is use pretty extensively throughout the test suite (including brand new stuff).
[23:36] <james_w> yeah
[23:37] <wgrant> It probably should be merged into the factory, but it does lots of stuff that the factory does not.
[23:38] <wgrant> I'm also not a fan of the factory's monolithic approach.
[23:38] <wgrant> It doesn't seem to be very compatible with the ongoing attempts at modularity.
[23:42] <james_w> indeed
[23:42] <james_w> I've only gone through about 10 methods so far, and they made little use of the publisher I think
[23:43] <wgrant> For a lot of things, makeBinaryPackagePublishingHistory will indeed suffice.