[01:39] <tsimonq2> Excellent.
[08:50] <Jasem[m]> wgrant: I noticed builds were _really_ fast now.. you guys upgraded the machines or this is just an artifact?
[08:53] <wgrant> Jasem[m]: Hm, nothing should have changed there for a month or so.
[08:53] <wgrant> What in particular seems fast?
[08:53] <Jasem[m]> wgrant: I just ran a build request and it finished in less than 10 mins which is abnormal (usually takes an hour)
[08:54] <Jasem[m]> not the build time itself, but it time until it starts building and uploading.
[08:54] <Jasem[m]> I guess maybe because queues were largely empty
[08:56] <wgrant> Jasem[m]: i386/amd64 queues at most times of day are empty. Unless you upload at a consistently bad time every day, you should have seen significant queue latency.
[08:56] <wgrant> (except for the past week or so)
[08:57] <Jasem[m]> wgrant: ah okay.. what happened recently that created the build delays?
[08:59] <wgrant> Jasem[m]: One of our internal clouds ran into some load problems, but we've hopefully fixed that today.
[22:38] <tsimonq2> cjwatson, wgrant: Seems like it's lcy01's turn to want to have all builders on "Cleaning:
[22:38] <tsimonq2> s/g:/g"/
[22:39] <wgrant> tsimonq2: Yep. Different issue there, but we're trying to work out what's gone wrong.
[22:41] <tsimonq2> wgrant: Glad to hear, thanks.
[22:41] <tsimonq2> (glad to hear that the issues are being worked out, not so much the issue itself...)
[22:41] <wgrant> Heh
[22:42] <acheronuk> thanks. murphy's law dictates it always seems to happen just before we or others kick off nightly builds!
[22:42] <wgrant> lgw01's issue should be solidly solved. Just ended up overloaded as LP started sharing the new cloud with other heavy workloads.
[22:42] <wgrant> lcy01's problem is more sinister and OpenStacky.
[22:43]  * acheronuk cheers for lgw01 at least
[22:51] <tsimonq2> wgrant: Do the problems keep /stacking up/?
[22:51]  * tsimonq2 runs
[22:51] <wgrant> tsimonq2: BAD
[22:51] <tsimonq2> XD
[22:56] <acheronuk> BAD^AWFUL