=== morphis93417 is now known as morphis
[09:15] <cpaelzer> wgrant: cjwatson: sorry to bother you again but since ppc usually is fast and >50% are in cleaning state I wanted to ask if ppc64el builders need a bump again
[09:15] <cjwatson> they are being automatically stabbed
[09:15] <cjwatson> if they're not working then the cloud is busted
[09:15] <cpaelzer> a rather high queue time on arm can happen, but if ppc > the rest that would be uncommon
[09:15] <cpaelzer> cjwatson: oh interesting
[09:16] <cjwatson> I cannot look more right now because I'm in a sprint
[09:16] <cjwatson> the automatic thing was the best I could do for the time being
[09:16] <cpaelzer> np, thanks for the info cjwatson
[09:16] <tjaalton> hi, wonder why linux-firmware sru uploads still show diff's as pending, like here https://launchpad.net/ubuntu/disco/+queue?queue_state=1
[09:16] <tjaalton> after 11h
[09:16] <cpaelzer> maybe ppc just has quite some leftover from the archive rebuild
[09:16] <cjwatson> cpaelzer: no there is a real problem with the cloud - we just haven't had a chance to work with IS to fix it
[09:17] <cjwatson> tjaalton: stuck job I guess, give me a bit
[09:18] <tjaalton> cjwatson: thanks, bionic/cosmic too
[09:18] <cjwatson> yeah, series won't matter when the job queue is stuck :)
[09:18] <cjwatson> apparently backlogged since Friday, I should check my cronspam more often
[09:19] <cjwatson> so it will take quite a while to clear
[09:20] <cjwatson> clearing gradually now
[09:20] <tjaalton> great, thanks
[18:48] <Eickmeyer> Is the arm build farm stuck or are we seriously looking at a long queue with hardly any movement on the builders?
[20:30] <Eickmeyer> cjwatson: Got any ideas?
[20:46] <tjaalton> apparently things are stuck again
[20:48] <teward> tjaalton: i already poked Canonical Sysadmin about it, since there was no response here
[20:49] <tjaalton> cool
[20:49] <tjaalton> I'm waiting for some sru debdiffs to be generated
[20:49] <tjaalton> since ~12h ago
[20:54] <cjwatson> tjaalton: I told you already that there was a huge backlog
[20:54] <cjwatson> it is not stalled and is still making progress
[20:56] <cjwatson> I poked the build farm some
[20:57] <cjwatson> people seriously need to chill about queue lengths during a test rebuild though
[20:58] <cjwatson> looks like a chunk of builders failed to reset during the unexpected cert expiry earlier - ppa-reset tried to mark the builders as active and failed with cert errors
[20:58] <cjwatson> so that's a known cause now fixed
[21:00] <tjaalton> cjwatson: ok, didn't realize it was that huge, sorry
[21:00] <cjwatson> I can't easily see how long, I'm afraid, but with four days of outage I'd expect it to take ... a while
[21:00] <tjaalton> I was just meant to review the sru's before being off again
[21:04] <Eickmeyer> cjwatson: Sorry for the noise then. I didn't realize there had been an outage, and I can't tell if it's a test rebuild.
[21:09] <cjwatson> yeah it's ok, I just came back from dinner to find that I had pings about the same thing in like five channels
[21:09] <cjwatson> which was a bit WHOA