| === morphis93417 is now known as morphis | ||
| cpaelzer | wgrant: cjwatson: sorry to bother you again but since ppc usually is fast and >50% are in cleaning state I wanted to ask if ppc64el builders need a bump again | 09:15 |
|---|---|---|
| cjwatson | they are being automatically stabbed | 09:15 |
| cjwatson | if they're not working then the cloud is busted | 09:15 |
| cpaelzer | a rather high queue time on arm can happen, but if ppc > the rest that would be uncommon | 09:15 |
| cpaelzer | cjwatson: oh interesting | 09:15 |
| cjwatson | I cannot look more right now because I'm in a sprint | 09:16 |
| cjwatson | the automatic thing was the best I could do for the time being | 09:16 |
| cpaelzer | np, thanks for the info cjwatson | 09:16 |
| tjaalton | hi, wonder why linux-firmware sru uploads still show diff's as pending, like here https://launchpad.net/ubuntu/disco/+queue?queue_state=1 | 09:16 |
| tjaalton | after 11h | 09:16 |
| cpaelzer | maybe ppc just has quite some leftover from the archive rebuild | 09:16 |
| cjwatson | cpaelzer: no there is a real problem with the cloud - we just haven't had a chance to work with IS to fix it | 09:16 |
| cjwatson | tjaalton: stuck job I guess, give me a bit | 09:17 |
| tjaalton | cjwatson: thanks, bionic/cosmic too | 09:18 |
| cjwatson | yeah, series won't matter when the job queue is stuck :) | 09:18 |
| cjwatson | apparently backlogged since Friday, I should check my cronspam more often | 09:18 |
| cjwatson | so it will take quite a while to clear | 09:19 |
| cjwatson | clearing gradually now | 09:20 |
| tjaalton | great, thanks | 09:20 |
| Eickmeyer | Is the arm build farm stuck or are we seriously looking at a long queue with hardly any movement on the builders? | 18:48 |
| Eickmeyer | cjwatson: Got any ideas? | 20:30 |
| tjaalton | apparently things are stuck again | 20:46 |
| teward | tjaalton: i already poked Canonical Sysadmin about it, since there was no response here | 20:48 |
| tjaalton | cool | 20:49 |
| tjaalton | I'm waiting for some sru debdiffs to be generated | 20:49 |
| tjaalton | since ~12h ago | 20:49 |
| cjwatson | tjaalton: I told you already that there was a huge backlog | 20:54 |
| cjwatson | it is not stalled and is still making progress | 20:54 |
| cjwatson | I poked the build farm some | 20:56 |
| cjwatson | people seriously need to chill about queue lengths during a test rebuild though | 20:57 |
| cjwatson | looks like a chunk of builders failed to reset during the unexpected cert expiry earlier - ppa-reset tried to mark the builders as active and failed with cert errors | 20:58 |
| cjwatson | so that's a known cause now fixed | 20:58 |
| tjaalton | cjwatson: ok, didn't realize it was that huge, sorry | 21:00 |
| cjwatson | I can't easily see how long, I'm afraid, but with four days of outage I'd expect it to take ... a while | 21:00 |
| tjaalton | I was just meant to review the sru's before being off again | 21:00 |
| Eickmeyer | cjwatson: Sorry for the noise then. I didn't realize there had been an outage, and I can't tell if it's a test rebuild. | 21:04 |
| cjwatson | yeah it's ok, I just came back from dinner to find that I had pings about the same thing in like five channels | 21:09 |
| cjwatson | which was a bit WHOA | 21:09 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!