[14:25] <DirtyToby> I can't get onto any Ubuntu site. Launchpad just keeps say "oh no!"
[14:25] <DirtyToby> "uh oh!" it turns out
[14:26] <DirtyToby> I can't be alone in this, surely?
[14:32] <mdeslaur> red is the new black https://status.canonical.com/
[14:33] <mdeslaur> not sure who Major Outage is, but he should get relieved of duty
[14:35] <tsimonq2> "What is your major malfunction?"
[14:39] <Guest83> wagafak
[14:39] <Guest83> i am have problem
[14:40] <Guest83> Uh oh!
[14:40] <Guest83> Something has gone wrong. We're sorry!
[14:40] <Guest83> If we are in the middle of an update, Launchpad will be back in a couple of minutes. Otherwise, we are working to fix the unexpected problems. Check @launchpadstatus on Twitter or @launchpadstatus@ubuntu.social on Mastodon for updates.
[14:40] <Guest83> If the problem persists, let us know in the #launchpad IRC channel on libera.chat.
[14:40] <Guest83> Technically, the load balancer took too long to connect to an application server.
[14:40] <Guest83> Reload this page or try again in a few minutes
[14:40] <tsimonq2> Guest83: watch https://status.canonical.com/ :)
[14:40] <tsimonq2> That should probably in the topic here... :)
[14:40] <tsimonq2> *be in
[14:40] <Guest43> just came to say the same thing :)
[14:41] <Kennybobs> oh, i'm in
[14:43] <Guest83> launchpad is doing a big fuck
[14:44] <tsimonq2> !language | Guest83 
[14:44] <tsimonq2> Guest83: tl;dr pls chill we know :)
[14:50] <Guest83> launchpad is a big raping my ass
[14:50] <Guest43> well... someone is mad because they don't know how to manually install stuff
[14:50] <tsimonq2> !ops | please keep an eye out, this channel will be high traffic until the outage is done
[14:50] <Guest83> Guest43 educate me big boy
[14:50] <Guest43> aint nobody got time for that :) go to school or something
[14:50] <Guest83> k thx smartman
[14:50] <Guest83> ETA till outage is done???
[14:50] <Guest83> Welcome back!
[14:52] <Guest83> k its b ack
[14:52] <Guest83> thank you open sauce people
[14:52] <Guest83> minus guest43
[14:52] <Guest83> bye
[14:54] <fungi> yep, working for me again. thanks everyone for all your hard work, as always!!!
[15:15] <Kennybobs> builds failing and no logs. i assume it's related - unable to upload logs, etc.
[15:15] <Kennybobs> new bugs won't process either
[15:16] <tsimonq2> Kennybobs: Sounds related, yeah, LP was down
[15:16] <Kennybobs> i shall try once more
[15:21] <KenSharp> darn. down again.
[15:24] <tsimonq2> The more we post, the less time they have to fix the issues ;P
[15:25] <nesda> Please be patient, we have people looking into fixing the outage as we speak. We will notify here once it is all been fixed
[15:25] <tsimonq2> <3
[15:26] <KenSharp> The cake is always a lie
[16:26] <nesda> Launchpad is back up and seems stable, although there are still some work going on, and we're still keeping an eye to understand if all services are back up. I will keep the header up until we are certain that the issues are fully fixed
[16:30] <nesda> Also note that there is some backlog of processes (code imports being one of such), so things might take a little longer than usual before they get back to normal. The queues seem to generally be decreasing
[16:43] <KenSharp> Some of the build machines seem stuck. Even cancelling a build seems stuck on cancelling.
[16:48] <KenSharp> Oh, wow, nearly every machine is stuck on "Cleaning for 40-odd minutes"
[16:51] <nesda> Yup, I was hoping that would be coming back down automatically after the reboots, but doesn't seem to be the case. I'm currently looking into that
[16:51] <nesda> Thanks for reporting!
[17:17] <kisak> those cleaning for X minutes indicators are misleading since someone put a hack in to keep build uploads from failing. It's likely longer than 60 minutes.
[17:39] <KenSharp> things are building now anyway
[18:30] <nesda> We still have some builder issues that require addressing.
[18:30] <nesda> Unfortunately there seems to be a second outage going on, so Launchpad is down again :(
[19:57] <KenSharp> Indeed :(
[20:23] <enigma9o7> rmadison no longer 503, but launchpad website still issue, guess it's half way fixed
[20:25] <clinton-fung> Yes it's still in flux
[20:25] <clinton-fung> I am currently on a call with the SREs that are looking into it
[20:39] <clinton-fung> \/4
[21:03] <kisak> Looks like I made a mistake and pushed a PPA build while in the last snafu and there's been no ingress confirmation email. Push again or wait, hmm.
[21:04] <KenSharp> Uploading isn't working for me
[21:07] <kisak> my stuff isn't urgent, so to the backburner it goes.
[21:11] <KenSharp> bedtime i reckon
[21:19] <clinton-fung> I'm told the infra is basically back to normal; we are now just looking into whatever fallout occurred after the last couple of hours.