=== ChanServ changed the topic of #launchpad to: Help contact: ialmeida (08:00-16:00 UTC, Mon-Fri) | Launchpad is an open source project: https://dev.launchpad.net/ | This channel is logged: http://irclogs.ubuntu.com/ | User Guide https://help.launchpad.net/ | Support and spam reporting: https://answers.launchpad.net/launchpad === fossfreedom__ is now known as fossfreedom_ === teward_ is now known as teward === frickler_ is now known as frickler === JanC is now known as Guest8432 === JanC_ is now known as JanC [14:25] I can't get onto any Ubuntu site. Launchpad just keeps say "oh no!" [14:25] "uh oh!" it turns out [14:26] I can't be alone in this, surely? [14:32] red is the new black https://status.canonical.com/ [14:33] not sure who Major Outage is, but he should get relieved of duty [14:35] "What is your major malfunction?" === DirtyToby is now known as Kennybobs [14:39] wagafak [14:39] i am have problem [14:40] Uh oh! [14:40] Something has gone wrong. We're sorry! [14:40] If we are in the middle of an update, Launchpad will be back in a couple of minutes. Otherwise, we are working to fix the unexpected problems. Check @launchpadstatus on Twitter or @launchpadstatus@ubuntu.social on Mastodon for updates. [14:40] If the problem persists, let us know in the #launchpad IRC channel on libera.chat. [14:40] Technically, the load balancer took too long to connect to an application server. [14:40] Reload this page or try again in a few minutes [14:40] Guest83: watch https://status.canonical.com/ :) [14:40] That should probably in the topic here... :) [14:40] *be in [14:40] just came to say the same thing :) [14:41] oh, i'm in [14:43] launchpad is doing a big fuck [14:44] !language | Guest83 [14:44] Guest83: Please avoid any language that may be considered offensive, including acronyms and obfuscation of such - also see https://wiki.ubuntu.com/IRC/Guidelines || The main channels are English only, for other languages, see https://wiki.ubuntu.com/IRC/ChannelList [14:44] Guest83: tl;dr pls chill we know :) [14:50] launchpad is a big raping my ass [14:50] well... someone is mad because they don't know how to manually install stuff [14:50] !ops | please keep an eye out, this channel will be high traffic until the outage is done [14:50] please keep an eye out, this channel will be high traffic until the outage is done: Help! lifeless, flacoste, jml, joey [14:50] Guest43 educate me big boy [14:50] aint nobody got time for that :) go to school or something [14:50] k thx smartman [14:50] ETA till outage is done??? [14:50] Welcome back! [14:52] k its b ack [14:52] thank you open sauce people [14:52] minus guest43 [14:52] bye [14:54] yep, working for me again. thanks everyone for all your hard work, as always!!! [15:15] builds failing and no logs. i assume it's related - unable to upload logs, etc. [15:15] new bugs won't process either [15:16] Kennybobs: Sounds related, yeah, LP was down [15:16] i shall try once more [15:21] darn. down again. [15:24] The more we post, the less time they have to fix the issues ;P [15:25] Please be patient, we have people looking into fixing the outage as we speak. We will notify here once it is all been fixed [15:25] <3 [15:26] The cake is always a lie === ChanServ changed the topic of #launchpad to: We are experiencing issues due to an outage. We are looking into it, and will notify the channel once everything is in the clear. || Help contact: ialmeida (08:00-16:00 UTC, Mon-Fri) | Launchpad is an open source project: https://dev.launchpad.net/ | This channel is logged: http://irclogs.ubuntu.com/ | User Guide https://help.launchpad.net/ | Support and spam reporting: https://answers.l === bdmurray_ is now known as bdmurray [16:26] Launchpad is back up and seems stable, although there are still some work going on, and we're still keeping an eye to understand if all services are back up. I will keep the header up until we are certain that the issues are fully fixed [16:30] Also note that there is some backlog of processes (code imports being one of such), so things might take a little longer than usual before they get back to normal. The queues seem to generally be decreasing [16:43] Some of the build machines seem stuck. Even cancelling a build seems stuck on cancelling. [16:48] Oh, wow, nearly every machine is stuck on "Cleaning for 40-odd minutes" [16:51] Yup, I was hoping that would be coming back down automatically after the reboots, but doesn't seem to be the case. I'm currently looking into that [16:51] Thanks for reporting! [17:17] those cleaning for X minutes indicators are misleading since someone put a hack in to keep build uploads from failing. It's likely longer than 60 minutes. [17:39] things are building now anyway [18:30] We still have some builder issues that require addressing. [18:30] Unfortunately there seems to be a second outage going on, so Launchpad is down again :( === locutusofborg_ is now known as locutusofborg [19:57] Indeed :( === rs20099 is now known as rs2009 [20:23] rmadison no longer 503, but launchpad website still issue, guess it's half way fixed [20:25] Yes it's still in flux [20:25] I am currently on a call with the SREs that are looking into it [20:39] \/4 [21:03] Looks like I made a mistake and pushed a PPA build while in the last snafu and there's been no ingress confirmation email. Push again or wait, hmm. [21:04] Uploading isn't working for me [21:07] my stuff isn't urgent, so to the backburner it goes. [21:11] bedtime i reckon [21:19] I'm told the infra is basically back to normal; we are now just looking into whatever fallout occurred after the last couple of hours.