=== dkarakasilis1 is now known as dkarakasilis
=== guruprasad changed the topic of #launchpad to: Short Launchpad downtime for upgrading the database server OS coming up | Help contact: guruprasad (05:00-13:00 UTC Mon-Fri) | Launchpad is an open source project: https://dev.launchpad.net/ | This channel is logged: http://irclogs.ubuntu.com/ | User Guide https://help.launchpad.net/ | Support and spam reporting: https://answers.launchpad.net/launchpad | Contribute to our documentation:
[03:19] <guruprasad> I have stopped some Launchpad services in preparation for upgrading the OS of the Launchpad PostgreSQL database primary via a failover.
[03:47] <tsimonq2> guruprasad: Thanks. I'm sure you know, but process-upload is sad.
[03:48] <guruprasad> tsimonq2, that is expected because we are preparing for a database primary failover for the OS upgrade.
[03:48] <guruprasad> as mentioned in the topic
[03:48] <tsimonq2> No problem, thanks.
[04:26] <guruprasad> The outage has started now.
[05:04] <guruprasad> The outage lasted ~4 minutes from the time I posted about its start.
[05:04] <guruprasad> Things should be back up and running.
[05:05] <guruprasad> Will start the stopped Launchpad services soon.
[05:13] <guruprasad> And done
=== guruprasad changed the topic of #launchpad to: Help contact: guruprasad (05:00-13:00 UTC Mon-Fri) | Launchpad is an open source project: https://dev.launchpad.net/ | This channel is logged: http://irclogs.ubuntu.com/ | User Guide https://help.launchpad.net/ | Support and spam reporting: https://answers.launchpad.net/launchpad | Contribute to our documentation:
[05:37] <tsimonq2> \o/
[05:42] <RikMills> guruprasad: getting errors from the lp api
=== JanC is now known as Guest7955
=== JanC_ is now known as JanC
[05:47] <RikMills> guruprasad: https://dpaste.org/k1OTc
[05:47] <guruprasad> RikMills, thanks for the report. We received some reports just before yours and are investigating.
=== guruprasad changed the topic of #launchpad to: Launchpad database is in read-only mode and we are investigating | Help contact: guruprasad (05:00-13:00 UTC Mon-Fri) | Launchpad is an open source project: https://dev.launchpad.net/ | This channel is logged: http://irclogs.ubuntu.com/ | User Guide https://help.launchpad.net/ | Support and spam reporting: https://answers.launchpad.net/launchpad | Contribute to our documentation:
[05:52] <tsimonq2> Can confirm the same errors.
[05:57] <guruprasad> RikMills, tsimonq2, can you retry and check if you are still running into errors?
[06:00] <tsimonq2> guruprasad: Works for me.
[06:00] <RikMills> so far so good :)
=== guruprasad changed the topic of #launchpad to: Help contact: guruprasad (05:00-13:00 UTC Mon-Fri) | Launchpad is an open source project: https://dev.launchpad.net/ | This channel is logged: http://irclogs.ubuntu.com/ | User Guide https://help.launchpad.net/ | Support and spam reporting: https://answers.launchpad.net/launchpad | Contribute to our documentation:
=== JanC_ is now known as JanC
=== graber1 is now known as graber
=== JanC is now known as Guest4504
=== JanC_ is now known as JanC
[07:43] <guruprasad> Stopping some Launchpad services for a Launchpad database failover
=== guruprasad changed the topic of #launchpad to: Some downtime is expected due to an upcoming Launchpad DB failover | Help contact: guruprasad (05:00-13:00 UTC Mon-Fri) | Launchpad is an open source project: https://dev.launchpad.net/ | This channel is logged: http://irclogs.ubuntu.com/ | User Guide https://help.launchpad.net/ | Support and spam reporting: https://answers.launchpad.net/launchpad | Contribute to our documentation:
[08:44] <guruprasad> There will be an outage now for performing a failover.
[08:56] <guruprasad> The outage is over and the web application and API should be up and running now.
[08:56] <guruprasad> Will start the stopped background services after verifying that everything is okay.
[09:06] <guruprasad> I have started the background services now.
=== guruprasad changed the topic of #launchpad to: Help contact: guruprasad (05:00-13:00 UTC Mon-Fri) | Launchpad is an open source project: https://dev.launchpad.net/ | This channel is logged: http://irclogs.ubuntu.com/ | User Guide https://help.launchpad.net/ | Support and spam reporting: https://answers.launchpad.net/launchpad | Contribute to our documentation:
=== gurupras- is now known as guruprasad
=== graber1 is now known as graber
[17:06] <robinhood2021> I cannot log in to Launchpad. I get the error ID OOPS-c40a1575d3c3dd0cf1bc8cb01c655846
[17:13] <cjwatson> robinhood2021: if you don't get a reply from an LP staff member here, then email feedback@launchpad.net with that OOPS ID and they should be able to fix your account
[18:21] <tsimonq2> The script that updates PPA quotas after deleting items, has that ran recently?
[18:22] <tsimonq2> My PPA is above quota and has been for about 9 hours now. I deleted a bunch of items about 10 hours ago that should have completely resolved that.
[18:38] <tsimonq2> riscv64 builders look sad. Half a dozen idle nodes that aren't picking up new jobs.
[18:42] <cjwatson> Seems like a buildd-manager problem rather than just riscv64, since I see a similar problem on other architectures.
[18:42] <cjwatson> Maybe a consequence of the DB outage earlier?  But I'm just guessing.
[18:43] <cjwatson> Given the failover it might have fallen off a perf cliff.
[18:44] <tsimonq2> That sounds about right.
[18:45] <tsimonq2> Looks like there was just a blip, maybe that's somewhat related.
[19:23] <tsimonq2> \o/ I have much more space in my PPA again
[21:58] <tsimonq2> Looks like a fair share of the arm64 builders are stuck at "Cleaning for 5 hours 10 minutes"