[12:04] <slyon> ubuntu-qa: I'm seeing plenty of "Traceback: The read operation timed out" when trying to queue autopkgtests. Is there anything we can do about it?
[12:06] <slyon> seems to be flaky
[12:29] <schopin> I think the HL is qa-help
[12:38] <paride> yup. andersson1234 ^^ is that something you are already aware of?
[12:40] <Skia> I'm taking a look
[12:42] <Skia> okay, rabbitmq was restarted by our magical watchdog "maybe-restart-rabbitmq" about 10 minutes ago, that could explain the flakyness if it was almost dead
[13:04] <Skia> slyon: I don't see errors anymore, do you confirm that it's working on your side too?
[13:05] <slyon> Skia: seems to be working again for me
[13:05] <Skia> great
[16:38] <andersson1234> mattermost is down, so let's co-ordinate here if any fires need fighting :) 
[16:52] <andersson1234> bdmurray: How often does update_excuses update?
[16:53] <bdmurray> andersson1234: hours
[16:54] <andersson1234> okay. noted
[16:54] <bdmurray> https://ubuntu-archive-team.ubuntu.com/proposed-migration/log/noble/2024-04-04/
[16:54] <bdmurray> yeah, looks like its avergaing 2 or 2.5 hours
[16:55] <Skia> just so you know the current state, bos03 armhf workers 04 and 06 have been redeployed, but are still missing some LXD images. They both at least have noble, but can be taken down by other releases if bad scheduling happens. At least it's only a matter of restart, not redeploy.
[16:55] <Skia> missing images are in the process of being rebuilt, I'll keep monitoring that
[16:56] <bdmurray> Copy that. Thanks.
[16:58] <Skia> also, `~/juju-config/autopkgtest-lxd-worker_lxd-remotes.yaml` has up-to-date config, but not the service bundle yet, I'll push a commit soon with that
[16:58] <Skia> the advantage of the juju config file is that it has comments on the current status of remotes
[17:05] <andersson1234> we could make a script which produces yaml files for each juju config option, then our workflow could be: modify service bundle -> use script to produce the yaml files -> update config option with a juju config call 
[17:05] <andersson1234> then we'd avoid having the yaml files and service bundle out of sync 
[17:06] <bdmurray> I do really wonder if we can just use the service-bundle but with the right juju config options
[17:08] <Skia> 6 is all good, 4 is missing focal and mantic, in an retry loop for building; I've pushed the current config in the service-bundle on master, and now I'm dropping off; EOW o7
[17:08] <Skia> forgot that: expect worker count at 29
[17:09] <Skia> and the reported one in error in the KPI is a remain that didn't get cleaned up by `juju config`, but shouldn't exist anymore
[17:25] <andersson1234> `eod` see you tomorrow, and you on monday skia :)