[00:06] <JanC> so what happened then?
[00:09] <sarnold> we lost a race condition in the way the archive is updated in the rsync source and when the rsync copies it all to the machines that serve it
[00:17] <JanC> you mean the rsync ran in the middle of an update?
[00:17] <oerheks> yes, probably, but that would not explain the missing key
[00:19] <JanC> I suppose if you copy one half before an update & one half after that can result in all sorts of weird errors
[00:21] <sarnold> JanC: and it looked like it managed to catch it between generating the InRelease and Release.gpg files or something just like that. unfortunately it also doesn't look like there's an easy solution here :(
[00:23] <JanC> well, the solution is easy in theory, but depending on how things are managed currently, and work together (or not) currently it might be a lot of work & changes   :)
[00:26] <sarnold> exactly :)
[00:26] <oerheks>  a glitch
[00:26] <JanC> basically, you'll need some sort of lock or such to make sure both can't run in parallel... or you need "double buffering" on at least one side & an atomic switch
[00:26] <oerheks> i was waiting for the rsync update
[00:27] <oerheks> so, maybe that was involved
[00:27] <oerheks> just thinking out loud
[00:28] <JanC> it's not really a glitch, it's a concurrency bug  :)
[00:31] <sarnold> one that'd be a real challenge to hit intentionally, but when we do this every hour of every day for years on end, it's going to happen again ..