[02:47] <teward> cjwatson: got an email from the build bot that it failed, looks like the timeout was RabbitMQ related, is this a failure in buildbot?
[03:03] <wgrant> teward: Yeah, known flakiness. I've retried it.
[05:46] <teward> wgrant: ah, thanks.  So not my fault :)
[05:46] <teward> looks like it passed.
[05:47] <teward> (and yes I can't sleep >.<  Time to try again with melatonin now)
[09:19] <tomwardill> Fix an OOPS if you search for OCI projects with no term: https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/383773
[09:25] <cjwatson> r=me, one comment
[09:31] <tomwardill> oh, it's not different, I just did a silly
[11:31] <cjwatson> Hmm, looking at the perennial problem where you need to manually clear launchpadlib's wadl cache
[11:31] <cjwatson> I don't think it's a client-side problem
[11:31] <cjwatson> At the server end we seem to be somehow changing the content without changing the Etag
[11:31] <cjwatson> So no wonder the poor client is confused
[11:35] <tomwardill> that would do it
[11:36] <cjwatson> Hm, this only seems to be the case on dogfood though.  I wonder if Apache is doing something in front of all the other instances?
[11:36] <cjwatson> Though I'm sure I remember seeing this problem across production upgrades too
[11:37] <cjwatson> Right now the etag on dogfood is literally the sha1 of '\0application/vnd.sun.wadl+xml' though
[11:44] <wgrant> The etag config is customised to ensure consistency across the FEs
[11:44] <wgrant> I forget what it is, maybe mtime and size
[11:50] <cjwatson> I'll see if I have to do anything manual after the next deployment
[11:58] <cjwatson> https://code.launchpad.net/~cjwatson/lazr.restful/fix-wadl-etag/+merge/383777
[12:04] <wgrant> That MP explains a lot.
[15:08] <tomwardill> Looks like the librarian file upload deduplication either doesn't work or wasn't actually implemented
[15:08] <tomwardill> (or I've lost a bunch of code somewhere)
[15:13] <tomwardill> fixing
[15:38] <tomwardill> oh no, I think I'm just being silly
[15:39] <tomwardill> cjwatson: remind me where the code that pulls files from the builder lives?
[15:46] <tomwardill> aha, buildbehaviour, of course
[15:50] <tomwardill> ah, right,that code does exist, I was just looking in the wrong place
[15:50] <tomwardill> (for future travellers, it's in the buildbehaviour, which ensures that all the files exist from the librarian on disk, ready for the uploadjob)
[15:55] <tomwardill> although the scope it's checking for existing files might be too limited there
[15:55]  * tomwardill stares at it a bit harder
[15:56] <cjwatson> lp.buildmaster.interactor too IIRC
[15:56] <cjwatson> Well, in fact that's more about causing the builder to pull files from LP
[15:56] <cjwatson> There's also lp.archiveuploader.ocirecipeupload
[15:57] <tomwardill> yeah, that's where I was originally
[15:58] <tomwardill> by the code in ocirecipebuildbehaviour, we're only looking for existing layer files within the same build (L13(
[15:58] <tomwardill> *L139
[15:58] <tomwardill> which isn't correct, or useful
[16:02] <tomwardill> cjwatson: so, the buildbehaviour grabs the (existing) file from the librarian and puts it on disk, so the uploadjob can reupload it. This seems a bit weird
[16:03] <cjwatson> I think this was partly in case of racing with GC, but I'm sure it's possible to do better, yes
[16:04] <tomwardill> yeah