[08:28] <TJ-> Seeing repeated timeout errors with OOPS last 5 minutes. (Error ID: OOPS-12be8b5060bbc1f92318e126ad954cb2)
[08:28] <ubot5> https://oops.canonical.com/?oopsid=OOPS-12be8b5060bbc1f92318e126ad954cb2
[08:45] <TJ-> cristiangsp: Seeing repeated timeout errors with OOPS last 5 minutes. (Error ID: OOPS-12be8b5060bbc1f92318e126ad954cb2)
[08:45] <ubot5> https://oops.canonical.com/?oopsid=OOPS-12be8b5060bbc1f92318e126ad954cb2
[08:45] <TJ-> cristiangsp: e.g. for https://launchpad.net/ubuntu/+source/linux
[08:57] <cristiangsp> yep, I can reproduce it too, checking...
[09:01] <cristiangsp> it seems that the overview section is fetching too much data which triggers the timeout. There is any specific are that you are interested on?
[09:02] <cristiangsp> https://code.launchpad.net/ubuntu/+source/linux seems loading fine for example
[09:03] <cristiangsp> (just trying to find a better route  for you as investigating and fixing a too big query can take a while
[09:03] <cristiangsp> (just trying to find a better route  for you as investigating and fixing a too big query can take a while)
[09:04] <TJ-> we were trying to get to the publishing history
[09:04] <TJ-> but it's not vital so don't worry about trying to find a route for us right now
[09:05] <cristiangsp> got it, I would say that if you pick a small project probably you will be able to figure out the route for that one
[09:06] <cristiangsp> like this: https://launchpad.net/ubuntu/+source/linux/+publishinghistory
[09:06] <TJ-> got it via https://launchpad.net/ubuntu/+source/linux/+publishinghistory
[09:07] <cristiangsp> nice synchronization :P
[09:09] <TJ-> but we wouldn't make the syncronised swimming team :)
[09:09] <cristiangsp> XD
[13:49] <kisak> good day, anyone happen to know if lgw01 has been working with degraded storage over the last couple days?
[13:53] <kisak> context: I have a kernel build with minor patches that shouldn't affect the overall size at https://launchpad.net/~kisak/+archive/ubuntu/steamvr/+build/20807185 which failed twice with out of disk space at https://launchpad.net/~kisak/+archive/ubuntu/steamvr/+build/20807185 , then last night it failed after 11 hours with no log.
[13:53] <kisak> whoops, sorry about the double link
[13:56] <kisak> also, something restarted https://launchpad.net/~kisak/+archive/ubuntu/kisak-mesa/+build/20812889 , and this llvm build is at about 1/3 complete after 6 hours, so it's going to take 18 hours to complete when 4-6 hours is expected
[13:57] <kisak> a raid array being rebuilt would explain it
[14:01] <cjwatson> There are quite a lot of reasons why OpenStack nodes might be slow; storage is only one possibility
[14:02] <cjwatson> We'd have to bring this up with our sysadmins.  There's nothing intentional from the Launchpad side of things, but we aren't the lowest layer of the stack here ...
[14:02] <kisak> yeah, the llvm build is just circumstantial, and I'm not trying to complain about it.
[14:05] <kisak> the practical part of my question is if I should give up on the kernel build after this, I don't want to be wasting resources if there's a blocker
[14:14] <cjwatson> I don't think so, but yours is the second report of lgw01 slowness recently so we'll need to dig
[16:25] <kisak> that kernel build hit another out of space on device failure
[16:26] <kisak> that's 3 out of space on device after ~2 hours and 1 unknown, log-less failure after 11 hours
[16:27] <kisak> all on lgw01
[16:27] <cjwatson> failures without a log mean that either the builder crashed or there was a network interruption
[16:27] <cjwatson> out of space can't really be due to infrastructure though.  are you sure your build-dependencies haven't changed
[16:27] <cjwatson> ?
[16:27] <cjwatson> each builder VM has had the same disk allocation for a couple of years at least
[16:28] <cjwatson> there are known bugs where launchpad-buildd doesn't always react well to ENOSPC, too
[16:28] <cjwatson> so could be all the same thing
[16:28] <kisak> I've not changed any build dependencies, it's inherited from https://git.launchpad.net/~ubuntu-kernel-test/ubuntu/+source/linux/+git/mainline-crack/tag/?h=cod/mainline/v5.10.6
[16:28] <cjwatson> you might not have changed any, but they might have been changed under you
[16:28] <cjwatson> for cases where you do get a build log, have you tried diffing it against a previous successful log to look for other changes?
[16:29] <kisak> I've not dug deeper yet.
[16:31] <cjwatson> I'd quite strongly advise doing a successful/failing build log diff to start with - it's a very useful approach in general
[16:32] <cjwatson> out of space is likely to be some kind of problem internal to the build, even if it's in something you're build-depending on rather than in your package directly
[16:32] <cjwatson> and diffing build logs might show you something you missed
[18:18] <kisak> so, it's failing while installing the files that go into linux-image-unsigned-5.10.6-steamvr-generic-dbgsym, I guess I have to disable debug symbols for the PPA, which I'm not thrilled about. There's no reason why a mainline build wouldn't hit the same issue.
[18:24] <cjwatson> Are you building an additional flavour, or perhaps something slightly larger in some way?
[18:24] <cjwatson> You might also consider removing intermediate output along the way once it's been processed
[18:24] <cjwatson> Since it sounds like your failure is pretty close to the end
[18:30] <kisak> no, there's about 72 KB of patches that gets applied to be base config from ubuntu. no additional packages or flavoring
[18:32] <kisak> it's about the middle of the build, since if it got further, it would finish with -generic- packages, delete it, and then do -lowlatency-
[18:35] <kisak> the previous kernel build for this PPA was back in November, and there's been ~15MB from normal package bumps in the container setup
[18:42] <kisak> happen to know if Icy has the same storage limit?
[18:50] <kisak> yeah, the dbgsym package grew another .1GB apparently https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+packages?field.name_filter=linux-oem&field.status_filter=published&field.series_filter=focal
[18:58] <cjwatson> Icy?  Oh, you mean lcy01
[18:59] <cjwatson> Yes, the VM flavours are identical, 60G disk
[18:59] <cjwatson> (they're airport codes for roughly where the relevant datacentres are physically located)
[19:00] <kisak> okay thanks ... uhh ... that's impossible to tell that's an 'l' instead of an 'I' on the webpage
[19:01] <kisak> it's stupid the build grows that large
[20:02] <mwhudson> has launchpad stopped including display names in email from: recently?
[20:45] <cjwatson> mwhudson: Not intentionally - what sort of email?
[20:46] <mwhudson> cjwatson: ones from uploads that go to -changes lists, for one
[20:48] <mwhudson> hmm
[20:49] <mwhudson> maybe it's gmail that's changed actually
[20:49] <doko> which MUA are you using?
[20:49] <cjwatson> mwhudson: Just looked at my hirsute-changes mailbox and they seem to have display names in From:
[20:49] <mwhudson> yeah indeed
[20:49] <mwhudson> doko: gmail in browser
[20:50] <mwhudson> sorry for the noise
[20:50] <cjwatson> np
[20:50] <doko> at least with groovy's Thunderbird I see -changes for my uploads as well