| TJ- | Seeing repeated timeout errors with OOPS last 5 minutes. (Error ID: OOPS-12be8b5060bbc1f92318e126ad954cb2) | 08:28 |
|---|---|---|
| ubot5 | https://oops.canonical.com/?oopsid=OOPS-12be8b5060bbc1f92318e126ad954cb2 | 08:28 |
| TJ- | cristiangsp: Seeing repeated timeout errors with OOPS last 5 minutes. (Error ID: OOPS-12be8b5060bbc1f92318e126ad954cb2) | 08:45 |
| ubot5 | https://oops.canonical.com/?oopsid=OOPS-12be8b5060bbc1f92318e126ad954cb2 | 08:45 |
| TJ- | cristiangsp: e.g. for https://launchpad.net/ubuntu/+source/linux | 08:45 |
| cristiangsp | yep, I can reproduce it too, checking... | 08:57 |
| cristiangsp | it seems that the overview section is fetching too much data which triggers the timeout. There is any specific are that you are interested on? | 09:01 |
| cristiangsp | https://code.launchpad.net/ubuntu/+source/linux seems loading fine for example | 09:02 |
| cristiangsp | (just trying to find a better route for you as investigating and fixing a too big query can take a while | 09:03 |
| cristiangsp | (just trying to find a better route for you as investigating and fixing a too big query can take a while) | 09:03 |
| TJ- | we were trying to get to the publishing history | 09:04 |
| TJ- | but it's not vital so don't worry about trying to find a route for us right now | 09:04 |
| cristiangsp | got it, I would say that if you pick a small project probably you will be able to figure out the route for that one | 09:05 |
| cristiangsp | like this: https://launchpad.net/ubuntu/+source/linux/+publishinghistory | 09:06 |
| TJ- | got it via https://launchpad.net/ubuntu/+source/linux/+publishinghistory | 09:06 |
| cristiangsp | nice synchronization :P | 09:07 |
| TJ- | but we wouldn't make the syncronised swimming team :) | 09:09 |
| cristiangsp | XD | 09:09 |
| kisak | good day, anyone happen to know if lgw01 has been working with degraded storage over the last couple days? | 13:49 |
| kisak | context: I have a kernel build with minor patches that shouldn't affect the overall size at https://launchpad.net/~kisak/+archive/ubuntu/steamvr/+build/20807185 which failed twice with out of disk space at https://launchpad.net/~kisak/+archive/ubuntu/steamvr/+build/20807185 , then last night it failed after 11 hours with no log. | 13:53 |
| kisak | whoops, sorry about the double link | 13:53 |
| kisak | also, something restarted https://launchpad.net/~kisak/+archive/ubuntu/kisak-mesa/+build/20812889 , and this llvm build is at about 1/3 complete after 6 hours, so it's going to take 18 hours to complete when 4-6 hours is expected | 13:56 |
| kisak | a raid array being rebuilt would explain it | 13:57 |
| cjwatson | There are quite a lot of reasons why OpenStack nodes might be slow; storage is only one possibility | 14:01 |
| cjwatson | We'd have to bring this up with our sysadmins. There's nothing intentional from the Launchpad side of things, but we aren't the lowest layer of the stack here ... | 14:02 |
| kisak | yeah, the llvm build is just circumstantial, and I'm not trying to complain about it. | 14:02 |
| kisak | the practical part of my question is if I should give up on the kernel build after this, I don't want to be wasting resources if there's a blocker | 14:05 |
| cjwatson | I don't think so, but yours is the second report of lgw01 slowness recently so we'll need to dig | 14:14 |
| kisak | that kernel build hit another out of space on device failure | 16:25 |
| kisak | that's 3 out of space on device after ~2 hours and 1 unknown, log-less failure after 11 hours | 16:26 |
| kisak | all on lgw01 | 16:27 |
| cjwatson | failures without a log mean that either the builder crashed or there was a network interruption | 16:27 |
| cjwatson | out of space can't really be due to infrastructure though. are you sure your build-dependencies haven't changed | 16:27 |
| cjwatson | ? | 16:27 |
| cjwatson | each builder VM has had the same disk allocation for a couple of years at least | 16:27 |
| cjwatson | there are known bugs where launchpad-buildd doesn't always react well to ENOSPC, too | 16:28 |
| cjwatson | so could be all the same thing | 16:28 |
| kisak | I've not changed any build dependencies, it's inherited from https://git.launchpad.net/~ubuntu-kernel-test/ubuntu/+source/linux/+git/mainline-crack/tag/?h=cod/mainline/v5.10.6 | 16:28 |
| cjwatson | you might not have changed any, but they might have been changed under you | 16:28 |
| cjwatson | for cases where you do get a build log, have you tried diffing it against a previous successful log to look for other changes? | 16:28 |
| kisak | I've not dug deeper yet. | 16:29 |
| cjwatson | I'd quite strongly advise doing a successful/failing build log diff to start with - it's a very useful approach in general | 16:31 |
| cjwatson | out of space is likely to be some kind of problem internal to the build, even if it's in something you're build-depending on rather than in your package directly | 16:32 |
| cjwatson | and diffing build logs might show you something you missed | 16:32 |
| kisak | so, it's failing while installing the files that go into linux-image-unsigned-5.10.6-steamvr-generic-dbgsym, I guess I have to disable debug symbols for the PPA, which I'm not thrilled about. There's no reason why a mainline build wouldn't hit the same issue. | 18:18 |
| cjwatson | Are you building an additional flavour, or perhaps something slightly larger in some way? | 18:24 |
| cjwatson | You might also consider removing intermediate output along the way once it's been processed | 18:24 |
| cjwatson | Since it sounds like your failure is pretty close to the end | 18:24 |
| kisak | no, there's about 72 KB of patches that gets applied to be base config from ubuntu. no additional packages or flavoring | 18:30 |
| kisak | it's about the middle of the build, since if it got further, it would finish with -generic- packages, delete it, and then do -lowlatency- | 18:32 |
| kisak | the previous kernel build for this PPA was back in November, and there's been ~15MB from normal package bumps in the container setup | 18:35 |
| kisak | happen to know if Icy has the same storage limit? | 18:42 |
| kisak | yeah, the dbgsym package grew another .1GB apparently https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+packages?field.name_filter=linux-oem&field.status_filter=published&field.series_filter=focal | 18:50 |
| cjwatson | Icy? Oh, you mean lcy01 | 18:58 |
| cjwatson | Yes, the VM flavours are identical, 60G disk | 18:59 |
| cjwatson | (they're airport codes for roughly where the relevant datacentres are physically located) | 18:59 |
| kisak | okay thanks ... uhh ... that's impossible to tell that's an 'l' instead of an 'I' on the webpage | 19:00 |
| kisak | it's stupid the build grows that large | 19:01 |
| mwhudson | has launchpad stopped including display names in email from: recently? | 20:02 |
| cjwatson | mwhudson: Not intentionally - what sort of email? | 20:45 |
| mwhudson | cjwatson: ones from uploads that go to -changes lists, for one | 20:46 |
| mwhudson | hmm | 20:48 |
| mwhudson | maybe it's gmail that's changed actually | 20:49 |
| doko | which MUA are you using? | 20:49 |
| cjwatson | mwhudson: Just looked at my hirsute-changes mailbox and they seem to have display names in From: | 20:49 |
| mwhudson | yeah indeed | 20:49 |
| mwhudson | doko: gmail in browser | 20:49 |
| mwhudson | sorry for the noise | 20:50 |
| cjwatson | np | 20:50 |
| doko | at least with groovy's Thunderbird I see -changes for my uploads as well | 20:50 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!