[01:00] wgrant: hey, I'm trying to do a livefs build against a private PPA, but it seems to be stuck after fetching from it: https://launchpad.net/~ubuntu-acs/+livefs/ubuntu/trusty/stable/+build/53895 [01:00] the build log has been on Ign https://private-ppa.launchpad.net trusty InRelease [01:01] for a long time, then progressed by a couple of lines of the same type very slowly, and has been stuck for 10+mn or so [01:10] alright, Connection timed out after 120001 milliseconds [01:10] I guess it's a firewall issue [01:11] I'll try passing this to the right admin group [01:28] lool: Sounds likely. I can't see the build to investigate further, since it's private. But that sounds plausible given that description. [01:48] hi [01:48] I'm having problem to build on arm64 [01:49] it's not finding dlopen for some reason === maclin1 is now known as maclin [08:59] cjwatson, wgrant: issue was tracked in a ticket and solved; firewall now lets private-ppa access through (over https) === [HeOS] is now known as HeOS [10:19] is there a problem with LP these days? Publishing seems to be taking a very long time compared to a few months ago? [10:49] clivejo: there's a known problem with bloat of a particular index due partly to the test rebuilds in progress. But can you give an example? [10:50] (Might or might not be the same thing, question is slightly too general to be able to tell) [10:51] cjwatson: builds in kubuntu-ci seem to be taking a very long time to publish - https://launchpad.net/~kubuntu-ci/+archive/ubuntu/unstable [10:56] Can you quantify that? [10:56] Anywhere up to about 15 minutes is normal. [10:56] Maybe even 20, it depends where you happen to land in the cycle. [10:57] we have a Jenkins instance monitoring it, some are taking up to 30mins [10:58] That can happen if there's a particularly long run. [11:00] there are some builds that used to finish in 8mins now taking 20mins or more [11:00] The average cycle time at the moment seems to be around 15 minutes, so if your build happens to enter the queue just after the start of a publisher run, it could end up waiting for near the end of the next one if it's unlucky. [11:00] Ah, are you now talking about builds, not about publishing? [11:00] just seems that there is a noticeable delay in this past few months [11:01] Some is just organic growth. [11:01] There is definitely a problem at the moment with collecting artifacts from builders at the end of a build, though, or at least was yesterday morning. [11:01] well the timer is the entire process, but investigation seems to show that the publishing is the hold up [11:01] Publishing is always the slowest part. [11:01] (On average.) [11:02] what is the reason for that? [11:02] But log analysis doesn't suggest anything particularly worse right now than e.g. a year ago. [11:02] Im guessing there is a technical bottleneck here? [11:02] There are a lot of PPAs and the publisher has some linear scaling. [11:03] It's walking through all PPAs that need work each cycle, and there are a lot of those. [11:03] But it's also not structured to permit bulk operations very well. [11:04] Still, if you're seeing something in the last couple of days that's noticeably worse than a few months ago, which is how you opened this conversation, it's likely not actually the publisher. [11:09] (Also for the last day we've had logging turned up to 11 to try to get good data on buildd-manager database handling ...) [11:10] cjwatson: can it depend on the PPA as well? [11:11] for example my own PPA seems to publish a lot faster than the kubuntu-ci one [11:11] clivejo: kubuntu-ci is GIANT [11:11] clivejo: But it's more that giant PPAs like kubuntu-ci slow the whole cycle down [11:12] clivejo: It will depend on how you check [11:12] We can hardly complain about people using the service of course, but kubuntu-ci does use it extensively :-) [11:13] clivejo: If you're in a cycle where kubuntu-ci hasn't done anything but your own PPA has, then it's possible that that particular cycle will be quicker [11:15] so the publisher is a batch job? [11:16] does that mean the actual build machine is idle until the files are copied off? [11:17] The publisher is a batch job, yes; it tries to run every five minutes, though only actually runs if another instance isn't running. [11:18] Yes, but the builder doesn't wait on the publisher; retrieving files from the builder is a separate piece. [11:18] interesting [11:51] cjwatson, hello! We have a problem with our commercial subscription on Launchpad. Launchpad is returned the following ID: OOPS-6169ec49f0d826b02ef3cddb17dbd80b. Could you help us? [11:51] https://oops.canonical.com/?oopsid=OOPS-6169ec49f0d826b02ef3cddb17dbd80b [12:02] HeOS: can you try again now? We've started a proxy that wasn't running for some reason. [12:16] cjwatson, thanks, I'm going to follow your recommendations. === psivaa_ is now known as psivaa === FourDollars_ is now known as FourDollars === igitoor_ is now known as igitoor === Guest86703 is now known as jelmer [23:34] hi [23:34] I'm having problem to build on arm64 [23:34] it's not finding dlopen for some reason [23:34] is this the right channel? [23:46] Checking function dlopen in -ldl ... no [23:46] isn't it from glibc lib?