[01:00] <lool> wgrant: hey, I'm trying to do a livefs build against a private PPA, but it seems to be stuck after fetching from it: https://launchpad.net/~ubuntu-acs/+livefs/ubuntu/trusty/stable/+build/53895
[01:00] <lool> the build log has been on Ign https://private-ppa.launchpad.net trusty InRelease
[01:01] <lool> for a long time, then progressed by a couple of lines of the same type very slowly, and has been stuck for 10+mn or so
[01:10] <lool> alright, Connection timed out after 120001 milliseconds
[01:10] <lool> I guess it's a firewall issue
[01:11] <lool> I'll try passing this to the right admin group
[01:28] <cjwatson> lool: Sounds likely.  I can't see the build to investigate further, since it's private.  But that sounds plausible given that description.
[01:48] <sergio-br2> hi
[01:48] <sergio-br2> I'm having problem to build on arm64
[01:49] <sergio-br2> it's not finding dlopen for some reason
[08:59] <lool> cjwatson, wgrant: issue was tracked in a ticket and solved; firewall now lets private-ppa access through (over https)
[10:19] <clivejo> is there a problem with LP these days?  Publishing seems to be taking a very long time compared to a few months ago?
[10:49] <cjwatson> clivejo: there's a known problem with bloat of a particular index due partly to the test rebuilds in progress.  But can you give an example?
[10:50] <cjwatson> (Might or might not be the same thing, question is slightly too general to be able to tell)
[10:51] <clivejo> cjwatson: builds in kubuntu-ci seem to be taking a very long time to publish - https://launchpad.net/~kubuntu-ci/+archive/ubuntu/unstable
[10:56] <cjwatson> Can you quantify that?
[10:56] <cjwatson> Anywhere up to about 15 minutes is normal.
[10:56] <cjwatson> Maybe even 20, it depends where you happen to land in the cycle.
[10:57] <clivejo> we have a Jenkins instance monitoring it, some are taking up to 30mins
[10:58] <cjwatson> That can happen if there's a particularly long run.
[11:00] <clivejo> there are some builds that used to finish in 8mins now taking 20mins or more
[11:00] <cjwatson> The average cycle time at the moment seems to be around 15 minutes, so if your build happens to enter the queue just after the start of a publisher run, it could end up waiting for near the end of the next one if it's unlucky.
[11:00] <cjwatson> Ah, are you now talking about builds, not about publishing?
[11:00] <clivejo> just seems that there is a noticeable delay in this past few months
[11:01] <cjwatson> Some is just organic growth.
[11:01] <cjwatson> There is definitely a problem at the moment with collecting artifacts from builders at the end of a build, though, or at least was yesterday morning.
[11:01] <clivejo> well the timer is the entire process, but investigation seems to show that the publishing is the hold up
[11:01] <cjwatson> Publishing is always the slowest part.
[11:01] <cjwatson> (On average.)
[11:02] <clivejo> what is the reason for that?
[11:02] <cjwatson> But log analysis doesn't suggest anything particularly worse right now than e.g. a year ago.
[11:02] <clivejo> Im guessing there is a technical bottleneck here?
[11:02] <cjwatson> There are a lot of PPAs and the publisher has some linear scaling.
[11:03] <cjwatson> It's walking through all PPAs that need work each cycle, and there are a lot of those.
[11:03] <cjwatson> But it's also not structured to permit bulk operations very well.
[11:04] <cjwatson> Still, if you're seeing something in the last couple of days that's noticeably worse than a few months ago, which is how you opened this conversation, it's likely not actually the publisher.
[11:09] <cjwatson> (Also for the last day we've had logging turned up to 11 to try to get good data on buildd-manager database handling ...)
[11:10] <clivejo> cjwatson: can it depend on the PPA as well?
[11:11] <clivejo> for example my own PPA seems to publish a lot faster than the kubuntu-ci one
[11:11] <cjwatson> clivejo: kubuntu-ci is GIANT
[11:11] <cjwatson> clivejo: But it's more that giant PPAs like kubuntu-ci slow the whole cycle down
[11:12] <cjwatson> clivejo: It will depend on how you check
[11:12] <cjwatson> We can hardly complain about people using the service of course, but kubuntu-ci does use it extensively :-)
[11:13] <cjwatson> clivejo: If you're in a cycle where kubuntu-ci hasn't done anything but your own PPA has, then it's possible that that particular cycle will be quicker
[11:15] <clivejo> so the publisher is a batch job?
[11:16] <clivejo> does that mean the actual build machine is idle until the files are copied off?
[11:17] <cjwatson> The publisher is a batch job, yes; it tries to run every five minutes, though only actually runs if another instance isn't running.
[11:18] <cjwatson> Yes, but the builder doesn't wait on the publisher; retrieving files from the builder is a separate piece.
[11:18] <clivejo> interesting
[11:51] <HeOS> cjwatson, hello! We have a problem with our commercial subscription on Launchpad. Launchpad is returned the following ID: OOPS-6169ec49f0d826b02ef3cddb17dbd80b. Could you help us?
[12:02] <cjwatson> HeOS: can you try again now?  We've started a proxy that wasn't running for some reason.
[12:16] <HeOS> cjwatson, thanks, I'm going to follow your recommendations.
[23:34] <sergio-br2> hi
 I'm having problem to build on arm64
 it's not finding dlopen for some reason
[23:34] <sergio-br2> is this the right channel?
[23:46] <sergio-br2> Checking function dlopen in -ldl ... no
[23:46] <sergio-br2> isn't it from glibc lib?