=== pieq_ is now known as pieq
SpecialK|CanonI'll take a look08:36
cjwatsonIt looked like William fixed that overnight but didn't mention it here08:37
cjwatsonrabbitmq OOM08:37
SpecialK|CanonI saw that, but that looked to be several hours ago, and it looks like there's a bunch of builders that might be stuck cleaning08:39
cjwatsonAh, sure, feel free to dig then :)08:43
SpecialK|Canontsimonq2: Builders reset, should now be much happier!09:08
=== ErichEickmeyer is now known as Eickmeyer
EickmeyerI don't know what's going on, but it seems that every day around this time some dummy recipe in a PPA takes up nearly every amd64 builder for a good half-hour. Seems suspicious to me, I don't recognize the name.18:08
EickmeyerIt also occurs whether or not the person has updated their code. Seems nobody has touched it since 2017?18:10
RikMillsEickmeyer: I guess you mean the kstars/indi ones?18:12
EickmeyerRikMills: Yes, more like mutlaqja/libinid, but yes.18:13
RikMillsit is the kstars developer18:13
EickmeyerAh, why is it hogging every single builder then?18:14
RikMillsI have never looked that closely, but I guess he has a bazillion recipes, and in the recipes the base code is a common 'dummy' repo with each recipe nesting other repos on that18:24
RikMillsso that he can build each indi driver separately I guess, instead of the whole indi source repo in one big build18:25
RikMills100 has drivers for over 100 different types of astronomy device IIRC18:29
RikMills*indi has18:29
cjwatsonEickmeyer: It's not that user's fault.20:28
cjwatsonI mean apart from having a bazillion recipes.20:28
cjwatsonBut it's not their fault that we aren't doing a good job of spacing them out.20:28
cjwatsonHowever until like six hours ago it would have been very hard for us to space them out without risking tipping buildd-manager further over a performance cliff.20:29
cjwatsonThe situation should be better now, but I'd rather not perturb it in newly-invented ways for a while ...20:29
cjwatsonThe recipe build description is misleading - it doesn't show enough information to indicate that all the builds are actually distinct.20:30
sarnoldit certainly sounds more like something to fiddle with near the start of a day than at the end of a day :) heh20:30
cjwatsonAt the moment it's not very much of a practical problem - it's noticeable if you happen to look at /builders at the right time, but it doesn't typically clog things up for too long.20:31
cjwatson(It was a good bit worse before we redeployed git.l.n on faster hardware.)20:31
sarnoldyay :)20:33
cjwatsonAs of the rollout earlier today, buildd-manager now does a batched candidate selection query roughly once per group of builders with the same properties per scan cycle (15 seconds), rather than one query per idle builder per 15 seconds, so it's potentially much more viable to complicate the candidate selection query than it was.  But as I say I want to make sure that this actually solves the ...20:38
cjwatson... performance problems before changing anything else there.20:38
cjwatsonWe think this is almost certainly related to why we've had to run two buildd-managers for the last couple of months to avoid catastrophically slow fetching of large build output files, and at least plausibly implicated in the occasional buildd-manager hangs.20:40
cjwatsonTurns out that doing lots of synchronous DB queries in your asynchronous event loop is a bad idea.  (We're still doing some synchronous work there, just much less of it.)20:41
Eickmeyercjwatson: Thanks for the info! Such a complicated topic for sure. I don't envy you your job here.22:08
kyrofaDid my build just get... paused? https://pasteboard.co/JbAxSpW.png22:23
cjwatsonLink?  Impossible to grep logs for just that fragment22:24
kyrofacjwatson, https://code.launchpad.net/~kyrofa/+snap/nextcloud-test/+build/98697822:25
kyrofaLooks like it's kinda back... but no log22:25
cjwatsonIt got requeued after probably some kind of network interruption to the builder I think22:25
kyrofaAh, so it's starting over?22:26
cjwatsonSlightly odd, not sure what went on with the cancellation there22:26
kyrofaNot a big deal, just nothing I'd seen happen in the past22:26
cjwatson(Could also be that your build managed to DoS the builder in some way)22:27
kyrofaThat would be interesting. Nextcloud has been building LP for years, but this is my attempt to get it using base: core1822:27
kyrofaWhich also means I specified the stable channel for snapcraft as opposed to the default22:28
kyrofaLet's see if it happens again22:28
kyrofaI also changed the distribution series to bionic from xenial. Not sure if that was required or not22:28
cjwatsonYou should in general just leave it unset22:29
cjwatsoni.e. the very misnamed "Ubuntu Core 16"22:29
cjwatson(sorry - we'll fix that soonish I hope)22:29
kyrofaOh really? So I should distro series set to ubuntu core 16 and it'll work regardless of base?22:29
kyrofa(what about a lack of a base?)22:30
cjwatson"Ubuntu Core 16" is a really unclear way to say "build for series: 16 (i.e. what everything is) and autodetect everything else from base:"22:30
cjwatsonIf there's no base: it uses xenial22:31
kyrofacjwatson, perfect, good to know, thank you! And yeah, that series 16 has thrown me in multiple places22:31
kyrofacurling the store, for example22:31
kyrofacjwatson, interesting, that removes the arch checkbox as well, so it will depend on my having architectures set in the snapcraft.yaml?22:33
cjwatsonLooks happier so far anyway22:33
cjwatsonWhich you should anyway, that's the modern way to spell it22:34
kyrofaDarn. I wish LP assumed I wanted the old behavior when there wasn't any architecture22:34
cjwatsonsnapcraft defines things according to base really22:34

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!