/srv/irclogs.ubuntu.com/2016/02/12/#ubuntu-ci-eng.txt

=== kalikiana_ is now known as kalikiana
Saviqrobru, didn't the train properly handle the case where MPs got removed from a silo? I mean with a default build job, should it not notice that there's an MP missing?00:27
Saviqhad to explicitly tell it which projects to build00:27
robruSaviq: uh it iterates over the branches that are there and checks them for new commits, I guess if you remove an MP it wouldn't notice that00:28
Saviqrobru, ok, I'll file a bug, thought it would00:46
Saviqrobru, uh oh https://requests.ci-train.ubuntu.com/#/ticket/93800:46
robruSaviq: lp:cupstream2distro00:46
Saviqrobru, yup, will do, tomorrow00:47
robruOoh Jesus00:47
robruSaviq: ah, looks like you caught it mid-rollout, one old script in memory calling a new script that updated out from under it. I triggered a DIFF_ONLY, should fix it right up00:49
Saviqrobru, ack,00:49
robruLooks good now00:52
=== salem_ is now known as _salem
Saviqrobru, btw, any idea why unity-api does not have a packaging diff generated? just content.diff ↑?01:57
robruSaviq: because there are no packaging changes.02:00
Saviqrobru, so changelog/cmakelists are not considered packaging any more?02:00
Saviqright, changelog would always trigger, duh02:01
robruSaviq: anymore? "packaging" diff only includes debian/ and excludes changelog02:01
Saviqrobru, ok maybe I'm confused, but would swear CMakeLists.txt were considered before02:01
Saviqrobru, anyway, wfm02:01
robruSaviq: I dunno maybe years ago. it's been just debian/ for a long time02:02
=== balloons16 is now known as balloons
robruheh07:35
=== ljp is now known as lpotter
SaviqMirv, morning, can I please ask you to press the ♻s on https://requests.ci-train.ubuntu.com/static/britney/xenial/landing-051/excuses.html and https://requests.ci-train.ubuntu.com/static/britney/vivid/landing-051/excuses.html08:52
MirvSaviq: sure08:56
MirvSaviq: doh, unity-api is in main and I haven't yet applied for core-dev. the qtmir-gles shows a weird "You submitted an invalid request: Package qtmir-gles does not have any test results" that I haven't seen before08:58
Mirvsil2100: please retry the regression from https://requests.ci-train.ubuntu.com/static/britney/xenial/landing-051/excuses.html08:58
sil2100Mirv: on it08:58
SaviqMirv, you not core-dev!? *gasp*08:58
Mirvsil2100: plus try if you get the same weird message as me from https://requests.ci-train.ubuntu.com/static/britney/vivid/landing-051/excuses.html08:58
sil2100Saviq: exactly! I have no idea why Mirv didn't apply yet ;)08:58
MirvSaviq: sorry :) I got so much PPU rights in addition to MOTU I'm too comfortable, but I really need to do the application08:58
sil2100Mirv: uh, same message09:01
Saviqsil2100, Mirv, so no idea about qtmir-gles in https://requests.ci-train.ubuntu.com/static/britney/vivid/landing-051/excuses.html ? shall I talk to pitti?10:05
Saviqalso, https://requests.ci-train.ubuntu.com/static/britney/xenial/landing-051/excuses.html is over an hour old now, any idea when would that get updated?10:06
MirvSaviq: sorry, yes ping and quote the error we're getting on trying to retry it10:36
MirvSaviq: it updates from my pov randomly :) roughly hourly, sometimes quicker. now updated 30min ago.10:37
SaviqMirv, yeah, with dozens still "in progress" and nowhere to be seen in http://autopkgtest.ubuntu.com/running.shtml10:37
Saviqwell, maybe not dozens :P10:38
Saviqasked about both10:38
robruSaviq: qtmir-gles is in running.shtml10:53
Saviqrobru, yeah, pitti just restarted them10:54
robruAh10:56
cjwatsonCan I have some train advice please?  https://requests.ci-train.ubuntu.com/#/ticket/960 needs an upgrade fix to avoid problems for apt users in xenial (it was a one-character typo in the Replaces field).  Can I put this through the same not-quite-completely-landed silo without incurring another pass through QA?11:04
robrucjwatson: well you'd have to clear that with qa, technically everything targeting vivid overlay goes through them11:05
robrucjwatson: strictly speaking any rebuild will clear the qa signoff on the ticket11:06
robruIf you ask them nicely they might just wave it through11:06
cjwatsonrobru: OK.  Is there some way I could just cause the change to go to xenial?  I could just manually upload to xenial-proposed ...11:07
cjwatsonIt's not particularly vital to have in the overlay since that won't typically be upgraded using apt11:07
robrucjwatson: yeah you could just upload to xenial.11:07
cjwatsonAll right, let's go with that for expediency's sake11:08
robrucjwatson: just make sure you sync to trunk after a xenial upload11:08
cjwatsonrobru: Yeah, I'll sync it all up after it lands11:08
robruThanks11:09
=== _salem is now known as salem_
=== alan_g is now known as alan_g|lunch
Saviqrobru, oh, you strip <foo> from bileto comments? could we instead html-entity it?13:32
Saviqjibel, silo 51 is good for QA, britney complains about a regression (known flaky test), but that actually re-ran already, just the results were not updated for over an hour ;/13:33
Saviqthere's a design ACK on it, too13:34
MirvSaviq: it's now updated but there is remaining issue (unity8 not considered) which I believe I can fix13:38
SaviqMirv, right, that's a leftover binary, was hoping that this would go away13:38
MirvSaviq: it won't automatically, but it will now that I manually deleted the superseded sources from the PPA13:39
Saviqah ctrl+r!13:39
SaviqMirv, ack13:39
MirvI ran into the same problem with my Qt PPA13:39
Saviqany case, jibel ↑ 051 ready for testing, despite britney being sloooooow13:39
* Saviq worried looking at "ready for testing" queue :(13:40
MirvSaviq: it took me ~2 weeks to get one of my silos into the queue when it was ready the whole time :) now the train is much more likely to get there with the automated signoff.13:43
Mirvyeah the queue is long, but priority silos will get priority treatment13:43
SaviqMirv, well, I'm happy with the approach altogether, not so much with the overhead13:43
Saviqand the waiting, most of all13:46
morphiscjwatson: any idea why my silo packages in https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-000/+packages waiting for publication since 20min?14:22
cjwatsonmorphis: 20 minutes isn't that long14:23
morphisreally?14:23
morphishad it faster in mind14:23
cjwatsonmorphis: the cron job in question tries to run every five minutes and typically takes around ten, IIRC14:23
sil2100Sometimes it can take longer, depends on how busy the publishers are14:23
morphisah I see14:23
morphisthanks!14:23
cjwatsonmorphis: so 20 minutes is in the normal worst-case ballpark14:23
=== alan_g|lunch is now known as alan_g
morphisok14:24
dobeyhmm, i wonder how long it will take the valid candidates to get through proposed now and have the MPs merged14:26
sil2100dobey: the scopes-api bits?14:27
sil2100hmmm14:29
cjwatsondobey: hahaha.  multiple entwined transitions here14:31
cjwatsonthere's a big complicated openmpi transition that it's welded to, unfortunately14:31
cjwatsonmorphis: Possibly some networking problem on the PPA publisher, though, it's working but very slowly14:32
morphiscjwatson: wonderful :-)14:33
dobeysil2100, cjwatson: i wonder if i should force merge the scopes-api silo, so we can get other silos moving along again, then14:33
cjwatsondobey: possibly, although I'd recommend not actually trying to publish anything else to the relevant set of packages in order to avoid complicating this set of transitions even further14:34
dobeyhmm14:36
sil2100cjwatson, dobey: I would opt for only force-merging this silo to unblock development, but as per cjwatson's recommendation not landing any other scope silos before this migrates14:40
sil2100I see a big list of uninstallable packages in the update_output, I'm worried it'll take some time for all the transitions to get resolved14:41
cjwatsonsil2100: it's actually fairly close I think, update_output can be a bit misleading here because it excludes anything not a valid candidate14:42
sil2100A force-merge would at least allow building other silos14:43
cjwatsonbut it's certainly painful to analyse14:43
sil2100Yeah, looks scary14:43
sil2100;)14:43
sil2100cjwatson: btw. do you know who I should poke about questions regarding mailing-list support in launchpad?14:47
cjwatsonsil2100: I guess if I don't know then William is your most likely option14:50
sil2100wgrant: hey! I would need to create a semi read-only mailing list in Launchpad - either with posting requiring moderation or with some other approach14:51
sil2100wgrant: do you know if I could create something like that in LP?14:52
wgrantsil2100: LP mailing lists are rather inflexible, unfortunately. There are no facilities for anything like that.14:52
sil2100wgrant: hmm, ok, thanks14:58
pstolowskisil2100, hey, any idea what the above status of silo 65 means ^?15:27
sil2100hm, never saw it before, let me take a look15:27
dobeysil2100: i guess merging 37 might help here?15:31
sil2100Ah, let's merge that one indeed, I guess it should be ok15:32
SaviqMirv, doesn't seem like that helped in silo 5115:34
dobeysil2100: i guess you need to do the merge/clean, since it claims i don't have job permission :)15:34
morphiscjwatson: some were moved but the important ones still sit there for close to two hours now15:36
cjwatsonmorphis: network issues, being worked on15:38
morphisah ok15:38
dobeywow, http 41215:53
dobeywell that confused the train15:54
MirvSaviq: weird. sil2100 any other ideas with silo 51 excuses not considered? it was exactly what I explained to you that I resolved by removing superseded packages from the PPA.16:00
Mirvbut for some mysterious reason it's not helping here16:00
davmor2sil2100: 926 could that be a priority for merging please morphis wantsssssssessss it the precioussssss </smegol_impression>16:13
morphisdavmor2: :-)16:14
sil2100morphis: eeek! Approve teh branchezzz16:16
sil2100(review and approve)16:17
morphistvoss, greyback__: can one of you approve the platform-api changes in https://requests.ci-train.ubuntu.com/#/ticket/926 ?16:17
tvossmorphis, done16:20
morphissil2100: ^^16:20
dobeysil2100: hmm, i think you need to merge/clean silo 37 again perhaps?16:24
sil2100o/16:28
sil2100morphis: still one merge needs approval...16:29
sil2100brb in a minute16:29
morphissil2100: done16:30
morphiscjwatson: any information available when the network problems will be solved?16:47
dobeyoh bah16:55
dobeysil2100: hrmm, i got the ppa/bzr version mismatch error now too :(16:55
cjwatsonmorphis: well, we've been working with IS, but it looks like a DOS, so you know17:01
morphiscjwatson: perfect :-)17:01
dobeyick :-/17:02
cjwatsonmorphis: the publisher is definitely still chugging along, so it'll get there, it's just tough for it to download files from the librarian over its saturated link17:02
morphisI see17:03
kgunntrainguards ^ sorry, unfamiliar with that one?18:17
kgunni do see a weird warning here18:19
kgunnhttps://launchpad.net/ubuntu/xenial/+source/mir18:19
kgunn"Mir 0.19.2 is older than the current packaged version. Launchpad may be missing release information for the 0.19 series or this package is linked to the wrong Mir series."18:19
kgunnoh...nvmd i think i know18:20
cjwatsonmorphis: all right, so this is an amusing self-DOS; the main libreoffice PPA has done on the order of 100GB outbound over the last hour, which has kind of flatlined everything else.  we think the best approach is to just wait for it to calm down18:25
morphiscjwatson: ohh18:26
morphiscjwatson: close to week so have to wait anyway :-)18:26
cjwatsonyeah, I don't think it's worth trying anything more complicated last thing on a Friday18:26
robruSo I guess most of the "ppa/bzr version mismatch" statuses are just due to ppas being slow to publish packages, should sort itself out18:41
dobeyrobru: ah, i guess slow to consume rather than publish. took forever for the uploads to show up for mine. then status was weird with "needs building" and had to do a diff_only rebuild to get it to look at the ppa again. now i just have to wait forever for the binaries to publish :-/18:58
robrudobey: uh, no? diff only just generates diffs. it doesn't "make it look at the ppa again". the train looks at the ppa every 15 minutes forever.18:59
Trevinhorobru: what's that error?18:59
Trevinho^18:59
dobeyrobru: oh; well then why did it say "needs building" instead of showing the built status?18:59
Trevinho(ppa/bzr version mismatch)18:59
dobeyTrevinho: DDoS19:00
robrudobey: "needs building" is a ppa build status.19:00
dobeyrobru: yes; but why would it show that *while* it is building?19:00
robrudobey: "needs building" means "ppa has your source package but all the buildds are busy"19:00
robrudobey: if that was the status for one of the arches then that would be reported as the status for the entire build.19:00
dobey75% were already built :)19:01
dobeyoh19:01
robrudobey: train has priorities the list of possible ppa build states when it decides which ones to report. so eg "failed to build" is highest, if any arch is a failure then the whole thing is considered a failure. "needs building" is in that priority order higher than "currently building" and 'successfully built' so if 75% succeed and a couple are still19:02
robrubuilding and one is "needs building", the whole thing is reported as needs building.19:02
robrudobey: the idea is that a failure needs to be reported asap but a success isn't a success until all arches succeed19:02
robruTrevinho: ppas are slow to publish packages, it means it's looking at a ppa and it's not seeing the package it just uploaded there. it's just slow, it'll fix itself19:03
Trevinhorobru: fine, thanks :)19:03
robruyw19:05
kgunntrainguards ok back to being stumped ^19:09
kgunnthe ppa/bzr version mismatch thing19:09
robrubruh19:09
kgunnrobru: is that a bruh like "you should know better" or a bruh like "tell me about it" ?19:16
kgunn:)19:16
dobeybruh, read the room ;)19:16
robrukgunn: it's like "bruh we were just talking about that"19:17
kgunnoh my bad19:17
robrukgunn: "bruh I'm writing documentation because I don't want to explain this a third time in 20 minutes"19:17
* kgunn recedes with tail between legs19:17
dobeyrobru: we can start an office pool on whether it will take 20 minutes for another complaint about that status ;)19:18
dobeyrobru: or maybe just change that status to "Waiting for uploaded source to appear in PPA (foo/series)19:18
robrudobey: yeah I dunno, because the same message could also appear if it fails to push the bzr branch.19:22
robrudobey: the point is that it's comparing the remote bzr branch to the remote ppa and the versions don't match. it doesn't really have a way of knowing which one is wrong.19:22
robrudobey: kgunn: Trevinho: https://wiki.ubuntu.com/citrain/LandingProcess#PPA.2BAC8-bzr_version_mismatch19:28
=== robru changed the topic of #ubuntu-ci-eng to: Train trouble? ping trainguards | CI problems? Use JenkaaS: http://bit.ly/jenkins-docs | Train: http://bit.ly/1hGZsfS | QA Signoffs: http://bit.ly/1qMAKYd | Known Issues: https://wiki.ubuntu.com/citrain/LandingProcess#PPA.2BAC8-bzr_version_mismatch
dobeynow you just need a scripted reply of "see the topic" whenever someone asks about it ;)19:30
robruheh19:31
Saviqrobru, hey, any idea how to convince britney to consider unity8 here https://requests.ci-train.ubuntu.com/#/ticket/938? it's complaining about a package from a previous build (that was removed by now)20:01
Saviqor jibel, did you have a look to force that into QA-ready ↑?20:01
dobeySaviq: launchpad publishing is very slow today, so it might catch up after the binary publish finishes20:10
dobeyoh, but looks like it has finished20:10
robrudobey: Saviq I dunno, that britney run is from nearly an hour ago...20:13
dobeycould be the same network saturation causing problems though20:14
robruSaviq: how long has it been since the packages were deleted?20:14
Mirvrobru: it's not the britney run delay. I deleted the packages 5h ago...20:14
robruoh, that's a lot20:14
dobeyit's been like 1.5 hrs waiting for binaries to publish in my silo20:15
Mirvdobey: oh... well that's abnormal20:15
robrui dunno man20:15
dobeyMirv: yeah. apparently this is what happens when the network gets saturated and the publisher can't publish things, though :)20:16
robrudobey: "can't publish" also including "can't delete"?20:17
dobeyrobru: i would presume so, yes20:18
robruhm20:18
dobeybut it really shouldn't matter if old binaries are around, if the new ones are there20:19
=== alesage_ is now known as alesage
Mirvit does in train britney currently if binary package is removed in new version. the superseded source needs to be deleted manually20:25
Mirvhappened to my qtbase and now Saviq's unity8 silo20:26
robrudobey: yeah I'm not sure why britney is looking at superceded sources. I guess it's some difference in how ppa package indexes look compared to -proposed (like I guess -proposed doesn't keep those in the indexes and ppas do)20:27
dobeyoh, it removed binary packages from debian/control? ok, yeah that is a bit different too20:34
cjwatsondobey: not sure I'd describe it as a DDoS, though there's certainly a distributed element :)20:35
cjwatsonMirv: yeah, last completed run was on the order of five hours ago :-/20:37
robrucjwatson: all of the "ppa/bzr version mismatch" messages cleared up though, implying that packages got published in ppas?21:01
cjwatsonrobru: it's making gradual progress, but it's still very gradual21:02
robruah21:02
cjwatsonrunning at a couple of percent of the usual speed21:02
dobeyrobru: i think finding the source uploads is a different process from publishing binaries?21:02
cjwatsonit's the same publisher, it just depends on how much is available to publish21:03
cjwatsonI mean even pretty much the same bit of the publisher21:03
dobeyoh21:03
cjwatsonaccepting source/binaries is a different phase from publishing source/binaries; but the ppa/bzr business depends on the source part of the latter21:04
dobeyit's seemed like source uploads get picked up faster than binaries are getting published, so i presumed there was some relevant difference there21:04
cjwatsondobey: they'd get picked up for build earlier, assuming public PPAs21:11
cjwatsondobey: but I don't believe the train looks at that bit; ICBW21:11
cjwatsondobey: oh; no, I stand corrected, you're right; the train looks for either Pending or Published builds21:12
cjwatsons/builds/publications/21:13
dobeyah ok21:13
cjwatsonboth will still have been very slow since the whole process is serial21:13
cjwatsonPPA publishing issues should hopefully improve very shortly22:04
=== salem_ is now known as _salem
robruSaviq: https://requests.ci-train.ubuntu.com/static/britney/xenial/landing-051/excuses.html well this looks better, just needs somebody to retry that regression22:37
Saviqoh oh, mterry ↑↑ :D22:37
alesagerobru question: any harm in landing the same MP twice?  I suppose it's just a vacuous merge?  seeing the same branch in 16 and 3822:47
Saviqsil2100, I'll trade you ↑↑ recycle https://requests.ci-train.ubuntu.com/static/britney/xenial/landing-051/excuses.html please? ;)22:48
robrualesage: depends what it is i think. When one silo lands the other will need to be rebuilt which may cause conflicts.22:49
alesagerobru, ack22:49
sil2100o/22:49
robrualesage: i mean the silo needs to be rebuilt anyway so worst case you just remove the duplicate before the rebuild22:49
Saviqoh crap, just missed req100022:50
sil2100Clickedy clicked22:50
alesagerobru, ok will advise22:50
Saviqsil2100, https://requests.ci-train.ubuntu.com/#/ticket/100122:50
sil2100Saviq: hah :)22:50
sil2100Thanks ;)22:50
Saviqjhodapp, congrats on request 1000! :D22:50
sil2100I still need to copy mir etc.22:50
robruOh god we're already over 1000 tickets? You maniacs are burning through them so fast! Those numbers will overflow at 9 quintjiliion you know, stop wasting them!22:51
sil2100Let's start using letters22:54
sil2100AZ12BJ doesn't sound confusing at all22:54
mterrySaviq, still need that press?22:55
robrusil2100: after the postgres db overflows at 9223372036854775807 I'll switch it to UUIDs i guess22:55
mterrySaviq, looks like no22:56
jhodappSaviq, ha!22:56
Saviqmterry, no, sil hooked me up, thanks22:56
jhodappthanks :)22:56
jhodappSaviq, what'd I win?22:56
Saviqrobru, customer for you ↑22:56
robruWhat?22:58
Saviqrobru, what did jhodapp win for grabbing req 100022:59
robruSaviq: he wins one internet for the day22:59
jhodapprobru, one whole internet to myself!?23:00
robrujhodapp: you got it, buddy!23:00
Saviqrobru, btw, did you even start thinking about git support in train? I started looking around citrain code to see how quickly this could be hacked in23:00
Saviqand it doesn't look bad23:00
jhodapprobru, thanks :)23:01
sil2100Saviq: ok, should be good to build23:01
sil2100The silo that is23:01
robruSaviq: yeah I've been thinking about it. There are some unanswered questions about how it will work. My top priority is getting ephemeral silos, then git after that23:01
* sil2100 goes to sleep23:01
sil2100o/23:01
Saviqrobru, can you remind me why http://code.launchpad.net/~ci-train-bot/ branches have the project name in them?23:01
SaviqI mean ~/$project/$project-$distro-$release-landing-$silo23:02
Saviqwhy the double $project?23:02
robruSaviq: they don't have the project name in them, they have the source package name in them. Ctrl+f for "gles" and you'll see the difference23:02
Saviqrobru, right!23:03
robruSaviq: so you probably saw the Branch class, right? In theory it would be easy to write a sister class that implements the same api but wrapping git instead of bzr, but the thing is that isn't a class that gets instantiated, it's a mixin that gets inherited, so I'm not sure how to have a class conditionally inherit from a bzr or git class as needed. Will23:05
robruprobably need to be split out into separate classes with some logic for choosing which one to use when.23:05
robruSaviq: also the lp api exposes different properties for bzr MPs as git MPs and i don't have a clear idea of what parts in the code would be affected by those differences. It's not fully contained inside Branch class.23:06
Saviqhum23:07
Saviqrobru, ack, will try and have a look when I find some time23:07
robruSaviq: talk to barry also, he expressed some interest in working on that23:08
Saviqrobru, ack23:09
Saviqrobru, any idea what happened here https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-046/+packages ?23:52
Saviqrobru, it says it's waiting for unity-api >= 0.105, which is in the "emergency snapshot PPA" as added by sil23:53
Saviqsame for UITK23:53
Saviqgranted, the qtmultimedia missing dep is valid, so the others might be red herrings23:54
robruSaviq: emergency snapshot ppa doesn't have anything to do with anything?23:54
Saviqrobru, yes it does, sil modified that silo to have the snapshot ppa as dep, not overlay23:54
robruoh, well23:55
robruSaviq: like, just now? lp will retry depwaits every 1.5 hours or so23:55
Saviqrobru, before I even started the build23:55
Saviqrobru, even the build log says the ppa is added https://launchpadlibrarian.net/238313480/buildlog_ubuntu-vivid-amd64.unity8_8.11+15.04.20160212-0ubuntu1_BUILDING.txt.gz23:56
Saviqmaybe they didn't publish yet, that snapshot is somewhat new23:56
Saviqrobru, that's probably it, will likely resolve itself soon23:57
robruSaviq: yeah could be same old publisher issues.23:57
robruSaviq: i just retried one arch to see what happens, but yeah depwaits will be retried automatically so as long as you're sure all the deps are in the ppa it should be fine eventually23:57
Saviqrobru, yup, tx23:58

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!