/srv/irclogs.ubuntu.com/2016/07/15/#ubuntu-ci-eng.txt

=== chihchun is now known as chihchun_afk
=== _salem is now known as salem_
=== chihchun_afk is now known as chihchun
=== salem_ is now known as _salem
michitrainguards: I have a process question with failing autopkgtests. Anyone around who might be able to advise?04:03
robrumichi: what's up?04:03
michirobru: Same question as yesterday.04:04
michiBasically, the thumbnailer test is failing due to some package dependency I didn’t even know existed.04:04
michiSo, what’s the right process?04:04
michiI don’t want to bug a core dev every day to kick the test until it finally passes.04:04
robrumichi: you need a core dev to hit retry then?04:04
michiYes.04:04
michiBut it might be days before this works.04:04
michiSo, I bug a core dev every day?04:05
robrumichi: only people with upload rights can retry the tests. So if it's a universe package you can get a MOTU04:05
michiI don’t know what  MOTU is...04:05
michiIt just seems really awkward and inefficient this way.04:06
michiCouldn’t we have something that realizes that an autopkgtest has failed due to a missing dependency and then automatically retry periodically?04:07
michiSeems that the process for an autopkg test failing due to missing dependencies is quite different from a failure when the actual test itself doesn’t pass.04:07
michiI’m wondering whether we could automate the former.04:08
robrumichi: https://launchpad.net/~motu but check if the package is in universe04:08
robrumichi: you'd have to take that up with pitti04:08
michiOK, will do.04:09
michiBTW, the failure is due to repowerd.04:10
michiWhich isn’t in universe, AFAIK.04:10
robrumichi: isn't that new? Universe is default until it gets promoted04:11
michiNot sure.04:11
michiThat’s part of the problem.04:11
michiOur autopkgtest fails due to some problem with a package I didn’t even know existed...04:11
robrumichi: either way, yeah you need a person with power to hit the switch, nothing I can do about that04:11
michiSure, I understand, thanks!04:12
robrumichi: pitti told me he required this to avoid flooding the system, it's a necessary bottleneck apparently04:12
robruYou're welcome04:12
michiCool, thanks.04:12
=== chihchun is now known as chihchun_afk
=== chihchun_afk is now known as chihchun
=== chihchun is now known as chihchun_afk
=== chihchun_afk is now known as chihchun
=== chihchun is now known as chihchun_afk
=== chihchun_afk is now known as chihchun
Saviqsil2100, morning, any idea why https://requests.ci-train.ubuntu.com/#/ticket/1525 is failed? everything seems fine but unity8 on xenial, which just has "not considered" :/07:38
sil2100uh07:41
sil2100No autopkgtests no nothing07:42
sil2100Saviq: I think pitti needs to be brought into this07:43
Saviqack /me → -devel07:50
sil2100cjwatson: hello! Houston, we have a problem, need your wisdom on how to resolve this08:02
sil2100cjwatson: situation: we had a new package called repowerd put into the overlay, this package was providing a virtual package of powerd to deprecate the existing powerd package for transition08:04
sil2100cjwatson: but repowerd seemed to cause a lot of regressions, we decided to 'revert' it temporarily from the vivid overlay08:04
sil2100cjwatson: since it was the only repowerd version, I reverted the dependent packages and removed repowerd from the overlay archive (so that the only thing providing powerd is now powerd itself)08:05
sil2100cjwatson: but this backfired, since now the image builders don't see powerd in the overlay at all, possibly because they see that repowerd that was providing the latest version of powerd binary is deleted and ignores the existing powerd package in the overlay08:06
sil2100cjwatson: and it's pulling in something archaic from the archives now08:06
sil2100cjwatson: what can we do in such a situation? We want anyone using the overlay to be able to use the powerd provided by powerd source, with repowerd removed08:07
sil2100(so get back to the situation from a few days before)08:07
seb128you probably want to reupload a proper powed to the ppa08:16
seb128powerd08:16
seb128it's likely that the previous binary got superseeded by the repowerd ones08:17
seb128so to restore them you need a rebuild08:17
seb128no-change rebuild should do I guess?08:17
sil2100seb128: but with the old version number of powerd, right?08:17
sil2100seb128: since repowerd had a much much higher version08:18
seb128that should work I guess08:18
sil2100seb128: ok, let me try that, thanks!08:18
* sil2100 has fingers crossed08:18
seb128well, I don't think the are controls over binary versions in ppas08:18
seb128it would create issues for people who use that ppa on desktop installs though08:18
seb128apt is not going to downgrade binaries for you08:18
seb128not sure if you have many desktop users there though08:19
seb128you might have some with convergence work...08:19
sil2100hm, I think they shouldn't have any need to install powerd though, it's only for touch I suppose08:19
sil2100seb128: see! If I didn't misunderstand you and listened to you blocking the preNEW review we wouldn't have had repowerd on the images yet! ;)08:21
sil2100eh eh08:21
sil2100Since it seems it wasn't ready on various fronts and the maintainer is away08:21
seb128haha08:22
seb128yeah :-/08:22
ograsil2100, first fallout on the mailing list ... i guess that deserves an announcement for rc-proposed users08:38
Saviqjibel, hey, this is ready for QA https://requests.ci-train.ubuntu.com/#/ticket/1525 - train forgot to test unity8 on xenial and that's why it's red, we'd need to wait for robru to resolve that, but elsewhere it's all green so if possible, I'd like to get it in your queue08:39
sil2100ogra: yeah, let me follow up on that08:43
sil2100hm, not sure if this powerd bump worked08:48
sil2100seb128: strange thing that on my chroot it still tries to give me the repowerd powerd package even though it's deleted08:49
sil2100Maybe cjwatson would have some power over removal of the binaries somehow?08:49
seb128https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/stable-phone-overlay/+delete-packages?field.name_filter=repower&field.status_filter=&field.series_filter=08:50
seb128doesn't show it as deleted08:50
seb128oh, that's xenial though08:50
seb128what serie are you on?08:50
sil2100Aaaah08:51
sil2100Ok, don't mind me, eh08:51
sil2100...08:51
* sil2100 was testing on a xenial chroot08:51
sil2100seb128: anyway, when using chdist, the powerd bump didn't seem to help, now it still sees the main-archive powerd only08:56
sil2100I'm worried that it's because the version number is still smaller, but I wouldn't want to bump the old powerd to the absurd versioning of repowerd08:58
seb128well, you just uploaded so maybe it needs another publisher round09:01
seb128or maybe it has memory of the binary version and it's creating issues09:01
seb128Colin would know about that09:01
seb128but if that's the case I need you need a version bump or to use an epoch (that's they are made for)09:01
sil2100uh, never used an epoch before, ok09:02
sil2100Let me wait a bit for Colin and if not I'll do it like that09:03
Laneywhich series?09:03
sil2100Laney: vivid overlay09:03
Laneyvivid hahaha09:03
Laneysorry :P09:03
sil2100;p09:03
seb128Laney, well, it's a ppa question, not a serie one09:03
Laneypackages are published per series09:04
Laneyso you need to know which one to analyse the problem09:04
seb128Laney, powerd has version 1, repowerd was uploaded with version 6 (making up versions) which built a powerd transational binary, repowerd was deleted ... what happens if powerd 1.1 is uploaded09:05
seb128does the ppa get a powerd 1.1 binary back?09:05
seb128or does the ppa has memory of having a powerd 6 binary09:05
seb128which creates issues09:05
LaneyYou can upload a no-change rebuild of powerd09:07
Laneyit won't work properly for apt upgrades, but that's not a problem here09:07
seb128right09:07
seb128well, he did a no change rebuild09:07
sil2100I uploaded version 1.1 but it still doesn't seem to work09:08
seb128and the ppa doesn't see the powerd binary from that rebuild09:08
sil2100At least when using chdist09:08
seb128but maybe it's still a publisher cycle behind09:08
seb128wait a bit...09:08
sil2100It doesn't seem to see the overlay PPA binary at all, but yeah, I'll wait a cycle still09:08
Laney  Candidate: 0.16+15.04.20160204.1-0ubuntu209:09
Laney  Version table:09:09
Laney     0.16+15.04.20160204.1-0ubuntu2 009:09
Laney        500 http://ppa.launchpad.net/ci-train-ppa-service/stable-phone-overlay/ubuntu/ vivid/main amd64 Packages09:09
seb128well, seems to have worked then09:16
seb128Sil's env is probably buggy09:16
sil2100seb128: no, it works here as well now09:17
sil2100seb128: \o/09:17
seb128k, so just needed to wait for the publisher09:18
seb128good!09:18
sil2100seb128: thanks for the pointer o/09:18
seb128thanks Laney for the input ;-)09:18
seb128yw!09:18
sil2100And Laney too :)09:18
sil2100Ok, let me rebuild09:18
seb128it's going to bite your apt users though09:18
seb128but I guess you don't have many of those09:18
Laneynice09:18
seb128you could add a conflicts on repowerd or something if you want to force them into fixing it manually rather than staying on the wrong version09:19
seb128like having unity8 to conflicts repowerd09:20
cjwatsonsil2100: looks like everyone else beat me to it09:47
cjwatsonsil2100: though for the record I think it might have been possible to copy the old powerd over the top of itself09:47
cjwatsonnot strictly necessary to crank the version in that case (depending of course on what you want to happen with apt clients)09:48
sil2100cjwatson: btw. wow, armhf image build took 24 minutes :O09:56
sil2100Is that due to the scaling-stack switch?09:57
cjwatsonsil2100: what was the previous time?09:58
sil210045 minutes09:58
cjwatsonsil2100: right, I am unsurprised but pleased09:58
cjwatsonsil2100: the arm64 VMs are generally rather more capable09:58
sil2100cjwatson: thanks for that! It's so, wow09:58
=== chihchun is now known as chihchun_afk
mzanettisil2100, hey, got multiple people reporting that their devices don't boot any more after the latest rc-proposed upgrade11:33
mzanettiafter the bootsplash it just turns black11:33
=== fginther` is now known as fginther
=== dpm is now known as dpm-afk
Saviqsil2100, can you add zsombi to train users please? thanks12:27
zsombi+1 thx12:27
=== dpm-afk is now known as dpm
=== _salem is now known as salem_
Wellarktrainguards: "No silos available! Please ask your friendly neighborhood trainguard to free some."13:48
=== salem_ is now known as _salem
=== _salem is now known as salem_
rvrbfiller: Silo 55 (messaging app) approved14:35
bfillerrvr: thanks14:35
Laneysil2100: can haz silos?14:48
sil2100uuuu14:50
sil210081 assigned :O14:50
Saviq@unity8 ↑ if you have a silo that you could give up, please do so :)14:51
Saviqjibel, did you see my msg about https://requests.ci-train.ubuntu.com/#/ticket/1525 ? it's ready for QA but britney is confused (similar to what rvr saw before with https://requests.ci-train.ubuntu.com/#/ticket/1378 - simply no results from britney for a certain package)14:52
Saviqrobru, could you have a look at why those two are missing britney results for unity8@xenial and ubuntu-push@vivid, respectively?14:53
rvrSaviq: jibel is on holidays14:53
Saviqrvr, ah, well deserved, he should log off then ;)14:53
rvrSaviq: Another "Not considered"14:54
Saviqrvr, yup14:54
rvrsil2100: Ah, you are back14:55
rvrsil2100: There are problems with a couple of silos. Automated signoff failed because packages are "not considered". Do you know how to solve that?14:56
sil2100rvr: sadly not, Saviq already poked me about that, we need to wait for robru14:56
sil2100We need more detailed logs from britney14:56
sil2100Even pitti couldn't help14:57
rvrI see14:57
sil2100zsombi: oh, missed that message, you're added14:59
sil2100ChrisTownsend, bregma: hey! I see this silo https://requests.ci-train.ubuntu.com/#/ticket/1513 got rejected for xenial15:03
sil2100ChrisTownsend, bregma: since we're out of silos, you want to abandon/finalize it maybe and start over again?15:03
ChrisTownsendsil2100: Right, trying to get input from the Technical Board on whether the SRU can be accepted.15:04
ChrisTownsendsil2100: Hmm, I'm not sure if the Tech Board will need a reference to the deb diff.15:04
bregmasil2100, let's see if larryprice will abandon hist acolyterm silo instead, while we resolve some design issues15:04
bregmaI'd rather keep ticket 1513 until the TB meeting next Tuesday15:05
larrypricebregma, ChrisTownsend, np i'll abandon15:06
sil2100Ok, I see 2 free silos15:08
sil2100Laney: ^15:08
sil2100larryprice, bregma, ChrisTownsend: thanks for the info15:08
Laneysil2100: merci!15:08
Laneylooked to me like there were a lot of old ones15:08
sil2100Yeah, still trying to free some up15:09
sil2100Maybe we'll have more15:09
rvrsil2100: Don't publish ticket 1676, I'm checking something15:33
sil2100rvr: ACK15:33
=== chihchun_afk is now known as chihchun
sil2100rvr: give me a sign once you're done15:40
rvrsil2100: Ok15:42
rvr(ok, I'll give you a sign)15:42
rvrsil2100: Cannot reproduce the problem... weird. Go ahead.+15:48
sil2100rvr: ACK! Thanks!15:48
bfillersil2100: ping16:07
sil2100bfiller: pong16:07
bfillersil2100: any ideas why ubuntu-push says "not considered" here https://requests.ci-train.ubuntu.com/static/britney/ticket-1378/landing-054-vivid/excuses.html16:07
bfillersil2100: it's blocking the landing16:07
sil2100bfiller: sadly I don't know, this is something that pitti would need to take a look at, it's a known bug it seems16:08
bfillersil2100: known bug with britney?16:08
sil2100robru might also be able to take a look into that16:08
sil2100bfiller: known from today, multiple silos had that16:09
bfillerah16:09
bfillerrvr: ^^^^16:09
bfillerrobru: any ideas?16:09
=== chihchun is now known as chihchun_afk
rvrbfiller: Poor sil2100, we have pinged him multiple times for the same issue :)16:09
bfillersorry sil2100 :)16:10
rvrbfiller: I asked him after Saviq16:10
sil2100I did not have enough knowledge to dive into that ;p16:10
* bfiller really wants silo 54 to land :)16:10
robrubfiller: I don't know anything about it. I just enabled debug logging so pitti can dig into it soon16:10
rvrrobru: :-/16:10
robruslangasek: https://requests.ci-train.ubuntu.com/static/britney/last-run.txt can you help us interpret this? Seems pitti isn't around and landings are blocked17:00
robrusil2100: ^^17:00
robruNobody?17:13
kenvandinerobru, i had a quick look but i don't know anything about britney17:15
robrukenvandine: yeah me either. That NBS message is the problem but I don't see how that could be possible or how to fix it17:18
slangasekrobru: which bit needs interpreting there?17:25
robruslangasek: grep through for NBS. bileto exclusively uploads source packages to PPAs but somehow britney can't find the sources that built the binaries17:28
robruslangasek: actually that might be a red herring? the first one in the list with NBS actually gets approved. so basically there are a few britney failures that I have no idea what's going on at all17:34
slangasekok, which failures are the problematic ones?17:34
robruslangasek: landing-054 for starters: https://requests.ci-train.ubuntu.com/static/britney/ticket-1378/landing-054-vivid/excuses.html17:35
slangasekfwiw looking at the first 'NBS' case, it does seem that address-book-app-dbg is not built from the address-book-app package on ppc64el in silo 32; so this appears to be accurate17:36
slangasekrobru: so I guess you're looking at e.g. ubuntu-push, which is 0 days old and 'Not considered'?17:37
robruslangasek: yes, there's at least 3 silos like that, I'll have to dig them up from the scrollback17:37
slangasekok17:37
slangasekI've seen this behavior on the main p-m run also17:37
slangasekand it's new and confusing behavior with the new code base17:37
slangasekevery time I've seen it, it's been because of missing package builds17:38
robruslangasek: are you able to figure out why ubuntu-push is not considered? pitti asked that I enable verbose britney logging, so I did17:38
robrubrb17:38
slangasekrobru: the log unfortunately only shows that ubuntu-push is 'excluded' from the autopkgtest run.  looking at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-054/+packages, I'm fairly certain the cause is that ubuntu-push isn't built on arm64 and ppc64el, and britney is (rightly or wrongly) waiting for those builds17:40
slangasekrobru: yes, this explains: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/stable-phone-overlay/+packages?field.name_filter=ubuntu-push&field.status_filter=published&field.series_filter=vivid17:42
slangasekrobru: the previous version of ubuntu-push was built successfully on arm64 and ppc64el, so this is a regression, which britney will never blindly ignore.  However, the fact that it doesn't report an /explanation/ on update_excuses for why the package is 'Not considered' is a regression introduced by pitti's code upgrade17:43
robruslangasek: ok thanks17:56
robruslangasek: is it a regression compared to the archive or compared to a previous build in the PPA though?17:56
robruslangasek: because the train status code checks if it's a regression against the archive and isn't reporting anything (ticket says 'Successfully built')17:57
robruslangasek: wait, what are you looking at? I'm looking at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/stable-phone-overlay/+packages?field.name_filter=ubuntu-push&field.status_filter=&field.series_filter=vivid and I'm seeing that some old builds failed on some arches but many newer builds are built on all arches17:58
robruslangasek: oh you're saying the packages in the ticket ppa are a regression compared to the overlay ppa17:59
robruslangasek: so what's our move here? delete the packages from the overlay ppa? kick it back to landers to fix the regressions? do we care about those arches?18:00
robrubfiller: apparently your ubuntu-push has regressed on some arches ^18:03
slangasekrobru: yes, the ticket ppa hasn't built the package for all architectures that the older version of the package was built for in the overlay ppa.  So the choices here are to either delete the old ppc64el,arm64 binaries from the overlay ppa, or fix the build of the new package18:05
slangasekrobru: the fact that it's an "either/or" is why p-m isn't going to help you out here - p-m can only point out the inconsistency, not tell you what the right way is to resolve it for a given case18:06
dobeyoh18:06
dobeythe ubuntu-push merges are all wrong anyway18:06
slangasekrobru: and I guess bileto should also know to check up-to-dateness vs. the overlay ppa, not just against the main archive18:06
dobeythat silo is all wrong wrt ubuntu-push18:07
dobeyrobru: is that bfiller's silo? i only see artmello's name on it.18:08
bfillerrobru: what is failing exactly on ubuntu-push?18:12
dobeybfiller: the MPs are not how ubuntu-push landings happen18:12
dobeyfor one18:12
bfillerdobey: it is now18:13
dobeybfiller: but ppc64el failed to build on vivid (i don't recall if there were successful arm64 builds on vivid before, but if so, then it failing is also part of the problem)18:13
bfillerdobey: we changed that18:13
dobeybfiller: eh? when?18:13
bfillerdobey: no more tarmac18:13
bfillerdobey: last week18:13
bfillerdobey: we are making MR's against trunk and landing in the same way we do other silos18:14
robrubfiller: dobey: the overlay PPA has successful builds on all arches18:14
dobeyoh, and OLS are ok with that?18:14
bfillerdobey: OLS?18:15
dobeybfiller: yes, the online services team that owns the server18:15
dobeybecause lp:ubuntu-push is a mix of client and server code18:16
bfillerdobey: we told them, and my team now basically maintains the package and it's running on our jenkins instance so we want to be consistent with how we are doing everything else18:16
dobeybecause last time i suggested we should get rid of the /automatic branch a couple weeks ago, pedronis was saying it would require some work to make feasible, and there wasn't any time allocated to making it happen18:17
bfillerdobey: hmn, not aware of that, jgdx discussed with pedronis and afaik there were no issues18:18
bfillerdobey: in fact, there were major problems with automatic and tarmac - it was getting build deps live from upstream rather than ubuntu archive18:18
dobeybecause the whole reason for the complicated extra thing was because of the tree having both server and client code in it18:18
bfillerand the upstream repo is now gone18:18
dobeywell that's a separate issue18:19
bfillerdobey, robru : so back to silo 54, what is the britney issue with autopackage test?18:20
bfillerassuming this converstation is related to that18:21
dobeybfiller: the issue is that it's missing binary builds18:21
dobeybfiller: ie, before this silo, pp64el and arm64 were building on vivid, and now in the silo, they fail to build18:21
robrubfiller: dobey is correct18:21
dobeyor well, the tests failed at least, which means the packages failed to build18:22
bfillerrobru: why is silo marked then as build passed? assumed these archs were not supported18:23
robrubfiller: yeah that's a good question18:23
dobeybecause someone didn't know what was going on and asked for it to be marked ready for qa even though it isn't18:23
bfillerdobey: no that is not true18:24
dobeywhy why else would it be marked ready? :)18:24
bfillerdobey: no one can change the Status line18:24
dobeyi can change the QA Signoff right now18:24
bfillerdobey: do we know for sure those arches built previously?18:24
dobeyyes18:25
bfillerdobey: I know that18:25
dobeythey are in the archive18:25
robrudobey: the status specifically says 'Successfully built' which implies there are no arch build regressions18:25
bfillerdobey: saying the silo should not marke the build as Passed if known arches fail18:25
robrubut britney is exploding over this arch build regression. I'm trying to figure out why the status doesn't report the regression18:25
dobeyrobru: oh, well i presume that is a bug18:25
dobeybfiller: sorry, terminoligy fail. read "passed" as relating to qa, not builds being built18:26
dobeyerr, terminology18:27
robrubfiller: ok I think I see what's happening.18:29
robrubfiller: the train status code is hard-coded to check main archive for arch regressions, so ubuntu-push in vivid archive only built on a subset of arches. so the ppa contents aren't a regression compared to vivid archive18:29
robrubfiller: but stable overlay somehow built on all arches, so the ppa is a regression compared to that. britney is failing on the regression compared to vivid overlay. the fact that the britney failure message is so useless is a bug18:30
bfillerrobru: thanks for the explanation, so wonder which is correct? vivid archive or overlay?18:31
dobeyand the fact that bileto isn't comparing to the target archive, is also a bug18:31
robrubfiller: well overlay is correct because that's what phones are using18:31
dobeybfiller: overlay is correct18:31
bfillerok18:32
bfillerso we'll have to fix the tests then on those arches18:32
robrudobey: ISTR we changed this to be main_archive for a reason, but I can't remember so I'll switch it to dest18:32
robrubfiller: dobey: ok I pushed a fix, in about an hour you'll see the silo status change to 'Failed to build' and the lander approval will clear18:33
dobeyrobru: maybe to work around another bug, since it seems there are likely others, especially since main archive is hardcoded for some other things (ie, first dest in a ticket)18:33
robrudobey: no the first dest in a ticket thing is unrelated, that's just because there's no way to specify more than one dest for a ticket, so tickets with multiple dests had to be hardcoded.18:34
dobeyrobru: but was it previously trying to compare archs in overlay for the first dest in that case?18:35
dobeyin which case it would not fail, because there are no archs in the overlay for the development release of ubuntu?18:35
dobey(just a wild associative guess at maybe why the change was there) :)18:36
robrudobey: no, there is a class that defines the behavior of packages, every instance of every package class has a 'dest' attribute that correctly represents an lp archive object, either main archive or overlay ppa. if a package wants to compare itself to it's correct destination it just has to reference 'self.dest'18:36
dobeyoh ok18:37
dobeywell comment hardcoded things next time so you'll know why :)18:37
robrudobey: more likely when we open a new release the overlay is empty for that release so we have to fall back on main_archive in that case. I should probably make it check dest first and then fall back on main_archive18:37
dobeyoh ugh. this silo adds skipping of tests on some archs, though that wasn't needed before18:38
robruIt begins...19:19
=== salem_ is now known as _salem

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!