=== chihchun is now known as chihchun_afk === _salem is now known as salem_ === chihchun_afk is now known as chihchun === salem_ is now known as _salem [04:03] trainguards: I have a process question with failing autopkgtests. Anyone around who might be able to advise? [04:03] michi: what's up? [04:04] robru: Same question as yesterday. [04:04] Basically, the thumbnailer test is failing due to some package dependency I didn’t even know existed. [04:04] So, what’s the right process? [04:04] I don’t want to bug a core dev every day to kick the test until it finally passes. [04:04] michi: you need a core dev to hit retry then? [04:04] Yes. [04:04] But it might be days before this works. [04:05] So, I bug a core dev every day? [04:05] michi: only people with upload rights can retry the tests. So if it's a universe package you can get a MOTU [04:05] I don’t know what MOTU is... [04:06] It just seems really awkward and inefficient this way. [04:07] Couldn’t we have something that realizes that an autopkgtest has failed due to a missing dependency and then automatically retry periodically? [04:07] Seems that the process for an autopkg test failing due to missing dependencies is quite different from a failure when the actual test itself doesn’t pass. [04:08] I’m wondering whether we could automate the former. [04:08] michi: https://launchpad.net/~motu but check if the package is in universe [04:08] michi: you'd have to take that up with pitti [04:09] OK, will do. [04:10] BTW, the failure is due to repowerd. [04:10] Which isn’t in universe, AFAIK. [04:11] michi: isn't that new? Universe is default until it gets promoted [04:11] Not sure. [04:11] That’s part of the problem. [04:11] Our autopkgtest fails due to some problem with a package I didn’t even know existed... [04:11] michi: either way, yeah you need a person with power to hit the switch, nothing I can do about that [04:12] Sure, I understand, thanks! [04:12] michi: pitti told me he required this to avoid flooding the system, it's a necessary bottleneck apparently [04:12] You're welcome [04:12] Cool, thanks. === chihchun is now known as chihchun_afk === chihchun_afk is now known as chihchun === chihchun is now known as chihchun_afk === chihchun_afk is now known as chihchun === chihchun is now known as chihchun_afk === chihchun_afk is now known as chihchun [07:38] sil2100, morning, any idea why https://requests.ci-train.ubuntu.com/#/ticket/1525 is failed? everything seems fine but unity8 on xenial, which just has "not considered" :/ [07:41] uh [07:42] No autopkgtests no nothing [07:43] Saviq: I think pitti needs to be brought into this [07:50] ack /me → -devel [08:02] cjwatson: hello! Houston, we have a problem, need your wisdom on how to resolve this [08:04] cjwatson: situation: we had a new package called repowerd put into the overlay, this package was providing a virtual package of powerd to deprecate the existing powerd package for transition [08:04] cjwatson: but repowerd seemed to cause a lot of regressions, we decided to 'revert' it temporarily from the vivid overlay [08:05] cjwatson: since it was the only repowerd version, I reverted the dependent packages and removed repowerd from the overlay archive (so that the only thing providing powerd is now powerd itself) [08:06] cjwatson: but this backfired, since now the image builders don't see powerd in the overlay at all, possibly because they see that repowerd that was providing the latest version of powerd binary is deleted and ignores the existing powerd package in the overlay [08:06] cjwatson: and it's pulling in something archaic from the archives now [08:07] cjwatson: what can we do in such a situation? We want anyone using the overlay to be able to use the powerd provided by powerd source, with repowerd removed [08:07] (so get back to the situation from a few days before) [08:16] you probably want to reupload a proper powed to the ppa [08:16] powerd [08:17] it's likely that the previous binary got superseeded by the repowerd ones [08:17] so to restore them you need a rebuild [08:17] no-change rebuild should do I guess? [08:17] seb128: but with the old version number of powerd, right? [08:18] seb128: since repowerd had a much much higher version [08:18] that should work I guess [08:18] seb128: ok, let me try that, thanks! [08:18] * sil2100 has fingers crossed [08:18] well, I don't think the are controls over binary versions in ppas [08:18] it would create issues for people who use that ppa on desktop installs though [08:18] apt is not going to downgrade binaries for you [08:19] not sure if you have many desktop users there though [08:19] you might have some with convergence work... [08:19] hm, I think they shouldn't have any need to install powerd though, it's only for touch I suppose [08:21] seb128: see! If I didn't misunderstand you and listened to you blocking the preNEW review we wouldn't have had repowerd on the images yet! ;) [08:21] eh eh [08:21] Since it seems it wasn't ready on various fronts and the maintainer is away [08:22] haha [08:22] yeah :-/ [08:38] sil2100, first fallout on the mailing list ... i guess that deserves an announcement for rc-proposed users [08:39] jibel, hey, this is ready for QA https://requests.ci-train.ubuntu.com/#/ticket/1525 - train forgot to test unity8 on xenial and that's why it's red, we'd need to wait for robru to resolve that, but elsewhere it's all green so if possible, I'd like to get it in your queue [08:43] ogra: yeah, let me follow up on that [08:48] hm, not sure if this powerd bump worked [08:49] seb128: strange thing that on my chroot it still tries to give me the repowerd powerd package even though it's deleted [08:49] Maybe cjwatson would have some power over removal of the binaries somehow? [08:50] https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/stable-phone-overlay/+delete-packages?field.name_filter=repower&field.status_filter=&field.series_filter= [08:50] doesn't show it as deleted [08:50] oh, that's xenial though [08:50] what serie are you on? [08:51] Aaaah [08:51] Ok, don't mind me, eh [08:51] ... [08:51] * sil2100 was testing on a xenial chroot [08:56] seb128: anyway, when using chdist, the powerd bump didn't seem to help, now it still sees the main-archive powerd only [08:58] I'm worried that it's because the version number is still smaller, but I wouldn't want to bump the old powerd to the absurd versioning of repowerd [09:01] well, you just uploaded so maybe it needs another publisher round [09:01] or maybe it has memory of the binary version and it's creating issues [09:01] Colin would know about that [09:01] but if that's the case I need you need a version bump or to use an epoch (that's they are made for) [09:02] uh, never used an epoch before, ok [09:03] Let me wait a bit for Colin and if not I'll do it like that [09:03] which series? [09:03] Laney: vivid overlay [09:03] vivid hahaha [09:03] sorry :P [09:03] ;p [09:03] Laney, well, it's a ppa question, not a serie one [09:04] packages are published per series [09:04] so you need to know which one to analyse the problem [09:05] Laney, powerd has version 1, repowerd was uploaded with version 6 (making up versions) which built a powerd transational binary, repowerd was deleted ... what happens if powerd 1.1 is uploaded [09:05] does the ppa get a powerd 1.1 binary back? [09:05] or does the ppa has memory of having a powerd 6 binary [09:05] which creates issues [09:07] You can upload a no-change rebuild of powerd [09:07] it won't work properly for apt upgrades, but that's not a problem here [09:07] right [09:07] well, he did a no change rebuild [09:08] I uploaded version 1.1 but it still doesn't seem to work [09:08] and the ppa doesn't see the powerd binary from that rebuild [09:08] At least when using chdist [09:08] but maybe it's still a publisher cycle behind [09:08] wait a bit... [09:08] It doesn't seem to see the overlay PPA binary at all, but yeah, I'll wait a cycle still [09:09] Candidate: 0.16+15.04.20160204.1-0ubuntu2 [09:09] Version table: [09:09] 0.16+15.04.20160204.1-0ubuntu2 0 [09:09] 500 http://ppa.launchpad.net/ci-train-ppa-service/stable-phone-overlay/ubuntu/ vivid/main amd64 Packages [09:16] well, seems to have worked then [09:16] Sil's env is probably buggy [09:17] seb128: no, it works here as well now [09:17] seb128: \o/ [09:18] k, so just needed to wait for the publisher [09:18] good! [09:18] seb128: thanks for the pointer o/ [09:18] thanks Laney for the input ;-) [09:18] yw! [09:18] And Laney too :) [09:18] Ok, let me rebuild [09:18] it's going to bite your apt users though [09:18] but I guess you don't have many of those [09:18] nice [09:19] you could add a conflicts on repowerd or something if you want to force them into fixing it manually rather than staying on the wrong version [09:20] like having unity8 to conflicts repowerd [09:47] sil2100: looks like everyone else beat me to it [09:47] sil2100: though for the record I think it might have been possible to copy the old powerd over the top of itself [09:48] not strictly necessary to crank the version in that case (depending of course on what you want to happen with apt clients) [09:56] cjwatson: btw. wow, armhf image build took 24 minutes :O [09:57] Is that due to the scaling-stack switch? [09:58] sil2100: what was the previous time? [09:58] 45 minutes [09:58] sil2100: right, I am unsurprised but pleased [09:58] sil2100: the arm64 VMs are generally rather more capable [09:58] cjwatson: thanks for that! It's so, wow === chihchun is now known as chihchun_afk [11:33] sil2100, hey, got multiple people reporting that their devices don't boot any more after the latest rc-proposed upgrade [11:33] after the bootsplash it just turns black === fginther` is now known as fginther === dpm is now known as dpm-afk [12:27] sil2100, can you add zsombi to train users please? thanks [12:27] +1 thx === dpm-afk is now known as dpm === _salem is now known as salem_ [13:48] trainguards: "No silos available! Please ask your friendly neighborhood trainguard to free some." === salem_ is now known as _salem === _salem is now known as salem_ [14:35] bfiller: Silo 55 (messaging app) approved [14:35] rvr: thanks [14:48] sil2100: can haz silos? [14:50] uuuu [14:50] 81 assigned :O [14:51] @unity8 ↑ if you have a silo that you could give up, please do so :) [14:52] jibel, did you see my msg about https://requests.ci-train.ubuntu.com/#/ticket/1525 ? it's ready for QA but britney is confused (similar to what rvr saw before with https://requests.ci-train.ubuntu.com/#/ticket/1378 - simply no results from britney for a certain package) [14:53] robru, could you have a look at why those two are missing britney results for unity8@xenial and ubuntu-push@vivid, respectively? [14:53] Saviq: jibel is on holidays [14:53] rvr, ah, well deserved, he should log off then ;) [14:54] Saviq: Another "Not considered" [14:54] rvr, yup [14:55] sil2100: Ah, you are back [14:56] sil2100: There are problems with a couple of silos. Automated signoff failed because packages are "not considered". Do you know how to solve that? [14:56] rvr: sadly not, Saviq already poked me about that, we need to wait for robru [14:56] We need more detailed logs from britney [14:57] Even pitti couldn't help [14:57] I see [14:59] zsombi: oh, missed that message, you're added [15:03] ChrisTownsend, bregma: hey! I see this silo https://requests.ci-train.ubuntu.com/#/ticket/1513 got rejected for xenial [15:03] ChrisTownsend, bregma: since we're out of silos, you want to abandon/finalize it maybe and start over again? [15:04] sil2100: Right, trying to get input from the Technical Board on whether the SRU can be accepted. [15:04] sil2100: Hmm, I'm not sure if the Tech Board will need a reference to the deb diff. [15:04] sil2100, let's see if larryprice will abandon hist acolyterm silo instead, while we resolve some design issues [15:05] I'd rather keep ticket 1513 until the TB meeting next Tuesday [15:06] bregma, ChrisTownsend, np i'll abandon [15:08] Ok, I see 2 free silos [15:08] Laney: ^ [15:08] larryprice, bregma, ChrisTownsend: thanks for the info [15:08] sil2100: merci! [15:08] looked to me like there were a lot of old ones [15:09] Yeah, still trying to free some up [15:09] Maybe we'll have more [15:33] sil2100: Don't publish ticket 1676, I'm checking something [15:33] rvr: ACK === chihchun_afk is now known as chihchun [15:40] rvr: give me a sign once you're done [15:42] sil2100: Ok [15:42] (ok, I'll give you a sign) [15:48] sil2100: Cannot reproduce the problem... weird. Go ahead.+ [15:48] rvr: ACK! Thanks! [16:07] sil2100: ping [16:07] bfiller: pong [16:07] sil2100: any ideas why ubuntu-push says "not considered" here https://requests.ci-train.ubuntu.com/static/britney/ticket-1378/landing-054-vivid/excuses.html [16:07] sil2100: it's blocking the landing [16:08] bfiller: sadly I don't know, this is something that pitti would need to take a look at, it's a known bug it seems [16:08] sil2100: known bug with britney? [16:08] robru might also be able to take a look into that [16:09] bfiller: known from today, multiple silos had that [16:09] ah [16:09] rvr: ^^^^ [16:09] robru: any ideas? === chihchun is now known as chihchun_afk [16:09] bfiller: Poor sil2100, we have pinged him multiple times for the same issue :) [16:10] sorry sil2100 :) [16:10] bfiller: I asked him after Saviq [16:10] I did not have enough knowledge to dive into that ;p [16:10] * bfiller really wants silo 54 to land :) [16:10] bfiller: I don't know anything about it. I just enabled debug logging so pitti can dig into it soon [16:10] robru: :-/ [17:00] slangasek: https://requests.ci-train.ubuntu.com/static/britney/last-run.txt can you help us interpret this? Seems pitti isn't around and landings are blocked [17:00] sil2100: ^^ [17:13] Nobody? [17:15] robru, i had a quick look but i don't know anything about britney [17:18] kenvandine: yeah me either. That NBS message is the problem but I don't see how that could be possible or how to fix it [17:25] robru: which bit needs interpreting there? [17:28] slangasek: grep through for NBS. bileto exclusively uploads source packages to PPAs but somehow britney can't find the sources that built the binaries [17:34] slangasek: actually that might be a red herring? the first one in the list with NBS actually gets approved. so basically there are a few britney failures that I have no idea what's going on at all [17:34] ok, which failures are the problematic ones? [17:35] slangasek: landing-054 for starters: https://requests.ci-train.ubuntu.com/static/britney/ticket-1378/landing-054-vivid/excuses.html [17:36] fwiw looking at the first 'NBS' case, it does seem that address-book-app-dbg is not built from the address-book-app package on ppc64el in silo 32; so this appears to be accurate [17:37] robru: so I guess you're looking at e.g. ubuntu-push, which is 0 days old and 'Not considered'? [17:37] slangasek: yes, there's at least 3 silos like that, I'll have to dig them up from the scrollback [17:37] ok [17:37] I've seen this behavior on the main p-m run also [17:37] and it's new and confusing behavior with the new code base [17:38] every time I've seen it, it's been because of missing package builds [17:38] slangasek: are you able to figure out why ubuntu-push is not considered? pitti asked that I enable verbose britney logging, so I did [17:38] brb [17:40] robru: the log unfortunately only shows that ubuntu-push is 'excluded' from the autopkgtest run. looking at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-054/+packages, I'm fairly certain the cause is that ubuntu-push isn't built on arm64 and ppc64el, and britney is (rightly or wrongly) waiting for those builds [17:42] robru: yes, this explains: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/stable-phone-overlay/+packages?field.name_filter=ubuntu-push&field.status_filter=published&field.series_filter=vivid [17:43] robru: the previous version of ubuntu-push was built successfully on arm64 and ppc64el, so this is a regression, which britney will never blindly ignore. However, the fact that it doesn't report an /explanation/ on update_excuses for why the package is 'Not considered' is a regression introduced by pitti's code upgrade [17:56] slangasek: ok thanks [17:56] slangasek: is it a regression compared to the archive or compared to a previous build in the PPA though? [17:57] slangasek: because the train status code checks if it's a regression against the archive and isn't reporting anything (ticket says 'Successfully built') [17:58] slangasek: wait, what are you looking at? I'm looking at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/stable-phone-overlay/+packages?field.name_filter=ubuntu-push&field.status_filter=&field.series_filter=vivid and I'm seeing that some old builds failed on some arches but many newer builds are built on all arches [17:59] slangasek: oh you're saying the packages in the ticket ppa are a regression compared to the overlay ppa [18:00] slangasek: so what's our move here? delete the packages from the overlay ppa? kick it back to landers to fix the regressions? do we care about those arches? [18:03] bfiller: apparently your ubuntu-push has regressed on some arches ^ [18:05] robru: yes, the ticket ppa hasn't built the package for all architectures that the older version of the package was built for in the overlay ppa. So the choices here are to either delete the old ppc64el,arm64 binaries from the overlay ppa, or fix the build of the new package [18:06] robru: the fact that it's an "either/or" is why p-m isn't going to help you out here - p-m can only point out the inconsistency, not tell you what the right way is to resolve it for a given case [18:06] oh [18:06] the ubuntu-push merges are all wrong anyway [18:06] robru: and I guess bileto should also know to check up-to-dateness vs. the overlay ppa, not just against the main archive [18:07] that silo is all wrong wrt ubuntu-push [18:08] robru: is that bfiller's silo? i only see artmello's name on it. [18:12] robru: what is failing exactly on ubuntu-push? [18:12] bfiller: the MPs are not how ubuntu-push landings happen [18:12] for one [18:13] dobey: it is now [18:13] bfiller: but ppc64el failed to build on vivid (i don't recall if there were successful arm64 builds on vivid before, but if so, then it failing is also part of the problem) [18:13] dobey: we changed that [18:13] bfiller: eh? when? [18:13] dobey: no more tarmac [18:13] dobey: last week [18:14] dobey: we are making MR's against trunk and landing in the same way we do other silos [18:14] bfiller: dobey: the overlay PPA has successful builds on all arches [18:14] oh, and OLS are ok with that? [18:15] dobey: OLS? [18:15] bfiller: yes, the online services team that owns the server [18:16] because lp:ubuntu-push is a mix of client and server code [18:16] dobey: we told them, and my team now basically maintains the package and it's running on our jenkins instance so we want to be consistent with how we are doing everything else [18:17] because last time i suggested we should get rid of the /automatic branch a couple weeks ago, pedronis was saying it would require some work to make feasible, and there wasn't any time allocated to making it happen [18:18] dobey: hmn, not aware of that, jgdx discussed with pedronis and afaik there were no issues [18:18] dobey: in fact, there were major problems with automatic and tarmac - it was getting build deps live from upstream rather than ubuntu archive [18:18] because the whole reason for the complicated extra thing was because of the tree having both server and client code in it [18:18] and the upstream repo is now gone [18:19] well that's a separate issue [18:20] dobey, robru : so back to silo 54, what is the britney issue with autopackage test? [18:21] assuming this converstation is related to that [18:21] bfiller: the issue is that it's missing binary builds [18:21] bfiller: ie, before this silo, pp64el and arm64 were building on vivid, and now in the silo, they fail to build [18:21] bfiller: dobey is correct [18:22] or well, the tests failed at least, which means the packages failed to build [18:23] robru: why is silo marked then as build passed? assumed these archs were not supported [18:23] bfiller: yeah that's a good question [18:23] because someone didn't know what was going on and asked for it to be marked ready for qa even though it isn't [18:24] dobey: no that is not true [18:24] why why else would it be marked ready? :) [18:24] dobey: no one can change the Status line [18:24] i can change the QA Signoff right now [18:24] dobey: do we know for sure those arches built previously? [18:25] yes [18:25] dobey: I know that [18:25] they are in the archive [18:25] dobey: the status specifically says 'Successfully built' which implies there are no arch build regressions [18:25] dobey: saying the silo should not marke the build as Passed if known arches fail [18:25] but britney is exploding over this arch build regression. I'm trying to figure out why the status doesn't report the regression [18:25] robru: oh, well i presume that is a bug [18:26] bfiller: sorry, terminoligy fail. read "passed" as relating to qa, not builds being built [18:27] err, terminology [18:29] bfiller: ok I think I see what's happening. [18:29] bfiller: the train status code is hard-coded to check main archive for arch regressions, so ubuntu-push in vivid archive only built on a subset of arches. so the ppa contents aren't a regression compared to vivid archive [18:30] bfiller: but stable overlay somehow built on all arches, so the ppa is a regression compared to that. britney is failing on the regression compared to vivid overlay. the fact that the britney failure message is so useless is a bug [18:31] robru: thanks for the explanation, so wonder which is correct? vivid archive or overlay? [18:31] and the fact that bileto isn't comparing to the target archive, is also a bug [18:31] bfiller: well overlay is correct because that's what phones are using [18:31] bfiller: overlay is correct [18:32] ok [18:32] so we'll have to fix the tests then on those arches [18:32] dobey: ISTR we changed this to be main_archive for a reason, but I can't remember so I'll switch it to dest [18:33] bfiller: dobey: ok I pushed a fix, in about an hour you'll see the silo status change to 'Failed to build' and the lander approval will clear [18:33] robru: maybe to work around another bug, since it seems there are likely others, especially since main archive is hardcoded for some other things (ie, first dest in a ticket) [18:34] dobey: no the first dest in a ticket thing is unrelated, that's just because there's no way to specify more than one dest for a ticket, so tickets with multiple dests had to be hardcoded. [18:35] robru: but was it previously trying to compare archs in overlay for the first dest in that case? [18:35] in which case it would not fail, because there are no archs in the overlay for the development release of ubuntu? [18:36] (just a wild associative guess at maybe why the change was there) :) [18:36] dobey: no, there is a class that defines the behavior of packages, every instance of every package class has a 'dest' attribute that correctly represents an lp archive object, either main archive or overlay ppa. if a package wants to compare itself to it's correct destination it just has to reference 'self.dest' [18:37] oh ok [18:37] well comment hardcoded things next time so you'll know why :) [18:37] dobey: more likely when we open a new release the overlay is empty for that release so we have to fall back on main_archive in that case. I should probably make it check dest first and then fall back on main_archive [18:38] oh ugh. this silo adds skipping of tests on some archs, though that wasn't needed before [19:19] It begins... === salem_ is now known as _salem