=== chihchun_afk is now known as chihchun === chihchun is now known as chihchun_afk === chihchun_afk is now known as chihchun === chihchun is now known as chihchun_afk === chihchun_afk is now known as chihchun === chihchun is now known as chihchun_afk === chihchun_afk is now known as chihchun [07:48] Mirv, hey, think you could force-merge https://requests.ci-train.ubuntu.com/#/silo/067 ? it passed britney in silo and we're waiting for it in another silo [08:16] Saviq: sure [08:17] tx [08:33] Mirv, can you please retry the failed build in https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-070 [08:33] not sure what the error is there [08:34] Saviq: done, it seems to have hanged since the build took 2.5h [08:34] Mirv, indeed [08:39] jibel, good morning! not sure whether you saw my message yesterday in the evening about silo 21? [09:03] sil2100, hey, we'll need a new ubuntu-touch meta with indicator-session added for silo 13 - could you please whip something up? since I don't think we're installing Recommends on touch at all? [09:03] Saviq: hey, ok, preparing - the recommends bit is indeed not-so-clear, I'll have to look into that later on [09:04] ubuntu-qa it seems https://requests.ci-train.ubuntu.com/#/silo/021 is pretty important, please force it to QA queue as requested by oSoMoN in the evening since the retry function does not work for the one failed autopkgtest test because of a known infra bug [09:08] Saviq: btw. any chance silo 13 landing today? :) [09:08] sil2100, yeah, we've found the issue + workaround [09:08] it was a nasty one [09:08] first boot, timezone related and triggered by rotation :P [09:09] Mirv, done [09:09] is actually a pre-existing issue in rc-proposed [09:09] Saviq, and if you remove the calendar widget from the indicator, the bug doesn't happen? [09:10] If that's worked-around now then awesome [09:16] thanks [09:18] jibel, yes, https://requests.ci-train.ubuntu.com/#/ticket/1167 is doing that, but I'd rather land those separately and re-enable timezone change in OOBE once we verify this is really helping [09:23] Saviq, heh, one of my favorite bug :) [09:28] jibel, so what do you think, we could land 13 with the timezone skipped, or put 70 into 13 to avoid the skip, or land the two separately knowing one will fix the other [09:31] Saviq, will you remove the timezone page from the wizard still? [09:32] let me think [09:33] jibel, let's talk after the turbo mtg [09:40] jibel, https://plus.google.com/hangouts/_/canonical.com/saviq [09:50] sil2100, ↑ is rebuilding for the last time, please let me know when you have ubuntu-touch ready, thanks! [10:23] Saviq: it's ready since a bit, I can push it into the overlay if needed [10:24] sil2100, ah, can you push into silo 13 then? [10:24] or is that not how we roll with ubuntu-touch bits? [10:24] Saviq: ah, ok, to the silo first then? Sure [10:24] We can, but we usually push those straight to the overlay [10:25] Will anything blow up if we have indicator-session on our images without silo 13? [10:26] sil2100, yeah, it will show up by default and with 13 it will be hidden [10:26] so it should land together [10:27] Pushing to silo 13 in that case [10:58] sil2100, can we have one for xenial, too? [10:58] Saviq: ok, I committed to the seeds already, just need to re-generate the meta [10:58] thanks [11:29] Saviq: I hope silo 13 will land today, right? [11:29] sil2100, that's the plan, yes [11:29] just waiting for the last builds on awkward arches that take an era [11:30] sil2100, oh [11:30] sil2100, the freeze is the last blocker for silo 13, everything else reported by rvr has been fixed and verified already [11:30] Ok, since there's some additional changes staged in the xenial [11:30] sil2100, it looks like the two powerpc builds are stuck https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-013/+packages [11:30] *xenial metapackage [11:31] we saw that overnight and Mirv restarted, not sure if possible to un-stuck it before it times out after 2.5h :/ [11:31] hmmm [11:31] Why would those tests suddenly get stuck? [11:33] sil2100, looks like a race, really - something must've changed in xenial :/ [11:33] but yeah ppc64el got stuck overnight, too [11:33] * Saviq should've bincopied :P [11:33] but then train wouldn't be happy [11:35] * Mirv is intimately familiar with "stuff changes in xenial" [11:38] :| [11:40] it's just weird we still get things changing in xenial at this point, but for example the gcc gets updated all the time with the assumption that upstream stable branch never regresses [11:41] I'm now trying to hunt why out-of-range file mappings don't fail anymore on i386/armhf while they did two weeks ago at least. ie, Qt unit test is failing. [11:42] the previous case was the famous grep update a month ago [11:47] Things like this should not happen so late in the cycle [11:50] Saviq: anyone with upload permission to the relevant thing can cancel, wait for cancel to complete, retry [12:02] Mirv, Saviq: you guys think we should re-try running the builds again? [12:04] sil2100, yes [12:11] sil2100: probably [12:12] It's re running [12:12] Fingers crossed... [12:19] Saviq, Mirv: I see the ppc64el already seems hanged on the same test :< [12:29] Oh! [12:29] Saviq, Mirv: both finished! [12:32] \o/ === _salem is now known as salem_ [12:36] nice [12:36] * Saviq files a bug for that test [12:40] Mirv, sil2100, bug #1560960 [12:40] bug 1560960 in indicator-datetime (Ubuntu) "test-live-actions frequently hangs on powerpc on xenial" [Undecided,New] https://launchpad.net/bugs/1560960 [12:53] jibel, https://requests.ci-train.ubuntu.com/#/ticket/1138 is up, I've Lander: ACKed it, running qml tests and will do some sanity checks on it now,too [12:54] it also has ubuntu-touch-meta now to pull indicator-session in [12:56] Saviq, okay, I'll install is now === alan_g is now known as alan_g|lunch === chihchun is now known as chihchun_afk [13:19] sil2100: can you override the automated sign-off for the android package landing at https://requests.ci-train.ubuntu.com/#/ticket/1145 again? [13:28] jibel, FYI - tests results on silo 13 https://unity8-jenkins.ubuntu.com/job/test-ppa-autopkgtest/47/testReport/ [13:35] jibel, not sure how to cause britney to rerun the "regressions" from https://requests.ci-train.ubuntu.com/static/britney/vivid/landing-013/excuses.html and https://requests.ci-train.ubuntu.com/static/britney/xenial/landing-013/excuses.html - the only really relevant ones are for the new unity8 package (20160323.2) === alan_g|lunch is now known as alan_g [14:37] sil2100: ping [15:03] morphis: pong [15:03] sil2100: you saw my message above about the current landing of a new android pacakge? [15:04] morphis: oh crap, missed it, ok [15:04] sil2100: need a override for britney again on https://requests.ci-train.ubuntu.com/#/ticket/1145 [15:04] Let's ping QA again [15:04] thanks [15:04] jibel, davmor2: hey! We'd need a manual insertion into the QA queue, android again ^ [15:05] pffff [15:05] (the QA automated scripts for adding landings to the queue will soon stop being used because of all those missing britney hints) [15:13] sil2100: can you also have a look at the package in silo 70? [15:13] need a coredev signoff on that, this needs to land really soon [15:19] morphis: looking [15:19] sil2100: thanks [15:19] awe: ^^ [15:19] Oh, the WiFi bug, yeah that's critical alright [15:19] k [15:19] sil2100: :-) [15:30] morphis: hmmm, one thing I don't like in silo 70 so far is that debian/patches/wifi-fix-cancel-scan.patch seemed to have been touched (and some parts removed) without any mention in the changelog [15:31] Any changes such as this should be documented as a changelog entry [15:31] sil2100: they don't [15:31] sil2100: there was only one patch added [15:31] https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_39a8dbb93caf4ec889f8a1b7f69885db/bileto-1168/2016-03-23_10:34:23/vivid/network-manager/packaging_changes.diff [15:32] The packaging change shows a change in the patch - maybe the packaging diff is wrong? (maybe it's outdated or something) [15:32] sil2100: possible [15:33] Let me manually check [15:34] * sil2100 runs a debdiff [15:36] morphis: sadly manual inspection confirms that debian/patches/wifi-fix-cancel-scan.patch has been changed [15:36] morphis: was that intentional? [15:37] http://paste.ubuntu.com/15480297/ <- this is what I get after running debdiff against what's in the overlay-ppa [15:41] morphis: anyway, if it's intentional and wanted, then it should have a mention in the changelog [16:52] trainguards, hi. it seems https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-064/+build/9390812 is hanged. can someone kill/restart that build? [16:56] oh seems it just died [16:56] so just a retry please [17:16] trainguards: also, can someone please retry https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-066/+build/9386439 ? [17:16] dobey: on it [17:16] Done [17:16] sil2100: can you do https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-064/+build/9390812 too please? [17:19] robru: I create silo 62 for QA. It only has a click package. Getting bad status and also need to set QA Signoff to Ready. Can you take a look? [17:19] https://requests.ci-train.ubuntu.com/#/silo/062 [17:20] hi trainguards! I need to land this MP: https://code.launchpad.net/~xavi-garcia-mena/ubuntu/vivid/upower/percentages-power-off/+merge/289935 [17:20] alecu: oh, that isn't going to work [17:20] I've never landed upower branches nor branches that are not upstream, is that something that we can do with the train? [17:21] dobey: how so? [17:22] alecu: because that's not upstream; we need to make a change to the debian source package, and get it uploaded [17:23] dobey: uploaded to debian? we want this change only for the phone overlay. [17:23] alecu: no, uploaded to a silo; that change needs to not be directly to the files in question, but adding a patch in debian/patches/ that makes that change. [17:24] dobey: ah, I see [17:24] and needs to have debian/changelog entry, etc [17:24] as we don't own upower [17:26] dobey: sure [17:26] sil2100: thanks [17:26] alecu: oh [17:26] alecu: ok, where would you like me to copy the source package? [17:26] alecu: is it critical to land that upower thing for ota10? [17:27] wtf, the ubuntuone-credentials tests crashed again on powerpc [17:27] alecu: and yes, as dobey, it would be better if the change was already a valid debian package, so added as a quilt package etc. [17:27] This would save me some work [17:28] sil2100: can retry https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-066/+build/9386439 yet again please? i wonder if maybe something is wrong with the builder perhapse :-/ [17:28] dobey: sure [17:28] dobey: mzanetti keeps bugging me about it. He has half a dozen devices, and whenever he needs to test something on one, and the device is completely dead, it takes 15 minutes for the battery to charge enough so the device can be restarted. [17:29] dobey: yeah, we had some issues with powerpc builds today... not sure if those were builder related, just was hard to get a package built completely [17:29] sil2100: how do silos that just have click packages work? [17:29] sil2100: the status keep saying it's failed [17:29] sil2100: silo 62 [17:29] dobey: so, I don't think it's a critical, but it's long in the tooth. [17:29] bfiller: silos with clicks don't work, really [17:30] dobey: it's how QA tests them [17:30] not for release [17:30] bfiller: hey! Yeah, for such click-only landings that do not have any deb-parts to build it should just stay as an unassigned request [17:30] sil2100: dobey: I'll work on a proper debian branch after lunch. [17:30] bfiller: right, you just make a silo and fill in the download url for the click, and then ask jibel to move it to trello [17:30] bfiller: since getting a silo is pointless [17:30] (nothing to build or install) [17:30] alecu: thanks! === olli_ is now known as olli [17:31] sil2100, dobey: ok let me abandon it then and start over [17:33] alecu: ok, if you need help from me, just ping; i am a bit hesitant about it though; would be nice if we could get a proper fix implemented where we can tweak that in device tarball or such to override, instead of having it that way in the package [17:37] dobey, its just that its for every device, there is some other package with config overrides which one escapes me [17:41] alecu, just send some extra power strips to mzanetti ;) [17:45] pmcgowan: oh right, well whatever package then perhaps [17:46] pmcgowan: i'm not sure that upower settings can be overridden the normal way though, is the thing [17:47] sil2100: looks like that ppc build failed again :( same builder, so maybe an issue with the builder [17:47] dobey, yeah looking now [17:47] and aiui its bad to write a file another package owns [17:48] hmm [17:48] https://launchpad.net/builders/sagari/+history doesn't suggest a general builder problem though :( [17:53] sil2100: can you kick off an amd64 xenial build for gallery-app on this silo please? https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-051 [17:59] ogra_: lol [18:01] bfiller: done [18:01] :) [18:08] sil2100: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-066/+build/9386439 <- hmm, third time's a charm maybe? [18:09] ogra_, is ubuntu-touch-session really reserved for session related configs or can stuff running as root go in there as well [18:10] pmcgowan: there's "ubuntu-touch-settings" btw [18:11] * pmcgowan looks [18:11] don't know if unity8-desktop-session-mir depends on it or not though [18:11] if it pulls it in we might not want it there, either [18:12] or, trainguards: can get another rebuild on https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-066/+build/9386439 please? [18:12] dobey: on it [18:12] thanks [18:13] yw === alan_g is now known as alan_g|EOW [18:14] \o/ [18:16] dobey, Im confused, the upstream project for that package is something unrelated [18:17] thanks for your work on this silo mterry, it was an interesting landing :) [18:18] jibel: likewise. glad we sorted it out [18:18] pmcgowan: ? [18:19] dobey, if I look here https://code.launchpad.net/ubuntu/+source/ubuntu-settings [18:19] well fml, another segfault on ppc [18:19] it points to an upstream for the code [18:19] robru, where do I learn the reason why automated sign-off failed? https://requests.ci-train.ubuntu.com/#/ticket/939 [18:20] jhodapp: "Automated Test Results" field is a little below that [18:20] i wonder what changed in xenial that would have caused such a crash only on ppc [18:20] robru, thanks, didn't see that [18:20] mterry, can you please copy shadow for vivid from silo 13 to overlay - we'll then need to pop it from silo and publish the rest [18:21] Saviq: oh ok [18:21] mterry, somehow you managed to put different sources for xenial in silo and archive [18:21] pmcgowan: huh, no. there isn't an upstream source tree for that afaict [18:21] so train is unhappy [18:21] pmcgowan: it's just a .deb that has some overrides we ship, in it [18:21] Saviq: yeah I pushed to xenial a long time ago, I was told to ease publishing just to put a bogus one in ppa. Guess tnot [18:21] dobey, ack [18:22] lp pahge is just wrong then [18:22] pmcgowan: not exactly; someone improperly registered an upstream project in lp [18:22] mterry, well, I asked you to copy from archive to silo, not "put a bogus one" - train would be happy with that :) [18:23] mterry, but you must've rebuild the source when dputting to ppa or something [18:23] Saviq: yeah it got rebuilt. I'll fix [18:24] pmcgowan: fixed it [18:25] Saviq: OK, shadow copied to overlay & deleted from silo 13 ppa [18:25] mterry, it's not even that "it got rebuilt" as in binaries - it's the source package that ended up different [18:25] mterry, thanks, deleted both distros? [18:25] Saviq: alright I must have messed that up [18:25] Saviq: yeah [18:25] tx [18:26] let's see if a diff_only build makes things work now [18:26] i wish lp builders could be configured to upload crash reports to errors.u.c and spit out the URL to the report in the build log [18:29] maybe i should just disable tests on powerpc [18:30] pmcgowan: so i can't land that ubuntuone-credentials change if i wanted to, because the build keeps failing on powerpc :( [18:31] dobey, I see powerpc tests on xenial bah [18:32] dobey, who could help look at it? [18:32] robru: was discussing with jibel, he was asking if any lander should be able to mark it ready for QA when there is no automated sign off [18:33] bfiller: hmmm, right [18:33] pmcgowan: someone with a 15 year old mac that still boots? [18:33] jibel, davmor2, thanks a lot guys, we know we caused you a lot of grief these past week+½, we really didn't wanna [18:33] dobey, lol [18:34] dobey, how bad would it be to disable the tests [18:34] pmcgowan: it's not hard to do. just feels incredibly dirty [18:35] pmcgowan, we used to use it for all things session related ... iirc there are also lightdm configs and such in it [18:35] Saviq, no worries, you know how we love your mega silos [18:35] ogra_, there are [18:35] so indeed you can have non-user related bits [18:35] vg ty [18:35] alecu, ^ [18:35] food for thought for your next landings [18:35] i wouldnt put totally unrelated stuff in there indeed [18:35] ogra_, it an override for the upower conf [18:36] jibel, others just hide the same size silos behind staging, so ;) [18:36] pmcgowan: it would be better in ubuntu-touch-settings [18:36] yeah [18:36] pmcgowan: unity8 depends on ubuntu-touch-session [18:36] that sounds less session realted [18:36] *related [18:36] jibel, I'd have landed all that in 4-5 smaller ones if I could [18:36] so it will be installed on laptops too for unity8-desktop-session-mir [18:36] dobey, ok if you say so, just there are no conf files there now [18:36] I see [18:36] but ubuntu-touch-settings doesn't seem to be [18:36] Saviq, indeed, I didn't say it couldn't have been worse [18:37] pmcgowan: there is one override file in it. we can add a conf though, i think :) [18:37] dobey, so I would disable the tests on powerpc for today, since that is wanted for turbo [18:37] bfiller: ok I pushed a fix to trunk, just waiting on #webops to roll it out, they're a bit busy (been waiting already an hour for a different rollout) [18:37] pmcgowan, and breap all the powerpc tablets out there ?!? [18:37] *break [18:37] pmcgowan: do we not also want silo 64 for turbo? who is determining what is "critical" there? [18:38] whats in 64 [18:38] robru: awesome, ty!! [18:38] bfiller: yw [18:38] jhodapp: sorry got a bit distracted. did you find the test results there? does the error make sense to you? [18:38] dobey, pmcgowan, I just filed bug #1560960 today about test failures on powerpc/ppc64el [18:38] bug 1560960 in indicator-datetime (Ubuntu) "test-live-actions frequently hangs on powerpc on xenial" [Undecided,New] https://launchpad.net/bugs/1560960 [18:38] dobey, I think we are ok with 64 for the ota [18:38] robru, yeah I've pinged Saviq and we're trying it again [18:39] thanks for checking [18:39] this sounds like something happened in xenial these past few days [18:39] you're welcome. I know that excuses page can be hard to read at times [18:39] OTOH it was a hang for i-datetime, this is SIGSEGV [18:40] dobey: the unity8 desktop session does not include ubuntu-touch-settings? [18:40] alecu: rdepends says no [18:40] great [18:40] dobey, did you guys try to build once more? [18:40] only ubuntu-touch depends on it [18:40] Saviq: this is the 4th time it's segfaulted [18:40] I mean just a retry here https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-066/+build/9386439 ? [18:40] ok :/ [18:40] I just need to figure where to put the env var in order for the system dbus-daemon to pick it. [18:41] i suppose i could ask to try again [18:41] but i guess it will crash again [18:41] yeah no point [18:43] oh come on train, whatch'a doin'? [18:43] we're waiting here [18:44] Saviq: it's a slooooooooooooooooooooooooooooooooooooooow train comin' [18:53] Saviq: dobey: what are you waiting for? [18:54] robru: someone to acknowledge my bob dylan reference === chihchun_afk is now known as chihchun [19:05] Saviq: I can publish [19:05] Would be my pleasure [19:06] mterry, please do, just wanted to run the YOU SHALL NOT PASS run so it's clear [19:06] Saviq: OK. pushing button.... [19:12] mterry, w000t [19:12] jibel: ref Saviq s earlier gratitude that's one and half weeks of drink on him right? That's the way it works now isn't it? [19:13] davmor2, not sure you want me to provide you 1½ weeks of drinking [19:13] remember I come from a ~Eastern European country [19:14] Saviq: I only drink coffee and soft drinks not an issue for me :P === daker_ is now known as daker === tedg_ is now known as tedg [19:26] now what [19:28] sigh [19:31] great success === cking is now known as king-afk [19:52] alright fellas, big train rollout is live, hope y'all are ready for a 3x speedup in source package builds. === robru changed the topic of #ubuntu-ci-eng to: Train trouble? ping trainguards | CI problems? Use JenkaaS: http://bit.ly/jenkins-docs | Train: http://bit.ly/1hGZsfS | QA Signoffs: http://bit.ly/1qMAKYd | Known Issues: experimental source package build code now live. ping robru at first sign of trouble [19:55] bfiller: ok that qa change should be live, try poking the qa field to see if it works (page should reload automatically) [19:55] robru: cool let me try [19:56] robru: seems to work, thanks [19:56] bfiller: you're welcome! [20:01] alright! huge experimental train rollout done, time to disappear for lunch! what could possibly go wrong? [20:02] reentrant call errors? [20:10] robru, what is the problem with this silo? https://requests.ci-train.ubuntu.com/#/ticket/1123 [20:23] renatu: "burned version number" means that the destination archive (in this case overlay PPA) contains a package with the same version number but with different source contents. the only fix is to rebuild so that you get a new version number. [20:24] robru, just push a new build? [20:24] renatu: most likely this happens when you have the same package in two silos, they both build on the same day (so they get the same version number), then one publishes, it "burns" the other one [20:24] robru, ok thanks [20:25] renatu: it's weird that xenial doesn't also indicate that it's burned, lemme look a little deeper [20:26] renatu: yeah these are different: https://launchpadlibrarian.net/249683044/indicator-datetime_15.10+15.04.20160323-0ubuntu1_source.changes https://launchpadlibrarian.net/249723640/indicator-datetime_15.10+15.04.20160323-0ubuntu1_source.changes [20:27] renatu: yeah the one in the overlay is from silo 13 [20:28] ok got it from trunk now [20:28] renatu: so if you rebuild now, it'll fix the "burned version number" error, but you're still inherently dealing with two conflicting silos, once silo 13 lands fully you'll need to rebuild again anyway, so maybe just wait [20:28] robru, sure, thanks [20:29] renatu: wait, 13 isn't assigned, wtf [20:29] robru, something has landed I got updates from trunk [20:30] renatu: my mistake, 13 is fully landed, so you should rebuild now [20:31] renatu: lol, that's the status I was expecting. 13 must have *just* landed [20:32] robru, nice, turnk merged. Rebuilding it right now === salem_ is now known as _salem [21:35] robru, o/ [21:35] tvoss: ahoy! [21:35] robru, mind checking if the publisher for https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-074/+packages is working correctly? [21:36] ping trainguards. I need to ship an alternate /etc/UPower/UPower.conf on the phone. And I see that several conf files in /etc are overridden with bind mounts from /userdata/system-data/etc/ [21:36] tvoss: i dunno anything about the publisher, but i did make a huge change to the train today, let me take a poke [21:36] so, do you guys know where those things in /userdata/system-data are coming from? they don't seem to be part of an installed .deb [21:36] alecu: no idea, sorry. That's a question for phone people, not train people [21:36] Though maybe sil2100 is both... [21:37] robru: ah, there lies my confusion, I usually ask these things to sil2100! [21:37] robru: thanks anyway[ [21:37] you're welcome [21:41] tvoss: everything looks fine on the train side. only armhf hasn't published (in the ppa) yet, so maybe just wait a bit more? if it's still not published in half an hour, maybe poke colin watson or william grant [21:43] robru, ack and thx [21:46] tvoss: yw [21:51] alecu: oh maybe check with john-mcaleely, that might be from the device tarball.