/srv/irclogs.ubuntu.com/2015/09/22/#ubuntu-ci-eng.txt

robruoh holy jesus00:29
tedgrobru: yes?00:30
robrutedg: I just did something like rm -rf $HOME on the production train00:30
tedgAh, that sucks.00:31
robrutedg: will check with webops for backups, seems a lot of people might need to rebuild silos00:32
tedgrobru: Good luck!00:33
robrutedg: hm actually it looks like I cancelled it before it deleted more than a couple silos... will try to figure out which silos got nuked and which are expected to be empty...00:42
tedgWhen you play with silos, sometimes the nukes go off.00:43
robrutedg: I'm literally implementing a new way to abandon silos. some variable got set wrong and it started deleting all silos instead of just one.00:44
tedgrobru: http://www.quickmeme.com/img/d6/d689b760f2debdb9d4cad9b7f755f8113d041875f98d7f2e8f1a755e6dc239ed.jpg00:48
tedg;-)00:48
robrutedg: I have unit tests and I tested this in staging! It's just a Monday, that's all00:48
tedgWell, it's also one of my favorite memes :-)00:49
=== chihchun_afk is now known as chihchun
=== chihchun is now known as chihchun_afk
Mirvmorning04:07
=== chihchun_afk is now known as chihchun
oSoMoNrobru, why on earth did you trigger a new build for silo 35?04:46
robruoSoMoN: I just sent out an email about that04:46
robruoSoMoN: TL;DR: that silo got deleted so I had to rebuild it04:46
oSoMoNrobru, it took me 6 attempts to get it to build successfully last night, and with your build it failed again04:47
robruoSoMoN: ohhh nooooo04:47
robruoSoMoN: I'm sorry dude.04:47
robruoSoMoN: I had no choice, the silo was accidentally deleted so there would be no way to publish it04:48
oSoMoNunderstood04:48
robruoSoMoN: is there anything I can do to help get it back into a good state? since I broke it and all04:50
oSoMoNrobru, not much I’m afraid, there is one unit test in the webbrowser-app test suite that started failing randomly yesterday, with no related change, and of course a normal PPA build doesn’t give any useful output, so I don’t know why it fails04:51
oSoMoNthat specific unit test (like all the others) had been rock-solid before yesterday04:51
oSoMoNit’s a complete mistery04:51
MirvoSoMoN: stop trying random rebuilds, let me (/robru) retry individual archs until it succeeds04:51
oSoMoNrobru, do you know by any chance if the pbuilders for the silos have been upgraded very recently?04:51
oSoMoNMirv, ah, that would be awesome, I didn’t know you could do that04:52
Mirvso wily ok already, now just amd i386 for vivid04:52
oSoMoNMirv, yes please04:52
MirvoSoMoN: we try to advertise every time possible that pleeease don't rerun build job for all archs when some archs fail, just get us to retry :)04:52
robruoSoMoN: not that I know of, but I only know about the train side (which only builds source packages) actual PPA builds would be a question for LP people04:52
oSoMoNMirv, sorry I must have missed all those announcements, I guess that’s because my silos never failed to build before yesterday :)04:53
robruoSoMoN: and actually colin has been poking at the builders I think... ;-)04:53
MirvoSoMoN: that's a good reason :)04:53
oSoMoNMirv, so you re-triggered both i386 and amd64 for vivid?04:54
MirvoSoMoN: yes04:55
oSoMoNMirv, both failed…04:58
MirvoSoMoN: I'm afraid it might be because of this https://launchpadlibrarian.net/218473113/qtbase-opensource-src_5.4.1%2Bdfsg-2ubuntu8_5.4.1%2Bdfsg-2ubuntu9.diff.gz - I'd need to know if it needs to be rolled back, but it seemed an important fix fixing a previous (very important) fix that was https://codereview.qt-project.org/#/c/110150/04:59
MirvoSoMoN: the retries are cheap, but it'd be very useful to know if the behavior is ~correct or not. all webbrowser AP:s passed among else.05:00
oSoMoNMirv, ah, thanks for pointing at that, I’ll see if I can reproduce locally with this patch05:01
oSoMoNMirv, in the meantime, can you please retry the failing jobs? they will eventually pass…05:01
MirvoSoMoN: yes, it's in overlay PPA now. see also the patch description if it helps you understand what has changed. the patch is fixing reported Connection Closed / etc problems leading to for example missing images on web pages (on another browser)05:02
MirvoSoMoN: sure05:02
MirvoSoMoN: the good news of course is that it always succeeded on armhf05:12
oSoMoNMirv, yes, the failures are only on amd64 and i386, never on armhf, and they are random05:15
oSoMoNMirv, both failed again :/05:16
oSoMoNMirv, it looks like the i386 build succeeded, can you retry the amd64 one only?05:26
MirvoSoMoN: yes, you don't need to poll it, I handle it :)05:30
oSoMoNMirv, awesome, thanks05:31
oSoMoNMirv, I’ll be offline for the rest of the morning, back online this afternoon, if anything comes up please send me an e-mail. Thanks!05:31
Mirvok05:32
robruwell that looks good06:17
robruMirv: thanks for doing those rebuilds, that was really my mess06:17
robruoooh, ticket 40006:58
Mirv:)07:00
robruat one point I was like "I dunno man, what about integer overflow?" and then I realized you'd have to create a new ticket every second for 80 years for that to be an issue.07:05
brendandrobru, careful that's what they said about storing dates07:08
robrubrendand: I made sure to use more than 2 digits ;-)07:10
dbarth_good morning07:16
dbarth_i made proper merge proposals for https://requests.ci-train.ubuntu.com/#/ticket/396 but assign still does not like me07:16
Mirvdbarth_: I think we have a ghost silo, just a moment07:19
Mirvrobru: I noticed the ONLY_FREE_SILO does not work anymore07:20
robruMirv: it... doesn't work? it shouldn't exist. I covered that in my recent mail07:21
robruMirv: are you trying to free dbarth_'s request? because it can be fixed in bileto without freeing07:23
robruerrr07:23
Mirvrobru: it's there, the option, but it seems to try normal merge&clean instead. the problem is that if the bileto doesn't know of a silo, like I think is the case with silo 041, I can't free it up any way https://ci-train.ubuntu.com/job/ubuntu-landing-041-3-merge-clean/6/console07:23
Mirvso https://ci-train.ubuntu.com/job/prepare-silo/6222/console lead me to think of another ghost silo07:23
Mirvhmm, no, that's a normal silo07:24
Mirvoh, now it has a silo07:24
Mirvsoooo... I think just confusion after confusion, but no actual problem :)07:24
robruMirv: "ghost silo" is not a ghost, it is really fully assigned in jenkins for 100% real and solid. it just doesn't say so in bileto.07:24
robruMirv: I'm not sure why merge&clean job still has ONLY_FREE_SILO option, that should not be there at all07:25
Mirvrobru: yeah, it shouldn't have it since it's just ignored07:25
Mirvand now we have the abandon job07:25
robruMirv: ugh, just missed the auto-rollout cutoff07:27
robruMirv: that option will go away in an hour07:27
Mirvgood! :)07:28
Mirvdbarth_: so in short, you have a silo07:28
robruMirv: if you see any ghost silos again, do the prepare, and it'll say "request found in /whatever/silo", just copy&paste the ubuntu/landing-NNN into the siloname field in bileto.07:28
Mirvright, indeed07:30
dbarth_robru: Mirv: thanks!07:32
robrudbarth_: you're welcome07:32
robruhm08:32
cjwatsonrobru: I see that it's been handled, but anyway, we haven't deployed any new buildd code for a month and a half08:49
cjwatsonaside from the new ppc64el virt builders, but silos aren't using those yet08:49
robrucjwatson: ah OK, just knew you were working on "stuff"08:50
cjwatsonI do that sometimes08:50
cjwatsonAlso things08:51
robrucjwatson: do you always read all the scrollback it so you have a highlight on "colin"? ;-)08:52
robru"Or do you have"08:52
robru(On phone)08:52
cjwatsonI don't read all scrollback in every channel I'm in, but I often page through a fair bit of it quickly08:53
cjwatsonI don't highlight on "colin", no :)08:54
sil2100Colin gets highlight pings on 'stuff' and 'things' though ;)09:22
robruMirv: ok I finally killed ONLY_FREE_SILO, and with that I'm going to go pass out. goodnight!09:32
Mirvrobru: thanks!09:48
jgdxfginther, hey, ci runs takes 4-8 hours just to start. What's up? :)09:55
=== chihchun is now known as chihchun_afk
rvrpstolowski: ping11:17
pstolowskirvr, pong11:17
rvrpstolowski: Silo 2511:19
pstolowskirvr, yes?11:19
rvrpstolowski: Is there anything to check?11:19
rvr"Scopes API fix: Loop through each argument of the custom scope runner command and ensure that all path arguments are absolute."11:19
pstolowskimarcustomlinson, ^ ?11:19
pstolowskirvr, in general - not really, but perhaps marcustomlinson has an idea11:23
marcustomlinsonrvr: hey, the fix in silo 25 is really for the SDK, it allows debugging of scopes who's runner executable is relative to the click path12:02
marcustomlinsonrvr: little tricky to test, but running through the test plan is good enough12:02
=== alan_g is now known as alan_g|lunch
marcustomlinsonrvr: which I did yesterday on krillin 128 and looked good12:03
rvrmarcustomlinson: Ack12:10
marcustomlinsonrvr: fyi, here's a test to confirm it works: http://paste.ubuntu.com/12520739/12:17
marcustomlinsonrvr: but like I said its tricky. I just tried this test myself (now on krillin 129) and works just fine12:17
=== alan_g|lunch is now known as alan_g
fgintherjgdx, do you have any more context, such as an MP that was slow? I don't see anything unusually right now although jobs requiring makos have been delayed due to a shortage of devices13:38
jgdxfginther, yeah, it was the shortage of devices I was referring to. Any plans for more devices?13:39
fgintherjgdx, we've already requested more. Just waiting now on the approval/installation process to move forward. Sorry that I don't have an ETA on when things might be better.13:41
jgdxfginther, okay. More makos or other devices too?13:42
fgintherjgdx, Other devices13:42
fgintherjgdx, makos are getting harder to find13:42
fgintherbut not impossible13:43
jgdxfginther, thank you.13:43
pstolowskiMirv, hey, silo 24 has been in proposed pocket for several hours and not merged yet, any idea when is it going to actually land?14:17
=== pat_ is now known as Guest18418
cjwatsonpstolowski: it's awaiting an autopkgtest on armhf: http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html#unity-scopes-shell14:29
cjwatson13:41 <pitti> Riddell: see scrollback from 1.5 h ago; I retried them, queues are catching up14:29
cjwatson13:41 <pitti> but ARM still has some ~ 290 tests to grind through14:29
cjwatsonpstolowski: ^-14:29
pstolowskicjwatson, ah, i see, thanks for the info!14:31
pstolowskialecu, ^14:31
oSoMoNMirv, I’m looking at the unit test failure for webbrowser-app in silo builds, I was hoping that the changes you pointed me to in QNAM would be the culprit, but I can’t reproduce the failure locally :/14:42
Saviqcihelp, hey, our only remaining failing jobs on s-jenkins are wily autopilot for unity8, and it falls over completely https://jenkins.qa.ubuntu.com/job/generic-deb-autopilot-runner-wily-mako/417/console14:45
Saviqlooking at the log it failed to unlock the screen in the first place (couldn't connect to dbus)14:46
Saviqit looks as if it can't even talk to the session upstart14:47
fgintherSaviq, hey14:49
fgintherSaviq, it looks like the screen unlock has been failing for a while14:50
fginthermterry, Is there a known issue with the automated screen unlock on wily? ^14:51
mterryfginther, not known, no14:51
bzoltanMirv:  do you know what is the state of the silo38?14:53
fginthermterry, are you still the best contact for the screen unlock? If so, shall we just open a bug report to track this?14:54
mterryfginther, uh I suppose I am14:58
oSoMoNMirv, I finally managed to reproduce the failure locally, looking into it now14:58
mterryfginther, yes please open bug.  I assume that's a pain point with some urgency?  I'm in the middle of something else now though14:58
fginthermterry, Saviq pointed out the issue and it's mostly hitting unity8. I'll defer to him on the urgency of this problem.14:59
MirvoSoMoN: thanks!15:04
Mirvbzoltan: well wily part is broken because of the GCC bug, vivid ok15:04
Mirvbzoltan: robert just accidentally deleted the silo so it was rebuilt by him as announced in e-mail15:04
bzoltanMirv:  I have read the mail. That was not causing any problem for us. I just wonder if the silo could be released now.15:05
Saviqfginther, mterry, it'd be good to have, but truthfully I don't think the problem is in the unlock code, there seems to be something wrong in the environment maybe, in a way that nothing can access the session upstart (or session dbus for that matter)15:09
oSoMoNMirv, and FYI, I filed bug #1498539 to track the issue15:11
ubot5bug 1498539 in webbrowser-app (Ubuntu) "FaviconFetcherTests random failures in silo builds" [Critical,In progress] https://launchpad.net/bugs/149853915:11
fgintherSaviq, mterry, for reference, the bug report is https://bugs.launchpad.net/ubuntu/+source/unity8/+bug/149854115:11
ubot5Ubuntu bug 1498541 in unity8 (Ubuntu) "Automated screen unlock not working on wily images" [Undecided,New]15:11
mterryfginther, thanks.  will put on my list to look at15:15
fginthermterry, thanks15:16
=== Guest18418 is now known as pmgowan
=== alan_g is now known as alan_g|EOD
Saviqfginther, I followed http://ubuntu-test-cases-touch.readthedocs.org/en/latest/ on wily and it's fine, not sure what jenkins runs might be doing different, but evidently the shell env is broken https://bugs.launchpad.net/ubuntu-test-cases/+bug/1498541/comments/317:28
ubot5Ubuntu bug 1498541 in unity8 (Ubuntu) "Automated screen unlock not working on wily images" [High,Incomplete]17:28
Saviqhmm --revision=276?17:31
Saviqfginther, jenkins is hardcoded at ↑17:32
* Saviq tries17:32
fgintherSaviq, looking17:36
fgintherSaviq, ugh... good catch. I'll remove that and give this another try17:37
* Saviq can't shake the feeling this is not the first time...17:40
Saviqwe need a better process for this (I imagine you hardcoded an image because there was breakage in further versions?)17:41
fgintherSaviq, indeed, this was done to workaround an unbootable image. I usually set a reminder to revisit and remove the hardcoded version, but I failed in this case.17:45
Saviqfginther, just realized, current image might not be good to use either, it doesn't reboot18:06
Saviqgets stuck after stopping services (so adb is gone, and won't be back until you forcefully reboot the device)18:06
fgintherSaviq, :-(18:35
fgintherSaviq, it appears to be working here (with image 311) - http://s-jenkins.ubuntu-ci:8080/job/generic-deb-autopilot-runner-wily-mako/421/console18:37
fgintherunlock too18:37
Saviq fginther, oh good18:43
Saviqthat looks promising then18:43
Saviqjeez system-image is slow recently :/18:49
Saviqand it timed out @ 97% for me...18:49
bfillerrobru: can you publish silo 53, just pot file changes19:44
bfillerfor string freeze19:44
robrubfiller: sure can't! You need a core dev for that19:45
bfillerrobru: how is it different than a normal silo that qa approves?19:46
robrubfiller: it isn't, the rules changed. I no longer have any special publishing powers over what you have, as per the email i sent out last night19:47
* bfiller reads email19:47
robrubfiller: also there's packaging changes, so it's not "just pot files"19:47
robrumterry: kenvandine: either of you around? https://ci-train.ubuntu.com/job/ubuntu-landing-025-2-publish/65/ needs ack, has new binary packages & symbols, huge diff too.20:08
robrubfiller: oh you're rebuilding 53?20:11
bfillerrobru: yes, then will hand it to QA. let them verify it and publish20:13
robrubfiller: well, QA can't publish either. it has to be a core dev.20:13
kenvandinerobru, i'm confused... silo 25 looks like a diff i already reviewed :)20:13
bfillerrobru: ok, now I'm confused. whatever the process is for publishing silos.. this one should be no different20:14
robrukenvandine: it's possible, I accidentally deleted some silos recently, and then when I was restoring them I potentially restored extra ones by mistake.20:14
kenvandinewe even had quite a bit of discussion over the breaks and replaces... etc20:15
kenvandine-Depends: libunity-scopes1.0 (= ${binary:Version}),20:15
kenvandine+Depends: libunity-scopes3 (= ${binary:Version}),20:15
robrubfiller: right, the process is that it's your responsibility to find a core dev to publish. if you need help I have a couple go-to guys (ken & mike) but generally publishing is your responsibility. you are empowered!20:15
bfillerrobru: hmnn, sounds like a step backwards. I just want to hand it off and when QA passes it should happen autoamtically20:16
bfillernot have to find someone to do it20:16
robrubfiller: no, it's the same. previously you had trainguards publishing, now you have core devs publishing. it's still a manual step20:18
bfillerrobru: ok20:18
robrukenvandine: well if you approved it in the past, it seems it was never published20:18
kenvandinemaybe there was an issue then... i really don't recall20:18
robrukenvandine: please re-review & publish then? ;-)20:19
kenvandinei'm not convinced it wasn't published20:20
kenvandine-Package: libunity-scopes1.020:20
kenvandine+Package: libunity-scopes320:20
kenvandineand the package in the overlay ppa has a libunity-scopes3 binary20:20
robrukenvandine: huh? the silo version is days ahead of the wily version or the overlay ppa version20:20
robrukenvandine: the diff is against wily20:20
kenvandinerobru, yeah... i see that20:20
kenvandinethe debian/changelog version is20:21
kenvandinebut the diff shows binary package renames20:21
kenvandinethat have already landed20:21
kenvandinerobru, and shouldn't that diff be from the package in the overlay ppa?20:22
kenvandine--- unity-scopes-api-1.0.1+15.10.20150915.1/debian/changelog20:22
kenvandine+++ unity-scopes-api-1.0.2+15.04.20150921/debian/changelog20:22
kenvandinelooks like it is20:22
kenvandinebut... again... that new binary package is already in the latest build in the overlay ppa20:22
robrukenvandine: no dual silos should be diffing against wily as far as I know...20:22
kenvandineoh this is a dual landing20:22
kenvandinei think we landed this same thing for vivid last week20:22
robrukenvandine: maybe they landed in overlay on the 15th and are re-syncing dual today?20:23
kenvandineyeah20:23
kenvandinethat's what i'm thinking now20:23
robrukenvandine: hang on a second don't publish yet20:24
robruohtoo late, nevermind20:24
robrulet it go20:24
kenvandineyeah... i did :)20:25
kenvandinei spent a bunch of time reviewing this once20:25
kenvandinefor a vivid only landing20:25
robrukenvandine: it seems like there's a bug in the diff generation, it seems to be diffing the vivid local build against the wily version, that's definitely wrong. I guess it was never an issue before because the only thing that should be different between wily & vivid in a dual silo is the first line of the changelog. but now they're doing this "different20:26
robrubinary package names depending on what release we build for" thing and the diff is more complicated.20:26
robruuh, that's really weird, that it would say Publishing, then Migration, then Publishing. wtf20:28
fginthermterry, https://bugs.launchpad.net/ubuntu/+source/unity8/+bug/1498541 may be a non-issue. The unlock is working on a more up-to-date image20:28
ubot5Ubuntu bug 1498541 in unity8 (Ubuntu) "Automated screen unlock not working on wily images" [High,Incomplete]20:28
robru'Publishing' is the last status from the publish job, it shouldn't go back to that after Migration starts20:28
mterryfginther, oh..  good.  I love bugs that fix themselves  :)20:29
mterryrobru, kenvandine: sorry I didn't see your ping earlier, but looks like it was just as well20:29
kenvandinesolved it :)20:29
robrumterry: no worries20:30
Saviqelopio, hey, is there a canonical way to detect ubuntu release (vivid vs. wily) in a autopilot test?20:34
robrukenvandine: hey, I need to test some code that involves 'epoch versions' in an ubuntu package, can you think of one off the top of your head?20:41
robrukenvandine: nm, compiz20:41
robrukenvandine: one more? https://ci-train.ubuntu.com/job/ubuntu-landing-010-2-publish/82/ ;-)20:45
kenvandinemterry, can you take silo 10?  i'm trying to finish something up here and still working on getting 53 published too20:48
kenvandinehe ran like hell :)20:48
kenvandine+Depends: @,20:49
robruhm20:49
kenvandineanyone know if that's valid in the package tests?20:49
cjwatsonthat's an autopkgtest thing20:49
cjwatsonyes it is20:49
kenvandinecool20:49
cjwatsonmeans "all binaries produced by this source"20:50
cjwatsonhttps://anonscm.debian.org/gitweb/?p=autopkgtest/autopkgtest.git;a=blob_plain;f=doc/README.package-tests.rst;hb=HEAD20:50
robruoh20:50
kenvandinecjwatson, you had that handy :)20:50
cjwatsonwell, not directly, but http://dep.debian.net/deps/dep8/ is rather easier to remember ...20:50
kenvandinerobru, grr... silo 10 has unapproved branches20:52
robrubah20:53
robruSaviq: kgunn: please approve the two merges listed here: https://ci-train.ubuntu.com/job/ubuntu-landing-010-2-publish/83/consoleFull20:53
Saviqrobru, sry, done20:56
robrukenvandine: please publish again ^20:56
Saviqdidn't know it got under testing already20:56
robruSaviq: yep, qa granted20:57
kenvandineok21:01
kenvandinebfiller, silo 53 is published21:02
bfillerkenvandine: nice, thanks21:02
kenvandinenp21:02
kenvandinerobru, Saviq: silo 10 published21:04
robrukenvandine: thanks a bunch21:04
kenvandinenp21:04
Saviqthank you21:05
robrukenvandine: LAWL, yeah there's a huge bug in the diff generation. here's the real diff for that silo: https://ci-train.staging.ubuntu.com/job/ubuntu-landing-002-2-publish/2/artifact/unity-scopes-api_packaging_changes.diff/*view*/21:32
robruanyway, that fix will hit trunk & production shortly21:40
robrubbl, lunch21:40
dobeytrainguards: hi, can someone please click retry on https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-026/+build/7930359 ? thanks.22:01
robrudobey: sorry I'm afk. kenvandine can you ^ ?22:09
cjwatsondobey: done22:10
cjwatsonkenvandine: ^-22:10
robrucjwatson: thanks22:11
dobeythanks22:11
robruI gotta find my otg cable so i can use my yubikey on my phone, then i can work anywhere ;-)22:11
* robru is a raging workaholic22:12
robru404, request not found23:03

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!