| imgbot | === trainguards: IMAGE 36 building (started: 20141128 02:05) === | 02:04 |
|---|---|---|
| imgbot | === trainguards: IMAGE 36 DONE (finished: 20141128 03:25) === | 03:24 |
| imgbot | === changelog: http://people.canonical.com/~ogra/touch-image-stats/36.changes === | 03:24 |
| === chihchun_afk is now known as chihchun | ||
| === chihchun is now known as chihchun_afk | ||
| === chihchun_afk is now known as chihchun | ||
| michi | cihelp: cloud-worker-06 seems to have a problem. It’s been compiling my code for well over and hour now, and is making progress only very, very slowly. Any chance of help with that? | 06:55 |
| michi | http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-vivid-amd64-autolanding/21/consoleFull | 06:55 |
| pstolowski | trainguards hello, may i ask to interrupt the build #61 in silo 13 which apparently got stuck, and also remove unity8 packages from that silo's ppa (I removed the MP for it)? | 08:36 |
| Mirv | pstolowski: ok | 08:46 |
| satoris | We have a problem with thumbnailer. The current package in rtm is a weird frankenpackage that does not seem to have a backing bzr branch and none of us knows where it came from. We need to add one commit to it which would then make it equivalent to current vivid trunk. | 08:55 |
| satoris | Can we just request a source copy from vivid and if not how should we create a new branch that tries to mimic what rtm currently has? | 08:55 |
| === chihchun is now known as chihchun_afk | ||
| sil2100 | satoris: let me try looking into it and trying to identify what's the status | 08:58 |
| satoris | sil2100: cool, thanks. | 08:58 |
| === chihchun_afk is now known as chihchun | ||
| satoris | sil2100: the commit we need to get into rtm is this: http://bazaar.launchpad.net/~unity-team/thumbnailer/trunk/revision/108 | 08:59 |
| satoris | Commits 110 and 111 are already in rtm. | 08:59 |
| satoris | The other are release commits. The common branch point is 107. | 09:00 |
| popey | hmm, my phone crashed at 5am... | 09:00 |
| ogra_ | mine didnt .... since i run silo 3 it has been stable | 09:01 |
| ogra_ | webapps crash every now and then (3 crashes since yesterday evening) but the device as a whole has been massively more stable | 09:02 |
| === zsombi_ is now known as zsombi | ||
| sil2100 | satoris: damn, I think I see what happened... | 09:02 |
| popey | no crash files | 09:03 |
| ogra_ | popey, yeah, likely cgproxy hanging | 09:03 |
| popey | is there anything I can look at to indicate what caused it? | 09:03 |
| popey | ah | 09:03 |
| sil2100 | satoris: so we had a freak situation here - it seems someone made a thumbnailer silo for utopic, built it, requested a sync to rtm from it, landed the sync to RTM and DIDN'T land the original utopic silo | 09:04 |
| sil2100 | satoris: this resulted in RTM getting a new package version, but the original changes didn't get merged into trunk and not released into utopic | 09:05 |
| sil2100 | satoris: which is why 1.3+14.10.20141020-0ubuntu1 doesn't exist there | 09:05 |
| sil2100 | satoris: let me think what we can do | 09:06 |
| satoris | sil2100: ok, so what do we need to do to get things fixed now? The simplest thing would be to just take the source package of vivid (that has everything we need and nothing extra) and put that in rtm. | 09:07 |
| sil2100 | satoris: if vivid only has changes that are approved for RTM, then a source copy from vivid is the right solution indeed | 09:08 |
| sil2100 | satoris: we can arrange a silo with that | 09:08 |
| satoris | The only commit in it is the exif one and that has been approved for rtm (or ota at least). | 09:08 |
| satoris | thostr_: is that correct? | 09:09 |
| michi | cihelp: Any chance of taking cloud-worker-06 off-line? It’s still broken, aborting the build after two hours, at a point where it hasn’t finished compiling yet. | 09:15 |
| === chihchun is now known as chihchun_afk | ||
| thostr_ | satoris: what was the bug id again | 09:30 |
| satoris | thostr_: there does not seem to be one (it was fixed in trunk without a bug). | 09:31 |
| popey | ogra_: I'd like to run this cgproxy thing on my phone... | 09:47 |
| ogra_ | popey, go ahead then :) | 09:47 |
| popey | where is it? | 09:47 |
| ogra_ | https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu-rtm/landing-003 | 09:47 |
| ogra_ | you want cgmanager and the lib | 09:47 |
| popey | ta | 09:47 |
| === vrruiz_ is now known as rvr | ||
| sil2100 | psivaa: ping! In case you're around IRC - the address_book_app test suite went into an infinite loop and timed out, so we're missing some test suites that were supposed to be ran on this device | 09:54 |
| === karni is now known as karni-otp | ||
| satoris | sil2100: let's hold the thumbnailer copy for the moment. I need to sync with jhodapp first. | 09:58 |
| sil2100 | satoris: ACK | 10:04 |
| === karni-otp is now known as karni | ||
| * sil2100 needs to jump out to the post office for a minute | 11:02 | |
| sil2100 | brb | 11:03 |
| robru | dbarth_: https://ci-train.ubuntu.com/job/ubuntu-rtm-landing-009-1-build/53/console started the build for you because I'm testing something. | 11:20 |
| robru | dbarth_: oh hrm, you need to fix your debian/changelog... | 11:24 |
| robru | dbarth_: https://code.launchpad.net/~mardy/signon/rtm-fixes/+merge/243137 yeah you're going to need to change vivid to UNRELEASED here, in order for citrain to generate the right version and build that. | 11:25 |
| Mirv | robru: didn't you talk about some work/life balance / sleep at night thing ...? | 11:43 |
| robru | Mirv: yeah I dunno what happened, I'm super sick and I can't sleep. | 11:43 |
| Mirv | :( :( | 11:43 |
| robru | Mirv: the good news is, watch_ppa supports all arches now! http://bazaar.launchpad.net/~cupstream2distro-maintainers/cupstream2distro/trunk/revision/846#citrain/watch_ppa.py | 11:44 |
| robru | Mirv: try a build if you have one... | 11:44 |
| robru | Mirv: sorry, "super" is an exaggeration, just some insomnia / indigestion... I'm not dying or anything. but I appreciate your concern. | 11:44 |
| dbarth_ | robru: hi; thanks for the silo yesterday; i saw in the logs | 11:45 |
| robru | dbarth_: you're welcome | 11:45 |
| dbarth_ | robru: what about the rtm one though; i need to fix the changelog? i thought the citrain was managing that part | 11:46 |
| dbarth_ | ah, unreleased, i see | 11:46 |
| robru | dbarth_: well, citrain manages it if you do a sync silo, but this is just a standard MP so standard MP rules apply. citrain doesn't consider it as a sync at all. | 11:46 |
| Mirv | robru: ok! :) I'm trying the watch_ppa out now. | 11:47 |
| robru | Mirv: I tested it in preprod and it has 100% test coverage, so there shouldn't be any surprises there. | 11:50 |
| robru | Mirv: oooh, is that right? gles only has 2 arches? | 11:53 |
| robru | seems so | 11:53 |
| robru | good, watch_ppa is working ;-) | 11:54 |
| davmor2 | sil2100, ogra_: this seems somehow suitable right now https://www.youtube.com/watch?v=gSq8ZBdSxNU | 11:56 |
| sil2100 | ;) | 11:56 |
| ogra_ | heh | 11:57 |
| Mirv | robru: yes, seems 100% correct :) | 11:59 |
| robru | ^^^ there you have it, new watch_ppa fix correctly identified a regression on ppc64el. | 12:19 |
| robru | sil2100: Mirv: ok I'll try sleeping again. I won't make it for the meeting, but after some sleep I'll start my shift late tomorrow. | 12:22 |
| === alan_g is now known as alan_g|lunch | ||
| sil2100 | robru: ACK | 12:29 |
| Mirv | ack ack | 12:30 |
| === karni is now known as karni-lunch | ||
| === alan_g|lunch is now known as alan_g | ||
| sil2100 | geh... | 12:56 |
| jibel | ogra_, sil2100 I've a good news. I was checking the crash files on my phone for something and found this apport-recoverable crash with the pid of cgmanager | 13:00 |
| jibel | ogra_, sil2100 the crash was generated at 10:55 this morning and the phone is still working | 13:00 |
| ogra_ | yay | 13:00 |
| jibel | uptime 23:54 | 13:00 |
| jibel | hours | 13:00 |
| ogra_ | heh, you have 20min more than me | 13:01 |
| sil2100 | \o/ | 13:03 |
| jibel | too bad the diff is so huge between .30 and .32, we could have landed it | 13:08 |
| ogra_ | yeah | 13:12 |
| ogra_ | well, i still think we should land it as is | 13:12 |
| ogra_ | and just give it more testing | 13:13 |
| sil2100 | ogra_: did anyone talk with stgraber? | 13:13 |
| ogra_ | (if stgraber doesnt object indeed) | 13:13 |
| ogra_ | not yet | 13:13 |
| ogra_ | i pinged him in #ubuntu-touch ... he usually reads the backlog there (once he is around) | 13:15 |
| sil2100 | jibel, ogra_: I'll be a bit late for the OTA meeting | 13:20 |
| jibel | sil2100, how late? | 13:24 |
| sil2100 | ~10 minutes | 13:24 |
| jibel | sil2100, okay, we'll start at 15 | 13:24 |
| jibel | 15:15 | 13:24 |
| sil2100 | Thanks :) | 13:24 |
| === karni-lunch is now known as karni | ||
| sil2100 | jibel: how about I give you the list in spreadsheet form \o/ | 15:08 |
| pstolowski | sil2100, hey, any idea why landing 013 notoriously fail on 2 tests on ppc64, apparently they take too long to complete. is this machine overloaded or so? | 15:30 |
| pstolowski | Mirv, ^ | 15:34 |
| sil2100 | pstolowski: looking | 15:35 |
| sil2100 | pstolowski: so, the tests are timing out? | 15:35 |
| pstolowski | sil2100, kind of, they do some statistical tests where time is a factor, and it apparently exceeds some threshold. it only started happening a few hours ago when I tried to rebuild the silo; there were no code changes that could impact it | 15:37 |
| sil2100 | hm, not much we can do here, we use the same builders as the archive - I can rebuild that package in the PPA again and check if it'll be better now | 15:38 |
| pstolowski | sil2100, we haven't been touching these tests for months, it's very weird they fail now. but they may do so if machine is under stress i think | 15:41 |
| jibel | sil2100, a spreadsheet is fine, I love them :) | 15:42 |
| sil2100 | pstolowski: ok, build re-ran | 15:43 |
| sil2100 | (in PPA) | 15:43 |
| pstolowski | sil2100, ok, thanks | 15:44 |
| sergiusens | sil2100: can I have a silo for line 43? | 15:50 |
| sil2100 | sergiusens: sure | 15:52 |
| sergiusens | ty | 15:52 |
| sil2100 | jibel: http://paste.ubuntu.com/9284663/ <- actually, decided pastebinit might be better ;) | 15:58 |
| sil2100 | You can copy-paste it to a spreadsheet if you want! | 15:58 |
| jibel | sil2100, thx | 15:58 |
| pstolowski | sil2100, where can I monitor the progress of that build? | 16:00 |
| sil2100 | pstolowski: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-013/+build/6603131 <- eeek, it failed again ;/ | 16:00 |
| pstolowski | oh | 16:01 |
| sil2100 | But it failed on a different ppc64 machine | 16:01 |
| sil2100 | hm hm hmmm | 16:01 |
| sil2100 | Maybe something got uploaded! | 16:01 |
| pstolowski | sil2100, it's very weird. it built correctly in the morning. then I pushed a single line change to debian/changelog. and it keeps failing since than | 16:02 |
| pstolowski | * then | 16:02 |
| * sil2100 checks the archive | 16:06 | |
| pstolowski | i' upgrading my slightly outdated vivid machine.. if something broke, is should see it locally (hopefully) | 16:07 |
| sil2100 | It only fails for ppc64el, so hm, you would have to try creating a pbuilder/chroot for that | 16:09 |
| sil2100 | But not sure that it would fail on virtual hardware | 16:09 |
| pstolowski | sil2100, hmm right.. no, i'm not going to go that route | 16:10 |
| pstolowski | sil2100, did we enable that arch just recently, or it has always been there? | 16:10 |
| sil2100 | It's been around for a while | 16:11 |
| pstolowski | sil2100, have you found anything in the archive? | 16:28 |
| sil2100 | pstolowski: I checked, but it's hard to say what could have impacted this - do you know around which hour the builds were still succeeding? | 16:44 |
| sil2100 | Was it before 13? | 16:44 |
| sil2100 | pstolowski: nvm, found the info | 16:46 |
| pstolowski | sil2100, should be around 11:15 PL time | 16:46 |
| sil2100 | pstolowski: the build succeeded on the same machine it failed a few times... | 16:47 |
| sil2100 | pstolowski: ok, so I think it's some transient issue | 16:48 |
| sil2100 | pstolowski: I see that unity-scopes-api failed to build because of the same tests yesterday as well | 16:49 |
| sil2100 | pstolowski: but then it was building fine again, and started failing again now | 16:49 |
| sil2100 | So it seems there has to be something racy in there | 16:49 |
| sil2100 | https://launchpadlibrarian.net/191337421/buildlog_ubuntu-vivid-ppc64el.unity-scopes-api_0.6.9%2B15.04.20141127.2-0ubuntu1_FAILEDTOBUILD.txt.gz <- these are logs from a yesterday's build | 16:50 |
| pstolowski | sil2100, hmmm, ok, i didn't notice we had failures yesterday since it was failing on some other project in this silo | 16:51 |
| sil2100 | pstolowski: I guess we need to either find out why the tests are so sensitive there, or simply rebuild until it passes | 16:52 |
| pstolowski | sil2100, i'll try both, just emailing my team, in the meantime will put a stapler on build button | 16:54 |
| pstolowski | sil2100, thanks for help | 16:55 |
| pstolowski | sil2100, can we rule out an overloaded build machine? | 16:56 |
| sil2100 | pstolowski: I don't think that would be the cause... every builder performs only one build at a time, and besides the ppc64el machines seem to be relatively not-overloaded | 17:00 |
| pstolowski | sil2100, ack | 17:01 |
| sergiusens | sil2100: https://ci-train.ubuntu.com/job/ubuntu-landing-003-1-build/76/console is taking forever in dependency wait | 17:05 |
| * sil2100 quickly googles 'new star wars' | 17:13 | |
| sergiusens | sil2100: ideas? | 17:44 |
| sergiusens | sil2100: seems it gets stuck on a dependency wait for things that never build... | 17:45 |
| sergiusens | trainguards? | 17:50 |
| === alan_g is now known as alan_g|EOW | ||
| sil2100 | sergiusens: hey! Sorry, had a meeting and robru is sick so he might start late | 18:31 |
| sil2100 | Looking | 18:31 |
| Ursinha | sergiusens: sorry, I stole sil2100 for a while :) | 18:32 |
| sil2100 | sergiusens: uuuh, seems like some aftermath of Robert's modifications! I think his new scripts actually wait for all archs in the PPA, and don't check if those were ever published in the target archive | 18:34 |
| sil2100 | Let me try briefly looking into that | 18:36 |
| sil2100 | sergiusens: so it seems robru prepared a fix for that but it doesn't seem to be working | 18:40 |
| sil2100 | sergiusens: ah, indeed... his fix is actually looking at what arches it builds in the silo PPA, not the target archive | 18:41 |
| sil2100 | robru: anyway, bug LP: #1397407 when you have a moment | 18:49 |
| ubot5 | Launchpad bug 1397407 in CI Train [cu2d] "The train should not wait infinitely for arches that the package wasn't released yet" [High,New] https://launchpad.net/bugs/1397407 | 18:49 |
| sergiusens | sil2100: Ursinha ok, I'll leave that there for a while, but these are urgent fixes so I've done a dput dance | 18:55 |
| sergiusens | thanks | 18:55 |
| sil2100 | sergiusens: yeah, in case something like this happens in the future, just abort the build job and give us a sign - we can still publish it | 19:00 |
| sil2100 | With an ignore flag | 19:00 |
| * sil2100 needs to go | 19:00 | |
| sil2100 | o/ | 19:00 |
| robru | sergiusens: sorry for the delay, I'm not feeling well. I'm testing an experimental fix: https://ci-train.ubuntu.com/job/ubuntu-landing-003-1-build/79/console | 20:09 |
| robru | looks good locally, so far. we'll see how it handles production | 20:10 |
| robru | sergiusens: ok you're good to go | 20:14 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!