| rsalveti | will trigger a new image so I can have a working emulator | 00:14 |
|---|---|---|
| plars | something happened with the download taking way too long for the latest image it seems... I restarted them, so results should start to show up in a bit | 03:25 |
| Mirv | good morning | 05:12 |
| Mirv | robru: I'll test unity8 AP too.. | 05:18 |
| Mirv | robru: I've gotten sometimes problematic results where another run(s) have been fluent | 05:18 |
| Mirv | cihelp daily-release-executor is reported offline | 05:31 |
| Mirv | so all builds halted as well | 05:31 |
| rsalveti | plars: there was a regression in the recovery image, which is fixed in the archive but needs a newer image | 05:33 |
| rsalveti | not sure if the regression would cause the device to fail in someway | 05:33 |
| didrocks | Mirv: hey! around already? :) | 07:12 |
| didrocks | cihelp: hey, all executors on q-jenkins are down, (so everything is blocked). Seems the reason why robru/ken couldnt publish Mir yesterday | 07:14 |
| vila | didrocks, Mirv: on it | 07:14 |
| didrocks | thanks | 07:15 |
| didrocks | vila: I guess the daily-release executor is the first one to investigate why it went down | 07:15 |
| vila | didrocks: yes, Mirv said that already, we all agree | 07:16 |
| Mirv | didrocks: yeah, as written on langing plan | 07:21 |
| Mirv | didrocks: robert also had some unity8 AP problem, but I couldn't reproduce it so it's probably his environment (like wrongly updated unity8 from daily-build, or click tests being run and messing up unity8 testing) | 07:22 |
| didrocks | Mirv: ah, the notes were not from yesterday | 07:22 |
| didrocks | Mirv: yeah, probably | 07:22 |
| didrocks | Mirv: I'm not sure to understand the unity-mir rebuild? | 07:22 |
| didrocks | if it wasn't rebuilt already, you can't test, right? | 07:23 |
| Mirv | didrocks: the mir 0.1.1 build-dep was not in, otherwise it's ok. same for u-s-c. | 07:23 |
| didrocks | Mirv: ah, it's just the bump build-dep MP | 07:23 |
| didrocks | Mirv: well, if we release without that one, I won't die TBH ;) | 07:23 |
| didrocks | as long as the things are rebuilt as expected | 07:23 |
| Mirv | didrocks: ok ;) yeah the libmirserver10 dependency is there anyway, as it was built after the mir build | 07:24 |
| didrocks | Mirv: ok, just don't stress on that, if you feel that we should skip the build, please do (if we are ready to publish) | 07:24 |
| didrocks | ogra_: asac: did one of you requested a new image? | 07:25 |
| didrocks | ogra_: asac: I see a new image and 2 android uploads in particular to the archive (the second to fix a regression it seems) | 07:25 |
| Mirv | didrocks: ok. we're ready to publish, aside from that I don't see a note if unity-system-compositor was smoke tested. | 07:25 |
| Mirv | so that needs to be done still | 07:25 |
| didrocks | ogra_: asac: but the image which was built was between the 2 android uploads, so the "recovery regression" is in the current proposed image? | 07:25 |
| didrocks | Mirv: ok, maybe you should kill in jenkins if there are some mir/platform/unity8 stacks planned to be rebuilt? | 07:26 |
| didrocks | so that one q-jenkins is repaired, you are not trapped by a rebuild :) | 07:26 |
| Mirv | didrocks: yes, I did that already | 07:26 |
| Mirv | (except for platform it seems, now did that too) | 07:27 |
| didrocks | thanks Mirv :) | 07:27 |
| didrocks | waow, it's snowing here | 07:30 |
| Mirv | didrocks: cool, it's just rain currently here :) | 07:31 |
| Mirv | I'd welcome snow already, it's been unusually warm (which means also dark) | 07:32 |
| didrocks | Mirv: well, I think it'll transform soon in rain as well :) | 07:32 |
| didrocks | ah ah, for some kind of "warm" ;) | 07:32 |
| didrocks | here, for us, it's freezing (like we have the temperature we do have generally in 3 weeks) | 07:33 |
| vila | daily-release-executor is back | 07:33 |
| didrocks | thanks vila | 07:41 |
| Mirv | ok got to the desktop with updated mir + u-s-c too | 08:04 |
| didrocks | Mirv: and still get graphics? ;) | 08:04 |
| Mirv | didrocks: even that :) not much multimonitor though | 08:04 |
| didrocks | Mirv: not a surprise, but not a bad surprise either :) | 08:04 |
| didrocks | sweetness! | 08:05 |
| Mirv | didrocks: the mir stack had already queued the prepare tasks before daily release executor broke earlier, so they got rebuilt. mir is a no-change rebuild and u-s-c got now the mir 0.1.1 b-d in. I'm rerunning unity8 AP still and after then asking for packaging acks to publish. | 08:31 |
| didrocks | Mirv: sounds good, just do minor dogfooding | 08:31 |
| didrocks | don't stress it too much in some words :) | 08:32 |
| didrocks | thanks! | 08:32 |
| Mirv | didrocks: ok then, http://q-jenkins.ubuntu-ci:8080/view/cu2d/view/Head/view/Mir/job/cu2d-mir-head-3.0publish/lastSuccessfulBuild/artifact/packaging_changes_mir_0.1.1+14.04.20131120-0ubuntu1.diff + http://q-jenkins.ubuntu-ci:8080/view/cu2d/view/Head/view/Mir/job/cu2d-mir-head-3.0publish/lastSuccessfulBuild/artifact/packaging_changes_unity-system-compositor_0.0.2+14.04.20131120-0ubuntu1.diff + http://q-jenkins.ubuntu-ci:8080/view/cu2d/view/Head/view | 08:50 |
| Mirv | so the actual u-s-c session moved to ubuntu-desktop-mir which I already noticed | 08:50 |
| didrocks | Mirv: sorry, my system is in autodestruction mode after removing the emulator, I can't use sudo, ls or any other tool like chromium, can you get someone to review it? | 08:50 |
| didrocks | I can't even open a webpage | 08:50 |
| Mirv | didrocks: ok, have a nice debugging session.. | 08:56 |
| didrocks | well, nice with no "ls", I hope that my home dir is still intact… | 08:57 |
| asac | didrocks: from backlog: | 09:01 |
| asac | 09:59 < -bip> 20-11-2013 00:08:09 -!- alesage!~alesage@c-98-212-4-39.hsd1.il.comcast.net has quit [Quit: Leaving] | 09:01 |
| asac | 09:59 < rsalveti> 00:15:21> will trigger a new image so I can have a working emulator | 09:01 |
| asac | :/ | 09:01 |
| didrocks | asac: well, and no second image with "recovery fixed" | 09:02 |
| didrocks | right? | 09:02 |
| didrocks | or did you find anything in the backlog? | 09:02 |
| asac | didrocks: let me look more careful :) | 09:02 |
| didrocks | asac: let's discuss in a few, my install is broken by the emulator I guess | 09:02 |
| asac | 09:59 < plars> 03:25:24> something happened with the download taking way too long for the latest image it seems... I restarted them, so results should start to show up in a bit | 09:03 |
| asac | 09:59 < rsalveti> 05:33:30> plars: there was a regression in the recovery image, which is fixed in the archive but needs a newer image | 09:03 |
| asac | 09:59 < rsalveti> 05:33:42> not sure if the regression would cause the device to fail in someway | 09:03 |
| asac | didrocks: next line spoken here was you pinging Mirv | 09:03 |
| asac | so guess not | 09:03 |
| asac | though the need for an image was identified | 09:03 |
| asac | didrocks: so feels we want another image? whatelse happened in the meantime? | 09:04 |
| didrocks | asac: yeah, I guess we want one (but again, I can't even ls right now, let me fix my box) | 09:05 |
| didrocks | and post a warning if it's really due to the emulator replacing libc6 with the armhf version on your system | 09:05 |
| asac | didrocks: ok. guess just boot in a live session and fix it from there | 09:05 |
| asac | :/ | 09:06 |
| didrocks | asac: I'm trying with busybox here | 09:06 |
| asac | or from initrd if you are brave | 09:06 |
| asac | yeah | 09:06 |
| didrocks | but dpkg from busybox doesn't show which arch is installed :p | 09:06 |
| asac | didrocks: guess check in /var/lib/dpkg on your own | 09:07 |
| didrocks | 2013-11-20 09:48:34 status half-configured libc-bin:amd64 2.17-93ubuntu4 | 09:07 |
| didrocks | from removing the emulator | 09:07 |
| * asac wonders what the emulator does that can cause this | 09:09 | |
| seb128 | didrocks, weird, usually app ask you to type the "yes, I'm sure I want to do that" before removing libc-bin | 09:09 |
| didrocks | seb128: I think it didn't remove, it has overriden with the armhf version | 09:10 |
| didrocks | seb128: I clearly didn't get "remove libc6" | 09:10 |
| asac | half-configured probably means that things fell over during upgrade | 09:10 |
| didrocks | ok, busybox didn't work from my needs, let me try to find an usb stick and fix from there | 09:10 |
| didrocks | yeah | 09:10 |
| didrocks | I guess there are some divert magic for the emulator | 09:10 |
| asac | didrocks: where did you get the emulator from? | 09:11 |
| didrocks | and something when went on removing | 09:11 |
| asac | ack | 09:11 |
| didrocks | asac: followed the wiki page | 09:11 |
| didrocks | (but then, tried to remove the android-emulator package) | 09:11 |
| seb128 | I hope other people don't try the emulator | 09:12 |
| didrocks | well, it's not 100% confirmed, but it was the only thing I was doing at this time | 09:12 |
| seb128 | didrocks, do you want me to follow up on the list to "be careful, Didier bricked his system with that"? | 09:12 |
| asac | didrocks: you hvae a URL to the wiki? | 09:12 |
| didrocks | seb128: would be nice to have someone confirming (in a vm first) | 09:12 |
| didrocks | asac: well, I just have a weechat window here :p it was on the ML | 09:12 |
| asac | kk | 09:13 |
| didrocks | ok, see you (hopefully) in a few | 09:13 |
| seb128 | asac, https://wiki.ubuntu.com/Touch/Emulator | 09:13 |
| didrocks | you should try on amd64 to reproduce, as jibel noted, it's probably a i386<->amd64 issue | 09:14 |
| didrocks | so seb128 is safe, you can try in production! :p | 09:14 |
| seb128 | lol | 09:14 |
| asac | it refers to an android-emulator package | 09:15 |
| asac | but thats noot in the archive... which ppa is it? | 09:15 |
| jibel | asac, binaries are only available for i386 | 09:15 |
| jibel | in the archive | 09:15 |
| seb128 | asac, it's in multiverse for i386 | 09:16 |
| seb128 | https://launchpad.net/ubuntu/+source/android/20131120-0225-0ubuntu1 | 09:16 |
| asac | oh its part of android | 09:16 |
| asac | yeah i can imagine that that might cause accidential havoc | 09:16 |
| seb128 | on i386 it wants to install libc6-amd64 | 09:20 |
| seb128 | not sure I want to risk trying on my system :p | 09:20 |
| asac | so the package doesnt look harmful by itself | 09:22 |
| jibel | trying in an amd64 VM, it's installing 723MB of i386 packages | 09:23 |
| asac | seb128: it includes stuff like: ./usr/share/android/emulator/out/host/linux-x86/bin/emulator64-arm | 09:23 |
| asac | so it somewhat makes sense that it brings in amd64 stuff | 09:23 |
| didrocks | Commandline: apt-get install android-emulator | 09:23 |
| didrocks | Install: android-emulator:i386 (20131120-0225-0ubuntu1), lib64gcc1:i386 (4.8.2-1ubuntu2, automatic), lib64stdc++6:i386 (4.8.2-1ubuntu2, automatic), libc6-amd64:i386 (2.17-93ubuntu4, automatic) | 09:24 |
| didrocks | (that was what I had) | 09:24 |
| didrocks | and it worked after that, it's really when I removed the package and autoremove the deps | 09:24 |
| seb128 | asac, it's a bit weird that it's an i386 package and that it pulls in amd64 binaries | 09:24 |
| jibel | didrocks, I reproduced your problem | 09:24 |
| asac | http://paste.ubuntu.com/6447226/ | 09:24 |
| jibel | and broke the VM after removing the emulator | 09:24 |
| jibel | asac, ^ | 09:24 |
| seb128 | jibel, can you paste the log of what apt did when you removed the emulator? | 09:25 |
| asac | right. that would be good to know | 09:25 |
| jibel | seb128, sure | 09:25 |
| didrocks | jibel: ah, so not rebooting immediatly, would be interesting to get your VM fix rather than me being offline :) | 09:25 |
| jibel | didrocks, indeed, 1 minute | 09:25 |
| didrocks | (ok, just finished getting an usb stick ready with an old trusty image) | 09:26 |
| asac | so the emulator package has nothing in the postrm postinst scripts | 09:26 |
| asac | so the bustage must come genuinely from apt and libc6-amd64 removal | 09:26 |
| didrocks | right | 09:26 |
| asac | my guess is that if i just instlal libc6-amf64 and remove it will also be broken | 09:27 |
| jibel | seb128, http://paste.ubuntu.com/6447229/ | 09:27 |
| asac | -> e.g. if thats the case thats a doko bug, no? | 09:27 |
| didrocks | at least, I'm looking at busybox dpkg and it doesn't have a lot of options | 09:27 |
| didrocks | asac: you're probably right, indeed | 09:27 |
| jibel | seb128, asac to reproduce on an amd64 VM, dpkg --add-architecture i386 && apt-get update && apt-get install -y android-emulator && apt-get autoremove --purge android-emulator | 09:28 |
| asac | libstdc++6:i386* | 09:28 |
| asac | odd. | 09:28 |
| didrocks | yeah, so it probably removed the libc6-amd64 files configured to be used against | 09:31 |
| didrocks | and then, can't run the postrm script to configure the new links? | 09:31 |
| asac | jibel: can you reproduce by just installing the libc-amd64 and purging that? | 09:32 |
| asac | (in case you have a VM image easily available) | 09:32 |
| jibel | asac, hold on, I'm trying to recover the system first to help didrocks save his machine | 09:32 |
| didrocks | I would appreciate, indeed :) | 09:32 |
| asac | sure | 09:32 |
| seb128 | that package has a postrm | 09:33 |
| seb128 | if [ "$1" = remove ]; then | 09:33 |
| seb128 | ARCH=${DPKG_MAINTSCRIPT_ARCH:-$(dpkg --print-architecture)} | 09:33 |
| seb128 | if [ "${ARCH}" = "amd64" ] && [ "libc6-amd64" = "libc6-i386" ]; then | 09:33 |
| seb128 | if [ -h /lib/ld-linux.so.2 ] && [ ! -f /lib/ld-linux.so.2 ]; then | 09:33 |
| seb128 | rm /lib/ld-linux.so.2 | 09:33 |
| asac | however, maybe first narrowing it down will help :) | 09:33 |
| asac | greedilyu removing ld-linux | 09:33 |
| didrocks | hum, I still have /lib/ld-linux.so.2 here | 09:34 |
| asac | seb128: that only happens on amd64, no? | 09:34 |
| asac | i think that might not be it | 09:34 |
| didrocks | but the postrm failed itself | 09:34 |
| didrocks | so it's not executed | 09:34 |
| didrocks | (as per log) | 09:34 |
| asac | all those postrm fail | 09:34 |
| didrocks | right | 09:34 |
| asac | seems stuff is already busted after prerm | 09:34 |
| didrocks | yep | 09:34 |
| asac | and after just removing the files | 09:34 |
| asac | int he packages | 09:34 |
| asac | maybe having the log from installing might show some diversions | 09:35 |
| asac | etc. | 09:35 |
| didrocks | I guess when removing libc6-amd64 files | 09:35 |
| ogra_ | are you 100% sure the last android build made it into the image ? | 09:36 |
| jibel | asac, trying now | 09:36 |
| * ogra_ sees that publishing and image build are very close | 09:36 | |
| didrocks | ogra_: was it supposed to be reflected in your diff? | 09:37 |
| ogra_ | no | 09:37 |
| ogra_ | the android package isnt inside anywhere | 09:37 |
| ogra_ | so the manifest generator cant pick it up | 09:37 |
| ogra_ | Get:1 http://ftpmaster.internal/ubuntu/ trusty/multiverse android all 20131120-0225-0ubuntu1 [348 MB] | 09:37 |
| ogra_ | Fetched 348 MB in 48s (7243 kB/s) | 09:37 |
| ogra_ | ok, seems it made it | 09:38 |
| ogra_ | (you can find it in the livefs build log) | 09:38 |
| ogra_ | http://people.canonical.com/~ubuntu-archive/livefs-build-logs/trusty/ubuntu-touch/20131120.1/livecd-20131120.1-armhf.out | 09:38 |
| didrocks | ogra_: I guess you need a new image then | 09:38 |
| ogra_ | ? | 09:39 |
| didrocks | ogra_: see the followup android upload at 5 UTC | 09:39 |
| ogra_ | 20131120-0225-0ubuntu1 is the last upload | 09:39 |
| didrocks | ah, you got with that one? | 09:39 |
| ogra_ | see above | 09:39 |
| didrocks | ogra_: can't click on links right now :p | 09:39 |
| didrocks | if you follow what was discussed, system hosed ^ | 09:40 |
| jibel | asac, confirmed: apt-get install libc6-amd64 && apt-get remove libc6-amd64 | 09:40 |
| jibel | and get a broken system for free | 09:40 |
| didrocks | I would have prefered to not have it at all than for free ;) | 09:40 |
| didrocks | thanks for digging jibel :) | 09:41 |
| ogra_ | didrocks, yeah | 09:41 |
| asac | jibel: thats on i386? | 09:42 |
| jibel | asac, amd64 with i386 enabled | 09:43 |
| jibel | which is the same cofigiration than didrocks. I can try on i386 too | 09:43 |
| asac | kk | 09:43 |
| asac | dont worry. guess file a bug so we can have doko etc. look at it | 09:43 |
| jibel | k | 09:44 |
| Mirv | didrocks: I filed bug #1253008 about pitti's worry about u-s-c configuration file removal (or actually it's a move to another package) | 09:59 |
| ubot5 | bug 1253008 in Unity System Compositor "Clean configuration file on upgrade" [Undecided,New] https://launchpad.net/bugs/1253008 | 09:59 |
| didrocks | Mirv: thanks | 10:00 |
| ogra_ | r26 looks ok on magureo | 10:35 |
| ogra_ | (havent done any call or sms tests, but it boots fine) | 10:35 |
| popey | blimey, 101MB to go to 26 | 10:42 |
| popey | did you add the kitchen sink to it? | 10:42 |
| === vrruiz_ is now known as rvr | ||
| didrocks | ogra_: mind updating the spreadsheet with latest infos? I'm a little bit lost as the android update is still to TODO | 11:08 |
| didrocks | so not sure if the upload was that one (it seems so though) | 11:08 |
| * ogra_ can look, but i wasnt involved in any of the last three android uploads | 11:12 | |
| ogra_ | (i didnt even know about them until i saw them on the changes ML) | 11:12 |
| didrocks | ogra_: I guess the planned image # needs to change as well | 11:16 |
| didrocks | ogra_: hum, ok, I think, I'll bump everyting to 27 | 11:26 |
| didrocks | well, system settings is in 25 from what I see | 11:27 |
| ogra_ | didrocks, oh, you wanted me to bump the others ? | 11:27 |
| ogra_ | sorry, i understood i should sort the android line | 11:27 |
| didrocks | ogra_: yeah, as they are not exact, but no worry :) | 11:27 |
| ogra_ | sorry | 11:28 |
| didrocks | jibel: lool: can one of you confirm https://bugs.launchpad.net/ubuntu/+source/ubuntu-download-manager/+bug/1240656? that will get it moved and then we can kick a saucy image build | 11:29 |
| ubot5 | Ubuntu bug 1240656 in ubuntu-download-manager (Ubuntu) "disable debug logging by default" [Critical,In progress] | 11:29 |
| psivaa | ogra_: didrocks: not sure if that's known, but with image 26, we see "failed to copy '/var/lib/jenkins/phablet-flash/imageupdates/pool/ubuntu-418162f2a9ba9a320a2b460938e3d333f84b6f35826200c2a6b6c342ba1e21ef.tar.xz' to '/cache/recovery//ubuntu-418162f2a9ba9a320a2b460938e3d333f84b6f35826200c2a6b6c342ba1e21ef.tar.xz': No space left on device" on mako | 11:40 |
| psivaa | have seen that in two devices | 11:40 |
| didrocks | interesting, as the broken recovery was installed in image 25? that can be it, right? | 11:41 |
| ogra_ | psivaa, the recovery from r25 was broken ... could be that you still havethat in place | 11:41 |
| ogra_ | i just did an OTA upgrade on maguro, that one worked fine | 11:41 |
| psivaa | i tried on two devices, not sure if both of them had attempted r25 as well, i'll try another device | 11:42 |
| ogra_ | didrocks, you should probably not use IRC nicks (or make clear they are IRC nicks) in mails to ubuntu-phone :) | 11:42 |
| didrocks | ogra_: good point | 11:42 |
| ogra_ | "infinity is looking into it" sounds a bit cryptic for non IRC users | 11:43 |
| ogra_ | :) | 11:43 |
| ogra_ | (like "eternity will solve it") | 11:43 |
| Mirv | go, builders, go | 11:44 |
| Mirv | waiting for amd64+i386 on https://launchpad.net/~ubuntu-unity/+archive/daily-build/+sourcepub/3671940/+listing-archive-extra ... | 11:45 |
| didrocks | ogra_: well, it can be true :) | 11:45 |
| ogra_ | haha | 11:45 |
| * didrocks doesn't want to hear about amd64 and i386 in the same sentence today | 11:46 | |
| Mirv | oh, how rude of me | 11:46 |
| ogra_ | yippiiee | 11:49 |
| ogra_ | root@ubuntu-phablet:/# lxc-console -nandroid -t0 | 11:49 |
| ogra_ | Connected to tty 0 | 11:49 |
| ogra_ | Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself | 11:49 |
| ogra_ | root@android:/ # | 11:49 |
| ogra_ | it works ! | 11:49 |
| Saviq | cihelp, hey, unity8 otto is getting stuck due to some dependency issue http://s-jenkins.ubuntu-ci:8080/job/autopilot-testrunner-otto-trusty/817/console | 11:51 |
| Saviq | I *feel* like it might be caused by https://code.launchpad.net/~kgunn72/unity-mir/bump-mir-dep-0.1.1/+merge/193521 | 11:54 |
| Saviq | if unity-mir built against mir > 0.1.1 somehow | 11:55 |
| Saviq | and is kept in the mbs repo | 11:55 |
| Saviq | but then there's no mir > 0.1.1 in the archive yet, so I'm not sure how that could happen | 11:55 |
| Saviq | Mirv, ideas ↑? | 11:55 |
| Saviq | how did that even get through autolanding? | 11:55 |
| didrocks | Saviq: the transition is in the daily-build ppa normally | 12:00 |
| Saviq | didrocks, right... but otto isn't using it | 12:00 |
| Saviq | grr | 12:01 |
| didrocks | Saviq: otto for integration tests is | 12:01 |
| Mirv | Saviq: aanyhow, once the amd64 PPA build of this https://launchpad.net/~ubuntu-unity/+archive/daily-build/+sourcepub/3671940/+listing-archive-extra finally starts, we're really close of having Mir 0.1.1 in archives | 12:01 |
| didrocks | maybe not for upstream merger? not sure | 12:01 |
| Mirv | my plan is 1. wait for amd64 build of that, 2. upgrade desktop, 3. reboot and login, 4. publish mir + platform-api + unity-mir + unity-system-compositor | 12:03 |
| Saviq | Mirv, ok | 12:03 |
| Saviq | we'll just have to wait then | 12:04 |
| Saviq | didrocks, you mean in cu2d it does? | 12:04 |
| didrocks | Saviq: yeah, when releasing, we are adding the ppa in the otto integration tests | 12:04 |
| Saviq | didrocks, in -ci -autolanding it doesn't, as I understood it using daily is unsafe 'cause you never know if what you've built against is really going to end up in distro | 12:04 |
| didrocks | Saviq: yeah, and we have issues when there are some transitions | 12:06 |
| didrocks | like today | 12:06 |
| didrocks | that's why we need CI Airline :p | 12:06 |
| Saviq | didrocks, ok, so nobody is doing anything wrong, just that our tools are not letting do us what we really need, gotcha | 12:06 |
| Saviq | we'll just wait, then | 12:06 |
| didrocks | Saviq: I guess yeah | 12:06 |
| === alan_g is now known as alan_g|afk | ||
| psivaa | didrocks: ogra_: so the issue of no space in /cache/recovery/ on mako is not due to image r25 remnants, i flashed a device which had r23 and it still occurs | 12:18 |
| ogra_ | weird | 12:19 |
| didrocks | psivaa: ok, if you changed the recovery as well and you're sure you don't have image 25 recovery, one less ;) | 12:19 |
| psivaa | on maguro this is not happening though | 12:22 |
| === alan_g|afk is now known as alan_g | ||
| === MacSlow is now known as MacSlow|lunch | ||
| psivaa | didrocks: the only files in /cache/recovery/ are very small log files so they can not be the reasons. | 12:31 |
| psivaa | i mean before flashing with r26 | 12:32 |
| ogra_ | check one level up in /cache | 12:41 |
| ogra_ | i think /cache is the partition, recovery/ is only a dir | 12:42 |
| psivaa | ogra_: 32k on /cache | 12:46 |
| ogra_ | used you mean ? | 12:47 |
| psivaa | yes | 12:47 |
| psivaa | before flashing with r26 | 12:47 |
| ogra_ | thats not much ... and should really not cause out of space warnings | 12:48 |
| psivaa | right during the flashing on *maguro i see 1069 M in /cach | 12:49 |
| psivaa | *cache | 12:49 |
| psivaa | ogra_: not sure what's the capacity though on each devices | 12:51 |
| ogra_ | http://phablet.ubuntu.com/gitweb?p=CyanogenMod/android_build.git;a=log;h=refs/heads/phablet-trusty | 12:51 |
| ogra_ | i see nothing that even touches recovery ... strange | 12:51 |
| psivaa | but as you said it could be anything that goes into /cache | 12:52 |
| psivaa | now it's 1386M | 12:52 |
| ogra_ | aha | 12:53 |
| ogra_ | http://phablet.ubuntu.com/gitweb?p=CyanogenMod/android_system_core.git;a=blobdiff;f=init/init.c;h=e5f265858d0e0890deff18424ab09b174a4ce3f0;hp=462130711e331820467e2b8c427c1e2f7be9cdf2;hb=a1527ce86c90842b693941723888f7739f3bf473;hpb=7a318cec11f017c5ae9c186dae2b01f2dc4d3ef4 | 12:55 |
| ogra_ | psivaa, does /sbin/recovery exist ? | 12:55 |
| Mirv | sil2100: I hoped you wouldn't block the stacks, that's why I asked for publishing jobs only | 12:56 |
| psivaa | ogra_: the installation on maguro is finished and now /sbin/recovery is not there.. but i dont know if it would have existed during the flash | 12:58 |
| ogra_ | psivaa, well, it should be existent in the recovery mode only | 12:59 |
| ogra_ | oh, meeting ... | 13:00 |
| * ogra_ totally forgot ... gimme a sec to relocate | 13:00 | |
| didrocks | sil2100: coming? | 13:02 |
| jdstrand | hey-- I'm looking at landing no 303 and wondering if now is the time to upload | 13:02 |
| jdstrand | (it is marked TODO) | 13:03 |
| sil2100 | didrocks: I'll be a bit late, start off without meee | 13:03 |
| sil2100 | Mirv: you told me that too late I guess, I just redeployed unity8 though | 13:05 |
| Mirv | sil2100: you didn't wait for answer :) you also started a build, so it's now waiting for the scope build | 13:05 |
| sil2100 | Mirv: ok, so I'm not building/doing anything until I get a greenish light from you now ;p | 13:08 |
| sil2100 | Mirv: since I would need to rebuild ubuntu-settings-components as well... | 13:08 |
| sil2100 | But I'll do that after you're done ;) | 13:09 |
| Mirv | sil2100: ok :) let's wait for the current build to finish, then I'll publish, then I'll let you a sign that you can publish / do what you want | 13:09 |
| sil2100 | ;) | 13:10 |
| Mirv | sil2100: when the build job is published, isn't it in theory ok if I cancel the check job since you need to rebuil more anyway? | 13:12 |
| Mirv | s/published/finished/ | 13:12 |
| sil2100 | Mirv: you can cancel everything besides build ;p | 13:13 |
| sil2100 | Mirv: since it's not being tested anyway anywhere | 13:13 |
| Mirv | sil2100: yeah. what I think I'll do that I'll wait for build to finish, then cancel check, then do my publishing and then I can let it run check again via foo | 13:13 |
| sil2100 | Mirv: ok, but I think that for my needs a check re-run won't be needed anyway | 13:15 |
| sil2100 | Since for me it's enough that it builds, so I can just do a direct force publish after your publishing is done | 13:15 |
| Mirv | right | 13:17 |
| ogra_ | rsalveti, could the above git commit have any impact on that "/cache running out of space" issue when flashing ? | 13:17 |
| Mirv | sil2100: and actually the check didn't run since unity8 build happens to fail | 13:18 |
| sil2100 | Oh | 13:18 |
| Mirv | we should look into that at some point too, and file a bug as usual, there just hasn't been time for that | 13:18 |
| Mirv | and we probably need to remind the other team mates too that the usual cu2d vanguarding starts again now that it's up | 13:20 |
| didrocks | Mirv: quite, publish mir then! | 13:21 |
| didrocks | quick* | 13:21 |
| didrocks | as the stack stopped :p | 13:21 |
| Mirv | didrocks: already ... done! | 13:22 |
| didrocks | sooo fast, it's not an airline, it's a jet \o/ | 13:22 |
| Mirv | sil2100: so I'll wait still to see with my own eyes that the packages are in LP, and ping you after that | 13:22 |
| ogra_ | rsalveti, oooh, i think i see whats wrong with your patch ... you actually want the mkdir for /proc and /sys outside of the "if" in http://phablet.ubuntu.com/gitweb?p=CyanogenMod/android_system_core.git;a=blobdiff;f=init/init.c;h=e5f265858d0e0890deff18424ab09b174a4ce3f0;hp=462130711e331820467e2b8c427c1e2f7be9cdf2;hb=a1527ce86c90842b693941723888f7739f3bf473;hpb=7a318cec11f017c5ae9c186dae2b01f2dc4d3ef4 | 13:24 |
| ogra_ | psivaa, could you boot one of the failing devices into recovery and check that ? | 13:28 |
| psivaa | ogra_: the devices are in the lab and the failing devices are not reachable by adb. | 13:29 |
| ogra_ | hmm | 13:29 |
| psivaa | i dont know any other way to connect them. | 13:29 |
| ogra_ | so we need someone in the lab to check them ... ok | 13:29 |
| psivaa | ogra_: yea waiting till rfowler comes online | 13:30 |
| === MacSlow|lunch is now known as MacSlow | ||
| Mirv | sil2100: so "ping", feel free | 13:36 |
| rsalveti | ogra_: that patch is fine | 13:45 |
| rsalveti | ogra_: the problem was the previous one that got us into a broken recovery | 13:45 |
| rsalveti | ogra_: http://phablet.ubuntu.com/gitweb?p=CyanogenMod/android_system_core.git;a=commitdiff;h=aefb9653a367a6fd528f2ad48a1cce8945aecb3c;hp=87b40ee4a93bd682506fa082f2c2d1a0f5a60f04 | 13:45 |
| ogra_ | rsalveti, well, that still doesnt create /proc or /sys | 13:46 |
| ogra_ | it tries to mount them to a nonexisting mountpoint | 13:46 |
| rsalveti | didrocks: I did trigger another one with the fix included in it | 13:46 |
| didrocks | rsalveti: hum, it seems to have never gotten anywhere (we only had one new image) | 13:46 |
| ogra_ | rsalveti, though i still dont have any explanation for the out of space issue | 13:46 |
| didrocks | or ogra's script is broken :p | 13:46 |
| ogra_ | didrocks, ? | 13:46 |
| rsalveti | ogra_: that is ok in our default image, because they are created by lxc | 13:47 |
| ogra_ | there are 20 and 20.1 | 13:47 |
| ogra_ | 20 is the broken one | 13:47 |
| rsalveti | yes | 13:47 |
| rsalveti | 20.1 is fine | 13:47 |
| didrocks | ogra_: but you did build 20.1, no? | 13:47 |
| ogra_ | rsalveti, in recovery ? | 13:47 |
| rsalveti | ogra_: in recovery it'll be created | 13:47 |
| ogra_ | oh, i see what you are saying | 13:47 |
| ogra_ | yeah | 13:47 |
| rsalveti | ogra_: + if (stat("/sbin/recovery", &s) == 0) { | 13:47 |
| ogra_ | well, still, all makos run our of space when flaching | 13:47 |
| rsalveti | that's the fix I did yesterday when I noticed the regression | 13:48 |
| rsalveti | how? | 13:48 |
| didrocks | 12:40:27 psivaa | ogra_: didrocks: not sure if that's known, but with image 26, we see "failed to copy | 13:48 |
| didrocks | | '/var/lib/jenkins/phablet-flash/imageupdates/pool/ubuntu-418162f2a9ba9a320a2b460938e3d333f84b6f35826200c2a6b6c342ba1e21ef.tar.xz' to | 13:48 |
| didrocks | | '/cache/recovery//ubuntu-418162f2a9ba9a320a2b460938e3d333f84b6f35826200c2a6b6c342ba1e21ef.tar.xz': No space left on device" on mako | 13:48 |
| didrocks | rsalveti: ^ | 13:49 |
| ogra_ | rsalveti, <psivaa> ogra_: didrocks: not sure if that's known, but with image 26, we see "failed to copy '/var/lib/jenkins/phablet-flash/imageupdates/pool/ubuntu-418162f2a9ba9a320a2b460938e3d333f84b6f35826200c2a6b6c342ba1e21ef.tar.xz' to '/cache/recovery//ubuntu-418162f2a9ba9a320a2b460938e3d333f84b6f35826200c2a6b6c342ba1e21ef.tar.xz': No space left on device" on mako | 13:49 |
| rsalveti | didrocks: why did you remove the emulator? | 13:50 |
| rsalveti | :P | 13:50 |
| rsalveti | ogra_: right, but is 26 = 20.1? | 13:50 |
| ogra_ | rsalveti, he did an apt-get autremove ... | 13:51 |
| ogra_ | rsalveti, it is | 13:51 |
| didrocks | rsalveti: yeah, I know you wanted to punish people doing that! :p | 13:51 |
| didrocks | I'll note this for next time "never ever remove what rsalveti installed on your system" :) | 13:51 |
| rsalveti | ogra_: right, not sure if that is indeed using the cache partition, need to give it a try here | 13:51 |
| ogra_ | rsalveti, r26 works fine on my maguro with OTA upgrade | 13:51 |
| rsalveti | ogra_: I know the cdimage one works fine | 13:51 |
| rsalveti | just flashed it | 13:51 |
| rsalveti | ogra_: are we flashing the lab devices with -b? | 13:53 |
| ogra_ | could be, not sure | 13:53 |
| ogra_ | doanac` should know | 13:53 |
| ogra_ | or sergio | 13:53 |
| psivaa | the devices in the lab use - phablet-flash ubuntu-system -b --channel trusty-proposed | 13:54 |
| plars | psivaa: I just got to my desk, looks like there's some trouble this morning? | 13:55 |
| plars | rsalveti, ogra_: we are using -b for quite some time now | 13:56 |
| psivaa | plars: yes, a couple but the latest impacting one is that flashing on mako fail due to no space available error on /cache/recovery | 13:56 |
| rsalveti | let me flash my mako to see | 13:56 |
| rsalveti | the cdimage one worked fine | 13:57 |
| Mirv | Mir 0.1.1 now in release pocket | 13:57 |
| Mirv | and time for UDS | 13:57 |
| plars | psivaa: we seem to have a lot of devices showing up with the 0123456789ABCDEF adb id again now | 14:04 |
| psivaa | plars: yes, vila mentioned about it. dont seem to be an id that's used for smoke | 14:05 |
| plars | psivaa: it's a generic one that's used when it can't sort out the real one | 14:05 |
| plars | psivaa: we'll probably need rfowler to reflash them | 14:06 |
| psivaa | plars: ahh ok, could be that we have a bunch of mako's that failed to flash due to the no space reason and they could be it | 14:06 |
| psivaa | plars: on another issue, we had adb protocol fault in kinnara in the morning | 14:07 |
| plars | psivaa: just one? :) | 14:07 |
| plars | psivaa: we do get it a lot less frequently I think, but it still happens from time to time | 14:07 |
| psivaa | even running 'adb devices' from kinnara constantly threw this error | 14:08 |
| psivaa | plars: how do you normally recover from it | 14:08 |
| plars | psivaa: oh, that's different | 14:08 |
| plars | psivaa: in a case like that, I guess we'd just need to restart adb server... I'm not sure I've seen that happen before | 14:09 |
| plars | psivaa: we added quite a few more devices to that system though | 14:09 |
| ogra_ | plars, psivaa, this will be fixed within the next weeks | 14:10 |
| psivaa | plars: yea, it turned out to be one of the old adb processes was causing the issue | 14:10 |
| rsalveti | plars: ogra_: mako worked fine here with system-image | 14:10 |
| ogra_ | very weird | 14:10 |
| psivaa | plars: killing that process fixed it | 14:10 |
| plars | rsalveti: you flashed with -b also? | 14:10 |
| psivaa | Installed: 1.0+14.04.20131108-0ubuntu1 is the phablet tools in the lab | 14:11 |
| rsalveti | plars: yes | 14:11 |
| ogra_ | i dont think it has anything to do with phablet-tools, after all the out of space message seems to come from the device itself | 14:11 |
| rsalveti | plars: http://paste.ubuntu.com/6448304/ | 14:12 |
| plars | brb | 14:12 |
| psivaa | ogra_: ok | 14:14 |
| rsalveti | ogra_: so I think the cache related issue could be a side effect from the broken image | 14:22 |
| ogra_ | rsalveti, thats what i thought too, but -b pushes a new recovery | 14:23 |
| ogra_ | so it should boot the fixed one | 14:23 |
| ogra_ | (it actually boots the new recovery via fastboot) | 14:24 |
| rsalveti | ogra_: right, but that doesn't necessarily mean that it'll clean up the cache partition | 14:25 |
| rsalveti | INFO:phablet-flash:Clearing /data and /cache | 14:26 |
| ogra_ | h,mm, i thought we do that from ubuntu_commands or how thats called | 14:26 |
| rsalveti | but it should in theory clean it as part of the flashing process | 14:26 |
| plars | so was the bad recovery actually on image 25 then? | 14:26 |
| ogra_ | right | 14:26 |
| ogra_ | plars, yes | 14:26 |
| psivaa | plars: ogra_ i tried flashing from image 23 as well | 14:28 |
| psivaa | and the same thing happend | 14:28 |
| ogra_ | psivaa, if you use -b it weill use the new recovery ... and if you were unlucky you had r25 | 14:29 |
| ogra_ | though looking at the above it smells more like the cleanup step failed | 14:30 |
| plars | indeed | 14:30 |
| ogra_ | but you should have seen that in the log | 14:30 |
| plars | ogra_: I don't see anywhere in phablet-tools where fastboot erase is called, could be that we need to really erase those partitions | 14:31 |
| ogra_ | i dont think it calls erase | 14:31 |
| rsalveti | plars: the clean is a shell call done in recovery | 14:32 |
| ogra_ | yeah | 14:32 |
| rsalveti | not via fastboot | 14:32 |
| rsalveti | so that might be failing | 14:32 |
| plars | rsalveti: yeah, it just does rm | 14:32 |
| rsalveti | plars: you might want to force a clean with fastboot then and retry to see | 14:48 |
| plars | rsalveti: yeah, I'm trying to reproduce locally. Flashing 25 right now. | 14:49 |
| sergiusens | fastboot -w created too many support issues | 14:55 |
| ogra_ | yeah | 14:55 |
| ogra_ | well, i can imagine that if there was no /dev and no /proc in r25's recovery that the rm might indeed have failed ... i just dont get why it would also fail in r26 | 14:56 |
| === alex-abreu|afk is now known as alex-abreu | ||
| rsalveti | plars: were you able to reproduce the issue locally? | 15:22 |
| plars | rsalveti: yes, but psivaa said he's seeing the issue going from 23->26 also | 15:23 |
| rsalveti | oh =\ | 15:23 |
| plars | rsalveti: I was also able to get it to flash just fine from 25->26 after it failed once... it was in recovery and not getting the bad adbid like we see in the lab | 15:23 |
| plars | rsalveti: and I just added -d mako | 15:23 |
| plars | rsalveti: it all just worked... so I'm trying to figure out what's going on | 15:23 |
| === alan_g is now known as alan_g|tea | ||
| === alan_g|tea is now known as alan_g | ||
| plars | psivaa, rsalveti: so I just watched it get the same error when trying to go from 26->25 | 15:55 |
| plars | wait, maybe not | 15:55 |
| plars | this one might have just been an adb failure | 15:55 |
| plars | adb/mtp thing probably | 15:56 |
| rsalveti | right | 15:56 |
| rsalveti | I think you might be able to extract the logs from the broken flags if you're able to boot the device | 15:57 |
| rsalveti | check /cache/recovery/last_log | 15:57 |
| plars | rsalveti: ok, after coming back up, all I have is busybox and there's no fstab or /cache. If I mount mmcblk0p22 by hand though, I found what seems to be a cache partition there, and here's the last_log: http://paste.ubuntu.com/6448925/ | 16:22 |
| plars | rsalveti: this was after installing 25, and then trying to install 26 | 16:22 |
| plars | oddly: 'Skipping missing file: version-25.tar.xz' was in the log | 16:23 |
| rsalveti | plars: hm, right, but the log is from 25 I believe | 16:25 |
| rsalveti | weird | 16:25 |
| plars | rsalveti: yeah, so maybe we didn't get far enough for that log to happen | 16:26 |
| ogra_ | if you are in busybox you are not in recovery | 16:26 |
| ogra_ | so indeed there is no /cache | 16:26 |
| rsalveti | yeah | 16:26 |
| plars | since it happened on the adb push, that makes sense | 16:26 |
| rsalveti | plars: let me try to flash 25 to see | 16:27 |
| rsalveti | plars: got the issue here | 16:40 |
| rsalveti | plars: so, what happens, when flashing 25 with -b it will first flash the recovery image with fastboot | 16:40 |
| rsalveti | plars: then it'll try to load recovery, which will fail | 16:40 |
| rsalveti | plars: meanwhile it waits for adb, thinking that the recovery will be up | 16:41 |
| rsalveti | as recovery fails to boot, the device boots the previous image instead | 16:41 |
| rsalveti | and phablet-tools thinks that the recovery is then up, because adb is there | 16:41 |
| plars | rsalveti: ah, I see. so we think we're booting 25 but really we're not | 16:41 |
| plars | rsalveti: I see | 16:42 |
| rsalveti | http://paste.ubuntu.com/6449025/ | 16:42 |
| rsalveti | sergiusens: we should be checking for /sbin/recovery when flashing | 16:42 |
| rsalveti | sergiusens: to check if we're indeed in recovery | 16:42 |
| sergiusens | rsalveti, sure, I'm already checking for something; checking for recovery specifically shouldn't be complicated | 16:43 |
| sergiusens | rsalveti, my wonder is, what did it boot? | 16:43 |
| sergiusens | plars, any indications of why it failed to boot? | 16:44 |
| rsalveti | sergiusens: it fails to load the recovery, then it tries to boot the previous image | 16:44 |
| rsalveti | but it seems that the 25->26 migration worked fine | 16:44 |
| sergiusens | rsalveti, yeah, but why? :-) | 16:44 |
| rsalveti | so the error when copying the file into /cache happens because it's not really in the recovery | 16:45 |
| plars | rsalveti: really? for me, I get a broken system - just boots to busybox | 16:45 |
| rsalveti | sergiusens: that's the default behavior | 16:45 |
| sergiusens | rsalveti, how could've recovery been broken :-) | 16:45 |
| rsalveti | sergiusens: we had a regression | 16:45 |
| sergiusens | rsalveti, that's the how I'm asking :-) | 16:45 |
| sergiusens | ack | 16:45 |
| plars | rsalveti: you are flashing cdimage? or ubuntu-system image? | 16:45 |
| sergiusens | plars, rsalveti super easy to have a better check if we are in recovery, no worries | 16:45 |
| rsalveti | sergiusens: http://phablet.ubuntu.com/gitweb?p=CyanogenMod/android_system_core.git;a=commitdiff;h=aefb9653a367a6fd528f2ad48a1cce8945aecb3c;hp=87b40ee4a93bd682506fa082f2c2d1a0f5a60f04 | 16:46 |
| rsalveti | which was fixed at http://phablet.ubuntu.com/gitweb?p=CyanogenMod/android_system_core.git;a=commitdiff;h=a1527ce86c90842b693941723888f7739f3bf473;hp=7a318cec11f017c5ae9c186dae2b01f2dc4d3ef4 | 16:46 |
| rsalveti | plars: ubuntu-system | 16:46 |
| didrocks | ogra_: can we get a new image build? | 16:46 |
| didrocks | ogra_: now that Mir entered :) | 16:46 |
| plars | strange... for me, I'm getting exactly what we have seen on multiple devices in the lab | 16:46 |
| rsalveti | didrocks: sure :-) | 16:46 |
| didrocks | rsalveti: thanks! | 16:47 |
| ogra_ | didrocks, yeah | 16:47 |
| sergiusens | why does everyone get to push but me? :-P | 16:47 |
| didrocks | it's christmas | 16:47 |
| ogra_ | rsalveti, do you already ? | 16:47 |
| didrocks | I saw some snow this morning ;) | 16:47 |
| rsalveti | didrocks: having newer images is always a good thing | 16:47 |
| rsalveti | didrocks: easier to test and revert broken changes as well | 16:47 |
| ogra_ | we should just build one every hour ! | 16:47 |
| rsalveti | ogra_: nops | 16:47 |
| rsalveti | ogra_: yeah | 16:47 |
| ogra_ | then i'll do | 16:47 |
| sergiusens | rsalveti, ogra_ you will starve the testing infra though | 16:47 |
| didrocks | rsalveti: fully agreed, we just need to have a way to communicate those to you and us more easily | 16:47 |
| rsalveti | didrocks: why do we need the communication here? | 16:48 |
| ogra_ | sergiusens, nah, we'll have our fast emulator and all will be fine | 16:48 |
| rsalveti | we don't | 16:48 |
| rsalveti | the communication is the changelog :-) | 16:48 |
| sergiusens | ogra_, big LOL | 16:48 |
| ogra_ | :) | 16:48 |
| didrocks | rsalveti: well, we do plan landings on certain image/try to communicate that | 16:48 |
| rsalveti | didrocks: and that's wrong | 16:48 |
| sergiusens | ogra_, the emulator would be mostly useful for apps, but not image testing ;-) | 16:48 |
| rsalveti | didrocks: you should only know when something lands, not planning when it'll land | 16:48 |
| ogra_ | === Image r27 building === | 16:48 |
| didrocks | rsalveti: will be easier once we have the CI Airline I hope to know more on feature-based what landed | 16:49 |
| sergiusens | ogra_, we can certainly only run on real hw if the emulator test pass | 16:49 |
| ogra_ | sergiusens, well, i thought the plan was to use it in all testing infra | 16:49 |
| rsalveti | we don't need to know when it'll be landing | 16:49 |
| rsalveti | we just need to know when it *landed* | 16:49 |
| didrocks | well, we want to know when big thangs are landing at the same time | 16:49 |
| didrocks | to avoid clashes | 16:49 |
| didrocks | but apart from that, yeah | 16:49 |
| ogra_ | right, only have a test on real HW if all emu tests were good | 16:49 |
| ogra_ | didrocks, and that ifno comes from the changelogs | 16:50 |
| rsalveti | right, but even if we get clashes, we want more images to test all the steps | 16:50 |
| didrocks | ogra_: note on a "feature-approach", and not planning clashes | 16:50 |
| ogra_ | rsalveti, the point is that asac wants us to have stuff tested in full context before we upload | 16:51 |
| ogra_ | nobody does that | 16:51 |
| ogra_ | (or do you run the full AP suite for an android change you do) | 16:51 |
| rsalveti | ogra_: right, but that's not a blocker here | 16:51 |
| rsalveti | having more images is *always* a good thing | 16:51 |
| ogra_ | thats true | 16:51 |
| ogra_ | but sergiusens has a point | 16:51 |
| ogra_ | we'd saturate the lab | 16:51 |
| rsalveti | right, that's our only limitation | 16:52 |
| rsalveti | but we can do as most images we can depending on how long it takes to validate an image | 16:52 |
| rsalveti | what would that be? | 16:52 |
| rsalveti | at every 4 hours? | 16:52 |
| asac | rsalveti: in theory we can produce images in parallel and validate them | 16:52 |
| asac | the problem is not parallelizing the validation | 16:52 |
| asac | but the building | 16:53 |
| asac | its not really clear what atomic snapshot you will take when you start the job i think :) | 16:53 |
| ogra_ | asac, we only talk about the actual images | 16:53 |
| ogra_ | not about any parallel test builds etc | 16:53 |
| asac | (though you could mitigate that by pulling the packages file out of the tool and passing itas an input) | 16:53 |
| ogra_ | we currenntly only build two per day at most | 16:53 |
| asac | right | 16:54 |
| rsalveti | and need to speed this up | 16:54 |
| asac | so for now we talk about the future anyway because emulator is not there etc. | 16:54 |
| ogra_ | having a more fine grained view in the changelogs through building more of them will make identifying issues a lot easier | 16:54 |
| rsalveti | right, but I think we can already improve by building at least a few images a day | 16:54 |
| asac | however, if the emulator is fully landed, i anticipate that we can produce as manages as we want a day. | 16:54 |
| ogra_ | from a builder POV we could build one every hour | 16:54 |
| rsalveti | as we always get stuff that lands in the archive which is not part of the landing spreadsheet | 16:54 |
| asac | just not test all on all phones | 16:54 |
| rsalveti | so it's *always* good to produce and test newer images | 16:54 |
| ogra_ | 30min for cdimage + 15min for system-image post processing | 16:55 |
| asac | we would basically validate all iamges in emlator and then pick those that we are interested in (for instance, because we consider them a release image) and test them everywhere | 16:55 |
| asac | we discussed hooking image productionm up with the publisher runs | 16:55 |
| asac | so we get one image somewhere stored for each injection of what really goes in the archive | 16:55 |
| asac | has to wait for cloud buildd which is worked on somewhere | 16:56 |
| * cjwatson forces android-emulator back in despite p-m not quite understanding multiarch | 16:56 | |
| asac | p-m? | 16:56 |
| cjwatson | proposed-migration | 16:56 |
| asac | ah | 16:56 |
| cjwatson | asac: https://rt.admin.canonical.com/Ticket/Display.html?id=62272 and https://bugs.launchpad.net/bugs/1247461 | 16:57 |
| ubot5 | Ubuntu bug 1247461 in launchpad-buildd "Move live filesystem building into Launchpad" [High,In progress] | 16:57 |
| ogra_ | right, that will give us parallel builds | 16:58 |
| josepht | asac: once an image + new packages is promoted don't all previous images need to be updated and retested? | 16:58 |
| asac | not really. all we do is snapshotting the archive after a publisher run in the form of an image | 16:59 |
| ogra_ | why would they ? | 16:59 |
| asac | so whatever is in there is already finally committed | 16:59 |
| asac | until overwritten for which a new image would exist | 16:59 |
| josepht | we create an image for archive+A and archive+B in parallel, archive+A passes all tests so now archive = archive+A, if archive+B passes all tests we don't know that archive+A+B will pass all tests and need to test that scenario, right? | 17:01 |
| rsalveti | asac: right, but let's try to improve our current situation | 17:02 |
| asac | josepht: yeah you are right, but you are thinking about a different level | 17:02 |
| rsalveti | how long does it take to test an image currently? | 17:02 |
| rsalveti | plars: ^ | 17:02 |
| asac | josepht: you are thinking about the premerge level... there indeed we need to invalidate A after we land B | 17:02 |
| plars | rsalveti: somewhere around 3 hours I believe | 17:02 |
| asac | hwoever, we can try to be smart and only invalidate if the changes are in the same dependency tree for instance | 17:02 |
| plars | rsalveti: assuming everything goes right :) | 17:02 |
| didrocks | plars: but then, you rekick jobs, right? | 17:02 |
| plars | didrocks: yes | 17:02 |
| asac | josepht: we talk about images produced after the fact from whatever landed in the real archive | 17:03 |
| didrocks | so how long until we get a clean state, I guess | 17:03 |
| didrocks | and you rekicked enough? :) | 17:03 |
| plars | didrocks: if I'm up and see it needs to be done, and there's not another image already coming | 17:03 |
| josepht | asac: ah I see | 17:03 |
| rsalveti | right, can we build a new image at every 6 hours then? | 17:03 |
| * asac hope that makes sense | 17:03 | |
| rsalveti | or 8? | 17:03 |
| plars | didrocks: for instance, if the devices could work right now, I wouldn't bother retesting because I see there's a new image on the way, and it would just delay | 17:03 |
| asac | rsalveti: well. so i dont want random images | 17:03 |
| asac | i want precisely cut images | 17:03 |
| rsalveti | asac: why not? | 17:03 |
| rsalveti | asac: that's *wrong* | 17:04 |
| asac | its not :) | 17:04 |
| asac | not sure how that can be wrong :) | 17:04 |
| rsalveti | asac: we want one image per change | 17:04 |
| rsalveti | or the most images we can | 17:04 |
| asac | right. | 17:04 |
| rsalveti | otherwise it's a pita to find regressions | 17:04 |
| rsalveti | one image a day is *wrong* and only causes pain | 17:04 |
| asac | one image per chagne... but now that we can't get that (for various reasons) short term, it doesnt mean we should give up | 17:04 |
| cjwatson | an image isn't any more precisely cut just because somebody asked for it manually | 17:04 |
| asac | and cut images in the middle of changes :) | 17:04 |
| cjwatson | there shouldn't be a "middle of changes" | 17:05 |
| rsalveti | asac: give me one example | 17:05 |
| asac | rsalveti: i am supportive of more images per day. just not at cronjob kicked | 17:05 |
| rsalveti | exactly | 17:05 |
| asac | cjwatson: depends on which level you define changes :) | 17:05 |
| rsalveti | once it's in the archive, there's no "middle of changes" | 17:05 |
| cjwatson | get your packaging metadata right so that proposed-migration can migrate it properly | 17:05 |
| rsalveti | exactly | 17:05 |
| cjwatson | and then roll | 17:05 |
| rsalveti | we should put it in cron | 17:05 |
| cjwatson | manual image building requests are a workaround for lazy metadata maintenance | 17:06 |
| rsalveti | image is just an archive snapshot | 17:06 |
| asac | "get your packaging metadata right" ... please give up on that hope. all you can hope for is "get youir packaging meta data right most of the times please" | 17:06 |
| asac | but even if everyone does it right most of the times, if the engineering team is big enough you will always have problems :) | 17:06 |
| cjwatson | I'm not going to give up on that hope, because working on that basis made, empirically, a *massive* difference to the day-to-day quality of Ubuntu. | 17:06 |
| ogra_ | cjwatson, *my* manually built images are *allways* more precisely cut, i sand and olish them :P | 17:06 |
| ogra_ | *polish | 17:06 |
| cjwatson | In fact I think it's mostly reality rather than hope. | 17:07 |
| rsalveti | as long we're not saturating our test lab, we should be fine to produce the most images we can | 17:07 |
| asac | exactly... its mostly | 17:07 |
| asac | :) | 17:07 |
| cjwatson | Yes, there's the odd exception, but they're, well, exceptional | 17:07 |
| cjwatson | We shouldn't structure processes around exceptions | 17:07 |
| rsalveti | +1 | 17:07 |
| asac | and will alwayus be mostly... but 600* mostly might mean rarely :) | 17:07 |
| ogra_ | the point of automated builds is to simply have an as small changeset as possible ... | 17:07 |
| ogra_ | doing only two manual builds completely goes against that | 17:08 |
| ogra_ | and causes lots and lots of guesswork on our side until we found the actual issue | 17:08 |
| rsalveti | yeah | 17:08 |
| asac | its human. noone is perfect. hence designing the process must work in continuous exceptions due to human failures induced | 17:08 |
| rsalveti | horrible | 17:08 |
| rsalveti | see the udev one we had a few weeks ago | 17:08 |
| ogra_ | which wastes more developer time than having more images per day | 17:08 |
| rsalveti | the package was uploaded at wednesday and we found the issue at saturday | 17:08 |
| ogra_ | asac, its wasting human resources | 17:08 |
| asac | ogra_: the problem can be solved by using a cronjob or by kicking builds again at a 3 times a day schedule | 17:09 |
| asac | with shbuman smartness | 17:09 |
| ogra_ | asac, how often did we get onto the wrong path in the recent times if an image was broken | 17:09 |
| ogra_ | wasting hours to find it was the other change that we didnt look at yet | 17:09 |
| rsalveti | right, even if we build it at every 12 hours, it's already better than what we have now | 17:09 |
| ogra_ | rsalveti, we do | 17:09 |
| asac | i agree that we should revisit the image schedule :) | 17:09 |
| rsalveti | ogra_: not currently | 17:09 |
| ogra_ | we usually have two manual built ones per day | 17:10 |
| rsalveti | well, it's manual | 17:10 |
| asac | not sure if crontab is the right approach. lets talk about it tomorrow on the standup | 17:10 |
| ogra_ | (if there isnt UDS) | 17:10 |
| asac | i would love to see at least 3 images :) | 17:10 |
| ogra_ | so the timing is already 12h | 17:10 |
| rsalveti | ogra_: right, I want it to be automatically triggered | 17:10 |
| rsalveti | ogra_: as the archive is always moving | 17:10 |
| ogra_ | right | 17:10 |
| ogra_ | 4 per day should be fine | 17:10 |
| ogra_ | automated and all | 17:10 |
| cjwatson | I'm completely boggled at the insistence that things must *not* be automated, frankly | 17:10 |
| rsalveti | can we try to put it back to the cron and see how it goes? | 17:10 |
| cjwatson | It goes against all the engineering practice I know | 17:11 |
| rsalveti | haha, exactly | 17:11 |
| ogra_ | cjwatson, all but asac and didrocks in here say we want automated | 17:11 |
| didrocks | ogra_: I didn't say we don't want automation, I said that we want to cut after each big change as a first approach | 17:11 |
| cjwatson | I understand "automation is hard and we can't quite manage it yet" (have said that myself on various occasions) - I just don't understand "the ideal is manual" | 17:11 |
| ogra_ | right, but why ? | 17:11 |
| asac | for the record: noone says "we dont want automation" :) | 17:12 |
| ogra_ | didrocks, you land all your stuff in one chunk anyway | 17:12 |
| asac | lets discuss it tomorrow | 17:12 |
| asac | i will join you guys there | 17:12 |
| ogra_ | where ? | 17:12 |
| cjwatson | OK, so you're arguing against regularity but not against automation, I see | 17:12 |
| asac | standup | 17:12 |
| cjwatson | I disagree but at least that isn't so boggling :-) | 17:12 |
| asac | hehe | 17:12 |
| asac | its also not exactly what i am saying :) | 17:12 |
| ogra_ | asac, ah, the landing team one | 17:12 |
| asac | but closer | 17:13 |
| rsalveti | asac: why tomorrow? | 17:14 |
| rsalveti | asac: let's just put it back to cron today and see how it goes | 17:14 |
| didrocks | ogra_: if you land all the stuff in one chunk, well, you have to debug it, you decided the size of the chunk :) | 17:14 |
| rsalveti | why would that cause us any pain now | 17:14 |
| asac | i want to first ensure people understand what we are saying :) | 17:14 |
| didrocks | ogra_: we need to be able to produce images after each "chunk" landing, but a chunk is defined by the developer itself | 17:14 |
| asac | I dont see us having any pain from having manual image building | 17:14 |
| asac | it gives us many good | 17:14 |
| rsalveti | you can still manually trigger new images | 17:15 |
| rsalveti | but we should *always* have it in cron | 17:15 |
| ogra_ | didrocks, thats not true ... your chunks are controlled in the spreadsheet | 17:15 |
| rsalveti | because the archive is *always* moving | 17:15 |
| ogra_ | not defined by a developer | 17:15 |
| didrocks | ogra_: I just want to avoid A + B entered, "A says I don't care, it's B", it's not me, and B says "I don't care, it's because of A" | 17:15 |
| asac | so ... i know for sure that we will spin more than enough images once the landing stuff is fully back operational again | 17:15 |
| rsalveti | all I'm asking is to add it back in cron at every 8/12 hours | 17:15 |
| didrocks | ogra_: well, not really in our current model, but yeah, we would need to define priorities rather than image #, agreed | 17:15 |
| asac | hence, i believe the issue you try to solve is the low velocity due to the 1ss move paired with a UDS going on now :) | 17:15 |
| didrocks | ogra_: and I think we do align with what rsalveti told seeing it that way | 17:16 |
| asac | and yes, we should build at least 2 images a day even if not much is going on | 17:16 |
| ogra_ | didrocks, so we can switch cron back on and set it to say 6h ? | 17:16 |
| rsalveti | I just want to make sure we always have a fallback in case we don't have humans to trigger a new image | 17:16 |
| ogra_ | (to have 4 images / day) | 17:16 |
| asac | well. as i said befeore the capacity on testing is really bound to emulator :) | 17:17 |
| didrocks | ogra_: well, for me, it would be a +1 but rather 8 hours TBH, to let the CI team to rekick the tests jobs and giving some slack for that). 3 images a day will be a start | 17:17 |
| ogra_ | asac, we should have as many as the infra allows imho | 17:17 |
| asac | so ... dont hurry | 17:17 |
| rsalveti | ogra_: yeah | 17:17 |
| didrocks | ogra_: but I don't think I'm the one able to take the decision :) | 17:17 |
| rsalveti | ogra_: I'd first try with 8h | 17:17 |
| ogra_ | didrocks, why do you care if it is 4 or 8 ? | 17:17 |
| rsalveti | ogra_: and we can improve that as we go | 17:18 |
| didrocks | ogra_: because then, the testing infra can't rekick the jobs | 17:18 |
| ogra_ | if your team only lands something every second image thats fine | 17:18 |
| asac | i care because i want to ensur3e the CI folks that support those images whenthey hit validation | 17:18 |
| didrocks | if a new image is produced | 17:18 |
| asac | know what to do and how to prioritize | 17:18 |
| didrocks | ogra_: they can't "restest an older image" | 17:18 |
| ogra_ | asac, but the CI team knows when cron runs | 17:18 |
| asac | anyway | 17:18 |
| didrocks | ogra_: it's not only that, most of their time is just "relaunch the jobs" | 17:18 |
| didrocks | because the whole thing is flacky | 17:18 |
| asac | not sure why is it suddenly such a big issue todya? | 17:18 |
| rsalveti | that's why let's try with either 8h or 12h | 17:18 |
| ogra_ | and as the one who does the manual builds i must say that the requirement for manual builds is largely always at the same time | 17:18 |
| didrocks | asac: and so you need to reboot the device | 17:19 |
| didrocks | rsalveti: +1 | 17:19 |
| ogra_ | so why not just have images inbetween too ? | 17:19 |
| ogra_ | rsalveti, i dont get it | 17:19 |
| didrocks | ogra_: because you don't let the CI team to get test results | 17:19 |
| rsalveti | ogra_: we can, I just want to make sure we're not saturating our test lab | 17:19 |
| ogra_ | why does having 2 images help more than 4 | 17:19 |
| didrocks | if you kick a new image | 17:19 |
| didrocks | because we give them more time to rekick the jobs | 17:19 |
| didrocks | when you finished building a new image | 17:20 |
| didrocks | they can't retest an older one | 17:20 |
| ogra_ | didrocks, but if they dont make this image they will make the one in 4h | 17:20 |
| ogra_ | i dont get it | 17:20 |
| rsalveti | yeah | 17:20 |
| asac | if noone can tell me why this is so much more important today than yesterday, lets really talk tomorrow in the standup. | 17:20 |
| asac | :) | 17:20 |
| ogra_ | there is nothing that puts time pressure on anyone | 17:20 |
| didrocks | ogra_: but then, there is the same issues, they need to rekick | 17:20 |
| rsalveti | that's why as long we're not saturating the lab, we're fine | 17:20 |
| didrocks | ogra_: and you just rebuild the next next image | 17:20 |
| didrocks | and the story repeats :) | 17:20 |
| ogra_ | just make sure you land your stuff inbetween two builds if you want to make sure all of it lands at the same time | 17:21 |
| rsalveti | asac: why can't we just add it there and see what happens | 17:21 |
| asac | rsalveti: right now until we have the emuilator fully rolled out its about saturating support folks | 17:21 |
| ogra_ | and the schedule is known | 17:21 |
| ogra_ | didrocks, why do they need to rekick | 17:21 |
| ogra_ | or even care for the images | 17:21 |
| didrocks | ogra_: repeating "because it's flacky" | 17:21 |
| asac | rsalveti: understand taht i have to figure how to deal with that with CI process | 17:21 |
| rsalveti | asac: sure, but all I'm asking is more images than we have today, and that's for sure something we can improve even without the emulator | 17:21 |
| ogra_ | they just need to check the results for the image their change landed in | 17:21 |
| didrocks | ogra_: see a vanilla test run | 17:21 |
| ogra_ | no matter when they landed their stuff | 17:21 |
| didrocks | before plars or psivaa starts looking at it | 17:21 |
| didrocks | you have a lot of tests not running | 17:22 |
| asac | rsalveti: not without undersanding and clarifyuing with my team what they should focus on testing etc. | 17:22 |
| didrocks | or with just 1 test | 17:22 |
| asac | rsalveti: on technology side its easy | 17:22 |
| didrocks | (and beeing representated as "green" btw :/) | 17:22 |
| didrocks | ogra_: so, they restart the jenkins jobs as of now | 17:22 |
| asac | if you guys feel so strongly about more images | 17:22 |
| rsalveti | right, but you don't get that we're also wasting resources by not triggering more images | 17:22 |
| asac | lets really check how we can get that | 17:22 |
| rsalveti | because it's really a pain to find and revert regressions | 17:22 |
| asac | i am not sayuing we shouldnt have more images | 17:23 |
| rsalveti | if we get tons of changes at every image we produce | 17:23 |
| asac | just lets discuss how to do that brest | 17:23 |
| didrocks | (btw, while we discuss, we still can't test an image I guess and powerpc is broken in distro ;)) | 17:23 |
| ogra_ | asac, it wastes immense amount of manpower to just make the CI team happy | 17:23 |
| ogra_ | while they would just have to obey to a cronned build schedule | 17:23 |
| ogra_ | finding an issue in an image that has 50 packages changed is lots and lots harder than finding it in one that only has 10 | 17:24 |
| rsalveti | +100 | 17:25 |
| rsalveti | it's just stupid to fight automation | 17:25 |
| ogra_ | right | 17:25 |
| rsalveti | as the archive is always moving | 17:25 |
| ogra_ | asac, i worked with this system for 4 months now ... and thats the result of my experience | 17:25 |
| ogra_ | we waste manpower and that doesnt need to happen | 17:26 |
| asac | i am the first who wants more images. if the landing team would be operational they would at least spin 2 images a day | 17:27 |
| asac | it should be closer to 3 | 17:27 |
| ogra_ | and its a trivial change ... just a crontab entry | 17:27 |
| asac | and as we add more "smart automation" we woulc increase the pace | 17:27 |
| ogra_ | but why do we have to make it depend on the landing team | 17:27 |
| ogra_ | the landing team can land stuff even with automated builds | 17:27 |
| rsalveti | yeah, that's the wrong, as the landing team doesn't have the power to freeze the archive :-) | 17:27 |
| rsalveti | *that's wrong | 17:28 |
| ogra_ | its not like it is hard, the schedule will be known so you know when your stuff gets into the next image | 17:28 |
| rsalveti | didn't make at one image? wait for the next | 17:28 |
| ogra_ | right | 17:28 |
| rsalveti | just don't make everyone to wait on your call | 17:29 |
| jibel | cihelp: publisher on d-jenkins fails to create new jobs on public instance | 17:29 |
| rsalveti | about triggering a new image | 17:29 |
| jibel | example http://d-jenkins.ubuntu-ci:8080/job/trusty-adt-python-misaka/1/? | 17:29 |
| retoaded | jibel, looking | 17:29 |
| jibel | retoaded, it is only the creation of a new jobs, publication of new builds work fine. | 17:32 |
| retoaded | jibel, it wouldn't happen to be a matrix job would it? | 17:34 |
| tedg | Not sure who to blame, but having DNS on the CI VPN is pretty cool. Thanks! | 17:35 |
| jibel | retoaded, it is | 17:37 |
| retoaded | jibel, http://10.98.3.6:8080/plugin/build-publisher/instance/0/publisherThread/currentState/output makes me think that it could be related to the just updated matrix-reloaded plugin. | 17:37 |
| retoaded | jibel, i will probably have to revert it back. | 17:37 |
| jibel | retoaded, I'm not sure, I see no reference to the plugin in the trace. But if it's a recent change revert it back and we'll see | 17:40 |
| retoaded | jibel, it near the bottom: Caused by: java.lang.NullPointerException at hudson.matrix.MatrixProject.onLoad(MatrixProject.java:474) | 17:43 |
| retoaded | jibel, and as soon as the currently running jobs complete I'll revert | 17:43 |
| jibel | retoaded, it is a reference to a Matrix object from jenkins core, not the plugin. | 17:49 |
| ogra_ | === Image r27 DONE === | 17:51 |
| * popey updates | 17:51 | |
| ogra_ | (totally forgot that over the abive discussion) | 17:51 |
| ogra_ | *above | 17:51 |
| jibel | retoaded, it is not a new problem apparently | 17:53 |
| jibel | retoaded, trusty-adt-simplejson failed to publish too | 17:53 |
| jibel | retoaded, oh, and it vanished from d-jenkins | 17:56 |
| jibel | WTF | 17:56 |
| Ursinha | ogra_, can I update my phone then? :P | 17:56 |
| jibel | retoaded, so on the filesystem there is a /var/lib/jenkins/jobs/trusty-adt-simplejson but not on the UI | 17:57 |
| ogra_ | Ursinha, well, my maguro just updated fine here | 17:57 |
| jibel | retoaded, and publication fails with the same error 500 | 17:57 |
| ogra_ | bah, since when is facebook on the home lens | 17:58 |
| Ursinha | ogra_, are you using -proposed? read-only? | 17:58 |
| ogra_ | yes | 17:58 |
| Ursinha | ogra_, at least since yesterday when I flashed mine with the current devel | 17:58 |
| ogra_ | well, i do OTA updates | 17:58 |
| ogra_ | Ursinha, heh, that was ricardos fault ... r25 was broken | 17:59 |
| Ursinha | ogra_, it rarely is rsalveti's fault, hehe | 17:59 |
| ogra_ | haha | 17:59 |
| ogra_ | never ever :) | 17:59 |
| Ursinha | rarely ;) | 17:59 |
| rsalveti | lol | 18:03 |
| Ursinha | ogra_, so I heard you test all the images :P I have a nexus 4 and whenever I put some music on and change to another app the music stops playing after a few seconds, is that a known issue? | 18:41 |
| popey | it shouldn't | 18:42 |
| ogra_ | Ursinha, i test only on maguro ... | 18:42 |
| Ursinha | I just updated with the latest -proposed image | 18:42 |
| popey | what image are you running Ursinha ? | 18:42 |
| ogra_ | popey, is the makoman | 18:42 |
| popey | Ursinha: are you using the music app or something else? | 18:42 |
| Ursinha | popey, Home lens, click on music and then play | 18:43 |
| === fginther changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: fginther | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: Some services are down (1SS move), network slowness | ||
| Ursinha | about this phone says r27 | 18:43 |
| popey | my phone is dead.. one moment | 18:43 |
| popey | works fine here | 18:45 |
| Ursinha | popey, music app has the same behavior, but the interesting thing is that the music stops playing but when I return to the music app it resumes playing as if it never stopped | 18:45 |
| Ursinha | there's no sound but it seems the app was playing it | 18:45 |
| popey | i am listening to music which is playing in the music app but I'm on the home screen | 18:45 |
| popey | it even plays when locked | 18:45 |
| popey | and continies to play after a song finishes | 18:46 |
| Ursinha | popey, what should I do then? | 18:46 |
| popey | is it launching the music app ? | 18:46 |
| Ursinha | popey, yes, it launches and I'm able to listen to music, the only problem is when I change screens it stops after a few seconds | 18:47 |
| popey | http://popey.com/~alan/phablet/device-2013-11-20-184641.png | 18:47 |
| popey | like that | 18:47 |
| Ursinha | it's playing as it should (I think), displaying the cover and band/song name | 18:47 |
| popey | was this a clean install or an update to an existing install? | 18:47 |
| Ursinha | popey, it was a clean install yesterday, with the current devel image | 18:48 |
| popey | odd | 18:48 |
| Ursinha | I switched to -proposed today and the problem is the same | 18:48 |
| popey | I dont understand why it's working for me and not you | 18:48 |
| Ursinha | hehe | 18:48 |
| Ursinha | popey, btw it keeps playing when screen is locked | 18:49 |
| Ursinha | popey, I'll try a clean install and let you know | 18:50 |
| cjwatson | http://people.canonical.com/~cjwatson/tmp/emulator-yay.png | 18:51 |
| cjwatson | didn't *quite* manage to get it up on the hangout just now ... | 18:51 |
| ogra_ | :) | 18:51 |
| ogra_ | thats quite a wide screen | 18:52 |
| cjwatson | dual monitor | 18:52 |
| * ogra_ has triple ... but screenshots always only show one screen | 18:52 | |
| cjwatson | import(1) doesn't really understand that and just gives me the whole thing. I should probably use the screenshot thing in the desktop but old habits die very hard | 18:52 |
| ogra_ | might be nvidias fault | 18:52 |
| ogra_ | ah | 18:53 |
| ogra_ | yeah, i only use alt+print | 18:53 |
| fginther | cyphermox, kenvandine, robru, we need to restart q-jenkins as soon as possible | 19:09 |
| robru | fginther, fine by me... | 19:10 |
| kenvandine | fginther, go for it | 19:10 |
| cyphermox | yeah, fine | 19:10 |
| fginther | there is a build in progress (for 11 hours) is it safe to kill ti? | 19:10 |
| fginther | it | 19:10 |
| robru | fginther, yeah, whatever. we can restart it if necessary. i'm not waiting on anything myself | 19:11 |
| robru | fginther, also 11 hours usually indicates a problem anyway | 19:11 |
| fginther | robru, I kinda figured that :-) | 19:11 |
| fginther | robru, kenvandine cyphermox thanks for the input | 19:11 |
| === Ursinha is now known as Ursinha-afk | ||
| === Ursinha-afk is now known as Ursinha | ||
| rsalveti | Ursinha: background music working fine with image 27, let me help you debugging the issue | 20:19 |
| Ursinha | rsalveti, okay, thanks | 20:19 |
| fginther | cyphermox, is it difficult to re-deploy all of the cu2d jobs? | 20:47 |
| cyphermox | well, i'd love to avoid it if possible, why? | 20:48 |
| fginther | cyphermox, the jenkins root moved from /var/lib/jenkins to /iSCSI/jenkins. and some of the scripts had the old path hard coded | 20:49 |
| fginther | cyphermox, I have MPs to update the scripts and the job. I just want a plan for how best to do this | 20:49 |
| fginther | would prefer to actually do the update tomorrow when more help is available | 20:50 |
| cyphermox | are you saying you would prefer or are you asking my opinion? | 20:58 |
| fginther | cyphermox, I just want some idea of what is involved? does it require lots of downtime, do we need to make special arrangments to make it happen, etc.? | 21:00 |
| cyphermox | not really, I think we just need to update the paths in lp:cupstream2distro and lp:cupstream2distro-config (if any there), redeploy all the jobs, etc, make sure all the necessary machines have their copies of those branches updated | 21:03 |
| cyphermox | it would take an hour of work maybe? | 21:03 |
| fginther | cyphermox, good. I do have MPs here: | 21:03 |
| cyphermox | but I could see it being done much more cleanly if we use a symlink as a temporary step to avoid breaking things | 21:03 |
| fginther | https://code.launchpad.net/~fginther/cupstream2distro/jenkins-home/+merge/195848 | 21:03 |
| fginther | https://code.launchpad.net/~fginther/cupstream2distro-config/update-stack_status/+merge/195956 | 21:03 |
| fginther | cyphermox, things are already broken :-( | 21:05 |
| cyphermox | how so? | 21:06 |
| fginther | the check jobs are looking for job data under /var/lib/jenkins. it's not there | 21:06 |
| cyphermox | right | 21:06 |
| cyphermox | that's partly why i mentioned a symlink | 21:06 |
| cyphermox | if not /var/lib/jenkins -> /iSCSI/jenkins then one for just cu2d | 21:07 |
| cyphermox | that allows a smooth transition, but not meant to stay in place | 21:07 |
| fginther | cyphermox, hmm, ok | 21:07 |
| cyphermox | so, both branches look fine | 21:07 |
| fginther | thanks, I'll email out a plan | 21:08 |
| cyphermox | fwiw I see other places in cupstream2distro-config that need a path change | 21:10 |
| fginther | cyphermox, ack! can you add them to the bug: https://bugs.launchpad.net/cupstream2distro/+bug/1252750 | 21:47 |
| ubot5 | Ubuntu bug 1252750 in Canonical Upstream To Distro "latest_autopilot_results hardcodes a jenkins root directory" [Undecided,New] | 21:47 |
| cyphermox | do you have an equivalent bug for cupstream2distro-config? | 21:48 |
| fginther | cyphermox, no | 21:48 |
| robru | fginther, no CI after 5 hours? can you look please? https://code.launchpad.net/~robru/friends-app/autopilot-py3/+merge/195979 thanks | 21:50 |
| fginther | robru, all the phones are blocked due to an image flash problem | 21:52 |
| fginther | robru, it is being worked | 21:52 |
| robru | fginther, oh ok thanks. just really curious about what happens with that branch ;-) | 21:53 |
| robru | no worries | 21:53 |
| Saviq | fginther, afraid makos started to get stuck in flashing, too: http://s-jenkins.ubuntu-ci:8080/computer/mako-04cb53b598546534/? http://s-jenkins.ubuntu-ci:8080/computer/mako-04cbcc545f5328a5/? | 23:01 |
| fginther | Saviq, thanks for the ping. Someone is working on it right now, but they have to manually reflash each device this time | 23:04 |
| fginther | Saviq, all three of them were dead a couple hours ago | 23:06 |
| asac | curious: what happened to them? | 23:06 |
| Saviq | fginther, ah | 23:06 |
| fginther | asac, AIUI there was a problem with the image today, plars do you know more? | 23:06 |
| asac | ok thanks. nevermind then | 23:08 |
| fginther | were also still being hit by https://bugs.launchpad.net/phablet-tools/+bug/1249162 | 23:09 |
| ubot5 | Ubuntu bug 1249162 in android-tools (Ubuntu) "Devices lose adb connection after phablet-flash loop" [Undecided,New] | 23:09 |
| * asac wonders what a phablet-flash loop is :) | 23:10 | |
| asac | ok i think i see what it tries to describe | 23:11 |
| fginther | the phablet-flash loop is a way to reproduce the problem we see on our devices | 23:13 |
| asac | check my comment please | 23:13 |
| === fginther changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: cihelp | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: Some services are down (1SS move), network slowness | ||
| sergiusens | asac, fginther Saviq recovery was broken, that was the problem | 23:47 |
| sergiusens | I'm adding a safeguard to phablet-flash to avoid flashing if recovery is broken | 23:48 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
