[00:14] <rsalveti> will trigger a new image so I can have a working emulator
[03:25] <plars> something happened with the download taking way too long for the latest image it seems... I restarted them, so results should start to show up in a bit
[05:12] <Mirv> good morning
[05:18] <Mirv> robru: I'll test unity8 AP too..
[05:18] <Mirv> robru: I've gotten sometimes problematic results where another run(s) have been fluent
[05:31] <Mirv> cihelp daily-release-executor is reported offline
[05:31] <Mirv> so all builds halted as well
[05:33] <rsalveti> plars: there was a regression in the recovery image, which is fixed in the archive but needs a newer image
[05:33] <rsalveti> not sure if the regression would cause the device to fail in someway
[07:12] <didrocks> Mirv: hey! around already? :)
[07:14] <didrocks> cihelp: hey, all executors on q-jenkins are down, (so everything is blocked). Seems the reason why robru/ken couldnt publish Mir yesterday
[07:14] <vila> didrocks, Mirv: on it
[07:15] <didrocks> thanks
[07:15] <didrocks> vila: I guess the daily-release executor is the first one to investigate why it went down
[07:16] <vila> didrocks: yes, Mirv said that already, we all agree
[07:21] <Mirv> didrocks: yeah, as written on langing plan
[07:22] <Mirv> didrocks: robert also had some unity8 AP problem, but I couldn't reproduce it so it's probably his environment (like wrongly updated unity8 from daily-build, or click tests being run and messing up unity8 testing)
[07:22] <didrocks> Mirv: ah, the notes were not from yesterday
[07:22] <didrocks> Mirv: yeah, probably
[07:22] <didrocks> Mirv: I'm not sure to understand the unity-mir rebuild?
[07:23] <didrocks> if it wasn't rebuilt already, you can't test, right?
[07:23] <Mirv> didrocks: the mir 0.1.1 build-dep was not in, otherwise it's ok. same for u-s-c.
[07:23] <didrocks> Mirv: ah, it's just the bump build-dep MP
[07:23] <didrocks> Mirv: well, if we release without that one, I won't die TBH ;)
[07:23] <didrocks> as long as the things are rebuilt as expected
[07:24] <Mirv> didrocks: ok ;) yeah the libmirserver10 dependency is there anyway, as it was built after the mir build
[07:24] <didrocks> Mirv: ok, just don't stress on that, if you feel that we should skip the build, please do (if we are ready to publish)
[07:25] <didrocks> ogra_: asac: did one of you requested a new image?
[07:25] <didrocks> ogra_: asac: I see a new image and 2 android uploads in particular to the archive (the second to fix a regression it seems)
[07:25] <Mirv> didrocks: ok. we're ready to publish, aside from that I don't see a note if unity-system-compositor was smoke tested.
[07:25] <Mirv> so that needs to be done still
[07:25] <didrocks> ogra_: asac: but the image which was built was between the 2 android uploads, so the "recovery regression" is in the current proposed image?
[07:26] <didrocks> Mirv: ok, maybe you should kill in jenkins if there are some mir/platform/unity8 stacks planned to be rebuilt?
[07:26] <didrocks> so that one q-jenkins is repaired, you are not trapped by a rebuild :)
[07:26] <Mirv> didrocks: yes, I did that already
[07:27] <Mirv> (except for platform it seems, now did that too)
[07:27] <didrocks> thanks Mirv :)
[07:30] <didrocks> waow, it's snowing here
[07:31] <Mirv> didrocks: cool, it's just rain currently here :)
[07:32] <Mirv> I'd welcome snow already, it's been unusually warm (which means also dark)
[07:32] <didrocks> Mirv: well, I think it'll transform soon in rain as well :)
[07:32] <didrocks> ah ah, for some kind of "warm" ;)
[07:33] <didrocks> here, for us, it's freezing (like we have the temperature we do have generally in 3 weeks)
[07:33] <vila> daily-release-executor is back
[07:41] <didrocks> thanks vila
[08:04] <Mirv> ok got to the desktop with updated mir + u-s-c too
[08:04] <didrocks> Mirv: and still get graphics? ;)
[08:04] <Mirv> didrocks: even that :) not much multimonitor though
[08:04] <didrocks> Mirv: not a surprise, but not a bad surprise either :)
[08:05] <didrocks> sweetness!
[08:31] <Mirv> didrocks: the mir stack had already queued the prepare tasks before daily release executor broke earlier, so they got rebuilt. mir is a no-change rebuild and u-s-c got now the mir 0.1.1 b-d in. I'm rerunning unity8 AP still and after then asking for packaging acks to publish.
[08:31] <didrocks> Mirv: sounds good, just do minor dogfooding
[08:32] <didrocks> don't stress it too much in some words :)
[08:32] <didrocks> thanks!
[08:50] <Mirv> didrocks: ok then, http://q-jenkins.ubuntu-ci:8080/view/cu2d/view/Head/view/Mir/job/cu2d-mir-head-3.0publish/lastSuccessfulBuild/artifact/packaging_changes_mir_0.1.1+14.04.20131120-0ubuntu1.diff + http://q-jenkins.ubuntu-ci:8080/view/cu2d/view/Head/view/Mir/job/cu2d-mir-head-3.0publish/lastSuccessfulBuild/artifact/packaging_changes_unity-system-compositor_0.0.2+14.04.20131120-0ubuntu1.diff + http://q-jenkins.ubuntu-ci:8080/view/cu2d/view/Head/view
[08:50] <Mirv> so the actual u-s-c session moved to ubuntu-desktop-mir which I already noticed
[08:50] <didrocks> Mirv: sorry, my system is in autodestruction mode after removing the emulator, I can't use sudo, ls or any other tool like chromium, can you get someone to review it?
[08:50] <didrocks> I can't even open a webpage
[08:56] <Mirv> didrocks: ok, have a nice debugging session..
[08:57] <didrocks> well, nice with no "ls", I hope that my home dir is still intact…
[09:01] <asac> didrocks: from backlog:
[09:01] <asac> 09:59 < -bip> 20-11-2013 00:08:09 -!- alesage!~alesage@c-98-212-4-39.hsd1.il.comcast.net has quit  [Quit: Leaving]
[09:01] <asac> 09:59 < rsalveti> 00:15:21> will trigger a new image so I can have a working emulator
[09:01] <asac> :/
[09:02] <didrocks> asac: well, and no second image with "recovery fixed"
[09:02] <didrocks> right?
[09:02] <didrocks> or did you find anything in the backlog?
[09:02] <asac> didrocks: let me look more careful :)
[09:02] <didrocks> asac: let's discuss in a few, my install is broken by the emulator I guess
[09:03] <asac> 09:59 < plars> 03:25:24> something happened with the download taking way too long for the latest image it  seems... I restarted them, so results should start to show up in a bit
[09:03] <asac> 09:59 < rsalveti> 05:33:30> plars: there was a regression in the recovery image, which is fixed in the  archive but needs a newer image
[09:03] <asac> 09:59 < rsalveti> 05:33:42> not sure if the regression would cause the device to fail in someway
[09:03] <asac> didrocks: next line spoken here was you pinging Mirv
[09:03] <asac> so guess not
[09:03] <asac> though the need for an image was identified
[09:04] <asac> didrocks: so feels we want another image? whatelse happened in the meantime?
[09:05] <didrocks> asac: yeah, I guess we want one (but again, I can't even ls right now, let me fix my box)
[09:05] <didrocks> and post a warning if it's really due to the emulator replacing libc6 with the armhf version on your system
[09:05] <asac> didrocks: ok. guess just boot in a live session and fix it from there
[09:06] <asac> :/
[09:06] <didrocks> asac: I'm trying with busybox here
[09:06] <asac> or from initrd if you are brave
[09:06] <asac> yeah
[09:06] <didrocks> but dpkg from busybox doesn't show which arch is installed :p
[09:07] <asac> didrocks: guess check in /var/lib/dpkg on your own
[09:07] <didrocks> 2013-11-20 09:48:34 status half-configured libc-bin:amd64 2.17-93ubuntu4
[09:07] <didrocks> from removing the emulator
[09:09]  * asac wonders what the emulator does that can cause this
[09:09] <seb128> didrocks, weird, usually app ask you to type the "yes, I'm sure I want to do that" before removing libc-bin
[09:10] <didrocks> seb128: I think it didn't remove, it has overriden with the armhf version
[09:10] <didrocks> seb128: I clearly didn't get "remove libc6"
[09:10] <asac> half-configured probably means that things fell over during upgrade
[09:10] <didrocks> ok, busybox didn't work from my needs, let me try to find an usb stick and fix from there
[09:10] <didrocks> yeah
[09:10] <didrocks> I guess there are some divert magic for the emulator
[09:11] <asac> didrocks: where did you get the emulator from?
[09:11] <didrocks> and something when went on removing
[09:11] <asac> ack
[09:11] <didrocks> asac: followed the wiki page
[09:11] <didrocks> (but then, tried to remove the android-emulator package)
[09:12] <seb128> I hope other people don't try the emulator
[09:12] <didrocks> well, it's not 100% confirmed, but it was the only thing I was doing at this time
[09:12] <seb128> didrocks, do you want me to follow up on the list to "be careful, Didier bricked his system with that"?
[09:12] <asac> didrocks: you hvae a URL to the wiki?
[09:12] <didrocks> seb128: would be nice to have someone confirming (in a vm first)
[09:12] <didrocks> asac: well, I just have a weechat window here :p it was on the ML
[09:13] <asac> kk
[09:13] <didrocks> ok, see you (hopefully) in a few
[09:13] <seb128> asac, https://wiki.ubuntu.com/Touch/Emulator
[09:14] <didrocks> you should try on amd64 to reproduce, as jibel noted, it's probably a i386<->amd64 issue
[09:14] <didrocks> so seb128 is safe, you can try in production! :p
[09:14] <seb128> lol
[09:15] <asac> it refers to an android-emulator package
[09:15] <asac> but thats noot in the archive... which ppa is it?
[09:15] <jibel> asac, binaries are only available for i386
[09:15] <jibel> in the archive
[09:16] <seb128> asac, it's in multiverse for i386
[09:16] <seb128> https://launchpad.net/ubuntu/+source/android/20131120-0225-0ubuntu1
[09:16] <asac> oh its part of android
[09:16] <asac> yeah i can imagine that that might cause accidential havoc
[09:20] <seb128> on i386 it wants to install libc6-amd64
[09:20] <seb128> not sure I want to risk trying on my system :p
[09:22] <asac> so the package doesnt look harmful by itself
[09:23] <jibel> trying in an amd64 VM, it's installing 723MB of i386 packages
[09:23] <asac> seb128: it includes stuff like: ./usr/share/android/emulator/out/host/linux-x86/bin/emulator64-arm
[09:23] <asac> so it somewhat makes sense that it brings in amd64 stuff
[09:23] <didrocks> Commandline: apt-get install android-emulator
[09:24] <didrocks> Install: android-emulator:i386 (20131120-0225-0ubuntu1), lib64gcc1:i386 (4.8.2-1ubuntu2, automatic), lib64stdc++6:i386 (4.8.2-1ubuntu2, automatic), libc6-amd64:i386 (2.17-93ubuntu4, automatic)
[09:24] <didrocks> (that was what I had)
[09:24] <didrocks> and it worked after that, it's really when I removed the package and autoremove the deps
[09:24] <seb128> asac, it's a bit weird that it's an i386 package and that it pulls in amd64 binaries
[09:24] <jibel> didrocks, I reproduced your problem
[09:24] <asac> http://paste.ubuntu.com/6447226/
[09:24] <jibel> and broke the VM after removing the emulator
[09:24] <jibel> asac, ^
[09:25] <seb128> jibel, can you paste the log of what apt did when you removed the emulator?
[09:25] <asac> right. that would be good to know
[09:25] <jibel> seb128, sure
[09:25] <didrocks> jibel: ah, so not rebooting immediatly, would be interesting to get your VM fix rather than me being offline :)
[09:25] <jibel> didrocks, indeed, 1 minute
[09:26] <didrocks> (ok, just finished getting an usb stick ready with an old trusty image)
[09:26] <asac> so the emulator package has nothing in the postrm postinst scripts
[09:26] <asac> so the bustage must come genuinely from apt and libc6-amd64 removal
[09:26] <didrocks> right
[09:27] <asac> my guess is that if i just instlal libc6-amf64 and remove it will also be broken
[09:27] <jibel> seb128, http://paste.ubuntu.com/6447229/
[09:27] <asac> -> e.g. if thats the case thats a doko bug, no?
[09:27] <didrocks> at least, I'm looking at busybox dpkg and it doesn't have a lot of options
[09:27] <didrocks> asac: you're probably right, indeed
[09:28] <jibel> seb128, asac to reproduce on an amd64 VM, dpkg --add-architecture i386 && apt-get update && apt-get install -y android-emulator && apt-get autoremove --purge android-emulator
[09:28] <asac> libstdc++6:i386*
[09:28] <asac> odd.
[09:31] <didrocks> yeah, so it probably removed the libc6-amd64 files configured to be used against
[09:31] <didrocks> and then, can't run the postrm script to configure the new links?
[09:32] <asac> jibel: can you reproduce by just installing the libc-amd64 and purging that?
[09:32] <asac> (in case you have a VM image easily available)
[09:32] <jibel> asac, hold on, I'm trying to recover the system first to help didrocks save his machine
[09:32] <didrocks> I would appreciate, indeed :)
[09:32] <asac> sure
[09:33] <seb128> that package has a postrm
[09:33] <seb128> if [ "$1" = remove ]; then
[09:33] <seb128>     ARCH=${DPKG_MAINTSCRIPT_ARCH:-$(dpkg --print-architecture)}
[09:33] <seb128>     if [ "${ARCH}" = "amd64" ] && [ "libc6-amd64" = "libc6-i386" ]; then
[09:33] <seb128> 	if [ -h /lib/ld-linux.so.2 ] && [ ! -f /lib/ld-linux.so.2 ]; then
[09:33] <seb128> 	    rm /lib/ld-linux.so.2
[09:33] <asac> however, maybe first narrowing it down will help :)
[09:33] <asac> greedilyu removing ld-linux
[09:34] <didrocks> hum, I still have /lib/ld-linux.so.2 here
[09:34] <asac> seb128: that only happens on amd64, no?
[09:34] <asac> i think that might not be it
[09:34] <didrocks> but the postrm failed itself
[09:34] <didrocks> so it's not executed
[09:34] <didrocks> (as per log)
[09:34] <asac> all those postrm fail
[09:34] <didrocks> right
[09:34] <asac> seems stuff is already busted after prerm
[09:34] <didrocks> yep
[09:34] <asac> and after just removing the files
[09:34] <asac> int he packages
[09:35] <asac> maybe having the log from installing might show some diversions
[09:35] <asac> etc.
[09:35] <didrocks> I guess when removing libc6-amd64 files
[09:36] <ogra_> are you 100% sure the last android build made it into the image ?
[09:36] <jibel> asac, trying now
[09:36]  * ogra_ sees that publishing and image build are very close 
[09:37] <didrocks> ogra_: was it supposed to be reflected in your diff?
[09:37] <ogra_> no
[09:37] <ogra_> the android package isnt inside anywhere
[09:37] <ogra_> so the manifest generator cant pick it up
[09:37] <ogra_> Get:1 http://ftpmaster.internal/ubuntu/ trusty/multiverse android all 20131120-0225-0ubuntu1 [348 MB]
[09:37] <ogra_> Fetched 348 MB in 48s (7243 kB/s)
[09:38] <ogra_> ok, seems it made it
[09:38] <ogra_> (you can find it in the livefs build log)
[09:38] <ogra_> http://people.canonical.com/~ubuntu-archive/livefs-build-logs/trusty/ubuntu-touch/20131120.1/livecd-20131120.1-armhf.out
[09:38] <didrocks> ogra_: I guess you need a new image then
[09:39] <ogra_> ?
[09:39] <didrocks> ogra_: see the followup android upload at 5 UTC
[09:39] <ogra_> 20131120-0225-0ubuntu1 is the last upload
[09:39] <didrocks> ah, you got with that one?
[09:39] <ogra_> see above
[09:39] <didrocks> ogra_: can't click on links right now :p
[09:40] <didrocks> if you follow what was discussed, system hosed ^
[09:40] <jibel> asac, confirmed: apt-get install libc6-amd64 && apt-get remove libc6-amd64
[09:40] <jibel> and get a broken system for free
[09:40] <didrocks> I would have prefered to not have it at all than for free ;)
[09:41] <didrocks> thanks for digging jibel :)
[09:41] <ogra_> didrocks, yeah
[09:42] <asac> jibel: thats on i386?
[09:43] <jibel> asac, amd64 with i386 enabled
[09:43] <jibel> which is the same cofigiration than didrocks. I can try on i386 too
[09:43] <asac> kk
[09:43] <asac> dont worry. guess file a bug so we can have doko etc. look at it
[09:44] <jibel> k
[09:59] <Mirv> didrocks: I filed bug #1253008 about pitti's worry about u-s-c configuration file removal (or actually it's a move to another package)
[10:00] <didrocks> Mirv: thanks
[10:35] <ogra_> r26 looks ok on magureo
[10:35] <ogra_> (havent done any call or sms tests, but it boots fine)
[10:42] <popey> blimey, 101MB to go to 26
[10:42] <popey> did you add the kitchen sink to it?
[11:08] <didrocks> ogra_: mind updating the spreadsheet with latest infos? I'm a little bit lost as the android update is still to TODO
[11:08] <didrocks> so not sure if the upload was that one (it seems so though)
[11:12]  * ogra_ can look, but i wasnt involved in any of the last three android uploads
[11:12] <ogra_> (i didnt even know about them until i saw them on the changes ML)
[11:16] <didrocks> ogra_: I guess the planned image # needs to change as well
[11:26] <didrocks> ogra_: hum, ok, I think, I'll bump everyting to 27
[11:27] <didrocks> well, system settings is in 25 from what I see
[11:27] <ogra_> didrocks, oh, you wanted me to bump the others ?
[11:27] <ogra_> sorry, i understood i should sort the android line
[11:27] <didrocks> ogra_: yeah, as they are not exact, but no worry :)
[11:28] <ogra_> sorry
[11:29] <didrocks> jibel: lool: can one of you confirm https://bugs.launchpad.net/ubuntu/+source/ubuntu-download-manager/+bug/1240656? that will get it moved and then we can kick a saucy image build
[11:40] <psivaa> ogra_: didrocks: not sure if that's known, but with image 26, we see "failed to copy '/var/lib/jenkins/phablet-flash/imageupdates/pool/ubuntu-418162f2a9ba9a320a2b460938e3d333f84b6f35826200c2a6b6c342ba1e21ef.tar.xz' to '/cache/recovery//ubuntu-418162f2a9ba9a320a2b460938e3d333f84b6f35826200c2a6b6c342ba1e21ef.tar.xz': No space left on device" on mako
[11:40] <psivaa> have seen that in two devices
[11:41] <didrocks> interesting, as the broken recovery was installed in image 25? that can be it, right?
[11:41] <ogra_> psivaa, the recovery from r25 was broken ... could be that you still havethat in place
[11:41] <ogra_> i just did an OTA upgrade on maguro, that one worked fine
[11:42] <psivaa> i tried on two devices, not sure if both of them had attempted r25 as well, i'll try another device
[11:42] <ogra_> didrocks, you should probably not use IRC nicks (or make clear they are IRC nicks) in mails to ubuntu-phone :)
[11:42] <didrocks> ogra_: good point
[11:43] <ogra_> "infinity is looking into it" sounds a bit cryptic for non IRC users
[11:43] <ogra_> :)
[11:43] <ogra_> (like "eternity will solve it")
[11:44] <Mirv> go, builders, go
[11:45] <Mirv> waiting for amd64+i386 on https://launchpad.net/~ubuntu-unity/+archive/daily-build/+sourcepub/3671940/+listing-archive-extra ...
[11:45] <didrocks> ogra_: well, it can be true :)
[11:45] <ogra_> haha
[11:46]  * didrocks doesn't want to hear about amd64 and i386 in the same sentence today
[11:46] <Mirv> oh, how rude of me
[11:49] <ogra_> yippiiee
[11:49] <ogra_> root@ubuntu-phablet:/# lxc-console -nandroid -t0
[11:49] <ogra_> Connected to tty 0
[11:49] <ogra_> Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself
[11:49] <ogra_> root@android:/ #
[11:49] <ogra_> it works !
[11:51] <Saviq> cihelp, hey, unity8 otto is getting stuck due to some dependency issue http://s-jenkins.ubuntu-ci:8080/job/autopilot-testrunner-otto-trusty/817/console
[11:54] <Saviq> I *feel* like it might be caused by https://code.launchpad.net/~kgunn72/unity-mir/bump-mir-dep-0.1.1/+merge/193521
[11:55] <Saviq> if unity-mir built against mir > 0.1.1 somehow
[11:55] <Saviq> and is kept in the mbs repo
[11:55] <Saviq> but then there's no mir > 0.1.1 in the archive yet, so I'm not sure how that could happen
[11:55] <Saviq> Mirv, ideas ↑?
[11:55] <Saviq> how did that even get through autolanding?
[12:00] <didrocks> Saviq: the transition is in the daily-build ppa normally
[12:00] <Saviq> didrocks, right... but otto isn't using it
[12:01] <Saviq> grr
[12:01] <didrocks> Saviq: otto for integration tests is
[12:01] <Mirv> Saviq: aanyhow, once the amd64 PPA build of this https://launchpad.net/~ubuntu-unity/+archive/daily-build/+sourcepub/3671940/+listing-archive-extra finally starts, we're really close of having Mir 0.1.1 in archives
[12:01] <didrocks> maybe not for upstream merger? not sure
[12:03] <Mirv> my plan is 1. wait for amd64 build of that, 2. upgrade desktop, 3. reboot and login, 4. publish mir + platform-api + unity-mir + unity-system-compositor
[12:03] <Saviq> Mirv, ok
[12:04] <Saviq> we'll just have to wait then
[12:04] <Saviq> didrocks, you mean in cu2d it does?
[12:04] <didrocks> Saviq: yeah, when releasing, we are adding the ppa in the otto integration tests
[12:04] <Saviq> didrocks, in -ci -autolanding it doesn't, as I understood it using daily is unsafe 'cause you never know if what you've built against is really going to end up in distro
[12:06] <didrocks> Saviq: yeah, and we have issues when there are some transitions
[12:06] <didrocks> like today
[12:06] <didrocks> that's why we need CI Airline :p
[12:06] <Saviq> didrocks, ok, so nobody is doing anything wrong, just that our tools are not letting do us what we really need, gotcha
[12:06] <Saviq> we'll just wait, then
[12:06] <didrocks> Saviq: I guess yeah
[12:18] <psivaa> didrocks: ogra_: so the issue of no space in /cache/recovery/ on mako is not due to image r25 remnants, i flashed a device which had r23 and it still occurs
[12:19] <ogra_> weird
[12:19] <didrocks> psivaa: ok, if you changed the recovery as well and you're sure you don't have image 25 recovery, one less ;)
[12:22] <psivaa> on maguro this is not happening though
[12:31] <psivaa> didrocks: the only files in /cache/recovery/ are very small log files so they can not be the reasons.
[12:32] <psivaa> i mean before flashing with r26
[12:41] <ogra_> check one level up in /cache
[12:42] <ogra_> i think /cache is the partition, recovery/ is only a dir
[12:46] <psivaa> ogra_: 32k on /cache
[12:47] <ogra_> used you mean ?
[12:47] <psivaa> yes
[12:47] <psivaa> before flashing with r26
[12:48] <ogra_> thats not much ... and should really not cause out of space warnings
[12:49] <psivaa> right during the flashing on *maguro i see 1069 M in /cach
[12:49] <psivaa> *cache
[12:51] <psivaa> ogra_: not sure what's the capacity though on each devices
[12:51] <ogra_> http://phablet.ubuntu.com/gitweb?p=CyanogenMod/android_build.git;a=log;h=refs/heads/phablet-trusty
[12:51] <ogra_> i see nothing that even touches recovery ... strange
[12:52] <psivaa> but as you said it could be anything that goes into /cache
[12:52] <psivaa> now it's 1386M
[12:53] <ogra_> aha
[12:55] <ogra_> http://phablet.ubuntu.com/gitweb?p=CyanogenMod/android_system_core.git;a=blobdiff;f=init/init.c;h=e5f265858d0e0890deff18424ab09b174a4ce3f0;hp=462130711e331820467e2b8c427c1e2f7be9cdf2;hb=a1527ce86c90842b693941723888f7739f3bf473;hpb=7a318cec11f017c5ae9c186dae2b01f2dc4d3ef4
[12:55] <ogra_> psivaa, does /sbin/recovery exist ?
[12:56] <Mirv> sil2100: I hoped you wouldn't block the stacks, that's why I asked for publishing jobs only
[12:58] <psivaa> ogra_: the installation on maguro is finished and now /sbin/recovery is not there.. but i dont know if it would have existed during the flash
[12:59] <ogra_> psivaa, well, it should be existent in the recovery mode only
[13:00] <ogra_> oh, meeting ...
[13:00]  * ogra_ totally forgot ... gimme a sec to relocate
[13:02] <didrocks> sil2100: coming?
[13:02] <jdstrand> hey-- I'm looking at landing no 303 and wondering if now is the time to upload
[13:03] <jdstrand> (it is marked TODO)
[13:03] <sil2100> didrocks: I'll be a bit late, start off without meee
[13:05] <sil2100> Mirv: you told me that too late I guess, I just redeployed unity8 though
[13:05] <Mirv> sil2100: you didn't wait for answer :) you also started a build, so it's now waiting for the scope build
[13:08] <sil2100> Mirv: ok, so I'm not building/doing anything until I get a greenish light from you now ;p
[13:08] <sil2100> Mirv: since I would need to rebuild ubuntu-settings-components as well...
[13:09] <sil2100> But I'll do that after you're done ;)
[13:09] <Mirv> sil2100: ok :) let's wait for the current build to finish, then I'll publish, then I'll let you a sign that you can publish / do what you want
[13:10] <sil2100> ;)
[13:12] <Mirv> sil2100: when the build job is published, isn't it in theory ok if I cancel the check job since you need to rebuil more anyway?
[13:12] <Mirv> s/published/finished/
[13:13] <sil2100> Mirv: you can cancel everything besides build ;p
[13:13] <sil2100> Mirv: since it's not being tested anyway anywhere
[13:13] <Mirv> sil2100: yeah. what I think I'll do that I'll wait for build to finish, then cancel check, then do my publishing and then I can let it run check again via foo
[13:15] <sil2100> Mirv: ok, but I think that for my needs a check re-run won't be needed anyway
[13:15] <sil2100> Since for me it's enough that it builds, so I can just do a direct force publish after your publishing is done
[13:17] <Mirv> right
[13:17] <ogra_> rsalveti, could the above git commit have any impact on that "/cache running out of space" issue when flashing ?
[13:18] <Mirv> sil2100: and actually the check didn't run since unity8 build happens to fail
[13:18] <sil2100> Oh
[13:18] <Mirv> we should look into that at some point too, and file a bug as usual, there just hasn't been time for that
[13:20] <Mirv> and we probably need to remind the other team mates too that the usual cu2d vanguarding starts again now that it's up
[13:21] <didrocks> Mirv: quite, publish mir then!
[13:21] <didrocks> quick*
[13:21] <didrocks> as the stack stopped :p
[13:22] <Mirv> didrocks: already ... done!
[13:22] <didrocks> sooo fast, it's not an airline, it's a jet \o/
[13:22] <Mirv> sil2100: so I'll wait still to see with my own eyes that the packages are in LP, and ping you after that
[13:24] <ogra_> rsalveti, oooh, i think i see whats wrong with your patch ... you actually want the mkdir for /proc and /sys outside of the "if" in http://phablet.ubuntu.com/gitweb?p=CyanogenMod/android_system_core.git;a=blobdiff;f=init/init.c;h=e5f265858d0e0890deff18424ab09b174a4ce3f0;hp=462130711e331820467e2b8c427c1e2f7be9cdf2;hb=a1527ce86c90842b693941723888f7739f3bf473;hpb=7a318cec11f017c5ae9c186dae2b01f2dc4d3ef4
[13:28] <ogra_> psivaa, could you boot one of the failing devices into recovery and check that ?
[13:29] <psivaa> ogra_: the devices are in the lab and the failing devices are not reachable by adb.
[13:29] <ogra_> hmm
[13:29] <psivaa> i dont know any other way to connect them.
[13:29] <ogra_> so we need someone in the lab to check them ... ok
[13:30] <psivaa> ogra_: yea waiting till rfowler comes online
[13:36] <Mirv> sil2100: so "ping", feel free
[13:45] <rsalveti> ogra_: that patch is fine
[13:45] <rsalveti> ogra_: the problem was the previous one that got us into a broken recovery
[13:45] <rsalveti> ogra_: http://phablet.ubuntu.com/gitweb?p=CyanogenMod/android_system_core.git;a=commitdiff;h=aefb9653a367a6fd528f2ad48a1cce8945aecb3c;hp=87b40ee4a93bd682506fa082f2c2d1a0f5a60f04
[13:46] <ogra_> rsalveti, well, that still  doesnt create /proc or /sys
[13:46] <ogra_> it tries to mount them to a nonexisting mountpoint
[13:46] <rsalveti> didrocks: I did trigger another one with the fix included in it
[13:46] <didrocks> rsalveti: hum, it seems to have never gotten anywhere (we only had one new image)
[13:46] <ogra_> rsalveti, though i still dont have any explanation for the out of space issue
[13:46] <didrocks> or ogra's script is broken :p
[13:46] <ogra_> didrocks, ?
[13:47] <rsalveti> ogra_: that is ok in our default image, because they are created by lxc
[13:47] <ogra_> there are 20 and 20.1
[13:47] <ogra_> 20 is the broken one
[13:47] <rsalveti> yes
[13:47] <rsalveti> 20.1 is fine
[13:47] <didrocks> ogra_: but you did build 20.1, no?
[13:47] <ogra_> rsalveti, in recovery ?
[13:47] <rsalveti> ogra_: in recovery it'll be created
[13:47] <ogra_> oh, i see what you are saying
[13:47] <ogra_> yeah
[13:47] <rsalveti> ogra_: +    if (stat("/sbin/recovery", &s) == 0) {
[13:47] <ogra_> well, still, all makos run our of space when flaching
[13:48] <rsalveti> that's the fix I did yesterday when I noticed the regression
[13:48] <rsalveti> how?
[13:48] <didrocks> 12:40:27   psivaa | ogra_: didrocks: not sure if that's known, but with image 26, we see "failed to copy
[13:48] <didrocks>                   | '/var/lib/jenkins/phablet-flash/imageupdates/pool/ubuntu-418162f2a9ba9a320a2b460938e3d333f84b6f35826200c2a6b6c342ba1e21ef.tar.xz' to
[13:48] <didrocks>                   | '/cache/recovery//ubuntu-418162f2a9ba9a320a2b460938e3d333f84b6f35826200c2a6b6c342ba1e21ef.tar.xz': No space left on device" on mako
[13:49] <didrocks> rsalveti: ^
[13:49] <ogra_> rsalveti, <psivaa> ogra_: didrocks: not sure if that's known, but with image 26, we see "failed to copy '/var/lib/jenkins/phablet-flash/imageupdates/pool/ubuntu-418162f2a9ba9a320a2b460938e3d333f84b6f35826200c2a6b6c342ba1e21ef.tar.xz' to '/cache/recovery//ubuntu-418162f2a9ba9a320a2b460938e3d333f84b6f35826200c2a6b6c342ba1e21ef.tar.xz': No space left on device" on mako
[13:50] <rsalveti> didrocks: why did you remove the emulator?
[13:50] <rsalveti> :P
[13:50] <rsalveti> ogra_: right, but is 26 = 20.1?
[13:51] <ogra_> rsalveti, he did an apt-get autremove ...
[13:51] <ogra_> rsalveti, it is
[13:51] <didrocks> rsalveti: yeah, I know you wanted to punish people doing that! :p
[13:51] <didrocks> I'll note this for next time "never ever remove what rsalveti installed on your system" :)
[13:51] <rsalveti> ogra_: right, not sure if that is indeed using the cache partition, need to give it a try here
[13:51] <ogra_> rsalveti, r26 works fine on my maguro with OTA upgrade
[13:51] <rsalveti> ogra_: I know the cdimage one works fine
[13:51] <rsalveti> just flashed it
[13:53] <rsalveti> ogra_: are we flashing the lab devices with -b?
[13:53] <ogra_> could be, not sure
[13:53] <ogra_> doanac` should know
[13:53] <ogra_> or sergio
[13:54] <psivaa> the devices in the lab use - phablet-flash ubuntu-system -b --channel trusty-proposed
[13:55] <plars> psivaa: I just got to my desk, looks like there's some trouble this morning?
[13:56] <plars> rsalveti, ogra_: we are using -b for quite some time now
[13:56] <psivaa> plars: yes, a couple but the latest impacting one is that flashing on mako fail due to no space available error on /cache/recovery
[13:56] <rsalveti> let me flash my mako to see
[13:57] <rsalveti> the cdimage one worked fine
[13:57] <Mirv> Mir 0.1.1 now in release pocket
[13:57] <Mirv> and time for UDS
[14:04] <plars> psivaa: we seem to have a lot of devices showing up with the 0123456789ABCDEF adb id again now
[14:05] <psivaa> plars: yes, vila mentioned about it. dont seem to be an id that's used for smoke
[14:05] <plars> psivaa: it's a generic one that's used when it can't sort out the real one
[14:06] <plars> psivaa: we'll probably need rfowler to reflash them
[14:06] <psivaa> plars: ahh ok, could be that we have a bunch of mako's that failed to flash due to the no space reason and they could be it
[14:07] <psivaa> plars: on another issue, we had adb protocol fault in kinnara in the morning
[14:07] <plars> psivaa: just one? :)
[14:07] <plars> psivaa: we do get it a lot less frequently I think, but it still happens from time to time
[14:08] <psivaa> even running 'adb devices' from kinnara constantly threw this error
[14:08] <psivaa> plars: how do you normally recover from it
[14:08] <plars> psivaa: oh, that's different
[14:09] <plars> psivaa: in a case like that, I guess we'd just need to restart adb server... I'm not sure I've seen that happen before
[14:09] <plars> psivaa: we added quite a few more devices to that system though
[14:10] <ogra_> plars, psivaa, this will be fixed within the next weeks
[14:10] <psivaa> plars: yea, it turned out to be one of the old adb processes was causing the issue
[14:10] <rsalveti> plars: ogra_: mako worked fine here with system-image
[14:10] <ogra_> very weird
[14:10] <psivaa> plars: killing that process fixed it
[14:10] <plars> rsalveti: you flashed with -b also?
[14:11] <psivaa> Installed: 1.0+14.04.20131108-0ubuntu1 is the phablet tools in the lab
[14:11] <rsalveti> plars: yes
[14:11] <ogra_> i dont think it has anything to do with phablet-tools, after all the out of space message seems to come from the device itself
[14:12] <rsalveti> plars: http://paste.ubuntu.com/6448304/
[14:12] <plars> brb
[14:14] <psivaa> ogra_: ok
[14:22] <rsalveti> ogra_: so I think the cache related issue could be a side effect from the broken image
[14:23] <ogra_> rsalveti, thats what i thought too, but -b pushes a new recovery
[14:23] <ogra_> so it should boot the fixed one
[14:24] <ogra_> (it actually boots the new recovery via fastboot)
[14:25] <rsalveti> ogra_: right, but that doesn't necessarily mean that it'll clean up the cache partition
[14:26] <rsalveti> INFO:phablet-flash:Clearing /data and /cache
[14:26] <ogra_> h,mm, i thought we do that from ubuntu_commands or how thats called
[14:26] <rsalveti> but it should in theory clean it as part of the flashing process
[14:26] <plars> so was the bad recovery actually on image 25 then?
[14:26] <ogra_> right
[14:26] <ogra_> plars, yes
[14:28] <psivaa> plars: ogra_ i tried flashing from image 23 as well
[14:28] <psivaa> and the same thing happend
[14:29] <ogra_> psivaa, if you use -b it weill use the new recovery ... and if you were unlucky you had r25
[14:30] <ogra_> though looking at the above it smells more like the cleanup step failed
[14:30] <plars> indeed
[14:30] <ogra_> but you should have seen that in the log
[14:31] <plars> ogra_: I don't see anywhere in phablet-tools where fastboot erase is called, could be that we need to really erase those partitions
[14:31] <ogra_> i dont think it calls erase
[14:32] <rsalveti> plars: the clean is a shell call done in recovery
[14:32] <ogra_> yeah
[14:32] <rsalveti> not via fastboot
[14:32] <rsalveti> so that might be failing
[14:32] <plars> rsalveti: yeah, it just does rm
[14:48] <rsalveti> plars: you might want to force a clean with fastboot then and retry to see
[14:49] <plars> rsalveti: yeah, I'm trying to reproduce locally. Flashing 25 right now.
[14:55] <sergiusens> fastboot -w created too many support issues
[14:55] <ogra_> yeah
[14:56] <ogra_> well, i can imagine that if there was no /dev and no /proc in r25's recovery that the rm might indeed have failed ... i just dont get why it would also fail in r26
[15:22] <rsalveti> plars: were you able to reproduce the issue locally?
[15:23] <plars> rsalveti: yes, but psivaa said he's seeing the issue going from 23->26 also
[15:23] <rsalveti> oh =\
[15:23] <plars> rsalveti: I was also able to get it to flash just fine from 25->26 after it failed once... it was in recovery and not getting the bad adbid like we see in the lab
[15:23] <plars> rsalveti: and I just added -d mako
[15:23] <plars> rsalveti: it all just worked... so I'm trying to figure out what's going on
[15:55] <plars> psivaa, rsalveti: so I just watched it get the same error when trying to go from 26->25
[15:55] <plars> wait, maybe not
[15:55] <plars> this one might have just been an adb failure
[15:56] <plars> adb/mtp thing probably
[15:56] <rsalveti> right
[15:57] <rsalveti> I think you might be able to extract the logs from the broken flags if you're able to boot the device
[15:57] <rsalveti> check /cache/recovery/last_log
[16:22] <plars> rsalveti: ok, after coming back up, all I have is busybox and there's no fstab or /cache. If I mount mmcblk0p22 by hand though, I found what seems to be a cache partition there, and here's the last_log: http://paste.ubuntu.com/6448925/
[16:22] <plars> rsalveti: this was after installing 25, and then trying to install 26
[16:23] <plars> oddly: 'Skipping missing file: version-25.tar.xz' was in the log
[16:25] <rsalveti> plars: hm, right, but the log is from 25 I believe
[16:25] <rsalveti> weird
[16:26] <plars> rsalveti: yeah, so maybe we didn't get far enough for that log to happen
[16:26] <ogra_> if you are in busybox you are not in recovery
[16:26] <ogra_> so indeed there is no /cache
[16:26] <rsalveti> yeah
[16:26] <plars> since it happened on the adb push, that makes sense
[16:27] <rsalveti> plars: let me try to flash 25 to see
[16:40] <rsalveti> plars: got the issue here
[16:40] <rsalveti> plars: so, what happens, when flashing 25 with -b it will first flash the recovery image with fastboot
[16:40] <rsalveti> plars: then it'll try to load recovery, which will fail
[16:41] <rsalveti> plars: meanwhile it waits for adb, thinking that the recovery will be up
[16:41] <rsalveti> as recovery fails to boot, the device boots the previous image instead
[16:41] <rsalveti> and phablet-tools thinks that the recovery is then up, because adb is there
[16:41] <plars> rsalveti: ah, I see. so we think we're booting 25 but really we're not
[16:42] <plars> rsalveti: I see
[16:42] <rsalveti> http://paste.ubuntu.com/6449025/
[16:42] <rsalveti> sergiusens: we should be checking for /sbin/recovery when flashing
[16:42] <rsalveti> sergiusens: to check if we're indeed in recovery
[16:43] <sergiusens> rsalveti, sure, I'm already checking for something; checking for recovery specifically shouldn't be complicated
[16:43] <sergiusens> rsalveti, my wonder is, what did it boot?
[16:44] <sergiusens> plars, any indications of why it failed to boot?
[16:44] <rsalveti> sergiusens: it fails to load the recovery, then it tries to boot the previous image
[16:44] <rsalveti> but it seems that the 25->26 migration worked fine
[16:44] <sergiusens> rsalveti, yeah, but why? :-)
[16:45] <rsalveti> so the error when copying the file into /cache happens because it's not really in the recovery
[16:45] <plars> rsalveti: really? for me, I get a broken system - just boots to busybox
[16:45] <rsalveti> sergiusens: that's the default behavior
[16:45] <sergiusens> rsalveti, how could've recovery been broken :-)
[16:45] <rsalveti> sergiusens: we had a regression
[16:45] <sergiusens> rsalveti, that's the how I'm asking :-)
[16:45] <sergiusens> ack
[16:45] <plars> rsalveti: you are flashing cdimage? or ubuntu-system image?
[16:45] <sergiusens> plars, rsalveti super easy to have a better check if we are in recovery, no worries
[16:46] <rsalveti> sergiusens: http://phablet.ubuntu.com/gitweb?p=CyanogenMod/android_system_core.git;a=commitdiff;h=aefb9653a367a6fd528f2ad48a1cce8945aecb3c;hp=87b40ee4a93bd682506fa082f2c2d1a0f5a60f04
[16:46] <rsalveti> which was fixed at http://phablet.ubuntu.com/gitweb?p=CyanogenMod/android_system_core.git;a=commitdiff;h=a1527ce86c90842b693941723888f7739f3bf473;hp=7a318cec11f017c5ae9c186dae2b01f2dc4d3ef4
[16:46] <rsalveti> plars: ubuntu-system
[16:46] <didrocks> ogra_: can we get a new image build?
[16:46] <didrocks> ogra_: now that Mir entered :)
[16:46] <plars> strange... for me, I'm getting exactly what we have seen on multiple devices in the lab
[16:46] <rsalveti> didrocks: sure :-)
[16:47] <didrocks> rsalveti: thanks!
[16:47] <ogra_> didrocks, yeah
[16:47] <sergiusens> why does everyone get to push but me? :-P
[16:47] <didrocks> it's christmas
[16:47] <ogra_> rsalveti, do you already ?
[16:47] <didrocks> I saw some snow this morning ;)
[16:47] <rsalveti> didrocks: having newer images is always a good thing
[16:47] <rsalveti> didrocks: easier to test and revert broken changes as well
[16:47] <ogra_> we should just build one every hour !
[16:47] <rsalveti> ogra_: nops
[16:47] <rsalveti> ogra_: yeah
[16:47] <ogra_> then i'll do
[16:47] <sergiusens> rsalveti, ogra_ you will starve the testing infra though
[16:47] <didrocks> rsalveti: fully agreed, we just need to have a way to communicate those to you and us more easily
[16:48] <rsalveti> didrocks: why do we need the communication here?
[16:48] <ogra_> sergiusens, nah, we'll have our fast emulator and all will be fine
[16:48] <rsalveti> we don't
[16:48] <rsalveti> the communication is the changelog :-)
[16:48] <sergiusens> ogra_, big LOL
[16:48] <ogra_> :)
[16:48] <didrocks> rsalveti: well, we do plan landings on certain image/try to communicate that
[16:48] <rsalveti> didrocks: and that's wrong
[16:48] <sergiusens> ogra_, the emulator would be mostly useful for apps, but not image testing ;-)
[16:48] <rsalveti> didrocks: you should only know when something lands, not planning when it'll land
[16:48] <ogra_> [16:49] <didrocks> rsalveti: will be easier once we have the CI Airline I hope to know more on feature-based what landed
[16:49] <sergiusens> ogra_, we can certainly only run on real hw if the emulator test pass
[16:49] <ogra_> sergiusens, well, i thought the plan was to use it in all testing infra
[16:49] <rsalveti> we don't need to know when it'll be landing
[16:49] <rsalveti> we just need to know when it *landed*
[16:49] <didrocks> well, we want to know when big thangs are landing at the same time
[16:49] <didrocks> to avoid clashes
[16:49] <didrocks> but apart from that, yeah
[16:49] <ogra_> right, only have a test on real HW if all emu tests were good
[16:50] <ogra_> didrocks, and that ifno comes from the changelogs
[16:50] <rsalveti> right, but even if we get clashes, we want more images to test all the steps
[16:50] <didrocks> ogra_: note on a "feature-approach", and not planning clashes
[16:51] <ogra_> rsalveti, the point is that asac wants us to have stuff tested in full context before we upload
[16:51] <ogra_> nobody does that
[16:51] <ogra_> (or do you run the full AP suite for an android change you do)
[16:51] <rsalveti> ogra_: right, but that's not a blocker here
[16:51] <rsalveti> having more images is *always* a good thing
[16:51] <ogra_> thats true
[16:51] <ogra_> but sergiusens has a point
[16:51] <ogra_> we'd saturate the lab
[16:52] <rsalveti> right, that's our only limitation
[16:52] <rsalveti> but we can do as most images we can depending on how long it takes to validate an image
[16:52] <rsalveti> what would that be?
[16:52] <rsalveti> at every 4 hours?
[16:52] <asac> rsalveti: in theory we can produce images in parallel and validate them
[16:52] <asac> the problem is not parallelizing the validation
[16:53] <asac> but the building
[16:53] <asac> its not really clear what atomic snapshot you will take when you start the job i think :)
[16:53] <ogra_> asac, we only talk about the actual images
[16:53] <ogra_> not about any parallel test builds etc
[16:53] <asac> (though you could mitigate that by pulling the packages file out of the tool and passing itas an input)
[16:53] <ogra_> we currenntly only build two per day at most
[16:54] <asac> right
[16:54] <rsalveti> and need to speed this up
[16:54] <asac> so for now we talk about the future anyway because emulator is not there etc.
[16:54] <ogra_> having a more fine grained view in the changelogs through building more of them will make identifying issues a lot easier
[16:54] <rsalveti> right, but I think we can already improve by building at least a few images a day
[16:54] <asac> however, if the emulator is fully landed, i anticipate that we can produce as manages as we want a day.
[16:54] <ogra_> from a builder POV we could build one every hour
[16:54] <rsalveti> as we always get stuff that lands in the archive which is not  part of the landing spreadsheet
[16:54] <asac> just not test all on all phones
[16:54] <rsalveti> so it's *always* good to produce and test newer images
[16:55] <ogra_> 30min for cdimage + 15min for system-image post processing
[16:55] <asac> we would basically validate all iamges in emlator and then pick those that we are interested in (for instance, because we consider them a release image) and test them  everywhere
[16:55] <asac> we discussed hooking image productionm up with the publisher runs
[16:55] <asac> so we get one image somewhere stored for each injection of what really goes in the archive
[16:56] <asac> has to wait for cloud buildd which is worked on somewhere
[16:56]  * cjwatson forces android-emulator back in despite p-m not quite understanding multiarch
[16:56] <asac> p-m?
[16:56] <cjwatson> proposed-migration
[16:56] <asac> ah
[16:57] <cjwatson> asac: https://rt.admin.canonical.com/Ticket/Display.html?id=62272 and https://bugs.launchpad.net/bugs/1247461
[16:58] <ogra_> right, that will give us parallel builds
[16:58] <josepht> asac: once an image + new packages is promoted don't all previous images need to be updated and retested?
[16:59] <asac> not really. all we do is snapshotting the archive after a publisher run in the form of an image
[16:59] <ogra_> why would they ?
[16:59] <asac> so whatever is in there is already finally committed
[16:59] <asac> until overwritten for which a new image would exist
[17:01] <josepht> we create an image for archive+A and archive+B in parallel, archive+A passes all tests so now archive = archive+A, if archive+B passes all tests we don't know that archive+A+B will pass all tests and need to test that scenario, right?
[17:02] <rsalveti> asac: right, but let's try to improve our current situation
[17:02] <asac> josepht: yeah you are right, but you are thinking about a different level
[17:02] <rsalveti> how long does it take to test an image currently?
[17:02] <rsalveti> plars: ^
[17:02] <asac> josepht: you are thinking about the premerge level... there indeed we need to invalidate A after we land B
[17:02] <plars> rsalveti: somewhere around 3 hours I believe
[17:02] <asac> hwoever, we can try to be smart and only invalidate if the changes are in the same dependency tree for instance
[17:02] <plars> rsalveti: assuming everything goes right :)
[17:02] <didrocks> plars: but then, you rekick jobs, right?
[17:02] <plars> didrocks: yes
[17:03] <asac> josepht: we talk about images produced after the fact from whatever landed in the real archive
[17:03] <didrocks> so how long until we get a clean state, I guess
[17:03] <didrocks> and you rekicked enough? :)
[17:03] <plars> didrocks: if I'm up and see it needs to be done, and there's not another image already coming
[17:03] <josepht> asac: ah I see
[17:03] <rsalveti> right, can we build a new image at every 6 hours then?
[17:03]  * asac hope that makes sense
[17:03] <rsalveti> or 8?
[17:03] <plars> didrocks: for instance, if the devices could work right now, I wouldn't bother retesting because I see there's a new image on the way, and it would just delay
[17:03] <asac> rsalveti: well. so i dont want random images
[17:03] <asac> i want precisely cut images
[17:03] <rsalveti> asac: why not?
[17:04] <rsalveti> asac: that's *wrong*
[17:04] <asac> its not :)
[17:04] <asac> not sure how that can be wrong :)
[17:04] <rsalveti> asac: we want one image per change
[17:04] <rsalveti> or the most images we can
[17:04] <asac> right.
[17:04] <rsalveti> otherwise it's a pita to find regressions
[17:04] <rsalveti> one image a day is *wrong* and only causes pain
[17:04] <asac> one image per chagne... but now that we can't get that (for various reasons) short term, it doesnt mean we should give up
[17:04] <cjwatson> an image isn't any more precisely cut just because somebody asked for it manually
[17:04] <asac> and cut images in the middle of changes :)
[17:05] <cjwatson> there shouldn't be a "middle of changes"
[17:05] <rsalveti> asac: give me one example
[17:05] <asac> rsalveti: i am supportive of more images per day. just not at cronjob kicked
[17:05] <rsalveti> exactly
[17:05] <asac> cjwatson: depends on which level you define changes :)
[17:05] <rsalveti> once it's in the archive, there's no "middle of changes"
[17:05] <cjwatson> get your packaging metadata right so that proposed-migration can migrate it properly
[17:05] <rsalveti> exactly
[17:05] <cjwatson> and then roll
[17:05] <rsalveti> we should put it in cron
[17:06] <cjwatson> manual image building requests are a workaround for lazy metadata maintenance
[17:06] <rsalveti> image is just an archive snapshot
[17:06] <asac> "get your packaging metadata right" ... please give up on that hope. all you can hope for is "get youir packaging meta data right most of the times please"
[17:06] <asac> but even if everyone does it right most of the times, if the engineering team is big enough you will always have problems :)
[17:06] <cjwatson> I'm not going to give up on that hope, because working on that basis made, empirically, a *massive* difference to the day-to-day quality of Ubuntu.
[17:06] <ogra_> cjwatson, *my* manually built images are *allways* more precisely cut, i sand and olish them :P
[17:06] <ogra_> *polish
[17:07] <cjwatson> In fact I think it's mostly reality rather than hope.
[17:07] <rsalveti> as long we're not saturating our test lab, we should be fine to produce the most images we can
[17:07] <asac> exactly... its mostly
[17:07] <asac> :)
[17:07] <cjwatson> Yes, there's the odd exception, but they're, well, exceptional
[17:07] <cjwatson> We shouldn't structure processes around exceptions
[17:07] <rsalveti> +1
[17:07] <asac> and will alwayus be mostly... but 600* mostly might mean rarely :)
[17:07] <ogra_> the point of automated builds is to simply have an as small changeset as possible ...
[17:08] <ogra_> doing only two manual builds completely goes against that
[17:08] <ogra_> and causes lots and lots of guesswork on our side until we found the actual issue
[17:08] <rsalveti> yeah
[17:08] <asac> its human. noone is perfect. hence designing the process must work in continuous exceptions due to human failures induced
[17:08] <rsalveti> horrible
[17:08] <rsalveti> see the udev one we had a few weeks ago
[17:08] <ogra_> which wastes more developer time than having more images per day
[17:08] <rsalveti> the package was uploaded at wednesday and we found the issue at saturday
[17:08] <ogra_> asac, its wasting human resources
[17:09] <asac> ogra_: the problem can be solved by using a cronjob or by kicking builds again at a 3 times a day schedule
[17:09] <asac> with shbuman smartness
[17:09] <ogra_> asac, how often did we get onto the wrong path in the recent times if an image was broken
[17:09] <ogra_> wasting hours to find it was the other change that we didnt look at yet
[17:09] <rsalveti> right, even if we build it at every 12 hours, it's already better than what we have now
[17:09] <ogra_> rsalveti, we do
[17:09] <asac> i agree that we should revisit the image schedule :)
[17:09] <rsalveti> ogra_: not currently
[17:10] <ogra_> we usually have two manual built ones per day
[17:10] <rsalveti> well, it's manual
[17:10] <asac> not sure if crontab is the right approach. lets talk about it tomorrow on the standup
[17:10] <ogra_> (if there isnt UDS)
[17:10] <asac> i would love to see at least 3 images :)
[17:10] <ogra_> so the timing is already 12h
[17:10] <rsalveti> ogra_: right, I want it to be automatically triggered
[17:10] <rsalveti> ogra_: as the archive is always moving
[17:10] <ogra_> right
[17:10] <ogra_> 4 per day should be fine
[17:10] <ogra_> automated and all
[17:10] <cjwatson> I'm completely boggled at the insistence that things must *not* be automated, frankly
[17:10] <rsalveti> can we try to put it back to the cron and see how it goes?
[17:11] <cjwatson> It goes against all the engineering practice I know
[17:11] <rsalveti> haha, exactly
[17:11] <ogra_> cjwatson, all but asac and didrocks in here say we want automated
[17:11] <didrocks> ogra_: I didn't say we don't want automation, I said that we want to cut after each big change as a first approach
[17:11] <cjwatson> I understand "automation is hard and we can't quite manage it yet" (have said that myself on various occasions) - I just don't understand "the ideal is manual"
[17:11] <ogra_> right, but why ?
[17:12] <asac> for the record: noone says "we dont want automation" :)
[17:12] <ogra_> didrocks, you land all your stuff in one chunk anyway
[17:12] <asac> lets discuss it tomorrow
[17:12] <asac> i will join you guys there
[17:12] <ogra_> where ?
[17:12] <cjwatson> OK, so you're arguing against regularity but not against automation, I see
[17:12] <asac> standup
[17:12] <cjwatson> I disagree but at least that isn't so boggling :-)
[17:12] <asac> hehe
[17:12] <asac> its also not exactly what i am saying :)
[17:12] <ogra_> asac, ah, the landing team one
[17:13] <asac> but closer
[17:14] <rsalveti> asac: why tomorrow?
[17:14] <rsalveti> asac: let's just put it back to cron today and see how it goes
[17:14] <didrocks> ogra_: if you land all the stuff in one chunk, well, you have to debug it, you decided the size of the chunk :)
[17:14] <rsalveti> why would that cause us any pain now
[17:14] <asac> i want to first ensure people understand what we are saying :)
[17:14] <didrocks> ogra_: we need to be able to produce images after each "chunk" landing, but a chunk is defined by the developer itself
[17:14] <asac> I dont see us having any pain from having manual image building
[17:14] <asac> it gives us many good
[17:15] <rsalveti> you can still manually trigger new images
[17:15] <rsalveti> but we should *always* have it in cron
[17:15] <ogra_> didrocks, thats not true ... your chunks are controlled in the spreadsheet
[17:15] <rsalveti> because the archive is *always* moving
[17:15] <ogra_> not defined by a developer
[17:15] <didrocks> ogra_: I just want to avoid A + B entered, "A says I don't care, it's B", it's not me, and B says "I don't care, it's because of A"
[17:15] <asac> so ... i know for sure that we will spin more than enough images once the landing stuff is fully back operational again
[17:15] <rsalveti> all I'm asking is to add it back in cron at every 8/12 hours
[17:15] <didrocks> ogra_: well, not really in our current model, but yeah, we would need to define priorities rather than image #, agreed
[17:15] <asac> hence, i believe the issue you try to solve is the low velocity due to the 1ss move paired with a UDS going on now :)
[17:16] <didrocks> ogra_: and I think we do align with what rsalveti told seeing it that way
[17:16] <asac> and yes, we should build at least 2 images a day even if not much is going on
[17:16] <ogra_> didrocks, so we can switch cron back on and set it to say 6h ?
[17:16] <rsalveti> I just want to make sure we always have a fallback in case we don't have humans to trigger a new image
[17:16] <ogra_> (to have 4 images / day)
[17:17] <asac> well. as i said befeore the capacity on testing is really bound to emulator :)
[17:17] <didrocks> ogra_: well, for me, it would be a +1 but rather 8 hours TBH, to let the CI team to rekick the tests jobs and giving some slack for that). 3 images a day will be a start
[17:17] <ogra_> asac, we should have as many as the infra allows imho
[17:17] <asac> so ... dont hurry
[17:17] <rsalveti> ogra_: yeah
[17:17] <didrocks> ogra_: but I don't think I'm the one able to take the decision :)
[17:17] <rsalveti> ogra_: I'd first try with 8h
[17:17] <ogra_> didrocks, why do you care if it is 4 or 8 ?
[17:18] <rsalveti> ogra_: and we can improve that as we go
[17:18] <didrocks> ogra_: because then, the testing infra can't rekick the jobs
[17:18] <ogra_> if your team only lands something every second image thats fine
[17:18] <asac> i care because i want to ensur3e the CI folks that support those images whenthey hit validation
[17:18] <didrocks> if a new image is produced
[17:18] <asac> know what to do and how to prioritize
[17:18] <didrocks> ogra_: they can't "restest an older image"
[17:18] <ogra_> asac, but the CI team knows when cron runs
[17:18] <asac> anyway
[17:18] <didrocks> ogra_: it's not only that, most of their time is just "relaunch the jobs"
[17:18] <didrocks> because the whole thing is flacky
[17:18] <asac> not sure why is it suddenly such a big issue todya?
[17:18] <rsalveti> that's why let's try with either 8h or 12h
[17:18] <ogra_> and as the one who does the manual builds i must say that the requirement for manual builds is largely always at the same time
[17:19] <didrocks> asac: and so you need to reboot the device
[17:19] <didrocks> rsalveti: +1
[17:19] <ogra_> so why not just have images inbetween too ?
[17:19] <ogra_> rsalveti, i dont get it
[17:19] <didrocks> ogra_: because you don't let the CI team to get test results
[17:19] <rsalveti> ogra_: we can, I just want to make sure we're not saturating our test lab
[17:19] <ogra_> why does having 2 images help more than 4
[17:19] <didrocks> if you kick a new image
[17:19] <didrocks> because we give them more time to rekick the jobs
[17:20] <didrocks> when you finished building a new image
[17:20] <didrocks> they can't retest an older one
[17:20] <ogra_> didrocks, but if they dont make this image they will make the one in 4h
[17:20] <ogra_> i dont get it
[17:20] <rsalveti> yeah
[17:20] <asac> if noone can tell me why this is so much more important today than yesterday, lets really talk tomorrow in the standup.
[17:20] <asac> :)
[17:20] <ogra_> there is nothing that puts time pressure on anyone
[17:20] <didrocks> ogra_: but then, there is the same issues, they need to rekick
[17:20] <rsalveti> that's why as long we're not saturating the lab, we're fine
[17:20] <didrocks> ogra_: and you just rebuild the next next image
[17:20] <didrocks> and the story repeats :)
[17:21] <ogra_> just make sure you land your stuff inbetween two builds if you want to make sure all of it lands at the same time
[17:21] <rsalveti> asac: why can't we just add it there and see what happens
[17:21] <asac> rsalveti: right now until we have the emuilator fully rolled out its about saturating support folks
[17:21] <ogra_> and the schedule is known
[17:21] <ogra_> didrocks, why do they need to rekick
[17:21] <ogra_> or even care for the images
[17:21] <didrocks> ogra_: repeating "because it's flacky"
[17:21] <asac> rsalveti: understand taht i have to figure how to deal with that with CI process
[17:21] <rsalveti> asac: sure, but all I'm asking is more images than we have today, and that's for sure something we can improve even without the emulator
[17:21] <ogra_> they just need to check the results for the image their change landed in
[17:21] <didrocks> ogra_: see a vanilla test run
[17:21] <ogra_> no matter when they landed their stuff
[17:21] <didrocks> before plars or psivaa starts looking at it
[17:22] <didrocks> you have a lot of tests not running
[17:22] <asac> rsalveti: not without undersanding and clarifyuing with my team what they should focus on testing etc.
[17:22] <didrocks> or with just 1 test
[17:22] <asac> rsalveti: on technology side its easy
[17:22] <didrocks> (and beeing representated as "green" btw :/)
[17:22] <didrocks> ogra_: so, they restart the jenkins jobs as of now
[17:22] <asac> if you guys feel so strongly about more images
[17:22] <rsalveti> right, but you don't get that we're also wasting resources by not triggering more images
[17:22] <asac> lets really check how we can get that
[17:22] <rsalveti> because it's really a pain to find and revert regressions
[17:23] <asac> i am not sayuing we shouldnt have more images
[17:23] <rsalveti> if we get tons of changes at every image we produce
[17:23] <asac> just lets discuss how to do that brest
[17:23] <didrocks> (btw, while we discuss, we still can't test an image I guess and powerpc is broken in distro ;))
[17:23] <ogra_> asac, it wastes immense amount of manpower to just make the CI team happy
[17:23] <ogra_> while they would just have to obey to a cronned build schedule
[17:24] <ogra_> finding an issue in an image that has 50 packages changed is lots and lots harder than finding it in one that only has 10
[17:25] <rsalveti> +100
[17:25] <rsalveti> it's just stupid to fight automation
[17:25] <ogra_> right
[17:25] <rsalveti> as the archive is always moving
[17:25] <ogra_> asac, i worked with this system for 4 months now ... and thats the result of my experience
[17:26] <ogra_> we waste manpower and that doesnt need to happen
[17:27] <asac> i am the first who wants more images. if the landing team would be operational they would at least spin 2 images a day
[17:27] <asac> it should be closer to 3
[17:27] <ogra_> and its a trivial change ... just a crontab entry
[17:27] <asac> and as we add more "smart automation" we woulc increase the pace
[17:27] <ogra_> but why do we have to make it depend on the landing team
[17:27] <ogra_> the landing team can land stuff even with automated builds
[17:27] <rsalveti> yeah, that's the wrong, as the landing team doesn't have the power to freeze the archive :-)
[17:28] <rsalveti> *that's wrong
[17:28] <ogra_> its not like it is hard, the schedule will be known so you know when your stuff gets into the next image
[17:28] <rsalveti> didn't make at one image? wait for the next
[17:28] <ogra_> right
[17:29] <rsalveti> just don't make everyone to wait on your call
[17:29] <jibel> cihelp: publisher on d-jenkins fails to create new jobs on public instance
[17:29] <rsalveti> about triggering a new image
[17:29] <jibel> example http://d-jenkins.ubuntu-ci:8080/job/trusty-adt-python-misaka/1/?
[17:29] <retoaded> jibel, looking
[17:32] <jibel> retoaded, it is only the creation of a new jobs, publication of new builds work fine.
[17:34] <retoaded> jibel, it wouldn't happen to be a matrix job would it?
[17:35] <tedg> Not sure who to blame, but having DNS on the CI VPN is pretty cool.  Thanks!
[17:37] <jibel> retoaded, it is
[17:37] <retoaded> jibel, http://10.98.3.6:8080/plugin/build-publisher/instance/0/publisherThread/currentState/output  makes me think that it could be related to the just updated matrix-reloaded plugin.
[17:37] <retoaded> jibel, i will probably have to revert it back.
[17:40] <jibel> retoaded, I'm not sure, I see no reference to the plugin in the trace. But if it's a recent change revert it back and we'll see
[17:43] <retoaded> jibel, it near the bottom: Caused by: java.lang.NullPointerException at hudson.matrix.MatrixProject.onLoad(MatrixProject.java:474)
[17:43] <retoaded> jibel, and as soon as the currently running jobs complete I'll revert
[17:49] <jibel> retoaded, it is a reference to a Matrix object from jenkins core, not the plugin.
[17:51] <ogra_> [17:51]  * popey updates
[17:51] <ogra_> (totally forgot that over the abive discussion)
[17:51] <ogra_> *above
[17:53] <jibel> retoaded, it is not a new problem apparently
[17:53] <jibel> retoaded, trusty-adt-simplejson failed to publish too
[17:56] <jibel> retoaded, oh, and it vanished from d-jenkins
[17:56] <jibel> WTF
[17:56] <Ursinha> ogra_, can I update my phone then? :P
[17:57] <jibel> retoaded, so on the filesystem there is a /var/lib/jenkins/jobs/trusty-adt-simplejson but not on the UI
[17:57] <ogra_> Ursinha, well, my maguro just updated fine here
[17:57] <jibel> retoaded, and publication fails with the same error 500
[17:58] <ogra_> bah, since when is facebook on the home lens
[17:58] <Ursinha> ogra_, are you using -proposed? read-only?
[17:58] <ogra_> yes
[17:58] <Ursinha> ogra_, at least since yesterday when I flashed mine with the current devel
[17:58] <ogra_> well, i do OTA updates
[17:59] <ogra_> Ursinha, heh, that was ricardos fault ... r25 was broken
[17:59] <Ursinha> ogra_, it rarely is rsalveti's fault, hehe
[17:59] <ogra_> haha
[17:59] <ogra_> never ever :)
[17:59] <Ursinha> rarely ;)
[18:03] <rsalveti> lol
[18:41] <Ursinha> ogra_, so I heard you test all the images :P I have a nexus 4 and whenever I put some music on and change to another app the music stops playing after a few seconds, is that a known issue?
[18:42] <popey> it shouldn't
[18:42] <ogra_> Ursinha, i test only on maguro ...
[18:42] <Ursinha> I just updated with the latest -proposed image
[18:42] <popey> what image are you running Ursinha ?
[18:42] <ogra_> popey, is the makoman
[18:42] <popey> Ursinha: are you using the music app or something else?
[18:43] <Ursinha> popey, Home lens, click on music and then play
[18:43] <Ursinha> about this phone says r27
[18:43] <popey> my phone is dead.. one moment
[18:45] <popey> works fine here
[18:45] <Ursinha> popey, music app has the same behavior, but the interesting thing is that the music stops playing but when I return to the music app it resumes playing as if it never stopped
[18:45] <Ursinha> there's no sound but it seems the app was playing it
[18:45] <popey> i am listening to music which is playing in the music app but I'm on the home screen
[18:45] <popey> it even plays when locked
[18:46] <popey> and continies to play after a song finishes
[18:46] <Ursinha> popey, what should I do then?
[18:46] <popey> is it launching the music app ?
[18:47] <Ursinha> popey, yes, it launches and I'm able to listen to music, the only problem is when I change screens it stops after a few seconds
[18:47] <popey> http://popey.com/~alan/phablet/device-2013-11-20-184641.png
[18:47] <popey> like that
[18:47] <Ursinha> it's playing as it should (I think), displaying the cover and band/song name
[18:47] <popey> was this a clean install or an update to an existing install?
[18:48] <Ursinha> popey, it was a clean install yesterday, with the current devel image
[18:48] <popey> odd
[18:48] <Ursinha> I switched to -proposed today and the problem is the same
[18:48] <popey> I dont understand why it's working for me and not you
[18:48] <Ursinha> hehe
[18:49] <Ursinha> popey, btw it keeps playing when screen is locked
[18:50] <Ursinha> popey, I'll try a clean install and let you know
[18:51] <cjwatson> http://people.canonical.com/~cjwatson/tmp/emulator-yay.png
[18:51] <cjwatson> didn't *quite* manage to get it up on the hangout just now ...
[18:51] <ogra_> :)
[18:52] <ogra_> thats quite a wide screen
[18:52] <cjwatson> dual monitor
[18:52]  * ogra_ has triple ... but screenshots always only show one screen 
[18:52] <cjwatson> import(1) doesn't really understand that and just gives me the whole thing.  I should probably use the screenshot thing in the desktop but old habits die very hard
[18:52] <ogra_> might be nvidias fault
[18:53] <ogra_> ah
[18:53] <ogra_> yeah, i only use alt+print
[19:09] <fginther> cyphermox, kenvandine, robru, we need to restart q-jenkins as soon as possible
[19:10] <robru> fginther, fine by me...
[19:10] <kenvandine> fginther, go for it
[19:10] <cyphermox> yeah, fine
[19:10] <fginther> there is a build in progress (for 11 hours) is it safe to kill ti?
[19:10] <fginther> it
[19:11] <robru> fginther, yeah, whatever. we can restart it if necessary. i'm not waiting on anything myself
[19:11] <robru> fginther, also 11 hours usually indicates a problem anyway
[19:11] <fginther> robru, I kinda figured that :-)
[19:11] <fginther> robru, kenvandine cyphermox thanks for the input
[20:19] <rsalveti> Ursinha: background music working fine with image 27, let me help you debugging the issue
[20:19] <Ursinha> rsalveti, okay, thanks
[20:47] <fginther> cyphermox, is it difficult to re-deploy all of the cu2d jobs?
[20:48] <cyphermox> well, i'd love to avoid it if possible, why?
[20:49] <fginther> cyphermox, the jenkins root moved from /var/lib/jenkins to /iSCSI/jenkins. and some of the scripts had the old path hard coded
[20:49] <fginther> cyphermox, I have MPs to update the scripts and the job. I just want a plan for how best to do this
[20:50] <fginther> would prefer to actually do the update tomorrow when more help is available
[20:58] <cyphermox> are you saying you would prefer or are you asking my opinion?
[21:00] <fginther> cyphermox, I just want some idea of what is involved? does it require lots of downtime, do we need to make special arrangments to make it happen, etc.?
[21:03] <cyphermox> not really, I think we just need to update the paths in lp:cupstream2distro and lp:cupstream2distro-config (if any there), redeploy all the jobs, etc, make sure all the necessary machines have their copies of those branches updated
[21:03] <cyphermox> it would take an hour of work maybe?
[21:03] <fginther> cyphermox, good. I do have MPs here:
[21:03] <cyphermox> but I could see it being done much more cleanly if we use a symlink as a temporary step to avoid breaking things
[21:03] <fginther> https://code.launchpad.net/~fginther/cupstream2distro/jenkins-home/+merge/195848
[21:03] <fginther> https://code.launchpad.net/~fginther/cupstream2distro-config/update-stack_status/+merge/195956
[21:05] <fginther> cyphermox, things are already broken :-(
[21:06] <cyphermox> how so?
[21:06] <fginther> the check jobs are looking for job data under /var/lib/jenkins. it's not there
[21:06] <cyphermox> right
[21:06] <cyphermox> that's partly why i mentioned a symlink
[21:07] <cyphermox> if not /var/lib/jenkins -> /iSCSI/jenkins then one for just cu2d
[21:07] <cyphermox> that allows a smooth transition, but not meant to stay in place
[21:07] <fginther> cyphermox, hmm, ok
[21:07] <cyphermox> so, both branches look fine
[21:08] <fginther> thanks, I'll email out a plan
[21:10] <cyphermox> fwiw I see other places in cupstream2distro-config that need a path change
[21:47] <fginther> cyphermox, ack! can you add them to the bug: https://bugs.launchpad.net/cupstream2distro/+bug/1252750
[21:48] <cyphermox> do you have an equivalent bug for cupstream2distro-config?
[21:48] <fginther> cyphermox, no
[21:50] <robru> fginther, no CI after 5 hours? can you look please? https://code.launchpad.net/~robru/friends-app/autopilot-py3/+merge/195979 thanks
[21:52] <fginther> robru, all the phones are blocked due to an image flash problem
[21:52] <fginther> robru, it is being worked
[21:53] <robru> fginther, oh ok thanks. just really curious about what happens with that branch ;-)
[21:53] <robru> no worries
[23:01] <Saviq> fginther, afraid makos started to get stuck in flashing, too: http://s-jenkins.ubuntu-ci:8080/computer/mako-04cb53b598546534/? http://s-jenkins.ubuntu-ci:8080/computer/mako-04cbcc545f5328a5/?
[23:04] <fginther> Saviq, thanks for the ping. Someone is working on it right now, but they have to manually reflash each device this time
[23:06] <fginther> Saviq, all three of them were dead a couple hours ago
[23:06] <asac> curious: what happened to them?
[23:06] <Saviq> fginther, ah
[23:06] <fginther> asac, AIUI there was a problem with the image today, plars do you know more?
[23:08] <asac> ok thanks. nevermind then
[23:09] <fginther> were also still being hit by https://bugs.launchpad.net/phablet-tools/+bug/1249162
[23:10]  * asac wonders what a phablet-flash loop is :)
[23:11] <asac> ok i think i see what it tries to describe
[23:13] <fginther> the phablet-flash loop is a way to reproduce the problem we see on our devices
[23:13] <asac> check my comment please
[23:47] <sergiusens> asac, fginther Saviq recovery was broken, that was the problem
[23:48] <sergiusens> I'm adding a safeguard to phablet-flash to avoid flashing if recovery is broken