[06:08] <didrocks> hey Mirv, how are you?
[06:10] <Mirv> didrocks: hey, fine. I've been fixing 10+ prepare jobs this morning for head/saucy, but there's still work to do
[06:10] <didrocks> Mirv: sil2100 didn't finish it and didn't email you?
[06:11] <Mirv> didrocks: I don't think he worked on those, I saw the same prepare jobs yellow as yesterday
[06:11] <Mirv> (no e-mail)
[06:11]  * didrocks wonders why fixing prepare jobs is taking so long.
[06:11] <didrocks> Mirv: urgh :/
[06:11] <didrocks> ok, I'll talk with him
[06:11] <didrocks> I'm looking at why the jobs FTBFS
[06:12] <Mirv> didrocks: oh, I started using the Stack status for trusty and filed first FTBFS bugs https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0AuDk72Lpx8U5dHFtUmlPOUtCRk8zR2dtaEpIbUVhMmc#gid=3
[06:12] <didrocks> Mirv: ah, excellent! looking :)
[06:16]  * didrocks adds https://bugs.launchpad.net/ubuntu/+source/mediascanner/+bug/1243536
[06:19] <didrocks> and https://bugs.launchpad.net/autopilot-gtk/+bug/1243538
[06:21] <didrocks> jibel: hey!
[06:26] <Mirv> didrocks: the prepare jobs are now fast to fix (I'm now at ~15/20), there was just so much to do before that. on Monday I started with the thought there's just small bits needing to be done, while actually most of the transition was not done at all.
[06:26] <didrocks> Mirv: urgh, ok… I hope we'll get better at next release opening
[06:27] <didrocks> Mirv: there are some in saucy as well it seems
[06:27] <didrocks> Mirv: I'm pocking people to get their FTBFS fixed
[06:28] <Mirv> didrocks: yeah.. as I was away for a few days, and you mentioned something along the lines that it's about done but sil missed a few, I was under wrong impression. during Mon & Tue then the actual transition was done with me + sil + fginther.
[06:28] <Mirv> didrocks: yeah, I've fixed around 8/14 of the saucy prepare job problems now
[06:29] <didrocks> Mirv: ok, good luck :)
[06:29] <didrocks> I'm more wondering why sil2100 didn't email/continue yesterday
[06:29] <didrocks> because the FTBFS were not handled
[06:29] <didrocks> nor the prepare jobs
[06:29] <didrocks> nor the -check issue (no access to archive.ubuntu.com)
[06:31] <didrocks> vila: did you get any email? this is blocking us
[06:31] <didrocks> vila: from magners-orchestra, I can't ping archive.ubuntu.com
[06:31] <didrocks> unfortunately, as sil2100 didn't share any info (I guess he pinged retoaded privately), we don't know where we stand
[06:31] <didrocks> asac: FYI ^
[07:01] <vila> didrocks: can't ping from m-o, filtered by the IS firewall, I use wget for that kind of tests, and wget archive.ubuntu.com works
[07:01] <vila> didrocks: so, what's the issue ?
[07:02] <didrocks> vila: http://10.97.0.1:8080/job/autopilot-trusty-daily_release/9/label=autopilot-nvidia/console for instance
[07:03] <didrocks> /var/log/upstart/otto-setup.log:   Could not resolve 'archive.ubuntu.com'
[07:03] <didrocks> vila: seems to work from outside the container
[07:03] <cjwatson> that doesn't sound like a firewall problem - you wouldn't be getting a DNS failure in that case
[07:03] <didrocks> (using wget)
[07:03] <sil2100> Hi
[07:03] <cjwatson> more likely busted resolv.conf or something
[07:03] <sil2100> didrocks: retoaded had this to say (I though he was fixing that):
 sil2100, looking through the console output in the job then looking at dx-autopilot-nvidia it appears that the LXC network bridge (lxcbr0) doesn't have an interface assigned so it doesn't have a network path outside if itself.
 nor are there any iptables rules in place to route the network traffic from lcxbr0 to eth0
[07:05] <sil2100> This was close to my EOD
[07:05] <didrocks> sil2100: hum, weird, something changes on the machine, we got a successful snapshot (and it was the raring machines)
[07:06] <sil2100> hmm
[07:06] <didrocks> jibel: do you remember we are doing any bridge manual setup on the hosts for otto? ^
[07:06] <didrocks> sil2100: btw, Mirv is finishing the prepare jobs, (apparently he didn't find any branches)
[07:07] <vila> didrocks: urhg :-(
[07:07] <jibel> didrocks, we don't do anything the bridge is handled by lxc in the upstart job lxc.conf
[07:07]  * didrocks continues on libaccounts-glib FTBFS
[07:07] <didrocks> maybe a regression from latest lxc?
[07:07] <didrocks> vila: are you on it? ^
[07:08] <vila> didrocks: not for now, I'm on upstream-merger for trusty but I can switch if needed
[07:08] <jibel> lxc-net.conf actually
[07:09] <vila> didrocks: which also involve otto and a container so I may run into the same issue...
[07:09] <didrocks> vila: ok, so upstream-merger is blocked as well?
[07:09] <vila> didrocks: for trusty yes (unless fginther did something I'm not aware of)
[07:10] <Mirv> sil2100: I noticed that you had two branches, but there were 17 prepare job problems all in all - now fixed
[07:10] <didrocks> Mirv: all fixed? \o/ (saucy as well?)
[07:10] <vila> didrocks: as for cu2d on phones, I've seen MPs from doanac but no firm confirmation is was ready to be plugged...
[07:11] <didrocks> vila: ok, keep us posted once he's online. I guess the priority is upstream-merger/otto, while we are working with upstream to fix FTBFS
[07:11] <vila> didrocks: /me nods
[07:11] <Mirv> didrocks: yep, saucy as well
[07:12] <didrocks> Mirv: thanks a lot! seems it wasn't that long after all ;) not sure why sil2100 wasn't able to finish it
[07:12] <didrocks> (the accounts-glib FTBFS is really puzzling)
[07:12] <vila> didrocks: still, for my sanity/understanding, http://10.97.0.1:8080/job/autopilot-trusty-daily_release is a step (or several) further than the setup we did with you jibel for otto ?
[07:13] <didrocks> vila: it's the otto job, the one running the tests
[07:13] <didrocks> but it can't get to archive.ubuntu.com for any reason inside the container
[07:13] <vila> didrocks: I thought we were able to create a container which includes an apt-get update right ?
[07:13] <didrocks> so the tests are failing
[07:13] <didrocks> vila: right, on Monday
[07:13] <vila> didrocks: so that's a regression right ?
[07:13] <didrocks> vila: sounds like it
[07:13] <vila> didrocks: ok, thanks
[07:14]  * vila restores a tiny bit of sanity
[07:14] <vila> didrocks: could it be something similar to the [[:space:]] issue encountered on the phone yesterday ?
[07:16] <ogra_> hmm, was the dashboard CSS updated or sis my browser just decide to use a completely different font on its own  ?
[07:17] <didrocks> vila: maybe
[07:17] <ogra_> s/sis/did/
[07:19] <ogra_> system-image-cli -c trusty-proposed -v
[07:19] <ogra_> ...
[07:19] <ogra_> [systemimage] Oct 23 09:19:13 2013 (22849) Already up-to-date
[07:19] <ogra_> HMPF
[07:20] <ogra_> root@ubuntu-phablet:/# system-image-cli -i|grep channel
[07:20] <ogra_> channel: saucy-proposed
[07:20] <ogra_> that doesnt look right
[07:21] <didrocks> ogra_: there are multiple files for the channels
[07:22] <didrocks> ogra_: at least 2
[07:22] <didrocks> so I won't be surprised there is a bug somewhere
[07:22] <ogra_> didrocks, well, the above should switch me to trusty-proposed
[07:22] <ogra_> according to the docs
[07:22] <didrocks> ogra_: right, maybe it's just updating the wrong file :)
[07:22] <ogra_> and i know it worked for pat mcgowan yesterday
[07:22] <didrocks> ah, interesting
[07:22] <didrocks> RO image?
[07:22] <ogra_> oh, wait
[07:22] <ogra_> i tested something ...
[07:23]  * ogra_ checks for writable_image
[07:23] <ogra_> right, its rw
[07:23]  * ogra_ removes and reboots
[07:23] <didrocks> ok, was worth looking…
[07:24] <ogra_> doesnt change even in ro
[07:25] <sil2100> Mirv: there were more branches, I pasted like 8 branches to cyphermox before going EOD
[07:27]  * ogra_ phablet-flash'es .... hoping his data wont be wiped
[07:27] <sil2100> didrocks: I pasted all the branches that needed merging to cyphermox, and I see 2 of those didn't merge in because of merger failures
[07:28] <didrocks> sil2100: branches that you handled?
[07:28] <didrocks> argh, so duplicate work because of no email/communcation :/
[07:29] <sil2100> hm, could be, I thought Mathieu handled those as I pasted those to him - but let me check what's the status with that
[07:31] <sil2100> Mirv: which branches did you have to fix? As I fixed all yellow prepare jobs that I saw yesterday, at least preparing merges for those from the Trusty side
[07:35] <didrocks> sil2100: btw, any opinion on #ubuntu-unity?
[07:39] <didrocks> vila: ok, seems 1.0.0~alpha2 is guilty
[07:39] <didrocks> downgraded to alpha1 and no issue
[07:39] <didrocks> (lxc)
[07:43] <vila> didrocks: lxc on the host right ?
[07:43] <didrocks> vila: right
[07:44] <didrocks> then, there is another issue I'm debugging
[07:44] <didrocks> aptsources.distro.NoDistroTemplateException: Error: could not find a distribution template for Ubuntu/trusty
[07:44] <didrocks> hum, lsb-release is fine though
[07:44] <vila> good, the use case for upstream-merger is slightly different as we use saucy on the host so I may be immune, will know soon
[07:45] <didrocks> vila: I'm going to hold lxc upgrade
[07:45] <vila> didrocks: rings a bell, heard something I didn't followup closely about patching distro-data manually :-/
[07:46] <didrocks> vila: well, ubuntu.csv contains 14.04 LTS,Trusty Tahr,trusty,2013-10-17,2014-04-17,2019-04-17
[07:47]  * vila sighs
[07:47] <vila> didrocks: something else then :-/ Sorry
[07:47] <ogra_> sigh
[07:48] <ogra_> so switching to trusty with pahbelt-flash works, but all installed apps are gone
[07:48] <ogra_> thats annoying :/
[07:48] <wgrant> didrocks: python-apt-common needs updating
[07:48] <vila> ogra_: I got that even without switching (i.e. image 101)
[07:49] <didrocks> wgrant: ah, that's the one! thanks :) I can do it
[07:49] <vila> ogra_: but I had to redo the restoring part of the backup manually (adb got stucked but I copied the backup.tar.gz during phablet-flash, lucky me)
[07:50] <wgrant> Amusingly there's a new one stuck in proposed because bzr-builddeb autopkgtest failed.
[07:51] <didrocks> ah, indeed :)
[07:51] <ogra_> vila, well, by the looks of it it didnbt even do a backup, it just didnt touch my homedir
[07:52] <vila> ogra_: so it failed to restore it , didn't phablet-flash hang ?
[07:52] <ogra_> nope
[07:52] <ogra_> just went through
[07:52] <ogra_> i also didnt get a full image ... which i thought i was supposed if i swithc channels
[07:53]  * ogra_ will wait for stgraber 
[07:53] <Mirv> sil2100: right, I only saw the unmerged ones. so also fix needing in trusty were libcolumbus, autopilot, mtp, compiz, and in saucy some 10+ more
[07:59] <Mirv> didrocks: can you redeploy head unity8 + mir for the unity-mir move? I can't deploy the latter since I don't have rights to lightdm
[08:00] <didrocks> Mirv: sure, only head, right? nothing to deploy for saucy?
[08:01] <Mirv> didrocks: I didn't touch saucy for those, and mir is already disabled there
[08:01] <didrocks> Mirv: reminder: when we move/removed things, you need to delete by hand the jenkins job
[08:01] <Mirv> didrocks: I remember that
[08:01] <didrocks> Mirv: ok, deployed
[08:02] <Mirv> thanks, cleaned the unity-mir from unity8
[08:02] <didrocks> thanks ;)
[08:04] <sil2100> Mirv: right, saucy! I only posted the trusty merges to cyphermox it seems!
[08:05] <didrocks> sil2100: can you relaunch the jobs where the prepare jobs failed?
[08:05] <didrocks> just to ensure we are all good on that one at least
[08:05] <sil2100> Mirv: sorry about that, we duplicated work because of that
[08:05] <sil2100> didrocks: the saucy failures you mean? Sure
[08:06] <Mirv> sil2100: I don't think there was any duplicates beside the autopilot + ui-toolkit, since you didn't start on the saucy, plus the trusty ones I did didn't have branches from you?
[08:06] <Mirv> FYI unity-system-compositor was missing from head before didrock's redeploy. I'll now compare all stacks side-by-side head vs. saucy and clean up eg. removed jobs
[08:08] <sil2100> didrocks: it seems all of those stacks were already ran and are now green for saucy \o/
[08:09] <sil2100> didrocks, Mirv: what should we do with stacks that have no components in them anymore?
[08:09] <sil2100> (in saucy)
[08:09] <sil2100> Should we remove those altogether?
[08:09] <didrocks> sil2100: and head?
[08:10] <didrocks> sil2100: if you can remove them from config + jenkins, I guess that will help clarifying, yeah
[08:10] <didrocks> thanks Mirv, sil2100
[08:10] <sil2100> didrocks: clean as well
[08:11] <didrocks> grrr, I can't reproduce the bzr-builddeb failure on saucy, even upgrading latest python-apt
[08:11] <didrocks> will need a trusty chroot I guess
[08:11] <sil2100> didrocks, Mirv: ok, I'll clean up the empty stacks then, as we see now that more or less things are working right now
[08:12] <sil2100> didrocks: !
[08:12] <sil2100> didrocks: I'm just checking my mail and I got a mail from Larry about the lxc issues, let me take a look
[08:13] <sil2100> didrocks: hah, let me forward it to you
[08:13] <cjwatson> didrocks: what was the lxc failure?
[08:13] <Saviq> didrocks, any word on unity-mir and lxc-android-config release?
[08:13] <sil2100> didrocks: it seems the issue is a bit bigger
[08:13] <didrocks> cjwatson: I didn't debug it yet, just downgraded, I'm on the bzr-builddeb autopkgtests failing which is blocking otto as well
[08:14] <didrocks> but hard to reproduce, can't start in the isolated env (following the doc from http://developer.ubuntu.com/packaging/html/auto-pkg-test.html#executing-the-test) but http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img doesn't exist yet
[08:14] <sil2100> didrocks: forwarded the e-mail, seems like retoaded forwarded the problem to Larry and he made an investigation
[08:14] <didrocks> not sure what adt is using then
[08:14] <didrocks> Saviq: we need first to be able to run the test infra
[08:14] <cjwatson> IIRC adt-virt-lxc isn't actually packaged yet
[08:14] <didrocks> Saviq: and trusty seems not ready for it yet, so working on it
[08:14] <Saviq> didrocks, :/
[08:14] <Saviq> didrocks, k thanks
[08:15] <didrocks> ok, I think I have to upgrade to trusty to hope reproducing the bzr-builddeb autopkgtests issue
[08:16] <didrocks> at least, the 3 errors seems to have a common ground
[08:17] <didrocks> the first one seems more hectic
[08:18] <didrocks> vila: getting any progress on your side on the upstream-merger one?
[08:19] <Mirv> ok head is up-to-date, with the only exception being cordova-ubuntu which will need some more info before adding it - ie. is the 3.0 branch ready to be released (I think not, there are among else hard-coded reference to 2.8 in the SDK)
[08:19]  * cjwatson would like to make it clear that the fact that you're blocked on python-apt in trusty is another casualty of Mark being slow to announce the name
[08:19] <sil2100> didrocks: it anyway seems that the DNS issues we're encountering are some strange apparmor issues
[08:20] <cjwatson> If we'd had the new name in a timely fashion then we could've had support for it in saucy
[08:20] <sil2100> Mirv: yeah, I saw cordova-ubuntu trunk when I was doing the splitting and didn't see a debian/ directory then even
[08:20] <sil2100> Not sure how it is now
[08:21] <Mirv> sil2100: I'm sure the webapps team will let us know when they want it - they've promised that they'll also update the sdk at that time
[08:21] <didrocks> cjwatson: I think it's clear to all of us TBH
[08:22] <didrocks> cjwatson: I'm trying to look at what exactly broke in lxc after the meeting
[08:22] <cjwatson> didrocks: Yeah, I just wanted to hammer it home :)
[08:22] <didrocks> ;)
[08:23] <didrocks> TBH, if we get everything running (including CI) less than a week after the release name was given, it's already an achievement TBH
[08:23] <didrocks> diverging all branches, and so on…
[08:23] <didrocks> we are not rolling, so we can't have a "no time interruption" as if we had it. I think people should understand that
[08:24] <cjwatson> I agree, but there's always pressure to be faster and I really resent having to break promises I've given because of something as simple as a name
[08:24] <vila> didrocks: almost ready to do a run but I lack some context from fginther :-/
[08:24] <cjwatson> And really, if we didn't have that blocker as we've had for the last three cycles, we could be open again almost immediately
[08:25] <didrocks> oh 3? I was thinking it was only 2…
[08:25] <cjwatson> Three
[08:25] <cjwatson> We got raring's name about a day before quantal released, which was too late to add proper support to quantal
[08:26] <cjwatson> And even the cycle before that, we only had three days' notice, which wasn't enough
[08:26] <cjwatson> The cycle before that (so precise's name) we had eight days' notice, which was still less than I'd like but it was manageable
[08:26] <cjwatson> The cycle before *that* we had 52 days' notice
[08:26] <sil2100> Mirv: right
[08:28] <cjwatson> I don't think I'll be on this morning's standup, I've had five hours' sleep due to kids invading the bed and I don't think I can manage coherent conversation in actual speech
[08:28] <vila> didrocks: it failed, that's progress ;) The trusty container doesn't exist, now to find where it's create...
[08:28] <vila> *created
[08:28] <didrocks> cjwatson: no worry ;)
[08:29] <vila> didrocks: it failed with an understandable error which is nice
[08:29] <didrocks> heh
[08:31] <didrocks> vila: joining?
[08:31] <vila> didrocks: yup
[08:51] <ogra_> popey, mind to test trusty-proposed ? we dont have call/sms tests for that yet
[08:52] <popey> ogra_: sure thing
[08:52] <ogra_> thx
[08:52] <ogra_> i think didrocks tested the rest already
[08:52]  * popey OTA's
[08:52] <ogra_> from saucy ?
[08:53] <popey> no, from 14.04
[08:53] <ogra_> ah, k
[08:53] <popey> I mean, OTA from saucy would be nice
[08:53] <popey> but I suspect we're not planning to support that?
[08:53] <ogra_> bug 1243573, bug 1243577
[08:53] <ogra_> thats non OTA though ...
[08:54] <popey> yeah, i flashed clean trusty-proposed on monday iirc
[08:54] <popey> maybe tues
[08:54] <ogra_> i tried to switch the channel from cmdline using system-image-cli -c ... but that didnt work :(
[08:54] <ogra_> it pulled the channel properly from what i saw with -v but told me i'm up to date
[09:02] <popey> ogra_: sms and calls work fine in both directions
[09:02] <ogra_> awesome, thanks
[09:02] <ogra_> didrocks, asac ^^^
[09:07] <didrocks> ogra_: please promote then! :)
[09:07] <didrocks> popey: ^
[09:07] <ogra_> will do
[09:07] <ogra_> (or try to, not sure if the script behaves ... first trusty image etc)
[09:15] <asac> ogra_: pay attention to what might have broken after :)
[09:15] <asac> i am sure all phablet-flash will go crazy hehe
[09:16] <didrocks> vila: so after a reboot, still nothing (worked on the nvidia machines, not on others FYI): http://10.97.0.1:8080/job/autopilot-trusty-setup_otto/label=qa-intel-4000/6/console
[09:16]  * didrocks will classify that as "unknown other issues"
[09:16] <ogra_> asac, i wont switch the devel or devel-proposed channels, thats stgraber stuff ...
[09:16]  * ogra_ will just promote trusty-proposed to trusty
[09:17] <popey> ogra_: lemme know when and I'll test reflashing without --no-backup
[09:19] <vila> didrocks: makes sense since update_host ... update the host and the container so get the latest lxc no ?
[09:19] <didrocks> vila: lxc in the container -> we don't really care
[09:19] <didrocks> it's only the host one which is used
[09:20] <vila> didrocks: update_host update the *host*
[09:20] <ogra_> popey, didrocks, asac trusty 3/20131022.1 promoted
[09:20] <didrocks> vila: yeah, but as told in the hangout, I hold lxc
[09:21] <didrocks> vila: just try apt-cache policy on the machine to check
[09:21] <didrocks> lxc:
[09:21] <vila> didrocks: I'll start with that once done with u-m
[09:21] <didrocks>   Installed: 1.0.0~alpha1-0ubuntu11
[09:21] <didrocks>   Candidate: 1.0.0~alpha2-0ubuntu3
[09:21] <didrocks>   Version table:
[09:21] <didrocks>      1.0.0~alpha2-0ubuntu3 0
[09:21] <didrocks>         500 http://us.archive.ubuntu.com/ubuntu/ trusty/main i386 Packages
[09:21] <vila> didrocks: ack
[09:21] <didrocks>  *** 1.0.0~alpha1-0ubuntu11 0
[09:21] <didrocks>         100 /var/lib/dpkg/status
[09:21] <didrocks> for instance
[09:21] <didrocks> ogra_: thanks!
[09:21] <popey> ogra_: thanks, will test
[09:22] <vila> didrocks: for u-m update_host needs a tweak since it want to create a container for the host release (aka saucy here when I want a trusty container)
[09:23] <didrocks> vila: are you using update_host to create the container? not otto directly?
[09:23] <vila> didrocks: I'm using nothing for now ;) Looking at update_host for at least getting which parameters otto wants
[09:27] <didrocks> asac: status email sent
[09:30] <vila> didrocks: container building
[09:33] <vila> didrocks: lxc_container: command get_init_pid failed to receive response harmless ?lxc-ls
[09:33] <vila> grrr
[09:33] <popey> ogra_: right, so flashing from saucy to trusty wipes /opt/click.ubuntu.com ☹
[09:33] <asac> didrocks: thanks!
[09:33] <popey> do we have a bug for that?
[09:33] <ogra_> popey, yup
[09:34] <ogra_> popey, bug 1243577
[09:34] <popey> ta
[09:40] <vila> didrocks: 2013-10-23 05:36:55,304 INFO Starting container 'trusty-amd64-20131023-0529'
[09:40] <vila> lxc_container: command get_init_pid failed to receive response
[09:40] <vila> didrocks: rings a bell ?
[09:41] <didrocks> vila: can be anything meaning that lxc-start didn't work
[09:42] <vila> ack, looking into that already, thanks
[09:42] <didrocks> https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1181136
[09:43] <vila> didrocks: great :) subscribed... read.... :-/
[09:45] <didrocks> vila: there was another one I can't find right now about no log at all in stdout when using the python bindings
[09:46] <vila> didrocks: seems to be related to the pre-start.sh indeed:
[09:46] <vila> + [ -e /sys/fs/cgroup/memory/lxc/memory.use_hierarchy ]
[09:46] <vila> + echo 1
[09:46] <vila> lxc-start: Script exited with status 1
[09:46] <didrocks> again the same issue :/
[09:47] <vila> whic is weird since that's commented out
[09:47] <didrocks> vila: is it really? we reverted IIRC
[09:48] <vila> didrocks: let me check on dx-autopilot-intel, on ps-radeon-hd8350 otto is up to date with lp:otto
[09:48] <didrocks> vila: dx-autopilot-intel is a saucy machine, right?
[09:49] <vila> roght
[09:57] <vila> didrocks: ~ubuntu/otto is up to date, but ~jenkins/otto is not, stooopid vila
[10:03] <didrocks> vila: you ps-radeon-hd8350?
[10:03] <vila> didrocks: http://10.97.0.26:8080/job/autopilot-testrunner-otto-trusty/9/console so, tests are running but seem to be failing
[10:03] <didrocks> vila: some are passing, so sounds good
[10:03] <didrocks> vila: what stack is it?
[10:04] <vila> didrocks: suspicious messages /var/log/syslog: Oct 23 10:01:36 trusty-amd64-20131023-0529 kernel: [60980.789436] [drm:radeon_dvi_detect] *ERROR* DVI-I-2: probed a monitor but no|invalid EDID
[10:05] <vila> didrocks: http://10.97.0.26:8080/job/autopilot-testrunner-otto-trusty/9/parameters/?
[10:05] <vila> didrocks: so unity8 IIUC
[10:06] <vila> didrocks: I used the same parameters as fginther did on the previous runs
[10:06] <didrocks> vila: making sense for unity8 to fail
[10:06] <vila> didrocks: I think I'll stop on that one as I've already far too many unknowns to check with fginther
[10:07] <vila> will look at qa-intel-4000 instead
[10:08] <didrocks> vila: ok, thanks
[10:21] <vila> didrocks: clarification, when you say 'I hold lxc' you mean on the host or in the archive ? I think we were having two discussions intermixed, me talking about ps-radeon and you talking about qa-intel.
[10:22] <vila> didrocks: on qa-intel the container creation is failing so my question is: can use update_host there or will that bring the latest lxc back ?
[10:22] <vila> didrocks: or will I get the same absence of usable error message by running update_host manually :-/
[10:26] <didrocks> vila: i hold the package on previous lxc on the 3 machines
[10:26] <vila> well, lxc not updated and same messages...
[10:26] <vila> didrocks: right, how do you achieve that ?
[10:27] <didrocks> vila: apt-mark hold <binary_package>
[10:27] <vila> didrocks: thanks
[10:27] <didrocks> yw
[10:33]  * vila breaks for lunch
[10:40] <sil2100> didrocks, vila: how's progress with otto? Any luck?
[10:54] <didrocks> sil2100: I guess it's for vila
[11:20] <vila> sil2100, didrocks: I don't think it solved itself during my lunch but I'll know that soon ;)
[11:21] <sil2100> ;)
[11:22] <vila> right, still the same, the container creation fails... with not a single msg when done from update_host, I need to drill down
[11:24] <Mirv> meanwhile another round of autobuilds is progressing nicely
[11:25] <sil2100> Mirv: were you able to contact anyone from upstream for https://bugs.launchpad.net/libfriends/+bug/1243527 ?
[11:27] <Mirv> sil2100: didier pinged robru about it
[11:30] <Mirv> sil2100: (as it says on the stack status page)
[11:30] <Mirv> sil2100: robert will ping ken in the morning
[11:31] <sil2100> Mirv: ah! Right, it's been such a long time since we used that, I need to get my habits back
[11:34] <Mirv> didrocks: I was finally able to do a proper retrace, it seems apport has various bugs in handling non-English locale that don't even show up as such. bug #1243665 "QUbuntu: Could not create application instance" so might be even mir or such problem
[11:34] <vila> right, so apparmor DENIED on host's /var/log/syslog during lxc-create AFAICS http://paste.ubuntu.com/6288480/
[11:36] <vila> didrocks: you did reboot after downgrading lxc right ?
[11:39] <didrocks> vila: yeah, on 2 of the 3 machines
[11:39] <didrocks> but as it's failing on 2
[11:40] <vila> didrocks: I'm on qa-intel-4000 for now, pinged stgraber in #ubuntu-server
[11:42] <vila> didrocks: the denied are for operation="mount" on /dev/input  /dev/dri but the weird thing is that the pid (16121) is the same for all creations but I can't find that process in 'ps fax'
[11:47] <vila> damn, re-trying again, the apparmor msgs don't come back...
[11:58] <vila> didrocks: no log files under /var/log/lxc for any container I tried to create, the last one if for trusty-i386-20131023-0008 but has been updated 10 minutes ago ???
[12:10] <didrocks> vila: you are trying to create with otto?
[12:11] <didrocks> or lxc-create directly?
[12:13] <vila> otto
[12:14] <vila> hoping to find some msgs somewhere and avoid issues reproducing the way otto calls lxc-create, will try to do that now
[12:15] <didrocks> *sigh* I pointed to https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1181136
[12:15] <didrocks> vila: that why I pasted that bug ^
[12:16] <vila> didrocks: for the lxc log itself, doesn't mean nothing can be found elsewhere
[12:17] <didrocks> vila: what other non lxc logs you were expecting to get in /var/log/lxc?
[12:18] <vila> didrocks: no idea, that 's why I looked and it's still weird that an old log file modification time changes, but I won't dig why as I doubt it's related
[12:32] <vila> didrocks: using otto directly doesn't provide more output so I need to drill down to the lxc-create command but otto "prepares" a lot of stuff first (which are relevant is still unclear...)
[12:34] <vila> didrocks: otto -d gives some hint but still fails too quickly.... that rings a bell
[12:35] <vila> <container:209 - MainThread> 2013-10-23 12:32:59,910 INFO Container 'trusty-i386-20131023-1216' started
[12:35] <vila> <commands:172 - MainThread> 2013-10-23 12:32:59,911 ERROR An error during creation occurred, trying to cleanup the container: The container didn't st
[12:35] <vila> 910 started, 911 ERROR, that's a little too fast
[12:36] <didrocks> vila: the issue is clearly in lxc creation
[12:36] <didrocks> and as per bug I posted above, you won't get any info from the container in lxc
[12:37] <didrocks> due to the python bindings
[12:37] <didrocks> so you need to start lxc-create by hand
[12:37] <vila> didrocks: which otto doesn't display nor the context it requires either as per   This method creates a new container from scratch. We don't want to use
[12:37] <vila>         the create method from LXC API because the bootstrapping phase is not
[12:37] <vila>         required and replaced by using an Ubuntu image directly.
[12:38] <didrocks> vila: otto can't display the lxc logs as per bug :/
[12:38] <didrocks> vila: it's call lxc-start
[12:38] <didrocks> calling*
[12:38] <vila> didrocks: but it could display which lxc-create command it want to execute or are you .... no lxc-create ??
[12:38] <vila> at all ?
[12:39] <didrocks> no lxc-create, lxc-start on the container
[12:39] <didrocks> so a classic lxc-start -n <last_container_name_without_broken._prefix>
[12:39] <didrocks> so you look at what container it created last
[12:39] <didrocks> you mv broken.<container_name> <container_namer>
[12:40] <didrocks> and sudo lxc-start -n <container_name>
[12:41] <vila> pfew...... mv broken. good to get the context creted by otto, I see
[12:42] <didrocks> vila: broken. is to not reuse the broken container
[12:42] <didrocks> but not removing it
[12:42] <didrocks> so that you can still debug
[12:42] <didrocks> and you stay on latest good container
[12:43] <vila> yeah, got that, confused because the last debug session I saw was Mirv extracting the delta, different case
[12:44] <vila> didrocks: and if I haven't read lxc-create when you wrote lxc-start earlier....
[12:45] <vila> didrocks: lxc-start succeeds....
[12:45] <didrocks> vila: well, the second time, I wrote lxc-create TBH (because of broken mind)…
[12:46] <vila> didrocks: nevermind, let's go ahead and fix that crap if we can without stgraber help
[12:47] <vila> didrocks: hmm, the container just logged me out and terminate
[12:47] <didrocks> because it finished its upgrade
[12:48] <vila> didrocks: now I'm confused. If it starts properly and halt properly, what are we searching for ?
[12:48] <didrocks> vila: are you sure it's doing everything properly?
[12:48] <didrocks> if it didn't exit 1, yeah, there is an issue
[12:49] <didrocks> if you are starting the right container
[12:49] <vila> didrocks: no, not even sure I started with good one, so let me re-start that from scratch, update_host, use that one
[12:56] <vila> didrocks: http://paste.ubuntu.com/6288876/
[12:57] <didrocks> vila: it exits 1, right?
[12:58] <vila> didrocks: lxc-start ? 255
[12:58] <didrocks> vila: so… not 0, hence the command failing I guess
[12:59] <vila> didrocks: but it's not the same behavioras when run from otto (,910 -> ,911 earlier) I'd say
[12:59] <vila> behavior as
[12:59] <didrocks> weird
[12:59] <vila> not a spanish variant ;)
[13:00] <vila> ISOID=ubuntu_14.04_lts_trusty_tahr_release_i386_20131021.1
[13:00] <vila> didrocks: expected ? nothing fresher ?
[13:01] <didrocks> vila: you can check yourself at http://cdimage.ubuntu.com/daily-live/current/ :)
[13:01] <didrocks> 21-Oct-2013 13:57
[13:01] <didrocks> which seems to match
[13:01] <vila> didrocks: yeah, thanks, sorry, memory overflow ;)
[13:02] <cjwatson> vila: we haven't re-cronned yet - that was a manual build to let didrocks get otto working
[13:02] <vila> didrocks: so, for this specific lxc-start we're booting from the iso and creates rootfs no ?
[13:03] <didrocks> vila: yeah, we did on Monday
[13:03] <vila> cjwatson: ha, good, thanks ! Yeah heard about that, now I *see* it ;)
[13:06] <vila> didrocks: ha, no, bases/<something>
[13:09] <vila> didrocks: is that correct ? first lxc-start creates the root fs under <lxc-name>/bases ?
[13:10] <vila> didrocks: and the others lxc-start will use a snapshot on top of that right ?
[13:12] <didrocks> vila: right
[13:13] <fginther> morning
[13:14] <vila> didrocks: nothing weird in the logs inside the container :-/
[13:15] <vila> didrocks: will create yet another and look under bases/ before starting
[13:15] <vila> fginther: hey ! Let's sync about trusty for u-m in a few minutes
[13:16] <fginther> vila, ack. let me catch up a moment first
[13:16] <vila> fginther: perfect
[13:18] <vila> didrocks: ha ! no logs (except for faillog and lastlog)
[13:18]  * didrocks in meetings
[13:19] <vila> so lxc-start failure is where I'll search next
[13:19] <vila> didrocks: yeah, no worries, thanks for saying
[14:29] <asac> bfiller: you able to come to our feedback meeting?
[14:30] <bfiller> asac: yes
[14:30] <asac> cool. cu there
[14:42] <vila> didrocks: any objection to putting qa-intel-4000 offline for jenkins while stgraber investigates the lxc issue ? (currently in #distro)
[14:43] <vila> didrocks: done, it's broken anyway, no jobs were running, they can still happily fail on the other slaves ;)
[14:43] <didrocks> vila: still in meeting, but no objection of course ;)
[14:43] <vila> didrocks: ack, thanks
[14:53] <asac> didrocks: dont wait
[14:53] <asac> have netwowkr issues
[15:02] <didrocks> asac: we didn't wait if that can reconfort you :)
[15:03] <asac> didrocks: good.never wait for me
[15:03] <asac> :)
[15:03] <didrocks> ;)
[15:03] <asac> even if you will repeat :)
[15:23] <cyphermox> Mirv: looking at autopilot--- are you sure that there weren't merges that needed to be forward-ported to trunk from the 13.10 branch?
[15:27] <elopio> hey, are the jenkins phones going to be flashed with the trusty channel?
[15:28] <vila> didrocks, asac, Mirv, sil2100: Looks like stgraber found a fix we can apply to otto in lxc.defaults/fstab http://paste.ubuntu.com/6289562/
[15:29] <didrocks> vila: excellent, that should entirely the issue?
[15:31] <vila> didrocks: that's my understanding, roughly the cause was: lxc now supports selinux, checks /sys/kernel/security, couldn't in the otto case, did nothing, the container was still using *host* profiles, messed them up, confusion followed
[15:31] <vila> didrocks: full version in #distro without any misunderstanding added by me ;)
[15:39] <sil2100> ;)
[15:40] <elopio> fginther: I suppose you would know ^
[15:43] <fginther> elopio, that's the plan, but if when we start supporting jobs needing multiple channels, then the image flashed just becomes part of the job configuration so that it flashes the right one
[15:46] <elopio> fginther: my ultimate question is when can I start using autopilot 1.4. Right now I can run autopilot 1.4 tests on my phone because I flashed it with trusty.
[15:46] <elopio> do you have an ETA for when will I be able to do it on Jenkins?
[15:48] <vila> didrocks, sil2100, Mirv: So, I'm going to apply https://code.launchpad.net/~vila/otto/lxc-alpha2/+merge/192347 on qa-intel-4000 and create a new container
[15:51] <sil2100> vila: excellent! Let's give it a try ;)
[15:51] <fginther> elopio, should be working tomorow, I'll get back to you
[15:51] <vila> sil2100: container creation already going further
[15:51] <vila> sil2100: do you know who can review that otto patch ?
[15:51] <elopio> fginther: tomorrow would be more awesome than what I was hoping. Thanks!
[15:52] <vila> jibel is EOD I think
[15:53] <vila> sil2100: container creation succeeded, rebooting
[15:53] <vila> sil2100: do you have a job we can run or re-run there ?
[15:53] <vila> once I put that node back online that is
[16:00] <didrocks> vila: I'll be late, can you start leading the meeting?
[16:00] <sil2100> vila: you mean, to check if it works?
[16:00] <didrocks> asac: ogra_ sil2100 ^
[16:00] <ogra_> didrocks, and i'll have to leave after a few mins
[16:00] <vila> sil2100: yup
[16:00] <ogra_> since my standup moved 1h later today
[16:05] <cyphermox> didrocks: I'm at phonedations standup btw
[16:05] <cyphermox> vila: asac: sil2100:  ^
[16:05]  * ogra_ will go there now too 
[16:07] <sil2100> cyphermox: ACK
[16:09] <sil2100> vila: can I use the intel machine for a test build now?
[16:09] <vila> sil2100: yup, it's good to go
[16:09] <sil2100> vila: I mean, to run some otto testing on the machines now - can I? I want to get the list of extra-packages we need to modify, so that we're done when all machines are fixed
[16:09] <sil2100> vila: thanks!
[16:09] <sil2100> ;)
[16:09]  * sil2100 is running some jobs that are to fail on purpose
[16:10] <vila> sil2100: the missing bit it landing https://code.launchpad.net/~vila/otto/lxc-alpha2/+merge/192347 then I'll update the otto branch there to avoid people getting confused by the actual state
[16:11] <vila> sil2100: that's for qa-intel-4000, the 2 others hosts need to be fixed, I'll do that asap while it's hot ;)
[16:14] <sil2100> didrocks: I'm fixing the extra packages lists now if anythign
[16:14] <sil2100> For all stacks
[16:14] <sil2100> (head for now only)
[16:20] <asac> vila: network issues kicked me out
[16:20] <asac> still having issues
[16:20] <asac> did didrocks manage to join yet?
[16:21] <didrocks> asac: did just before you left
[16:21] <vila> asac: yeah, that bug had very effects on various people ;) Or you were scared by didrocks ;)
[16:21] <vila> *very weird
[16:22] <asac> didrocks: will try after another reboot (resetted my whole network now now)
[16:25] <didrocks> ogra_: so green light to start build #4
[16:25] <ogra_> didrocks, will do after the meeting
[16:26] <ogra_> didrocks, i just said (before i left and you joined i guess) that instead of enabling cronned builds again, i will just fire a build at the end of my day every day
[16:26] <ogra_> so we have something to look at in the morning
[16:26] <ogra_> and can still have a fine grained view on changes
[16:29] <cjwatson> Why not just cron it for the end of your day every day? :)
[16:30] <ogra_> cjwatson, well, i saw the crontab and didnt really want to be the only entry in there :P
[16:30] <asac> ogra_: doesnt need every day. since we seem to resume CI landings, we will have images kicked on demand  all thet ime from tomrrow
[16:30] <asac> so its just about today
[16:30] <ogra_> ok
[16:30] <cjwatson> ogra_: the rest will be turned on soon enough
[16:31] <ogra_> cjwatson, right, and i thought touch can just go with that
[16:31] <cjwatson> ogra_: just getting the installer working first, around everything else
[16:31] <ogra_> yeah
[16:31] <cjwatson> ogra_: it'll probably be tomorrow or Friday
[16:31] <ogra_> k
[16:31] <cjwatson> I did another chunk of it today
[16:31] <ogra_> well, as soon as we do CI landings again and the new process hasnt been discussed i think we'll just stay on manual
[16:32]  * ogra_ was slapped already today for blocking packages in proposed
[16:32] <cjwatson> as you like, just seems odd to be playing human-cron
[16:32] <cjwatson> ogra_: you were?  didn't see any block hints from you
[16:32] <ogra_> well, i'm dont that since a few montjhs now :)
[16:33] <ogra_> cjwatson, i had to remove them again
[16:33] <ogra_> people complained
[16:33] <cjwatson> oh, I'm looking in the wrong branch
[16:33] <ogra_> rev 51-53
[16:58] <didrocks> sil2100: robru: otto ready for all machines \o/
[16:58] <didrocks> thanks vila for tracking ;)
[17:02] <sil2100> \o/
[17:02] <sil2100> didrocks: extra packages lists almost all updated
[17:03] <sil2100> I don't have one for platform yet, as mir needs rebuilding, but we'll get to that later
[17:04] <didrocks> sil2100: ok, QA seems to still need (it's the one I used for tests)
[17:04] <didrocks> but you probably already updated it
[17:05] <sil2100> didrocks: I didn't redeploy the stacks yet, but I have a branch pushed and soon ready for review
[17:06] <didrocks> ah great!
[17:08] <sil2100> didrocks, robru: https://code.launchpad.net/~ubuntu-unity/cupstream2distro-config/updating_extra_packages_for_T/+merge/192364 <- could you guys take a look? I'll redeploy in the meantime
[17:08] <sil2100> Platform I only modified what I could
[17:12] <didrocks> sil2100: I'll let have robru having a closer look ;)
[17:12] <sil2100> I'm redeploying in the meantime :)
[17:13] <robru> didrocks, is there any technical reason those lines can't wrap? if so, can we work on eliminating it this cycle?
[17:13] <didrocks> robru: not sure if yaml accept wrapping? do you know?
[17:14] <didrocks> robru: if you can get something understandable to yaml and the deploy stack command, I'm happy :)
[17:14] <robru> didrocks, why wouldn't it? we wrap in other places. all this time i've simply assumed that cu2d itself was being brittle about the wrapping here
[17:15] <didrocks> robru: so, what I do for deploying is:
[17:15] <didrocks> import yaml
[17:16] <didrocks> yaml.load(this file)
[17:16] <didrocks> then I have a dictionary
[17:16] <didrocks> if this is working with wrapping, I'm all in favor of it :)
[17:17] <robru> didrocks, ok, i am going to try that locally in an interactive session and see what i find...
[17:17] <didrocks> yeah ;)
[18:17] <renato_> fginther, hi
[18:18] <renato_> fginther, could you take a look on these MRs: https://code.launchpad.net/~renatofilho/address-book-app/dynamic-backend/+merge/192326
[18:18] <renato_> and https://code.launchpad.net/~renatofilho/ubuntu-ui-toolkit/fix-1236464/+merge/190496
[18:18] <renato_> jenkins is falling for some strange reason
[18:28] <ogra_> [18:29] <fginther> renato_, https://code.launchpad.net/~renatofilho/address-book-app/dynamic-backend/+merge/192326 failed because the armhf build node failed, it's been re-approved
[18:31] <fginther> renato_, https://code.launchpad.net/~renatofilho/ubuntu-ui-toolkit/fix-1236464/+merge/190496 has a couple of failures. The maguro failure appears to be transient, possibly networking related. I've seen it a few times now since we switched to trusty, I'll try a workaround for that.
[18:32] <fginther> renato_, the mako failure is more strange. It looks like the autopilot tests finish (with some failures) but the results don't get recorder. Will investigate
[18:32] <renato_> fginther, thanks
[21:05] <plars> thomi: did josepht or doanac ping you about autopilot not showing the new skips in the camera-app tests?
[21:12] <thomi> plars: nope
[21:12] <thomi> plars: maybe I missed it in the backscroll?
[21:13] <josepht> thomi: you didn't, we haven't
[21:13] <thomi> ok then :)
[21:14] <josepht> thomi: autopilot needs to include 'skipped="N"' when testcases are skipped
[21:14] <josepht> thomi: in the <testsuite> element
[21:14] <thomi> josepht: in junitxml output format?
[21:14] <josepht> thomi: yes
[21:14] <thomi> it doesn't already!?
[21:15] <josepht> thomi: no sir
[21:15] <thomi> huh
[21:15] <josepht> http://jenkins-dev-image-test:8080/job/trusty-touch_mir-mako/2/artifact/clientlogs/camera_app/test_results.xml
[21:15] <thomi> can you file a bug please?
[21:15] <josepht> thomi: sure
[21:15] <thomi> it does actually skip the tests though, right?
[21:15] <josepht> thomi: I'm not sure
[21:16] <thomi> I'd be very surprised if it was that broken
[21:16] <josepht> I'd say it is skipping them since they're not failing :)
[21:20] <josepht> thomi: #1243928 filed
[21:24] <thomi> cool
[21:25] <thomi> that's an interesting bug, since it's not actually autopilot that formats that output...