didrocks | hey Mirv, how are you? | 06:08 |
---|---|---|
Mirv | didrocks: hey, fine. I've been fixing 10+ prepare jobs this morning for head/saucy, but there's still work to do | 06:10 |
didrocks | Mirv: sil2100 didn't finish it and didn't email you? | 06:10 |
Mirv | didrocks: I don't think he worked on those, I saw the same prepare jobs yellow as yesterday | 06:11 |
Mirv | (no e-mail) | 06:11 |
* didrocks wonders why fixing prepare jobs is taking so long. | 06:11 | |
didrocks | Mirv: urgh :/ | 06:11 |
didrocks | ok, I'll talk with him | 06:11 |
didrocks | I'm looking at why the jobs FTBFS | 06:11 |
Mirv | didrocks: oh, I started using the Stack status for trusty and filed first FTBFS bugs https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0AuDk72Lpx8U5dHFtUmlPOUtCRk8zR2dtaEpIbUVhMmc#gid=3 | 06:12 |
didrocks | Mirv: ah, excellent! looking :) | 06:12 |
* didrocks adds https://bugs.launchpad.net/ubuntu/+source/mediascanner/+bug/1243536 | 06:16 | |
ubot5 | Ubuntu bug 1243536 in mediascanner (Ubuntu) "mediascanner FTBFS in trusty" [Undecided,New] | 06:16 |
didrocks | and https://bugs.launchpad.net/autopilot-gtk/+bug/1243538 | 06:19 |
ubot5 | Ubuntu bug 1243538 in autopilot-gtk "autopilot-gtk FTBFS on trusty" [Undecided,New] | 06:19 |
didrocks | jibel: hey! | 06:21 |
Mirv | didrocks: the prepare jobs are now fast to fix (I'm now at ~15/20), there was just so much to do before that. on Monday I started with the thought there's just small bits needing to be done, while actually most of the transition was not done at all. | 06:26 |
didrocks | Mirv: urgh, ok… I hope we'll get better at next release opening | 06:26 |
didrocks | Mirv: there are some in saucy as well it seems | 06:27 |
didrocks | Mirv: I'm pocking people to get their FTBFS fixed | 06:27 |
Mirv | didrocks: yeah.. as I was away for a few days, and you mentioned something along the lines that it's about done but sil missed a few, I was under wrong impression. during Mon & Tue then the actual transition was done with me + sil + fginther. | 06:28 |
Mirv | didrocks: yeah, I've fixed around 8/14 of the saucy prepare job problems now | 06:28 |
didrocks | Mirv: ok, good luck :) | 06:29 |
didrocks | I'm more wondering why sil2100 didn't email/continue yesterday | 06:29 |
didrocks | because the FTBFS were not handled | 06:29 |
didrocks | nor the prepare jobs | 06:29 |
didrocks | nor the -check issue (no access to archive.ubuntu.com) | 06:29 |
didrocks | vila: did you get any email? this is blocking us | 06:31 |
didrocks | vila: from magners-orchestra, I can't ping archive.ubuntu.com | 06:31 |
didrocks | unfortunately, as sil2100 didn't share any info (I guess he pinged retoaded privately), we don't know where we stand | 06:31 |
didrocks | asac: FYI ^ | 06:31 |
vila | didrocks: can't ping from m-o, filtered by the IS firewall, I use wget for that kind of tests, and wget archive.ubuntu.com works | 07:01 |
vila | didrocks: so, what's the issue ? | 07:01 |
didrocks | vila: http://10.97.0.1:8080/job/autopilot-trusty-daily_release/9/label=autopilot-nvidia/console for instance | 07:02 |
didrocks | /var/log/upstart/otto-setup.log: Could not resolve 'archive.ubuntu.com' | 07:03 |
didrocks | vila: seems to work from outside the container | 07:03 |
cjwatson | that doesn't sound like a firewall problem - you wouldn't be getting a DNS failure in that case | 07:03 |
didrocks | (using wget) | 07:03 |
sil2100 | Hi | 07:03 |
cjwatson | more likely busted resolv.conf or something | 07:03 |
sil2100 | didrocks: retoaded had this to say (I though he was fixing that): | 07:03 |
sil2100 | <retoaded> sil2100, looking through the console output in the job then looking at dx-autopilot-nvidia it appears that the LXC network bridge (lxcbr0) doesn't have an interface assigned so it doesn't have a network path outside if itself. | 07:04 |
sil2100 | <retoaded> nor are there any iptables rules in place to route the network traffic from lcxbr0 to eth0 | 07:04 |
sil2100 | This was close to my EOD | 07:05 |
didrocks | sil2100: hum, weird, something changes on the machine, we got a successful snapshot (and it was the raring machines) | 07:05 |
sil2100 | hmm | 07:06 |
didrocks | jibel: do you remember we are doing any bridge manual setup on the hosts for otto? ^ | 07:06 |
didrocks | sil2100: btw, Mirv is finishing the prepare jobs, (apparently he didn't find any branches) | 07:06 |
vila | didrocks: urhg :-( | 07:07 |
jibel | didrocks, we don't do anything the bridge is handled by lxc in the upstart job lxc.conf | 07:07 |
* didrocks continues on libaccounts-glib FTBFS | 07:07 | |
didrocks | maybe a regression from latest lxc? | 07:07 |
didrocks | vila: are you on it? ^ | 07:07 |
vila | didrocks: not for now, I'm on upstream-merger for trusty but I can switch if needed | 07:08 |
jibel | lxc-net.conf actually | 07:08 |
vila | didrocks: which also involve otto and a container so I may run into the same issue... | 07:09 |
didrocks | vila: ok, so upstream-merger is blocked as well? | 07:09 |
vila | didrocks: for trusty yes (unless fginther did something I'm not aware of) | 07:09 |
Mirv | sil2100: I noticed that you had two branches, but there were 17 prepare job problems all in all - now fixed | 07:10 |
didrocks | Mirv: all fixed? \o/ (saucy as well?) | 07:10 |
vila | didrocks: as for cu2d on phones, I've seen MPs from doanac but no firm confirmation is was ready to be plugged... | 07:10 |
didrocks | vila: ok, keep us posted once he's online. I guess the priority is upstream-merger/otto, while we are working with upstream to fix FTBFS | 07:11 |
vila | didrocks: /me nods | 07:11 |
Mirv | didrocks: yep, saucy as well | 07:11 |
didrocks | Mirv: thanks a lot! seems it wasn't that long after all ;) not sure why sil2100 wasn't able to finish it | 07:12 |
didrocks | (the accounts-glib FTBFS is really puzzling) | 07:12 |
vila | didrocks: still, for my sanity/understanding, http://10.97.0.1:8080/job/autopilot-trusty-daily_release is a step (or several) further than the setup we did with you jibel for otto ? | 07:12 |
didrocks | vila: it's the otto job, the one running the tests | 07:13 |
didrocks | but it can't get to archive.ubuntu.com for any reason inside the container | 07:13 |
vila | didrocks: I thought we were able to create a container which includes an apt-get update right ? | 07:13 |
didrocks | so the tests are failing | 07:13 |
didrocks | vila: right, on Monday | 07:13 |
vila | didrocks: so that's a regression right ? | 07:13 |
didrocks | vila: sounds like it | 07:13 |
vila | didrocks: ok, thanks | 07:13 |
* vila restores a tiny bit of sanity | 07:14 | |
vila | didrocks: could it be something similar to the [[:space:]] issue encountered on the phone yesterday ? | 07:14 |
ogra_ | hmm, was the dashboard CSS updated or sis my browser just decide to use a completely different font on its own ? | 07:16 |
didrocks | vila: maybe | 07:17 |
ogra_ | s/sis/did/ | 07:17 |
ogra_ | system-image-cli -c trusty-proposed -v | 07:19 |
ogra_ | ... | 07:19 |
ogra_ | [systemimage] Oct 23 09:19:13 2013 (22849) Already up-to-date | 07:19 |
ogra_ | HMPF | 07:19 |
ogra_ | root@ubuntu-phablet:/# system-image-cli -i|grep channel | 07:20 |
ogra_ | channel: saucy-proposed | 07:20 |
ogra_ | that doesnt look right | 07:20 |
didrocks | ogra_: there are multiple files for the channels | 07:21 |
didrocks | ogra_: at least 2 | 07:22 |
didrocks | so I won't be surprised there is a bug somewhere | 07:22 |
ogra_ | didrocks, well, the above should switch me to trusty-proposed | 07:22 |
ogra_ | according to the docs | 07:22 |
didrocks | ogra_: right, maybe it's just updating the wrong file :) | 07:22 |
ogra_ | and i know it worked for pat mcgowan yesterday | 07:22 |
didrocks | ah, interesting | 07:22 |
didrocks | RO image? | 07:22 |
ogra_ | oh, wait | 07:22 |
ogra_ | i tested something ... | 07:22 |
* ogra_ checks for writable_image | 07:23 | |
ogra_ | right, its rw | 07:23 |
* ogra_ removes and reboots | 07:23 | |
didrocks | ok, was worth looking… | 07:23 |
ogra_ | doesnt change even in ro | 07:24 |
sil2100 | Mirv: there were more branches, I pasted like 8 branches to cyphermox before going EOD | 07:25 |
* ogra_ phablet-flash'es .... hoping his data wont be wiped | 07:27 | |
sil2100 | didrocks: I pasted all the branches that needed merging to cyphermox, and I see 2 of those didn't merge in because of merger failures | 07:27 |
didrocks | sil2100: branches that you handled? | 07:28 |
didrocks | argh, so duplicate work because of no email/communcation :/ | 07:28 |
sil2100 | hm, could be, I thought Mathieu handled those as I pasted those to him - but let me check what's the status with that | 07:29 |
sil2100 | Mirv: which branches did you have to fix? As I fixed all yellow prepare jobs that I saw yesterday, at least preparing merges for those from the Trusty side | 07:31 |
didrocks | sil2100: btw, any opinion on #ubuntu-unity? | 07:35 |
didrocks | vila: ok, seems 1.0.0~alpha2 is guilty | 07:39 |
didrocks | downgraded to alpha1 and no issue | 07:39 |
didrocks | (lxc) | 07:39 |
vila | didrocks: lxc on the host right ? | 07:43 |
didrocks | vila: right | 07:43 |
didrocks | then, there is another issue I'm debugging | 07:44 |
didrocks | aptsources.distro.NoDistroTemplateException: Error: could not find a distribution template for Ubuntu/trusty | 07:44 |
didrocks | hum, lsb-release is fine though | 07:44 |
vila | good, the use case for upstream-merger is slightly different as we use saucy on the host so I may be immune, will know soon | 07:44 |
didrocks | vila: I'm going to hold lxc upgrade | 07:45 |
vila | didrocks: rings a bell, heard something I didn't followup closely about patching distro-data manually :-/ | 07:45 |
didrocks | vila: well, ubuntu.csv contains 14.04 LTS,Trusty Tahr,trusty,2013-10-17,2014-04-17,2019-04-17 | 07:46 |
* vila sighs | 07:47 | |
vila | didrocks: something else then :-/ Sorry | 07:47 |
ogra_ | sigh | 07:47 |
ogra_ | so switching to trusty with pahbelt-flash works, but all installed apps are gone | 07:48 |
ogra_ | thats annoying :/ | 07:48 |
wgrant | didrocks: python-apt-common needs updating | 07:48 |
vila | ogra_: I got that even without switching (i.e. image 101) | 07:48 |
didrocks | wgrant: ah, that's the one! thanks :) I can do it | 07:49 |
vila | ogra_: but I had to redo the restoring part of the backup manually (adb got stucked but I copied the backup.tar.gz during phablet-flash, lucky me) | 07:49 |
wgrant | Amusingly there's a new one stuck in proposed because bzr-builddeb autopkgtest failed. | 07:50 |
didrocks | ah, indeed :) | 07:51 |
ogra_ | vila, well, by the looks of it it didnbt even do a backup, it just didnt touch my homedir | 07:51 |
vila | ogra_: so it failed to restore it , didn't phablet-flash hang ? | 07:52 |
ogra_ | nope | 07:52 |
ogra_ | just went through | 07:52 |
ogra_ | i also didnt get a full image ... which i thought i was supposed if i swithc channels | 07:52 |
* ogra_ will wait for stgraber | 07:53 | |
Mirv | sil2100: right, I only saw the unmerged ones. so also fix needing in trusty were libcolumbus, autopilot, mtp, compiz, and in saucy some 10+ more | 07:53 |
Mirv | didrocks: can you redeploy head unity8 + mir for the unity-mir move? I can't deploy the latter since I don't have rights to lightdm | 07:59 |
didrocks | Mirv: sure, only head, right? nothing to deploy for saucy? | 08:00 |
Mirv | didrocks: I didn't touch saucy for those, and mir is already disabled there | 08:01 |
didrocks | Mirv: reminder: when we move/removed things, you need to delete by hand the jenkins job | 08:01 |
Mirv | didrocks: I remember that | 08:01 |
didrocks | Mirv: ok, deployed | 08:01 |
Mirv | thanks, cleaned the unity-mir from unity8 | 08:02 |
didrocks | thanks ;) | 08:02 |
sil2100 | Mirv: right, saucy! I only posted the trusty merges to cyphermox it seems! | 08:04 |
didrocks | sil2100: can you relaunch the jobs where the prepare jobs failed? | 08:05 |
didrocks | just to ensure we are all good on that one at least | 08:05 |
sil2100 | Mirv: sorry about that, we duplicated work because of that | 08:05 |
sil2100 | didrocks: the saucy failures you mean? Sure | 08:05 |
Mirv | sil2100: I don't think there was any duplicates beside the autopilot + ui-toolkit, since you didn't start on the saucy, plus the trusty ones I did didn't have branches from you? | 08:06 |
Mirv | FYI unity-system-compositor was missing from head before didrock's redeploy. I'll now compare all stacks side-by-side head vs. saucy and clean up eg. removed jobs | 08:06 |
sil2100 | didrocks: it seems all of those stacks were already ran and are now green for saucy \o/ | 08:08 |
sil2100 | didrocks, Mirv: what should we do with stacks that have no components in them anymore? | 08:09 |
sil2100 | (in saucy) | 08:09 |
sil2100 | Should we remove those altogether? | 08:09 |
didrocks | sil2100: and head? | 08:09 |
didrocks | sil2100: if you can remove them from config + jenkins, I guess that will help clarifying, yeah | 08:10 |
didrocks | thanks Mirv, sil2100 | 08:10 |
sil2100 | didrocks: clean as well | 08:10 |
didrocks | grrr, I can't reproduce the bzr-builddeb failure on saucy, even upgrading latest python-apt | 08:11 |
didrocks | will need a trusty chroot I guess | 08:11 |
sil2100 | didrocks, Mirv: ok, I'll clean up the empty stacks then, as we see now that more or less things are working right now | 08:11 |
sil2100 | didrocks: ! | 08:12 |
sil2100 | didrocks: I'm just checking my mail and I got a mail from Larry about the lxc issues, let me take a look | 08:12 |
sil2100 | didrocks: hah, let me forward it to you | 08:13 |
cjwatson | didrocks: what was the lxc failure? | 08:13 |
Saviq | didrocks, any word on unity-mir and lxc-android-config release? | 08:13 |
sil2100 | didrocks: it seems the issue is a bit bigger | 08:13 |
didrocks | cjwatson: I didn't debug it yet, just downgraded, I'm on the bzr-builddeb autopkgtests failing which is blocking otto as well | 08:13 |
didrocks | but hard to reproduce, can't start in the isolated env (following the doc from http://developer.ubuntu.com/packaging/html/auto-pkg-test.html#executing-the-test) but http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img doesn't exist yet | 08:14 |
sil2100 | didrocks: forwarded the e-mail, seems like retoaded forwarded the problem to Larry and he made an investigation | 08:14 |
didrocks | not sure what adt is using then | 08:14 |
didrocks | Saviq: we need first to be able to run the test infra | 08:14 |
cjwatson | IIRC adt-virt-lxc isn't actually packaged yet | 08:14 |
didrocks | Saviq: and trusty seems not ready for it yet, so working on it | 08:14 |
Saviq | didrocks, :/ | 08:14 |
Saviq | didrocks, k thanks | 08:14 |
didrocks | ok, I think I have to upgrade to trusty to hope reproducing the bzr-builddeb autopkgtests issue | 08:15 |
didrocks | at least, the 3 errors seems to have a common ground | 08:16 |
didrocks | the first one seems more hectic | 08:17 |
didrocks | vila: getting any progress on your side on the upstream-merger one? | 08:18 |
Mirv | ok head is up-to-date, with the only exception being cordova-ubuntu which will need some more info before adding it - ie. is the 3.0 branch ready to be released (I think not, there are among else hard-coded reference to 2.8 in the SDK) | 08:19 |
* cjwatson would like to make it clear that the fact that you're blocked on python-apt in trusty is another casualty of Mark being slow to announce the name | 08:19 | |
sil2100 | didrocks: it anyway seems that the DNS issues we're encountering are some strange apparmor issues | 08:19 |
cjwatson | If we'd had the new name in a timely fashion then we could've had support for it in saucy | 08:20 |
sil2100 | Mirv: yeah, I saw cordova-ubuntu trunk when I was doing the splitting and didn't see a debian/ directory then even | 08:20 |
sil2100 | Not sure how it is now | 08:20 |
Mirv | sil2100: I'm sure the webapps team will let us know when they want it - they've promised that they'll also update the sdk at that time | 08:21 |
didrocks | cjwatson: I think it's clear to all of us TBH | 08:21 |
didrocks | cjwatson: I'm trying to look at what exactly broke in lxc after the meeting | 08:22 |
cjwatson | didrocks: Yeah, I just wanted to hammer it home :) | 08:22 |
didrocks | ;) | 08:22 |
didrocks | TBH, if we get everything running (including CI) less than a week after the release name was given, it's already an achievement TBH | 08:23 |
didrocks | diverging all branches, and so on… | 08:23 |
didrocks | we are not rolling, so we can't have a "no time interruption" as if we had it. I think people should understand that | 08:23 |
cjwatson | I agree, but there's always pressure to be faster and I really resent having to break promises I've given because of something as simple as a name | 08:24 |
vila | didrocks: almost ready to do a run but I lack some context from fginther :-/ | 08:24 |
cjwatson | And really, if we didn't have that blocker as we've had for the last three cycles, we could be open again almost immediately | 08:24 |
didrocks | oh 3? I was thinking it was only 2… | 08:25 |
cjwatson | Three | 08:25 |
cjwatson | We got raring's name about a day before quantal released, which was too late to add proper support to quantal | 08:25 |
cjwatson | And even the cycle before that, we only had three days' notice, which wasn't enough | 08:26 |
cjwatson | The cycle before that (so precise's name) we had eight days' notice, which was still less than I'd like but it was manageable | 08:26 |
cjwatson | The cycle before *that* we had 52 days' notice | 08:26 |
sil2100 | Mirv: right | 08:26 |
cjwatson | I don't think I'll be on this morning's standup, I've had five hours' sleep due to kids invading the bed and I don't think I can manage coherent conversation in actual speech | 08:28 |
vila | didrocks: it failed, that's progress ;) The trusty container doesn't exist, now to find where it's create... | 08:28 |
vila | *created | 08:28 |
didrocks | cjwatson: no worry ;) | 08:28 |
vila | didrocks: it failed with an understandable error which is nice | 08:29 |
didrocks | heh | 08:29 |
didrocks | vila: joining? | 08:31 |
vila | didrocks: yup | 08:31 |
ogra_ | popey, mind to test trusty-proposed ? we dont have call/sms tests for that yet | 08:51 |
popey | ogra_: sure thing | 08:52 |
ogra_ | thx | 08:52 |
ogra_ | i think didrocks tested the rest already | 08:52 |
* popey OTA's | 08:52 | |
ogra_ | from saucy ? | 08:52 |
popey | no, from 14.04 | 08:53 |
ogra_ | ah, k | 08:53 |
popey | I mean, OTA from saucy would be nice | 08:53 |
popey | but I suspect we're not planning to support that? | 08:53 |
ogra_ | bug 1243573, bug 1243577 | 08:53 |
ubot5 | bug 1243573 in ubuntu-system-settings (Ubuntu) "Timezone setting is gone after upgrade to Ubuntu Touch touchy" [Undecided,New] https://launchpad.net/bugs/1243573 | 08:53 |
ubot5 | bug 1243577 in system-image (Ubuntu) "All apps are gone after upgrade to trusty" [Undecided,New] https://launchpad.net/bugs/1243577 | 08:53 |
ogra_ | thats non OTA though ... | 08:53 |
popey | yeah, i flashed clean trusty-proposed on monday iirc | 08:54 |
popey | maybe tues | 08:54 |
ogra_ | i tried to switch the channel from cmdline using system-image-cli -c ... but that didnt work :( | 08:54 |
ogra_ | it pulled the channel properly from what i saw with -v but told me i'm up to date | 08:54 |
=== vrruiz_ is now known as rvr | ||
popey | ogra_: sms and calls work fine in both directions | 09:02 |
ogra_ | awesome, thanks | 09:02 |
ogra_ | didrocks, asac ^^^ | 09:02 |
didrocks | ogra_: please promote then! :) | 09:07 |
didrocks | popey: ^ | 09:07 |
ogra_ | will do | 09:07 |
ogra_ | (or try to, not sure if the script behaves ... first trusty image etc) | 09:07 |
asac | ogra_: pay attention to what might have broken after :) | 09:15 |
asac | i am sure all phablet-flash will go crazy hehe | 09:15 |
didrocks | vila: so after a reboot, still nothing (worked on the nvidia machines, not on others FYI): http://10.97.0.1:8080/job/autopilot-trusty-setup_otto/label=qa-intel-4000/6/console | 09:16 |
* didrocks will classify that as "unknown other issues" | 09:16 | |
ogra_ | asac, i wont switch the devel or devel-proposed channels, thats stgraber stuff ... | 09:16 |
* ogra_ will just promote trusty-proposed to trusty | 09:16 | |
popey | ogra_: lemme know when and I'll test reflashing without --no-backup | 09:17 |
vila | didrocks: makes sense since update_host ... update the host and the container so get the latest lxc no ? | 09:19 |
didrocks | vila: lxc in the container -> we don't really care | 09:19 |
didrocks | it's only the host one which is used | 09:19 |
vila | didrocks: update_host update the *host* | 09:20 |
ogra_ | popey, didrocks, asac trusty 3/20131022.1 promoted | 09:20 |
didrocks | vila: yeah, but as told in the hangout, I hold lxc | 09:20 |
didrocks | vila: just try apt-cache policy on the machine to check | 09:21 |
didrocks | lxc: | 09:21 |
vila | didrocks: I'll start with that once done with u-m | 09:21 |
didrocks | Installed: 1.0.0~alpha1-0ubuntu11 | 09:21 |
didrocks | Candidate: 1.0.0~alpha2-0ubuntu3 | 09:21 |
didrocks | Version table: | 09:21 |
didrocks | 1.0.0~alpha2-0ubuntu3 0 | 09:21 |
didrocks | 500 http://us.archive.ubuntu.com/ubuntu/ trusty/main i386 Packages | 09:21 |
vila | didrocks: ack | 09:21 |
didrocks | *** 1.0.0~alpha1-0ubuntu11 0 | 09:21 |
didrocks | 100 /var/lib/dpkg/status | 09:21 |
didrocks | for instance | 09:21 |
didrocks | ogra_: thanks! | 09:21 |
popey | ogra_: thanks, will test | 09:21 |
vila | didrocks: for u-m update_host needs a tweak since it want to create a container for the host release (aka saucy here when I want a trusty container) | 09:22 |
didrocks | vila: are you using update_host to create the container? not otto directly? | 09:23 |
vila | didrocks: I'm using nothing for now ;) Looking at update_host for at least getting which parameters otto wants | 09:23 |
didrocks | asac: status email sent | 09:27 |
=== psivaa is now known as psivaa-afk | ||
vila | didrocks: container building | 09:30 |
vila | didrocks: lxc_container: command get_init_pid failed to receive response harmless ?lxc-ls | 09:33 |
vila | grrr | 09:33 |
popey | ogra_: right, so flashing from saucy to trusty wipes /opt/click.ubuntu.com ☹ | 09:33 |
asac | didrocks: thanks! | 09:33 |
popey | do we have a bug for that? | 09:33 |
ogra_ | popey, yup | 09:33 |
ogra_ | popey, bug 1243577 | 09:34 |
ubot5 | bug 1243577 in system-image (Ubuntu) "All apps are gone after upgrade to trusty" [Undecided,New] https://launchpad.net/bugs/1243577 | 09:34 |
popey | ta | 09:34 |
vila | didrocks: 2013-10-23 05:36:55,304 INFO Starting container 'trusty-amd64-20131023-0529' | 09:40 |
vila | lxc_container: command get_init_pid failed to receive response | 09:40 |
vila | didrocks: rings a bell ? | 09:40 |
didrocks | vila: can be anything meaning that lxc-start didn't work | 09:41 |
vila | ack, looking into that already, thanks | 09:42 |
didrocks | https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1181136 | 09:42 |
ubot5 | Ubuntu bug 1181136 in lxc (Ubuntu) "Empty log file when a container is started with the API" [Medium,Confirmed] | 09:42 |
vila | didrocks: great :) subscribed... read.... :-/ | 09:43 |
didrocks | vila: there was another one I can't find right now about no log at all in stdout when using the python bindings | 09:45 |
vila | didrocks: seems to be related to the pre-start.sh indeed: | 09:46 |
vila | + [ -e /sys/fs/cgroup/memory/lxc/memory.use_hierarchy ] | 09:46 |
vila | + echo 1 | 09:46 |
vila | lxc-start: Script exited with status 1 | 09:46 |
didrocks | again the same issue :/ | 09:46 |
vila | whic is weird since that's commented out | 09:47 |
didrocks | vila: is it really? we reverted IIRC | 09:47 |
vila | didrocks: let me check on dx-autopilot-intel, on ps-radeon-hd8350 otto is up to date with lp:otto | 09:48 |
didrocks | vila: dx-autopilot-intel is a saucy machine, right? | 09:48 |
vila | roght | 09:49 |
vila | didrocks: ~ubuntu/otto is up to date, but ~jenkins/otto is not, stooopid vila | 09:57 |
didrocks | vila: you ps-radeon-hd8350? | 10:03 |
vila | didrocks: http://10.97.0.26:8080/job/autopilot-testrunner-otto-trusty/9/console so, tests are running but seem to be failing | 10:03 |
didrocks | vila: some are passing, so sounds good | 10:03 |
didrocks | vila: what stack is it? | 10:03 |
vila | didrocks: suspicious messages /var/log/syslog: Oct 23 10:01:36 trusty-amd64-20131023-0529 kernel: [60980.789436] [drm:radeon_dvi_detect] *ERROR* DVI-I-2: probed a monitor but no|invalid EDID | 10:04 |
vila | didrocks: http://10.97.0.26:8080/job/autopilot-testrunner-otto-trusty/9/parameters/? | 10:05 |
vila | didrocks: so unity8 IIUC | 10:05 |
vila | didrocks: I used the same parameters as fginther did on the previous runs | 10:06 |
didrocks | vila: making sense for unity8 to fail | 10:06 |
vila | didrocks: I think I'll stop on that one as I've already far too many unknowns to check with fginther | 10:06 |
vila | will look at qa-intel-4000 instead | 10:07 |
didrocks | vila: ok, thanks | 10:08 |
vila | didrocks: clarification, when you say 'I hold lxc' you mean on the host or in the archive ? I think we were having two discussions intermixed, me talking about ps-radeon and you talking about qa-intel. | 10:21 |
vila | didrocks: on qa-intel the container creation is failing so my question is: can use update_host there or will that bring the latest lxc back ? | 10:22 |
vila | didrocks: or will I get the same absence of usable error message by running update_host manually :-/ | 10:22 |
didrocks | vila: i hold the package on previous lxc on the 3 machines | 10:26 |
vila | well, lxc not updated and same messages... | 10:26 |
vila | didrocks: right, how do you achieve that ? | 10:26 |
didrocks | vila: apt-mark hold <binary_package> | 10:27 |
vila | didrocks: thanks | 10:27 |
didrocks | yw | 10:27 |
* vila breaks for lunch | 10:33 | |
sil2100 | didrocks, vila: how's progress with otto? Any luck? | 10:40 |
didrocks | sil2100: I guess it's for vila | 10:54 |
vila | sil2100, didrocks: I don't think it solved itself during my lunch but I'll know that soon ;) | 11:20 |
sil2100 | ;) | 11:21 |
vila | right, still the same, the container creation fails... with not a single msg when done from update_host, I need to drill down | 11:22 |
Mirv | meanwhile another round of autobuilds is progressing nicely | 11:24 |
sil2100 | Mirv: were you able to contact anyone from upstream for https://bugs.launchpad.net/libfriends/+bug/1243527 ? | 11:25 |
ubot5 | Ubuntu bug 1243527 in libfriends "libfriends FTBFS on trusty" [Critical,New] | 11:25 |
Mirv | sil2100: didier pinged robru about it | 11:27 |
Mirv | sil2100: (as it says on the stack status page) | 11:30 |
Mirv | sil2100: robert will ping ken in the morning | 11:30 |
sil2100 | Mirv: ah! Right, it's been such a long time since we used that, I need to get my habits back | 11:31 |
Mirv | didrocks: I was finally able to do a proper retrace, it seems apport has various bugs in handling non-English locale that don't even show up as such. bug #1243665 "QUbuntu: Could not create application instance" so might be even mir or such problem | 11:34 |
ubot5 | bug 1243665 in qtdeclarative-opensource-src (Ubuntu) "qmlscene crashed with SIGABRT in raise()" [Undecided,Confirmed] https://launchpad.net/bugs/1243665 | 11:34 |
vila | right, so apparmor DENIED on host's /var/log/syslog during lxc-create AFAICS http://paste.ubuntu.com/6288480/ | 11:34 |
vila | didrocks: you did reboot after downgrading lxc right ? | 11:36 |
didrocks | vila: yeah, on 2 of the 3 machines | 11:39 |
didrocks | but as it's failing on 2 | 11:39 |
vila | didrocks: I'm on qa-intel-4000 for now, pinged stgraber in #ubuntu-server | 11:40 |
vila | didrocks: the denied are for operation="mount" on /dev/input /dev/dri but the weird thing is that the pid (16121) is the same for all creations but I can't find that process in 'ps fax' | 11:42 |
vila | damn, re-trying again, the apparmor msgs don't come back... | 11:47 |
=== alan_g is now known as alan_g|lunch | ||
vila | didrocks: no log files under /var/log/lxc for any container I tried to create, the last one if for trusty-i386-20131023-0008 but has been updated 10 minutes ago ??? | 11:58 |
didrocks | vila: you are trying to create with otto? | 12:10 |
didrocks | or lxc-create directly? | 12:11 |
vila | otto | 12:13 |
vila | hoping to find some msgs somewhere and avoid issues reproducing the way otto calls lxc-create, will try to do that now | 12:14 |
didrocks | *sigh* I pointed to https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1181136 | 12:15 |
ubot5 | Ubuntu bug 1181136 in lxc (Ubuntu) "Empty log file when a container is started with the API" [Medium,Confirmed] | 12:15 |
didrocks | vila: that why I pasted that bug ^ | 12:15 |
vila | didrocks: for the lxc log itself, doesn't mean nothing can be found elsewhere | 12:16 |
didrocks | vila: what other non lxc logs you were expecting to get in /var/log/lxc? | 12:17 |
vila | didrocks: no idea, that 's why I looked and it's still weird that an old log file modification time changes, but I won't dig why as I doubt it's related | 12:18 |
=== psivaa-afk is now known as psivaa | ||
vila | didrocks: using otto directly doesn't provide more output so I need to drill down to the lxc-create command but otto "prepares" a lot of stuff first (which are relevant is still unclear...) | 12:32 |
vila | didrocks: otto -d gives some hint but still fails too quickly.... that rings a bell | 12:34 |
vila | <container:209 - MainThread> 2013-10-23 12:32:59,910 INFO Container 'trusty-i386-20131023-1216' started | 12:35 |
vila | <commands:172 - MainThread> 2013-10-23 12:32:59,911 ERROR An error during creation occurred, trying to cleanup the container: The container didn't st | 12:35 |
vila | 910 started, 911 ERROR, that's a little too fast | 12:35 |
didrocks | vila: the issue is clearly in lxc creation | 12:36 |
didrocks | and as per bug I posted above, you won't get any info from the container in lxc | 12:36 |
didrocks | due to the python bindings | 12:37 |
didrocks | so you need to start lxc-create by hand | 12:37 |
vila | didrocks: which otto doesn't display nor the context it requires either as per This method creates a new container from scratch. We don't want to use | 12:37 |
vila | the create method from LXC API because the bootstrapping phase is not | 12:37 |
vila | required and replaced by using an Ubuntu image directly. | 12:37 |
didrocks | vila: otto can't display the lxc logs as per bug :/ | 12:38 |
didrocks | vila: it's call lxc-start | 12:38 |
didrocks | calling* | 12:38 |
vila | didrocks: but it could display which lxc-create command it want to execute or are you .... no lxc-create ?? | 12:38 |
vila | at all ? | 12:38 |
didrocks | no lxc-create, lxc-start on the container | 12:39 |
didrocks | so a classic lxc-start -n <last_container_name_without_broken._prefix> | 12:39 |
didrocks | so you look at what container it created last | 12:39 |
didrocks | you mv broken.<container_name> <container_namer> | 12:39 |
didrocks | and sudo lxc-start -n <container_name> | 12:40 |
vila | pfew...... mv broken. good to get the context creted by otto, I see | 12:41 |
didrocks | vila: broken. is to not reuse the broken container | 12:42 |
didrocks | but not removing it | 12:42 |
didrocks | so that you can still debug | 12:42 |
didrocks | and you stay on latest good container | 12:42 |
vila | yeah, got that, confused because the last debug session I saw was Mirv extracting the delta, different case | 12:43 |
vila | didrocks: and if I haven't read lxc-create when you wrote lxc-start earlier.... | 12:44 |
vila | didrocks: lxc-start succeeds.... | 12:45 |
didrocks | vila: well, the second time, I wrote lxc-create TBH (because of broken mind)… | 12:45 |
vila | didrocks: nevermind, let's go ahead and fix that crap if we can without stgraber help | 12:46 |
vila | didrocks: hmm, the container just logged me out and terminate | 12:47 |
didrocks | because it finished its upgrade | 12:47 |
vila | didrocks: now I'm confused. If it starts properly and halt properly, what are we searching for ? | 12:48 |
didrocks | vila: are you sure it's doing everything properly? | 12:48 |
didrocks | if it didn't exit 1, yeah, there is an issue | 12:48 |
didrocks | if you are starting the right container | 12:49 |
vila | didrocks: no, not even sure I started with good one, so let me re-start that from scratch, update_host, use that one | 12:49 |
vila | didrocks: http://paste.ubuntu.com/6288876/ | 12:56 |
didrocks | vila: it exits 1, right? | 12:57 |
vila | didrocks: lxc-start ? 255 | 12:58 |
didrocks | vila: so… not 0, hence the command failing I guess | 12:58 |
vila | didrocks: but it's not the same behavioras when run from otto (,910 -> ,911 earlier) I'd say | 12:59 |
vila | behavior as | 12:59 |
didrocks | weird | 12:59 |
vila | not a spanish variant ;) | 12:59 |
vila | ISOID=ubuntu_14.04_lts_trusty_tahr_release_i386_20131021.1 | 13:00 |
vila | didrocks: expected ? nothing fresher ? | 13:00 |
didrocks | vila: you can check yourself at http://cdimage.ubuntu.com/daily-live/current/ :) | 13:01 |
didrocks | 21-Oct-2013 13:57 | 13:01 |
didrocks | which seems to match | 13:01 |
vila | didrocks: yeah, thanks, sorry, memory overflow ;) | 13:01 |
cjwatson | vila: we haven't re-cronned yet - that was a manual build to let didrocks get otto working | 13:02 |
vila | didrocks: so, for this specific lxc-start we're booting from the iso and creates rootfs no ? | 13:02 |
didrocks | vila: yeah, we did on Monday | 13:03 |
vila | cjwatson: ha, good, thanks ! Yeah heard about that, now I *see* it ;) | 13:03 |
vila | didrocks: ha, no, bases/<something> | 13:06 |
vila | didrocks: is that correct ? first lxc-start creates the root fs under <lxc-name>/bases ? | 13:09 |
vila | didrocks: and the others lxc-start will use a snapshot on top of that right ? | 13:10 |
didrocks | vila: right | 13:12 |
fginther | morning | 13:13 |
vila | didrocks: nothing weird in the logs inside the container :-/ | 13:14 |
vila | didrocks: will create yet another and look under bases/ before starting | 13:15 |
vila | fginther: hey ! Let's sync about trusty for u-m in a few minutes | 13:15 |
fginther | vila, ack. let me catch up a moment first | 13:16 |
vila | fginther: perfect | 13:16 |
vila | didrocks: ha ! no logs (except for faillog and lastlog) | 13:18 |
* didrocks in meetings | 13:18 | |
vila | so lxc-start failure is where I'll search next | 13:19 |
vila | didrocks: yeah, no worries, thanks for saying | 13:19 |
=== alan_g|lunch is now known as alan_g | ||
=== Ursinha is now known as Ursinha-afk | ||
=== Ursinha-afk is now known as Ursinha | ||
asac | bfiller: you able to come to our feedback meeting? | 14:29 |
bfiller | asac: yes | 14:30 |
asac | cool. cu there | 14:30 |
vila | didrocks: any objection to putting qa-intel-4000 offline for jenkins while stgraber investigates the lxc issue ? (currently in #distro) | 14:42 |
vila | didrocks: done, it's broken anyway, no jobs were running, they can still happily fail on the other slaves ;) | 14:43 |
didrocks | vila: still in meeting, but no objection of course ;) | 14:43 |
vila | didrocks: ack, thanks | 14:43 |
asac | didrocks: dont wait | 14:53 |
asac | have netwowkr issues | 14:53 |
didrocks | asac: we didn't wait if that can reconfort you :) | 15:02 |
asac | didrocks: good.never wait for me | 15:03 |
asac | :) | 15:03 |
didrocks | ;) | 15:03 |
asac | even if you will repeat :) | 15:03 |
cyphermox | Mirv: looking at autopilot--- are you sure that there weren't merges that needed to be forward-ported to trunk from the 13.10 branch? | 15:23 |
elopio | hey, are the jenkins phones going to be flashed with the trusty channel? | 15:27 |
vila | didrocks, asac, Mirv, sil2100: Looks like stgraber found a fix we can apply to otto in lxc.defaults/fstab http://paste.ubuntu.com/6289562/ | 15:28 |
didrocks | vila: excellent, that should entirely the issue? | 15:29 |
vila | didrocks: that's my understanding, roughly the cause was: lxc now supports selinux, checks /sys/kernel/security, couldn't in the otto case, did nothing, the container was still using *host* profiles, messed them up, confusion followed | 15:31 |
vila | didrocks: full version in #distro without any misunderstanding added by me ;) | 15:31 |
sil2100 | ;) | 15:39 |
elopio | fginther: I suppose you would know ^ | 15:40 |
fginther | elopio, that's the plan, but if when we start supporting jobs needing multiple channels, then the image flashed just becomes part of the job configuration so that it flashes the right one | 15:43 |
elopio | fginther: my ultimate question is when can I start using autopilot 1.4. Right now I can run autopilot 1.4 tests on my phone because I flashed it with trusty. | 15:46 |
elopio | do you have an ETA for when will I be able to do it on Jenkins? | 15:46 |
vila | didrocks, sil2100, Mirv: So, I'm going to apply https://code.launchpad.net/~vila/otto/lxc-alpha2/+merge/192347 on qa-intel-4000 and create a new container | 15:48 |
sil2100 | vila: excellent! Let's give it a try ;) | 15:51 |
fginther | elopio, should be working tomorow, I'll get back to you | 15:51 |
vila | sil2100: container creation already going further | 15:51 |
vila | sil2100: do you know who can review that otto patch ? | 15:51 |
elopio | fginther: tomorrow would be more awesome than what I was hoping. Thanks! | 15:51 |
vila | jibel is EOD I think | 15:52 |
vila | sil2100: container creation succeeded, rebooting | 15:53 |
vila | sil2100: do you have a job we can run or re-run there ? | 15:53 |
vila | once I put that node back online that is | 15:53 |
=== gatox is now known as gatox_lunch | ||
didrocks | vila: I'll be late, can you start leading the meeting? | 16:00 |
sil2100 | vila: you mean, to check if it works? | 16:00 |
didrocks | asac: ogra_ sil2100 ^ | 16:00 |
ogra_ | didrocks, and i'll have to leave after a few mins | 16:00 |
vila | sil2100: yup | 16:00 |
ogra_ | since my standup moved 1h later today | 16:00 |
cyphermox | didrocks: I'm at phonedations standup btw | 16:05 |
cyphermox | vila: asac: sil2100: ^ | 16:05 |
* ogra_ will go there now too | 16:05 | |
sil2100 | cyphermox: ACK | 16:07 |
sil2100 | vila: can I use the intel machine for a test build now? | 16:09 |
vila | sil2100: yup, it's good to go | 16:09 |
sil2100 | vila: I mean, to run some otto testing on the machines now - can I? I want to get the list of extra-packages we need to modify, so that we're done when all machines are fixed | 16:09 |
sil2100 | vila: thanks! | 16:09 |
sil2100 | ;) | 16:09 |
* sil2100 is running some jobs that are to fail on purpose | 16:09 | |
vila | sil2100: the missing bit it landing https://code.launchpad.net/~vila/otto/lxc-alpha2/+merge/192347 then I'll update the otto branch there to avoid people getting confused by the actual state | 16:10 |
vila | sil2100: that's for qa-intel-4000, the 2 others hosts need to be fixed, I'll do that asap while it's hot ;) | 16:11 |
sil2100 | didrocks: I'm fixing the extra packages lists now if anythign | 16:14 |
sil2100 | For all stacks | 16:14 |
sil2100 | (head for now only) | 16:14 |
asac | vila: network issues kicked me out | 16:20 |
asac | still having issues | 16:20 |
asac | did didrocks manage to join yet? | 16:20 |
didrocks | asac: did just before you left | 16:21 |
vila | asac: yeah, that bug had very effects on various people ;) Or you were scared by didrocks ;) | 16:21 |
vila | *very weird | 16:21 |
asac | didrocks: will try after another reboot (resetted my whole network now now) | 16:22 |
didrocks | ogra_: so green light to start build #4 | 16:25 |
ogra_ | didrocks, will do after the meeting | 16:25 |
ogra_ | didrocks, i just said (before i left and you joined i guess) that instead of enabling cronned builds again, i will just fire a build at the end of my day every day | 16:26 |
ogra_ | so we have something to look at in the morning | 16:26 |
ogra_ | and can still have a fine grained view on changes | 16:26 |
cjwatson | Why not just cron it for the end of your day every day? :) | 16:29 |
ogra_ | cjwatson, well, i saw the crontab and didnt really want to be the only entry in there :P | 16:30 |
asac | ogra_: doesnt need every day. since we seem to resume CI landings, we will have images kicked on demand all thet ime from tomrrow | 16:30 |
asac | so its just about today | 16:30 |
ogra_ | ok | 16:30 |
cjwatson | ogra_: the rest will be turned on soon enough | 16:30 |
ogra_ | cjwatson, right, and i thought touch can just go with that | 16:31 |
cjwatson | ogra_: just getting the installer working first, around everything else | 16:31 |
ogra_ | yeah | 16:31 |
cjwatson | ogra_: it'll probably be tomorrow or Friday | 16:31 |
ogra_ | k | 16:31 |
cjwatson | I did another chunk of it today | 16:31 |
ogra_ | well, as soon as we do CI landings again and the new process hasnt been discussed i think we'll just stay on manual | 16:31 |
* ogra_ was slapped already today for blocking packages in proposed | 16:32 | |
cjwatson | as you like, just seems odd to be playing human-cron | 16:32 |
cjwatson | ogra_: you were? didn't see any block hints from you | 16:32 |
ogra_ | well, i'm dont that since a few montjhs now :) | 16:32 |
ogra_ | cjwatson, i had to remove them again | 16:33 |
ogra_ | people complained | 16:33 |
cjwatson | oh, I'm looking in the wrong branch | 16:33 |
ogra_ | rev 51-53 | 16:33 |
=== gatox_lunch is now known as gatox | ||
didrocks | sil2100: robru: otto ready for all machines \o/ | 16:58 |
didrocks | thanks vila for tracking ;) | 16:58 |
=== alan_g is now known as alan_g|EOD | ||
sil2100 | \o/ | 17:02 |
sil2100 | didrocks: extra packages lists almost all updated | 17:02 |
sil2100 | I don't have one for platform yet, as mir needs rebuilding, but we'll get to that later | 17:03 |
didrocks | sil2100: ok, QA seems to still need (it's the one I used for tests) | 17:04 |
didrocks | but you probably already updated it | 17:04 |
sil2100 | didrocks: I didn't redeploy the stacks yet, but I have a branch pushed and soon ready for review | 17:05 |
didrocks | ah great! | 17:06 |
sil2100 | didrocks, robru: https://code.launchpad.net/~ubuntu-unity/cupstream2distro-config/updating_extra_packages_for_T/+merge/192364 <- could you guys take a look? I'll redeploy in the meantime | 17:08 |
sil2100 | Platform I only modified what I could | 17:08 |
didrocks | sil2100: I'll let have robru having a closer look ;) | 17:12 |
sil2100 | I'm redeploying in the meantime :) | 17:12 |
robru | didrocks, is there any technical reason those lines can't wrap? if so, can we work on eliminating it this cycle? | 17:13 |
didrocks | robru: not sure if yaml accept wrapping? do you know? | 17:13 |
didrocks | robru: if you can get something understandable to yaml and the deploy stack command, I'm happy :) | 17:14 |
robru | didrocks, why wouldn't it? we wrap in other places. all this time i've simply assumed that cu2d itself was being brittle about the wrapping here | 17:14 |
didrocks | robru: so, what I do for deploying is: | 17:15 |
didrocks | import yaml | 17:15 |
didrocks | yaml.load(this file) | 17:16 |
didrocks | then I have a dictionary | 17:16 |
didrocks | if this is working with wrapping, I'm all in favor of it :) | 17:16 |
robru | didrocks, ok, i am going to try that locally in an interactive session and see what i find... | 17:17 |
didrocks | yeah ;) | 17:17 |
renato_ | fginther, hi | 18:17 |
renato_ | fginther, could you take a look on these MRs: https://code.launchpad.net/~renatofilho/address-book-app/dynamic-backend/+merge/192326 | 18:18 |
renato_ | and https://code.launchpad.net/~renatofilho/ubuntu-ui-toolkit/fix-1236464/+merge/190496 | 18:18 |
renato_ | jenkins is falling for some strange reason | 18:18 |
ogra_ | === Build of image #4 running === | 18:28 |
fginther | renato_, https://code.launchpad.net/~renatofilho/address-book-app/dynamic-backend/+merge/192326 failed because the armhf build node failed, it's been re-approved | 18:29 |
fginther | renato_, https://code.launchpad.net/~renatofilho/ubuntu-ui-toolkit/fix-1236464/+merge/190496 has a couple of failures. The maguro failure appears to be transient, possibly networking related. I've seen it a few times now since we switched to trusty, I'll try a workaround for that. | 18:31 |
fginther | renato_, the mako failure is more strange. It looks like the autopilot tests finish (with some failures) but the results don't get recorder. Will investigate | 18:32 |
renato_ | fginther, thanks | 18:32 |
plars | thomi: did josepht or doanac ping you about autopilot not showing the new skips in the camera-app tests? | 21:05 |
thomi | plars: nope | 21:12 |
thomi | plars: maybe I missed it in the backscroll? | 21:12 |
josepht | thomi: you didn't, we haven't | 21:13 |
thomi | ok then :) | 21:13 |
josepht | thomi: autopilot needs to include 'skipped="N"' when testcases are skipped | 21:14 |
josepht | thomi: in the <testsuite> element | 21:14 |
thomi | josepht: in junitxml output format? | 21:14 |
josepht | thomi: yes | 21:14 |
thomi | it doesn't already!? | 21:14 |
josepht | thomi: no sir | 21:15 |
thomi | huh | 21:15 |
josepht | http://jenkins-dev-image-test:8080/job/trusty-touch_mir-mako/2/artifact/clientlogs/camera_app/test_results.xml | 21:15 |
thomi | can you file a bug please? | 21:15 |
josepht | thomi: sure | 21:15 |
thomi | it does actually skip the tests though, right? | 21:15 |
josepht | thomi: I'm not sure | 21:15 |
thomi | I'd be very surprised if it was that broken | 21:16 |
josepht | I'd say it is skipping them since they're not failing :) | 21:16 |
josepht | thomi: #1243928 filed | 21:20 |
thomi | cool | 21:24 |
thomi | that's an interesting bug, since it's not actually autopilot that formats that output... | 21:25 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!