/srv/irclogs.ubuntu.com/2013/10/23/#ubuntu-ci-eng.txt

didrockshey Mirv, how are you?06:08
Mirvdidrocks: hey, fine. I've been fixing 10+ prepare jobs this morning for head/saucy, but there's still work to do06:10
didrocksMirv: sil2100 didn't finish it and didn't email you?06:10
Mirvdidrocks: I don't think he worked on those, I saw the same prepare jobs yellow as yesterday06:11
Mirv(no e-mail)06:11
* didrocks wonders why fixing prepare jobs is taking so long.06:11
didrocksMirv: urgh :/06:11
didrocksok, I'll talk with him06:11
didrocksI'm looking at why the jobs FTBFS06:11
Mirvdidrocks: oh, I started using the Stack status for trusty and filed first FTBFS bugs https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0AuDk72Lpx8U5dHFtUmlPOUtCRk8zR2dtaEpIbUVhMmc#gid=306:12
didrocksMirv: ah, excellent! looking :)06:12
* didrocks adds https://bugs.launchpad.net/ubuntu/+source/mediascanner/+bug/124353606:16
ubot5Ubuntu bug 1243536 in mediascanner (Ubuntu) "mediascanner FTBFS in trusty" [Undecided,New]06:16
didrocksand https://bugs.launchpad.net/autopilot-gtk/+bug/124353806:19
ubot5Ubuntu bug 1243538 in autopilot-gtk "autopilot-gtk FTBFS on trusty" [Undecided,New]06:19
didrocksjibel: hey!06:21
Mirvdidrocks: the prepare jobs are now fast to fix (I'm now at ~15/20), there was just so much to do before that. on Monday I started with the thought there's just small bits needing to be done, while actually most of the transition was not done at all.06:26
didrocksMirv: urgh, ok… I hope we'll get better at next release opening06:26
didrocksMirv: there are some in saucy as well it seems06:27
didrocksMirv: I'm pocking people to get their FTBFS fixed06:27
Mirvdidrocks: yeah.. as I was away for a few days, and you mentioned something along the lines that it's about done but sil missed a few, I was under wrong impression. during Mon & Tue then the actual transition was done with me + sil + fginther.06:28
Mirvdidrocks: yeah, I've fixed around 8/14 of the saucy prepare job problems now06:28
didrocksMirv: ok, good luck :)06:29
didrocksI'm more wondering why sil2100 didn't email/continue yesterday06:29
didrocksbecause the FTBFS were not handled06:29
didrocksnor the prepare jobs06:29
didrocksnor the -check issue (no access to archive.ubuntu.com)06:29
didrocksvila: did you get any email? this is blocking us06:31
didrocksvila: from magners-orchestra, I can't ping archive.ubuntu.com06:31
didrocksunfortunately, as sil2100 didn't share any info (I guess he pinged retoaded privately), we don't know where we stand06:31
didrocksasac: FYI ^06:31
viladidrocks: can't ping from m-o, filtered by the IS firewall, I use wget for that kind of tests, and wget archive.ubuntu.com works07:01
viladidrocks: so, what's the issue ?07:01
didrocksvila: http://10.97.0.1:8080/job/autopilot-trusty-daily_release/9/label=autopilot-nvidia/console for instance07:02
didrocks/var/log/upstart/otto-setup.log:   Could not resolve 'archive.ubuntu.com'07:03
didrocksvila: seems to work from outside the container07:03
cjwatsonthat doesn't sound like a firewall problem - you wouldn't be getting a DNS failure in that case07:03
didrocks(using wget)07:03
sil2100Hi07:03
cjwatsonmore likely busted resolv.conf or something07:03
sil2100didrocks: retoaded had this to say (I though he was fixing that):07:03
sil2100<retoaded> sil2100, looking through the console output in the job then looking at dx-autopilot-nvidia it appears that the LXC network bridge (lxcbr0) doesn't have an interface assigned so it doesn't have a network path outside if itself.07:04
sil2100<retoaded> nor are there any iptables rules in place to route the network traffic from lcxbr0 to eth007:04
sil2100This was close to my EOD07:05
didrockssil2100: hum, weird, something changes on the machine, we got a successful snapshot (and it was the raring machines)07:05
sil2100hmm07:06
didrocksjibel: do you remember we are doing any bridge manual setup on the hosts for otto? ^07:06
didrockssil2100: btw, Mirv is finishing the prepare jobs, (apparently he didn't find any branches)07:06
viladidrocks: urhg :-(07:07
jibeldidrocks, we don't do anything the bridge is handled by lxc in the upstart job lxc.conf07:07
* didrocks continues on libaccounts-glib FTBFS07:07
didrocksmaybe a regression from latest lxc?07:07
didrocksvila: are you on it? ^07:07
viladidrocks: not for now, I'm on upstream-merger for trusty but I can switch if needed07:08
jibellxc-net.conf actually07:08
viladidrocks: which also involve otto and a container so I may run into the same issue...07:09
didrocksvila: ok, so upstream-merger is blocked as well?07:09
viladidrocks: for trusty yes (unless fginther did something I'm not aware of)07:09
Mirvsil2100: I noticed that you had two branches, but there were 17 prepare job problems all in all - now fixed07:10
didrocksMirv: all fixed? \o/ (saucy as well?)07:10
viladidrocks: as for cu2d on phones, I've seen MPs from doanac but no firm confirmation is was ready to be plugged...07:10
didrocksvila: ok, keep us posted once he's online. I guess the priority is upstream-merger/otto, while we are working with upstream to fix FTBFS07:11
viladidrocks: /me nods07:11
Mirvdidrocks: yep, saucy as well07:11
didrocksMirv: thanks a lot! seems it wasn't that long after all ;) not sure why sil2100 wasn't able to finish it07:12
didrocks(the accounts-glib FTBFS is really puzzling)07:12
viladidrocks: still, for my sanity/understanding, http://10.97.0.1:8080/job/autopilot-trusty-daily_release is a step (or several) further than the setup we did with you jibel for otto ?07:12
didrocksvila: it's the otto job, the one running the tests07:13
didrocksbut it can't get to archive.ubuntu.com for any reason inside the container07:13
viladidrocks: I thought we were able to create a container which includes an apt-get update right ?07:13
didrocksso the tests are failing07:13
didrocksvila: right, on Monday07:13
viladidrocks: so that's a regression right ?07:13
didrocksvila: sounds like it07:13
viladidrocks: ok, thanks07:13
* vila restores a tiny bit of sanity07:14
viladidrocks: could it be something similar to the [[:space:]] issue encountered on the phone yesterday ?07:14
ogra_hmm, was the dashboard CSS updated or sis my browser just decide to use a completely different font on its own  ?07:16
didrocksvila: maybe07:17
ogra_s/sis/did/07:17
ogra_system-image-cli -c trusty-proposed -v07:19
ogra_...07:19
ogra_[systemimage] Oct 23 09:19:13 2013 (22849) Already up-to-date07:19
ogra_HMPF07:19
ogra_root@ubuntu-phablet:/# system-image-cli -i|grep channel07:20
ogra_channel: saucy-proposed07:20
ogra_that doesnt look right07:20
didrocksogra_: there are multiple files for the channels07:21
didrocksogra_: at least 207:22
didrocksso I won't be surprised there is a bug somewhere07:22
ogra_didrocks, well, the above should switch me to trusty-proposed07:22
ogra_according to the docs07:22
didrocksogra_: right, maybe it's just updating the wrong file :)07:22
ogra_and i know it worked for pat mcgowan yesterday07:22
didrocksah, interesting07:22
didrocksRO image?07:22
ogra_oh, wait07:22
ogra_i tested something ...07:22
* ogra_ checks for writable_image07:23
ogra_right, its rw07:23
* ogra_ removes and reboots07:23
didrocksok, was worth looking…07:23
ogra_doesnt change even in ro07:24
sil2100Mirv: there were more branches, I pasted like 8 branches to cyphermox before going EOD07:25
* ogra_ phablet-flash'es .... hoping his data wont be wiped07:27
sil2100didrocks: I pasted all the branches that needed merging to cyphermox, and I see 2 of those didn't merge in because of merger failures07:27
didrockssil2100: branches that you handled?07:28
didrocksargh, so duplicate work because of no email/communcation :/07:28
sil2100hm, could be, I thought Mathieu handled those as I pasted those to him - but let me check what's the status with that07:29
sil2100Mirv: which branches did you have to fix? As I fixed all yellow prepare jobs that I saw yesterday, at least preparing merges for those from the Trusty side07:31
didrockssil2100: btw, any opinion on #ubuntu-unity?07:35
didrocksvila: ok, seems 1.0.0~alpha2 is guilty07:39
didrocksdowngraded to alpha1 and no issue07:39
didrocks(lxc)07:39
viladidrocks: lxc on the host right ?07:43
didrocksvila: right07:43
didrocksthen, there is another issue I'm debugging07:44
didrocksaptsources.distro.NoDistroTemplateException: Error: could not find a distribution template for Ubuntu/trusty07:44
didrockshum, lsb-release is fine though07:44
vilagood, the use case for upstream-merger is slightly different as we use saucy on the host so I may be immune, will know soon07:44
didrocksvila: I'm going to hold lxc upgrade07:45
viladidrocks: rings a bell, heard something I didn't followup closely about patching distro-data manually :-/07:45
didrocksvila: well, ubuntu.csv contains 14.04 LTS,Trusty Tahr,trusty,2013-10-17,2014-04-17,2019-04-1707:46
* vila sighs07:47
viladidrocks: something else then :-/ Sorry07:47
ogra_sigh07:47
ogra_so switching to trusty with pahbelt-flash works, but all installed apps are gone07:48
ogra_thats annoying :/07:48
wgrantdidrocks: python-apt-common needs updating07:48
vilaogra_: I got that even without switching (i.e. image 101)07:48
didrockswgrant: ah, that's the one! thanks :) I can do it07:49
vilaogra_: but I had to redo the restoring part of the backup manually (adb got stucked but I copied the backup.tar.gz during phablet-flash, lucky me)07:49
wgrantAmusingly there's a new one stuck in proposed because bzr-builddeb autopkgtest failed.07:50
didrocksah, indeed :)07:51
ogra_vila, well, by the looks of it it didnbt even do a backup, it just didnt touch my homedir07:51
vilaogra_: so it failed to restore it , didn't phablet-flash hang ?07:52
ogra_nope07:52
ogra_just went through07:52
ogra_i also didnt get a full image ... which i thought i was supposed if i swithc channels07:52
* ogra_ will wait for stgraber 07:53
Mirvsil2100: right, I only saw the unmerged ones. so also fix needing in trusty were libcolumbus, autopilot, mtp, compiz, and in saucy some 10+ more07:53
Mirvdidrocks: can you redeploy head unity8 + mir for the unity-mir move? I can't deploy the latter since I don't have rights to lightdm07:59
didrocksMirv: sure, only head, right? nothing to deploy for saucy?08:00
Mirvdidrocks: I didn't touch saucy for those, and mir is already disabled there08:01
didrocksMirv: reminder: when we move/removed things, you need to delete by hand the jenkins job08:01
Mirvdidrocks: I remember that08:01
didrocksMirv: ok, deployed08:01
Mirvthanks, cleaned the unity-mir from unity808:02
didrocksthanks ;)08:02
sil2100Mirv: right, saucy! I only posted the trusty merges to cyphermox it seems!08:04
didrockssil2100: can you relaunch the jobs where the prepare jobs failed?08:05
didrocksjust to ensure we are all good on that one at least08:05
sil2100Mirv: sorry about that, we duplicated work because of that08:05
sil2100didrocks: the saucy failures you mean? Sure08:05
Mirvsil2100: I don't think there was any duplicates beside the autopilot + ui-toolkit, since you didn't start on the saucy, plus the trusty ones I did didn't have branches from you?08:06
MirvFYI unity-system-compositor was missing from head before didrock's redeploy. I'll now compare all stacks side-by-side head vs. saucy and clean up eg. removed jobs08:06
sil2100didrocks: it seems all of those stacks were already ran and are now green for saucy \o/08:08
sil2100didrocks, Mirv: what should we do with stacks that have no components in them anymore?08:09
sil2100(in saucy)08:09
sil2100Should we remove those altogether?08:09
didrockssil2100: and head?08:09
didrockssil2100: if you can remove them from config + jenkins, I guess that will help clarifying, yeah08:10
didrocksthanks Mirv, sil210008:10
sil2100didrocks: clean as well08:10
didrocksgrrr, I can't reproduce the bzr-builddeb failure on saucy, even upgrading latest python-apt08:11
didrockswill need a trusty chroot I guess08:11
sil2100didrocks, Mirv: ok, I'll clean up the empty stacks then, as we see now that more or less things are working right now08:11
sil2100didrocks: !08:12
sil2100didrocks: I'm just checking my mail and I got a mail from Larry about the lxc issues, let me take a look08:12
sil2100didrocks: hah, let me forward it to you08:13
cjwatsondidrocks: what was the lxc failure?08:13
Saviqdidrocks, any word on unity-mir and lxc-android-config release?08:13
sil2100didrocks: it seems the issue is a bit bigger08:13
didrockscjwatson: I didn't debug it yet, just downgraded, I'm on the bzr-builddeb autopkgtests failing which is blocking otto as well08:13
didrocksbut hard to reproduce, can't start in the isolated env (following the doc from http://developer.ubuntu.com/packaging/html/auto-pkg-test.html#executing-the-test) but http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img doesn't exist yet08:14
sil2100didrocks: forwarded the e-mail, seems like retoaded forwarded the problem to Larry and he made an investigation08:14
didrocksnot sure what adt is using then08:14
didrocksSaviq: we need first to be able to run the test infra08:14
cjwatsonIIRC adt-virt-lxc isn't actually packaged yet08:14
didrocksSaviq: and trusty seems not ready for it yet, so working on it08:14
Saviqdidrocks, :/08:14
Saviqdidrocks, k thanks08:14
didrocksok, I think I have to upgrade to trusty to hope reproducing the bzr-builddeb autopkgtests issue08:15
didrocksat least, the 3 errors seems to have a common ground08:16
didrocksthe first one seems more hectic08:17
didrocksvila: getting any progress on your side on the upstream-merger one?08:18
Mirvok head is up-to-date, with the only exception being cordova-ubuntu which will need some more info before adding it - ie. is the 3.0 branch ready to be released (I think not, there are among else hard-coded reference to 2.8 in the SDK)08:19
* cjwatson would like to make it clear that the fact that you're blocked on python-apt in trusty is another casualty of Mark being slow to announce the name08:19
sil2100didrocks: it anyway seems that the DNS issues we're encountering are some strange apparmor issues08:19
cjwatsonIf we'd had the new name in a timely fashion then we could've had support for it in saucy08:20
sil2100Mirv: yeah, I saw cordova-ubuntu trunk when I was doing the splitting and didn't see a debian/ directory then even08:20
sil2100Not sure how it is now08:20
Mirvsil2100: I'm sure the webapps team will let us know when they want it - they've promised that they'll also update the sdk at that time08:21
didrockscjwatson: I think it's clear to all of us TBH08:21
didrockscjwatson: I'm trying to look at what exactly broke in lxc after the meeting08:22
cjwatsondidrocks: Yeah, I just wanted to hammer it home :)08:22
didrocks;)08:22
didrocksTBH, if we get everything running (including CI) less than a week after the release name was given, it's already an achievement TBH08:23
didrocksdiverging all branches, and so on…08:23
didrockswe are not rolling, so we can't have a "no time interruption" as if we had it. I think people should understand that08:23
cjwatsonI agree, but there's always pressure to be faster and I really resent having to break promises I've given because of something as simple as a name08:24
viladidrocks: almost ready to do a run but I lack some context from fginther :-/08:24
cjwatsonAnd really, if we didn't have that blocker as we've had for the last three cycles, we could be open again almost immediately08:24
didrocksoh 3? I was thinking it was only 2…08:25
cjwatsonThree08:25
cjwatsonWe got raring's name about a day before quantal released, which was too late to add proper support to quantal08:25
cjwatsonAnd even the cycle before that, we only had three days' notice, which wasn't enough08:26
cjwatsonThe cycle before that (so precise's name) we had eight days' notice, which was still less than I'd like but it was manageable08:26
cjwatsonThe cycle before *that* we had 52 days' notice08:26
sil2100Mirv: right08:26
cjwatsonI don't think I'll be on this morning's standup, I've had five hours' sleep due to kids invading the bed and I don't think I can manage coherent conversation in actual speech08:28
viladidrocks: it failed, that's progress ;) The trusty container doesn't exist, now to find where it's create...08:28
vila*created08:28
didrockscjwatson: no worry ;)08:28
viladidrocks: it failed with an understandable error which is nice08:29
didrocksheh08:29
didrocksvila: joining?08:31
viladidrocks: yup08:31
ogra_popey, mind to test trusty-proposed ? we dont have call/sms tests for that yet08:51
popeyogra_: sure thing08:52
ogra_thx08:52
ogra_i think didrocks tested the rest already08:52
* popey OTA's08:52
ogra_from saucy ?08:52
popeyno, from 14.0408:53
ogra_ah, k08:53
popeyI mean, OTA from saucy would be nice08:53
popeybut I suspect we're not planning to support that?08:53
ogra_bug 1243573, bug 124357708:53
ubot5bug 1243573 in ubuntu-system-settings (Ubuntu) "Timezone setting is gone after upgrade to Ubuntu Touch touchy" [Undecided,New] https://launchpad.net/bugs/124357308:53
ubot5bug 1243577 in system-image (Ubuntu) "All apps are gone after upgrade to trusty" [Undecided,New] https://launchpad.net/bugs/124357708:53
ogra_thats non OTA though ...08:53
popeyyeah, i flashed clean trusty-proposed on monday iirc08:54
popeymaybe tues08:54
ogra_i tried to switch the channel from cmdline using system-image-cli -c ... but that didnt work :(08:54
ogra_it pulled the channel properly from what i saw with -v but told me i'm up to date08:54
=== vrruiz_ is now known as rvr
popeyogra_: sms and calls work fine in both directions09:02
ogra_awesome, thanks09:02
ogra_didrocks, asac ^^^09:02
didrocksogra_: please promote then! :)09:07
didrockspopey: ^09:07
ogra_will do09:07
ogra_(or try to, not sure if the script behaves ... first trusty image etc)09:07
asacogra_: pay attention to what might have broken after :)09:15
asaci am sure all phablet-flash will go crazy hehe09:15
didrocksvila: so after a reboot, still nothing (worked on the nvidia machines, not on others FYI): http://10.97.0.1:8080/job/autopilot-trusty-setup_otto/label=qa-intel-4000/6/console09:16
* didrocks will classify that as "unknown other issues"09:16
ogra_asac, i wont switch the devel or devel-proposed channels, thats stgraber stuff ...09:16
* ogra_ will just promote trusty-proposed to trusty09:16
popeyogra_: lemme know when and I'll test reflashing without --no-backup09:17
viladidrocks: makes sense since update_host ... update the host and the container so get the latest lxc no ?09:19
didrocksvila: lxc in the container -> we don't really care09:19
didrocksit's only the host one which is used09:19
viladidrocks: update_host update the *host*09:20
ogra_popey, didrocks, asac trusty 3/20131022.1 promoted09:20
didrocksvila: yeah, but as told in the hangout, I hold lxc09:20
didrocksvila: just try apt-cache policy on the machine to check09:21
didrockslxc:09:21
viladidrocks: I'll start with that once done with u-m09:21
didrocks  Installed: 1.0.0~alpha1-0ubuntu1109:21
didrocks  Candidate: 1.0.0~alpha2-0ubuntu309:21
didrocks  Version table:09:21
didrocks     1.0.0~alpha2-0ubuntu3 009:21
didrocks        500 http://us.archive.ubuntu.com/ubuntu/ trusty/main i386 Packages09:21
viladidrocks: ack09:21
didrocks *** 1.0.0~alpha1-0ubuntu11 009:21
didrocks        100 /var/lib/dpkg/status09:21
didrocksfor instance09:21
didrocksogra_: thanks!09:21
popeyogra_: thanks, will test09:21
viladidrocks: for u-m update_host needs a tweak since it want to create a container for the host release (aka saucy here when I want a trusty container)09:22
didrocksvila: are you using update_host to create the container? not otto directly?09:23
viladidrocks: I'm using nothing for now ;) Looking at update_host for at least getting which parameters otto wants09:23
didrocksasac: status email sent09:27
=== psivaa is now known as psivaa-afk
viladidrocks: container building09:30
viladidrocks: lxc_container: command get_init_pid failed to receive response harmless ?lxc-ls09:33
vilagrrr09:33
popeyogra_: right, so flashing from saucy to trusty wipes /opt/click.ubuntu.com ☹09:33
asacdidrocks: thanks!09:33
popeydo we have a bug for that?09:33
ogra_popey, yup09:33
ogra_popey, bug 124357709:34
ubot5bug 1243577 in system-image (Ubuntu) "All apps are gone after upgrade to trusty" [Undecided,New] https://launchpad.net/bugs/124357709:34
popeyta09:34
viladidrocks: 2013-10-23 05:36:55,304 INFO Starting container 'trusty-amd64-20131023-0529'09:40
vilalxc_container: command get_init_pid failed to receive response09:40
viladidrocks: rings a bell ?09:40
didrocksvila: can be anything meaning that lxc-start didn't work09:41
vilaack, looking into that already, thanks09:42
didrockshttps://bugs.launchpad.net/ubuntu/+source/lxc/+bug/118113609:42
ubot5Ubuntu bug 1181136 in lxc (Ubuntu) "Empty log file when a container is started with the API" [Medium,Confirmed]09:42
viladidrocks: great :) subscribed... read.... :-/09:43
didrocksvila: there was another one I can't find right now about no log at all in stdout when using the python bindings09:45
viladidrocks: seems to be related to the pre-start.sh indeed:09:46
vila+ [ -e /sys/fs/cgroup/memory/lxc/memory.use_hierarchy ]09:46
vila+ echo 109:46
vilalxc-start: Script exited with status 109:46
didrocksagain the same issue :/09:46
vilawhic is weird since that's commented out09:47
didrocksvila: is it really? we reverted IIRC09:47
viladidrocks: let me check on dx-autopilot-intel, on ps-radeon-hd8350 otto is up to date with lp:otto09:48
didrocksvila: dx-autopilot-intel is a saucy machine, right?09:48
vilaroght09:49
viladidrocks: ~ubuntu/otto is up to date, but ~jenkins/otto is not, stooopid vila09:57
didrocksvila: you ps-radeon-hd8350?10:03
viladidrocks: http://10.97.0.26:8080/job/autopilot-testrunner-otto-trusty/9/console so, tests are running but seem to be failing10:03
didrocksvila: some are passing, so sounds good10:03
didrocksvila: what stack is it?10:03
viladidrocks: suspicious messages /var/log/syslog: Oct 23 10:01:36 trusty-amd64-20131023-0529 kernel: [60980.789436] [drm:radeon_dvi_detect] *ERROR* DVI-I-2: probed a monitor but no|invalid EDID10:04
viladidrocks: http://10.97.0.26:8080/job/autopilot-testrunner-otto-trusty/9/parameters/?10:05
viladidrocks: so unity8 IIUC10:05
viladidrocks: I used the same parameters as fginther did on the previous runs10:06
didrocksvila: making sense for unity8 to fail10:06
viladidrocks: I think I'll stop on that one as I've already far too many unknowns to check with fginther10:06
vilawill look at qa-intel-4000 instead10:07
didrocksvila: ok, thanks10:08
viladidrocks: clarification, when you say 'I hold lxc' you mean on the host or in the archive ? I think we were having two discussions intermixed, me talking about ps-radeon and you talking about qa-intel.10:21
viladidrocks: on qa-intel the container creation is failing so my question is: can use update_host there or will that bring the latest lxc back ?10:22
viladidrocks: or will I get the same absence of usable error message by running update_host manually :-/10:22
didrocksvila: i hold the package on previous lxc on the 3 machines10:26
vilawell, lxc not updated and same messages...10:26
viladidrocks: right, how do you achieve that ?10:26
didrocksvila: apt-mark hold <binary_package>10:27
viladidrocks: thanks10:27
didrocksyw10:27
* vila breaks for lunch10:33
sil2100didrocks, vila: how's progress with otto? Any luck?10:40
didrockssil2100: I guess it's for vila10:54
vilasil2100, didrocks: I don't think it solved itself during my lunch but I'll know that soon ;)11:20
sil2100;)11:21
vilaright, still the same, the container creation fails... with not a single msg when done from update_host, I need to drill down11:22
Mirvmeanwhile another round of autobuilds is progressing nicely11:24
sil2100Mirv: were you able to contact anyone from upstream for https://bugs.launchpad.net/libfriends/+bug/1243527 ?11:25
ubot5Ubuntu bug 1243527 in libfriends "libfriends FTBFS on trusty" [Critical,New]11:25
Mirvsil2100: didier pinged robru about it11:27
Mirvsil2100: (as it says on the stack status page)11:30
Mirvsil2100: robert will ping ken in the morning11:30
sil2100Mirv: ah! Right, it's been such a long time since we used that, I need to get my habits back11:31
Mirvdidrocks: I was finally able to do a proper retrace, it seems apport has various bugs in handling non-English locale that don't even show up as such. bug #1243665 "QUbuntu: Could not create application instance" so might be even mir or such problem11:34
ubot5bug 1243665 in qtdeclarative-opensource-src (Ubuntu) "qmlscene crashed with SIGABRT in raise()" [Undecided,Confirmed] https://launchpad.net/bugs/124366511:34
vilaright, so apparmor DENIED on host's /var/log/syslog during lxc-create AFAICS http://paste.ubuntu.com/6288480/11:34
viladidrocks: you did reboot after downgrading lxc right ?11:36
didrocksvila: yeah, on 2 of the 3 machines11:39
didrocksbut as it's failing on 211:39
viladidrocks: I'm on qa-intel-4000 for now, pinged stgraber in #ubuntu-server11:40
viladidrocks: the denied are for operation="mount" on /dev/input  /dev/dri but the weird thing is that the pid (16121) is the same for all creations but I can't find that process in 'ps fax'11:42
viladamn, re-trying again, the apparmor msgs don't come back...11:47
=== alan_g is now known as alan_g|lunch
viladidrocks: no log files under /var/log/lxc for any container I tried to create, the last one if for trusty-i386-20131023-0008 but has been updated 10 minutes ago ???11:58
didrocksvila: you are trying to create with otto?12:10
didrocksor lxc-create directly?12:11
vilaotto12:13
vilahoping to find some msgs somewhere and avoid issues reproducing the way otto calls lxc-create, will try to do that now12:14
didrocks*sigh* I pointed to https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/118113612:15
ubot5Ubuntu bug 1181136 in lxc (Ubuntu) "Empty log file when a container is started with the API" [Medium,Confirmed]12:15
didrocksvila: that why I pasted that bug ^12:15
viladidrocks: for the lxc log itself, doesn't mean nothing can be found elsewhere12:16
didrocksvila: what other non lxc logs you were expecting to get in /var/log/lxc?12:17
viladidrocks: no idea, that 's why I looked and it's still weird that an old log file modification time changes, but I won't dig why as I doubt it's related12:18
=== psivaa-afk is now known as psivaa
viladidrocks: using otto directly doesn't provide more output so I need to drill down to the lxc-create command but otto "prepares" a lot of stuff first (which are relevant is still unclear...)12:32
viladidrocks: otto -d gives some hint but still fails too quickly.... that rings a bell12:34
vila<container:209 - MainThread> 2013-10-23 12:32:59,910 INFO Container 'trusty-i386-20131023-1216' started12:35
vila<commands:172 - MainThread> 2013-10-23 12:32:59,911 ERROR An error during creation occurred, trying to cleanup the container: The container didn't st12:35
vila910 started, 911 ERROR, that's a little too fast12:35
didrocksvila: the issue is clearly in lxc creation12:36
didrocksand as per bug I posted above, you won't get any info from the container in lxc12:36
didrocksdue to the python bindings12:37
didrocksso you need to start lxc-create by hand12:37
viladidrocks: which otto doesn't display nor the context it requires either as per   This method creates a new container from scratch. We don't want to use12:37
vila        the create method from LXC API because the bootstrapping phase is not12:37
vila        required and replaced by using an Ubuntu image directly.12:37
didrocksvila: otto can't display the lxc logs as per bug :/12:38
didrocksvila: it's call lxc-start12:38
didrockscalling*12:38
viladidrocks: but it could display which lxc-create command it want to execute or are you .... no lxc-create ??12:38
vilaat all ?12:38
didrocksno lxc-create, lxc-start on the container12:39
didrocksso a classic lxc-start -n <last_container_name_without_broken._prefix>12:39
didrocksso you look at what container it created last12:39
didrocksyou mv broken.<container_name> <container_namer>12:39
didrocksand sudo lxc-start -n <container_name>12:40
vilapfew...... mv broken. good to get the context creted by otto, I see12:41
didrocksvila: broken. is to not reuse the broken container12:42
didrocksbut not removing it12:42
didrocksso that you can still debug12:42
didrocksand you stay on latest good container12:42
vilayeah, got that, confused because the last debug session I saw was Mirv extracting the delta, different case12:43
viladidrocks: and if I haven't read lxc-create when you wrote lxc-start earlier....12:44
viladidrocks: lxc-start succeeds....12:45
didrocksvila: well, the second time, I wrote lxc-create TBH (because of broken mind)…12:45
viladidrocks: nevermind, let's go ahead and fix that crap if we can without stgraber help12:46
viladidrocks: hmm, the container just logged me out and terminate12:47
didrocksbecause it finished its upgrade12:47
viladidrocks: now I'm confused. If it starts properly and halt properly, what are we searching for ?12:48
didrocksvila: are you sure it's doing everything properly?12:48
didrocksif it didn't exit 1, yeah, there is an issue12:48
didrocksif you are starting the right container12:49
viladidrocks: no, not even sure I started with good one, so let me re-start that from scratch, update_host, use that one12:49
viladidrocks: http://paste.ubuntu.com/6288876/12:56
didrocksvila: it exits 1, right?12:57
viladidrocks: lxc-start ? 25512:58
didrocksvila: so… not 0, hence the command failing I guess12:58
viladidrocks: but it's not the same behavioras when run from otto (,910 -> ,911 earlier) I'd say12:59
vilabehavior as12:59
didrocksweird12:59
vilanot a spanish variant ;)12:59
vilaISOID=ubuntu_14.04_lts_trusty_tahr_release_i386_20131021.113:00
viladidrocks: expected ? nothing fresher ?13:00
didrocksvila: you can check yourself at http://cdimage.ubuntu.com/daily-live/current/ :)13:01
didrocks21-Oct-2013 13:5713:01
didrockswhich seems to match13:01
viladidrocks: yeah, thanks, sorry, memory overflow ;)13:01
cjwatsonvila: we haven't re-cronned yet - that was a manual build to let didrocks get otto working13:02
viladidrocks: so, for this specific lxc-start we're booting from the iso and creates rootfs no ?13:02
didrocksvila: yeah, we did on Monday13:03
vilacjwatson: ha, good, thanks ! Yeah heard about that, now I *see* it ;)13:03
viladidrocks: ha, no, bases/<something>13:06
viladidrocks: is that correct ? first lxc-start creates the root fs under <lxc-name>/bases ?13:09
viladidrocks: and the others lxc-start will use a snapshot on top of that right ?13:10
didrocksvila: right13:12
fginthermorning13:13
viladidrocks: nothing weird in the logs inside the container :-/13:14
viladidrocks: will create yet another and look under bases/ before starting13:15
vilafginther: hey ! Let's sync about trusty for u-m in a few minutes13:15
fginthervila, ack. let me catch up a moment first13:16
vilafginther: perfect13:16
viladidrocks: ha ! no logs (except for faillog and lastlog)13:18
* didrocks in meetings13:18
vilaso lxc-start failure is where I'll search next13:19
viladidrocks: yeah, no worries, thanks for saying13:19
=== alan_g|lunch is now known as alan_g
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
asacbfiller: you able to come to our feedback meeting?14:29
bfillerasac: yes14:30
asaccool. cu there14:30
viladidrocks: any objection to putting qa-intel-4000 offline for jenkins while stgraber investigates the lxc issue ? (currently in #distro)14:42
viladidrocks: done, it's broken anyway, no jobs were running, they can still happily fail on the other slaves ;)14:43
didrocksvila: still in meeting, but no objection of course ;)14:43
viladidrocks: ack, thanks14:43
asacdidrocks: dont wait14:53
asachave netwowkr issues14:53
didrocksasac: we didn't wait if that can reconfort you :)15:02
asacdidrocks: good.never wait for me15:03
asac:)15:03
didrocks;)15:03
asaceven if you will repeat :)15:03
cyphermoxMirv: looking at autopilot--- are you sure that there weren't merges that needed to be forward-ported to trunk from the 13.10 branch?15:23
elopiohey, are the jenkins phones going to be flashed with the trusty channel?15:27
viladidrocks, asac, Mirv, sil2100: Looks like stgraber found a fix we can apply to otto in lxc.defaults/fstab http://paste.ubuntu.com/6289562/15:28
didrocksvila: excellent, that should entirely the issue?15:29
viladidrocks: that's my understanding, roughly the cause was: lxc now supports selinux, checks /sys/kernel/security, couldn't in the otto case, did nothing, the container was still using *host* profiles, messed them up, confusion followed15:31
viladidrocks: full version in #distro without any misunderstanding added by me ;)15:31
sil2100;)15:39
elopiofginther: I suppose you would know ^15:40
fgintherelopio, that's the plan, but if when we start supporting jobs needing multiple channels, then the image flashed just becomes part of the job configuration so that it flashes the right one15:43
elopiofginther: my ultimate question is when can I start using autopilot 1.4. Right now I can run autopilot 1.4 tests on my phone because I flashed it with trusty.15:46
elopiodo you have an ETA for when will I be able to do it on Jenkins?15:46
viladidrocks, sil2100, Mirv: So, I'm going to apply https://code.launchpad.net/~vila/otto/lxc-alpha2/+merge/192347 on qa-intel-4000 and create a new container15:48
sil2100vila: excellent! Let's give it a try ;)15:51
fgintherelopio, should be working tomorow, I'll get back to you15:51
vilasil2100: container creation already going further15:51
vilasil2100: do you know who can review that otto patch ?15:51
elopiofginther: tomorrow would be more awesome than what I was hoping. Thanks!15:51
vilajibel is EOD I think15:52
vilasil2100: container creation succeeded, rebooting15:53
vilasil2100: do you have a job we can run or re-run there ?15:53
vilaonce I put that node back online that is15:53
=== gatox is now known as gatox_lunch
didrocksvila: I'll be late, can you start leading the meeting?16:00
sil2100vila: you mean, to check if it works?16:00
didrocksasac: ogra_ sil2100 ^16:00
ogra_didrocks, and i'll have to leave after a few mins16:00
vilasil2100: yup16:00
ogra_since my standup moved 1h later today16:00
cyphermoxdidrocks: I'm at phonedations standup btw16:05
cyphermoxvila: asac: sil2100:  ^16:05
* ogra_ will go there now too 16:05
sil2100cyphermox: ACK16:07
sil2100vila: can I use the intel machine for a test build now?16:09
vilasil2100: yup, it's good to go16:09
sil2100vila: I mean, to run some otto testing on the machines now - can I? I want to get the list of extra-packages we need to modify, so that we're done when all machines are fixed16:09
sil2100vila: thanks!16:09
sil2100;)16:09
* sil2100 is running some jobs that are to fail on purpose16:09
vilasil2100: the missing bit it landing https://code.launchpad.net/~vila/otto/lxc-alpha2/+merge/192347 then I'll update the otto branch there to avoid people getting confused by the actual state16:10
vilasil2100: that's for qa-intel-4000, the 2 others hosts need to be fixed, I'll do that asap while it's hot ;)16:11
sil2100didrocks: I'm fixing the extra packages lists now if anythign16:14
sil2100For all stacks16:14
sil2100(head for now only)16:14
asacvila: network issues kicked me out16:20
asacstill having issues16:20
asacdid didrocks manage to join yet?16:20
didrocksasac: did just before you left16:21
vilaasac: yeah, that bug had very effects on various people ;) Or you were scared by didrocks ;)16:21
vila*very weird16:21
asacdidrocks: will try after another reboot (resetted my whole network now now)16:22
didrocksogra_: so green light to start build #416:25
ogra_didrocks, will do after the meeting16:25
ogra_didrocks, i just said (before i left and you joined i guess) that instead of enabling cronned builds again, i will just fire a build at the end of my day every day16:26
ogra_so we have something to look at in the morning16:26
ogra_and can still have a fine grained view on changes16:26
cjwatsonWhy not just cron it for the end of your day every day? :)16:29
ogra_cjwatson, well, i saw the crontab and didnt really want to be the only entry in there :P16:30
asacogra_: doesnt need every day. since we seem to resume CI landings, we will have images kicked on demand  all thet ime from tomrrow16:30
asacso its just about today16:30
ogra_ok16:30
cjwatsonogra_: the rest will be turned on soon enough16:30
ogra_cjwatson, right, and i thought touch can just go with that16:31
cjwatsonogra_: just getting the installer working first, around everything else16:31
ogra_yeah16:31
cjwatsonogra_: it'll probably be tomorrow or Friday16:31
ogra_k16:31
cjwatsonI did another chunk of it today16:31
ogra_well, as soon as we do CI landings again and the new process hasnt been discussed i think we'll just stay on manual16:31
* ogra_ was slapped already today for blocking packages in proposed16:32
cjwatsonas you like, just seems odd to be playing human-cron16:32
cjwatsonogra_: you were?  didn't see any block hints from you16:32
ogra_well, i'm dont that since a few montjhs now :)16:32
ogra_cjwatson, i had to remove them again16:33
ogra_people complained16:33
cjwatsonoh, I'm looking in the wrong branch16:33
ogra_rev 51-5316:33
=== gatox_lunch is now known as gatox
didrockssil2100: robru: otto ready for all machines \o/16:58
didrocksthanks vila for tracking ;)16:58
=== alan_g is now known as alan_g|EOD
sil2100\o/17:02
sil2100didrocks: extra packages lists almost all updated17:02
sil2100I don't have one for platform yet, as mir needs rebuilding, but we'll get to that later17:03
didrockssil2100: ok, QA seems to still need (it's the one I used for tests)17:04
didrocksbut you probably already updated it17:04
sil2100didrocks: I didn't redeploy the stacks yet, but I have a branch pushed and soon ready for review17:05
didrocksah great!17:06
sil2100didrocks, robru: https://code.launchpad.net/~ubuntu-unity/cupstream2distro-config/updating_extra_packages_for_T/+merge/192364 <- could you guys take a look? I'll redeploy in the meantime17:08
sil2100Platform I only modified what I could17:08
didrockssil2100: I'll let have robru having a closer look ;)17:12
sil2100I'm redeploying in the meantime :)17:12
robrudidrocks, is there any technical reason those lines can't wrap? if so, can we work on eliminating it this cycle?17:13
didrocksrobru: not sure if yaml accept wrapping? do you know?17:13
didrocksrobru: if you can get something understandable to yaml and the deploy stack command, I'm happy :)17:14
robrudidrocks, why wouldn't it? we wrap in other places. all this time i've simply assumed that cu2d itself was being brittle about the wrapping here17:14
didrocksrobru: so, what I do for deploying is:17:15
didrocksimport yaml17:15
didrocksyaml.load(this file)17:16
didrocksthen I have a dictionary17:16
didrocksif this is working with wrapping, I'm all in favor of it :)17:16
robrudidrocks, ok, i am going to try that locally in an interactive session and see what i find...17:17
didrocksyeah ;)17:17
renato_fginther, hi18:17
renato_fginther, could you take a look on these MRs: https://code.launchpad.net/~renatofilho/address-book-app/dynamic-backend/+merge/19232618:18
renato_and https://code.launchpad.net/~renatofilho/ubuntu-ui-toolkit/fix-1236464/+merge/19049618:18
renato_jenkins is falling for some strange reason18:18
ogra_=== Build of image #4 running ===18:28
fgintherrenato_, https://code.launchpad.net/~renatofilho/address-book-app/dynamic-backend/+merge/192326 failed because the armhf build node failed, it's been re-approved18:29
fgintherrenato_, https://code.launchpad.net/~renatofilho/ubuntu-ui-toolkit/fix-1236464/+merge/190496 has a couple of failures. The maguro failure appears to be transient, possibly networking related. I've seen it a few times now since we switched to trusty, I'll try a workaround for that.18:31
fgintherrenato_, the mako failure is more strange. It looks like the autopilot tests finish (with some failures) but the results don't get recorder. Will investigate18:32
renato_fginther, thanks18:32
plarsthomi: did josepht or doanac ping you about autopilot not showing the new skips in the camera-app tests?21:05
thomiplars: nope21:12
thomiplars: maybe I missed it in the backscroll?21:12
josephtthomi: you didn't, we haven't21:13
thomiok then :)21:13
josephtthomi: autopilot needs to include 'skipped="N"' when testcases are skipped21:14
josephtthomi: in the <testsuite> element21:14
thomijosepht: in junitxml output format?21:14
josephtthomi: yes21:14
thomiit doesn't already!?21:14
josephtthomi: no sir21:15
thomihuh21:15
josephthttp://jenkins-dev-image-test:8080/job/trusty-touch_mir-mako/2/artifact/clientlogs/camera_app/test_results.xml21:15
thomican you file a bug please?21:15
josephtthomi: sure21:15
thomiit does actually skip the tests though, right?21:15
josephtthomi: I'm not sure21:15
thomiI'd be very surprised if it was that broken21:16
josephtI'd say it is skipping them since they're not failing :)21:16
josephtthomi: #1243928 filed21:20
thomicool21:24
thomithat's an interesting bug, since it's not actually autopilot that formats that output...21:25

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!