[00:33] @ci: I'm seeing some errors in the CI jobs for this MP: https://code.launchpad.net/~thomir/autopilot/1.3-fix-processsearcherror/+merge/191312 [00:33] specifically: "ERROR:pbuilderjenkins:Could not execute hook: H05set_package_version" [00:34] any ideas? [00:38] fginther: still around? [00:39] thomi, for a moment [00:39] looking [00:39] thanks. Does the '@ci' thing actually do anything? [00:39] thomi, I don't know what it's supposed to do :-) [00:39] ahhh. ok then. [00:42] thomi, the failures were both caused by bzr branch failing. I restarted the job and it's working now [00:46] fginther: thanks. [00:47] it'd be awesome if that happened automatically :-/ [06:31] o/ [07:42] didrocks: where is that udev landing ask? [07:42] xnox: do you know where the fix for the udev event spam on maguro is? [07:42] asac: I don't remember seeing ogra_ filing one before it was ready [07:43] * asac remembers taht someone said yesterday that it is ready [07:43] but lets check with xnox ... maybe that was a dream [07:50] lool, hey, you filed a bug about utah truncating .crash files did you not? [07:59] asac, not uploaded afaik [08:05] ogra_: why? [08:05] do you know? [08:06] asac, SRU process ? [08:06] lets ask xnox, i definitely didnt see an upload last night for it [08:06] (and am uü since <10min) [08:06] *up [08:07] well, i hoped for a heads up [08:07] ogra_: isnt xnox in EURO timezone? e.g. is he on soon? [08:07] london [08:08] Hi [08:08] so there is a regression? [08:10] Saviq: I did not file a bug; let me check if Steve did [08:10] didrocks: did you see the request to kick another image with session fix? [08:10] lool, if not, we should - and one more about collecting all the .crashes, not just one per project [08:10] ah yes, 99 is out [08:11] we only have one image to go now, we should be careful ;D [08:11] Saviq: there are two sdes [08:11] Saviq: one is apport itself [08:11] Saviq: the other is lab infra [08:11] Saviq: do you know where to file bugs on latter? [08:11] lool, no idea :/ [08:12] lool: you mean rsalveti's one? [08:12] (image 99?) [08:12] yes [08:12] yes [08:12] yeah, excellent to get an email with the info :) [08:12] thanks! [08:12] oh this was still done yesterday http://people.canonical.com/~ogra/touch-image-stats/20131015.2.changes [08:13] yeah [08:13] yes [08:13] lool, didrocks can you guys please kick https://code.launchpad.net/~ps-jenkins/unity8/latestsnapshot-7.83+13.10.20131015.4-0ubuntu1/+merge/191184 ? [08:13] it was with my permission, i'll take all the blame, sorry for not mailing [08:13] lool: do you know how to do that? ^ [08:14] ogra_: I got an email from Ricardo [08:14] so perfect :) [08:14] oh [08:14] cool [08:14] * ogra_ is up since a few mins only [08:14] ogra_: nah, failed testing. filters all udev events which is not nice. [08:14] havent read mail yet [08:14] ;) [08:15] xnox, hmm, but that could definitely gain us a lot of performance ... we couls just revivew hotplug.sh then :) [08:15] *could [08:15] *revive [08:15] *sigh* [08:15] didrocks: yes [08:15] asac, ^^^ no udevd fix [08:16] thomi, fginther: '@ci' is not supposed to be wired to the bot, but I haven't been bothered enough by it yet to fix it myself [08:16] Saviq: building [08:16] * ev goes back to conferencing [08:16] lool, thanks [08:17] xnox: what exactly happened? [08:17] ev: oh right, you are at a conference! emailing you then with 2 top priorities [08:17] ev: do you know who can fix to get autopilot-intel back on? [08:18] this is blocking us [08:18] didrocks: is this a downed machine? How badly is it blocking you / what is it blocking? [08:18] ev: we have to click to stop all jobs hanging and disregard testing [08:19] ev: tried myself to reboot the machine by CDU, no chance [08:19] ev: do you know if retoaded put the nagios in place as discussed last week? [08:19] didrocks: He has munin set up. I'm not aware of any nagios checks (though I have plans to start adding them in) [08:20] nagios wont really help us resolve this though [08:20] ev: it basically blocks validating all desktop related package validation i believe [08:20] ev: ok, so who can work on the autopilot-intel machine today? [08:20] asac: touch as well [08:20] rfowler and retoaded... send them an SMS so they know there is urgency [08:20] didrocks, asac: we could wake Rick up [08:20] and all the jobs are hanging [08:20] we need to restart them by hand [08:20] yes, indeed [08:20] I'll take care of that now [08:20] thanks ev :) [08:21] sure thing! [08:21] asac: udevd daemon needs to learn how to ignore the VSYNC events spam from omapfb. To even avoid wake ups, pitti and I tried to work on the socket filter within udevd, but at the moment it filters out everything. [08:21] ev: i guess we might want to even consider calling him :) [08:21] ev: for the less urgent one, i'll email you so that you can get to the conference [08:21] this sprint is such poor timing :-/ [08:21] right, stepping out [08:21] poor rfowler [08:25] so why cant the nvidia machine run them? [08:29] lool: because it was agreed at some point that not regressing on intel is key [08:30] we can relax that requirement and hope that we dont do anything bad there, but... :) [08:31] well in my experience one of the two is often down [08:31] ogra_: coming? [08:32] it's important for desktop stuff that we test on both [08:32] not for touch stuff though [08:33] ev: ping [08:34] rfowler: pong [08:34] rfowler: didrocks can give you more details of whats going on [08:35] rfowler: thanks for coming in at such a terrible hour [08:35] ev: rfowler: so dx-autopilot-intel don't answer to ping [08:35] (and the jenkins node isn't started) [08:35] rfowler: ds-autopilot-intel seems dead and can't be restarted even when power-cycling from the cdu [08:35] yeah, restarting from the cdu didn't help [08:35] rfowler: this blocks autopilot tests for desktop and is blocking landings [08:35] Saviq, i dont see the sigstop fix in there [08:37] 4:30 am [08:37] rfowler: :-/ [08:37] * rfowler face in palm [08:37] :) [08:38] didrocks: was it running before? [08:38] rfowler: yes, it's running continously [08:38] ogra_, not merged yet [08:39] Saviq, can you please ? [08:39] ogra_, can't, yet - not in mir trunk [08:39] ogra_, we need to land all three in concert [08:39] we need it to fix all the crashing maliit tests [08:39] oh [08:39] ok [08:39] ogra_, should happen around 3pm our time when kgunn's here to ACK a release of Mir [08:40] Saviq, ok, it needs to land today at some point :) [08:40] rfowler: hold on ! It's back [08:41] didrocks: ^ [08:41] \o/ [08:41] ogra_, yeah, I'm making sure it will [08:41] cjwatson: hi [08:41] cjwatson: we have some questions how to handle packages that might go to saucy-updates etc. in the CI standup [08:42] cjwatson: if you have 3 minutes time to talk about that feel free to join... otherwise chat with didrocks [08:42] and discuss details how to go about things [08:43] vila: omg [08:43] vila: soo... what was the deal? [08:43] tracking here: https://wiki.canonical.com/UbuntuEngineering/CI/IncidentLog/2013-10-16 [08:44] asac: be there in a minute, just got to London [08:44] rfowler: sorry about the trouble, I have no idea what happened, I power-cycled it via the cdu as didrocks said he did, he may give more details after the ho [08:45] vila: rfowler: I tried it once, giving 30 minutes between the try [08:45] (to ping it) [08:46] vila, didrocks: we sure this is back up and running? [08:46] didrocks: and you got on the kvm to see if it got stuck at grub right? [08:46] is there anything you can do to confirm with some certainty [08:47] ev: i'm sitting in a shell on the machine [08:47] ev: looking at syslog and munin [08:47] rfowler: I don't really know how to use the kvm (never tried) [08:47] didrocks: unity8 testing looks good, manual testing as well - I'll publish qtubuntu if you don't mind [08:47] sil2100: please do :) [08:47] rfowler: excellent (for certain definitions) [08:48] didrocks: good time to learn :) [08:48] rfowler: leading a meeting, but yeah :) [08:48] vila: is the jenkins node up? [08:48] didrocks: yes, I waited for that before telling rfowler to hold on [08:48] rfowler: please feel free to go back to bed and sleep in. :) Sorry for waking you up, but it did seem like we were very much stuck. [08:49] vila, didrocks: can you please fill out some more details of what went wrong here in the incident report [08:49] didrocks: what probably happened is you cut power and booted and it got stuck at the grub and needed manual input... which you could have done from the kvm [08:49] and your thoughts for the successes, problems, and recommendations sections [08:50] ev: no problem [08:50] rfowler: if you have opinions for those sections, please feel free to fill them out at a more reasonable hour [08:50] I want to make sure we don't end up here again [08:50] rfowler: more than possible, ok, got it, thanks agian! [08:51] ev: I think we need to spread the knowledge in people from your team, but that's a discussion for later [08:51] thanks vila, rfowler, ev [08:51] didrocks: please add that to the incident report [08:51] but I definitely agree :) [08:51] ev: will do, when I have more time [08:52] thanks guys! [08:52] understood [08:52] didrocks: i'll show you how to use it later [08:52] thanks rfowler [08:53] (even if my gut feeling is that people from CI should handle that, not the landing side) [08:53] didrocks: you're right but it's helpful for you to be able to check thing like this too [08:54] yep [08:54] bunch of Oct 16 04:57:18 dx-autopilot-intel kernel: [63563.604695] [drm:i915_hangcheck_el [08:54] apsed] *ERROR* stuck on render ring [08:54] in syslog [08:59] Damn, what happened that Mir is so fast on mako right now?! [08:59] I even started wondering if I don't have SF running, but no..! [08:59] Fast [09:09] didrocks: hey this may be a very silly question, what's the easiest way for me to tell when a specific revno of trunk for a project will be released in an image? [09:09] sil2100, damn,. fix it ! [09:09] i.e. I would like to know when this merge will be available: https://code.launchpad.net/~veebers/ubuntu-keyboard/update_autopilot_emulators/+merge/190319 [09:10] veebers: there is a landing ask for it (ubuntu-keyboard), you will see on the spreadsheet when/if it landed [09:11] ogra_: on it...! [09:11] veebers: do you have specific AP tests that are failing? [09:11] sil2100: the Mir perf fixes I mentionned yesterday [09:12] didrocks: with the current image all the ubuntu_keyboard test are failing as that MR updates them to use updates int he keyboard itself [09:12] veebers: see the landing ask, robru reported that AP tests are still failing with it [09:13] ev, didrocks: see https://wiki.canonical.com/UbuntuEngineering/CI/IncidentLog/2013-10-16 no trace of 2 power-cyclings, only one, mine. But the jenkins master/slave connection was broken, rebooting restored it but it could have been restored more simply (see playbook) [09:13] didrocks: ok, next silly Q :-) can you link me to the landing ask please? [09:14] I grabbed the autopilot tests from trunk and ran them on my device and they all pass for me [09:14] veebers: https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0Au6idq7TkpUUdGNWb0tTVmJLVzFZd0doV3dVOGpWemc#gid=0 [09:14] veebers: see line 259 (req 240) [09:14] didrocks: awesome, much appreciated [09:14] check the comments [09:14] yw! [09:14] vila: are you sure? I did one quite early [09:15] where do you see no trace of 2 power cyclings? [09:15] didrocks: that's what I want to clarify with you after meeting [09:15] didrocks: syslog clearly shows a single reboot this morning, when did you do yours ? [09:15] I guess near 8am [09:15] so 6 UTC [09:16] didrocks: then we should understand what you did because this does not appear in syslog [09:28] in the latest image I have no osk anymore… is that a known bug? [09:37] Hey guys, I still see no mediaplayer app in the scope nor when doing an explicit search - is that normal? [09:38] sil2100: search failing is normal [09:38] sil2100: I also remember it redisappeared, I pinged some people and I didnt find why [09:38] vila: http://paste.ubuntu.com/6244724/ [09:38] sil2100: so might be worth double-checking [09:39] lool: ok, thought it got fixed in the meantime, will double check [09:40] Oct 16 04:57:23 dx-autopilot-intel kernel: [63569.615422] [drm:i915_set_reset_status] *ERROR* render ring hung flushing bo (0x4990a000 ctx 1) at 0x5c [09:40] last message logged ^ [09:41] Oct 16 08:25:53 dx-autopilot-intel kernel: imklog 5.8.11, log source = /proc/kmsg started. [10:21] thostr_: around? [10:22] didrocks: yes [10:22] thostr_: so, for indicator-bluetooth, we need now to follow the SRU process [10:22] didrocks: :( [10:22] thostr_: meaning, having bugs that follow the SRU rules with test cases, regression potential and so on [10:22] thostr_: this is for all components that are common between desktop and touch [10:23] thostr_: can you fix the bugs for indicator-bluetooth? (I think you will have the same with indicator-datetime) [10:23] I'm testing those meanwhile [10:23] didrocks: me, personally? [10:23] thostr_: well, you or the developer [10:23] where are the sru rules? [10:23] the requester of the landing [10:24] thostr_: https://wiki.ubuntu.com/StableReleaseUpdates?action=show&redirect=StableReleaseUpdate#Procedure [10:24] but does that mean we cannot get bluetooth landed any more for 13.10? [10:24] thostr_: you can assess the impact on desktop as well [10:24] thostr_: if you do that, we can get it for the touch image [10:24] ok, will do [10:24] basically, after my testing, it will just pend on the bug following the procedure [10:24] keep me posted [10:26] thostr_: from my testing, then, it's a +1 as soon as you are ready [10:28] lool: did you post deferred? I think we don't know for code that are not in trunk then, do we? [10:28] (on the landing asks) [10:28] or maybe it's asac? [10:28] didrocks: I marked deferred the things which were for Friday [10:29] so that we could quickly skip over them today [10:29] lool: but some didn't even get the code ready in trunk and we lost this info [10:29] hmm which one? [10:29] well, I don't remember, but some for sure [10:29] grumph, /me look at the snapshots [10:29] didrocks: the one with issues dont have "Friday" and have a red comment on the right [10:30] didrocks: Note that I've updated the unity-mir one with Code ready now [10:30] the MPs are both in [10:30] I dont know whether we want it [10:31] ok, so unity-mir was waiting, not anymore then [10:32] the others were waiting on upstream feedback === alf_ is now known as alf|lunch [10:32] but ok, we do have this red on the right for those [10:33] Saviq: seems like the tests for https://code.launchpad.net/~ps-jenkins/unity8/latestsnapshot-7.83+13.10.20131015.4-0ubuntu1/+merge/191184 failed, did you have a look? [10:34] seems it can't even install the version [10:34] didrocks, ,mediumtests-touch are broken for us still [10:34] didrocks, the qml tests is flaky [10:34] Saviq: do you mind pushing it manually? if you want an additional unity8 landing, it will be needed [10:34] didrocks: check https://bugs.launchpad.net/ubuntu/+source/indicator-datetime/+bug/1227106 [10:34] Ubuntu bug 1227106 in indicator-datetime (Ubuntu Saucy) "indicator-datetime menu buttons do nothing" [Medium,Confirmed] [10:35] didrocks, hmm mediumtests-saucy should pass, let me see [10:35] thostr_: you need to subscribe ubuntu-sru team, otherwise perfect! [10:35] thostr_: please do the same for indicator-bluetooth :) [10:35] Saviq: we have failures on the dashboard as well [10:35] didrocks, either way, yes will merge [10:35] didrocks: done [10:35] Saviq: thanks [10:36] thostr_: great! tell me for indicator-bluetooth and I'll try to get that one in :) [10:36] didrocks: yes, working on it [10:36] didrocks, those are unity8 crashes that we don't know much about yet - need to get proper .crash files out of there - locally we didn't get anywhere [10:36] Saviq: ok, thanks for having a look at them :) [10:39] Saviq: tell me once merged, I want to get unity8 built for testing with the setcap thing [10:39] didrocks, k [10:41] didrocks: here's the bt one: https://bugs.launchpad.net/indicator-bluetooth/+bug/1230275 [10:41] Ubuntu bug 1230275 in indicator-bluetooth (Ubuntu) "Bluetooth cannot be switched on/off on Nexus 4 (rfkill issues)" [Undecided,New] [10:41] thostr_: I'm nomating for series [10:42] didrocks: thanks === alan_g is now known as alan_g|afk [10:51] thostr_: the second one is ready? [10:51] * didrocks wants to poke the release team before being afk === alan_g|afk is now known as alan_g [10:54] sil2100: everything's fine? good progress on indicator-datetime? [10:56] Saviq: if you can't, do you mind if I merge the changelog myself? [10:56] Saviq: I want to kick the unity8 rebuild asap [10:56] didrocks, kicking now [10:56] ok, thanks [10:57] didrocks, should be there in 3 [10:58] greatness! [11:01] so system-image still has some issues when stopping wifi when download was paused [11:01] lool: with latest ubuntu-download-manager? [11:01] but seems to cope with losing active wifi connection and gaining it while downloading [11:01] download stopped then resumed by itself [11:01] didrocks: yup [11:01] and pause/resume works [11:01] so net improvement [11:01] ok, so yeah, improvement :) [11:01] trying the timeout thing now [11:01] as this one is annoying [11:02] right [11:02] will be back in 10mn to check this one [11:02] ok, thanks! :) [11:03] popey: ogra_: btw, did you promote image 99? [11:03] nope [11:03] should i ? [11:03] if your tests are good, please do :) [11:03] lets keep up the suspense before release ;) [11:03] (on maguro) [11:04] ogra_: well, we'll either have 99, 100 or 101 ;) [11:04] right [11:04] so let's promote 99 now ;) [11:04] i would say lets promote the release image on release day [11:04] and pick out of the three tomorrow [11:05] well, no need to potentially retain the nice fixes on image 98 ;) [11:05] I'm sure we'll have another image to promote [11:05] ok [11:05] * didrocks really confidents with what we have [11:05] can someone try weather app? [11:05] let me finish the test i'm doing atmn and i'll do some call tests on maguro [11:05] ogra_: not related, but I merged req 242 with 255 [11:05] ok, thanks! [11:06] didrocks, uh [11:06] popey: no weather data for lyon… [11:06] hmm [11:06] didrocks, ok, sounds risky ... as the oom stuff is very intrusive while the sigstop one is trivial [11:06] ogra_: they are in trunk already [11:06] ah, k [11:06] ogra_: would have been good to not have it [11:06] (in trunk) [11:06] ogra_: let's all test it, ok? [11:07] if we see too many issues, we'll back the oom stuff out [11:07] yeah [11:07] popey: same with paris, seems it's a small unknown city :) [11:07] yeah, same here [11:08] looking to see if it's an upstream api issue [11:08] So is this indicator-bluetooth change one of the things you want for touch but not desktop? [11:08] in unapproved [11:08] * ogra_ is happy he lives in a province city then :) [11:08] http://api.openweathermap.org/data/2.5/ [11:08] cjwatson: right, I was waiting on thostr_ to complete the SRU process for the second bug before pinging you [11:08] cjwatson: we'll have indicator-datetime later on (this bug is already ready) [11:09] popey: well, that can explain :p [11:09] popey: their whole site is unreachable [11:09] good [11:09] ⍨ [11:10] http://api.openweathermap.org/data/2.5/weather?q=London,uk [11:10] works [11:10] didrocks: so, somebody still needs to test this on desktop ASAP, so that we can respin desktop if we need to [11:10] http://api.openweathermap.org/data/2.5/weather?q=Paris,fr [11:10] didrocks: and we're going to defer this from -updates a bit because we haven't done the desktop respins yet [11:10] cjwatson: oh, let me do it [11:10] one sec [11:11] didrocks: however, as infinity suggested, if there's an emergency respin we can always temporarily delete it from -updates and then copy it back in :-) [11:12] cjwatson: right, but let me test it quickly first on the desktop (the code isn't impacting normally, but you know…) [11:17] * didrocks apt-get update is taking ages today [11:17] ok, let's download from the ppa [11:19] * didrocks restarts, one sec [11:22] hum, seems I can't turn on the bluetooth again (but even with previous version and in g-c-c) [11:22] on the desktop [11:22] lool: do you have one minute for testing this one your desktop? [11:22] (if your bluetooth works here) [11:22] * didrocks will debug that calmy later [11:25] waow, I had to power off/on the switch while the session started [11:25] ok, let's upgrade now [11:25] cjwatson: +1 on the desktop, playing with disabling/enabling works well here [11:26] (and changing visibility status) [11:27] ok, unity8 rebuild started === alf|lunch is now known as alf_ [11:28] didrocks: ok, so we could just put this in saucy rather than saucy-updates? [11:28] cjwatson: yeah, it should be fine [11:28] indicator-datetime won't make it, probably, but this could [11:28] cjwatson: I'm waiting on sil2100's feedback [11:28] (for datetime) [11:28] It might just be physically too late [11:28] For desktop [11:28] no worry in that case, -updates is really fine IMHO [11:32] thostr_: 2 things on indicator-datetime [11:33] thostr_: you missed one bug to follow the SRU process: https://bugs.launchpad.net/bugs/1238043 [11:33] Ubuntu bug 1238043 in indicator-datetime (Ubuntu) "indicator-datetime-service is opening /etc/localtime every 5 seconds even on a totally "idle" system" [Medium,In progress] [11:33] thostr_: and https://code.launchpad.net/~charlesk/indicator-datetime/use-mobile-alarm-icon/+merge/191091 needs to ensure it only impact ubuntu touch, not desktop [11:33] (writing that in one of the 2 bug reports would help) [11:35] ogra_: Saviq: who is getting that merged? https://code.launchpad.net/~saviq/unity8/raise-sigstop/+merge/191212 (if we want to have it tested for image 100 as well) [11:35] didrocks, I am [11:35] ok, great! [11:36] * ogra_ will happily test once we have debs [11:36] didrocks, we need mir released for that too [11:36] didrocks, er... merged, first, too [11:36] didrocks, when's image 100? [11:36] Saviq: isn't unity-mir? [11:36] didrocks, that as well [11:36] what about mir? [11:37] didrocks, we need to pull dev into trunk and release [11:37] Saviq: there is no ABI breakage at least? [11:37] didrocks, there is [11:37] ok, no chance to get in now I guess [11:37] before Friday [11:37] Saviq: so please don't merge the unity8 side [11:37] didrocks, I won't, not before everything's in [11:38] ogra_: asac: FYI ^ (we won't have the sigstop thing before Friday because of mir ABI breakage) [11:38] didrocks, that means we will release with massive maliit crashers [11:38] * ogra_ thinks thats a critical one [11:38] ogra_: well, the ABI breakage means a complex operation (once again) [11:39] and I'm unsure about the release team feeling about that one (at least, it's not seeded) [11:39] cjwatson: infinity: wdyt? ^ [11:39] nervous [11:39] Saviq: is there any way to get a fix without an ABI break? [11:39] well, we need to do something about maliit ... we could add sleeps to the upstart job, but that will be racy [11:39] even if it's not "the right long term solution" [11:40] well, let's see [11:40] didrocks, not from our PoV [11:40] IIRC none of the packages are on any other images? [11:40] (and i'm not sure maliit is the only thing suffering by mir sockets not being available, it is just the obvious one) [11:40] didrocks, we don't know when Mir is ready without the ABI break [11:40] So maybe you can get away with it [11:40] cjwatson: none, it's just a build-dep on xorg [11:41]