[04:21] Laney: I talked to thomi about the USS tests, he said they are planning a new AP release and we should have a fix for it soon: https://bugs.launchpad.net/autopilot/+bug/1278272 [04:21] Launchpad bug 1278272 in Autopilot "Autopilot cannot attach process stdout or stderr that contain non-ascii characters" [Critical,In progress] [04:21] * plars -> zzz [04:32] veebers, are you still there? [04:32] fginther: Hi, yep still here [04:32] veebers, what's up with autopilot/cupstream2distro-config [04:32] ? [04:33] fginther: I'm not too sure. I noticed earlier today this failure here: https://jenkins.qa.ubuntu.com/job/autopilot-1.4-trusty-amd64-ci/13/console [04:33] and see that it's trying to use lp:autopilot/1.4 instead of lp:autopilot [04:34] veebers, ok, autopilot/1.4 was removed I see [04:34] sorry about that fginther. [04:37] I'm confused as to why the autopilot-1.4 job was used, there still a set of jobs dedicated to lp:autopilot [04:37] brb [04:43] fginther: just heading back home, will be online again shortly (or file me an email :-)) [04:43] veebers, ack [04:43] veebers, I think I have it figured out, will send a message [04:45] fginther: awesome thanks [09:00] s! [09:01] (ups, xchat changing channels on start while typing) [09:31] Mirv: meeting! [09:34] möte [09:42] sil2100: https://jenkins.qa.ubuntu.com/job/trusty-touch-mako-smoke-daily/28/artifact/clientlogs/ubuntu_system_settings/_usr_sbin_system-image-dbus.32011.crash/*view*/ [09:42] probably for Laney to look at? [09:43] may be not.. [09:48] PermissionError: [Errno 13] Permission denied: '/var/log/system-image/client.log' [09:48] sil2100: also, this morning, a kernel crash on an otto node you may want to track: http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-daily_release/label=qa-intel-4000/1419/console [09:49] sil2100: so was there a discussion last week while I was away about who should dogfood? [09:49] I am happy not to do any more dogfooding if someone from QA is taking it on [09:49] Mirv, have you seen my mail about rootstock-ng ? you should be able to procude testable images for Qt 5.2 now [09:50] (I have other things I can of course be doing) but if it's still needed, happy to do it. [09:50] *produce [09:53] psivaa: not me, system-image is barry [09:53] I actually got this a lot on my desktop already and filed https://bugs.launchpad.net/ubuntu/+source/system-image/+bug/1260237 [09:53] Launchpad bug 1260237 in system-image (Ubuntu) "system-image-dbus crashed with PermissionError in initialize(): [Errno 13] Permission denied: '/var/log/system-image/client.log'" [Medium,New] [09:53] Laney: ack, got that late, my bad [09:54] and there's https://bugs.launchpad.net/ubuntu-system-image/+bug/1222984 which is Won't Fix... [09:54] Launchpad bug 1222984 in Ubuntu system image "traceback when run as non-root" [Low,Won't fix] [09:54] well [09:55] i would think that system-image-dbus talks to a root owned process on the other side [09:55] weird if it doesnt [09:55] (and that process should own the logging preferably) [09:57] popey: i brought it up in the meeting since i saw someone from QA team started doing it last week. I could have wrongly assumed about it. [09:57] I haven't looked into the details, but crashing is certainly wrong [09:57] ogra_: yes, very interesting! the QA team should be able to use that too for fully flashed testing. [09:57] a) it should print an error [09:57] b) why is it a d-bus activated service if only root can make use of it? [09:58] anyway, I'm not really here :-) [10:04] sil2100: can you publish silo 1 and 3. also reconfigure silo 5? [10:07] popey: so, I know there were discussions, I know Didier and Jason were chatting with Julien a lot, but nothing definite got set [10:08] popey: at least I don't know about it [10:08] thostr_: hello! I'll reconfigure silo 5, with publishing we try to wait a little bit until we get one failure tracked down [10:09] thostr_: but I promise to publish them before evening today [10:12] psivaa: I asked omer to test while I was away at the sprint. [10:12] popey: ohh, then my assumption was wrong. sorry :) [10:13] np [10:20] sil2100: reconfigured? [10:22] thostr_: ...done! [10:23] sil2100: thanks [10:41] Mirv, right ... [11:09] psivaa: any luck with downgrading telepathy? [11:09] I mean, telephony [11:10] sil2100: the dialer app test failure is not as deterministic as we thought before. [11:10] the tests sometimes pass [11:10] sil2100: so i am having to run the test a few times before confirming anything [11:11] psivaa: oh, right, the smoketesting gave the impression it's reproducible all the time [11:11] psivaa: if you manage to find something out, just ping me here please [11:11] sil2100: right it failed on the first attempt, but a couple of reruns passed, then it failed again, so there is flakiness. [11:12] sil2100: reverted the package and running now to see if the flakiness is going and i'll update you [11:13] Thanks :) === mhr3__ is now known as mhr3 [12:12] morning [12:15] Morning === MacSlow is now known as MacSlow|lunch [12:17] sil2100: ogra_: still no luck on the dialer-app hangup test failure. [12:17] one thing i see is that in all the attempts where the test fails, i see http://pastebin.ubuntu.com/6908750/ [12:18] sil2100: ogra_ but i couldn't see which package that could cause it, [12:18] i tried reverting the dbus related packages in http://people.canonical.com/~ogra/touch-image-stats/20140206.2.changes too but to no avail [12:20] psivaa, did you try telephony-service too ? [12:20] (http://people.canonical.com/~ogra/touch-image-stats/20140207.changes) [12:20] ogra_: that was the first one and it dint make any difference [12:20] hmmm [12:27] sil2100: ogra_ https://bugs.launchpad.net/ubuntu/+source/telepathy-ofono/+bug/1226298 appear to have some similarities. but this was fixecd [12:28] Launchpad bug 1226298 in telepathy-ofono (Ubuntu) "[mako] After 5-10 incoming calls ( ended remotely ), no more ring/snap decision" [Critical,Fix released] [12:28] and quite a while ago it seems [12:29] yes, the kernel log is kind of similar though [12:29] it actually points to pulse/alsa though [12:30] ogra_: are there any recent uploads on these? [12:30] http://people.canonical.com/~ogra/touch-image-stats/20140207.1.changes [12:30] libasound2 [12:30] (and *-data) [12:31] ok, let me try them next === alan_g is now known as alan_g|lunch [12:45] reverting that doesn't help [12:45] ;/ === josepht changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: josepht | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: - [13:10] cihelp: josepht: autopilot project configuration was broken, so https://code.launchpad.net/~xnox/autopilot/use-fb-sizes/+merge/199295 jenkins bot at the end disapproved things. [13:11] cihelp: josepht: can that be re-triggered please? [13:11] xnox: looking === MacSlow|lunch is now known as MacSlow [13:12] josepht: also I need to land https://code.launchpad.net/~xnox/autopilot/use-fb-sizes/+merge/199295 into the archive ASAP. Can we trigger the CI-train upload for that branch? [13:14] sil2100, ping [13:29] sil2100: who will review unity-mir's landing now that Saviq is in vacation? [13:29] kgunn: ^ [13:30] that's a blocker for latest mir landing request [13:39] xnox: I've kicked off the autopilot-ci job once that's finished successfully I'll kick off the autolanding job. [13:40] josepht: not sure about the autolanding job, since it's under ci-train management actually. [13:40] josepht: and i'm not the person who requested this. [13:40] josepht: err... please dont do autolandings for stuff in CI train. [13:41] josepht: these thigns are in CI train silo staged, most likely waiting for veebers and thomi to get up finish their testing [13:41] xnox, asac: ack no autolanding [13:41] thx === tvoss is now known as tvoss|test === alan_g|lunch is now known as alan_g [14:01] updated libunity in archive now [14:02] rsalveti: thanks....we can get gerry to do it (he's wanting this mir anyway) [14:03] kgunn: thanks, otherwise this will block us for days [14:03] and we don't have days [14:03] rsalveti: what do I need to do? [14:03] greyback: https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0AuDk72Lpx8U5dFlCc1VzeVZzWmdBZS11WERjdVc3dmc&usp=drive_web#gid=22 [14:04] greyback: there was a landing request for unity-mir that Saviq created a few days ago [14:04] still needs testing, so it can be approved and landed [14:04] rsalveti: oh...you were talking about Saviq's [14:04] rsalveti: he might have left that with mzanetti let me check [14:05] greyback: i thot rsalveti meant a review of this one... https://code.launchpad.net/~kgunn72/unity-mir/um-mir0.1.5-bump [14:05] kgunn: ah ok [14:05] I want the big landing first :-) [14:06] * kgunn loves that rsalveti doesn't consider mir a big landing anymore :) [14:09] rsalveti: ok, tsdgeos is looking at the 004 silo build [14:09] well i'm not [14:09] tsdgeos: huh ? [14:09] *yet* [14:09] i'm getting cimi and/or mzanetti to give me access to their maguros [14:09] ah [14:09] and then i'll get it fixed [14:09] sil2100: no luck at all with the dialer app failure.. there is a lxc-android-config change in http://people.canonical.com/~ogra/touch-image-stats/20140206.2.changes [14:09] ...oh yeah...how could i forget [14:10] hopefully [14:10] not so easy to get a thing fixed over the interwebs :D [14:10] but we'll see [14:10] tsdgeos: hey...wait, i thot that ended up being due to AP update ? [14:10] maybe davmor2 can give you a hand as well? [14:10] and davmor2 and elopio were looking into that? [14:10] ...but maybe i'm missing some new info [14:10] ? [14:11] psivaa, [14:11] lxc-android-config (0.136) trusty; urgency=medium [14:11] * 30-no-surface-flinger: adding logic to start/stop surface flinger via [14:11] properties (needed by the SDK team to compare performance against MIR) [14:11] unrelated [14:11] kgunn: i don't know, i told elopio to send me an email on friday night after his shift ended, but i never got anything [14:11] kgunn: it may be [14:12] ogra_: ack, it was a last ditch attempt [14:15] tsdgeos: what do you need to do with maguro? [14:15] tsdgeos: I can open a port for you [14:15] rsalveti: cimi just gave me one [14:15] tsdgeos: awesome then [14:15] rsalveti: well, figure out those autopilot tests that fail and are blocking us [14:16] which honestly if we don't support it [14:16] makes no sense to me [14:16] but i'm not the one that decides if that makes sense or not, so i'll just fix them [14:16] haha, alright [14:18] tsdgeos: i feel you brother... [14:18] rsalveti: so when do we officially move to 4.4 ? :) [14:18] good morning. [14:18] kgunn: well, I'm also waiting on you for that :-) [14:18] elopio: hey there! [14:18] I'm just waking up, so trying to parse what you are saying. [14:18] rsalveti: oh no...chicken egg? [14:18] elopio: no worries [14:18] kgunn: first I need mir 1.5, then the backend packaging split that alf_ is working on [14:19] tsdgeos: davmor2 leaves pretty much at the same time you leave, so we couldn't collect more maguro information. [14:19] yep....chicken egg [14:19] kgunn: yeah [14:19] elopio: ok [14:19] tsdgeos: the other bug you found is already reviewed and approved by mterry. [14:19] i saw :) [14:19] elopio: so basically we still don't know why and how to fix the other 2 AP fails, right? [14:20] tsdgeos: right. But I understand from your recent messages that you now have access to a maguro, right? [14:20] kind of yes [14:20] i'll take care of it [14:20] will ping you if need help [14:20] elopio: or maybe, let's both try to get something out of it if you can get davmor2 or rsalveti to give you another maguro [14:21] tsdgeos, kgunn, elopio: what's needed on what now save me digging into a lot of backscroll [14:21] so if one fails the other gets it to work [14:21] we need to be able to land [14:21] davmor2: sorry... [14:21] davmor2: is there any chance you can give ssh to elopio in a maguro so he can debug the AP failures we're having in unity8? [14:21] davmor2: we're trying to determine what remains an unknown on the AP failures [14:21] josepht: thanks for that. There are a few other autopilot jobs that need a re-kick now that it's correctly configured: [14:22] https://code.launchpad.net/~thomir/autopilot/trunk-add-output_stream_tests/+merge/204793 [14:22] https://code.launchpad.net/~veebers/autopilot/fix_deprecate_pick_app_launcher/+merge/202784 [14:22] https://code.launchpad.net/~thomir/autopilot/trunk-fix-functional-tests/+merge/205512 [14:22] I didn't know rsalveti could give access to maguros. rsalveti, how does that work? I could use one. [14:22] https://code.launchpad.net/~veebers/autopilot/fix_1275913_launch/+merge/204815 [14:22] https://code.launchpad.net/~thomir/autopilot/trunk-add-unit-test-coverage/+merge/205517 [14:22] elopio: sure, with system-image as rw I'd guess? [14:22] davmor2: basically those 2 AP failures are the current landing process log jam....unity8, and then mir is also help up due to that...etc [14:22] elopio: which image as well? [14:23] kgunn, tsdgeos, elopio: ah right is this what we started digging into on Friday night? [14:23] davmor2: yes, before you left on friday, I thought you could reproduce it. What I would love is to see a video from it failing. [14:23] xnox: sure, is just the autopilot-ci job enough or do you need autopilot-1.4-ci as well? the second seems to fail a lot. [14:23] tsdgeos: are we seeing the error in the latest image too? Or which image should I get? [14:24] josepht: the autopilot-1.4-ci is obsolete, and shouldn't be used any more. [14:24] josepht: just autopilot-ci please. [14:24] elopio: no idea :D [14:24] xnox: okay [14:24] elopio: i'm just trying to get it to fail now [14:25] elopio: is there a way to run just the one test? Other wise there is no way I can grab it. elopio I don't think the video capture will work on maguro for the same reason that the screenshot doesn't [14:25] rsalveti: the most recent image please. [14:25] alright, flashing [14:27] davmor2: yes, you can run only one with phablet-test-run test unity8.shell.tests.test_emulators.DashEmulatorTestCase.test_open_applications_scope [14:28] elopio: right give me a couple of minutes I'll see what I can rig up [14:28] davmor2: and if it happens every time, maybe you can record it with a camera? Maybe I'm asking too much :) [14:28] elopio: no that is what I was going to do [14:29] davmor2: thanks! [14:29] elopio: hmmm hang on phone doesn't seem to be accepting charge give me about 10 minutes to remove the battery and get a small amount of charge in it. [14:31] davmor2: sure. We'll try to gather more information from our side. [14:32] sil2100: so, i'm going out for lunch... one thing that we could do is to flash image 169 to see if this issue is there. and then upgrade the packages one by one to see if the issue pops up. [14:32] sil2100: but that will take quite a lot of time.. [14:36] xnox: I've re-kicked the autopilot-ci job for all of those MPs [14:37] josepht: thanks! [14:46] elopio: I'm guessing this isn't the right response http://paste.ubuntu.com/6909483/ :D [14:47] davmor2: that means it didn't find the test. [14:48] davmor2: do you have unity8-autopilot installed on the phone? [14:48] well, that's not a phone. [14:48] elopio: hmmm I bet had to the reinstall it to test 170. I'll grab the stuff again D'oh [14:51] davmor2: https://bugs.launchpad.net/autopilot/+bug/1278462 [14:51] Launchpad bug 1278462 in Autopilot "Test runner should report an error when no tests where found" [Undecided,New] === tvoss|test is now known as tvoss [15:00] psivaa: ouch... [15:01] psivaa: how often can you reproduce the problem locally on the device? [15:02] psivaa: and, is it also possible to encounter the problem during normal usage? [15:02] bfiller: hello! [15:03] bfiller: even though it doesn't seem to be caused by any of the changes you and your team made, we seem to have a problem with one of dialer-app's AP tests since some images [15:04] bfiller: psivaa tried bisecting which package could have caused this failure to appear, but we seem to not have much luck with that [15:05] sil2100: crashes still? probably the same mir bug [15:05] bfiller: could you have someone investigate this failure here? http://ci.ubuntu.com/smokeng/trusty/touch/mako/173:20140210:20140115.1/6527/dialer_app/756161/ [15:05] sil2100: ok [15:05] bfiller: not the crash thing sadly, something new - seems like a problem with hangup [15:06] bfiller: from what I know the test failure is reproducible (but not always) [15:06] psivaa mentioned that he usually required some re-runs of the test to get the failure [15:13] bfiller: thanks! [15:30] elopio: okay this is getting frustrating now I can run phablet-test-run -n -p unity8-autopilot unity8 and that runs fine if I run phablet-test-run test unity8.shell.tests.test_emulators.DashEmulatorTestCase.test_open_applications_scope I get the 0 tests run [15:31] elopio: and I have unity8 listed in the the /home/phablet/autopilot dir [15:31] davmor2: let me double check the module path. [15:31] elopio: will do [15:32] unity8.shell.tests.test_emulators.DashEmulatorTestCase.test_open_applications_scope [15:32] seems correct, that's what I've been running. [15:32] davmor2: but, I have run the single test 20 times without failures, on this magic rsalveti's maguro. [15:32] I'm trying to run the whole suite now to see if it's because there's a test war. [15:33] davmor2: oh, you have an extra "test" on your command === alan_g is now known as alan_g|tea [15:34] phablet-test-run -n unity8.shell.tests.test_emulators.DashEmulatorTestCase.test_open_applications_scope [15:35] elopio: thanks === alan_g|tea is now known as alan_g [15:45] elopio: http://paste.ubuntu.com/6909764/ note the last one [15:46] elopio: not sure why they are so vastly different in times [15:46] davmor2: yes, I'm looking that here too. [15:47] according to tsdgeos, the device can get busy and it takes a lot of time to settle. [15:47] elopio: yeah only on a full test run there is no settle time [15:47] i may be lying :D [15:47] tsdgeos: ^ [15:48] yes i've seen that too [15:48] honestly it's not what bothers me [15:48] if you add some prints [15:48] you'll see it's at the end [15:48] it's not what's causing this problem [15:58] sil2100: http://people.canonical.com/~ogra/touch-image-stats/20140208.changes thats the changes that made the dialer_app problem firs tappear, right? [15:58] and we cant reproduce at all? [15:58] asac, thats not clear [15:59] ogra_: i dont see it before int he results [15:59] asac, psivaa rolled back piece by pice of any phone related packages since 20140206 [15:59] oh wait [15:59] http://people.canonical.com/~ogra/touch-image-stats/20140207.1.changes that one seems to have ti [15:59] there were a few images where everything was broken [15:59] elopio: off hand do you know what the test is looking for to confirm that the screen is now on the applications page? [16:00] * ogra_ needs to re-locate for meeting, one sec [16:01] ogra_: on 7 (with .1) it was green [16:01] http://ci.ubuntu.com/smokeng/trusty/touch/mako/169:20140207:20140115.1/6492/ [16:01] there were crashes though [16:02] psivaa: there? [16:02] asac: yep, just came back from lunch [16:03] kk [16:03] and reading the backlog [16:03] psivaa: the bisecting for dialer-app regression failed or you still have options to try out? [16:03] asac: not in terms of reverting packages. [16:03] psivaa: so you reverted everything and the issue was still there? [16:03] asac, on 7 ? [16:04] asac: yea, reverted everything that we thought might cause the issue, but the issue is still there [16:04] asac, the last green image was 6.1 [16:04] ogra_: 6.2 also had a green dialer-app [16:04] http://ci.ubuntu.com/smokeng/trusty/touch/mako/168:20140206.2:20140115.1/6487/ [16:04] with crashes, but still [16:04] The problem is that even when it was green, the issue could still have been there [16:05] asac: not really knowing when this was introduced is making decision difficult [16:05] As the problem is not reproducible in 100% [16:05] asac, after 6.1 (167) there were several images that were so broken that you cant take any results serious) [16:05] so i guess [16:05] http://people.canonical.com/~ogra/touch-image-stats/20140207.changes [16:05] davmor2: it waits until the current index is the previous index + 1 [16:05] ogra_: 168 above has green dialer-app [16:05] I flashed my phone and will try investigating a bit further [16:05] asac, ignore 168 and 169 please [16:05] ogra_: why? they are green :) [16:05] on dialer-app [16:05] so 170 is the first to show this problem [16:06] asac, there were low level issues, i wouldnt take any of the tests serious on these images [16:06] ogra_: well, i think a green can be taken into account [16:06] low level issues might invalidate a red [16:06] but not a green :) [16:06] http://people.canonical.com/~ogra/touch-image-stats/20140207.changes [16:07] psivaa: did you work against 170 and backed out the ones fromt he changes in the line above? [16:08] asac, we went through the phone related packages, he tested while backing them out one by one [16:08] asac: i worked with 173 and reverted packages [16:08] (back to 07) [16:08] psivaa: try start from 170 ... and revert the few from above [16:08] most likely telephony-service :) [16:08] 169 as green on dialer app [16:08] tried already [16:08] 170 had the same issue we see [16:09] Looking at the failure right now [16:09] was my first shot too :) [16:09] maybe there was confusion or a mistake [16:09] asac: ack, will do that. in the meeting and once this is over i'll install 170 and try [16:10] elopio: so why would the test suddenly take nearly 4 times as long to run :/ [16:11] no didrocks? [16:11] tsdgeos: what do you need? [16:11] tsdgeos: no, he's on holidays [16:11] sil is here for you [16:11] bfiller: any luck regarding the issue? [16:12] sil2100: so about the failing AP tests for the unity8 silo [16:12] sil2100: boiko is looking at it, no update yet [16:12] sil2100: i can make them fail without any of the MPs from that silo [16:12] tsdgeos: oh oh! Did you manage to reproduce it? [16:12] sil2100: so they were already broken [16:12] tsdgeos: yes, we know [16:12] do you? [16:12] ok [16:12] tsdgeos: that's why we don't want to land anything until it's fixed [16:13] i thought we were blocking because it was a regression [16:13] sil2100: when did that rregress? [16:13] but it's not a regression [16:13] tsdgeos: Didier's point is: no new landings of unity8 as long as it's not fixed, as it seems to have regressed in one of the earlier landings [16:13] it's just an unstable test [16:13] tsdgeos: it probably regressed last week sometimes [16:13] sil2100: that's bad point [16:13] it's new tests [16:13] nothing regressed [16:14] it's just that CI doesn't run galaxy nexus [16:14] so nothing runs those tests [16:14] tsdgeos: maybe you are talking about something different? [16:14] sil2100: ? [16:14] except the ultra blocker silo thing [16:14] i'm talking about [16:14] "2 test failures on maguro that we need to understand: [16:14] http://ci.ubuntu.com/smokeng/trusty/touch/maguro/167:20140206.1:20140115.1/6480/unity8-autopilot/741839/ [16:14] http://ci.ubuntu.com/smokeng/trusty/touch/maguro/168:20140206.2:20140115.1/6490/unity8-autopilot/744272/" [16:14] that the ci train document mentions [16:15] if you say we can't merge new stuff until those tests don't fail anymore [16:15] sil2100: can you check that those never succeeded and are new? if so, it might be indeed not right to block on them [16:15] asac: they do succeed [16:15] eventually [16:16] i mean i run it 10 times in a loop [16:16] sure [16:16] it succeeds aroud 50% [16:16] thats not the point :) [16:16] the point is [16:16] tsdgeos: when were they introduced? [16:16] that it may have succeed in an earlier run [16:16] and you'll claim "look they worked" [16:16] and i'll claim "they worked the same they work now :D" [16:16] http://ci.ubuntu.com/smokeng/trusty/touch/maguro/169:20140207:20140115.1/6491/unity8-autopilot/ [16:16] we were just lucky [16:16] tsdgeos: are they in there at all? [16:17] one is old [16:17] so yes it is there [16:17] the other let me check [16:17] tsdgeos: which one is old? [16:17] yes it is there too [16:17] the hud_click_one [16:17] tsdgeos: do you know why this test is so flacky then? You think it can be fixed? [16:17] cant find the string :/ [16:18] cant find "click_one" [16:18] asac: sorry unity8.shell.tests.test_hud.TestHud.test_hide_hud_click [16:18] sil2100: i think it can be fixed, yes [16:18] sil2100: i am not sure it is worth doing the effort of fixing them *now* [16:18] tsdgeos: check if you can find this happening regularly in the past https://jenkins.qa.ubuntu.com/job/trusty-touch-maguro-smoke-unity8-autopilot/ [16:19] tsdgeos: if its flaky it should have been there a few time in that list [16:19] tsdgeos: I just looked at maguro results from the past weeks and I didn't see these tests failing, so we had to be *really* lucky before ;) [16:19] tsdgeos: did you figure why they fail? [16:19] asac, just an observation, but between the dialer-app working and it failing there was the utah change ... [16:20] doanac: what changed in utah? [16:20] ogra_: but it fails also on local devices [16:20] asac: no, if i had figured why they fail i'd be fixing them and not arguing here :) [16:20] sil2100: true [16:21] tsdgeos: so the way of htinking is that on maguro you see nasty stuff that will strike us later. hence its very valuble to check them out if they are reproducible right now [16:21] e.g. if its non-hw related races [16:21] you certainly want to get rid of them [16:21] ogra_: do you know what changed? [16:21] asac, note that the test links have different names between 169 and 170 [16:21] asac: i can reproduce them now, i also have 13 branches landing that are starting to conflict amonst themselves [16:21] so it'd be helpful if we could land them [16:21] ogra_: you talking about the dialer-app failure now? [16:21] but if you guys are blocking until we get that test fixed [16:21] asac, nope, i see that it seems to have gotten a lot more reliable ... zero change images come out with the same results now [16:22] tsdgeos: if you land your branches how do the APs look? [16:22] we'll get it fixed [16:22] tsdgeos: did you run every AP [16:22] ? [16:22] sil2100, yes [16:22] and you get exactly this 1 failyre only? [16:22] asac: you get the same errors [16:22] ogra_: you think it might be caused by utah? [16:22]