[00:14] alright, I'm off for dinner, will be back later though [01:22] robru: btw, silo 11 had an issue uploading the the PPA, I'm not sure why [01:22] a following run of the build job seems successful now [01:23] do you have access to the ps-jenkins list to see why the uploads were rejected or something? [01:25] 2014-04-03 00:07:12 INFO Failed to parse changes file '/srv/launchpad.net/ppa-queue/incoming/upload-ftp-20140403-000629-025651/~ci-train-ppa-service/landing-011/ubuntu/indicator-applet_12.10.2+14.04.20140402-0ubuntu1_source.changes': GPG verification of /srv/laun [01:26] chpad.net/ppa-queue/incoming/upload-ftp-20140403-000629-025651/~ci-train-ppa-service/landing-011/ubuntu/indicator-applet_12.10.2+14.04.20140402-0ubuntu1_source.changes failed: Verification failed 3 times: ["(7, 9, u'No public key')", "(7, 9, u'No public key')", "(7, [01:26] 9, u'No public key')"] [01:26] looks like transient trouble talking to the keyserver [01:26] (is my guess) [01:52] just looking for a silo for line 51 [01:53] (btw, silo 12 ready to merge...so... :) [02:05] === trainguard: IMAGE 275 building (started: 20140403 02:05) === [04:16] morning [04:17] hmm, how that 275 has taken >2h to build [04:23] there's a bit of backlog on the system-image importer [04:24] it was in manual mode while setting up generic_x86 and I forgot to turn it back on (looks like nobody noticed until now). I've re-enabled the cronjob and it'll start processing the latest images now. [04:24] since we now have two rootfs images to process, expect the processing time to almost double (not much we can do there...) [04:25] right [04:26] thanks for fixing it [04:46] and it seems testing is also halted for a bit longer time probably [05:10] === trainguard: IMAGE 275 DONE (finished: 20140403 05:10) === [05:10] === changelog: http://people.canonical.com/~ogra/touch-image-stats/275.changes === [05:11] ogra_, Mirv: wow, 3 hour image build, longest one I've ever seen! [05:14] that was indeed a long one, but great that it happened at last. too bad we won't get test results. [06:29] psivaa_: hey, I think we'll have to run all the tests ourselves manually to know what's the image status is if we don't have devices, you do have a mako, right? [06:54] didrocks: no. i dont have a mako [06:57] ok, Mirv: do you think you would have time for that? [06:57] (like a background task, and don't do landings meanwhile) [06:58] didrocks: I can squash two bugs in a one run, since I'm soon about to test a qtdeclarative update [06:58] Mirv: yeah, that's a good plan then! :) [06:59] oh, the date format fix? [06:59] it's already there, nice! [06:59] well Pat suggested that "maybe these two lines would work", so I'm testing that now and if it works properly without regressions I'll submit it upstream [07:00] first findings is that yes it seems to work, but the real test is with RSS Reader and then seeing nothing regresses [07:01] yeah === psivaa changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: psivaa | CI Train support - US: robru, cyphermox, rsalveti - EU: sil2100, Mirv, didrocks | CITrain support no answer: use mup bot after 30 minutes, but choose right timezone | Known issues: - [08:24] didrocks, seems stgraber added i386 to the system-image builds last night ... which he said he can only run serialized ... which in turn resulted in the last image taking 3:05h to build [08:25] (i'll talk to him tonight to rip it out again and find a way to parallelize that, but todays builds will most likely take that long :/ ) [08:25] ogra_: ok, thanks for the head's up [08:27] ogra_: He said fairly clearly in scrollback that most of the delay was due to the cronjob being accidentally disabled for a while [08:27] cjwatson, oh, right, so thats not the real time yet ... he warned me that i should look at the build time since it would be significantly longer [08:27] ogra_: So please see how long the next build takes before taking any action [08:27] right [08:28] thanks for the heads up, i totally missed that above [08:34] ogra_: could you please tekk me what do I do wrong? -> http://pastebin.ubuntu.com/7197747/ FAILED (failures=18) [08:38] bzoltan: I wonder if phablet-config autopilot --dbus-probe enable is needed, that's what I do before running tests as mentioned at https://wiki.ubuntu.com/Touch/Testing#Running_Click_tests [08:39] Mirv: OK... I try that [08:41] didrocks: packaging ACK needed -> https://ci-train.ubuntu.com/job/landing-015-2-publish/4/artifact/packaging_changes_unity-scope-click_0.1+14.04.20140402-0ubuntu1.diff (new dep, it's in universe but -scope-click is as well) [09:05] sil2100: +1 [09:44] ogra_: popey: contrary to what i said in the meeting, we now have a flo impacted by the reboot issue, with 275 [09:45] ugh [09:45] psivaa, can you ask whoever does the reboot to capture /proc/last_kmsg from recovery ? [09:46] ogra_: ack, will do that. [09:46] ogra_: http://paste.ubuntu.com/7197823/ [09:46] (it is important that there is no reboot between the hang and getting that log) [09:47] ogra_: hum, with 275, which is supposed to adb wait, have all tests failing, and then reboot? === vrruiz_ is now known as rvr [09:48] didrocks, well, depends where it actually hangs ... if it is run-init there is no way to get any adbd up [09:48] ok [09:48] what psivaa posted above doesnt seem to hang in run-init though [09:49] yeah [09:49] can we centralize all that into a bug? [09:49] that will make tracking/conversation better [09:49] [ 25.702487] transport_store: usb cable is not connected [09:49] interesting [09:51] Mirv: that dbus-probe does not help [09:52] if it actually thinks the cable is disconnected we cant really do much :( [09:52] Mirv, ogra_, didrocks: stock image ... not even a single app gives good results. [09:52] bzoltan: hum, that's not what Mirv is experiencing though? [09:52] as he's rerunning everything manually [09:52] bzoltan: unrelated to the boot issue I guess? [09:53] (you are booted) [09:53] bzoltan, note that the shell handling changed, make sure to be on the latest phablet-tools etc [09:53] didrocks: I am booted [09:53] ogra_: I check that [09:53] foundations dropped a few things from the login process (and added others) [09:54] ogra_: Installed: 1.0+14.04.20140401-0ubuntu1 [09:54] seems to be fresh [09:55] in the logs I see %CLICK_FRAMEWORK%-%CLICK_ARCH% [here comes the directory listing of my ~] (process:8732): WARNING **: Unable to register app: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: Invalid application ID [10:09] bzoltan: so far I haven't had problems with #275 [10:10] Mirv: I do not know what the problem is ... I wish to understand. I am sure it is not about the image... but the tools [10:11] Morning all [10:13] ogra_: it thinks it's disconnected on the host or device side? [10:13] are we back to the device online and restart adb server to get it back situation? [10:13] sergiusens, well, it shouldnt [10:13] oh, i havent tried that yet [10:14] let me reproduce the error ... i'm at the 25th loop, still working [10:14] it is so annoying that it takes forever too even get it ... i hate such bugs [10:14] aha ! [10:15] #27 hangs :) [10:15] nope [10:15] restarting didnt help [10:15] lets see what last_kmsg says ... i added some extra panic line after run-init [10:17] hmm, so it doesnt pass run-init http://paste.ubuntu.com/7198063/ [10:17] (else it would have printed something ) [10:17] * ogra_ adds more printing [10:18] oh ! [10:18] dang [10:20] heh, trying to work on the device while the loop script runs in another terminal isnt helpful :P === MacSlow is now known as MacSlow|lunch [10:25] Mirv: like this http://pastebin.ubuntu.com/7198089/ [10:25] popey, hmm, your pastebin is differnt from the mail content [10:26] oh [10:26] did i grab the wrong one [10:26] well, it moves forward there [10:26] what is irritating is that systemd-udevd starts that early [10:26] it should start after the contsainer [10:28] sorry, http://paste.ubuntu.com/7197970/ [10:28] too many pastebins open [10:30] bzoltan: hrm. something clearly wrong, but what... [10:30] bzoltan: what I do before starting the tests is apt-get install ubuntu-ui-toolkit-autopilot address-book-app-autopilot messaging-app-autopilot dialer-app-autopilot friends-app-autopilot webbrowser-app-autopilot mediaplayer-app-autopilot unity8-autopilot ubuntu-system-settings-online-accounts-autopilot ubuntu-system-settings-autopilot [10:30] and I don't see such errors as that missing module === psivaa changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: cihelp | CI Train support - US: robru, cyphermox, rsalveti - EU: sil2100, Mirv, didrocks | CITrain support no answer: use mup bot after 30 minutes, but choose right timezone | Known issues: - [10:44] can someone give me a jand with jenkins, I have tested the tests in a branch on my machine and the phone and the tests pass but it seems that PS jenkins is "slow" and the timeouts of the QTRY_COMPARE are not long enough.. :-/ [10:46] popey: didrocks: so I found an app in Ubuntu Netwalk that does the renaming issue I have installed it twice from available but on once does it show up in click list http://paste.ubuntu.com/7198136/ So from that I can only assume it is safer than we think my big concern would be updates not sure on a happy way to test that though [10:47] s/but on/but only [10:48] davmor2: ok, if you can test updates, that would be excellent! [10:48] ogra_: you don't need anything else from my phone do you? [10:48] popey, nope [10:48] thanks [10:48] mandel: jenkins doesn't build itself your requests, it's the distro builders building it [10:49] didrocks, hm.. bummer, tests used to pass with no problems, but now the simply fail and I'm quite sure is due to timing issues [10:49] * mandel adds even more timeouts while thinks of a diff way to test qt signals [10:50] didrocks: also I can't reproduce the mediascanner crashers so I'm going to assume for now that they are invalid and refile if it happens again so those can be knocked off the blocker list along with the unity8 crasher I guess [10:50] mandel: basically we do build in the exact same environment/machines (really the same, as it's the final product we build) than distro. [10:50] davmor2: ok, please tell that on the ML [10:51] * sil2100 reminds thostr_ about m&c o/ [10:51] didrocks, ok, I wonder what has changed to it not to run as fast as it used.. anyway, I'll think a work around [10:51] didrocks: will do [10:51] mandel: Qt, distro… a lot of things changes continously [10:52] didrocks: I just realised you have a time machine that you have been hiding from the rest of the team no wonder you get so much work done ;) 02.04.13 indeed :D [10:52] didrocks, well, yes but in a clean chroot in my system works so I don't see a "code" reason to see this issues, for example, tests do pass in amd64 but fail in arm on the buildservers while th tests do pass in the phone [10:55] mandel: yeah, clearly timing-related [10:55] davmor2: yeah, mentionned it yesterday :p [10:56] mandel: didrocks; when mandel said jenkins, he might mean the ps-bot running for the MRs (not clear from the conversation) [10:56] didrocks, sergiusens the issues are in the MR, yes [10:56] sigh ... 84 reboots ... no hang [10:56] sergiusens: oh, good point! [10:56] * ogra_ curses this day [10:57] "almost done" #275 report that all AP:s seem to pass so far but I've a couple of weird things that I'll document in more detail [10:57] mandel: ok, so not us, ping the CI team :) [10:57] didrocks, ok [10:57] Mirv: ok, thanks [10:57] didrocks, channel? [10:57] mandel: that one, ping the vanguard [10:57] Mirv: thanks ;) [10:59] didrocks: So you mentioned you had a time machine I must of missed that :) [10:59] mandel: citeam is supposed to have a highlight setup for 'cihelp' -> read topic ;-) [11:01] cihelp, any idea on why the following tests are failing => https://jenkins.qa.ubuntu.com/job/ubuntu-download-manager-trusty-amd64-ci/377/console when they do pass on a machine and on the phone (I can confirm that the tests do have a timeout nad that it can be increased but it would make no sense to do so) [11:01] mandel: /me looks [11:02] mandel: the funny thing is that I was thinking about you not 5 minutes ago ;) [11:02] vila, the tests have been passing for a long time, the issue here is that they are slow in those machines... [11:02] vila, what did I do? [11:02] mandel: hey, good stuff, don't panic ;) [11:03] mandel: let's talk about that later [11:03] vila, ok :) [11:04] vila, one of the possible only reasons I can think of if that I moved to cmake to run the tests and that uses ctest.. [11:04] mandel: ctest never succeeded or ? [11:05] didrocks: https://bugs.launchpad.net/ubuntu/+source/unity8/+bug/1301871 [11:05] Ubuntu bug 1301871 in unity8 (Ubuntu) "Last track visible on the music carousel can not be used" [High,New] [11:06] davmor2: thanks [11:08] mandel: and http://s-jenkins.ubuntu-ci:8080/job/ubuntu-download-manager-trusty-amd64-ci/378/consoleFull succeeded [11:08] vila, some tests do run, but the issue is that I'm testing that qt signals are emitted, to do so I use a qt macro that try to check that the signal was emitted with a timeout and allows the events to be processed during the execution http://qt-project.org/doc/qt-5.0/qttestlib/qtest.html#QTRY_COMPARE [11:08] vila, exactly, while the arm ones fail.. [11:09] vila, so it looks like the arm machines are slow or (if they use qemu) something is going on there [11:09] alecu: how is the fix for https://bugs.launchpad.net/ubuntu/+source/unity-scope-click/+bug/1279481 progressing it's possibly about to become a blocker if it screws up updates :( [11:09] Ubuntu bug 1279481 in unity-scope-click (Ubuntu) "scope uses app title to uniquify list of installed applications" [Critical,In progress] [11:09] vila, in theory I can pass a longer timeout, so that the test runs for longer in the psbot but feels like a dirty workaround [11:10] mandel: and 377 is on arm ? Doesn't look like it can you help me find the right pieces to compare ? [11:10] mandel: yeah, we had that discussion already some months ago right ? [11:11] vila, correct [11:11] vila, let me point you to an error with the same tests in trunk that failed [11:11] vila, give me a few mins [11:11] davmor2: it will be fixed by this branch: https://code.launchpad.net/~alecu/unity-scope-click/use-tested-search/+merge/212252 [11:12] mandel: I need to grab some food, bb in a few minutes [11:12] davmor2: but, if we need it to land faster I can backport it on a smaller branch so it gets to trunk earlier [11:12] didrocks: ^ [11:12] davmor2: because I'm still a day away from finishing the big branch above, and I'm not making any progress by having all these meetings in the scopes sprint :-) [11:12] davmor2: that has already landed AFAIK, as told during the meeting (in proposed or unapproved) [11:12] * ogra_ sighs ... i cant reproduce the hang at all anymore [11:13] I thought? [11:13] * didrocks looks [11:13] didrocks: you probably mean the unicode fix? [11:13] alecu: yeah, I was told it was the fix for the issue davmor2 mentionned [11:13] didrocks: this is about some apps that are duplicated because they have a different title in the .desktop than in the click package index webservice. [11:14] davmor2: right? ^ [11:14] alecu: ah ok [11:14] didrocks: davmor2: what landed was this unicode issue: https://bugs.launchpad.net/ubuntu/+source/unity-scope-click/+bug/1298327 [11:14] Ubuntu bug 1298327 in unity-scope-click (Ubuntu) "Wrong character encoding in app previews" [Critical,Fix released] [11:15] can someone tell me how I can re-trigger a click build at http://s-jenkins:8080/view/click/job/music-app-click/ ? [11:15] davmor2: so, please let me know if I need to hurry the fix for "some apps are duplicated" or if it can wait a day more. [11:16] * popey discovers a "Build now" button and presses it [11:16] alecu: I still see that on image 275 [11:16] davmor2: which one? [11:16] the unicode issue [11:16] davmor2: it has probably landed less than an hour ago, I don't know if it's on the image yet [11:17] Ah okay [11:17] so next image should have that fixed then [11:17] davmor2: please let me know if it's not fixed on the next image [11:17] davmor2: ditto [11:19] didrocks: so we need to decide how important the double apps is so alecu knows whether to land it as a smaller fix I think getting it in sooner rather than later we help us in a possible promotable image right? [11:23] Mirv: ogra_: even after installing all thepackages -> http://pastebin.ubuntu.com/7198266/ That is on a stock image... [11:24] vila, no code was changed from trunk related to the tests in the following => https://jenkins.qa.ubuntu.com/job/ubuntu-download-manager-trusty-amd64-ci/272/console [11:24] mandel: just back, what a stnc... [11:25] vila, as you can see, is in amd64 and the line that says task-0: FAIL! : TestDownload::testTotalSizeError() Compared values are not the same is a perfect example [11:29] bzoltan, a) this is not the latest image (the patch to disable adjust_soc messages from the mako kernel landed on monday), b) you seem to have ofono-phonesim-autostart installed, this isnt the case in automated testing (all packages used in previous tests get uninstalled there) [11:29] alecu: so personally the quicker that fix lands the better but if it is landing tomorrow that might be soon enough but I'll leave that in the hands of didrocks :) [11:31] ogra_: this error was the same with #275 ... [11:31] might be, just saying that the log is not from a recent image [11:34] mandel: ack, revno 251 is a no-change rebuild, good, now let's try to see if we can find some correlation with genie load... [11:34] mandel: no wait it's kinnara for 271 and 272 (at the start of the job: Building remotely on kinnara ) [11:35] alecu: yeah, the quicker fix if possible [11:36] didrocks: packaging app for mediascanner2 dropping its unused Unity scope https://ci-train.ubuntu.com/job/landing-012-2-publish/lastSuccessfulBuild/artifact/packaging_changes_mediascanner2_0.100+14.04.20140403-0ubuntu1.diff [11:36] ack, not app [11:37] Mirv: +1 [11:37] thx [11:37] sil2100: is there a CI problem? Jenkins doesn't seem to have done anything for 10 hours or so: https://code.launchpad.net/mir/+activereviews [11:39] mandel: except, I can't get usable load data :-/ [11:39] vila, I wonder, is there a way we can test that, for example, you can trigger a rebuild, correct? [11:40] vila, I don't have the rights to do it [11:40] mandel: sure, can't you ?? [11:40] >-/ [11:40] alan_g: I told you to ping the banfuard, so "cihelp" [11:40] (see /topic) [11:40] cihelp: is there a CI problem? Jenkins doesn't seem to have done anything for 10 hours or so: https://code.launchpad.net/mir/+activereviews [11:40] thanks :) [11:41] the landing team isn't piloting those CI jobs [11:41] alan_g: I guess it's due to the phone breakage in the datacenter though [11:41] as per the phone ML [11:41] proposed-migration et al stopping for battery replacement on their host [11:41] mandel: job/ubuntu-download-manager-trusty-amd64-ci <- that one right ? [11:42] alan_g: do you know about the corresponding jenkins jobs to speed things up ? [11:42] vila, http://s-jenkins.ubuntu-ci:8080/job/ubuntu-download-manager-ci/383/rebuild [11:42] vila, right? [11:43] mandel: ha ! better, thanks [11:44] mandel: but... then you can trigger the rebuilds no ? [11:44] mandel: or jenkins shouldn't give you a way to get that url (unless you crafted it manually " ;) [11:45] vila, the ps bot adds it as a comment, I don't have access [11:45] vila, => https://code.launchpad.net/~mandel/ubuntu-download-manager/upload-interface/+merge/213458 [11:45] mandel: ha great [11:46] vila: you mean something like: http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-ci/ [11:46] alan_g: perfect ! [11:47] alan_g: but there a re a bunch of jobs running there [11:47] alan_g: Waiting for the completion of mir-mediumtests-trusty-touch in http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-ci/1235/console [11:47] alan_g: image #274 killed all our makos [11:48] vila, I need to grab some food, I'll be back asap [11:48] vila: if didrocks is right about the phone hardware being AWOL then that's the explanation === mandel is now known as mandel|lunch [11:48] alan_g: fginther sent an email on the phone ML I think [11:48] * alan_g knows nothing about that email or list [11:48] grr, missed that in the log, yeah, didrocks is definitely right [11:49] vila, didrocks: thanks [11:49] alan_g: https://launchpad.net/~ubuntu-phone if you run tests on phones you should subscribe... [11:50] vila: now I know about it... [11:50] alan_g: good ;) Sorry, it's a bit messy today ;) [11:53] didrocks, davmor2: ack [11:57] mandel|lunch: so, http://s-jenkins.ubuntu-ci:8080/job/ubuntu-download-manager-trusty-armhf-ci/379/consoleFull and http://s-jenkins.ubuntu-ci:8080/job/ubuntu-download-manager-trusty-armhf-ci/380/consoleFull are currently running on the same arm node (cyclops-08) [11:57] mandel|lunch: that means you can encounter different load levels ... [11:58] mandel|lunch: which I suspect you never see when running on a phone [12:00] mandel|lunch: From the jobs I've looked at so far, your tests can run on (genie, kinnara) amd64, and cyclops-{08,09,10,11,12,13} arm [12:00] mandel|lunch: the amd64 nodes have 12 executors each, the arm ones 2 each === cjohnston changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: cjohnston | CI Train support - US: robru, cyphermox, rsalveti - EU: sil2100, Mirv, didrocks | CITrain support no answer: use mup bot after 30 minutes, but choose right timezone | Known issues: makos are down [12:00] mandel|lunch: so you have no guarantee about the load [12:02] mandel|lunch: executor is what jenkins call the bit that run a job, so 2 executors == up to 2 jobs running at the same time [12:03] ogra_: #275 on mako, I just rebooted and get the google logo, but adb is available to me.. [12:03] popey: the light ! [12:03] popey, great, so the container failed to start i assume [12:03] how do i tell? [12:04] you should see a lot of /system/bin/foobar in your processlist if it rins [12:04] guys, can someone please try adding a U1 account on the phone [12:04] *runs [12:04] nope [12:04] and thre should also be /init as well as /sbin/init in the processlist [12:04] nope, only /sbin/init [12:04] ogra_: r275 -> http://pastebin.ubuntu.com/7198400/ === alan_g is now known as alan_g|lunch [12:05] popey, check /var/log/lxc/android.log and /var/log/upstart/lxc-android-config.log [12:05] (and indeed syslog) [12:05] for me it seems to fail a lot earlier here though (see the ML) [12:06] /var/log/upstart/lxc-android-config.log: No such file or directory [12:06] ok [12:06] nothing lxc* in /var/log/upstart [12:06] well, see the android.log [12:07] lxc-start 1396526456.109 ERROR lxc_cgfs - cgroupfs failed to detect cgroup metadata [12:07] lxc-start 1396526456.109 ERROR lxc_start - failed initializing cgroup support [12:07] lxc-start 1396526456.109 ERROR lxc_start - failed to spawn 'android' [12:07] hmm [12:07] cgroup issues ... we had lots of changes in that area recently [12:08] let me upgrade to 275 too [12:08] (will take 30min or so) [12:08] ah, no wait, i can do ota [12:11] at least we know my emergency adb shell works :P [12:14] popey, no issues with 275here for me after OTA [12:15] hmm, odd [12:15] * ogra_ fires up the loop script [12:15] i did ota from 274 to 275 [12:15] popey, file a bug, but thats not the same issue [12:28] didrocks: so I keep getting this issue where the screen is greyed out and I think I know what the cause is. if you swipe the screen left to right to unlock it the welcome screen is not 100% off the screen same thing happens with apps from time to time so I'm going to add some more details to popey 's bug report [12:30] ok, i can reproduce on 275 === jhodapp|afk is now known as jhodapp [12:34] ogra_: tadda ! How many loop cycles ? [12:34] 23 [12:35] :-/ better than 62 ! :-} [12:37] cjwatson: any idea what's going wrong here? http://91.189.93.70:8080/job/generic-mediumtests-trusty/1917/testReport/junit/calendar_app.tests.test_calendar/TestMainView/test_new_event_with_mouse_/ [12:37] i watched the video and can't see why the test fails. [12:41] popey: hm, you sure you meant me rather than cjohnston? [12:41] geh, need coffee ;/ [12:44] goops [12:44] cjwatson: correct ☻ [12:44] proposed-migration back up, btw [12:45] popey: beats me.. maybe when fginther gets in he will have an idea [12:46] ok. [12:46] popey: is it only with that MP that its failing? [12:46] seems so. [12:49] ogra_, vila: yeah it seems pretty random - how many loops to reproduce it [12:49] right [12:50] ogra_, vila: I had one run through several hundred cycles last night [12:50] with no problem [12:50] i noticed the phone gets pretty warm when it hangs [12:50] oh my... if temperature comes into play.... [12:50] I'm having better luck with increasing the sleep like popey suggested though [12:50] Mirv: thanks for pressing m&c for webbrowser ;) [12:50] so i'm currently trying to hack top output into the initrd right before rin-init [12:50] Mirv: I'll assign a silo for the other webbrowser landing once this finishes [12:50] *run-init [12:50] thostr_: some more m&c for you! [12:51] sil2100: just pressed the button [12:51] sil2100: yep, that was the plan [12:52] plars: better luck with longer sleeps... could that be that the phone warms *more* ? [12:52] vila: wouldn't think so... if any thing it has more time to settle after rebooting [12:52] ogra_, plars: and cut power because of that ? [12:53] vila: I could set it on fire [12:53] plars: that's the idea, it doesn't settle but loop on something [12:53] plars: hehe [12:53] there is some process that hangs i think [12:53] vila: if that were the case, we would have seen a lot more systemsettle issues during testing I think [12:53] and we didn't see any [12:53] if i could top output anything but "unknown terminal type." then i could do another debug run :P [12:54] vila: if we were getting some process chewing up the cpu that is [12:54] plars: but systemsettle happens far later no ? [12:54] vila: happens pretty soon after boot, and again after the tests run [12:55] plars: right, but aiui, an lxc container is not even started when the phone hangs [12:55] it happens inside the initrd [12:55] before it goes to the rootfs [12:55] sil2100: well, thanks for monitoring the new silo assignment, I guess I should end my day again anyway... [12:55] ogra_: so before systemsettle ? [12:55] vila, before it even started booting [12:56] sil2100: landing-008 would be ready to be published as I wrote there, but somehow I stuck to the idea that someone else could confirm also on the bug report that the bug was fixed (which is what I already tested myself)... [12:56] ack, anyway, I'm not helping, muting myself [12:56] Mirv: ok, I'll try taking a look at that landing then - thanks :)! [12:56] fginther: when you awaken, could you please take a look at why https://code.launchpad.net/~pkunal-parmar/ubuntu-calendar-app/CalManagement/+merge/213355 fails, thanks! [12:57] vila, a boot goes (simplyfied): bootloader->kernel->initrd->rootfs->android-container->lightdm/unity [12:57] sil2100: thanks! [12:57] vila, when we hang we hang while exiting the initrd [12:57] popey, sure === alan_g|lunch is now known as alan_g [13:05] didrocks, btw, I can confirm I still get redirected to http:///job from time to time - usually after some time I've not been at the train (maybe reconnected network?) === wgrant_ is now known as wgrant [13:05] it would be nice if you guys could archive some of the landed line [13:05] it makes the list easier to scroll/read ;-) [13:06] Saviq, that happens to me as well [13:06] seb128: Saviq: the only ones having a clue about that are on the webops channels I'm afraid [13:06] maybe open a RT? [13:06] * Saviq does [13:07] I've another issue also when the first click on "build" doesn't work [13:08] it seems to just refresh the page [13:08] then clicking again works [13:08] seb128: can be a related issue, like as well, I can sometimes get logged in [13:08] and it refreshes to a page I'm not logged in [13:11] popey, I see this in the console log for the failing test: file:///usr/share/calendar-app/calendar.qml:89: TypeError: Cannot call method 'isSameDay' of undefined [13:11] popey, could that result in the app aborting? [13:13] could be. [13:19] dobey: can you review this branch? it will hopefully fix davmor2's issue: https://code.launchpad.net/~alecu/unity-scope-click/unique-name/+merge/214021 [13:28] hmm, no runaway processes when it hangs [13:33] didrocks: do you have time for a little chat with me ? [13:33] kgunn: I have for 1.5h of meetings, free afterwards [13:33] kgunn: that will be good for you? [13:33] didrocks: :) yep...that'll work [13:34] just question about updating a project with new packages etc (glmark2) [13:35] kgunn: ok, let's sync on that [13:38] seb128, boiko, Saviq: I assigned silos for you guys, but it seems the spreadsheet is taking its time with refreshing the status [13:38] sil2100, thanks [13:38] sil2100: thanks === mandel|lunch is now known as mandel [13:39] sil2100, ok, thanks [13:42] hm, actually, there seems to be a bug on the spreadsheet [13:43] didrocks: ok, I noticed that suddenly all rows from line 55 to the end of the spreadsheet have a different formula in columns C and L [13:44] didrocks: how are those added to those columns? Are they manually set in the spreadsheet or does the scripts fill those formulas in there? [13:44] * sil2100 looks into the code [13:48] Ah [13:48] didrocks: ok, so it seems that someone (?) modified the formula of the first landing request ;/ [13:49] didrocks: and the script uses the first landing row as the 'base' for the formula and all the new landing rows are broken - fixing that [13:51] didrocks: ok, fixed the first landing row and all the 'added' landing rows - now everything should have the 'correct' formula [13:52] Saviq, boiko, seb128: the spreadsheet fixed, you can build and work normally [13:52] sil2100, \o/ [13:53] sil2100, \o\ |O| /o/ [13:56] popey: what was the bug number for the grid layout issue I can't find it [13:57] davmor2: bug 1300302 [13:57] bug 1300302 in unity8 (Ubuntu) "Dead area at bottom when scrolling in scopes" [Undecided,Incomplete] https://launchpad.net/bugs/1300302 [13:57] popey: thanks dude [14:00] seb128: I'm doing a tender loving m&c of silo 18 [14:01] sil2100, thanks ;-) it's getting difficult to browse that list (did you see my comment about that an hour ago?) [14:03] seb128: argh, really? :/ [14:03] oopsss [14:03] sil2100 ^ [14:03] (sorry, in meetings) [14:03] sil2100: the script copy the formulas there [14:03] sil2100: copying from line 4 [14:08] didrocks: I know, as mentioned, I checked, noticed and fixed ;) [14:09] didrocks: no worries, your code is made easy-readable - sorry to have bothered, just wanted to keep you up-to-date [14:10] hello [14:10] can someone trigger a CI run for this MR? https://code.launchpad.net/~andrew-hayzen/ubuntu-ui-toolkit/fix-swipe-delete-002/+merge/202171 [14:11] t1mp: just leave an approval and it should re-trigger, right fginther ? [14:14] that was not the case before as far as I know. Is that new? [14:14] popey, t1mp, for CI runs, a push of a new version will automatically trigger a rebuild [14:14] davmor2: can you tell me the name of any of the apps that show up duplicated? [14:15] fginther: that is only the case if it is pushed by a canonical person, that is not the case here [14:15] t1mp, ahh, I wasn't aware that was the case... [14:15] alecu: ubuntu netwalk [14:15] fginther: but we are happy to accept patches from outside (as long as we review it) [14:15] great [14:15] fginther: if I leave an approval (not top-approve yet), it will trigger CI? [14:16] t1mp, one moment [14:17] didrocks, huh... I just saw http://people.canonical.com/~didrocks/citrain/silos/landing-004/ [14:17] didrocks, it contains the hud, but the hud isn't there in the silo [14:20] Saviq: let me see [14:20] sil2100, btw, that silo can be published [14:21] Saviq: I would have to check the source, but maybe it wasn't yet removing the old sources [14:21] Or maybe the rsync works somehow differently [14:21] uh oh, silo 004?! [14:21] Awesome [14:21] Saviq: let me publish then [14:25] davmor2, didrocks: I've got the fix for the duplicated icons bug, I've tested the deb from jenkins and it seems to work ok: https://jenkins.qa.ubuntu.com/job/unity-scope-click-trusty-armhf-ci/302/? [14:26] davmor2: if you can test it too, I'll ask for a silo for it in the meanwhile [14:27] didrocks: can I ask you to land it? all the landers in this sprint are in a meeting right now. [14:28] this is the branch: https://code.launchpad.net/~alecu/unity-scope-click/use-tested-search/+merge/212252 [14:29] alecu: sure (or maybe sil2100 can handle it as I'm in meetings) [14:29] thanks a lot alecu :) [14:29] sil2100: weird, there was a first publication for the hud, I'm not sure where the second .project comes from [14:30] t1mp, sorry for the confusion, adding an approved comment does nothing. For an MP from a non canonical member to be automatically tested, it must be top-approved. So if you just need a ci run, that needs to be triggered manually, which I can do [14:31] alecu: will do [14:31] fginther: please do [14:31] t1mp, already done :-) [14:32] fginther: thanks [14:32] fginther: is there a configuration that gives me a way to trigger it? then I don't need to bother you in future [14:39] didrocks: hm hmmm... I'll look further [14:39] alecu: let me backlog [14:40] ogra_: sil2100: didrocks: We have a problem with the r275 ... The QtC can not connect to the device with that image [14:41] what is the QtC ? [14:41] ogra_: QtCreator [14:41] sil2100: yes [14:41] alecu: can you set the branch to 'Needs review'? [14:42] alecu, didrocks: I'll add a landing for that one [14:42] bzoltan, via adb you mean ? [14:43] bzoltan: i'm here [14:44] kgunn: available now [14:44] didrocks: packaging ACK! https://ci-train.ubuntu.com/job/landing-004-2-publish/lastSuccessfulBuild/artifact/packaging_changes_unity8_7.85+14.04.20140403.1-0ubuntu1.diff <- from the packaging side, simply one additional file to be installed [14:45] second meeting was shorter than expected :) [14:45] plars, with the rebooted makos, did you start another test run for 275 ? [14:45] sil2100: ack for the "nonmirplugins" :p [14:45] ;p [14:45] didrocks: thanks! [14:46] davmor2: 5. and 6. are new? [14:47] davmor2: you didn't list the adb blocking one [14:47] davmor2: as the device doesn't reboot, this seems a promotion blocker to me [14:47] didrocks: ok... :) [14:48] kgunn: https://plus.google.com/hangouts/_/7acpjkahe4pge84i6aa08l6d9c [14:48] didrocks: 5 and 6 were ones that were really hard to reproduce popey and I have hit them on and off . The one is the one I showed in the call this morning the other is one that is annoying when it hits but is temperamental at best and were just missed out I think [14:49] didrocks: I've not been hit by the adb bug but I would agree that is a blocker it there a bug for it? [14:50] alecu: I changed it to 'Needs review' if anything [14:50] alecu: and assigning a silo [14:50] sil2100: I'm sorry, I pasted a wrong url [14:50] sil2100: that's not the branch [14:50] sil2100: it's this one: [14:50] https://code.launchpad.net/~alecu/unity-scope-click/unique-name/+merge/214021 [14:51] ...crap... [14:51] :| [14:51] Grrrr [14:51] I'm so so sorry :-( [14:53] ogra_: you do have a bug for the "I lost my phone" bug? [14:53] or even "OMG I lost my phone"? :p [14:54] didrocks: I don't know about that but there is a Oh crap how do I find my phone [14:54] ogra_: sil2100: no, from the Ubuntu SDK (QtCreator) just plug in the device and it should be able to connect to it [14:54] davmor2: heh ;) [14:54] sil2100: I asked kalikiana in #ubuntu-app-devel about https://bugs.launchpad.net/ubuntu-calculator-app/+bug/1289695 - is there an upcoming slot he can have to get that in soon? [14:54] ogra_: sil2100: apparently it needs a dpkg-reconfigure openssh-server [14:54] Ubuntu bug 1289695 in Ubuntu UI Toolkit "Title for all Core apps should be improved/localized for Desktop" [High,In progress] [14:54] didrocks: ring it seems to be the answer not very productive :D [14:55] popey: let me take a look [14:55] bzoltan: is it relying on being able to ssh as root with a password? [14:55] bzoltan: oh? [14:55] that would be odd, that's the only thing that changed [14:55] well, not only, but the most likely thing to matter [14:55] cjwatson: no...we do as phablet and we create key [14:55] bzoltan: so I don't get how dpkg-reconfigure would matter, given that permitrootlogin without-password is the only thing handled by debconf [14:56] is something forgetting to create host keys on deployment? [14:56] cjwatson: All I know that with the recent image the device connectivity stopped working [14:56] popey: we might have an additional free silo in a moment - the problem is that there is one landing for UITK prepared anyway [14:56] I was asking the channel as a whole [14:56] popey: so it's basically locking UITK for now until that's released... [14:57] cjwatson: and dpkg-reconfigure fixes it [14:58] dpkg-reconfigure would no doubt have the side-effect of creating host keys, but that hasn't changed lately - if they were in place before, they should still be [14:58] cjwatson: yes > http://pastebin.ubuntu.com/7198938/ [14:58] right, so I wonder what was creating those before? [14:58] cjwatson: ogra_: I did install the image with --wipe [14:59] bzoltan, why do you fiddle with ssh-server ? [14:59] cjwatson, lxc-android-boot.conf was creating them on first boot [14:59] ogra_: I do not :) [15:00] you just said you did dpkg-reconfigure [15:00] ogra_: start on staring ssh [15:00] ogra_: that was what fixed it [15:00] spot the typo [15:00] ogra_: while you're there, you should sync up the set of host keys generated with openssh-server.postinst [15:00] cjwatson, yeah, my unconcious developer brain probably thought it shouild stare back :P [15:00] staring ssh :) [15:00] * didrocks loves that [15:01] to add ecdsa/ed25519 [15:01] cjwatson, will do [15:01] bzoltan, does any of you code call "start ssh" ? it would be nice if QtC started using the property instead [15:02] alecu: so that deb doesn't seem to want to install http://paste.ubuntu.com/7199048/ [15:02] (until we have a proper developer-mode that sets writalke and the ssh property) [15:02] *writable [15:03] ogra_: we start the ssh if it is not started [15:03] ogra_: what do you suggest to use instead? [15:04] bzoltan, one sec ... i have to look up the property name and am in a meeting [15:04] so, yeah, regression in lxc-android-config 0.156. that's a relief, I thought it might somehow have been a regression in openssh 1:6.6p1-1 [15:05] ogra_: thanks [15:05] cjwatson, if it comes to touch ssh issues, ping me first :) [15:07] bzoltan: I'm running the tests for silo 007 but something is messed up [15:07] bzoltan: even though I ran adb shell powerd-cli display on &, the display turns off some times [15:08] t1mp: not cool === doanac changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: doanac | CI Train support - US: robru, cyphermox, rsalveti - EU: sil2100, Mirv, didrocks | CITrain support no answer: use mup bot after 30 minutes, but choose right timezone | Known issues: makos are down [15:09] bzoltan: results so far http://paste.ubuntu.com/7199076/ [15:09] bzoltan: gallery-app is supposed to fail because its test code is broken. I don't know why it is OK on the dashboard, maybe different version of python? But that's weird since we have the same image.. [15:11] bzoltan: other failures (system settings) I have no idea what that is ProcessSearchError: Search criteria (pid = 14512, dbus bus = 'session', object path = '/com/canonical/Autopilot/Introspection', process object = '') returned no results [15:11] I'll let the tests continue to see what the other apps do [15:29] bregma: reminding about m&c! :) [15:29] sil2100, thanks [15:34] ogra_: did you see my question about having a bug number for the initrd issue? [15:40] bzoltan, you want to use "setprop persist.service.ssh true" ... that sets a persistent property so ssh keeps running across reboots [15:40] ogra_: thanks [15:41] didrocks, yes, sorry, didnt egt to file one yet ... [15:41] *get [15:41] * ogra_ is actually having some breakfast atm :P [15:43] ogra_: enjoy! :) [15:52] Saviq, I resolved the conflict in silo 13 and hit rebuild for you, should be ready to test soonish [15:52] robru, oh thanks [15:52] hrm, now there's a new merge conflict... [15:53] robru, we'll take care of it [15:53] pete-woods, https://ci-train.ubuntu.com/job/landing-013-1-build/5/console [15:54] Mirv: when you have a moment could you please push music 409 from http://s-jenkins:8080/view/click/job/music-app-click/ - http://s-jenkins:8080/view/click/job/music-app-click/lastSuccessfulBuild/artifact/out/com.ubuntu.music_1.3.409_armhf.click to the store? [15:54] Saviq: davmor2 pinged you about: [15:54] https://bugs.launchpad.net/ubuntu/+source/unity8/+bug/1300326 [15:54] Ubuntu bug 1300326 in unity8 (Ubuntu) "Shell randomly freezes with grey tint" [High,Confirmed] [15:54] https://bugs.launchpad.net/ubuntu/+source/unity8/+bug/1300302 [15:54] Ubuntu bug 1300302 in unity8 (Ubuntu) "Dead area at bottom when scrolling in scopes" [High,Incomplete] [15:54] right? [15:55] (as he set them as blockers) [15:55] alecu: the silo has packages built - silo 17 ;) [15:55] didrocks: no that was my next job [15:56] didrocks: I just got a list together initially to ensure we weren't missing any [15:56] davmor2: hum, you should ping them asap so that they don't discover new issues [15:56] davmor2: seems we lost a day on the second for instance [15:57] didrocks: indeed but Saviq picked up on both but it is only today going back over all the recent bugs that I realised that they had fallen through the cracks [15:58] sil2100: thanks, testing [15:58] \o/ [15:58] ok [16:03] cyphermox: coming? [16:03] yes [16:10] didrocks, davmor2, Albert and Michael will look into those [16:11] Saviq: awesome thanks [16:12] davmor2, sil2100: I'm testing silo 17, and the issue seems to be fixed. [16:12] alecu: I might of grabbed it before it finished [16:12] davmor2: I was unable to reproduce the broken .deb from jenkins, it worked ok for me. [16:12] alecu: awesome - did you try the whole unity-scope-click test-plan? [16:13] sil2100: I'm on it [16:14] alecu: thank you :) [16:14] it will take a little while, since I'm on 274 and need to reinstall === jhodapp is now known as jhodapp|lunch [16:27] Saviq: thanks! [16:28] robru: published 001 [16:28] sil2100, thanks [16:36] sil2100, davmor2: I've completed the unity-scope-click test plan. It works as expected (minus the bugs noted in the test plan) [16:36] sil2100, davmor2: I think it's ready to go. [16:36] thanks for all your help! [16:36] alecu: \o/ [16:39] alecu: \o/ [16:39] alecu: thanks! [16:39] plars, i believe I have a fix for the suspend-blocker issue - i've updated asana with some notes [16:40] ogra_: you mean in the regular ci tests? or at home to try to see if this bug is reproducible in 275 (sorry for the delay, had to take my son to the doctor) [16:40] cking: cool, I'll take a look. thanks! [16:40] plars, yeah, i heard, all ok ? [16:41] plars, i meant in the lab [16:41] ogra_: yeah, he was having really bad swelling and knee pain, and had already confounded one specialist as to the cause [16:41] ogra_: I heard the device were back up, but that was after I had already left. Let me take a look now [16:42] only try on one so we dont kill them all :) [16:43] ogra_: yeah, no kidding! [16:44] ogra_: looks like psivaa started one but it's stuck in the delay for image. I just restarted it without that so we can get it rolling quicker [16:44] yeah [16:44] ogra_: it's running at http://q-jenkins.ubuntu-ci:8080/job/trusty-touch-mako-smoke-daily/215/console [16:45] now if my laptop had a working vpn :P [16:47] robru: published 17 now, leaving the landings in your hands now ;) [16:47] sil2100, great thanks [16:48] ogra_: any progress on this problem with boot getting stuck? Any ideas yet? [16:48] plars, not really, we decided to look at different images now ... it is clear that it hangs at or right after run-init [16:49] plars, we (popey, davmor2, cyphermox and I) are running your reboot script with different image numbers. so far image 271 is up to 23rd reboot without issue [16:49] i checked the processlists and top output right before run-init when the issue happens, nothing special visible there [16:49] robru: ogra_: I had started something similar while I was away, I'm running it on image 273 [16:49] my best guess would be a kernel issue [16:49] I'm on loop 88 (with 2 min sleep) [16:50] also popey claims it crashes earlier if you raise the sleep [16:50] so we agreed to use 30sec at least with your script [16:50] yeah, we raised it from 10s to 30s [16:50] with the 10 sec sleep, I got over 500 loops last night on 274 [16:50] so it's really hit or miss [16:50] damn [16:50] plars,how many reboots doe a device usually have during a day [16:50] *does [16:51] (hust a rough guess) [16:51] *just [16:51] ogra_: roughly 30 reboots per image (setup + tests) [16:51] mine has run 40 times now [16:52] 30 isnt that much [16:52] no, it isn't [16:52] i had it often enough only die at 80 or above today [16:52] ogra_: yeah, but remember that it runs tests beforehand [16:52] ogra_: so, can become hot because of that [16:52] if your theory is right [16:53] well, my theory is born out of desparation :P [16:53] heh, I had that feeling! [16:53] dont trust it :) [16:53] ogra_: do you think I did? :p [16:53] ha [16:53] * didrocks thinks it's time to run away :p [16:54] didrocks: see you next week [16:54] ☻ [16:54] popey: oh right, enjoy your Friday! [16:54] thanks [16:54] vacation ? [16:54] popey: keep us posted on the ML on oyur latest findings [16:54] ogra_: you should really listen to the meeting, you can't blame popey for the accent! :) [16:55] lol [16:55] well, i was watching a phone reboot :P [16:55] popey: see, ogra_ prefers watching a phone than you! [16:55] not sure how I would take it… [16:55] see you guys ;) [16:55] bye [16:57] ogra_: yeah, day off while kids are off school. going to a science museum ☻ [16:57] nice ! [16:57] enjoy === alan_g is now known as alan_g|EOD [17:04] ogra_: so you suggest to apply this on the script we use -> http://pastebin.ubuntu.com/7199576/ [17:11] seb128, got you silos 4 and 5 [17:11] robru, thanks! [17:11] seb128, you're welcome! [17:12] bzoltan: (BTW it's a good idea to use ">/dev/null 2>&1" to avoid gratuitously bash-specific shell code) [17:13] (rather than "&> /dev/null") === plars changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: plars | CI Train support - US: robru, cyphermox, rsalveti - EU: sil2100, Mirv, didrocks | CITrain support no answer: use mup bot after 30 minutes, but choose right timezone | Known issues: makos are down [17:18] cjwatson: OK, thanks... but generally that replacement should be OK, right? [17:22] bzoltan, right, you might want to run the ssh start once if you dont reboot the device, the property will only kick in after a reboot [17:23] ogra_: so you suggest to keep the line we use now? Because I do not really want to reboot the device [17:26] bzoltan, http://paste.ubuntu.com/7199689/ ... that way you will only set it once [17:26] ogra_: 100 reboots [17:26] ogra_: ohh... that looks smart. Thanks a lot [17:26] #269 [17:26] popey, no hangs ? [17:26] nope [17:26] i'm at 64 on 275 [17:27] still looks fine [17:29] ogra_: https://pastebin.canonical.com/107744/ is a last_kmsg from a device that had this issue in the lab. rfowler grabbed it [17:29] psivaa, yeah, shows the same as all the others [17:29] ogra_: ack [17:29] hangs at run-init or right afterwards when starting /sbin(init [17:30] popey, ogra_ so far I'm at 67 reboots on 271. [17:30] results are starting to flow for 275 now on mako - so far the device is still up [17:30] robru: mine is still going on 271 at 105 reboots, do you know if anyone has managed to reproduce this on anything less than 274 yet? [17:31] plars, I'm not aware of anything less than 274 having the issue, no [17:31] robru: I'm going to let mine run a bit longer, but just checking [17:32] plars, ok [17:32] popey, ogra_ : who was doing what image again? popey had 269, i had 271... i forget who else is doing this [17:33] robru, i'm doing 275 now and will stop at ~150 loops ... thn i'll reinstall and try 275 in readonly mode [17:34] ohhhhh, readonly mode. that's probably a good idea to try. [17:34] ogra_, how do you get back to readonly mode? bootstrap option? [17:35] yeah, i usually have --bootstrap and --wipe set when doing test installs [17:35] (might be redundant :) ) [17:35] hmmmm ok i'll try that if I get to 200 or so [17:36] robru, popey, ogra_: have you guys hit the reboot issue on your images? [17:37] davmor2, not on 271 [17:37] davmor2, not yet [17:38] mine just seems to be chugging along no issues yet on 273 [17:38] i have a feeling that making the sleep longer actually makes it fail less than more [17:45] davmor2: not on #269 - 130 reboots [17:45] I can't say I'm surprised, given we only saw this in #274 really, it came on overnight [17:45] tea got in the way for me so I'm only on 30 [17:46] you give your phone tea ? [17:46] popey: I agree but at least taking 3 older images if they are all perfect then we know for sure that it is only 274 so it is a change from 273 to 274 that is causing it [17:47] sure [17:47] well, i saw it for sure on 275 [17:47] ogra_: no I had tea :P [17:47] but not with the 30sec sleep [17:47] well, i got it with 2 min sleep on 274 [17:48] after only 12 goes [17:48] yeah, you said so [17:48] i really wonder if it is because the image is writable and we dont do proper reboots [17:48] (adb reboot is like hitting the reset button on a PC) [17:48] mine isn't writable. [17:48] your 274 [17:48] ? [17:49] i haven't made it un-writable since last night [17:49] and there's no /userdata/.writable_image on mine [17:49] oh, interesting [17:49] that saves me from doing some test [17:49] (but also trashes another potential theory) [18:04] 163 and still going [18:05] 43 and still going [18:05] 102 [18:06] and many many nautilus windows :P [18:07] oh, 102 actually got me the hang ... [18:07] ogra_: right click the icon in the launcher and quit [18:08] so 275 still has it [18:08] davmor2, yep, i know :) === jhodapp|lunch is now known as jhodapp [18:17] yeah I'm at 117 on 271, no issue [18:22] 60 now on 273 [18:30] 207 on #269 [18:41] image 271 got up to 142, I'm gonna reflash in readonly mode and try again [18:44] ogra_: how far do you want me to go with this? I'm at 230 reboots of #269... [18:45] popey: what timeout did you have [18:45] 30s [18:45] i can try again with longer or shorter timeout [18:45] while I make dinner [18:46] so why am I still on 80 I was on 30 and you were on 130 [18:47] http://paste.ubuntu.com/7200015/ [18:47] thats my script [18:47] popey: ah I'm using the one that plars posted [18:48] only 30 secs rather than 10 [18:48] Thu Apr 3 17:27:59 BST 2014 [18:48] thats when i started [18:49] so its been 2.5 hours [18:49] I upped mine to 120 after popey said that was working better for him [18:49] davmor2: popey: which images are y'all testing? [18:49] I've made it through 137 loops now with 273 [18:50] #269 here [18:50] plars: we are all doing different ones so I'm testing 273, robru 271 and popey 296 [18:50] 269 even [18:51] 85 [18:51] and none of you have reproduced so far? [18:51] plars: nope [18:51] ogra_: has on 275 [18:51] lol, I just did a --bootstrap --wipe flash of 271 and unity won't start [18:52] plars: and popey and ogra_ did on 274 [18:52] I suspect my mako in the lab running on 275 just died [18:53] robru: D'oh forgot about that, that is when didrocks broke the image [18:53] I'm watching for a bit longer but it's been stuck on the dialer-app test reboot for a while now [18:53] davmor2, but earlier when I flashed 271 it was loading fine. it was specifically the --bootstrap --wipe that broke it [18:53] davmor2, at least, adb is able to reboot the device for the purposes of this test, but yeah, i'm not getting unity8 starting at all [18:54] robru: if you install without wiping it is fine as it would of had the missing deps the wipe kills it :) [18:54] ahhh ok [18:59] plars, before i go to bed i'll fire up a test with all kernel debug options enabled etc ... lets see, perhaps we have an oops we cant see [19:02] 95 [19:04] right, bored of this game now. going to re-run with 2 min delay again on #269, just in case 30s isnt long enough [19:04] 265 successive reboots and no lockup, seems enough to me ☻ [19:07] * ogra_ adds "debug loglevel=6 log_buf_len=1M" to the kernel commandline ... lets see if that spits out more [19:07] LOOP 1 [19:07] worked :) [19:09] ogra_: well done ! hold it tight ! [19:10] hmm, i was expecting more noise from loglevel=6 [19:10] ogra_: publish it [19:10] 103 [19:12] dmesg looks pretty normal with loglevel=6 http://paste.ubuntu.com/7200123/ [19:12] * ogra_ changes to 7 [19:22] loglevel=7 --verbose debug ... that looks mildly informative [19:23] (and log_buf_len=1M else dmesg eats itself) [19:30] ogra_: after 120 loops I got this instead of the nuatilus window Unable to open a folder for Nexus 4 Cache invalid, retry (internally handled) [19:30] yeah [19:30] known [19:30] ogra_: that's fine then I'll ignore it :) [19:30] happens on the manta more often than on the mako [19:31] 122 [19:31] well, not sure, i think cyphermox has a bug open for it already [19:31] if not, opening one might make sens [19:31] e [19:31] eh? [19:32] cyphermox, an mtp popup message that rarely shows up on the desktop [19:32] "Cache invalid, retry (internally handled)" [19:33] i see that too every 100th boot or so [19:33] (or 200 ... or whatever) [19:33] ah [19:34] yeah that's an issue in gvfs actually, I think [19:34] if you click in nautilus all is fine ... its just that starting of the nautilus window thats missing when it happens [19:35] I suspect at some point the caching fails, it's a little weird [19:36] yeah, its a low prio bug for sure [19:36] it works 99% of the time [19:37] cosmetic noise :) [19:45] * davmor2 calls it a night and leaves the looper looping [19:45] heh [19:52] can I get a silo for l62? [20:07] doanac: ping [20:08] sergiusens: what's up? [20:09] doanac: I have silo 10 with phablet-tools changes, was wondering if we could trigger a full run again [20:10] sergiusens: i'll give it a shot. we tried yesterday but all the makos keep getting stuck offline [20:10] doanac: right... forgot about that [20:10] plars: mind if i run: http://q-jenkins:8080/job/andy-smoke-daily-test/ for sergiusens ? [20:11] or do we need the makos for other stuff right now? [20:11] doanac: I could take a leap of faith; the changes look sane; but there's an execution change in phablet-test-run for shelling in and I don't want to find out later that this broke something [20:11] doanac: you are welcome to try, but... [20:12] plars: if nothing is passing its probably not worth trying [20:12] is it worth it? [20:12] doanac: when you can get it to go, it's working: http://ci.ubuntu.com/smokeng/trusty/touch/mako/275:20140403:20140331/7545/ [20:13] doanac: but at some point, it's likely to fail [20:13] sergiusens: so here's the limited results from yesterday before we killed the job: http://q-jenkins:8080/job/andy-smoke-daily-test/5/testReport/ [20:14] looks like it could test friends-app, so maybe we can call it good? [20:17] doanac: yeah, I guess so; I ran a couple of mine; just wanted to make sure the infra didn't break [20:17] seems it didn't [20:17] thanks, should be fine [20:17] thanks [20:18] robru: hey, can we publish silo 10? [20:18] sergiusens, done! === jhodapp is now known as jhodapp|bbl [20:19] ty [20:19] yw [20:24] ogra_, popey, davmor2, plars: 250+ loops on image 271, no sign of the problem. want me to try 272 now? or is there a plan? [20:25] robru, i think i found something [20:25] for whatever reason udev starts while the container is coming up [20:27] (it should only start after the container) [20:30] hmm, and i also see why it starts to early ... i just dont get why that hasnt bitten us before [20:37] ogra_: shall i stop rebooting my device? [20:38] 44 runs at 2 min interval on #269, its fine [20:38] popey, is there a bug report for this one? [20:38] popey, well, you could try a fix on an image that we know has the issue [20:38] pmcgowan, nope [20:39] popey, in 274 or 275 edit /etc/init/udev.override ... change lxc-android-boot to lxc-android-config and see if that fixes it [20:39] robru, ^^^ in case you want to try [20:40] seems i cant trigger it in 275 if all debugging is enabled :( [20:40] 95 loops yet [20:40] pmcgowan, not yet [20:40] ogra_, ok, i'll try that [20:41] your blaming of systemd wasnt that wrong [20:41] i guess [20:41] just that the bug isnt in systemd/udev itself [20:42] * ogra_ hopes that doesnt make the boot to slow :( [20:42] ogra_, lol, that was just a joke (actually i mailed that privately, not on list) because it seems popular to hate on systemd ;-) [20:42] lol [20:42] i didnt notice it was private, heh, sorry [20:43] i see it now :P [20:44] ogra_: happy to try that :D [20:45] nice one to leave running over night i guess [20:45] popey, did you file a bug on UI slowness? or was that a rumor [20:45] gah, never rotate phone in system-settings -> updates [20:45] pmcgowan: anecdotal [20:45] need to test more [20:48] ogra_, I can file a bug for this if you tell me where and what to say ;) [20:52] pmcgowan, booting mako hangs randomly right after run-init [20:53] pmcgowan, Every 10-100 boots (in a really random manner) the boot of mako devices gets completely stuck. This started with image 274. [20:54] ogra_, which package/project? [20:54] thats the issue :P [20:54] hah [20:54] file it against lxc-android-config for now, if my theory is right the fix will live in there [20:54] ubuntu [20:54] ok [20:55] ogra_, hah, I just flashed 275 for the first time and it got stuck at the Google logo on the very first boot [20:55] ouch [20:55] ogra_, complete with "device not found". are there any logs I can provide? what/how? [20:56] robru, reboot into recovery (by holding vol-down), once there: adb shell cat /proc/last_kmsg ... [20:57] robru, if the last thing (before the power button shuts down the CPUs) you see in there is the swapon output, it is the same issue [20:58] ogra_, crap, rebooted into the phone first, I guess I lost that log. [20:58] yeah [20:58] ogra_, https://bugs.launchpad.net/ubuntu/+source/lxc-android-config/+bug/1302174 [20:59] Ubuntu bug 1302174 in lxc-android-config (Ubuntu) "Booting mako hangs randomly right after run-init" [Critical,Confirmed] [20:59] thanks ! [20:59] that will make all the difference fixing it ;) [21:00] robru: no I think the plan is that with me covering 273 you 271 and popey 269 we would lower down the releases that the issue was introduced looks like it was 274 I'm on 232 no issues [21:01] ogra_: think that means 274 is where the issue was introduced at least right? [21:01] yes [21:01] davmor2, but i could reproduce it reliably on 275 too [21:03] ogra_: oh yeah I'm saying we were lowering down where the issues was introduced so we know it is 274 now but the effect of that will be with us I guess until someone fixes the issue right? [21:06] ogra_, I can't ssh into image 275... just says 'connection closed'. I tried both ssh over wifi, and also over a port-forward bridge through adb/usb [21:07] robru, yeah, there is a typo ... dpkg-reconfigure openssh-server [21:07] i'll fix that with the next lxc-android-config upload [21:08] (key generation was moved to its own upstart job, which has a typo in the start condition) [21:08] ok thanks [21:11] right /me calls it a night catch you in the morning [21:20] boiko, i'm just about to take lunch, do you need a hand with silo 9 before I go?(just noticed the build error) [21:21] robru: well, renato updated the MR but it seems jenkins didn't notice that when I tried to rebuild [21:21] robru: I just used force rebuild, let's see if this works [21:21] boiko, yeah, i was just gonna say, force rebuild should do it [21:21] robru: great, thanks [21:21] boiko, you're welcome [21:21] ok, bbl [21:22] boiko, oh, that failed too :-/ [21:22] boiko, try force rebuild + ignore step [21:22] boiko, nm, i just did it [21:23] boiko, looks good: https://ci-train.ubuntu.com/job/landing-009-1-build/6/console [21:23] bbl for real ;-) [21:23] robru: weird, it is trying to apply revision 48, but the MR already got the revision 49 [21:23] robru: ah ok, now it got the rev 49 === plars changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: cihelp | CI Train support - US: robru, cyphermox, rsalveti - EU: sil2100, Mirv, didrocks | CITrain support no answer: use mup bot after 30 minutes, but choose right timezone | Known issues: makos are down === jhodapp|bbl is now known as jhodapp