/srv/irclogs.ubuntu.com/2014/04/03/#ubuntu-ci-eng.txt

robrualright, I'm off for dinner, will be back later though00:14
cyphermoxrobru: btw, silo 11 had an issue uploading the the PPA, I'm not sure why01:22
cyphermoxa following run of the build job seems successful now01:22
cyphermoxdo you have access to the ps-jenkins list to see why the uploads were rejected or something?01:23
cjwatson2014-04-03 00:07:12 INFO    Failed to parse changes file '/srv/launchpad.net/ppa-queue/incoming/upload-ftp-20140403-000629-025651/~ci-train-ppa-service/landing-011/ubuntu/indicator-applet_12.10.2+14.04.20140402-0ubuntu1_source.changes': GPG verification of /srv/laun01:25
cjwatsonchpad.net/ppa-queue/incoming/upload-ftp-20140403-000629-025651/~ci-train-ppa-service/landing-011/ubuntu/indicator-applet_12.10.2+14.04.20140402-0ubuntu1_source.changes failed: Verification failed 3 times: ["(7, 9, u'No public key')", "(7, 9, u'No public key')", "(7,01:26
cjwatson 9, u'No public key')"]01:26
cjwatsonlooks like transient trouble talking to the keyserver01:26
cjwatson(is my guess)01:26
kgunnjust looking for a silo for line 5101:52
kgunn(btw, silo 12 ready to merge...so... :)01:53
imgbot=== trainguard: IMAGE 275 building (started: 20140403 02:05) ===02:05
Mirvmorning04:16
Mirvhmm, how that 275 has taken >2h to build04:17
stgraberthere's a bit of backlog on the system-image importer04:23
stgraberit was in manual mode while setting up generic_x86 and I forgot to turn it back on (looks like nobody noticed until now). I've re-enabled the cronjob and it'll start processing the latest images now.04:24
stgrabersince we now have two rootfs images to process, expect the processing time to almost double (not much we can do there...)04:24
Mirvright04:25
Mirvthanks for fixing it04:26
Mirvand it seems testing is also halted for a bit longer time probably04:46
imgbot=== trainguard: IMAGE 275 DONE (finished: 20140403 05:10) ===05:10
imgbot=== changelog: http://people.canonical.com/~ogra/touch-image-stats/275.changes ===05:10
robruogra_, Mirv: wow, 3 hour image build, longest one I've ever seen!05:11
Mirvthat was indeed a long one, but great that it happened at last. too bad we won't get test results.05:14
didrockspsivaa_: hey, I think we'll have to run all the tests ourselves manually to know what's the image status is if we don't have devices, you do have a mako, right?06:29
psivaa_didrocks: no. i dont have a mako06:54
didrocksok, Mirv: do you think you would have time for that?06:57
didrocks(like a background task, and don't do landings meanwhile)06:57
Mirvdidrocks: I can squash two bugs in a one run, since I'm soon about to test a qtdeclarative update06:58
didrocksMirv: yeah, that's a good plan then! :)06:58
didrocksoh, the date format fix?06:59
didrocksit's already there, nice!06:59
Mirvwell Pat suggested that "maybe these two lines would work", so I'm testing that now and if it works properly without regressions I'll submit it upstream06:59
Mirvfirst findings is that yes it seems to work, but the real test is with RSS Reader and then seeing nothing regresses07:00
didrocksyeah07:01
=== psivaa changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: psivaa | CI Train support - US: robru, cyphermox, rsalveti - EU: sil2100, Mirv, didrocks | CITrain support no answer: use mup bot after 30 minutes, but choose right timezone | Known issues: -
ogra_didrocks, seems stgraber added i386 to the system-image builds last night ... which he said he can only run serialized ... which in turn resulted in the last image taking 3:05h to build08:24
ogra_(i'll talk to him tonight to rip it out again and find a way to parallelize that, but todays builds will most likely take that long :/ )08:25
didrocksogra_: ok, thanks for the head's up08:25
cjwatsonogra_: He said fairly clearly in scrollback that most of the delay was due to the cronjob being accidentally disabled for a while08:27
ogra_cjwatson, oh, right, so thats not the real time yet ... he warned me that i should look at the build time since it would be significantly longer08:27
cjwatsonogra_: So please see how long the next build takes before taking any action08:27
ogra_right08:27
ogra_thanks for the heads up, i totally missed that above08:28
bzoltanogra_: could you please tekk me what do I do wrong? -> http://pastebin.ubuntu.com/7197747/ FAILED (failures=18)08:34
Mirvbzoltan: I wonder if phablet-config autopilot --dbus-probe enable is needed, that's what I do before running tests as mentioned at https://wiki.ubuntu.com/Touch/Testing#Running_Click_tests08:38
bzoltanMirv: OK... I try that08:39
sil2100didrocks: packaging ACK needed -> https://ci-train.ubuntu.com/job/landing-015-2-publish/4/artifact/packaging_changes_unity-scope-click_0.1+14.04.20140402-0ubuntu1.diff (new dep, it's in universe but -scope-click is as well)08:41
didrockssil2100: +109:05
psivaaogra_: popey: contrary to what i said in the meeting, we now have a flo impacted by the reboot issue, with 27509:44
ogra_ugh09:45
ogra_psivaa, can you ask whoever does the reboot to capture /proc/last_kmsg from recovery ?09:45
psivaaogra_: ack, will do that.09:46
popeyogra_: http://paste.ubuntu.com/7197823/09:46
ogra_(it is important that there is no reboot between the hang and getting that log)09:46
didrocksogra_: hum, with 275, which is supposed to adb wait, have all tests failing, and then reboot?09:47
=== vrruiz_ is now known as rvr
ogra_didrocks, well, depends where it actually hangs ... if it is run-init there is no way to get any adbd up09:48
didrocksok09:48
ogra_what psivaa posted above doesnt seem to hang in run-init though09:48
didrocksyeah09:49
didrockscan we centralize all that into a bug?09:49
didrocksthat will make tracking/conversation better09:49
ogra_[   25.702487] transport_store: usb cable is not connected09:49
ogra_interesting09:49
bzoltanMirv: that dbus-probe does not help09:51
ogra_if it actually thinks the cable is disconnected we cant really do much :(09:52
bzoltanMirv, ogra_, didrocks: stock image ... not even a single app gives good results.09:52
didrocksbzoltan: hum, that's not what Mirv is experiencing though?09:52
didrocksas he's rerunning everything manually09:52
didrocksbzoltan: unrelated to the boot issue I guess?09:52
didrocks(you are booted)09:53
ogra_bzoltan, note that the shell handling changed, make sure to be on the latest phablet-tools etc09:53
bzoltandidrocks:  I am booted09:53
bzoltanogra_: I check that09:53
ogra_foundations dropped a few things from the login process (and added others)09:53
bzoltanogra_: Installed: 1.0+14.04.20140401-0ubuntu109:54
bzoltanseems to be fresh09:54
bzoltanin the logs I see %CLICK_FRAMEWORK%-%CLICK_ARCH% [here comes the directory listing of my ~]  (process:8732): WARNING **: Unable to register app: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: Invalid application ID09:55
Mirvbzoltan: so far I haven't had problems with #27510:09
bzoltanMirv: I do not know what the problem is ... I wish to understand. I am sure it is not about the image... but the tools10:10
davmor2Morning all10:11
sergiusensogra_: it thinks it's disconnected on the host or device side?10:13
sergiusensare we back to the device online and restart adb server to get it back situation?10:13
ogra_sergiusens, well, it shouldnt10:13
ogra_oh, i havent tried that yet10:13
ogra_let me reproduce the error ... i'm at the 25th loop, still working10:14
ogra_it is so annoying that it takes forever too even get it ... i hate such bugs10:14
ogra_aha !10:14
ogra_#27 hangs :)10:15
ogra_nope10:15
ogra_restarting didnt help10:15
ogra_lets see what last_kmsg says ... i added some extra panic line after run-init10:15
ogra_hmm, so it doesnt pass run-init http://paste.ubuntu.com/7198063/10:17
ogra_(else it would have printed something )10:17
* ogra_ adds more printing10:17
ogra_oh !10:18
ogra_dang10:18
ogra_heh, trying to work on the device while the loop script runs in another terminal isnt helpful :P10:20
=== MacSlow is now known as MacSlow|lunch
bzoltanMirv: like this http://pastebin.ubuntu.com/7198089/10:25
ogra_popey, hmm, your pastebin is differnt from the mail content10:25
popeyoh10:26
popeydid i grab the wrong one10:26
ogra_well, it moves forward there10:26
ogra_what is irritating is that systemd-udevd starts that early10:26
ogra_it should start after the contsainer10:26
popeysorry, http://paste.ubuntu.com/7197970/10:28
popeytoo many pastebins open10:28
Mirvbzoltan: hrm. something clearly wrong, but what...10:30
Mirvbzoltan: what I do before starting the tests is apt-get install ubuntu-ui-toolkit-autopilot address-book-app-autopilot messaging-app-autopilot dialer-app-autopilot  friends-app-autopilot webbrowser-app-autopilot mediaplayer-app-autopilot unity8-autopilot ubuntu-system-settings-online-accounts-autopilot ubuntu-system-settings-autopilot10:30
Mirvand I don't see such errors as that missing module10:30
=== psivaa changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: cihelp | CI Train support - US: robru, cyphermox, rsalveti - EU: sil2100, Mirv, didrocks | CITrain support no answer: use mup bot after 30 minutes, but choose right timezone | Known issues: -
mandelcan someone give me a jand with jenkins, I have tested the tests in a branch on my machine and the phone and the tests pass but it seems that PS jenkins is "slow" and the timeouts of the QTRY_COMPARE are not long enough.. :-/10:44
davmor2popey: didrocks: so I found an app in Ubuntu Netwalk that does the renaming issue I have installed it twice from available but on once does it show up in click list http://paste.ubuntu.com/7198136/ So from that I can only assume it is safer than we think my big concern would be updates not sure on a happy way to test that though10:46
davmor2s/but on/but only10:47
didrocksdavmor2: ok, if you can test updates, that would be excellent!10:48
popeyogra_: you don't need anything else from my phone do you?10:48
ogra_popey, nope10:48
popeythanks10:48
didrocksmandel: jenkins doesn't build itself your requests, it's the distro builders building it10:48
mandeldidrocks, hm.. bummer, tests used to pass with no problems, but now the simply fail and I'm quite sure is due to timing issues10:49
* mandel adds even more timeouts while thinks of a diff way to test qt signals10:49
davmor2didrocks: also I can't reproduce the mediascanner crashers so I'm going to assume for now that they are invalid and refile if it happens again so those can be knocked off the blocker list along with the unity8 crasher I guess10:50
didrocksmandel: basically we do build in the exact same environment/machines (really the same, as it's the final product we build) than distro.10:50
didrocksdavmor2: ok, please tell that on the ML10:50
* sil2100 reminds thostr_ about m&c o/10:51
mandeldidrocks, ok, I wonder what has changed to it not to run as fast as it used.. anyway, I'll think a work around10:51
davmor2didrocks: will do10:51
didrocksmandel: Qt, distro… a lot of things changes continously10:51
davmor2didrocks: I just realised you have a time machine that you have been hiding from the rest of the team no wonder you get so much work done ;)  02.04.13 indeed :D10:52
mandeldidrocks, well, yes but in a clean chroot in my system works so I don't see a "code" reason to see this issues, for example, tests do pass in amd64 but fail in arm on the buildservers while th tests do pass in the phone10:52
didrocksmandel: yeah, clearly timing-related10:55
didrocksdavmor2: yeah, mentionned it yesterday :p10:55
sergiusensmandel: didrocks; when mandel said jenkins, he might mean the ps-bot running for the MRs (not clear from the conversation)10:56
mandeldidrocks, sergiusens the issues are in the MR, yes10:56
ogra_sigh ... 84 reboots ... no hang10:56
didrockssergiusens: oh, good point!10:56
* ogra_ curses this day10:56
Mirv"almost done" #275 report that all AP:s seem to pass so far but I've a couple of weird things that I'll document in more detail10:57
didrocksmandel: ok, so not us, ping the CI team :)10:57
mandeldidrocks, ok10:57
didrocksMirv: ok, thanks10:57
mandeldidrocks, channel?10:57
didrocksmandel: that one, ping the vanguard10:57
sil2100Mirv: thanks ;)10:57
davmor2didrocks: So you mentioned you had a time machine I must of missed that :)10:59
sergiusensmandel: citeam is supposed to have a highlight setup for 'cihelp' -> read topic ;-)10:59
mandelcihelp, any idea on why the following tests are failing => https://jenkins.qa.ubuntu.com/job/ubuntu-download-manager-trusty-amd64-ci/377/console when they do pass on a machine and on the phone (I can confirm that the tests do have a timeout nad that it can be increased but it would make no sense to do so)11:01
vilamandel: /me looks11:01
vilamandel: the funny thing is that I was thinking about you not 5 minutes ago ;)11:02
mandelvila, the tests have been passing for a long time, the issue here is that they are slow in those machines...11:02
mandelvila, what did I do?11:02
vilamandel: hey, good stuff, don't panic ;)11:02
vilamandel: let's talk about that later11:03
mandelvila, ok :)11:03
mandelvila, one of the possible only reasons I can think of if that I moved to cmake to run the tests and that uses ctest..11:04
vilamandel: ctest never succeeded or ?11:04
davmor2didrocks: https://bugs.launchpad.net/ubuntu/+source/unity8/+bug/130187111:05
ubot5Ubuntu bug 1301871 in unity8 (Ubuntu) "Last track visible on the music carousel can not be used" [High,New]11:05
didrocksdavmor2: thanks11:06
vilamandel: and http://s-jenkins.ubuntu-ci:8080/job/ubuntu-download-manager-trusty-amd64-ci/378/consoleFull succeeded11:08
mandelvila, some tests do run, but the issue is that I'm testing that qt signals are emitted, to do so I use a qt macro that try to check that the signal was emitted with a timeout and allows the events to be processed during the execution http://qt-project.org/doc/qt-5.0/qttestlib/qtest.html#QTRY_COMPARE11:08
mandelvila, exactly, while the arm ones fail..11:08
mandelvila, so it looks like the arm machines are slow or (if they use qemu) something is going on there11:09
davmor2alecu: how is the fix for https://bugs.launchpad.net/ubuntu/+source/unity-scope-click/+bug/1279481 progressing it's possibly about to become a blocker if it screws up updates :(11:09
ubot5Ubuntu bug 1279481 in unity-scope-click (Ubuntu) "scope uses app title to uniquify list of installed applications" [Critical,In progress]11:09
mandelvila, in theory I can pass a longer timeout, so that the test runs for longer in the psbot but feels like a dirty workaround11:09
vilamandel: and 377 is on arm ? Doesn't look like it can you help me find the right pieces to compare ?11:10
vilamandel: yeah, we had that discussion already some months ago right ?11:10
mandelvila, correct11:11
mandelvila, let me point you to an error with the same tests in trunk that failed11:11
mandelvila, give me a few mins11:11
alecudavmor2: it will be fixed by this branch: https://code.launchpad.net/~alecu/unity-scope-click/use-tested-search/+merge/21225211:11
vilamandel: I need to grab some food, bb in a few minutes11:12
alecudavmor2: but, if we need it to land faster I can backport it on a smaller branch so it gets to trunk earlier11:12
davmor2didrocks: ^11:12
alecudavmor2: because I'm still a day away from finishing the big branch above, and I'm not making any progress by having all these meetings in the scopes sprint :-)11:12
didrocksdavmor2: that has already landed AFAIK, as told during the meeting (in proposed or unapproved)11:12
* ogra_ sighs ... i cant reproduce the hang at all anymore11:12
didrocksI thought?11:13
* didrocks looks11:13
alecudidrocks: you probably mean the unicode fix?11:13
didrocksalecu: yeah, I was told it was the fix for the issue davmor2 mentionned11:13
alecudidrocks: this is about some apps that are duplicated because they have a different title in the .desktop than in the click package index webservice.11:13
alecudavmor2: right?  ^11:14
didrocksalecu: ah ok11:14
alecudidrocks: davmor2: what landed was this unicode issue: https://bugs.launchpad.net/ubuntu/+source/unity-scope-click/+bug/129832711:14
ubot5Ubuntu bug 1298327 in unity-scope-click (Ubuntu) "Wrong character encoding in app previews" [Critical,Fix released]11:14
popeycan someone tell me how I can re-trigger a click build at http://s-jenkins:8080/view/click/job/music-app-click/ ?11:15
alecudavmor2: so, please let me know if I need to hurry the fix for "some apps are duplicated" or if it can wait a day more.11:15
* popey discovers a "Build now" button and presses it11:16
davmor2alecu: I still see that on image 27511:16
alecudavmor2: which one?11:16
davmor2the unicode issue11:16
alecudavmor2: it has probably landed less than an hour ago, I don't know if it's on the image yet11:16
davmor2Ah okay11:17
davmor2so next image should have that fixed then11:17
alecudavmor2: please let me know if it's not fixed on the next image11:17
alecudavmor2: ditto11:17
davmor2didrocks: so we need to decide how important the double apps is so alecu knows whether to land it as a smaller fix I think getting it in sooner rather than later we help us in a possible promotable image right?11:19
bzoltanMirv: ogra_: even after installing all thepackages -> http://pastebin.ubuntu.com/7198266/ That is on a stock image...11:23
mandelvila, no code was changed from trunk related to the tests in the following => https://jenkins.qa.ubuntu.com/job/ubuntu-download-manager-trusty-amd64-ci/272/console11:24
vilamandel: just back, what a stnc...11:24
mandelvila, as you can see, is in amd64 and the line that says task-0: FAIL!  : TestDownload::testTotalSizeError() Compared values are not the same is a perfect example11:25
ogra_bzoltan, a) this is not the latest image (the patch to disable adjust_soc messages from the mako kernel landed on monday), b) you seem to have ofono-phonesim-autostart installed, this isnt the case in automated testing (all packages used in previous tests get uninstalled there)11:29
davmor2alecu: so personally the quicker that fix lands the better but if it is landing tomorrow that might be soon enough but I'll leave that in the hands of didrocks  :)11:29
bzoltanogra_: this error was the same with #275 ...11:31
ogra_might be, just saying that the log is not from a recent image11:31
vilamandel: ack, revno 251 is a no-change rebuild, good, now let's try to see if we can find some correlation with genie load...11:34
vilamandel: no wait it's kinnara for 271 and 272 (at the start of the job: Building remotely on kinnara )11:34
didrocksalecu: yeah, the quicker fix if possible11:35
Mirvdidrocks: packaging app for mediascanner2 dropping its unused Unity scope https://ci-train.ubuntu.com/job/landing-012-2-publish/lastSuccessfulBuild/artifact/packaging_changes_mediascanner2_0.100+14.04.20140403-0ubuntu1.diff11:36
Mirvack, not app11:36
didrocksMirv: +111:37
Mirvthx11:37
alan_gsil2100:  is there a CI problem? Jenkins doesn't seem to have done anything for 10 hours or so: https://code.launchpad.net/mir/+activereviews11:37
vilamandel: except, I can't get usable load data  :-/11:39
mandelvila, I wonder, is there a way we can test that, for example, you can trigger a rebuild, correct?11:39
mandelvila, I don't have the rights to do it11:40
vilamandel: sure, can't you ??11:40
vila>-/11:40
didrocksalan_g: I told you to ping the banfuard, so "cihelp"11:40
didrocks(see /topic)11:40
alan_gcihelp:  is there a CI problem? Jenkins doesn't seem to have done anything for 10 hours or so: https://code.launchpad.net/mir/+activereviews11:40
didrocksthanks :)11:40
didrocksthe landing team isn't piloting those CI jobs11:41
didrocksalan_g: I guess it's due to the phone breakage in the datacenter though11:41
didrocksas per the phone ML11:41
cjwatsonproposed-migration et al stopping for battery replacement on their host11:41
vilamandel: job/ubuntu-download-manager-trusty-amd64-ci <- that one right ?11:41
vilaalan_g: do you know about the corresponding jenkins jobs to speed things up ?11:42
mandelvila, http://s-jenkins.ubuntu-ci:8080/job/ubuntu-download-manager-ci/383/rebuild11:42
mandelvila, right?11:42
vilamandel: ha ! better, thanks11:43
vilamandel: but... then you can trigger the rebuilds no ?11:44
vilamandel: or jenkins shouldn't give you a way to get that url (unless you crafted it manually " ;)11:44
mandelvila, the ps bot adds it as a comment, I don't have access11:45
mandelvila, => https://code.launchpad.net/~mandel/ubuntu-download-manager/upload-interface/+merge/21345811:45
vilamandel: ha great11:45
alan_gvila: you mean something like: http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-ci/11:46
vilaalan_g: perfect !11:46
vilaalan_g: but there a re a bunch of jobs running there11:47
vilaalan_g: Waiting for the completion of mir-mediumtests-trusty-touch in http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-ci/1235/console11:47
vilaalan_g: image #274 killed all our makos11:47
mandelvila, I need to grab some food, I'll be back asap11:48
alan_gvila: if didrocks is right about the phone hardware being AWOL then that's the explanation11:48
=== mandel is now known as mandel|lunch
vilaalan_g: fginther sent an email on the phone ML I think11:48
* alan_g knows nothing about that email or list11:48
vilagrr, missed that in the log, yeah, didrocks is definitely right11:48
alan_gvila, didrocks: thanks11:49
vilaalan_g: https://launchpad.net/~ubuntu-phone if you run tests on phones you should subscribe...11:49
alan_gvila: now I know about it...11:50
vilaalan_g: good ;) Sorry, it's a bit messy today ;)11:50
alecudidrocks, davmor2: ack11:53
vilamandel|lunch: so, http://s-jenkins.ubuntu-ci:8080/job/ubuntu-download-manager-trusty-armhf-ci/379/consoleFull and http://s-jenkins.ubuntu-ci:8080/job/ubuntu-download-manager-trusty-armhf-ci/380/consoleFull are currently running on the same arm node (cyclops-08)11:57
vilamandel|lunch: that means you can encounter different load levels ...11:57
vilamandel|lunch: which I suspect you never see when running on a phone11:58
vilamandel|lunch: From the jobs I've looked at so far, your tests can run on (genie, kinnara) amd64, and cyclops-{08,09,10,11,12,13} arm12:00
vilamandel|lunch: the amd64 nodes have 12 executors each, the arm ones 2 each12:00
=== cjohnston changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: cjohnston | CI Train support - US: robru, cyphermox, rsalveti - EU: sil2100, Mirv, didrocks | CITrain support no answer: use mup bot after 30 minutes, but choose right timezone | Known issues: makos are down
vilamandel|lunch: so you have no guarantee about the load12:00
vilamandel|lunch: executor is what jenkins call the bit that run a job, so 2 executors == up to 2 jobs running at the same time12:02
popeyogra_: #275 on mako, I just rebooted and get the google logo, but adb is available to me..12:03
vilapopey: the light !12:03
ogra_popey, great, so the container failed to start i assume12:03
popeyhow do i tell?12:03
ogra_you should see a lot of /system/bin/foobar in your processlist if it rins12:04
Saviqguys, can someone please try adding a U1 account on the phone12:04
ogra_*runs12:04
popeynope12:04
ogra_and thre should also be /init as well as /sbin/init in the processlist12:04
popeynope, only /sbin/init12:04
bzoltanogra_:  r275 -> http://pastebin.ubuntu.com/7198400/12:04
=== alan_g is now known as alan_g|lunch
ogra_popey, check /var/log/lxc/android.log and /var/log/upstart/lxc-android-config.log12:05
ogra_(and indeed syslog)12:05
ogra_for me it seems to fail a lot earlier here though (see the ML)12:05
popey/var/log/upstart/lxc-android-config.log: No such file or directory12:06
ogra_ok12:06
popeynothing lxc* in /var/log/upstart12:06
ogra_well, see the android.log12:06
popey      lxc-start 1396526456.109 ERROR    lxc_cgfs - cgroupfs failed to detect cgroup metadata12:07
popey      lxc-start 1396526456.109 ERROR    lxc_start - failed initializing cgroup support12:07
popey      lxc-start 1396526456.109 ERROR    lxc_start - failed to spawn 'android'12:07
ogra_hmm12:07
ogra_cgroup issues ... we had lots of changes in that area recently12:07
ogra_let me upgrade to 275 too12:08
ogra_(will take 30min or so)12:08
ogra_ah, no wait, i can do ota12:08
ogra_at least we know my emergency adb shell works :P12:11
ogra_popey, no issues with 275here for me after OTA12:14
popeyhmm, odd12:15
* ogra_ fires up the loop script 12:15
popeyi did ota from 274 to 27512:15
ogra_popey, file a bug, but thats not the same issue12:15
davmor2didrocks: so I keep getting this issue where the screen is greyed out and I think I know what the cause is.  if you swipe the screen left to right to unlock it the welcome screen is not 100% off the screen same thing happens with apps from time to time so I'm going to add some more details to popey 's bug report12:28
ogra_ok, i can reproduce on 27512:30
=== jhodapp|afk is now known as jhodapp
vilaogra_: tadda ! How many loop cycles ?12:34
ogra_2312:34
vila:-/ better than 62 ! :-}12:35
popeycjwatson: any idea what's going wrong here? http://91.189.93.70:8080/job/generic-mediumtests-trusty/1917/testReport/junit/calendar_app.tests.test_calendar/TestMainView/test_new_event_with_mouse_/12:37
popeyi watched the video and can't see why the test fails.12:37
cjwatsonpopey: hm, you sure you meant me rather than cjohnston?12:41
sil2100geh, need coffee ;/12:41
popeygoops12:44
popeycjwatson: correct ☻12:44
cjwatsonproposed-migration back up, btw12:44
cjohnstonpopey: beats me.. maybe when fginther gets in he will have an idea12:45
popeyok.12:46
cjohnstonpopey: is it only with that MP that its failing?12:46
popeyseems so.12:46
plarsogra_, vila: yeah it seems pretty random - how many loops to reproduce it12:49
ogra_right12:49
plarsogra_, vila: I had one run through several hundred cycles last night12:50
plarswith no problem12:50
ogra_i noticed the phone gets pretty warm when it hangs12:50
vilaoh my... if temperature comes into play....12:50
plarsI'm having better luck with increasing the sleep like popey suggested though12:50
sil2100Mirv: thanks for pressing m&c for webbrowser ;)12:50
ogra_so i'm currently trying to hack top output into the initrd right before rin-init12:50
sil2100Mirv: I'll assign a silo for the other webbrowser landing once this finishes12:50
ogra_*run-init12:50
sil2100thostr_: some more m&c for you!12:50
thostr_sil2100: just pressed the button12:51
Mirvsil2100: yep, that was the plan12:51
vilaplars: better luck with longer sleeps... could that be that the phone warms *more* ?12:52
plarsvila: wouldn't think so... if any thing it has more time to settle after rebooting12:52
vilaogra_, plars: and cut power because of that ?12:52
plarsvila: I could set it on fire12:53
vilaplars: that's the idea, it doesn't settle but loop on something12:53
vilaplars: hehe12:53
ogra_there is some process that hangs i think12:53
plarsvila: if that were the case, we would have seen a lot more systemsettle issues during testing I think12:53
plarsand we didn't see any12:53
ogra_if i could top output anything but "unknown terminal type." then i could do another debug run :P12:53
plarsvila: if we were getting some process chewing up the cpu that is12:54
vilaplars: but systemsettle happens far later no ?12:54
plarsvila: happens pretty soon after boot, and again after the tests run12:54
vilaplars: right, but aiui, an lxc container is not even started when the phone hangs12:55
ogra_it happens inside the initrd12:55
ogra_before it goes to the rootfs12:55
Mirvsil2100: well, thanks for monitoring the new silo assignment, I guess I should end my day again anyway...12:55
vilaogra_: so before systemsettle ?12:55
ogra_vila, before it even started booting12:55
Mirvsil2100: landing-008 would be ready to be published as I wrote there, but somehow I stuck to the idea that someone else could confirm also on the bug report that the bug was fixed (which is what I already tested myself)...12:56
vilaack, anyway, I'm not helping, muting myself12:56
sil2100Mirv: ok, I'll try taking a look at that landing then - thanks :)!12:56
popeyfginther: when you awaken, could you please take a look at why https://code.launchpad.net/~pkunal-parmar/ubuntu-calendar-app/CalManagement/+merge/213355 fails, thanks!12:56
ogra_vila, a boot goes (simplyfied): bootloader->kernel->initrd->rootfs->android-container->lightdm/unity12:57
Mirvsil2100: thanks!12:57
ogra_vila, when we hang we hang while exiting the initrd12:57
fgintherpopey, sure12:57
=== alan_g|lunch is now known as alan_g
Saviqdidrocks, btw, I can confirm I still get redirected to http:///job from time to time - usually after some time I've not been at the train (maybe reconnected network?)13:05
=== wgrant_ is now known as wgrant
seb128it would be nice if you guys could archive some of the landed line13:05
seb128it makes the list easier to scroll/read ;-)13:05
seb128Saviq, that happens to me as well13:06
didrocksseb128: Saviq: the only ones having a clue about that are on the webops channels I'm afraid13:06
didrocksmaybe open a RT?13:06
* Saviq does13:06
seb128I've another issue also when the first click on "build" doesn't work13:07
seb128it seems to just refresh the page13:08
seb128then clicking again works13:08
didrocksseb128: can be a related issue, like as well, I can sometimes get logged in13:08
didrocksand it refreshes to a page I'm not logged in13:08
fgintherpopey, I see this in the console log for the failing test: file:///usr/share/calendar-app/calendar.qml:89: TypeError: Cannot call method 'isSameDay' of undefined13:11
fgintherpopey, could that result in the app aborting?13:11
popeycould be.13:13
alecudobey: can you review this branch? it will hopefully fix davmor2's issue: https://code.launchpad.net/~alecu/unity-scope-click/unique-name/+merge/21402113:19
ogra_hmm, no runaway processes when it hangs13:28
kgunndidrocks: do you have time for a little chat with me ?13:33
didrockskgunn: I have for 1.5h of meetings, free afterwards13:33
didrockskgunn: that will be good for you?13:33
kgunndidrocks: :) yep...that'll work13:33
kgunnjust question about updating a project with new packages etc (glmark2)13:34
didrockskgunn: ok, let's sync on that13:35
sil2100seb128, boiko, Saviq: I assigned silos for you guys, but it seems the spreadsheet is taking its time with refreshing the status13:38
Saviqsil2100, thanks13:38
boikosil2100: thanks13:38
=== mandel|lunch is now known as mandel
seb128sil2100, ok, thanks13:39
sil2100hm, actually, there seems to be a bug on the spreadsheet13:42
sil2100didrocks: ok, I noticed that suddenly all rows from line 55 to the end of the spreadsheet have a different formula in columns C and L13:43
sil2100didrocks: how are those added to those columns? Are they manually set in the spreadsheet or does the scripts fill those formulas in there?13:44
* sil2100 looks into the code13:44
sil2100Ah13:48
sil2100didrocks: ok, so it seems that someone (?) modified the formula of the first landing request ;/13:48
sil2100didrocks: and the script uses the first landing row as the 'base' for the formula and all the new landing rows are broken - fixing that13:49
sil2100didrocks: ok, fixed the first landing row and all the 'added' landing rows - now everything should have the 'correct' formula13:51
sil2100Saviq, boiko, seb128: the spreadsheet fixed, you can build and work normally13:52
seb128sil2100, \o/13:52
Saviqsil2100, \o\ |O| /o/13:53
davmor2popey: what was the bug number for the grid layout issue I can't find it13:56
popeydavmor2: bug 130030213:57
ubot5bug 1300302 in unity8 (Ubuntu) "Dead area at bottom when scrolling in scopes" [Undecided,Incomplete] https://launchpad.net/bugs/130030213:57
davmor2popey: thanks dude13:57
sil2100seb128: I'm doing a tender loving m&c of silo 1814:00
seb128sil2100, thanks ;-) it's getting difficult to browse that list (did you see my comment about that an hour ago?)14:01
didrocksseb128: argh, really? :/14:03
didrocksoopsss14:03
didrocks sil2100 ^14:03
didrocks(sorry, in meetings)14:03
didrockssil2100: the script copy the formulas there14:03
didrockssil2100: copying from line 414:03
sil2100didrocks: I know, as mentioned, I checked, noticed and fixed ;)14:08
sil2100didrocks: no worries, your code is made easy-readable - sorry to have bothered, just wanted to keep you up-to-date14:09
t1mphello14:10
t1mpcan someone trigger a CI run for this MR? https://code.launchpad.net/~andrew-hayzen/ubuntu-ui-toolkit/fix-swipe-delete-002/+merge/20217114:10
popeyt1mp: just leave an approval and it should re-trigger, right fginther ?14:11
t1mpthat was not the case before as far as I know. Is that new?14:14
fgintherpopey, t1mp, for CI runs, a push of a new version will automatically trigger a rebuild14:14
alecudavmor2: can you tell me the name of any of the apps that show up duplicated?14:14
t1mpfginther: that is only the case if it is pushed by a canonical person, that is not the case here14:15
fginthert1mp, ahh, I wasn't aware that was the case...14:15
davmor2alecu: ubuntu netwalk14:15
t1mpfginther: but we are happy to accept patches from outside (as long as we review it)14:15
alecugreat14:15
t1mpfginther: if I leave an approval (not top-approve yet), it will trigger CI?14:15
fginthert1mp, one moment14:16
Saviqdidrocks, huh... I just saw http://people.canonical.com/~didrocks/citrain/silos/landing-004/14:17
Saviqdidrocks, it contains the hud, but the hud isn't there in the silo14:17
sil2100Saviq: let me see14:20
Saviqsil2100, btw, that silo can be published14:20
sil2100Saviq: I would have to check the source, but maybe it wasn't yet removing the old sources14:21
sil2100Or maybe the rsync works somehow differently14:21
sil2100uh oh, silo 004?!14:21
sil2100Awesome14:21
sil2100Saviq: let me publish then14:21
alecudavmor2, didrocks: I've got the fix for the duplicated icons bug, I've tested the deb from jenkins and it seems to work ok: https://jenkins.qa.ubuntu.com/job/unity-scope-click-trusty-armhf-ci/302/?14:25
alecudavmor2: if you can test it too, I'll ask for a silo for it in the meanwhile14:26
alecudidrocks: can I ask you to land it? all the landers in this sprint are in a meeting right now.14:27
alecuthis is the branch: https://code.launchpad.net/~alecu/unity-scope-click/use-tested-search/+merge/21225214:28
didrocksalecu: sure (or maybe sil2100 can handle it as I'm in meetings)14:29
didrocksthanks a lot alecu :)14:29
didrockssil2100: weird, there was a first publication for the hud, I'm not sure where the second .project comes from14:29
fginthert1mp, sorry for the confusion, adding an approved comment does nothing. For an MP from a non canonical member to be automatically tested, it must be top-approved. So if you just need a ci run, that needs to be triggered manually, which I can do14:30
davmor2alecu: will do14:31
t1mpfginther: please do14:31
fginthert1mp, already done :-)14:31
t1mpfginther: thanks14:32
t1mpfginther: is there a configuration that gives me a way to trigger it? then I don't need to bother you in future14:32
sil2100didrocks: hm hmmm... I'll look further14:39
sil2100alecu: let me backlog14:39
bzoltanogra_: sil2100: didrocks: We have a problem with the r275 ... The QtC can not connect to the device with that image14:40
ogra_what is the QtC ?14:41
sil2100ogra_: QtCreator14:41
bzoltansil2100:  yes14:41
sil2100alecu: can you set the branch to 'Needs review'?14:41
sil2100alecu, didrocks: I'll add a landing for that one14:42
ogra_bzoltan, via adb you mean ?14:42
zbenjaminbzoltan: i'm here14:43
didrockskgunn: available now14:44
sil2100didrocks: packaging ACK! https://ci-train.ubuntu.com/job/landing-004-2-publish/lastSuccessfulBuild/artifact/packaging_changes_unity8_7.85+14.04.20140403.1-0ubuntu1.diff <- from the packaging side, simply one additional file to be installed14:44
didrockssecond meeting was shorter than expected :)14:45
ogra_plars, with the rebooted makos, did you start another test run for 275 ?14:45
didrockssil2100: ack for the "nonmirplugins" :p14:45
sil2100;p14:45
sil2100didrocks: thanks!14:45
didrocksdavmor2: 5. and 6. are new?14:46
didrocksdavmor2: you didn't list the adb blocking one14:47
didrocksdavmor2: as the device doesn't reboot, this seems a promotion blocker to me14:47
kgunndidrocks: ok... :)14:47
didrockskgunn: https://plus.google.com/hangouts/_/7acpjkahe4pge84i6aa08l6d9c14:48
davmor2didrocks: 5 and 6 were ones that were really hard to reproduce popey and I have hit them on and off .  The one is the one I showed in the call this morning the other is one that is annoying when it hits but is temperamental at best and were just missed out I think14:48
davmor2didrocks: I've not been hit by the adb bug but I would agree that is a blocker it there a bug for it?14:49
sil2100alecu: I changed it to 'Needs review' if anything14:50
sil2100alecu: and assigning a silo14:50
alecusil2100: I'm sorry, I pasted a wrong url14:50
alecusil2100: that's not the branch14:50
alecusil2100: it's this one:14:50
alecuhttps://code.launchpad.net/~alecu/unity-scope-click/unique-name/+merge/21402114:50
sil2100...crap...14:51
sil2100:|14:51
sil2100Grrrr14:51
alecuI'm so so sorry :-(14:51
didrocksogra_: you do have a bug for the "I lost my phone" bug?14:53
didrocksor even "OMG I lost my phone"? :p14:53
davmor2didrocks: I don't know about that but there is a Oh crap how do I find my phone14:54
bzoltanogra_: sil2100: no, from the Ubuntu SDK (QtCreator) just plug in the device and it should be able to connect to it14:54
didrocksdavmor2: heh ;)14:54
popeysil2100: I asked kalikiana in #ubuntu-app-devel about https://bugs.launchpad.net/ubuntu-calculator-app/+bug/1289695 - is there an upcoming slot he can have to get that in soon?14:54
bzoltanogra_: sil2100: apparently it needs a  dpkg-reconfigure openssh-server14:54
ubot5Ubuntu bug 1289695 in Ubuntu UI Toolkit "Title for all Core apps should be improved/localized for Desktop" [High,In progress]14:54
davmor2didrocks: ring it seems to be the answer not very productive :D14:54
sil2100popey: let me take a look14:55
cjwatsonbzoltan: is it relying on being able to ssh as root with a password?14:55
sil2100bzoltan: oh?14:55
cjwatsonthat would be odd, that's the only thing that changed14:55
cjwatsonwell, not only, but the most likely thing to matter14:55
bzoltancjwatson:  no...we do as phablet and we create key14:55
cjwatsonbzoltan: so I don't get how dpkg-reconfigure would matter, given that permitrootlogin without-password is the only thing handled by debconf14:55
cjwatsonis something forgetting to create host keys on deployment?14:56
bzoltancjwatson:  All I know that with the recent image the device connectivity stopped working14:56
sil2100popey: we might have an additional free silo in a moment - the problem is that there is one landing for UITK prepared anyway14:56
cjwatsonI was asking the channel as a whole14:56
sil2100popey: so it's basically locking UITK for now until that's released...14:56
bzoltancjwatson: and dpkg-reconfigure fixes it14:57
cjwatsondpkg-reconfigure would no doubt have the side-effect of creating host keys, but that hasn't changed lately - if they were in place before, they should still be14:58
bzoltancjwatson: yes > http://pastebin.ubuntu.com/7198938/14:58
cjwatsonright, so I wonder what was creating those before?14:58
bzoltancjwatson: ogra_: I did install the image with --wipe14:58
ogra_bzoltan, why do you fiddle with ssh-server ?14:59
ogra_cjwatson, lxc-android-boot.conf was creating them on first boot14:59
bzoltanogra_: I do not :)14:59
ogra_you just said you did dpkg-reconfigure15:00
cjwatsonogra_: start on staring ssh15:00
bzoltanogra_: that was what fixed it15:00
cjwatsonspot the typo15:00
cjwatsonogra_: while you're there, you should sync up the set of host keys generated with openssh-server.postinst15:00
ogra_cjwatson, yeah, my unconcious developer brain probably thought it shouild stare back :P15:00
didrocksstaring ssh :)15:00
* didrocks loves that15:00
cjwatsonto add ecdsa/ed2551915:01
ogra_cjwatson, will do15:01
ogra_bzoltan, does any of you code call "start ssh" ? it would be nice if QtC started using the property instead15:01
davmor2alecu: so that deb doesn't seem to want to install http://paste.ubuntu.com/7199048/15:02
ogra_(until we have a proper developer-mode that sets writalke and the ssh property)15:02
ogra_*writable15:02
bzoltanogra_: we start the ssh if it is not started15:03
bzoltanogra_: what do you suggest to use instead?15:03
ogra_bzoltan, one sec ... i have to look up the property name and am in a meeting15:04
cjwatsonso, yeah, regression in lxc-android-config 0.156.  that's a relief, I thought it might somehow have been a regression in openssh 1:6.6p1-115:04
bzoltanogra_: thanks15:05
ogra_cjwatson, if it comes to touch ssh issues, ping me first :)15:05
t1mpbzoltan: I'm running the tests for silo 007 but something is messed up15:07
t1mpbzoltan: even though I ran adb shell powerd-cli display on &, the display turns off some times15:07
bzoltant1mp: not cool15:08
=== doanac changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: doanac | CI Train support - US: robru, cyphermox, rsalveti - EU: sil2100, Mirv, didrocks | CITrain support no answer: use mup bot after 30 minutes, but choose right timezone | Known issues: makos are down
t1mpbzoltan: results so far http://paste.ubuntu.com/7199076/15:09
t1mpbzoltan: gallery-app is supposed to fail because its test code is broken. I don't know why it is OK on the dashboard, maybe different version of python? But that's weird since we have the same image..15:09
t1mpbzoltan: other failures (system settings) I have no idea what that is ProcessSearchError: Search criteria (pid = 14512, dbus bus = 'session', object path = '/com/canonical/Autopilot/Introspection', process object = '<subprocess.Popen object at 0xb5fe6430>') returned no results15:11
t1mpI'll let the tests continue to see what the other apps do15:11
sil2100bregma: reminding about m&c! :)15:29
bregmasil2100, thanks15:29
didrocksogra_: did you see my question about having a bug number for the initrd issue?15:34
ogra_bzoltan, you want to use "setprop persist.service.ssh true" ... that sets a persistent property so ssh keeps running across reboots15:40
bzoltanogra_: thanks15:40
ogra_didrocks, yes, sorry, didnt egt to file one yet ...15:41
ogra_*get15:41
* ogra_ is actually having some breakfast atm :P15:41
didrocksogra_: enjoy! :)15:43
robruSaviq, I resolved the conflict in silo 13 and hit rebuild for you, should be ready to test soonish15:52
Saviqrobru, oh thanks15:52
robruhrm, now there's a new merge conflict...15:52
Saviqrobru, we'll take care of it15:53
Saviqpete-woods, https://ci-train.ubuntu.com/job/landing-013-1-build/5/console15:53
popeyMirv: when you have a moment could you please push music 409 from http://s-jenkins:8080/view/click/job/music-app-click/ - http://s-jenkins:8080/view/click/job/music-app-click/lastSuccessfulBuild/artifact/out/com.ubuntu.music_1.3.409_armhf.click to the store?15:54
didrocksSaviq: davmor2 pinged you about:15:54
didrockshttps://bugs.launchpad.net/ubuntu/+source/unity8/+bug/130032615:54
ubot5Ubuntu bug 1300326 in unity8 (Ubuntu) "Shell randomly freezes with grey tint" [High,Confirmed]15:54
didrockshttps://bugs.launchpad.net/ubuntu/+source/unity8/+bug/130030215:54
ubot5Ubuntu bug 1300302 in unity8 (Ubuntu) "Dead area at bottom when scrolling in scopes" [High,Incomplete]15:54
didrocksright?15:54
didrocks(as he set them as blockers)15:55
sil2100alecu: the silo has packages built - silo 17 ;)15:55
davmor2didrocks: no that was my next job15:55
davmor2didrocks: I just got a list together initially to ensure we weren't missing any15:56
didrocksdavmor2: hum, you should ping them asap so that they don't discover new issues15:56
didrocksdavmor2: seems we lost a day on the second for instance15:56
davmor2didrocks: indeed but Saviq picked up on both but it is only today going back over all the recent bugs that I realised that they had fallen through  the cracks15:57
alecusil2100: thanks, testing15:58
sil2100\o/15:58
didrocksok15:58
didrockscyphermox: coming?16:03
cyphermoxyes16:03
Saviqdidrocks, davmor2, Albert and Michael will look into those16:10
davmor2Saviq: awesome thanks16:11
alecudavmor2, sil2100: I'm testing silo 17, and the issue seems to be fixed.16:12
davmor2alecu: I might of grabbed it before it finished16:12
alecudavmor2: I was unable to reproduce the broken .deb from jenkins, it worked ok for me.16:12
sil2100alecu: awesome - did you try the whole unity-scope-click test-plan?16:12
alecusil2100: I'm on it16:13
sil2100alecu: thank you :)16:14
alecuit will take a little while, since I'm on 274 and need to reinstall16:14
=== jhodapp is now known as jhodapp|lunch
didrocksSaviq: thanks!16:27
sil2100robru: published 00116:28
robrusil2100, thanks16:28
alecusil2100, davmor2: I've completed the unity-scope-click test plan. It works as expected (minus the bugs noted in the test plan)16:36
alecusil2100, davmor2: I think it's ready to go.16:36
alecuthanks for all your help!16:36
davmor2alecu: \o/16:36
sil2100alecu: \o/16:39
sil2100alecu: thanks!16:39
ckingplars, i believe I have a fix for the suspend-blocker issue - i've updated asana with some notes16:39
plarsogra_: you mean in the regular ci tests? or at home to try to see if this bug is reproducible in 275 (sorry for the delay, had to take my son to the doctor)16:40
plarscking: cool, I'll take a look. thanks!16:40
ogra_plars, yeah, i heard, all ok ?16:40
ogra_plars, i meant in the lab16:41
plarsogra_: yeah, he was having really bad swelling and knee pain, and had already confounded one specialist as to the cause16:41
plarsogra_: I heard the device were back up,  but that was after I had already left. Let me take a look now16:41
ogra_only try on one so we dont kill them all :)16:42
plarsogra_: yeah, no kidding!16:43
plarsogra_: looks like psivaa started one but it's stuck in the delay for image. I just restarted it without that so we can get it rolling quicker16:44
ogra_yeah16:44
plarsogra_: it's running at http://q-jenkins.ubuntu-ci:8080/job/trusty-touch-mako-smoke-daily/215/console16:44
ogra_now if my laptop had a working vpn :P16:45
sil2100robru: published 17 now, leaving the landings in your hands now ;)16:47
robrusil2100, great thanks16:47
plarsogra_: any progress on this problem with boot getting stuck? Any ideas yet?16:48
ogra_plars, not really, we decided to look at different images now ... it is clear that it hangs at or right after run-init16:48
robruplars, we (popey, davmor2, cyphermox and I) are running your reboot script with different image numbers. so far image 271 is up to 23rd reboot without issue16:49
ogra_i checked the processlists and top output right before run-init when the issue happens, nothing special visible there16:49
plarsrobru: ogra_: I had started something similar while I was away, I'm running it on image 27316:49
ogra_my best guess would be a kernel issue16:49
plarsI'm on loop 88 (with 2 min sleep)16:49
ogra_also popey claims it crashes earlier if you raise the sleep16:50
ogra_so we agreed to use 30sec at least with your script16:50
robruyeah, we raised it from 10s to 30s16:50
plarswith the 10 sec sleep, I got over 500 loops last night on 27416:50
plarsso it's really hit or miss16:50
robrudamn16:50
ogra_plars,how many reboots doe a device usually have during a day16:50
ogra_*does16:50
ogra_(hust a rough guess)16:51
ogra_*just16:51
plarsogra_: roughly 30 reboots per image (setup + tests)16:51
popeymine has run 40 times now16:51
ogra_30 isnt that much16:52
plarsno, it isn't16:52
ogra_i had it often enough only die at 80 or above today16:52
didrocksogra_: yeah, but remember that it runs tests beforehand16:52
didrocksogra_: so, can become hot because of that16:52
didrocksif your theory is right16:52
ogra_well, my theory is born out of desparation :P16:53
didrocksheh, I had that feeling!16:53
ogra_dont trust it :)16:53
didrocksogra_: do you think I did? :p16:53
ogra_ha16:53
* didrocks thinks it's time to run away :p16:53
popeydidrocks: see you next week16:54
popey16:54
didrockspopey: oh right, enjoy your Friday!16:54
popeythanks16:54
ogra_vacation ?16:54
didrockspopey: keep us posted on the ML on oyur latest findings16:54
didrocksogra_: you should really listen to the meeting, you can't blame popey for the accent! :)16:54
ogra_lol16:55
ogra_well, i was watching a phone reboot :P16:55
didrockspopey: see, ogra_ prefers watching a phone than you!16:55
didrocksnot sure how I would take it…16:55
didrockssee you guys ;)16:55
ogra_bye16:55
popeyogra_: yeah, day off while kids are off school. going to a science museum ☻16:57
ogra_nice !16:57
ogra_enjoy16:57
=== alan_g is now known as alan_g|EOD
bzoltanogra_:  so you suggest to apply this on the script we use -> http://pastebin.ubuntu.com/7199576/17:04
robruseb128, got you silos 4 and 517:11
seb128robru, thanks!17:11
robruseb128, you're welcome!17:11
cjwatsonbzoltan: (BTW it's a good idea to use ">/dev/null 2>&1" to avoid gratuitously bash-specific shell code)17:12
cjwatson(rather than "&> /dev/null")17:13
=== plars changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: plars | CI Train support - US: robru, cyphermox, rsalveti - EU: sil2100, Mirv, didrocks | CITrain support no answer: use mup bot after 30 minutes, but choose right timezone | Known issues: makos are down
bzoltancjwatson: OK, thanks... but generally that replacement should be OK, right?17:18
ogra_bzoltan, right, you might want to run the ssh start once if you dont reboot the device, the property will only kick in after a reboot17:22
bzoltanogra_: so you suggest to keep the line we use now? Because I do not really want to reboot the device17:23
ogra_bzoltan, http://paste.ubuntu.com/7199689/ ... that way you will only set it once17:26
popeyogra_: 100 reboots17:26
bzoltanogra_:  ohh... that looks smart. Thanks a lot17:26
popey#26917:26
ogra_popey, no hangs ?17:26
popeynope17:26
ogra_i'm at 64 on 27517:26
ogra_still looks fine17:27
psivaaogra_: https://pastebin.canonical.com/107744/ is a last_kmsg from a device that had this issue in the lab. rfowler grabbed it17:29
ogra_psivaa, yeah, shows the same as all the others17:29
psivaaogra_: ack17:29
ogra_hangs at run-init or right afterwards when starting /sbin(init17:29
robrupopey, ogra_ so far I'm at 67 reboots on 271.17:30
plarsresults are starting to flow for 275 now on mako - so far the device is still up17:30
plarsrobru: mine is still going on 271 at 105 reboots, do you know if anyone has managed to reproduce this on anything less than 274 yet?17:30
robruplars, I'm not aware of anything less than 274 having the issue, no17:31
plarsrobru: I'm going to let mine run a bit longer, but just checking17:31
robruplars, ok17:32
robrupopey, ogra_ : who was doing what image again? popey had 269, i had 271... i forget who else is doing this17:32
ogra_robru, i'm doing 275 now and will stop at ~150 loops ... thn i'll reinstall and try 275 in readonly mode17:33
robruohhhhh, readonly mode. that's probably a good idea to try.17:34
robruogra_, how do you get back to readonly mode? bootstrap option?17:34
ogra_yeah, i usually have --bootstrap and --wipe set when doing test installs17:35
ogra_(might be redundant :) )17:35
robruhmmmm ok i'll try that if I get to 200 or so17:35
davmor2robru, popey, ogra_: have you guys hit the reboot issue on your images?17:36
robrudavmor2, not on 27117:37
ogra_davmor2, not yet17:37
davmor2mine just seems to be chugging along no issues yet on 27317:38
ogra_i have a feeling that making the sleep longer actually makes it fail less than more17:38
popeydavmor2: not on #269 - 130 reboots17:45
popeyI can't say I'm surprised, given we only saw this in #274 really, it came on overnight17:45
davmor2tea got in the way for me so I'm only on 3017:45
ogra_you give your phone tea ?17:46
davmor2popey: I agree but at least taking 3 older images if they are all perfect then we know for sure that it is only 274 so it is a change from 273 to 274 that is causing it17:46
popeysure17:47
ogra_well, i saw it for sure on 27517:47
davmor2ogra_: no I had tea :P17:47
ogra_but not with the 30sec sleep17:47
popeywell, i got it with 2 min sleep on 27417:47
popeyafter only 12 goes17:48
ogra_yeah, you said so17:48
ogra_i really wonder if it is because the image is writable and we dont do proper reboots17:48
ogra_(adb reboot is like hitting the reset button on a PC)17:48
popeymine isn't writable.17:48
ogra_your 27417:48
ogra_?17:48
popeyi haven't made it un-writable since last night17:49
popeyand there's no /userdata/.writable_image on mine17:49
ogra_oh, interesting17:49
ogra_that saves me from doing some test17:49
ogra_(but also trashes another potential theory)17:49
popey163 and still going18:04
davmor243 and still going18:05
ogra_10218:05
ogra_and many many nautilus windows :P18:06
ogra_oh, 102 actually got me the hang ...18:07
davmor2ogra_: right click the icon in the launcher and quit18:07
ogra_so 275 still has it18:08
ogra_davmor2, yep, i know :)18:08
=== jhodapp|lunch is now known as jhodapp
robruyeah I'm at 117 on 271, no issue18:17
davmor260 now on 27318:22
popey207 on #26918:30
robruimage 271 got up to 142, I'm gonna reflash in readonly mode and try again18:41
popeyogra_: how far do you want me to go with this? I'm at 230 reboots of #269...18:44
davmor2popey: what timeout did you have18:45
popey30s18:45
popeyi can try again with longer or shorter timeout18:45
popeywhile I make dinner18:45
davmor2so why am I still on 80 I was on 30 and you were on 13018:46
popeyhttp://paste.ubuntu.com/7200015/18:47
popeythats my script18:47
davmor2popey: ah I'm using the one that plars posted18:47
davmor2only 30 secs rather than 1018:48
popeyThu Apr 3 17:27:59 BST 201418:48
popeythats when i started18:48
popeyso its been 2.5 hours18:49
plarsI upped mine to 120 after popey said that was working better for him18:49
plarsdavmor2: popey: which images are y'all testing?18:49
plarsI've made it through 137 loops now with 27318:49
popey#269 here18:50
davmor2plars: we are all doing different ones so I'm testing 273, robru 271 and popey 29618:50
davmor2269 even18:50
davmor28518:51
plarsand none of you have reproduced so far?18:51
davmor2plars: nope18:51
davmor2ogra_: has on 27518:51
robrulol, I just did a --bootstrap --wipe flash of 271 and unity won't start18:51
davmor2plars: and popey and ogra_ did on 27418:52
plarsI suspect my mako in the lab running on 275 just died18:52
davmor2robru: D'oh forgot about that, that is when didrocks broke the image18:53
plarsI'm watching for a bit longer but it's been stuck on the dialer-app test reboot for a while now18:53
robrudavmor2, but earlier when I flashed 271 it was loading fine. it was specifically the --bootstrap --wipe that broke it18:53
robrudavmor2, at least, adb is able to reboot the device for the purposes of this test, but yeah, i'm not getting unity8 starting at all18:53
davmor2robru: if you install without wiping it is fine as it would of had the missing deps the wipe kills it :)18:54
robruahhh ok18:54
ogra_plars, before i go to bed i'll fire up a test with all kernel debug options enabled etc ... lets see, perhaps we have an oops we cant see18:59
davmor29519:02
popeyright, bored of this game now. going to re-run with 2 min delay again on #269, just in case 30s isnt long enough19:04
popey265 successive reboots and no lockup, seems enough to me ☻19:04
* ogra_ adds "debug loglevel=6 log_buf_len=1M" to the kernel commandline ... lets see if that spits out more19:07
ogra_LOOP 119:07
ogra_worked :)19:07
vilaogra_: well done ! hold it tight !19:09
ogra_hmm, i was expecting more noise from loglevel=619:10
davmor2ogra_: publish it19:10
davmor210319:10
ogra_dmesg looks pretty normal with loglevel=6 http://paste.ubuntu.com/7200123/19:12
* ogra_ changes to 719:12
ogra_loglevel=7 --verbose debug ... that looks mildly informative19:22
ogra_(and log_buf_len=1M else dmesg eats itself)19:23
davmor2ogra_: after 120 loops I got this instead of the nuatilus window Unable to open a folder for Nexus 4  Cache invalid, retry (internally handled)19:30
ogra_yeah19:30
ogra_known19:30
davmor2ogra_: that's fine then I'll ignore it :)19:30
ogra_happens on the manta more often than on the mako19:30
davmor212219:31
ogra_well, not sure, i think cyphermox has a bug open for it already19:31
ogra_if not, opening one might make sens19:31
ogra_e19:31
cyphermoxeh?19:31
ogra_cyphermox, an mtp  popup message that rarely shows up on the desktop19:32
ogra_"Cache invalid, retry (internally handled)"19:32
ogra_i see that too every 100th boot or so19:33
ogra_(or 200 ... or whatever)19:33
cyphermoxah19:33
cyphermoxyeah that's an issue in gvfs actually, I think19:34
ogra_if you click in nautilus all is fine ... its just that starting of the nautilus window thats missing when it happens19:34
cyphermoxI suspect at some point the caching fails, it's a little weird19:35
ogra_yeah, its a low prio bug for sure19:36
ogra_it works  99% of the time19:36
ogra_cosmetic noise :)19:37
* davmor2 calls it a night and leaves the looper looping19:45
ogra_heh19:45
seb128can I get a silo for l62?19:52
sergiusensdoanac: ping20:07
doanacsergiusens: what's up?20:08
sergiusensdoanac: I have silo 10 with phablet-tools changes, was wondering if we could trigger a full run again20:09
doanacsergiusens: i'll give it a shot. we tried yesterday but all the makos keep getting stuck offline20:10
sergiusensdoanac: right... forgot about that20:10
doanacplars: mind if i run: http://q-jenkins:8080/job/andy-smoke-daily-test/  for sergiusens ?20:10
doanacor do we need the makos for other stuff right now?20:11
sergiusensdoanac: I could take a leap of faith; the changes look sane; but there's an execution change in phablet-test-run for shelling in and I don't want to find out later that this broke something20:11
plarsdoanac: you are welcome to try, but...20:11
doanacplars: if nothing is passing its probably not worth trying20:12
doanacis it worth it?20:12
plarsdoanac: when you can get it to go, it's working: http://ci.ubuntu.com/smokeng/trusty/touch/mako/275:20140403:20140331/7545/20:12
plarsdoanac: but at some point, it's likely to fail20:13
doanacsergiusens: so here's the limited results from yesterday before we killed the job: http://q-jenkins:8080/job/andy-smoke-daily-test/5/testReport/20:13
doanaclooks like it could test friends-app, so maybe we can call it good?20:14
sergiusensdoanac: yeah, I guess so; I ran a couple of mine; just wanted to make sure the infra didn't break20:17
sergiusensseems it didn't20:17
sergiusensthanks, should be fine20:17
sergiusensthanks20:17
sergiusensrobru: hey, can we publish silo 10?20:18
robrusergiusens, done!20:18
=== jhodapp is now known as jhodapp|bbl
sergiusensty20:19
robruyw20:19
robruogra_, popey, davmor2, plars: 250+ loops on image 271, no sign of the problem. want me to try 272 now? or is there a plan?20:24
ogra_robru, i think i found something20:25
ogra_for whatever reason udev starts while the container is coming up20:25
ogra_(it should only start after the container)20:27
ogra_hmm, and i also see why it starts to early ... i just dont get why that hasnt bitten us before20:30
popeyogra_: shall i stop rebooting my device?20:37
popey44 runs at 2 min interval on #269, its fine20:38
pmcgowanpopey, is there a bug report for this one?20:38
ogra_popey, well, you could try a fix on an image that we know has the issue20:38
ogra_pmcgowan, nope20:38
ogra_popey, in 274 or 275 edit /etc/init/udev.override ... change lxc-android-boot to lxc-android-config and see if that fixes it20:39
ogra_robru, ^^^ in case you want to try20:39
ogra_seems i cant trigger it in 275 if all debugging is enabled :(20:40
ogra_95 loops yet20:40
ogra_pmcgowan, not yet20:40
robruogra_, ok, i'll try that20:40
ogra_your blaming of systemd wasnt that wrong20:41
ogra_i guess20:41
ogra_just that the bug isnt in systemd/udev itself20:41
* ogra_ hopes that doesnt make the boot to slow :(20:42
robruogra_, lol, that was just a joke (actually i mailed that privately, not on list) because it seems popular to hate on systemd ;-)20:42
ogra_lol20:42
ogra_i didnt notice it was private, heh, sorry20:42
ogra_i see it now :P20:43
popeyogra_: happy to try that :D20:44
ogra_nice one to leave running over night i guess20:45
pmcgowanpopey, did you file a bug on UI slowness? or was that a rumor20:45
popeygah, never rotate phone in system-settings -> updates20:45
popeypmcgowan: anecdotal20:45
popeyneed to test more20:45
pmcgowanogra_, I can file a bug for this if you tell me where and what to say ;)20:48
ogra_pmcgowan, booting mako hangs randomly right after run-init20:52
ogra_pmcgowan, Every 10-100 boots (in a really random manner) the boot of mako devices gets completely stuck. This started with image 274.20:53
pmcgowanogra_, which package/project?20:54
ogra_thats the issue :P20:54
pmcgowanhah20:54
ogra_file it against lxc-android-config for now, if my theory is right the fix will live in there20:54
pmcgowanubuntu20:54
pmcgowanok20:54
robruogra_, hah, I just flashed 275 for the first time and it got stuck at the Google logo on the very first boot20:55
ogra_ouch20:55
robruogra_, complete with "device not found". are there any logs I can provide? what/how?20:55
ogra_robru, reboot into recovery (by holding vol-down), once there: adb shell cat /proc/last_kmsg ...20:56
ogra_robru, if the last thing (before the power button shuts down the CPUs) you see in there is the swapon output, it is the same issue20:57
robruogra_, crap, rebooted into the phone first, I guess I lost that log.20:58
ogra_yeah20:58
pmcgowanogra_, https://bugs.launchpad.net/ubuntu/+source/lxc-android-config/+bug/130217420:58
ubot5Ubuntu bug 1302174 in lxc-android-config (Ubuntu) "Booting mako hangs randomly right after run-init" [Critical,Confirmed]20:59
ogra_thanks !20:59
pmcgowanthat will make all the difference fixing it ;)20:59
davmor2robru: no I think the plan is that with me covering 273 you 271 and popey 269 we would lower down the releases that the issue was introduced looks like it was 274  I'm on 232 no issues21:00
davmor2ogra_: think that means 274 is where the issue was introduced at least right?21:01
ogra_yes21:01
ogra_davmor2, but i could reproduce it reliably on 275 too21:01
davmor2ogra_: oh yeah I'm saying we were lowering down where the issues was introduced so we know it is 274 now but the effect of that will be with us I guess until someone fixes the issue right?21:03
robruogra_, I can't ssh into image 275... just says 'connection closed'. I tried both ssh over wifi, and also over a port-forward bridge through adb/usb21:06
ogra_robru, yeah, there is a typo ... dpkg-reconfigure openssh-server21:07
ogra_i'll fix that with the next lxc-android-config upload21:07
ogra_(key generation was moved to its own upstart job, which has a typo in the start condition)21:08
robruok thanks21:08
davmor2right /me calls it a night catch you in the morning21:11
robruboiko, i'm just about to take lunch, do you need a hand with silo 9 before I go?(just noticed the build error)21:20
boikorobru: well, renato updated the MR but it seems jenkins didn't notice that when I tried to rebuild21:21
boikorobru: I just used force rebuild, let's see if this works21:21
robruboiko, yeah, i was just gonna say, force rebuild should do it21:21
boikorobru: great, thanks21:21
robruboiko, you're welcome21:21
robruok, bbl21:21
robruboiko, oh, that failed too :-/21:22
robruboiko, try force rebuild + ignore step21:22
robruboiko, nm, i just did it21:22
robruboiko, looks good: https://ci-train.ubuntu.com/job/landing-009-1-build/6/console21:23
robrubbl for real ;-)21:23
boikorobru: weird, it is trying to apply revision 48, but the MR already got the revision 4921:23
boikorobru: ah ok, now it got the rev 4921:23
=== plars changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: cihelp | CI Train support - US: robru, cyphermox, rsalveti - EU: sil2100, Mirv, didrocks | CITrain support no answer: use mup bot after 30 minutes, but choose right timezone | Known issues: makos are down
=== jhodapp|bbl is now known as jhodapp

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!