[01:50] any core devs around for a quick ack ^ ? [01:50] infinity, cjwatson ^ ? [01:57] robru: I assume there are appropriate code changes to match the s/python-six/python-pil/ bit? [01:58] infinity, I also assume that. [01:58] (I'm not sure I agree with the truncated debian-dir-only diff review, it's a blind review, matching packaging changes to upstream bits we don't see) [02:00] infinity, yeah, just checked the upstream diff, they rip out six and introduce PIL [02:00] robru: So, the changes look fine, my only complaint is that the changelog doesn't mention python-pil, nor the XDG_RUNTIME_DIR fix. [02:00] infinity, k, thanks [02:05] === trainguard: IMAGE 175 building (started: 20140806 02:05) === [02:09] infinity, robru: ah rats, I must have missed that change when putting together the change logs === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha [03:33] great that the autopilot got in! [03:40] === trainguard: IMAGE 175 DONE (finished: 20140806 03:40) === [03:40] === changelog: http://people.canonical.com/~ogra/touch-image-stats/175.changes === [06:58] sil2100, good morning! can silo 15 be published? [07:21] sil2100, you around? [07:22] oSoMoN: hello! Sorry, was AFK for a moment [07:22] sil2100, no worries! would you mind publishing silo 15 for me? [07:23] oSoMoN: sure, one moment [07:27] good morning [07:28] davmor2, you around? [07:30] hmmm [07:30] oSoMoN: ok, we seem to have some problems, one moment [07:31] sil2100, what sort of problems? [07:32] woot, this error message doesn’t make sense [07:32] oSoMoN: trying to dig into why CI Train doesn't see the oxide upload [07:34] oSoMoN: this might take a minute... [07:35] if it’s only a minute, I’m fine with it :) [07:55] oSoMoN: ok, fixed... I *should* be able to publish your package in the nearest minutes ;) [07:59] oSoMoN: done \o/ [08:03] Saviq, reading zoltan's update mail and wondering, if the UbuntuShape optimizations have landed in the image, yet? [08:06] tvoss: since 173, yes [08:07] Mirv, ah, just flashed devel-proposed and I was expecting some visible improvements in the app scope [08:08] sil2100: Which package(s) did you push to dogfood? [08:09] sil2100: I'm pretty worried that we aren't going to have a recent promotion before the point when it comes time to branch :-/ [08:10] tvoss: I guess the app scope perf problems are more complicated than just ubuntushape [08:18] cjwatson: I pushed indicator-location till the end [08:18] cjwatson: yeah... today I'll push more on various fixes to get a promotable image ASAP, but we're a bit worried about that as well [08:21] ogra_: hey, do you remember if plars and jdstrand were able to find someone who can fix the bug causing the AP failures? [08:22] grrr [08:22] Even more failing tests [08:22] I think we need to stop the line and only concentrate on fixes [08:22] tvoss, don't think so [08:23] Saviq, Mirv just told me that they landed in 173 [08:24] sil2100, in 174 i see the gallery-app test has passed fine ... so there is hope [08:25] ogra_: yeah, but address-book-app has 13 failures, messaging and dialer suddenly got worse as well [08:25] ogra_: while I have been told that the uploads for those were supposed to *fix* [08:25] tvoss, ah correct http://people.canonical.com/~ogra/touch-image-stats/173.changes [08:25] sil2100, thanks! [08:25] the batching of ubuntu shapes was introduced here: https://launchpad.net/ubuntu/+source/ubuntu-ui-toolkit/1.1.1179+14.10.20140804-0ubuntu1 [08:25] hmm, it seems the changelog was broken [08:25] sil2100, https://launchpad.net/ubuntu/utopic/+source/dbus-cpp/4.0.0+14.10.20140730-0ubuntu1 this landed in 163 ... i wonder if tearing down the dbus daemon could cause such apparmor issues [08:26] in that none of the bugs were actually closed automatically [08:27] ogra_, it tears down its private instance of the daemon, not the actual daemons [08:28] tvoss, right, but does apparmor know (or care) and could this cause all the denials [08:29] ogra_, apparmor does not know, the testing buses are used during build time tests only [08:29] ogra_, and don't run on the image [08:29] tvoss, the denials we see are for a specific autopilot introspection interface [08:29] which could well be the one you have just torn down [08:30] ogra_, nope, the testing buses are not run during image tests [08:30] ogra_, they are solely run at package build time [08:30] ogra_: anyway, the address-book-app failures do not seem apparmor related, need to find renato___ or boiko... [08:30] tvoss, we have to call out TRAINCON0 since we have 100s of tests failing due to that ... i'm grabbing for straws here [08:30] renato___: ping [08:30] ogra_, do you have the denials in a pastebin somewhere? [08:31] tvoss, we have them in every console log for every test run :) one sed [08:31] *sec [08:31] https://jenkins.qa.ubuntu.com/job/utopic-touch-mako-smoke-daily/666/artifact/clientlogs/gallery_app/syslog/*view*/ [08:31] either there [08:31] http://jenkins.qa.ubuntu.com/job/utopic-touch-mako-smoke-daily/666/consoleFull or there [08:31] just search for "DEN" [08:32] Aug 6 05:11:51 ubuntu-phablet dbus[2210]: apparmor="DENIED" operation="dbus_method_call" bus="session" path="/com/canonical/Autopilot/Introspection" interface="com.canonical.Autopilot.Introspection" member="GetVersion" name=":1.88" mask="receive" pid=3704 profile="com.ubuntu.gallery_gallery_2.9.1.1025" peer_pid=3689 peer_profile="unconfined" [08:32] thats a typical one [08:33] sil2100, hey, reconfigure silo 1 please, added unity-api there [08:35] ogra_, interesting ... the peer calling into gallery is unconfined [08:35] ogra_, also, apparmor wouldn't deny if the daemon wasn't running [08:35] ogra_, did you verify that the peer pid actually *is* autopilot? [08:36] tvoss, i dont know what was researched by plars and jdstrand, they spent quite some time on this already (apparently unsuccessfull) [08:43] what is the generic-deb job for uss and why can't it find a package that all the other jobs find? [08:44] trainguards, reconfigure of silo 1 please, added unity-api there [08:44] Saviq: one moment, in a meeting :) [08:44] sil2100, pfft! ;P [08:45] Saviq, camako I'm about to set silo 10 to testing done, please make sure to rebuild your qtmir(-gles) silos [08:45] tvoss, kk [08:45] tvoss, ack.. thanks for the heads up! [08:46] ogra_, is there a known issue with 175? it kinda doesn't boot [08:46] stuck on google logo [08:46] theerare plenty of issues, but booting isnt one [08:46] http://ci.ubuntu.com/smokeng/utopic/touch/mako/175:20140806:20140805.2/9528/ tests run ... [08:46] Saviq, also https://code.launchpad.net/~thomas-voss/qtmir/gles-sync-20140804/+merge/229425 [08:46] can adb in though [08:47] check with top if apparmor still parses stuff [08:47] ogra_, nope, not doing anything really [08:47] werid [08:47] and thats a normal readonly OTA upgrade ? [08:48] ogra_, no, normal ubuntu-device-flash [08:48] with non-wiped data [08:48] with --bootstrap ? [08:48] no [08:48] that might be your issue, we got new android bits (OTA would have been cleverer here) [08:49] (your kernel/initrd could be out of sync) [08:49] i'll bootstrap then [08:51] sil2100, could you check why silo10 never finishes building? [08:51] tvoss, not published on powerpc/ppc64el yet [08:52] tvoss, that's gonna be a problem [08:52] Saviq, for which package? [08:52] tvoss, location-service [08:52] tvoss, it was available in all arches before https://launchpad.net/ubuntu/+source/location-service/2.0.1+14.10.20140731-0ubuntu1 [08:53] Saviq, ack ... deps on trust-store [08:53] Saviq, which now deps on mir, which is not available on those architectures [08:54] tvoss, yeah, that means some archive guru needs to have a look [08:54] Saviq, I will just make the build dep optional [08:54] tvoss, like we need to drop the unsupported arches' packages from archive [08:54] tvoss, or yeah, if you can make it go for other arches still, sure [08:58] sil2100, do you have the details on how the apparmor denials were introduced? === vila changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | CI Train support: robru, trainguards | Vanguard (general help): vila | CI Train Status: #157 promoted | CI Train Dashboard: http://bit.ly/1mDv1FS | Known issues: - [09:01] sil2100: awesome, so that made it through proposed-migrationd [09:01] sil2100: maybe we should consider branching from not a promoted image ... [09:15] Saviq: looking at timestamps you didn't get the reconfiguration yet, so doing that [09:16] Mirv, nope, didn't, thanks [09:16] cjwatson: \o/ We're announcing TRAINCON-0 today, as we want to resolve the image situation ASAP [09:18] Saviq: sil2100: reconfigure done [09:19] NOOOOOOOOO [09:19] sil2100, is it now? [09:19] Saviq: so sad about reconfigure? ;) (I know, I know) [09:24] Saviq: we're announcing it in a moment! [09:24] sil2100, can you wait until I publish silo 1? ;D [09:24] ;p [09:28] Saviq, I rebuilt qtmir and version changed to 0.4.0+14.10.20140806-0ubuntu1. I need to update the twin. If I use 'dch -v' then your old entry remains. Should I manually update your entry? [09:29] camako, was the previous version released? [09:29] Saviq, no [09:29] camako, if it was UNRELEASED, then it should get updated, let me have a look [09:31] Saviq, qtmir still building armhf though (not completely finished) [09:31] camako, shouldn't be a problem [09:31] camako, that's silo 7 is it? [09:31] Saviq, yes [09:32] camako, why did you rebuild btw? [09:32] camako, if on tvoss's orders then too early ;) [09:32] Saviq, getting there with trust-store [09:32] Saviq, a lot of packaging changes broke other downstreams, then we changed mir to fix.. thought I'd rebuild qtmir as well [09:33] camako, ah ok [09:33] Saviq, yes... not on tvoss' orders [09:34] camako, in any case, http://paste.ubuntu.com/7968849/ [09:34] camako, if I go `dch -v 0.4.0+14.10.20140806-0ubuntu1 ""`, only the version and time gets updated [09:35] camako, is that not what's happening for you? [09:35] * camako looks [09:35] sil2100: new blocker I'm just going to dig into when it landed. The guide swipes away and leaves the phone in a position where none of the apps actually open [09:37] sil2100: ok [09:38] Saviq, I get http://pastebin.ubuntu.com/7968870/ [09:38] I guess my identity needs updating [09:38] camako, yeah, that, too [09:39] camako, but yeah, *inside* a version entry clear it up to taste [09:39] Saviq, ok thanks. [09:39] camako, export DEBEMAIL="michal.sawicz@canonical.com" in .profile [09:39] camako, and DEBNAME accordingly [09:39] Saviq, ah I see [09:40] well, export your own email of course ;) [09:40] :-) [09:40] I always wanted to have dreadlocks... [09:53] Saviq, the twin needs to be rebuilt after the version update, right? [09:53] camako, yup [09:54] davmor2: yeah, I added that to the blocker list already [09:54] davmor2: oh, or wait, no, that's something different I gues? [09:57] sil2100: I'm just digging back to find when it landed, Saviq already thinks it is likely the Scope as an app landing but still going to confirm [10:03] sil2100: you approved the ms2 branch, do you know when that will hit the archive? https://bugs.launchpad.net/music-app/+bug/1350529 [10:03] Launchpad bug 1350529 in Ubuntu Music App "[music-app] Trunk fails autopilot tests on jenkins" [High,In progress] === tvoss is now known as tvoss|test === tvoss|test is now known as tvoss === ricmm_ is now known as ricmm === sil2100 changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | CI Train support: robru, trainguards | Vanguard (general help): vila | CI Train Status: #157 promoted | CI Train Dashboard: http://bit.ly/1mDv1FS | Known issues: Forceful TRAINCON-0! Please work together on getting all blockers resolved [10:12] * popey lols at sil2100's mail - "incontinences" [10:21] popey, sil2100: inconvenience I bet is the word you were after [10:22] Typooo! [10:23] popey: anyway, hm, let me check ms2 [10:23] popey: ok, it's in the archive now [10:24] ogra_, could you merge this pls? https://code.launchpad.net/~pete-woods/click-sync/add-youtube-scope/+merge/229065 [10:24] it's been sitting there for a week [10:24] ish [10:24] sil2100: how did the apparmor denials shake out overnight? [10:24] slangasek: what do you mean? [10:25] sil2100: did the fixes that plars/jdstrand landed fix things, or are we still broken? [10:25] slangasek: they're still broken, I didn't know that jdstrand/plars had any fixes - I only heard they identified/triaged the problem, but as it's something on the system level they could not do much [10:26] hmm [10:26] slangasek: in any case, 174 is still broken [10:26] not sure why it being on the system level prevents them from doing things [10:26] but if they can't, who owns it now? [10:26] slangasek: the problem seems that they don't know how exactly to proceed and in which component the problem lies, and this is something that needs to be done [10:28] right, so, who owns this now? Given how critical this is, it would be awesome if someone were working on this in European time [10:28] slangasek: it seems that sometimes the system clock is not set correctly at the right time (if I understood it correctly?) and it causes all those apparmor problems with profiles [10:28] yes, that's roughly what I understood but I don't have the details [10:28] We have no idea, we were trying to think of someone during the meeting but we're clueless [10:28] sil2100, it looks like most of the failures are down to a uitk change [10:28] hmm ;) [10:28] The only info we have is from the backlogs [10:29] sil2100, since messaging, address-book and dialer all share the same cause [10:29] We can provide those to the general public, but that's why I called out for people that could help out with this issue [10:29] brendand: oh! [10:29] sil2100, StateNotFoundError: Object not found with name 'UbuntuShape' and properties {'objectName': 'bottomEdgeTip'}. [10:29] * sil2100 gives bzoltan the evil eye (again) [10:29] ;D [10:29] sil2100: what? [10:30] brendand: is that an autopilot test? [10:30] bzoltan, yes [10:30] bzoltan: j/k with that evil eye! But yeah, brendand mentions that all our recent failures are related to the UbuntuShape bottomEdgeTip [10:30] bzoltan: do you know anything regarding that? [10:30] t1mp: hey! [10:30] sil2100: yes [10:30] bzoltan, and actually it seems to indicate a functional failure [10:31] sil2100: that is the fix I sent for the short app... [10:31] brendand: No it indicates that the APs should not use the UbunuShape as objecttype [10:31] bzoltan: ok, so it seems that we'll need the same fixes for the other apps then [10:32] zsombi: hello! Any progress on bug LP: #1351024 ? [10:32] sil2100: how it did not pop out on the autopilot tests? [10:32] Launchpad bug 1351024 in Ubuntu Calendar App "Date & Time picker is not working on device." [Critical,Triaged] https://launchpad.net/bugs/1351024 [10:32] bzoltan, that's not a justification [10:32] brendand: what is not a justification? [10:32] bzoltan, if the tests break, then they need to be fixed [10:33] brendand: What I say that the tests did not break when I run them [10:33] jdstrand, plars: btw. guys, did you have a bug for the autopilot apparmor denials caused by the clock? [10:34] bzoltan, brendand: yeah, so hm, maybe the problem was in some recent address-book+dialer+messaging landing? [10:35] bzoltan, brendand: since when UITK was landing, I remember 1-2 landings of those apps as well [10:35] sil2100: I have like 3 OK logs for these apps with the UITK I landed [10:36] bzoltan, the animationn broke on all 3 apps too [10:37] bzoltan, i don't think they could all break in the same way simultaneously [10:37] bzoltan, i guess it's possible, but not likely [10:37] brendand: could you please give me some more context [10:37] bzoltan, try it on 175 [10:37] sil2100: just got it, I have an other ugly bug to fix before that [10:37] bzoltan, it's a bit hard to explain. basically when you pull the bottom edge up, the motion is all strange [10:38] bzoltan, i'll post a video :) [10:38] brendand: try what? I still do not understand whatthe problem is. I can help... but tell me what should I look for [10:40] bzoltan, best way to sum it up is the bottom edge tip is not drawn at the top of the page during the animation [10:40] bzoltan, but again, only a video, or trying it yourself really does it justice [10:41] sil2100: what's up? [10:41] t1mp: ah, sorry, unping ;) Since I remembered you assigned on the date-picker bug, but then noticed that it got reassigned to zsombi [10:42] t1mp: sorry for the disturbance! [10:44] sil2100: ok :) [10:47] sil2100: if I understand well the new UITK landed on rev173 and these problems appeared on rev175. So I think the UITK i snot the prime suspect [10:49] bzoltan: right, as mentioned it might be one of the messaging/dialer/address-book landings, just wanted to make sure it's unrelated as brendand pointed out UITK as a possibility [10:49] Anyway, I will poke boiko once he's up anyway [10:49] sil2100, i was pointing at uitk because it was common [10:50] sil2100, but since it broke when all those apps got updated then the finger points in a different direction now [10:50] sil2100, i'm really curious how people are actually testing stuff [10:50] brendand, bzoltan: yeah, and thanks to this we know more or less where to poke further :) [10:50] sil2100, several really bad regressions here [10:51] brendand: indeed... need to make sure boiko runs all his tests on a mako device [10:51] sil2100: me and the whole SDK team is at your disposal if you need more investigation or bugfix. [10:53] sil2100, where can i get more details about those app landings? [10:53] sil2100, so i can try and find what might be the cause of the problem === MacSlow is now known as MacSlow|lunch [11:02] sil2100: ha that was nice timing I'd already sent out a heads up email for the traincon0 and then your email landed shortly after :) [11:04] brendand: you can use http://people.canonical.com/~lzemczak/landing-team/ to check what landed when :) [11:04] davmor2: hah ;) [11:10] brendand, that issue in dialer/messaging/address-book you're talking about isn't it caused by the 'visual updates' of these applications that landed yesterday? [11:11] jibel, well yes - that part we know [11:11] jibel, trying to figure out how they missed it [11:12] sil2100, anyway i'll get a bug filed for all three projects [11:12] brendand: ok, thanks :) I'll poke boiko when he's up automatically anyway [11:15] sil2100, https://bugs.launchpad.net/address-book-app/+bug/1353420 [11:15] Launchpad bug 1353420 in messaging-app "Autopilot tests that use the bottom edge failing since #174" [Undecided,New] [11:16] sil2100, ogra_, Saviq: so the guide works as expected in 173, trying 174 now [11:16] davmor2: ok, so a recent regression then! [11:16] davmor2, really, you can stop :) [11:16] brendand: thanks! Adding to teh blockerz [11:16] davmor2, it's dash-as-app's fault 100% [11:16] davmor2, I'll be on it just after I eat something [11:18] Saviq: I'm just being thorough it's what we do you know ;) No worries enjoy your food :) [11:19] sil2100, video of the issue :) https://plus.google.com/u/1/110434705244077414661/posts/YVnQPrvmWJE?pid=6044400154069719906&oid=110434705244077414661 [11:25] Saviq, ogra_, sil2100: yeap so 174 is the broken landing, I'll write up a bug for it and completely blame Saviq ;) [11:26] davmor2, there is one already [11:26] Saviq: oh nice [11:26] davmor2, bug #1353351 [11:26] bug 1353351 in Unity 8 "Edges intro finishes after right edge is performed and always runs on boot" [Critical,Triaged] https://launchpad.net/bugs/1353351 [11:27] ah brendand beat me too it :) [11:30] brendand: sil2100: bottom edge swipe problems - seems the apps are using their own one, and seem recently in 174 one app copied the component from teh other... [11:30] brendand: sil2100: so it aint came from UITK [11:31] zsombi, yeah - i filed the bug against the apps themselves [11:31] brendand: sil2100: all UITK provides is a Panel component (invisible one) which provides bottom-edge swipe detection, all the rest comes from apps [11:31] brendand: ok, thx!! [11:54] sil2100, hi === cprov changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | CI Train support: robru, trainguards | Vanguard (general help): cprov | CI Train Status: #157 promoted | CI Train Dashboard: http://bit.ly/1mDv1FS | Known issues: Forceful TRAINCON-0! Please work together on getting all blockers resolved [12:14] sil2100, for your viewing pleasure: bug #v [12:14] 1353451 [12:14] bug #1353451 [12:14] dammit [12:14] bug 1353451 in Canonical Upstream To Distro "build job in train should check whether built version is higher than in silo" [Undecided,Confirmed] https://launchpad.net/bugs/1353451 [12:15] sil2100, the problem with address-book dialer messaging app is that the UbuntuShape has changed the type name, we need to update the autopilot tests for that [12:45] sil2100, so it appears that the app updates were never tested with 173, which had the version of UITK that impacted them [12:45] bzoltan, ^ - fyi [12:45] sil2100, we need to have a tighter test/release cycle [12:46] brendand: sil2100: what is the problem? [12:46] sil2100, silos must be sure to be tested with the image prior to the one they are to be published in [12:46] bzoltan, the apps were broken by the uitk update in 173 [12:46] bzoltan, but they were never tested with 173 [12:46] brendand: what apps? [12:47] bzoltan, so uitk did break them, but it's not strictly speaking your fault [12:47] brendand: how can I test with an image what does not exist? [12:47] just a faulty process [12:47] bzoltan, you can't - i'm not placing any blame on you [12:47] brendand: Than please do not :) [12:48] brendand: The UITK did not break anything ... I have tested the UITK with the latest and most fresh available image. UITK can not be blamed for failures integrated after it lands. [12:48] sil2100: ^ [12:49] bzoltan, that's what i said - it's not the fault of uitk or it's release process, the fault is in the general landing process [12:49] brendand: I kind of expected that after I run 24 apps autopilot test suites and ~800 tests I will be asked to test against the development branches of the apps too :D [12:49] brendand: what if the apps would test against the latest released UITK at the first place? [12:50] bzoltan, they should - and that's what went wrong [12:50] brendand: I am bagging for a staged releasing system for ages... we need an image with the staging UITK and one for the staging Qt to foresee the possible upcoming issues. [12:51] bzoltan - that certainly would be helpful given the present release process [12:51] brendand: we should have channels where we can customize the development branches and possible set PPAs with development versions [12:51] brendand: this is something I am asking for very long time. [13:05] sil2100, are you guys doing a standup with ppl working on the promotion blockers [13:05] renato___: could you guys take care of those tests? :) === Ursinha changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | CI Train support: robru, trainguards | Vanguard (general help): Ursinha | CI Train Status: #157 promoted | CI Train Dashboard: http://bit.ly/1mDv1FS | Known issues: Forceful TRAINCON-0! Please work together on getting all blockers resolved [13:07] brendand: hm, so, in other words - the behavior changed in UITK but this did not break the tests, but when the new apps were released those were not tested against the new UITK and tried using 'old' features that got removed in UITK, right? [13:07] olli: we're pushing people about the blockers all the time (except for lunch periods) [13:08] sil2100: I did not file a bug. I was thinking plars would after his investigations [13:08] zsombi: give me a sign once you have any updates regarding the date-picker in calendar [13:09] sil2100, are you getting the right attention [13:09] sil2100, kind of. i'll explain more in the landing meeting [13:09] jdstrand: ok, since we would like to find someone who could actually work on fixing the root cause - the bug itself is a bit strange (or maybe I lack some understanding here), but do you have any idea who could be the best person or at least which component should be worked on to get this fixed? [13:10] olli: so far yes, everyone's responding to pings when they're available, the only thing left is finding someone who could work on the apparmor-smoketesting problem [13:10] jdstrand: sil2100: I don't have one yet, or know yet where to file it. We know "what" is happening, but still have no idea why [13:11] plars: right... this is exactly what I was asking jdstrand now, and we were also thinking about that during the morning meeting [13:11] As we don't really even know 'who' to poke about getting this fixed [13:11] Any ideas? [13:11] sil2100: honestly, I have no idea. it could be a provisioning issue, it could be a hardware issue, it could be a bug in Ubuntu [13:12] is that the clock issue ? [13:12] there is probably something else it could be that I am not thinking of [13:12] ogra_: yes [13:13] so i was wondering ... if the hwclock doesnt get updated (i dont think we do that) ... your clock wil be off until ntpdate is called by /etc/if-up.d [13:13] it could be worked around in the CI infrastructure, but I agree with plars that it is unwise to paper over it in case it is a bug in Ubuntu since we don't want phones running around thinking it is 1970 [13:13] that will only be called once NM starts [13:14] sil2100: I'd be happy to put a placeholder for it somewhere though, do we have a catch-all place for stuff like this on touch? [13:14] if you reboot and the apparmor profiling gets re-run at that point, the clock will still be off [13:14] ogra_: right, but why is it off by so much sometimes, and not others? [13:14] ogra_: there is a hwclock-save upstart job on the device that runs in runlevels 0 and 6, but aiui, adb reboot (what the CI tools do) doesn't run through those runlevels [13:14] i wonder if making the apparmor upstart job "start on started network-manager" would help here [13:14] ogra_: for some runs, when the device is installed and boots, it's off by maybe 1-8 seconds. For others, it's gone way back in time to January 12, 1970 [13:15] plars, because we never set it [13:15] ogra_: it isn't the apparmor upstart job [13:15] plars, you always operate with the same hwclock setting until ntpdate kicks in [13:15] ogra_: I think we'd still have the race even if we wait to ntpdate after nm starts if that's what you mean [13:15] ogra_: it might narrow the window a bit, but that's all [13:16] jdstrand, i thought it was the time discrepancy of the generated files [13:16] ogra_: sure, but it's never set for any of them [13:16] ogra_: yet some preserve a fairly close time, some are just way off [13:16] we have precompiled policy cache on the device with a date of 2014. when the clock is off, aa-clickhook is run with the clock still at 1970. it generates new policy with timestamps of 1970. it tries to load them into the kernel, but see the precompiled cache has date of 2014 [13:16] so it skips them [13:17] jdstrand, right ... and making sure the clock is right first (by starting after NM) might be a (not very good, but usable)workaround [13:17] this isn't the apparmor upstart job [13:17] aa-clickhook is run by the CI tools [13:17] aa-clickhook -f --include=... [13:17] hmm [13:18] hmmm [13:18] that command ^ is run when the clock is at 1970 [13:18] we could just wrap that with: ntpdate ntp.ubuntu.com [13:18] the clock needs to be correct when that runs [13:18] sure, there are various workarounds [13:18] we could after running ntpdate do 'start hwclock-save' [13:18] traingaurds, can I please get a silo for line 26 ? [13:19] but that doesn't address why the clock is at 1970 in the first place [13:19] jdstrand, right, it should only happen when the battery and the RTC backup are totally depleted [13:19] ogra_: sure, we can force ntpdate before we do anything, we could even force hwclock as a provisioning step. But we don't want to potentially conceal a bug here that when a user installs and boots a phone, sometimes they could have a good date, other times they won't. Also, if they never shutdown the phone normally they may never run hwclock at all and the whole thing could start over [13:19] we should actually make it fail if $clock < $installation date [13:20] ogra_: my gut feeling is that most people would never go through a "normal" shutdown, but rather wait until the phone is stuck and hold the power button for a hard reboot [13:20] plars: pmcgowan had an interesting point. is there a way to detect that? [13:20] but lab devices are all plugged in I assume [13:20] jdstrand: sure, hwclock by default will read what it's set to [13:20] plars, that would be sad for the people that implemented the new shutdown/reboot dialog :P [13:21] ogra_: sure, but do you ever reboot your personal phone for fun? :) [13:21] (which btw, that dialog comes up every time I touch the power button it seems) [13:21] * jdstrand assumes that is a bug fixed in proposed... [13:21] plars, nah, never indeed [13:23] jdstrand, plars do you have the kernel logs/logcat available from the lab devices? [13:23] well, hwclock -s/-w both work fine [13:23] so we should simply make sure that we set it on boot [13:23] or if ntpdate retrieved a new time or ... or ... [13:24] currently the upstart job that does that only runs in 0 and 6 [13:24] tvoss: yes, one moment and I'll point you at some [13:24] I was wondering why the hwclock wasn't set after ntpdate [13:24] ogra_, it still would just hide the issue [13:24] (just as a matter of course) [13:25] tvoss: basically anytime you see these apparmor "DENIED" results, it's typically this problem (for now at least): http://ci.ubuntu.com/smokeng/utopic/touch/mako/175:20140806:20140805.2/9528/ubuntu_calculator_app/1496111/ [13:25] tvoss: from there, you can link to syslog below for the full syslog [13:25] tvoss: and in syslog, you'll find: Aug 6 05:06:31 ubuntu-phablet ntpdate[2787]: step time server 91.189.89.199 offset 1400004685.653897 sec [13:26] tvoss, huh ? [13:26] plars: interesting, that has a date of Mar 26 instead of Jan 12 [13:26] tvoss, the issue is that whenever we set the time we dont set the hwclock [13:26] jdstrand: that would at least keep it from recurring on later boots [13:26] s/whenever/has/ [13:26] jdstrand: oh, didn't notice - yeah the ones we looked at yesterday were all Jan 12 [13:26] we're moving forward in time at least :) [13:26] ogra_, are we sure that that is the actual issue? [13:26] Mar 26 is enough behind to cause a problem though [13:27] indeed [13:27] plars: we'll get there! [13:27] hehe [13:27] tvoss, if the HW clock is permanently at 1970 i'm 99.5% sure, yes [13:27] sloooowly [13:27] ogra_, why is that only showing up now? [13:27] tvoss, no idea ... [13:27] ogra_, that's the point I'm trying to make [13:27] well, it is a race [13:27] probably apparmor didnt care about timestamps before ? [13:27] but we are certainly hitting it a lot now, where we didn't seem to at all before [13:27] ogra_, I'm pretty sure it did [13:28] jdstrand, ^ [13:28] or it simply ran a nanosecond later [13:28] the issue is only showing up now I think because we only started precompiling apparmor policy [13:28] which is the nanosecond that ntpdate runs delayed today [13:28] recently [13:28] right [13:28] I don't know the exact date [13:28]