[00:06] robru: are you able to paste me the link where I can follow that test run please? is it this? http://10.97.0.1:8080/job/autopilot-saucy-daily_release/ [00:07] thomi, not much to see yet: http://10.97.0.1:8080/job/cu2d-qa-saucy-2.1build/34/console [00:08] robru: awesome, I just wanted to catch it when it ended === Ursinha is now known as Ursinha-afk [01:17] robru: fginther: this job looks stuck to me? http://10.97.0.1:8080/job/autopilot-saucy-daily_release/2126/ [01:17] I'm still waiting for that qa stack run to end [01:18] thomi, i see a test failure: http://10.97.0.1:8080/job/autopilot-saucy-daily_release/label=autopilot-intel/2125/testReport/junit/unity.tests.test_panel/PanelKeyNavigationTests/test_panel_indicators_key_navigation_prev_works_Single_Monitor_/ [01:18] robru: but that's the unity tests, I'm after the autopilot tests [01:31] thomi, dunno then. it could be stalled. there's been lots of infrastructure issues lately, but I dunno many details. maybe didrocks would be better to answer that question? he'll be on in a few hours === Ursinha-afk is now known as Ursinha [01:33] yeah, I guess I' [01:33] ll wait [01:33] oops [01:33] thanks for your help anyway [06:03] Mirv: do you have some free slots? [06:04] good morning btw ;) [06:06] didrocks: morning! not really right now, I've still not been able to start what I suppoed to start with 2h ago because I was debugging sdk problems (which turned out not to probably be sdk problems) [06:06] Mirv: ok, good luck then! [06:06] cu2d would also have some bugs to file, but I'll let it run the next tick first [06:06] ok [06:06] libfriends was hanging cu2d since last evening, and will probably do that again so I'll keep an eye on that [06:07] Mirv: robru didn't kill it? [06:07] Mirv: I explicitely asked him to kill it [06:07] didrocks: no, it was still running [06:07] ok, let's remove libfriends then [06:07] from the config [06:07] Mirv: we can't have timeout on building because some builds can lag a lot (not the case recently, was the case in the past) [06:07] ok [06:07] that's true [06:08] robru: removing libfriends, please fix the buildd issue I pointed you at yesterday (and see spreadsheet) [06:08] didrocks: ah, correction, too many things this morning. it was actually that intel + nvidia jenkins slaves had died.. [06:08] Mirv: both? again? :/ [06:09] didrocks: and that was since yesterday evening, and then only after that libfriends hung [06:09] didrocks: could we get QA team to somehow have autokicker/monitor/alerting on that? [06:09] Mirv: let's raise it in this morning call [06:10] Mirv: libfriends removal deployed [06:10] didrocks: I just noticed, thanks [06:11] Mirv: do not hesitate to do that next time btw ;) [06:11] (as long it's logged somewhere like in the spreadsheet) [06:11] right.. [06:17] didrocks, Mirv: Not fully awake yet but a bit of explanation would help ? (Context, I'm ramping up with how the ci engine works so speak slowly ;) [06:17] intel + nvidia jenkins slaves ? slave names ? [06:17] vila: the 2 slaves being the test machines [06:18] it happens quite often that we have a kernel panic [06:18] and so the machine is stuck [06:18] the jenkins node don't work [06:18] so we can't contact them [06:18] kenerl panic seen in the jenkins logs or requiring shell access ? [06:18] vila: seen in logs (jibel most of the time get those) [06:18] but before, the machines needs to be rebooted [06:18] electrically [06:19] through the CDU [06:19] ouch [06:19] so we would need something monitoring those slaves [06:19] yup [06:19] and if they don't answer for 5 minutes, reboot electrically [06:19] probably not jenkins itself but yeah [06:19] and ensuring the jenkins slave started [06:19] yeah [06:19] * vila nods [06:19] vila: mind adding that to the TODO? ;) [06:19] as this is blocking everything when it happens and nobody monitors [06:20] * vila 's.head.TODO++ [06:20] (everytime we report to the kernel team, it's "have you tried latest kernel update?") [06:20] thanks [06:20] didrocks: I'm using this channel log as a huge TODO for now :-/ [06:20] heh [06:21] "libfriends removal deployed" how ? What does it mean ? [06:22] didrocks, Mirv: a commit reference will help me understand how the config work ;) [06:22] vila: latest commit on lp:cupstream2distro-config [06:24] http://bazaar.launchpad.net/~cupstream2distro-maintainers/cupstream2distro-config/trunk/revision/814?start_revid=814 [06:24] ok ,that's the how [06:24] what does it trigger/stop to trigger ? [06:24] it doesn't build anymore libfriends [06:25] in the daily release machinery [06:25] (basically no prepare-libfriends jobs) [06:25] so the last good version is used instead ? [06:25] right [06:25] \o/ [06:26] which version is that then ? ppa/proposed/archive whichever comes first ? [06:27] whicherver comes first [06:27] the highest version [06:27] great [06:27] and then you trigger a stack build again ? [06:27] well, it's triggered every 4 hours [06:27] and we just had a tick [06:27] so should be fine [06:27] haaa, the tick! [06:28] any wau to trigger that manually or will that result in a huge mess ? [06:28] *way [06:28] we can, but not needed in that case [06:28] ok [06:28] but when you trigger manually you need to know what's the current dep are [06:28] elaborate ? [06:29] if you have indicators building [06:29] and you trigger unity [06:29] maybe you are building against half the new indicators [06:29] and half the old ones [06:29] oh right, bad idea [06:29] so you don't know which state you are having [06:29] any way to abort all in flight builds ? or some of them ? [06:30] manually in launchpad [06:30] or is that too tricky/unreliable ? [06:30] but again, you are in an unknown state [06:30] in lp ??? [06:30] as you maybe have binaries published [06:30] yeah [06:30] oh, the builds happen in lp after you dput to some ppa ? [06:31] ? [06:31] that's how the system work [06:31] or I don't understand the question? ;) [06:31] I'm learning so I may miss some obvious things ;) [06:31] look at my presentation [06:31] the prepare dput to a ppa [06:31] then, we monitor the build into that ppa [06:34] https://docs.google.com/a/canonical.com/presentation/d/1-1qB0GamQhdvCHFa49fe6x2emqSe0e_LNnITZRiZLuE/edit#slide=id.gf421d6b8_1107 ? [06:34] vila: right [06:35] how do you find the the lp build url ? [06:40] didrocks, vila retoaded could add a nagios check and sned a fence_cdu command if the mahcine doesn't respond after a timeout [06:41] didrocks, 10min is safer than 5 because the mahcines are rebooted every day and if there is an fsck it can take a while [06:41] jibel: I understand what you mean but I have no idea on how to make it real :) Does retoaded know enough about that to turn it into a working solution ? [06:41] and in this case there is a risk of boot loop [06:41] vila, yes he knows everything [06:41] jibel: yeah, making sense :) [06:42] vila: why do you need it? the build monitors it [06:42] jibel: great, will check with him then [06:42] it only print the launchpad build url in case of failure [06:42] if you want to play a little bit, look at the mir stack, mir failed on i386 [06:42] didrocks: I don't need it *now*, I'm trying to understand what I will do the day I need it ;) [06:43] vila: really, try poking around in the UI first [06:43] me or anybody with the required access, whatever [06:43] as it seems it's blurry for you ;) [06:43] with the presentation on the other side [06:43] I think it will make sense [06:44] ok, will try, but if I feel I'm blindly poking around I will come back for more hand holding ;) [06:45] vila: if you don't remember my presentation you should read https://wiki.ubuntu.com/DailyRelease (and the links under More specific topic) [06:45] didrocks: oh, I do remember it, but that doesn't mean I understood it enough ;) [06:53] didrocks: starting at http://s-jenkins:8080/view/Mir/job/mir-android-saucy-i386-build/2144/console ? [06:53] failed to open trace file: [Errno 2] No such file or directory: '/home/ubuntu/.bzr.log' [06:53] is a Bad Thing ! [06:55] didrocks: may be unrelated but '1 conflicts encountered. [06:55] bzr: ERROR: Conflicts from merge' is obscure, is this a common failure ? [06:55] vila: this is not daily release [06:55] vila: I don't know that one [06:55] you have a cu2d view [06:55] daily releases are there [06:55] crap, wrong jenkins server ? [06:55] to view the chained jobs, you should use the private instance [06:55] 110.97.0.1:8080 [06:56] 10* [06:57] my kingdom for a dns entry ! [07:01] hmm http://10.97.0.1:8080/view/cu2d/ 3 reds 2 greys, really ? grey as disabled ? Too bad it so looked liked stack directory names :-/ [07:01] http://10.97.0.1:8080/view/mir/job/cu2d-mir-saucy/14/ is more on track ? [07:02] then http://10.97.0.1:8080/job/cu2d-mir-saucy-2.1build/13/console [07:02] then https://launchpad.net/~ubuntu-unity/+archive/daily-build/+build/5050734 ? [07:03] vila: right [07:03] \o/ [07:04] a bit weird that it ends up in a failed compilation... could it be that the dev never saw that failure because some update happened after he successfully compiled or are the builds stricter ? [07:04] vila: doesn't happen on other archs [07:05] hmm [07:05] but it's not an infrastructure issue right ? [07:05] no [07:05] just something the dev could't reproduce locally so a valid output from the ci engine right ? [07:06] everything is valid, I pinged upstream, they just investigate [07:08] ... from there they fix it however they see fit, land on whatever package's trunk in the mir stack and wait for the next tick ? [07:08] or we will relaunch it as it's very urgent [07:08] and block other stacks [07:08] oooooook [07:09] enough food for thought during shave/shower ;) [07:09] didrocks: thanks a ton, sorry for all the silly questions ;) [07:10] no worry ;) [07:20] didrocks: ok now platform is stalled because it waits for mir https://launchpad.net/~ubuntu-unity/+archive/daily-build/+build/5050812 [07:20] canceling to kick it further [07:21] morning guys [07:21] didrocks: two questions [07:21] didrocks: in http://10.97.0.1:8080/view/mir/job/cu2d-mir-saucy/, where is that '(blocking)' coming from for the subprojects ? [07:21] didrocks: first, how's the mir development/experimental branch published? [07:21] Mirv: yeah, alf is looking at it, you can kill both platform and unity8 stack if you want [07:21] right [07:21] didrocks: that is, are we helping land changes from the dev branch of mir into some PPA, or is that some other setup than cu2d? [07:22] lool: there is no more experimental branch for Mir [07:22] Mirv: care to paste the urls you use to kill ? [07:22] there is a mode on cu2d for that, a pity you were not at the Boston sprint when I presented it [07:22] didrocks: second, I didn't figure out parameters to build just one package out of a stack, or everything but one package; I looked at the configured scripts, but didn't see any input vars, so I guess you might be driving these manually, with the shell? [07:23] lool: can be done using the shell command, but as well, you have REBUILD_ONLY [07:23] it's a space separated values of packages [07:23] didrocks: I mean ~mir-team/mir/development-branch [07:23] lool: I don't know about that branch, so no, we don't do anything special with it [07:23] didrocks: so I have to set REBUILD_ONLY in the top job, the master control job for a stack, if I wand this to work, correct? [07:24] right [07:24] ok cool [07:24] it will take the previous state [07:24] but only rebuild those [07:24] (if they have things to release) [07:24] awesome [07:24] you have FORCE_REBUILD parameter [07:24] to tell "please, even if there is nothing new to release, rebuild it" [07:24] (it will add a new changelog entry with "manual build forced" [07:25] there are also the skip dest check parameters, but we can discuss that later on [07:25] didrocks, Mirv: the '(see spreadsheet)' in http://bazaar.launchpad.net/~cupstream2distro-maintainers/cupstream2distro-config/trunk/revision/814?start_revid=814 refers to line 5 in https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0AuDk72Lpx8U5dHFtUmlPOUtCRk8zR2dtaEpIbUVhMmc#gid=3 'Stack status' right ? [07:25] vila: no, current tab, last item [07:26] vila: I use the red 'x':s on the stack's build + check subjob pages [07:26] vila: when logged in to the internal jenkins instance [07:28] didrocks: I see [07:28] Mirv: page url ? [07:29] grr was not logged in couldn't see any running/pending jobs [07:33] Mirv: don't worry about the unity8 failures [07:33] Mirv: it's because of unity-mir not being installable [07:33] because of mir on i386 not producing the -dev [07:33] and the domino world starts! [07:35] 'ping upstream' in ' everything is valid, I pinged upstream, they just investigate' ~== mail allan.lesage@canonical.com since he's 'contact_email' for libfriends ? [07:35] vila: no, pinging #ubuntu-mir on IRC [07:35] emails is too slow [07:36] and we are not talking about friends [07:36] but Mir here [07:38] ok, grep 'mir' stacks/saucy/friends.cfg is empty, how do you know it's mir related ? expertise ? [07:38] vila: http://10.97.0.1:8080/view/cu2d/view/Saucy/view/Platform/ - from there 2.1build and 2.2check. on those pages. [07:41] Mirv: damn, why are they all grey in jenkins ? I thought grey meant 'disabled' :-/ [07:43] is this aborted instead and bubbles up from aborted build to parent jobs ? [07:45] *s/parent jobs/ parents jobs or views/ [07:54] vila: they're grey because they were aborted on this tick [07:54] vila: yep [07:55] didrocks: regarding 'it will add a new changelog entry with "manual build forced"', care to point me an url where the changelog is displayed when you come to it ? IIRC, that was one use case were the --multimaint dch fix was required or two entries for were created [07:55] Mirv: thanks ! Makes far more sense this way ! Pfew [07:55] vila: I also aborted unity already before since because of some previous delays this tick wouldn't have started on time otherwise (unity autopilot tests take the longest, about 1h 45min) [07:55] vila: not now, I'm trying to unblocking production, those are corner cases that don't happen everyday (extra functionality) [07:55] vila: not sure it will be a good use of your time as well ;) [07:56] didrocks: it should be fixed and display a single entry, just want to look if I'm right. If I'm wrong, so be it ;) [07:56] in fact 2 entries [07:56] the "manual build forced" [07:56] + "latest snapshot from rev xxx" [07:56] (the last one is always present) [07:57] vila: you do have examples in the testssuite IIRC [07:57] didrocks: ok, so there is a corner case I don't understand yet. Good, pushed on stack. [07:57] didrocks: ha, vila.TODO++ ^ [08:08] ogra_: hey! how was the image yesterday evening? build went fine? [08:08] didrocks, yes, but the gallery app test on mako failed badly and had to be restarted [08:08] second run seems better [08:08] i didnt want to wait anymore at 1am so its not released yet [08:08] yeah ;) [08:09] vila: FYI pressing red 'x' on http://10.97.0.1:8080/view/cu2d/view/Saucy/view/Unity8/job/cu2d-unity8-saucy-2.2check/ [08:09] will do so once someone to announce at avengers is around [08:09] notes-app regressed? [08:10] i wouldnt see how it had no update [08:10] sdk was updated, right? [08:10] * didrocks looks for ogra's page [08:10] http://people.canonical.com/~ogra/touch-image-stats/ [08:11] Mirv: thanks, so here the reason is in https://launchpadlibrarian.net/151505876/buildlog_ubuntu-saucy-armhf.unity8_7.81.3%2B13.10.20130926-0ubuntu1_FAILEDTOBUILD.txt.gz right ? [08:11] Mirv: as well as in https://launchpadlibrarian.net/151506228/buildlog_ubuntu-saucy-amd64.unity8_7.81.3%2B13.10.20130926-0ubuntu1_FAILEDTOBUILD.txt.gz [08:11] didrocks: http://people.canonical.com/~ogra/touch-image-stats/20130925.1.changes [08:11] there are ui-toolkit changes in that one [08:11] ah [08:11] asac: yeah [08:11] so possibly toolkit regressed those notes-app [08:12] maybe we need jibels page also project what a release changelog would look like against last current [08:12] :) [08:12] Mirv: mind poking the sdk guys with that failing tests? [08:12] vila: true, and handling missing dependencies is not detected at the moment well, it just waits building [08:12] Mirv: http://reports.qa.ubuntu.com/smokeng/saucy/touch_ro/4443/notes-app-autopilot/ [08:12] didrocks: notes app has traditionally been flaky [08:12] have you checked with psivaa if he retried? [08:12] psivaa: ^^ [08:12] http://reports.qa.ubuntu.com/smokeng/saucy/touch_ro/4443/ [08:13] didrocks: I actually did in the morning, and they analyzed it and came into conclusion that it's not toolkit but everything related to 'tapping' so maybe more low-level [08:13] didrocks: I then also noticed that in the landing plan it says ui-toolkit landed in 65, while the regressions started in 64 [08:13] so maybe mir upgrade then? [08:13] http://reports.qa.ubuntu.com/smokeng/saucy/touch_ro/4441/ notes app was still ok on the .1 build [08:13] Mirv: thanks, vila.sanity++, sanity is good, we want moar ;) [08:13] i really believe its a retry thing [08:13] psivaa: ^^ :) [08:14] Mirv: hum, right [08:14] asac: not on maguro though [08:14] or maybe just flackiness, I don't see apart from the lxc config I don't know about, what can impact on 65 [08:14] and there's rssreader failure on both starting from 64, plus calendar failure on maguro starting from 64 [08:15] ogra_: yo [08:15] ogra_: we releasing [08:15] ? [08:15] popey, hey, yeah [08:15] gimme a sec [08:15] kk [08:17] k, should be publishing [08:17] i see it already on my device [08:17] 65 right? [08:17] and it's auto-downloaded already [08:17] man this is great! [08:18] what are you looking for? [08:19] popey, right, 25.2/65 was released [08:20] hey [08:20] hey seb128 [08:20] sil2100: hey, around? feeling better today? [08:20] can someone add system-settings to the landing board? or do I need to wait on Jason to be up at my end of day, [08:20] ? [08:20] didrocks, hey ;-) [08:21] we take bribes [08:21] lol [08:21] seb128: you are supposed to have acceess so you can add your own asks [08:21] seb128: i shared it with you now [08:21] asac, I didn't, let me try again [08:21] didrocks: morning, still feeling bad, but not bad enough for coding ;) [08:21] sil2100: I think we'll need you for some landing today ;) [08:21] didrocks: mir >> it's great to see that there are as many changes in code as in tests, but oh $DEITY the amount of code changes :-) can't review the ABI changes I'm afraid [08:22] asac, that works now, thanks! ;-) [08:22] didrocks: so how do you handle the package renames exactly? did you preNEW them already? [08:22] didrocks: please feel free to add me to some tasks, and I'll do those whenever possible ;) [08:22] lool: package rename? are you on the multimedia stuff? [08:22] didrocks: no, mir [08:22] oh please someone ! What does preNEW mean ??? [08:22] sil2100: please come to the HO meeting [08:22] libmirserver3 -> libmirserver4 [08:22] lool: wait for the meeting, will be easier ;) [08:23] right now, I'm trying to unblock Mir FTBFS [08:23] * lool checks PPA [08:24] * lool checks #ubuntu-mir [08:24] didrocks, lool: What does preNEW mean ? [08:24] ogra_: asac mail sent [08:24] it's an irritating blind spot ;) [08:24] popey: about what? [08:24] 65 [08:24] why do we release it with regressions? [08:25] ogra_: didrocks: ^? [08:25] yeah, we shouldn't as there is those apps test failing [08:25] gah [08:25] we dont release with regressions [08:25] (which mail was sent?) [08:25] asac, which regressions ? [08:25] notes app [08:25] popey: did you try the notes app AP tests manually? [08:25] no. [08:25] are they flacky tests or reproducible issues? [08:25] right [08:25] can you try that please? [08:25] we cant continue until we know [08:25] so that we can assess [08:26] gah, sorry [08:26] vila: preNEW is basically a 'review' that an archive admin needs to do of a new package, ending with a thumbs up from that admin [08:27] running now [08:27] vila: it's because the automated archive copying we're doing bypasses the normal processes [08:27] vila: so when we have new binary or source packages, some archive admin should review them before they get automatically in [08:28] lool, sil2100: thanks ! archive or -proposed ? [08:29] vila: they get into proposed [08:29] but not to binary new [08:29] IIRC [08:29] lool: thanks, one blind spot down, there are N blin spots on the wall... Hey ho ! [08:29] vila: normally, new sources and new binaries uploaded by a developer or a human get stuck into the archive NEW queue [08:30] but IIRC, the way we're copying them it bypasses binary NEW [08:30] lool: archive admin == ubuntu dev ? [08:30] no [08:30] archive admin is like an über-elite secret club of people that can rm -rf launchpad [08:30] j/k [08:30] asac: sorry just saw the message :), notes is a consistent failure [08:31] right [08:31] ogra_: so please try to remove the 56 build from release channel [08:31] vila: essentially a set of people with deep understanding of how archive works, and with access to special host to do maintenance tasks such as accepting packages, creating new releases etc. [08:31] lool: hmm, so, infinity, didrocks, yourself ? [08:31] lool: ok, names please ;) [08:31] I'm not archive admin [08:31] https://launchpad.net/~ubuntu-archive [08:32] puuurfect [08:32] psivaa: we have 3 failures on mako [08:32] psivaa: we have 1 failyre on maguro... doesnt sound very consitant [08:32] asac: joining? [08:32] asac, i'm a bit scared that everything explodes then [08:32] lool: much much clearer :) [08:32] ogra_: well. we have to learn [08:32] lets talk in a second [08:32] asac: yea, i was talking about the info from last night.. let me retry :) [08:32] yeah [08:32] lool: there are a few names I should be able to bride there ;) [08:33] vila: it's kind of orthogonal to developer membership, since these are different tasks, but I guess the levels of trust and understanding of archive / packages goes up from MOTU, to core dev to archive admin [08:33] lool: received loud and clear, thanks ;) [08:33] asac: ogra_ all the tests that failed on jenkins, succeeded on my device locally [08:34] yeah the notes app ones are really random [08:34] not the same across the devices [08:35] vila: can you do something about upstream merges not being processed for music-app? [08:35] vila: https://code.launchpad.net/~lool/music-app/singleton/+merge/187583 [08:36] lool: not quite right - they still land in NEW, although a bit differently (there are some problems, but I don't think those problems are why preNEW exists) [08:36] popey: did all tests succeed? [08:37] lool: AIUI the problem is that if cu2d copies something then the workflow expects that it will appear in the archive before *too* long, and things can get a bit confused if it sits there for ages [08:37] lool: so preNEW is to try to make sure that the actual NEW check doesn't take too long [08:37] asac: ran it 3 times, same result each time http://paste.ubuntu.com/6157852/ http://paste.ubuntu.com/6157871/ http://paste.ubuntu.com/6157875/ [08:37] lool: (also NEW for copies is harder to review than it ought to be - I have some LP bugs to fix) [08:37] ok [08:38] lool: not now as I would have to dig to find where somthing can be triggered without knowing if I will have to trigger so -> vila.TODO++ ^ [08:38] meh [08:38] vila: ~ubuntu-archive would be a strict subset of ~ubuntu-core-dev if we hadn't been forced to make it otherwise by past management [08:38] asac: (and different from jenkins) [08:38] vila: hopefully we'll get back to that again though [08:38] lool: not now as I would have to dig to find where something can be triggered without knowing if I will have the right access to trigger so -> vila.TODO++ ^ [08:38] popey: you have to unlock a screen [08:38] hey, actually, maybe it is a subset again now [08:38] asac: i did [08:38] (aside from the robot) [08:39] can we make the robot a core dev? [08:39] would help a lot [08:39] cjwatson: thanks, also lool's 'kind of orthogonal to developer membership' made sense to me [08:39] lool: why? [08:39] quite obviously robots are more productive than humans [08:39] :-) [08:39] they complain less [08:39] work crazy hours^W^W^W actually we do that [08:40] :-D [08:40] * vila goes dark a bit to concentrate on some stuff [08:41] asac: and again... same http://paste.ubuntu.com/6157890/ [08:41] popey: do you manually unlock the screen? [08:41] before? [08:41] you need to do that for apps [08:42] check https://wiki.ubuntu.com/Touch/Testing#Testing_your_Ubuntu_Touch_Code_before_submission [08:42]