[02:05] === trainguard: IMAGE 207 building (started: 20140825 02:05) === [03:35] === trainguard: IMAGE 207 DONE (finished: 20140825 03:35) === [03:35] === changelog: http://people.canonical.com/~ogra/touch-image-stats/207.changes === [04:45] veebers: https://code.launchpad.net/~veebers/autopilot/legacy-lp1352889/+merge/230241 not approved [04:46] Mirv: ah rats, not the first time I've done that either :-P Thanks for the heads up, sorting it now [04:46] ;) [04:47] Mirv: Have approved now, is there a button that I need to (re)push? [04:48] veebers: no buttons needed, thanks! [04:48] Mirv: nw, thanks for sorting that out [04:48] now we just need a core-dev to approve the pkg changes :S but I'll have it done today. [04:49] Mirv: awesome, thanks again :-) [07:00] sil2100: hi! Do you know why silo 15 hasn't yet landed? [07:03] mardy: there seems to have been some sort of build error according to the spreadsheet [07:03] mardy: or maybe that thing is an erronous report in itself, but that's why it's red in the spreadsheet so no-one has noticed it has actually been tested.. [07:04] mardy: I think that's a false alarm that's totally Wellark's fault :) https://ci-train.ubuntu.com/job/ubuntu-landing-015-1-build/9/console - he probably erronously tried to build a wrong silo [07:04] Mirv: yes, I've seen the error message, but I honestly didn't understand what it means :-) The silo was built and tested [07:04] mardy: I'm running 'watch only' build on it now so that the status gets refreshed correctly [07:12] Mirv: cool, that seems to have worked, thanks a lot! :-) [07:17] mardy: no problem :) but the MP:s are not approved [07:17] https://ci-train.ubuntu.com/job/ubuntu-landing-015-2-publish/5/console [07:17] or at least not top approved [07:18] dbarth: could you top-approve also those in that console log ^ ? you've already approved them in comments. [07:19] Mirv: I just top-approved them myself :-) [07:19] dbarth: unping :) [07:20] so, now only packaging acks [07:24] mardy: hey. unfortunately the libaccounts-glib needs changes. the diff https://ci-train.ubuntu.com/job/ubuntu-landing-015-2-publish/lastSuccessfulBuild/artifact/packaging_changes_libaccounts-glib_1.17+14.10.20140819.1-0ubuntu1.diff reveals that manual uploads from spring enabling multi-arch did not get merged to trunk, and they'd need to be added back (including changelog entries) :( [07:24] https://launchpad.net/ubuntu/+source/libaccounts-glib/+changelog [07:25] ubuntu2 was just a rebuild, but ubuntu3 added the two multi-arch lines. so it should be just syncing changelog and adding back those two multi-arch lines [07:25] Mirv: ah, OK, I'll try to fix that; the multi arch lines should be already in trunk, only the changelog is missing [07:26] mardy: the diff claims for some reason that the multi-arch lines are being removed in the current build [07:26] Mirv: what diff? [07:27] * mardy checks [07:27] Mirv: yes, I guess you are right; I'll add those lines [07:28] Mirv: is it important that I add the changelog entries as well? [07:28] mardy: thanks. it's a bit unfortunate this happens every now and then. [07:28] mardy: yes it is, since we don't want to erase history :) possibly easiest to copy paste from https://launchpad.net/ubuntu/+archive/primary/+files/libaccounts-glib_1.15%2B14.04.20131126.2-0ubuntu3.debian.tar.gz (open in file-roller, open debian/changelog in gedit) [07:36] Mirv: I pushed a couple of commits fixing that, can you please check? [07:37] Mirv: hi; while you're on webapps stuff, silo 2 can be freed; it's not ready for re-testing [07:38] Mirv: and line 12 is obsolete; can i just remove it from the spreadsheet? [07:48] mardy: looks perfect, thanks! [07:48] Mirv: cool! So, what is the next step? [07:49] mardy: rebuilding the libaccounts-glib only [07:49] dbarth: ok, freed. should it be marked as "Ready?" "No" also? [07:49] dbarth: you can simply remove the line 12, go ahead [07:52] mardy: I just kicked a libaccounts-glib build, then [07:54] Mirv: thanks [08:17] trainguards, can I have silos for line 39 and line 40? [08:18] tvoss: sure o/ [08:18] sil2100, thanks [08:18] tvoss: those are for utopic for now, yes? [08:21] Mirv: I see you're ssigning a silo for oSoMoN ? ;) [08:22] sil2100: yes :) [08:23] thanks guys :) [08:23] mardy: landing-015 is now built again, you maybe want to have a quick smoke test? the diff was of course only packaging and doesn't affect functionality at all. [08:29] dbarth_: What do you think? I don't see the need to re-test it ^ [08:30] Mirv: i can do a quick smoke test no worries [08:30] dbarth_: thanks! [08:31] omg, meeting [08:31] It will be a very sad meeting [08:35] hi trainguards; can i have a silo for line 29; thanks [08:53] dbarth_: utopic, yes? not rtm/14.09 [08:54] (assuming) === tvoss is now known as tvoss|test === tvoss|test is now known as tvoss [09:06] Mirv: sil2100: I do not know if anybody is actively working on this issue -> https://bugs.launchpad.net/ubuntu/+source/phablet-tools/+bug/1360582 I have not seen mvo online yet. But this bug killed the SDK, like totally. Not little, not just a bit, but ultimately. [09:06] Ubuntu bug 1360582 in phablet-tools (Ubuntu) "Can't manually install clicks "Signature verification error" since #205" [Undecided,Confirmed] [09:08] bzoltan: we discussed that on the meeting, we're waiting for mvo and or jdstrand, although it seems to be caused by mvo's landing [09:11] sil2100: OK, I will make a simple AP tests for the SDK tools what people could run before landing something what effects the SDK [09:14] dbarth_: we'll need to finish the 015 and 010 landings first before line 29. [09:19] thostr_: pstolowski: please coordinate with Saviq on whether you can build, test and publish your line 34 before his landing-017 or not [09:19] Mirv: will do [09:20] Mirv: 15 smoke tested; you can land [09:20] Mirv: for 10, that's fine; i can do a rebuild if oSoMoN lands first [09:20] dbarth_: thanks, we'll just need packaging acks still [09:25] Anyone have an idea what is causing this failure: https://jenkins.qa.ubuntu.com/job/thumbnailer-utopic-amd64-ci/18/console [09:27] Mirv, it's fine, Saviq will remove it from his entry [09:29] satoris: looks like a jenkins problem and something for cihelp, but I'm not sure what's their availability today [09:29] pstolowski: ok then [09:30] Thanks. [09:42] Mirv, mardy: whoops, sorry! [09:43] 15 and 12 are not that far of each other :) [09:44] :) [09:50] Mirv: cool, silo 15 is migrating, can we have a new silo then? ^^ ;) [09:50] remember i freed silo 2 as well, as a special bonus [10:21] dbarth_: done now [10:23] 015 not yet merge&cleaned though since it's still in proposed [10:24] Mirv: ty! === tvoss is now known as tvoss|lunch [10:58] 12:40 < ogra> heh, so nothing landed in RTM since my last upload on friday ... [10:58] sil2100: ^ [10:59] asac, it is painful ans awful for people using bzr atm [10:59] i know, doesnt change a thing though [10:59] we need that fixed before we force *everyone* into having to maintain two out of sync branches [11:00] which is something we wont be able to easily revert later [11:00] well, whatever isnt landing in stable now, isnt in stable [11:00] (and which contradicts the whole plan we initially had) [11:01] doesnt help [11:01] asac, right, because you only can land something in stable if you create a diverged branch atm [11:01] then do that [11:01] Well, *theoretically* you can already do this: [11:01] i wont, i will rather use debs now ... like most of us [11:01] (sneakily) [11:02] Fill in a landing with MR's for a ubuntu [11:02] And then below fill in the same landing for ubuntu-rtm but without any MPs, just source package names listed [11:02] then pull the source package out of the silo and re-land that in rtm [11:02] right [11:02] yeah, then do that [11:02] thats what we need [11:02] if you dont land you will not land [11:02] Yeah, troublesome (I'll try to make that much easier today) but still [11:02] and thats where we need to work towards in automation [11:03] dont hold your breath for automation. people that land in ubuntu only now are risking that their changes won't make it [11:04] -> diverging the baseline further, making live even harder for everyone who wants to land in both directions [11:04] lfie [11:05] asac, well, i think we need some wiggle room anyway ... we dont have rtm tests for krillin (well, we dont even have automated tests yet, its all manual) [11:05] so if we want to target that, there is still a good bunch of things missing [11:06] thats going to happen today [11:06] i would expect a few days til everything is fallen in place on all sides ... and while thats goinng on sil2100 can work on that extra PPA love [11:06] now that the channel is avail [11:06] ogra_: we wont get the stuff over then [11:06] thats the point. land your stuff in stable with N4 targetted then [11:06] I'll be loving it much more after lunch [11:06] we shouldnt allow folks to shoot themselves even more [11:06] asac, right, it is tricky to get anything over if you dont even have a reference image test [11:07] we have N4 and later today krillin too [11:07] right [11:08] anyway, deb landing is trivisal in both ... the bzr/MP landing needs some love first else we paint ourselves in an RTM corner with forcing two branches for each landing [11:08] we have the trick sil said [11:08] (in both -> ubuntu and rtm) [11:08] no need to be stuck [11:08] right [11:08] we need to stop now and replay whoevfer landed without landing in ubuntu-rtm [11:08] i am not stuck [11:08] but i think many people hold their feet still hoping for improvement first [11:09] thats not coming [11:09] if it comes it comes, but dont wait [11:09] i guess someone should outlin the source package step on the ML ;) [11:09] since up to now people onlykone the MP way [11:09] *only know [11:09] right, and so their changes are not landing in stable [11:10] yes ... since they dont know that process is also allowed [11:10] (see the ML discussion from the weekend) [11:10] for me those changes landed today are not meant for stable then. we certainly won't do the replaying [11:10] for them === mandel is now known as mandel|lunch === renato is now known as Guest9436 [12:14] * jdstrand notes that bug #1360582 is not his bug (I got pinged earlier) [12:14] bug 1360582 in phablet-tools (Ubuntu) "Can't manually install clicks "Signature verification error" since #205" [Undecided,Confirmed] https://launchpad.net/bugs/1360582 [12:16] jdstrand, yes, thats mvo's [12:16] but it was fixed afaik [12:16] ok cool [12:16] * jdstrand is curious how [12:16] dunno, he had me test it and later provided a new landing [12:17] (which makes me assume he attempted to fix it with tht) [12:17] ok, thanks [12:18] oh, thats the --allow-unauthenticate bit [12:18] the issue i tested for was "no clicks can be installed at all because click tries to create some dirs in /root/" [12:19] i know he fixed that one ... the one above was likely fallout of this [12:22] ogra_: that one's not really fixed; the workarount is to use click install directly which I was told we shouldn't do [12:22] how else would you sideload a click package ? [12:22] jedi waves over your screen ? [12:24] ogra_: pkcon install-local ... [12:24] oh [12:24] that, yeah [12:25] (in fact thats what i use locally in my scripts ... thats what you get using scripts and never looking what you programmed initially :P ) [12:25] that's the dream; fire and forget and work always :-) [12:26] yeah [12:26] thats what makes my dev mode changes so hard :( [12:41] can anybody reconfigure silo 12, please? [12:43] Mirv, hey is there an image to test ? [12:43] Mirv, on mako I mean [12:57] Saviq: hey! [12:58] sil2100, elo [12:59] Saviq: so, I see you have a silo ready for releasing [12:59] sil2100, actually... [12:59] sil2100, I need to take it back and push one small change and rebuild :/ [13:00] sil2100, sorry for the noise, will have it ready in ~30mins [13:00] Saviq: I was thinking... I know that you have already published something to RTM for unity8 (and did a separate unity8 branch probably), but I was thinking if maybe we could allocate you an RTM silo to which we'll do a srccopy from the ubuntu one? [13:01] Saviq: no worries :) [13:01] sil2100, yeah, let's do that [13:01] sil2100, I need to look back for dependencies, though [13:01] sil2100, I'll let you know when we're ready [13:01] Saviq: for now all manual, so I'll help you out with the copies and such, just tell me when you [13:01] are ready [13:01] sil2100, will do [13:01] (premature newline) [13:01] We'll get a silo for you then [13:02] I just need to consume my lunch and I'll be back === tvoss|lunch is now known as tvoss [13:20] om26er: which kind of mako image you'd want? :) [13:20] om26er: I think we're not considering mako promotions today at least [13:21] om26er: and there are enough unsolved blockers in the image now (apparently some new over the weekend even) [13:25] sil2100: good morning, can I get a reconfigure in silo 1, please? [13:27] Mirv, alright, I just wanted to inquire. [13:32] ralsina: I can reconfigure it [13:32] Mirv: awesome, thanks! [13:36] Saviq: i would recommend to not publish your ubuntu silo if you plan to get that into stable (using sil2100's new trick) as is as we will have QA double check for stable silo, so you might have to iterate there one more time before publishing [13:36] ... which probably means you want to iterate with devel in sync too (and stay in sync) [13:36] ralsina: uh oh, the csv output is again broken at Google, which means many functionalities including reconfigure is broken at the moment :( a manual reconfiguration worked however but hopefully google will fix itself soon [13:36] sil2100: ^ csv again broken, prepare-silo-manual still works [13:36] Mirv: oops, thanks! [13:36] asac, stable = rtm ? [13:37] yeah [13:37] asac, I don't have a devel, not sure what you mean there [13:37] * ogra_ thinks our terms get heavily overused :( [13:37] Saviq: you have a silo targetted normal utopic, no? [13:37] ogra_, do you know how to get add-apt-repository to recognize 14.09 as a distro? [13:37] sil2100: dholbach: we have not seen mvo so far today. He might be on vacation. [13:37] sil2100: unping, a couple of minutes later csv works again, thanks google [13:37] brendand, should just work on rtm installs [13:37] * asac will try to stick to long form: "ubuntu" "ubuntu-rtm" in near future [13:38] brendand, if not, file a bug :) [13:38] ogra_, well /etc/lsb-release is not correct in 14.09 so it's not [13:38] Saviq: devel i used as synonym for -> ubuntu ... stable for -> ubuntu-rtm [13:38] brendand, oh, that, yeah [13:38] ogra_, where's the best place for that to go? [13:39] asac, if you want to say that I should file separate MPs into ubuntu and ubuntu-rtm, that's not gonna happen, I don't have the time to retarget a dozen or more MPs [13:39] brendand, in lsb i guess [13:39] Saviq: sil has a trick he wanted to try with your silo that uses source copy [13:39] asac, please don't use synonyms like that unless we rename the channels as well, we have devel-proposed, devel, stable, rtm, ubuntu etc. [13:39] Saviq: but you can only use that if you havent gotten out of sync yet [13:40] asac, I don't think that's a problem [13:40] Saviq: yeah, i will use devel-propose and rtm [13:40] the rest doesnt matter now [13:40] right [13:40] rtm-proposd actually [13:40] asac, with the first srccopy landing it will get back in sync [13:41] asac, rtm-proposed doesnt exist for devs ... thats happening automatically (like uploading to ubuntu ending up in ubuntu-proposed) [13:41] ogra_, or maybe against 14.09, if possible [13:41] Saviq, the plan was some kind of auto-copy of the source packages to the rtm silo ... so you dont need two branches unless you do dircet rtm uploads [13:42] Saviq: cant say for sure - you are most likely right. the further things diverge the more stepping stones you might encounter. anyway, sil think he has a trick to do these with source copy's... would be good to try so we can announce that more widely as an option. [13:42] ogra_, yes, that's exactly what I want indeed [13:42] right [13:42] everyone does :) [13:42] "auto" is not avail, but semi-auto [13:42] ogra_: have you guys heard about that the SDK tools are broken since the #205? https://bugs.launchpad.net/ubuntu/+source/phablet-tools/+bug/1360582 The developers can not run their apps on the device. [13:42] is what we want to try NOW [13:42] Ubuntu bug 1360582 in phablet-tools (Ubuntu) "Can't manually install clicks "Signature verification error" since #205" [Undecided,Confirmed] [13:42] sil2100 is fixing :) [13:42] auto will come later if that really turns out feasible [13:42] bzoltan, yes, we need mvo to fix that [13:43] bzoltan, and like most debianners he is at debconf this week [13:43] ogra_, asac was just saying I shouldn't do that because I already have an rtm branch, which I don't see as a problem (I will just drop the rtm serie for now or just keep it manually updated) [13:43] ogra_: mvo is no around [13:43] bzoltan, right [13:43] ogra_: would this situation qualify as a reason for reverting? [13:43] bzoltan, he might be later today ... as he is (like many from UE, is in portland) [13:43] bzoltan, but it didnt land in rtm yet [13:43] Saviq: ah so you have branched and will use branches, then yes, you are not a good target to try; remember to deliver early and often though so you dont risk big issues with this [13:43] so no worries [13:44] bzoltan, shouldnt even affect you atm [13:44] ogra_: and should not [13:44] asac, no I won't [13:44] asac, use branches [13:44] well, it wont until mvo lands it there [13:44] ogra_: it does [13:44] bzoltan, how, it didnt land yet [13:44] ogra_, Saviq: the auto-mechanism will give the ability to request dual-silos (if there are silos free) that would basically build stuff for ubuntu and just with every build do the same for the RTM silo [13:44] bzoltan, note that we dont really care for ubuntu anymore [13:44] Just source-only for the RTM ones [13:44] ogra_: it is on the image since #205 [13:44] bzoltan, it isnt in image 6 [13:45] bzoltan, ignore ubuntu ... [13:45] rtm is our target [13:45] file a bug (which already happened i think) and wait for mvo to fix ubuntu and then to land in rtm [13:46] Saviq: right, then what i said was that you could try landing in rtm with source copying now; just talk to sil [13:46] ogra_: so that is the ubuntu-touch/ubuntu-rtm/14.09-proposed [13:46] bzoltan, exactly [13:46] yes [13:46] ogra_: thank you [13:46] :) [13:46] bzoltan, i agree its a bit annoying that we leave ubuntu broken now though [13:47] but there are rtm channels for all devices [13:47] broken? [13:47] asac, yep [13:47] we dont want to ignore ubuntu [13:47] asac: https://bugs.launchpad.net/ubuntu/+source/phablet-tools/+bug/1360582 [13:47] Ubuntu bug 1360582 in phablet-tools (Ubuntu) "Can't manually install clicks "Signature verification error" since #205" [Undecided,Confirmed] [13:48] bzoltan: how does this block you? [13:48] asac: since the ubuntu image 205 developers can not run their apps on the device [13:48] asac, you need to hack round it by breaking security [13:48] you can install clicks locally ... but not by using the right tools [13:48] right [13:49] so ignoring ubuntu doesnt help as we have the same on rtm i am sure [13:49] ogra_: as root and not as phablet [13:49] asac, only once that landed in rtm [13:49] which didnt happen [13:49] bzoltan, yeah [13:50] ogra_: and we do not do that :) [13:50] bzoltan, well, as a workaround you can use sudo ... surely not a long term solution :) [13:50] but if i would be a dev iÄd rather use sudo than waiting a day for a fix :) [13:51] bzoltan, i think we sadly cant roll back ... so we need mvo to fix it [13:51] ogra_: it is not about what a dev would do... it is about releasing a known to be broken and super ugly hack to LTS [13:51] the server signs the packages now [13:51] so rolling back might mmake them uninstallable [13:52] bzoltan, oh, i didnt mean you should hack your scripts ... just tell devs how to work around it manually [13:52] ogra_: I would roll back without hesitation. I respect and love mvo .. but even he should consider what effects the changes he commits have on the app development. [13:52] a fix has to happen and will eventually [13:52] bzoltan, but rolling back will break even worse [13:52] making the store unusable [13:53] at least thats what i suspect [13:53] so if w dont have this on stable [13:53] it means you cannot install apps on stable? [13:53] err [13:53] rtm :) [13:53] ogra_: it is not about working around .. our customers expect the "green triangle" run the app they edit. Most of them do not even know what adb and click tools are. [13:53] :) [13:53] * asac hits is heads [13:53] asac, can you ? try it [13:53] ogra_: well, if it works, we can back it out [13:53] no? [13:54] asac, right, someone on rtm needs to install a click from the store and verify [13:54] if that works fine we can also roll back === mandel|lunch is now known as mandel [13:54] ogra_: does it matter whether its a webapp or a native? [13:54] ogra_: let's hope that not so many developers update to the 205+ image [13:54] https://lists.ubuntu.com/archives/rtm-14.09-changes/2014-August/thread.html shows the new click definitely didnt land yet [13:54] asac, no, just a click package with the new gpg signing from the store [13:54] bzoltan: we've already had our own core apps developers hit by this [13:54] people working on our own apps [13:55] ogra_: which one can i pick? [13:55] asac: ^ [13:55] do you know? [13:55] asac, any ? === Guest9436 is now known as renatu [13:55] i installed delta [13:55] webapp [13:55] popey: yeps.. [13:55] if click doesnt bail you shoudl eb fine [13:55] and that worked on r6 [13:55] cool [13:55] well, then we should be able to roll back [13:55] can someone check that its really signed? [13:55] beuno should be able tto tell you i think [13:55] popey: do you have a better example i could validate on rtm branch to be sure that the backout will not break app install by default? [13:56] asac: a better example of what? [13:56] /me thinks all packages are auto-signed now on the store side [13:56] its just the question if click bails on that or not [13:56] popey: of an app that might not be installable with the old click that doenst check signatures [13:56] but it seems it doesnt [13:56] yeah [13:56] i could install the "delta webapp" on r6 [13:56] so we should be fine [13:56] ack [13:56] Is it about reverting click? [13:57] asac: no, it's that people developing our apps locally in qtc are having difficulty with their apps [13:57] (and hope that ,mvo shows up soon :P ) [13:57] sil2100, yes [13:57] asac: e.g. clock, calendar, weather, calculator, music etc [13:57] popey, they shouldnt even have noticed [13:57] if you get trunk and develop in qtc, then push to phone to debug [13:57] If yes then I would ne +1 if it doesn't break anything in the store, which I was afraid from the start [13:57] ogra_: why? [13:57] popey, nothing of that is in rtm [13:57] :P [13:57] means they use the wrong image [13:57] uh, hang fire a moment [13:57] right [13:57] :P [13:57] this bug is on #205, the nexus 4 image [13:57] so 1. lets back this out from devel [13:58] popey, right [13:58] 2. lets mvoe devs to rtm [13:58] which _all_ the developer use [13:58] popey, devs should target rtm since last week [13:58] popey: app devs should move to rtm now [13:58] or soonish [13:58] since the announcement that we all switch our focus to rtm [13:58] well, it would be nice if we didn't have an utterly confused messaging in that regard! [13:58] some can stay on devel of course to continue dogfooding that it doesnt break [13:58] popey, did we ? [13:58] ogra_: I can prepare the source packages with the revert - do we have certainty that a revert will not cause any new issues? [13:58] * ogra_ saw several announcements [13:58] where? [13:59] on tubuntu-phone ? [13:59] link? [14:00] [Ubuntu-phone] ANNOUNCEMENT: Landing team - RTM landings now officially open! [14:00] thats the subject [14:00] Nothing in there suggests that core apps developers should reflash their phone. [14:00] unless I missed it [14:01] well, probably asac could send a clearification mail then [14:01] as a higher entity :) [14:02] My e-mail was only from the landing perspective ;) [14:02] given none of the clicks are in the archive or via ci train, the mail only addresses individuals affected by those changes. [14:02] exactly [14:03] robru: sil2100: is there a way to use that citrain tool with rtm silos? [14:03] popey: ogra_: i will wait until the initial dust around rtm/devel fork has settled; then send a call to recommend switching channel for dogfooding and app development target baseline [14:03] ok [14:03] I didn't send out an official annoucement since back then I didn't have clarification of what the procedures are to be [14:04] popey: ogra_: in the end it shouldnt' matter much; popey could tell them that rtm branch might be better from stability point of view etc. as landings get more thorough testing there [14:04] also app devs probably never wanted to run devel-proposed btw [14:04] yeah, they totally do! [14:04] popey: they run devel-proposed? [14:04] e.g. clock depends on new / fixed features in EDS [14:04] asac, they have to [14:04] yeah, all of them. [14:04] devel is like debian stable to them [14:05] ogra_: so, I'll prepare the revert packages in case we want that, and then you can simply pull that in and dput to the archive [14:05] right, thats something we should fix. folks should only temporarily switch to-proposd [14:05] asac, how else would you verify your changes against the latest image ? [14:05] to validate that a fix in there helps [14:05] that would only work if we had way more promotions [14:05] besides that dont be on -proposed as that is wild west with reressions potentially busting you [14:05] sil2100, ok [14:05] and yes, more promotions could help, but then they could also temporarily switch to -proposed if they need sometihng now [14:06] anyhow, later [14:06] * popey gets back to holiday [14:06] asac, if the last promotion is a week old you are way to much behind to make sure your changes work against latest [14:06] and nobody switches temporary :) [14:07] ogra_: thats good feedback; folks said we wanted to allow longer periods of non-promotion s they can self organize to fix their issues [14:07] but thef act that we never had any promotion without TRAINCON-0 [14:08] well, we had a pretty bad cycle wrt promotions and traincon [14:08] that needs to become better next round [14:08] sows taht their i can self organize claim is true on a micro level, but not on a golballevel [14:08] we need someting beteween -2 and -0 [14:08] and promote even more broken with more exceptions ? [14:08] that doesnt slow down folks that are having no problems, but gives incentives for those that have problems to bring things back to green [14:08] no [14:09] our prob is that we never had a green image in utopic [14:09] part of it was an experiment [14:09] and whitelist because of time pressure to get something out [14:10] last cycle we blocked hard til it was green [14:10] yes [14:10] this cycle was experiment based on what upstreams wanted [14:10] right [14:10] i think we need something in between and then its a touch down and perfect [14:10] and i dont think there is any sane middle ground :( [14:11] i think there is :) [14:11] heh, ok [14:11] * ogra_ will trust you [14:11] there are multiple knobs to tune still [14:11] that we bounced to the other extreme this time [14:11] btw sent mail to foundations folks [14:11] great [14:11] telling them we consider backing out their click [14:12] if they have better guidance they should let us know [14:12] not sure how long to wait [14:12] maybe half a day [14:12] traincon 0 should be per project team [14:12] no reason why non related components should be blocked [14:12] sergiusens: right, we have that basically in traincon-1 ... where slow down only happens for those that are affected [14:12] and that is supposed to start sooner [14:12] and should be triggered that same day [14:13] right, we have that, just not implmeneted [14:13] I've suffered enough this cycle, more so because I don't have the staging branch [14:13] after 2 days without promotions, components with troubles would get a similar treatment as traincon-0 while the rest can continue [14:13] only if that isnt envough for a week or so we bring down the big axe [14:13] which just allows people to delay their fixes and kill the ones with package == trunk [14:14] sil2100, ok, I'm ready for publishing, what do I do? [14:14] Saviq: one moment :) [14:14] sil2100, sure, whenever you're ready [14:14] sergiusens: TRAINCON-0 is a means that blocks projects that are currently broken and makes sure (or decreases the risk of) no new regressions appearing [14:15] sergiusens: that's why TRAINCON-0 affects everyone [14:15] sil2100: right, but they say we should do -1 :) [14:15] sil2100: earlier, so we dont need to do -0 [14:15] * ogra_ didnt say that :O [14:15] sil2100: but if I have no involvement; I just get to stop working [14:15] or miss the feature freeze [14:15] i dont think we can find a sane middle ground ... (but happy to be proven wrong) [14:15] right, hence we had a middle stage where we just slow those with issues [14:15] which is -1 :) [14:15] lol [14:15] Sure, we can do that, but it won't really help in this particular case, as we're getting constantly blocked by new things [14:16] -1 doesn't protect us at all from new blockers appearing [14:16] right [14:16] sil2100: look historically and you will see that the things that cause traincon live around the same projects [14:16] and everyone gets to suffer [14:16] sil2100: we havent tried, so we can only guess with gut feeling. we can try and see [14:17] I know from experience that everytime we're not able to get a promotion it's because when we fix one blocker, suddenly another appears in a different project [14:17] sil2100: asac I also want to see a correlation with the traincon 0 causers and the projects with staging branches [14:18] sil2100: we should try to make statistics :) ... also there is a psychological component that another, earlier alert level might mean earlier peer pressure and earlier cautious behaviour. [14:18] sergiusens: that would be intersting :) [14:18] heh [14:18] you are assuming peer pressure wheer we have none [14:18] and direct uploads [14:18] thats the issue [14:18] sergiusens, asac: we're having incident reports for TRAINCON-0's, but I do not have an 'archive' of issues that are blockers [14:18] we have some imo [14:19] SInce this list would be really big [14:19] people wont put pressure on the person causing the issue [14:19] I might consider archiving those in the issue tracker though [14:19] they just lean back and moan but dont pester the peer specifically [14:19] sil2100: we have the mails and can check which issues were blockers when we announced TRRAINCON [14:19] asac: I'm not saying -1 is a bad idea, I think it's good, but I just know it won't stop us from getting to -0 [14:19] (because next time it coulld be them causing -0) [14:20] asac: right, that's true, but parsing that would be a pain ;) [14:20] sil2100: it will not stop us forever, sure. heence there is still -0. however, if we manage to not get to -0 before making a promotion 50% of time we would have onw [14:20] won [14:20] even once would be good :) [14:20] hehe [14:20] never seen us promote without -0 :/ [14:21] anyway [14:21] lets focus on RTM [14:21] Hey, we did promote many times without -0 [14:21] sil2100: asac so think about this, when in traincon 0; projects with trunk == package can't merge anymore; but projects (which caused traincon 0) can continue working as it didn't exist; so the full team doesn't focus on traincon 0 [14:21] We only started having TRAICON-0's all the time since like 1-2 months [14:22] asac, we did promote without 0 in trusty ... several times actually [14:22] which is what you want with traincon 0 [14:22] in trusty yes [14:22] in utopic --> very rare [14:22] asac, but that was because we had one green image we could refer to [14:22] certainly not in last 3 month [14:22] which we never managed in utopic [14:22] sergiusens: right, that shouldn't happen, managers should make sure that when in TRAINCON-0 everyone is working on getting things unblocked [14:22] asac: no no, in utopic as well [14:22] sergiusens: so thats true and thats why i said they shouldnt do it [14:22] I'm sick of everyone forgetting the good promotions in utopic ;p! [14:23] sergiusens: hwoever, they have also pressure to land stuff, which is when they get hurt too [14:23] * sil2100 looks for some data [14:23] sergiusens: ;) [14:23] err [14:23] sil2100: :) [14:23] sil2100: dont worry; also its not your fault anyway! [14:23] :) [14:23] yes we had an image that was almost green [14:23] except 1 failure :/ [14:23] http://ci.ubuntu.com/smokeng/utopic/touch/ some data :P [14:23] that was cool and very close [14:23] all non-green except for the ones that were identical to trusty [14:23] That doesn't mean any promotions ;) [14:23] we should make nice marker which image got promoted [14:24] so you can see easily the stream of images and promotions there [14:24] asac, http://system-image.ubuntu.com/ubuntu-touch/utopic/mako/ [14:24] there you have your list :) [14:25] ogra_: yeah, now just a nice overlay for the dashboard made out of that [14:25] asac: I have pressure too; but I don't do the staging branches as it wasn't the recommended way [14:25] asac: so it's screw the people who work by the book [14:25] asac, yeah [14:25] so for ubuntu-rtm one has to create the "rtm-14.09" branch for a project, by branching trunk at the revision last synced to ubuntu-rtm, and then make an MP for trunk -> rtm-14.09 to get things back up to date? [14:25] sergiusens, just go back to debs ... so much easier :P [14:26] dobey: we're trying out something easier [14:26] dobey, no, wait ... sil2100 is working on a better process [14:26] sil2100: maybe dobey can help trying? :) [14:26] Saviq: ok, so I'm working on this - just one question, does your landing require the new click? === boiko_ is now known as boiko [14:26] dobey, you only land your MP in utopic ... and then when creating an rtm silo your source package gets copied over and re-built [14:26] to just sync debs from ubuntu? or what/ :) [14:26] dobey: yeah, that's the plan ;) [14:27] oh, ok, sil2100 can handle that one :) pkging changes anyhow. [14:27] oh ok [14:27] dobey, only if you want rtm only changes you need an rtm branch then [14:27] Mirv: wait with Saviq's changes ;) [14:27] +1 for marking promoted images clearly at http://ci.ubuntu.com/smokeng/utopic/touch/ [14:27] ogra_: that would be excellent [14:27] yep [14:27] send flowers to sil2100 ;) [14:28] sergiusens: so you say we should work something into the rules that gives incentives for components that have trunk == archive? [14:28] sil2100: you want me to test this? :) [14:28] asac: well that is the model, right? [14:28] dobey, Saviq is already ... but you could be another datapoint [14:28] dobey: we're testing it manually on Saviq's branch now, but you can be first for the automatic one! [14:28] sergiusens: i think the model is: deliver early and often, otherwise your landing is more risky and you will have troubles landing without regressions [14:29] asac: and if it isn't I'll just dput every now and then as it would be the same [14:29] with correlary: if you needf a staging branch you might not do it right [14:29] sil2100: ok, just ping me and let me know what to do, when ready for me to test it [14:29] asac: yeah; break often and fast is also missed [14:29] sergiusens: we want to land through silos to ensure we get testplan systematically run [14:29] which means, when it breaks; it's hard to fix [14:29] ideally no breakages; but they should be easy to identify [14:29] traincon 0 just makes me create megaMRs [14:29] as I can't land anything === balloons_ is now known as balloons === alecu_ is now known as alecu === psivaa_ is now known as psivaa [14:33] sergiusens: can you tell me what makes it hard for you to land your stuff during TRAINCON-0? is it hte QA sign off step that takes too long? === cprov__ is now known as cprov [14:34] asac, it is the fact that his branch has to be identical to whats in the archive [14:34] sergiusens: by design what should happen is: you can just land your stuff as usual and if you have good quality stuff, QA will come around and sign it off and it just goes into the image [14:35] so if he wants to move forward he cant push stuff into his branch and things pile up [14:35] sergiusens: can you tell me if thats the case or if there are other roadblocks that are not in that line [14:35] i.e. the feature branches will grow [14:35] ogra_: thats what i am saying; traincon allows you to land [14:35] or you get a bigger amount of branches to merge into trunk etc [14:35] its designed to allow everything to land still except if your component has a blocker issue, then you cannot land anything that doesnt fix that issue [14:36] asac: requiring QA sign off last time took 3 to 4 days for a packaging change [14:36] ogra_: i understand that thats the observed symptom, asking him why that ends up to be the case exactly [14:36] right [14:36] sergiusens: ok, so its just that? [14:36] QA ia busy trying to help to get the blockers fixed [14:36] "just" :) [14:36] Saviq: hah, so it seems the PPA copy-package view in LP needs to be updated, as right now it doesn't list the other's distribution seriesses [14:36] so your stuff gets lower prio [14:36] Saviq: from the automatic-dputting view it's not a problem, but doing it manually is more irritating [14:36] asac: mostly; and silos get full as everything is mostly blocked on the same bottleneck [14:36] ok, so a smarter/fairer queue management and QA time allocation might help [14:37] right [14:37] asac, i personally gave up trying to land during traincon ... just to not cause more work ... [14:37] thats an outcome [14:37] right [14:37] and i know in one/two days i can land without having caused extra QA work [14:37] asac: the biggest problem we noticed during traincon is also the lack of silos [14:37] so i think we should really try the middle step whhere we only put folks with issues and risky landings (e.g. huge merges from staging branches migth often be risky) through QA sign off [14:37] but that only works if you have enough other stuff to work on and if your future features dont depend on what you try to land [14:37] asac: since things get slowed down natually, landings pile up and we can't even assign silos [14:38] and then see if we can be fairer/better in case yo uneed QA sign off [14:38] sil2100: right. [14:38] asac: right; I didn't break the image in the first place; and I do't think I have caused any blocker issue from history; but I get exta qa even though I'm careful not to break [14:38] I test on 3 to 4 devices! [14:38] I might be a special case; but why would I need QA signoff if I wouldn't outside of traincon 0 [14:39] well, [14:39] the state of highest alert wants to prevent new regressions [14:39] guys, any idea why one package from row #34 failed with "'libunity-scopes>=0.6.2' not found", if it's supposed to be in the same silo (and that dependency check passes for me locally)? [14:39] so that has to stay [14:39] sergiusens: will you guarantee that you won't cause an regression with your landing? [14:39] sil2100, ^ [14:39] what is missing is something in the middle [14:39] really [14:39] :) [14:39] sil2100: when you set testing to Yes; I expect people to guarantee that [14:39] sil2100: there have been regressions with QA Sign off [14:39] sergiusens: but they don't [14:40] from macro perspective that seems not the case [14:40] so why would you expect that [14:40] hence we needed to introduce qa sign off [14:40] in the end we all are humans [14:40] sergiusens: there have been hundrets of landings that were set to testing done and caused regressions, some of them weren't even properly tested since somethings were skipped [14:40] sil2100: asac the best way to make this easy is if any ci train changes would also go through ci train [14:40] then you would get a feel for the process and it's pain [14:42] sil2100: you need history and to track the lander that set it to yes; after a couple of red flags; they get QA signoff required always [14:42] period [14:42] this is like the anti good behavior; you follow the process and suffer due to other people [14:43] but it makes you god like ! [14:43] TRAINCON-0 is supposed to be an emergency case where we're making sure nothing bad happens, not trusting anyone - TRAINCON-0 by principle should happen rarely [14:43] :) [14:43] asac promised me that good behaviored citizents of the train would get benefits [14:43] sil2100, aham [14:43] yes, we are working on it [14:43] :) [14:43] sil2100, if we srccopy, is QA signoff blocking landing into utopic as well>? [14:43] sergiusens: problem is that all think they are good citizen if you ask them :) [14:44] so we need to be smarter, implicit etc. [14:44] thats the tricky part of adjusting these rules [14:44] asac: that can only be proven by numbers [14:44] Saviq: yes [14:44] Saviq: no no, to utopic it can go normally, only the RTM bits need to move - but well, it'll block you to proceed further if your main target is RTM [14:44] asac: I expect them to be collected [14:44] Saviq: whats the whole point. you need to go through qa for rtm [14:44] asac, uh, no [14:45] asac, utopic landings should juet be landings ... you want to have QA for the rtm one though [14:45] *just [14:45] ogra_: yes, thats what i said [14:45] * Saviq thinks we should batch RTM landings [14:45] no need to QA twice in different context [14:45] if you land your stuff in RTM you need QA [14:45] asac, I was asking whether it's blocking landing into *utopic* [14:45] asac, right, but only on the RTM silo [14:46] else you have to do it twice [14:46] Saviq: we assume that you land them in both at the same time. because of that assumption we said we wont need qa sign off for normal utopic and would also not do traincon business there [14:46] if behaviour goes too far off we have to change that [14:47] that wont work unless you rais the QA team headcount massively :) [14:47] *raise [14:47] we have put more folks in there [14:47] still, if you want every landing for both targets QAed thats like 300% more than we had before [14:47] adding a few people wont cut it [14:48] ogra_: no... we want landing in both at same time [14:48] but only the landing in the rtm silo gets QA [14:48] (up to now we only have QA signoff at all during traincon-0) [14:48] train → halt [14:48] asac, right [14:48] asac: I wouldn't block the utopic landing until RTM lands - if it fails QA sign-off for RTM then we would just reject the RTM part of the landing, request a fix for utopic then publish it for utopic and RTM again [14:48] asac, i'm just saying we dont have the resources to QA both landings [14:49] sil2100: i didnt say we block utopic landing; what i said we would like to see QA sign off on the RTM silo which hopefully will exist at same time as the utopic landing silo [14:49] ogra_: and we are not doing that [14:49] you just said worst case we'd have to :) [14:49] asac: right ;) [14:49] thats what i commented on [14:50] asac: I think what Saviq wanted to know is if his utopic part of the landing will be blocked from releasing until the RTM version of the landing gets qa sign-off [14:50] Which I think shouldn't be the case [14:50] sil2100: i dont think it needs to be forced unless it makes it hard for Saviq to iterate on the fixes tahht QA fill find and also put them into utopic [14:50] sil2100, asac, ogra_, still, that sounds to me like we need to batch ubuntu landings for QA sign off into ubuntu-rtm [14:51] because they will die trying to sign off every landing separately [14:51] Saviq: well, just land whatever you want now in obht and we see how good QA is at catching up [14:51] let me know [14:51] the better your quality delivered is the faster we can go [14:51] if QA is the bottleneck then let me know asap [14:54] jfunk: is QA ready to do silo sign off for stable? [14:54] :) [14:54] err for rtm [14:54] asac: they are in place and ready to act [14:55] ack [14:55] asac: but there was some blockers [14:55] jfunk: are those resolved? [14:55] or is it about krillin rtm images not there yet? [14:55] Saviq: ^^ go and grab those for your first landings. whatever is in, is in and doesnt need to be batched anymore I would say :P [14:59] asac, lsb info being wrong is another one [14:59] asac, i.e. you cant easily add ppas === plars changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | CI Train support: trainguards | Vanguard (general help): plars | CI Train Status: #203 promoted | CI Train Dashboard: http://bit.ly/1mDv1FS | Known issues: citrain struggles with source packages, don't WATCH_ONLY until *after* the source is built in PPA. http://youtu.be/-Rnw0D2AdYU. [15:01] ogra_: who should fix lsb? [15:01] ogra_: isnt that you? [15:02] ogra_: you cannot add ppas? [15:02] asac, i was taking a brief look but dev mode is as important [15:02] i dont understand what lsb has to do with that [15:02] ogra_: yeahh was kidding. does it really need to happen urgently? [15:02] the tool checks for the distro name in lsb-release as i understood [15:02] which tool? [15:02] add-apt-repository [15:02] and where do we use that? [15:02] sil2100: can you reconfigure silo 3 please [15:03] during testing? [15:03] to enable silos for testing [15:03] so if i install a rtm image [15:03] you shoudl be able to hack around that ... but i havent found the time to look yet [15:03] i cannot enable the rtm silos? [15:03] right [15:03] sure? [15:03] they tool looks for ubuntu. not ubuntu-rtm [15:03] ogra_: what about ubuntu-bug? [15:04] does it use that as well? [15:04] yeah, same thing [15:04] probably [15:04] sergiusens: ogra_: can you file a bug with that info? want to do a batch mail with issues for colin later today [15:04] asac, there is one [15:04] there is even a ML discussion [15:05] asac, i'll try to take a look later today [15:05] plars: could you reconfigure silo 3? [15:05] thostr_: sure [15:06] plars: ^ sil2100 already took care.. [15:06] thostr_: you need trainguards for that [15:06] ogra_: reminding you about poking plars ^ ;) [15:06] ogra_: thanks. let me know if i shall inlcude that in the batch mail for steve/colin to fix [15:06] ogra_: is there a way to workaround that we can tell landers to use? [15:06] sil2100, already done :) [15:07] asac, for sure, but i need to test it before i suggest ... after the meeting mumbo jumbo in 2h [15:07] thanks [15:08] ogra_: so, no revert? ;) [15:08] sil2100, thats about lsb [15:08] not about click [15:08] Ah, ok ;) [15:08] sil2100, for click lets try to catch mvo first, then revert and give him a chance to fix etc [15:09] (perhaps there is justa one line that fixes the issue ... so i'd like to get feedback frist) [15:21] sil2100, anything I need to do for silo 4 packaging ACK? [15:21] Saviq: no, I'll ACK it in a moment, want to push the RTM versions first, so it'll be published in ~5 mins [15:21] sil2100, ok thanks [15:23] Saviq: ok, so I got some XP points from this, let me now publish your silo [15:29] sil2100, :) [15:36] ogra_: ok, the revert package is ready - just tell when we proceed [15:39] sil2100: for silo 1 which belongs to ralsina and me, can we get an rtm silo? [15:41] sergiusens: sure, are the packages in that silo final and good? If yes, I'll assign a silo and push the src packages from it to the RTM one [15:41] sil2100: so I need to test first on regular and then on rtm? [15:41] Since I suppose both projects are RTM specific, right? [15:41] sil2100: everything we do is rtm specific [15:42] the people not rtm specific are outliers actually [15:42] That would be best, yeah... since it would be good if everything is still working on utopic as well [15:42] sergiusens: I'll prepare a silo for you and make sure your packages get there [15:42] sil2100: well, I'll test on regular ubuntu and have qa test on rtm; I can be flashing all day [15:42] sergiusens: and promise it will be automatic tomorrow [15:45] sil2100, dont give him to much, next time he will expect you to have it ready in 24h :) [15:46] ;p [15:47] The ETA would be like that, yes! [15:47] * sil2100 is an optimist [15:48] be more like mr. scott :) [15:48] "it will take three days" .... [15:48] "oh look, i made it work in two" ... [15:50] sil2100: are we suppose to be testing ubuntu silos against ubuntu image and rtm silos agains rtm images? [15:51] bfiller: yes... [15:51] ogra_: btw. did you confirm if we have ubuntu-rtm images for krillin? [15:51] sil2100, we do [15:52] sil2100, see janis mail :) [15:52] sil2100: hmnn, that's not good [15:52] Ah! I see it [15:52] means a reflash to land something in both places [15:53] bfiller: I know :/ If you have mako and krillin then I guess you can have krillin on ubuntu-rtm and mako on ubuntu [15:53] And at least test it this way [15:53] The situation is worse if you only have one of those... [15:54] But there's not much we can do here, this whole split is sadly very painful in many ways [15:54] at least for me, lately everything i do has to be tested on krillin [15:54] Since it would be nice to still have ubuntu up to shape === gatox is now known as gatox_lunch [15:54] kenvandine: right, make sure you test it on ubuntu-rtm on krillin though [15:54] bfiller, you only need QA signoff on the rtm ones [15:55] yeah... so much time flashing back and forth :/ [15:55] bfiller, test the ubuntu ones on your own ... then poke QA for rtm [15:55] ogra_: well, I think the idea is not only leaving the tests to QA, but I can understand that it might be redundant [15:55] there is no point in testing rtm yourself [15:55] since QA needs to sign it off anyway [15:56] sil2100, oh, then i misunderstood [15:56] Right, not sure what are the exact rules for ubuntu-rtm landings, but when we have TRAINCON-0 and expect QA sign-off, we actually want landers to test their stuff and QA just does a 'safety check' [15:56] ogra_: that will be quick :) [15:57] ogra_: no no, maybe you're right, I don't know ;) asac might have to shed some light on this [15:57] sil2100, well, i was called a "bad ogra" when i tzested my rtm silo myself for my last landing [15:57] so if Qa has to test it anyway, why shoudl i [15:57] i can then invest my time into testing ubuntu [15:57] Then maybe it's indeed like that, I only know what we do on TRAINCON-0 [15:57] Not sure here [15:58] asac, ^^^^^ his masters voice please [15:58] :) [15:58] whats the exact test plan [15:58] In overall the idea of the double QA sign-off was always to put an additional pair of eyeballs looking at the landing [15:58] Well, in case of RTM, I'm indeed not sure if that's needed ;) [15:58] right, but double QA just wastes time [15:58] test yourself in ubuntu, have QA test for RTM ... and we should be safe [15:59] and traincon should anyway only target RTM [15:59] Right, might be enough here, since we can more or less basically say that ubuntu and ubuntu-rtm are rather similar [15:59] ogra_: but then... [15:59] ogra_: no offense to QA but I like to test all of our releases myself [15:59] sil2100, no we cant say that [15:59] ogra_: if we only force landers to test on ubuntu, this means they basically develop for ubuntu... where the target should be ubuntu-rtm, right? [16:00] how do you know there isnt a minor toolchain change in ubuntu next week that we wont have in rtm [16:00] sil2100, well, then dont test ubuntu at all [16:00] but lets not duplicate the work for all of us [16:00] Right, but then we might end up having stuff broken in ubuntu, which will be sad [16:00] It's actually a very hard topic ;) [16:01] harps !! [16:02] in the end I personally like bfiller's original idea [16:02] landing first in RTM (and testing more there) [16:03] then syncing in Ubuntu (like we do with debian) [16:03] cant do that [16:03] ubuntu is upstream [16:03] right, that's why we sync there later [16:03] you need to land there first [16:03] like we do for debian [16:03] we dont do that usually [16:03] ogra_: that's a rule we created [16:03] only for very specific bits developed only in ubuntu [16:03] the default is the other way round [16:03] right, but here it seems we care the testing more when landing on ubuntu [16:04] while RTM is for sure the higher priority here [16:04] right [16:04] I'd just land on rtm by default and sync from rtm to ubuntu from time to time [16:04] but whats the issue with having the silos created at the same time [16:04] as we don't actually care much if ubuntu is worse than rtm atm [16:04] and then only test the one we care for [16:04] more work basically [16:04] we will have to drop non rtm image builds at some point anyway [16:04] we can have folks doing the sync from rtm to ubuntu from time to time [16:05] unless we add more builders [16:05] as we have people working on upstreaming ubuntu changes in debian [16:05] would just be faster [16:05] ogra_: right, before we get to that point we just sync everything :-) [16:05] I think the model we used to do in OEM worked very well: freeze automatically syncing ubuntu into RTM release, put current work into RTM and test there, backport changes in ubuntu at some point decoupled from RTM release [16:05] might be ok for a one time exceptio [16:05] but surely not as general rule [16:05] bfiller: yeah, would be way easier and faster [16:05] rsalveti: worked great for all of our custom OEM projects for many years [16:06] adding more work at the same time we're already suffering to find time to finish the rest of the stuff is not ideal [16:06] bfiller, except that your changes only returned to ubuntu a release later (if at all) [16:07] once we release rtm we need to switch back to ubuntu [16:07] rsalveti, but it isnt more work [16:07] we only care about rtm now anyway [16:07] ogra_: of course it is [16:07] how can it be ? [16:07] ogra_: yup, so there is overhead there but it's less of a priority imo, we are developing for the phone and not the desktop right now [16:07] even if QA on both sides, because we're kind of doing the sync at the same point we're landing rtm related stuff [16:07] you have to test once today you will have to test once in the future [16:08] why would you QA on both sides at all [16:08] ogra_: because that's the current process [16:08] land on ubuntu, qa, land on rtm, qa [16:08] while we could only be doing land on rtm, qa now :-) [16:08] rsalveti, who said so ? [16:08] and care about landing on ubuntu later on [16:08] ogra_, so should I still ask sil2100 to setup a rtm silo for my ofono changes??? Seems like the discussion is still happening...??? [16:08] ogra_: that's the process [16:08] the current process is test yourself in ubuntu and have QA sign off rtm [16:08] we're not landing on rtm without testing [16:08] and we're not landing on ubuntu without testing [16:08] yes [16:08] that's totally broken ogra_ [16:09] we *care* about rtm [16:09] huh ??? [16:09] per asacs mail [16:09] I don't want to leave QA to test my detailed ofono/urfkill cases [16:09] awe_, you need QA signoff in *any* case for the rtm silo [16:09] that's fine, but I'm going to do more in-depth testing than they are [16:09] that's why I'd just land stuff in rtm, make it easier for our devs [16:09] but well :-) [16:09] and if you know it works in ubuntu that should be relatively safe [16:10] ogra_: yes you need signoff (another slowdown) but that's in addition to develper testing (or should be) [16:10] so is RTM traincon-0 by default? [16:10] * rsalveti lunch [16:10] safe enough for QA to pick it up in rtm then [16:10] awe_, yes [16:10] sigh [16:10] awe_, well, QA needed by default [16:10] not traincon ... that might even become worse :) [16:11] so back to my original question... you mentioned that I should ask sil2100 to setup a rtm silo based on silo-020 ( which contains ofono ). Is that still a legit request? [16:11] yes [16:12] awe_, if it doesnt work yet someone who can needs to pull your silo sources and dpout for you in the rtm silo ... as interim solution [16:12] ogra_, ok [16:12] how will I know if it doesn't work? [16:15] sil will tell you :) [16:15] once we are out of that meeting i guess :) [16:16] sil2100: bug #1355130 seems to have slipped to the (promition) blocker list on Friday by accident [16:16] bug 1355130 in indicator-network (Ubuntu) "indicator-network crashing during dialer-app and default tests on smoketesting" [Low,New] https://launchpad.net/bugs/1355130 [16:16] please remove it [16:16] Wellark, but the crash is still there ? [16:17] http://ci.ubuntu.com/smokeng/utopic/touch/mako/207:20140825:20140811.1/9930/dialer_app/ [16:17] sure it is [16:17] thats the lates dialer test [16:17] it's a Low priority bug. [16:19] it is preventing a green image [16:19] which is a pretty high prio nowadays [16:21] ogra_: oh, how come? as it's explained in the bug that the issue has absolutely no side effects on any tests that are being run [16:21] sil2100: could I have your comment on this [16:22] Wellark: hey, one moment :) In a meeting [16:23] Wellark: sooo, reading up what happened [16:25] Wellark: the thing is, sure, this wasn't top-priority, but since we reported this 10 days ago we started counting the so called timer - we have a policy that every non-impacting bug that appears constantly and has no movement in it, it turns into a blocker after 7 business days [16:26] jdstrand, did you notice the new failures in the security testing ? [16:26] Wellark: so it's actually on the list for a reason :) [16:27] Wellark: this rule is irritating, but it's our way of making sure such small things like crashers or some misc autopilot failures do not get lost somewhere in development [16:27] Wellark: there is pressure on pushing further, we know, but we also have to consider quality and make sure we're crasher-free [16:27] ogra_: where? mako seems to be ok [16:27] (even in cases when it's not visible to the user) [16:28] jdstrand, on mako :) [16:28] http://ci.ubuntu.com/smokeng/utopic/touch/mako/207:20140825:20140811.1/9930/security/ [16:28] Wellark: otherwise we could end up with an end-version of our images that have like 10 crashes on every boot [16:29] Cannot install /tmp/qrt_tests/assets/com.example.am-i-confined_0.1_armhf.click: Signature verification error: debsig: Origin Signature check failed. [16:29] This deb might not be signed. [16:29] ok, I'll need to fix that [16:29] ogra_: thanks [16:29] jdstrand, i guess thats fallout of the other click issue [16:29] jdstrand, the SDK broke too with the gpg signing stuff [16:29] yeah-- I use pkcon to install packages [16:30] jdstrand, right [16:30] jdstrand, like all other tools everywhere [16:30] well, that is what we were instructed to do :) [16:30] right [16:30] but with the gpg signed clicks that doesnt work anymore [16:31] yeah [16:31] we will most likely have to revert this landing [16:31] (but it will come back) [16:31] I guess should either just warn (which is what I thought we discussed) or gain an option like click did to allow installing them [16:32] (pkcon install-local that is) [16:32] right [16:32] * ogra_ votes for the latter [16:34] jdstrand: ogra_ like android, allow unsigned clicks (an option in developer mode) [16:35] the spec says not to expose it in the gui [16:35] uh, why would you expose it at all :) [16:35] right [16:35] a command line option seems totally reasonable [16:36] yep [16:37] sadly mvo is gone this week :/ [16:43] sil2100: I kindly request that you whitelist the issue. As is stands, it's a Low priority bug and it will not get any attention (as there simply is not enough people to work on Low stuff) until this list is cleared: https://bugs.launchpad.net/~kaijanmaki/+assignedbugs [16:44] just having the issue on an email will not bump it's priority [16:44] I'm sorry, but this is the reality. [16:45] Wellark: it's not just 'having it on an e-mail'... when it's on the e-mail in the Blocker field it means it's a blocker, and if a blocker is not fixed no promotions will happen [16:45] Wellark: so the priority will have to be bumped or things will get halted [16:46] Wellark: managers will need to make sure that either this gets the right attention, or actions will have to be performed to get this whitelisted [16:47] Wellark: if you and your manager do not want to fix this blocker, you can reach out to QA and convince them, which is when we will whitelist it then [16:49] Wellark: but the rules that we have now have been set up with QA's blessing to make sure the quality of images does not degrade too much [16:49] sil2100, now i'm curious which issue :) [16:49] robru: has citrain device-upgrade been updated to point at RTM silos yet? [16:49] brendand: it's the indicator-network crasher we're seeing in autopilot - it passed the 7 day period [16:49] brendand: so it became a blocker now as there has been no attention on it [16:49] sil2100, is it one found by the soak tests? [16:50] brendand: the crashers in smoketesting, not affecting users (so counter counted days) [16:50] brendand, yes, the ever re-occuring one in dialer and messaging tests [16:51] sil2100: so what has to happen to get this whitelisted? [16:51] Wellark, why not fix it [16:51] we can not whitelist something eternally [16:52] ogra_: because there are more important things to implement. nobody asks it to be whitelisted eternally. This is not the time to artificially bump up priorities on Low bugs [16:52] (we should definitely not whitelist anything more than twice at all imho) [16:52] so I ask again, what needs to happen to get it whitelisted? [16:53] sil2100 told you above [16:53] QA needs to approve it [16:53] Wellark: go contact QA (brendand or jfunk) and try to convince them to get it whitelisted [16:53] They need to give a +1 on a recurring crasher happening at least since 10 days reproducible every time [16:53] thank you for the names. [16:53] brendand, jfunk: --^ [16:53] sil2100, i'd say it can be whitelisted if it does not impact on one of our readiness criteria [16:54] sil2100, which i have a feeling it might [16:54] brendand, jfunk: please see the backlog for 50 minutes [16:54] starting from [16:54] * ogra_ thinks we need ageneral rule about how oiften we can whitelist at all ... so things dont slip to long [16:54] 19:16 < Wellark> sil2100: bug #1355130 seems to have slipped to the (promition) blocker list on Friday by accident [16:54] bug 1355130 in indicator-network (Ubuntu) "indicator-network crashing during dialer-app and default tests on smoketesting" [Low,New] https://launchpad.net/bugs/1355130 [16:55] whitelisting two promotions curretly means the devleoper had two extra weeks to fix it for example === Ursinha is now known as Ursinha-afk [16:55] so that you have to have a fix in the third week [16:55] I can understand there are more important things to work on indeed, this is why we're not blocking on it instantly but give it a grace period [16:56] Maybe we should increase the grace period then, hm [16:56] bfiller: the trick is that the ppa:foo/bar URL scheme has no way to specify what distro you're using, so if your phone is non-rtm, then 'citrain device-upgrade' will get you the non-rtm silo, but if your phone has an RTM image flashed then you'll get the RTM silo. [16:56] c'mon, we don't have divisions or multi-developer teams working on single components [16:56] it's usually just _one_ developer who is working on multiple components [16:57] robru: that works, boiko ^^^^^^^ [16:57] robru: remember you can use ppa:foo/ubuntu-rt/bar === ubot5` is now known as ubot5 [16:57] robru: and ppa:foo/ubuntu/bar [16:57] sil2100: can you? nobody ever told me that [16:57] *ppa:foo/ubuntu-rtm/bar [16:58] forcing the priorities of that one developer will mean that other Critical stuff is not getting done [16:58] sil2100: in fact I raised this with colin and he just said "yeah, you get the RTM silo on RTM, or the ubuntu silo on ubuntu" [16:58] robru: yes! That's an old change cjwatson made, ppa:foo/bar is now an implicit alias to the correct ppa:foo/ubuntu/bar form [16:58] Wellark, slipping forever means that it is never fixed :) [16:58] robru: although you need to modify the dput.cf a bit [16:58] (if you want to push to it) [16:58] sil2100: so this means you can install a rtm silo on a non-rtm phone image? [16:58] sil2100, someone should write a mail about the dput changes you need ;) [16:59] *nudge* *nudge* *wink* *wink* [16:59] Wellark, probably best to email jfunk [16:59] robru: it should be possible! I never tried though ;) [16:59] Wellark, you can cc me as well [16:59] sure, I can fix the bug. That means: Critical RTM feature of the system-settings gets even more delayed, indicator-network Critical missing features get delayed, critical bugs fixes for real production system get delayed [17:00] the bugs will not get lost [17:00] they are in launchpad [17:00] they are on the plate [17:00] Wellark, well, friday is the final day for features ... so get it whitelsted one more time and you have next week to work on it ... [17:00] robru: didn't work actually, I have an rtm image (at least I think I do) and ran device upgrade and it gave me an ubuntu silo not the ubuntu-rtm silo [17:00] Wellark: it's a very old rule that we have and we do understand your concerns, we just want to make sure in some way that this bug does not get forgotten and left without action for a very long time [17:00] but forcing priorities to go up on Low bugs "just because" is not acceptable [17:00] again: it's in LP [17:00] it will not get abandoned [17:01] it isnt "just because" ... not every dev looks at the landing mails or smoke tests regulary [17:01] Wellark: we have many many bugs that are laying around unfixed for months and they're on LP [17:01] ogra_: no, I'm not asking for one additional week. I'm saying it will not get fixed before all of the more important bugs are dealt with [17:01] and not everyone is a good enough citizen to not forget about these bugs [17:01] bfiller: erk, really? [17:01] bfiller: I never tried it either but colin told me it should work ;-) [17:01] if it takes more than a week then that is what it is [17:02] Wellark: there's always something more important to work on, we can't let some bugs get starved because of that, we need to work like schedulers making sure everything gets attention [17:02] robru: if you look at this ppa: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu-rtm/landing-003 [17:02] robru: seems it's references this ppa:ci-train-ppa-service/landing-003 === plars changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | CI Train support: trainguards | Vanguard (general help): cihelp | CI Train Status: #203 promoted | CI Train Dashboard: http://bit.ly/1mDv1FS | Known issues: citrain struggles with source packages, don't WATCH_ONLY until *after* the source is built in PPA. http://youtu.be/-Rnw0D2AdYU. [17:03] robru: which is incorrect, but when you expand the sources.list entry on that page it looks correct [17:03] bfiller, you *must* adjust your dput.cf [17:03] ogra_: I'm not dputting.. just fetching [17:03] ah, yeah, fetching seems to have other issues [17:03] (lsb-release giving the wrong distro name would be one) [17:04] sil2100: low priority bug is a low priority bug. they have low priority for a reason and forcing them to be fixed on expence of higher priority tasks will gain us nothing [17:04] bfiller: yes exactly. if you look at the code (`cat $(which citrain)`), it runs the line "phablet-config writable-image --ppa $PPA/$SILO" and the value of $PPA/$SILO works out to "ppa:ci-train-ppa-service/landing-003" which is supposed to just Do The Right Thing because it doesn't specify which distro to use. [17:05] ogra_: so are you saying that installing from RTM silos is broken because ubuntu-rtm doesn't know that itself is not ubuntu? and my script is fine? [17:05] bfiller: anyway, try editing the line that says 'PPA="ppa:ci-train-ppa-service"' to instead say 'PPA="ppa:ci-train-ppa-service/ubuntu-rtm"' and let me know if that works for you [17:05] but if you say that I would get "one additional" week of whitelisting only to "fix it next week", then there is no point continuing this discussion. I will reply to the ML and if you decide to go to TRAINCON-0 in the end for Low priority bug then it's your decision [17:06] robru, i havent seen your script, sorry if you have any way that makes silos work i guess thats fine for now :) [17:07] ogra_: http://bazaar.launchpad.net/+branch/phablet-tools/view/head:/citrain if you're curious. As far as I can tell it should work without modification in RTM but I guess RTM is just broken [17:07] robru: trying [17:07] Wellark: as I said - I do understand your concerns and I will certainly bring those up with QA, but you have to understand our rationale as well: experience shows that bugs like these get left alone without attention for a very long time [17:07] Wellark: I'm sure QA can be convinced for a whitelist [17:08] sil2100, btw, see my conversation with mvo in -touch [17:09] Wellark: since if we don't do this, I'm afraid that we might become overwhelmed by seemingly low-priority bugs which, after pilling up, will not make our product top-quality [17:09] Wellark, dont forget that OEMs might use the test results as a base for signoff (not sure they do, but they surely could) [17:09] As a crasher is NEVER a good thing and no one can say that it's supposed to be acceptable [17:09] And if we are supposed to one day move away from manual testing and get everything automated, well, a crash during automated testing will certainly be a problem [17:10] plars: what determines what artifcats are included in a failing test case? [17:10] robru: didn't exactly work, sources.list.d ended up with this deb http://ppa.launchpad.net/ci-train-ppa-service/ubuntu-rtm/ubuntu utopic main and should have been this: deb http://ppa.launchpad.net/ci-train-ppa-service/landing-003/ubuntu-rtm 14.09 main [17:10] Wellark: which crasher are we talking about? [17:11] Wellark: but that being said, I'm not QA so I'm not the one to have a final word on how quality is measured [17:11] bfiller: bah. sounds like a bug in phablet-config then. sergiusens ! [17:11] bdmurray: pretty much anything the test drops, plus other things that we collect (such as all of /var/crash, certain system logs, etc) What are you missing? [17:11] jfunk: there's this bug: LP: #1355130 [17:11] Launchpad bug 1355130 in indicator-network (Ubuntu) "indicator-network crashing during dialer-app and default tests on smoketesting" [Low,New] https://launchpad.net/bugs/1355130 [17:11] ogra_: when they do, then it becomes Critical. unless proven otherwise, it stays Low [17:11] robru, i dont think that script will work atm since add-apt-repository will use ubuntu instead of ubuntu-.rtm or 14.09 [17:11] jfunk: it's 100% reproducible on smoketesting and didn't have any attention for 10 days now [17:11] plars: given that OOPSIDS are now logged in /var/log/upstart/whoopsie.log instead of /var/log/syslog it seems to be we should gather /var/log/upstart/whoopsie.log. [17:12] jfunk: but yeah, it's only appearing in smoketesting and does not affect users, so it might be whitelisted [17:12] bdmurray: I'll add it [17:12] plars: cool, thanks! [17:12] ogra_: but i don't call add-apt-repository, unless phablet-config does (checking) [17:12] ogra_: oh, yeah, it does. [17:13] ogra_: so it's a bug in add-apt-repository then? [17:14] robru: the above defect mentioned by Wellark, does it show up as a top crasher? [17:14] in the LRT [17:14] robru, most likely ... mvo just showed up ... i was planning to ask him about it and then i'll fix it ... most likely lsb-release just returns the wrong distro name [17:15] jfunk: I dunno, I haven't been tracking that one. sil2100 might know? [17:15] robru: nm, sorry for some reason I got you mixed with robotfuel [17:15] robotfuel: see above [17:15] no worries [17:16] ogra_: fix the --allow-unauthenticated? [17:17] mvo, no, being able to add ubuntu-rtm PPAs to the ubuntu-rtm images [17:17] mvo, does add-apt-repository use lsb-release info for that ? [17:17] or wheer does it get the distro name [17:18] ogra_: so... does it mean the revert is no longer needed? === gatox_lunch is now known as gatox [17:18] ogra_: (it's in http://people.canonical.com/~lzemczak/packaging/ if anything) [17:18] the ubuntu-rtm concept is an actual derivative distro ... not just a flavour [17:18] sil2100, see #ubuntu-touch ... but it looks liek a simple pkcon fix would work [17:20] ogra_: in case it's needed, it's there [17:20] * sil2100 goes of to prepare the e-mail [17:21] sil2100, right, thanks, but i think mvo has ideas for a proper fix instead (for click) [17:23] * sil2100 seems to turn into a bad-guy after the recent happenings [17:24] sil2100, well, let me land the developer mode and everyone will be distracted by things not working anymore ... and hate me instead [17:24] Can't wait! [17:24] its a temporary thing :) [17:24] ogra_: work faster! [17:24] haha [17:24] yeah [17:29] yay [17:33] jfunk: it does not matter if it's the most frequent crasher or not (no, it's not). The point is that it's a Low priority bug and there are major Critical ones (including the most frequent crasher) that need to be fixed first. So my question is, can I get a whitelisting for undetermined time (until the higher priority bugs are dealt with) or not. [17:34] Wellark, on what base do you say its not ? [17:36] indicator-network is by far the most crashy things in testing over the whole cycle ... we only had very few images where we didnt have any crashes from it [17:36] *thing [17:36] ogra_: it so happens that the #1 crasher is in the indicator-network [17:36] and is not this defect [17:37] Wellark: who is pressuring you to fix this? is it landing? [17:37] (i understand that the former crashes were induced by ofono phonesim etc, but still its by far the most crashy piece in testing) [17:37] jfunk, it was added to the blocker list [17:37] since it did hit the timer [17:37] ogra_: how was it on the timer list at all as a low defect? [17:38] jfunk: the fact that it's been marked as "Blocker" which will lead to TRAINCON-0 if not fixed [17:38] jfunk, its is a constantly re-occuring crasher [17:38] ogra_: as I've explained to QA, features first, crasher after that. The most frequent crasher is already Critical. [17:38] jfunk, after a certain period these get on the timer ... we dont judge prioriity in tegh landing team, just occurence [17:38] ogra_: do we have a diff between ubuntu-touch/utopic-proposed and ubuntu-touch/ubuntu-rtm/14.09-proposed anywhere? [17:39] jfunk, if y crash doesnt get worked on in a while it gets on the timer which then counts down [17:39] weird I'm seeing an old version of ubuntu-keyboard on ubuntu-rtm [17:39] was released last weds [17:39] bfiller, no, we have a -cahnges ML ... nothing landed or changed since friday [17:39] bfiller, https://lists.ubuntu.com/archives/rtm-14.09-changes/2014-August/thread.html [17:40] bfiller, last manual ubuntu -> rtm sync was on the 19th ... [17:40] everything after this is expected to be landed via the new landing process [17:40] ogra_: ok [17:40] ogra_: Wellark: I am inclined to agre with Wellark on this, maybe if the crasher is so important, then the priority of the defect should be raised? Am I missing somethign? I don't feel that we should blindly obey a process. [17:40] bfiller, we should perhaps ask cjwatson to do one final sync though [17:42] ogra_: yes, seeing that sil2100 email went out on Aug 21st and everything was branched then I would expect RTM to contain everything up until then [17:42] jfunk, it isnt about bug importantce at all ... nobody looks at bugs in the landing team in that context [17:42] ogra_: I would *love* to get the crashers fixed. could you get me a team of two or three developers? We have priorities on the bugs. When there are multiple Criticals it's up to the people responsible of the component to prioritize between them. Having a process that forces Low bugs over Criticals and blindly mandates artificial priorities on the developers and owners of any component is just unacceptable. [17:42] jfunk, it is simply about stuff not being worked on at all [17:43] jfunk, if we need a whitelisting, so be it ... by the default rules this crasher had to be put onot the blocker list after it has been around for a certain timeframe [17:43] ogra_: how did this get on the list in the first place, was it a regression? [17:43] jfunk: remember our discussions in Malta? [17:43] jfunk, we look at the test results twice a day ... [17:43] jfunk, if something new shows up we contact the dev [17:43] jfunk: we agreed that any non-impacting-users issue when it's reproducible we give a grace period of 7 days, and after it passes we set it to a blocker [17:43] ahh ok, I see now [17:44] jfunk, after a defined timeframe if there has been no work on the issue the issue gets a countdown [17:44] after this it turns into a blocker [17:44] jfunk: it happens all the time so we're just acting according to the rules we set [17:44] so this is a case of a test which was passing has started to fail [17:44] everyone hits these all the times ... [17:44] correct? [17:44] and if somebody migh think that "Wellark would have fixed this issue already if instead of bitching about it" rest assured. It's almost 9pm here (although I'm sure nobody here is counting). I'm having this conversation simply because I care of the end result and also about having a process that does not inflict additional pain on our most important assets; the people. [17:45] jfunk: not a test, it's a crasher, a crash happening all the time - it wasn't there before and it started happening since some time [17:45] jfunk, well, in this case it isnt a test failure but a crash ... [17:45] jfunk: it's a crash visible on smoketesting [17:45] jfunk, we cant do anything with these crashes ... we only see them happen, the dev who owns the componnent needs to retrace it, judge severity etc [17:45] jfunk: no tests are failing. [17:45] impact 0 [17:46] Wellark, can you prove that ? [17:46] ogra_: yes. if it has an impact it will come through errors.ubuntu.com as crasher on a production system [17:46] sorry to be an ass here ... but "impact 0" need to be claimed based on some data [17:46] and all of those have priority Critical [17:46] ogra_: It's not me [17:47] look at the bug [17:47] Wellark, and how do yu knwo it doesnt taint any testing results ? [17:47] A crasher is never expected behavior [17:47] right [17:47] If crashers are expected behaviors, then why do we even care to list their count on smoketesting? [17:47] Wellark, even through apport-collect running in the next test this could have impact on test results [17:47] If crashers are normal, then I would remove the 'Crashes' field in the dashboard, then it will not be an issue [17:47] because your CPU is hogged or whatnot [17:48] ogra_: which AP test is failing because of this? [17:48] Wellark, i dont know, but i also dont claim "impact 0" [17:48] which component is affected because of this? [17:48] well, the current data we have says "impact 0" [17:48] i'm just doubting you can reliably claim impact 0 [17:48] the current data says we have a crasher every image [17:48] ogra_: sil2100: perhaps we need to revisit the value of the process surrounding crashers in the smoke test [17:49] Sure [17:49] atm, we have a "top crasher" list [17:49] we had times where we didnt have a crasher of this componnent every image [17:49] this is not the top crasher [17:49] so somethig *is* broken now [17:49] which to me, will get the most value from time spent solving crashes [17:49] or even if it is [17:49] and we dont now if it has impact or not [17:49] jfunk: ok, then the rules we set in Malta are no longer valid? Why did we discuss this and agree to this in the first place? [17:49] Anyway, I'm all for changing the rules [17:49] sil2100: I have a feeling some things have evolved since Malta [17:50] Right, then why no one informed us about this that the QA criteria have changed? [17:50] I'm not saying we throw the "rules" away, but perhaps we look at the value we are getting from them, and the risks we run if we change them [17:51] as I said. get me a team of couple of people to crunch down the list of Criticals and High and then this will be fixed. unless that happens this Low bug will not be fixed until I myself will get to it on my list. [17:51] sil2100: no one informed you because the rules haven't changed [17:51] there's a discussion happening here [17:52] jfunk, in trusty we had the rule to only release fully green images ... that brought us a reasonable quality ... but also caused immense pain for devs not being able to land stuff ... if you look at http://ci.ubuntu.com/smokeng/utopic/touch/ you will notice how much green there is when scrolling [17:52] Well, I think I'm tired already and a bit agitated [17:52] and as an enduser and dogfooder i can only say that very much reflects the quality difference between trusty and utopic today [17:53] I'm fine for revisiting the rules, but it doesn't change the fact that we just do as per our rules that have been set, and it's not that we are doing this because we want to be irritating and slow people down [17:53] jfunk, we were very lax in utopic wrt blocking stuff for promoting images ... but we need to find some middle ground to actually get one green image *once* the timer was one approach towards that [17:54] I will send my reply to the ML. I don't think this is something that gets resolved here. [17:54] Wellark: +1 [17:54] if we start ignoreing crashers we can as well stop testing altogether ... or doing traincon states etc [17:55] at least fro the automated part [17:55] ogra_: sil2100: perhaps someone can spell out the rules which surround crashers found in the smoke tests [17:55] ogra_: sil2100: in response to Wellark's mail [17:55] jfunk: there was an announcement regarding that like a month ago [17:55] having crashers aroudn for months and nobody working on them wont get us any better quality [17:55] sil2100: great, do you have the subject handy [17:55] it would be great to read [17:55] * ogra_ guesses in one of the langing mails [17:55] I need to find the landing e-mail, as it was in the daily mail [17:55] *landing [17:56] ogra_: I see the risk being that if we allow new crashes in the smoke tests then we will accumulate debt in the form of crashes [17:56] The only thing that irritates me here is that it was some people from QA that had concerns about us ignoring crashers and AP test failures that have no impact on users [17:56] jfunk, right, that is exactly what happens atm [17:56] Since we were whitelisting them normally before [17:57] and why we had the timer setup discussed in malta [17:57] Then QA had concerns, so we started this [17:57] And now we're being pointed out as the bad people for this [17:57] so the dev has time to put an issue on low prio for a while but needs to fix it after some time [17:57] ogra_: the reality is that we don't have enough people to work on _all_ of the crashers [17:57] this was the alternative to blocking completely to retain some basic quality [17:58] Wellark: that's true [17:58] we need to prioritice them [17:58] we dont want to go back [17:58] but seemingly the loosening of the rules just makes us being shouted it [17:58] Wellark: right... in a perfect world though, I would like those low priority ones to also get some attention [17:58] sil2100: me too [17:58] *at [17:59] well, in the old ubuntu world you would have asked a community enthusiast to look at it while doing the important stuff [17:59] but these guys get rare [17:59] Wellark: so what I am starting to see from this discussion is that the crasher they are mentioning is important because we need to incrementally improve our quality, and if we let it slide, we are basically opening the gate to incrememntally accumulating new debt [17:59] Wellark: anyway, sorry for being a bit rough, in any case if jfunk or some QA representative thinks it's fine to whitelist, then we can do that without any problems [17:59] Since we did that in the past and it didn't kill us ;) [17:59] sil2100: it's ok. [18:00] * sil2100 gets back to his e-mail since he's grumpy [18:00] it's me who is constantly second guessing myself with this [18:00] ;p [18:00] I don't want to argue [18:00] but the reality me and a lot of devs face with the pressures we are having I just can't leave this be [18:01] nothing personal [18:01] sil2100, ogra_: -^ [18:03] True [18:03] There's not that much time and much work [18:07] * ogra_ hugs Wellark [18:09] sil2100: silo 1 is tested on utopic, can we get the srccopy to the rtm silo? === Ursinha-afk is now known as Ursinha [18:11] ogra_: <3 [18:11] sil2100: <3 [18:11] jfunk: <3 [18:11] * sil2100 hugs Wellark too [18:11] ;) [18:12] Anyway, all sides expressed their concerns, the arguments have been set so I think, if QA (jfunk!) gives a blessing we can whitelist it [18:13] oh fun [18:13] whats going on there ? [18:13] urk [18:13] ogra_: I ran 'archive landed requests' and now the bot is freaking out [18:13] looks like a complete spreadsheet refresh [18:13] ogra_: no idea why... queuebot is supposed to index based on landing descriptions, not line numbers. [18:14] * jfunk is in PM negotiations right now, will have word shortly [18:14] heh, tell stgraber :) [18:14] ogra_: it occurs to me that the original "index on landing description" code was leaky, since it would always add new descriptions to the dict and never clear them... maybe stgraber "fixed" it... [18:15] bfiller: you got silos 15 and 16 [18:16] robru: thanks [18:16] ralsina: request on line 36 conflicts with silo rtm-5 [18:16] bfiller: you're welcome [18:16] sergiusens: ^ [18:18] robru: you mean silo 5 of rtm ? [18:20] ralsina: yes that's what I said. although I'm confused, it looks like the thing in silo 5 is missing from the spreadsheet, I wonder if those are the same landings but just got disconnected somehow [18:20] could be! We asked sil2100 for a rtm silo earlier, maybe that's 5? [18:21] ralsina: yes! rtm silo 5 is yours :) [18:21] (didn't do the binary copy yet) [18:21] ok, so no conflict. I don't need a new silo, I just need the things copied whhen you can :-) [18:21] robru: is there a dashboard for that? [18:21] where to I click "build" [18:21] sergiusens: http://people.canonical.com/~platform/citrain_dashboard/#?distro=ubuntu-rtm&q= same dashboard grew an rtm page [18:22] sil2100: robru since this is all go and staticky; ralsina and myself are highy confident it would just work [18:22] true [18:22] a bin copy would work even, but we can tackle that another day [18:22] sergiusens: thank go for saving us from ourselves [18:23] :) [18:23] sergiusens, please no bin copies ... [18:23] nice pun [18:23] unless cjwatson asks you to :P [18:23] ogra_: now you are being cautios ;-) [18:23] heh [18:24] ogra_: doesn't matter; the package takes time to build due to the packag install; actual build takes 2 seconds ;- [18:24] ;-) [18:24] sergiusens, not cautious ... didnt you notice, i just passed the bucket to someone not here ;) [18:24] ogra_: :-P [18:24] thats not cautious, thats *clever* [18:24] * ogra_ grins [18:25] ogra_: well cjwatson was fine with a bin copy taking the necessary precautions; we did test the utopic target on an rtm channel fwiw ;-) [18:25] yeah, the general rule shouldnt be bin copies though [18:25] Bin copies can work most of the time as well! [18:25] i agree it is unlikely to cause issues [18:25] Buuuuuut [18:25] Not sure if we have the tools for that right now [18:25] LP doesn't cope well with copying anything between ubuntu and ubuntu-rtm PPAs right now [18:26] launchpad-lib should have everything you need [18:26] at least fro copying around stuff between PPAs === robru changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | CI Train support: trainguards | Vanguard (general help): cihelp | CI Train Status: #203 promoted | CI Train Dashboard: http://bit.ly/1mDv1FS | Known issues: There are vastly more silo requests than silos available. They are being assigned on a first-come, first-serve basis. Please ping trainguards if you want a silo and are here to use it right now. [18:29] sil2100: ogra_: I am ok to whitelist the defect until after FF, can you please begin the countdown anew the day after FF is complete? [18:29] Wellark: ^ [18:30] sil2100, see jfunk [18:30] jfunk, sill maintins the counters [18:30] *maintains [18:30] robru: sil2100 that packages build above is a false; https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu-rtm/landing-005 [18:30] it's empty [18:31] did you put on the right glasses ? [18:31] do I have to mention the packages? [18:31] sergiusens: sounds like somebody ran a WATCH_ONLY, it sets the status stupidly to 'packages built.' [18:31] jfunk: thanks [18:31] yeah [18:31] robru: well the job finished as well [18:31] you need to add the package names to the spreadhseet [18:31] robru: https://ci-train.ubuntu.com/job/ubuntu-rtm-landing-005-1-build/1/console [18:31] and do a watch only build after they have built [18:31] didn't do anything [18:32] We didn't push the packages yet, one moment [18:32] sergiusens: yeah, because there's nothing to do. the packages need to get dput in before the build job is run. the build job is just to acknowledge the packages were uploaded. [18:32] ogra_: they show up in the dashboard :-) [18:32] oh [18:32] sil2100: robru ah, I thought build would do that for us now :-) [18:32] sergiusens: yeah the dashboard reflects what was copied into it from the spreadsheet. [18:32] sergiusens: it will be done tomorrow! [18:32] I mean... that's the ETA for what at least ;/ === robru is now known as robru_breakfast [18:33] sil2100: oh, that's what will be done, no worries [18:33] brb [18:33] yeah [18:33] we were a bit premature with all the rtm switch [18:34] sergiusens: actually it will be working in such a way that you can fill in a landing that will be marked as 'target: dual' and you will get 2 silos assigned at once, and pressing Build on the ubuntu silo will cause source packages of your uploads be pushed to the ubuntu-rtm silo as well [18:34] sergiusens: ...more or less [18:34] sil2100: that is the dream :-) [18:35] sergiusens: the end result might be a bit different from the dream design but yeah ;p === slangase` is now known as slangasek [18:35] but close :) [18:38] Indeed! [18:41] sergiusens: I just uploaded the modified source packages, they should appear in the PPA soon - pretty soon you can do a WATCH_ONLY build [18:41] Ok, I really need to go and write that e-mail [18:47] ogra_: revert click anyway I suppose [18:49] yeah, just got a PM about thet too :) [18:50] I guess we should really be more willing to revert! ;) [18:51] well ... [18:52] sil2100, i was just saying to slangasek, reverting immediately might make us miss issues the breakage causes ... which then hit us the next time it lands [18:52] so i think a short period of breakage is actually a good thing [18:54] Right, but theoretically if the breakage has been noticed, well, we anyway have it in an image [18:54] indeed [18:54] So we can test it by installing this particular image in case we want to take a good look at it [18:54] At least... [18:54] ogra_: people can continue to test with the existing image [18:54] * slangasek nods [18:55] But true, instant revert = no, quick revert = yes ;) i.e. no longer waiting for someone to create a fix for half a day [18:55] right [18:56] i think thats what we can agree on :) [19:02] See you tomorrow everyone o/ [19:02] sil2100, wait [19:02] sil2100, where was that soource package ? [19:02] uh, what's up? Did I revert something wrong? [19:02] Ah [19:02] http://people.canonical.com/~lzemczak/packaging/ [19:02] ah, got it, enjoy your evening [19:02] (there's some old revert there as well, ignore that) [19:03] Thanks o/ [19:03] yup [19:03] i think i uploaded the older one before :) [19:03] Right, you're my main-reverter ;) [19:20] ogra_,slangasek: I disagree with reverting all of click when we can revert just one revision instead - see mail [19:20] cjwatson, to late ... [19:20] GRRRRR [19:20] already uploaded ... [19:21] tempted to immediately upload with the better revert [19:21] feel free [19:22] now incomprehensible version [19:24] Oh I can use 0.4.31.2.is.0.4.30 [19:24] Er, 0.4.31.2 [19:24] SIGH, 0.4.31.3 [19:27] what happened to silo 12? was that taken away on purpose? [19:27] ci-train dashboard still lists it for my MPs... [19:27] ogra_,slangasek: ok, more surgical revert uploaded now, https://launchpad.net/ubuntu/+source/click/0.4.31.3 [19:27] http://paste.ubuntu.com/8143369/ [19:27] cjwatson, great, thanks [19:27] heh [19:28] thats simply [19:28] *simple [19:28] the full revert deleted a facility that the error tracker is either now using or is about to rely on [19:28] so it was inappropriate IMO [19:28] well, i wouldnt have expected it to be reverted for long ... but yeah [19:29] cjwatson, do you have any idea what we need to do for add-apt-repository to work with rtm ? seems it defaults to add ubuntu and not ubuntu-rtm [19:29] ogra_: wgrant had a to-do item for that I think [19:29] oh, cool [19:29] ogra_: I think it probably just needs to be taught about something like new-style archive references [19:29] asac had asked me to check that [19:30] with a fallback to the old default [19:30] so you'd do add-apt-repository ppa:ci-train-ppa-service/ubuntu-rtm/landing-001 [19:30] it doesnt read from lsb-release, right (someone claimed that) [19:30] ? [19:30] lsb-release is irrelevant [19:30] right [19:30] well, we should change it nontheless ... but not urgent [19:30] new-style archive references are the way forward [19:31] this has all been canonicalised in Launchpad since we started the derived distros resurrection [19:31] we just have a few more things to go round and update [19:31] right, QA was a bit desparate though [19:42] cjwatson: how to we get the silos back into ci sheet? [19:42] thostr_: in case of discrepancy, the dashboard is always more correct than the spreadsheet. I can fix that soon [19:43] robru_breakfast: right. BUT, I cannot trigger a rebuild since it cannot find any references in ci sheet any more [19:43] thostr_, you mean http://bit.ly/1mDv1FS ? or the spreadsheet ? [19:43] thostr_: build should be fine. Just reconfigure will fail. One sec [19:44] ogra_: yes, that is fine... I tried a recon and that failed then... [19:44] ah [19:44] thostr_: err I have no idea, not my field [19:45] thostr_: the build failure is unrelated [19:46] robru_breakfast: it was a recon failure === Ursinha is now known as Ursinha-afk === renato is now known as Guest55515 [19:53] thostr_: it quite clearly said build failure, anyway I fixed the spreadsheet for you [19:54] thostr_: hmm, looking at the jenkins build timestamps, it does look like the recon was run since the most recent build. i guess there's a new bug that recon failures aren't updating the status. yay. [19:54] robru_breakfast: yes, it was a build failure which we fixed by now. that's why I wanted to recon/rebuild [19:55] thostr_: ok, sorry about the mixup, spreadsheet just loses status sometimes. haven't been able to track it down. it's ready to recon if you need it. [19:55] robru_breakfast: if it's the very same MP I don't need to recon, do I? [20:01] robru_breakfast: ogra_ seems bunch of packages are missing from the rtm sync; how do I make sure teams do the right thing? [20:02] robru_breakfast: ogra_ I need https://launchpad.net/ubuntu/+source/powerd and https://launchpad.net/ubuntu/+source/indicator-datetime in rtm silo 5 [20:03] can we dput that as well? [20:03] ralsina: ^^ [20:04] thostr_: no, recon is only for new mps. [20:04] robru_breakfast: ok [20:04] sergiusens: i thought sil was handling that? I can get to it shortly if not [20:05] robru_breakfast: I thought he took of for the day [20:05] robru_breakfast: in any case needs silo reconfig and those to sources added [20:05] no need to rebuild what already there thanks to dbus [20:05] sergiusens: ack will do soon === robru_breakfast is now known as robru [20:15] sergiusens: ok sorry, what should I put in rtm-5? is there an ubuntu silo to copy from or do you just want the copy from ubuntu archive? [20:16] robru: from the archive as these two packages are from last week [20:16] sergiusens: ok no worries [20:16] robru: I guess this will happen a bit until things normalize and the things that missed the sync get synced [20:18] sergiusens: ack [20:23] ralsina: sergiusens ^ don't build yet [20:24] for some reason it's taking me an absurdly long time to download those source packages. === Ursinha-afk is now known as Ursinha [20:36] sergiusens: ok sorry that took so long, it's my first time doing one of these super-manual RTM syncs. just did the upload, hopefully it was accepted. will keep an eye for that [20:37] sergiusens: ok it looks as though the upload has been accepted, source packages are just rebuilding right now. [20:38] sergiusens: unfortunately I have no clue how to track what packages are in sync and which ones are out of sync between RTM vs ubuntu, so I guess we'll just need to discover those on a case by case basis. === robru changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | CI Train support: trainguards | Vanguard (general help): cihelp | CI Train Status: #203 promoted | CI Train Dashboard: http://bit.ly/1mDv1FS | Known issues: There are vastly more silo requests than silos available. They are being assigned on a first-ping, first-serve basis. Please ping trainguards if you want a silo and are here to use it right now. [20:59] ralsina: I'm testing the packages now to see what gives [21:03] meh, will have to wait after aikido [21:03] need to updated powerd from recovery... [21:19] Weird. Slow day for silos. [21:24] ToyKeeper: everything ground to a halt with the RTM confusion. nobody knows where to land or what to do. [21:24] ToyKeeper: I'm sure I can find something for you to test. how about rtm-5? [21:24] Darn, I was hoping that would be smoothed out by today. [21:24] I have no shortage of other things to do though. :) [21:24] ToyKeeper: http://people.canonical.com/~platform/citrain_dashboard/#?distro=ubuntu-rtm&q= btw not sure if you saw, but here's the RTM dashboard. the link to switch from ubuntu to rtm is in the top left of the page. [21:25] robru: Thanks, I had not seen that yet... and it looks like there's one waiting on QA now. [21:25] I just wasn't checking in the right view. [21:29] ... and a few script mods later, I think I can probably get rtm silos installed. [21:29] ToyKeeper: yeah I initially made those as separate pages simply because it was the easier thing to do, and then I justified it by thinking "well, it would be overwhelming to display 40 silos on the same page." maybe I need to rethink that, the rtm sub-page is a little bit ghetto-ized... [21:30] ToyKeeper: yeah citrain script fails as well, apparently the bug is in add-apt-repository, I heard colin was working on a fix for that [21:31] I think it would probably be easier to number them 1 .. 40, and arbitrarily decide that 20-40 are for RTM. [21:31] Unless, of course, we intend to scale further for more branches. [21:34] ToyKeeper: yeah, that was exactly the reason that it was decided to duplicate the names. so we don't paint ourselves into a corner with the numbering. although I've wanted to increase the number of silos by at least 50% on several occaisions and it never happened. I'm afraid that the ability to add extra silos has bitrotted away [21:38] Well, with apt-add-repository out of the loop, the silo-add script runs faster. :) [21:39] Oh, oops. I need to mod my flash script too, to pull rtm images instead of utopic. [22:30] robru: I didn't say I was, no - at DebConf so a bit tricky. I thought wgrant had a work item for it [22:33] robru, hey, can silo 10 be published? [22:33] Oh, citrain actually uses add-apt-repository? [22:34] looks like the auto-updated status in the CI train spreadsheet is messed up, btw [22:34] wgrant: only indirectly. `citrain` tool (a script for end users to install silos on their devices, not used internally in the train) shells out to phablet-config, which in turn shells out to add-apt-repository. [22:35] wgrant: cjwatson it would be nice if 'add-apt-repository ppa:foo/bar' did the right thing, eg installed the ppa with the distro that matches the system. [22:35] oSoMoN: looking [22:36] oSoMoN: ugh, yeah it seems the spreadsheet has lost track of many silos. you claim silo 10 is tested? [22:36] robru, it is indeed, tested, re-tested, and then tested again [22:37] oSoMoN: ok, I'm gonna need you to test it one more time, then rebuild, then test again, then file your request in triplicate, and you'll need to include your gradeschool report cards from grades 1, 2, 3, not 4, 5, and not 6. [22:38] robru, mmm, this CI train thingy is getting a little bureaucratic lately :) [22:39] but ok, let me get back to you with a recommendation letter for my silo from my previous employer in a bit [22:39] oSoMoN: oh it's not that bad, but we are going to need those reports notarized, signed in blood, and blessed by the pope. good luck. [22:40] oSoMoN: ok, seems everything is in order here ;-) https://ci-train.ubuntu.com/job/ubuntu-landing-010-2-publish/11/console [22:40] robru, thanks :) [22:40] * oSoMoN heads to bed [22:40] oSoMoN: sweet drams! [22:40] cheers [22:44] wow [22:44] what's the status of silo 1? anybody? [22:45] stgraber wtf ^ [23:00] ... is it safe? wow [23:00] ugh [23:04] restarted queuebot, let's see if it gets confused again [23:05] stgraber: hey, is lp:queuebot in sync with the code that's actually in production? it's been doing some odd stuff lately that doesn't seem to correspond to the code I wrote [23:06] robru: just checking and yeah, it's in sync with the LP branch [23:06] *checked [23:06] stgraber: like, I ran 'archive landed requests' in the spreadsheet (which causes the rows to get moved upwards) and then queuebot re-reported a bunch of statuses. it didn't make sense, I thought we indexed the statuses by column A, not by row number. [23:06] ok, so it's still confused, turning off that plugin [23:07] done, it'll now only run the silo plugin and not the landing one [23:07] stgraber: ok, I guess I'll tinker with the code and see if I can't get it working and then submit a branch. [23:07] stgraber: thanks for stopping the flood [23:18] ralsina: sergiusens: so you have this superceded MP in your silo: https://code.launchpad.net/~sergiusens/account-polld/qtcontact/+merge/228969 not sure if you want to replace that one with the one that replaces it, or what your intentions are there... [23:18] robru: right.... forgot to update === robru is now known as robru_brb [23:19] robru: from my last status I didn't test this yet (well I did test utopic) but not rtm as powerd required me to go to recorery' [23:19] robru_brb: let me get back to you; today is going to be a looooong night [23:47] sergiusens: no worries, I'll be available for at least 2 more hours officially, and then I'll be out for dinner but reachable with some delay. === robru_brb is now known as robru === slangase` is now known as slangasek [23:59] robru: installing powerd is just a tad complicated; requires chrooting into recovery as there are overlays from krillin