=== ChanServ changed the topic of #ubuntu-uds-core-1 to: Currently no events are active in this room - http://summit.ubuntu.com/uds-1403/core-1/ - http://irclogs.ubuntu.com/2014/03/12/%23ubuntu-uds-core-1.html [13:30] ho [13:30] hohoho [13:30] :) [13:30] ogasawara: you run this track, right? [13:31] asac: yep [13:31] ogasawara: when will we boot the room? [13:31] asac: I usually start it up about 15min before [13:31] in 20 minutes early enough? [13:31] cool [13:31] asac: then I ping the url here [13:32] good [13:46] asac, didrocks: https://plus.google.com/hangouts/_/hoaevent/AP36tYejx4a57LGNk7fbW9HeUUpDvo-i-SHKLuRhKzSha-qxFpMxUA?authuser=0&hl=en [13:46] thanks ogasawara [13:47] ok lets see if this thing works here === ChanServ changed the topic of #ubuntu-uds-core-1 to: Track: Core | Landing process rules for ubuntu Touch | Url: http://summit.ubuntu.com/uds-1403/meeting/22174/core-1403-landing-process-touch/ [13:51] bfiller: https://plus.google.com/hangouts/_/hoaevent/AP36tYejx4a57LGNk7fbW9HeUUpDvo-i-SHKLuRhKzSha-qxFpMxUA?authuser=0&hl=en [13:53] hi folks [13:57] hi olli_ [14:05] Someone could drop the presentation URL into the session pad, once it's over. [14:07] karni: will get it posted afterwards [14:07] elopio: you might want to mute yourself [14:07] asac: thanks! [14:07] sorry. I'm muted now [14:07] dont want to interrupt didier [14:07] anyone who is not talking should mute themselves [14:07] in every session :) [14:07] elopio: np [14:08] hmm, cant get video here [14:09] ogra_, didier is signing Hollywood music, so it's restricted. [14:09] heh, well, it doesnt load on yourtube (nor in the embedded thing) [14:09] ogra_, I restarted firefox for it to work... [14:09] * ogra_ only gets a spinner . [14:10] sergiusens, igh ... certainly wont do that [14:10] (too many open things here, cant restart it) [14:10] didrocks, for the "fix ready" line ... [14:10] (other videos work fine ) [14:10] ogra_, pkill firefox and restart [14:10] does that include reversions? [14:10] ogra_, will restore all your tabs :-) [14:10] sergiusens, i will lose several oppen endit forms [14:11] *edit [14:11] ogra_, you are so complicated :-P [14:11] and i can play any other video on youtube, just this one doesnt load at all [14:11] ogra_, maybe try joining the hangout? [14:12] .. [14:12] rickspencer3, cant from this device ... [14:12] autopilot did for sure catch failures. [14:12] heh [14:12] ogra_: try shift-reload to reload without cache? [14:12] it also produced a lot [14:12] dobey, all tried [14:12] ogra_: i've found i have to do that for all the uds videos when the session actually starts [14:13] QUESTION: with most apps moving to autopilot and most test being against apps; is there a plan for the ci train to accommodate that? [14:13] or is the focus going to move to running autopilot on system components (osk, unity8, uitoolkit) [14:14] yes we can [14:14] asac: rick is talking! [14:14] lol [14:15] i can hear rick! [14:15] we can hear him [14:15] lol [14:15] we can hear rickspencer3 [14:15] asac, restart your hangout [14:15] asac: you need to stop talking [14:15] laaaaag! [14:15] asac: you're watching the youtube stream or something [14:15] sergiusens: well, leann also cant [14:15] yeah; the delay doesn't help :-) [14:15] know we know why asac and ogasawara are so productive [14:15] heh [14:16] :P [14:16] are they leaving 2 minutes in the future? [14:16] *living [14:16] oh i remember i installed the hangout block feature [14:16] for rick [14:16] haha :) [14:16] rickspencer3: clearly some people clicked "ignore this person" button =)))))))))) [14:16] whatever works! [14:16] It seems like a lot of the delay is in rebuilding images. But it seems like building multiple images is just a matter of CPU time. [14:17] Can we build images for "instant revert" proactively? [14:17] tedg: that's getting fixed, by moving cd building into launchpad and have a full pull of all builders which will be able to build image per each silo. [14:17] tedg: image building is only 1/4th of the time [14:17] s/pull/pool/ [14:17] the testing is what takes a long time (4-6h) [14:17] image builds take ~1h [14:17] we a) work on allowing parallel testing which will allows us to pipeline [14:17] That actual building. But let's say we built all the image combinations. So we were just choosing which one. [14:18] i.e. there's already an image without a certain silo landed. [14:18] b) later do swarm testing so it will take just 1h or even less [14:18] swarm waits for emulator [14:18] kgunn, wasn't the number of silos somethnig you mentioned in an earlier conversation [14:18] x86 emulator will make testing a lot faster [14:18] tedg: we have a bottleneck in our official image build infra [14:18] asac, By swarm you mean AP with gemgem? [14:18] ogra_ knows what that bottleneck is [14:18] do we ? [14:18] tedg: well, taking 10 phones and running one AP on each instead of sequence [14:18] as an example [14:18] olli_: yes, more specifically - dedicated silo, available all the time to a team/proj [14:19] ogra_: we cant produce multiple images in parallel, right? [14:19] people told me we wait for buildd cloud [14:19] asac, ah, right, they get queued ... [14:19] ogra_: what part of infra cant do parallel? [14:19] olli_: but this is what didrocks calls "airline" vs train ....so some x amount of infrastructure or change to it needed to accomplish that [14:19] but that part of the build only takes 20-25min [14:19] Can we "swarm test" manual testing by using the QA test tracker? [14:19] so we could raise the frequency a bit [14:19] Basically make one entry per-image like we do with daily CDs? [14:20] tedg: we could look at swarming manual testing, but problem is that with manual testing the testers need to be calibrated [14:20] so you cannot just test once in a month one test case [14:20] usually you will report confusing things etc. [14:20] our problem is really getting the AP results asap [14:21] Sure, but couldn't the calibrated testers just test the cases that are raised by the swarm? [14:21] I wonder if there is a way to make reversion faster and more reliable, but without slamming upstream trunk [14:21] rickspencer3: we revert without slamming upstream trunk [14:21] like could we keep copies of the last binaries around and put those into the image or something [14:21] rickspencer3: reverts are done in archive [14:21] ? [14:21] right, I think that's a good thing that we want to keep [14:22] but maybe we could make rollbacks easier and keep that goodness [14:22] I guess for me I'd say we want rollbacks quicker so that they can be decisive and make it so that other things can land. [14:22] in other words, can we make it so that there is nothing that is "hard to rollback" [14:22] rickspencer3: old binaries are available from librarian, but there is no guarantee they are installable when everything else moved on. Using old version binaries works for e.g. image-based upgrades. But doesn't help other silos/packages (they still see higher/broken version) or apt-get based machines. [14:23] xnox, but could we not calculate all the binaries that changes in the rollback? [14:23] i think we could do it by grapping old binaries, but it becomes pretty tricky for everything not a leave package [14:23] grepping [14:23] * ogra_ thinks we just need to become better with testing so we dont have to roll back *after* something entered [14:23] err :) you know what i mean [14:23] rickspencer3: from legal point of view, we build from the archive, as we publish matching source-code for those versions. [14:24] rickspencer3: calculation could be done, but it would need verification that it indeed doesn't regress. [14:24] that doesn't seem like a blocker, I mean, the source code is available in the commit history [14:24] so it should be easy to get the source that is published [14:24] rickspencer3: what i have seen with reverts, is that revert happened without consulting the uploader (myself), which is a no-win revert as it re-introduces fixed things. [14:24] to be clear, my implementation is maybe not good, but I think it's worth thinking about how to do rollbacks smarter [14:24] Question: are all regressions treated equally or is there some analysis of minor regression vs major feature landed and needs to be released [14:25] rickspencer3, +1 [14:25] I think we *are* doing rollbacks quite smartly, and people are asking for them to be done more quickly but less smartly ... [14:25] that's the strong impression I get [14:25] cjwatson, interesting point [14:26] For me I'd like rollback, etc, to be more automatic. Not a "someone decides how important" it's just a part of the machine. [14:26] rickspencer3: the way we do reverts, of a single package, not a full revert to previous image number, it still needs to go though full testing. As it's a new state. It has been identified to resolve /a regression/, but it's rarely verified fully to not introduce any /new regression/ as those reverts don't go through full test-plan. [14:26] the pushback always seems to be in cases when we're in the middle of doing proper root-cause analysis [14:26] more quickly but also still smart sounds good [14:26] So then things move on quickly. [14:26] cjwatson, I'm curious why that impacts the developers, I would think didr0cks could do a rollback to keep the image green while the developers keep going with their root cause anlaysis [14:26] what am I missing? [14:27] didrocks: QUESTION: why reverts don't go through silo and a full test-plan execution? [14:27] rickspencer3: I've suggested before that this could trivially be done by having a separate PPA that we copy old versions into [14:27] interesting [14:27] and having that pinned through the roof in the image build process [14:27] in the meantime, profile before you optimise [14:27] so far my suggestion has been largely ignored though *shrug* [14:27] * ogra_ sighs ... no dice for me with the video, even on another machine with different DSL connection :/ [14:27] the 5 hour automated test lag is clearly where we should focus [14:27] cjwatson, then the image does not match trunk or archive, is that a concern? [14:28] pmcgowan: I think that's fine as long as we keep the list of active rollbacks small [14:28] which should be good engineering practice anyway [14:28] QUESTIONS: one thing that Google CI supposedly does well is to relate code changes to relevant test cases automatically and then run only that subset (automated). Have we any plans on targeting testcases more precisely? [14:28] rickspencer3: re:automated lag - it takes less than 5 hours on armhf emulator, yet adding emulator to ci has been stalled it seems. [14:29] xnox, I thought you were going to add x86 emulator so that we could quickly scale out automated testing [14:30] didrocks, asac, can we come up with a metric for "risky" so that people can understand it? [14:30] rickspencer3: armhf emulator is available and it scales well on top of openstack. x86 emulator is in progress in bring up. [14:30] i.e. not assigned by people, but by an algorithm [14:30] rickspencer3: i made intial proposal to add amr emulato to jenkins / ci, and someone from #ci was working on it, but no updates for a few weeks now. [14:31] s/amr emulato/arm emulator/ [14:31] rickspencer3, when you want to talk mute asac and unmute when done; should provide better sync and collision avoidance :-) [14:31] :) [14:31] didrocks: QUESTION: there was full-train stop, which did delay python2-removal for example. [14:31] just like irl [14:31] duct tape ? [14:32] ah rick is talking [14:32] hehe [14:32] asac: yeah :) [14:32] asac: will tell you here [14:33] (I hope someone can relay the question in the hangout, I've my slides focused) [14:33] for me the definition of risky landing could be: if your testplan is affected by current regressions, you are at risk of landing more regressions [14:33] that you dont see [14:33] I don't think it does. I think we need a way to ensure that "risky" is transparent. If you can't measure it, it really can't be part of the determination. [14:33] so its not so much where in the stack you are, but its kind of correlated [14:33] [14:34] tedg: is that a comment on my proposal of risk above? [14:34] measure> yeah, absolutely; I get the sense that the "risky" category is kind of prejudice / based on something that happened months ago [14:34] feel thats pretty measurable. you have a risky landing if your component testplan is currently not 100% green [14:34] sometimes, anyway [14:34] rickspencer3, Isn't that really for images that get released to users. For instance, we're not going to do a daily release to all users. I think we need a different level for daily and to-user releases. [14:34] often enough, yeah [14:34] i think with this it will usually correlate on where you are on the stack [14:34] but it wouldnt be just "you are low, so you are risky" [14:35] asac, I'm not sure what you mean by "component testplan" [14:35] tedg, the testplan for the component you maintain [14:35] tedg: every component like mir has a testplan. thats manual checks + a set of autopilots that folks run in the silo [14:35] Yes, but how is that "green or not" ? [14:35] tedg, either your change passes or not [14:35] I think even in normal states, it's difficult because nothing can merge to trunk [14:35] We have no place that it's getting recorded. It's "green" in someone's head. [14:35] tedg: if you can run all the manual checks and autopilots tha tyou need for your landing in the current image and it is 100% succeess [14:35] its green [14:36] trunk should not be the same as the archive [14:36] tedg: no its recorded [14:36] the manual part not. but the APs [14:36] well, the manual part is kind of as well [14:36] right. [14:36] by you setting the silo stuff [14:36] asac, So if all the APs for a component is green, you consider it non-risky? [14:36] but we haven't figured how to ensure people that have testplans know whether their manual testplan is currently affected [14:36] if you say "tests passed" we have to assume your manual tests worked [14:37] tedg: all APs that you need to run for a component landing ... which is more than just the AP for that component [14:37] under a normal process, you would have a release branch, a development branch, and feature branches. Features land into development, and occasionally you would push those then to a release branch [14:37] tedg: e.g. if you land mir we also run a bunch of app APs, unity8 AP etc. [14:37] tedg: so the APs in the component testplan being green == non-risky in the sense of phonecon1 [14:37] asac, How is that list generate? [14:37] generated? [14:37] tedg, you as the developer define it [14:37] tedg: we start with a good initial guess out of experience ... [14:38] tedg: and if we identify that something slipped through, we grow that list [14:38] tedg: we try to not start with the complete list of reverse dependencies to stay efficient [14:38] kgunn, isn't that how Mir is working atm? [14:38] you have to provide an initial testplan when getting the silo [14:38] asac, Where is it recorded? [14:38] tedg: supposed to be recorded in the testplan [14:38] tedg, on the landing sheets [14:38] .. [14:38] tedg: https://wiki.ubuntu.com/Process/Merges/TestPlan/ [14:39] * asac is brave and reconnects [14:39] a year and a bit ago we had the concept of a release commit which only then would push to our archive (which was a PPA at that time) [14:39] e.g. with ubuntu-ui-toolkit we tried to land 17 branches in one go, which ofcourse, failed to merge and 4 branches did conflict between themself. [14:39] tedg, when you request a silo, you have to link your testplan ... when you did the testing, you have to note in the silo spreadsheet that it passed (or not) [14:39] so people could merge to trunk and _release_ when it's ready [14:39] \o/ i can hear rick :) [14:39] and therefore was pain to land, as there was no sensible conflict resolution. 4 branches got kicked out, and eseentially landed one by one, weeks later. [14:40] rickspencer3: anything that provides an API to anything else has a clear incentive to get things into the archive so that they can land changes that depend on new APIs, IME [14:40] So, I'm confused. So for every image, QA takes the list of every silo landing and generates a list of APs to run. [14:40] tedg, no [14:40] Where is the list of AP tests that'll be run on that image? [14:40] rickspencer3: (e.g. I had a strong incentive to get libclick landed so that I could start landing the things that depend on that) [14:40] tedg, for your very own changes you use the silo ... [14:40] tedg, you do your testing, if your testing passes you can land it in the archive [14:41] cjwatson, yeah, that's when stopping the line is the most burden [14:41] .. [14:41] tedg, images are built out of sync here [14:41] right, it's terrible for anything with multi-level stacking [14:41] Or 100's of MR's queued... [14:41] tedg, and the images run *all* AP tests automatically before we consider them releasable [14:42] olli_: yes - mir works this way today on lp:mir/devel [14:42] and lp:mir [14:42] tedg, so if at that point a regression is found in the component you landed, we roll it back (or as you to) [14:42] *ask [14:42] I'm confused. So project X is risky if any run of that it declares in its test plan has failed. [14:42] all projects are risky [14:42] right [14:43] So how do I determine if my project is risky? [14:43] the issue when your testplan passes but the image one doesnt is most likely that your testplan misses something [14:43] tedg: if it has code, it's risky :) [14:43] tedg: I think there's some talking past each other here; some people seem to be talking about risky projects, others about risky landings [14:43] tedg's project are always risky [14:43] tedg, I think that the landing team tells you if it's risky [14:43] those are obviously quite different things [14:43] and what olli says ;) [14:43] * ogra_ would like to get away from that "risky/non risky" stuff [14:43] rickspencer3: but that should be based on science [14:43] and vaguely predictable [14:43] the issue we have is that the first level testing isnt good enough very often [14:44] cjwatson, indeed, I was subtly pointing out a potential problem [14:44] to mee, it just seems like CI needs to be more async [14:44] if that is 100% reliable, you wont have to roll back ever [14:44] I think it should be predictable, but I'd settle for transparent at this point :-) [14:44] rickspencer3: *nod* [14:44] I do wonder if there is some kind of rdepends that we can run that predicts such things [14:44] like, a certain weight of things that depend on you equals risk level or something [14:44] not really, since we are working image based [14:45] more async and more automation [14:45] one of the probs is that you test on the last image that was built... the stuff that you land lands in the next image that gets built ... [14:46] which means you have other changes that go in alongside .... [14:46] which can change the whole situation amnd stability [14:46] Sure, but that doesn't effect how risky the project itself is. [14:46] .. [14:46] ideally we'd have integration branch for landing, ahead of the landing. [14:47] tedg: ideally, every package/project would be treated as equally risky [14:47] tedg, no, but you have to asses the risk always in combination with all the other changes [14:47] I agree with Saviq and want to add that merging a component of yours that depends on changes on many other projects (like mir or unity8) you sort of block everyone [14:47] it's silo-cked [14:47] heh [14:47] and you eliminate the thinking about whether something is risky or not, because everything is [14:47] lol [14:47] the problem with this is that it puts massive friction on improving our core [14:47] "silo-cked" word of the day [14:47] now, it also puts massive friction on breaking our core [14:47] sergiusens: it'd still be an issue if we had devel/release branches [14:47] so I get the trade-off, but still [14:48] would not block trunk, but the issue would still be around [14:48] rsalveti, oh, I'm not stating solutions today; just wanted ane xcuse to say silo-cked :-) [14:48] cf. the gigantic project of Qt 5.2 [14:48] sergiusens: and then the silo-ns will nuke us all [14:48] ++ [14:49] maybe risky = time since landing + weight of rdepends + history of regressions caused [14:50] it was rsalveti :-P [14:50] ? [14:50] oops [14:50] sorry sergiusens and rsalveti :) [14:50] np [14:50] yeah, but rsalveti said "i dont want my team to do that" :) [14:50] another guy from south america :P [14:50] we ar the same team and have the same opinion [14:50] rickspencer3: also I think some judgement of the actual changes should be involved [14:50] size of patch? [14:50] it's been this way for us since we started this project [14:50] the stuff that's already in the image, well, we've accepted the risk [14:50] 2 other "small" things i would ask for today - self service reconfigure & not locking silo's on project use (only lock landing) [14:50] maybe risky = time since landing + weight of rdepends + history of regressions caused + size of patch? [14:51] asac: ^^ [14:51] but there are some corners where we might consider flag days perhaps [14:51] asac: because we want to share the responsibility, and we can't afford having someone coordinate trunk/release merges [14:51] as much as the android bump was tested; it wasn't enough; and qt 5.2 will have a similar problem most likely [14:51] sergiusens, yup, exactly [14:52] and that's not necessarily representing failure conditions [14:52] sergiusens: risk-aversion is a continuum - it doesn't IMO make sense to crank the dial all the way to one side [14:52] some things are just tough [14:52] sergiusens, thats what i'm saying all the time above ... we need to get our pre-testing better [14:52] the problem is that any time something goes wrong people ask "how could we have avoided this" [14:52] which is healthy in one sense [14:52] cjwatson, well, we have the promotion as the ultimate safe gaurd [14:52] but it means it's very hard to say "sometimes, it just wouldn't have been worth the degree of friction required to catch this" [14:52] managers enforcing releases once a week> that means the managers have to have eyes on all the individual components that their team has touched, right? [14:52] so, it seems to me that the "risk" really comes down to a lag between promoted images [14:52] rickspencer3, but we most of the time only discover the issues at that point [14:52] which is what causes us to roll back [14:53] We could have sync points where everyone makes sure to have a release. [14:53] i would land MPs in trunk as fast as possible, and automate daily releases and testing [14:53] also, what if one team is ready to do their weekly release, and it needs a change to a package from another team, and the other team has staged changes that are not yet releasable? [14:53] we need to get better (a lot) on the first test run and with the first test plan in the line [14:53] rickspencer3: i'm ok for integration branches, but it shouldn't be an open-ended branches, but e.g. "next-landing" branch which would be limited to what is desired to land, not something open ended. Once it's full, it must land before merging more stuff into it. [14:53] ogra_, right, I guess I'm saying that just because we do a landing and we find issues, it's not a freak out situation [14:53] tedg: we could have them every six months :) [14:53] ogra_, I think we all agree with that [14:53] cjwatson, I was going to propose some interim ones, we could call them alpha/beta ;-) [14:53] once we get the x86 emulator we'll at least the get the test results sooner [14:53] tedg: heresy [14:53] as I say, it seems that we are good at keeping the impact to us as developers [14:53] and that promotion protects users [14:53] and another issue is that we can't yet give a custom image to be tested like the ones we have in the dashboard [14:53] rsalveti, but the x86 emulator will test x86 binaries [14:54] most of the issues are arch-indep [14:54] we will only see over time how reliable that even is [14:54] ogra_: it's still catch a lot of things [14:54] *it'll [14:54] sure [14:54] the click problems over the last couple of days would absolutely have been caught by that [14:54] for instance [14:54] yep [14:54] bfiller: +1 [14:54] like, if I want to land hybris or android, I'd like to be able to create a custom image and give it to be automatically tested [14:54] but i assume there will still be many false positives [14:55] it'll get better over time [14:55] didrocks, asac: gotta start wrapping up... [14:55] didrocks, asac: as I'll need to shut down this hangout in a few mins [14:55] rsalveti, use rootstock until cjwatson's PPA builds start working then === ChanServ changed the topic of #ubuntu-uds-core-1 to: Track: Core | Networking Health-Check | Url: http://summit.ubuntu.com/uds-1403/meeting/22215/core-1403-networking-healthcheck/ [14:55] I think it's important that we keep branches to releases for maintenance. [14:55] ogasawara: ok, got it, just have 30s to say [14:55] cyphermox: will get your hangout up as soon as this one is done [14:55] didrocks: ack [14:55] ogra_: my issue is not creating the image, but using that to be automatically tested :-) [14:55] ogasawara: ah, ok [14:56] We are supporting those releases and need to base off of those. [14:56] * stgraber waves [14:56] But we use symbolic naming in LP for that. [14:56] so later on we can have landing silo ppa -> image -> dashboard (tested with the x86 emulator) [14:56] rsalveti, well, doanac is working on a "home use utah" for that [14:56] lp:foo goes to lp:~team/foo/trunk.14.04 [14:56] ogra_: right, but that's a feature I asked before we started trusty :-) [14:56] heh [14:56] tedg: that's a bit annoying though [14:57] ogra_, I think that utah is mostly done; at least my tools plan is to ask him to test against that :-P [14:57] you can often create the maintenance branches for older series lazily [14:57] lots of projects never need them [14:57] sergiusens, well, you dont really want to run systemsettle so testing locally will only take 1h instead of 3-4 [14:57] Sure, I find that's easy for us, but people who are more drive-by have a more difficult time finding those points. [14:58] merging to trunk and releasing to archive/image being separate, but closely knit, seems like the best solution to me [14:58] rickspencer3++ [14:58] rickspencer3, +1 [14:58] ogra_, oh, these take 4h so I might be in full blown mode ;-) [14:58] sergiusens, yeah, its awful [14:58] cjwatson: lp:project/stable [14:58] does anyone notice the artwork behind didrocks [14:58] (not need to encode the codename) [14:58] those are by his wife, julie, she's an amazing artist [14:58] rickspencer3: i have yeah [14:58] :) [14:59] rickspencer3, she has an afro haricut ? [14:59] haha, indeed [14:59] oh ... "by his" [14:59] :) [14:59] yeah, the "by" is an important word there ;) [14:59] next sessions are starting fwiw [14:59] * didrocks move the camera to make more ad :p [15:00] the session ended 5 minutes ago guys :) [15:00] haha, you need to put prices on [15:00] should we schedule a followup session ? [15:00] bfiller, asac for automerge; the auto tests were ran [15:00] cyphermox: sorry for the delay, I'm proding these guys to wrap up [15:01] ogasawara: I m following that session no worries [15:01] mine will be short and sweet I think [15:01] maybe do it on the phone mailing list or something? [15:01] sergiusens: i know :) [15:01] rickspencer3: sure. [15:01] just reconfirming [15:01] thanks guys [15:01] thanks [15:01] was a good session ... and i was even able to hear rickspencer3 at some point :) [15:02]