=== infinity_ is now known as infinity === seb128_ is now known as seb128 [08:55] vila, the link /etc/apparmor.d/disable/usr.bin.rsyslogd is part a the default setup of the system not specific to our setup or anything required by otto. [08:55] s/a/of [08:56] jibel: wow.... ok thanks [08:57] vila, also, I think I'll drop swap control from the default config file and add it dynamically only if swapaccount is enabled. [08:57] vila, unless you beat me to it and send me a patch of course:) [08:58] jibel: hehe, ack, probably not, I'm still un-piling a bunch of other stuff ;) [09:00] jibel: out of curisosity, what should have I done to track where /etc/apparmor.d/disable/usr.bin.rsyslogd was coming from, 'dpkg -S' doesn't help there [09:00] jibel: in the mean time, removed from the doc and pushed [09:07] vila, it must be created by rsyslog itself, check dpkg *inst scripts of rsyslog. [09:08] jibel: ack [09:08] vila, ⟫ grep apparmor.d/disable /var/lib/dpkg/info/rsyslog.*inst [09:08] /var/lib/dpkg/info/rsyslog.preinst: APP_DISABLE="/etc/apparmor.d/disable/usr.sbin.rsyslogd" [09:09] jibel: excellent, thanks ! [09:10] anyhow it is irrelevant to the configuration of otto. [09:10] jibel: yup, fixed in the mp [09:26] rhaaaa, damn it vila, you updated that doc to capture knowledge and failed to add the knowledge acquired during the review ! [09:28] jibel: sorry about that https://code.launchpad.net/~vila/otto/update-setup-instructions/+merge/193201 [09:57] ogra_: we spinning another build today? [09:57] popey, waiting for bug 1245958 to be fixed i fear [09:58] bug 1245958 in unity-mir "Apps crash with image 8" [Critical,New] https://launchpad.net/bugs/1245958 [09:58] ok [09:58] nobody assigned [10:01] i havent herad from the Mir team, i think alan_g and kdub wanted to look into it though [10:01] * ogra_ hopes that gets fixed asap ... we couldnt release an image since friday [10:01] and i fear rolling back the whole stack is as much work as fixing it :( [10:03] ogra_: I *guess* that what is needed is https://code.launchpad.net/~afrantzis/mir/remove-client-rpc-timeout/+merge/193094 - but I'm still setting up phone to test. [10:06] balloons / fginther any idea what's going wrong here? http://91.189.93.70:8080/job/generic-mediumtests/1247/? (from https://code.launchpad.net/~vthompson/music-app/fixes-1234990/+merge/192771 ) [10:06] alan_g, awesome, good to know someone looks into it, thans a lot ! [10:06] +k === alf_ is now known as alf|upgrade === alan_g is now known as alan_g|afk === alan_g|afk is now known as alan_g === alf|upgrade is now known as alf_ [12:52] popey, morning [12:52] popey, the last run for that MP passed (30 minutes ago) [12:56] thanks fginther, can we prepare a click package soon for music app sergiusens ? I'd like to do some tesing before we upload to the store [12:56] popey, sure, the app should be autogenerated still [12:57] popey, fginther now, the problem with all click apps is that they need to be tested on stable AND on devel-proposed [12:58] i can do that [12:58] i have two devices setup exactly like that [12:58] (manual, not automated testing that is) [12:59] sergiusens, hmm [12:59] fginther, that's before publishing to the store [13:00] sergiusens, stable == saucy right? [13:00] fginther, would be good to start hooking up the apps in http://10.97.0.26:8080/view/click/? to some sort of automated testing and do auto pushes; it would be a daily release extension I guess [13:00] fginther, today, yes [13:01] fginther, I'm not sure how you are going to deal with the non backwards compatible autopilot 1.4/1.3 though [13:01] sergiusens, then we have a problem... autopilot tests... [13:02] beat me to it [13:03] fginther, I'll leave that issue to QA; tbh I thought api breakages were a thing of the past [13:03] sergiusens, does this also apply to internal apps? [13:04] notes, webbrowser, etc [13:04] fginther, yes [13:04] fginther, since click apps are exposed to devices by the framework they support [13:08] sergiusens, https://bugs.launchpad.net/ubuntu-ci-services-itself/+bug/1246299 [13:08] Ubuntu bug 1246299 in Ubuntu CI Services "Need to test apps as click packages on touch devices" [Critical,Triaged] [13:08] fginther, webbrowser I'm particularly not sure of it's path, but notes, camera and gallery for sure [13:09] fginther, subscribed, thanks [13:14] are there new known issues with jenkins? Since yesterday or so CI is failing our MRs again [13:14] for example, this one https://code.launchpad.net/~kalikiana/ubuntu-ui-toolkit/aviator/+merge/193219 [13:16] t1mp: not that I know of [13:18] t1mp, there is a unity8 and qmlscene crash on that mako run [13:19] fginther: what does that mean, that unity8 is broken and we need to wait for a fix? [13:20] t1mp, unity8 crash is probably there from the failed unlock attempt. It eventually succeeded and the first few tests passed... [13:20] t1mp, but I've never seen a qmlscene crash. looking for something in the logs that might indicate anything [13:21] Mir is crashy in trusty atm ... (or rather apps are due to Mir) [13:21] see bug 1245958 [13:21] bug 1245958 in unity-mir "Apps crash with image 8" [Critical,Confirmed] https://launchpad.net/bugs/1245958 [13:21] kalikiana: ^ fyi [13:21] not sure how much impact this can have on MP tests [13:23] ogra_, a lot if the MR test runner installs trusty-proposed [13:24] yeah, i was suspecting that [13:24] ogra_, fginther would temprarily switching to surfaceflinger unblock people? [13:24] sergiusens, ogra_, t1mp, MPs are tested with trusty-proposed [13:24] not so sure ... the behavior will be different (i.e. you might have other issues) [13:24] but probably worth a try [13:25] fginther, ogra_ either that or set the phablet-flash to flash -r -2 [13:25] or -r 6 (7 and 8 being the broken ones) [13:26] why wasn't it noticed before? [13:27] how did it pass despite you sounding as if it's obviously crashy as known? [13:27] kalikiana, it doesnt happen immediately ... so if you are lucky all manual AP tests pass ... and even testing it manually works if you dont test for more than 10min [13:27] sergiusens, here's a better question, should apps even be tested with -proposed? [13:28] (testing it manually as in "using it" ) [13:29] hmm another memory issue? there was the one about not releasing closed apps that took n test runs to break [13:29] there is a proposed fix on the bug [13:29] sergiusens, just use -proposed for unity8, uitk and other plumbing components? [13:30] kalikiana, this one is rather "you use the app and watch it disapper underneath your fingers while using it" [13:31] fginther, well apps should work everywhere given they have no dependencies [13:31] fginther, it's a tough question [13:31] fginther, utah already tests them on proposed; maybe would be good to test the apps on non proposed [13:32] sergiusens, my thought is to give the apps a more stable environment, keep the plumbing in -proposed. We could also add some apps to the uitk and unity8 tests to make sure they continue to work [13:33] sergiusens, adding more test suites to uitk and unity8 has been the plan for a while, we just have to solve the (lack of) resource issue [13:34] first we need ci to be reliable otherwise it's hard to get better coverage in [13:35] a number of planned things never made it [13:35] kalikiana, right, if the tests aren't stable then adding more will just cause more issues [13:36] fginther: basically improvements in the uitk are being discussed all the time among the team - but we struggle to get anything in at all :-( [14:21] so, if a new version of unity is proposed for a merge (or in the image), can it be tested with UITK trunk (should be known stable, right?) autopilot tests before merging? [14:22] like that our MRs would be tested with an image that should be stable [14:23] +1000 [14:27] do any ci-eng people have time to help with reviewing branches for tarmac? [14:27] fginther: ^^ would you maybe? [14:28] dobey: I could take a look [14:28] dobey, I would like to look as well [14:30] cjohnston, fginther: thanks. added both of you to ~tarmac-devs team. [14:30] https://code.launchpad.net/~dobey/tarmac/simple-setup/+merge/193147 [14:30] and https://code.launchpad.net/~dobey/tarmac/bzr-export/+merge/193130 [14:30] those could use a couple reviews right now [14:33] t1mp, that is something that can be added, but we have to build this up gradually. The last time we added additional test suites, we overtaxed our resources and created a huge wait queue [14:46] fginther: on the other hand, we now have a large amount of MRs approved for UITK, but we know that they will fail CI/autolanding. So actually it is a waste of resources to execute the tests for those right now [14:46] s/large amount of MRs/some MRs [14:47] t1mp, we can disable the autolanding job until we have a new image [14:48] fginther: yes, okay. [14:48] t1mp, since there is other test feedback provided by the other non-touch jobs I would recommend keeping ci going [14:48] kalikiana: ^ that's what happened to your MR. autolanding failed. [14:49] executing all the tests first with the last-known stable image, and then with the proposed one, may be helpful in the future [14:50] but I don't know if it is feasible considering the resources, and maybe a smarter solution can be thought of [14:53] t1mp, it would take more resourced, but for a few projects, it might be a better solution. [14:54] t1mp, If an MP passed stable, but failed proposed, that most likely indicates an issue in the image itself, what's the action for the MP owner? [14:54] if any [14:55] fginther: find someone to fix it in the image ;) [14:55] fginther: now we test with proposed, right? [14:56] t1mp, yes [14:56] fginther: a lot of times, a bunch of tests "suddenly" start to fail for us, and then we spend quite an effort to track down what we did wrong [14:56] ..until we figure out the bug is in the image. [14:56] that can be avoided by testing with a stable image firs [14:56] *first [14:57] t1mp, I can see that as a result we would be beating harder on the proposed image... [14:57] and then, depending on the developer, we can decide whether we debug more and help with solving the issue, or simply wait for a good image to come again [14:57] fginther: I don't understand your sentence [14:58] t1mp, if we test an MP on both stable and proposed, that would provide more test coverage of the proposed image [14:59] vs the stable one to expose issues [14:59] yeah [15:00] mostly just thinking out loud [15:00] me too [15:00] it is difficult to find a way to have everything well-tested because there are so many large and small things coming together in an image [15:01] agreed [15:23] fginther: wouldn't a passing test on stable indicate it can land? given stable is a moving target that updates all the time and gets the latest from -proposed anyway [15:23] maybe that's too obvious to be an option? [15:25] For apps yes, for stuff that actually goes through trusty-proposed in the archive that could create deadlock [15:25] (Since the contents of trusty-proposed may be blocked waiting for what you're trying to land ...) [15:25] Though I guess if you mean the trusty-proposed *image* that's not relevant [15:26] I see deadlock right now with always testing proposed and mir/ unity keeps introducing new bugs resulting in a race with low chance of winning [15:26] oh yes [15:26] image is what I meant, sorry [15:30] kalikiana, the issue that comes to mind is that the MP being tested is not released on top of stable, it's released as the next -proposed. you don't want to get too far away from that target. Apps should be image agnostic (although a change to the image can break them as well) === alan_g is now known as alan_g|tea [15:31] fginther: apps fail the same way as the toolkit when the underlying system changes [15:31] I'm not convinced making a difference there is a genuine win [15:32] kalikiana, I need to think about it more... [15:36] fginther: is there any way for jenkins to compare test results for equality? for example to say "apparently 20 mrs fail lexactly ike this, please ask a qa engineer about it" [15:36] reducing the effort of investigating would at least alleviate some pain [15:37] kalikiana: wow. how did you move the l of like to lexactly? [15:38] kalikiana, jenkins can't do this by itself. but there are ways to pull the info out of jenkins to create a running report of test failurs [15:38] It's been on my wishlist for a while, but fires keep coming up [15:38] t1mp: I don't use the same finger to type an e, probably rested on the l for longer and accidentally touched it [15:39] ah; lazy finger. [15:42] fginther: *nod* I would even say a brief message w/o any details could be helpful; doesn't have to replace anything but reduce time spent on the wrong end of the problem [15:43] though it might be equally non-trivial to implement I guess === alan_g|tea is now known as alan_g [15:48] kalikiana, https://bugs.launchpad.net/ubuntu-ci-services-itself/+bug/1246380 [15:48] Ubuntu bug 1246380 in Ubuntu CI Services "Provide running reports for upstream merger test failures" [Wishlist,Triaged] === retoaded changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: retoaded | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: - [16:10] ogra_: hey! [16:10] didrocks, yo [16:10] how are you? [16:10] why did you revert the seed change again ? [16:10] fine, thanks [16:10] ogra_: yeah, missing stuff on the Mir… [16:10] MIR* [16:10] should be finally resolved today [16:10] well, we can drop the package from the seed anyway [16:11] do you know why image 8 didn't run the unity8 AP tests on mako? [16:11] independently of the MIR i think [16:11] ogra_: yeah, we want to do everything in a shot (late yesterday, so prefered to not do it) [16:11] didrocks, because Mir crashes ? [16:11] ok, it's confirmed it's the cause? [16:11] no idea, its definitely one cause [16:11] do you think it worth rekicking an image now? [16:12] the tests are all shaky with that bug [16:12] no, it isnt [16:12] we need the Mir team to fix the crasher [16:12] didrocks, still bug 1245958 ... [16:12] bug 1245958 in unity-mir "Apps crash with image 8" [Critical,In progress] https://launchpad.net/bugs/1245958 [16:13] i wouldnt consider any test reliable until thats gone [16:13] (and apparently it broke other stuff like CI too) [16:13] ogra_: ok, we have a branch apparently [16:13] right, no package though [16:13] i know davmor2 is eager to test it [16:14] ken is testing as well [16:14] and we'll deliver ASAP [16:14] yay [16:14] no idea if there are other subsequent issues, i fear this one shields other issues that we will only see after the fix [16:15] if I have a deb I can just not got the time to learn to cross-compile from source [16:16] especially something with complex deps === gatox is now known as gatox_lunch === retoaded changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: fginther | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: - [17:24] Hello CI team... Pretty soon we're going to be porting the autopilot test suites to python 3. [17:24] However, we'd like to be able to port them one by one [17:24] but, that means that some test suites will need to be run with the python3-package, and others with the older python-autopilot package [17:24] I'd like a way so we can do this without that being a massive PITA for you guys [17:25] like, maybe we can somehow specify which suites should be run with python3 somehow so you gusy don't need to update your configuration every time we port another test suite [17:25] ...or something like that. [17:25] Who should I be talking to to make something like that happen? [17:26] I'd guess fginther [17:26] As far as timeline is concerned, we're probably looking at a couple of weeks away... [17:26] maybe 3 weeks, actually [17:26] plars, i see a i386 power test has completed, can you kick off an amd64 power test and then another i386 and amd64 test so I can sanity check the results? [17:27] cking: will do - did they results look like they were ok with your new changes? [17:27] plars, they look OK to me [17:27] cking: great [17:28] plars, just like to see some results of more runs now to ensure I've not introduced too much variability [17:28] cking: understand [17:30] ogra_: we've confirmed and top approved the fix for bug 1245958 [17:30] bug 1245958 in unity-mir "Apps crash with image 8" [Critical,In progress] https://launchpad.net/bugs/1245958 [17:30] * ogra_ hugs alan_g [17:33] thomi, yes, lets talk [17:34] thomi, is there going to be a new python3-autopilot package? [17:39] fginther: already exists [17:39] :) [17:40] what doen't exist is python 3 versions of the meta-packages [17:40] i.e.- autopilot-touch and autopilot-desktop. but since those are meta-packages we can create them easily enough, if they'd help [17:43] thomi, the apps themselves will need to depend on the right python3-autopilot or python-autopilot, right. Does that not solve the problem? [17:43] fginther: is that what you guys do? just install that package? [17:43] if so, then yeah, that's easy :) [17:44] thomi: can't you just make it depend on both? python-* and python3-* should be coinstallable [17:44] thomi, that might not be how it works now. [17:44] cjwatson: no, they both install /ust/bin/autopilot [17:44] that's a bug :) [17:44] should be in a common package [17:44] cjwatson: patches welcome :) [17:45] fginther: maybe we should do a single porting and see what happens [17:45] * cjwatson runs away to the warm friendly coreutils build [17:45] thomi, but if just don't pre-install any autopilot, then the deps should get us the right version [17:45] thomi, I'm all for a test case, I'm sure it will expose some issues [17:46] thomi, I can see it being a problem for image testing. [17:47] fginther: yeah, ok. [17:47] and testing as a click package [17:47] indeed [17:47] thomi, so yeah, lots of issue to fix first [17:47] well, I'm not doing this till 1.4 is out anyway === gatox_lunch is now known as gatox [17:52] thomi, https://bugs.launchpad.net/ubuntu-ci-services-itself/+bug/1246425 [17:52] Ubuntu bug 1246425 in Ubuntu CI Services "Need to support simultaneous testing of python-autopilot and python3-autopilot test suites" [High,Triaged] [17:53] fginther: sweet, thanks === alan_g is now known as alan_g|EOD [18:14] ogra_: so when does image 9 land :) [18:14] davmor2, once the bug is fixed [18:14] (once there is a package with the fix in the archive) [18:16] * davmor2 curses lp for not being instant at creating packages, why do you have to take minutes damn you :D [18:39] davmor2, minutes ?!? [18:39] LOL [18:39] lets talk again in a few hours [19:01] ogra_: shhhh [19:01] * ogra_ is grumpy [19:02] all google SMS services dont work for me :( [19:04] Mirv, can you dput qtcreator for trusty into ppa:ubuntu-sdk-team/ppa? [19:10] fginther: sure, bzoltan was working on getting it sponsored to archives but it has not yet happened [19:11] * Mirv pushes [19:11] Mirv, bzoltan just pinged me in another channel, he may already be working on updating the ppa [19:12] fginther: I see the discussion [19:12] Mirv, do we also need a trusty qtdeclarative-opensource-src? I can't find any packages [19:12] fginther: we sure have qtdeclarative on top of which all apps run https://launchpad.net/ubuntu/+source/qtdeclarative-opensource-src [19:12] ... I can't find any packages using http://packages.ubuntu.com [19:13] fginther: what we do need is qtcreator-plugin-ubuntu, after the qtcreator. so the cu2d-config rule should be extended to trusty [19:15] Mirv, I'm not sure what you mean [19:17] fginther: qtcreator-plugin-ubuntu has autolanding to SDK PPA for precise,quantal,raring,saucy in head/sdk.cfg, trusty should be added. it needs the qtcreator-dev [19:18] Mirv, ah, yes, you're correct [19:19] Mirv, argh! they were all updated except for qtcreator-plugin-ubuntu [19:20] fginther: well without this manual upload of qtcreator it wouldn't have compiled anyhow [19:20] since the trunk doesn't work against QtC 2.7 [19:20] Mirv, I'll get if fixed before the next MP lands [19:20] fginther: thanks [19:30] Mirv: hi! Remember guys that if you have some stuff that needs to be done, just assign it to me in the Landing Plan or the Ubuntu Unity Team Work, ok? ;) [19:31] Mirv: in the meantime I'll deal with misc issues and the appmenu Qt5 for 5.2 [19:33] Mirv: have a nice work-day guys \o/ ;) [20:02] Mirv, thomi, not sure who to contact. They're might be a regression in autopilot in the daily-build ppa, but based on a very small sample size... [20:02] I have a notes-app MP that fails on desktop when using the daily build PPA, but passes when the PPA is removed [20:03] there are only a few differences in the dpkg log, one of them being autopilot 1.3.1+13.10.20131003.1-0ubuntu1 -> 1.3.1+14.04.20131030.2-0ubuntu1 [20:03] but there have been any autopilot changes since 2013-10-14 [20:04] * fginther realizes he was looking at the wrong branch [20:05] still, no changes on autopilot/1.3 since 2013-10-15 [20:26] fginther: hey, sorry - was at lunch [20:26] fginther: so, autopilot hasn't changed in that period? [20:26] so we're off the hook? [20:29] thomi, I honestly don't know. I'm still trying to make sense of the logs [20:30] OK. I don't think we've merged much into 1.3 [20:36] thomi, AFAIK notes-app is the only thing that's failing and it hasn't passed yet with trusty [20:40] * fginther updates laptop to trusty [20:48] Hi guys - I realise this is probably going to be a massive PITA, but I wonder if we could somehow run the Ap tests for the branches listed here: http://pad.ubuntu.com/autopilot-1-4 with autopilot 1.4? [20:49] We're porting these to be compatible with 1.4, but the CI runs use 1.3, so our tests fail in CI. [20:49] it'd be nice to know that the CI system passes the tests as wel [20:49] l [20:49] outside of developer machines [20:49] but if it's too hard to achieve, then I guess we'll do withotu [20:49] *wihtout [20:49] dammit [21:01] thomi, let me see what I can do [21:03] thomi, do you have a PPA with 1.4? [21:04] fginther: ppa:autopilot/experimental [21:05] thomi, thx [21:43] fginther: hey! [21:44] didrocks, greetings! [21:44] fginther: so, I have feelings, but I would prefer numbers on the upstream merger :) [21:44] didrocks, mir is stuck in proposed [21:44] ogra_: ken should look at that [21:45] (and i'm really to tired to fix this tonight) [21:45] ogra_: I'll have a look, don't worry [21:45] fginther: with our "silos" for stack, how often do we have failures because of something else moved outside of the local repo? [21:45] cyphermox: can you either grab ken or look at why Mir is stuck in proposed? [21:46] sure [21:46] didrocks, very rarely [21:46] I'll look now [21:46] didrocks, it depends on ust ... which in turn depends on ltt-bin, lttng-tools, python-autopilot-trace ... one of them is stuck as i understand [21:47] fginther: ok, thanks for your feedback :) [21:47] cyphermox: thanks! [21:47] correct, that's the reason [21:47] ust is blocking it [21:48] yep [22:10] cyphermox: blocking because tests are running? [22:12] didrocks: No, SONAME transition. === fginther changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: cihelp | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: - [22:12] didrocks: autopilot and ltt-contol need transitioning to the new ltt-whatever2. [22:12] (In progress) [22:13] infinity: ok, thanks! (quite ironic Mir being block on another SONAME transition) [22:13] blocked* [22:13] didrocks, if i'm still around, i'll trigger a build, else tomorrow morning [22:13] (or if infinity likes to ...) [22:13] ogra_: don't worry, go home… well, go to bed :) [22:14] heh, cant be more home :P [22:14] i'm still up for a while and will take a look before going to bed later ... if its ready i'll trigger a build [22:14] ogra_: ok, just ping me if it's the case (as there is no way for me to know if a build was kicked) [22:15] yeah, i'll ping here [22:20] cihelp can someone help me figure out why this is failing? https://jenkins.qa.ubuntu.com/job/generic-mediumtests-runner-maguro/2839/console [22:21] it ends with "top write error", which sounds to me like a CI problem [22:22] thomi, did you mean this one? https://jenkins.qa.ubuntu.com/job/generic-mediumtests-runner-maguro/2838/consoleFull [22:24] thomi, the failure is caused by ppa:phablet-team/ppa not having any packages, /me wonders if this job even needs it [22:24] errr, any 'trusty' packages [22:26] fginther: yeap, i've pinged #qa team to upload _anything_ for trusty such that a pocket for trusty is published, but they didn't do that yeat. [22:26] fginther: wait phablet-team? i might have access. let me check. [22:26] fginther: hmmm [22:27] fginther: a better question to ask would be: why is this MP failing to land? https://code.launchpad.net/~elopio/ubuntu-ui-toolkit/fix1244518-composer_sheet_object_names/+merge/192641 [22:27] nah i'm not. [22:27] I can't see anything obvious in any of the failing jenkins jobs [22:28] thomi, there is one test failure: https://jenkins.qa.ubuntu.com/job/generic-mediumtests-runner-maguro/2839/testReport/ [22:29] fginther: oh.. thanks. Not sure how I missed that [22:30] thomi, xnox, I have to drop off for family time. I'll be back in 3 hours or so if I can help further [22:30] thanks fginther [22:46] cihelp, can someone tell me how many machines and what spec are being used to do build on commit in our infrastructure, and are we generally keeping up with the queue (are we over/under provisioned)? [22:46] retoaded: ^ ? [22:46] ev: probably more for fginther ^ [22:47] * ev nods [22:48] also, can someone tell me how many builds we're doing, and the total build time on average? [23:06] ev: sounds like an email question and allow ~a day to respond. I realize your probably in a meeting and that came up, but its after 5 for I think everyone on the team [23:15] cjohnston: I didn't ask for a quick response [23:20] fginther: just so you have the context, elmo is trying to get some idea of the load if we moved to a PPA-build-per-commit setup in the London DC [23:21] so whatever numbers you can provide when you're back online would be much appreciated