[03:04] === trainguard: IMAGE 256 building (started: 20140324-03:05) === [04:09] === trainguard: IMAGE 256 DONE (finished: 20140324-04:10) === [04:09] === changelog: http://people.canonical.com/~ogra/touch-image-stats/256.changes === [06:45] SDK folks are reporting phablet-flash doesn't work for them [06:46] I upgrade just by system-settings [07:40] Mirv: I have booted to the bootloader and flashed with ubuntu-device-flash --channel=devel --bootstrap successfully [07:40] bzoltan: phew [07:41] Mirv: yes... relieved. But not a pleasant experience :) [07:42] bzoltan: landing-009 for QtC plugins, click build when ready [07:42] Mirv: thanks .. building right now [08:18] cjwatson: hey! I think you read the phone ML and some people complaining of not being able to start some apps anymore. I'm not entirely familiar with the click layout, but it seems to me from latest email of Sam's bull that the installed version is latest one for music app (1.1.359) where his local desktop file is matching 1.3.389 which is… higher? (also look at the click list crash) [08:19] Mirv: hi! mind doing psivaa_'s backup on the image status as he's not around this week? :) [08:22] ah, right, he's not around this week [08:22] didrocks: vila mentioned something he can fill up for psivaa_ if anything, not sure if that changed [08:23] sil2100: excellent! he's going to do the image status then? [08:24] maybe better to have 2 pairs of eyes, just in case :) [08:24] didrocks: that's what I remember from Friday's morning meeting, but I guess that better being safe than sorry ;) [08:27] heh [08:27] didrocks: sil2100: sure, I can keep an eye on them too in addition to vila.. of course, if we'd have image testing working in the first place [08:27] sil2100: could you check line 9 in ci sheet? apparently ted used the wrong ffe bug to document the ffe [08:28] Mirv: yeah, I count on you guys to refresh me on the image testing story today :) [08:30] didrocks: latest update is from Paul on ubuntu-qa mailing list [08:30] Mirv: do you have a direct link? that would be easier (etoomanyemails) [08:31] so hosts seem down. popey emailed ev and francis on the weekend and now they've kicked it a bit but getting a lot of failing. [08:31] yeah 185… [08:31] didrocks: direct link seems a bit problematic, search for "Upgrade of ADB hosts within the CI Lab" [08:32] didrocks: sil2100: on the good news side, I've ran all AP tests this morning for 256 + updated UITK, and mostly everything passes 100% [08:32] I've 1 gallery-app test and 1 address-book-app test that seem flaky or broken [08:32] Mirv: did you try rerunning them then? [08:32] and they pass? [08:32] thostr_: checking... [08:33] thostr_: thanks for updating the bug btw :) [08:34] ffe bug? [08:34] jibel: I've got another bug where rosa maria assigned himself [08:34] thostr_: yep :) [08:34] (this ffe bug) [08:34] didrocks: address-book-app test passed individually at least, but gallery_app.tests.test_photo_viewer.TestPhotoViewer.test_photo_delete_works seems to fail consistently [08:34] kicked him out [08:34] Mirv: and with downgrading the sdk? [08:34] and I just tested that those are not related to the UITK but are on #256 in general [08:34] ok [08:34] please not that down [08:34] I mean, that, gallery-app [08:34] Mirv: any gallery-app update recentaly? [08:34] yes [08:34] recently* [08:35] how to check a 'changelog' for click package, ie upload log [08:35] thostr_: yeah, I remember the core-dev discussions around Friday [08:35] thostr_: let me see how we're standing with silos [08:35] sil2100: thanks [08:36] Mirv: I think start looking at the image content diff [08:36] calendar is at 0.4.214, I believe that was also on #250 and passed [08:36] generated in ogra's directory [08:36] ok, looked at wrong app, sorry :) [08:36] but, yes #256 had click:com.ubuntu.gallery from 2.9.1.927 to 2.9.1.931 [08:36] so that could explain it [08:37] filing a bug, although it'd be nice if we would have also automated test results [08:38] So, to make sure, sergio is currently the only one who can do a click-app landing to the store, right? [08:38] bug #1296573 [08:38] bug 1296573 in gallery-app "gallery-app AP regression on image #256" [Critical,New] https://launchpad.net/bugs/1296573 [08:40] Saviq: hi! Did you get any further with the Qt bi-secting? :) [08:40] sil2100, not really, qt is a bitch [08:40] sil2100, have pulled tsdgeos is for help [08:40] *in [08:40] fyi xnox ran all the ap tests for gallery #931 before it hit the store.. Ran 34 tests in 623.078s [08:40] Saviq: good choice, thanks! Keep us updated if anything, and good luck [08:41] sil2100, there's also upstream folk online today, so we'll be hitting them soon [08:41] sil2100, will do [08:41] Saviq: if you need any additional help, just ping [08:42] popey: aha, aha, good to know [08:43] I wonder why I'm hitting that, or is there something else on #256 (or before) that causes it [08:43] ie what base image xnox was using [09:06] sil2100: as it seems there is no-one from CI around, do you mind retrying the whole set of job? [09:06] for the tests [09:06] to see if that's going to fix it [09:07] (the master job I pointed to all of you last week) [09:08] didrocks: which set of jobs? [09:09] sil2100: ah, you didn't write it down? [09:09] let me refind it [09:11] I don't seem to have it anywhere, hmm [09:11] sil2100: http://q-jenkins.ubuntu-ci:8080/job/trusty-touch-mako-smoke-master/ [09:11] Mirv: please note it down too ^ [09:11] this is the master job to relaunch iso testing [09:11] nice to know, thanks, bookmarking [09:14] It seems it's waiting for the next available executor to start [09:15] hum [09:15] asac: so, we're stuck, no device available ^ [09:16] It seems the only one available is running touch_custom right now? [09:16] yep [09:22] apparmor.click.AppArmorException: "Could not find '/usr/share/autopilot-touch/apparmor/click.rules'" [09:22] hmpf [09:22] sil2100, didrocks, Mirv : Yes, only one mako available right now. Thanks to plars and fginther` efforts during the week-end 1 > 0 [09:22] vila: it's the same device than touch_custom and so we are blocked on those tests to finish? [09:23] (for the rerun) [09:23] vila: seems like we don't have the AP tests installed on the device on previous 256 run though, and I can't figure why we don't bzr branch the tests [09:23] asac: FYI ^ [09:24] didrocks: probably sergio topic [09:24] asac: or infra? [09:24] didrocks: we could compare logs [09:24] ogra_: what you broken now [09:24] but yeah, basically all click apps didn't run tests [09:24] didrocks: yeah leave it run I'd say [09:24] ogra breaks everything [09:24] ok [09:24] single source of obstructions :) [09:24] davmor2, nothing, thats the crash we have on all failing tests [09:24] vila: well, I think we'll have the same issue on the rerun, don't you? [09:24] didrocks: yeah, there seems to be a single common failure related to click apps not launching [09:25] autopilot-touch from 1.4+14.04.20140319.1-0ubuntu1 to 1.4+14.04.20140319.1-0ubuntu2 [09:25] apparmor denies execution [09:25] http://people.canonical.com/~ogra/touch-image-stats/20140323.changes [09:25] didrocks: sounds likely but it's a wild guess, it's either a click/apparmor issue or some unforeseen fallout from re-provisioning the device [09:26] .oO( we really need servers without BIOS in the lab ) [09:27] my DSL reconnect seems to have moved to 10:30 ... i might be a few mins late to the meeting btw [09:27] ogra_: IIUC said server has a memory issue and the debug they attempted *locally* (i.e. physical access with a physical keyboard) didn't help [09:28] vila, yep, i saw the mail [09:28] ok, great [09:31] bah, google talk crash [09:31] * popey restarts browser [09:32] \o/ freed up 10GB RAM [09:32] deleted your mail ? [09:32] closed chromium [09:32] hah [09:33] apparmor.click.AppArmorException: "Could not find '/usr/share/autopilot-touch/apparmor/click.rules'" [09:35] popey: once shutting down all those html5 games… :p [09:35] didrocks: I still don't have the high score in Google Docs! [09:38] |HELP [09:38] |HELP [09:38] I am the firendly image watchbot [09:39] k [09:39] !bots [09:39] Hi! I'm #ubuntu-ci-eng's favorite infobot, you can search my brain yourself at http://ubottu.com/factoids.cgi | Usage info: http://ubottu.com/devel/wiki/Plugins | Bot channels and general info: https://wiki.ubuntu.com/IRC/Bots [09:39] ogra_: should probably add your bot to that page [09:39] :) [09:39] so people know about it [09:39] ogra_: fire ndly ? Put fire on all images ? :-p [09:40] popey, thanks [09:41] vila, it only kindles a little :P [09:41] ;) [09:41] popey: I can implement it in the CI train spreadsheet if you want… :p [09:42] np [09:52] autopilot (1.4+14.04.20140319.1-0ubuntu2) trusty; urgency=medium [09:52] * Add python-gi and python3-gi dependencies which provide gi.repository [09:52] modules. (since gir1.2-upstart-app-launch is accessed via [09:52] gi.repository) [10:05] WTF [10:05] sil2100: hahaha [10:05] Now google hangouts won't start! [10:06] didrocks: ok, it seems both my internet and google hate me today [10:06] "Please wait" [10:06] ... [10:07] didrocks: as for landings, there is one content-hub landing that is ready for landing, but I need sergio around to publish the new gallery-app along with it [10:07] sil2100: it's okay we pushed all the work over to you [10:07] didrocks: no other were marked as 'testing done', but I will consult the scopes one I guess [10:07] popey, did you mean the movies ? [10:07] I have 12 items showing in gallery [10:08] and 12 media items in the gallery database [10:08] sil2100: ok, and new scopes? [10:08] i think it is supposed to [10:08] maybe mhr3 would know? ^ [10:08] http://paste.ubuntu.com/7145536/ [10:08] sil2100: yeah :) [10:08] (sorry was bcklogging) [10:09] I'll file a bug [10:09] didrocks: yeah, will poke them about that :) [10:09] mhr3, can you maybe land/clean l23 (code cleanup in the clip scope)? it has been built and holding a silo for at least half of a week [10:09] popey: ok, so I guess it's a bug for the apps team? (but if you can try the upgrade stuff…) [10:09] * sil2100 has lags so sorry if answering slow [10:09] popey: maybe one version picked the videos that you have at some point and didn't flush it? [10:09] sil2100: no worry [10:10] * popey shrugs [10:10] popey: the videos name makes sense at least? [10:10] like it's videos you had at any point on your device? [10:10] they are on my device now [10:10] those are the videos I test with [10:11] seb128, it didn't pass testing last time [10:11] maybe gallery is hooked up to mediascanner now, dunno, bug will reveal all [10:11] * popey files [10:11] mhr3, if so can you drop it and give back the silo and ask for one again once it's ready? ;-) [10:12] seb128, fine, drop it [10:12] sil2100, Mirv, didrocks: it would be nice to keep some "desktop silos" with beta freeze today, so we don't block desktop changes to land because there are not silo left [10:12] mhr3, I don't have powers for that, just ^ [10:12] seb128, eh, sorry, i thought i'm talking to sil2100 :P [10:13] lol [10:13] mhr3, you need more coffee! [10:13] mhr3, btw there is a fix up for the manpages scope issue we discussed some days ago, if you want to review/approve that [10:14] seb128, waiting for david to confirm [10:15] mhr3, you always have a good reason! ;-) [10:15] mhr3, thanks [10:15] seb128: sil2100: I now went one below the set minimum of 3 free silos to have at least one for desktop, landing-018, and kicked a build [10:15] Mirv, thanks [10:15] seb128: I agree with you and we can use silo-000 if you need anything [10:15] Mirv: FYI ^ [10:15] Mirv, don't worry about the desktop ones, I'm going to drive them through [10:15] didrocks: seb128: aha, ok. [10:16] e.g they should be freed in a few hours every time, no days locking [10:16] didrocks, thanks [10:16] seb128: maybe start from the second one and use 000 for that, while I check freeing up the 018 again [10:17] Mirv, your call, I don't think it's an issue to be down to 2 for an hour [10:18] davmor2: didrocks bug 1296634 [10:18] bug 1296634 in gallery-app (Ubuntu) "gallery shows blank dates and photos" [Undecided,New] https://launchpad.net/bugs/1296634 [10:19] popey: confirmed [10:20] seb128: ok, you can have it, I just had a configuration messup but now it's building, for real this time. [10:21] tvoss: can you see about landing the dbus-cpp ppc64el fix that has built in landing-001? [10:21] Mirv, sure, flashing latest image right now [10:21] Mirv, thanks [10:25] popey: thanks [10:28] popey, didrocks: Oho some of the apps are installing on 237 [10:29] davmor2: wdym? [10:29] if they have the 13.10 framework, they will [10:29] not every app bumped the framework in their latest updates [10:30] popey, didrocks: they did for the qt5.2 rebuild I thought [10:30] no. not all [10:30] some [10:30] popey: ah oky [10:30] apps can still install post-qt5.2 if they have the older framework [10:32] just leaving myself a list of app to try, gallery, terminal, camera, music, notes all have command error, calculator, calendar and sudoku all installed. [10:35] didrocks: Yeah, I have a data dump from sergiusens that looks like the same kind of thing - was planning to sit down and stare at it today. I don't think it's a particularly recent regression, if it's a regression at all - it's the sort of thing that can build up as people install apps, upgrade images, etc. [10:36] cjwatson: yeah, I was on same assumption, can be an upgrade before updating the image, or multiple upgrades + removal… [10:37] Saviq: hum, just read kgunn's email? Was I implying it was Mir's fault? I was just explaining the bug… [10:37] didrocks: there's definitely something weird, can't tell you what yet [10:38] didrocks, it read a little bit like it, don't worry [10:38] didrocks, it might still be in the end ;) [10:38] Saviq: argh, ok, sorry for that, I'll talk to kgunn [10:39] not a bug per se, but an incompatibility between how Mir and Qt handle things [10:39] cjwatson: yeah, I think it's worth having on the radar and see if we can get a clear reproducer [10:39] but yeah, not anything that will block as probably not new but a followup of people upgrading manually some apps [10:39] didrocks: let's not have anyone spend effort on it right now, as I say I have a data dump [10:40] Saviq: yeah, we do agree, I just tried to explain [10:40] cjwatson: good :) [10:57] didrocks: So I just flashed 237, updated all the apps that would, seen the command error issue on all the others. I've then upgraded to 256 and everything works everything opens with no issues [10:59] davmor2: ok, I think just concentrate on normal dogfooding until we have image test results now :) [10:59] sil2100: progress on the gallery-app test failure? [10:59] didrocks: that's my plan now i'm a work ;) [10:59] thanks davmor2 :) [10:59] Morning all [11:00] didrocks: had to reflash with bootstrap, as it seems I had some deb version of gallery-app and the click one did not want to work [11:00] didrocks: so I'll be running the real test just now [11:00] sil2100: ok, let's see if the test failing is flaky or not… [11:00] hey davmor2! [11:14] hmm, to bad psivaa isnt around ... having a test run with rolled back AP would really be interesting [11:16] grr [11:16] ogra_: yeah, let's see with Mirv's first test result run [11:16] webapps really misbehave on my phone [11:17] with a clean bootstrap image [11:17] ogra_: misbehaving like? [11:17] didrocks, well, i always have like 5 open constantly ... flicking through them one or two always die [11:17] randomly [11:17] not sure when that started [11:18] (i'm on 254 btw) [11:18] * ogra_ upgrades to 256 [11:20] ogra_: what webapps are you trying? so that I can try on latest [11:21] heise, golem n-tv.de are the ones i have always open [11:21] and G+ [11:21] i'll see after upgrade if it happens right from the start or only after a while ... havent rebooted since the last upgrade [11:23] doesnt happen after upgrade/reboot yet ... i'll watch it over the day [11:23] aha, now it did [11:23] (and is fine again after the app restarted) [11:24] i'll leave the phone alone for a while and re-check again later [11:24] ogra_: yeah, I can't reproduce it until now here [11:24] session restore would really be an awesome feature :) [11:25] so you dont have to always re-open all your apps after upgrade [11:28] yeah [11:28] lol [11:28] http://www.ebay.com/bhp/nexus-4-broken [11:28] they even have a category for that [11:29] waow [11:29] we should buy all of these and put them in the lab :) [11:29] (i use my broken one here for doing bootcharts ... works excellent) [11:42] didrocks: one broken phone per test how quick would result be in then ;) [11:42] heh :) [11:46] davmor2, thats the idea [11:46] parallelize all tests [11:47] ogra_: No don't parallelize them, it's cruel to put tests in wheelchair [11:47] :P [11:52] is this the correct channel to request to trigger a CI run on this MR? https://code.launchpad.net/~andrew-hayzen/ubuntu-ui-toolkit/fix-swipe-delete-002/+merge/202171 [11:52] t1mp: it is, please ask the citeam (if no vanguard, just use cihelp) ^ [11:56] !cihelp [11:56] ? [11:56] t1mp: you just use it in hilight and they'll get it [11:56] for hilight [11:56] that magic string ci_help without underscore [11:56] Mirv: ahh :) [11:57] t1mp: looking [11:57] josepht: thank you. [11:57] FWIW so far even after bootstrap I'm not getting test failures (in click packages) [11:58] didrocks: ^ on #256 [11:58] Craaap [11:59] Mirv: ok, so seems to be infra-side only… :( [11:59] asac: FYI ^ [11:59] so it seems [12:02] sil2100, Mirv: I freed my 2 silos, if you want to give me 1 pack for some other desktop line ;-) (need to keep the ball rolling with beta freeze today) [12:02] Mirv, no aa-click crashes ? [12:02] 1 back* [12:03] didrocks: cant be long until they come on [12:03] seb128: ;) Thanks! [12:03] seb128: doing! [12:03] ogra_: where they would be visible? [12:03] sil2100, thanks [12:04] Mirv, in /var/crash [12:04] ogra_: nope. I do have one _usr_lib_arm-linux-gnueabihf_upstart-app-launch_desktop-hook.32011.crash [12:04] that seems to have happened during notes AP tests (which succeeded) [12:07] seb128: ok, assigned - line 55 will have to wait as u-c-c is already locked by silo 11 [12:07] From bregma's landing [12:07] oh [12:08] sil2100, thanks for spotting that, and that's fine ;-) [12:08] popey, didrocks: I just noticed that my 256 image has no oddities in the gallery app. That is on the upgrade from 237 to 256 so I wonder if it is something on the platform from 255 to 256 rather than gallery itself? [12:08] Mirv, yeah, doesnt look like the one we get on the infra === josepht changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: josepht | CI Train support - US: robru, cyphermox, rsalveti - EU: sil2100, Mirv, didrocks | CITrain support no answer: use mup bot after 30 minutes, but choose right timezone | Known issues: ADB host ashes is down [12:13] sil2100, you didn't free 002? [12:13] sil2100, pinged you last week that i'm giving it up [12:13] sil2100, and now i want silo for 57 [12:15] didrocks: ok thats semi-good news i think [12:15] Mirv: do you use the phablet-flash command? [12:16] or ubuntu-device-flash? [12:16] asac: which news? [12:16] didrocks: that its infra only [12:16] mhr3: I freed it, I guess someone else assigned it? [12:16] asac: yep… [12:16] didrocks: on the bad side we are blind [12:16] but maybe we are still good :P [12:16] indeed [12:16] or we are regressing [12:16] as long as we produce enough images we are probably fine [12:16] and maybe have someone once a day crunch all APs manually :( [12:18] yeah, now having doanac's "run all AP tests like UTAH did" script would be nice to have [12:18] mhr3: let me free it again then [12:20] asac: that's what Mirv did for a sdk landing === MacSlow is now known as MacSlow|lunch [12:20] didrocks: no failures on my device on latest image for gallery... ran the whole test suite 4 times already [12:20] sil2100: can you just upgrade from Mirv's silo and see if you get the gallery failure he has? [12:48] didrocks: hmm, seems to look ok as well, 2 test runs and no failures [12:48] not sure what Mirv is experiencing, maybe the phone doesn't like colder temperatures… [12:48] or cats [12:49] sil2100: thanks for looking at it! [12:49] * didrocks classes that at NOISSUETHERE for now [12:50] cat is a possible explanation [12:51] Waiting just in case for the third test suite run [12:55] asac: too many highlights, so I used ubuntu-device-flash this time for the newest bootstrap, and I was using phablet-flash earlier [12:56] i can definitely run the aa-clickhook command from the .crash manually and it properly works and finds the file [12:57] * ogra_ has no idea wh it wouldnt work during testing [12:57] *why [12:58] ok in my latest gallery-app rerun after bootstrap flash I got one fail, but it was different from before, so no I can't trustworthily reproduce the TestPhotoViewer.test_photo_delete_works ... [12:59] Mirv: that might be the bug that appears on an initial install IIRC it is something to do with the icon on notes [12:59] didrocks: 256 is looking similar to 250 for me [12:59] davmor2: apart from the blank dates and photos, right? [13:00] didrocks: see my message earlier I don't see that on the upgrade from 237 to 256 [13:00] davmor2: your first trial was a direct flash? [13:01] and you did wipe when getting back to 237? [13:01] didrocks: no upgrade from 255 or 254 whatever was released early on Sunday [13:01] ok [13:01] and on 237, you did wipe, imported some photos and upgrade? [13:01] didrocks: I did a bootstrap to get 237 on [13:02] and imported some photos/videos? [13:02] or all blank and fresh? [13:02] didrocks: blank and fresh I can add some images if you want and try again [13:03] davmor2: yeah, let's see that [13:03] so get 237 [13:03] import some photos and videos [13:03] run the gallery-app [13:03] and then update [13:04] didrocks: will do I'll give you a ping in a bit [13:06] * sil2100 lunch === alan_g is now known as alan_g|lunch [13:07] great! [13:09] not that this would help much with all tests failing :P [13:10] Sorry for the -proposed logjam, folks, having some problems with the x264 transition causing proposed-migration to take eons [13:10] s/all tests/all click tests/ [13:11] I *think* this run might sort it out === MacSlow|lunch is now known as MacSlow [13:21] balloons: with the new qtdeclarative 5.2.1-3ubuntu11 there's a fix for bug #1294181 from qtdeclarative side, ie the revert-212 could be reconsidered since it shouldn't be crashing anymore [13:21] bug 1294181 in qtdeclarative-opensource-src (Ubuntu) "Autopilot tests crashing in switch_to_tab helper" [Undecided,In progress] https://launchpad.net/bugs/1294181 [13:23] Mirv, yes, did kunal re-propose the merge? [13:23] it should pass and yes, we should add it bacl [13:28] balloons: I don't see such a branch that would revert the revert, but anyhow such could be proposed now [13:29] Mirv, would you mind doing it? I can help approve/review [13:31] balloons: tried, a few problems. bzr merge -r 213..212 has conflicts in WeekView.qml and calendar.qml [13:33] I might be able to cope with it even with my zero calendar knowledge [13:33] just a minute [13:34] Mirv: when you have 10 mins, we could do with a calendar release. lots of good UI stuff landed over the weekend. [13:35] sil2100, didrocks: can we get silo14 released? it needs gallery app click to be released at same time [13:36] bfiller: gallery-app is part of the silo 14 isn't it? [13:37] bfiller: and please, ping all EU people, so that Mirv can help you as well if around :) [13:37] (see /topic) [13:37] howdy [13:37] bfiller: who is doing the gallery-app part from the click store, is that coordinated? [13:37] hey cyphermox [13:38] didrocks: I don't know, sergio usually does it but he's not here today [13:38] yo yo yo, how's everyone and everything doing? [13:38] didrocks: sil2100 pinged me earlier, and I suggested xnox if he's about [13:38] * ogra_ is banging his head against the aa-clickhook crashers and doesnt get anywhere [13:38] otherwise balloons is around and can upload to store too [13:39] didrocks: yes gallery is part of silo 14 but needs a click release at the same time of deb release [13:39] bfiller: please get xnox or balloons in though, so what we don't release in a broken state [13:39] we don't have access to click store here [13:40] ty Mirv [13:40] didrocks: this is going to be a common occurance in the next few weeks where we'll need coordination [13:41] balloons: can you help get gallery-app click ready for release (using branch in silo-14)? [13:41] bfiller: yeah, I expect the landers to coordinate on that. That's part of the issue of having your app in click [13:41] didrocks: great [13:41] didrocks: wishing we hadn't done that (: [13:42] we really need the train to release both click and deb [13:42] bfiller: I warned about it (not sure it was to you :)) [13:44] didrocks: is there a roadmap for having CI Train support both click and deb? [13:44] bfiller: not with the current workload [13:44] popey: um, should I know about publishing click apps? [13:44] bfiller: the airline should have it, it's on the roadmap of it [13:44] bfiller: not sure about the train [13:44] Mirv: it wouldn't hurt if you did, xnox does. would be good to have more .eu people able to [13:45] popey: sure, I'd love a guide to it [13:45] (note: I don't have the power to publish clicks, and I don't want to, I am happy keeping the approval auth only) [13:45] Mirv: if xnox isn't around, balloons may be able to help here ☻ [13:46] xnox is off today afaik [13:46]