/srv/irclogs.ubuntu.com/2013/12/17/#ubuntu-ci-eng.txt

balloonscihelp, jenkins isn't building music-app00:26
balloonshttps://code.launchpad.net/~vthompson/music-app/fixes-1259962/+merge/19860100:26
plarscwayne: sorry, didn't get a good chance to reconfigure the jobs before, but I merged the change already. I just pushed and merged it all now, and re-started the custom jobs here: http://q-jenkins:8080/view/Touch/view/Ubuntu%20Touch%20Master%20Jobs/job/trusty-touch_custom-mako-smoke-master/05:05
plarscwayne: we should check on them tomorrow morning, but they will all run now anytime there's a new custom image05:05
sil2100Morning07:57
sil2100Mirv: hello! Did you notice any infra failures today maybe?09:13
Mirvsil2100: intel failing right now there running apps stack ap:s09:14
Mirvnothing else09:14
MirvI got one good result before that09:15
=== ev changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: cihelp (EU CI members sprinting this week) | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: -
loolev: joining hangout?09:33
sil2100psivaa: hello!09:43
loolpsivaa: heya09:43
loolpsivaa: could you retry terminal and unity8 on maguro after recording hte URLs of failing tests?09:44
psivaalool: will do09:44
loolpsivaa: also would need webbrowser on mako09:44
psivaalool: will do09:44
loolpsivaa: cool, could you ping us when done?09:45
loolpsivaa: would like to kick an image build once it's done09:45
psivaalool: sure09:45
asacpsivaa: seems we duplicate here :)09:45
asacerr in -touch09:45
psivaa:)09:46
asacso yeah. let us know when done so we can swiftly kick09:46
evlool: sorry, couldn't make it. Running a sprint on juju and ci in bluefin today09:50
loolok09:52
Mirvsil2100: building it now first. I think we had the rule for this week that cu2d desktop tests need to pass as well before releasing anything, so let's see them first and I'll continue on my Qt stuff09:53
sil2100Mirv: thanks!09:56
sil2100Mirv: regarding testing the bug, from the merge request I see that it's indeed just switching between 12h and 24h locale, looking at some PM time and checking if the indicator changes10:03
psivaalool: asac: the three tests  have been run and passed. sending you the earlier failed links in a mail10:19
sil2100psivaa: \o/10:20
loolawesome10:21
loolsil2100: good to start a build?10:21
sil2100lool: fire it up!10:22
lool99.5%10:22
loolthat's pretty good10:22
loolbuilding10:23
loolsil2100: Do we want to promote image 70?10:23
loolsil2100: maybe check with popey and others10:24
loolsil2100: just let me know here if you want it promoted10:24
popeyI can check on mako, but you need someone (omer) with maguro to test too10:24
sil2100hmm10:25
asaclool: how can we get those final tests fixed?10:25
asacom26er: ^^10:25
asacom26er: we have like 3 AP tests... can you just crush them? :)10:25
sil2100popey: if you have a free moment then check, if you can - I would personally prefer promoting an image with the datetime fix, but any promotion doesn't hurt ;)10:26
asacom26er: http://ci.ubuntu.com/smokeng/trusty/touch/mako/70:20131217:20131211.2/5503/ubuntu-weather-app-autopilot/10:26
asacom26er: if you can fix those i will give you whatever you want :)10:26
asac2!!!10:26
om26erasac, sure. get me a pay raise ;)10:34
asacom26er: how much? :)10:35
timpwhat is needed for a new ubuntu-ui-toolkit release? It has been in the landing asks page for a while now10:35
om26erasac, i'll settle for 10% but before april.10:36
om26erworking on the failures now10:36
asacom26er: seems its just a DB not being created10:37
asacnot sure whats going on with the .ini files10:37
asacand who should create those10:37
asacin create_blank_db10:38
asacsomething is missing10:38
asac:)10:38
asacfeels all it wants is to figure what location to use for the sqlite file10:38
asacnot sure why the algo is so weird and goes through .ini files in Databases directory10:38
=== xnox_ is now known as xnox
popeysil2100: 70 looks good to me on mako10:50
sil2100popey: thanks!10:51
sil2100om26er: could you give image 70 a spin on maguro as well, just like popey ?10:52
om26ersil2100, Ok, updating. maguro is too slow to be fun :)10:52
sil2100;D10:52
om26ersil2100, how to run autopilot tests for click packages ?10:56
popeyom26er: http://paste.ubuntu.com/6588007/  i use that script10:56
popey./testing.sh lp:ubuntu-rssreader-app ubuntu_rssreader_app10:57
popeylike that10:57
om26erpopey, thanks10:58
sil2100om26er: https://wiki.ubuntu.com/Touch/Testing <- this guide works for me as well10:58
sil2100So just phablet-config autopilot --dbus-probe enable10:58
sil2100And then: phablet-click-test-setup --click com.ubuntu.calculator; phablet-test-run ubuntu_calculator_app10:58
sil2100etc.10:58
sil2100Damn, my neighbour again started drilling10:59
sil2100Doesn't help with concentrating10:59
popeybug 1192711 is a little irritating11:07
ubot5bug 1192711 in mediaplayer-app "video player won't play a second video without closing completely" [High,Confirmed] https://launchpad.net/bugs/119271111:07
popeynot a regression, seems to have been there a while, but I'd not noticed11:07
Mirvsil2100: https://code.launchpad.net/~timo-jyrinki/cupstream2distro-config/indicators_missing_deps/+merge/19926511:21
sil2100Mirv: do you know if both libupstart-app-launch1 and libupstart-app-launch2 are required to be installed right now?11:22
asacom26er: any clue yet?11:24
asacom26er: sergiusens knows how to run AP test for click11:24
asacoh already answerd :)11:24
LaneyCan someone help me figure out why most of the autopilot tests for ubuntu-system-settings are borked (the ones which try to select anything, it seems), please?11:27
om26erasac, so the issue is the test suite expects a certain directory to be there but its not able to find it because that directory is only created once the weather app is started11:27
asacom26er: i got so far... but it also expects .ini files in that directory :)11:28
asacso i dont know where that .ini is supposed to come from11:29
asaci think this is super bogus code to be honest :)11:29
om26erasac, works fine for me after I started the weather app manually11:29
asacom26er: where is that directory created?11:29
Laneyborked on the device only, that is --- they work on the desktop...11:29
asacom26er: i didnt see anythign in code creating that directory ... nor the .ini file11:29
Mirvsil2100: in the previous cases it seemed to be, at least earlier, that if the older one was removed it did not work11:29
om26erasac, ~/.local/share/com.ubuntu.weather/11:29
asacom26er: where in code i mean :)11:30
asacoh wait11:30
asacyeah .../Databases11:30
asaci still dont see anywhjere in code that this dir is created11:30
asacalso the code checks for .ini files11:30
om26erasac, exactly that directory is not created by the test suite, rather it is automatically created when the app is started11:30
asacanyway. seems you already have it running, so debugging shoudl be simple11:30
asacom26er: which code is doing that?11:31
asacom26er: is that something our sdk does? does it also create the .ini files?11:31
om26erasac, the app is using QtLocalStorage for that11:31
sergiusensasac, the dir is created when using LocalStorage11:31
asacok seems the test doesnt use it11:31
asackk11:31
asacso the tests should be using that :)11:32
Mirvsil2100: anyhow, the cu2d tests are not green so if I should test on device or not11:32
asacinstead of raw mkdirs etc.11:32
asacerr raw os. stuff11:32
asacinteresting that i get no direct hit when googling for QtLocalStorage11:32
asacis that no public API?11:32
om26ersergiusens, I am thinking to check if that directory exists and if it does not just start the app and close it in the suite. sounds hacky ?11:33
Mirvsil2100: I understand that we won't release unity stack until it's green, but what about indicators? do you think I can go ahead and test on device with those results currently in cu2d and publish if device ok?11:33
asacsounds hacky :)11:33
asacbut makdirs that directory would make sense11:33
asac:)11:33
sergiusensom26er, that was there before and actually removed from the test suite to make it more stable11:33
asachehe... just dont know about the .ini and if that info in ther ematters11:33
sergiusensom26er, I don' t get it; I tested by deleting the dir completely when first accepting11:33
asacso why cant the test use qt local storage as well?11:34
asacor call an app func to do the setup11:34
asachmm. guess its not nicely available in python :)11:34
sergiusensasac, QtQuick.LocalStorage11:34
asacright. so if we would have qmltests rather than AP things would be easier?11:35
asacanyway. i feel fro the test all that matters is that we give it SOME db filepath11:35
asacthat is  valid11:35
om26erbut qmltests are not integration tests11:35
asacso whatever we give too it its good and if its even in the right .local/ directory owned by the app its even better11:36
sergiusensasac, the app is supposed to create everything it needs on it's own, and it does11:36
asacsergiusens: right, so its a race?11:37
asacsergiusens: where does the test start the app?11:37
sergiusensom26er, asac, the test code just needs to contemplate the inexistence of a prior database, unless it mangles with it before launch11:37
asacsergiusens: so we might want to change the find_db into a poll that waits for 60 seconds?11:37
asacand only then gives up?11:37
asac:)11:37
sergiusensasac, that may be the case; not sure if it waits for everything11:37
sergiusensasac, yeah, that's what's been happening with the tests11:38
sil2100Mirv: let's test it on the device, since I guess not-green tests on indicators right now basically might be related to some desktop issues11:38
sil2100Mirv: let's make this one exception11:38
sergiusensbtw ci, need a release for autopilot to get some emulator love http://www.youtube.com/watch?v=n2pVpjBFiaI11:42
sergiusenscihelp that is11:42
asacnice one11:43
asacsergiusens: lets get one 100% image11:43
asacthen we land autopilot... hwoever, i wouldprefer if we could pick just what you need11:43
asacand leave the rest out11:43
sil2100There's a lot of autopilot changes pending still11:43
asaccheck much stuff people have accumulated http://people.canonical.com/~platform/cu2d/results11:43
asaci would prefer if we only land exactly what we understand out of those11:43
sil2100I guess we could release this, but I would like to get all the other pending bits out separately first11:43
asacat best all, but only after looking at each change11:44
asac:)11:44
asacsil2100: lets get 100% green before pulling in AP11:44
asacAP folks should be testing the hell out of their AP11:44
asacso once we take it its clean11:44
sergiusensasac, sil2100 would a package patch do? It's a one line add to a map, that's about it for now11:45
timprobru: ping11:45
sergiusensI can prepare a debdiff11:45
asacsergiusens: not before we have 100% :)11:45
asacget us the fix for this test :)11:45
asacthen we cna land whatever you want11:46
asachehe11:46
sergiusensasac, ack, I can wait11:46
asacsergiusens: help fixing the test11:46
asacits just this DB thing11:46
asacat best next image would have the fix11:46
sil2100sergiusens: that one-liner, would it help in getting us green, or is it related to something else?11:46
asacsergiusens: check with om26er if he needs help etc.11:46
sergiusenssil2100, not related11:46
sergiusensom26er, do you need help?11:46
sergiusensom26er, I can review whaever you've done so far11:47
sil2100sergiusens: then let's wait and release it with all the AP changes then - we need to release those ASAP anyway ;)11:47
asacsil2100: please send a mail to thomi,sergiusenss etc. that they should start running APs all day long against their AP package from daily-build11:47
asacand give us a list of APs that are green11:47
asacafter that ... at best with logs :)11:47
Mirvsil2100: alright11:49
om26ersergiusens, I only figured what was the problem. now working on workarounds/solutions11:52
om26ersergiusens, will get help from you if i run out of ideas11:52
sergiusensack11:54
timpMirv, sil2100 do you know what is blocking the UITK release? the flaky test from https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0Au6idq7TkpUUdGNWb0tTVmJLVzFZd0doV3dVOGpWemc&usp=drive_web#gid=0 was fixed a while ago11:54
timpit was this one https://bugs.launchpad.net/ubuntu-ui-toolkit/+bug/125947611:55
ubot5Ubuntu bug 1259476 in ubuntu-ui-toolkit (Ubuntu) "Flaky gallery.test_textinput.TextInputTests.test_textfield_numbers, fix or disable" [Undecided,New]11:55
sil2100timp: hi!11:55
timpsil2100: hello :)11:55
sil2100timp: from what I see we required an ACK from you guys and a re-addition to the Landing Asks once the flacky test is gone :)11:55
sil2100timp: so, if you can, please add it and we'll do everything we can to release it ASAP11:56
timpsil2100: I don't have edit rights for that doc11:56
timpsil2100: can I get rights or should I ask zoltan to add it tot he list?11:56
timp*to the list11:56
popeyom26er: did I see sil2100 bribe you to look at the weather test failures? Are you looking at those?11:57
om26erpopey, yes, I am working on it11:58
Mirvtimp: it's usually the manager that adds it11:58
popeyom26er: great. let me know if you need any testing of it done11:58
timpMirv: okay.11:58
sil2100timp: if you can, please ask Zoltan :) Thanks!11:58
timpokay, asking. thanks12:00
timprobru: unping :)12:00
timpsil2100: it is there nwo :)12:03
timp*now12:03
timpI seem to have a typing problem today12:03
om26ersergiusens, summary is: the test code adds certain fixed locations to the database (hamburg and London in this case) before starting the app so that the tests have some data in the app to interact with. for injecting these pre-defined db entries it needs to have an already created database.12:15
om26ersergiusens, the notes-app does exactly the same thing. it check if the db directory is there and if its not it starts and quits the app12:15
sergiusensom26er, ok, there's old code in there that calls a launch_and_close_app12:18
=== gatox is now known as gatox_dr
timpsil2100: can you please keep me informed about the uitk release? If there are any issues please let me know so we can fix them :) thanks12:40
sil2100timp: sure, thanks! :) We'll add it to our todays release list12:40
timpgreat12:40
sil2100Mirv: how's the progress on indicator-datetime ?12:41
sil2100popey: hmm, you said you have some bluetooth audio device for Ubuntu Touch testing?12:41
Mirvsil2100: it's starting to be done, I'm wondering though who to bug with packaging changes12:41
asaclool: did we ever build an image :)?12:42
asacah seems we have 7112:43
asacnice12:43
asacpsivaa: something was going wrong on mako it seems12:43
asacguess you are on it?12:43
sil2100Mirv: that's a valid question! Maybe try poking Laney or lool I guess?12:44
sil2100Later we can always pester Ken or Mathieu12:44
asacack. pick any friendly core-dev :)12:45
LaneyThere's loads in #ubuntu-devel, but sure12:46
MirvLaney: http://pastebin.ubuntu.com/6588664/ (to fix http://pad.lv/1256061)12:46
ubot5Launchpad bug 1256061 in indicator-datetime (Ubuntu) "clock forcefully switches from 24h to 12h AM/PM format once the panel clock is loaded" [Undecided,New]12:46
Laneymaybe you could look up my question from earlier that went unanswered to thank me back :-)12:46
Laneyoh it's this12:47
Mirvthe merge request does not explain the details (small fixes) https://code.launchpad.net/~charlesk/indicator-datetime/lp-1256061/+merge/198968 but seem sok12:47
LaneyI told him it's better to use locale-gen directly12:47
Laneybut fine12:47
popeysil2100:  no12:47
popeysil2100: oh, actually, my daughter has a speaker, let me see if that works12:48
sil2100popey: since someone mentioned on the ML that bluetooth devices still don't work with the latest image12:48
MirvLaney: thanks. ooh, autopilot test failures.. did you see the logs already?12:48
Laneywhich logs?12:48
MirvLaney: settings failures, there are some tests failing in the CI12:49
Laneyshow me?12:49
Laneyit's probably the same thing12:49
Mirvhttp://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-daily_release/957/label=autopilot-nvidia/ + http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-daily_release/957/label=qa-intel-4000/12:49
Mirvfor some reason on intel test_about.StorageTestCase.test_installed_apps and on nvidia test_about.LicenseTestCase.test_licenses_page12:49
Laney/var/local/autopilot/autopilot.log: AssertionError: The following apps were started during the test and not closed: [<Application 'Report a problem...'>]12:50
Laneyis it that?12:50
popeysil2100: it doesn't detect my bluetooth speaker12:50
popeybut I don't know if it ever did, because I have never tested this12:50
sil2100CraB12:50
MirvLaney: looks like that. a) it seems like apport, b) it's different from your "most tests failing"12:50
Laneyit is12:51
Mirvyep, http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-daily_release/957/label=autopilot-nvidia/artifact/results/autopilot/videos/ubuntu_system_settings.tests.test_about.LicenseTestCase.test_licenses_page.ogv12:51
Laneysystem-image-dbus crash12:51
sil2100psivaa: hello! Do you have a Ubuntu Touch compatible bluetooth device by any chance?12:52
Laneyseems like it's not our fault12:52
Laneyalso that runs on desktop ...12:52
Laneythe problem I had was only on the phone12:52
MirvLaney: bypassing those, are the selection problems using selection via UI Toolkit somehow? just wondering if SDK people could help debugging the errors12:53
Laneyit uses select_single from autopilot, don't know what that does12:53
Mirvthat's at least used also by ubuntu-ui-toolkit which has all tests passing, so maybe it'd be worth asking AP people like thomi directly12:54
Laneyok12:56
=== alan_g is now known as alan_g|lunch
om26erpopey, here: https://code.launchpad.net/~om26er/ubuntu-weather-app/fix_db_tests/+merge/19927812:59
om26erpopey, will the CI run on that branch ?12:59
=== cjohnston changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: cjohnston | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: -
popeyom26er: lemme test it first13:06
popeyit wont if nobody approves it13:07
popeytesting now13:09
loolsil2100, Mirv: Feel free to bug me for packaging reviews if I'm around13:17
sil2100lool: thanks :)13:18
asacom26er: sergiusens do we have a breakthrough on those tests yet :)?13:20
om26erasac, I gave something to popey to test13:20
om26erWFM13:20
popeywfm too13:20
asaccool13:20
asac:)13:20
asacnice13:20
* popey commented on the mr13:20
asacwho need to approve so this can land?13:20
asacdid something else land as well on trunk after we updated in image last time?13:21
asaclets get that removed :)13:21
popeymhall119: can probably approve https://code.launchpad.net/~om26er/ubuntu-weather-app/fix_db_tests/+merge/199278 for us13:21
popeywhen he awakens shortly.13:21
sil2100popey, om26er: with this fix in, the weather app AP tests are fixed?13:23
popeythats the plan13:23
om26ersil2100, atleast the one's that I was given link to13:23
sil2100om26er: +beer for you!13:23
om26ersil2100, ;)13:25
sergiusensasac, nothing; the apps with the most failures in testing have been apps that only hd test updates for the past two months13:26
psivaasil2100: i dont have any devices with me at the moment :/13:32
sil2100om26er: sorry to bother you again, but maybe you by some lucky chance have some UT compatible bluetooth devices? ;)13:33
om26ersil2100, I don't have. the one in my car is not supported by Ubuntu touch13:33
sil2100sergiusens: hi! I had some questions related to LP: #123436113:35
ubot5Launchpad bug 1234361 in lxc-android-config (Ubuntu) "Bluetooth panel not getting scan results" [High,Fix released] https://launchpad.net/bugs/123436113:35
sil2100sergiusens: someone reported on the ML that it's not fixed - and I can't find anyone that would have the equipment for re-testing this13:35
sil2100sergiusens: do you have the tools to check if this is fixed with the latest image?13:37
asacsergiusens: ok... please help the fix get into the store and then lets build an image13:38
asacpsivaa: mako still has no resutls for 7113:38
asachttp://ci.ubuntu.com/smokeng/trusty/touch/maguro/71:20131217.1:20131211.2/5521/13:39
asac(sorry if you answered)13:39
asacplars: ^^13:39
psivaaasac: looking at that, sorry dint notice that13:39
asacsil2100: did we promote 70 or do we have that in the bank and already well tested, but hope for 71 to be at least as good and push that?13:40
sergiusenssil2100, I don't have bluetooth headsets, but I can check from the cli and see if I find anything13:42
sil2100asac: from what I see, 71 doesn't have anything more than 70 (no packages updated)13:46
sil2100asac: so let's promote 70 anyway, since 72 will only be ready for tomorrow morning13:47
sil2100asac: if there is of course no cost of promoting an image that is13:47
sil2100It doesn't matter if we promote 70 or 71 basically13:47
sil2100But just to have a clear concious we can promote 70, since that's what popey and om26er used for testing13:48
sil2100lool: ^ could you promote 70 ?13:48
loolsil2100: yup13:49
sil2100Thanks!13:49
plarsasac: just got to my desk, looking now, unless psivaa has already13:50
psivaaplars: the devices is shown as offline soon after flashing13:50
psivaaprobably need to assign another device. mako-06 looks ok to do that13:51
plarspsivaa: wow, it even shows in adb but shows offline13:52
plarspsivaa: yeah, you want me to switch it, or do you want to?13:53
psivaaplars: i am in the sprint,so if you could. that would help13:53
plarspsivaa: ack13:53
psivaathanks plars13:53
plarsrfowler: when you are around, could you take a look at mako-02? It's marked offline for some reason, so nothing can run on it. I've moved all the jobs from it to 06 for now13:56
plarsasac: jobs are restarting now for mako on image 7113:57
sergiusensom26er, popey where's the MR13:59
sergiusens?13:59
popeyhttps://code.launchpad.net/~om26er/ubuntu-weather-app/fix_db_tests/+merge/19927813:59
fginthermorning14:00
loolFYI, image promoted14:01
lool(70)14:01
popeylool: my phone can't see it14:01
ogra_pfft ... gimme 72 ... with 24h clock14:01
sergiusenspopey, om26er ironically, the tests fail on jenkins14:02
om26ersergiusens, that's not related to my code14:02
om26erfalky tests are flaky :/14:02
sil2100ogra_: patience!14:04
ogra_:D14:04
loolpopey: something is wrong with the push mirrors14:05
popeyindeed14:07
loolhmm maybe it's a cron and I had never realized14:08
popeyit's usually immediately after ogra_ says it's published14:09
ogra_lool, nusakan seems dead here14:11
ogra_or at least inaccessible14:11
* ogra_ cant ssh 14:11
loolpopey: it's being fixed14:11
loologra_: you need to use sshebang14:11
loologra_: or update your ssh config to use FQDN14:11
cjwatsonogra_: works fine for me14:12
ogra_since when ?14:12
cjwatsonI don't use sshebang)14:12
loologra_: since some hours ago14:12
ogra_it worked on monday14:12
loolit worked wihtout .canonical.com up til this morning  :-)14:12
ogra_ah14:12
cjwatsonmy config has used nusakan.canonical.com for ages14:12
loolI raised it with #is, but it's intended14:12
* ogra_ fixes /etc/hosts 14:12
ogra_hmm, nothing to fix there it seems14:13
ogra_91.189.89.127nusakannusakan.canonical.com14:13
ogra_but indeed, using the FQDN works14:13
loolcjwatson: ssh nusakan used to work without .canonical.com from the jump host, allowing to have a single .ssh/config entry for *.canonical.com + nusakan; now I have two14:13
loologra_: this is in your .ssh/config14:14
cjwatsonI doubt that /etc/hosts matters.  Fix ~/.ssh/config14:14
loologra_: your proxycommand needs a FQDN14:14
loologra_: nc -q0 nusakan.canonical.com or nc -q0 %h.canonical.com14:14
cjwatsonThe host name is looked up on the proxy jumphost, not on your machine14:14
cjwatsonAnd I'm pretty sure you can't write to chinstrap's /etc/hosts :)14:14
ogra_oh, ok14:14
ogra_thanks, that fixes it14:15
=== alan_g|lunch is now known as alan_g
popey#70 on the way \o/14:21
sil2100\o\14:22
sil2100It's a bird, it's a plane! No, it's #70!14:22
loolIS should have fixed it entirely now14:23
loolI see #70 on trusty14:23
sergiusensom26er, check my comment on the MR14:26
rfowlerplars: I'm going to 1ss in a bit... my car was completely frozen and had to thaw it before the next storm hits14:27
plarsrfowler: ugh, be safe14:29
plarsrfowler: it's no rush on our end, its up and running for now14:30
=== gatox_dr is now known as gatox
om26ersergiusens, pushed. looks better now ?14:46
sil2100sergiusens: did you have a moment to do that cli bluetooth sweep?14:47
sergiusensom26er, let me check; let's also get balloons on now that he's online14:47
sergiusenssil2100, no14:47
sergiusensom26er, looks better, let's wait for ci14:48
balloonsom26er, are you looking @ weather or calendar?14:48
sergiusensballoons, weather14:48
om26erballoons, weather, and its cold.14:48
sergiusensballoons, https://code.launchpad.net/~om26er/ubuntu-weather-app/fix_db_tests/+merge/19927814:48
asacsergiusens: om26er: are we close :)?14:48
sergiusensom26er, it's 35 C14:48
asacthe next image will be cut in 2h approx14:48
asacwould be nice to have it in :)14:48
sergiusensasac, I can't approve MRs for community apps, but if it works, I'm fine with the solution14:49
asacballoons: can you plesae check that out?14:49
asacballoons: we have the fix pending for 100% dashboard gren on mako14:50
asacand i want to get this in opportunistically for next image14:50
balloonsom26er, why are you readding launch and quit?14:50
om26erballoons, these tests assume ~/.local/share/com.ubuntu.weather/Databases/ to be there. It is only created once the app is started14:51
om26erballoons, since we manipulate an already existent database, and if the suite is not able to find one it fails without telling why it failed14:52
balloonsom26er, we have the ability to create a db if one doesn't exist in the test14:53
balloonsI don't believe this changes anything14:53
balloonslaunch and quit would be a regression imho14:53
om26erballoons, remove ~/.local/share/com.ubuntu.weather/Databases/ on your phone and run those tests, they fail with the traceback that's in the dashboard14:54
om26erballoons, then start the app with hands and run those tests again and they will pass14:54
om26erballoons, from what I saw from the test code is that it tries to add a few database entries to an already existent database, it does not create a whole new database14:55
om26erballoons, line 165 in __init__.py14:55
balloonsom26er, I don't doubt your assessment. However, the fix imho is not to go back to the launch and quit thing. Let's just create a db from scratch if we need to14:56
om26erballoons, not really sure if that's going to be a simple task ? these are Qt LocalStorage databases, and there aren't nice APIs for python14:59
balloonsom26er, well I understand the desire to push this in, but it's going to be tough to review beyond I would really like to not see launch and quite come back into this test15:00
om26erballoons, another nasty thought is: clear the db, launch the app, copy the db and ship that with weather app tests. In the test suite make sure to move any old db, copy the shipped one over to the default location, run the tests. Profit!15:02
balloonsom26er, I would be happier with that kind of approach15:04
balloonsif we can't build it on the fly, just ship it with the tests and copy it it to set it up that way15:04
om26erballoons, ok, I can work on that. btw how do these tests get in in the first place ?15:05
om26erah, i would assume CI test runner passed those tests because that db was already there due to not having a clean environment.15:07
sergiusensballoons, popey om26er we need a happrover15:12
balloonssergiusens, om26er is going to make the tweak I asked for15:13
sergiusensballoons, ack, feels lke a recap from an earlier conversation ;-)15:13
balloonssergiusens, :-) I'm sure he works quick and we'll have it15:13
plarscwayne: looks like the attempt  I made last night with custom on the new mako had a strange problem, going to retry it and see what happens, but that's not something I've seen happen before15:34
cwayne plars ah, ok15:36
cwaynei was going to ping you since i saw all 0's, but I thought it might be too early :)15:36
nic-doffayHey, having some CI issue I can't seem to figure, anyone mind taking a look? https://code.launchpad.net/~nicolas-doffay/ubuntu-ui-toolkit/fix-1242647/+merge/19717615:46
asacsergiusens: balloons: om26er: stuff in?15:54
asacpeople are waiting with aggressive uploads15:54
asacthat will brick us for weeks15:55
asaci can hold the line for a bit longer :)15:55
asacbut... :)15:55
om26erasac, just pushing the fix15:56
sergiusensasac, which means it's another hour until it's fully in15:59
sil2100sergiusens: we can wait with kicking the build a bit after the meeting, so hour more is ok16:00
=== Ursinha-afk is now known as Ursinha
sergiusenssil2100, really depends on the infra after the review...16:02
asacsergiusens: yes, thats fine16:03
asacright in time for our standup16:03
=== doanac changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: doanac | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: -
asacto agree on kicking16:03
balloonsom26er, please ping when you need approval16:05
mhall119popey: do you still need somebody to top-approve https://code.launchpad.net/~om26er/ubuntu-weather-app/fix_db_tests/+merge/199278 ?16:14
popeymhall119: i think balloons is on it.16:17
rfowlerplars: mako-02 is up16:23
plarsrfowler: cool, thanks - what was going on with it?16:23
rfowlerplars: it visually seemed just fine16:24
plarsrfowler: odd, adb had it marked offline16:24
rfowlerplars: ya... not sure16:24
balloonsom26er, how's it going?16:44
om26erballoons, https://code.launchpad.net/~om26er/ubuntu-weather-app/db_stab2/+merge/19932116:50
balloonsom26er, :-)16:51
sergiusensballoons, om26er I added 2 comments16:54
sil2100sergiusens, om26er, balloons: so I guess we won't make it with the fix before the meeting ;) ?16:59
balloonsit'll be during the meeting :-)16:59
sil2100robru, kenvandine, cyphermox, lool: meeting?17:00
sergiusensballoons, om26er also need to use the module path to determine where to copy the database from, look at the music app as an example17:00
om26ersergiusens, so I do os.path.join inside os.path.expanduser wouldn't that hurt readability more ?17:01
sil2100asac: coming?17:01
asacsil2100: yep17:01
sergiusensom26er, do it in two steps, but doing a + is not pretty and could concatenate the wrong path17:02
sergiusensom26er, alternatively, just have db_file and drop db_path and use path.dirname17:03
balloonstests do run fine on my device17:06
balloonsI think Sergio has nailed the syntax bits17:06
asacsergiusens: all in?17:06
asac:)17:06
asace.g. can we kick ?17:06
=== gatox is now known as gatox_lunch
looldo I need to start a build?17:07
sergiusensasac, read like 5 lines above; MR got in 10 minutes ago17:07
sergiusensasac, still under review17:07
om26ersergiusens, pushed17:09
asacsergiusens: i dont understand the steps it takes17:11
asacbefore we can build an imge17:11
asachence i am only asking about the final YES its all ready17:11
asacor no, i have to do X,Y still :)17:11
sergiusensasac, ok, fyi, we are at step 117:11
sergiusensasac, I'll get back to you when it's in17:12
asacsergiusens: so what steps are left?17:12
asacand whats the ETA?17:13
asaci need to tell folks that they have to wait with uploads etc.17:13
kenvandinesil2100, https://code.launchpad.net/~ken-vandine/cupstream2distro-config/no_ap_settings/+merge/19932517:13
sergiusensasac, needs to get review approved, tested and then merged by the autobuilders, then build the click; test the click, then push to store; then have it synced17:14
kenvandinesil2100, that should be enough to keep it from block on the check step17:14
sergiusensasac, as I said, an hour after the MR is in17:14
balloonsahh sergiusens sorry.. I thought it was a bit quicker after the MR was in17:14
balloonsI said we'd laid in the MR very shortly, and I didn't think it would take much beyond that17:15
sergiusensballoons, well it's mostly waiting on infra most of the time17:15
sergiusensballoons, I can also blindly push without testing myself17:15
popeysergiusens: ping when you want the click package testing17:16
sergiusenspopey, ack17:16
balloonssergiusens, I'm running the MR with and without db prexisting17:17
sergiusensballoons, the problem isn't going to be click, but when running on a clean env as a deb17:18
sergiusensas a click we cd into the autopilot dir17:18
sergiusensbut as a deb we on desktop, that's not the case17:19
sil2100kenvandine: could I ask for commenting that out instead of removing? :) Otherwise I'm sure I'll forget that the test should be there ;p17:26
kenvandineok17:26
kenvandinesil2100, pushed17:28
sil2100Thank you! Approving17:30
balloonsom26er, sergiusens made some further suggestions. I'm happy with it as he is suggesting. It runs great for me.17:32
balloonsom26er, with those tweaks I think we're good17:32
om26erballoons, done17:34
om26eraah there are more suggestions17:35
sergiusensom26er, balloons also look at how I'm getting the module path http://paste.ubuntu.com/6589958/17:36
sergiusensdidnt' test that part yet though17:36
balloonsom26er, thanks for keeping on this.. It's worth getting it right before it goes in..,17:42
sergiusensballoons, om26er with the changes I proposed, still get all tests passing as click17:46
balloonstested on desktop and device with and without db17:46
Laneywe're releasing u-s-s?17:46
Laneyneat17:46
sil2100Laney: yes \o/17:47
Laneywould have reviewed the outstanding branches if I knew17:48
sil2100Laney: kenvandine handles that, we'll deal with the AP tests later - no need to keep the package waiting, since we're not yet using those extensively17:48
asacballoons: isnt the current code good enough? you can still improve after its out :) ... all we need is code that fixes the tests for now17:48
kenvandineLaney, we can land it again17:48
Laneyit's okay, I'm looking at something there17:48
kenvandineit had just been ages17:48
om26erballoons, sergiusens pushed you suggestions.17:48
om26er*yours17:49
* asac sorry for being inheritently impatient :)17:49
Laneythose tests select a QQuickView17:49
Laneytrying to copy what gallery-app does instead17:49
* asac goes off and waits for another hour17:49
sergiusensasac, it is prone to fail on desktop17:50
sergiusensom26er, did you add the module path detection?17:50
sil2100;)17:50
om26ersergiusens, no, I am not adding that ;)17:50
sergiusensom26er, why?17:51
om26ersergiusens, I don't understand that well and I am not really productive at this O'clock17:51
asacdo we care about desktop? not sure what it means if we say it fails there17:51
sergiusensasac, you don't today, but you will tomorrow and when you do you won't be able to pinpoint the problem as easily17:52
sergiusensasac, I'll abstain and leave it to balloons and popey17:53
asacok.17:53
asacballoons: popey: whats next? did we give up?17:53
sil2100hmmm17:53
* popey is on a hangout, will catch up in a moment17:53
sergiusensasac, they need to approve the MR17:53
sil2100sergiusens, balloons, om26er: so no weather-app update today, right?17:53
popeyor you can summarise in one tweet size line17:53
sergiusensasac, and then I continue17:53
asacsil2100: i hope the answer is that we get it17:54
asacsoonish17:54
balloonssil2100, I was under the impression we could go as soon as the MP landed, but sergiusens informed me it takes a bit.17:54
sergiusenssil2100, asac it will be in17:54
sil2100\o/17:54
sil2100sergiusens, balloons: could you guys poke us once the fix is in and ready to be fetched by a new image?17:55
asacballoons: when will the MP land though17:55
asaci think thats where sergiusens is stuck17:55
sergiusenssil2100, that was the plan all along17:55
balloonssergiusens, om26er  commented with "Not creating the dir in the test code. shutil.copytree calls makedirs so no need"17:56
sergiusensasac, I was reviewing the MR and detecting possible failures for the future; that's all; it's been like this forever with these apps; just want to do it once and not have to go over this again17:56
sergiusensom26er, balloons let me take over the MR17:56
asacok. dont make rocket science out of this17:57
asac:)17:57
asacbut keep engineering standards high17:57
balloonsyes, at this point we're not after quick fixes.. we'd had those for awhile. Being agonizingly detailed on these MP's is what has gotten us to this point17:58
balloonsas in, so close to 100% with core apps not falling over randomly :-)17:58
asacwell, the goal shoujld be to have a fix soon17:58
asacwhatever best solution is a vailable17:58
asacif code is not good enough you can iterate17:58
sergiusensballoons, I fixed that in my proposal17:58
asacballoons: well, so we have big destructuve landings happening later today17:59
asachence i want this in to get to complete cgreen etc.17:59
asaclet me know what is left17:59
asacand then lets do it17:59
balloonssergiusens, you make a new mp or ?17:59
sergiusensballoons, I will resubmit18:01
sergiusensballoons, I don't have privs, but can you resubmit that MR with lp:~sergiusens/ubuntu-weather-app/db_create18:02
=== alan_g is now known as alan_g|EOD
balloonssergiusens, https://code.launchpad.net/~sergiusens/ubuntu-weather-app/db_create/+merge/19933318:04
balloonssergiusens, add a logging statement under if not os.path.exists(db_path): if you do create a db18:04
sergiusensballoons, done18:07
balloonsfirst results on desktop and phone are good.. running again18:11
balloonsjenkins builder is running18:12
=== gatox_lunch is now known as gatox
cwayneplars, looks like touch_custom still has some issues18:19
sil2100balloons: soooo... will it be merged soon :) ?18:24
sil2100\o/18:24
plarscwayne: indeed, that's very strange considering how stable the other builds have been18:29
plarscwayne: anything odd about that mako?18:29
cwaynenot that i know of plars18:31
cwayneplars, hm, looks like it is having an issue with the theming perhaps?18:33
plarscwayne: is it confusing the ap tests somehow?18:34
cwayneplars, i'm not 100% sure, is it installed with --no-backup or -b?18:34
cwaynei wonder if it's looking for an old theme or something18:34
cwayneis there any way to get ssh access to it?18:34
plarscwayne: -b, yes18:35
cwaynemaybe our customizations really is breaking stuff, in which case, good thing we did this!18:35
plarscwayne: if you need ssh access, talk to rfowler or retoaded about getting ssh to a system called "ashes", it hangs off of there18:36
cwaynerfowler, ^18:36
robruwho's doing the latest image build? can somebody ping me when it's done?18:43
asacwe are waiting on MP landing still18:51
asacand store upload i guess18:52
rfowlercwayne: try logging into ashes18:55
cwaynerfowler, what's the ip?18:56
rfowlercwayne: it's in dns18:56
cwayneor is it just ashes.ubuntu-ci18:56
rfowlercwayne: yes18:56
cwaynerfowler, k im in18:56
rfowlercwayne: enjoy18:57
cwaynerfowler, which device is it?18:57
rfowlercwayne: 01ce848e48dfa6a218:58
cwaynehm, the theming stuff is all correct there19:00
cwaynemaybe autopilot doesn't like it for some reason?19:00
balloonssil2100, yes "soon". have  you heard that before?19:01
sil2100:)19:13
balloonssil2100, mp is top approved so it's up to jenkins now ;)19:13
sil2100\o/19:14
sil2100Do you know when it will be available for the image to pull it in?19:14
sil2100sergiusens: ^ ?19:15
sergiusenssil2100, 30'19:15
sergiusensspeeding19:15
cwayneplars, so i see how the tests are failing, i still don't get *why* though, especially since it seems to work when i run it manually19:16
sergiusenssil2100, code hasn't merged yet... it's in the pipes, as soon as that's done, the equivalent of a daily release for a click package happens, goes to the store, then there's a sync to the ubuntu-archives19:19
plarscwayne: I was going to try rerunning some of them, but the number of failures on each one of the failing tests looks to be the same as the last time it ran too19:19
plarscwayne: pretty much the same failures on both 67 and 6819:19
cwayneplars, so it's apparmor denying requests for the custom theme we have19:20
plarscwayne: ah19:20
cwayneplars, but what doesnt make sense to me is that when you actually are using it, the custom theme shows up just fine19:20
plarscwayne: doesn't seem to affect all apps, but it does seem to affect certain apps consistently19:20
balloonsdoanac, my friend. vanguard i trust :-) Can you have a look at this: https://code.launchpad.net/~vthompson/music-app/fixes-1259962/+merge/198601. Jenkins isn't landing the merge for some reason. The builders are failing19:21
cwayneplars, right, but im looking at an emulator instance now with the devel-proposed-customized, and the gallery app is 100% properly themed19:21
cwayneplars, and there's no errors, so something's weird19:22
cwaynesomehow running it from autopilot is making it denied19:22
sil2100sergiusens: ok, let's wait till then19:24
cwayneplars, is autopilot confined in some way?19:24
plarscwayne: yes, it all runs identically to how we run on the other images19:26
doanacballoons: sorry. was grabbing lunch. looking now19:26
cwayneplars, so what's happening is it's getting denied looking for /custom/xdg/data/themes when running from autopilot19:26
cwaynebut when actually using the device, it's allowed19:26
fginthersergiusens, re: https://bugs.launchpad.net/ubuntu-ci-services-itself/+bug/1259568 - I don't have anything scripted yes19:27
ubot5Ubuntu bug 1259568 in Ubuntu CI Services "Create android builder to build AOSP branch from phablet.ubuntu.com repo" [High,Triaged]19:27
fginther'yet'19:28
sergiusensfginther, ack; so what we need to script is rather easy and was sort of stripped out of the build setup when going public, what we can do something similar19:28
sergiusensfginther, if you are working on something; I can review and add your missing parts19:29
sergiusensfginther, are you working on something that sets up the environment and another that works on the build?19:29
fginthersergiusens, right, a script to just set up the env would be adequate19:30
cwayneplars, is there a reason syslog isn't an artifact of the test runs?19:30
fginthersergiusens, I can put something together quick with the steps I performed as a start19:31
sergiusensfginther, sounds good19:31
sergiusenspopey, https://myapps.developer.ubuntu.com/dev/click-apps/147/19:35
plarscwayne: iirc, it should come in the console output stream19:36
plarsdoanac: can you confirm? is that what we get in utah with the 'with device.tail_syslog()'?19:36
popeysergiusens: ack19:37
doanacplars: that is correct19:38
doanacplars: ie - the syslog should be in the jenkin job's console output19:40
doanacballoons: making progress on your issue. looks like the clock drifted on the build slave. working to get it fixed now19:41
cwaynei'm not seeing it there, i should be seeing the apparmor denials19:42
popeysergiusens: approved19:46
doanacballoons: you can top-approve and it should work now19:46
balloonsdoanac, k.. ty!19:46
sergiusenspopey, yay19:51
sergiusenscjwatson, do you mind manually triggering the click sync please?19:52
ChrisTownsendfginther: Hey, to follow on that Unity armhf debugging from yesterday, I know have a test mesa in a PPA we can try.19:56
ChrisTownsend*now19:57
fgintherChrisTownsend, cool, what's the PPA? I'll kick off a build19:57
ChrisTownsendfginther: ppa:rocket-scientists/staging19:58
ChrisTownsendFor armhf using that branch you used yesterday.19:58
ChrisTownsendfginther: Thanks!19:58
sil2100sergiusens: how's the sync?20:12
sergiusenssil2100, it's in, automated sync hits at 11' or 13' after the hour20:13
sergiusenswe are good20:13
asacsergiusens: once an hour?20:13
sil2100\o/20:13
sil2100So I guess we're ok20:14
sil2100cyphermox: hi! Do you have a moment to press the build button?20:14
asacQ: how can we confirm that its really in (like: the image build will pick it up)? (like in archive we use rmadison)?20:14
=== doanac changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: cihelp | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: -
sergiusensasac, it's in click_list and livecd-rootfs has a hook that picks it from there20:18
cyphermoxsure20:18
sergiusensasac, so if it doesn't, we have a mitm :-)20:18
asackk20:18
asacjust checking :)20:18
asaclets go then20:18
asacif you confirmed it like that20:19
ChrisTownsendfginther: Could you send me the link to the build whenever you have it kicked off?20:24
cwayneplars, alright, so we've got an anwser for why all the clicks were failing on touch_custom20:26
cwaynestill unsure of gallery-app though20:26
cyphermoxsil2100: done20:26
fgintherChrisTownsend, http://s-jenkins.ubuntu-ci:8080/job/unity-trusty-armhf-ci/61/console20:27
ChrisTownsendfginther: Thank you20:28
thomicihelp: I just noticed today that the job that updates the autopilot documentation has been broken for a while - look s like a networking issue: http://s-jenkins.ubuntu-ci:8080/job/autopilot-docs-upload/20:29
thomiI wonder if someone could take a look for me please?20:29
fgintherthomi, looking20:29
thomiMy guess is that since the machine moved lab, the firewall rules that IS look after will have to be changed20:30
thomibut that's just a guess :)20:30
sil2100\o/20:30
sil2100cyphermox: thanks!20:30
sil2100See you tomorrow everyone20:30
fgintherthomi, timing of when the job started to fail would support that guess20:30
thomifginther: yeah20:31
fgintherthomi, I'll get a ticket filed to update the firewall20:31
thomithanks20:31
thomifginther: I assume once the FW is fixed you'll re-run the job?20:31
fgintherthomi, I can if I remember to do so :-)20:32
fgintherit usually takes them a while to respond to the request20:32
thomifginther: thanks - I guess you'll run it to test the FW rules anyway20:32
thomiyeah, I usually hassle them on IRC :)20:33
fgintherthomi, is maitri.canonical.com the correct destination?20:40
thomifginther: yes20:43
sergiusensfginther, btw can we get automerge on lp:usensord? and some upcoming daily release prep?20:59
fginthersergiusens, I can help with setting up automerge, sil2100 and company can help with the daily-release prep21:03
sergiusensfginther, sure, just not sure how to make the request these days :-) seems the team is pending a merge :-P21:04
fginthersergiusens, this will take more time than usual, lp:usensord requires dh-golang on the host which isn't available for raring (which the build machines are currently running)21:30
sergiusensfginther, we just want trusty21:34
sergiusensoh21:34
sergiusensstill raring?21:34
fginthersergiusens, it's a build host thing: http://s-jenkins.ubuntu-ci:8080/job/usensord-trusty-i386-ci/2/console21:36
fginthersergiusens, my plan was to move them to saucy eventually, this appears to be the trigger to start that process21:37
cyphermoxfginther: daily-release-executor?21:41
sergiusensfginther, it's not super hard to backport dh-golang btw; just a bunch of perl scripts21:41
fginthercyphermox, I was waiting for the job to finish before change the connection method, looks like that just happened21:42
fginthercyphermox, should be back in a few minutes now21:42
fginthercyphermox, it's back up now21:48
cyphermoxthanks21:49
fginthersergiusens, can I get back to you tomorrow on lp:usensord?22:00
fgintherChrisTownsend, the unity armhf build passed with your PPA22:01
sergiusensfginther, sure22:17
sergiusensfginther, I will have around 5 more jobs using dh-golang aprox fwiw22:17
sergiusenssoon more to come22:17
thomifginther: got a second?22:26
fgintherthomi, yep22:26
fgintherthomi, I do have a hard stop in 30 minutes22:27
thomifginther: before I can get the new autopilot released to distro, I need to prove that it doesn't break anything...22:27
thomiin the past you made us a jenkins job to do this, I believe?22:27
thomiI'm trying to figure out how I can do this again, but there are so many 'autopilot' jenkins jobs out there I'm not sure what I should be looking at22:27
fgintherthomi, yes, we did that for the 1.4 effort22:27
thomifginther: right22:28
thomido you remember which job it was?22:28
fgintherthomi, found it: http://s-jenkins.ubuntu-ci:8080/job/autopilot-trusty-master/22:31
thomisweet, let's see....22:31
fgintherthomi, it worked by pulling the latest python-autopilot from the experimental ppa22:32
fgintherthomi, is this for python3-autopilot?22:32
thomifginther: sure22:32
thomifginther: actually, I'm not sure22:32
thomifginther: this is for whatever you use normally22:32
fgintherpython-autopilot in that case22:33
thomiyeah22:33
thomiso target_branch is the project whose tests I want to run... what do I put in landing_candidate?22:34
fgintherlanding_candidate is actually the value that holds the branch to test, target_branch is the 'trunk' branch22:35
fginthersorry, that's confusing22:35
fgintherlanding_candidate is the branch to test, target_branch is the 'trunk' branch. They can be identical22:36
thomifginther: ok.. and this won't actually land anything though, right?22:37
fgintherthomi, nope, it just buids and tests22:37
thomiawesome, thanks22:38
fgintherthomi, hope that helps22:38
thomifginther: testing it out now22:38
thomifginther: are there still only 4 test runners in the pool? Just wondering how far I can push this22:40
thomidon't want to starve the rest of Canonical22:40
fgintherthomi, yes still have just the 422:40
thomiok, thanks22:40
Laneythomi: if you have some time, can you cast a look at the autopilot tests in lp:ubuntu-system-settings on the device please? All but the most basic ones are broken & none of us has the autopilot smarts to figure it out22:46
thomiLaney: got a link to a failure page?22:47
Laneythomi: umm, a pastebin, don't know if there was an official run22:48
Laneythomi: http://paste.ubuntu.com/6586198/ was one robru did yesterday22:48
* thomi looks22:49
LaneyI guess it's some weirdness around how vrruiz set it up originally22:49
thomiok, it looks like maybe the object names have changed or something. I can add it to my list of things to look at, but it's unlikely I'll get to it before tomorrow morning...22:50
Laneyit passes on desktop22:50
Laneythat's probably an important piece of information that I forgot to say :-)22:50
thomiinteresting22:58
robruLaney, thomi : yep those failures are on mako23:20
Laneymy brain is too small to understand it23:22

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!