Mirvcihelp why does q-jenkins say it's being shut down?05:38
didrockshey Mirv, are you building Mir? (is cowbuilder back up?)07:19
Mirvdidrocks: I was, but there is a test failure07:24
Mirvdidrocks: a fix is being merged now, but q-jenkins states it's shutting down to which I haven't yet gotten a response from cihelp07:24
MirvI don't know about the cowbuilder07:25
didrocksMirv: ok, you didn't see issues in the previous cu2d run?07:26
didrocksev: vila: seems no cihelp, hence the direct ping (excuse us), but the cu2d jenkins is marked for shutting down, any idea?07:30
Mirvdidrocks: so actually mir visited my mind on Saturday (shame on me) when I pushed the cu2d mir build and back then cu2d was fine. today I checked the build result + noticed q-jenkins is marked as shutting down ie further builds blocked.07:35
didrocksMirv: ok, so cowbuilder fixed, great!07:35
Mirvdidrocks: ken and robru marked that because of AP machine problems they didn't continue on mir on Friday, although they could have just built and skipped check jobs I believe back then07:35
didrocksyeah, let's see for today what's up with q-jenkins…07:36
Mirvdidrocks: another note that mir stack build job was pending with arm64 build jobs, not sure why those weren't auto-ignored. but we'll see it again soon enough when q-jenkins is operational again.07:37
didrocksev: vila: ~desktop-team/cupstream2distro was **weeks** old07:51
didrocksat least 6-8 weeks07:51
didrockson jatayu07:51
didrocksMirv: that's why arm64 wasn't ignored07:52
didrocksasac: I can't tell when we are going to do releases again, see issues like that ^ (we are in an unknown state)07:52
* didrocks worries as well about the network, seems bzr pull took ages (not sure if it's launchpad or the local CI network)08:00
tsdgeoswaht's the new sjenkins ip?08:10
Mirvtsdgeos: seems, but you can also configure the DNS08:15
tsdgeosMirv: yep, i saw the part of the email to do that, but just changing the hosts file seems easier :D08:16
tsdgeosmanual dns ftw!08:16
viladidrocks: and I suppose there is no health check to warn you about that ?08:22
didrocksvila: to ensure that your home directory was restored with a version that is 3 months ago? No really08:23
viladidrocks: we all highlight 'cihelp' no need for a direct ping08:23
didrocksvila: seems you didn't have an health check either to ensure that what you migrate are consistant with latest in production?08:23
viladidrocks: could we get a bit more professional and stop pointing fingers, I can do that too but I don't think it's productive especially during fire fight08:24
didrocksvila: well, seems like you do something similar with "urgh", "we should stop doing that" and other expression like that without providing solutions08:25
didrocksI wish just that we could get back to work after a week and half, and I don't think we didn't help you08:25
viladidrocks: you did help08:26
viladidrocks: but not by yelling08:26
didrocksso, can you help in keeping/figuring out why jenkins want to shut down and ensuring that we do have the latest code in production?08:27
viladidrocks: and if I wasn't fire fighting an undocumented engine I would have more time to implement solutions I did propose08:27
didrocksvila: see, pointing fingers…08:27
viladidrocks: yeah, "yelling" is finger pointing at you, nothing good comes out of panic mode so please just stop, let's get back to facts and solutions08:29
didrocksvila: so, facts is that ~desktop-team wasn't up to date with latest in magners08:29
didrockscan you help ensuring it's now the case08:29
didrocksand no other thing like cowbuilder are forgotten between the 2 copies?08:29
didrockssecond fact: can you figure out why jenkins is setup to stop?08:30
didrocks(and so, we can build anything)08:30
vilaI asked why jenkins was in shutdown mode on Friday, didn't get an answer08:31
vila"no other thing like" is what I meant by undocumented, my crystal ball doesn't know either, do you ?08:31
didrocksvila: well, I think making a diff between old machine, at least in system path, would have been a first step for the transition08:32
didrocksvila: like as well asking for the tool prerequisite for the undocumented part from stackholders08:32
didrocksvila: on jenkins, so, what's the next step?08:32
vilaI don't care about "would have", do you have "will" ?08:32
didrocksvila: "will" you make a diff between old machine and new one then?08:33
didrocksvila: and what are going to you do about jenkins?08:33
didrocksif you prefer future :)08:33
vilayes I do, thanks08:35
vilaI'll ask again why it is in shut down08:35
vilaand since I won't get an answer I will restart it08:36
vilawhich is not the smartest thing to do08:36
didrocksvila: so, in term of speaking futures, please apply this to you and stop telling "this stucks", "this should have never been done like that", "this was undocumented". That will help both sides, thanks :)08:36
didrocksvila: +1 on restarting if needed, everything is blocked (the new planned jobs are stucked and don't move)08:37
didrocksvila: from the list of stuck jobs, all the cu2d can be killed08:38
didrocksnot sure about:08:38
viladidrocks: I said it *is* undocumented and this makes it harder to check after the 1ss move and harder to get it back up. That how I explain to the stakeholders why it's not up yet08:40
didrocksvila: but you keep ranting about everything single piece of the architecture, which isn't helpful either and doesn't fix anything08:40
didrocksanyway, let's move on, I hope the list above can help you ^08:41
viladidrocks: you're still pointing fingers, do we need to solve that with a third party ?08:41
didrocksvila: come on, I was just showing you that the issue is not unilateral, but it seems you want to continue when I'm proposing to move on and close the topic08:42
didrocksvila: and that's why I started to list the jobs I don't know if you can kill and you bring back the topic :/08:42
didrocksanyway, *sigh*08:42
* vila moves on08:42
vilaautopilot-nvidia needs a new power supply08:44
vilathe alternatives are 1) setup a new host with any nvidia card 2) disable autopilot-nvidia in cu2d-config08:45
didrocksvila: go the road you prefer, you are handling the production of this. We can go over and hope that if you choose 2), there will be no real card regression (which is unlikely until we release an unity7)08:46
Saviqcihelp: hey guys, seems the otto runner for unity8-autolanding often fails with a lock password entry... https://jenkins.qa.ubuntu.com/job/autopilot-testrunner-otto-trusty/682/artifact/results/autopilot/artifacts/unity8.shell.tests.test_hud.TestHud.test_show_hud_button_appears%20%28Desktop%20Nexus%2010%29.ogv08:53
Saviqis that known? resolved?08:53
vilaSaviq: not known to me08:57
Saviqvila, http://s-jenkins:8080/job/autopilot-testrunner-otto-trusty/683/ seems to be the last job that failed like this, maybe it got resolved since08:59
Saviqvila, or maybe simply one of the machines is locked...09:00
Saviqno, seems all the jobs ran on ps-radeon-hd835009:00
Saviqand now are green09:00
vilaSaviq: good, no time to investigate that one right now, especially if you think it's back to green09:01
viladidrocks: you did update q-jenkins:~desktop-team/cusptream2distro right ?09:02
Saviqvila, yeah, will let you guys know09:02
didrocksvila: yeah, I had to to unblock Mirv (but that was before seeing jenkins was getting shut down)09:02
didrocksvila: only thing I did this morning on the machine09:02
* didrocks spent some time to look if there was a bug in the code first09:03
viladidrocks: sorry about that09:03
didrocksno worry09:03
vilarestarting q-jenkins09:04
viladidrocks: I won't do a diff between two systems that had different purposes and now have  new different purposes, I would have no idea about how to interpret the diff (and I'm not even sure I know how to compare two systems anyway)09:10
viladidrocks: q-jenkins is back09:10
didrocksvila: hum, so we will go on try and error procedure then?09:10
didrocksvila: nice! want me start a stack?09:10
viladidrocks: yes please, as gently as possible, we're still on thin ice09:10
* didrocks starts just one stack: Mir09:11
didrocks(no AP for that one)09:11
didrocksMirv: starting Mir ^09:11
didrocksvila: I'll let you decide on the autopilot-nvidia side09:12
didrocks* Mir started *09:12
Mirvdidrocks: thanks, although the test fix is not yet in https://code.launchpad.net/~vanvugt/mir/fix-1252144.trunk/+merge/19554709:12
didrocksMirv: at least, we'll ensure we can start building something :)09:13
Mirvyes, and it'll be nice to see no arm64 stuckness09:13
didrocksvila: confirming the network is really really slow: http://q-jenkins.ubuntu-ci:8080/view/cu2d/view/Head/view/Mir/job/cu2d-mir-head-1.1prepare-unity-system-compositor/47/console09:16
didrocksFetched 12.0 MB in 1min 1s (196 kB/s)09:16
didrocksFetched 11.8 MB in 1min 48s (110 kB/s)09:24
didrockson another job09:24
didrocksand some error: http://q-jenkins.ubuntu-ci:8080/view/cu2d/view/Head/view/Mir/job/cu2d-mir-head-1.1prepare-unity-greeter-session-broadcast/119/console09:24
didrocksev: coming?09:31
evon my way in09:31
viladidrocks: https://code.launchpad.net/~vila/cupstream2distro-config/no-nvidia/+merge/19556309:31
Mirvcihelp please check I'm seeing jenkins network problems and timeouts at merge proposal https://code.launchpad.net/~vanvugt/mir/fix-1252144.trunk/+merge/195547 - especially concerning connecting naartjie09:44
didrocksvila: just to confirm, the radeon machine is now working? (as it's in the config?)09:51
viladidrocks: well, last I checked it was... you mean the qa-radeon-7750 right ?09:52
didrocksvila: yep, ok, good, deploying with it, thanks09:52
didrocksMirv: sil2100: think about reconfiguring your ~/.cu2d.cred with q-jenkins.ubuntu-ci btw09:53
Mirvdidrocks: did that already09:53
didrocksvila: reprovisionning a recent trusty otto container: http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-setup_otto/23/console09:57
sil2100didrocks: updated \o/09:58
didrocksand removed nvidia from it09:58
didrockssil2100: great! :)09:58
asacdidrocks: the tests we usually would have run on those nvidia machines are AP tests, right?09:58
ogra_=== Image r23 building ===09:58
didrocksasac: yeah09:59
didrockscihelp: ok, can't reprovision the otto machines, seems it can't download the iso (it tries from q-jenkins)09:59
didrockssee http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-setup_otto/label=qa-intel-4000/23/console09:59
didrocks+ wget --progress=dot:mega http://q-jenkins//iso/trusty//trusty-desktop-i386.iso -O autopilot-trusty-setup_otto_label=qa-intel-4000.QgzUAk10:00
didrocks--2013-11-18 09:57:35--  http://q-jenkins//iso/trusty//trusty-desktop-i386.iso10:00
didrocksResolving q-jenkins (q-jenkins)...
didrocksConnecting to q-jenkins (q-jenkins)||:80... connected.10:00
didrocksHTTP request sent, awaiting response... 404 Not Found10:00
didrocks2013-11-18 09:57:36 ERROR 404: Not Found.10:00
didrocksogra_: great!10:00
didrocksMirv: we'll have to wait on the provisionning before running the Mir stack (all setup done apart from this FYI) ^10:00
* asac wonders why it doesntloads from q-jenkins10:00
didrockswe should do a wrapper one day for wget, that a 404 doesn't create an empty file…10:01
=== vrruiz_ is now known as rvr
asacdidrocks: do you know how this normally works?: e.g. who ensures that there is an .iso on q-jenkins?10:01
didrocksasac: I think QA cached it some times ago for their smoke testing and they had a process for that10:02
didrocksasac: maybe it's a jenkins job, I don't really know10:02
didrocksah, also:10:02
didrocksErrors were encountered while processing:10:02
didrocks grub-pc10:02
didrocksE: Sub-process /usr/bin/dpkg returned an error code (1)10:02
didrockson radeon10:02
didrocksvila: I guess this is the conffile you changed ^10:02
asacwonder why they dont use squid10:03
viladidrocks: looking10:05
viladidrocks: first attempt at reprovisioining since the move probably10:06
didrocksvila: yeah, confirmed10:06
viladidrocks: /iso content is good, searching for :80 config :-/10:09
xnoxCan this be merged please? https://code.launchpad.net/~xnox/ubuntu-keyboard/libpinyin4/+merge/19385910:12
xnoxthe r101 was a merge from trunk to resolve a merge conflict.10:12
xnoxand the bot now thinks it needs to be re-reviewed.10:12
didrocksxnox: I approved it for you :)10:15
didrocks(seeing it was already reviewed)10:15
retoadeddidrocks the URL for the ISO works now10:20
didrocksretoaded: thanks! vila: did you look at the grub-pc thing or should I just retake a setup snapshot?10:21
vilaretoaded: thanks, haven't figured that one out, there is no /var/www/iso/trusty on m-o yet the log says there was one a couple of days ago 8-/10:22
viladidrocks: what grub-pc stuff ?10:22
didrocks11:02:38 didrocks | ah, also:10:23
didrocks11:02:40 didrocks | Errors were encountered while processing:10:23
didrocks11:02:40 didrocks |  grub-pc10:23
didrocks11:02:40 didrocks | E: Sub-process /usr/bin/dpkg returned an error code (1)10:23
retoadedvila, it is an alias setup within apache; see /etc/apache2/conf.d/isos.conf10:23
didrocks11:02:43 didrocks | on radeon10:23
didrocks11:02:47     asac | ok10:23
didrocks11:02:51 didrocks | vila: I guess this is the conffile you changed ^10:23
viladidrocks: the only ring it bells was to pin a kernel version... hold on reading10:23
didrocksvila: yeah, that's why I'm pinging you, in case you changed a conffiles and grub isn't happy about it10:24
vilaretoaded: thanks10:25
viladidrocks: ok, no I've restored everything to pristine after giving back qa-radeon-7750, thanks for thinking about that10:25
vilaasac: the isos are downloaded by otto, the magic links were missing server side, just fixed10:26
didrocksvila: ok, so you think we can ignore this grub error then?10:27
viladidrocks: first time I see this grub error :-/10:29
viladidrocks: oh, locally modified...hmm, may be a leftover from debugging by the kernel team, I remember they had to force things a bit at one point to get a specific kernel10:30
didrocksvila: ah, so you are reverting back to vanilla? \o/10:30
viladidrocks: sounds like the safer bet10:31
didrocksand then I guess sudo apt-get install -f10:31
didrockskeep me posted10:31
didrocksI'll rerun a setup() then to try to get your image10:31
didrocks(on both)10:31
evretoaded: good morning10:33
viladidrocks: reproduced with 'apt-get install grub-pc', confirmed it complained about the comment10:33
retoadedev, it is definitely morning. I'm not sure about the "good" part though10:34
cjwatsonvila: what's this?10:34
cjwatson<- grub maintainer10:34
evretoaded: ev@jatayu:~$ time curl https://launchpad.net -> 21s; ev@mayura:~$ time curl https://launchpad.net -> 0.7s10:34
evany idea what's going on?10:35
vilacjwatson: http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-setup_otto/label=qa-radeon-7750/23/console10:35
retoadedev, no but will tale a peek10:35
vilacjwatson: caused by a comment in /etc/default/grub10:35
cjwatsonvila: remind me how I set up DNS for .ubuntu-ci?10:36
vilacjwatson: https://wiki.canonical.com/UbuntuEngineering/QA/VPN10:36
vilacjwatson: you had one working before ? NM or openvpn ?10:37
cjwatsonNM, I'll just update it now10:37
cjwatsonah, right, yeah, that error isn't my problem :)10:39
cjwatsonalthough the update from 2.00-19ubuntu3 -> 2.00-20 shouldn't have triggered a conffile prompt ...10:39
cjwatson<cjwatson@amber ~>$ diff -u <(deb-extract-file grub2-common_2.00-19ubuntu3_amd64.deb /usr/share/grub/default/grub) <(deb-extract-file grub2-common_2.00-20_amd64.deb /usr/share/grub/default/grub)10:41
cjwatson<cjwatson@amber ~>$10:41
vilacjwatson: but it did, I installed the maintainer version which removed the comments, hold on10:42
cjwatsonno substantive change to postinst either10:43
vilacjwatson: scratch, that,10:43
vilawas looking at the wrong machine10:43
vilaon qa-radeon-7750:/etc/default/grub comments after the GRUB_DEFAULT line10:44
cjwatsonI did change config a bit, but that change would have been a no-op unless GRUB_CMDLINE_LINUX_DEFAULT had been removed from the config file10:45
cjwatsonAnyway, I probably can't debug any further unless there's some way to see the diff between the versions here10:46
vilacjwatson: ctrl-alt-del10:46
vilacjwatson: I did say: keep the installed version and the comments are still there10:47
cjwatsonAnd of course if you change package-supplied configuration files directly then it's inevitable that you'll occasionally have to resolve conflicts10:47
vilacjwatson: (since I was looking at the wrong machine and stop seeing the comment I revised my memory and said I did chose the maintainer version, clearly not the case)10:47
cjwatsonCould you show me "diff -u /usr/share/grub/default/grub /etc/default/grub" then?10:47
vilacjwatson: ha ! great: here is the *needed* change:10:48
vila-GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"10:48
vila+GRUB_CMDLINE_LINUX_DEFAULT="quiet swapaccount=1"10:48
cjwatsonOK, so you can fix this by making that change in debconf10:48
cjwatsondpkg-reconfigure grub-pc10:48
cjwatsonThough it should have done that automatically ...10:49
cjwatsonBefore you do the above, "debconf-show grub-pc"10:49
vilacjwatson: http://paste.ubuntu.com/6436818/10:50
cjwatsonHuh, so that's already in debconf10:50
vilacjwatson: because I did 'apt-get install grub-pc' ? That change was made.... when the machine was setup, say, two weeks ago ?10:51
vilacjwatson: no need for dpkg-reconfigure grub-pc then ?10:52
cjwatsonWas the change originally made directly to /etc/default/grub, or via some user interface?10:52
* vila forgot dentist, needs to run10:52
cjwatson(Either should have worked, but I need to know how to reproduce this)10:52
viladirectly in file10:53
cjwatsonvila: Oh, but you say there was a comment added above it?10:54
cjwatsonI guess that would have been sufficient to confuse matters10:54
cjwatson(I'd hoped for the diff pastebinned, not just a part of the diff pasted into IRC)10:54
Saviqvila, it's back again... https://jenkins.qa.ubuntu.com/job/autopilot-testrunner-otto-trusty/691/ :/10:55
Saviqdidrocks, session locking under otto ↑, ideas?10:56
didrocksSaviq: I guess it's something for cihelp ^10:56
Saviqdidrocks, yeah, vila responded to cihelp before, asked me to follow up if it happened again10:57
didrocksSaviq: hard to tell without having access to kvm, which I don't anymore (for good reasons btw ;))10:58
Saviqdidrocks, mhm, thanks10:58
Saviqcihelp, so yeah, otto runner has issues with session getting locked in some jobs - all autopilot videos are either black or show the screen lock password entry, and obviously the tests fail11:00
* Mirv back in 1h11:01
retoadedev, leworks@jatayu:~$ time curl https://launchpad.net -> real0m0.611s11:12
evretoaded: whoop. What was the problem?11:12
retoadedev, /etc/resolv.conf. utah seems to modify it to put a nameserver first in the order11:13
retoadedso resolving launchpad.net tries that first then fails over to the next nameserver in order11:14
retoadedI removed it11:14
retoadedev, that could just be remnants of an older utah install and was sync'd over from m-o11:15
evugh, I knew there'd be something I hadn't checked11:18
vilacjwatson: sorry, being late didn't help, here is the full diff http://paste.ubuntu.com/6436934/11:21
vilacjwatson: and I probably missed that GRUB_CMDLINE_DEFAULT when apt-get install presented the diff :--/11:26
cjwatsonI tried a saucy chroot, apt-get install grub-pc vim, edit /etc/default/grub to apply that diff, edit /etc/apt/sources.list to upgrade to trusty, apt-get update, apt-get install grub-pc - no prompt11:26
vilacjwatson: so, reformulating to ensure I get it right: pat-get saw a conflict on GRUB_CMDLINE_DEFAULT and warned. It won't do it next time correct ? (Unless a new change introduce a new conflict)11:27
cjwatsonI don't think so but I can't tell for sure because this is a bit out of the ordinary11:27
cjwatsonIs the immediate problem dealt with?  If so I propose ignoring this until/unless it happens again :-)11:27
vilacjwatson: good enough for me, will tell you if I encounter such a case11:28
cjwatson(Or until/unless somebody can get me a reproduction recipe I can run locally)11:28
cjwatsonFWIW I suspect the conflict would have related to the comment and not to the GRUB_CMDLINE_LINUX_DEFAULT change, since the latter appears to have been correctly carried over into debconf11:29
cjwatsonCan't prove it right now though11:29
vilacjwatson: the weird thing is that qa-intel-4000 has the same changes and didn't encounter the issue: http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-setup_otto/label=qa-intel-4000/23/console11:30
ogra_== Image r23 DONE ===11:31
cjwatsonvila: Ah well, let's see if it happens on the next grub2 change11:32
* popey updates to 2311:32
cjwatsonProbably not worth ratholing on now11:32
vilacjwatson: agreed11:33
viladidrocks: so, grub-pc issue cleared for now, re-trying an otto setup11:40
didrocksvila: keep me up to date!11:41
viladidrocks: images downloaded on both, progress11:42
viladidrocks: and both failed later, reading the logs11:43
vilalxc_container: command get_cgroup failed to receive response >-/ not sure that's the trigger but...11:45
viladidrocks: nope, that one should be harmless, happened during the last successful run11:53
=== MacSlow is now known as MacSlow|lunch
tsdgeosguys, i had a test segfault in jenkins autolanding while it passed fine on jenkins CI and the crash seems to be no related to my changes, we've just re top approved the change but t1mp thinks its a good idea to mention here, in case it happens again to someone else which may mean faulty hardware or something else12:01
tsdgeosi.e. not asking for help, just saying this happened in case it repeats12:01
vilatsdgeos: thanks, cihelp ^12:01
t1mp^the MR that had issues https://code.launchpad.net/~aacid/ubuntu-ui-toolkit/ima_do_not_filter_disabled/+merge/19483012:03
didrocksvila: otto should be ready then? ;)12:06
Mirvtsdgeos: yep, I think it's good to report the hitches so that they can be tracked down one by one12:07
viladidrocks: :-/ apparently not despite update_host returning 0 (AFAICS) the jobs fail12:08
didrocksah ok :/12:08
viladidrocks: last run succeeded, while I'm happy to report, I'm unhappy to not understand why :-/12:10
viladidrocks: I did some apt-get upgrade/dist-upgrade manually though...12:10
viladidrocks: so let's move on the next step12:11
Mirvcihelp more 'naartjie' host issues (ssh connection fails) https://code.launchpad.net/~thomas-voss/process-cpp/add_fork_and_run_facilities/+merge/19484212:11
viladidrocks: I also noticed that autopilot-nvidia last otto setup run failed before the move... we may have other issues to investigate when we get a nvidia replacement :-/12:11
viladidrocks: so, what's the next step ? Do you have a small stack in mind that is expected to succeed ?12:15
didrocksvila: was discussing in a hangout12:39
didrocksvila: so, otto should be up for those 2 machines? Mirv: can you launch Mir now?12:40
didrocksMirv: Mir & co of course :)12:40
didrocksif the fix branches were merged12:40
didrocksvila: on the failure: I know that sometimes, the machine exits before the job finishes to process12:40
didrocksso you see the java stack12:40
didrocksand it's marked as failed12:40
Mirvdidrocks: ok!12:45
Mirvthe branch is still not merged, though12:45
MirvI'm wondering if I could help from ci_help to get ETA if it'll work soon, or merge manually12:46
psivaaMirv: naartjie is resolved ok from s-jenkins but not from the slave cyclops nodes12:46
psivaaretoaded or fginther could add more information12:47
didrocksMirv: yes, please ;)12:47
viladidrocks: otto setup succeeded so yeah, that should mean otto is up, but I'd rather not rely on that without at least one successful run of one job12:48
Mirvdoing manually for now, and good if the naartjie problems are on radar12:48
retoadedpsivaa, which cyclops node(s)? node-06 resolves it.12:48
didrocksvila: yeah, we need to be able to start the Mir stack (pending Mirv to get merged/merge the Mir branch) ^12:49
viladidrocks: ok12:49
vilaretoaded: node07 is the only one my test script can't reach12:49
retoadedvila, ack.12:49
psivaaretoaded: ohh? http://s-jenkins:8080/job/process-cpp-trusty-armhf-autolanding/11/console says 'Unable to connect to naartjie:http:'12:50
retoadedvila, it's possible node07 is not up.12:50
psivaawhich uses node-0612:50
vilapsivaa: ha, good, retoaded, my test script exercises ssh only12:51
vilaretoaded: from my desktop12:51
vilaMirv: let me know when you start so I can monitor12:52
retoadedpsivaa, I don't think that is an issue with being able to connect to naartjie but, instead, not finding what it is looking for:  http://naartjie trusty/ InRelease has spaces in it12:52
retoadedbut that mey not be the actual URL12:53
Mirvvila: I kicked the https://code.launchpad.net/~thomas-voss/process-cpp/add_fork_and_run_facilities/+merge/194842 now for example12:55
retoadedpsivaa, because the other job https://jenkins.qa.ubuntu.com/job/process-cpp-trusty-armhf-ci/16/console has the URL  http://naartjie/archive//head.mir/trusty/InRelease and InRelease does not exist under  http://naartjie/archive//head.mir/trusty12:55
vilaMirv: err, on s-jenkins ?12:56
psivaaretoaded: yea, that's true. so an issue with the job config i guess.12:56
retoadedpsivaa, possibly12:57
viladidrocks, Mirv: I was asking for some validation on q-jenkins, can't we do that ?12:57
psivaaretoaded: thanks, i'll dig in if there is anything that is obvious12:57
didrocksvila: Mirv is going to run the Mir and dependant stacks, which will trigger that job12:57
didrocksvila: but the CI isn't able to merge one of the upstream branch it seems which enables to build Mir12:58
Mirvvila: ok I'm just not clear which issue you're referring to. I also kicked the Mir stack now running, after merging the CI failing commit manually.12:58
didrocksso Mirv told he will merge manually the mir branch12:58
didrocksand run the stacks :)12:58
didrocksand, already done12:58
vilaha ok, sorry, was confused12:58
didrocksMirv: then, you take care of the platform stack? (so that it does build against latest Mir)12:58
Mirvvila: yeah platform + unity-mir (and recheck unity-system-compositor too)12:59
didrocksMir Mirv… this is so confusing! :)12:59
didrocksthanks ;)12:59
Mirvhopefully then there'd be something ready for robru to test for example12:59
didrocksMirv: is the network seems to still be slow btw?12:59
viladidrocks: hehe, yeah, funny, didn't fall for that Mir Mirv one ;)12:59
Mirvdidrocks: yes, the prepare jobs seem to take ages13:00
Mirv"Fetched 12.0 MB in 2min 43s (73.2 kB/s)"13:00
* didrocks goes for a run outside13:01
Mirv512kbit/s not really 'internal network' speeds13:01
* vila sighs, and we postponed the upgrades for the monitoring we had...13:02
vilabah, would probably won't help, I doubt we are monitoring the part that is acting right now13:02
=== alan_g is now known as alan_g|lunch
vilaMirv: unity-system-compositor failed to build on amd64 and i38613:09
vilaMirv, didrocks: do we have something simpler to try ?13:09
=== psivaa is now known as psivaa-lunch
=== MacSlow|lunch is now known as MacSlow
vilaMirv: ?13:16
vilaMirv: is there a way to manually trigger a simple stack on q-jenkins ?13:16
Mirvvila: unity-system-compositor in itself is not too big. but the problem there is maybe related to boost transition, not these other problems?13:17
Mirvvila: the prepare jobs now completed without timeouts13:17
vilaMirv: no idea :-/13:17
vilaMirv: I try to stay focus on the otto nodes if I can :-/13:18
Mirvvila: or maybe it's CI after all, I can install libboost-all-dev just fine locally13:18
vilaMirv: the failed to build are on lp, so not related to otto is my point :-/13:19
=== josepht changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: josepht | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: Some services are down (1SS move)
Mirvvila: ok, but why did you ask about it failing then, or what were you searching to try?13:19
vilaMirv: ctrl-alt-del13:19
=== gatox is now known as gatox_lunch
vilaMirv: I asked for a simple stack to run and validate the otto stuff, you pointed to an MP and I didn't make the connection. When mir-head came, I thought I got that job anyway and looked at how it went13:20
vilaMirv: it ended up failing *before* reaching otto IIUC, so I'm now asking if have a simpler way to validate otto13:21
Mirvvila: ok, so you want to see check job running?13:21
Mirvvila: sdk stack now running just the check job13:21
vilaMirv: thanks, sorry for not expressing my need more clearly13:22
Mirvexcept that it failed to start13:22
vilaMirv: http://q-jenkins:8080/job/cu2d-qa-head-2.2check/358/ ?13:23
Mirvvila: qa stack now running13:23
Mirvand yes,that13:24
vilaMirv: thanks13:24
Saviqjosepht, we've see https://jenkins.qa.ubuntu.com/job/unity8-trusty-armhf-autolanding/86/console a few times now - connection time out to naartje13:24
Saviqjosepht, that expected / known?13:25
josephtSaviq: looking, I know there've been some reports of networking issues in the lab so perhaps this is related.13:26
vilaMirv: http://q-jenkins:8080/job/autopilot-trusty-daily_release/513/console hu ho, autopilot-nvidia and qa-intel-4000, what did we miss ?13:26
vilaMirv: my understanding is that we should have qa-intel-4000 and qa-radeon-7750 there instead13:27
cjohnstoncu2distro-config was updated, did that maybe run after it was updated? (or was the config maybe never merged?)13:28
vilaand http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-daily_release/label=qa-intel-4000/513/console failed, no containers which IIRC is an issue with right access somewhere triggered by an lxc update (but it should not have triggered again >-/)13:28
vilacjohnston: or the jobs may need to be re-deployed, didrocks said he will after the hangout, let's wait for confirmation13:30
vilaMirv: qa-intel-4000 fixed (lxc upgrade striked again, already documented in the playbook), qa-radeon-7750 is not affected13:35
vilaMirv: do you know how to check that lp:cupstream2distro-config revno 925 has been deployed or do we need to check some other job (I think they are documented somewhere)13:37
vilaMirv: by deployed I mean I think some jobs needs to be injected again in jenkins but I don't know how13:37
vilaMirv: and didrocks mentioned having to tweak some other jobs too13:38
vilaMirv: http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-daily_release/configure is the one that mention autopilot-nvidia instead of qa-radeon-7750 should I fix it or is it one that should re-generated by cu2d ?13:43
josephtSaviq: retoaded sorted it out13:43
Saviqjosepht, ok thanks, will report back if we stumble upon it again13:44
vilaMirv: I fixed the slaves in the configuration matrix, we'll check with didrocks if that was appropriate when he's back13:44
Mirvvila: I guess no deploying of the configuration?13:45
vilaMirv: no idea13:45
vilaMirv: can you rebuild cu2d-qa-head or should I just click the build button (no specific parameters ? )13:46
Mirvvila: so lp:cupstream2distro-config is correct at the moment?13:46
Mirvvila: I can to the config redeploying, please don't run anything yet in that case yet13:46
vilaMirv: trunk has my proposal for disabling nvidia so AFAIK yes13:46
vilaMirv: ok, not touching my mouse , just my keyboard to reply here ;)13:51
vilaMirv: and sorry for the flood here, not pushing you, just trying to stay in sync13:51
vila(and have notes to summarize and document when the fires go down)13:51
vilano no no, I won't believe that I can't reach q-jenkins anymore, I just won't13:51
vilapfew, local network hiccup13:51
=== josepht changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: - use 'cihelp' | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: Some services are down (1SS move)
Mirvvila: all deployed, and started qa stack again13:57
Mirvalso continuing on the mir builds but seeing problems still13:57
vilaMirv: thanks, I'll look at the q-jenkins part while you do that13:59
=== psivaa-lunch is now known as psivaa
vilaMirv: autopilot tests running on both hosts14:08
vilaMirv: I haven't seen any failure yet but I didn't check thoroughly, wating for the jobs to finish14:08
sergiusensdidrocks, seems it's nice day to upload clicks14:11
=== alan_g|lunch is now known as alan_g
vilaMirv: http://q-jenkins:8080/job/autopilot-trusty-daily_release/lastCompletedBuild/testReport/ I think that's good enough to validate the otto setup14:16
viladidrocks: ^14:16
Mirvrobru: passing the mir ball to you/ken. status is that bug #1252144 still affects mir in daily-build PPA, please continue to ping upstream about it. then I don't know what's this u-s-c fail about: https://launchpadlibrarian.net/156843289/buildlog_ubuntu-trusty-amd64.unity-system-compositor_0.0.1%2B14.04.20131118.1-0ubuntu1_FAILEDTOBUILD.txt.gz14:17
ubot5bug 1252144 in mir (Ubuntu) "A test failing on builders with mir 0.1.1: PublishedSocketConnector.drm_auth_magic_is_processed_by_the_server" [Critical,In progress] https://launchpad.net/bugs/125214414:17
Mirvrobru: however mir build for armhf, and so did u-s-c. platform-api and unity-mir are about to compile now in cu2d, so if the build for armhf you could run the mir 0.1.1 touch tests even without mir x86 builds or u-s-c, and report them to the landing plan.14:19
Mirvsil2100: didrocks: FYI too14:19
Mirvrobru: correction, please force a rebuild of platform-api, it didn't have changes so didn't rebuild like others in the stack. unity-mir has changes so it'll rebuild.14:23
didrocksvila: I did redeploy, any pointer?14:43
didrocksvila: well, I deployed the one you changed, maybe I missed one14:44
viladidrocks: as mentioned above, http://q-jenkins:8080/job/autopilot-trusty-daily_release had autopilot-nvidia instead of qa-radeon-7750 I fixed that14:44
didrocksvila: hum, interested, I changed the setup job, and then check the otto job and I didn't see nvidia on it, I thought you did change it14:45
didrocksvila: maybe was missing coffee :)14:45
didrockssergiusens: exactly! notes-app and all with AP failing please :)14:45
didrockssergiusens: I guess http://reports.qa.ubuntu.com/smokeng/trusty/touch/maguro/22:20131115:20131111.1/4976/ gives a good list :)14:45
didrocksvila: nice on the otto setup! :)14:46
didrocksMirv: thanks for your work!14:47
viladidrocks: yeah, at least we have some green around there (I'm not clear if we suffer from the slow network there but at least it doesn't break otto)14:48
sergiusensdidrocks, landing plan doesn't show notes app from what I saw14:49
didrockssergiusens: in fact, it was on it and treated deb-side14:50
didrockssergiusens: but as it's both .deb and .click, I think you need to do it .click side as well14:50
sergiusensdidrocks, ack, I'll tackle it as well14:50
dobeyis there a way to see what args are being passed to the autoland script for current/past CI jobs? or any way to get that information at all?15:00
dobeyfginther: ^^15:00
ogra_ev, there seem to be quite a lot of whoopsie crashes in r2315:15
ogra_(along with unity and maliit)15:15
evogra_: crashes of whoopsie, or crashes that whoopsie is uploading?15:15
ogra_whoopsie .crash files15:16
evwell that's really the problem of the crashing program, surely15:16
ogra_(which indicates crashes of whoopsie)15:16
evoh, I see what you're saying15:16
evin a meeting, will come back to this15:16
didrockssergiusens: btw, just updated it as landing 30415:18
fgintherdobey, one moment, in a meeting15:20
=== alan_g is now known as alan_g|tea
=== gatox_lunch is now known as gatox
=== alan_g|tea is now known as alan_g
sergiusensballoons, didrocks this fix won't work on click http://bazaar.launchpad.net/~ubuntu-filemanager-dev/ubuntu-filemanager-app/trunk/revision/8815:43
didrockssergiusens: do not hesitate to annotate the landing spreadsheet with those infos btw (and turn the thing to a big, blinking, RED! :)15:44
balloonssergiusens, looking15:44
sergiusensdidrocks, give me comment access and I will :-)15:44
didrockssergiusens: oh? you don't have this? one sec15:44
didrockssergiusens: you have POWER!15:45
balloonssergiusens, you mean temp_dir = tempfile.mkdtemp(dir=os.path.expanduser("~"))?15:45
sergiusensballoons, no, you can't patch home with click15:45
balloonssergiusens, ahh, so         patcher = mock.patch.dict('os.environ', {'HOME': temp_dir}) is no go :-)15:46
didrockssergiusens: click has access to a separate tmp btw?15:46
sergiusensballoons, you need to re set the upstart environment and that might be a mess ;-)15:46
sergiusensballoons, the environment is managed by upstart15:47
sergiusensballoons, so you can get away with it; but if you don't reset properly; you break the entire env until you reboot/restart the session15:47
balloonssergiusens, ok makes sense. Well, hmm. I guess we've no choice but to redo the tests without patching home. I'm not sure if there are any implications we can't work around or not15:48
balloonssergiusens, I will say the patching home is not new.. the current tests do the same15:49
sergiusensballoons, was my implementation that bad?15:49
balloonssergiusens, ohh, you mean the if not click logic?15:49
sergiusensballoons, yes15:49
balloonsit's been a bit since I did this. I assumed if I removed it, it didn't work15:49
balloonsbut perhaps this was me trying to be more elegant15:50
sergiusensballoons, it worked for me when I did it15:50
balloonssergiusens, ok, well let me re-enable it and do a quick test. if it works, I've no complaints15:51
dobeyfginther: sure. i had a meeting as well. i'm just wondering how all that code is being called in practice at the moment, to see what exactly there is left to do to switch it to running tarmac for the landing15:55
fgintherdobey, can you view http://s-jenkins:8080/job/generic-land/ ?16:10
dobeyfginther: i don't have whatever vpn config is required for that, no16:11
dobeyfginther: is it not visible from the public URLs?16:11
fgintherdobey, no, those details are not published16:11
fgintherdobey, I can pastebin some to you16:12
dobeyfginther: that would be fine. i'm only interested in the arguments passed to autoland such as --ppa and whatnot16:13
fgintherdobey, http://paste.ubuntu.com/6438151/ and http://paste.ubuntu.com/6438155/16:14
alesagefginther, tedg noting this Jenkins failure, that dir apparently created in a second pass, maybe something in the "custom hooks" script to fix? https://jenkins.qa.ubuntu.com/job/indicator-sound-trusty-amd64-ci/16/console16:31
fgintheralesage, looking into it16:57
alesagefginther, thanks16:57
didrockssil2100: kenvandine: joining?17:01
didrocksplars: ^17:01
plarsdidrocks: yes, brt17:01
didrocksrobru: you can't hear us/rejoin?17:03
robrudidrocks, nope, can't hear a thing. gonna update & reboot17:03
kenvandinerobru, uitk is building in the PPA now17:21
robrukenvandine, oh, great17:21
kenvandinerobru, i need to run out for lunch, when the build is done can you get started testing?17:22
kenvandinerobru, and when i get back i'll jump in too17:22
robrukenvandine, sure thing17:22
* kenvandine runs17:22
Saviqcihelp, can I file a bug somewhere to track issues with the CI infra? i.e. otto encounters locked unity7 session from time to time17:24
fgintherSaviq, https://bugs.launchpad.net/ubuntu-ci-services-itself/+bugs17:24
fgintheralesage, tedg, indicator-sound-ci is working now. still trying to figure out why it failed earlier17:26
tedgfginther, Cool, thanks!17:26
vilaSaviq: from *time to time* ??? I could understand everytime, but from time to time... screen saver ?17:27
alesagefginther, yes maybe one pass was necessary to create that hook directory?  and then subsequent runs passed b/c it existed17:27
Saviqvila, yes, screen saver17:28
fgintheralesage, hmm17:28
Saviqvila, I would expect for some projects the time to set up (apt-get etc.) is long enough that the lock screen kicks in17:28
vilaSaviq: why now ?17:28
Saviqbug #125238617:28
ubot5bug 1252386 in Ubuntu CI Services "otto runner has locked unity7 session from time to time" [Undecided,New] https://launchpad.net/bugs/125238617:28
vilaSaviq: ha ! slow network then, worked on17:29
Saviqvila, just wanted to say that maybe network is slow17:29
Saviqvila, still, we should disable screen lock completely on otto, shouldn't we ;)17:29
vilaSaviq: oh yes17:29
Saviqor at least inhibit it during test runs17:29
vilaSaviq: there is no screen to save there !17:30
Saviqvila, I imagined so17:30
vilaSaviq: so when it happens it's for all tests but it doesn't happen for all jobs ?17:31
Saviqvila, yes, when it happens it's 100% failure17:31
plarsdidrocks: it looks like pitti has already fixed the whoopsie-upload-all crash I mentioned, should be better as soon as we pull in a newer apport17:32
vilaSaviq: thanks17:32
Saviqvila, and it happens most often on unity8, gallery - the biggest dependencies probably17:32
didrocksplars: well… that's pitti, no surprise :)17:38
didrocksplars: thanks for the head's up!17:38
sil2100fginther: hello!17:50
sil2100fginther: https://code.launchpad.net/~sil2100/cupstream2distro-config/add_unity-scopes-api/+merge/195643 <- can you take a look and then maybe enable the automerger for this project :) ?17:50
sil2100fginther: thanks!17:51
dobeyfginther: so there's a single jenkins job, and it runs the autoland script synchronously for every single MP that is approved?18:07
=== alan_g is now known as alan_g|EOD
fgintherdobey, yes, that's how it works18:31
dobeyfginther: and puppet isn't being used to manage jenkins config or anything, is it?18:32
fgintherdobey, nope18:33
balloonssergiusens, https://code.launchpad.net/~nskaggs/ubuntu-filemanager-app/disable-patch-home-click/+merge/195658. I remember the issue now. The tests fail if your /home directory is full of files I believe20:01
balloonsI'm running now, we'll see20:01
sergiusensballoons, it shouldn't, I added a different count mechanism as well20:02
sergiusensballoons, did popey tell you the music app tests fail btw?20:02
balloonssergiusens, yes I know you count the files before running20:02
balloonssergiusens, yes, I'm looking after this run.20:02
balloonsthe trouble is so many files you'd have to scroll and the tests don't account for that20:03
sergiusensballoons, oh, that explains it20:05
balloonsI think I should just next a folder under home and have the tests do there thing in there20:07
fgintherrobru, kenvandine, can either of you review: https://code.launchpad.net/~fginther/cupstream2distro-config/remove-duplicate-compiz/+merge/19567321:03
kenvandinefginther, looking21:03
kenvandinefginther, are we sure you should remove lp:compiz instead of lp:compiz/0.9.11?21:04
kenvandinefginther, i suspect so21:05
robrufginther, kenvandine: I'm not familiar with the 'no-dailies' stack, is that new?21:05
kenvandinei'm not either21:05
fgintherrobru, kenvandine "no-dailies" is to hold projects that need upstream merger support but are not part of daily release21:06
fgintherfor various reasons21:06
fgintherlp:compiz/0.9.11 was updated on 10/22 by timo21:07
fgintherso it's the "newer one"21:07
robrufginther, well, considering that we have been in manual publishing mode for *months* (eg, no dailies EVER), I don't see any value in having a "no-dailies" stack, so anything that makes it smaller looks good to me.21:08
fgintherrobru, kenvandine, the problem is that lp:compiz and lp:compiz/0.9.11 are the same branch. Having two different configs results in upstream merger running 2 different jobs for the compiz MPs21:09
robrufginther, ahh ok21:09
kenvandinefginther, makes sense21:10
robru(I'm not familiar with compiz version numbers)21:10
robrufginther, there's also a compiz-0.9.10 in the no-dailies stack as well, should that also be removed?21:10
fgintherrobru, looking21:11
robrufginther, oh, no, my mistake. so after your change lands, then no-dailies stack will have lp:compiz/0.9.10 and then head stack will have lp:compiz/0.9.1121:12
robrufginther, ok ok, looks good to me, approved.21:12
fgintherrobru, right, they are two different branches in that case21:12
kenvandinerobru, ubuntu_keyboard passed, trying notes_app again21:15
robrukenvandine, hmmm ok. strange, so many flaky tests or weird issues today21:16
robrukenvandine, ok, gone for lunch for real this time21:16
tedgSo it seems that dbus-test-runner has it's own PPA.21:25
tedgIt is the only person building there.21:25
tedgWhich, well, sucks.21:25
tedgAnyone know why that might be?21:25
tedgalesage perhaps?21:25
alesagetedg investigating21:26
tedgOr, wait, does that mean it gets released normally?21:27
alesagetedg not fully grokking the problem there21:28
tedgalesage, I need other packages to be able to dep on it.21:28
alesagefginther, quick q: does this ppa get added to every build under trusty, at least? https://launchpad.net/~ubuntu-unity/+archive/daily-build21:44
alesageor fginther am I recalling a bygone era21:44
fgintheralesage, yes, that's the default behavior. There are a couple projects configured to *not* do that.21:44
fginthertedg, it's probably historical reasons or someone needed to build it for multiple series21:46
alesagetedg would you want to transpute your dbus-test-runner to this ppa?  or would indicators et.al. want a separate ppa?21:46
tedgI think there should probably one PPA to rule them all.21:46
fginthertedg, alesage, there are still some PPAs configured to 'backport' the current trunk21:46
tedgSo let's get everything pointing there.21:47
alesagefginther, tedg but for the record, should what's in this ppa be "fresher" than daily-release?21:47
tedgThe only reason I'd want a custom PPA is if it could remove my projects from the release craziness.  But I'm guessing that's inescapable.21:47
alesageI mean for projects so configured--is dbus-test-runner one?21:48
fgintheralesage, it's not fresher.  The difference is that only trusty packages are in daily-build, the indicator-staging-ppa has raring, saucy and trusty21:49
alesagefginther, o ok I see21:49
fginthertedg, alesage, the custom PPAs are really only used to provide trunk to older series, if you don't need that for dbus-test-runner, we should eliminate that step21:51
tedgI'm not using the older builds for anything.21:53
tedgSeems distro version should be fine on older releases.21:53
fginthertedg, thanks21:55
tedgfginther, Is it possible to trigger a push of dbus-test-runner trunk until the daily-build PPA?  I'd like to use API that's in trunk.22:01
fginthertedg, not directly, that PPA is managed by the daily release tools...22:05
tedgOh, my faves.22:06
tedgfginther, Then will we get a daily release to that PPA even if the target PPA isn't there?22:06
popeysergiusens: basically only one app passed22:07
fgintherrobru, is the daily release machinery working yet?22:07
robrufginther, hahahahhahhahahahaha22:07
robrufginther, oh, it probably 'works', sure. but it'll be manually published for the forseeable future22:07
fgintherrobru, what about builds to the daily-build PPA?22:08
tedgrobru, Let's separate broken process and broken tools :-)22:08
sergiusenspopey, notes?22:08
robrufginther, hmmmm. last time i tried to build friends, it built but the check step failed. so it should be building in the PPA fine.22:09
sergiusenspopey, well, for the weather app, they lowered the failures by 122:09
robrufginther, haven't had a chance to investigate fully. /still on lunch22:09
sergiusenspopey, all the terminal ones passed for me22:09
tedgYeah, it looks like Friends hit the PPA about 4h 40m ago.22:09
fgintherrobru, any chance you could build the qa stack?22:11
robrufginther, sure22:11
fginthertedg, that should get you dbus-test-runner22:11
popeysergiusens: let me reboot and re-run all the tests and get you logs22:11
tedgSweet!  Thanks guys!22:11
robrutedg, fginther: ok, just started the build. not sure how long it usually takes. watch the PPA I guess (it shows up in the PPA before the job finishes running)22:12
popeysergiusens: http://paste.ubuntu.com/6439836/ notes crapped out part way through22:13
* popey reboots and tries again22:14
sergiusenspopey, worked fine for me, but I only have maguro22:14
* tedg hits reload repeatedly to make it build faster22:14
sergiusenspopey, wait; after you reboot, don't start the tests until mtp is ready22:14
sergiusenspopey, it will reset your adb connection22:14
popeysergiusens: http://paste.ubuntu.com/6439879/ one failure on notes22:21
popeysergiusens: balloons http://paste.ubuntu.com/6439903/ music app 2 fails22:28
sergiusenspopey, ack, going to run notes tests again22:31
sergiusenspopey, that said, it's a lot better than current http://reports.qa.ubuntu.com/smokeng/trusty/touch/maguro/23:20131118:20131111.1/5017/notes-app-autopilot/22:32
sergiusenspopey, and http://reports.qa.ubuntu.com/smokeng/trusty/touch/mako/23:20131118:20131111.1/5018/notes-app-autopilot/22:33
sergiusenspopey, so I'd approve that one regardless if you agree22:33
sergiusenspopey, you are right, test_delete fails on trusty, but works on saucy for notes app22:41
sergiusenspopey, the slide emulation isn't enough22:44
robrutedg, ok, looks like dbus-test-runner is in the ppa22:44
tedgrobru, Yup, thanks!  Requeueing all the other builds. :-)22:44
popeysergiusens: ok, notes I'll approve22:50
popeysergiusens: but not weather or music for now... also, bed22:51
sergiusenspopey, I'm rerunning weather now, I'll leave a comment on that one :-)22:51
sergiusenspopey, thanks for the run!22:51
popeywill refresh and look in the morning22:51
sergiusenspopey, enjoy sleep while you can!22:51
robrufginther, seems not all is well just yet: http://q-jenkins.ubuntu-ci:8080/job/cu2d-sdk-head-3.0publish/295/console23:41
robrufginther, also, it seems like some communication issue between jenkins and test machines? http://q-jenkins.ubuntu-ci:8080/job/cu2d-friends-head-2.2check/253/console shows check job failing because it can't find the junit results from the (successful!) subjob.23:43

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!