[05:38] cihelp why does q-jenkins say it's being shut down? [07:19] hey Mirv, are you building Mir? (is cowbuilder back up?) [07:24] didrocks: I was, but there is a test failure [07:24] didrocks: a fix is being merged now, but q-jenkins states it's shutting down to which I haven't yet gotten a response from cihelp [07:25] I don't know about the cowbuilder [07:26] Mirv: ok, you didn't see issues in the previous cu2d run? [07:30] ev: vila: seems no cihelp, hence the direct ping (excuse us), but the cu2d jenkins is marked for shutting down, any idea? [07:35] didrocks: so actually mir visited my mind on Saturday (shame on me) when I pushed the cu2d mir build and back then cu2d was fine. today I checked the build result + noticed q-jenkins is marked as shutting down ie further builds blocked. [07:35] Mirv: ok, so cowbuilder fixed, great! [07:35] didrocks: ken and robru marked that because of AP machine problems they didn't continue on mir on Friday, although they could have just built and skipped check jobs I believe back then [07:36] yeah, let's see for today what's up with q-jenkins… [07:37] didrocks: another note that mir stack build job was pending with arm64 build jobs, not sure why those weren't auto-ignored. but we'll see it again soon enough when q-jenkins is operational again. [07:51] urgh [07:51] ev: vila: ~desktop-team/cupstream2distro was **weeks** old [07:51] at least 6-8 weeks [07:51] on jatayu [07:52] Mirv: that's why arm64 wasn't ignored [07:52] asac: I can't tell when we are going to do releases again, see issues like that ^ (we are in an unknown state) [08:00] * didrocks worries as well about the network, seems bzr pull took ages (not sure if it's launchpad or the local CI network) [08:10] waht's the new sjenkins ip? [08:15] tsdgeos: seems 10.98.3.13, but you can also configure the DNS [08:16] Mirv: yep, i saw the part of the email to do that, but just changing the hosts file seems easier :D [08:16] manual dns ftw! [08:16] tx [08:22] didrocks: and I suppose there is no health check to warn you about that ? [08:23] vila: to ensure that your home directory was restored with a version that is 3 months ago? No really [08:23] didrocks: we all highlight 'cihelp' no need for a direct ping [08:23] vila: seems you didn't have an health check either to ensure that what you migrate are consistant with latest in production? [08:24] didrocks: could we get a bit more professional and stop pointing fingers, I can do that too but I don't think it's productive especially during fire fight [08:25] vila: well, seems like you do something similar with "urgh", "we should stop doing that" and other expression like that without providing solutions [08:25] I wish just that we could get back to work after a week and half, and I don't think we didn't help you [08:26] didrocks: you did help [08:26] didrocks: but not by yelling [08:27] so, can you help in keeping/figuring out why jenkins want to shut down and ensuring that we do have the latest code in production? [08:27] didrocks: and if I wasn't fire fighting an undocumented engine I would have more time to implement solutions I did propose [08:27] vila: see, pointing fingers… [08:29] didrocks: yeah, "yelling" is finger pointing at you, nothing good comes out of panic mode so please just stop, let's get back to facts and solutions [08:29] vila: so, facts is that ~desktop-team wasn't up to date with latest in magners [08:29] can you help ensuring it's now the case [08:29] and no other thing like cowbuilder are forgotten between the 2 copies? [08:30] second fact: can you figure out why jenkins is setup to stop? [08:30] (and so, we can build anything) [08:31] I asked why jenkins was in shutdown mode on Friday, didn't get an answer [08:31] "no other thing like" is what I meant by undocumented, my crystal ball doesn't know either, do you ? [08:32] vila: well, I think making a diff between old machine, at least in system path, would have been a first step for the transition [08:32] vila: like as well asking for the tool prerequisite for the undocumented part from stackholders [08:32] vila: on jenkins, so, what's the next step? [08:32] I don't care about "would have", do you have "will" ? [08:33] vila: "will" you make a diff between old machine and new one then? [08:33] vila: and what are going to you do about jenkins? [08:33] if you prefer future :) [08:35] yes I do, thanks [08:35] I'll ask again why it is in shut down [08:36] and since I won't get an answer I will restart it [08:36] which is not the smartest thing to do [08:36] vila: so, in term of speaking futures, please apply this to you and stop telling "this stucks", "this should have never been done like that", "this was undocumented". That will help both sides, thanks :) [08:37] vila: +1 on restarting if needed, everything is blocked (the new planned jobs are stucked and don't move) [08:38] vila: from the list of stuck jobs, all the cu2d can be killed [08:38] not sure about: [08:38] daily-release_update-touch-images [08:38] autopilot-ubuntu-applications [08:38] jenkins-autocheck [08:39] power-trusty-desktop-amd64-power-test-1 [08:39] power-trusty-desktop-i386-power-test-1 [08:40] didrocks: I said it *is* undocumented and this makes it harder to check after the 1ss move and harder to get it back up. That how I explain to the stakeholders why it's not up yet [08:40] vila: but you keep ranting about everything single piece of the architecture, which isn't helpful either and doesn't fix anything [08:41] anyway, let's move on, I hope the list above can help you ^ [08:41] didrocks: you're still pointing fingers, do we need to solve that with a third party ? [08:42] vila: come on, I was just showing you that the issue is not unilateral, but it seems you want to continue when I'm proposing to move on and close the topic [08:42] vila: and that's why I started to list the jobs I don't know if you can kill and you bring back the topic :/ [08:42] anyway, *sigh* [08:42] * vila moves on [08:44] autopilot-nvidia needs a new power supply [08:45] the alternatives are 1) setup a new host with any nvidia card 2) disable autopilot-nvidia in cu2d-config [08:46] vila: go the road you prefer, you are handling the production of this. We can go over and hope that if you choose 2), there will be no real card regression (which is unlikely until we release an unity7) [08:53] cihelp: hey guys, seems the otto runner for unity8-autolanding often fails with a lock password entry... https://jenkins.qa.ubuntu.com/job/autopilot-testrunner-otto-trusty/682/artifact/results/autopilot/artifacts/unity8.shell.tests.test_hud.TestHud.test_show_hud_button_appears%20%28Desktop%20Nexus%2010%29.ogv [08:53] is that known? resolved? [08:57] Saviq: not known to me [08:59] vila, http://s-jenkins:8080/job/autopilot-testrunner-otto-trusty/683/ seems to be the last job that failed like this, maybe it got resolved since [09:00] vila, or maybe simply one of the machines is locked... [09:00] no, seems all the jobs ran on ps-radeon-hd8350 [09:00] and now are green [09:01] Saviq: good, no time to investigate that one right now, especially if you think it's back to green [09:02] didrocks: you did update q-jenkins:~desktop-team/cusptream2distro right ? [09:02] vila, yeah, will let you guys know [09:02] vila: yeah, I had to to unblock Mirv (but that was before seeing jenkins was getting shut down) [09:02] vila: only thing I did this morning on the machine [09:02] ok [09:03] * didrocks spent some time to look if there was a bug in the code first [09:03] didrocks: sorry about that [09:03] no worry [09:04] restarting q-jenkins [09:10] didrocks: I won't do a diff between two systems that had different purposes and now have new different purposes, I would have no idea about how to interpret the diff (and I'm not even sure I know how to compare two systems anyway) [09:10] didrocks: q-jenkins is back [09:10] vila: hum, so we will go on try and error procedure then? [09:10] vila: nice! want me start a stack? [09:10] didrocks: yes please, as gently as possible, we're still on thin ice [09:11] * didrocks starts just one stack: Mir [09:11] (no AP for that one) [09:11] Mirv: starting Mir ^ [09:11] \o/ [09:12] vila: I'll let you decide on the autopilot-nvidia side [09:12] * Mir started * [09:12] didrocks: thanks, although the test fix is not yet in https://code.launchpad.net/~vanvugt/mir/fix-1252144.trunk/+merge/195547 [09:13] Mirv: at least, we'll ensure we can start building something :) [09:13] yes, and it'll be nice to see no arm64 stuckness [09:13] right [09:16] vila: confirming the network is really really slow: http://q-jenkins.ubuntu-ci:8080/view/cu2d/view/Head/view/Mir/job/cu2d-mir-head-1.1prepare-unity-system-compositor/47/console [09:16] Fetched 12.0 MB in 1min 1s (196 kB/s) [09:24] Fetched 11.8 MB in 1min 48s (110 kB/s) [09:24] on another job [09:24] and some error: http://q-jenkins.ubuntu-ci:8080/view/cu2d/view/Head/view/Mir/job/cu2d-mir-head-1.1prepare-unity-greeter-session-broadcast/119/console [09:31] ev: coming? [09:31] on my way in [09:31] didrocks: https://code.launchpad.net/~vila/cupstream2distro-config/no-nvidia/+merge/195563 [09:44] cihelp please check I'm seeing jenkins network problems and timeouts at merge proposal https://code.launchpad.net/~vanvugt/mir/fix-1252144.trunk/+merge/195547 - especially concerning connecting naartjie [09:51] vila: just to confirm, the radeon machine is now working? (as it's in the config?) [09:52] didrocks: well, last I checked it was... you mean the qa-radeon-7750 right ? [09:52] vila: yep, ok, good, deploying with it, thanks [09:53] Mirv: sil2100: think about reconfiguring your ~/.cu2d.cred with q-jenkins.ubuntu-ci btw [09:53] didrocks: did that already [09:57] vila: reprovisionning a recent trusty otto container: http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-setup_otto/23/console [09:58] didrocks: updated \o/ [09:58] and removed nvidia from it [09:58] sil2100: great! :) [09:58] didrocks: the tests we usually would have run on those nvidia machines are AP tests, right? [09:58] === Image r23 building === [09:59] asac: yeah [09:59] cihelp: ok, can't reprovision the otto machines, seems it can't download the iso (it tries from q-jenkins) [09:59] see http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-setup_otto/label=qa-intel-4000/23/console [10:00] + wget --progress=dot:mega http://q-jenkins//iso/trusty//trusty-desktop-i386.iso -O autopilot-trusty-setup_otto_label=qa-intel-4000.QgzUAk [10:00] --2013-11-18 09:57:35-- http://q-jenkins//iso/trusty//trusty-desktop-i386.iso [10:00] Resolving q-jenkins (q-jenkins)... 10.98.3.12 [10:00] Connecting to q-jenkins (q-jenkins)|10.98.3.12|:80... connected. [10:00] HTTP request sent, awaiting response... 404 Not Found [10:00] 2013-11-18 09:57:36 ERROR 404: Not Found. [10:00] ogra_: great! [10:00] Mirv: we'll have to wait on the provisionning before running the Mir stack (all setup done apart from this FYI) ^ [10:00] ok [10:00] * asac wonders why it doesntloads from q-jenkins [10:01] we should do a wrapper one day for wget, that a 404 doesn't create an empty file… === vrruiz_ is now known as rvr [10:01] didrocks: do you know how this normally works?: e.g. who ensures that there is an .iso on q-jenkins? [10:02] asac: I think QA cached it some times ago for their smoke testing and they had a process for that [10:02] asac: maybe it's a jenkins job, I don't really know [10:02] ah, also: [10:02] Errors were encountered while processing: [10:02] grub-pc [10:02] E: Sub-process /usr/bin/dpkg returned an error code (1) [10:02] _____________________________________________________________________________ [10:02] ok [10:02] on radeon [10:02] vila: I guess this is the conffile you changed ^ [10:03] wonder why they dont use squid [10:05] didrocks: looking [10:06] didrocks: first attempt at reprovisioining since the move probably [10:06] vila: yeah, confirmed [10:09] didrocks: /iso content is good, searching for :80 config :-/ [10:12] Can this be merged please? https://code.launchpad.net/~xnox/ubuntu-keyboard/libpinyin4/+merge/193859 [10:12] the r101 was a merge from trunk to resolve a merge conflict. [10:12] and the bot now thinks it needs to be re-reviewed. [10:15] xnox: I approved it for you :) [10:15] (seeing it was already reviewed) [10:20] didrocks the URL for the ISO works now [10:21] retoaded: thanks! vila: did you look at the grub-pc thing or should I just retake a setup snapshot? [10:22] retoaded: thanks, haven't figured that one out, there is no /var/www/iso/trusty on m-o yet the log says there was one a couple of days ago 8-/ [10:22] didrocks: what grub-pc stuff ? [10:23] 11:02:38 didrocks | ah, also: [10:23] 11:02:40 didrocks | Errors were encountered while processing: [10:23] 11:02:40 didrocks | grub-pc [10:23] 11:02:40 didrocks | E: Sub-process /usr/bin/dpkg returned an error code (1) [10:23] vila, it is an alias setup within apache; see /etc/apache2/conf.d/isos.conf [10:23] 11:02:43 didrocks | on radeon [10:23] 11:02:47 asac | ok [10:23] 11:02:51 didrocks | vila: I guess this is the conffile you changed ^ [10:23] didrocks: the only ring it bells was to pin a kernel version... hold on reading [10:24] vila: yeah, that's why I'm pinging you, in case you changed a conffiles and grub isn't happy about it [10:25] retoaded: thanks [10:25] didrocks: ok, no I've restored everything to pristine after giving back qa-radeon-7750, thanks for thinking about that [10:26] asac: the isos are downloaded by otto, the magic links were missing server side, just fixed [10:27] vila: ok, so you think we can ignore this grub error then? [10:29] didrocks: first time I see this grub error :-/ [10:30] didrocks: oh, locally modified...hmm, may be a leftover from debugging by the kernel team, I remember they had to force things a bit at one point to get a specific kernel [10:30] vila: ah, so you are reverting back to vanilla? \o/ [10:31] didrocks: sounds like the safer bet [10:31] and then I guess sudo apt-get install -f [10:31] keep me posted [10:31] I'll rerun a setup() then to try to get your image [10:31] (on both) [10:33] retoaded: good morning [10:33] didrocks: reproduced with 'apt-get install grub-pc', confirmed it complained about the comment [10:34] ev, it is definitely morning. I'm not sure about the "good" part though [10:34] :) [10:34] vila: what's this? [10:34] <- grub maintainer [10:34] retoaded: ev@jatayu:~$ time curl https://launchpad.net -> 21s; ev@mayura:~$ time curl https://launchpad.net -> 0.7s [10:35] any idea what's going on? [10:35] cjwatson: http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-setup_otto/label=qa-radeon-7750/23/console [10:35] ev, no but will tale a peek [10:35] cjwatson: caused by a comment in /etc/default/grub [10:36] vila: remind me how I set up DNS for .ubuntu-ci? [10:36] cjwatson: https://wiki.canonical.com/UbuntuEngineering/QA/VPN [10:37] ta [10:37] cjwatson: you had one working before ? NM or openvpn ? [10:37] NM, I'll just update it now [10:39] ah, right, yeah, that error isn't my problem :) [10:39] good [10:39] although the update from 2.00-19ubuntu3 -> 2.00-20 shouldn't have triggered a conffile prompt ... [10:41] $ diff -u <(deb-extract-file grub2-common_2.00-19ubuntu3_amd64.deb /usr/share/grub/default/grub) <(deb-extract-file grub2-common_2.00-20_amd64.deb /usr/share/grub/default/grub) [10:41] $ [10:42] cjwatson: but it did, I installed the maintainer version which removed the comments, hold on [10:43] no substantive change to postinst either [10:43] cjwatson: scratch, that, [10:43] was looking at the wrong machine [10:44] on qa-radeon-7750:/etc/default/grub comments after the GRUB_DEFAULT line [10:45] I did change config a bit, but that change would have been a no-op unless GRUB_CMDLINE_LINUX_DEFAULT had been removed from the config file [10:46] Anyway, I probably can't debug any further unless there's some way to see the diff between the versions here [10:46] cjwatson: ctrl-alt-del [10:47] cjwatson: I did say: keep the installed version and the comments are still there [10:47] And of course if you change package-supplied configuration files directly then it's inevitable that you'll occasionally have to resolve conflicts [10:47] cjwatson: (since I was looking at the wrong machine and stop seeing the comment I revised my memory and said I did chose the maintainer version, clearly not the case) [10:47] Could you show me "diff -u /usr/share/grub/default/grub /etc/default/grub" then? [10:48] cjwatson: ha ! great: here is the *needed* change: [10:48] -GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" [10:48] +GRUB_CMDLINE_LINUX_DEFAULT="quiet swapaccount=1" [10:48] OK, so you can fix this by making that change in debconf [10:48] dpkg-reconfigure grub-pc [10:49] Though it should have done that automatically ... [10:49] Before you do the above, "debconf-show grub-pc" [10:50] cjwatson: http://paste.ubuntu.com/6436818/ [10:50] Huh, so that's already in debconf [10:51] cjwatson: because I did 'apt-get install grub-pc' ? That change was made.... when the machine was setup, say, two weeks ago ? [10:52] cjwatson: no need for dpkg-reconfigure grub-pc then ? [10:52] No [10:52] Was the change originally made directly to /etc/default/grub, or via some user interface? [10:52] * vila forgot dentist, needs to run [10:52] (Either should have worked, but I need to know how to reproduce this) [10:53] directly in file [10:54] vila: Oh, but you say there was a comment added above it? [10:54] I guess that would have been sufficient to confuse matters [10:54] (I'd hoped for the diff pastebinned, not just a part of the diff pasted into IRC) [10:55] vila, it's back again... https://jenkins.qa.ubuntu.com/job/autopilot-testrunner-otto-trusty/691/ :/ [10:56] didrocks, session locking under otto ↑, ideas? [10:56] Saviq: I guess it's something for cihelp ^ [10:57] didrocks, yeah, vila responded to cihelp before, asked me to follow up if it happened again [10:58] Saviq: hard to tell without having access to kvm, which I don't anymore (for good reasons btw ;)) [10:58] didrocks, mhm, thanks [11:00] cihelp, so yeah, otto runner has issues with session getting locked in some jobs - all autopilot videos are either black or show the screen lock password entry, and obviously the tests fail [11:01] * Mirv back in 1h [11:08] Cim [11:08] bleh [11:12] ev, leworks@jatayu:~$ time curl https://launchpad.net -> real 0m0.611s [11:12] retoaded: whoop. What was the problem? [11:13] ev, /etc/resolv.conf. utah seems to modify it to put a nameserver 192.168.122.1 first in the order [11:14] so resolving launchpad.net tries that first then fails over to the next nameserver in order [11:14] I removed it [11:15] ev, that could just be remnants of an older utah install and was sync'd over from m-o [11:18] ugh, I knew there'd be something I hadn't checked [11:18] thanks [11:21] cjwatson: sorry, being late didn't help, here is the full diff http://paste.ubuntu.com/6436934/ [11:26] cjwatson: and I probably missed that GRUB_CMDLINE_DEFAULT when apt-get install presented the diff :--/ [11:26] I tried a saucy chroot, apt-get install grub-pc vim, edit /etc/default/grub to apply that diff, edit /etc/apt/sources.list to upgrade to trusty, apt-get update, apt-get install grub-pc - no prompt [11:27] cjwatson: so, reformulating to ensure I get it right: pat-get saw a conflict on GRUB_CMDLINE_DEFAULT and warned. It won't do it next time correct ? (Unless a new change introduce a new conflict) [11:27] I don't think so but I can't tell for sure because this is a bit out of the ordinary [11:27] Is the immediate problem dealt with? If so I propose ignoring this until/unless it happens again :-) [11:28] cjwatson: good enough for me, will tell you if I encounter such a case [11:28] (Or until/unless somebody can get me a reproduction recipe I can run locally) [11:29] FWIW I suspect the conflict would have related to the comment and not to the GRUB_CMDLINE_LINUX_DEFAULT change, since the latter appears to have been correctly carried over into debconf [11:29] Can't prove it right now though [11:30] cjwatson: the weird thing is that qa-intel-4000 has the same changes and didn't encounter the issue: http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-setup_otto/label=qa-intel-4000/23/console [11:31] == Image r23 DONE === [11:32] vila: Ah well, let's see if it happens on the next grub2 change [11:32] * popey updates to 23 [11:32] Probably not worth ratholing on now [11:33] cjwatson: agreed [11:40] didrocks: so, grub-pc issue cleared for now, re-trying an otto setup [11:41] vila: keep me up to date! [11:42] didrocks: images downloaded on both, progress [11:43] didrocks: and both failed later, reading the logs [11:45] lxc_container: command get_cgroup failed to receive response >-/ not sure that's the trigger but... [11:53] didrocks: nope, that one should be harmless, happened during the last successful run === MacSlow is now known as MacSlow|lunch [12:01] guys, i had a test segfault in jenkins autolanding while it passed fine on jenkins CI and the crash seems to be no related to my changes, we've just re top approved the change but t1mp thinks its a good idea to mention here, in case it happens again to someone else which may mean faulty hardware or something else [12:01] i.e. not asking for help, just saying this happened in case it repeats [12:01] tsdgeos: thanks, cihelp ^ [12:03] ^the MR that had issues https://code.launchpad.net/~aacid/ubuntu-ui-toolkit/ima_do_not_filter_disabled/+merge/194830 [12:06] vila: otto should be ready then? ;) [12:07] tsdgeos: yep, I think it's good to report the hitches so that they can be tracked down one by one [12:08] didrocks: :-/ apparently not despite update_host returning 0 (AFAICS) the jobs fail [12:08] ah ok :/ [12:10] didrocks: last run succeeded, while I'm happy to report, I'm unhappy to not understand why :-/ [12:10] didrocks: I did some apt-get upgrade/dist-upgrade manually though... [12:11] didrocks: so let's move on the next step [12:11] cihelp more 'naartjie' host issues (ssh connection fails) https://code.launchpad.net/~thomas-voss/process-cpp/add_fork_and_run_facilities/+merge/194842 [12:11] didrocks: I also noticed that autopilot-nvidia last otto setup run failed before the move... we may have other issues to investigate when we get a nvidia replacement :-/ [12:15] didrocks: so, what's the next step ? Do you have a small stack in mind that is expected to succeed ? [12:39] vila: was discussing in a hangout [12:40] vila: so, otto should be up for those 2 machines? Mirv: can you launch Mir now? [12:40] Mirv: Mir & co of course :) [12:40] if the fix branches were merged [12:40] vila: on the failure: I know that sometimes, the machine exits before the job finishes to process [12:40] so you see the java stack [12:40] and it's marked as failed [12:45] didrocks: ok! [12:45] the branch is still not merged, though [12:46] I'm wondering if I could help from ci_help to get ETA if it'll work soon, or merge manually [12:46] Mirv: naartjie is resolved ok from s-jenkins but not from the slave cyclops nodes [12:47] retoaded or fginther could add more information [12:47] Mirv: yes, please ;) [12:48] didrocks: otto setup succeeded so yeah, that should mean otto is up, but I'd rather not rely on that without at least one successful run of one job [12:48] doing manually for now, and good if the naartjie problems are on radar [12:48] psivaa, which cyclops node(s)? node-06 resolves it. [12:49] vila: yeah, we need to be able to start the Mir stack (pending Mirv to get merged/merge the Mir branch) ^ [12:49] didrocks: ok [12:49] retoaded: node07 is the only one my test script can't reach [12:49] vila, ack. [12:50] retoaded: ohh? http://s-jenkins:8080/job/process-cpp-trusty-armhf-autolanding/11/console says 'Unable to connect to naartjie:http:' [12:50] vila, it's possible node07 is not up. [12:50] which uses node-06 [12:51] psivaa: ha, good, retoaded, my test script exercises ssh only [12:51] retoaded: from my desktop [12:52] Mirv: let me know when you start so I can monitor [12:52] psivaa, I don't think that is an issue with being able to connect to naartjie but, instead, not finding what it is looking for: http://naartjie trusty/ InRelease has spaces in it [12:53] but that mey not be the actual URL [12:53] s/mey/may [12:55] vila: I kicked the https://code.launchpad.net/~thomas-voss/process-cpp/add_fork_and_run_facilities/+merge/194842 now for example [12:55] psivaa, because the other job https://jenkins.qa.ubuntu.com/job/process-cpp-trusty-armhf-ci/16/console has the URL http://naartjie/archive//head.mir/trusty/InRelease and InRelease does not exist under http://naartjie/archive//head.mir/trusty [12:56] Mirv: err, on s-jenkins ? [12:56] retoaded: yea, that's true. so an issue with the job config i guess. [12:57] psivaa, possibly [12:57] didrocks, Mirv: I was asking for some validation on q-jenkins, can't we do that ? [12:57] retoaded: thanks, i'll dig in if there is anything that is obvious [12:57] vila: Mirv is going to run the Mir and dependant stacks, which will trigger that job [12:58] vila: but the CI isn't able to merge one of the upstream branch it seems which enables to build Mir [12:58] vila: ok I'm just not clear which issue you're referring to. I also kicked the Mir stack now running, after merging the CI failing commit manually. [12:58] so Mirv told he will merge manually the mir branch [12:58] and run the stacks :) [12:58] and, already done [12:58] ha ok, sorry, was confused [12:58] Mirv: then, you take care of the platform stack? (so that it does build against latest Mir) [12:59] vila: yeah platform + unity-mir (and recheck unity-system-compositor too) [12:59] Mir Mirv… this is so confusing! :) [12:59] thanks ;) [12:59] hopefully then there'd be something ready for robru to test for example [12:59] yep [12:59] Mirv: is the network seems to still be slow btw? [12:59] didrocks: hehe, yeah, funny, didn't fall for that Mir Mirv one ;) [13:00] didrocks: yes, the prepare jobs seem to take ages [13:00] "Fetched 12.0 MB in 2min 43s (73.2 kB/s)" [13:01] * didrocks goes for a run outside [13:01] 512kbit/s not really 'internal network' speeds [13:02] * vila sighs, and we postponed the upgrades for the monitoring we had... [13:02] bah, would probably won't help, I doubt we are monitoring the part that is acting right now === alan_g is now known as alan_g|lunch [13:09] Mirv: unity-system-compositor failed to build on amd64 and i386 [13:09] Mirv, didrocks: do we have something simpler to try ? === psivaa is now known as psivaa-lunch === MacSlow|lunch is now known as MacSlow [13:16] Mirv: ? [13:16] Mirv: is there a way to manually trigger a simple stack on q-jenkins ? [13:17] vila: unity-system-compositor in itself is not too big. but the problem there is maybe related to boost transition, not these other problems? [13:17] vila: the prepare jobs now completed without timeouts [13:17] Mirv: no idea :-/ [13:18] Mirv: I try to stay focus on the otto nodes if I can :-/ [13:18] vila: or maybe it's CI after all, I can install libboost-all-dev just fine locally [13:19] Mirv: the failed to build are on lp, so not related to otto is my point :-/ === josepht changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: josepht | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: Some services are down (1SS move) [13:19] vila: ok, but why did you ask about it failing then, or what were you searching to try? [13:19] Mirv: ctrl-alt-del === gatox is now known as gatox_lunch [13:20] Mirv: I asked for a simple stack to run and validate the otto stuff, you pointed to an MP and I didn't make the connection. When mir-head came, I thought I got that job anyway and looked at how it went [13:21] Mirv: it ended up failing *before* reaching otto IIUC, so I'm now asking if have a simpler way to validate otto [13:21] vila: ok, so you want to see check job running? [13:21] vila: sdk stack now running just the check job [13:22] Mirv: thanks, sorry for not expressing my need more clearly [13:22] except that it failed to start [13:22] dang [13:23] Mirv: http://q-jenkins:8080/job/cu2d-qa-head-2.2check/358/ ? [13:23] vila: qa stack now running [13:24] and yes,that [13:24] Mirv: thanks [13:24] josepht, we've see https://jenkins.qa.ubuntu.com/job/unity8-trusty-armhf-autolanding/86/console a few times now - connection time out to naartje [13:25] josepht, that expected / known? [13:26] Saviq: looking, I know there've been some reports of networking issues in the lab so perhaps this is related. [13:26] Mirv: http://q-jenkins:8080/job/autopilot-trusty-daily_release/513/console hu ho, autopilot-nvidia and qa-intel-4000, what did we miss ? [13:27] Mirv: my understanding is that we should have qa-intel-4000 and qa-radeon-7750 there instead [13:28] cu2distro-config was updated, did that maybe run after it was updated? (or was the config maybe never merged?) [13:28] and http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-daily_release/label=qa-intel-4000/513/console failed, no containers which IIRC is an issue with right access somewhere triggered by an lxc update (but it should not have triggered again >-/) [13:30] cjohnston: or the jobs may need to be re-deployed, didrocks said he will after the hangout, let's wait for confirmation [13:35] Mirv: qa-intel-4000 fixed (lxc upgrade striked again, already documented in the playbook), qa-radeon-7750 is not affected [13:37] Mirv: do you know how to check that lp:cupstream2distro-config revno 925 has been deployed or do we need to check some other job (I think they are documented somewhere) [13:37] Mirv: by deployed I mean I think some jobs needs to be injected again in jenkins but I don't know how [13:38] Mirv: and didrocks mentioned having to tweak some other jobs too [13:43] Mirv: http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-daily_release/configure is the one that mention autopilot-nvidia instead of qa-radeon-7750 should I fix it or is it one that should re-generated by cu2d ? [13:43] Saviq: retoaded sorted it out [13:44] josepht, ok thanks, will report back if we stumble upon it again [13:44] Mirv: I fixed the slaves in the configuration matrix, we'll check with didrocks if that was appropriate when he's back [13:45] vila: I guess no deploying of the configuration? [13:45] Mirv: no idea [13:46] Mirv: can you rebuild cu2d-qa-head or should I just click the build button (no specific parameters ? ) [13:46] vila: so lp:cupstream2distro-config is correct at the moment? [13:46] vila: I can to the config redeploying, please don't run anything yet in that case yet [13:46] Mirv: trunk has my proposal for disabling nvidia so AFAIK yes [13:51] Mirv: ok, not touching my mouse , just my keyboard to reply here ;) [13:51] Mirv: and sorry for the flood here, not pushing you, just trying to stay in sync [13:51] (and have notes to summarize and document when the fires go down) [13:51] no no no, I won't believe that I can't reach q-jenkins anymore, I just won't [13:51] pfew, local network hiccup === josepht changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: - use 'cihelp' | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: Some services are down (1SS move) [13:57] vila: all deployed, and started qa stack again [13:57] also continuing on the mir builds but seeing problems still [13:59] Mirv: thanks, I'll look at the q-jenkins part while you do that === psivaa-lunch is now known as psivaa [14:08] Mirv: autopilot tests running on both hosts [14:08] Mirv: I haven't seen any failure yet but I didn't check thoroughly, wating for the jobs to finish [14:11] didrocks, seems it's nice day to upload clicks === alan_g|lunch is now known as alan_g [14:16] Mirv: http://q-jenkins:8080/job/autopilot-trusty-daily_release/lastCompletedBuild/testReport/ I think that's good enough to validate the otto setup [14:16]