/srv/irclogs.ubuntu.com/2013/11/15/#ubuntu-ci-eng.txt

rsalveticyphermox: still around?00:12
cyphermoxback now yeah00:54
cyphermoxrsalveti: pong00:55
rsalveticyphermox: seems fine now, I just copied over the new phablet-tools that was pending in the ppa since nov-0800:56
cyphermoxok00:56
rsalvetithat had a few fixes that are useful for the emulator bootstrap00:56
rsalveticyphermox: everything is in manual still, right?00:57
rsalveticyphermox: which then probably means that someone had to file a landing request at that time00:57
cyphermoxrsalveti: yeah00:57
rsalvetiwhich didn't happen, so it was stuck in the ppa00:57
rsalvetibut since jenkins if down, and this only affects the emulator use case, I decided to push push it directly00:58
cyphermoxhey, if it's pressing just upload directly to the archive ad we'll sync it back00:58
rsalvetiright, I know there might be a missing step in there00:59
rsalvetilike publishing in another ppa (for different series), and also updating trunk00:59
rsalvetiwith the released version00:59
rsalvetibut is that part of a jenkins job as well?00:59
cyphermoxyes00:59
cyphermoxbut like I said, you could jsut as well upload phablet-tools to distro directly and I'll merge the changelog01:00
rsalveticyphermox: right, that's kind of what I did, but I decided to copy the binary packages instead01:00
rsalvetiif you can merge it manually it'd indeed be helpful01:00
cyphermoxok I'll take a look at that first thing tomorrow01:05
rsalvetigreat, thanks :-)01:05
=== kalikiana_ is now known as kalikiana
=== Ursinha-afk is now known as Ursinha
Mirvlove the amount of jenkinses up! I wonder what's the status of mergers?05:26
Mirvah, up but jobs don't launch at jatayu at least05:29
ogra_hmm, something tried an image test it seems but with a weird version08:40
didrocksyep08:44
didrocksand I see stuck jobs on ex-magners08:44
didrocksI guess vila would know ^08:44
didrocksogra_: btw, can we publish 101 on saucy? request from stephan :)08:44
didrocksogra_: IIRC, this one is correct08:44
ogra_didrocks, i was waiting for asac's "go" on that ...08:45
viladidrocks: on it but it's mostly guess work at that point08:46
didrocksogra_: hum? what asac knows more than us on publishing a new image in the stable release (between minor ones, 100 and 101)08:46
didrockswe are not talking about saucy->trusty, but rather saucy->saucy08:46
sil2100didrocks: is mangers up already but on some other IP or something? ;)09:08
didrockssil2100: you do have http://q-jenkins.ubuntu-ci:808009:09
didrocksbut it's not fully up yet09:09
didrocksmissing executors, firewall rules for syncing on the archive admin machine and the otto machines09:10
didrockssil2100: I guess vila is on it, but let's sync up during a short meeting09:10
sil2100I would have to know the IP of q-jenkins.ubuntu-ci anyways!09:10
didrockssil2100: hum, no, you need to change the vpn config as per wiki09:11
vilasil2100: it's IP changed overnight, stop *thinkinh* about IPs ;)09:11
sil2100:<09:11
* sil2100 has a lot of syncing up as he does not know about any wiki09:12
viladidrocks: we're on it with jibel but blocked by access rights :-/ no sudo anymore on jatayu which is the new home for q-jenkins and inherits the restricted sudo policy from m-o apparently09:12
Mirvdidrocks: those stuck jobs were started by me earlier today, just to see what's up. daily-release-executor seemed down.09:12
vilaMirv: starting that slave wasn't documented for m-o and wasn't migrated, jibel just documented the needed steps but we lack the rights to process them09:13
Mirvvila: ok09:13
didrockssil2100: look at the engineering ML (mails from yesterday)09:17
didrocksvila: and do you know about the firewall? this is an orthogonal question I guess :)09:17
sil2100didrocks: what e-mail is it? Since I don't see anything, could you give me the title?09:18
viladidrocks: nope09:18
sil2100I might not be on that mailing list?09:18
sil2100Ah, the earlier update09:19
sil2100Got it09:19
didrockssil2100: great ;)09:19
didrocksvila: do you know who is tracking it then?09:19
vilaev: ^09:20
didrocks(better to figure it out now than too late)09:21
=== ev changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: ev | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: Some services are down (1SS move)
evdidrocks: when you say firewall, do you mean https://rt.admin.canonical.com//Ticket/Display.html?id=65887 ?09:23
* didrocks looks09:23
didrocksev: is tachash q-jenkins?09:25
viladidrocks: no09:25
viladidrocks: q-jenkins was magners but it's now hosted on jatayu09:26
didrocksI don't see then the request between snakefruit and jatayu09:26
didrockswhich is used for daily release09:26
* didrocks tries09:26
didrocks$ rsync rsync://jatayu.ubuntu-ci09:27
didrocksrsync: failed to connect to jatayu.ubuntu-ci (10.98.3.12): Connection refused (111)09:27
didrocksrsync error: error in socket IO (code 10) at clientserver.c(122) [Receiver=3.0.9]09:27
didrocksfrom snakefruit09:27
* didrocks add this to the ticket09:27
* didrocks updates the cu2d config to point the final mirror copy from jatayu.ubuntu-ci instead of magners09:29
didrocksplars: ogra_: vila: sil2100: Mirv: ev: short meeting09:30
viladidrocks: we're trying to identify service names instead of machine names, can you come up with a name for that one ? (doesn't have to be definitive we'll fix them later)09:30
didrocksvila: the cu2d machine?09:30
didrocksthen, call it as you wish09:30
viladidrocks: no, the need to use rsync, what's the service there, isos ?09:31
viladidrocks: archive ?09:31
vilajoining in a sec09:32
ogra_=== Image #22 building ===09:35
asacdidrocks: ogra_: lets talk about the saucy update thingy09:37
asacso we have now 101?09:37
sil2100didrocks: ok, I lost my account on that jenkins, and now when I re-created it and log in I don't see cu2d actually - what do I need to get access to it?09:37
sil2100Or is it not visible?09:37
ogra_asac, since a month, idling in saucy-proposed ... i remember you asked me to hold it back, but i dont remember why anymore09:38
ogra_asac, stgraber would like us to promote it so we test the stable channel promotion09:38
asacogra_: was 101 produced _before_ we started producing trusty?09:38
ogra_yes09:38
ogra_it only has fixes for the uevent spam on maguro09:38
ogra_built from SRU fixes only09:39
asaccan we in theory produce a 102?09:40
sil2100didrocks: ok, I was looking in the wrong jenkins09:40
ogra_asac, any time, yes, but there were a lot more SRUs now ...  so that would need careful testing and looking at09:41
asacof course09:41
ogra_http://reports.qa.ubuntu.com/smokeng/saucy/touch_mir/09:41
ogra_101 looks even better than 10009:41
ogra_hmm or was that under touch_ro09:41
ogra_(where it looks even better better :) )09:42
asacsure. i just wonder if we should also test other parts of the update infra09:42
asace.g. that we can produce saucy still09:42
asacthat we can test it09:42
ogra_we surely can roll another saucy image ... but i would wait until we can get test results09:42
didrocksasac: ogra_: we can promote 101 to stable right now and kick an 102 build09:43
asacyeah. how about we target saucy update for next week thu09:43
ogra_didrocks, no09:43
didrockswhy?09:43
ogra_didrocks, lets wait with kicking a new build until the infra is fully back09:43
ogra_i dont want to experiment with that09:43
asacso they idea was always that we release an update after one month09:43
ogra_right09:43
didrocksogra_: ok, but we can promote 101 right now still :)09:43
ogra_so lets promote 101 and roll an 102 build on monday09:44
didrocks+109:44
asaclets discuss final things monday09:44
ogra_right, so that we see image testing works as expected09:44
didrocksbut we still promote 101 now?09:45
asacnot please09:45
didrocksasac: can you please tell that to stephan then?09:45
asacsure09:45
didrocksasac: because he's blocking on it09:45
didrocksthanks09:45
asache is blocking? not: he feels blocked :)?09:45
didrocksnot sure…09:45
asacok let me figure09:46
ogra_right he wants to finish stuff on the weekend but needs results from our side09:46
didrocksI don't know why we are not promoting 101 without any reason, but well :)09:46
didrocksI think we're not the stackholder there09:46
asacthe plan was to release an image to stable channel after one month09:46
asacthats what we sold to exec mgmt etc.09:46
* ogra_ doesnt care about stakeholders ... but it is simply a better image than 100 09:47
ogra_i.e. brings benefit to our users09:47
asacthat was true for the last 3 weeks :)09:47
didrocksasac: so a month, we do release on sunday? :)09:47
ogra_asac, well the plan was surely too that we do more regular image builds then ;)09:47
asacso i am not sure why the sense of urgency today09:47
ogra_asac, and not have something rotting in proposed for 4 weeks09:47
asacdidrocks: just saying: for me it was last thursday09:48
asachence i am not ready mentally09:48
ogra_asac, because stephane needs info if the setup works09:48
asacbecause for me it was in the longer future and i want to talk to a few folks first09:48
ogra_well, then lets keep it til monday ...09:48
ogra_but it seems weird to leave images rot for 4 weeks "just because"09:48
asaci will check with him09:49
asaconce he is up09:49
ogra_there were a lot of SRUs, if 102 breaks it will become really hard to find out why09:49
didrocks"The goal behind this image, besides fixing the issue on maguro was to09:49
didrockstest our upgrade process for 14.04.09:49
didrocks"09:49
didrocks"This end to end test was a requirement before we can officially09:49
didrocksdiscontinue support of the saucy image in favor of the trusty one, so09:49
ogra_(unrealted to when we promote 101 ... we should do more frequent proposed builds)09:49
didrocksit'd be good if this could be done ASAP."09:49
didrocksso I guess we can wait for Monday09:50
didrocksbut would be good not say on Monday "let's wait even more"09:50
asacthats his perspective. there is more parts of the process we should pipeclean09:50
didrocksasac: however, if we promote image 101 next week09:50
didrocksit will mean we flip the stable channel to trusty in a month?09:50
didrocks(to not have more than one update a month?)09:50
ogra_didrocks, no09:50
ogra_didrocks, stable is stable09:51
ogra_it wont switch before we release trusty09:51
ogra_(as stable)09:51
didrocksogra_: that's not what I understood…09:51
asachehe09:51
asacdont argue about that part09:51
ogra_everything else would be nonsense09:51
asaci disagree with that statement, but lets not go down here09:51
ogra_trusty is in constant flux ... it isnt stable we cant call it stable09:52
didrockswell, if we take a snapshot, are happy about it quality-wise09:52
ogra_thast still not stable09:52
didrocksI don't see the difference with saucy unmaintained :)09:52
ogra_api versions might still change etc09:52
ogra_you have no guarantee your click packages still work and the like09:52
ogra_(we have a devel channel for a reason ... why would you make them the same)09:53
ogra_anyway09:53
ogra_(just sounds pretty messed up to me )09:54
popeyogra_: is a build in progress?10:30
ogra_hmm, might actually be done10:30
ogra_no, not yet10:30
ogra_and yes, see above10:30
ogra_22 is in the making10:30
ogra_(cdimage part is done)10:31
popeyah10:39
* popey does "/hilight ==="10:39
davmor222 installing10:46
ogra_right10:52
ogra_=== image #22 DONE ===10:52
davmor2ogra_, popey: if you send yourself a bunch of text messages in the indicator does it change,  send about 5.  I'll try and get a photo10:58
ogra_does the keyboard work again (it didnt in 21 ... after you rotated the device it didnt come up anymore)11:00
popeydavmor2: i sent a bunch of texts and they were all received11:00
popeyogra_: unlikely, that bug was only filed last night11:01
* ogra_ doesnt have his maguro around atm11:01
popeydavmor2: can you be more specific about the issue11:01
ogra_yeah, thats what i expected11:01
popeydavmor2: http://popey.com/~alan/phablet/device-2013-11-15-110127.png11:01
davmor2popey: no I get them but in the indicator the icon changes from text to globe and doesn't give the same effect if you click on it to send back11:01
popeyogra_: yes, still happens in #2211:05
ogra_as expected then11:05
popeybug 125159711:34
ubot5bug 1251597 in Unity 8 "Scope header sometimes appears in the middle of the scope" [Undecided,New] https://launchpad.net/bugs/125159711:34
popeyanyone else ever see that?11:34
popey(not a regression, seen it quite a bit)11:34
didrockspopey: yeah, I think there is already a bug from asac related to it11:38
ogra_popey, all the time11:38
ogra_confirmed11:40
popeyah okay11:42
davmor2popey: yes11:43
davmor2popey, ogra: http://ubuntuone.com/20tSyBkGQWHDRmOVznvWcw this is what you normally see right,  every now and again I get this http://ubuntuone.com/4OPQyVQdfhliEcdIyq3jlK11:54
ogra_ah, i never noticed that11:54
ogra_(but i must admit i also never paid much attention to this icon)11:55
davmor2ogra_: it's normally the 5-7 text but it's not just the icon I can't reply to this one11:55
davmor2ogra_: I noticed it on 21 but couldn't seem to reproduce it but now I can woohoo \o/11:57
popeynot seen that12:01
=== MacSlow is now known as MacSlow|lunch
asaccool. i something on dashboard about todays image :)12:19
asaceven if 0%12:19
ogra_oh, a proper versioned entry ?12:19
* ogra_ checks 12:19
asacnot proper12:19
asacbut something :)12:19
asac20131115 ?12:20
ogra_well, thats just the dashboard going mad i think12:20
ogra_(there was no such thing like 20131114 )12:20
asacits more than all the days before :)12:20
asace.g. there is a heart beating somewhere again12:20
ogra_right, but most likely still not proper12:20
asacof course not... i would hope we are better than 0% and still have a proper version12:21
* ogra_ guesses another firewall issue, though i dont know the actual setup 12:21
didrocksogra_: no, it's what we discussed this morning, the phone don't start12:23
ogra_ah, that12:23
ogra_yeah12:23
didrocksso the job is stuck in install12:23
ogra_well, the odd versioning usually means the phones are unreachable12:23
ogra_i just didnt know why12:23
psivaadidrocks: the jenkins timeout of 30 mins is not working :/12:23
didrockspsivaa: I hope it's not an uninstalled plugin :p12:24
psivaadidrocks: Build timeout plugin is shown as installed, but not sure if the correct version was restored after the migration..12:29
=== ev changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: - Type: "cihelp" for help | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: Some services are down (1SS move)
=== alan_g is now known as alan_g|lunch
=== josepht changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: - josepht | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: Some services are down (1SS move)
=== josepht changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: josepht | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: Some services are down (1SS move)
plarspsivaa: which plugin are you concerned about?13:37
psivaaplars: the 'Build timeout plugin'13:38
plarspsivaa: that's a plugin? I thought that was just part of jenkins13:38
psivaathe isntalled version is 1.11 but 1.12. is available and the jobs are not timing out with 30 mins13:38
psivaaplars: http://q-jenkins:8080/pluginManager/ has more details13:39
plarspsivaa: interesting, but I'm more concerned with why the phones are dying when we try to install at the moment :(13:39
psivaaplars: yea, that is concerning. have seen in 4 devices13:40
psivaabut i ran provision.sh in the same manner locally and it works fine13:40
plarspsivaa: I've been through multiple runs locally and can't get it to happen13:40
psivaaplars: not sure if any kinnara side pkg versions are any different to what we had in phoenix13:41
plarspsivaa: I think I saw somewhere that they got some sort of new usb hub, so I wonder if that has something to do with it13:41
=== MacSlow|lunch is now known as MacSlow
psivaaplars: possibly, not sure why so determinant soon after flashing. I tried doing some reboots with one of the devices and it comes up fine in adb13:50
fginthermorning14:02
=== alan_g|lunch is now known as alan_g
=== alan_g is now known as alan_g|tea
=== alan_g|tea is now known as alan_g
asacvila: ev: retoaded: so the otto machines... i thought those were in the DC and wired up yet.16:05
asacis that not the case?16:05
retoadedasac, they are all in the DC and wired as far as KVM, network and power are concerned. There are still a few systems that were migrated out of their desktop cases into rack mount cases that require some internal wiring (extended power leads, system fan adapters, etc ...). rfowler is picking those pieces up today so the can be brought online.16:07
asacretoaded: are those the otto machiens?16:08
asacor are those wired up etc.?16:08
=== gatox is now known as gatox_lunch
* asac thinks those are desktop machines and its hard to migrate those into server cases16:08
asacbut guess rackmount is something else16:08
vilaasac: see https://wiki.canonical.com/UbuntuEngineering/CI/1ss-move-current-issues missing harware pieces16:08
evasac: they were desktop machines - they were reconfigured into rackable units16:09
fgintherasac, yes, those are the otto machines16:09
retoadedasac, the otto machines were desktop machines now in rack mount cases. The problem was that some of the smaller form factor desktops had very short outputs from the power supplies.16:09
asacok so those are our x86 test boxes.16:10
asacwhat about the phones?16:10
asacdo we know yet what the issues are with running our jobs?16:10
asachttp://reports.qa.ubuntu.com/smokeng/trusty/touch/16:10
ogra_looks like the dashboard has at least one proper version there now16:10
asac:)16:10
* asac is purely color driven :)16:10
evWe have 20 phones with two down16:11
* vila sends green all over asac 16:11
asaclol16:11
asacev: that sounds good. but why dont the job run?16:11
asacok i heard that a new attempt is currently running16:12
asaclets see16:12
asacand wait a bit16:12
* ogra_ was playing with the emulator all day ... 16:12
ogra_i doubt we'll ever get it to make the systemsettle tests :P16:13
plarsok, the image smoke tests seem to be rolling along on both mako and maguro now16:19
plarsinstall_and_boot is done on both, and default has passed on maguro, should start to see results on reports.qa.ubuntu.com soon16:20
plarsogra_: I'm less worried about systemsettle tests, and more concerned with whether sensitive timings in the autopilot tests will be a problem16:25
ogra_thy most likely will16:28
plarshttp://reports.qa.ubuntu.com/smokeng/trusty/touch/16:32
balloonsdidrocks, are you going to be able to pick up the new source for the core apps?16:33
balloonssorry, I mean to say, are you going to land the stuff in the pipeline for core apps?16:34
balloonsfginther, I'm confused by the output from the merge bot on this: https://code.launchpad.net/~nskaggs/ubuntu-rssreader-app/add-activity-indicator-check/+merge/195322. http://91.189.93.70:8080/job/generic-mediumtests-trusty/147/console16:36
cjohnstonballoons: use the vanguard to start with please :-)16:37
fgintherballoons, interesting. looks like something fell over. Will investigate16:37
* balloons smacks balloons around a little bit16:38
balloonssorry cjohnston.. bad habits16:38
doanacplars: smoke is showing up on the dashboard now: http://reports.qa.ubuntu.com/smokeng/trusty/touch/mako/22:20131115:20131111.1/497716:39
doanacwell done16:39
plarsdoanac: yep, I posted just a bit ago ^ :)16:39
plarsdoanac: looks promising at least, will be interesting to see the overall results16:40
balloonscjohnston, can I bug the vanguard about landing asks as well? :-)16:41
cjohnstonballoons: probably not... I guess it would depend on what it is, but we don't yet have alot to do with that16:41
didrocksballoons: I think only sergiusens can (once the CI infra back)16:44
ogra_doanac, and its all GREEN !!!16:44
ogra_asac will love that16:44
balloonsdidrocks, ty.. I figured the infrastructure might be the holdup16:44
didrocksheh16:44
balloonsthere should be merges for everything.. so I'm curious to see it all get pulled and go green16:45
sil2100I wouldn't too hastily hope for 'go green' in the nearest days ;)16:45
balloonshope springs eternal sil210016:46
ogra_sil2100, well, if we stop the image tests right now they will be green :)16:46
sil2100hahah ;)16:47
=== gatox_lunch is now known as gatox
davmor2ogra_: you need to perfect the following line,  " asac these aren't the results you're looking for, move along, move along"16:53
ogra_haha16:53
ogra_that only works in hangouts, else he cant see the specific hand wave16:53
davmor2o| o/ o_16:59
didrockssil2100: ogra_: robru: kenvandine: cyphermox: ev: vila: plars: landing meeting time!17:00
ogra_pfft17:00
ogra_your late17:00
sil2100!17:00
* sil2100 joins17:00
ogra_*you're17:00
sil2100kenvandine: ping!17:02
kenvandinesil2100, pong17:04
evcyphermox, vila: oh hai17:11
evso the problem we're trying to solve, to recap:17:11
evwe want the openvpn server to push the DNS server's IP address to the client, but we want this to be a fallback for whatever the person currently has set up17:11
evso if I have 8.8.8.8, I don't want to start suddenly routing all DNS requests through batun17:12
cyphermoxright17:12
cyphermoxso, using openvpn directly, there's just one way to do it; via the up script as per the VPN wiki page17:12
cyphermoxwe can tweak that to some degree to be even more useful if we were to push config to dnsmasq rather than to resolvconf directly17:13
cyphermoxotherwise, NM should be able to do the right thing without change... if not, there's a bug17:13
evwould `push "dhcp-option DNS 10.99.244.1"` not work?17:14
vilamy understanding is that even for the openvpn trick documented on the wiki you end up with the pushed dns on top of yours in /etc/resolv.conf17:14
vilai.e. the one pushed from the vpn becomes the primary one17:14
vilacurrently mine is: nameserver 10.99.244.117:15
vilanameserver 192.168.0.25417:15
vilasearch ubuntu-ci17:15
cyphermoxev: correct, push dhcp-option DNS (and DOMAIN) should be all you really no17:15
cyphermoxargh17:16
cyphermox.. really all you need17:16
ev:)17:16
cyphermoxvila: that's why you need resolvconf or dnsmasq to properly handle the multiple DNS servers and domains17:16
evcyphermox: so to be clear, that wouldn't overwrite 8.8.8.8?17:16
cyphermoxshouldn't, no17:16
evyay17:17
vilacyphermox: additional magic then ?17:17
cyphermoxresolvconf and dnsmasq will do things differently, but both should work provided they are set up properly17:17
vilaev: hold on your enthusiasm ;)17:17
vilacyphermox: I'm fine if we have proper instructions to give but as of today that won't work out of the box17:17
cyphermoxe.g. resolvconf depends on putting , iirc, the VPN dns first, and making sure it only responds for the domain and anything else NXDOMAIN (I think)17:17
cyphermoxand dnsmasq can do everything nicely17:18
cyphermoxvila: are the two push commands currently in the VPN config?17:18
vilahow ?17:18
cyphermoxdnsmasq is able to say hey, I have this dns server but only for domain XYZ, and ship the requests only to it17:18
vilacyphermox: no, but the trick I used in the wiki fakes it so that should give the same end result17:19
cyphermoxright17:19
robrudidrocks, hey wait17:20
didrocksrobru: yep?17:20
robrudidrocks, i tried to kick a friends stack build but it failed immediately without even trying.17:21
didrocksrobru: can you point vila and I to it?17:21
vilacyphermox: so currently the VPN dns is first but last I checked with wireshark, it receives all dns requests17:21
cyphermoxvila: I'm just saying, what you need is that the push lines are there in the config, or adding the settings to NM; and NM out of the box will do the right thing, otherwise, if you use openvpn directly, you need an up / down script17:21
didrocksrobru: and please, don't land friends before we know the image is green :p17:21
didrocksrobru: but nice for trying ;)17:21
cyphermoxvila: with resolvconf, that's expected17:21
robrudidrocks, no, i won't land it, i just wanted to see some recent friends commits built in PPA ;-)17:21
robrudidrocks, http://q-jenkins.ubuntu-ci:8080/view/cu2d/view/Head/view/Friends/job/cu2d-friends-head/347/17:21
didrocksyeah ;)17:21
cyphermoxvila: with resolvconf, you'd have the first DNS server get all the requests, and possibly skip to the next if it answers with SRVFAIL17:22
vilacyphermox: ha, nm is a different story, indeed I have only namesserver 127.0.1.1\nnseach ubuntu-ci there and I didn't check with wireshark17:22
cyphermoxvila: I'll get you an updated up script to try17:23
cyphermoxahah, or actually, just a file that could be left in place always...17:23
didrocksrobru: "only one instance of a stack can be queued for building17:23
didrocks"17:23
didrocksrobru: did you see this?17:24
didrocksrobru: but yeah, nothing is running17:24
didrocksvila: ev: it seems jenkins was backed up in a very very weird state17:24
robrudidrocks, yes... so where does it show that the stack is already building? jenkins lists zero builders active, it says totally idle.17:24
didrocksrobru: yeah, the backup was done while stuff was running. I know where to clean that17:24
robrudidrocks, ok, otherwise i won't be able to build anything today, right? ;-)17:25
didrocksrobru: well, at least not friends, maybe other stacks are in a more lucky state ;)17:25
didrocksrm: cannot remove `head/platform/stack.started': Permission denied17:26
didrocksgrumph the ACL are not restored :/17:26
didrocksretoaded: ^17:26
didrocksI can't restore the state until then17:26
didrocksnormally desktop-team can touch /var/lib/jenkins/cu2d17:26
didrocksrobru: when someone from the CI team will be available, can you ask on jenkins-q.ubuntu-ci to:17:28
didrockscd /var/lib/jenkins/cu2d/work17:28
didrocksrm */*/*started */*/*building17:28
robrudidrocks, who? can fginther do this?17:28
didrocksprobably, not sure17:28
didrockswe used to have access to that directory as desktop-team17:28
didrocksso restoring the ACL can be helpful17:28
robrujosepht, need some help with q-jenkins: http://paste.ubuntu.com/6422247/17:30
vilageeee17:31
evdidrocks, robru: josepht is your guy. He's the vanguard right now.17:31
viladidrocks: do you expect *started to match both getting-started and .started ?17:31
cyphermoxvila: confirm that you have dnsmasq running?17:31
didrocksvila: getting-started?17:31
didrocksvila: where is it?17:31
vila./head/misc.bak/ubuntu/cordova-docs/cordova-docs-3.0.0+13.10.20130930/docs/en/2.4.0/guide/getting-started for example17:32
cyphermoxev: you use NM or openvpn to connect to the VPN?17:32
evcyphermox: both17:32
evI use openvpn, as do a few other people. Others use NM17:32
didrocksvila: that doesn't match rm */*/*started17:32
evideally we'd like a solution for both17:32
cyphermoxev: yes17:32
vilafind . -name '*started' -ls | wc -l17:32
vila12517:32
vilavila@jatayu:/var/lib/jenkins/cu2d/work$ find . -name 'getting-started' -ls | wc -l17:32
vila10617:32
didrocksvila: yeah, this is in a subdirectory of subdirectory of subdirectory of…17:32
didrocksnot in */*/*started which is just 2 level down17:33
cyphermoxev: stop-gap while the openvpn config doesn't push the info, is a file in /etc/NetworkManager/dnsmasq.d and /etc/dnsmasq.d that contains just "server=/ubuntu-ci/10.99.244.1"... doesn't need to be removed or modified ever17:33
cyphermoxev: and I'm writing a proper up/down script to handle openvpn when the settings do get pushed17:34
cyphermox^^ that stop-gap requires removing the up/down script for openvpn17:34
didrocksvila: I think you are handling it and will give a sign to robru, right? (also, can you add to your list of "things to do" to restore the desktop-team ACL?)17:35
viladidrocks: rm done, that's the kind of commands we don't want to have to run anymore17:35
didrocksvila: thanks!17:36
didrocksrobru: should be good now17:36
viladidrocks: ask ev, I think that's not how we want to proceed in the future, instead we would probably want to reduce that kind of access17:36
didrocksvila: ok, as long as the vanguard can answer quickly17:36
didrocksand you handle it/understand the system17:37
didrocksno worry for me, one thing less I have to handle :)17:37
evI'm making exceptions for AU/NZ17:37
robrudidrocks, vila: thanks17:37
=== josepht changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: - use 'cihelp' | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: Some services are down (1SS move)
evbecause they're wonderful people who happen to live at a timezone entirely not conducive to me ever sleeping if we're to cover them with the vanguard17:37
retoadeddidrocks, try again17:38
vilaretoaded: I did the rms17:39
didrocksretoaded: working, thanks!17:39
retoadedvila, ack. the ACLs will all be changed anyway when we move to userdir-ldap17:40
viladidrocks: so that cu2d/work dir is the one you were referring to as your fs sync back in lexington ?17:40
didrocksvila: sorry, what do you mean?17:40
viladidrocks: you mention some use of the file system to sync jenkins jobs17:40
didrocksvila: hum, not sure I mentionned that. I mentionned a while ago that we are using a shared fs for all the stacks17:41
didrocksthat is that one, yeah17:41
viladidrocks: yeah, that17:41
didrocksrobru: http://q-jenkins.ubuntu-ci:8080/job/cu2d-friends-head-1.1prepare-friends/337/console17:42
didrocksvila: it seems cowbuilder isn't setup ^17:42
didrocks(so cu2d not available)17:42
* didrocks is late for his appointment already17:42
robrubah17:43
robruvila, what's the deal with cowbuilder then17:55
robru?17:55
vilarobru: no idea yet17:56
vilarobru: it should be part of cu2d setup but I can't find where it's documented for now, I did come across that at some point but doing it once is not the same as knowing how do it  from memory ;)17:57
robruvila, ok, no worries.17:58
vilahttps://wiki.ubuntu.com/DailyRelease/MovingNewRelease#First_setup_on_the_jenkins_server17:59
vilaFirst setup on the jenkins server17:59
vila    we need to create a release+1 pbuilder. Ping jibel for it.17:59
vilabased on the assumption that we need to re-create it, it may be something else but 'ping' is all the doc I can found17:59
vilafind17:59
vilaev, cihelp: EOW here, dead end on cowbuilder for q-jenkins18:01
retoadedvila, what needs to be done for cowbuilder? I remember seeing that as a package on m-o but was not able to find it in the repository.18:02
vilaretoaded: that's what I'd like to know... cowbuilder is installed but /var/cache/pbuilder/trusty-amd64/base.cow is not found18:08
retoadedvila, ack18:12
retoadedvila, pushing /var/cache/pbuilder/trusty-amd64 now from m-o18:15
vilaretoaded: oh my just when I was checking and found it there 8-)18:24
vilarobru: can you retry ?18:24
vilaretoaded: pushed ? or did I ping robru too soon ?18:25
vilaretoaded: and sorry for disturbing you one more time :-/18:25
robruvila, well, trying. we'll see18:25
plarsI had to kill the notes test run on maguro, it was stuck18:26
retoadedvila, was pushing but it is done so can now be called pushed18:27
plarsretoaded: we should probably update the build timeout plugin as psivaa mentioned earlier, unless you have any reason for wanting the earlier one on there: http://q-jenkins:8080/pluginManager/18:29
retoadedplars, I have the plugin downloaded already. Just need to find the moments to update it (and a few others that have newer versions out).18:30
retoadedplars, what I will need from you though is a list of systems that need to be available for kernel sru testing come Monday.18:31
plarsretoaded: ok, will sort that out18:32
retoadedplars. thx18:32
plarsretoaded: looks like the jobs might be there, but the views are definitely missing from http://d-jenkins:8080/18:35
vilaretoaded: thanks, http://q-jenkins.ubuntu-ci:8080/job/cu2d-friends-head-1.1prepare-friends/ turned to green congrats18:39
vilarobru: unblocked then ?18:39
retoadedvila, sweet18:39
robruvila, retoaded: yep, looks good, thanks.18:39
retoadedvila, robru: that was all cu2d related correct?18:40
robruretoaded, well i needed that for cu2d, yeah, but i couldn't say if anybody else is using it or not18:40
retoadedrobru, ack. thx18:40
vilaretoaded: 80% sure it's only for cu2d18:41
retoadedvila, ack18:41
vilarobru: what job should be tracked to get a feeling on whether cu2d is working ? I.e. what do *you* expect ? ;)18:44
robruvila, well i just kicked friends job instinctively because that's my stack ;-)18:44
robruvila, also friends is a relatively stable stack that should never have failing tests, so any problems there are usually cu2d problems, not friends problems.18:45
robruvila, not sure really. i'll kick webapp stack as well just to see what happens, since I'm more familiar with that stack as well18:46
vilarobru: yeah, better start with a friendly one ;)18:49
vilacyphermox: sorry got diverted and almost forgot you ;-/ EOW'ed but if you send me an email with the stop-gap and post-stop-gap scripts I'll look into them and see how to document their use in our wiki (or get back to you if needed)19:09
cyphermoxsure19:09
cyphermoxI just need to know if you have dnsmasq running though19:09
cyphermoxbut my script is ready19:09
vilacyphermox: I have two configs, alptop uses NM, desktop using openvpn, will have to check for dnsmasq, but can't dig more right now19:11
vilaalptop... laptop !19:11
cyphermoxok19:11
sergiusenscihelp is there an ETA for http://s-jenkins.ubuntu-ci:8080/ ?19:11
cjohnstonnot to my knowledge19:12
retoadedsergiusens, can you be more specific? The calxeda nodes are now online and we will be working the VMs from naartjie next.19:12
sergiusensretoaded, there's a "Jenkins is going to shut down"19:12
sergiusensretoaded, so someone thinks it's not ready, right?19:12
fginthersergiusens, it's still missing some critical slaves19:13
sergiusensretoaded, I'm not rushing, just want to plan  bit19:13
sergiusensfginther, ok, thanks19:13
fginthersergiusens, the VM server is still not available, once that comes online and all checks out, jenkins will be enabled19:14
sergiusensfginther, that's good to hear19:14
robruvila, so i haven't seen a failure yet, but i dunno, seems slow to me. friends stack has been running for 50 minutes now. i can't remember how long it normally takes, but that just seems really slow to me. it builds locally in just a few minutes.19:15
fgintherballoons, I think i have your test issue fixed. There appears to have been a regression in the package python-simplejson 3.3.1-1ubuntu219:53
balloonsfginther, thanks. So I'll just try a  rebuild then eh19:57
fgintherballoons, yep, one test made it through with only test failures19:57
balloonsahh.. https://code.launchpad.net/~nskaggs/ubuntu-calendar-app/fix-new-event-test/+merge/195421 :-)19:58
fgintherballoons, thankfully someone else already spotted the regression and uploaded a new python-simplejson already19:59
balloonsyea, that's what it looked like. honestly I suppose I should have pushed it more at that level20:00
fgintherballoons, no worries. I think they right course of action is to contact the ci team first. Although it's always helpful if you can offer deeper insight20:01
fgintherin the case that a new package was not available, we would have needed to address it in ci to shutdown the job or find a workaround20:01
robruvila, nearly 2 hours and friends still isn't done building. something is definitely wrong here.20:04
evrobru: he's gone for the night. Please use cihelp instead.20:10
cjohnstonrobru: link?20:11
robrucjohnston, http://q-jenkins.ubuntu-ci:8080/view/cu2d/view/Head/view/All/job/cu2d-friends-head/20:11
evThanks cjohnston20:12
kenvandinerobru, cjohnston: it's waiting for friends-app to build on arm6420:13
kenvandinebut there is no build job for arm6420:13
robrukenvandine, yeah, just noticed that...20:13
robruwait, no build job?20:13
kenvandinenope20:13
kenvandinenot in the PPA20:13
robrukenvandine, oh i see, just armhf20:14
robrukenvandine, so should i cancel those jobs?20:14
kenvandinethat won't fix it though20:15
kenvandinenot sure why cu2d is waiting from arm64, if the ppa doesn't build those20:15
robrukenvandine, well, no, but it's never going to finish, right?20:15
kenvandineright20:15
robrukenvandine, same thing happened to webapp stack, it's waiting for arm64 builds.20:15
robrucjohnston, kenvandine: ok, so i cancelled the build jobs, now the check jobs are starting and it says "Configuration autopilot-nvidia is still in the queue: autopilot-nvidia is offline" so there's another problem20:17
cjohnstonits possible that it still isn't back20:17
cjohnstonthere are still some autopilot systems that aren't functioning, I'm not sure what the reasoning for this one being off is20:20
retoadedcjohnston, robru: the only autopilot system up atm is the intel system; the nvidia and radeon systems are waiting on some cables for the inside (either to make sure all of the fans are plugged up and running or to extend a power lead somewhere).20:25
robruretoaded, any ETA?20:25
retoadedrobru, the parts were going to be picked up today but rfowler was diverted to another task. Might be tomorrow at earliest.20:26
robruretoaded, ah, ok. i won't wait around then. thanks ;-)20:26
retoadedrobru, ack20:27
kenvandineogra_, is there an image 23 building?20:36
ogra_nope20:36
kenvandineok, were we supposed to be waiting for the smoke tests on image 22 to be green?20:36
ogra_(nobody asked for one)20:37
kenvandinedidrocks told us to test the apps when we had a green image20:37
kenvandineso maybe he was hopeful that things would look better with 2220:37
vilarobru: basically it's your choice: either we keep the whole line blocked because an hardware piece is missing (so may have to wait until monday in the worst case) or we take the risk to exposing ourselves to a regression on nvidia only (until monday in the worst case), ev, asac, didTAB thoughts ?23:35

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!