=== _salem is now known as salem_ === salem_ is now known as _salem [05:12] you === chihchun_afk is now known as chihchun [06:56] Me? [07:19] jibel: hey! Did alesage finish testing image number 10? [07:28] trainguards, jibel hello! i'm not sure what happened to line #22 in the cisheet, looking at the dashboard looks like it landed? [07:29] pstolowski: the spreadsheet often loses the status after landing happened [07:29] sil2100, ToyKeeper did it but she couldn't test MMS on affected networks (giffgaff and vodafone) [07:29] sil2100, everything else is OK [07:29] Mirv, ah, ok, cool then, thanks [07:29] pstolowski, what was line 22? [07:30] good morning :) [07:30] jibel, Review editing (Fix: LP #1318144) for vivid ppa-overlay, don't remember what was the silo [07:30] Launchpad bug 1318144 in unity-scope-click (Ubuntu) "Can't edit reviews" [High,In progress] https://launchpad.net/bugs/1318144 [07:30] jibel: silo 029 [07:31] pstolowski, yes it landed in silo 029 [07:31] as Mirv said [07:31] pstolowski, rvr did the verification yesterday [07:31] jibel: you think I can copy it over to the RC channel? [07:31] jibel, great, ty [07:31] jibel: or do you want davmor2 to run tests before I do that? [07:32] jibel: anyway, I can finally feel the release clos [07:32] *close [07:32] sil2100, I'd like davmor2 to confirm it's good to go [07:33] sil2100, he can test on giffgaff [07:34] sil2100, what is the story with shell rotation? silo 18 landed and silo 4 must be rebuilt now? [07:35] jibel: yes, but I'm still not sure if that will help... let me try [07:43] sil2100, apparently the is a build started 5h ago by kgunn https://ci-train.ubuntu.com/job/ubuntu-landing-004-1-build/202/ do you know what is going on? [07:43] there is* [07:53] Ok, now at least it's clear why it doesn't build, let me dive into that [08:14] jibel: anyway, looks like silo 4 will be ready soon :) [08:31] sil2100, don't think so... [08:32] sil2100, the job was started 5h ago [08:32] seems to hang :( [08:32] mzanetti: now the i386 in wily fails [08:32] mzanetti: I thought that was transient, but cmake seems to die [08:32] yeah.. saw that yesterday [08:32] this one right now is again different tho [08:32] seems to be locked [08:34] mzanetti: yeah, so... [08:34] mzanetti: Mirv said cmake has some issues on i386 on wily now [08:34] So it's unrelated, we need to wait for that to get fixed :D [08:37] I'm so sure something else will break in the meantime [08:59] mzanetti: that's my thoughts with wily atm, trying to land qt 5.4.2 before my holidays in a week, and things just keep on breaking :) [08:59] but now I think if it's all down to ciborium / Go / OpenGL on armhf [09:01] bug #1464569 about the CMake issue [09:01] bug 1464569 in gnutls28 (Ubuntu) "3.3.15-5ubuntu1 breaks CMake on i386" [Undecided,New] https://launchpad.net/bugs/1464569 [09:04] sil2100, so, what's the ota-4 status? === tvoss is now known as tvoss|test === tvoss|test is now known as tvoss [09:15] john-mcaleely: ...almost there! (you're probably annoyed by this already) [09:15] john-mcaleely: so, ToyKeeper did some tests on the candidate yesterday, now davmor2 confirms a fix that reqired giffgaff and we'll copy to RC [09:15] It should be copied to RC really really soon [09:15] Like, minutish-soon [09:16] sil2100, not annoyed at all. fatalistic :-) [09:17] and rc soon sounds excellent [09:20] mzanetti: would you be fine with rebuilding unity8 in the PPA? Since I think I'll disable -proposed in it and get unity8 building finally [09:33] sil2100: jibel: mms works in both directions on giffgaff and ee [09:33] sil2100: also mms group is off [09:38] john-mcaleely: ^ [09:38] \o/ [09:38] sil2100: have you pushed it yet? [09:38] sil2100: is it nearly there yet? [09:38] sil2100: is it nearly there yet? [09:38] sil2100: is it nearly there yet? [09:39] sil2100: is it nearly there yet? [09:39] sil2100: is it nearly there yet? [09:39] Pushing! [09:39] Daaamn [09:39] \o/ [09:39] oh oh [09:39] Daamn doesn't sound good [09:39] sil2100: john-mcaleely ah no vpn [09:40] hm? [09:42] yeah, seems VPN and IRC are down [09:42] yes [09:43] jibel, john-mcaleely, davmor2: krillin image 10 copying to rc right now [09:43] 10? [09:43] this all gets very confusing :-) [09:44] john-mcaleely, just wait til we get rid of system-image :) [09:44] Yeah, image 10 from the -factory-proposed channel is what we want ;) [09:44] (there wont be image numbers anymore then) [09:44] lol [09:45] jibel, john-mcaleely, davmor2: besides that I'll also promote the vegetahd, arale, generic and mako images [09:45] (only versions of the different snaps) [09:45] arale? not sure about that [09:45] Ok, I thought we wanted to release everything at once [09:45] ie, changes in stable/meizu.en might need to be separately managed [09:46] I think so too, but there's a discussion in my inbox this morning about thta [09:46] that will head your way shortly :-) [09:46] Anyway, the -factory-proposed channel also has a correct arale image based off the snapshot PPA + the right device and custom tarballs [09:46] perfect [10:33] jibel, john-mcaleely: ok, both krillin and vegetahd are on rc now [10:34] sil2100, thank you. shame my build server is down with the golden factory bits on it [10:34] hopefully I have a local copy [10:34] (looking now) [10:35] davmor2, jibel: can you guys do the OTA testing? [10:35] :) [10:36] john-mcaleely: don't send it to BQ just yet tho, we just want to confirm OTA works fine [10:36] sil2100: this is in the standard rc channel right? [10:36] sil2100, ack [10:36] davmor2: yeah, the ubuntu-touch/rc/bq-aquaris.en [10:36] sil2100: sure on it [10:36] \o/ [10:37] * sil2100 copies over the generic image [10:44] sil2100: Could you please throw me a silo? :) [10:45] Suar [10:46] Throwing [10:46] ;) [10:49] sil2100: thanks ... [10:49] * bzoltan catches it [10:53] bzoltan: fumbles.....recovers......catches it with his foot [10:56] sil2100, do we know why the builders are just stuck? [10:57] mzanetti: didn't get an answer from you on my last question I think :) [10:57] sil2100, I had serious troubles with connectivity in the past hour [10:57] mzanetti: would it be fine to rebuild all unity8 archs in silo 4? [10:57] sorry [10:58] sil2100, yeah, works for me [10:58] bzoltan: hmmm [10:58] bzoltan: not sure why, but the prepare-silo job is stuck :O [10:59] So you didn't get the silo yet [10:59] I have no idea what's going on with everythin today [11:00] mzanetti: hey, did you press rebuild? [11:00] sil2100, yeah, like some 15 minutes back [11:00] before that last chat we had [11:01] not gonna touch it now [11:01] That's too early, I didn't disable -proposed, but yeah... it's stuck anyway so no worries [11:01] I think the jenkins is dead somehow [11:01] sil2100, can you try to get it built, I'd go for an early lunch then [11:01] to be back when it's build so I can do the testing asap [11:02] Yeah... but our train jenkins seems hm, dead [11:02] wouldn't the build happen in the ppa anyways? I thought you'd upload it manually... [11:02] but anyhow... whatever works... [11:02] or not works :D [11:02] Well, jenkins needs to prepare the packages first ;p [11:03] ah, I see [11:05] jibel, john-mcaleely, sil2100: image 16 in place, backup installed, waiting on ota to download now [11:05] \o/ [11:05] ooh [11:07] full image download 427.7mb [11:08] sil2100: you're saying my next obstacle today will be that train is practically dead? :D [11:08] ;/ [11:08] I'm poking webops, not sure where the problem is [11:08] sil2100: I've gotten cmake, go and UITK broken for me today so far [11:08] Mirv: think of it as training in futility [11:09] davmor2: or sisu [11:09] Mirv: do you have any spare silo you could run the build watch-only job with debugging? [11:09] oh there's an article about it in Wikipedia https://en.wikipedia.org/wiki/Sisu [11:10] sil2100: feel free to abort https://ci-train.ubuntu.com/job/ubuntu-landing-016-1-build/174/console and fire it again with "ciborium" in pacakges but with debug. [11:10] Mirv: hm, I don't like breaking the state in the middle of that, but ok :) Thanks! [11:11] Mirv: looks like SSO issues probably [11:11] sil2100: it's been there for 15 minutes doing nothing.. [11:11] davmor2: full image download 427.7mb! Full of awesome goodies ;) ...lets hope we get it by monday [11:11] haha, "See also: Cojones" [11:13] hm, SSO for users seems to work, but CI Train doesn't seem able to authorize [11:13] At least that's what I think is happening [11:14] mzanetti: looks like the world doesn't want silo 4 to land [11:18] sil2100: image build 24/ubuntu 15.04 -armhf (20150611-173952) so that looks right :) [11:18] sil2100: and the world does not want a silo for the UITK either :) [11:18] ;/ [11:19] wth is going on [11:20] davmor2, hm, this 427MB will be a problem. It has to be downloaded completely to receive the notification, correct? [11:21] jibel: indeed [11:21] davmor2, if the connection drops during download, does it resume or restart from scratch? [11:22] sil2100: When did the broken bits of the train last work? [11:22] jibel: no idea. Not sure if it currently registers the fact there is an update, I'll try that afterwards [11:23] wgrant: not sure, would have to browse the history, not sure if anyone ran anything in the morning [11:23] Mirv: did you run any CI Train code successfully? [11:23] sil2100: I haven't tried anything else besides building now [11:23] sil2100: I wouldn't have expected anything to break in the last 24 hours, but login.ubuntu.com moved to new IP addresses a couple of days ago, which could have broken firewall rules. [11:24] hm, SSO works fine on my machine [11:24] Yes, SSO itself is fine. [11:24] I wonder what can be happening, I don't have access to the train machine so I can't debug [11:25] CI Train uses a credentials file when logging in, could that be a problem [11:25] ? [11:25] When logging into what? [11:25] SSO doesn't have the concept of a credentials file. [11:26] sil2100: What is the exact symptom you're seeing? [11:26] davmor2, can you test 22 (latest promoted image) to 24 too? [11:27] jibel: sure [11:28] wgrant: all of our CI Train jenkins build jobs that use LP just hang somewhere in the beginning, when aborting it looks as if it had problems authorizing [11:28] sil2100: Hang where? [11:28] https://ci-train.ubuntu.com/job/ubuntu-landing-016-1-build/174/console [11:28] What's the message? [11:28] Oh, that doesn't looks like anything related to SSO. [11:28] I have no idea, no message, it's just waiting on something and blocking [11:28] SSO is login.launchpad.net or login.ubuntu.com [11:28] That just looks like your Launchpad OAuth token has expired or been revoked. [11:28] There's no message if you don't abort [11:29] hm, but why does it hang then and not just abort straight away? [11:29] Because launchpadlib was written by fools, mostly. [11:29] uuuh! [11:29] It assumes it's running interactively. === sil2100 changed the topic of #ubuntu-ci-eng to: Need a silo or CI Train support? ping trainguards | Need help with something else? ping cihelp | Train Dashboard: http://bit.ly/1mDv1FS | QA Signoffs: http://bit.ly/1qMAKYd | Known Issues: CI Train outage! The CI Train Bot user token expired, waiting for IS/webops [11:41] jibel: test of 16 passes trying on 22 now [11:48] davmor2, do it quick before pmcgowan tries to shoehorn another fix ;) [11:49] jibel: installing now [11:49] 68% downloaded === alan_g is now known as alan_g|lunch [12:05] jibel: it resumes [12:06] jibel: download image on 22 now same 427.7 MB [12:10] Yeah, full delta [12:10] I mean, full image [12:11] uh, [12:12] and 2h boot time ... if you have a popeyesque amount of apps installed :) === MacSlow is now known as MacSlow|lunch [12:15] I think I am getting a bad rap [12:18] sil2100: you want to land silo 4 still today, in case the train gets fixed, right? I mean, I just need to rebuild unity8, qtmir, qtmir-gles, qtubuntu, qtubuntu-gles in my Qt silo if you land 4 first [12:19] which is ok, just needs to be known [12:19] I need an approval for my ciborium branch anyway [12:21] Mirv: yes, I'd like to land 4 ASAP [12:21] But there's no one to help us... [12:21] pmcgowan: you only think that? [12:22] hey [12:22] davmor2: so how are the OTA tests? [12:22] pmcgowan: I've heard rap dude you are aweful at it ;) [12:22] davmor2, they call me mc-gowan [12:24] lol [12:26] sil2100: pmcgowan jibel john-mcaleely 22-24 installed looks good https://docs.google.com/spreadsheets/d/1j91dnYhnffw_6BQCebNncXl4lcS6HMo-qVCdn_YdGmY/edit#gid=1940054 is for 16 → 24 I'll do the same for 22→24 after relocate [12:26] and with that I am out of here [12:27] sil2100: ok. and yes, no help yet :( [12:28] eh, I have no juju knowledge and they request some info about that [12:28] I never had time to dig into that [12:31] cihelp: there seems to be a problem with jenkins builders now? i tried to rebuild silo 22, but it never proceeds past the "INFO Including $packages" line https://ci-train.ubuntu.com/job/ubuntu-landing-022-1-build/99/console [12:32] dobey, that's a question for trainguards ^ [12:33] fginther: oh, they own those jenkins jobs? i thought ci owned them? [12:34] fginther: also, go back to sleep! :) [12:34] dobey, yep, they look after ci-train [12:35] dobey, I think it's the same problem sil2100 was mentioning earlier about lp credentials for the citrain [12:35] dobey: look at topic [12:37] oh ok [12:37] bot lost its train ticket [12:40] sil2100: ah, the topic is too long and trails off the end of my double-wide terminal, so unless topic changes while i'm looking at the channel (ie, not at my 02:00 AM :), i generally won't know to look at it. maybe would be good practice to start prepending important notices at the beginning of topic, rather than appending to the end, so they are more visible? :) [12:40] Maybe, I'm not the owner of this channel ;) [12:46] trainguards: could someone hit rebuild on lp for https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-022/+build/7526571 please? [12:47] dobey: sure [12:47] Mirv: thanks === Ursinha changed the topic of #ubuntu-ci-eng to: Need a silo or CI Train support? ping trainguards | Need help with something else? ping cihelp | Train Dashboard: http://bit.ly/1mDv1FS | QA Signoffs: http://bit.ly/1qMAKYd | Known Issues: CI Train outage! This means CI Train jenkins builders are failing. The CI Train Bot user token expired, waiting for IS/webops. [12:55] sil2100, what happened? === alan_g|lunch is now known as alan_g === joet is now known as josepht === _salem is now known as salem_ [13:22] I can't publish that yeet! [13:22] Aargh [13:26] :-/ [13:30] sil2100: so what's the deal atm ? (i've read all the scroll back...just wondering if we're doomed for the day) [13:31] Webops is on it, there seems to be something wrong with the train instance [13:32] well, there was quite an outage today [13:32] (vpn and irc at least) .... [13:32] probably fallout of that [13:33] From what we were able to determine, with the migration to PS4.5 it seems that someone didn't deploy the train correctly and the credentials and keys were not set in stone [13:33] So probably due to some outage those got erased or something [13:35] ...and I think jacekn resolved the issue [13:35] Yaay! === sil2100 changed the topic of #ubuntu-ci-eng to: Need a silo or CI Train support? ping trainguards | Need help with something else? ping cihelp | Train Dashboard: http://bit.ly/1mDv1FS | QA Signoffs: http://bit.ly/1qMAKYd | Known Issues: there was a CI Train instance outage, everything should be back but inform us if you see any problems - builds need to be re-run [13:37] bzoltan: silo for you! [13:38] dobey: train should be good now, just re-run the build job if it's stuck [13:38] davmor2: how's the tests? Finished? :) [13:38] sil2100: ok, thanks [13:39] davmor2: I'll have to AFK soon to get the cat, but if you finish the OTA tests with success then just poke john-mcaleely to push the image to BQ [13:40] sil2100, [13:40] sil2100: pmcgowan jibel john-mcaleely 22-24 installed looks good https://docs.google.com/spreadsheets/d/1j91dnYhnffw_6BQCebNncXl4lcS6HMo-qVCdn_YdGmY/edit#gid=1940054 is for 16 → 24 I'll do the same for 22→24 after relocate [13:40] and with that I am out of here [13:41] is that done? ^ sil2100 [13:42] I thought "I'll do the same for 22→24 after relocate" meant it's not done yet, but maybe I misunderstood [13:42] If all is tested than yes, we're good :) [13:42] ah, yes [13:42] jibel: ^ are we good? [13:42] you've parsed better than me [13:43] I agree we're still waiting [13:43] sorry [13:44] john-mcaleely, today fer sure! [13:45] pmcgowan, promise? [13:45] :-) [13:47] sil2100, john-mcaleely davmor2 did the upgrade from 22 to 24 and it "looks good" but he didn't report the results to the spreadsheet. [13:47] today fer sure then! === MacSlow|lunch is now known as MacSlow [13:50] and back [13:52] davmor2: are the OTA tests done? :) [14:04] train build are still broken, thus that dput failure [14:11] john-mcaleely: sil2100: there is nothing missing from 22→24 I'm just going to take the time to fill out the form now [14:17] trainguards, can I get a silo for line 69? [14:35] john-mcaleely, pmcgowan, sil2100, jibel: https://docs.google.com/spreadsheets/d/1j91dnYhnffw_6BQCebNncXl4lcS6HMo-qVCdn_YdGmY/edit#gid=938740349 22→24 is good too [14:35] \o/ [14:35] sorry lunch and relocate got in the way but thems the hits [14:37] sil2100: thot i saw you say it was fixed, so tried a unity8 rebuild, failed dput in silo4 [14:37] ? [14:48] bdmurray, ted mentioned that the recoverable error API will be in the next release of libwhoopsie, dyk when that'll be? === infinity1 is now known as infinity [15:25] bah [16:27] sil2100: do we have a date/time for OTA-4 yet? (sorry) [16:28] popey: not sure, I think it's up to BQ now [16:28] They should be done around Monday? [16:28] ok [16:34] sil2100: I realized that 004 doesn't have QA approval yet, so I guess it wouldn't land today anyway? [16:35] Mirv, how would it have QA approval it didn't even build [16:35] jibel: well it built for vivid just fine, and wily is not QA checked [16:35] jibel: only wily would need rebuilds, vivid could stay as is [16:36] I'm just thinking about landing order and when my wily Qt could go in. UITK needs it so that they can begin their landing. [16:37] Mirv, right but I think that's the point of dual landing otherwise it should land in 2 separate silos [16:54] well, hopefully 004 lands so I can land 016 on Monday === chihchun is now known as chihchun_afk === alan_g is now known as alan_g|EOW [17:23] jibel, Mirv: silo 004 built [17:23] mzanetti: is silo 004 good for QA? :) [17:24] uh... did it build? [17:24] * mzanetti grabs a device and tests [17:24] wow! it did build [17:24] back in a bit... just doing sanity tests before asking QA to spend efforts [17:25] Yeah, I got yelled on for that, but I hope at least this won't cause any breakage of the world [17:25] kgunn, you just missed the conversation of the week :D [17:25] Siiince I disabled -proposed for the time of the build [17:25] :| [17:40] trainguards, can someone rebuild this? Seems like it failed a signature check from the archive?!? https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-036/+build/7535097 [17:40] cjwatson, wgrant: hey! Could you take a look at that one? ^ [17:40] Before we rebuild :) [17:40] Ah [17:40] cjwatson, wgrant: unping! [17:41] cjwatson, wgrant: nevermind that [17:41] tedg: ok, so this is the current breakage caused by an ongoing transition in proposed [17:41] Oh, okay. [17:41] tedg: https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1464569 [17:41] Launchpad bug 1464569 in gnutls28 (Ubuntu) "3.3.15-5ubuntu1 breaks CMake on i386" [Undecided,Confirmed] === tvoss is now known as tvoss|test === tvoss|test is now known as tvoss [17:41] This is our bug, but it's actually related to another one [17:43] sil2100, okay, so I need to wait for that to get fixed. But I can test the armhf binaries. [17:45] The signature check thing there is perfectly normal, just cosmetic. We should silence it at some point; LP checks the signature much earlier, and individual buildds don't have the ability to do so. [17:45] tedg: ^- [17:46] Ah, I see. It was the previous verbose message :-) === Trevinho is now known as Trevinho|Holiday [18:44] trainguards, I marked line 45 as tested, because it is. But it shouldn't be published until the i386 binary can be rebuilt. I put a note to that effect in the spreadsheet. [18:45] tedg: hm, can you not mark it as tested if it's not ready to publish. the dashboard will say "YOU CAN PUBLISH" in really big print and the warning not to publish is very small underneath that [18:45] robru, K, changed it from "Yes" to "Yeah", does that take it off the dashboard? [18:46] * tedg can do "Yup" too [18:46] tedg: not sure, let me check [18:47] tedg: what row are we talking about? [18:47] robru, 45 [18:47] oh, row 45. i thought you said silo 45. duh [18:48] tedg: oh, actually the "failed to build" status is great, that stops it from saying "ready to publish" anyway. so whatever. is somebody going to follow up that i386 build? [18:48] robru, Yeah, it's being tracked. Apparently a lib and cmake crashing. [18:49] robru, In the backlog [18:49] tedg: ok, but I mean when somebody fixes that issue in the archive, is somebody going to remember to actually retry this particular build? because if not that'll just sit there ;-) [18:50] robru, I'm not 100%, if not I'll be annoying people about it :-) [18:50] tedg: ok great. yeah once it's good to go just ping me and I can publish it later. [19:07] * robru -> lunch === chihchun_afk is now known as chihchun [20:51] good god === ahayzen_ is now known as ahayzen === salem_ is now known as _salem