=== salem_ is now known as _salem === _salem is now known as salem_ === salem_ is now known as _salem [04:37] michi: stop it it's not ready yet [04:38] robru: Won’t be building anything for a while. [04:38] Thinking of train wrecks and the Addams Family… [04:38] You just sounded like Uncle Fester. “It’s not plugged in yet" [04:39] :) [04:40] Ever played the Addams Family pinball? Best pinball machine in the history of the game. [04:42] not pinball, no. I remember the movies and the NES game... [04:42] I’m trying to find a sound clip… [04:42] If you ever see that machine, play it. [05:06] michi: ok try stuff now [05:07] robru: cool, thanks! [05:08] argh [05:08] https://ci-train.ubuntu.com/job/ubuntu-landing-022-1-build/1/console [05:10] michi: what request number and silo number was it? we have a stale backup apparently [05:10] silo 22 [05:11] No, sorry. [05:11] Sec... [05:11] It was silo 6, I think. [05:11] But it seems to have disappeared [05:12] michi: not the one that you just assigned 5 minutes ago. did you have a silo you were using from before? [05:12] I abandoned that one this morning [05:12] That’s 26, I think. [05:12] Seems like it’s still kicking around now. [05:13] michi: yeah, stale backup [05:13] You can blow that one away if you like. [05:13] michi: what request number? [05:13] I don’t need it any more [05:13] Where do I find that? [05:13] Got it: 756. 2015-12-08 [05:13] michi: /ticket/NNN? [05:13] yeah [05:14] Don’t know [05:14] Where do I find that? [05:14] michi: ok one sec, working on stuff [05:54] alright, production rollout FROM HELL entering 9th hour, no end in sight [06:01] robru: I feel with you. [06:03] michi: it's just a missing oauth token. generating it is completley beyond me. we've been using the same one for a year. [06:03] :( [06:03] I’m oauth ignorant, I’m sorry. [06:04] michi: yeah I dunno, we can generate it but for some reason it's not being saved in a way that lets us actually use it. total mystery [07:16] Mirv: train is basically offline, don't try to run any jobs right now [07:19] have some scripts been not-quite-ported to Python 3? [07:26] robru: can you do me a favor and drop the ubuntu-touch-meta package from silo 0? [07:27] jamesh: yeah apparently lplib [07:27] morphis: sure [07:27] robru: thanks [07:31] ok :( [07:46] Mirv: yeah, we're approaching hour 11 of downtiime [07:46] hour 11 of this 30 minute redeploy [07:48] look on the bright side. Tomrrow's 30 minute redeploy should be a lot quicker [08:21] morning [08:22] trainguards, there is something weird with bileto. Some request set to 'passed' yesterday are marked 'ready for qa' this morning and there is no trace of the change in the history. [08:23] for example: requests 753, 767, 773 [08:24] has the state of bileto been reverted somehow? [08:25] jibel: yes everything is horribly broken and restored from an old backup, please do your best to mark stuff as passed again, I'm very sorry [08:26] robru: I tried assigning a silo, and got this failure: https://ci-train.ubuntu.com/job/prepare-silo/4/console [08:26] robru, how old is the backup, so I know what I have to 'fix' [08:26] ? [08:27] jamesh: yeah it's broken, sorry [08:27] jibel: about a day [08:36] jamesh: OK try again now [08:36] robru: thanks. [08:36] michi: try again now if you're still around [08:36] robru: Wow, I’m amazed that YOU are still around! [08:36] Mirv: i think things are coming back, try stuff if you've been waiting [08:37] michi: yes, i started a deploy at 1pm and saw it through right to 1am. [08:37] robru: Looks like it’s working, but it’s low on silos. [08:38] robru: same here: https://ci-train.ubuntu.com/job/prepare-silo/6/console [08:38] michi: you should abandon that old one, i didn't, sorry [08:39] will try [08:39] Mirv: you around to assign some? [08:41] Argh [08:44] check [08:44] hello? [08:47] robru: sorry [08:47] In a hangout. [08:47] trainguards, did we lose requests overnight https://requests.ci-train.ubuntu.com/#/ticket/786 ?? [08:48] Saviq: yes, catastrophic failure & old backup [08:48] robru: I think I got rid of the old silo, or maybe you did? But still low on silos. [08:48] robru, ack [08:48] michi: unfortunately my internet has cut out and i'm tethering off my phone and it is just *AWFUL* I can barely chat, no pages are loading [08:48] thought you shouldn't be here at this time [08:49] Sure. Go to bed! I think you’ve definitely done enough for one day! === robru changed the topic of #ubuntu-ci-eng to: Train trouble? ping trainguards | CI problems? ping cihelp | Train: http://bit.ly/1hGZsfS | QA Signoffs: http://bit.ly/1qMAKYd | Known Issues: CATASTROPHIC TRAIN FAILURE restored from stale backup, please do your best, I'm sorry [08:49] trainguards: If anyone other than robru is around, can someone help me get a silo please? [08:50] There are 52 in use, and I can’t get one automatically. [08:50] michi: Mirv should be here [08:50] michi: sure I can do whatever can be done with the train [08:50] sil2100 will also be here very soon now [08:50] Mirv: If you could get me a silo for ticket 783, that would be great. [08:50] michi: done [08:50] Sweet, thank you! [08:51] trainguards, should I be recreating the lost tickets? [08:51] * Mirv reabandons one silo [08:51] I've https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-035 assigned but the request for it is gone [08:51] Saviq: yes [08:51] Mirv, still failing. [08:51] Ah, no. [08:51] Saviq: oh... [08:51] Saviq: let's see... [08:52] Saviq: if there's no ticket, could you file the same ticket again and I copy the already built packages there? I don't believe the 035 is "assigned" from what train thinks if there's no ticket for it [08:52] Mirv: yes apologies everything is in shambles, the train / bileto backups are noteven from the same time so stuff is out of sync [08:52] Mirv, nw, I don't care much about packages, will recreate the request [08:53] robru: yeah we'll just reassign, reabandon and move things around, I think it's eventually ok [08:53] michi: no no you have to check in the train for what is assigned ~/silos/ubuntu/landing-035 may still contain a reference to a nonexisting ticket [08:53] god this i a mess [08:53] Saviq: ok, then I'll simply empty the silo since the packages shouldn't be there [08:53] robru: oh ok [08:53] checking [08:54] robru: Seems to be building now. [08:54] So, I’m good. Thanks heaps for all your hard work! [08:55] robru: nope 035 dir is empty [08:55] Mirv: ok [08:55] Mirv: if my phone stops cutting out I'm going to try to figure out what's been lost and recreate some of it [08:55] robru: or you can just go to sleep, we'll manage with the current state well enough [08:56] I can't believe I'm leaving such a mess [08:56] Mirv: 2015-12-15 08:54:13,751 WARNING [08:56] debconf: delaying package configuration, since apt-utils is not installed [08:56] I: unlink for ilistfile /var/cache/pbuilder/build//cow.25390/.ilist failed, it didn't exist? [08:57] That’s here: https://ci-train.ubuntu.com/job/ubuntu-landing-026-1-build/1/console [08:57] Ah, looks like it got past that. I haven’t seen that before. [08:57] michi: let's see how it goes from there [08:58] There’s a bunch of other unusual errors in that log. [08:58] at least diff generation builds go alright [08:59] michi: it's creating a new pbuilder because it's a fresh deploy [09:00] robru: Cool [09:00] robru: do you know if someone is recreating jobs for silos 031-060 that are missing? [09:00] michi: first build will be really slow as it debootstraps vivid & xenial but afterwards should be normal [09:00] or if they are restorable [09:00] Sure, that’s fine. [09:00] Mirv: wat [09:00] I won’t die if it takes a few hours :) [09:01] Mirv: thanks for pointing that out, I can fix that. holy crap [09:02] that would explain the low silo errors [09:02] robru: ok. I was first a bit "wtf" of 404 but then noticed they're simply missing [09:02] god I can't wait o get rid of jenkns [09:03] Mirv: so basically all the "artifacts" links will be broken, you'll needto do DIFF_ONLY builds for silos that you want the diffs back [09:03] Mirv: I mean technically the diffs are in the silos but we've lost the jenkins artifacts [09:04] robru: yes, I think that's the default thing to do by everyone, run diff_only build. I've just run such on my own. === Mirv changed the topic of #ubuntu-ci-eng to: Train trouble? ping trainguards | CI problems? ping cihelp | Train: http://bit.ly/1hGZsfS | QA Signoffs: http://bit.ly/1qMAKYd | Known Issues: CATASTROPHIC TRAIN FAILURE restored from stale backup, please do your best, I'm sorry. Please run "DIFF_ONLY" build job on your silos. [09:07] Mirv: the last few lines here: https://ci-train.ubuntu.com/job/ubuntu-landing-026-1-build/1/console [09:07] Is that normal? [09:08] sil2100: welcome to a trainwreck. in short, old backups restored, all silos need manually fixing to get them back up to date. abandoned tickets need to be reabandoned, filed and assigned tickets need to be refiled and reassigned. bileto and train backup restores are from different times so if there's a silo full of content better check the silo directory on jenkins before removing the packages (if ther [09:08] e's erronous ticket reference). [09:09] Mirv: Never mind... [09:09] michi: yes I think nowadays stuff is kept clean / using low amount of space, so that'd sound like normal thing to do everything important has been pushed to LP. [09:10] Cool, thanks. [09:10] I think everything looks good with my silo now. At least, on the PPA page, it shows packages building. [09:11] Mirv: holy flip, https://requests.ci-train.ubuntu.com/static/britney/vivid/landing-047/excuses.html this is the first britney run I've seen with all-passing tests! [09:11] incredible [09:15] robru: wow :) [09:20] IT WAS ALL WORTH IT JUST FOR THAT [09:20] IT WAS ALL WORTH IT JUST FOR THAT [09:20] Mirv: uh oh [09:21] * sil2100 walks by as if he didn't see or hear anything [09:21] :D [09:21] ;) [09:21] there was no trainwreck when I glanced at that direction, so I just walked along [09:38] Mirv: So, it appears the that builds have finished: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-026/+packages [09:38] Is it possible to get rid of the IBM build for thumbnailer? [09:38] It’s not relevant. [09:39] But the train page does seem to want to update: https://requests.ci-train.ubuntu.com/#/user/michi [09:39] It still shows everything as building. [09:39] Sorry, not everything. armhf [09:40] Sheesh, I’m always one step ahead of the train :) [10:21] michi: the train updates every 15 min or so [10:21] Mirv: Yes, all good. It’s all done and dusted now. === vrruiz_ is now known as rvr [10:50] brendand: jibel: hello. I have a super critical UITK landing in the silo60. It is fixing 10 OTA9 bugs ... I would like to ask you to please put that silo on a fast track, so I can spin off a new landing this week with even more OTA fixes. [10:50] sil2100: ^ [10:51] bzoltan_, ack [10:52] jibel: thank you [10:52] bzoltan_: ok, nice === _salem is now known as salem_ === salem_ is now known as _salem [13:05] trainguards, something's not right here https://requests.ci-train.ubuntu.com/#/ticket/784 - I had to abandon the silo due to old/wrong versions sticking in the PPAs, but it still says "Name of Assigned Silo ubuntu/landing-023" and assigning does pick that same silo again [13:05] help? === alan_g is now known as alan_g|lunch [13:19] sil2100, Mirv: can one of you upload something for me? [13:20] Saviq: uuuh [13:20] Saviq: might have something to do with the trainwreckage from yesterday night [13:20] Saviq: I'll try looking into that in a minute [13:21] morphis: sure, what's up? [13:22] sil2100: see PM [14:04] sil2100, my request got assigned to a different silo after all [14:04] so I'm good again [14:04] Saviq: I saw some stale files there === alan_g|lunch is now known as alan_g [14:24] jibel, hey, can you confirm this comment https://bugs.launchpad.net/canonical-devices-system-image/+bug/1526328/comments/3 ? [14:24] Ubuntu bug 1526328 in Canonical System Image "blank area at the bottom of the screen in full screen mode" [High,Confirmed] [14:25] jibel, I can't seem to get gallery to do this now (although I've seen it yesterday and initially attributed to the u8 regression indeed) [14:27] fginther: hello. do you know if it is possible to set up jenkins to autobuild packages from lp:camera-app/staging and lp:qtubuntu-camera/staging ? [14:31] Saviq, I confirm that rotating the app makes the issue go away. [14:31] Saviq, gallery bug then? [14:31] jibel, yes [14:50] nerochiaro, we can get it added. do you need just the -ci jobs or -autolanding as well to merge MPs? [14:51] fginther: just jobs that autobuild packages [14:51] fginther: so that people can test them [14:51] nerochiaro, ack, thanks [14:51] fginther: thanks ! [15:26] cjwatson: hi. i've been seeing this dangling symlink error quite a lot the past few days. wonder if you might have some idea about it? https://launchpadlibrarian.net/230091672/buildlog_ubuntu-vivid-i386.pay-service_15.10%2B15.04.20151215-0ubuntu1_BUILDING.txt.gz [15:31] xavigarcia_: Silo 51 failed https://trello.com/c/59x9w35y/2573-645-ubuntu-landing-051-unity8-indicator-sound-xavi-garcia-mena-nick-dedekind [15:31] dobey: not off the top of my head; in that context that particular symlink would be expected to be dangling (it'll be pointing to a file in the runtime library package), but I'd have expected the "! -type l" there to exclude it [15:31] dobey: have you tried reproducing it in a local build? [15:34] cjwatson: it seems to only happen on vivid on launchpad builders. it doesn't happen in my jenkins builds, and there haven't been changes that would cause it. a retry usually gives a working build again [15:37] dobey: that would suggest a parallel-building race or similar [15:37] dobey: there's nothing special here, a local sbuild run with matching sources.list should reproduce it (eventually) [15:37] hmm, ok [15:47] trainguards... ugh... last build job from https://requests.ci-train.ubuntu.com/#/ticket/784 uploaded all the packages... somewhere into the void, they're not there in the silo PPA :/ [15:48] help/ [15:49] Saviq: maybe those got rejected? [15:49] Let me check the logs [15:49] sil2100, not sure why they would, FWIW I can't see anything like "uploading foo to silo 031" in the logs either [15:49] eh... [15:50] it's as if it signed them... and went away [15:50] a no wait there they are [15:50] Saviq: those got rejected... the reason is: this PPA already has the 15.04.20151215 versions of the packages [15:50] wha [15:51] https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-031/+builds?build_text=&build_state=all [15:51] sil2100, ok then those came from https://ci-train.ubuntu.com/job/ubuntu-landing-031-1-build/7/console [15:51] sil2100, but who deleted them... [15:51] hmmm [15:52] Didn't you re-assign the silo or something? [15:52] rvr: sorry, I was at a standup [15:52] sil2100, I did, but the previous one was 023 [15:52] Ah, hm [15:53] sil2100, we can try abandoning and reassigning again, yay [15:53] or I can wait until tomorrow to get version bumps :P [15:53] but would be good to know what happened anyway [15:53] I'm really shocked that the trian doesn't take Deleted packages into account when creating the version number [15:53] *train [15:53] rvr: that issue you point out is an already known issue... related with qml. Any app using that qml component to launch short sounds will make the audio role change from "multimedia" to "alert" [15:54] sil2100, trueth [15:54] rvr: so the game is running in "alert" sound mode... but changes to "multimedia" everytime it plays a sound [15:55] xavigarcia_: I understand the silo fixes that problem [15:55] rvr: and the warning is only shown when playing multimedia. But the sound is so short that the user has no time to cancel or approve the warning, as when it finishes it changes the audio role back to alert [15:55] rvr: nope, that was not the intention of the silo [15:55] rvr: in fact we cannot fix that from the sound indicator [15:56] xavigarcia_: https://bugs.launchpad.net/ubuntu/+source/indicator-sound/+bug/1484589 [15:56] Ubuntu bug 1484589 in pulseaudio (Ubuntu) "Spurious volume notification when changing output role with headphones" [Undecided,New] [15:58] rvr: the issue that is fixed for that bug is different [15:59] rvr: it was related with the normal notifications [15:59] xavigarcia: Ok, volume notification is no more [15:59] xavigarcia: But now we see the warning [15:59] rvr: yeah, but that's the one we cannot fix from the indicator [15:59] renatu: what's up with the qtpim landing, did you get the issue resolved / rechecked? [16:00] rvr: because it changes the game changes the audio role, which is managed in pulseaudio [16:01] rvr: we also have an issue related to that with videos [16:01] Mirv, I was able to reproce the problem, but I can confirm that is not related with the qptim changes [16:01] Mirv, I still investigating what is causing the problem [16:02] rvr: anyway... the main reason of the silo was the notifications that appear when the output changes (headphones/speakers) and so on... [16:02] rvr: and the problem is again that the code is adding new strings that need to be translated for OTA9 [16:03] rvr: so... What I suggest is to open a separated bug for the warning (although it's been already reported) [16:03] rvr: for example... here: https://bugs.launchpad.net/ubuntu/+source/pulseaudio/+bug/1485522 [16:03] Ubuntu bug 1485522 in Canonical System Image "Volume obtained is 100% in multimedia sink for sounds from app" [High,Confirmed] [16:04] rvr: and here https://bugs.launchpad.net/canonical-devices-system-image/+bug/1478506 [16:04] Ubuntu bug 1485522 in Canonical System Image "duplicate for #1478506 Volume obtained is 100% in multimedia sink for sounds from app" [High,Confirmed] [16:05] Saviq: anyway, could you try reassigning *again* ? ;p [16:05] I know it's stupid, but not much we can do in this PPA today [16:05] sil2100, nah, I'll just build tomorrow ;) [16:05] renatu: ok, so are you planning to fix that issue in the same silo or get the current silo landed first? [16:05] ...or that [16:05] ;) [16:06] Mirv, I do not think that is a qtpim problem, but I want to have 100% sure [16:06] renatu: ok, let QA know then if you want them to re-evaluate it [16:07] ok [16:25] trainguards: hey, silo 34 is sitting in the garage. Is there anything I should do? [16:36] Mirv, could you try to reproduce the problem? I tried 4 times and I can get this failing anymore [17:08] camako: Hello, we've just enabled devel-proposed for mir-mediumtests-runner-xenial-touch jobs: https://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-xenial-touch/16/console following your yesterday's conversation with fginther about it [17:12] psivaa, thanks I'lll check things out [17:51] jgdx: what's the issue? [17:52] robru, it has not landed [17:55] jgdx: looks like it's waiting for qa, right? === alan_g is now known as alan_g|EOD [18:14] robru, it passed [18:14] unless I dreamt it [18:15] jgdx: well the db did lose a day's worth of info due to catastrophic failure so it's possible that it passed and got reverted, but it doesn't say it got passed. please confirm with qa people [18:15] bzoltan_: eyes up any second [18:16] davmor2: https://www.youtube.com/watch?v=itAOGRiYRLI [18:16] robru, can't we use https://trello.com/c/I6sg1anH/2564-767-ubuntu-landing-034-ubuntu-push-jonas-drange ? [18:17] jgdx: ok right [18:17] thx [18:19] jgdx: publishing is unfortunately broken right now so please find a core dev to copy the packages to the archive manually [18:20] seb128, hey, could you help me out? ^ [18:24] jgdx, what package/ppa is that? [18:25] seb128, it's ubuntu-push-client to xenial and vivid+overlay [18:25] you don't have the copy commands handy right? [18:25] like things I could copy/paste [18:29] robru: would you be the right person to merge the silo60 UITK to the trunk and release the packages to the Overlay PPA and to Xenial? We might need a jedi's ack on the debian/ changes... like seb128 :) [18:34] bzoltan_: I don't have any publish powers unfortunately and also the train is broken again so I can't merge either. working on it [18:34] robru: does not sound good ... who has publish power? [18:35] bzoltan_: any core dev can copy the packages manually for you right now [18:35] bzoltan_: or you might need to wait if the diffs are not available for reviewing [18:36] robru: I see ... do you know when the train will be able to creat diff at least? [18:36] bzoltan_: I dunno maybe like an hour? this shit is fucked [18:37] robru: it is ... I know I am not much use, but if I can do anything for you I am still here for few more hours [18:38] bzoltan_: no it's ok, I'm working with magical webops people to get things sorted but it's like performing surgery blindfolded [18:44] Mirv, rvr, I can confirm that the problem that you faced on silo 10 is not related with the changes in the silo. Is related with a Know problem on QtNetwok that sometimes report a wrong connection state [18:49] jibel, so none of the bugs in silo 51 are actually fixed [18:50] I should change the status [18:54] pmcgowan, none of them since the silo has been reverted. I reverted the status. [18:54] jibel, thanks [18:54] too bad :( [18:56] alecu, do you want to break out that one change from the silo to land it? [18:59] pmcgowan: sorry, which change? I understand that the silo is a rework of a few things, so I have to check with xavi if it can be split. [19:00] alecu, silo 51 has 4 fixes and the one for headphone warning is blocked [19:05] pmcgowan: the first three bugs are closely related, and are fixed by the same branch. I'll ask xavi to see if we can split just the parts that belong to that headphone bug. [19:06] alecu, assuming the other fix is not coming soon but asked john about it [19:41] hmm [19:57] ok... [19:57] bzoltan_: things should be working if you want to try publishing === ljp is now known as lpotter [20:41] robru: I am afraid that the s390x build failure blocks me [20:44] robru: I have the fix for that here -> https://code.launchpad.net/~bzoltan/ubuntu-ui-toolkit/link_Gestures_more_explicitely/+merge/280630 [20:45] robru: What I need the most is to merge this landing branch to the UITK trunk, so I could start working on our very last landing of this year... [20:56] bzoltan_: ah ok, which silo? I can merge it [21:01] bzoltan_: oh, it would only say failed on s390x if that was a regression, so even if the publish job let you publish that it'd just get stuck in proposed... [23:38] robru: Simple question for you... [23:38] michi: what's up? [23:38] How can I stop a build in the train that’s currently in progress? [23:38] I just pushed another change to the branch and want to build the latest version, cancelling the current build. [23:39] Is there a “stop building” button somewhere? [23:42] michi: yeah there's a big red x on the jenkins job page [23:42] How do I navigate to that? [23:42] michi: be careful not to click that after the log says 'POINT OF NO RETURN' though otherwise stuff'll be corrupted [23:43] michi: if you're looking at the bileto ticket you can click 'build' and it opens the jenkins page [23:43] Ah, OK, I didn’t know that. [23:43] michi: you should see on the left a list of past jobs, including the running one, with an x [23:43] Doesn’t look like it: https://ci-train.ubuntu.com/job/ubuntu-landing-026-1-build/build [23:43] I see the past jobs, but not the one that’s building right now. [23:44] The arm builds are still in progress right now. [23:45] michi: oh, well, then, no, you can't cancel it. you can only cancel the source package build pre-upload. if the builds are in the PPA then you have no way to cancel [23:45] OK. Not a big deal. [23:45] I just thought it would be nice to release the resources early. [23:45] Thanks! [23:45] michi: yeah unfortunately the train doesn't expose that. I guess I could cancel but meh. just upload a new one when you're ready [23:46] Sure