=== jamesh_ is now known as jamesh [09:31] hey folks [10:06] om26er: hello! Are you free for some maguro testing of image 76? :) [10:06] sil2100, sure, let me upgrade [10:06] om26er: thank you! [10:16] sil2100: what was the blocker bug for promotion yesterday? something related to setting or upgrades? [10:16] sil2100: is it fixed now? [10:18] lool: hm, there was no particular promotion blocker, since we promoted all we could - we didn't promote any other image because we didn't manual-test those through ;) [10:19] lool: there was an online settings test crash on maguro, but it's just flaky [10:19] can someone test video playback on the latest image? [10:20] it's broken here [10:20] I'll try upgrading [10:21] also. why does mediaplayer do facebook/twitter/friends stuff? [10:22] sil2100: ah ok [10:22] I just realised that I only get notification of tweets / facebook messages _after_ I open a video [10:22] * popey will file a bug [10:23] popey: video playback >> regression from :73? [10:23] yes [10:25] sil2100: did we keep the URL of the online settings issue? would be good to escalate [10:26] lool: not yet, someone noticed it late late evening yesterday, so it was already my EOD [10:27] sil2100, on the latest image video player does not play videos, seems it crashes [10:27] sil2100, on both mako and maguro that is [10:28] Ouch [10:28] Ok, so it's confirmed then, let's see the changes between those images [10:29] in one of the image gstreamer0.10-ffmpeg was dropped, probably that ? [10:29] http://people.canonical.com/~ogra/touch-image-stats/20131218.changes [10:29] Yes, see that as well, hm hm [10:29] sil2100, where is the official place to check changes between images ? [10:30] I wonder why? [10:30] om26er: I always use the pages generated by ogra_'s magic [10:31] haha seems popey already reported that ^^ :p [10:37] http://paste.ubuntu.com/6598991/ [10:39] om26er, popey: anyway, try filling out a bug for this one, in the meantime I'll try to check what dep-change could have really caused this [10:40] ya [10:40] will file a bug [10:47] om26er: bug 1262600 [10:47] bug 1262600 in mediaplayer-app (Ubuntu) "mediaplayer no longer plays video in image #76" [Undecided,New] https://launchpad.net/bugs/1262600 [10:53] also bug 1262607 [10:53] bug 1262607 in friends (Ubuntu) "Social notifications only appear after mediaplayer is opened" [Undecided,New] https://launchpad.net/bugs/1262607 [10:57] sil2100, did ralsina tell you want was needed for udm? I had to EOD because it was very late and told him to take care of that [10:57] mandel: he got the rebuild started, it's in the landing plan, manual testing was discussed [10:57] ralsina, awesome! [10:57] ralsina, buenos dias! ;) [10:57] mandel: but it all happened in like 3 minutes after you left :-) [10:58] ralsina, great! [10:59] also bug 1262611 [10:59] bug 1262611 in unity8 (Ubuntu) "Network indicator doesn't display until you pull down" [Undecided,New] https://launchpad.net/bugs/1262611 [11:00] mandel, ralsina: hello! [11:00] So, I expected that Ken or Robert will handle the release of this one yesterday, but sadly it didn't happen [11:01] I'll pick it up in a moment and do it till EOD [11:01] sil2100: awesome, thanks! [11:01] (before EOD) [11:01] Just need to finish something first and deal with the mediaplayer-app regression [11:02] sil2100, superb! thx a lot [11:08] if you have an image which still has the gst stuff installed, install aptitude and then run aptitude why [11:22] sil2100: lool: maguro tests completed, re-running unity8 now. pls let me know if you want me to rerun anything next [11:23] psivaa: thanks! [11:23] lool: I think we can kick a new build in the meantime [11:24] But no promoting until the mediaplayer-app flock is resolved [11:24] sil2100: is the gst thing fixed then? [11:28] sil2100: ok [11:29] sil2100: buildin [11:30] popey: no, but we won't make it for the morning build anyway - if we fix it we'll catch it in the evening [11:44] sil2100: who needs to help on the flock issue? [11:44] or is it all sorted? [11:45] asac: for now I don't have time for that, we have a big regression with video playback and I'm trying to get to the root cause [11:45] sil2100: i referred to: [11:45] 12:23 < sil2100> But no promoting until the mediaplayer-app flock is resolved [11:46] hence i thought that was the main blocker :) [11:46] sil2100: who owns mediaplayer? [11:46] sil2100: was that regression in the promoted image? [11:46] or is that new? [11:47] asac: from what popey said it wasn't in the promoted image [11:47] popey: you tested on 73 and it was ok there, right? [11:48] 73 works, 76 does not [11:48] did not test 74 or 75 [11:48] popey: ok, so it's not gst lack's fault [11:49] Damn, it doesn't make much sense - I tried it on my device and I get a segfault as well [11:50] asac: bfiller and renato are responsible for mediaplayer, both are not on - but let's see what we can do by ourselves [11:50] sil2100: by ourselves? thats not really the way we should do it [11:50] get them involved [11:50] of course if you have identified a source through bisecting fior them thats good [11:50] but rather escalate early and often [11:51] than holding on to things for ourselves to clean [11:51] we got this: http://people.canonical.com/~ogra/touch-image-stats/20131218.changes [11:51] dropping stuff [11:52] Yes, get them involved, but I'm not going to block on them not being around [11:52] asac: but it's 73, which was good [11:52] sure not block [11:52] thought they were around [11:52] if we are sure [11:53] then its probably lxc-android-config and friends [11:53] and lxc [11:53] try downgrading those [11:53] http://people.canonical.com/~ogra/touch-image-stats/20131218.1.changes [11:53] other candidate (maybe even more related frfom topic) is https://www.flyn.org/projects/libdmapsharing/ [11:53] we upgraded that [11:54] http://people.canonical.com/~ogra/touch-image-stats/20131218.2.changes [11:54] sil2100: try downgrading those two things [12:14] asac: I already tried libdmapsharing earlier but didn't help, now I'm trying to recover from my device being broken by a downgrade of lxc [12:14] But I'm slowly triaging [12:19] sil2100: check with -touch folks [12:19] i think its not trivial to downgrade lxc [12:19] etc. [12:19] ogra_: ^^ do you nkow :) [12:19] (sorry) [12:20] asac, hmm, ask stgraber ? ... (he is in europe) [12:20] is that about video playback ? [12:21] (why are you discussing lxc ?? ) [12:21] ogra_: yeah. video playback regression [12:21] ogra_: because its pretty much the only thing that changed [12:21] but look :) [12:21] right, must be a dependency change somewhere [12:21] oh wait [12:21] i think i misrad the changes :) [12:21] gstreamer0.10-ffmpeg was dropped for whatever reason [12:22] its either http://people.canonical.com/~ogra/touch-image-stats/20131218.2.changes [12:22] i see that xnox did demote it recently to a recommends in another package (which would remove it from touch if that was the dep we need) but thats was end of nov and doesnt seem related [12:22] http://people.canonical.com/~ogra/touch-image-stats/20131219.1.changes [12:22] or http://people.canonical.com/~ogra/touch-image-stats/20131219.changes [12:22] ogra_: the image produced based on 18.1 was good [12:22] It's not lxc's fault anyway [12:23] definitely not [12:23] Image 74 plays fine [12:23] ogra_: it's dropped for very good reasons, which are outlined in the bug report. [12:23] right [12:23] hence [12:23] http://people.canonical.com/~ogra/touch-image-stats/20131218.2.changes [12:23] xnox, might be, it is essential for touch [12:23] http://people.canonical.com/~ogra/touch-image-stats/20131219.changes [12:23] http://people.canonical.com/~ogra/touch-image-stats/20131219.1.changes [12:23] are the candidates [12:24] we checked libdmapsharing [12:24] guess if lxc we can rule out [12:24] there is only upstart-app-launch [12:24] left [12:24] sil2100: tried that? [12:24] ogra_: it is not. we legally cannot ship ffmpeg, it's blacklisted, it's not installable in the archive and about to fall out. If you need back the support for those codecs, the app in question must be ported to gstreamer1.0 and use libav [12:24] i guess if its about "cant start playback" [12:24] it smeels like its again upstart-app-launch [12:24] ogra_: the bug has been set at critical priority, since saucy, yet it hasn't been touched nor fixed yet. [12:25] xnox, well, without gsdtreamer.-ffmpeg, no video playback [12:25] ogra_: the image 73 and 74 are good [12:25] asac: I'll check that, but damn the app is starting fine, just crashes suddenly [12:25] the drop happened for 73 [12:25] so... [12:25] ogra_: it's not installable either any more. it's about to fallout on all architectures from the archive. [12:25] sil2100: are you sure its not on 74 and 73? [12:25] xnox, we added it on request of management [12:25] asac: sure, it's not gst [12:25] xnox, that means no video playbaclk on touch then [12:25] ogra_: good, it's also blacklisted from all ubuntu releasable images as per TC. [12:25] asac: I just have 74 on my device and it's fine, 73 was fine as well as popey said [12:25] xnox: yes. it was added explicitely. who triggered the removal? [12:25] ogra_: please use other codecs. [12:26] xnox, there were long discussions with legal and management before pulling it in [12:26] xnox, we cant [12:26] I'll just continue digging, no need for everyone to look into that [12:26] if you want to override the TB you should talk to the TB [12:26] not Canonical management [12:26] ogra_: in that case please use gstreamer1.0-libav as soon as possible. let me look up the bug report. [12:26] asac: what's the progress on https://bugs.launchpad.net/gallery-app/+bug/1221968 ? [12:26] Ubuntu bug 1221968 in gallery-app (Ubuntu) "GStreamer 1.0 port of video thumbnailing" [Critical,In progress] [12:26] cjwatson, lool and pmcgowan were involved in the discussion ... i even refused ot have my name on the seed change [12:27] asac: this has been critical since september and still not resolved. [12:27] please talk to them [12:27] xnox: dot know the status [12:27] all i know is that ffmpeg is currently needed [12:27] asac: this is the correct way to pull in gstreamer ffmpeg backend in trusty [12:27] xnox: still, who triggered the removal>? [12:27] ogra_: I didn't address you specifically; anyone who's attempting to override the TB for something within the domain of the Ubuntu project must talk to the TB [12:27] rather than working around them [12:27] asac: can you please get it fixed faster? [12:27] i dont know [12:27] it's not OK to just ignore it [12:27] i am not familiar with the technical reasons [12:28] i just want to ensure that we dont shart shoot each others and talk to each other [12:28] before doing things like this [12:28] asac: the demotion of ffmpeg1.0 dependency was blocking approximately 200 packages migration in -proposed, including gtk stack, server stack, samba4 stack. This was done part of the elaborate libav9 transitions. [12:29] i dont understand [12:29] asac: it's part of the plan to resolve _not_ having two sets of gstreamer stacks on the touch & desktop images. [12:29] i dont want to know the reaons, just who did it and why there was not even a single line of talk about it [12:29] xnox, but we cant remove the working stuff before we have the new stuff ready [12:29] are you sure there was not a single line of talk about it, rather than talk you didn't see? [12:30] I saw lots of chatter about the libav transition [12:30] asac: the problem is that gallery-app is not ported to gstreamer1.0 and all requests to get that sorted have been ignored by gallery-app upstream, which is holding back the rest of the ubuntu. [12:30] noone appraoched those that are doinmg the touch image and try to bring a phone together [12:30] at least i didnt hear it [12:30] asac: i've appoach lool and assignee of bug report bug 1221968 about it. [12:30] bug 1221968 in gallery-app (Ubuntu) "GStreamer 1.0 port of video thumbnailing" [Critical,In progress] https://launchpad.net/bugs/1221968 [12:31] xnox, so instead of trying harder to get uphstream to fix it you just make the touch images unusable ? [12:31] asac: who should I have talked to about it? [12:31] ok [12:31] well. [12:31] xnox: landing team [12:31] ogra_: how is this a productive approach [12:31] xnox, upstreams manager ? [12:31] asac: landing team does not right code? [12:31] xnox: if you feel you might change the features of the touch image, alwasy go to the lnding team [12:31] ogra_: how would i know who that is? [12:31] asking ? [12:31] xnox: its not about the solution. its about telling someone that you are about to kick ffmpeg out of our iamge [12:32] anyway. so whats the fallout? [12:32] ogra_: ? [12:32] cjwatson, it is productive to find a way to solve this differently than by making touch images unreleasable the next time it happens [12:32] do we have no working audio? [12:32] people said that image 73 was good [12:32] so not sure if they didnt test it [12:32] ogra_: shouting at people is not productive [12:32] i didnt shout [12:32] shouting? [12:33] didnt hear that :) [12:33] i asked why it was not tried harder to get upstream to fix it [12:33] I assume that gallery-app must have gone through the landing pipeline, since it was daily-released [12:33] asac: this was discussed on #ubuntu-release and #ubuntu-devel and was proposed and reviewed and accepted upstream which you should see plenty of bug-mail and code review changes, which has passed all of the CI and autopilot on premerge and post-merge. [12:33] asac: and it was done a long time ago. [12:33] however due to the abysmally crappy spreadsheet I cannot check history [12:33] cjwatson, it sure did [12:33] therefore, the landing team was consulted [12:33] kk [12:33] if that was done then ignore [12:33] but you have chosen to micromanage everything so you probably can't see the wood for the trees as a result ... [12:34] so meh [12:34] I should go on holidy [12:34] a [12:34] * ogra_ should go back to that again too [12:35] sil2100, so did you try to roll back gstreamer and reinstall ffmpeg already ? [12:35] ogra_: the image that has the gstreamer change didnt have the regression [12:35] hence this whole discussion is slightly moot :) [12:35] how do I check this bug? [12:35] ogra_: yes, actually - using an image that's already missing the gstreamer stuff it still works fine [12:35] ogra_: but you said you knew that there is now something broken, what exactly is it? [12:35] asac: you made me feel really bad just now. i do not feel welcome. And the tone of above discussion and how it was started, is over the top for me, what i consider a healthy working environment. Especially, when it's unwarranted. [12:35] ogra_: so it's not gst [12:35] * sil2100 is bisecting further [12:36] One by one, I'll get to it [12:38] ogra_: i tried reinstalling gstreamer on #76 and it didn't fix the issue [12:38] k [12:38] I still couldn't play video [12:38] popey, ffmpeg as well ? [12:38] it pulled in all the dropped packages [12:38] the ones listed in the bug [12:38] and rolled back the updated ones ? [12:39] no, i just added gstreamer and the libav friends [12:39] didnt roll anything else back [12:39] ogra_: but as I said, image 74 which is already after the dropping works fine [12:39] oh, please do that too [12:39] revert what? specifically? [12:40] This regression makes me sad :< [12:40] meh, shit happens [12:40] libgstreamer0.10-0 [12:42] revert to 0.10.36-1.2ubuntu2 right? [12:42] as per http://people.canonical.com/~ogra/touch-image-stats/20131218.changes [12:42] yep [12:42] k, ta [12:42] xnox: yeah sorry, wasnt very well done well by me. you are very welcome [12:43] i didnt want to know about the WHO because i want to complain about him doing something wrong, i just wanted to understand who to talk to to figure if the coordination on this was done right or if we have a process problem. [12:43] asac: apologies accepted. [12:44] E: Version '0.10.36-1.2ubuntu2' for 'libgstreamer0.10-0' was not found [12:44] bah [12:44] xnox, sorry, i think it was my fault steering it in that direction ... i didnt mean to blame you specifically [12:44] ☃ ☃ ☃ ☃ ☃ ☃ ☃ ☃ ☃ ☃ [12:44] popey: you have to dl it from LP manually, sadly [12:44] oof [12:45] xnox: so by now i think i guessed it was you and you did it properly and we dont even have a known regressiono from the issue. so i am happy :) [12:45] popey, https://launchpad.net/ubuntu/+source/gstreamer0.10/0.10.36-1.2ubuntu2 [12:45] ogra_, popey: the sad thing... I think I upgraded every package that makes sense from image 74 and I still have working video! All that is left is autopilot-related packages o_O [12:45] https://launchpad.net/ubuntu/+source/gstreamer0.10/0.10.36-1.2ubuntu2/+build/5326960/+files/libgstreamer0.10-0_0.10.36-1.2ubuntu2_armhf.deb looks good [12:45] asac: right, but that would be WHAT and WHERE =))) e.g. upload changelogs, merge proposals, landing spreadsheet / bug tracker mail / history. The more audit logs, the better. Which of all the listed, landing spreadsheet is the one that has the least audit history. [12:46] asac: anyway, let's work on resolving any anomalies that are present. [12:46] sil2100, thats why i think it is gstreamer .... probably there is some user db in ~/ thats not properly set up on fresh installs ... none of the package changes after r73 can have caused this [12:47] nope [12:47] downgraded libgstreamer and added gstreamer0.10-ffmpeg and friends, and video still doesn't play [12:47] k [12:47] hm, maybe ogra_ is right, like... [12:47] the one thing we dont reflect are android changes ... [12:47] Maybe I should do a whole wipe of my system and then try 74 [12:48] were there any android uploads the last days [12:48] that could have broken the codecs on the android side ? [12:48] ogra_: do you know of a place where we could check that? [12:48] well, android is a package like any other :) [12:49] uploads should have been asked at the landing meetings [12:49] * ogra_ checks the changes ML [12:49] is that in /system/vendor/lib ? [12:49] I just did a apt-get dist-upgrade on my device even and no breakage ;p [12:50] not sure where the codecs actually live [12:50] looks like /system/lib [12:50] sil2100, that kind of points to android, there were two uploads yesterday, but neitrher should have any impact [12:50] hm hm [12:50] popey, rsalveti or jhodapp should be able to tell you what to look for [12:51] ogra_: thanks for your help! [12:51] ogra_: now, I think it's time to get back on holidays ;) We'll try to poke people around for this [12:51] heh, k [12:51] ogra_: at least now we know a possible direction in which to go! [12:52] http://paste.ubuntu.com/6214995/ ... thats the way to replace the android bits [12:53] (grabbed from the android deb you want to use) [12:53] you want to test 20131202-2236-0ubuntu2 [12:54] * ogra_ goes back to play with his home automation ... [12:54] sil2100: daker says #75 has the problem too [12:54] which narrows it down a bit [12:55] android changed between 74 and 75 [12:56] http://ci.ubuntu.com/smokeng/trusty/touch/ ... 20131211.2 in 74 vs 20131218.2 in 75 [12:56] Ok [12:56] So... [12:56] ok so i guess we dont have a good view on those changes? [12:56] We at least know that there is a 90% chance that it's android [12:56] (that should really use the package version ... not sure why we add a date there) [12:57] asac, trusty-changes has ... first one was just emulator related ... second replaces the opendjk version that we dont really use in the device images (also emulator related) [12:57] asac, but it could be that i.e. something in the toolchain changed or some such [12:57] doesnt sound like it could cause issues, but well. what else could it be [12:58] how can we test this? [12:58] right, someone needs to roll back the android part on a i.e. 76 image [12:58] see the pastebin above [12:58] just downgrading android package and run some tool to update the container? [12:58] garb the android deb, unpack and then do the steps from the past [12:58] kk [12:58] *paste [12:59] sil2100: you think thats good enough to try? http://paste.ubuntu.com/6214995/ [12:59] .... we did have eglibc 2.17 -> 2.18 changed recently. it was validated a lot, but there are subtle breakage here & there (e.g. i did have to fix cross toolchains, and java exposed a random / non-deterministic crash) [12:59] can eglibc down-grade be tried? [12:59] asac: I'm reading this up, I guess I can try this - but I would have to upgrade to the latest image [12:59] popey: or maybe you feel adventurous? What image does your device have? [13:00] xnox: when did that change happen? [13:00] you can theoretically omit the boot.img stuff from the paste btw ... [13:00] system.img is the important change you want to look at [13:01] (kernel and initrd didnt change in a while) [13:02] sil2100: 76 currently [13:02] xnox, http://people.canonical.com/~ogra/touch-image-stats/ nothing there shows a libc update ... (between 20131218 and 20131219.1) [13:02] i see a libneon though [13:02] * popey lunches [13:02] ogra_: neon only had a ftbfs fix on !armhf [13:02] but only in the last image [13:03] right [13:03] ogra_: I tried upgrading libneon as well, so damn [13:03] cant be it [13:03] well, try the android rollback [13:03] Will do [13:03] popey: can you try that ^ I'm upgrading now to something newer so that I can try downgrading [13:03] only line 6 to 15 [13:04] (and 1+2) === cjohnston changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: cjohnston | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: - [13:05] ogra_: is this the package's deb we need to fetch? https://launchpad.net/ubuntu/+source/android/20131202-2236-0ubuntu2 [13:05] right [13:06] https://launchpad.net/ubuntu/+source/android/20131202-2236-0ubuntu2/+build/5325024/+files/android_20131202-2236-0ubuntu2_all.deb in fact [13:07] void QGstreamerPlayerSession::processInvalidMedia(QMediaPlayer::Error, const QString&) [13:07] void QGstreamerPlayerSession::stop() [13:07] 0:00:01.014680298 3457 0x1a9c600 WARN mirbufferpool mirpool.c:318:gst_mir_buffer_pool_release_buffer: delegate is NULL, rendering will not function [13:08] void QGstreamerPlayerSession::setSeekable(bool) false [13:08] AbstractPlayer: Internal data stream error. [13:11] sil2100: try what? [13:11] aha, android rollback [13:12] GST_DEBUG=*:3 mediaplayer-app /home/phablet/Videos/small.ogg --desktop_file_hint=/usr/share/applications/mediaplayer-app.desktop [13:12] is useful [13:15] Laney++ [13:16] not sure why reverting stuff is the first step in debugging [13:17] Laney: yeah, reverting individual pieces takes one into a _new_ state, not into a _known_ state (good/bad) [13:26] It wasn't the first step in debugging, I actually tried looking at the output but since I'm completely unfamiliar with the codebases, I moved on to a different way of finding the problem [13:26] Not to block on waiting for upstream to appear, as it's the upstream who should help in finding the bug essentially [13:34] ogra_: where can I get simg2img ? [13:34] android-tools-fsutils [13:35] ogra_: thanks again! [13:35] Since dlocate was silent today [13:44] \o/ [13:44] ogra_, popey, asac: after downgrading android, video playback works! [13:44] good [13:45] sil2100: nice one :) [13:45] rsalveti: ^^ [13:45] :) [13:45] now the question is why xnox' uploads with only unrelared changes can have caused this ... i guess you need to ask rsalveti [13:45] * ogra_ guesses there are additional in-tree changes that were picked up [13:46] Strangeness, but at least the component is locked on and we can poke specific people [13:47] ogra_: thanks for all the good advice, I wouldn't know how to downgrade android completely ;/ [13:47] now you do :) [13:51] ogra_: well no change rebuilds do trigger bugs. [13:51] ogra_: we've had that in the past. compare build-logs / build-deps used? [13:52] xnox, for android ? [13:52] ogra_: yeah. it fetches stuff from the archive at build-time. [13:52] i dont think the deps changed at all [13:53] though indeed someone should check the logs ;) [13:53] but thats up to someone not on vacation ;) [13:57] morning [14:16] ogra_, didn't the kernels change? [14:17] sergiusens, not within this week i think [14:17] ogra_: I'll try taking a look after lunch [14:26] ogra_: hey [14:26] ogra_: thought you were in vacation [14:26] what's up? [14:26] let me read the backlog [14:27] rsalveti, i'm working on my home automation system (well, mainly on the heating control) [14:27] so i'm around on the kbd ;) [14:28] rsalveti, looks like one of the recent android rebuilds broke video playback [14:29] rsalveti, rolling back to 20131202-2236-0ubuntu2 makes it work again [14:30] wow, interesting, wonder how [14:31] hm, maybe because the video compat side changed as well as it used the latest hybris version [14:31] which device? [14:31] yeah, i was thinking about hybris too [14:31] mako mainly ... not sure someone tested on maguro [14:32] yes, maguro too [14:32] i tested mako, om26er tested maguro [14:33] great, let me take a look [14:36] Back [14:36] rsalveti: it's on both mako and maguro, only tested those two [14:37] yeah, I should have enough now to debug it further [14:37] thanks a lot [14:38] rsalveti: thanks for taking a look into that! Let us know once you have some info ;) [14:39] sure :-) [14:53] hm, the hybris compat side didn't change comparing with the previous version [14:58]