[00:57] LocutusOfBorg1: I'm uploading a stack of rebuilds for the libxpv transition now [00:57] FYI since you were asking about it [04:05] Is anyone around how can help with publishing a silo? [04:06] cjwatson: ^ are you around? [04:07] I need someone to delete binary package for s390x for this: https://launchpad.net/ubuntu/+source/thumbnailer/2.3+16.04.20151102.2-0ubuntu1 [04:07] It never should have been built in the first place. [04:08] We can’t support this arch because gstreamer codecs don’t work there. [04:08] Now the existence of this package is preventing publication of silo 26, and I’m blocked on that for something else. [04:12] michi: But it passed the testsuite in the previous build? [04:12] No, not really. [04:12] That was a lie. [04:12] A lie? [04:13] Bascally, gstreamer codes for mp3 and mp4 do not exist on 390x [04:13] 100% tests passed, 0 tests failed out of 21 [04:13] At one point, I modified the tests to lie and just skip the mp3 and mp4 testing on s390x [04:13] That’s why they passed that one time. [04:13] Then jamesh, robru, and me discussed this. [04:13] So, now it has rdeps. [04:14] We decided that we do not want to support 390 because, once the package is published, we can’t regress, but we have no way to really make it work on that arch. [04:14] There are no 390 packages in the official archives, I believe. [04:14] We have not published since. [04:15] Err, what? [04:15] 21:14 < michi> There are no 390 packages in the official archives, I believe. [04:15] So we are not going to break anyone’s dependencies by deleting this now. [04:15] ? [04:15] The official archive is what you're asking me to remove packages from. :P [04:15] Has this been actually published? [04:15] It shouldn’t have been. [04:15] On a mainframe, no-one needs a thumbnailer. [04:15] The UI doesn’t exist, there is no desktop, etc. [04:16] You don't grasp how X works, I take it? :P [04:16] The whole 390 thing was simply a mistake on my part. [04:16] People run X clients on mainframes. [04:16] And they listen to music collections on mainframes? [04:16] Who knows what people do. [04:16] Look, we never meant to publish this. [04:16] It happened by mistake. [04:16] Anyhow, "we don't think people will use it on this platform" is never an excuse. [04:16] we cannot support the thumbaniler on s390x. [04:17] If we publish for that, then we will publish packages that don’t work when people try to use them. [04:17] And they can't be fixed, why? [04:17] Because gstreamer does not work on s390x, no-one maintains codecs for that. [04:17] Anyhow, it's been published since December. [04:17] michi: the build in the archive is not from your silo, it's from November 2 [04:17] Steve Langasek made it clear that ubuntu touch packages do not need to pass on 390 [04:18] Yes, robru, remember the discussion with james back then? [04:18] This was simply a mistake on my part and should never have happened. [04:18] Touch packages are becoming desktop-next, and desktop should work on s390x eventually. [04:18] michi: yeah i do remember that unfortunately it's not up to me to remove it [04:18] The s390x builds were added to the silos without anyone telling us. [04:18] I'm not arguing against removing it *for now*, I'm arguing against the "we can't possibly fix it" POV. [04:18] Then, before we got more tight on our tests, the thing slipped out. [04:19] The tests originally didn’t detect that failure at all [04:19] Then I made the test pass at one point by lying to the testing harness. [04:19] * Ignored failing tests on PPC due to gstreamer problems. [04:19] infinity: Well, no-one here at Canonical can fix the codecs. [04:19] Is that for the same issue? [04:19] infinity: yes. [04:19] If so, it sounds like it's a big-endian issue. [04:19] Nothing to do with mainframes. [04:20] Just bad math somewhere. [04:20] Similar problems, because the gstreamer codecs fall over on PPC [04:20] We have no interest in publishing the thumbnailer for things that are not on the phone. [04:20] Anyhow, if you're ignoring it on one arch, why not on both? [04:20] It has exactly one customer, which is the software on the phone. [04:20] infinity: I can do that. [04:20] Think to the future. Right now, it's the phone. Eventually, it'll be the desktop. [04:20] Except that we’ll be obliged then to keep publishing for s390x. [04:20] I know. [04:21] And it won’t be on the desktop unless someone fixes the codes where they are broken. [04:21] Which isn’t our job. [04:21] And if we had that attitude, we'd never be ready for arm64 phones either (oh, wait...) [04:21] infinity: Well that’s an entirely different kettle of fish. [04:21] Same fish. [04:21] Same attitude. [04:21] We have had repeated arm64 test failures too, due to bugs in components we have no control over. [04:21] "No one cares about !armhf, cause that's the phone". [04:22] No, I actually care a lot. [04:22] But I can’t make software work on a machine that I have no access to. [04:22] We have no arm64 hardware to work on. [04:22] Anyhow, if you're ignoring some tests on PPC for now, I recommend ignoring them on s390x too, make a big note that it's almost certainly an endian issue, and file a gstreamer bug to investigate further. [04:22] The porter boxes cannot be used because they are not available with the overlay [04:22] No one's forcing you to investigate, but a bug would be nice for reference. [04:23] I would be much happier if we could just delete the archive. [04:23] It was never meant to be published for 390. [04:23] This only happened by mistake. [04:23] Like I said, it has rdeps, I'd really rather just let it be. [04:23] So, you are recommending that I make the tests lie for 390 too? [04:24] michi: I recommend you make s390x and powerpc both skip the same test for now and file a gstreamer bug, yes. [04:25] OK. So, what is the correct procedure then? [04:25] silo 26 has been approved by QA. [04:25] Now I need to make change. [04:25] How do I do that? [04:25] Endian bugs are often bad math that comes back to bite you anyway, so worth looking at at some point. [04:25] Trust me, I have plenty of experience with endian bugs, and they don’t happen in my code... [04:26] michi: No idea what the silo procedure is. I'd just fix it in the archive if it were me. :P [04:26] So, what’s the right process to follow now? [04:26] We cannot get things into the archive without going through a silo because this is for touch. [04:26] And we are about to miss ota 9 [04:26] michi: But I assume you want to reupload to the silo, and skip QA if there are no code changes, only packaging/testsuite tweaks. [04:26] robru: ^? [04:26] And round and round we go. [04:26] Ah, that seems feasible. [04:28] michi: Sorry, not trying to be a dick here, and get your POV today. But, on the other hand, while a thumbnailer on s390x might not be something someone wants this year, gstreamer being busted is a big problem for a server, potentially. [04:28] (server side re-encoding, etc) [04:28] Sure, I take your point. [04:28] So, best to make sure the bug is known and someone's looking at it than to just say "anything using gstreamer can't work on s390x". [04:28] I also agree with you, in the sense that we shouldn’t just let shoddy software go. [04:29] michi: Anyhow, if you file the bug (after you sort out your silo -- priorities), I can make sure it gets some eyes. [04:29] It’s just that this is not our software and that, for components that run only on touch, we were told explicitly by slangasek that we do not need to pass on 390 [04:29] michi: And if we fix it, you can stop skipping tests, and everyone wins. [04:29] Not holding my breath. [04:29] infinity: not sure what you're asking me. Yes silos are fit touch landings and they build s390x if applicable [04:29] The gstreamer codecs are a complete train wreck in many cases. [04:30] michi: Sure, he's not wrong. You're not expected to have all your existing software start working. But you're expected to not have regressions. I get that this regression is a weird corner case, but. Meh. [04:30] robru: No, I was asking you if he can fasttrack and skip QA if he does a quick reupload with no code changes, just the testsuite tweak. [04:30] yeah, well, except for the accidental publication, it’s not a regression. [04:30] michi: There was on accidental publication, it was built in the archive. [04:30] robru: if I could do that, it would make a huge difference. [04:31] s/on/no/ [04:31] infinity: sorry i didn't read all the backlog. Did you guys decide to go back to skipping tests or something? [04:31] Disabling the tests for 390 is trivial. [04:31] Yes [04:31] robru: He already skips tests on PPC for the same issue. Makes no sense to be inconsistent there. [04:31] Ooh OK [04:31] michi: Would be better if you only skipped the one failing test instead of the testsuite, but not going to be picky when you're on a deadline. [04:32] Yeah if it's a trivial change i guess it's fine to skip qa but you do have to smoke test it to make sure nothing blows up [04:32] It’s thoroughly smoke tested. [04:32] Will test again, no problem. [04:32] michi: i mean a new build will need a new smoke test [04:33] K [04:33] So, should I add this change to the existing MR, or add a separate MR? [04:33] And once I’ve done that, how do I get the silo published? [04:33] michi: Thanks for putting up with my pedantry. And please do file the gst bug and point it at me after you've re-landed. [04:33] Won’t the QA approval revert? [04:33] michi: I can land it with force if no one's around to do it properly. :P [04:33] infinity: Will do. Thanks for your help! [04:33] michi: the qa field doesn't change unless somebody changes it [04:33] michi: (ie: I can copy to the archive and let robru pick up the pieces) [04:33] Any recommendation as to a separate MR or just commit to the existing branch? [04:34] infinity: the train detects that case and acts accordingly, nothing to pick up on my end [04:34] robru: ^-- Your machinery, what does he have to do? [04:34] michi: either is fine, whatever you want [04:34] robru: Oh, nice, if I do the copy out of band, it'll record the silo as landed and free it? [04:34] robru: Cool, thanks! [04:34] infinity: yup [04:34] robru: Shiny. [04:34] michi: you're welcome [04:35] infinity: it's slowly beginning to not suck so much ;-) [04:35] robru: Sadly, lack of suck almost always goes hand in hand with increased complexity. I bet I don't want to know HOW it doesn't suck, do I? :) [04:36] robru, infinity: silo is rebuilding now. [04:36] The train is bloody awesome. [04:36] It’s really easy to drive. I like it. [04:36] infinity: it pings the archive every 15 minutes on every silo to check if anything has changed and if it detects the version in archive has same hash then it merges ;-) [04:37] robru: Oh, sure, I meant the overall lack of suck. Each individual feature always looks elegant in isolation (or should), but when it's all put together, strong men weep and children run screaming. [04:38] At least, this is what I've learned over the decades of system architecture. [04:38] infinity: i didn't even make this feature on purpose, i wrote a status poller to make sure that new archive uploads didn't invalidate our builds, then i was like "wait, it's now trivial to also detect manual copies" [04:38] Yeah [04:41] infinity: Looking through the build log... [04:42] _XSERVTransmkdir: Owner of /tmp/.X11-unix should be set to root. [04:42] s390x is misconfigured in the silo, by the looks of things. [04:43] That shouldn't even be in the chroots. Lemme look if I oopsed. [04:43] We are running a test that uses xvfb to test that thumbnails have the right contents. [04:43] We’ve seen this before on some of the arm builders. [04:44] Oh. If you've seen it before on some builders sometimes, then it's not the builder's fault. :P [04:44] Since they all use the same chroots. [04:44] And indeed, that file isn't in the s390x chroot. [04:44] So it's a race in the process somewhere. [04:45] Well, it’s not anything we can fix. [04:45] that’s totally out of our control. [04:45] We just run xvfb [04:45] I suspect that's always in the log file. [04:45] And a red herring. [04:45] So, the dir is missing completely? [04:45] No. [04:45] But you only see it on failures. [04:45] 9: FAIL! : Thumbnailer::Photo::test_rotation_scaled() Unexpected pixel value at (0, 0) [04:45] Don’t think so. [04:45] 9: actual: #000000 [04:46] 9: expected: #FE0000 [04:46] Right. [04:46] That sort of thing being the real failure, I'd assume. [04:46] The thumbnailer painted black. [04:46] No, this is not a real failure. [04:46] It fails because xvfb can’t display the image. [04:46] On the previous failures, we got this problem because the ownership of that dir was wrong. [04:46] Does the dir exist for 390? [04:46] The directory doesn't exist at all in the chroots. [04:46] I just checked. [04:47] So it comes about at runtime. [04:47] It needs to be there, I believe. [04:47] Owned by root. [04:47] If it’s not owned by root, xvfb fails. [04:47] Doesn’t paint anything, and we read a zero pixel value. [04:48] It's not there in the x86 chroots either. [04:48] Aha. [04:48] So it should be created on the fly then. [04:48] Yes. [04:48] I have no idea why that’s not happening. [04:48] Anyhow, we can look at the why later. [04:48] Cool, thanks! [04:48] For now, get your quick fix through. [04:48] It’s building as we speak. [04:48] Ahh, so it is. [04:49] Oh, curious that ppc64el is in that list too.. [04:49] H 264 codec broken. [04:49] 100% tests passed, 0 tests failed out of 24 [04:49] From memory. [04:49] Or it doesn't need to be in the list anymore. [04:49] Yes, because we lie, as I said. [04:49] Heh. [04:50] See debian/rules [04:50] Yeah. Should definitely revisit all of this shortly. [04:50] BAsically, we still build and run the tests, but ignore the failures. [04:50] No, I'm saying that ppc64el no longer has failures. [04:50] ppc and s390x have 1 failure that you ignore. [04:50] Ah, it’s working suddenly? [04:50] ppc64el has 0, but you still ignore. ;) [04:50] That’s the problem with ignoring stuff. [04:50] But also, don't fix that right now. Deadlines and all. [04:50] you don’t notice when it breaks and starts working again :( [04:50] And I agree. Ignoring sucks. [04:51] Happy to help hunt and fix when it's not 10pm. [04:51] What a mess. [04:51] the problem with PPC is the same old one: [04:51] None of us has access to the hardware, so we can’t debug anything. [04:51] That's not true. [04:51] How are we supposed to make things work without hardware to do it with? [04:51] How so? [04:51] We have PPC porters. [04:51] They don’t work, I tried that. [04:51] None of them is available with vivid + overlay [04:52] But you're also building for xenial. [04:52] Yes, so? [04:52] So you could fix the bug on xenial... [04:52] Just because it works on xenial doesn’t mean that it’ll work on vivid. [04:52] At least in theory. [04:52] Well, I certainly can’t fix the bug myself. [04:52] But overlay literally only matters for armhf. [04:52] Don’t have the expertise to fix gstreamer codecs. [04:53] We discussed all this at great length with slangasek. [04:53] The whole gstreamer story is a disaster. [04:53] We have a kernel bug on arm that causes gstreamer processes to spin at 100% CPU. [04:53] They spin in the kernel, and a kill -9 won’t kill them. [04:53] I opened a bug for this months ago. [04:54] Huh. Fun. [04:54] tvoss pushed reallly hard to get it fixed. [04:54] No-one is interested. [04:54] On all kernels, or just some of the icky Android kernels? [04:54] Android. [04:54] The problem is somewhere with the low-level Android media stack. [04:54] Yeah. We can't support most of the Android stuff. Which is a whole extra mess. :/ [04:54] And, presumably, some driver is spinning at uninteruptible priority somehwere. [04:55] We had months of serious grief working around all the problems. [04:55] michi: Your PPA is all built for !armhf, if you intended to smoketest on x86 or something. [04:55] (armhf is catching up) [04:55] michi: so anything that we discussed, I would have been very clear in expressing that the packages should be fixed in a way (i.e. by dropping architectures from the source packages, and by handling removals of any reverse-dependencies that thereby become uninstallable) that does not require ongoing manual intervention from archive admins [04:55] On arm, we have disabled concurrency for gstreamer pipelines because, as soon as there is more than one running at a time (in different processes, mind you!), things go belly-up. [04:56] slangasek: The whole 390 publication thing was a huge mistake and basically happened by accident. [04:56] because the tests gave a false positive and were later fixed [04:56] Way back, we had a test suite that wasn’t anywhere near as thorough, and didn’t test mp3 and mp4 at all. [04:56] You keep saying that, but you didn't publish on s390x at all, we did. [04:56] so yes, always-failing tests can be ignored [04:56] Well, the problem is that the s390x thing slipped through completely behind our backs. [04:57] Because s-jenkins doesn’t build on s390x. [04:57] So, nothing in our process alerted us to the new architecture. [04:57] the silo just built and things worked because they weren’t tested. [04:57] then I tightened the test suite. [04:57] right [04:57] Found out about the build failure only when the stuff hit the silo. [04:57] then made the tests lie for s390x [04:58] After discussion with robru and jamesh, I backed that out again [04:58] Because of your comment in the email you sent that we can’t regress. [04:58] Anyhow, the current hack works for now. I'm happy to throw someone as investigating the "why" because while no one cares about thumbnailers, we should care about gst. [04:58] But, by then, the thing was published already. [04:58] That’s my best recollection of the sequence of events. [04:58] michi: It was built in the archive, no one accidentally published anything. [04:58] We have a *huge* problem with the s-jenkins and silo story. [04:58] Because the silos build on arches that s-jenkins doesn’t build on. [04:59] That means we find out about problems only at 5 minutes to midnight. [04:59] Or even not at all. [04:59] infinity: I don’t really know what that even means. [04:59] We build in the silo. The silo pushes to trunk, then the archive publication machinery picks it up from there. [04:59] michi: it means that two months after it was published, it built on s390x in the archive because we added a new arch and *all* packages were tried. [05:00] In that case, it most likely built in the silo for 390 without anyone of us even noticing. [05:00] And passed the tests only because the failing parts were not tested. [05:00] Sure. [05:00] As Steve points out, though, adding a new test that fails isn't a "regression" in general QA terms. [05:01] I see. [05:01] Hmmm... [05:01] Fixing it would be nice, but ignoring it is also valid while the fix is hard. [05:01] But I still have to lie, don’t I? Because, otherwise, I’ll never get the silo published. [05:01] (GCC and glibc run into this all the time, we create tests for things we *want* to fix, then get around to fixing them :P) [05:01] Wow :) [05:01] michi: Right, you have to lie to the testsuite (or invent an XFAIL), and then we sort things. [05:02] Well, I guess I’m lying thoroughly now. Because no matter what breaks on s390x, it’ll claim it worked. [05:02] I will go back and make that more selective [05:02] infinity: when I spoke of regressing, I was referring to the archive being unhappy with dropping an architecture once built and test-passed, without manual intervention [05:03] slangasek: Right, that more or less matches my definition, just in project terms instead of archive terms. [05:03] silo has “successfully build” s390x now :) [05:03] *built* [05:04] slangasek: But, basically, if a new test fails and it would have failed on the previous successful build, it's not a regression. [05:04] sure [05:04] infinity: Good point. Otherwise, it gets rather hard to improve the tests. [05:04] but autopkgtest expects the test *suite* to pass [05:05] adding a test to the testsuite that fails, and uploading that to the archive, doesn't help us [05:05] michi: I don't subscribe to the new world concept of TDD much, but for glibc, it's worked wonders to do things like "we want to clean up POSIX compliance in all our headers, we'll write a testsuite that checks how awful we are and then iterate". Cause on projects that large, something like that can be a 2-year process. [05:05] unless you XFAIL it, as mentioned [05:06] michi: So, yeah, we had to learn to XFAIL quite extensively to make the results usable. :P [05:06] Seems like everyone is lying ;) [05:06] It’s all an illusion anyway... [05:06] And, in the toolchain world, you always have things like "well, the kernel just plain can't do this yet on $arch, and someday we'll hook up the syscall" sorts of XFAILs too. [05:06] testing is messy. [05:07] It’s also saved my arse more often than I can count. [05:07] I think I’ll marry my test suite, actually. [05:07] It has saved me more often than my wife has by now ;) [05:07] She might not be cool with this plan. [05:08] I like my test suite better anyway ;) [05:08] Ouch. [05:08] just kidding... [05:09] I’ve been married for 32 years. I’m allowed to say such things. [05:09] Well, you're in QLD, hard to tell. [05:09] Ouch. :) [05:09] You haz armhf binaries now too. [05:10] robru: If you're still around, you going to push michi's buttons for him after he smoketests, so he can stop panicking? [05:10] robru: If I get no reply, I'll stick around and do the manual copy for him instead. [05:10] I just tried publishing and was refused. [05:10] I don’t have privileges. [05:11] infinity: yeah I'm off today so if you could copy that's be great [05:11] robru: My pretty all blue silo page has just gone all red again, because I tried to publish and wasn’t allowed to... [05:11] Do I need to kick off yet another build? [05:12] I need this to merge into trunk. [05:12] michi: why would you need to build again? Did you push a new commit? [05:12] No. It’s just all red now, complaining about the missing publish permission. [05:13] michi: that'll go away on its own [05:13] Sweet. [05:14] michi: Just let me know when you've checked for magic smoke, and I'll take care of the rest. [05:14] I’ve already tested locally with this change. [05:15] And seeing that I can’t test s390x binaries on a Nexus 4, I think we are good to go. [05:15] michi: I think robru was implying it's the binaries in the PPA that need smoketesting. [05:15] michi: Obviously not the s390x ones. :) [05:15] They are tested already to death, as in 30 minutes ago. [05:15] Yeah since they were rebuilt we just want to confirm that nothing broke due to build flakiness or something [05:15] And the previous few hours. [05:15] Ah. [05:15] Give me five please... [05:16] Sure. [05:16] Just poke me when you're done, I'll be around. [05:16] Make that a bit more, 20-30. [05:17] Oh Christ... [05:17] I flashed the phone with the latest rc-proposed, and now it doesn’t boot anymore. [05:18] infinity: How much longer will you be around? [05:18] michi: Couple of hours. [05:18] I’ll try an reflash from the boot loader. [05:18] michi: I'm in no rush, just trying to be helpful since I made you go through this effort. :P [05:18] michi: I'll be kicking around doing !work things and checking in here and there. [05:18] I do appreciate that! [05:19] It’s sometimes kind of lonely, given the timezone I’m in. [05:19] michi: Yeah, I found that when I lived in Melbourne. I fixed it by being awake 23h/d. [05:19] It’ll take half an hour to download the latest image. [05:19] How nice... [05:58] infinity: Mirv helped me out and has just pressed the publish button. [05:59] He’s done the smoke testing. [05:59] michi: Shiny. [05:59] I just got my phone back to life after bricking it. [05:59] Doing more testing now too, just in case. [05:59] Thanks again for stepping in to help! [09:24] thanks cjwatson :) [10:25] infinity: so, should I just start uploading lts-w though libdrm is still in the queue? [10:25] tjaalton: Does it all have a versioned build-dep on the new libdrm? [10:25] (either declared or undeclared...) [10:25] infinity: just the ones that need it [10:26] tjaalton: Hold on. Lemme try to review this a bit. [10:32] tjaalton: The changelog and the package don't agree... [10:33] http://paste.ubuntu.com/14429036/ [10:33] tjaalton: Changelog claims you dropped the revert patch, but it's clearly still there. [10:36] huh, so it seems [10:37] tjaalton: That said, if that .symbols change goes with it, I would say you need that patch. [10:37] tjaalton: And I question how we let upstream get away with this in later versions. :P [10:38] libdrm2 is not libdrm2 anymore if it dropped a symbol. [10:38] tjaalton: Anyhow, please either fix the changelog to match reality or reality to match the changelog, and explain why you chose the option you did. :P [10:39] infinity, is that the entire diff, if so there is nothing left if you drop that patch too ? [10:39] infinity: I'll figure it out [10:39] been 1,5mo since that :) [10:39] apw: Well, yes, but it's a backport, so what would be left is the changelog. [10:40] infinity, right, but the changelog also says there is a change to control [10:40] apw: In the previous backport, not this one. [10:41] point [10:46] infinity: apparently they were only used internally [10:46] within libdrm [10:46] tjaalton: Then why revert the commit in the past? [10:46] can't remember [10:47] tjaalton: Possible, perhaps, that an older version of X or some X driver abused that symbol? [10:47] I'll grep through the driver trees [10:47] tjaalton: And we need it to run trusty X/drivers with the backported librm? [10:47] libdrm, even. [10:48] tjaalton: Anyhow, seems safest to me to maintain that status quo, even if you can't find the original reason. [10:49] tjaalton: Though, no need to have the rest of that commit reverted, it's really just the visibility of the symbol that you're worried about. [10:49] right [12:02] Could someone remove the php-fpm binary from xenial-proposed please? It's blocking the proposed migration of php-defaults 21 I think, which no longer builds that binary. I'm not sure we want to commit to php-7.0 in Xenial yet, but I think it would be easier see that other things aren't blocking proposed migration if/when we attempt to move across, and presumably we can always delete it from the rele [12:02]