[10:59] <Odd_Bloke> I'm hitting a failed to upload on https://launchpad.net/~daniel-thewatkins/+livefs/ubuntu/wily/cpc/+build/37706
[11:00] <Odd_Bloke> I've managed to do it twice now, so I don't think it was a random failure.
[11:00] <Odd_Bloke> I'm playing around with the files that I output from live-build, so it's probably something to do with that.
[11:00] <Odd_Bloke> cjwatson: wgrant: Can you shed any light on why I'm hitting ^?
[11:00] <wgrant> Let me see.
[11:01] <wgrant> UploadFailed: Invalid length: 0
[11:01] <wgrant> Odd_Bloke: You have an empty file, which makes the librarian sad.
[11:01] <wgrant> I thought we excluded those in launchpad-buildd, but maybe not.
[11:01] <Odd_Bloke> OK, that's easy enough to remedy.
[11:01] <Odd_Bloke> wgrant: Ta.
[11:11] <cjwatson> wgrant: I think I was a little reluctant to possibly end up papering over problems by ignoring such things.
[11:11] <cjwatson> Though the OOPS of course isn't ideal.
[11:12] <wgrant> Indeed.
[12:10] <jgdx> I can't assign/change status on bugs. Known issue?
[12:10] <jgdx> OOPS-ca98aa95bdece3ad7bf7accced65c91d is one example
[12:14] <jgdx> the js error also comes back as a html-page, so it's hard to read the error: http://i.imgur.com/8EvY2uU.png
[12:17] <cjwatson> jgdx: We get occasional reports of this, and don't know the exact cause, but it normally clears up quite quickly
[12:18] <cjwatson> It's a very slow UPDATE on the BugTask table, probably a long transaction somewhere else ...
[12:19] <wgrant> Not exactly: https://lpstats.canonical.com/graphs/AppServer5XXsLpnet/20150908/20150915/
[12:19] <wgrant> Something is up.
[12:19] <wgrant> Some Ubuntu bug searches timing out.
[12:19] <wgrant> It's possible that a particular scripted search has fallen off a cliff and those long queries are somehow causing contention.
[12:19] <cjwatson> Worth a pg_stat_activity check?
[12:20] <pc_m> Hi! Anyone else noticing issues with launchpad? I'm trying to create a bug under neutron, and get an "Uh oh!" message repeatedly.
[12:20] <wgrant> It probably won't show anything, but worth a try.
[12:20] <jgdx> cjwatson, okay. I know there's the occational hiccup, but this is now 100% of the time.
[12:21] <cjwatson> pc_m: That's the current discussion.
[12:21] <pc_m> cjwatson: Ah, just joined channel.
[12:22] <pc_m> cjwatson: FYI, it complained about load balancer timing out. After 4th try, it created the bug.
[12:24] <pc_m> I did the reload button, and once it did work, it created multiple instanced (so looked like each request did work).
[12:28] <cjwatson> Hammering it isn't going to give us additional information at this point
[12:46] <wgrant> jgdx, pc_m: It should be OK now, let us know if you have any more issues.
[12:46] <wgrant> One of our DB slaves went a little overboard.
[12:51] <jgdx> wgrant, the failing action now completed. Thanks!
[12:51] <wgrant> Great, thanks for confirming.
[14:22] <Odd_Bloke> https://launchpad.net/~daniel-thewatkins/+livefs/ubuntu/wily/cpc/+build/37714 seems to be stuck; can anyone shed light on why that might be?
[14:23] <Odd_Bloke> (https://launchpad.net/~daniel-thewatkins/+livefs/ubuntu/wily/cpc/+build/37717 is also stuck in the same place, for confirmation)
[14:24] <Odd_Bloke> wgrant: ^ ?
[14:28] <cjwatson> Odd_Bloke: Do you have a similar build already that you've cancelled to see what the unflushed tail of the log says?
[14:28] <Odd_Bloke> cjwatson: I don't; cancelling that one now.
[14:34] <Odd_Bloke> cjwatson: Cancellation finished a couple of minutes ago, but no build log yet.  Am I just being impatient?
[14:36] <cjwatson> Odd_Bloke: It failed to cancel and gave up.
[14:36] <cjwatson> So that's unfortunate.
[14:37] <cjwatson> This usually means that you managed to crash the builder somehow.
[14:38] <Odd_Bloke> The only difference that this run should have amounts to "cp chroot/boot/vmlinuz-4.2.0-7-generic livecd.vmlinuz-generic.kernel" in a binary hook.
[14:38]  * cjwatson peers at the custom livecd-rootfs
[14:39] <Odd_Bloke> Hmm, does the builder blow up on unexpected suffixes?  That rings a very vague bell from last time I was playing with this sort of stuff?
[14:40] <cjwatson> No, my guess is that the copied files aren't readable by the builder.
[14:40] <Odd_Bloke> Ah, that's probably a better guess.
[14:40] <cjwatson> Notice that livecd-rootfs/live-build/auto/build does chmod 644 in similar situations.
[14:41] <cjwatson> In fact, why doesn't the existing stuff in live-build/auto/build work for you?
[14:41] <cjwatson> It's doing pretty similar stuff to copy out kernels.
[14:41] <Odd_Bloke> Oh, I must have missed it.
[14:42] <cjwatson> The only difference is that you need to explicitly list the flavours you want in the project declaration in live-build/auto/config.
[14:42] <cjwatson> And it also handles initrds and signed images.
[14:43] <Odd_Bloke> Oh, hmph.
[14:43] <Odd_Bloke> We _were_ using our own fork of live-build and still doing this outside of live-build.
[14:43] <Odd_Bloke> So I guess my assumption that we had to do it outside of live-build doesn't hold.
[14:44] <cjwatson> I think this kind of interaction with the buildd belongs in livecd-rootfs rather than live-build.
[14:44] <cjwatson> Albeit in its live-build hooks.
[14:44] <Odd_Bloke> Yeah, that looks precisely like what we need.
[14:45] <cjwatson> You definitely want to modify live-build/auto/config to set up the config you need.
[14:45] <cjwatson> Every distinct "project" field passed to a livefs build should have such a thing.
[14:45] <cjwatson> Oh, but ubuntu-cpc is already in there, OK.
[14:46] <Odd_Bloke> Yeah, we're already doing most of our stuff there sensibly; I'd just missed that we could do this with configuration rather than custom hooks.
[14:46] <cjwatson> So it probably just needs to set KERNEL_FLAVOURS.
[14:47] <cjwatson> Yeah, just hadn't realised it was in trunk rather than in your delta, good.
[14:47] <Odd_Bloke> Well, at least I've learned more about the difference between binary and chroot hooks this morning. :p
[14:47] <cjwatson> Heh
[15:19] <Odd_Bloke> cjwatson: Ah, that isn't working because it's looking in binary/boot which only contains filesystem.ext4, filesystem.packages and filesystem.packages-remove in our builds.
[15:26] <Odd_Bloke> Which I think would only work if LB_CHROOT_FILESYSTEM was none.
[15:26] <Odd_Bloke> Because all of the other code paths in lb_binary_rootfs just shove a file there.
[15:31] <cjwatson> Odd_Bloke: I think it's because you're using --linux-packages=none, which disables the relevant code path in lb_binary_linux-image.
[15:31] <Odd_Bloke> Aha, that looks promising.
[15:31] <cjwatson> Surely either you need to copy them out or you have no kernel packages installed, but not both. :-)
[15:32] <cjwatson> In fact you have KERNEL_FLAVOURS=none in that same section, so those probably need to be changed together.
[15:34] <cjwatson> --linux-packages defaults to "linux" on Ubuntu, so you should probably instead set KERNEL_FLAVOURS in the per-architecture case statement and drop the manual installation of linux-*.
[15:34] <cjwatson> I think.
[15:34] <cjwatson> Though maybe linux-virtual doesn't correspond to the kernel image name, so that might still need some work.
[15:35] <cjwatson> Maybe that's why you have this stuff to begin with.
[15:35] <Odd_Bloke> I remember there being some pain around this before, but I don't remember precisely what problems I was seeing.
[15:35] <Odd_Bloke> But, yeah, I wouldn't be surprised to get a definite reminder in a few minutes time. :p
[15:36] <cjwatson> So that might in fact be reasonable cause for a cpc-specific hook, but I'd probably do it by doing something in line with what lb_binary_linux-image would do if you could persuade it to DTRT without also changing the behaviour of lb_chroot_linux-image in undesirable ways.
[15:36] <cjwatson> Since that code is simpler.
[15:38] <Odd_Bloke> cjwatson: At the moment, we publish the tab-delimited file that is produced by that hook, which is why it was happening in that somewhat circumspect way.
[15:43] <cjwatson> That seems of dubious utility when the manifest is there as well.
[15:43] <cjwatson> So, I mean, the trivial fix is to chmod 644, just wondering if it could be done better.
[15:47] <Odd_Bloke> cjwatson: Yes, indeed.  Sadly we have to maintain that interface for trusty.
[15:48] <Odd_Bloke> I'm building a livecd-rootfs package with some different kernel options, I'll see what that does.
[17:51] <Odd_Bloke> cjwatson: Yeah, your suspicious is correct; we either fail to find something named *linux-virtual* in /boot or we install linux-generic in the image.
[17:51] <Odd_Bloke> *suspicion
[17:52] <Odd_Bloke> So tomorrow I'll look at rewriting that hook in a less hidious fashion.
[21:36] <Odd_Bloke> cjwatson: Hmm, alternatively I could teach the part of live-build that 'virtual' kernels are named 'generic' in /boot.
[22:29] <cjwatson> Odd_Bloke: I don't know which is better.  Your call.