[10:34] <Odd_Bloke> https://launchpad.net/~daniel-thewatkins/+livefs/ubuntu/wily/cpc/+build/39160 just 'Failed to build' without giving me a buildlog; can anyone see what happened?
[10:35] <Odd_Bloke> cjwatson: wgrant: ^?
[10:35] <wgrant> Let me see.
[10:35] <Odd_Bloke> Cheers.
[10:36] <wgrant> 2015-09-29 10:30:20+0000 [-] Failure: twisted.internet.defer.CancelledError:
[10:36] <wgrant> 2015-09-29 10:30:20+0000 [-]
[10:36] <wgrant> 2015-09-29 10:30:20+0000 [-] Judged builder lgw01-53 (1 failures) with job LIVEFSBUILD-39160 (4 failures): None, False
[10:36] <wgrant> CancelledError == TCP timeout
[10:37] <wgrant> And it did it four times.
[10:37] <wgrant> You seem to be causing the buildds some serious grief.
[10:37] <wgrant> Are you OOMing or something, possibly?
[10:37] <wgrant> There is no general fault; that build is the only one to hit such timeouts in recent hours.
[10:37] <Odd_Bloke> Yeah, that's possible.
[10:40] <Odd_Bloke> wgrant: Might I see a similar error if I was running out of disk space, or would that still get a build log back?
[10:43] <Odd_Bloke> wgrant: Also, is there any harm in doing these builds (with minor modifications for debugging)?
[10:45] <wgrant> Odd_Bloke: It's fine, the builder VMs are automatically restarted on failure.
[10:45] <wgrant> Odd_Bloke: Disk exhaustion would not cause this.
[10:46] <Odd_Bloke> Okie doke.
[11:07] <cjwatson> Disk exhaustion can cause other problems that you might not be able to distinguish from this, though.
[11:08] <cjwatson> I have some patches lying around somewhere to improve launchpad-buildd's resilience to ENOSPC, but never got round to working out how to test them.
[11:08] <cjwatson> It wouldn't cause a TCP timeout, but it could certainly cause launchpad-buildd to crash.
[11:14] <wgrant> Right, it could cause another "Failed to build" with no log, but AFAIK it wouldn't cause the CancelledError that we see in buildd-manager logs.
[11:27] <Odd_Bloke> Is https://launchpad.net/~daniel-thewatkins/+livefs/ubuntu/wily/cpc/+build/39163 stuck in some way I can't see?  It's taking a lot longer than it normally would...
[12:22] <Odd_Bloke> cjwatson: wgrant: ^ any ideas?
[12:23] <cjwatson> I don't have any magic way to tell.
[12:24] <cjwatson> I suggest cancelling it and seeing if you can divine anything from the log afterwards.
[12:24] <cjwatson> There might be something unflushed in the log.
[12:24] <Odd_Bloke> Done/doing.
[13:12] <cjwatson> Odd_Bloke: Hm, not clear, is it?  I wonder if there's still some kind of race with loop0p1 creation, though I can't imagine what.  Is it repeatable?
[13:13] <Odd_Bloke> cjwatson: I think so; this is using GPT partition tables which is the difference from before.
[13:14] <Odd_Bloke> cjwatson: The '[ ! -b /dev/mapper/loop0p1 ]' should catch the race condition now.
[13:18] <Odd_Bloke> https://launchpad.net/~daniel-thewatkins/+livefs/ubuntu/wily/cpc/+build/39164 was exhibiting the same behaviour, but hasn't ended up with a log.
[13:20] <Odd_Bloke> cjwatson: I might try omitting the dd and see if we have something that looks like a sensibly empty disk image at the other end.