[00:24] * wgrant experiments with BranchRevision [00:26] \o/ [00:26] It may not come out of this very healthily [00:26] But then again, neither may mawson... [00:26] bai [00:26] Bye lifeless [00:27] wgrant: can you give StevenK a hand on the jenkins thing? [00:27] wgrant: I'd love to see it work :) [00:27] I was intending to, yes :) [00:27] StevenK: Shell pls [00:47] StevenK: So, xvfb-run still won't start? === matsubara is now known as matsubara-afk [00:48] wgrant: It didn't for me when doing it by hand [00:48] Let me see [00:49] StevenK: What are the creds for authing to the container? [00:50] wgrant: jenkins@ [00:51] Oh, right [00:51] wgrant: I will usually sudo -u jenkins -i and go from there [00:51] Yep [00:51] Do you normally hack the container's sudoers? [00:51] I did yesterday [00:51] But not normally [00:57] Hm [00:57] So under xvfb-run it passes [00:58] In the root container [00:58] Spews locale errors, but passes [01:00] Sigh, there are tests that insist that the Registered branches link doesn't show up if there aren't any [01:01] Which makes it the only link to do so, so I crippled that [01:01] Right [01:01] Fix the tests :) [01:01] Or, possibly better still, delete them. [01:02] test: lib/lp/app/javascript/tests/test_multicheckboxwidget.html [01:02] Segmentation fault (core dumped) [01:03] \o/ [01:03] Hm [01:03] But that same thing passes in the root container [01:03] * wgrant blames aufs [01:04] Haha [01:04] aufs and I have history [01:06] Ah [01:07] Looking at syslog, there are some instances of what look like the symlink/hardlink restrictions [01:07] Ah ha [01:07] Mmmm, how do we turn that off? [01:07] Well [01:08] I just had another failure and didn't get the same message [01:08] And I don't actually know what the message was [01:08] Ah [01:08] [1236262.033939] /usr/bin/python[15312]: segfault at 14 ip 00000000f5496411 sp 00000000ff8a7160 error 4 in libgtk-x11-2.0.so.0.2000.1[f5310000+3cd000] [01:09] Ah right [01:09] dmesg is intact, indeed [01:09] wgrant: Right, so all tests pass. I'm going to toss this FF death branch at buildbot [01:09] So it is the hardlink restrictions [01:09] StevenK: Great [01:09] lp-setup was meant to disable that restriction, I thought [01:09] Or did we get a kernel fix that isn't in this ec2 image... [01:09] that would be pretty hilarious [01:10] Hm, the kernel's not even a month old [01:10] Yeah, the hardlink restrictions are still enabled. [01:10] * wgrant disables [01:10] If you give me the runes, I can edit my setup script [01:11] Testing that it stops segfaulting now [01:12] It's pretty nice of xvfb-run to just carry on even though its setup failed [01:12] successful: lib/lp/app/javascript/tests/test_multicheckboxwidget.html [01:12] sudo sysctl -w kernel.yama.protected_nonaccess_hardlinks=0 [01:12] Hah, right [01:12] is what you want [01:12] So, with that enabled, I'll shut down the containers and we can try a full --concurrency=4 [01:13] Yeah, I'll need to add the executor to Jenkins, so tell me when you're done [01:15] StevenK: All torn down [01:15] Fire it up [01:16] wgrant: Done. #1106 kicked off [01:16] concurrency=4? [01:16] I don't want to wait all day, idaelly [01:16] Yeah, it's still 4 [01:17] Great [01:17] Thought it might still be down to 1 [01:17] It changed back days ago [01:17] Better safe than waiting for four hours :) [01:19] StevenK: Have you restored subunit2junitxml etc. after the debugging? [01:19] I have not [01:22] k [01:22] Let's see how it goes [01:22] It's fairly easy to restore and it only tells Jenkins which tests fail [01:22] Oh [01:22] I can just tail the stream directly now [01:22] :) [01:22] Haha [01:23] How long did it take you to realise that? :-) [01:23] ew [01:23] There are jars in ~/launchpad/lp-branches [01:23] die [01:23] Those are for jenkins, hands off [01:23] I know, but still ew [01:23] Haha [01:24] ~/launchpad/lp-branches is the base dir for jenkins, due to shared repo [01:24] Launchpad is a Java-free workplace :) [01:24] Ah [01:24] Because then it goes ahead with ~/launchpad/lp-branches/workspace/devel [01:25] This should take about 90 minutes, right? [01:25] Roughly [01:25] I guess it might be a bit longer, since the testrunners won't all drop dead when they start doing JS tests [01:25] Haha [01:31] wgrant: You said you understood the issue with bug 828133 ? [01:31] <_mup_> Bug #828133: Archive:+edit-dependencies OOPSes when Ubuntu dependency changed via API

< https://launchpad.net/bugs/828133 > [01:31] StevenK: Yeah [01:32] StevenK: IIRC the UI only exposes None/multiverse options for the primary archive dep [01:32] But you can set eg. main through the API [01:32] causing the UI to crash when it tries to render the current value [01:32] Haha [01:32] but reading the bug now to see if I remember anything else [01:32] Why would the UI do that? It's daft [01:32] Right, yeah, what I said [01:33] The UI gives "the source's component in the primary archive" and "all components" options [01:33] None and multiverse respectively. [01:33] Right, this should be pretty easy to test [01:34] ... unless Archive:+edit-dependencies is tested via a doctest ... [01:35] StevenK: The jenkins console thingy will show stream bits if the subunit breaks, right? [01:35] Because testr will let it through [01:35] I think testr will only show it right at the end [01:35] It doesn't seem to print out during [01:36] Ah [01:36] No subprocess deaths yet, anyway [01:36] Sigh, it's tested via two doctests [01:37] Which both make up 2000 lines [01:37] :-( [01:42] ahoy there [01:43] lifeless: Hi [01:43] We fixed Jenkins [01:43] And found that Launchpad itself has more than 900 million branchrevisions [01:43] That about covers the horrors that you missed :) [01:43] 1billion! [01:43] dollars, yes [01:44] cool [01:44] Flight not delayed by 12 hours yet? [01:48] not yet [01:49] what was causing the segfaults ? [01:49] hardlink restrictions [01:50] orly [01:50] -fun- [01:50] Yeah [01:50] xvfb-run fails to start X properly [01:50] So it just continues [01:51] Without X... [01:51] nice [01:51] did you file a bug ? [01:51] Well, I'm not sure if it actually sees an error from the syscall [01:51] It probably does, but I'm waiting to see if the build succeeds first [01:56] ProgrammingError: permission denied for relation archivedependency [01:56] Aw fer ... [02:04] wgrant: So I should add main/restricted/universe to the vocab as well? [02:05] StevenK: Maybe. [02:05] StevenK: How would the UI show that? [02:05] Hmmm [02:05] That is a good point [02:05] You could perhaps add an extra option if the component is set to an unsupported value [02:05] Let any value be preserved by the UI, but not necessarily set [02:06] Right [02:06] I think I'm going to be forced to rewrite this doctest, it's annoying me. [02:07] wallyworld_: I think your MP fixes bug #618399 [02:07] <_mup_> Bug #618399: DistroSeries:+nominations timeouts

< https://launchpad.net/bugs/618399 > [02:08] oh, ok. thanks [02:08] i guess it does if it's not there anymore :-) [02:09] Right, it's no longer going to call canApprove on 75 nominations [02:09] Because it doesn't need to decide whether to show the control :) [02:10] yep [02:11] I note Development:Coding has been wgrant'd [02:12] It doesn't help that wallyworld_ has 3 cards in Review [02:13] 3503073kB 369kB/s | Fetching revisions:Inserting stream:Estimate 2710795/2795277 [02:13] <3 lp:chromium-browser [02:14] wait, 3.5GB ? [02:14] Yes... [02:14] binary blobs in bzr? [02:14] Recall that chromium includes all its deps [02:14] It's not much overhead, either [02:14] 2.9G /tmp/chromium-browser/.bzr/repository/upload/ [02:15] wgrant: Why are you branching it? [02:15] To perform a manual incremental scan [02:15] To unbreak the LP DB state [02:15] hopefully [02:15] I could do it by manually hacking last-revision, but it's easier this way [02:15] Not quicker [02:15] But easier.. [02:16] uhm [02:16] so this is in the DC? [02:16] No [02:16] why not branch --stacked; bzr push ? [02:16] It's local [02:16] I need to be able to examine history to chunk it properly [02:16] ah [02:16] well, gl. [02:16] I guess network latency in the DC wouldn't be too bad, though [02:17] hey, its your internets quota. [02:17] * bigjools strokes unlimited internets [02:17] * wgrant strokes 500GB Internets [02:18] * lifeless strokes it to the left [02:18] * bigjools doesn't want to see a pelvic thrust [02:19] I have 60GB, but that's peak. Off peak is unlimited. [02:19] My plan has been grandfathered about seven times and Exetel keep bugging me to switch [02:20] StevenK: let me guess, you told them to stroke it ? [02:21] Last time they rang me, I told them I'd think about it and get back to them. [02:21] I did the first part for about a second and have yet to do the second. [02:23] lifeless: Mmm [02:23] lifeless: Aren't DB reviews different because they have a habit of being catastrophic and difficult to fix? [02:25] wgrant: so entry to the club may be harder [02:25] wgrant: only folk in the club can sign more folk in :) [02:25] wgrant: we have a qa pipeline to catch gross problems [02:26] True. [02:30] ok, flap flap [02:37] One of the slaves has finished [02:39] No errors, failures, or subprocess deaths :) [02:42] There's two slaves done [02:42] three [02:42] Three [02:42] Haha [02:42] So [02:42] Now we just need to make sure result reporting works [02:42] Four [02:42] ideally getting failure counts etc. [02:43] Yippie, build fixed! [02:43] Project devel build #1106: FIXED in 1 hr 26 min: https://lpci.wedontsleep.org/job/devel/1106/ [02:43] Then we need a tarmac config and to tie up hloeung and spm next week [02:43] :D [02:43] Right, let me fix the script to tie subunit2junit back in [02:43] 5126232kB 665kB/s | Fetching revisions:Inserting stream:Estimate 2925464/3035215 [02:44] I'm glad I didn't try to branch this on a FAT32 volume [02:44] -rw-rw-r-- 1 wgrant wgrant 4439527424 Oct 4 12:43 /tmp/chromium-browser/.bzr/repository/upload/oclqmgbw90pn25rdjjp0.pack [02:44] Haha [02:44] wgrant: Right, done. Shall we force another build? [02:45] StevenK: Do I have enough privs to examine the config? [02:45] I'm going to compare it to buildbot and what I know about production [02:46] So we get something as close to prasé as possible [02:46] I'm not sure [02:46] Try? :-) [02:46] I have a Build Now button [02:46] https://lpci.wedontsleep.org/job/devel/ and 'Configure' [02:46] But not much else [02:46] No Configure link [02:47] wgrant: http://pastebin.ubuntu.com/1259265/ [02:47] That's what Jenkins runs [02:48] Bleh, this Archive:+edit-dependencies code is horrible [02:48] Ah [02:48] So the only big missing thing is the .testrepository preservation [02:49] Nah, it will be [02:49] (that's init_testr.py) [02:49] It's ignored by bzr [02:49] Oh, will it? [02:49] So it won't get reverted out [02:49] Oh [02:49] I see [02:50] That sounds sensible, then. [02:50] And it will only run testr init if it needs to [02:50] The script might need fixing for parameterized builds but eh [02:50] So, fire off 1107 [02:51] Great [02:51] + [ ! -d .testrepository ] [02:51] + [ -d download-cache ] [02:51] So it is indeed preserved [02:52] StevenK: You've added the sysctl call to the instance setup script? [02:52] Not that it's relevant to production. [02:52] I did [02:53] I'm tempted to try a m2.4xlarge with concurrency 8 [02:54] Not much point [02:54] Does the maas jenkins+tarmac stuff require parameterised builsd? [02:57] wgrant: This bug is making me very sad. [02:59] StevenK: https://wiki.canonical.com/UbuntuEngineering/QA/UnityAutolandingSetup [03:01] On jenkins create a freestyle job with 3 string parameters - dir, trunkrev and branch - this job will be duplicated and renamed for all the projects. [03:01] So yeah [03:01] I suspect we only need branch [03:05] The pack is now 5.5GB... [03:05] This is a few times bigger than I expected [03:05] Google, what have you done [03:19] blargh what [03:19] JUnit XML doesn't stream? [03:19] :( [03:34] StevenK: Has the test count graph disappeared because the last $LOTS runs didn't return any XML? [03:44] wgrant: It has been streaming [03:45] wgrant: The test count graph disappeared because I turned off that bit due to no XML. [03:45] It's back on for this build, so we'll see [03:50] 8.4GB of download later, it's finally building the tree [03:50] RSS of 2GB, as well [03:50] * StevenK stabs this bug [03:59] * StevenK gives up and finds something that doesn't make him want to scream [04:12] Hmmm. A test for bug 1061374 might be complex [04:12] <_mup_> Bug #1061374: PlainPackageCopyJob.attemptCopy() crashes if only binaries were copied

< https://launchpad.net/bugs/1061374 > [04:28] StevenK: Not hugely [04:29] Publish source in one pocket [04:29] Publish that source + a new binary in a second pocket [04:29] Copy from second to first [04:29] boom [04:34] wgrant: 25 line test, though ... [04:37] :( [04:38] Now 30 [04:40] Hmmm, no boom [04:43] wgrant: And see the graph is back is all its graphiness [04:43] Indeed [04:45] wgrant: So what is next for Jenkins? [04:46] StevenK: A good question [04:46] StevenK: Perhaps try setting up tarmac against qastaging or something, so we can know how it works? [04:47] wgrant: I can set up a parameterized build while you tackle tarmac if you wish? [04:54] StevenK: Sounds reasonable. [04:56] (Pdb) p error_text [04:56] u"copyme 2.8-1 in breezy-autotest (The signer of this package has no upload rights to this distribution's primary archive. Did you mean to upload to a PPA?)" [04:56] Thanks for nothing, mr test [05:05] copyme 2.8-1 in breezy-autotest (No such distro series breezy-autotest in distribution ubuntutest.) [05:05] :-( [05:05] wgrant: I'm done. Do we have an LP branch on qas to test with? [05:06] StevenK: Nothing that's reasonably up to date [05:06] I'm trying to fight with Canonistack [05:06] Since I only have 512Kbps upstream [05:06] Ah, it works now [05:06] * StevenK is also wondering why breezy-autotest is the distroseries [05:08] StevenK: It's in the PCJ JSON metadata, I believe [05:08] If you're using makePCJ then it probably defaults to it [05:08] I'm using STP.getPubSource and friends, so I don't see why the copy is failing with no such series [05:08] How're you creating the PCJ? [05:09] wgrant: http://pastebin.ubuntu.com/1259368/ [05:10] wgrant: Oh, blah, the branch doesn't need to be on qas. Something simple kicking around that isn't merged? [05:10] StevenK: Oh, you just want to test the parameterised job now? [05:11] Right [05:11] A good start [05:11] See if it starts and then kill it [05:11] lp:~wallyworld/launchpad/dynamic-bug-listing-nominations2-1060015 [05:12] Project test-branch build #1: FAILURE in 7.6 sec: https://lpci.wedontsleep.org/job/test-branch/1/ [05:13] Heh heh [05:13] + bzr merge http://bazaar.launchpad.net/~wallyworld/launchpad/dynamic-bug-listing-nominations2-1060015 [05:13] bzr: ERROR: No WorkingTree exists for "file:///home/jenkins/launchpad/lp-branches/workspace/test-branch/.bzr/checkout/". [05:14] :-( [05:19] Project test-branch build #2: ABORTED in 3 min 48 sec: https://lpci.wedontsleep.org/job/test-branch/2/ [05:20] Project test-branch build #3: ABORTED in 31 sec: https://lpci.wedontsleep.org/job/test-branch/3/ [05:20] Did it work? [05:20] Yes [05:22] copyme 2.8-1 in breezy-autotest (source has no binaries to be copied) [05:22] LIES! [05:22] The binaries are in the wrong pocket. [05:23] Silence. [05:23] But thanks. [05:26] wgrant: Same message :-( [05:26] paste test [05:26] Wrong archive? [05:27] wgrant: http://pastebin.ubuntu.com/1259385/ [05:27] Wrong archive. [05:27] Putting archive=archive into getPubBinaries changes nothing [05:27] Hm [05:27] Have you checked that it actually creates any binaries? [05:27] Remember that it respects createMissingBuilds [05:28] Hmmm [05:28] It returns two BPPHs [05:29] pdb [05:29] compare the BPPHs and SPPH [05:29] Check that SPPH.getBuiltBinaries() returns something useful [05:29] That's what the copier uses [05:30] It should return a BPPH for every distinct BPR built from that source and published in the same suite and archive [05:30] [] [05:30] Oh [05:30] Heh [05:30] It's picking the wrong SPPH [05:30] Um [05:30] You need to create them the other way around [05:30] Create the target source first [05:31] We ran into this this morning [05:31] It determines the source by a simple key of (archive, name, version) [05:31] No suit [05:31] Hahaha [05:31] So it just takes the latest [05:32] copyme 2.8-1 in breezy-autotest (a different source with the same version is published in the destination archive) [05:32] Bleh [05:32] Make sure it's the same SPR [05:32] It may be better to use the factory here [05:32] Since you really only need one binary [05:37] (Pdb) s [05:37] ForbiddenAttribute: ('sourcepackagerelease', ) [05:37] Boom [05:37] :) [05:37] But the test passes [05:37] :( [05:38] wgrant: It's been a while since I used STP, I forgot that getPubSource is jealous of any other SPR and will create one everytime [05:39] Yes [05:40] # Otherwise, rely on the job runner to do the final commit, [05:40] # and do not consider a failure of a copy to be a failure of [05:40] # the job. [05:40] Hmmm [05:40] So if runJob doesn't blow up, how can I die? [05:40] Hm [05:40] You might have to get the exception from elsewhere [05:41] Like where, though? [05:42] Where'd you get it before? [05:43] In pdb? [05:43] Yeah [05:43] Hm, but run re-raises it [05:46] Right, test fails [05:46] reference = [05:46] actual = [05:48] wgrant: How is tarmac? [05:48] Nearly working [05:48] Not talking to jenkins yet, but it seems to be otherwise OK [05:49]