[00:23] StevenK: is https://lpci.wedontsleep.org/ still the jeknins? [00:24] StevenK: I don't see an attached executor [00:32] lifeless: That's because I haven't done it yet because I was noming [00:33] StevenK: ah kk [00:35] lifeless: Your key has been added [00:36] lifeless: ubuntu@23.22.38.59 [00:37] lifeless: I've not updated xvfb did you want to hack that in before we kick off a build? [00:37] sure [00:39] win: $ lxc-ls [00:39] lptests [00:39] lptests [00:39] two for the price of one [00:39] Haha [00:39] StevenK: ah, lxc-start is still running [00:39] lptests setup still happening ? [00:40] No ... [00:40] the android-in-magazine thing is kindof cool [00:40] ok, well the base is running [00:40] so I'll just use that [00:42] StevenK: whats the password for jenkins, do you know ? [00:42] StevenK: doesn't seem to have passwordless sudo within the container [00:42] It ought to [00:42] jenkins@lptests:~$ sudo apt-get install xvfb [00:42] [sudo] password for jenkins: [00:42] It certainly does outside the container [00:43] lifeless: And it doesn't have one, the user is created with --disabled-password [00:44] StevenK: I am confuse [00:44] StevenK: how does lpsetup install stuff within it then ? [00:44] I have no idea how lpsetup does its thing [00:45] anyhow I've edited the file from outside [00:45] so xvfb-run is fixed [00:45] should I stop the container ? [00:45] Yeah, let's do that and I'll kick off a build [00:45] done [00:45] and done [00:51] what concurrency did you use ? [00:51] 4 [00:51] hmm [00:51] That hasn't changed at least [00:51] lxc-ls shows 8 [00:52] I detect a bug in lxc-ls [00:52] ls /var/lib/lxc/ [00:52] lptests lptests-temp-1FaKWwb lptests-temp-c8jcW0W lptests-temp-qoZ6ncr lptests-temp-wbvIQLE [00:52] $ [00:52] testr run --parallel --concurrency=4 [00:52] yeah [00:52] I know [00:52] check out lxc-ls on the executor though :P [00:52] Haha [00:52] lxc has bugs? I'm SHOCKED [00:53] lifeless: It could be once is for the container and another to show its running? [00:55] argh [00:55] cp's should really be cp dest source [00:56] cp -t . <- awkward [00:56] Hah [00:56] StevenK: 'ps fax | grep python.*load-list | grep -v resume-layer | awk '{ print $14 }' | xargs cp -t .' [00:56] StevenK: captured the lists being executed. [00:57] You can't get that from the temp directory inside workspace/devel? [00:57] lifeless, StevenK: I know little, but read this post that said "The lxc-list command lists anything that’s a directory in /var/lib/lxc": http://jonathancarter.org/2012/09/29/llxc-my-little-python3-lxc-based-project/ [00:58] Ah ha! [00:58] raise LocationError(subject, name) [00:58] LocationError: (None, 'branch_type') [00:58] abentley: it also looks in cgroups [00:59] abentley: and the mount table - its just a shell script if you want to look at it, it is AFAICT just confused by the unionfs going on. Or something like that. [01:00] lifeless: So far, I haven't used lxc. Tried to set up juju-on-lxc last week, ran into some glitch with zookeeper. [01:00] abentley: ah, so yeah - juju-on-lxc does things more differently than juju-on-openstack, for instance, which I find odd. [01:03] abentley: lxc on its own is pretty nice, and lpsetup does a decent job of configuring it. [01:03] wgrant: / StevenK: I think a 'here is how I did it' post to the -dev list might be useful for folk. [01:03] lpsetup needs a -y :-( [01:03] lifeless: Once we've actually done it, yeah :) [01:03] the new shiny stuff hasn't been socialised nearly enough. [01:05] lifeless: Unrelatedly, I'd like to basically abolish the explicit slave/master store selectors from normal code. [01:05] Do you see any reason to keep them? [01:05] A few places explicitly use a slave store for searching, but that seems pretty pointless. [01:06] I haven't done an audit but the idea sounds fine to me [01:06] I agree that forcing a slave for searching is unnecessary [01:09] lifeless: Thanks. [01:10] If basically everything uses the policy's default store, then we can avoid terrible fallback hacks in the DB policies. [01:10] Plus code gets shorter [01:10] eg. a lot of scripts use IMasterStore everywhere [01:10] When scripts always default to the master anyway [01:11] wallyworld_: Looks like you may have broken the build [01:11] But that's impressively few test failures considering what you've done. [01:11] iy passed ec2 [01:12] it [01:12] hmmm [01:12] looking [01:12] http://lpbuildbot.canonical.com/builders/lucid_lp_lxc/builds/83/steps/shell_9/logs/summary [01:14] i can click that myself you know [01:14] Yeah, but it's like a buildbot bookmark and 3 clicks away [01:16] i already had bb open to monitor the build [01:16] Ahh [01:16] wgrant: Do you know about the code.simplified_branches_menu.enabled FF ? [01:16] StevenK: It's been on for everyone for more than a year, I'm pretty sure [01:16] It can probably be removed. [01:17] Yeah [01:17] Lets do that [01:18] Even better if you remove the view code but forget the model stuff, so I can delete it myself later :P [01:21] Hah [01:21] I don't think this includes any model code === matsubara is now known as matsubara-afk [01:29] wgrant: PersonBranchesMenu makes me sad. It has these link properties and they're so generically named that I can't tell if they're used [01:35] wgrant: So if a method returns a link a template still has to reference it? I can't any results for '/owned' [01:36] StevenK: Most of the time [01:36] StevenK: But sometimes entire menus are probably rendered [01:36] This is PersonBranchesMenu and I want to destroy more code [01:37] Delete them and see what breaks, perhaps [01:45] 6 files changed, 66 insertions(+), 180 deletions(-) [01:45] I think I've deleted enough to make up for I added [02:02] wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/skip-private-dev-focus/+merge/127622 [02:13] Project devel build #1105: STILL FAILING in 1 hr 28 min: https://lpci.wedontsleep.org/job/devel/1105/ [02:15] That's wallyworld_'s fault too [02:52] StevenK: this is where I think some pragmatism is needed with the LOC thing - your branch combines 2 totally unrelated changes just for the sake of artificially reducing LOC when really 2 branches would have been better, and now it makes it harder to review because you have to constantly ask what unit of work the various parts of the diff are for [02:54] The LoC policy doesn't say anything about single branches being LoC-negative. [02:54] i know [02:54] That would be perfectly acceptable as two separate branches [02:54] it explicitly avoids that in fact [02:54] which i why i questioned it [02:55] So there's no pragmatism required :) [02:55] well, that's my bias coming through [02:55] I can delete the MP and blow them apart [02:55] if a better solution requires slightly more zLOC, then so be it [02:55] LOC should be a secondard consideration, always [02:56] secondary [02:56] Sure [02:56] But in most cases there's an easy win [02:56] StevenK: that would be great if you could [02:56] * StevenK stops glaring daggers at LibrarianFormatter [02:59] StevenK: ok, so what I'm doing to reproduce is this: [02:59] cd /home/jenkins/launchpad/lp-branches/workspace/devel/ [02:59] jenkins@ip-10-218-67-147:~/launchpad/lp-branches/workspace/devel$ sudo /usr/local/bin/lp-setup-lxc-test lptests /home/jenkins/.ssh/id_rsa --load-list /home/jenkins/runs/tmpzqc6_R > ~/runs/stream 2>~/runs/stream_err [02:59] using the captured load-list [02:59] which contained one of the tests that got spewed to https://lpci.wedontsleep.org/job/devel/1105/console [03:00] StevenK: when it completes, if the stream is corrupt, we'll have reproduction. [03:00] Nice [03:00] StevenK: if it isn't, we'll need to keep trying harder. [03:01] Once we have reproduction, then I'll start on trimming down the size neede dto trigger the failure: first delete all the tests that ran *after* the corruption (allowing 2- or 3 to stay for buffer flushing just-in-case) [03:01] then delete all the tests before it, leaving enough to trigger the fail [03:01] rinse and repeat - binary search needed for some of these 'delete all the' bits. [03:01] wallyworld_: Why don't we just restrict +bugs and searchTasks to users with launchpad.View? [03:01] because every alteration will permute things [03:01] Or zope.Public for public artifacts. [03:02] wallyworld_: Unauthorized is the correct exception to raise; it triggers the 403 page like any other forbidden page [03:02] We don't need a more user-friendly message than that. [03:02] wgrant: no, what's there in the mp is restoring existing behaviour which i broke with the ff removal [03:02] Or if we do, then we need it generally. [03:03] wallyworld_: Right, but is it valuable existing behaviour? [03:03] yes [03:03] Oh, is this the "this information is not shared with you" thing? [03:03] That we added recently? [03:03] for LimitedView? [03:03] it was specifically asked for during disclosure development [03:03] Right, that makes more sense [03:03] Not actual sense [03:03] yes, the "this info is no shared " thing [03:03] But more :) [03:04] * wallyworld_ doesn't want to speculate on the usefulness of the feature, just implementing what's been asked for [03:04] r=me [03:04] Sure [03:04] thank you [03:04] I just thought it was probably some terrible ancient thing [03:05] fair enough :-) [03:05] And terrible ancient things that I don't know about can usually be deleted without anyone noticing. [03:05] the recent bb breakage was me attempting to fix the failing test [03:05] so i neutered the test to get bb going [03:05] Right, I saw you'd dropped the +bugs team check thingy [03:05] and then provided the proper fix [03:06] StevenK: general design rule testr honours is to expose data / state so folk can go digging :) [03:07] But not quite enough data in this case [03:09] StevenK: what would you like added ? [03:14] /usr/bin/python 27917 jenkins 0r FIFO 0,8 0t0 18555269 pipe [03:14] seems saner [03:14] /usr/bin/python 27917 jenkins 1w FIFO 0,8 0t0 18555270 pipe [03:14] /usr/bin/python 27917 jenkins 2w FIFO 0,8 0t0 18555271 pipe [03:14] so the xvfb-run fix is intact [03:38] wgrant: installing the cert works great for chrome but now ff constantly complains about (Error code: ssl_error_bad_cert_domain) even if i add an exception. any idea? [03:38] lifeless: Merely pointing out that testr does not provide quite enough state or data to be able to tell what's leaking [03:38] StevenK: but it does, AFAICT [03:39] StevenK: I mean, I'm doing this because I've lots of experience and I have the time atm [03:39] StevenK: but there was nothing hidden that would make it hard to reproduce [03:39] no need for a debugger, for instance. [03:39] or extra print statements [03:40] StevenK: I mean, clearly its going wrong :P [03:45] wallyworld_: https://code.launchpad.net/~stevenk/launchpad/skip-private-dev-focus/+merge/127629 should be more to your liking [03:48] wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/destroy-simplified-branch-ff/+merge/127630 [03:49] StevenK: bit of a nitpick on line 52 - s/an/a [03:49] Ah yes. It made sense before I added 'private' in :-) [03:50] StevenK: so thst i understand, will self.branch be None if the user cannot see the devfocus branch? [03:53] wallyworld_: Yes, I won't paste the code block, but the development_focus_branch property will only return the branch if the user has permission to see it. In the cases there is no devfocus or it's invisible, it returns None. [03:54] StevenK: right, and this is the self.branch attribute used in external_visible? [03:54] wallyworld_: Right, self.branch backs onto self.development_focus_branch [03:54] ok, thanks [03:55] r=me [03:56] wallyworld_: I'll push up that nitpick before landing too [03:56] StevenK: and now you get to close a second bug :-) [03:56] since there are 2 x mp [03:57] wallyworld_: Which domain are you going to? [03:57] wgrant: bugs.launchpad.dev [03:57] wallyworld_: Which second bug? [03:58] now works without complain in chrome [03:58] Very tempted to just Won't Fix bug 393407 [03:58] StevenK: the one you raise for removing the feature flag [03:58] <_mup_> Bug #393407: PPA doesn't allow signed contributed builds

< https://launchpad.net/bugs/393407 > [03:58] wallyworld_: Bleh [03:58] wallyworld_: Does the Firefox cert error show the right cert? [03:58] It's possible you're fighting with SNI and losing, or something like that [03:58] hmm. not sure, let me look [04:01] wgrant: it's labelled as "Invalid Certificate" when i use ff to get the cert from the error screen. not sure how to know what the right one is [04:03] wgrant: i actually ran the entire rf-setup-certs.sh script once, but it complained about an invalid file or dir so i just ran the last 2 lines and that seemed to work [04:03] which it did for chrome [04:04] wgrant: I wonder if bug 583392 is fixed now [04:04] <_mup_> Bug #583392: IntegrityError raised setting a branch for a project series.

< https://launchpad.net/bugs/583392 > [04:05] wallyworld_: Try adding the cert to Firefox's certstore manually, I guess. [04:07] StevenK: It appears to be, indeed. [04:11] ok, it reproduced on that one stream [04:11] now to divide and conquer [04:11] cp tmpzqc6_R{,.orig} [04:11] and delete all but the 50 or so tests before the glitch [04:12] stream_err has only this: [04:12] less stream_err [04:12] Warning: Permanently added '10.0.3.8' (RSA) to the list of known hosts. [04:12] kill: 186: No such process [04:12] Stopping lxc [04:12] and a blank line [04:12] Hm [04:12] Did someone disable qastaging updates? [04:12] it can mess things up, but there is so little of it, and its all (AFAICT) outside of the xvfb-run anyhow, which is why xvfb-run isn't directly griefing us [04:12] s/directly/frequently/ [04:13] StevenK: http://paste.ubuntu.com/1257313/ list of tests [04:15] lifeless: Nice [04:26] StevenK: and for tracking progress https://bugs.launchpad.net/launchpad/+bug/1060616 [04:26] <_mup_> Bug #1060616: subunit stream corruption with jenkins test runs < https://launchpad.net/bugs/1060616 > [04:26] lifeless: So, thanks for filing that, but not another critical :-( [04:31] StevenK: https://bugs.launchpad.net/launchpad/+bug/1060616/comments/1 [04:31] <_mup_> Bug #1060616: subunit stream corruption with jenkins test runs < https://launchpad.net/bugs/1060616 > [04:31] wgrant: got it working. had to convert the x509 cert for chrome to a pkcs12 one for firefox. i have no idea about all this stuff [04:31] StevenK: now that we have an unadulterated stream, we can see pretty clearly whats fucked up [04:32] Indeed [04:32] I'll file a testr bug about making it easier to get said stream. [04:32] wallyworld_: Hm, Firefox should be able to import plain x509 too [04:33] wgrant: it didn't seem to give me that option, only pkcs12 [04:33] i tried to import the x509 and it complained [04:34] at least it all works now, and the other ff lp.dev issue is gone now too [04:34] StevenK: so this is critical, because if it breaks on buildbot it will equally badly and mysteriously. [04:34] wallyworld_: Ah, were you trying to import into the Personal section? [04:34] * wallyworld_ shrugs [04:34] that needs a private key too, so it requires PKCS#12 [04:34] yes, looks like i was [04:35] Servers/Authorities should work with x509 [04:35] ok, will try that next time it breaks [04:37] lifeless: Indeed [04:38] "USN-1551-1 fixed vulnerabilities in Thunderbird. The new package caused a [04:38] regression in the message editor and certain performance regressions as [04:38] So it wasn't just me! [04:38] well. This update fixes the problems." [04:52] StevenK: its worse than I thought: [04:52] see comment 2 [04:55] Wheee [04:57] That's normal [04:57] If the subprocess dies, the rest of the layer goes with it. [04:57] And the parent often doesn't notice. [04:57] There's a bug for that [04:58] Subprocess death unheeded by parent or so [04:58] Right [04:59] well [04:59] so the question is [04:59] is python failing to process a finally: [04:59] or is something else fubared [04:59] Which test is it dying on? [05:00] Probably something that uses webkit [05:00] wgrant: lib/lp/code/javascript/tests/test_productseries-setbranch.html and lp/testing/tests/test_yuixhr_fixture_facet [05:00] Ah yes [05:00] So it is [05:00] So it'll be segfaulting. [05:00] You can reproduce that easily [05:00] so, subunit layer, it may make sense to have a 'close off the stream no matter what' escape clause [05:00] It usually happens when run outside xvfb-run, though [05:01] which the parent could use [05:03] however [05:04] need to determine first how the parent-child stuff works [05:04] it may be that the parent is just being stupid and not reporting a close when it knows already [05:04] StevenK: so, you know now whats wrong - webkit fuckage [05:04] StevenK: and I have the data on how the system was failing, so can reproduce externally. [05:05] Then why doesn't buildbot fall over into a screaming heap? [05:05] its webkit isn't segfaulting [05:05] (duh) [05:05] But why not? :-( [05:05] Run it manually [05:05] dunno [05:05] You might get a GTK error [05:06] And I'm curious what is causing it to SEGV [05:06] It normally segfaults because there's no X [05:06] nothing on stderr [05:06] But the xvfb-run should sort that out [05:06] Right [05:06] gdb :) [05:06] indeed [05:06] start an ephemeral container [05:06] ssh into it [05:06] run one xvfb-run bash [05:07] run gdb python --args -- bin/test lp/testing/tests/test_yuixhr_fixture_facet [05:07] (or thereabouts) [05:07] Yep [05:07] lifeless: You're still on the executor, are you going to try that? [05:08] The stacktraces tend to be reasonably self-explanatory, fortunately. [05:10] StevenK: no [05:11] Aww [05:11] StevenK: a) cynthia time, b) I'm pursuing the test reporting side of it, to make future runs less likely to fail in this mysterious manner [05:11] StevenK: daylight saving kicked in here last weekend, we're 3 hours out atm [05:12] StevenK: https://bugs.launchpad.net/subunit/+bug/1060628 [05:12] <_mup_> Bug #1060628: cannot 'fix' a subunit stream that may be corrupted < https://launchpad.net/bugs/1060628 > [05:13] lxc-start -n lptests will start the ephemeral container? [05:13] StevenK: no [05:13] lxc-start-ephemeral [05:13] StevenK: lxc-start-ephemeral -d -n lptests [05:13] lxc-start -n lptests will start the real container [05:13] StevenK: so you can ssh in and not have the awful console emulator [05:15] root@ip-10-218-67-147:~# lxc-start-ephemeral -d -n lptests [05:15] getopt: invalid option -- 'n' [05:15] And the usage message isn't very helpful either [05:15] oh blah, [05:15] -o maybe? [05:15] I can't remember [05:15] May just be a positional arg [05:16] -o orig is in the usage [05:16] yes [05:16] -o lptests [05:16] I was remembering base container startup [05:16] A consistent UI is a bad UI [05:16] this isn't inconsistent [05:16] Back in my day it only had positional args, and we liked it. [05:16] because you wouldn't be naming it [05:17] it got 'fixed' [05:18] Bleh, and I run right into the 'jenkins has a password?' thing :-( [05:18] muhhaha [05:19] * StevenK forcibly edits the lxc's sudoers [05:23] jenkins@lptests-temp-k0hHY4M:~$ sudo apt-get install gdb [05:23] Reading package lists... Done [05:23] stub: Do you recall if there was a reason for translations-export-to-branch to do its search on the slave DB? [05:23] It then casts the branch up to a master object to write to it [05:23] Which seems odd. [05:24] jenkins@lptests-temp-k0hHY4M:~$ xvfb-run bash [05:24] xvfb-run: error: Xvfb failed to start [05:24] * StevenK grumbles [05:24] It does like its helpful error messages [05:24] Beware, 'tis a hideous shell script [05:24] Yes, and I'm trying to avoid bash -x [05:30] wgrant: Sounds like one I would have nagged about because of performance and/or long running transactions [05:30] * StevenK stabs xvfb-run [05:31] wgrant: IIRC it had long running transactions, and when fixing that it was easy to switch it to using the slave for most of the work too. [05:31] wgrant: that was avoiding lock driven bloat on the master [05:31] The thing is the DB work is really really trivial [05:31] Hmm [05:31] wgrant: but without slony the bloat impact happens no matter what slave you are reading from [05:31] so without slony there is no benefit to that jumping around [05:31] It's the only remaining case of things that need to write querying from the slave store [05:31] wgrant: Yeah, but if you look sideways at a Storm object at the wrong time it gets refreshed from the DB and suddenly you have a new, open transaction sitting idle. [05:32] jtv did the fixup IIRC, I might be mistaken for the rationale. [05:33] It's used the slave store since the script first landed in the tree [05:33] But the basic principal of 'thou shalt use a slave instead of the master whenever possible' still applies. [05:33] Excellent. We should do more of that. [05:34] Should we? [05:34] The DB load from the two things that do that is miniscule. [05:34] And it's a fair bit of added complexity [05:35] Yes, the master isn't horizontally scalable. [05:35] Why is it complex? [05:36] I suspect it is complex because it is bending over backwards to ensure a transaction doesn't get held open while doing possibly lengthy operations, like anything to do with importing or exporting pofiles or anything to do with bzr. [05:37] Well, most things are very well served by just using their policy's default store [05:37] We have a lot of code that uses IMasterStore/ISlaveStore/IMasterObject unnecessarily. [05:38] Yeah, but something like exporting translations you can say 'this always uses the slave store' because we never care that the information is up to date. [05:39] The scripts policy could encode this, but other callers to the code don't get that policy. [05:42] Do we care now if things are a little out of date? [05:44] Mmm [05:48] I'm trying to think of a cleaner way to do the slave fallback. Currently you get a slave object that's actually connected to the master, which is a bit ugly. [05:48] Because we need master->slave fallback to some extent. [05:52] Though I guess if I get a fake-slave object and keep it around over a connection reset, it could end up reattached to a real slave [05:52] Our Storm object lifetimes are sometimes a little odd [06:02] stub: Does our postgres config log statement timeouts? [06:03] Yes [06:03] https://oops.canonical.com/oops/?oopsid=OOPS-b01fe5e323fdcb85e9bfda27beca4466 [06:03] Is it easy enough to find out the second that the final statement there was terminated? [06:03] Since it took 24 seconds [06:04] I suspect the delay is on the Python side [06:04] But I'd like to know for sure [06:04] Since this goes back to the random delays we see sometimes that can't all be explained by GIL contention [06:04] wallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/assertion-review-token/+merge/127639 [06:04] They usually exhibit themselves as very long DB queries [06:05] From my POV, there is nothing wrong with ISlaveStore returning a Store connected to the master db. However, the Zope adapter mechanics require that IFoo(bar) return an object that provides the IFoo interface. [06:05] Oh, it actually checks? [06:05] I've thought of making all objects that provide IMasterStore also provide ISlaveStore [06:05] I think it does in some cases? All cases? I forget the details [06:06] I guess having the slave store connected to the master DB isn't too bad, but it might be useful to log the current connection string somewhere. [06:06] We don't need to use the adapter spelling. It was the nicest spelling that met our needs at the time. [06:06] Currently in OOPSes we log whether it was master or slave, which can be helpful for diagnosing slowness. [06:07] StevenK: i'll pass since i'm not fully across the subtlties of the oauth stuff [06:07] I'll look in a sec [06:08] wallyworld_: Pft. It's subtle as a housebrick to the face. [06:08] We need to, at a minimum, switch from using dbname to dbname+hostname. Full connection string might be too noisy in some contexts. [06:08]