[00:01] <StevenK> wgrant: http://pastebin.ubuntu.com/1229519/
[00:01] <wgrant> StevenK: That looks most excellent. Perhaps try some tests that exercise the DB?
[00:01] <StevenK> Hmmm
[00:01] <StevenK> -t bugs ?
[00:02] <wgrant> Though DatabaseLayer setup worked, so it's a good sign
[00:02] <StevenK> And it can run while I nom
[00:02] <wgrant> -t bugs is huge
[00:02] <wgrant> Try archiveuploader
[00:02] <wgrant> Relatively quick and terrible
[00:37] <StevenK> wgrant: Right, that looks pretty excellent too
[00:39] <wgrant> Right.
[00:39] <wgrant> So, jenkins :)
[00:41] <StevenK> wgrant: Yes
[00:41] <wgrant> StevenK: Should be pretty easy, I guess
[00:41] <wgrant> No need for the complex build rules any more
[00:41] <wgrant> Just a .testr.conf, lp-setup-lxc-build, testr run --parallel
[00:41] <wgrant> Or so
[00:42] <StevenK> Do I need the testr from testing-cabal?
[00:48] <wgrant> StevenK: You might need new testrepository, testtools and subunit
[00:48] <wgrant> Might as well grab them all from there
[00:48] <StevenK> Right
[00:48] <StevenK> Just installing java and fixing ssh access
[00:52] <wallyworld> lol
[00:53] <StevenK> The Jenkins slave is java, no choice in the matter
[00:53] <wallyworld> i know, just trolling
[00:55] <StevenK> Slave successfully connected and online
[00:55] <StevenK> wgrant: So, .testr.conf ?
[00:55] <StevenK> This is different from the .testr.conf already in the LP tree?
[00:56] <wgrant> StevenK: Yeah. Grab it from testr.conf in lp:~launchpad/lpbuildbot/public
[00:56] <wgrant> It has a list command and uses lp-setup-lxc-test rather than bin/test
[00:57] <StevenK> So I should move it into the tree as the first step?
[00:57] <wgrant> Just clobber the .testr.conf that's in trunk locally for now
[00:58] <wgrant> buildbot copies it in at the start of each build
[00:59] <StevenK> I've added a build step
[01:01] <LPCIBot> Project devel build #1091: STILL FAILING in 1 min 3 sec: https://lpci.wedontsleep.org/job/devel/1091/
[01:01] <wgrant> Heh
[01:02] <StevenK>  sudo lp-setup-lxc-build lptests /home/jenkins/.ssh/id_rsa
[01:02] <StevenK> sudo: lp-setup-lxc-build: command not found
[01:02] <StevenK> :-(
[01:02] <wgrant> Is it in PATH?
[01:03] <StevenK> It's in /home/jenkins/bin, but it looks like that doesn't work
[01:03] <wgrant> ~/bin isn't in PATH by default
[01:03] <StevenK> Well, yes
[01:04] <LPCIBot> Project devel build #1092: STILL FAILING in 8.9 sec: https://lpci.wedontsleep.org/job/devel/1092/
[01:06] <LPCIBot> Project devel build #1093: STILL FAILING in 28 sec: https://lpci.wedontsleep.org/job/devel/1093/
[01:06] <StevenK> Haha
[01:06] <StevenK> Need testr init first
[01:06] <wgrant> Ooh
[01:06] <wgrant> It nearly worked
[01:06] <StevenK> Are you spying? :-)
[01:06] <wgrant> Yes
[01:07] <wgrant> How many threads does the instance have?
[01:07]  * StevenK firewalls wgrant off
[01:07] <StevenK> 4 processors, apparently
[01:07] <wgrant> You'll want testr run --parallel --concurrency=SOMETHING --subunit --full-results
[01:07] <wgrant> And you'll need to set a shared TEMP, which probably means it needs to be in the code dir
[01:08] <wgrant> buildbot creates temp/ in the tree at the start of each build
[01:10] <StevenK> Oooh
[01:10] <StevenK> This is looking okay
[01:10] <wgrant> Indeed
[01:10] <wgrant> concurrency=8 is probably going to die
[01:10] <wgrant> But we'll see
[01:11] <StevenK> Die how?
[01:11] <StevenK> We have 34GiB of RAM
[01:11] <wgrant> It'll thrash
[01:11] <wgrant> Ah, so at least RAM won't be a problem
[01:11] <StevenK> Oh, yeah, java.
[01:11] <wgrant> Also, your build failed
[01:11] <StevenK> So we have 4GiB of RAM available
[01:11] <wgrant> lazr.restful missing from download-cache?
[01:11] <wgrant> buildbot's happy with it, so I assume you just failed to update
[01:11] <LPCIBot> Project devel build #1094: STILL FAILING in 2 min 19 sec: https://lpci.wedontsleep.org/job/devel/1094/
[01:12] <wgrant> Yeah
[01:12] <wgrant> But it otherwise worked
[01:12] <StevenK> I guess I need to run bzr pull in the download-cache
[01:12] <wgrant> You'll want to add bzr up download-cache, and probably update-sourcecode, steps
[01:12] <wgrant> But you can just do a one-off for now, I suppose
[01:13] <StevenK> I don't need db setup?
[01:13] <wgrant> lp-setup-lxc-build should do that
[01:13] <StevenK> I thought so
[01:14] <StevenK> Here's a script I prepared earlier
[01:14] <LPCIBot> Project devel build #1095: STILL FAILING in 10 sec: https://lpci.wedontsleep.org/job/devel/1095/
[01:14] <wgrant> Evidently not :)
[01:14] <wgrant> StevenK: scripts/init_testr.py from the lpbuildbot branch may be handy
[01:15] <wgrant> StevenK: It preserves .testrepository between runs, which makes balancing the tests between runners more effective
[01:15] <wgrant> And prevents the issue that killed this build :)
[01:16] <StevenK> Hmmm
[01:16] <StevenK> Right
[01:17] <StevenK> So far I'm blowing it away, but that can be fixed
[01:18] <wgrant> This is looking better
[01:18] <StevenK> Indeed
[01:19] <StevenK> wallyworld: Are you watching too, or just ignoring us?
[01:19] <wallyworld> huh?
[01:19] <wgrant> Doing proper work, hopefully :P
[01:20] <StevenK> This is proper work!
[01:20] <wallyworld> watching what?
[01:20] <StevenK> Also known as buildbot and buildbot-poll die in a large fire
[01:20] <StevenK> Jenkins
[01:20] <wgrant> StevenK: Check in temp/
[01:21] <wgrant> Are there 8 files with lots of tests listed?
[01:21] <StevenK> Yes
[01:21] <wgrant> It's starting up 8 new containers, so it looks good
[01:21] <wgrant> Great
[01:21] <wgrant> I think we're good :)
[01:21] <wgrant> Yay
[01:21] <StevenK> We are
[01:22] <wgrant> And it's even tagged properly
[01:22] <StevenK> jenkins@domU-12-31-39-17-3C-7F:~/launchpad/lp-branches/workspace/devel$ uptime
[01:22] <wgrant> I must work that out on prod buildbot with gnuoy next week
[01:22] <StevenK>  01:22:17 up  3:08,  1 user,  load average: 6.70, 2.75, 1.14
[01:22] <wgrant> :)
[01:22] <wallyworld> wgrant: i recall reading somewhere why the test count is inflated now. can you remind me?
[01:23] <wgrant> wallyworld: The Zope testrunner layer setup/teardowns appear as tests
[01:23] <wgrant> eg 'test: lp.testing.layers.BaseLayer:tearDown'
[01:23] <wallyworld> really?
[01:23] <StevenK> I wonder how it will take with 8
[01:23] <wallyworld> surely we can filter that out
[01:23] <wgrant> If you grep them out then the count is accurate
[01:24] <wgrant> StevenK: Didn't jenkins previously show live test stats?
[01:24] <StevenK> It will show progress, not stats
[01:25] <StevenK> But it takes like 3 or 4 builds to prime
[01:25] <wgrant> :(
[01:25] <wgrant> Yeah
[01:25] <wgrant> Doesn't subunit2junitxml let it show live counts?
[01:25] <StevenK> ENOIDEA
[01:27] <wallyworld> wgrant: StevenK: if you want a break from jenkins, a quickie.... https://code.launchpad.net/~wallyworld/launchpad/embargoed-sharing-policy-ui-1055617/+merge/126585
[01:27] <StevenK> 8 threads is probably what, 2 hours?
[01:27] <StevenK> wallyworld: It's building
[01:28] <StevenK> So now wgrant and I should do a critical or something
[01:28] <wallyworld> do a review first :-)
[01:28] <wgrant> I'm waiting for a local testr run too
[01:29] <wallyworld> oh bollocks. legit bb failure
[01:29]  * wallyworld fixes
[01:29] <StevenK> Line 53 can be sucked back onto 47?
[01:29] <StevenK> Which probably makes the branch neutral
[01:29] <wallyworld> yeah
[01:30] <StevenK> jenkins@domU-12-31-39-17-3C-7F:~/launchpad/lp-branches/workspace/devel$ uptime
[01:30] <StevenK>  01:30:10 up  3:16,  1 user,  load average: 11.62, 9.77, 5.15
[01:30] <wgrant> StevenK: Well, you did ask it to run 8 multiprocess test suites on 4 cores :)
[01:31] <StevenK> I should have gone bigger?
[01:31] <wgrant> Doesn't matter too much
[01:31] <StevenK> Pft, we have 20GiB of RAM free
[01:32] <wgrant> I'm not sure what yellow used for scalability testing, but we don't really need that here
[01:32] <wgrant> We don't care about scaling, just working
[01:33] <StevenK> It seems to work
[01:33] <wgrant> Indeed
[01:34] <wgrant> StevenK: You should probably steal the remaining build steps from master.cfg in the lpbuildbot branch
[01:34] <wgrant> With scripts/init_testr and co
[01:34] <StevenK> Aww, can't we just move on and get this on prase? :-P
[01:34] <wgrant> Also shake Julian until MAAS Tarmac and Jenkins configs fall out!
[01:35] <StevenK> I think other projects have used it as well
[01:35] <StevenK> openstack is one I can remember. Until they switched to git, at least
[01:52] <wallyworld> StevenK: you ok to +1 that branch you looked at?
[01:53] <StevenK> wallyworld: Sorry, done.
[01:53] <wallyworld> np thank you
[01:54] <wallyworld> i was doing the testfix anyway till now
[02:24] <lifeless> wgrant: pqm 0.6 up on LP, and in trunk.
[02:25] <lifeless> wgrant: YMMV, may the farce be with U
[02:26] <wgrant> Thanks
[02:26] <wgrant> Of course I might just dump PQM and use Tarmac instead :)
[02:26] <lifeless> fine by me
[02:26] <lifeless> long as you're not running tests under it
[02:27] <lifeless> tarmac isn't even faintly paranoid enough about *that*
[02:27] <StevenK> Tarmac+Jenkins ?
[02:27] <wgrant> Right, Tarmac + Jenkins is the sensible solution
[02:27] <lifeless> though I don't know why you'd have tarmac in the picture for that.
[02:28] <wgrant> Other projects (eg. maas) have you submitting to Tarmac, Tarmac asks Jenkins to build it, then Tarmac commits if Jenkins succeeds
[02:29] <lifeless> you may not know this, but PQM can get its queue from LP
[02:30] <lifeless> or could before the code team destroy the API :)
[02:30] <lifeless> anyhow
[02:30] <lifeless> there is something funky with that layering
[02:30] <lifeless> I need to reset my whiteboard to make a case, i think
[02:33] <wallyworld> do we know why TestArchiveWebservice.test_getAllPermissions_constant_query_count sometimes fails?
[02:34] <wgrant> wallyworld: It might depend on whether it's the first browser test in that process
[02:35] <wgrant> lifeless: Yeah, but Tarmac is at least less crufty and abandoned than PQM is today...
[02:39] <bigjools> tarmac + jenkins is working extremely well for maas so far
[02:40] <StevenK> bigjools: Are the configs available so we can duplicate it?
[02:50] <StevenK> wallyworld: You fail. Again.
[02:50] <wallyworld> huh?
[02:51] <wallyworld> how?
[02:51] <StevenK> Oh, the failure wasn't your fault.
[02:51] <wallyworld> nope
[02:51] <StevenK> Carry on
[02:54] <wgrant> Hm
[02:54] <wgrant> That subunit stream looks a little broken
[02:54] <StevenK> wgrant: I wonder if bug 1020439 is limited to those few branches
[02:54] <_mup_> Bug #1020439: AttributeError: 'NoneType' object has no attribute 'items'  in +preview-diff <code-review> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1020439 >
[02:54] <wgrant> StevenK: Yes
[02:54] <StevenK> Oh, they have no merge_diff ?
[02:54] <wgrant> So assign to you and close
[02:54] <wgrant> Oh oops, wrong bug
[02:55]  * wgrant -> lunch
[02:55] <StevenK> Haha
[02:55] <StevenK> Helpful much
[02:55] <wgrant> It may be the same, but I can't remember
[02:55] <wgrant> And am on my way out
[02:55] <StevenK> wgrant: It still happens
[02:55] <StevenK> wget a URL from the OOPS gives 00
[02:55] <StevenK> *500
[02:58] <StevenK> The MP exists, and has a diff
[03:01] <LPCIBot> Project devel build #1096: STILL FAILING in 1 hr 46 min: https://lpci.wedontsleep.org/job/devel/1096/
[03:02] <StevenK> Wha?
[03:02]  * StevenK prods Jenkins
[03:04] <StevenK> Hm, it misreports the failure, too :-(
[03:39]  * wallyworld has an appointment with the tax man :-(
[03:39] <StevenK> wallyworld: To pay him?
[03:41] <wallyworld> no, to figure out some of my shit for last year
[03:41] <wallyworld> bbiab
[03:56] <nigelb> Don't you pay him for that? :)
[03:57] <lifeless> We don't pay him for his shit.
[03:57] <lifeless> That would just be wrong wrong wrong
[03:58] <nigelb> Heh
[04:48] <StevenK> wgrant: Halp?
[04:49] <StevenK> wgrant: bug 1020439 is triggered when previewdiff.diff.diffstat = None and I'm trying to figure out how to not make lazr.restful eat a bullet
[04:49] <_mup_> Bug #1020439: AttributeError: 'NoneType' object has no attribute 'items'  in +preview-diff <code-review> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1020439 >
[04:52] <lifeless> StevenK: does lazr.restful want diff to never be None ?
[04:52] <StevenK> lifeless: Where is that defined?
[04:53] <lifeless> the schema for the interface
[04:53] <lifeless> I'm wagging here, of course
[04:53] <LPCIBot> Project devel build #1097: STILL FAILING in 1 hr 46 min: https://lpci.wedontsleep.org/job/devel/1097/
[04:53] <StevenK> Jenkins, you make me very sad.
[04:53] <StevenK> lifeless: readonly=True, I'm not sure what the other defaults are
[04:54] <lifeless> me neither
[04:54] <StevenK> They are not set to required=True
[04:54] <StevenK> So I guess they're False
[04:56] <StevenK> _StringException: lost connection during test 'lp/registry/javascript/tests/test_milestone_creation'
[04:56] <StevenK> Hm
[04:56] <lifeless> StevenK: so - marshaller
[04:57] <lifeless> StevenK: look at the marshallers available, I'll bet that its one with a bug in it
[04:58] <StevenK> <class 'lazr.restful.marshallers.DictFieldMarshaller'>
[05:00] <lifeless> StevenK: so a few lines up
[05:00] <lifeless>         marshaller = getMultiAdapter((field, self.request), IFieldMarshaller)
[05:00] <lifeless> that suggests that that is going off the rails, to me at least
[05:01] <StevenK> lifeless: How?
[05:01] <StevenK> The field is declared as an exported dict
[05:01] <StevenK> And it's attempting to unmarshall the contents
[05:01] <StevenK> It just doesn't deal with None at all
[05:01] <lifeless> right
[05:03] <StevenK> Right, if value is None: return {} works nicely
[05:03] <StevenK> HAH
[05:03] <StevenK>     def _get_diffstat(self):
[05:03] <StevenK>         if self._diffstat is None:
[05:03] <StevenK>             return None
[05:04] <StevenK> GIGO
[05:04] <lifeless> EWTF
[05:04] <lifeless> how is that different to
[05:04] <lifeless> return self._diffstat ?
[05:04] <StevenK> Below that it returns a dict
[05:04] <lifeless> ah
[05:08] <StevenK> Fantastic, I don't need to touch lazr.restful
[05:12] <StevenK> lifeless: So, can you look at the last Jenkins build?
[05:12] <StevenK> lifeless: I have this inkling there is something screwy with the subunit stream
[05:13] <lifeless> url me up
[05:13] <StevenK> lifeless: https://lpci.wedontsleep.org/job/devel/lastBuild/
[05:18] <lifeless> ERROR: cannot verify lpci.wedontsleep.org's certificate, issued by `/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA':
[05:18] <lifeless>   Issued certificate has expired.
[05:21] <StevenK> Yes
[05:21] <StevenK> We should fix that by moving to prase
[05:21]  * StevenK bangs that drum a little harder
[05:23] <lifeless> StevenK: ok, so tricks:
[05:23] <lifeless> cat consoleText| subunit-stats | less
[05:23] <lifeless> page down till you see
[05:23] <lifeless> Starting up the container...
[05:23] <lifeless> after that you should see nothing as subunit will be eating it
[05:24] <lifeless> when you start seeing something, its because subunit thinks a test prior to it has not finished
[05:24] <lifeless> and is in pass-through mode on everything other than a terminal for that test.
[05:24] <lifeless> so
[05:24] <lifeless> test: lib/lp/soyuz/stories/ppa/xx-private-ppas.txt
[05:24] <lifeless> is the first output
[05:24] <lifeless> use that to seek in the raw stream to debug
[05:24] <lifeless> less consoleText
[05:24] <lifeless>  / lib/lp/soyuz/stories/ppa/xx-private-ppas.txt
[05:24] <lifeless> and work back:
[05:25] <lifeless> test: Could not communicate with subprocess
[05:25] <lifeless> time: 2012-09-27 04:39:15.455646Z
[05:25] <lifeless> test: lib/lp/soyuz/stories/ppa/xx-private-ppas.txt
[05:25] <lifeless> ^ thats invalid yo
[05:26] <StevenK> Could not communicate with subprocess is zope.testing garbage, isn't it?
[05:26] <lifeless> yes, its odd that it says 'test: Could not communicate with subprocess'
[05:27] <lifeless> theres no immediate evidence of interleaving going on
[05:27] <lifeless> it *could* be a race condition with gc or something
[05:27] <lifeless> or just bad code for handling that situation
[05:27] <lifeless> e.g.
[05:27] <lifeless> there is a reporter thingy in zope.testing
[05:27] <lifeless> it has a special class for subunit
[05:27] <lifeless> that class may have a bug
[05:28] <lifeless> where it reports this situation via the literal 'test: Could not communicate with subprocess\n'
[05:28] <lifeless> which would be a bug.
[05:28] <StevenK> Then why doesn't buildbot see that :-(
[05:28] <lifeless> it may not be encountering 'Could not communicate with subprocess'
[05:28] <lifeless> or it may be one of the things gary keeps posting a bug link to
[05:29] <lifeless> StevenK: s    def error_with_banner(self, message):
[05:30] <lifeless>         self._emit_fake_test(message, self.TAG_ERROR_WITH_BANNER)
[05:30] <StevenK> Haha
[05:31] <StevenK> That does seem a little bit like crack, yes.
[05:31] <lifeless> which calls startTest(test), _emit_tag(tag), addSuccess(test)
[05:31] <lifeless> so, that should be ok, but..
[05:32] <lifeless> and in fact..
[05:32] <lifeless> test: Could not communicate with subprocess
[05:32] <lifeless>  is line 108890
[05:32] <lifeless> successful: Could not communicate with subprocess
[05:32] <lifeless> is 108996
[05:33] <StevenK> subunit doesn't like tests with whitespace?
[05:33] <lifeless> so there is an interleaving problem
[05:33] <lifeless> subunit doesn't care about that
[05:33] <lifeless> open consoleText with vim
[05:34] <lifeless> something wrote, to the same fd, a tonne of other test info between the three lines of that self._emit_fake_test function
[05:34] <StevenK> wallyworld, wgrant: https://code.launchpad.net/~stevenk/launchpad/preview-diff-none-type/+merge/126601
[05:35] <StevenK> lifeless: Yeah, the line numbers are staggeringly different
[05:36] <lifeless> StevenK: part of the issue here is that you're getting the demultiplexed stream out of testr
[05:36] <StevenK> testr run --parallel --subunit --full-results --concurrency=8 | subunit2junitxml -f -o test-results.xml
[05:37] <lifeless> yeah
[05:37] <lifeless> testr is running 8 streams on separate fd's and demultiplexing them.
[05:37] <lifeless> its possibly a bug in your testtools version
[05:37] <lifeless> what testtools and subunit versions are you running ?
[05:38] <StevenK> ii  python-testtools                0.9.14+bzr267~ppa37~precise1    Extensions to the Python unittest library
[05:38] <StevenK> ii  subunit                         0.0.8+bzr176~ppa126~precise1    command line tools for processing Subunit streams
[05:38] <StevenK> Both from testing-cabal
[05:39] <lifeless> hmmm
[05:39] <lifeless> I have to run, cynthia time
[05:39] <lifeless> suggestion
[05:39] <lifeless> drop the subunit2junitxml -f -o test-results.xml for now
[05:39] <nigelb> 54
[05:39] <lifeless> and the --full-results and the --subunit
[05:39] <nigelb> argh, sorry!
[05:40] <lifeless> testr will write its output to .testrepository, the jenkins fs explorer thingy will let us pull the stream down and see what it gets
[05:40] <lifeless> I will return in ~2 hours
[05:40] <StevenK> Which is how long the build will take
[06:18] <wgrant> StevenK: Did the build look successful other than the dead runner and corrupt stream?
[06:19] <StevenK> wgrant: Mostly
[06:19] <StevenK> I'm doing what lifeless' suggested, so we'll see
[06:20] <StevenK> But since testr likes hiding things, we have no output at all
[06:20] <wgrant> StevenK: Tail the stream
[06:20] <wgrant> It's in ~/.testrepository
[06:20] <wgrant> It'll be tmpFOO
[06:21] <StevenK> Right
[06:21] <StevenK> wgrant: Can haz review?
[06:21] <wgrant> Er
[06:21] <wgrant> Not ~/, but the tree
[06:22] <StevenK> Yes, I found it
[06:22] <StevenK> -rw------- 1 jenkins jenkins 3.4M Sep 27 06:22 .testrepository/tmpTDk8xw
[06:23] <wgrant> StevenK: Nothing else relies on diffstat being None sometimes?
[06:24] <StevenK> Hm, maybe
[06:28] <StevenK> wgrant: Good catch. http://pastebin.ubuntu.com/1229847/
[06:28] <wgrant> Well
[06:28] <wgrant> This is interesting
[06:29] <wgrant> I think our bugtracker may be sentient
[06:29] <wgrant> https://bugs.launchpad.net/launchpad?field.searchtext=quote_like
[06:29] <wgrant> Look at the first result
[06:30] <wgrant> First thought was "these results are terrible". Open first result, "Cannot search for identifier containing underscores... well I can search for it but the results aren't any good."
[06:30] <StevenK> Haha
[06:31] <StevenK> wgrant: I can't see any other code that relies on diffstat being None
[06:31] <wgrant> Did you consider making lazr.restful deal with it?
[06:32] <StevenK> I did, and then decided it was a case of GIGO
[06:32] <wgrant> Well, we allow object references to be nullable
[06:32] <wgrant> Why not dicts?
[06:34] <StevenK> Like you say, it could be fixed in lazr.restful's marshaller. But I decided that fixing LP was a little easier
[06:35] <StevenK> And it didn't involve working out how to run its tests and then spinning a new release, uploading to pypi and download-cache and then making an LP branch anyway
[06:37] <StevenK> If you strongly object then I can do that on Tuesday
[06:39] <wgrant> StevenK: It's not that hard; wallyworld did it yesterday
[06:40] <wallyworld__> and finished the job today
[06:40] <StevenK> I know it's not hard. It's easier to just change LP.
[07:06] <StevenK> wallyworld__: I don't think lp:lazr.restful contains your changes
[07:06] <wallyworld__> hmm. i'm sure i merged, let me check
[07:07] <StevenK> Hmm, yeah, revision 200
[07:07] <wallyworld__> ah, merged restfulclient
[07:07] <wallyworld__> forgot restful
[07:07] <wallyworld__> let me do it now
[07:08] <StevenK> The version.txt in the tree and pypi did not match
[07:08] <StevenK> And I didn't want to unfix your carefully fixed bug
[07:09] <wallyworld__> StevenK: done, sorry
[07:15] <StevenK> https://code.launchpad.net/~stevenk/lazr.restful/dict-unmarshaller-none/+merge/126615
[07:16] <wgrant> StevenK: Why isn't None None?
[07:16] <wgrant> None shouldn't be {}
[07:16] <wgrant> It should be None
[07:16] <wgrant> probably
[07:16] <StevenK> Well, it's better than what we have currently, which is AttributeError
[07:17] <wgrant> Exposing an unstable interface in lazr.restful is not better than AttributeError :)
[07:19] <wallyworld__> wgrant: StevenK: i had to reboot again after another unity crash and now lp.dev won't load in ff, but loads in chrome. "The connection was interrupted". any idea?
[07:20] <StevenK> Stop stabbing zope while loading it in Firefox?
[07:20] <wallyworld__> i have no knife
[07:20] <wgrant> wallyworld__: Tried restarting Apache? That sometimes indicates it's SSL listening on a non-SSL port or vice-versa
[07:20] <wallyworld__> wgrant: yeah, tried that
[07:21] <StevenK> wgrant: Diff updated
[07:22] <wgrant> StevenK: Have you tried this with lazr.restfulclient?
[07:22]  * StevenK de-prams his own toys
[07:23] <StevenK> wgrant: No, I haven't
[07:27] <LPCIBot> Project devel build #1098: STILL FAILING in 1 hr 45 min: https://lpci.wedontsleep.org/job/devel/1098/
[07:28] <StevenK> That isn't fun.
[07:52] <adeuring> good morning
[13:17] <jtv> Say... I don't have a working launchpad setup at the moment.  Would anyone be able to test & land this branch for me?  https://code.launchpad.net/~jtv/launchpad/format-relative-imports/+merge/124878
[13:53] <stub> jtv: sending via ec2
[13:53] <jtv> ?
[13:53] <jtv> Oh, branch.  Thanks!
[13:53] <stub> responsive r us
[14:04] <tumbleweed> stub: talking of responsiveness, have a moment to eyeball https://code.launchpad.net/~stefanor/launchpad/edit-packagesets/+merge/124555 ?
[14:12] <stub> tumbleweed: done
[14:13] <tumbleweed> stub: thanks
[16:29] <adeuring> jcsackett: thanks for your review!
[16:29]  * adeuring nearly forgot the MP
[16:30] <jcsackett> adeuring: you're welcome.
[18:28] <lifeless> sinzui: https://bugs.launchpad.net/launchpad/+bug/736010 is affecting openstack
[18:28] <_mup_> Bug #736010: Milestone:+index timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/736010 >
[18:29] <lifeless> sinzui: wondering if one of your timeout demolishing machines could cast an eye over it.
[18:30] <abentley> deryck: I'm getting a test failure on stable.  Both test and code are old.  Can you reproduce?  http://pastebin.ubuntu.com/1230926/
[18:32] <deryck> abentley, let me see....
[18:35] <lifeless> sinzui: oh, I see ttx is in #launchpad talking abou tit
[18:42] <sinzui> lifeless, We have discussed that bug a lot. We think a lot of work is needed t fix this issue. This is a design problem, not a a bad query problem
[18:44] <deryck> abentley, I don't get that failure.
[18:44] <lifeless> sinzui: Why do you say its a design problem ?
[18:45] <lifeless> sinzui: I see 2000ms of time between query 66 and 67 in https://oops.canonical.com/oops/?oopsid=OOPS-c3db3abe60cbad08100ecb38fcac389a#statementlog
[18:45] <abentley> deryck: Wacky.  It looks legit to me, but it also doesn't seem to happen on ec2.
[18:46] <deryck> WEIRD
[18:46] <deryck> sorry to shout, fat fingers :)
[18:46] <deryck> I wonder what's different for you that you see it?
[18:47] <sinzui> lifeless, the page only works if you can display all milestones and blueprints. It does not scale. We can improve the queries, but it still will not scale for large projects. I think a proper fix is to load just the crucial/first items, then use ajax to complete the listings, then do the analysis.
[18:47] <abentley> deryck: Maybe python version.  Are you on quantal 2.7.3?
[18:47] <lifeless> sinzui: I am confused; the OOPS doesn't show queries as a problem at all.
[18:47] <lifeless> sinzui: Are we talking about the same issue ?
[18:48] <deryck> abentley, ah, no.  I still have upgraded actually. :(
[18:48] <deryck> abentley, but I am on 2.7.3
[18:48] <sinzui> yes we are, This bug has been around a long time and making queries faster does not fix the issue
[18:48] <lifeless> sinzui: ok, so we're slightly in sync :) - I see bad python code / tal code.
[18:49] <sinzui> You can fix that query, but it want users the believe you have fixed the issue for there hundreds of items, we need to think about the solution differently
[18:49] <sinzui> YES
[18:49] <lifeless> sinzui: I agree that doing latency hiding and incremental loading would be better.
[18:49] <sinzui> Late evaulation that because the page relies on bug and blueprint rendering rules
[18:49] <lifeless> sinzui: but is there not a low hanging just-be-more-efficient step to get Ubuntu and openstack unstuck ?
[18:49] <lifeless> sinzui: its not late evaluation, only 68 queries on the page before it timed out .
[18:50] <sinzui> I think we just want to pull the data and have the browser do the analysis...such as I do in the burndown charts I wrote
[18:50] <lifeless> sinzui: those charts are sweet ;)
[18:51] <sinzui> I don't believe the issue is low hanging. I have landed more than a dozen speed ups in 2 years. so have you and William
[18:51] <sinzui> I asks jcsackett to not leap into that bug given the tremendous failure rate
[18:52] <lifeless> hmm
[18:52] <lifeless> loaded for me
[18:52] <jcsackett> lifeless, sinzui: timeout fixed.
[18:52] <lifeless> 69 queries in 6.83 seconds
[18:52] <lifeless> jcsackett: how ?
[18:52] <jcsackett> as i said, was increasing the wrong hard_timeout rule.
[18:52]  * jcsackett misread the flags.
[18:52] <jcsackett> the flag was for ProjectMilestone; we just now fixed milestone
[18:52] <jcsackett> well "fixed" ;-p
[18:53] <sinzui> lifeless, wgrant, jcsackett, and wallyworld__can discuss the effort to make that page properly fast for all project sizes in a few hours
[18:54] <lifeless> sinzui: my curiosity is piqued
[18:54] <lifeless> sinzui: I agree on the long term
[18:54] <lifeless> but I'm wondering why the short term is so poor
[18:54] <sinzui> certainty.
[18:55] <sinzui> 1 day of effort to fix one bug is high certainty to deliver value
[18:55] <sinzui> 5 days to fix 5 bugs is less certain, but equal value
[18:56] <sinzui> 10 days to fix 1 bug is even less certain to deliver less value.
[18:57] <sinzui> lifeless, we are a "machine" because we are taking the most certain bugs first. I asked jcsackett to not work on that one yet because there is more risk involved.
[18:57]  * sinzui hoped to fix another escalated bug before having to do feature analysis to fix criticals
[18:57] <lifeless> sinzui: all cool
[18:58] <lifeless> sinzui: I meant, what about the code makes this one so poor
[18:58] <lifeless> sinzui: not about your team or your choices ;)
[19:00] <sinzui> we often choose to make pages do less, or batch. milestones do need to show all items. The tales calls are an embarrassment. I choose to reuse code to keep presentation consistent, but the adaptions are costly.
[19:02] <sinzui> lifeless, the analysis is costly in Python I think. I know I can do it in JS now.
[19:05] <lifeless> sinzui: I'm getting a profile
[19:06] <lifeless> https://qastaging.launchpad.net/++profile++show/ubuntu/+milestone/ubuntu-10.10/+index#
[19:06] <lifeless> just sneaked in: 3378720 function calls (3296251 primitive calls) in 19.845 CPU seconds
[19:12] <lifeless> sinzui: there are some low hanging fruit that might make a lot of things better (or might be profile artifacts)
[19:12] <lifeless>      5134    0.810    0.000    0.979    0.000 enumcol.py:39(__init__)
[19:12] <lifeless> ^ 4% of total runtime, I believe wgrant has complained about enumcol performance before
[19:12] <sinzui> yes
[19:13] <lifeless> a few more like that
[19:14] <sinzui> this page might break now that there are private bluerprints too. milestones do the private bugs query then cache the permissions, but I did not see such as change land for blueprints
[19:15] <lifeless> yay
[19:19] <lifeless> sinzui: the ubuntu timing out page is only 140k in size
[20:24] <abentley> jcsackett: I have a sequence of four branches that need review.  Are you up for it?
[20:24] <jcsackett> abentley: if said sequence starts with new-tests, i'm already looking at it.
[20:25] <abentley> jcsackett: Excellent.  Continues with https://code.launchpad.net/~abentley/launchpad/storm-sprint-queries/+merge/126770 https://code.launchpad.net/~abentley/launchpad/specification-cleanup/+merge/126789 https://code.launchpad.net/~abentley/launchpad/hide-sprint-blueprints/+merge/126792
[20:27] <jcsackett> abentley: cool. i can get 'em all.
[21:05] <sinzui> jcsackett, do you have time to review https://code.launchpad.net/~sinzui/launchpad/email-authentication-errors/+merge/126801
[21:06] <jcsackett> sinzui: sure, you're in the queue.
[21:06] <jcsackett> still looking through abentley's series.
[21:12] <sinzui> abentley, rick_h_. was the privacy banner or information portlets updated in the last few days
[21:12] <sinzui> JS is broken on pages that show the privacy banne like bugs and archives
[21:13] <abentley> sinzui: Yes, rick_h_ changed the banner code yesterday IIRC.
[21:13] <sinzui> fab, maybe the diff will help me find the problem
[21:14] <abentley> sinzui: (rick_h is away)
[21:14] <sinzui> okay, diff it will be
[21:35] <jcsackett> abentley: i'm through your series of branches. r=me on all but one which has a few questions.
[21:35] <abentley> jcsackett: I've replied.
[21:35] <abentley> jcsackett: Thanks!
[21:38] <jcsackett> abentley: missed both things you pointed out. given those, r=me on that one as well.
[21:38]  * jcsackett moves on to sinzui's.
[21:39] <sinzui> wallyworld__, If your waking: read this to learn what I have learned so far https://bugs.launchpad.net/launchpad/+bug/1057714
[21:39] <_mup_> Bug #1057714: javascript is broken on pages that show the privacy banner <bugs> <javascript> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1057714 >
[21:40] <wallyworld__> sinzui: haven't read that bug but i think it may be a dupe of mine from yesterday, for the mp ypu approved
[21:40] <wallyworld__> mine is bug 1057248
[21:40] <_mup_> Bug #1057248: javascript error on bugtask page <javascript> <regression> <Launchpad itself:In Progress by wallyworld> < https://launchpad.net/bugs/1057248 >
[21:40] <sinzui> wallyworld__, does this look familiar: http://pastebin.ubuntu.com/1231240/
[21:41] <sinzui> This is the change, but I do not know why info_type is not defined in the privacy module
[21:41] <wallyworld__> sinzui: yes. i changed privacy.js and information_type.js
[21:41] <sinzui> okay,
[21:41] <wallyworld__> there were vrious issues - 'throw' was misspelt, lots of lint
[21:42] <sinzui> ah@
[21:42] <wallyworld__> plus the info_type issue
[21:42] <sinzui> !
[21:42] <sinzui> okay
[21:42] <wallyworld__> landing my branch now
[21:42] <lifeless> wgrant: enumcol would benefit from a cache I suspect
[21:42] <lifeless> wgrant: one of the rare times I'll say that
[21:42] <wallyworld__> we can deploy this morning
[21:42] <sinzui> we can run the yui layer locally, then lp-land
[21:42] <wallyworld__> sinzui: yes, doing that now
[21:44] <sinzui> sorry wallyworld__I remember the diff, but I did not read all the details int he bug. I suck
[21:45] <jcsackett> sinzui: r=me on your branch.
[21:45] <sinzui> wallyworld__, I was rushing to review branches because I had a day trip to look at houses
[21:45] <sinzui> thank you jcsackett
[21:52] <wallyworld__> np
[22:54] <cjwatson> wgrant: We've flushed nearly all the queue now following beta-2, and it all seemed pretty snappy; no timeouts in 80+ accepts.  At this point I'm confident with going ahead and removing the queue script.  Is there any remaining review you feel the need to do on https://code.launchpad.net/~cjwatson/launchpad/remove-queue-tool/+merge/114464 ?
[23:01] <wgrant> cjwatson: Let me look
[23:03] <wgrant> lifeless: Hmm
[23:03] <wgrant> lifeless: I might try to profile it a bit harder now that someone other than me thinks it might be a problem.
[23:05] <rick_h_> wallyworld__: man, so sorry. Just thought I added unused stuff. Tests passed, etc. I shouldn't have tried to rush that before I left
[23:06] <wgrant> cjwatson: r=me. Are you also going to remove the feature flag check?
[23:07] <wallyworld__> rick_h_: no problem. shit happens :-)
[23:08] <rick_h_> wallyworld__: I owe you one. broke it apart and tried to stage it in nice simple bits that didn't get too involved and boom
[23:09] <wallyworld__> rick_h_: we may need to look at the events - seems there are 2 sets (show/hide banner, info type public/private) that do the same thing. may be able to be refactored to just one set
[23:10] <rick_h_> wallyworld__: well so I can explain that
[23:10] <wallyworld__> i didn't get into it too far
[23:10] <cjwatson> wgrant: That's going through EC2 at the moment (https://code.launchpad.net/~cjwatson/launchpad/pabj-remove-feature-flag/+merge/126090)
[23:10] <rick_h_> because the filbug.js uses the privacy banner directly with a custom message, you need to be able to talk to the privacy banner with a custom message
[23:10] <wgrant> cjwatson: Ah, great
[23:10] <wgrant> I have a fair bit of MP mail; haven't quite caught up this morning yet
[23:11] <cjwatson> Yep.  Thanks.  I look forward to my giant LoC credit
[23:11] <rick_h_> wallyworld__: but the normal use case is that someone changes the information_type widget, which fires the information_type:changed, and files information_type:is_public/private and the banner will auto show/hide based on that event
[23:11] <cjwatson> And I haven't even removed delayed copies yet ...
[23:11] <rick_h_> wallyworld__: so there's really two use cases and almost should be a different banner, since privacy and security are sharing code tbh
[23:11] <lifeless> wgrant: there is a kcachegrind file for you
[23:11] <wallyworld__> rick_h_: ok, thanks for the explanation
[23:11] <wallyworld__> my main aim was to add the missing tests and get the fix out
[23:12] <rick_h_> wallyworld__: and in the next branch the banners will render themselves vs the html being in the .pt file, so the events will be used as ways to say "need a banner html setup here"
[23:12] <wallyworld__> cool :-)
[23:12] <rick_h_> wallyworld__: right, understand. and it's not 100% tested because it's part 1 of 4 and in general this was 'new unused' code since nothing is firing the events yet
[23:13] <wallyworld__> rick_h_: one thing. i had to move the info_type namespace variable from the top of the module or else it was undefined
[23:13] <wallyworld__> i don't know why
[23:13] <rick_h_> wallyworld__: so like I said appreciate it. Been offline most of the day on vaca so bummed to log in and see I did wrong for you guys
[23:13] <rick_h_> wallyworld__: hmmm, let me look
[23:14] <wgrant> lifeless: But it's from qastaging, so it's not completely useful
[23:14] <wgrant> Not useless, though
[23:14] <lifeless> not useless
[23:15] <rick_h_> wallyworld__: ok that's strange. I'll check it out when I get to the follow up branch when I get back
[23:15] <wgrant> rick_h_: Missing context here, but we only recently moved the banners *into* the HTML, to avoid them popping into the page later on once the JS loads.
[23:15] <wallyworld__> rick_h_: thanks. i didn't get into the root cause. i'd love to know why
[23:15] <rick_h_> wallyworld__: nothing looks odd, but running on 3hrs sleep after driving 600+ miles overnight
[23:16] <wallyworld__> rick_h_: where did you drive from/to?
[23:16] <rick_h_> wallyworld__: https://maps.google.com/maps?saddr=clarkston,+mi&daddr=Charlottesville,+VA&hl=en&sll=37.6,-95.665&sspn=38.593229,86.396484&geocode=FRAWjAIdYh8H-ymTC47xX5ckiDGKYHxzNWKthw%3BFfpHRAIdeopS-ymPpFDqLYaziTH8dIvDlvCGkA&oq=charlottesvi&mra=ls&t=m&z=6
[23:16] <rick_h_> doh
[23:17] <wallyworld__> looks like a nice drive
[23:17] <rick_h_> wgrant: yea, I'll hopefully be able to take care of that still. The big thing is we need banners on more places and it takes a bunch of code to wire it up it seems. Trying to split it into contained parts
[23:18] <rick_h_> wgrant: for instance trying to not have the privacy banner need to know about the information type, LP.cache locations, etc.
[23:18] <wgrant> Indeed.
[23:18] <rick_h_> wgrant: but thanks for the heads up. I didn't realize it was moved due to a flash effect so good to keep an eye on
[23:19] <rick_h_> wallyworld__: yea, well with a 2yr old those overnight drives still the best so he sleeps through most of it. Thanksfully we also had the lion king movie soundtrack for backup lol
[23:19] <wallyworld__> ah, those were the days. long gone for me now
[23:19] <wallyworld__> rick_h_: wgrant still listens to the Lion King for his beddy byes time
[23:19] <wgrant> Heh
[23:20] <rick_h_> hah
[23:21] <lifeless> wgrant: the other thing is there is a huge amount of content being re-rendered
[23:22] <lifeless> wgrant: a simple static result cache for rendering of person priority importance status might help a great deal.
[23:22] <wgrant> We can possibly just also make that not terrible
[23:22] <lifeless> or do both
[23:23] <lifeless> evaluating the same template with the same parameters in the same request to the same bytes seems non horrible to me
[23:23] <wgrant> Says he who removed template fragment caching :P
[23:23] <wgrant> (yes, yes, it was the wrong way to cache, but blah)
[23:23] <wgrant> :)
[23:24] <lifeless> I'm not against caches /per se/
[23:57] <wgrant> wallyworld__: qas is updated, if you want to check out the JS stuff
[23:57] <wgrant> Not sure why it's [testfix]
[23:57] <wallyworld__> finally, been waiting
[23:57] <wallyworld__> wgrant: because db bb was borked and i needed to get it landed
[23:57] <wallyworld__> we should fix that
[23:58] <wgrant> wallyworld__: In that situation you should force the build and wait 5 minutes
[23:58] <wgrant> [testfix] skips all QA
[23:58] <wallyworld__> ok