[00:33] thumper: mumble? [00:36] http://searchengineland.com/google-fans-throw-eggs-at-houses-blurred-in-google-street-view-56763 [01:11] lifeless: :( [01:11] You made us the top timeout again :( [01:12] wgrant: there's an easy answer for that [01:12] wgrant: Fix my scaling test for binary packages. [01:13] I looked at that last week. [01:13] But it wasn't obvious what was doing it. [01:13] the test ? [01:13] I couldn't see what was causing all those queries. [01:13] I think the test is flawed, the page doesn't show binaries without sources [01:13] wgrant: fix the test, it will be trivial to see after that ;) [01:18] lifeless: Which test? [01:19] wgrant: the one I landed [01:19] wgrant: sorry to be vague [01:19] Heh, OK. [01:19] look at log for my most recent Archive: landing [01:19] late last week I think [01:19] it adds two tests [01:19] test_source_query_counts and co? [01:19] one for source package scaling which works well - I got the test to demonstrate terrible scaling [01:20] * wgrant fixes. [01:20] the binary version *doesn't* demonstrate scaling [01:20] so I think its misspelt somehow [01:26] mwhudson: hi [01:26] mwhudson: test running. you were asking.. [01:26] lifeless: hello [01:26] ah yeah [01:26] I replied. [01:26] over there -> [01:26] didn't realize testtools had discovery [01:26] all the shiny [01:27] AFAIK your options are testtools/subunit .run, trial --reporter=subunit, nose with the subunit plugin. [01:27] hmm [01:27] AssertionError: Unable to use discovery, must use python 2.7 or greater, or install the discover package. [01:27] mwhudson: well, have you installed it ? :) [01:27] no, but i have unittest2 installed [01:27] is it packaged? [01:27] it may have discover as a private module [01:28] http://pypi.python.org/pypi/discover [01:34] wallyworld_: hey [01:34] ho [01:37] thumper: lib/lp/codehosting/vfs/branchfs.py [02:52] === Top 10 Time Out Counts by Page ID === [02:52] Hard / Soft Page ID [02:52] 99 / 5038 Archive:+index [02:52] 75 / 219 BugTask:+index [02:52] 34 / 361 Distribution:+bugs [02:52] 32 / 128 ProjectGroupSet:CollectionResource:#project_groups [02:52] 23 / 378 POFile:+translate [02:52] 17 / 51 DistroSeries:+queue [02:52] 15 / 10 Person:+commentedbugs [02:52] 11 / 2 Person:+bugs [02:52] 7 / 87 Archive:+packages [02:52] 7 / 5 DistributionSourcePackage:+changelog [02:52] wgrant: so, did you make any progress? === lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: - | https:/​/​dev.launchpad.net/��� | Get the code: https:/​/​dev.launchpad.net/​Getting [03:08] poolie: hey [03:08] poolie: what was the right spelling for LPNET_SERVICE_ROOT ? [03:08] hi thar [03:09] the trick is that they are now in launchpadlib.uris [03:09] lifeless: ^^ [03:09] poolie: got a code fragment ? [03:09] bzr pull -d ~/src/hydrazine ☺ [03:09] poolie: I'm writing a blog post [03:09] totally different space [03:09] thats a very robert answer to give [03:10] is it [03:10] karma :) [03:10] 'from launchpadlib.uris import LPNET_SERVICE_ROOT' ? [03:10] hang on [03:10] i wish it was fewer clicks from the project page to the source :/ [03:10] poolie: don't you have it local ? [03:11] I was assuming you could just copy paste a single line [03:11] ok [03:11] from launchpadlib.uris import (LPNET_SERVICE_ROOT, ) [03:11] thanks [03:11] return Launchpad(credentials, service_root, [03:11] lplib_cachedir) [03:12] etc [03:16] btw is it just me or are the apis getting faster too? [03:16] also, is there any point filing bugs about api timeouts, or should i assume the oops system will tell you about it [03:17] if they are not on https://bugs.launchpad.net/launchpad-project/+bugs?field.tag=timeout [03:17] then feel free to file [03:18] please include the page id in the title to make it easy to identify dupes [03:18] apis will be getting faster as we fix core issues. [03:18] https://dev.launchpad.net/LEP/PersistenceLayer may interest you [03:37] lifeless: did you mean to add control characters to the /topic? [03:39] mwhudson: It was a copy paste job. [03:39] mwhudson: they were presumably there already ;) === mwhudson changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: - | https:/​/​dev.launchpad.net/ | Get the code: https:/​/​dev.launchpad.net/​Getting [03:39] doesn't look like it [03:39] home time, probably back later [03:39] bai [03:42] lifeless: what do we do for a self review again? [03:42] lifeless: do I just approve myself? [03:43] you review it youself [03:43] as normal [03:43] just like you'd review someone else [03:45] gah [03:46] where do I change my login token? [03:46] * thumper goes to hunt through the email [03:47] qastaging is still down? [03:48] * thumper stabs mail [03:50] poolie: its working for me [03:50] poolie: https://qastaging.launchpad.net/ comes up ok, if slow. [03:50] bug 680759 [03:50] <_mup_> Bug #680759: qastaging api 503 errors [04:21] poolie: I think my further updates to that bug, and your mails, crossed paths. [04:22] yes they did [04:26] is qastaging now the best place to nondestructively test api clients? [04:27] lifeless: Yeah, I got the test to fail. [04:27] Will look at fixing the actual bug shortly. [04:31] wgrant: \o/ [04:31] lifeless: just to be clear i'm not complaining, just wanted to flag this in case it wasn't known && matters [04:31] wgrant: can you show me what I should be doing differently ? [04:31] poolie: qastaging is an ok place [04:32] you just have to be robust against timeouts [04:32] but its not resourced enough to be a reliable test environment (neither was staging - the DB massively outweighs the RAM on the db server) [04:32] fair enough; that's a good practice anyhow [04:32] yeah [04:32] lifeless: Your guess was correct -- the index doesn't show binaries without sources. [04:32] wgrant: whats the diff look like ? [04:32] for the sake of my education is RequestExpired also a timeout? [04:32] poolie: yes [04:33] lifeless: http://pastebin.ubuntu.com/535763/ [04:33] poolie: exact error depends on the exact bit that decided time was over [04:33] It would be nice if mBPPH had an arg to do that. [04:33] wgrant: yeah [04:34] wgrant: +1 on any patch doing that :) [04:34] I need to look at the factory and the Soyuz tests and make them match. [04:35] Since at the moment each seems to try to pretend that the other didn't exist at the time of writing. [04:46] lifeless: Bug #676776 should have nothing to do with the buildd-manager. [04:46] <_mup_> Bug #676776: build stuck with "Uploading build" status [UI] [04:46] wgrant: julian said 'with the workaround in place builds get stuck and I have to clean them up' [04:47] wgrant: IMBW of course. [04:47] but the symptoms sound exactly as he described [04:47] lifeless: They'll get stuck in PENDING/BUILDING. [04:47] Not UPLOADING. [04:48] there's no xmlrpc calls involved during uploading? [04:48] No. UPLOADING is set once the master has downloaded the files from the slave and thrown them into the upload queue. [04:48] Then it's in p-u's hands, not b-m. [04:48] ok [04:48] thanks for clarifying [07:12] "no module gettextpo" indicates out of date sourcecode, i guess? [07:12] yeah [07:12] utilities/update-sourcecode [07:12] + make [07:17] yup [07:21] and i guess "ImportError: cannot import name LPModerate" would be similar, but update-sourcecode hasn't fixed it [07:21] mm [07:21] dunno about that [07:28] poolie: That's some mailman breakage. make clean usually fixes it for me. [07:35] i think it's a test ordering dependency [07:36] iow these tests won't pass unless something else runs first in the same process :/ [07:36] oh well === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan [08:47] good morning [08:59] morning [09:34] bigjools: Did the log parser eventually finish? [09:34] wgrant: yes but it's got a new problem [09:34] :( [09:35] What is it? [09:35] hang on [09:35] http://librarian.dogfood.launchpad.net/57530685/1z7LhiYuLtwxWSJXAxHNvbyksXZ.txt [09:35] ... pardon? [09:35] it's not exactly a useful traceback :/ [09:36] The real exception wasn't logged at all? [09:36] "Unhandled exception" is all you get [09:36] and then I stopped working on it as I need to debug the b-m === matsubara-afk is now known as matsubara === mrevell is now known as mrevell-lunch [12:01] lunch already [12:01] * jml is not really succeeding at work today [12:04] hey [12:04] can I persuade anyone to test Launchpad against the latest pre-release of Twisted? === al-maisan is now known as almaisan-away [12:45] c'mon, there must be at least ten people with commit access around besides me. [12:47] jml: I can help you [12:48] bigjools: thanks. Want to test Launchpad against the latest pre-release of Twisted? http://twistedmatrix.com/users/glyph/10.2.0pre3/ [12:48] jml: what are you expecting me to test? [12:49] Launchpad. Update the version of Twisted that we use, run the tests, see what breaks. [12:50] the test suite is what I needed to hear [12:50] Congrats folks on getting rid of edge :) [12:50] jml: now, I need to learn how to update eggs :) [12:50] bigjools: doc/buildout.txt (from the root of the checkout) has a guide [12:51] yeah reading that already [12:55] bigjools: thanks. [12:56] ok that was easy [12:57] yeah. while I'm still not sold on buildout, it definitely makes upgrading dependencies easier than any other equivalent I've tried. [13:05] jml: ok tests running, I'll let you know in about 189 minutes [13:05] bigjools: woot. thanks. === mrevell-lunch is now known as mrevell [13:07] it'd be nice to just select trial tests [13:10] + distributionmirror_prober tests! [13:10] heh [13:10] and tests that use blocking APIs to exercise Twisted servers, I guess. [13:11] maybe it's worth ensuring we do import as TrialTestCase everywhere [13:11] nah [13:11] gimme a better idea [13:11] not using trial very soon anyway [13:11] oh, your testtools? [13:11] yeah [13:11] nyshe [13:12] run_tests_with = AsynchronousDeferredRunTest [13:12] (yeah, it's a horrible name) [13:12] ummm [13:12] or @run_test_with(ADRT) for individual tests. [13:33] great [13:34] I have to somehow figure out which revisions lifeless self-reviewed over the last three weeks. [13:35] * jml plays with log & grep, wishing again that Bazaar had a query language [13:36] jml: what about using the API? [13:37] bigjools: it only has by-status filtering, so I might as well check bzr. [13:37] :( [13:37] can you get all his MPs instead? [13:38] jml: bzr search? [13:39] jelmer: is that actually useful? I thought it was mostly a text search. [13:39] jml: I'm not sure how advanced its query language is. [13:39] jml: What are you trying to find? [13:40] jelmer: revisions of stable merged from branches authored by lifeless marked as being reviewed by lifeless [13:41] jelmer: I'm almost done in my search anyway, but it would be nice to know for next time [13:42] jml: Ah. No idea how to do that using the command line. [13:42] But it should be possible using a few lines of Python code and the API [13:44] jelmer: in general, I wish I didn't have to write so much Python to get stats about trunk landings to PQM-managed branches [13:46] jml: Do you mean the fact that it requires Python code at all? I'd think that it would only be a few lines using the graph API. [13:47] jelmer: yeah, pretty much. === almaisan-away is now known as al-maisan [13:54] bigjools: any progress on the timeout thing? [14:06] sinzui, Hey, around? [14:06] I am [14:07] sinzui, I was wondering if we could have a call later today. I'm revising the docs for someone who's setting up a new project (so right from registration through to getting each individual app configured) and I wanted to know if you have any views on what I should do. [14:08] sinzui, Also, is there a plan for resolving bug 4449? [14:08] <_mup_> Bug #4449: Project "title" is redundant with "display name" [14:08] mrevell: there's a spec by mpt that would be a good starting point. [14:08] * mrevell looks [14:09] mrevell: https://dev.launchpad.net/RegistrySimplifications [14:09] I would love to talk about this. My last effort to improve th page was a disappointment. Maybe docs and inline help will help users accomplish their task [14:09] Yeah, I think a mix is right ... it may also be time to de-mothball the screencasting apparatus, heh. [14:10] mrevell, jml: as a matter of fact, I was considering fixing all the dead attributes during the bug jam. [14:10] Thanks jml [14:10] sinzui, Ah, cool [14:10] sinzui, How does 17.15 UTC sound for a call? [14:11] mrevell, I think I am in a team lead meeting at that time [14:11] mrevell, 15:00 or 16:00? [14:12] sinzui, 15.00 works for me [14:12] okay [14:33] * jml hugs addDetail [14:34] anyone seen anything like this before? http://paste.ubuntu.com/535913/ (librarian db constraint violation) === matsubara is now known as matsubara-lunch [14:41] jml: no progress, although I've written 3 lines of code that will lock the slave manager in memory :) [14:41] bigjools: how will that help? [14:42] jml: I have a theory that it's getting paged out and thus can't respond quick enough to connections [14:43] although I'm starting to doubt that [14:43] but it's worth trying nonetheless === salgado is now known as salgado-lunch [14:45] jml: I've also analysed the logs a bit to see if there's any pattern in the builders that are failing [14:45] a minimal repro client might be a better starting point -- at least then we can completely eliminate lp.buildmaster. [14:45] some are failing a lot more than others [14:45] bigjools: oh. interesting. [14:45] https://bugs.edge.launchpad.net/launchpad-buildd/+bug/586359/comments/5 [14:45] <_mup_> Bug #586359: Virtual builders are sometimes very slow to accept connections [14:46] this librarian failure is perplexing [14:47] jml: I also looked at logs from April and the problem is exactly the same there [14:47] jml: when are you getting that librarian error? [14:47] henninge: can't get onto the other channel right now, but that export-to-branch failure you pasted was simply yet again the same general config problem. No reason to believe that the script itself hit any real problem otherwise. [14:47] when running lp.codehosting.codeimport.tests.test_workermonitor.TestWorkerMonitorUnit.test_finishJob_uploads_nonempty_file_to_librarian [14:48] I notice it's in retry_transaction_decorator [14:48] which is suspicious [14:48] jtv: it ran fine mostly. [14:49] henninge: yes, it was just logging that it was backing off from one branch to avoid a race condition. But that caused the oopsing machinery to log an oops, and that part was broken as elsewhere. [14:50] jtv: I filed bug 680908 about what we could do about it. [14:50] <_mup_> Bug #680908: Script translations-export-to-branch needs its own config section [14:56] it's intermittent [14:57] but only seems to be a thing w/ my testtools branch... [14:57] * jml does some bisecting [14:59] mrevell, mumble? [14:59] sinzui, Sure. [15:04] rockstar: hi, do you know if there's a reviewer meeting today and who's chairing? [15:04] jelmer, no idea. bac would be the guy to ask. [15:04] rockstar: thanks === al-maisan is now known as almaisan-away [15:16] jtv: Hey === almaisan-away is now known as al-maisan [15:19] mrevell, http://curtis.hovey.name/2010/11/02/encouraging-contribution-on-the-project-page/ [15:21] bigjools: as best as I can tell, (curse this intermittency), it's some interaction between the builder tests and the workermonitor tests. [15:21] or, to be precise, that's at least one way of reproducing the error. [15:21] jml: suck :/ [15:21] if you run both? [15:22] if I run both test_builder and test_workermonitor, then it sometimes fails. [15:22] so far haven't got the error from just running test_workermonitor [15:22] got a while loop going running it over & over. [15:22] but only sometimes .... urgh [15:23] yeah. [15:23] speaking of only sometimes [15:24] * jml goes downstairs to get some chocolate === matsubara-lunch is now known as matsubara [15:35] sinzui, ping [15:35] hi rockstar === al-maisan is now known as almaisan-away [15:36] sinzui, I wonder if you might be able to help me sort out some bug notification/ownership issues, since the bugs guys are gone and you seem to always know about permissions. [15:37] okay [15:37] * jcsackett hates cleaning up a bad merge state... [15:38] sinzui, the U1 team isn't getting notified on bugs filed on the u1 client filed through apport. They're trying to find out why. [15:38] hrmm, there have been some changes to privacy in lp recently haven't there? [15:39] rockstar, I could be via a structural subscription in a indirect team [15:39] * sinzui looks [15:40] so it looks like crash reports are all private until the rescanner does some magic [15:40] and this seems to be our issue === almaisan-away is now known as al-maisan [15:42] dobey, but also that once the rescanner is done, it still doesn't send notifications. [15:42] sure [15:42] but i'm ok with not getting more e-mail ;) [15:43] * jml sees more evidence for merging #launchpad & #launchpad-dev [15:46] jml, +1 [15:51] rockstar, ~ubuntuone-hackers is the bug supervisor or ubuntuone-client. They should be getting all bug reports via email [15:53] sinzui, https://bugs.launchpad.net/malone/+bug/425127 [15:53] <_mup_> Bug #425127: private bugs in packages people with access to private bugs are subscribed to don't generate emails [15:54] I was looking at the project, I will look at the package [15:55] If my reciprocal bug links were in production I would have seen that [15:56] hah [15:57] rockstar, Ubuntu One hackers has a structural subscription. A team admin can remove it [15:57] https://launchpad.net/ubuntu/+source/ubuntuone-client === salgado-lunch is now known as salgado [16:12] * jml blags === beuno is now known as beuno-lunch [16:39] :( [16:40] If I run any subset of the buildmaster.tests.test_builder tests, I don't see the error. :( [16:44] jml: May I add the Risks section (and the sub-section of experiment, goal/design/result) from https://dev.launchpad.net/Foundations/Proposals/Template to the LEP template so I can delete it ? [16:45] gary_poster: sure. [16:45] thanks [16:46] gary_poster: I think I would like the LEP template to imply that it's good to think of more than one implementation [16:47] gary_poster: but I think those are definitely good things to have when thinking about an implementation. [16:50] sinzui, ping, do you have a moment to help with a CHR documentation question? [16:51] * bigjools shakes fist at Flash plugin [16:54] jml: I put the things I did into the wiki page [16:54] mars, I am going into a meeting. I can type on irc [16:54] jml: but you can also query for merged mps, and then check the reviewers vs the mp branch owner [16:55] sinzui, just to clarify, so I don't do the wrong thing for reassigning Launchpad Answers (again) - "Retarget" means "Change the assigned project", correct? [16:56] lifeless: 'bzr log --line -r date:2010-10-31.. | grep r=lifeless' worked well enough [16:57] mars, yes, retarget does mean change the assigned project or distribution [16:57] jml: sure [16:57] thank you [16:57] sinzui, perfect, thank you [16:59] np [17:01] jml, I wasn't going to include the word "Implementation" in the copy to the LEP because it is confusing. That said, I think of "Workflows" as a bit of an implementation proposal. If the goal is to do blah blah for the webservice, how we accomplish blah blah is described in a workflow. That workflow is a possible approach to the goals. The workflow is a design that has risks. [17:02] So that's kind of where I think it goes [17:02] gary_poster: *nod* [17:02] cool [17:02] gary_poster: I don't like having workflows in the lep, actually :) [17:02] jml: gotcha [17:02] gary_poster: I only added it because I wanted to get beuno to stop shouting [17:02] but then all the LEP would be about is agreeing on goals? [17:02] gary_poster: yep [17:02] hm [17:02] gary_poster: goals, constraints, requirements [17:03] analysis, rather than design [17:03] I see [17:03] I'd be happy to have that discussion separately, jml [17:03] gary_poster: but if it gets used for design because managing two docs is too much work, I'm ok with that [17:04] jml, ok, we'll try separate docs then === al-maisan is now known as almaisan-away === beuno-lunch is now known as beuno [17:27] oh hai jml! [17:27] beuno: hi [18:01] lifeless: would you be available for a quick call nowish? [18:02] sure [18:02] thanks [18:03] I'm off for the day. Night [18:07] night all [18:09] lifeless: the testing-cabal PPA now has testtools and testrepository building in it [18:09] lifeless: I've done enough learning for now, so I'm going to open it up for the experts to fix up my mess :) === benji is now known as benji-lunch [18:26] jml: cool [18:27] back [18:44] you know it's getting serious when you have to draw a truth table to help you debug something [18:45] TTF [18:46] thirty-two combinations of fail [18:47] TTTTF? [18:47] yeah [18:49] lifeless: in case you have any insights: http://paste.ubuntu.com/535913/ is the error I'm seeing. It's during lp.codehosting.codeimport.tests.test_workermonitor.TestWorkerMonitorUnit.test_finishJob_uploads_nonempty_file_to_librarian but the error doesn't appear to happen when the that module's tests are run in isolation [18:50] lifeless: I can reproduce the error semi-reliably running the buildmaster.tests.test_builder tests before, but I've yet to get close to a minimal set [18:50] (debugging this one by the numbers, since my brain is too shot for creative debugging) [18:51] fuller error: http://paste.ubuntu.com/536005/ [18:52] jml: brief call? [18:52] lifeless: very brief, sure. === benji-lunch is now known as benji [19:53] lifeless: it *looks* like there was another test in test_workermonitor that was dirtying the db via the librarian without declaring as such... [19:53] lifeless: I have no idea why I could only reproduce the error with test_builder in the mix [19:53] weird [19:53] jml: I'm glad you found it [19:54] it would be relatively straightforward to put a thing in the librarian layer that checked to see if rows were added ... [19:54] lifeless: I'm not going to dig any deeper, fwiw. [19:55] jml: fair enough [19:55] * jml ec2 tests & goes away to eat, drink (medicine) and be lazy [19:55] lifeless: thanks for the help. [19:58] anytime === salgado is now known as salgado-afk [20:29] morning === matsubara is now known as matsubara-afk [21:15] can someone explain why some methods in launchpad take a quantity argument instead of using slices? [21:21] cr3: Do you have an example? [21:22] It's possibly webservice-related, but it's hard to say. [21:23] wgrant: grep -r quantity= returns examples such as: IBranchView.latest_revisions, IProductSet.latest [21:24] wgrant: there are some comments relating to the webservice interface, which decorates the functions using call_with(quantity=None), whereas the default is a hard coded value like 10 for example. [21:24] wgrant: I would've expected the other way around though, ie limiting the webservice interface to some hard coded value and allowing internal calls to slice and dice [21:29] hi all [21:31] cr3: slices require lazy objects, these are method calls [21:32] as soon as you're not in ResultSet territory, you'll find we don't slice [21:32] and where we do, its generally a bug. [21:32] lifeless: some calls like Branch.latest_revisions return a Storm ResultSet which could be sliced and diced though, haven't looked at many others, but it might just be remnants from the good ol' days [21:32] lifeless: But the methods should be returning (Decorated)ResultSets :/ [21:32] wgrant: not as of yesterday. [21:33] lifeless: How far through the design are you? [21:33] wgrant: do you guys normally use the launchpad DecoratedResultSet rather than the default Storm one? [21:33] cr3: Is there a Storm DRS? [21:33] cr3: when we eager load, yes. [21:33] wgrant: no, the default ResultSet I meant [21:33] cr3: we eager load in many, but not enough places. [21:34] cr3: We use the standard ResultSet in most places. [21:35] lifeless: wait, you use DecoratedResultSet or quantity when eager loading? (I need to finish reading that related thread on canonical-tech) [21:35] DRS is for eager loading. [21:36] DRS for eager loading. quantity when specifying up front what we need (which is a different, not necessarily better or worse) pattern. [21:36] quantity is for broken methods that don't return a ResultSet or derivative. [21:36] s/broken/other/ [21:36] Until yesterday they were broken, now they might not be :P [21:36] wgrant: they weren't broken before [21:36] they are symptoms of our lack [21:36] resultsets do not cache [21:37] wgrant: was there a fix in DRS yesterday? is it in trunk so that I can pull it and see? [21:37] so they aren't suitable to pass into methods [21:37] cr3: No, lifeless is just changing everything. [21:37] EVERYTHING> [21:37] wgrant: just when I thought I understood a thing or two about launchpad :) [21:38] we have several overlapping problems [21:38] firstly, our can-read assertions are scattered all over [21:39] we should have them in one layer so that we don't *return from queries* data the user [whose behalf we are acting on] shouldn't see. [21:39] secondly, templates and logic like canonical_url is per-object [21:39] but for efficiency and to map well to our data storage layer we need to work per-set [21:40] lifeless: is the API going to be similar to the unannounced new webservice model? [21:40] we should have a layer that enforces set based access to the data store and makes it easy to put set/group based idioms higher in the codebase [21:40] wgrant: they have strong parallels and I know that the webservice folk have added to their thoughts the ideas I've put up about the group based idioms in LP's internal code. [21:41] wgrant: Gary expressed a desire to have them be harmonious and perhaps even trivially layerable. [21:41] wgrant: I think they have different constraints, but they should be similar in style even if not in exact detail - initially. [21:41] wgrant: the webservice stuff is on dev.l.n if you're interested [21:42] wgrant, also fwiw, the new webservice model won't be announced but proposed [21:42] lifeless: Yeah, I've been reading it. [21:42] It looks really great. [21:42] thirdly, the failure mode we have today is death-by-queries [21:42] we should have a mechanism such that when we think we've prepped for a page, we turn off data access [21:43] gary_poster: You're silently working out how to make the webservice fantastic. I think that deserves announcement. [21:44] cr3: ^ [21:44] :-) thanks wgrant. (where "you" includes a bunch of people.) And I hear you. I'll do it. [21:44] cr3: so I've proposed an explicit persistence layer [21:44] lifeless: death-by-queries reminds me of an idea I had recently about unit testing where it might be desirable to decorate or provide some assert methods to make sure a certain number of queries are being performed, ie to detect when fetching a list of items results in a query per item for example [21:45] cr3: it will only expose things the current user can see, will return objects that have no backend-connection so cannot trigger bad behaviour by mistake [21:45] cr3: search for HasQueryCount in the lp code base [21:45] lifeless: awesome! will do [21:46] cr3: the persistence layer will also be a clear boundary, which we can switch out for tests that are not testing the data storage story [21:46] gary_poster: My main remaining gripe is the lack of transactions, but that's probably impossible to do, and it's mostly mitigated by the collection fixes. [21:46] wgrant: batch PATCH gives you near-enough IMO [21:46] Exactly. [21:46] wgrant, not impossible--leonardr talks about how to do it in his REST book--but not terribly attractive IMO [21:46] wgrant: we're -not- going to expose sql transaction objects on the net. [21:46] yay [21:47] lifeless: Batch PATCH and the atomic collection retrieval does almost everything needed, I think. [21:47] yeah [21:47] lifeless: that reminds me, I didn't notice any unit tests for the database functions themselves, ie no database/tests directory for example. I decided to have tests specifically for my database code in my latest project [21:47] cr3: lib/canonical/database/ftests/ ? [21:48] cr3: How are they different from the model tests, given that the layers are one? [21:48] there are some big questions about where 'is allowed to be related' logic should go in my new layer [21:49] but I think things like 'when a bug changes notify X Y and Z' very clearly do not belong in it. [21:49] lifeless: well, I would expect tests to exercise each of the functions defined in trusted.sql for example, which it doesn't look like lib/canonical/database/ftests/ is for [21:49] equally 'who will be notified' isn't a storage problem, its built on it. [21:50] cr3: oh. mmm, thats kindof bootstrap code. [21:50] sure, you could test it. It might be nice to do so. [21:51] wgrant: not necessarily true, some database integrity should be caught at the database layer rather than the application layer, so exercising the model layer does not validate database integrity per se [21:53] wgrant: I also find it easier when defining a function in sql to have tests specifically exercise that function rather than going through the model layer [21:54] cr3: this asumes you have different layers. [21:54] cr3: triggers are evil though and we should get rid of ours. They are workarounds for poor ORM support. [21:54] perhaps. We can keep them now we have them, but we've triggered storm bugs by using them, and will in the future) [21:56] lifeless: do you mean problems refreshing the cache in storm when triggers update objects without the orm knowing? [21:56] I mean bugs. [21:56] like updating all columns rather than changed objects [21:56] changed columns that is. [21:57] lifeless: oh, has that been fixed yet? I've experience that problem myself and it's been a pain :( [21:57] in 0.17 [21:57] it broke other stuff, which is fixed in 0.18 [21:57] lifeless: thanks for the info, will have a look [21:58] you may find https://dev.launchpad.net/LEP/PersistenceLayer my high level problem statement interesting [21:59] lifeless: you're ambitious: # None of our tests of Launchpad functionality require a real database to execute. [22:00] we can't tell if its complete if we don't do that [22:01] lifeless: I would've gone for some ratio or some rough best effort figure, I would be afraid of the 80-20 rule where you'll spend 80 percent of your time finishing off the 20 percent remaining tests hitting the database [22:02] lifeless: whereas I'd consider it a tremendous success if 80 percent of tests didn't hit the database, testing would be blazingly fast (modulo windmill and doctests) [22:04] cr3: if we're going to do it, we need to -do it- [22:04] cr3: the last thing we need is a half (or 80%) deployed thing. [22:04] that would leave the remaining 20% still existing : having 20% of our pages be terribly slow is not acceptable. [22:06] lifeless: another way of looking at it is 20% of each page being slow, which still means 5 times quicker than before. but I can appreciate your point of view though, and I suspect you already have a design in mind that actually makes it feasible. in lifeless we trust :) [22:09] What do Landscape and U1 do? [22:10] It seems that we are trying to solve all sorts of problems that they probably solved years ago. [22:10] landscape makes heavy use of 'collections' idioms. [22:10] In my discussions with jkakar he has expressed interest in what I've sketched out, and said they suffer the same things we do. [22:10] so, for them, they work harder at performance. Its not intrinsically easier for them. [22:11] U1, AIUI, are built on django [22:11] And U1 uses Couch and all sorts of other evil, I suppose. [22:11] which has a less strictly-object flavour and so is less driven to the pathology Launchpad is [22:11] but they've had plenty of requests taking thousands of queries. [22:12] I'm not aware of any Canonical web property that /has/ this solved pervasively, nor solved at the scale (of data model complexity) that Launchpad needs. [22:12] ... I've been asking :) [22:13] Has anyone else? [22:13] hibernates HQL, sqlalchemy's QL - these are great things to look at. [22:14] Right. [22:14] functional approaches, and things like map-reduce are very useful inspiration too [23:52] how/when is the sampledata updated? [23:53] fridayish [23:53] iow why has my merge from devel got large sampledata changes compared to devel? [23:53] oh [23:53] in https://code.launchpad.net/~mbp/launchpad/dkim/+merge/41689 [23:53] when commits are made [23:53] did you do 'make sampledata' [23:59] large diffs tend to come from (sometimes tiny) version differences in postgres