[00:02] spm: hi [00:02] spm: I'm sure you have urgents, but I have some today [00:02] spm: firstly, gmb needs to know some stuff about the release. [00:02] gmb: ^ [00:02] lifeless: heya [00:02] * gmb is paying attention, may mumble further questions [00:03] Have run out of Monday. [00:03] spm: specifically; how long did the staging db migration take (for downtime estimation); secondly, are the lucid upgrades tom is doing sequential or concurrent with the schema migration [00:03] also we seem to have a dead-but-accepting launchpad edge server - see #launchpad backlog with nightrose [00:03] and can I have a pony. please. [00:04] sorry. no ponies. we do have shiny ponies tho? does that count? [00:04] shiny will do [00:05] lifeless: spm: 25 mins for the db patches to apply anyway [00:05] gmb: ^ [00:05] spm: Ta. [00:05] tho. um. migration? I perhaps misunderstood the question? [00:06] did you mean the 8.3 -> 8.4 migration? or just the regular do the DB juju? cause the ans provided is for the latter; not former. [00:06] staging DB has not been migrated. [00:06] regular DB juju [00:06] coolio [00:07] oh and speaking of staging [00:08] I'd like to get a profile pretty please [00:13] * thumper is finding :(( [00:15] * gmb exits stage right in search of sleep. [00:15] 'Night folks. [00:15] shiny shiny - http://wiki.postgresql.org/wiki/Postgres-XC [00:23] "Failover/High Availability: No" [00:24] yeah [00:24] but its the first multi master thing in the pg world that I've heard of [00:24] so things are starting to move [00:24] ah no [00:24] I was all about to go \o/ [00:24] but no failover is somewhat sucky [00:25] oh god no, we can't /move/ to this today [00:25] its simply 'if you need more writes than one machine can give you. I think it would be possibly to run slony on top if it if we wanted (arrghh :P) [00:38] jelmer: ping? [00:38] thumper, pong [00:38] jelmer: why am I not too surprised that you are up? [00:38] jelmer: http://staging.launchpadlibrarian.net/54835281/chicken-chicken-git-mirror.log [00:38] jelmer: looking at some staging import logs [00:38] jelmer: Exception AttributeError: "'NoneType' object has no attribute 'close'" in ignored [00:39] jelmer: any ideas? [00:39] jelmer: seems on bzr-git and bzr-svn imports [00:39] thumper: as far as I can tell they are to do with paramiko [00:39] jelmer: ah [00:39] thumper: I've seen them from e.g. update-sourcecode as well [00:39] thumper: but I'm not clear as to what the problem is exactly [00:39] ok [00:40] jelmer: I've got a question about bug 599397 [00:41] <_mup_> Bug #599397: code imports break when tdb is installed

[00:41] jelmer: should we be running the import machines with tdb? [00:42] thumper: Yes-ish [00:42] ? [00:42] meaning? [00:42] thumper: There are some advantages to using tdb - it allows concurrent write access for example, so multiple clones from the same svn repository at the same. In my experience it's also faster than SQLite. [00:42] I feel a but coming... [00:43] there is.. :-) [00:43] maxb has reported he's seen worse performance from tdb than from sqlite [00:44] there is also the fact that all existing caches are already in sqlite, moving to tdb would mean regenerating them or keeping the existing ones in sqlite and the new ones in tdb [00:46] concurrent access isn't really an issue except when people register svn imports for different branches in the same svn repository at the same time. after the initial import the incremental imports are quite quick so they hardly ever overlap, and even if they do they will be retried. [00:46] jelmer: suggestion on an import to try to verify bug 617078? [00:46] <_mup_> Bug #617078: RedirectRequest errors trying to access .git/objects [00:46] any github http import i think? [00:47] thumper: xwax, lp:~dholbach/xwax/trunk [00:47] ok [00:47] mwhudson: I think most non-github failing git imports over http are failing because of that particular bug [00:48] ah ok [00:48] mwhudson: github still won't work with the new rollout, they don't support "dumb" access to git repositories over http [00:48] mwhudson: s/github/github over http/ [00:54] spm: can we do a profile on stagin gplease? [00:55] lifeless: sure; sorry was just stopping the librarian from running outa space and doing a major update on how docco on how to achieve that happy state :-) [00:55] yeah [00:55] I'd like to do the multi-disk-support thing properly. [00:55] if you have design constraints/ideas add em to the bug. [00:55] https://wiki.canonical.com/InformationInfrastructure/OSA/LPHowTo/IncrementTopLevelLibrarianDir for the bored [00:56] I wonder if we could just move to an S3 impl [00:56] written for the "AAAAA! Librarian! No Space! FIX NOW" I don't wanna have to work this out from first principles in a hurry losa [00:57] obvious jokes aside; it is something we've rambled about internally. I'm unsure if backups and/or IO would be sufficient tho. [00:57] spm: challenges challenges [00:58] anyhow, lets get our current system bug free [00:58] then we can transform what it does into a requirements spec for evaluating replacements. [00:59] spm: so profiling ? yes/no ? [00:59] getting there.... :-) [01:00] lifeless: no [01:00] lifeless: spm is doing something for me first :) [01:00] lifeless: I'll take 3 minutes [01:00] ok [01:06] there, thats 6 :P [01:07] lifeless: I'm all done [01:09] spm: oh hhhaaaaaiiiii [01:09] heh, you have not been forgotten; just another crit state I had to deal with... [01:10] I know [01:10] only 10 reds atm ; 2 since dealt with. progress... :-/ [01:10] >< [01:10] any I can help with ? [01:11] kill the c o u c h d b developers? [01:11] I have a small moral objection there [01:11] oh. did that come out loud. oops. [01:13] :) [01:20] lifeless: go for it! [01:22] grabbing the new docs [01:22] the new one only permits it rather than doing it on every request (unless you set another key as well) [01:23] wgrant, I disagree. They provide a good clean system, but they also provide a system that can be safely compromised. [01:25] 30782 0 16.7424 1.1619 storm.store:669(_load_object) [01:25] that might be related to the assignments perf problem [01:26] ✁☹ [01:26] that's how I'm feeling right now [01:30] where are enums again? [01:31] spm: can you kick the profiling rsync? [01:32] lifeless: which enums? [01:32] lazr.enum [01:33] or for a particular lp part? [01:33] yarp [01:33] thumper: particular part [01:33] registry [01:34] try lp.registry.enum [01:34] I know they were moving them [01:34] they may not have moved all yet though [01:34] thumper: and th eold place? [01:34] lifeless: interface imports [01:34] lifeless: done [01:36] spm: thanks I'm done [01:36] turn off allow profiling please [01:37] so, storm's load-objects is 80% of Distribution:+assignments [01:37] * lifeless again considers removing storms object cache. [01:37] rockstar: The VMs protect from compromise. Not the chroots. [01:38] damn [01:38] something is wrong with my groovey new code [01:38] * thumper lunches and mulls it over [01:39] lifeless: reverted [01:41] * spm goes to correct a ZOMG-ULTRA_VIOLENT Critical alert: -ENOCOFFEE [01:50] spm: zomg the librarian thing scares me shitless [01:51] lifeless: ahh. welcome to the club. [01:51] how often do you do this? [03:02] can I interest one of you folks to do a session during https://wiki.ubuntu.com/UbuntuAppDeveloperWeek [03:03] what sort of thing did you have in mind? [03:03] its about appp development, so something about how lp fits in, recipies, dailies, etc [03:04] how LP can be awesome to an app developer [03:05] recipes are a significant differentiator over e.g. sf [03:13] jamesh: hi [03:13] lifeless: hi [03:13] https://bugs.edge.launchpad.net/blueprint/+bug/618367 [03:13] <_mup_> Bug #618367: Distribution:+assignments is taking a long time maybe 20% of the time [03:14] jamesh: storm seems to spend about 1/2ms per object in load_objects [03:14] the last three comments are relevant [03:15] lifeless: sf? [03:15] source forge and others [03:16] yes, so that's why asked if somone could do a session about how LP could help :) [03:16] I htink its a good idea [03:16] ask on the lp-dev mailing list though [03:16] you'll capture more peoples attention [03:17] ok, I will :) [03:17] err, which team is that list attached to? [03:18] nm, got it [03:25] jamesh: any thoughts on that bug ? [03:25] jamesh: I'm going to reduce the columns and not return full objects for the eager loading [03:27] lifeless: I guess one obvious question is how many distinct objects are you getting from that query? [03:27] 3000 distinct -rows- [03:27] many repeated person/account/email triples [03:27] While it is 3000 rows x 10 tables, I assume it isn't 30K different objects, right? [03:27] thats right [03:28] if storm could skip deserialising when the primary key for a thing was already retrieved in the same query that would be useful [03:34] from the third last comment, is it set_values or _set_values you saw in the profile? [03:36] _set_values, attached the kcachegrind to the bug for you [03:37] to do seperate person + specs queries without code contortions at higher layers we need an even smarter resultset [03:37] so, from how _load_objec() is structured, the _set_values() calls occur when we're updating the values in an existing Python wrapper [03:37] I think investing in a sqlalchemy like mapping layer is probably well worth it [03:38] the get_object_info() calls are when _load_object() creates a new wrapper [03:38] jamesh: so having a way to skip that if its from the same query would be uesful ? [03:38] the _load_object() calls with neither are the cases where the row was all NULLs (I assume you're doing a left join here) [03:39] yes [03:39] the one I was pulling my hair out over the other week [03:45] I don't think I can give you a quick answer to the problem: we don't really keep track of whether an ObjectInfo has been fully filled out, so we'd need to track that to decide whether to attempt to fill it in [03:45] maintaining such a flag would need to track when the user assigns lazy values to properties too [03:46] hmmm [03:53] so the challenge is that we have apis which return resultsets, which shouldn't cost a lot to e.g. call and do count() [03:53] but /have/ to eager load this related data [03:54] its quite possible the db would be faster doing two queries as well (though in theory not; only if its misphrased) [03:54] I'd like to get rid of the use of count() in tales [03:55] creating a ResultSet and calling its count() method won't hit _load_object() [03:55] you should only hit that when calling an API that actually returns objects [03:55] let me rephrase [03:55] if we restructure this into two queries [03:55] there will still only be one attribute (specifications) that the browser/view/template iterate [03:56] first grab all specs, then grab all referenced (person, account, email) triples? [03:56] so it needs to eager load the people, for the rows being returned, before iteration, and only when iteration is hjappening [03:57] jamesh: right,thats what one would want to happen. [03:59] I doubt that we'll save much time doing that on the DB end: its 500ms to get the specs alone with no people [03:59] probably due to freakishly wide rows or something [04:54] mwhudson: I need to talk through this problem if you have a minute [04:55] thumper: ok, but a minute would be about right [04:56] mwhudson: you don't have long? [04:56] thumper: i just seem to have ever mounting distractions [04:56] :( [05:24] garh [05:24] what is [05:24] +class Array(NamedFunc): [05:24] + """Implements the postgres "array" function in Storm.""" [05:24] + name = 'array' [05:24] + [05:24] doing in product.py?! [05:25] lifeless: because that is where it is being used? [05:27] also lib/lp/services/database/prejoin.py should be deleted. [05:27] thumper: it should in storm itself or lp.services.database.storm [05:27] so other folk can find it [05:27] lifeless: agreed [05:36] * thumper just found a bug... [05:50] spm: hi [05:50] yo [05:50] spm: I have a small favour to ask [05:50] LIES! [05:51] I'd like you to pull a patch onto the staging appserver so I can see if it actually helps [05:51] sure. you have a patch file? [05:51] its one commit - rev 11516 in lp:~lifeless/launchpad/bug-618367 [05:51] I'll make a patch, one sec [05:52] or you can just do a merge -c 11516 lp:~lifeless/launchpad/bug-618367 [05:52] all it changes is specifications (hah what could go wrong) and it passes the tests I wrote for this the other day [05:53] "hah what could go wrong".... oh where do I start.... ;-) [05:54] http://paste.ubuntu.com/489589/ [05:54] but the merge may be easier to do; you tell me :) [05:55] generally a patch is tbh [05:55] the pastebin should do then ? [05:56] yup. ta [05:56] lp is a little easier these days being open; but any locked branches - eg configs - are a pita. so patches are generally better. [05:57] lifeless: patched clean; restarting... [05:59] ok, its not enough [05:59] can you please enable profiling again [05:59] sure [05:59] and I'll get a profile and then be out of your hair for a while [06:00] * spm restrains mightily against an outpouring of cynical laughter [06:02] lifeless: should be abck again [06:03] kick the profile rsync ? [06:05] oh yeah. sorry - should have anticipated that one. [06:09] spm: ok, I have it [06:09] profiling off, patch off, gtg. [06:09] thanks [06:10] spm: actually, hang a sec, i fyou haven't [06:10] I haven't. [06:11] trying to see if its DB IO [06:14] ok, there is clearly 10 seconds of storm parsing time in there [06:34] whats the status of 'mentoring' [06:34] should we be deleting stuff for it ? [06:36] i think so [06:39] k [06:39] I've got the main fubar fixed [06:41] lifeless: can I reset that btw? [06:46] spm: sorry, yes please [06:46] np [06:55] lifeless: So I can't maintain the data structures for the page performance report in RAM now that devpad has less RAM [06:55] lifeless: So I need to spool parsed requests to disk [06:57] lifeless: sqlite3 or spooling to an xdr file (xdrlib) seem suitable [06:59] lifeless: xdrlib will be fastest to write, but generating 500 rows in the report will involve scanning it 500 times. sqlite3 is much slower to write as it needs to update indexes, but the whole file does not need to be scanned to generate a row as data can be pulled out with an index [07:20] stub: we'll be getting more ram agian [07:20] lifeless: So don't bother? [07:20] it seems to be generating OK at the moment [07:20] Whatever I do, it will make things slower. [07:20] so yeah, I wouldn't bother. [07:21] k. I'll dump the kanban card and see if IS keep wining. [07:21] whining even. [07:21] rt 41200 [07:22] charlie mentions a different ticket to swap th ebits back [07:24] the monthlies may need a disk spool regardless [07:24] but dailies should be fine I think - no? [07:27] stub: we can whine no matter what you do. [07:27] rotfl [07:27] timing is everything [07:27] reckon. /me sulks [07:31] hrm, wgrant isn't around today? [07:31] nigelb: Am I not? [07:31] I'm reasonably busy. [07:31] But I'm around. [07:31] Although I'm about to be unbusy. [07:32] wgrant: I was asking around eariler if somone wanted to talk about LP awesomeness on app developer week... interested? [07:32] * nigelb hugs spm [07:32] someday you'll get the timing right ;) [07:33] :-) [07:33] nigelb: Not really, sorry. [07:33] wgrant: you bought a gun and found your project team? [07:33] lifeless: Sadly not. [07:33] lifeless: wait, what? [07:34] wgrant: you just did all the work yourself? [07:34] wgrant: np, I'll look for more victims..err candidates [07:34] mwhudson: It's looking that way, although we're only half-way through. [07:45] nigelb: ? [07:46] lifeless: talk of guns and project teams.. nm, I figured it out [07:51] wgrant: did you get a chance to look at the +history thing? [07:51] ok, I'm off for a while, I'll be back later to nab jkakar / therve etc and talk about big picture refactorings of storm [08:24] jamesh: ping [08:26] lifeless: pong [08:42] good morning [08:54] jamesh: wanting to discuss storm perf with you more [08:54] jamesh: looks like 66% of the time was in 10% of the objects [09:12] Hello [09:31] bigjools: That fix hasn't landed yet, has it? [09:31] wgrant: I'll find out shortly :) [09:33] statik: if you want to catch up nowish I could do a relatively brief skype call [09:35] lifeless: i'm toggling between an in-person meeting on OpenERP and a security bug, i'll probably have to wait until next week. sorry man [09:35] thats ok [09:35] just bein gopportunistic [09:35] I have to wait around a bit to stalk jkakar about storm performance anyhow [11:31] lifeless: still around? [11:34] Is there a staging codehosting environment? [11:35] allenap: yes [11:35] allenap: but it's a bit odd because it has hardly any other underlying branches [11:35] mwhudson: It's not refreshed at the same time as the database? [11:39] allenap: all 600 gigs of it? [11:43] mwhudson: Yeah :) Okay, guess not. If I'm interested in manipulating some of those branches, I suppose I can push them first, and things will work? [11:44] yeah [11:44] the bzrtools branchess are synced [11:44] iirc [11:54] mwhudson: Excellent, thank you. [12:00] Morning, all. [12:13] bigjools: not really; whats up? [12:13] lifeless: it can wait. I want to talk about how zopless scripts interact with the webapp with regard to table locking and transactions [12:13] 'badly' :P [12:14] I figured [12:14] let me re-phrase [12:14] I want to make it work :) [12:14] I'm growing more and more of the opinion that we need the same protection the webapp has, in scripts [12:14] which protection? [12:14] there are a few ways to do that [12:14] timeouts [12:14] enforced [12:14] hmmm [12:15] in my case I want to add a build cancellation feature, but the webapp would need to change build states potentially at the same time as the buildd-manager [12:16] how so, and how is it different to two people changing a bug status at the same time ? [12:17] because the b-m is a script [12:17] and that is where my uncertainty lies [12:18] there's a race condition where the build is NEEDSBUILD and the webapp wants to cancel it and the b-m wants to dispatch it [12:18] I need to make sure that they behave. In the old days you'd have mutexes or something. [12:19] ok [12:20] I can avoid it if I do something like make the webapp use another status that's a "request to cancel" and then make the b-m handle it [12:20] so you can model this with a table [12:20] but that's not ideal from a user PoV [12:22] you have NEEDSBUILD-> [12:22] and you have two mutators [12:22] webapp chooses cancelled [12:22] bm chooses dispatched [12:24] if the webapp ins the bm commit will fail [12:25] if the bm wins, the webapp request will be retried [12:25] (and won't see it as NEEDSBUILD the next time around) [12:25] if the bm commit fails, it shouldn't dispatch [12:25] what makes the commit fail? [12:25] conflict [12:25] what detects that? [12:26] pg [12:26] open up two pgsql launchpad_dev sessions [12:26] does it need a special txn mode? [12:26] in both do 'begin;' [12:26] The webapp uses READ COMMITTED. [12:26] So watch out. [12:26] then do an update in one like the bm will do, and in the other like the webapp - just the status field they both write to is enough [12:26] Really really watch out. [12:26] exactly my point wgrant :) [12:27] the b-m also uses the @write_transaction decorator which retries IIRC? [12:27] write locks are still txn wide in read committed [12:28] bigjools: It does retry, yes. [12:28] Does python have a sh -n equivlant? [12:28] * StevenK is trying to remember [12:29] I think it will work; you will definitely ant to verify that it does - even if only by a manual test script [12:29] I'm going to run the b-m and the webapp in 2 debug sessions and see what happens [12:30] I don't trust it simply because I think this sort of thing has gone wrong before [12:30] plenty of ways it can [12:30] * lifeless would be happiest if b-m didn't talk to the db [12:30] it would make this trivial [12:30] you mean talk through a webapp using the api don't you [12:30] * bigjools shudders [12:31] or the internal xml server [12:31] * bigjools shudders again [12:31] xmlsucks.org [12:31] Oh dear, this argument again? [12:31] :) [12:31] I'd be very happy to be persuaded that it would all work fine [12:31] oh xml is terrible, no argument. [12:32] but I'm unconvinced since it's pushing the problem to a different place [12:32] the LP API already has semantics for in flight collisions [12:32] it gives a patch error [12:32] FWIW [12:33] what about the webapp? [12:33] well, the UI I mean [12:33] lifeless: I think we need to make the API *much* quicker before we can argue about b-m using it with any credence. [12:33] when it retries -AIUI, I haven't checked this code myself- it will start over from scratch. [12:34] StevenK: yes, would you like a list of soyuz API calls that are shocking? Available on demand. [12:34] StevenK: I think that's already understood :) [12:34] lifeless: Not assigning blame [12:34] StevenK: the API is not slow because its the API - in general; its slow because LP is slow and the API is a thin skin over the LP model objects. [12:34] StevenK: its -trivial- to make a set of dedicated extremely fast API calls for services like the codehosting machines have [12:35] part of the problem with the API is that it serializes all of the object - even bits I don't care about [12:35] anyhow, yes thats a definite constraint [12:35] and often the developers don't remember this when adding new "properties" [12:35] yes [12:36] * StevenK cries since making the wadl is dying with SyntaxError [12:36] I need to collate some of the lessons in this area for general discussion [12:36] but back to the question [12:36] the webapp starts over [12:36] lifeless: anyway, I shall try some debug sessions to see how the webapp interacts with a b-m under conflict conditions [12:37] it will read the *new* state, and as long as the form handler is sensible (that is that a 'cancel' returns a 'cannot cancel, its ' when the thing wasn't in NEEDSBUILD, then the web ui will be fine. [12:37] And the SyntaxError is *so* helpful :-( [12:37] lifeless: the problem is that the transaction boundaries in the b-m are shit [12:38] I need to review it and figure out WTF it has commit() inside @write_transaction [12:38] bigjools: I can believe that; thats part of the stuff making me speculate about APIs [12:38]