=== beuno_ is now known as beuno
StevenKwallyworld_: I've had breakfast, but I think I need at least one cup of tea before our chat.00:15
wallyworld_StevenK: np00:16
StevenKAlso got distracted by finishing my double-word-death branch00:16
* StevenK tosses it at ec200:16
lifelesswgrant: ?00:18
wgrantlifeless: You said "heh, doit." but didn't actually vote.00:19
lifelesstwill be clear to onlookers :>00:20
wgrantAnd the subsequent 6 lines were about that tag subscription bug that you commented on.00:20
lifelessah yes00:20
lifelesswhich is beautiful win00:20
StevenKwallyworld: Sorry, now I'm waiting for the jerk with the leaf blower to move00:25
wallyworldStevenK: what? can't hear you over the noise00:25
StevenKOr I get annoyed enough to go downstairs and ... re-position the nozzle00:26
wgrantlifeless: Can we migrate to SQLAlchemy now please?00:26
StevenKAdd it sourcedeps. I dare you.00:26
lifelesswgrant: raise it on the list00:27
StevenKlifeless: You have no objections?00:27
* StevenK deals lifeless a penalty card for 'Lying'00:27
lifelessStevenK: oh?00:28
lifelessStevenK: deal yourself one00:28
lifelessStevenK: then ask Francis what I've said in the past.00:28
StevenKlifeless: You have strong opinions on everything we use, so I find it hard to believe you don't have one about this.00:28
lifelessI never claimed no opinion00:28
rick_hwgrant: is my new hero for that comment00:28
StevenKif( vs if (00:39
rick_hpep8 says if (00:40
StevenKwallyworld: http://pastebin.ubuntu.com/852074/00:42
* StevenK stabs Firefox01:01
=== 13WAAIDLL is now known as wallyworld
=== wallyworld is now known as Guest76072
abentleywgrant: it looks like bulk.create with a 0-length values will perform an execute and get a ProgrammingError.  What do you think about changing it to handle 0-length values by returning?02:04
abentleywgrant: Also, is it efficient to load when the values aren't needed?02:05
wgrantabentley: Much like the rest of Storm's SQL compilation stuff it is a bit fragile, yeah.02:06
wgrantAnd loading should indeed probably be made optional.02:06
wgrantSo, yeah, 0-length values should probably just be a no-op, and loading should be disablable.02:06
abentleywgrant: Cool.02:07
abentleywgrant: The 0-length values is the only reason I kept insert_many in my latest branch.02:07
wgrantabentley: I'm intrigued as to how your experiment was so slow.02:07
wgrantPerhaps sqlvalues is awful.02:07
wgrantabentley: How fast is the original code, btw?02:08
wgrantWithout any bulk optimisations.02:08
abentleywgrant: I'm curious too, but I've got lots of things piling up at the moment.02:08
abentleywgrant: I took over from gmb, so I didn't try it.  I know it exceeded the branch scanner job timeout.02:09
mwhudsonwhich bit of the scanner is timing out?02:09
lifelessthe wrapper02:09
wgrantBoth BranchRevision and Revision insertion have been problematic.02:09
wgrantAnd there was a libreoffice non-DB timeout yesterday.02:10
wgrantI didn't investigate.02:10
abentleymwhudson: The db insertions of Revisions and BranchRevisions.02:10
mwhudsoni worked on BranchRevision pretty hard at one point02:10
mwhudsonas you have probably noticed02:10
wgrantIIRC it uses manual string SQL now?02:10
wgrantLike the rest of LP's existing bulk inserts.02:10
wgrantWhich I'm going to port to the new shiny tomorrow, probably.02:10
mwhudsonthat'll be good02:10
abentleywgrant: Are you fixing the same bug as me?02:10
mwhudson(even if it doesn't end up being faster)02:10
wgrantabentley: No.02:10
wgrantabentley: I'm working on disclosure stuff.02:10
wgrantabentley: jelmer is working on the bzr side of scanner timeouts, though.02:11
abentleywgrant: I've ported it over to insert_many already.02:11
wgrantabentley: I informed him of your potential conflicts.02:11
wgrantabentley: Yeah, but there are other places that do manual bulk inserts.02:11
wgrantBPPH creation and a few others.02:11
abentleywgrant: Oh, I misread you.02:11
wgrantI'm not going to touch your stuff :)02:11
abentleywgrant: Well, if you insist on doing it better than  me, I'll grumble, but not too much :-)02:12
wgrantabentley: You've wrapped create() rather than fixed it directly?02:13
wgrantThe initial version will land in about 90 minutes, so I'll land a quick followup with the changes you suggested unless you already have one.02:13
abentleywgrant: No, I hadn't seen create.  I was using "Store.execute(Insert( ..."02:13
wgrantI'll make those two changes now, then.02:14
wgrantHopefully this will make people avoid bulk creation a bit less.02:15
wgrant'cause there are a lot of places that could use it.02:15
wgrantabentley: I apologise for colliding with you. I didn't realise you were working on this until I was proposing my Storm branch and saw yours.02:16
abentleywgrant: No worries.  Strange coincidence that we both tried to fix this longstanding gap in the same month.02:17
* wgrant benchmarks the old one too, just for curiosity's sake.02:21
abentleywgrant, jelmer: In last week's profiles of lp:s~irar/gcc-linaro/slp-for-any-nops-4.6/, only 13% of branch scanner time was spent in getBazaarRevisions, so I wasn't too concerned about it.02:24
abentleywgrant, jelmer: Though wgrant's bulk-insert will make the db side faster.02:25
wgrantwgrant@lucid-test-lp:~/launchpad/lp-branches/devel$ ls -lh /var/tmp/lperr02:25
wgranttotal 191M02:25
wgrantdrwxr-xr-x 2 wgrant wgrant 189M 2012-02-22 13:24 2011-12-0502:25
wgrantThat can't be good...02:25
StevenKA 189M dentry? Niiiiiiiiice02:27
wgrantI think that's the day I reproduced the poppy rabbit explosion.02:27
wgrantWhich could mean there are millions of files in there.02:27
StevenKls -1 /var/tmp/lperr/2011-12-05 | wc -l ? :-)02:28
wgrantI unfortunately already deleted part of it, but I'm already doing that.02:28
wgrantBeen going for 3 minutes...02:28
wgrantIt's probably statting all the children :*(02:28
StevenKIt will, yes02:28
wgrantMight write something to just readdir instead.02:29
wgrantOh no02:29
wgrantIt's still calling getdents02:29
wgrant512 at a time02:29
wgrantls is only using 3% of my RAM so far, though.02:30
wgrantAt least 4 statements per Revision insert in devel.02:40
wgrantAnd often 702:41
wgrantAnd oh look it timed out.02:41
wgrantLess than half way through.02:42
* wgrant starts it without a timeout and goes away for 15 minutes.02:46
wgrantIt doesn't really seem to be inserting them in any kind of order.02:51
wgrantabentley: Just tested on lp:launchpad. 9:00 without bulk inserts, 1:45 with mine.03:06
wgrantProbably doesn't improve the normal case much, but the initial scan of a large new history is 5x faster even with no DB latency.03:07
wgrantYeah, doesn't help the 0:47 new branch with existing Revisions case at all.03:13
wgrantmwhudson: Did you ever look at sharing history?03:15
wgranteg. bisect to find the latest shared history and then delegate to that.03:17
lifelesswgrant: it was examined, based on the work jam had done in history-db03:29
lifelesslol, pybars has had 65 downloads03:30
lifelessyou think?03:30
* wgrant finds out.03:31
mwhudsonwgrant: for branch revision?03:34
wgrantmwhudson: Yeah03:34
wgrantIt would presumably reduce the table size by something like 99%.03:34
wgrantlifeless: Oh, bah, hosted on PyPI rather than LP?03:35
wgrantNo referers for me :(03:35
lifelesspypi -> centre of the python universe03:36
wgrantYeah, but often our downloads are hosted on LP.03:36
StevenKwgrant, lifeless: Can I Disapprove https://code.launchpad.net/~linaro-infrastructure/launchpad/workitems-migration-script/+merge/93883 ?03:38
StevenKgarbo job or API script, damn it03:38
wgrantAs I said a couple of days agoo, I think it should be an API script. But it needs discussion.03:39
lifelessI recommended garbo to salgado a week or two back03:39
StevenKAll of three of us are in agreement about "Oh god, not that>"03:40
lifelessI think its reasonable to say that you want to discuss it03:40
lifelessand yes, I agree, API or garbo makes much more sense03:40
StevenKlifeless: "I'm going to disapprove this. This requires discussion, and certainly doesn't require a one-shot script -- the usual modus operandi for migrations like this is either a garbo job or an API script."03:42
StevenKlifeless: Sound fine?03:46
mwhudsonwgrant: i'm not sure how that would help things like mapping a revision to a branch03:46
lifelessStevenK: personally, I would needs-info rather than disapprove, but if you're reviewing, do what you prefer03:46
mwhudsonyou can do cleverer things that BR of course, but you end up doing something like historydb i think03:46
lifelessStevenK: I'll note that its a tunableloop already03:46
mwhudsonthere is a wiki page somewhere about being cleverer than BR, come to think of it03:46
lifelessStevenK: which means it can trivially sit in the garbo03:47
wgrantwallyworld__: Om nom nom XSS03:47
wgrantYour new sharing picker thing is nice and XSSable.03:47
StevenKlifeless: Meh, needs fixing03:47
wgrantAnd not feature-flagged :(03:47
lifelessStevenK: sure, or that03:47
wallyworld__wgrant: it's on a view that is not used03:48
mwhudsonwgrant: https://dev.launchpad.net/Code/BranchRevisions03:48
wgrantwallyworld__: That's only barely mitigation.03:48
lifelesswallyworld__: if someone can craft a url to it, they can exploit it03:48
lifeless(if there is an xss hole...)03:48
wallyworld__i'm not sure if the view is featured flags or not03:49
wgrantIt's not.03:49
wallyworld__so we need to ff the view03:49
StevenKlifeless, wgrant: You both chimed in on https://code.launchpad.net/~sinzui/launchpad/team-titles/+merge/93675 ? Do you have any remaining objections or were you just commenting?03:50
lifelessStevenK: sinzui took my suggestion into account03:50
wgrantI'm OK as long as the bug title changes are reverted.03:50
lifelessStevenK: the title is now preserved - see his QA instructions03:50
lifelesswgrant: they aren't reverted, but titles are shown in the page title03:51
wallyworld__lifeless: i'm in iharness and using Launchpad.login_with(). but there's no get() or named_get() method on the lp instance that is returned. what's the best way to get an instance of WebServiceCaller with those methods?03:53
wgrantLaunchpad.login_with() is a full launchpadlib03:53
wgrantThis is actually a two-layer XSS03:54
StevenKwgrant: Are you okay with the MP given what lifeless said?03:54
wallyworld__wgrant: there's no get or named_get methods though it seems03:54
wgrantwallyworld__: No, launchpadlib uses attribute access instead.03:54
wallyworld__so how to i get something i can invoke named_get or get on?03:55
wgrantDo you want to do that, or do you just want to experiment with the API?03:55
wallyworld__i have a service i think  have exposed and  i want to try and call it03:56
StevenKWrite a webservice test, then?03:56
wgrantMight as well just use launchpadlib. It's a nicer interface than WebServiceCaller03:56
wgrantAnd I don't know if one can use WebServiceCaller from outside AppServerLayer03:56
wgrantlp = Launchpad.login_with(fewfwefwehfuwef)03:57
wgrantobj = lp.load('/path/to/your/resource')03:57
wallyworld__.load is what i was missing03:57
wallyworld__it's not a domain object though03:57
wgrantHave you regenerated your WADL?03:57
wallyworld__i'm treating it as a service03:57
wgrantYou should be able to access it by lp.<whatever name you registered it as>03:58
wgrantBut note it's the top-level collection name, not its URL.03:58
wallyworld__so i assume export_as_webservice_entry() is the correct way to register it03:59
wgrantwallyworld__: Anyway, these XSSes are now blocking deployment.03:59
wgrantCollections aren't entries.03:59
wallyworld__it's not a collection03:59
wallyworld__it's a service03:59
wgrantHm, so it's a singleton entry?03:59
wgrantI guess that's OK, then.03:59
wgrantThese things generally get exported as collections, but I guess your way makes more sense.04:00
wallyworld__the ws api is geared to export entries and collections sadly04:00
wgrantJust means there's no way to access it without lp.load04:00
wallyworld__ideally i'd like to say something like lp.get('/myservicename?ws.op=something')04:00
wallyworld__or lp.post(/myservicename?ws.op=something&data=foo)04:01
wallyworld__it that possible?04:01
wallyworld__wgrant: so where's the xss hole? in the picker config passed to the client from the view?04:04
wgrantwallyworld__: LaunchpadWebServiceCaller is the interface you get in tests.04:04
wgrantIt looks like it speaks TCP04:04
wgrantSo you can probably use it.04:04
wgrantLaunchpadWebServiceCaller(protocol='https') should work04:05
wgrantI hope04:05
lifelesswallyworld__: so why do you want that rather than lp.thing.something() and lp.thing.something(foo) ?04:06
wallyworld__lifeless: whatever works. i tried that and couldn;t get it to work04:06
wallyworld__that would be my preferred way04:06
wgrantThen make that work :)04:06
wgrantWhat's the error?04:06
lifelesswallyworld__: so, suggestion, and I'm not being snarky - don't talk about what you don't want. Talk about what you do want!04:07
wallyworld__the error is that lp has no attribute thing04:07
wgrantWhat's the expression?04:07
* StevenK distracts wgrant with something shiny04:07
wgrantStevenK: Oh?04:08
StevenKwgrant: Are you okay with the MP given what lifeless said?04:08
wallyworld__so i'm not registering something properly. actually it's using the system launchpadlib04:08
* wgrant reads the diff04:08
wgrantwallyworld__: You sure you're connected to dev and not prod?04:08
wallyworld__yep lp = Launchpad.login_with('testing', service_root='https://api.launchpad.dev', version='devel')04:09
wgrantAnd what is failing, and what is the exact text of the error?04:09
wallyworld__<launchpadlib.launchpad.Launchpad object at 0x85c4e90> object has no attribute 'accesspolicies'04:09
wgrantRight, you didn't register it as a top-level collection, so it won't appear there.04:09
wallyworld__clearly i haven't set something up correctly04:09
wgrantYou have to use lp.load to get at it.04:10
wallyworld__so how do i avoid the lp.load() step?04:10
wgrantYou can't.04:10
wgrantUnless you say define it as an empty collection.04:10
wallyworld__ideally lp.accesspolicies would map to my AccessPolicyService04:11
lifelesswhat are the methods that you will have on this thing04:11
wallyworld__at the moment, i have foo()04:12
wallyworld__just to test it04:12
lifelessok, but really04:12
lifelesshelp me understand04:12
wgrantStevenK: Bug titles are missing a colon04:12
wgrantIt's now "Bug #1 blah blah blah"04:12
wallyworld__ok, getAccessPoliciesForProducts(product_collection, user)04:12
_mup_Bug #1: Microsoft has a majority market share <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Confirmed for compscibuntu-bugs> <LibreOffice Productivity Suite:New> <dylan.NET.Reflection:Invalid> <dylan.NET:Invalid> <EasyPeasy Overview:Invalid by ramvi> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <LibreOffice:In Progress by bjoern-michaelsen> <Linux:New> <Linux Mint:In Progress> <The Linux OS Project:In Progress> <metacity:In Prog04:12
wgrantWhich doesn't make sense.04:12
wallyworld__lifeless:  so i want to go lp.accesspolicies.getAccessPoliciesForProducts(product_collection, user)04:13
wallyworld__or something like that04:13
StevenKwgrant: Right, noted in my +1, thanks04:13
wallyworld__lifeless: and in the browser, there'd be an XHR call to the same api to get the data for the view04:14
lifelesswallyworld__: you want to pass *in* a vector of products?04:14
wallyworld__a set based interface04:14
wallyworld__but there'd be just the one normally04:15
lifelessso, I'm hugely in favour of set based calls, as you know04:15
lifelessbut all our existing api stuff would just make that a method on IProduct and IProject and IDistribution04:15
lifeless(or IPillar probably)04:15
wallyworld__putting service methods in domain objects is just so wrong04:15
wallyworld__and i'd rather avoid that in this new tranch of work if possible04:16
wallyworld__so i want to try a service based approach04:17
lifelessThat sounds good, but I htink then you need to commit to fixing the environment so that your new work is no worse than the current (which swimming upstream would be)04:17
lifelessspecifically, you'll need to teach launchpadlib about named objects or some such04:18
lifelessone hack you could use is to create a collection of services - IService or something04:18
wallyworld__right, that's what i didn;t know - i didn't realise it couldn't do that04:18
wgrantOr just define this as a collection with [] as the default content.04:18
wgrantcollections have methods too04:18
lifelessyou'd need to fix the launchpadlib bug where the interface the wadl specifies the type of collection members even when they return a more specific type (and self-annotate as such)04:19
wgrantI'm not actually sure if /branches is usefully iterable, for example.04:19
lifelessor as wgrant says, an empty collection will get you a named service fairly cheaply04:19
wallyworld__and i'd still need .load() right?04:19
wgrantTop-level collections show up as lp.foo04:20
lifelesswallyworld__: note though that the web API /only/ exports domain objects as far as its concerned, there isn't a separation there - and in most restful things I've seen there isn't a service layer as such04:20
wgrantSee eg. IBranchSet.04:20
lifelesswallyworld__: so I'm not-at-all-sure your approach makes sense for the *external* API.04:20
lifelesswallyworld__: I'd kind of like to talk it through with you and curtis04:20
lifelesswallyworld__: I'd like to understand the vision04:20
lifelesswallyworld__: where you want to take it04:21
wallyworld__lifeless: i haven't talked to much to curtis about it04:21
wallyworld__lifeless: want to jhoin our standup tomorrow?04:21
wgrantThis is very un-lazr.restful-ish.04:21
wgrantBut I think that's probably a good thing.04:21
wallyworld__i'm experimenting a bit first to see what can be done04:21
lifelesswallyworld__: I probably have conflicting calls; I have 3 in a row; what time is it ?04:22
wallyworld__8:00am AEST04:22
wgrantSo 11:00am04:22
wgrantDamn Queenslanders.04:22
lifeless11am? surely you jest, he's not 3 hours out from you is he ?04:22
wgrantJust realised what time the TL call is going to be for bigjools.04:22
wgrantlifeless: He's 3 hours from you.04:23
lifelessso, 11am I can do04:23
wgrantOne hour from us.04:23
StevenKwgrant: Do share?04:23
lifelessat the moment04:23
wgrantlifeless: 6am for you, right?04:23
wgrantWhich makes it 3am for bigjools :)04:23
lifeless9am at the moment04:23
lifelesshugely civilised04:23
wgrantWhen did that change?04:23
lifelesswhen mrevell said roughly 'uhm, I need to put my kids to bed at the current time, can we change it'04:24
wallyworld__lifeless: so i want to try and adopt a service oriented approach, using popo for the business model objects, separate from the storm db layer objects, and views getting flattened data from api calls etc04:24
wallyworld__and still support "sensible" api access from launchpadlib04:25
lifelessso this in some ways fits with where I want to take LP itself: remember I want to gut the server render layer to have no DB access at all04:26
wallyworld__it's how i've always built these types of systems previously04:26
lifelesshowever I think it would be good to separate the discussion into what you want the API to do and look and feel, and what you want to do in the appserver; because all the appserver work will be irrelevant as we move to more SOA backends04:27
lifelessthats not a reason not to do appserver work - because some things pay for themselves very quickly, and improvements now are improvements now04:27
lifelessbut I still want to get a handle on the specifics you're intending04:27
wallyworld__sure, np. it's still handwavy04:27
lifelessand how we'll measure the success of the experiment; and how we'll roll all the way forward, or roll back (depending on the assessed success)04:28
wgrantAnyway, we need to revert this picker thing.04:28
wallyworld__wgrant: what's the xss issue?04:28
wgrantwallyworld__: Put some stuff in a product title04:28
wgrantAnd visit +sharing04:28
wallyworld__so the json from the view doesn't get escaped properly?04:29
wallyworld__i thought it did?04:29
wgrantYou may recall that I strongly discourage ever encoding JSON manually.04:29
wgrantAnd always using JSONRequestCache wherever possible.04:29
wallyworld__wgrant: rather than revert, it's a simple one line fix to get the product title out of the picker header04:30
wgrantThat's not a fix.04:30
wgrantIt's getting the one bit of bad data that I've found so far out of the unescaped section of the page.04:30
wgrantRather than removing the injection of user-provided data into unescaped section of the page.04:31
wallyworld__we do that same pattern in a few other places in lp too04:31
wgrantYes, and they're all bugs.04:31
wallyworld__eg the inline picker widget04:31
wgrantJust because something is embarrasingly terrible already doesn't mean we should perpetuate it.04:32
wallyworld__sure. that's the trouble when it's in the code - the pattern can be reused04:32
wgrantHmm, I'm sure I've posed that point before and you've debated it :)04:32
wallyworld__so given the product title is the only user editable exploit, we could simply remove that for now04:33
lifelesswallyworld__: it is the problem, and thats why we need to finish our transitions and migrations more04:33
wallyworld__lifeless: well, i'm sure people thought the lazr picker work was finished when they did it04:34
lifelesswallyworld__: software is never finished ;)04:34
wallyworld__that's an answer to a slightly different question04:34
lifelesswallyworld__: we can only hope to put some of it in the ground soon :)04:34
lifelesswallyworld__: ok, so what was the question ?04:35
wallyworld__i was merely saying that the work in question was probably considered to be "finished", without the expectation there was still stuff to do04:36
wgrantIt was finished.04:36
wgrantIt was just wrong.04:36
StevenKNo jobs running04:36
wallyworld__wgrant: or the jsonrequestcache encoder could be used in this case04:36
wallyworld__rather than stuffing new stuff in the cache04:37
wallyworld__so the json would be properly escaped04:38
wgrantThat's what we've done in the past.04:38
wgrantBut nobody remembers.04:38
wgrantThis is why manual encoding is no longer permitted.04:38
wallyworld__i'll do that then for now04:39
wgrant(this is all because Microsoft are fucking morons, btw)04:39
wallyworld__so manual encoding works if done properly using the right encoder04:40
wgrantBut I wasn't going to say that, because it's always a bad idea.04:40
wallyworld__and in this case, i forgot to use the right encoder04:41
wgrantThat's why it's always a bad idea.04:41
wallyworld__wonder if we patch something  to force the correct encoder to be used by default04:41
wgrantThere's also a second layer of XSS after this.04:41
wgrantSo please just revert the whole thing :)04:41
wallyworld__wgat's the other xss?04:42
wgrant(an XSS I pointed out a few months ago during code review, but I was assured it would never be a problem)04:42
wallyworld__which is?04:42
wgrantThe picker title is treated as HTML.04:42
wgrantSo even once you encode the JSON properly, the product title is then injected unescaped into the picker.04:42
wgrant=> revert it all until someone looks through it thoroughly.04:43
wallyworld__i'd have to look to check, but the picker title is in the json data so should be escaped04:43
wgrantBonus points for deleting the rest of the pickers, but that seems unlikely.04:43
wgrantThat's not how escaping works :)04:44
wallyworld__hmm? we use the correct encoder and the json data is all properly escaped04:44
wgrantRight, when using the correct encoder the JSON is escaped fine.04:44
wgrantBut the JSON isn't really relevant for the second one.04:45
wallyworld__so that's what i was saying - i'd have to check but i thought the picker title was in the json data04:45
wallyworld__so would be escaped04:45
wgrantIt is.04:45
wgrantThat doesn't mean it's escaped properly in the picker.04:45
wallyworld__but if it comes from json data that is escaped....04:46
* wgrant reverts it.04:46
wallyworld__hang on a minute04:46
wallyworld__the issue can be easily corrected using the proper encoder04:47
wgrantThe first issue.04:47
wallyworld__and thye second issue is moot04:47
wgrantIt happens, so it's not moot.04:47
wallyworld__it is since the title is escaped in thje json04:48
wgrantThe JSON is escaped.04:48
wgrantThe title is not escaped in the JSON.04:48
wgrantThe point of escaping the JSON is to get the JSON through unharmed.04:48
wallyworld__if that's the case, then ALL of our pickers have this issue04:48
wgrantThat doesn't meant he contents of the JSON are escaped.04:48
wgrantBut few/none have variable data in the title.04:49
wgrantMost have no variable data in the config at all.04:49
wgrantOtherwise I would have started reverting every picker branch a year ago until it was fixed.04:49
wallyworld__ok, i'll revert this one.04:49
wgrantThis is similar to the reason I want to patch our mustache implementations to replace {{{ with {IAcknowledgeThatWgrantWillProbablyMaulMe{04:53
lifelesswgrant: s/mustache/handlebars/04:59
lifelesswgrant: and I know where the plumbing is to do that04:59
wgrantTempting to try making the factory more declarative at some point.05:05
wgrantProbably cut 40% off the test suite time.05:05
StevenK2.4 hours is better than 4 :-)05:07
wgrantStevenK: https://code.launchpad.net/~wgrant/launchpad/bulk-insert-2/+merge/9408405:13
StevenKwgrant: +105:20
* StevenK wonders if he can force a bug to be a dupe05:22
StevenKSince dupe.markAsDuplicate(bug) doesn't work since dupe.owner can't see bug05:22
wgrant(one X crash later): StevenK: Thanks05:22
StevenKPerhaps we need to bribe huwshimi to 'pay a visit' to RAOF.05:23
huwshimiStevenK: I can be there in 7 minutes05:24
wgranthuwshimi: You worked on the current loggerhead theme, right?05:29
huwshimiwgrant: I'm not sure how to appropriately answer that question...05:30
huwshimiwgrant: Hypothetically, yes. If-stabbing-is-a-possibility, no.05:30
wgrantJust wondering if you'd complain if I removed the 4px of padding around each line.05:31
wgrantI only stab for security holes :)05:31
huwshimiwgrant: Let me take a look05:31
StevenK(Pdb) p user05:31
StevenK<lp.registry.model.personroles.PersonRoles object at 0xfd81990>05:31
wgrantwallyworld__: Any luck with bending launchpadlib to your will?05:33
wallyworld__wgrant: not yet, had to revert that branch and redo the mp05:34
huwshimiwgrant: So you want to remove the 4px from the tds?05:36
wgranthuwshimi: Yeah05:37
huwshimiwgrant: Why would you want to do that?05:37
wgrantThe code is pretty unreadable at present.05:37
wgrantIt is currently at a density of 0.505:38
huwshimiwgrant: Oh, you're not talking about the file listing then...05:38
wgrantOh, no, http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/bootstrap.py05:39
StevenKwgrant, wallyworld__: Do either of you want to review https://code.launchpad.net/~stevenk/launchpad/bug-limitedview/+merge/94088 ?05:39
wallyworld__i can do it05:39
wgrantThe file listing and changelog are pretty dense.05:39
StevenKwallyworld__: Thanks. How long before you need to disappear?05:39
wallyworld__StevenK: gotta go to soccer in about 90 mins05:40
huwshimiwgrant: I have no problem with that at all :)05:40
wallyworld__got a reprieve from picking up the kid today05:40
wgranthuwshimi: Thanks.05:40
huwshimiwgrant: It may require a little padding, but I'll leave that up to you05:40
huwshimiwgrant: Just check you don't break the other listings at the same time05:40
StevenKwallyworld__: If it takes you 90 minutes to review, I'd be concerned. :-P05:41
wgranthuwshimi: Yeah, I think 1px either side is possibly OK05:41
* wgrant checks what other sites do.05:41
huwshimiwgrant: (I might have inadvertently added that spacing when I was trying to fix the other padding)05:41
wgrantAh, indeed.05:41
huwshimiwgrant: I'll leave it to your judgement :)05:41
huwshimiwgrant: Thanks that had been bothering me too05:41
wallyworld__StevenK: typo line 8805:42
wallyworld__StevenK: are there existing tests for the bugs security adaptor?05:43
wgrantwallyworld__: Oh, that reminds me, you misspelt duration in that commit too :P05:43
wgranthuwshimi: I might also try to make that view more friendly for copying.05:43
wgrantI wonder if side-by-side <pre>s works.05:43
StevenKwallyworld__: I'm not certain05:43
wgrantRather than a terrifying table.05:43
huwshimiwgrant: yeah, you could easily float divs next to each other with numbers in one and code in the other05:44
StevenKwallyworld__: I've added another test: http://pastebin.ubuntu.com/852269/05:45
wallyworld__StevenK: the new unit tests that are there are good but are incomplete. perhaps a comment on the test case would be good explaining that mainly limitedView is being tested05:45
StevenKwallyworld__: Incomplete? You think I'm missing a scenario?05:46
wallyworld__StevenK: get_bug_privacy_filter() is the method that is used05:46
wallyworld__to see if someone can view the bug05:46
wallyworld__it covers subscribers also i think05:47
wallyworld__IBug.userCanView() calls get_bug_privacy_filter()05:47
StevenKwallyworld__: You've lost me05:49
wallyworld__StevenK: so the security adaptor calls IBug.userCanView() to check for view permission05:50
wallyworld__and IBug.userCanView() calls get_bug_privacy_filter()05:50
wallyworld__which checks for subscribers05:50
wallyworld__which the tests don't currently do05:50
StevenKwallyworld__: Okay, I'm fairly sure that should be well-tested, so I can drop my unittests, except for the last one?05:51
wallyworld__StevenK: i think so. just add a comment on the test case that just limitedview access is being tested05:52
wallyworld__and leave it as an exercise for the reader to figure out where the other stuff is tested :-)05:52
StevenKwallyworld__: http://pastebin.ubuntu.com/852278/05:53
wallyworld__StevenK: looks good to me05:53
StevenKwallyworld__: Pushing05:54
* wallyworld__ still waiting for diff05:57
StevenKwallyworld__: Its updated for me05:58
wallyworld__StevenK: just updated for me too. took a while. r=me05:59
* StevenK fills up QA-Landing06:01
wgranthuwshimi: http://people.canonical.com/~wgrant/launchpad/lh-view/old-view.png -> http://people.canonical.com/~wgrant/launchpad/lh-view/new-view.png06:26
wgrant(the new one is copyable, since it's a table of two <pre>s)06:27
huwshimiwgrant: Awesome06:27
wgrantAlthough there's still too much space above the top line06:28
wgrantStevenK, wallyworld__: Any comments?06:28
wgrantTrying to make loggerhead a bit less painful for reading code06:29
wallyworld__wgrant: +1 from me, looks nice06:29
wgrantThe great LP bug migration corrupted things.07:13
pooliemrevell, hi there07:33
adeuringgood morning08:40
=== almaisan-away is now known as al-maisan
al-maisanGood morning adeuring, how are things?09:35
adeuringhi al-maisan! things are fine here. how about you?09:35
al-maisandoing well, thanks :)09:36
gmblifeless, Around?09:53
* jml keeps hitting refresh10:47
jelmer'morning abentley13:30
abentleyjelmer: morning.13:30
jelmerabentley: I heard you're working on fixing the scaling issue in the branch scanner?13:31
abentleyjelmer: Yes, mainly by fixing the DB access.13:31
jelmerabentley: cool13:31
jelmerabentley: I'm working on removing the use of Branch.revision_history and Repository.get_ancestry from the lp codebase, and as such was also touching that bit of the code13:32
jelmerit sounds like our changes shouldn't overlap though13:32
abentleyjelmer: No, I don't think I touched the bzr side.13:32
abentleyjelmer: It was only 13% of runtime when I profiled it.13:33
abentleyjelmer: lp:~abentley/launchpad/bulk-insert13:33
abentleyjelmer: You're just wanting to avoid deprecated interfaces?13:33
czajkowskiare there any plans in the works to support proxy on launchpad? have two bugs today with issues about using launchpad in one country and having to use a proxy13:34
jelmerabentley: mostly, yeah. In the process I'm also changing things to not access all of the branch history/ancestry if it doesn't have to13:34
abentleyczajkowski: I haven't heard of any plans, but I hadn't heard of any issues, either.13:35
czajkowskiabentley: https://bugs.launchpad.net/launchpadlib/+bug/938542  and https://bugs.launchpad.net/launchpad/+bug/93858013:36
_mup_Bug #938542: launchpadlib doesn't support system proxy <launchpadlib :Triaged> < https://launchpad.net/bugs/938542 >13:36
_mup_Bug #938580: launchpad not opening in Syria.. <Launchpad itself:Triaged> < https://launchpad.net/bugs/938580 >13:36
abentleyczajkowski: Oh, launchpadlib.13:36
czajkowskiwell one is launchpadlib and the other cant use LP without going through a proxy13:37
abentleyczajkowski: I would be surprised if they were related.13:38
abentleyczajkowski: So it sounds like one issue is that lazr.restfulclient (which launchpadlib uses) may not respect the system proxy settings.13:41
abentleyczajkowski: And the other issue is that access to Launchpad in Syria doesn't work, *except* when a proxy is used.  Which may be due to government interference with the Internet.13:42
abentleyczajkowski: I would ask webops about the second one.13:44
czajkowskiok cheers13:44
abentleyczajkowski: The first one needs investigation, but can certainly be fixed if true.13:46
czajkowskiabentley: thanks13:47
salgadomrevell, hey there. did you have a chance to talk to huw/dan about that new page we'd like to implement in LP?14:03
=== danhg_ is now known as danhg
deryckadeuring, abentley -- https://plus.google.com/hangouts/extras/talk.google.com/orange-standup14:32
deryckrick_h, I'm recalling you being away today, if not, see ^^14:32
=== matsubara is now known as matsubara-lunch
=== abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: abentley | Firefighting: - | Critical bugtasks: 4*10^2
=== al-maisan is now known as almaisan-away
danilosmrevell, heya, we have a chat scheduled in ~11 mins15:49
mrevelldanilos, We do15:50
danilosmrevell, ok, just confirming, let's do mumble15:51
mrevelldanilos, Dude, it's not 2010 any more.15:52
=== matsubara-lunch is now known as matsubara
danilosmrevell, ah, right, sorry, proprietary software is "in" again ;)15:53
mrevellheh, Mumble it is, then.15:53
salgadomrevell, oh, is that the recurring one for which I've never gotten the invite?15:53
danilossalgado, you don't deserve one15:54
mrevellsalgado, It's the recurring one, yeah. Let me see if I can fix the invite problem.15:54
salgadodanilos, I know I don't but with mrevell being English and all I thought I'd get one if only for his politeness ;)15:54
danilossalgado, it's supposed to be in your canonical.com calendar15:54
danilossalgado, fair point15:55
mrevellI've sent an invite to your Linaro.com address, salgado15:56
salgadoaha, now I got it15:56
=== salgado is now known as salgado-lunch
lifelessgmb: hi17:12
gmblifeless, Hi. W.r.t my earlier ping, I was checking whether the fix for bug 518016 was in the subunit 0.0.8beta that's in the LP sourcedeps. From what I can tell from commit timestamps it is (you added the snapshot 9 minutes after committing the bugfix) but I just wanted to be sure.17:13
_mup_Bug #518016: No public API for tagging on TestProtocolClient <paralleltest> <Launchpad itself:Fix Committed> <subunit:Fix Committed> < https://launchpad.net/bugs/518016 >17:13
gmb(I assumed "yes" for the purposes of marking the bug fixed)17:13
lifelessgmb: yes makes sense to me :)17:19
lifelessgmb: or even fix released, as its not something that needs deploying to be fixed17:19
gmblifeless, Righto, works for me. Thanks.17:20
lifelessabentley: AWS SWF looks interesting17:22
abentleylifeless: yes, indeed.17:23
=== salgado-lunch is now known as salgado
=== Ursinha is now known as Ursinha-lunch
danhgWhen is Pay Day? Does anyone know?19:25
lifelessgary_poster: 2406119:36
lifelessbug 2406119:36
_mup_Bug #24061: GPG error with apt-get/aptitude/update-manager behind proxy (BADSIG 40976EAF437D05B5) <apt (Ubuntu):In Progress> <update-manager (Ubuntu):Won't Fix> <apt (Ubuntu Precise):In Progress> <update-manager (Ubuntu Precise):Won't Fix> < https://launchpad.net/bugs/24061 >19:36
gary_posterlifeless, ah-ha, with workaround included! thanks19:37
lifelessgary_poster: basic troubleshooting is - remove the files from /var/cache/apt/ etc19:37
lifelessgary_poster: if that doesn't work, you're into figuring out what host is serving inconsistent data19:38
gary_posterlifeless, do you agree with workaround listed in bug?19:38
lifelessthe Acquire::BrokenProxy thing? if it works yes ;)19:51
lifelessgary_poster: ^19:52
gary_posterlifeless, heh19:53
lifelessI don't recall the code changes that triggers, but working >> most anything else19:53
lifelessyou may, when running precise, naturaly encounter skew mid-archive-push, but the archive update scripts work hard to make the skew interval sub-second19:54
=== garyposter is now known as gary_poster
=== Ursinha-lunch is now known as Ursinha
abentleylifeless_: Is https://code.launchpad.net/~abentley/launchpad/callgrind/+merge/94283 also worthy of a LOC waiver?22:10
czajkowskiflacoste: should all questions end up being 'solved' rather than marked as 'answered' after a length of time22:10
lifeless_abentley: It seems to me that if the bzrlib lsprof module supported just a little more glue, you could delete the entire profiling support for scripts from LP and not miss it22:24
wgrantsinzui, jcsackett: https://wiki.ubuntu.com/Bugs/Status22:25
sinzuiStevenK, wallyworld, jcsackett, wgrant: http://pastebin.ubuntu.com/853310/22:32
=== matsubara is now known as matsubara-afk
=== lifeless_ is now known as lifeless
lifelessjames_w: hey, so, django22:48
lifelessjames_w: what does your group do when getting a new django API/site up and running; do you use packages of django (are they in lucid-backports or a PPA) , ....22:49
james_wyes, we use packages22:49
james_wfrom lucid-cat currently22:49
james_w(to get django 1.3)22:50
lifelessdo you use buildout or anything, or do sysadmins manually manage settings.py?22:50
lifeless(+ mod-wsgi glue?)22:50
james_wdjango.wsgi is delivered by puppet22:51
james_wwe *may* use buildout, but we don't currently22:51
james_wsettings are stored in a hierarchical way22:52
james_wmain.cfg for settings used everywhere22:52
james_wdev.cfg and production.cfg for settings that are only appropriate in those environments, with a switch between them22:52
hingoHello #launchpad-dev. Is it possible my launchpad ppa ignores the epoch number given in .changes file?22:53
james_wproduction_credentials.cfg delivered by puppet for secrets22:53
james_whingo, I doubt it22:53
james_wall settings via django-configglue22:53
hingoI'm trying to: dput ppa:drizzle-developers/ppa drizzle_7.1.31~rc-1~oneiric1_source.changes22:54
james_wdependencies via a mix of packages (most things) and branches (things we develop that are tightly bound to the service)22:54
hingo...but nothing (not even a failure) appears at https://launchpad.net/~drizzle-developers/+archive/ppa/+builds22:54
james_wdevstaging deployments in ec2/canonistack using fab and some custom puppet (not related to memento unfortunately)22:55
james_whingo, do you get an email about the upload?22:55
hingoThe thing is, oneiric ships with drizzle_2011.03.13-something. So I have increased epoch to 1.22:55
hingojames_w: No.22:55
flacosteczajkowski: nah, answered is fine in the end, since it's up to the reported to mark them solved22:55
wgrantczajkowski: Users mark questions as solved once they confirm the answer.22:55
wgrantczajkowski: We should generally not set them to Solved ourselves.22:55
james_whingo, then it is 99% likely that the .changes file is incorrectly signed22:55
james_whingo, what is your LP username?22:56
hingojames_w: Ok. I created a generig GPG key. Do I need to use my own / matching email address?22:56
hingojames_w: hingo22:57
czajkowskiflacoste: wgrant ok, just wondered as there are many not set as solved there. just curious22:57
wgrantczajkowski: A lot of users don't do it.22:57
wgrantBut it doesn't matter much.22:57
james_whingo, you need to register the GPG key that you are signing the upload with: https://launchpad.net/~hingo22:57
wgrantGPG verification of /srv/launchpad.net/ppa-queue/incoming/upload-ftp-20120222-215212-001713/~drizzle-devel22:58
wgrantopers/ppa/ubuntu/drizzle_7.1.31-rc-1_source.changes failed: Verification failed 3 times: ["(7, 9, u'No public key')", "(7, 9, u'No public key')", "(7, 922:58
wgrant, u'No public key')"]22:58
hingojames_w: Ok, this makes sense. I'll upload the public key.22:58
hingojames_w: That explains it. Thanks a lot!22:59
wgrantStevenK: You also need to QA your double word thing.23:00
wallyworld_what happened to buildbot?23:01
wgrantwebops: did someone kill it?23:01
thedacwgrant: that was me. We had defunct processes that needed cleaning up23:02
* wgrant forces.23:02
=== abentley changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugtasks: 4*10^2
abentleylifeless: I can see how you could reduce the number of lines of profiling support for scripts, but how would you remove it entirely?23:06
hingoThere she goes: https://launchpad.net/~drizzle-developers/+archive/ppa/+builds?build_text=&build_state=pending23:12
hingojames_w: Thanks.23:13
lifelessabentley: If bin/py -m bzrlib.lsprof cronscripts/foo .... profiled cronscripts/foo in the same way bin/py -m pdb cronscripts/foo ... profiles it, then you don't need the script to know that it is being profiled23:13
james_wlifeless, does that cover your questions?23:14
wgrantlifeless, poolie: I want to make the soft timeout flaggable like the hard timeout is. But unfortunately the feature controller is destroyed before the soft timeout is checked.23:14
wgrantI'm considering simplifying things by just storing the hard and soft timeouts in thread-locals at the start of the request.23:15
wgrantRather than relying on the feature controller for every timeout check.23:15
abentleylifeless: Ah, I hadn't seen that before.  Yes, that might be worthwhile.  (Though profililing via bzrlib is a somewhat strange idea anyhow.)23:15
wgrantA lot of bzrlib seems like it should be called "this should really be in the standard library"23:16
poolieespecially osutils23:17
pooliearguably also transport23:17
abentleywgrant: IIRC, the lsprof module was rejected from the standard library.23:17
pooliewgrant, so you're saying you'll basically get it from the feature controller and remember it23:17
poolieoh also selftest23:17
wgrantpoolie: Yeah23:17
wgrantpoolie: We already preread it at the start of the request23:17
james_wif a language had excellent distribution support then I think it would take over the world, but that may be naive23:18
wgrantTo get it cached so it doesn't require DB access later.23:18
wgrantBut it relies on the feature controller surviving.23:18
lifelessjames_w: I think so, basically if you look at oops-tools is uses buildout to drop a local copy of django23:18
lifelessso its not getting bugfixes23:18
lifelessjames_w: is there a ppa with django 1.3 in it ?23:18
james_wlifeless, yes23:18
wgrantI could make the controller destruct later, but the way it's done now is reasonably nice.23:18
james_whttps://launchpad.net/~canonical-ca-hackers/+archive/production/+packages contains the stuff we use from CAT23:18
abentleywgrant: The problem with the "batteries included" approach is when the batteries are all the same size :-)23:18
lifelessjames_w: any objection to e.g. LP using that, or nabbing packages from it ?23:19
james_wlifeless, none, we stole them from CAT after all :-)23:19
james_wobviously co-ordination is encouraged, as we all end up having to share again when things are pushed back to CAT23:20
lifelessI was more checking you're not tossing known-broken stuff in there23:20
james_wlifeless, I have all the dependencies of oops-tools packaged fwiw23:21
james_wlifeless, that PPA contains only what is deployed in production23:21
james_wlifeless, we have a staging PPA for the packages we are waiting for copying to CAT23:21
james_wso it's all very much things we expect to work23:21
pooliewgrant, thanks for the loggerhead improvements23:34
=== broder_ is now known as broder

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!