[00:02] Someone linked to bug #1234 on the ISO tracker. [00:02] I expect. [00:02] Also, ubuntuqa uses edge. [00:08] another bug to file [00:46] * StevenK grumbles at TALES being useless. [00:59] . [01:04] lifeless: http://pastebin.ubuntu.com/730574/ [01:09] win [01:09] uhm, nothing jumps out at me [01:14] What's annoying me is that I saw it last week, and I fixed it, and I can't remember how. [01:15] wgrant: Do you have any clues from the above traceback? [01:16] Not really. [01:23] No traceback when not logged in. Curious. [01:26] Hardly. [01:26] It's in all probability the user info in the top right corner. [01:27] That's what uses link-display-id or whatever it is the most. [01:27] But factory.makePerson() should set all of that up, surely ... [01:28] Which layer are you in? [01:28] DatabaseFunctional [02:10] wgrant: Is DatabaseFunctional too low? [02:27] StevenK: shouldn't be. [02:32] wgrant: Same error with LaunchpadFunctional :-( [02:32] Oh, damn it, I know what I'm missing [02:47] Is anyone else getting sporadic "Exception: RabbitMQ server is not running." errors when trying to run 'make run'? [03:09] huwshimi: not I [03:09] huwshimi: but make run isn't a common thing for me atm [03:09] huwshimi: poolie had that recently, but I don't know what was causing it for him [03:09] I use make start and make run frequently. [03:09] No trouble. [03:09] But this is lucid. [03:10] hmmm [03:11] AssertionError: queries do not match: 0 != 1817 [03:11] Whee! [03:11] can has review ? https://code.launchpad.net/~lifeless/python-oops-twisted/bug-873030/+merge/81427 [03:11] haha yes [03:11] And 200 SPPH queries, so success [03:11] huwshimi, that's actually launchpad's cute way of saying "rabbitmq server *is* running" :) [03:12] bbiab, -> shopping [03:12] i put up a patch to make it give a better message but i think it's not deployed yet [03:12] look for a stray rabbit process and kill it [03:12] poolie: Oh right. That is cute [03:12] poolie: It doesn't need to be deployed, it just needs to be on devel [03:12] apparently our tone of voice is 'extremely sarcastic' [03:12] StevenK, does that come from a branch? [03:13] poolie: lp:launchpad [03:13] * poolie goes to look [03:13] no, it needs to be fixed in rabbitfixture [03:13] Ah [03:13] Where does that live? [03:13] huwshimi, was that in fact your problem? [03:14] It's an egg. [03:14] if so that nudges me to make sure my thing lands [03:14] poolie: I'm not sure, I'll try next time it complains [03:14] The only things that come from branches are bugs. [03:14] by 'deployed' i meant 'packaged updated etc' [03:14] huwshimi, if there's an orphaned instance i think it will happen again next time until you do kill it [03:15] poolie: it hasn't complained for a while [03:15] wgrant, ha ha [03:15] but, what are the things in sourcecode/ then? [03:16] poolie: We have three ways of adding dependencies. [03:16] poolie: They're bugs. [03:16] poolie: They shouldn't be branches. [03:16] i know this [03:16] StevenK, i know there are 3 ways [03:16] is this wgrant's way of saying that one of those ways is deprecated? [03:17] Or at least generally discouraged, yes. [03:17] (also, there's a fourth way: including them in the tree) [03:18] that's what you mean by saying they're bugs? [03:18] Yes. [03:19] Oh, ugh, I didn't know we linked from lib into sourcecode. [03:19] That's how sourcecode works :) [03:19] That's disgusting, kill it. [03:20] wgrant: I have a test that works! [03:20] wgrant: It's a bit evil at the moment since it creates 100 BugNominations, but it proves the query count is excessive [03:23] Oh, it's even worse. lib/mailman is a copy of sourcecode/mailman [03:25] * StevenK tries to work out how to prefetch the entire batch to seed Storm's cache [03:30] I'm guessing in Launchpad there is no intelligent way to create user icons from there logos or mugshots right? [03:31] huwshimi: No. [03:31] #$% [03:31] huwshimi: For projects we ask them to upload a separate 14x14 image. [03:31] Particularly since you can't sensibly scale much down to 14x14 [03:31] well you can, we just don't have the infrastructure [03:32] image scaling without jaggies is not -all- that simple. We might have been better off asking for a single svg. [03:33] O_o [03:33] not sure if serious [03:33] No, [03:33] this is ridiculous [03:34] why do we have to overthink everything [03:34] this the web [03:34] people scale images ALL THE TIME [03:34] Yes. [03:34] and they don't have massive scaling SVG mega frameworks, just to do the simplest of things [03:35] Using SVG here is entirely ridiculous. [03:36] But I'm not convinced that we can usefully scale normal sized images all the way down to our 14x14 icon format. [03:36] it all depends on what the criteria you're aiming for are. If we are happy with browser scaling, just set attributes on the image. [03:37] Anything below 32x32 tends to be pretty hideous, IME. [03:37] If we aren't happy with browser scaling, I think its a safe assumption we won't be happy with unattended resizing either. [03:37] (Because browsers keep iterating on their scaling logic to make things look good, so they are pretty solid) [03:38] lifeless: Really and use all the extra bandwidth, you really think that's the best practice here? [03:38] lifeless: And how about background images? [03:39] lifeless: Shall we exclude older browsers from having scaled images too? [03:39] huwshimi: I don't know - I'm not *setting* the goals here - [03:39] huwshimi: you'll know much better than I the limitations of the browsers we're dealing with [03:40] lifeless: Right, but this is a solved problem. Let's not invent a new ethos for handling images [03:40] huwshimi: thats great [03:41] huwshimi: I didn't realise it was so well solved :) [03:42] Hmm? Server-side thumbnailing isn't exactly an uncommon, new concept... [03:42] wgrant: Thankyou [03:42] Perhaps lifeless is one of our four IE6 users. [03:43] wgrant: not down to 14x14 [03:43] I recall implementing it in some of my early web projects in like 2002, and it was probably popular before then and 10-year-old me just didn't realise. [03:43] No. [03:43] Not down to 14x14. [03:43] I agree that we can't do that. [03:43] thumbnailing is one thing [03:43] but extremely large, or extremely small scales are damn tricky [03:44] lifeless, wgrant: You seriously think we shouldn't scaled down to ~14 (or 20 or whatever we want for small icon sizes)? You seriously think most people aren't just going to scale it before they upload it anyway (if they're a project) [03:44] heck, Next machines in 1997 happily thumbnailed images in a directory listing, if memory serves. [03:44] It is a rare website that uses small icons like that. [03:44] We should probably stop. [03:45] huwshimi: No, I argue that we shouldn't be using 14x14 icons for projects and people. [03:45] I don't think anyone else does. [03:45] They use more sensible stuff of at least 20x20, from what I've seen. [03:45] huwshimi: I know that its hard to get a good representation of a logo at 14x14 without manual fixups - e.g. an icon editor [03:46] huwshimi: I don't know if our users care about that [03:46] huwshimi: and I don't know if we need 14x14 for any reason other than having it already. [03:46] Exactly. [03:46] huwshimi: I am not about to argue for complexity we don't need : see francis bug report for the cost of complexity on us as a team. [03:46] right, but that shouldn't stop us from forcing users and ourselves into set sizes (192, 64, 14) [03:47] huwshimi: one answer, for instance, is to scale by default and allow overrides for folk to fix things up [03:48] stub: https://code.launchpad.net/~lifeless/python-oops-twisted/bug-873030/+merge/81427 (OCR penance time) [03:48] huwshimi: if your new design will permit a larger micro size, it may make a substantial difference to auto-scaled images [experimentation or citatin needed] [03:50] My prefetch query doesn't impact the query count :-( [03:50] Ugh, it just turns my job into a pipe-dream, as everything I design requires weeks of work to implement the simplest thing (and would never get priority over other work) [03:50] It could just be a stupid query [03:52] lifeless: k [03:52] StevenK: might be the cache size as well [03:52] StevenK: dev runs with a storm cache of only 100 objects. [03:52] StevenK: Product uses 10000 [03:52] Production, that is. [03:53] huwshimi: are there js only scaling libraries? I assume there must be [03:53] huwshimi: you could scale on the client and upload three things from one js form [03:53] huwshimi: s/you could/could you/? [03:53] wgrant: Right, so my test might just be too large at the moment [03:53] lifeless: Well there kind of are, but they still require the client to download the full-size image [03:53] StevenK: or just about right [03:53] wgrant: TBH, I'm tempted to just blame my current query [03:54] huwshimi: I meant in the edit form for the person; do all the prep on the client [03:54] lifeless: I'm creating 100 bug nominations, and then browsing the view with ?batch=100. It seems excessive to me, and I wrote it. [03:54] huwshimi: that would avoid nearly-all backend changes [03:54] lifeless: Oh, I see. Well there's no javascript version of PIL, Imagemagick etc. no. [03:54] huwshimi: which is where we are not really prepped to do dynamic stuff [03:56] huwshimi: we have reasonable tools to run e.g. imagemagick in the backend, but they aren't trivial to use if you're not familiar with them, which IIRC was one of your goals [03:57] lifeless: So you do want a grammar and coding style review, or wait for someone who knows a bit about twisted and wsgi? [03:57] stub: jamesh is eyeballing it [03:57] stub: I hope he's wiring it up to test in the u1 stack [03:57] stub: as much of a review as you want to give would be great. [03:57] lifeless: I'm not sure quite what you mean. It's pretty easy to set up PIL for thumbnailing etc. (it even has .thumbnail()) [03:58] huwshimi: I would expect that to be done in a 'job' straight after the upload succeeds, to avoid timeout issues. [03:59] huwshimi: which involves a new table; a new config entry to run under the shared job environment, some tests for your code, and a Job definition [03:59] And 64 million interfaces [04:00] we're iterating towards this being lighter still [04:00] lifeless: Sure, or it is often set up as a service when required (e.g. launchpad.net/thumbnail-service/14/14/file.jpg) [04:00] and, pick yourself off the flow, it used to be much worse [04:00] huwshimi: s/flow/floor/ [04:01] huwshimi: I'd have some scaling worries about a JIT service, though yes we could do one. [04:01] lifeless: what is the number after your name in NEWS? [04:01] stub: did I forget the # ? its a bug number [04:01] lifeless: Right, but anyway, I'll stop talking about this now as it is only a frustrating mental exercise [04:02] lifeless: And the word 'Bug' :) [04:02] stub: not needed ;) [04:02] stub: the rest of the file - and in other projects too - just does #1234 [04:03] Yay for confusing [04:03] seen a Debian/changelog recently ? :) [04:03] 'Better than a Debian changelog' is damning with faint praise [04:04] But at least with the '#' it won't look like you are giving your body measurements or date of birth [04:04] :) [04:13] Hm, my query doesn't help with only 10 bug nominations either, still 197 queries. [04:13] StevenK: is your theory that they *can* come out of cache false? [04:14] StevenK: if the deeper code is doing anything *other* than attribute traversal, you can't optimise by cache loading. [04:14] StevenK: you would need to actually setify the logic (which is a good thing anyhow) [04:14] lifeless: I thought I could cache the bugnominatons themselves in the first case. [04:15] But it seems that is pointless [04:16] lifeless: So, I'm tempted to declare that cache seeding won't help, but then I'm stuck about what I can do instead [04:16] have you identified the cause of the problem ? [04:17] I've identified what is causing the masses of SPPH queries. [04:17] go on [04:17] I'm not sure if that is the cause, but it's one thread to tug on. [04:19] StevenK: This is for latest_published_component? [04:19] StevenK: ok, so can you pastebin the call stack ? [04:19] wgrant: So latest_published_component wants SPPH, and I then I think verifyUpload() does too [04:19] StevenK: How are you preloading it? [04:19] You'll need to turn it into a cachedproperty. [04:20] And populate it manually in a set-based manner. [04:20] wgrant: Currently, I was trying to cache the bug nominations themselves first, but that hasn't changed anything. [04:20] Are there lots of BugNomination queries? [04:21] wgrant: 2 per row [04:21] That doesn't make much sense. [04:21] Do you have a query log? [04:21] Yup [04:22] wgrant: http://people.canonical.com/~stevenk/test_query_log.txt [04:23] none of those are attribute access queries [04:23] Ah, so SPPH isn't really the problem at all. [04:23] The whole thing is. [04:23] they are all explicit lookups from procedural code AFAICT in a quick glance [04:23] Heh [04:23] cache cannot help you at all [04:23] Well, the built-in Storm cache can't. [04:23] ^storm ... [04:23] cachedproperty can [04:24] So, you are going to need to radically setify canApprove. [04:24] Unless you want to do it all manually beforehand, which I would discourage. [04:25] Right, so canApprove() is the crux of the problem. I certainly agree. [04:25] so what you need to do is write a set based version of it [04:25] s/crux of/ [04:25] that takes many nominations (or something like that) and figures out the answer for all of them. [04:25] Exactly. [04:25] make the existing canApprove forward to that new function [04:26] and finally migrate this code path over to use the new function directly with all of the nominations for the page [04:27] But canApprove() is hideous :-P [04:27] yes [04:27] take small bits [04:27] *bites* [04:27] Hm, there is a BugNominationSet ... [04:28] So there are two issues I can see. [04:28] 1. My batch in the view is bugtasks. [04:28] 2. canApprove() sucks [04:29] the other alternative is to eliminate nominations; is possible that the new security rules mean they are not useful now. [04:29] so 1) is something that cachedproperty may help with [04:31] Oh, sure, eliminating nominations doesn't sound controversial *at all* :-P [04:46] wgrant: Can I beg you for a pre-impl about this new method? [04:47] It's pretty simple really, but OK. [05:03] wgrant, re bug 435905, i wonder if you would agree we should just close it [05:03] and keep wrapping roughly where we currently do [05:03] until we get proper markup [05:04] 3K timeouts. Hmm [05:04] lifeless: Network outage overnight. [05:04] stub: so no review results ? [05:04] stub: or just the # is it ? [05:05] poolie: Do you really mean me? [05:06] i do; istr you had some opinion about always using greasemonkey before [05:06] and your 'fix the defaults' reminded me of it too [05:06] Ah. [05:06] Well, I think the fix for it is markup. [05:06] lifeless: Getting there, distractions distractions [05:06] then i agree [05:06] thanks [05:06] stub: ;) [05:07] aieee, gmail, what have you done [05:07] poolie: The current uploads work mostly. We shouldn't keep hacking it to make various special cases work. We should do what the rest of the world does. [05:07] We should follow what the rest of the world does for an awful lot of things. [05:07] We needed to do our own thing 7 years old. [05:07] s/old/ago/ [05:08] But it's no longer 7 years ago, and the world has moved on. [05:08] 'current uploads'? [05:08] Er. [05:08] Sorry, multiple conversations. [05:08] current css? [05:08] Yes. [05:08] i agree with the general thing anyhow [05:09] i do wonder how much say feature flags or timelines are available externally in other frameworks [05:09] rpm.net [05:10] ? [05:10] it lapsed? [05:11] huh [05:11] let me find their current home [05:11] did you mean rpmfind? [05:11] Was that the fork, or the not fork? [05:11] I can't remember. [05:12] the description of the hardware on http://rpmfind.net/ is awesome [05:12] though i bet we have some that is older [05:12] http://newrelic.com/ [05:12] huh, they have added python since I last strolled by [05:14] http://newrelic.com/docs/python/instrumented-python-packages [05:14] mm [05:14] there may even be free versions [05:15] s//alternatives [05:15] well, oops-tools & sentry are in that space, though neither is anywhere -near- as advance. [05:16] oops-tools is the most metric enabled open end-to-end one I know of for Python [05:17] * stub wonders why magic asynchronous tests are a good idea [05:19] stub: magic? [05:19] is there any kind of trick about $PYTHONWARNINGS? [05:19] test methods returning deferreds with callbacks that the test runner somehow executes because the test case class has a property set to a particular value [05:20] stub: well, you need *something* [05:20] stub: this has the nice property that its extensible to different async environments in one library (vs e.g. twisted trial) [05:21] because tests being nearly the ultimate in synchronous code require asyncronicity [05:22] yeah, they very much do when the code is callback based [05:22] and the callback engine doesn't guarantee pause/resume/single step facilities [05:22] but a helper invoked explicitly would remove the magic [05:23] And would even allow you to test post state without registering these tests as callbacks. [05:24] see under callback engine guarantees [05:24] d.addCallback(lambda _: self.assertEqual(expected, self.calls)) would be much more readable as just self.assertEqual(expected, self.calls) [05:25] That doesn't mean anything to me. [05:26] but there's a difference ,right [05:26] the assertion is: _eventually_ when this is called, there are N calls [05:28] stub: I agree that it would be nicer if it was single stepped and looked just like regular python [05:28] I don't see the difference between 'd.addCallback(lambda: blah blah assert blah); return d' and 'self.run_deferred(d); self.assert' [05:28] in principle inlineCallbacks could be used. [05:28] A review for some lovely person, if possible: https://code.launchpad.net/~huwshimi/launchpad/avatars-everywhere-712894/+merge/81430 [05:29] Yer, just the more I see twisted code the more I think that twisted was a bad idea and should never have been shoe horned into Python. [05:30] Code that makes you go 'blergh' when you haven't gone through the indoctrination process keeps you isolated. [05:35] huwshimi: I've given it a once over [05:35] ah, the trick is it's new in 2.7? [05:35] poolie: what is new in 2.7 ? [05:36] $PYTHONWARNINGS [05:36] ah, dunno. [05:36] added by one B. Warsaw [05:45] *yawn*, morning [05:50] hi nigel [05:50] Hey poolie :) [05:50] is there some easy way to override buildout and make it use a dependency from a tree, while i'm futzing with it? [05:50] i guess replace the egg with a symlink [05:52] poolie: You should be able to symlink the tree into the root of your LP tree, then add the dir name to the 'develop' setting in buildout.cfg. [05:53] poolie: bin/buildout buildout:develop=". ../path/to/tree" [05:54] wgrant: no need to symlink [05:54] Oh, or that. [05:54] Didn't know that worked. [05:56] wow, ok [05:57] so those two trees come before everything else? [05:58] develop should just be '.' by default... what do you mean? [05:58] huwshimi: whoa! [05:59] wgrant, i meant before things pulled in from eggs [05:59] huwshimi, that would be a good thing to add behind a user 'labs' switch [05:59] though i'm not suggesting you block on adding one [05:59] stub: hi, i have a sql question for you if you have a moment sometime [05:59] poolie: Yeah, we'll see how much it breaks launchpad :) [06:00] wgrant: http://pastebin.ubuntu.com/730709/ [06:00] at least a flag would be good [06:00] wallyworld_: Just ask, it may not be stub that answers. [06:00] poolie: Wouldn't labs be similar to launchpad-beta-testers? [06:00] StevenK: it's a performance related thing [06:01] wallyworld_: Then lifeless or wgrant would probably also have opinions [06:01] fair enough. although i didn't want to bug them :-) [06:01] but anyway, i think this will be ok, hopefully: https://pastebin.canonical.com/55351/ [06:02] it is to check whether a given person owns any pillar [06:02] and i'm looking to use it amongst other places in a storm validator [06:02] so i want it to perform well [06:03] there's a similar query for isSecurityContact [06:03] huw, it would be similar with a few differences [06:03] - the ui is more obvious than joining/leaving a team [06:03] - teams are kind of used for too many things [06:04] - we can have more than one concurrent experiment [06:04] wallyworld_: That query seems very odd. [06:04] - we could hook it up to feedback [06:04] - oh, users would have a clearer idea what things were experimental [06:04] stub: maybe. the unit tests pass :-) [06:04] but if it can be improved... [06:04] at the moment they just see everything (though i believe they're going to get some kind of systematic indication in future) [06:04] wallyworld_: I'd prefer that it was in a setter, not a storm validator. [06:05] Yes, but selecting the first row from the person table if the EXISTS matches, throwing it away and returning true... [06:05] stub: the only reason is use select from person was to keep dtore.find happy [06:06] stub: i can constuct sql without any table "select exists ...." but storm complains [06:06] i guess i could do a store.execute() with raw sql? [06:06] by complains i mean store.find() says "there's no table" [06:07] if i try and run something like "select exists (select something where condition)" [06:08] wgrant: storm validators are bad? [06:08] they seem to be used in the type of thing i'm doing already [06:08] eg is_valid_person and friends [06:09] Lots of things are already used for lots of types of things :) [06:09] I think Storm validators impede clarity. [06:10] And are often used to do very bad things. [06:10] i don't agree that they hinder clarity [06:10] anything can be misused [06:11] Storm validators become problematic when eg. you want to pass an argument into them. [06:11] For example, admins might be allowed to override some restrictions. [06:11] https://pastebin.canonical.com/55352/ might work better [06:11] * wallyworld_ looks [06:12] huwshimi: I suspect it will need the eager loading changes I reference done before you can enable the flag :( [06:12] huwshimi: Its very cool to see it being done [06:12] lifeless, huwshimi: It also has an XSS hole or two. [06:12] wallyworld_: But what you have is just as fast despite looking a bit WTF? [06:13] stub: thanks. i'll use that. although the one i pasted also works, but your is easier to grok i guess. [06:13] lifeless: Oh, sorry I mis-understood [06:13] stub: i guess i was trying to avoid the count(*) [06:13] wallyworld_: I'm not sure of how to shoe horn it into Storm [06:13] huwshimi: I was probably unclear [06:13] lifeless: It's all good [06:13] huwshimi: Landing with a flag is fine, I just think its near-certain that enabling it will make things timoeut [06:13] lifeless: Sure [06:13] huwshimi: so you could, if you think I'm wrong try it and possibly save yourself some time [06:13] wallyworld_: If you go for yours, change the UNION to UNION ALL [06:13] stub: i think because yours does a select from... i can get it in [06:13] (IMBW) [06:14] I don't think I am, in this case. [06:14] lifeless: What's involved in making the eager loading changes? [06:14] stub: ok. i'll have a look at it. if it all goes pear shaped, i can always revert to my original sql since the performance is ok [06:14] thanks [06:15] lifeless: Is it something I can take a look at? [06:15] wallyworld_: We are looking at <10ms, so whatever ;) [06:15] stub: cool. the perf figure is what i was after. thanks :-) [06:15] wgrant: i agree about the argument passing thing. so validators are good for invariant conditions [06:16] wallyworld_: It's *presently* invariant. [06:17] wgrant: Where are the XSS holes? This is mostly just existing code, but I'm happy to fix it up a bit [06:17] yes. i don't see this one changing though [06:17] but perhaps it could [06:17] huwshimi: the key function is the one I mentioned in my review [06:17] StevenK: if reason is not None and not (is_new and getComponentsForUpload) [06:17] getPrecachedPersonsFromIDs [06:17] which calls [06:17] _getPrecachedPersons [06:18] the related but separate cacheBrandingForPeople already grabs all three sizes [06:18] so you don't need to touch that [06:19] lifeless: Ah I see [06:19] huwshimi: The URL contains user-controlled content (in the filename), and it's now in the page unescaped. cgi.escape is also no good for attribute values -- it doesn't escape quotes by default. [06:20] huwshimi: I recommend that you use structured() [06:20]