[00:02] lifeless, well, on a happier note, can i get a kind of architecture review on markdown? [00:02] sure === poolie_ is now known as poolie [00:03] so [00:03] it will be behind a flag [00:04] istm it might be good to make the rendering be a timeline action? [00:04] or would that be redundant with just profiling it [00:04] if its really 200ms per text [00:04] it could possibly go in memcached but perhaps that could be done later [00:04] I think we'll have issues [00:04] ugh no, memcached is very much not the right hammer here [00:05] so structurally whats going on is that you're generating html out of some content, which is exactly what the template engine does === wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wgrant | Critical bugtasks: 292 [00:05] we're then embedding that result literally in the larger template [00:05] Hmm? memcached would be reasonable here, I think. [00:06] the caching could be done in the template [00:06] s//controlled by [00:06] if it really is 200ms that's almost infeasible [00:06] wgrant: one word: bug comments [00:06] lifeless: Myes? [00:06] ow [00:07] wgrant: a situation where we timed out *because* we used memcache [00:07] wgrant: and a place where markdown is [eventually] desired [00:07] Right, but it was timing out because the pages sucked, and there was absolutely minimal benefit to using memcache there. [00:08] It would need testing, but I think it could be useful if implemented properly. [00:08] so we want to use memcache on things that are both expensive to generate and able to be generated once for all viewers [00:08] And if the page didn't suck already. [00:08] but more so we want the first hit to be fast [00:08] wgrant, tangentially, re your email bug, i will add a test that forces an email.Header into it and check its handled correctly [00:08] so we either want preloading (with change updates) or we want rendering to be so fast that memcache communication overhead is slower. [00:09] poolie: you could timeline it yes; if there are questions over its performance thats a great way to ensure we have data. [00:09] so if it's really that slow [00:09] that might be a problem for applying it to all bug comments [00:09] poolie: as there seem to be questions around it, perhaps you want to pull on that thread a little first [00:09] there is python-markdown2 which claims to be an unspecified amount faster [00:09] poolie: so that we don't have markdown for home pages and then get stuck on bug comments [00:09] exactly [00:10] poolie: I'm fine with landing with a restrictive feature flag to find out early adopter issues [00:10] right [00:10] performance to some extent we can test offline [00:11] there is also this thing that api clients will now see markdown in the results [00:11] i think that's ok [00:11] it's close enough to text [00:11] if they eventually want to start rendering it, they can [00:11] so will folk not behind the feature flag [00:11] right [00:11] so people probably don't want to go too crazy til it's widespread [00:12] I think timelining it for now is a good idea [00:12] till the issues/concerns are put to rest [00:12] oh hang on [00:13] rvb's measurement is actually that it's 240µs. [00:14] isn't it? [00:16] yep [00:16] that's not so bad then [00:16] but, worth testing [00:16] no, its not [00:16] the timeit CLI reports time per loop [00:17] the timeit python API reports total time [00:17] but each iteration of the loop formats it 1000 times [00:17] ah [00:17] let me have a play [00:17] ./bin/py -m timeit -vv -s 's="a *b*\n" ; import markdown; md = markdown.Markdown(safe_mode="escape",extensions=[ "tables",])' 'md.convert(s);' [00:17] 1000 loops, best of 3: 231.1 usec per loop [00:17] ./bin/py -m timeit -vv -s 's="a *b*\n" *100; import markdown; md = markdown.Markdown(safe_mode="escape",extensions=[ "tables",])' 'md.convert(s);' [00:18] 100 loops, best of 3: 7.816 msec per loop [00:18] btw, choosing the packaged version is irrelevant to LP: we need things on lucid, and eggs are much more reliable than debian packages (because of version migration) [00:18] yep [00:18] i have it as an egg [00:18] was just using 'packaged' as a very rough metric for 'supported/popular/worthwhile' [00:18] also kind of handy for things like this [00:19] ok, yes, <1 ms is good [00:19] don't timeline it [00:19] poolie: you know about bin/py ? [00:19] bin/py -m timeit ... [00:20] see the lines i pasted? [00:20] heh, I do know [00:20] *now* [00:20] * lifeless slaps his fingers [00:20] :) is ok [00:20] the scaling on longer strings is a little worrying [00:21] we would have trouble at 8ms per bug comment, for instance [00:21] ok so if we say a large comment has about 100 lines of text, it's about 28.69ms to convert it [00:21] hm, with no actual markdown in it, 100 lines, 1.798ms [00:21] so 100 comments is 2.8 seconds, which is 50% of the budget for an entire load [00:22] anyhow, no Big Red Flags [00:22] one 100 line code block, 0.4ms [00:22] strangely [00:22] it needs some love I guess [00:22] arguably typical 10 line comment, 0.3ms [00:22] it might have some pathological cases that can DOS us [00:22] yeah [00:22] OTOH one laptop can DOS us, so not goig to worry about this yet [00:23] especially because the parsing is regexps [00:23] wgrant: one word: bug comments [00:23] can i buy another word? [00:23] timeouts :) [00:23] oh i see [00:23] yeah [00:23] was going to say that, i think [00:23] this is the "you eventually fall off a cliff" case? [00:24] ⸘that⸘ [00:24] github run pygments but clearly limit the highlighting of any segment to say 2s [00:24] would be nice to have that sort of thing [00:24] (also for pygments...) [00:24] poolie: no, it was that we spent more time in memcache chatter than in rendering the comments; we also retrieved them from the DB no matter what [00:24] yes, that can hook in [00:24] huh [00:24] poolie: very poor interactions between late evaluation and a chatty protocol [00:24] so, the other approach is to do it on the client side [00:25] poolie: nowadays memcache is just turned off, and things are much faster. [00:25] which possibly makes rendering the page actually easier on the server, if we can reliably detect to just send json not html [00:41] Someone want to review https://code.launchpad.net/~wgrant/launchpad/observer-merging/+merge/82952? [00:42] wgrant: Line 68 of the diff -- why do you bother pulling in store? [00:43] StevenK: Because I forgot to remove that line when I replaced the check just now. [00:43] Haha [00:43] wgrant: That's my only niggle, r=me [00:43] IAPGS.get only came into existence late last night. [00:43] So I previously queried directly. [00:43] Thanks. [00:43] is http://appbuntu.com a legit site or a spammer ? [00:44] lifeless: NXDOMAIN ? [00:45] WFM [00:45] StevenK: get a better DNS resolver ? [00:45] Oh, *buntu*, not *ubuntu* [00:45] Transcribe error [00:45] UI fail :P [00:46] Well, I didn't click the link, I typed it out from memory so I could whois it [00:46] And my memory fails [00:46] Often [00:46] * StevenK waits for lifeless to say "Frequently" [00:47] there is a link to http://appbuntu.com/2011/04/mengapa-ubuntu-menggunakan-launchpad/ in the blog, marked as spam but it doesn't look like spam to me [00:47] trying to determine if it is / isn't [00:47] StevenK: I did but you've clearly forgotten :P [00:47] I was about to suggest wgrant, but I suspect wgrant is too young for Gilbert & Sullivan [00:48] there was also an andi comment, but francis has address that separately, so I just deleted it [01:21] lifeless, memcached is just off across the board? [01:21] in a flag i think? [01:22] more or less [01:22] I need to go through and rip out the inappropriate uses of it [01:22] hm [01:22] the rules ui could do with some love too [01:23] any new use needs rather deep thought about what its actually saving [01:23] there is currently one reasonable use of memcache in LP - the front page blog contents [01:23] the rest have deep flaws (due to the interaction between views, eager loading and late evaluation), as well as various cachability rule interactions [01:24] do you have a rule of thumb for how long it takes to query it? [01:24] between 1 and 5ms [01:24] sometimes up to 12ms [01:27] lifeless: TBH, since we aren't using it, I feel we should cache the blog contents differently and rip it out [01:27] StevenK: I'm open to that. [01:27] StevenK: OTOH it is a reasonable cache engine and I wouldn't want to reinvent a cache just because we haven't got much use for it [01:29] possibly comments could do it reasonably well [01:29] i mean, markdown could use it [01:29] if the rendering was independent of the request (including the user)\ [01:29] and keyed off the place the text comes from, eg comment N on bug M [01:30] anyhow, later [02:06] one of the problems with the tal:cache stuff (it seems to me) is that it doesn't let you use multi_get at all [02:09] mwhudson: I've been assuming that's the entire performance problem we saw, yes. [02:10] Making a thousand round-trips is not cheap, just like with postgres. [02:10] mwhudson, meaning one api call that gets many results? [02:10] s/api/network [02:10] poolie: yes [02:11] the usual 'don't do potato programming' sort of thing [02:11] sure [02:11] it seems like in theory that could be hooked in to tal [02:11] depending on what it exposes [02:12] It would be nice if we could do multi-stage renders :/ [02:12] you'd have to defer rendering parts of the template, or walk the template twice or something wouldn't you? [02:12] But TAL doesn't make that easy. [02:13] So we would probably have to do something like the current preloading. [02:13] Where we work out what we need beforehand, and poke it all into the caches. [02:14] poolie: Not sure if you saw my note about adding a test for the tag ordering (I had to rush offf shortly afterwards). Just wanted to check if it looked ok (line 59 onwards: https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689) [02:14] hi huw [02:14] Rather than the probably more sensible setup of partially rendering the template, finding all the markdown that needs to be rendered, pulling stuff out of the cache, rendering what's left, filling in the template. [02:14] poolie: Hey [02:15] huwshimi, can you look at / think about the markdown branch too [02:15] i don't know if it especially needs anything from you [02:15] poolie: No problems [02:15] is that screenshot up to date? [02:16] wgrant: do you know a template engine that lets you do that sort of thing? [02:16] mwhudson: No [02:17] poolie: That screenshot is up to date, yes [02:17] poolie: This is the branch: lp:~mbp/launchpad/391780-markdown ? [02:17] yep [02:17] the test looks good to me [02:17] mm [02:18] you could add a comment specling out the meaning [02:18] ie that the most popular tags come firstn [02:18] perhaps it's obvious [02:18] poolie: No, you're right [02:18] poolie: Thanks [02:19] poolie: thanks for the link and update in the bug. That's a different place than I was looking.working with. [02:20] I had been looking at https://bugs.launchpad.net/launchpad/+bug/814696 and the link "Diff: 45 lines..." [02:20] <_mup_> Bug #814696: Link to show inline diffs in merge proposals should be green < https://launchpad.net/bugs/814696 > [02:20] mwhudson: wgrant: poolie: also that we don't want to do DB lookups for stuff we're then going to get from memcache. [02:20] mwhudson: wgrant: poolie: the whole template-view interaction is brittle here [02:20] yeah [02:20] Yeah. [02:20] Particularly since the BugMessages are actually subviews. [02:20] i've been very suspicious of passing anything other than dumb data to templates [02:20] Doing a pipelined render would be much nicer, but no :( [02:21] s/been/become/ [02:21] ok [02:21] we can worry about it later [02:21] or not [02:21] :) [02:21] huwshimi, i'm not a very reliable reviewer for the js stuff [02:22] poolie: No problems. I spent quite a bit of time chatting to wallyworld about that anyway [02:22] aside from that it looks good [02:22] i'm keen to see the results live [02:22] good on you for deleting those horrible doctests [02:22] have a badge :) [02:23] what is the actual rule, if any, for tal indentation? [02:23] Nothing special for TAL. You mean XML/HTML? [02:24] yes [02:25] huwshimi: any reason not to make the taglist stuff a module with ATTR, initialize, etc? [02:25] huwshimi: CSS is somewhat broken. [02:25] huwshimi: https://bugs.qastaging.launchpad.net/launchpad/+bug/1234 [02:25] <_mup_> Bug #1234: Gina is an unmaintainable mess of command line options, environment variables and shell scripts < https://launchpad.net/bugs/1234 > [02:25] huwshimi: status/importance are blue [02:26] huwshimi, can i see your current theme things some time? [02:27] rick_h_: I'm not quite sure what you mean, care to elaborate? [02:27] wgrant: ick! [02:28] poolie: Sure, I haven't had a chance to work on it recently, but I'll try and push out a first stage to show people soon [02:28] huwshimi: ok so you declared a namespace, but normally we seem to be using YUI modules, extending base or some decendent of that. For instance, see lp/app/javascript/picker/picker.js [02:28] mwhudson: excellent, the meme is spreading [02:28] see http://yuilibrary.com/yui/docs/base/ for the YUI docs of the pattern [02:29] so you'd end up with more var tagcloud = new TagCloud({any: settings}); [02:30] it allows for others to extend and customize later, more OOP version of things [02:30] rick_h_: No reason other than ignorance [02:30] huwshimi: ok, sorry, a bit new. But if you wanted review I'd definetely say to head down that route. [02:31] overlay for instance is a good example [02:31] there's a base, and several extentions that are used [02:31] extensions that is [02:31] huwshimi: Ah, the stylesheets are combined in the wrong order [02:32] rick_h_: Thanks I'll take a look. This was mostly me pulling the code out of a template and adding a few things/getting it working again, so I didn't really put enough thought into what I was doing :) [02:32] huwshimi: totally understand, been there done that [02:32] typography.css is included later, and it sets link colours that override colours.css [02:32] rick_h_: But, I'll take another look [02:32] huwshimi: Should I roll this back? [02:32] Or do you want to fix quickly? [02:32] awesome, let me know if you need a hand with anything and I'll be happy to help in the morning [02:32] wgrant: I can fix quickly [02:32] rick_h_: Thankyou [02:33] * rick_h_ is heading towards bed here [02:33] huwshimi: Needs to be landed in the next three hours, so not completely urgent. [02:33] wgrant: That's ok, I'm on it [02:34] Thanks. [02:43] wgrant: What should I do with these changes? Commit to the same branch and lp-land? [02:44] huwshimi: Ideally customising the commit message in lp-land to something sensible, but yeah, that sounds reasonable. [02:47] oh breakages [02:47] huwshimi: rick_h_: with the yui module vs namespace thing, what is used depend on the heritage of the code. if it came from lazr-js, then it tends to extend the yui component framework. if it was originally lp code, then it tended to be more procedural and just use functions scoped using namespaces [02:47] lifeless, is there a timeline matcher? [02:48] poolie: not really [02:48] wallyworld___, rick_h_: in this case, I guess I would need to make it a little more generic, ie, not have a hardcoded url etc. right? [02:48] poolie: closest is the underlying matcher of BrowsesWithQueryCount which doesn't use the timeline, it uses a custom storm filter instead [02:48] huwshimi: rick_h_: so the preference moving forward is to use more of the yui component and widget framework for new stuff [02:48] wallyworld___: Is the component still appropriate for this? [02:48] but for simple refactoring, it think "whatever works" is fine [02:49] huwshimi: so, we are talking about the bug tags stuff? [02:49] wallyworld___: Yeah [02:49] -p 5432 [02:49] Blah [02:50] huwshimi: personally, i think the way it was done was fine - it was refactoring what was there already and was a very simple bit of code [02:51] wallyworld___: If you say that I'm not going to fix it :) [02:52] Mostly cause I would like to finish with this bracnh [02:52] but very happy to fix it if it means doing it right [02:53] huwshimi: i'm not sure what value it would add for the effort involved. but as a learning exercise.... [02:55] wallyworld___: Maybe I'll consider it a lesson learned for next time :) [02:56] huwshimi: sure. fwiw, as i said, i would have done it the same way, since it was refactoring existing code and was a simple change [02:57] wallyworld___: thanks for the info [02:57] the stuff I've been doing and seen most of is the YUI module stuff [02:58] so it just struck me as odd when I peeked at it [02:59] rick_h_: yes, i suspect you've been exposed to either the new distro pages or the former standalone lazr-js library that we sucked into the lp code base [03:00] rick_h_: then there's all the other lp javascript, which is mainly the procedural stuff [03:00] rick_h_: it's a bit of a mess, but we do want to evolve to use the yui component and widget framework for significant new stuff [03:00] yea, overlay, inline editor, etc. [03:01] yea, I had heard that, which I +1 :) [03:01] I'm a bit of a YUI fanboi so happy to be working with this stuff [03:01] rick_h_: yep, that's all the lazr-js stuff which was written to be a reusable js lib and formerly packaged separate to lp [03:02] rick_h_: then no-one else was using lazr-js and it was too hard to use separately so we just sucked in all the source code [03:02] into the lp tree [03:02] ah, yea I saw some bzr log stuff for "merge in lazr" [03:02] when I tried to chase some older history items for it [03:03] yeah, we did that at the last epic, in july [03:03] 'bzr lp-land' errors with this: http://paste.ubuntu.com/745525/ any thoughts? [03:04] rick_h_: i'm having "fun" today trying to upgrade to yui 3.4.1, but there's errors thrown by the yui.js code. i may perhaps ping to tomorrow about it. very frustrating [03:04] huwshimi: haven't seen that before [03:04] grrr [03:05] your gnome session isn't [03:05] wallyworld___: yuck, devel is currently on .0 then? [03:05] rick_h_: use currently are using 3.3 [03:05] I really regret not making a fuss about the desktop integration thing [03:05] we've had -lots- of fallout [03:05] s/use/we [03:05] oh, that's right, deryck mentioned skipped 3.4 to jump to 3.5 1st qrt [03:06] qtr that is [03:06] rick_h_: oh, i didn't know that. in that case i may drop it and stick with 3.3 till then [03:06] yea, there's a release schedule for 3.5 and deryck mentioned that would be the migration target [03:06] huwshimi: are you logged into gnome? [03:06] lifeless: That's strange, I'm not running this from ssh or anything [03:06] rick_h_: there's been a fairly major re-packaging in 3.3 -> 3.4 [03:06] so I'd not kill yourself on it [03:06] huwshimi: or ssh'd into the machine ? [03:06] lifeless: Yeah [03:06] huh [03:06] wallyworld___: yea, I was checking out their MVC stuff which moved a few things around [03:06] that is odd [03:07] did unity crash on you or something ? [03:07] rick_h_: the reason for doing it is that i wanted to start looking to use the mvc stuff [03:07] lifeless: Things have been a bit weird today, maybe a reboot is in order [03:07] wallyworld___: yea, I'm porting backbone to the YUI MVC on my bookmark app on the side [03:07] rick_h_: since we are embarking on a major new piece of development (the managing disclosure ui) and i wanted to look at using that [03:08] brb [03:08] wallyworld___: ah, yea it's definitely something to plan from the start from [03:08] rick_h_: so i'm still tempted to try to get 3.4.1 working [03:08] to catch the new way of doing the events and such [03:08] yep [03:09] rick_h_: so you used 3.4.1? [03:09] well, let me know how it goes. I'd be curious to see it [03:10] wallyworld___: yea, 3.4.1 [03:10] but I started out with that, so not sure what the migration points are from 3.3, I've been porting from jquery + backbone + historyjs [03:11] rick_h_: we have a "unique" way of packaging yui which may be part of the issue, not sure [03:11] yea, I've not gotten my head all the way around all of that yet [03:11] references and use of LEP() or something vs YUI() [03:11] rick_h_: we basically cat selected yui js files to a big launchpad.js file [03:12] yea, I've gotten that part and added some things to that [03:13] hopefully have my auto resizing textarea widget done tomorrow and in there soon :) [03:14] rick_h_: so 3.4.1 has split stuff out and re-done the options for seeding yui etc. a trivial change to yui-deps.py to pull in all the new files seems to generate a reasonable launchpad.js but there's an error inside yui when pages load :-( [03:14] lifeless: It worked, then I discovered that I hadn't pushed my change but now can't run lp-land again. [03:14] (it worked, after a restart [03:14] ) [03:14] huwshimi: same error? [03:14] lifeless: Sorry, yes [03:15] rick_h_: cool about the text area widget. great to have that fixed [03:15] lifeless: Can I restart my keyring or something? [03:15] wallyworld___: hmm, I can't find that file [03:15] but makes sense [03:15] huwshimi: possibly, might be listnening on dbus [03:15] rick_h_: utilities/yui-deps.py [03:15] huwshimi: have alook for a .*keyring.* process [03:16] rick_h_: it's run by make [03:16] bah, helps if I tell it to do it recursively [03:16] lifeless: gnome-keyring-daemon? [03:16] sounds like it [03:17] if its dbus activated, killing it should be pretty transparent [03:17] I don't know if it is [03:17] rick_h_: it print out all the js files we want to use and these are cat'ed to launchpad.js [03:17] lifeless: I'll give it a try [03:17] wallyworld___: gotcha, yea not peeked at that. [03:18] http://yuilibrary.com/forum/viewtopic.php?p=27897 [03:18] lifeless: Ah, that seems to have worked. Thanks :) [03:19] that seems sucky, but maybe? [03:19] lol "should be fixed in 3.5.0" [03:19] rick_h_: so you would think that being conservative and grabbing everything and yui-base and using that would work but sadly not. it fails inside YArray.hash :-( [03:19] wallyworld___: well I'd hand modify the file and start small [03:20] just try to get yui concat'd to launchpad.js and get it to load on some url [03:20] and then start adding a module/two at a time [03:21] oh failure [03:21] Failure? [03:21] rick_h_: yeah, i was hoping not to have to resort to doing stuff like that. i may just leave it as i've got lots of other stuff to get done :-( [03:22] wallyworld_: yea, sorry. There's so much to the build of this stuff I think the only sane way is to break it down step by step [03:22] wgrant: Still trying to get my keyring stuff happening [03:22] so I can lp-land that change [03:22] might at least try a JS test file, .html [03:22] huwshimi: :( [03:22] and see if you can just make the few files in one of those work with yui 3.4.1 [03:22] just load it into the html from the cdn [03:23] and see if there's anything in the modules we're writing that's harmful (like the forum post on using requires: ...) [03:23] rick_h_: i'll see what i can do in the time i have left. [03:24] yea, have fun! off to bed for real this time. [03:24] rick_h_: thanks. ttyl [03:24] Night rick_h_. [03:26] wgrant: Ok, done, that change is running through pqm now [03:28] huwshimi: Thanks. [03:29] wgrant: No problems [03:39] I don't know how but I appear to have completely broken my ssh/keyring on my computer [03:43] huwshimi: What's the issue? [03:44] StevenK: Well, now when I try and ec2 land I get "ec2: ERROR: You must have an ssh agent running with keys installed that will allow the script to access Launchpad and get your branch." [03:44] What does ssh-add -l say? [03:45] StevenK: "Could not open a connection to your authentication agent." [03:45] ssh-agent should be started on login [03:46] StevenK: ssh-agent gives me output [03:46] It will [03:46] what I assume to be normal looking output anyway [03:46] No matter if it is running or not [03:47] StevenK: oh, right, that's helpful [03:47] StevenK: How do I get it to run then? [03:47] huwshimi: So you can logout and back in and then see if ssh-add -l behaves or you can fix it for one terminal [03:48] StevenK: I'll try the log out first [03:48] brb [03:50] StevenK: Ok, that fixed it this time. Thankyou! [03:50] I restarted a few minutes ago, but that didn't work [03:50] maybe it needed a logout rather than a restart [04:00] wgrant: Can haz review? https://code.launchpad.net/~stevenk/launchpad/use-userHasPriviledges/+merge/82967 [04:01] StevenK: You can't spell privileged. [04:02] Not "privileges" [04:02] s/Not/Nor/ [04:03] StevenK: Why isn't userHasPrivileges implemented in terms of userHasPrivilegesContext? [04:04] wgrant: Because I couldn't think of a clean way to do it. Suggestions welcome. [04:05] StevenK: Make isOneOfDrivers work on distroseries/productseries if it doesn't already. [04:06]