[00:03] wgrant: mumble [00:54] thumper: hi [00:54] hi poolie [00:56] thumper: did you find out about kanbans? [00:56] poolie: yeah... although it seemed to have missed out one of the bugs and I don't know why [00:57] (just reading my scrollback from my holiday) [00:57] neither do i :) [00:57] which bug? and what mode of kanban were you using? [01:04] Project windmill-devel build #140: STILL FAILING in 1 hr 18 min: https://lpci.wedontsleep.org/job/windmill-devel/140/ [01:27] cjohnston: I suspect your link - I see none [02:10] o/ lifeless [02:36] wgrant: I've pushed all of packageupload out of notify() [02:37] Now to fix the tests [02:38] StevenK: Great! [02:38] hi poolie [02:42] wgrant: I had a thought about binary notification too -- We can call notify_spr_less() if all of spr, bprs and customfiles are None. Or we can just fix model/queue to call notify_spr_less() directly [02:45] StevenK: Possibly. [02:45] StevenK: What in the main function still needs the SPR? [02:46] wgrant: def notify()? [02:46] Right. [02:46] It's difficult because the SPR has lots of the data we need. [02:46] eg. Changed-By and Maintainer. [02:46] wgrant: _buildUploadedFilesLists, _buildSummary, and _sendNotification [02:47] Which BPR only has normally because we get it from the changes file... [02:47] but we can possibly get that from the linked SPR. [03:40] Any reviewers in town? [03:42] hi wgrant, stevenk, jtv [03:42] bac, hi? [03:42] hi poolie [03:43] poolie: Hai [03:43] and hi StevenK, while we're at it :) [03:56] NCommander: If you show me a diff or a WIP MP, happy to look it over. [03:59] wgrant: Right, I have nascentupload-announcements.txt passing -- except for the notify_spr_less() call [03:59] wgrant: Since notify_spr_less() wants the file-object, so it can get the filename [04:04] I haven't touched JS in a while and need to get back into it. Can anyone help with some Ajax interaction issues? [04:04] Two questions right off: [04:05] 1. When the client waits for stuff to happen on the server, have we got anything better than dumb regular polling yet? [04:06] 2. What's the current pattern for re-rendering a piece of TAL once stuff happens? [04:10] thumper: I know you're not here, but any spontaneous pearls of wisdom would be most welcome. :) [04:14] lifeless maybe? [04:14] jtv: for what? [04:14] thumper: see above [04:14] "I haven't touched JS in a while" etc. [04:14] jtv: I don't think so [04:14] booo! [04:14] jtv: I believe there is a plan for long-poll [04:14] but nothing is implemented yet AFAIK [04:14] Ah yes, plans. Don't we love those. :) [04:15] :) [04:26] jtv: All we have is dumb regular polling :( [04:26] You can ask for an HTML representation from lazr.restful. [04:26] But in general I think it's better to alter the DOM yourself. [04:26] What exactly are you wanting to do> [04:26] Re-render DSD entries on +localpackagediffs [04:26] …because syncing completes, for instance. [04:27] I think there could be so much to that (especially as changes progress) that it makes more sense to re-render the TAL than to try and mimick the same logic in JS. [04:28] I suppose I could move more into the view so that more of that logic is still done server-side and transferred as data, but then I fear I may end up with, in essence, a custom representation of the HTML. [04:40] wallyworld_: wasn't "4+1" a Vietcong methodology? Approach quickly, strike quickly, disengage quickly, retreat quickly; prepare slowly. [04:42] jtv: maybe it was. perhaps the software world copied :-) [04:43] But… but… copying is a _bad_ thing isn't it? [04:44] At least in Chinese-influenced cultures, everything catchy needs at least one number in it. And there aren't that many catchy numbers. [04:45] "One [x], one [y]." A thousand points of light. The Four Precepts. And so on. === almaisan-away is now known as al-maisan [05:07] wallyworld_: How goes QA for bug #302097? [05:07] <_mup_> Bug #302097: Trying to access rss feeds of a private bug OOPSes

< https://launchpad.net/bugs/302097 > [05:08] wgrant: Seems I can't just call notify_spr_less() in do_reject() :-( [05:08] StevenK: Oh? [05:09] wgrant: So for nascentupload-announcements.txt at least, there are 2 rejections -- one with full content, and one with an incorrect .changes filename [05:09] wgrant: waiting for staging to be updated [05:15] StevenK: Could you QA bug #787904? [05:15] <_mup_> Bug #787904: Copying a package into a derived series fails

< https://launchpad.net/bugs/787904 > [05:16] Oh, it is [05:16] I already know the fix is fine [05:16] Ah, great. Please mark it as such. [05:16] wgrant: Done [05:16] Thanks. [05:17] wgrant: Sorry, I tossed it at ec2 and meant to do so when the qa tagger nagged me [05:21] wallyworld_: I think I just QA'd it on qastaging. [05:22] wallyworld_: Needs feeds.qastaging added to /etc/hosts, and has to be accessed over HTTPS, but it works. [05:22] Are you not sure? :-) [05:22] wgrant: i was told rss feeds don't work on qastaging and that staging is the system to use [05:22] wgrant: Would an RT to get feeds.qastaging added to DNS be a good step? [05:23] wallyworld_: It's not in DNS and needs an Apache reconfig to be complete, but feeds.qastaging.launchpad.net is accessible if you poke it hard enough. [05:23] StevenK: Probably. [05:23] ok :-) [05:23] "The requested bug is private. Feeds do not serve private bugs." [05:23] Looks reasonable. [05:24] i think so. better than a yucky error page [05:24] Hm, ENOLIFELESS [05:24] netsplit [05:25] For over ten minutes? Go go Freenode [05:25] lib/lp/archiveuploader/tests/nascentupload-announcements.txt [05:25] Ran 1 tests with 0 failures and 0 errors in 9.389 seconds. [05:25] \o/ [05:26] How'd you manage that? [05:26] By doing evil things [05:26] That goes without saying. [05:26] changes['_filename'] = changes_file_object.name [05:26] ^ That, mostly [05:26] OK, not that evil. [05:26] :( [05:26] The only thing that cares about the filename is notify_spr_less [05:27] And I can't call it from do_reject, since it has no idea about the callstack [05:27] Anyway, running -vvm soyuz over it now [05:27] Great. [05:27] Then commit, push, and I will examine it and work out what we can do next. [05:27] To denauseatify it. [05:28] After the tests or during? [05:28] After. [05:33] wgrant: hard hard is it to get another field added to a fti and where do i look to see how an existing fti is defined? [05:33] how hard [05:34] wallyworld_: database/schema/fti.py [05:35] Reindexing everything is harder. [05:35] What are you looking to add? [05:35] person . irc nick [05:35] That's a bit harder. [05:35] Since that's in a separate table. [05:35] of course. silly me [05:35] Is the join too expensive? [05:35] There are not many IRC nicks. [05:36] emailaddress is the one that needs optimisation, if anything odes. [05:36] Does that need FTI? [05:36] I mean, IRC nicks? [05:36] i'm just thinking out loud [05:36] jtv: no, i said something stuoid without thinking [05:36] no worries then — we all do that [05:36] Who's up for a JS client-server pre-imp call? [05:37] StevenK: http://paste.ubuntu.com/613055/ (just realized I forgot to put the paste link) [05:38] NCommander: Indeed you did [05:40] NCommander: Making a BPR should make builds for you [05:41] jtv: what are you looking to do? [05:42] wallyworld_: just otp back in a sec [05:42] ack [05:43] NCommander: It looks good -- your imports are all over the place, so I'd suggest running utilities/format-imports over it. You have some spelling errors, and every comment should be a complete sentence -- you're missing some full stops. [05:43] NCommander: Eight lines of self.assertEquals is mind-numbling, can I suggest a for loop? [05:43] wgrant: ^ [05:43] There may be a matcher for that. [05:43] NCommander: Oh, and you import pdb without using it [05:44] Or it may be in james_w's unlanded branch. [05:44] yeah, I think that's in more-matchers [05:45] land it! :-) [05:45] wallyworld_: back. [05:45] NCommander: However, if that test passes and you have the publisher changes finalized, put up an MP! [05:46] Here's what I want to do: you've got a page displaying a bunch of items with somewhat complicated state, and I want live updates for that state. [05:46] james_w: Link me the MP? [05:46] the page stays open for a while? [05:47] Possibly, yes. [05:47] so you want a long poll which we don't currently do :-( [05:47] Ideally, yes. [05:47] But I have what I hope is a pretty efficient poll with some mitigation measures for the load problem. [05:48] i *think* that the option for now would be to set up a js timer and poll yourself [05:48] Right. [05:48] every so often [05:48] NCommander: But I'd suggest making the changes I suggest first, since any reviewer is likely to pick up on the exact same points. [05:48] https://code.launchpad.net/~james-w/launchpad/more-matchers/+merge/32057 [05:48] wallyworld_: and possibly less often for large numbers of objects. [05:48] and i think yuo has some apis which support that [05:48] yui [05:48] wallyworld_: wgrant: that feeds fix [05:49] wallyworld_: wgrant: I bet I can still make it oops [05:49] jtv: you can say "execute this emthod after x seconds" [05:49] without url hacking [05:49] PublishedStateIs along with the matcher that loops over the input and you'd have a test that would make lifeless proud :-) [05:49] wallyworld_: yes, I've seen that. [05:49] lifeless: fire away [05:49] lifeless: Probably. [05:49] I don't care about whether the fix is good. [05:49] I care about whether it breaks stuff. [05:49] wallyworld_: open two tabs on a public bug. In one make it private. In the other follow the rss link. [05:49] jtv: i think you'd just requeue the method after each invocation [05:50] lifeless: You failed. [05:50] wgrant: yes, its deployable. [05:50] lifeless: It redirects. [05:50] lifeless: i accounted for that case :-P [05:50] wallyworld_: yes—that avoids piling up overlapping poll attempts. [05:50] wallyworld_: excellent [05:50] james_w: I suspect that branch needs devel merged in [05:50] lifeless: you don't need two pages. just do an inline js edit of prvacy [05:50] wallyworld_: I only remembered the work around the link rel=... [05:51] so, cool. [05:51] wallyworld_: I was thinking: render each item on the page with a "status cookie" that encodes everything about the item that might change and require it to be re-rendered. To poll, the client sends its item ids with their cookies; the server responds with HTML and fresh cookies for any items that have changed. [05:51] lifeless: thanks for asking though. best to be thorough [05:51] wallyworld_: The cookie would be computed in one place only, so nobody else needs to care what goes into it. [05:52] wallyworld_: indeed! [05:52] jtv: sounds like a good solution. could you use an etag like lazr restful? [05:52] jtv: that essentially is the same problem [05:52] sounds heinous http://www.theregister.co.uk/2011/05/25/intel_hybrid_cloud_appup_service/ [05:52] wallyworld_: I had no idea lazr.restful did etags! Yes, that'd be an ideal place for the cookie to go. [05:53] jtv: i *think* lazr restful has js for computing the hash/cookie you could perhaps look to reuse too [05:53] james_w: No fair having 5 dependant branches deep [05:54] wallyworld_: jtv: etag can only talk about one object at a time [05:54] which drives lots of round trips [05:54] wallyworld_: wouldn't that by nature have to be based on the HTML? [05:54] whats the problem with just doing the UI in js ? [05:54] if the user doesn't have js, they won't get updates anyway [05:54] But they want to see the same items. [05:54] lifeless: i didn't realise that etag limitation [05:55] wallyworld_: etags apply to an entity from a url [05:55] And I stupidly thought wallyworld_ was saying lazr.restful had it deeper down where you could do it per item. :) [05:56] So scratch etags. [05:56] jtv: the recipes UI does updates using the html representation [05:56] jtv: lifeless: i wasn't sure about how it was implemented. just pointing out it had them in some form in case they were useful for this, but not so [05:56] sorry :-) [05:56] * lifeless doesn't understand the problem yet [05:57] lifeless: jtv wants a long poll type scenario to asynchronously update a page i think [05:57] Yes, items on a page. [05:57] oh, I get that bit [05:57] I don't understand why the existing answers don't work here [05:57] I asked earlier about existing answers. [05:58] we don't currently have anything like that atm in lp do we? [05:58] we do page updates in js all the time [05:58] wgrant: http://pastebin.ubuntu.com/613059/ [05:58] pickers, subscriptions, recipes, bugs [05:58] lifeless: yes, but that;s in response to a user action [05:58] wallyworld_: so? [05:58] not a server side event [05:59] this is a different scenario afaiu [05:59] wallyworld_: fire a background poll, it notifies when an object is updated, call into the local js [05:59] lifeless: that's not a very helpful description. [05:59] lifeless: yes, that's the solution that is being canvased [06:00] wgrant: A lot of the failures are due to tests tossing in StringIO for changes_file_object, which doesn't have a .name [06:00] StevenK: :( [06:00] we are discussing the implementaion options [06:00] wallyworld_: jtv: what I'm saying is that finding out about the change, and updating the UI in response, are largely separate problems [06:01] Which is why I posed them as separate questions from the start. [06:01] lifeless: You may want to read about 2 hours back, search for your name. [06:01] We're not confused on that point. [06:01] ok [06:01] sure. we are more discussing the "finding out about the changes" bit i think [06:01] wgrant: And getattr(changes_file_object, 'name') makes me cry [06:01] i think jtv has what seems like a reasonable solution [06:02] I was sketching out the solution I had in mind, based on the answers I got earlier about existing solutions. [06:02] The answers were: no, we have nothing better than polling yet; and even though we generally prefer to manipulate the DOM directly for object changes, it is possible to request HTML from lazr.restful. [06:03] jtv: so, I'm confused again [06:03] you seem to be describing a sort of complex page-state-diff algorithm. [06:03] Not particularly complex, when you get right down to it; just "send me any updates" but with a single roundtrip. [06:04] Whats the concrete use case here? Is it really 'any part of the page has changed', or is it 'show the status of jobs for this DSD' ? [06:04] It's showing the status of a DSD—which may include the status of the DSD itself, but also different types of jobs that may be in progress for it. [06:05] some thoughts: [06:05] - detecting a change generally involves as much as work as rendering the object again [06:06] - the server knows what would be on the page so why would the client supply stuff to check? [06:06] Why does the server know what would be on the page? [06:07] because the server renders pages [06:07] It knows _when_ it renders the page. But after that, it blissfully forgets. [06:08] yes, But it can redetermine that trivially as needed [06:08] Excuse me? [06:09] How can it do that? [06:09] render the page? [06:09] That's a different page. [06:10] It's got different DSDs on it. [06:10] That's not the page the user is looking at. [06:10] if there are 100 clients connected at different times, how can the server know what's on each and every client? that's not scalable [06:10] Exactly. [06:11] wallyworld_: so, in the future, ajax subscription does *exactly that* [06:11] wallyworld_: but thats not today. [06:12] jtv: why does it have different DSDs ? [06:12] even in that case though, the server would just push out new state for a domain object and it wouldn't know what's already been rendered [06:12] Because it's a filtered view. DSDs change status. [06:12] it would be up to each client to "do the right thing" [06:12] A DSD gets resolved, it falls off the page. [06:12] A package change may push a new DSD onto the page. [06:14] lifeless: storm doesn't support sql case statements does it? [06:14] wallyworld_: sure does [06:14] oh cool. [06:15] * wallyworld_ looks for it [06:16] jtv: then perhaps just request that the dsds present get re-rendered ? [06:16] StevenK: I don't have changes for it yet, I got tied up withspecs so I only finished the test suite today to see if its sane, as I'm unsure if this is the proper solution. I kinda felt it mightbe clearly to maintain publishing for SUPERSEDED items ... [06:17] lifeless: that's going to be very similar, except with unnecessary rendering and a bigger payload. Where's the gain? [06:17] NCommander: Hopefully, the test fails, then? [06:17] jtv: its simpler [06:18] jtv: you could just return domain objects, but I got the impression you didn't want to do the ui in js [06:18] That's right, I didn't want to do the UI in JS. [06:18] though domain objects are often *more* expensive to generate than rendering a page. [06:19] I don't think rending all items takes away much complexity: instead of keeping track of {id: cookie} we'd need to keep track of [id], and that's about all I see changing. [06:19] StevenK: indeed since everything expect the i386 binary should still be PUBLISHED [06:19] *rendering [06:19] lifeless: sorry. i can't see any native support. i know i can inject the required sql into the find(), i was looking for a built in Expr subclass [06:19] wgrant: The first 14 errors fixed [06:19] StevenK: That was easy. [06:20] 15 [06:20] test_realiseUpload_for_delayed_copies is still broken [06:20] :( [06:20] What is its complaint? [06:20] jtv: it means you don't need a cookie, generation of which is usually expensive [in all similar systems I've sen] [06:20] wgrant: It was expecting an annoucement mail [06:21] Hmm [06:21] wgrant: Let me commit and push [06:21] wallyworld_: hmm, sorry :(. Would you like some pointers on adding a case statement expr ? [06:21] % bzr di | wc -l [06:21] 413 [06:21] I am such a bad person [06:21] lifeless: even if generating the cookie is as expensive as rendering an item, big win. [06:22] lifeless: i should be ok to do that. i'm not sure if i need it. just experimenting at the moment [06:22] lifeless: the cookies can be generated in batch. [06:22] jtv: rendering can be done in a batch too? [06:22] Are you asking or saying? [06:23] both I guess [06:24] What I'm saying is, I don't see why generating the cookie and in most cases doing no rendering at all and sending basically no data across the wire is going to be anywhere near as expensive as re-rendering all items on the page and sending them to the client. [06:24] Also, I think saw IE running JS once. [06:24] i/win 21 [06:24] There's no longer a monitor attached to that computer, so I don't know how well it does. But not replacing items on a page that don't need touching may also be a plus. [06:26] so, I've fixed something like 10+ timeouts related to generating API objects now [06:26] How does that follow? [06:26] the API definition for an object includes grabbing *all* the adacent objects and calculating their urls [06:27] Good point: another reason to do it the way I had in mind I guess. [06:28] the reason I suggested rendering is that rendering is *targeted* [06:28] They're going to have to do something with the internet here, so I'll be offline for a bit. [06:28] k [06:28] pick this up in a bit [06:28] Hm. [06:28] Did someone break PQM? [06:29] Nobody fixed the conflict, but it has stopped complaining. [06:29] ah, buildbot failure. [06:29] "Set changed size during iteration" [06:29] WYay [06:32] wallyworld_: Found any remarkable insights from your ranking experiments? [06:33] wgrant: there is a relatively simple change that will improve the ordering of results while not changing the query too much. the query is quite complex so i'm loath to mess with it too much. i'm also experimenting a bit with a few other things [06:35] wallyworld_: You mean actually using the rank instead of throwing it away and ordering by displayname? [06:35] wgrant: The diff should be updated. [06:35] wgrant: yes, in a nutshell [06:36] wallyworld_: Good luck doing that while not changing the query too much :( [06:36] exact matches to name (lp id) > exact matches to displayname > exact matches to irc nick > partial matches to email [06:36] What's an exact match to displayname? [06:37] as in a full string match? [06:37] wgrant: i've got a solution which i expect wiol work but i'm messing with other things before i implement [06:37] yes. [06:37] wallyworld_: We also need to consider affiliation in ranking, or maybe just filtering... and I'm not sure how trusted affiliation wants to be. [06:38] Hmm. [06:38] I'd prefer an FTI rank over a string match. [06:38] Otherwise you have to get it just right. [06:38] wgrant: we discussed on the call this morning not doing affilliation in rankings first up [06:38] if you're checking displayname be sure to add a db index for it [06:38] will do [06:40] wgrant: wrt the fti match vs string match. the fti match is still being done too. but if the user does type in an exact match, that should take precedence [06:40] trust fti to do that [06:40] wallyworld_: But the FTI rank is not taken into account. [06:40] Right. [06:40] If we take FTI rank into account, we'll get full matches and partial matches, in the right order. [06:40] lifeless: the fti will do it but the ranking will be off [06:41] we want exact matches ranked higher [06:41] lifeless: "BugTask.sourcepackagename = ss4.sourcepackagename OR ss4.sourcepackagename IS NULL" is better written as "BugTask.sourcepackagename IS NOT DISTINCT FROM ss4.sourcepackagename" [06:41] stub: doh of course... thanks [06:41] wallyworld_: FTI rank should do that. [06:42] wgrant: not when fti rank is 0..1 and we use 100 for exact string match on name and 10 for irc nick [06:42] wallyworld_: Better to tweak those than work around FTI, surely. [06:43] i could look at adjusting the other raqnking values currently in use [06:43] but there must have been a reason for doing it the way it is [06:43] You've been around Launchpad for long enough to know that's not necessarily true :) [06:43] does our local db on our laptops have fti enabled? [06:44] Yes. [06:44] cool. [06:44] launchpad_dogfood=# SELECT Person.name, Person.displayname FROM person ORDER BY rank(fti, ftq('william grant')) DESC LIMIT 10; [06:44] name | displayname [06:44] -----------------+------------------------------ [06:44] me-williamgrant | William Grant [06:44] wgrant | William Grant [06:44] me+testing | William Grant (test account) [06:44] shuilongzi | Basil Bakhtin [06:44] We just have to balance them with exact name matches. [06:44] Those results aren't bad. [06:44] And IRC nick matches. [06:45] wgrant: wrt the exidting ranking. i realise that it may not be "correct" but i do want to try and understand why it was done the way it was in case there's some corner casse or something i'm missing [06:46] Those constants were just a guess that have never been tweaked IIRC [06:46] Just there to get all exact matches above fti fuzzy matches [06:46] stub: makes sense. wonder why if fti ranks from 0..1 that string matches used 100 though [06:47] if fti gives a ranking of 1 for exact match, then wou;dn't we have used 1 for for explicit exact matches? [06:48] wallyworld_: Exact match in a particular field. fti match does stemming, and searches other fields too probably, but will give the same score for 'wgrant' as it would 'wgrants' [06:48] And miss you entirely if your name happens to be a stopword. [06:49] Try a search for 'Ng' for instance... [06:49] fti is setup for english word searches, and names don't really fit that. [06:51] ok. thank for the explanation. sounds like we need to explicitly do any exact string matches we require. and there's none for displayanme yet [06:51] * wallyworld_ has to go to soccer training. back later [06:52] wallyworld_: We likely need an index created to do exact matches fast and case insensitive. Are exact matches on name useful though? Or do you really want a substring match (slow without further work...) [06:53] stub: i think they are useful. if you know you want to find "Joe Bloggs" you should be able to type that. you may not know his lp nick or email [06:54] i don't think we need substring. fti will cover that? just an index for exact match [06:54] for displayname [06:55] i'm assuming there is already an index for person.name [06:55] * wallyworld_ really has to go now [06:55] stub: Oh yeah, person.fti doesn't have an index on production. [06:55] stub: We discovered this yesterday. [07:06] wgrant: New list: http://pastebin.ubuntu.com/613073/ [07:07] StevenK: What's up with soyuz-set-of-uploads? [07:08] wgrant: distroseriesqueue.txt [07:08] Sigh [07:09] wgrant: distroseriesqueue.txt is complaining that spr.name is None for an announcement mail [07:09] :( [07:09] spr.name is None, or spr is None? [07:09] The latter, sorry [07:09] Right. [07:09] We probably have to infer it from the BPR or something. [07:10] So, if spr: name = spr.name, if bprs: bprs.... [07:11] wgrant: The soyuz-set-of-uploads failure is odd: http://pastebin.ubuntu.com/613074/ [07:16] StevenK: Why's it not emailing name16 any more? :( [07:17] It should be? [07:17] It was before. === al-maisan is now known as almaisan-away [07:18] wgrant: distroseriesqueue.txt fixed [07:21] Ah, it's a bug in notify_spr_less [07:23]