[00:32] wgrant: storebuildinfo, really? [00:32] (bug 828017) [00:32] <_mup_> Bug #828017: UnparsableDependencies raised by retry-depwait script < https://launchpad.net/bugs/828017 > [00:32] wgrant: for my education, what makes this a dup?> [00:33] lifeless: Check the referenced builds. [00:33] lifeless: They don't have a builder, or datefinished, or... [00:33] wgrant: are you saying that if they did, it wouldn't raise that oops ? [00:34] It would not, since the information would be set. [00:34] lifeless: The error is raised when the dependencies are unparseable, which means either lp-buildd is buggy and returning bad data, or LP isn't handling it properly. [00:34] In this case, storeBuildInfo isn't being committed. [00:34] None of the metadata is stored. [00:34] ok, so they don't have crap in their headers, for instance. [00:34] No. [00:34] The data is just None. [00:40] OK, JS hackers, what's the Launchpad way of moving the JS in this template into its own file? http://bazaar.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/view/head:/lib/lp/bugs/templates/bugtarget-portlet-bugtags.pt [00:41] * wgrant nominates wallyworld_ [00:41] I assume I can just move it into a file in lib/lp/bugs/javascript [00:41] * wallyworld_ looks [00:41] but I want to make sure it gets namespaced properly and uses whatever global stuff we have [00:43] huwshimi: much of the js for the bugtask page lives in bugtask_index.js but you may want to use a new namespace for this [00:43] wallyworld_: What would you recommend? [00:43] huwshimi: simply have the on domready in the tales call the method you have moved to the new namespace [00:44] perhaps lp.bug.bugtask.tagcloud [00:44] but i'm not the best at making up names [00:45] it depends on if you think there may be more stuff to add which would warrant a new namespace [00:45] i think that lp.bugs.bugtask_index has too much stuff at the moment [00:46] might be good to have the tag cloud stuff separate [00:46] typo, should be lp.bugs.bugtask.tagcloud, or even lp.bugs.tagcloud [00:46] wallyworld_: Where abouts is that tales domready? [00:47] huwshimi: the existing js embedded in the tales have a domready call [00:47] just replace the js in there with a call to the new namespaced method [00:48] huwshimi: do you see what i mean? [00:49] wallyworld_: I know what you mean, I'm just not sure which file it is in [00:49] huwshimi: the one from your link [00:49] bugtarget-portlet-bugtasks.pt [00:51] wallyworld_: Ah right. [00:51] wallyworld_: Is there a way I can modify things so I can remove the javascript from that page completely? [00:52] huwshimi: the on domready callback has to go somewhere [00:52] there's an argument that it belongs with the template [00:53] but it could also go on the template for the index page which contains the tag cloud i guess [00:53] so if it is removed from the template for the tag cloud, it would need to be put separately wherever the tag cloud is used [00:54] i guess the tag cloud is only used in the one place? [00:54] on the bug task index page [00:56] wallyworld_: At some stage we will be moving to having no javascript in our templates [00:56] \o/ [00:56] wallyworld_: As I was editing this code already I thought I would try and move this out of the template now [00:56] huwshimi: so as far as i know, we don't yet have a well defined plan to deal with that [00:56] wallyworld_: I guess really it needs to happen with a proper js loader [00:57] yep [00:57] wallyworld_: maybe I'll leave this for now then [00:57] wallyworld_: Well actually I'll move it into its own file, but leave the call [00:57] huwshimi: the best you can do, afaik, is to just have the on domready stub in the template and the core business logic elsewhere [00:57] for now [00:58] wallyworld_: Yep, ok that will set it up for the future complete removal. Thanks :) [00:58] i've used this exact pattern where i need to be able to invoke the js in an xhr callback [00:58] as well as the page loaf [00:58] benji: still around? I'm going to guess not ... [00:58] huwshimi: np. good luck with it [00:58] wallyworld_: Thanks [01:17] wallyworld_: hi; did you see my ping earlier ? [01:18] thumper: number of possible reviews - performance or visualisation ? [01:18] lifeless: ah yes, thanks. i tested it and yes it does do the right thing [01:18] lifeless: yes [01:18] lifeless: both [01:18] thumper: so consider https://code.launchpad.net/launchpad-project/+activereviews [01:19] thumper: or ubuntu/+activereviews [01:19] thumper: for a developer in either project, thats going to be a large fraction - maybe 80 or 90 percent - of what we'd be showing on their ~/+activereviews, simply due to the activity of the projects [01:19] thumper: or do you think it would be a much smaller fraction ? [01:20] approved merges from mid 2009 is a bit sad [01:20] it is [01:20] lifeless: I have no metrics [01:20] so I can't make any real call on this [01:20] one way to get some is to implement it, flagged, and see [01:21] sure [01:21] at the time of development we didn't have feature flags [01:21] yeah, they make things easier :) [01:21] and I chose the solution that was less likely to explode with timeouts [01:21] but feel free to "fix" it [01:22] :) I might scratch at the itch [01:22] jml's request for -a- page is reasonable, the main thing I didn't understand is why that shouldn't be on the existing per user page [01:23] since the data volume is ~the same either way. [01:24] lifeless: And how do you determine a person is involved in the project? [01:24] StevenK: current heuristic of being a reviewer for the branch [01:24] StevenK: can iterate on that, obviously. [01:25] StevenK: As a dashboard for reviews, +activereviews is pretty good. The only issue at the moment, for a single dev, is having to visit one per project [01:25] Indeed. [01:26] consider that looking at 'requested reviews I can do' for all projects in LP is equivalent to just seeing those you are a reviewer for - because the list is empty when you aren't an oficial reviewer. [01:28] wallyworld_: Do you mind doing a quick once over of what I did? https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689 [01:28] * wallyworld_ looks [01:33] huwshimi: for testability, you want to avoid Y.io [01:33] you want: lp_client = new Y.lp.client.Launchpad(); [01:34] lp_client.io_provider(xxx) [01:34] a bit pseudo ish [01:34] but the above allows a mock io provider to be used for tests [01:34] wallyworld_: Ah, I didn't do that part, but this is a good opportunity to fix it [01:34] see existing stuff in bugtask_index.js or elsewhere [01:35] wallyworld_: Thanks, I'll take a look [01:35] huwshimi: also, set('innerHTML') i think we are trying to avoid where possible, can't recall exactly, but setContext() may work [01:36] huwshimi: or new_node = Y.Node.create(xxxx); old_node.replace(new_node) [01:36] or something like that [01:36] wallyworld_: Sure, I'll fix that up too [01:37] i know that the server may generate only legit html and all that, but it is a potential security hole [01:37] other than that, looks ok [01:38] huwshimi: in case you don't notice it, the lp_client construction is usually inside a setup type function so the client is only constructed once [01:39] wallyworld_: Yeah ok thanks [01:39] good luck :-) there should be several examples to cargo cult. let me know if you are stuck [01:39] wallyworld_: No problems, will do [01:44] wallyworld_: I think it should be fine in the setup function I already have there. [01:45] the lp_client construction? yes, so long as setup is only called once which i guess it will be [01:46] you will want to make your setup function take a config dict as an argument so you can pass in the io_provider stub for tests [01:47] wallyworld_: Oh ok, in that case would it be easier to have its own function? [01:47] wallyworld_: Easier for testing that is [01:48] wallyworld_: I guess it would be [01:48] huwshimi: perhaps. getClient(config) or something [01:49] i think setup_lp_client(config) is also used, which assigns a module scoped lp_client variable [01:49] then the code can just use lp_client and assume it's been all set up [01:50] and the test would call setup_lp_client() with the correct config to pass in the io+provider [01:50] whereas the prod code would call with no confif [02:01] wallyworld_: Should that setup function check to see if a lp_client already exists and if so do nothing? [02:01] huwshimi: yes [02:01] wallyworld_: Ah good [02:02] so var lp_client = null; somewhere in the module, and then the setup function sets it [02:03] wallyworld_: Yup [02:04] wallyworld_: Well, "var client;" is enough right? [02:04] erm [02:04] var lp_client; [02:04] i prefer to be explicit, but your call [02:04] wallyworld_: I don't mind. I'd prefer to be consistent [02:05] sure. i'm not sure what our coding standard says for that to be honest [02:05] wallyworld_: let me check [02:05] wallyworld_: Most of these files seem to just declare the variable name [02:06] sounds good [02:08] wallyworld_: The docs don't seem to say though [02:08] as you say, just use what's gone before :-) [02:27] wallyworld_: Ok, here's where I'm at: https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689 [02:28] * wallyworld_ looks [02:35] huwshimi: i have led you astray a bit i'm sorry. if all you are doing is making a straight Y.io call and don't need the lp_client extras, you just do: [02:35] var io_provider = Y.lp.client.get_configured_io_provider(config) [02:35] where config is what's passed into your setup [02:35] it is either undefined or a dict with an io_provider key [02:37] if you do need the full lp_client, you do lp_client = new Y.lp.client.Launchpad(config); [02:37] so in your current code, you are missing the config param for constructing the lp_client, but as per the above, you may not need that [02:38] \o/ no oopses on asuka since the 2nd (all via amqp) [02:38] lifeless: Oh, that config parameter did exist, I lost it at some point [02:38] lifeless: sorry that was meant of wallyworld_ [02:39] wallyworld_: Or maybe it only ever existed in the code in my brain [02:39] huwshimi: perhaps :-) [02:39] older for tellurium. [02:39] so if you want, you can just get yourself the io_provider [02:39] good news about the oopses [02:41] wallyworld_: So this is how it should look? http://paste.ubuntu.com/733801/ [02:46] * wallyworld_ looks [02:48] huwshimi: that looks ok to me. your test setup then just calls setup_io_provider(). if io_provider is only used the once in setup_taglist, you may consider ditching the separate setup [02:48] wallyworld_: yeah ok [02:49] so make setup_taglist take the config param [02:50] wallyworld_: ok sure [02:51] huwshimi: sorry about the false start before, i'm used to needed the whole lp_client object [02:51] wallyworld_: No, all good, I've learnt some things [02:51] excellent [02:51] the mock io provider we have is pretty good [02:52] lp.app.javascript.testing.mockio.js [02:54] wallyworld_: Should I still have a check for if a io_provider already exists? [02:54] wallyworld_: Or is that really only important for the lp_client [02:56] huwshimi: it doesn't really matter in this case because it's just grabbing a value from a dict [02:57] wallyworld_: ok sure [03:01] lifeless: bugsummary is pretty impossible to modify :/ [03:01] lifeless: There's like 5 patches, and they weren't flattened into trusted.sql. [03:01] And they aren't in order :*( [03:08] wallyworld_: Ok, hopefully this is what you meant :) https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689 [03:08] wgrant: sorry :) - twitch. [03:08] hey, anyone know if there is a good 'touched recently' field on question ? [03:12] I guess we need a new baseline schema soon anyway. [03:13] lifeless: you probably have to take the max of all the date fields :/ [03:15] just query response I think will do [03:15] they seem to combine to cover it [03:16] They even cover answers? [03:16] last-response [03:16] last-response + last-query forms the rule for who needs to act next [03:17] Right, but what about a proposed answer? [03:17] arguably doesn't need to cover messages - they are scanned separately [03:17] Erm, what does it cover, then? [03:17] Is datecreated OK? [03:17] whiteboard, title description [03:17] Oh, whiteboard. [03:17] Blah. [03:17] so datecreated isn't enough [03:18] down to 5s query [03:18] big improvement [03:18] (for 1 week) [03:21] seq scanning question [03:27] huwshimi: i would reverse the config and io_config names, otherwise ok i think. the setup config can contain more than just io_provider. and io_config seems better suited for use in the io call [03:28] wallyworld_: Ok, no problems [03:30] 500ms with two simple indices [03:30] OTOH I could just limit those to their containing scope. [03:30] probably easier I think. [03:30] lifeless: For a full week? [03:31] yes [03:34] wallyworld_: OK, done, https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689 [03:34] * wallyworld_ looks [03:36] jtv: For arch-parallel a-f, see shelf 21 on DF. [03:37] huwshimi: one more improvement - use start and end callbacks in to show/hide spinner [03:37] start and end callbacks in io_config [03:37] that way you don't need to call hide in success handler [03:37] and again in failure callback [03:38] use something like "end: Y.bind(namespace.hideSpinner, namespace)" [03:43] wallyworld_: What does the bind give me that just doing "end: namespace.hide_spinner" doesn't? [03:44] huwshimi: ah, sorry. i referenced an example where i had to pass a parameter to the hide api call, and i think bind is required then [03:44] wallyworld_: Ah right [03:44] np [03:44] the js tests will ensure that it all works in any case [03:46] I'd like to return count(archive.id) in a SELECT query, but only if it's > 1 [03:46] Oh, wait, they will not disappear. [03:46] StevenK: HAVING [03:46] So I need to filter on state [03:47] SELECT owner, COUNT(*) FROM archive WHERE status = 0 GROUP BY owner HAVING COUNT(*) > 1 [03:47] wgrant: Can I do select count(...) AS foo .... HAVING foo > 1 ? [03:48] Probably. [03:48] * StevenK tries [03:48] Probably not, that is. [03:49] ok [03:49] wgrant: what did you want me to look at ? [03:49] now I have my query for oops pruning sorted, I'll look at your thing [03:49] lifeless: Thanks. [03:50] lifeless: So, http://paste.ubuntu.com/732768/ is the suggested undenormalisation. [03:50] lifeless: It's only ~10% slower for Ubuntu. [03:50] But 2.5x slower for ubuntuone-servers, because this ends up doing a seqscan on apg. [03:50] Not sure why. [03:51] I guess I need to regen sample data [03:51] ah itsi nthe bin [03:51] The paste contains 4 queries at the end. [03:51] The third is fastest. [03:51] Sample data is in the bin? [03:51] It's ~600ms for Ubuntu, vs 1200-1500ms for the other three. [03:51] But it still does a seqscan. [03:52] And is 120msish for ubuntuone-servers, rather than the old denormed 30-40ms version. [03:52] the old version was slow on ubuntu ? [03:52] or is this just data that the oldver was better overall [03:52] ? [03:52] Your version was just under 600ms for Ubuntu on DF. [03:52] The new one is a little over. [03:53] whats changed in the schema ? [03:53] APG.{artifact,policy} are mutually exclusive, and APA.policy exists. [03:53] So each artifact's policy is stored on APA, instead of many times in APG. [03:54] ah yes [03:54] so this is a normalisation [03:54] not a denorm :P [03:54] undenorm, I said. [03:54] It's not normalising, it's reverting the denormalisation :) [03:54] double negative parsing fail [03:55] Heh [03:55] ok, which query sucks again ? [03:56] The four at the end all do the same thing. The third is fastest. [03:56] One CTE and a couple of subselects. [03:58] Try against product 9514 to see the greater slowness. [03:58] But the plans are the same. [03:58] Er, the plans for distro 1 and product 9514, that is. [04:08] wgrant: plan 3, product 9514 - 114ms [04:09] plan 4 65ms [04:09] Bah, really? [04:12] Indeed, 30% faster. [04:12] But slower for Ubuntu. [04:12] http://paste.ubuntu.com/733853/ [04:13] thats the plans you had [04:14] #3 there has a different context (ubuntuone-servers rather than ubuntu), but yes. [04:14] bah [04:14] wgrant: do you mean that parallel a-f has been sitting around as uncompleted work? That would be a serious waste. [04:14] wgrant: 410ms for #3 on ubuntu [04:14] jtv: Well, we don't really know how much of a win it is. [04:15] wgrant: do you want the plan ? [04:15] lifeless: That's 650ms on DF. #4 is 1.1s. So it is behaving somewhat differently to qastaging :/ [04:15] lifeless: Not really. [04:15] wgrant: try raising your stats [04:15] wgrant: thing is, it won't be at all hard to make the MF manage a sensibly-sized pool of processes. So if 2 parallel processes is a win but 5 aren't, we can do 2. [04:16] wallyworld_: I made the change to start/end and also changed the show show/hide to using classes instead of styles. As the show/hide are one liners do you think they should still be in separate functions? [04:16] wallyworld_: https://code.launchpad.net/~huwshimi/launchpad/tag-cloud-removal-709009/+merge/81689 [04:18] lifeless: Mmm, the slow bits of the plan aren't really different [04:19] wgrant: those were all hot btw [04:20] The plan for #3 on 9514 is identical, and performance is almost the same. But it still hsa the seq scan, which seems unfortunate. [04:20] But if it's only 410ms for Ubuntu then I guess that's not too bad... [04:21] wgrant: 123557 rows [04:21] wgrant: 16ms to seq scan [04:21] wgrant: (just to read the rows once through, no processing) [04:22] lifeless: How'd you measure that? SELECT *? [04:23] wgrant: yes [04:23] 59785 rows are ubuntu [04:23] wgrant: thats 50% - seqscan is always going to be fastest ;) [04:24] That takes 160ms on DF. [04:24] Hmm. [04:24] But that can't be right, so maybe not. [04:24] lifeless: Yeah. [04:24] http://paste.ubuntu.com/733866/ [04:24] But U1 and other products will be smaller. [04:24] sure [04:24] 9514 was the 'slow' one right ? [04:24] 100ms instead of 40ms, yes. [04:25] 10294 is LP, which may be interesting. [04:26] 4K grants for 10294 and 9514 each [04:26] sorry, 60K for ubuntu, not 600K [04:26] my eyes, me eyes, my eyes [04:29] wgrant: still 5% of a table with narrow rows will hit ~ all pages unless its clustered on the selector [04:30] True. [04:30] It's also only a 9MB table and should be nice and hot. [04:35] wgrant: you can defeat the seqscan by unioning [04:35] if you want [04:36] Yeah, but that's really slow for Ubuntu, at least on DF. [04:36] distroseries-edit-datereleased => devel [OK] (up for 4:39:27.158866) [04:36] Oh, come on, ec2. [04:36] 800ms here [04:38] wgrant: 180ms in [04:38] -> Index Scan using accesspolicygrant__artifact__person__key on accesspolicygrant (cost=0.00..7.14 rows=2 width=8) (actual time=0.005..0.006 rows=2 loops=33978) [04:38] Index Cond: (pg_temp_43.accesspolicygrant.artifact = accesspolicyartifact.id) [04:38] rm, something like that [04:38] # XXX cprov 2007-02-22 bug=87098: [04:38] # Since we only allow one PPA for each user, [04:38] Ha. Hahaha. Hahahahahahaahahahahahaha [04:39] hm [04:39] so the buildd package can be installed pretty easily as a dpkg on developer systems [04:39] lifeless: OK, 800ms is better than I get, but still 100% slower than #3 [04:39] at the moment it automatically starts a buildd [04:39] i suspect this is not very safe or desirable [04:40] poolie: It's a nice easy remote root vulnerability, right. [04:40] i'm inclined to handle it by splitting off an "actually run it" package [04:40] which can be installed on the real buildds [04:40] with that still called launchpad-buildd [04:40] and then eg launchpad-buildd-bin or python-lpbuildd [04:40] or maybe both [04:40] that has all the stuff that will be developer dependencies [04:41] Mmm. [04:41] I'd prefer not to have launchpad-buildd in any readily accessible apt repo. [04:41] :) [04:41] someone will run it [04:41] Yes. [04:41] well, i could take this course and just not upload it [04:41] Huh. [04:42] Speaking of remote root vulnerabilities. [04:42] lolpython [04:42] the other way these things tend to be handled is to have the daemon off by default and an 'are you really sure' in /etc/default/something [04:42] ? [04:42] yaml.load isn't safe. It loads pickles. [04:42] What a sensible default. [04:43] seriously? [04:43] Yeah. [04:43] You have to use yaml.safe_load for that. [04:43] as a kind of dwim if you feed it a pickle? [04:44] Not sure. I guess they define a way for YAML to contain pickles. [04:45] Say wgrant, how do I dput a package onto a PPA on dogfood? [04:45] wallyworld_: What happened to TPG? [04:45] ? [04:45] wgrant, not pickles as such as far as i can see [04:46] but you can call python objects from inside it [04:46] wallyworld_: "rev.aanet.com.au" [04:46] Yeah, not actual pickles. [04:46] Its own implementation. [04:46] But still. [04:46] it's similar to doing eval [04:46] Who could possibly think that was a reasonable idea. [04:46] StevenK: oh, i've relocated today to a different place, change of scenery [04:46] anyhow [04:46] > off by default and an 'are you really sure' in /etc/default/something [04:46] the main drawback is [04:47] jtv: [df] [04:47] method = ftp [04:47] fqdn = upload.dogfood.launchpad.net:21 [04:47] incoming = %(df)s [04:47] login = anonymous [04:47] losas need to turn it on i guess [04:47] jtv: dput df:ubuntu something_source.changes [04:47] and, some people will turn it on and get in to trouble [04:47] poolie: And it breaks the automated installation. [04:47] wgrant: plan three is 640ms [04:47] wgrant: That's to ubuntu, not a PPA [04:47] wgrant: so no mention of the PPA at all? [04:47] lifeless: How fast was your query? [04:47] wgrant: uhm, variable. just got a 450 [04:47] so... [04:47] jtv: Ah, for a PPA. Use a PPA path. [04:47] wgrant: lots of noise [04:48] lifeless: Er, your query from a couple of days ago, that is. [04:48] I can't remember. [04:48] Was 300ms or something. [04:48] oh right [04:48] but that was old schema wasn't it ? [04:48] It was. [04:48] wgrant: is a PPA path something like “ppa:user/ppa”, without changes for dogfood? [04:49] jtv: The FTP path is ~user/ppa/distribution [04:49] So df:~user/ppa/distribution [04:49] Oh [04:49] Thanks [04:52] lifeless: But I would like to get a real performance comparison between the two schemas. [04:52] lifeless: The queries in the denormed one are already slow. [04:54] StevenK: Erm. [04:54] wgrant: Hm? [04:55] Ah, I see. [04:56] Just somewhat bemused at the misspelt instance variables your rev added. [04:56] but there was actually a reason for it, and the misspelling wasn't yours. [04:56] Oh? [04:56] stub: any luck with Q/A for bug 826653? [04:56] <_mup_> Bug #826653: preflight check could report patches to be applied

< https://launchpad.net/bugs/826653 > [04:57] Needed to wait for a staging update... lets have a look [04:57] StevenK: "derivative" is not spelt "derivitive", and it's sort of odd to set the instance variables in setUpFields. [04:57] Probably should avoid a rollout today since we are doing the slony upgrade btw. [04:58] Do we have a GSA scheduled to do the package swap? :) [05:01] stub: that's a shame—we could roll out all 14 available revisions once your Q/A on this is done. [05:01] qa-ok [05:01]