[00:02] yeah, one little change got rid of most of the errors [00:22] g'night all [00:36] jml: ciao [02:08] === Top 10 Time Out Counts by Page ID === [02:08] Hard / Soft Page ID [02:08] 283 / 2484 Distribution:EntryResource:searchTasks [02:08] 103 / 242 BugTask:+index [02:08] 94 / 209 Distribution:+bugs [02:08] 52 / 4427 Archive:+index [02:08] 38 / 85 Branch:+index [02:08] 23 / 3 ProjectGroup:+milestones [02:08] 15 / 258 Question:+index [02:09] 11 / 1 Distribution:+builds [02:09] 10 / 1 DistributionSourcePackage:+filebug [02:09] 8 / 2 Person:+bugs [03:06] Project db-devel build (318): FAILURE in 4 hr 54 min: https://hudson.wedontsleep.org/job/db-devel/318/ [03:06] Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12231, [03:06] 12232 included. [04:21] lifeless, have a minute for a review? [04:21] https://code.launchpad.net/~ursinha/qa-tagger/681099-consider-untestable-bugs/+merge/46870 === Ursinha is now known as Ursinha-afk [04:26] Ursinha-afk: done === Ursinha-afk is now known as Ursinha [05:01] thanks lifeless [05:03] de nada [05:05] :) [08:22] Yippie, build fixed! [08:22] Project db-devel build (319): FIXED in 5 hr 15 min: https://hudson.wedontsleep.org/job/db-devel/319/ [08:22] Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12233, [08:22] 12234 included. === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan [13:30] \o/ [13:46] hmm .. this channel used to be livelier [13:46] its < 8am where everyone in the team is [13:47] sprint time :) [13:47] ah, I see :) === al-maisan is now known as almaisan-away === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha === almaisan-away is now known as al-maisan [15:27] jml: are you going to show testr/tribunal? [15:28] lifeless: yes [15:29] lifeless: testr, tribunal, subunit, fixtures, testdoc, etc. [15:30] \o/ === beuno is now known as beuno-lunch [16:20] leonardr, is bug https://bugs.edge.launchpad.net/launchpad/+bug/486974 actually fix released? [16:20] <_mup_> Bug #486974: no API documentation of Person

< https://launchpad.net/bugs/486974 > [16:21] it was marked as fix released so the qa-tagger is ignoring it [16:31] leonardr, it's not actually released yet, as production doesn't have it, right? can I change it back to Fix Committed? [16:40] ursinha: yeah, sure [16:40] sorry [16:40] leonardr, no problem, really [16:42] jcsackett, is the partial fix of bug 618400 ok to be deployed? [16:42] <_mup_> Bug #618400: Distribution:+archivemirrors timing out in 1% of requests < https://launchpad.net/bugs/618400 > [16:42] as in it doesn't need qa or it won't break anything [16:43] ursinha: yeah, i took a look at the affected pages last night--all seems groovy. [16:43] jcsackett, cool. I'll mark it as qa-ok :) [16:43] thanks! [16:43] np. [16:43] with luck, full fix will land today. [17:13] huwshimi: please can I talk to you at the start of squad time? === beuno-lunch is now known as beuno === al-maisan is now known as almaisan-away [17:31] thumper: Will it take long. I need to have a meeting with someone else and have that scheduled straight up [17:31] huwshimi: I just want to point to some styling bugs in LP [17:32] huwshimi: should just be 1 or 2 minutes [17:32] thumper: ok no problems then [17:39] is this the right address if I fail to build a source package? [17:42] vanguard: it is good enough, what's up? [17:43] I just fell for the - and _ in the orig.tar.gz name [17:43] huwshimi: we might have to reschedule. I wasn't expecting this talk. [17:43] now I got another problem, I have Depends: default-jre (>= 1.6) in my control file [17:43] when I run debuild, it tells me that (>= 1.6 is no valid version [17:44] wgrant: can you help? [17:44] where did that ) go? [17:44] or bigjools [17:44] vanguard: Please pastebin your control file. [17:45] sure [17:46] http://pastebin.ubuntu.com/556236/ === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha [17:49] vanguard, you need a comma before default-jdk on the Build-Depends line [17:50] huwshimi: bug 705548 [17:50] <_mup_> Bug #705548: Multiline editor needs CSS tweaks < https://launchpad.net/bugs/705548 > [17:51] thumper: Thanks for that [17:51] np [17:52] james_w`: okay, there is .deb now. I have to check it out [17:53] the package is opened by gdebi, but in the file list is nothing of my project. How do I tell debuild to pack the .jar file into the .deb? [17:54] sry, g2t [17:54] g2g [18:05] mpt: [18:05] mpt: http://people.canonical.com/~jml/convergence/ [18:42] I build a source package, but after "debuild", I have a package that only contains the changelog and stuff, but not my actual .jar file. How do I add it? [18:45] vanguard: try #ubuntu-motu for packaging questions [18:46] Project db-devel build (321): FAILURE in 5 hr 10 min: https://hudson.wedontsleep.org/job/db-devel/321/ [18:46] Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12239, [18:46] 12240, 12241 included. [18:56] hi, lifeless - rather than just fix the cve case, I've started a search for all dubious SQL which employs a sub-select [18:57] my search is based on grep and manual checking so far, but if I come up with a list, I'll file a meta-bug for the issue [18:57] bugtask.py is looking like a strong culprit [18:58] Any chance you could paste me some sanitized stats on where I might most usefully look to fix first? [19:07] In case it is of interest to anyone, my commit hook robot code: [19:07] (WIP) [19:07] http://pastebin.com/8m2cWXZ3 [19:08] and the git part: http://pastebin.com/ahF1U3RX [19:08] It is a two-part hook / update process, the hook uses git shell commands to populate a sqlite DB [19:08] The update process pulls work items out of the sqlite DB and uses launchpadlib to make updates to bugs [19:09] The only snag so far is that it can't handle private bugs, for the project as it is not subscribed to them. I could add the robot to our developers team, but I can't handle the extra email it would generate - the robot's email address forwards to me [19:22] So far I've found 11 candidates in the code-base for SQL improvement [19:23] (And that is with the very crude search assuming the appropriate statements are on the same line) [19:24] http://pastebin.com/3Ln9z4qg === Ursinha is now known as Ursinha-afk [19:35] https://bugs.launchpad.net/launchpad/+bug/705582 [19:35] <_mup_> Bug #705582: Metabug: Inefficient sub-select SQL in LP causes performance issues < https://launchpad.net/bugs/705582 > [19:36] Not much chat in here at the moment - wrong time-zone, or is there a conference or something [19:38] The canonical folks are all still at their sprint, IIRC [19:40] ah, that would explain their absence [19:52] I just found this: http://www.markshuttleworth.com/archives/239 [19:52] That video mockup is pretty amazing - is that the shape of things to come? [19:52] (I realise the post is old) [20:00] pcjc2: adding nice ajax-y stuff to improve usability? Yes [20:01] pcjc2: hi :) want to see more crazy sql ? === Ursinha-afk is now known as Ursinha [20:11] is a subselect something that performs another select based on every single result from the first query? [20:12] thats a correlated subquery [20:12] a subselect is a form of subquery [20:13] whether its correlated or not depends on whether it depends on any variables in the containing query [20:30] hi lifeless - fire away with the SQL [20:30] I'm not promising I'll fix them all - just felt bad for not filing the bugs after I discovered those other cases [20:31] I'm thumb-twiddling whilst I wait for our project's "maintainer" / founder to get on and test my LP update robot scripts no his server. [20:31] Absentee landlord is no fun - and he's the only one with root / admin access on that server [20:33] * pcjc2 is like a child with ADHD demanding constant attention - likes feedback soon, does not like waiting 2 or more days to test the code he's just finished writing [20:34] pcjc2: :) [20:35] The video here.. http://www.markshuttleworth.com/archives/239 [20:35] bug 705224 was entertaininy [20:35] <_mup_> Bug #705224: Distribution:EntryResource:searchTasks timeout < https://launchpad.net/bugs/705224 > [20:35] was that a "live" mockup witih real AJAX, or heavily edited? [20:36] was the regression in bug 705224 related to my recent change to fix the timeout searching for -* tags? [20:36] <_mup_> Bug #705224: Distribution:EntryResource:searchTasks timeout < https://launchpad.net/bugs/705224 > [20:36] I think it was a manual mockup [20:36] certainly we don't have that yet [20:37] pcjc2: it was related yes, thats why I mention it to you :) [20:38] I'll take a look [20:38] pcjc2: the relation is that the UNIONs before used to be a single query over the tags [20:38] pcjc2: oh, its fixed. [20:38] pcjc2: but sure, have a look at the fix if you're interested [20:38] I did wonder if we ought to be combining the multiple tag queries inside the WHERE clause [20:45] rvb: me! [20:46] allnap: same name on lp? [20:51] lifeless - are tags gauranteed unique? [20:52] if so, "might" be able to optimise the all tags case too, abusing count [20:53] (SELECT COUNT(*) FROM (SELECT TRUE FROM BugTag WHERE BugTag.bug = Bug.id AND BugTag.tag IN ('foo', 'bar', 'baz')) = 3) [20:54] assuming the input tags are all distinct (could write some python to ensure that if necessary), the query should return an equal number of rows to the number of tags we asked for IFF all those tags exist [20:55] question is... would this "hack" / creative use of SQL give any useful speed-up for any real-world queries being seen? [21:00] I presume that if this was a regression - the DB engine was doing a better job of merging the uncorrelated UNION subqueries [21:06] allenap: I'm browsing through trivial bugs. Many of them look like they've been fixed already ... can you check this one https://bugs.launchpad.net/launchpad/+bug/298288 to make sure I'm not completely wrong about this? [21:06] <_mup_> Bug #298288: Feature Request: PPA file display should allow sort by header < https://launchpad.net/bugs/298288 > [21:07] rvb: Yeah. Do you want to mark it Invalid? [21:09] allenap: I'll do it [21:10] allenap: and to a few others bugs as well [21:11] rvb: Cool :) [21:11] http://people.canonical.com/~jkakar/storm/storm.sqlobject.SQLObjectBase.html#selectOne [21:11] Undocumented [21:11] What kind of SQL fragment does selectOne expect? (Or what kind of query would it expand to?) *** (checks storm sources) [21:12] pcjc2: You should ideally not use selectOne, since that's part of the legacy SQLObject compat layer. [21:12] grep around for '.one()' [21:12] That's the new Storm Way® [21:12] not new code - I was just looking to see if I could identify an SQL optimisation [21:12] depends on what type of result set he's got [21:13] consistency FTW [21:13] job = CodeImportJob.selectOne( [21:13] """id IN (SELECT id FROM CodeImportJob [21:13] WHERE date_due <= %s AND state = %s [21:13] ORDER BY requesting_user IS NULL, date_due [21:13] LIMIT 1)""" [21:13] bigjools: .selectOne is a method on a SQLObject class. [21:13] Not a resultset. [21:14] * bigjools needs more coffee [21:14] The question in theory - is whether that expands to something which gets correlated against for lots of rows [21:14] .... huh? [21:14] Does that make any sense? [21:14] id IN (SELECT id ...) [21:15] given that line of code... does it produce a horribly inefficient query, as was the cause for bug 501945 [21:15] I'd use this: [21:15] I wasn't convinced it would do anything other than return TRUE or FALSE [21:15] which seemed like an odd kind of thing to pass to a method which allegedly selects one result [21:16] lp/code/model/codeimportjob.py line 134 [21:16] It happened to be the first on my list of suspicious SQL things I found and noted in Bug #705582 [21:16] job = IStore(CodeImportJob).find(CodeImportJob.date_due <= whatever, CodeImportJob.state == whatever).order_by(Desc(CodeImportJob.requesting_user), CodeImportJob.date_due).first() [21:16] <_mup_> Bug #705582: Metabug: Inefficient sub-select SQL in LP causes performance issues < https://launchpad.net/bugs/705582 > [21:17] I'm going to file bugs for all cases where I can determine some better SQL, but without being able to link these to real issues people are seeing, change for the sake of change seems like regression potential [21:18] IStore is part of storm? [21:18] pcjc2: perhaps; we're not seeing timeouts on the 'all' case atm [21:18] pcjc2: we have -very- good feedback on slow queries, so I'm kindof inclined to change broken things only. [21:19] sounds fair enough [21:19] that siad, CIJ had some issues [21:19] Any matchup between the ones I noted and the hit-list of bad SQL? [21:20] and that select seems a workaround for separation between clause and results [21:20] That code-import fragment I posted has me confused to what it even does [21:21] id IN (...) should evaluate to TRUE or FALSE, right? [21:21] it acts as a where clause that sqlobject will use reasonably [21:21] yes [21:21] pcjc2: or NULL perhaps, I'd need to check. [21:21] job = CodeImportJob.selectOne() seems like an odd bit of code to run [21:22] pcjc2: It selects the single row that matches the condition. [21:22] yeah, its odd because of an old layer we're migrating from [21:22] so it is a WHERE clause gragment? [21:22] fragment, sorry [21:23] its a where clause [21:23] SELECT ... WHERE id IN (...) [21:23] sqlobject is magic and special [21:25] so it could be "CodeImportJob.id = id AND date_due <= %s AND state = %s ORDER BY requesting_user IS NULL, due_date LIMIT 1" [21:25] no [21:25] not with sqlobject [21:25] needs to be stormified [21:25] is sqlobject pretending to execute SQL on something which is not in the DB? [21:27] no, its pretending not to execute sql [21:27] its a complex story [21:27] best thing is to move to Store.find(CIJ, ....) instead [21:27] with .one() [21:28] I doubt I can do anything truly useful here, but I will file a bug for someone with area-knowledge to triage [21:28] I don't think code-import is something I'll be able to test very easily locally [21:29] so that doesn't look like a bug [21:29] its oddly written [21:29] but should be fast [21:29] why do you think its a bug? [21:29] lifeless: It's actually .first(), not .one(), I think. But it's spelt as a selectOne with a subquery that only matches the first. [21:30] meh, yes. [21:35] lifeless: Like pcjc2 is saying, he is following a pattern that a badly-written subselect was causing timeouts, so maybe others do the same. [21:36] all depends on row count really - bugtags is probably a very high count table, at about 700k rows [21:36] StevenK: thanks; I meant why does he think *this* one is a problem. [21:36] deryck: Available when you're ready [21:36] I suspect it just happens to be the first one on the list. [21:36] pcjc2: this query is already constrained though: note that the query is looking for next thing due [21:36] its answerable from index. [21:36] huwshimi, ah, cool. give me 5 minutes to wrap and we can. [21:37] huwshimi, shall I come to you or you to me? [21:37] yes, just first on list - it seemed odd, and I still can't quite map what SQL it ends up executing [21:37] you are right though - work from known timeouts, rather than poking at supposed problems [21:38] deryck: Great. I can come to you. Everyone is working quietly here so lets not disturb them :) [21:38] IStore, etc.. is Storm? [21:38] huwshimi, ok, first room on left going down left-side hallway. [21:38] deryck: ok see you soon [21:41] Hmm - example from storm docs: [21:41] store.find(Person, Person.name == u"Joe") and store.find(Person, name=u"Joe") [21:42] Seems like unholy magic to me [21:42] 'Person.name == u"Joe"' is somehow passed as argument, not evaluated immediately to give True / False? [21:46] It is passed as an argument and Storm intreprets it [21:47] thumper, the javascript side of things is in lib/canonical/launchpad/icing/build/bugs/bugtask_index.js and it's pretty hairy [21:48] lifeless: resubmitted. [21:50] deryck: Just held up here for a bit, will be over soon [21:51] StevenK, I need to go read up more on Python... the fact it is somehow an argument and not evaluated before the call is magical to me [21:51] huwshimi, no worries [21:51] does it have anything to do with lazy evaluation? [21:53] pcjc2: the implementation of the == operator doesn't have to return true or false... [21:53] ah - so it IS evil magic going on ;) [21:53] And presumably the "Person" object in the example is typed such that the == operator has a specific implementation? [21:57] It's likely done via SQLObject *and* Storm so that both implement __eq__ [21:57] I'm reading the code, and I'm still non the wiser how it works [21:58] * StevenK is fairly ignorant of Storm internals [21:58] NM though.. it will keep me entertained trying to learn how.. Evil evil magic is still my best guess [21:58] Is it a design choice to go more ORM style, and less explicit SQL queries? [21:59] jelmer: thanks [21:59] pcjc2: operator overloading [22:00] pcjc2: yes, its evil [22:00] It seems pretty evil [22:00] ORM is a bit of a mistake; late evaluation leads to death-by-single-row-retrieval [22:00] I'm sure someone more knowledgeable than I has decided it is a good idea though [22:00] long term goal is a mapping layer that answers service-point entire answers [22:01] service-point is for example, http://lp.net/project/+bugs ? [22:01] a search page, sure [22:02] launchpadlib API is a "sort" of ORM type affair? [22:02] I found that very intuitive to work with [22:03] sort of yes [22:03] its why its so slow ;P [22:04] can't have everything I guess [22:04] When people are not so busy sprinting, I'd love to have a chat about various ideas [22:05] Mostly feature-request stuff, but the kind most usefully bounced of other people before just filing a bug titled "wouldn't this be nice..." [22:05] sure [22:05] anytime is good [22:05] I was thinking that LP might usefully grow a "person" type object for automatons / robots [22:06] (I've been writing one recently, so it is in my mind) [22:06] They would be owned by a person or team, but delegate contact via that team - not requiring their own Email address or OpenID login / whatever [22:07] IE.. a way to auth against LP for API use which doesn't require setting up a pretend person to do it [22:08] jtv: From the API, a list of Bugzilla trackers can be obtained with: [bt for bt in root.bug_trackers if bt.bug_tracker_type == 'Bugzilla'] [22:08] Have also been thinking about controls to navigate collections of bug search results from within the bug page. UI still going through my head though [22:08] pjdc: you could call the object IRobot [22:09] might be a TM on that one [22:10] so, in page refresh is definitely already desired [22:10] what would be awesome is a way to use the API with basaic auth [22:10] basic even [22:10] or digest anyway [22:10] I have extremely mixed feelings about that [22:10] https://launchpad.net/~gpleda-launchpad-robot [22:11] The fact that the robot is a person doesn't map well to our usage [22:11] It needs to get subscribed to security bugs so it can close them [22:11] but I _dont_ want it to get bugmail [22:11] i don't want *me* to get bugmail [22:11] that too ;) [22:12] ok [22:12] I understand that is a WIP regarding bugmail granularity though [22:12] so we have an answer for this already [22:12] yeah [22:12] its well advance [22:12] if you ask gary_poster nicely you may be able to be in the alpha [22:12] ok, will do [22:13] lifeless: what reservations re: basic/digest auth? [22:13] dobey: well, we use ssl so on that front its fine [22:13] handshaking with canonical-identity-provider / other openid environments is darn tricky. [22:14] In page refresh would be awesome - I was thinking about remembering the "view" or "search query" / whatever, you were doing when you clicked on the bug, and giving "Next" "Prev" controls [22:14] in terms of standards... there are none [22:14] pcjc2: that would be awesome too [22:14] In page update when people comment (reverse AJAX push) would be amazing, but perhaps a corner case [22:14] yeah i know, which is why doing oauth is so horribly painful [22:15] i guess that launchpad isn't exactly the openid provider any more though, so basic/digest auth is trickier now [22:16] OpenID seems like a nice proposition [22:17] openid/oauth are solutions looking for a problem [22:17] if it were my server / wiki, I would re-write the auth for our project's wiki to link to user's LP OpenIDs [22:17] We could tie the permissions to team membership on LP [22:17] teams in openid are some launchpad specific extension though aren't they? [22:17] which is a bit of a shame [22:18] I'll admit I don't fully understand OpenID, [22:18] I looked at oauth, and it seemed a vaguely sensible way for an API to authenticate [22:19] i just don't want to have to authenticate my browser twice [22:20] dobey: can you be more specific about "authenticate my browser twice"? [22:21] I assume dobey is talking about having to login to the SSO / login.launchpad.net, and then auth launchpad.net to those credentials [22:21] dobey: right, we no longer maintain an authentication db [22:21] leonardr: i want to make an extension for my browser that uses lp api to do certain things when viewing certain launchpad pages, and having to authenticate that separately is weird [22:22] weird, but IMO necessary [22:22] i don't think it's necessary [22:22] it is however, cumbersome and painful [22:22] dobey: if you're in the browser, you can use the same-domain version of the web service that the ajax application uses, which will carry over the browser's authentication [22:23] but it depends on how close your code is to the browser [22:23] leonardr: if i'm in the browser, or "if i'm in the JS" ? [22:23] dobey: if you have access to the Cookie header being sent by the browser [22:23] by "extension" i mean "c, python, similar" [22:24] leonardr: so i can just re-use the sessionid cookie to talk to the api? [22:24] dobey: that's what the web browser does [22:25] leonardr: so i can pass that cookie through launchpadlib? [22:25] no [22:25] :( [22:26] but that's what allows the browser to make these requests [22:26] we could add that functionality but it would immediately be abused by everyone who's not writing a browser extension [22:27] and it would have all sorts of bad edge cases surrounding logout [22:27] i really do not want to reimplement launchpadlib :( [22:28] dobey: does it make you feel better to know that in natty, all desktop-wide apps will use the same oauth authorization? [22:28] no [22:29] dobey: do you understand my position? [22:29] How will they share that auth token? DBus or something? [22:29] gnome-keyring-manager? [22:29] pcjc2: keyring-manager by default [22:30] ubuntu-sso-client [22:30] which is per-app, not per-target [22:30] but anyway [22:32] leonardr: i understand what you're saying, but i don't think it's a particularly strong argument for doing it the way it's currently done. it wouldn't really take much more effort to abuse the browser stuff as it is, it seems. although it's tedious enough that i might not bother with it myself [22:32] brb [22:33] dobey: "tedious enough that i might not bother with it myself" is the only hurdle i need to clear [22:34] huwshimi, with reference to bug 263652, you rock [22:34] <_mup_> Bug #263652: {i} and other icon codes no longer work on help wiki < https://launchpad.net/bugs/263652 > [22:36] leonardr: yes, but everyone isn't an old bitter and jaded developer like me :) [22:36] dobey: i just need to stop this code from getting into supported ubuntu packages [22:37] dobey: i think you can implement Cookie: passing pretty easily by overriding some methods in Launchpad [22:37] and implementing your own Http subclass [22:38] leonardr: i don't want to touch javascript [22:38] i mean the Launchpad class of launchpadlib [22:38] ah ok [22:39] if this is code for yourself then that's fine, but if you want to put it into ubuntu we will have to reach some other compromise [22:43] i'm writing a browser, and would like to keep everything contained within it [22:46] dobey: if you're writing the whole browser, you should be able to intercept launchpadlib's webbrowser.open call and open that page in your already-authenticated browser, avoiding the second authentication [22:47] does that make any sense? [22:48] slightly [22:49] dobey: i have absolutely no problem helping you with this, if this is the way you want to go [22:50] i'm a long way from that point still, but it will require a lot more thought [22:51] mrevell: Yeah no problems. Still needs to be pushed live at some stage [22:52] dobey: ok, ping me. i think we can get a solution that makes us both happy [22:53] ok [22:53] cheers for now [23:32] lifeless: allenap just did an experiment with TAL to see if that macro re-evaluation was true, and it doesn't seem to re-eval if it's an int at least. Maybe it's only for complex objects.