[01:49] * thumper has lunch calling [03:03] gary_poster: ping, if you're around [03:04] gary_poster: I'd like https://bugs.edge.launchpad.net/launchpadlib/+bug/552419 to be addressed by squelching the warning; we can do it in bzrlib but really doing it in launchpad would be closer to the source [03:04] Bug #552419: Multiple module import warning from ubuntu-bug

[03:36] hmm... [03:36] where to general launchpad bugs live? [03:37] as in cross app ones? [03:38] ah ha [03:38] launchpad-web === Ursinha is now known as Ursinha-afk === henninge_ is now known as henninge === g4tsu_ is now known as g4tsu [08:28] good morning [08:53] Morning! [09:49] lifeless, i woke up very early--if you're around and want to talk about web service things, let me know [09:50] hey [09:50] I'd love to know enough to get useful traces of behaviour for you [09:50] I like some of the things you're talking about [09:51] traces of behavior == my plan? [09:51] traces of behavior == what requests are made by launchpadlib when do something that uses API's [09:51] s/when do/when I do/ [09:52] ah [09:52] like you pasted in your email [09:52] you know precisely whats going on and going wrong [09:52] sure [09:52] the simplest way to do it is: [09:52] import httplib2 [09:52] httplib2.debuglevel = 1 [09:52] how would you feel about that being in the docstring for launchpadlib/__init__.py ? [09:52] sounds good [09:53] would you like me to propose a merge for that ? [09:53] or do you want to JFDI ? :) [09:53] sure. i can also put timing information in lazr.restfulclient that's controlled by httplib2.debuglevel or a separate debuglevel [09:53] httplib2's debuglevel won't give your elapsed time [09:53] ok [09:54] is there a PPA with latest (launchpadlib and deps) ? [09:54] maybe not nightlies, but something moving faster than lucid :) [09:55] we have this great search facility for PPAs [09:55] *cough* :) [09:55] bigjools: we do? [09:55] https://edge.launchpad.net/ubuntu/+ppas/ [09:56] you can also vist the source package page and it lists PPAs with the same source [09:56] bigjools: so, I visited https://launchpad.net/ - no links towards PPAs there [09:56] i don't have a ppa--i know pretty much nothing about building debs. if i can get a script from james_w or something i can run it as part of my release process [09:56] leonardr: we can hook you up with a script, for sure. [09:56] bigjools: the problem here is that I want social data, not machine data. [09:57] bigjools: teasing you about discoverability aside. [09:57] lifeless: they're linked to distros, but it seems the link from the Ubuntu page has disappeared :/ [09:57] bigjools: also, https://edge.launchpad.net/+search?field.text=launchpadlib is pretty useless (global search) [09:57] oh wait, top right of https://edge.launchpad.net/ubuntu/ [09:57] bigjools: it would be nice for global search to say 'this package is in X PPAS [-> for more]' [09:58] indeed [09:58] need a bug ? [09:58] I think PPAs should be linked from the front page too [09:58] yeah, file it on foundations [09:58] the latter on registry I think? [09:58] what is the latter? [09:59] [09:58:23] I think PPAs should be linked from the front page too [09:59] I wonder what mrevell thinks about that one actually? [09:59] * mrevell reads up [10:00] bigjools, Yeah, I agree. We get complaints that Ubuntu isn't always that easy to find from the LP home page, which is kinda related. [10:00] bigjools, What do you have in mind, other than the link we have already? [10:01] mrevell: on the home page? [10:01] mrevell: PPAs should generally be more prominent I think [10:01] lifeless, In the list below "Launchpad is a software collaboration platform that provides:" ... that you didn't see it suggest it's a failure... [10:02] mrevell: https://launchpad.net/ [10:02] mrevell: where is that text on that page ? [10:02] I don't see it either [10:02] bigjools, We need to redesign the home page generally, I think, so that it's easier to get stuck into some work, as well as just a promotion for LP. [10:02] lifeless, bigjools: Damnit, I'm looking at the "not logged in" version. [10:03] * mrevell logs in [10:03] mrevell: perhaps you have a super secret launchpad instance ? [10:03] mrevell: yeah, the front page has sucked for years :( [10:03] lifeless, bigjools: I have a call just now but I'd love to talk about this some more later. [10:03] mrevell: bug 583707 [10:03] Bug #583707: show a link to PPAs on the launchpad home page [10:04] leonardr: I'll file a couple of small bugs for you re what we just chatted about [10:04] thanks lifeless [10:04] lifeless, great [10:05] leonardr: I think that generally things are heading in a good direction; I appreciate that you have a gargatuan challenge; I see and get your concerns about manpower and API management. [10:05] lifeless: anyway did you find the PPA using the search? [10:05] bigjools: no [10:05] has [10:05] I've been filing bugs [10:05] LP from NZ man. [10:05] bigjools: are you saying there is one ? [10:06] https://edge.launchpad.net/ubuntu/+source/python-launchpadlib [10:06] open the expander at the bottom [10:06] lifeless: supposedly i have another person coming on to the team soon, but even so we don't really have any option but to put a facade on top of what already exists in launchpad [10:06] leonardr: well, thats certainly true while your separate team is totally responsible for the web api [10:07] leonardr: I'm not suggesting that that should change, just noting that its not the entire space of ways-to-structure-the-problem [10:07] agreed [10:07] bigjools: so there are some ppas. That doesn't tell me which are reliable, and which track what leonardr does ;) [10:09] the main alternative is to delegate the work to the launchpad team as a whole. that's what we did for the initial implementation, and it worked out okay [10:09] certainly it was the solution that got us a working web service the fastest [10:10] but, apart from any quibbles i have about the way it was done, i don't think it's reasonable to be able to call on a certain amount of work from everyone on the launchpad team whenever we need to make a web service change. and not all the changes can be rolled out gradually as the different teams schedule work for it [10:11] I kind of feel that your role should be an empowerment one [10:11] lifeless: those ones are sorted by karma [10:11] to help the team scale out [10:11] I think we suck at packaging lplib [10:11] bigjools: which is interesting but doesn't tell me tiether of the things that I need to know. [10:12] leonardr: bug 583710 about some docs [10:12] Bug #583710: debugging docs [10:12] i would like to take on that role as well. the web service needs an 'audit' to make it self-consistent, but after doing that audit i may be able to step back a bit [10:12] in that each team appears keen to garden their part of the API already [10:13] lifeless: we'll be adding PPA download stats and ratings as part of the software centre work [10:13] bigjools: what I want is the answer to ', what PPA do you recommend?' [10:14] but in that role i would like to say, here's a new tool and how to adapt your part of launchpad to use it, rather than direct the creation of entire alternate apis. even distributed over the launchpad team, alternate apis are a lot more work for everyone [10:14] lifeless: what did say? :) [10:14] bigjools: he said 'hook me up with daily buidls' [10:14] heh [10:15] leonardr: yup, if everything else is equal that is obviously massively preferrable. And stays preferrable even when many things aren't equal. [10:17] https://bugs.edge.launchpad.net/launchpadlib/+bug/583712 for PPA packaging [10:17] Bug #583712: nightly / release builds in a PPA? [10:17] its actually pretty easy, but you'll want oh, 3-4 hours set aside to come up the learning curve. [10:19] all right [10:28] leonardr: or someone may take pity and give you a little script to put in your tree and run to do an upload. However I think its advantageous to you to know how to get releases 'up there'. [10:28] On the u1 team, they take responsibility for packaging all the way to Ubuntu for their products. [10:28] even if i get a little script i'll need to hack the script for lazr.restfulclient [10:28] where most of the code lives [10:29] I think this is a good model, for all that it has some overhead, it makes understanding how the distro hangs together (and thus what the crazy folk over there want :P) easier. [10:29] naturally. [11:05] Morning, all. === mrevell is now known as mrevell-lunch === mrevell-lunch is now known as mrevell [12:40] leonardr: around? (Pdb) mp.all_comments[-1] [12:40] *** ValueError: Collection slices must have a nonnegative start point. [12:40] leonardr: is that by design or by time ? [12:41] if the latter, where should I file the bug. [12:41] hmm [12:41] I want to 'show the user the most recent review comment' [12:41] it's by time in that we don't have any fundamental desire to not have that feature [12:41] mp is a branch_merge_proposal [12:42] but, making it work in general is pretty difficult [12:42] sure, but lets go step by step;) [12:42] is the all_comments collection sorted by anything at the moment / [12:43] nothing seems to indicate any which way, that I can see [12:43] i don't know, that's up to the database implementation [12:43] ok [12:43] so separate concerns: is -1 useful for what I need, and how to make -1 work at all, assuming it would be useful. [12:44] right [12:44] so let's assume -1 is what you want [12:44] first step is to find out how big the collection is [12:44] why ? [12:44] actually, where should I record this ? lazr.restful ? [12:45] that depends on the answer to 'why?', actually [12:45] heh, ok. [12:45] we could fix this on the server side or the client side [12:45] [or both?] [12:45] but file it against lazr.restfulclient, because there will definitely be a client component [12:46] let me start over [12:46] in the easy case, all the comments will fit on one page [12:46] so, the client will get /mp/comments and get back a collection that has no next_link [12:47] then it can just give you the last one [12:47] so what we see as a colletion is a series of pages [in the VM sense] with a forward pointer ? [12:47] yes [12:47] you can simulate this with [x for x in mp.comments][-1], since it's unlikely a merge proposal will have a huge number of comments [12:48] right [12:48] I'm just doing 'is there a better way' analysis [12:48] yeah [12:48] well, you're doing it. [12:48] I'm joe schmo asking silly questions :) [12:48] let's consider what happens when there are too many items to fit on one page [12:49] let me say something and then we can optimize it [12:49] we can make a request for /mp/comments. we'll get back a document full of useless comments, but we will also get back the total size of the collection [12:49] let's say it's 780 [12:49] thats pages or items ? [12:49] items [12:50] we can then calculate 780-1=779 on the client side [12:50] and make a second request for /mp/comments?ws.start=779 [12:50] that will give us -1 [12:50] that will work without changing the server, but it requires two requests, one of which will send a lot of useless data [12:51] sure [12:51] server side could accept -1 though, right? and give us a 'last page' [12:51] yes, that's what i was thinking of [12:52] the client could make a request to /mp/comments?ws.start=-1 and get back a collection of one item [12:52] one request, less bandwidth [12:53] or perhaps, assume negative iteration, and give the last page-count items [12:53] that would make ws.start mean different things--a page if the number was negative, an offset if the number was positive [12:53] RTT dominates bandwidth I'd expect here. Clearly tunable later though, which ever way [12:54] leonardr: ah, I see what you mean. [12:54] leonardr: I was thinking the vector would invert for a -value [12:54] so it would mean the same thing [12:54] go 'X' away from the origin [12:54] if we consider the collection a circle [12:54] consider this: [12:54] ok [12:55] collection[-1]; collection[-2]; .... [12:55] wouldn't want to send us all the things we have every time. [12:55] OTOH the client could cater for that and supply absolute values after the first one [12:55] as well as a ws.stop [12:55] * lifeless guesses at parameter names [12:55] ws.size [12:55] it's the same as we use internally, i think [12:55] in launchpad [12:57] well, if lazr.restful sees -1 and sends the last page lazr.restfulclient would still handle that wouldn't it? [12:57] you don't get one element if you do collection[0] [12:57] or at least, could handle it [12:58] leonardr: that seems to be pretty easy actually ;). Anything else I should capture ? [12:58] yes, but i still think it would be inconsistent for lazr.restful to see -1 and send the last *page* [12:58] james_w: I would think so, yes. [12:58] why? [12:58] if you do collection[0] lazr.restful gets a page _starting_ at 0 [12:58] * lifeless waves the I have a bug open flag [12:58] lifeless: what we have now is fine [12:59] leonardr: thanks! brain dump completed, your ticket number is bug 583761, have a nice day. [12:59] i guess lazr.restfulclient could get the largest possible page that _includes_ the item you want [12:59] Bug #583761: collection[-1] raises an error : *** ValueError: Collection slices must have a nonnegative start point. [12:59] but without other changes, that wouldn't buy us anything. we don't cache collections client-side because we have no idea of a max-age for the cache and no algorithms for invalidating the cache === matsubara-afk is now known as matsubara [13:00] the most efficient thing to do when given a single-item slice is to make a request for that specific item--anything else uses more bandwidth without reducing the number of http requests [13:00] (at least i'm pretty sure that's how it works) [13:01] ok [13:01] so [0] only gets one item as well ? [13:01] no [0] gets a page [13:02] and that page will be reused if you then to [1] I believe [13:02] hang on, thats inconsistent [13:02] with not caching collections client-side [13:03] actually, that's a bit general [13:03] let me find out what actually happens [13:03] if you do [X] then it gets a page [13:03] but [X+1] will not re-use that page and do a new GET [13:04] so the current behavior uses more bandwidth than it should, given how many http requests it makes [13:04] however, there is a special case, if on the first request you get a representation then that is stored, and any request from that page will re-use it [13:04] james_w, do you have a line number handy? [13:04] 722 resource.py [13:05] http://paste.ubuntu.com/437272/ [13:05] so we only keep one 'page' (the first page) in memory [13:06] this has all the problems of client-side caching [13:06] an app could run for 3 days, contain inaccurate data, and the client would never know [13:06] but in most ordinary cases it's not a big deal [13:07] yes [13:07] however, if we start getting serious about this, caching the 'last' page or any other pages of the collection, it would become a bigger deal [13:09] I really need to track /usr/lib/pymodules/python2.6/lazr/uri/__init__.py:19: UserWarning: Module launchpadlib was already imported from /usr/lib/pymodules/python2.6/launchpadlib/__init__.py, but /usr/lib/pymodules/python2.6 is being added to sys.path [13:09] import pkg_resources [13:09] it is tres annoying [13:09] it may make sense to limit our caching of the first 'page' to the results of named operations, which are very expensive, always deliver a single 'page' of results, and are represented as ad hoc collections rather than as things you can get to by dereferencing attributes of launchpad [13:09] down [13:09] lifeless, gary might be able to help, if james_w can't [13:10] oh, he has mail ;) [13:10] I have NFI, I've briefly looked previously [13:10] james_w: do you see it ? [13:10] the message is kind of silly anyway [13:10] yeah [13:10] spiv doesn't [13:10] can't remember what from though [13:10] he's on lucid too [13:11] james_w: try hydrazine, bughugger, pqm ? [13:11] lp-propose [13:11] don't think it was them [13:11] so, summary: once we have some standards for client-side cache, it would make sense to have lazr.restfulclient ask for a whole page when you try [-1]. until that point, it makes sense to have lazr.restfulclient ask for a page just big enough to handle whatever slice it's given [13:11] could you try 'bzr lp-propose' somewhere ? [13:12] hmmm [13:12] that was unexpected [13:12] File "/usr/lib/pymodules/python2.6/lazr/restfulclient/resource.py", line 860, in __getitem__ [13:12] raise KeyError(key) [13:12] KeyError: u'' [13:13] bzr bug, not lp - don't bother looking for it [13:15] bac, ping [13:16] lifeless: I don't get it from lp-propose [13:16] interesting [13:16] what if you run from source? [13:17] I am running from source [13:17] ok [13:17] so its not the packaged status [13:17] and its not the python code itself, as lp-propose does it for me [13:18] :!echo $PYTHONPATH [13:18] /home/robertc/lib/python:/home/robertc/local/lib/python2.6/dist-packages/ [13:22] anyhow, getting close to 1am, sleeeeeeep === Ursinha-afk is now known as Ursinha [14:05] BjornT, hi. Did we ever support +bugs-text for a person context in the past? [14:06] deryck: hmm. can't remember. if we don't today, then probably not. [14:07] BjornT, yeah, this is my impression, just on glancing at the code. That we don't and likely never did. [15:27] hey, [15:27] I got some issues when trying to branch: $ bzr branch lp:~ubuntu-branches/debian/sid/evince/sid/ debian [15:27] bzr: ERROR: Invalid http response for https://xmlrpc.edge.launchpad.net/bazaar/: Unable to handle http code 503: Service Unavailable [15:27] (for the last few minutes) [15:28] ah, this time it's ok, temporary issue so :) === deryck is now known as deryck[lunch] [15:43] henninge: merged and pushed [15:43] lp:~adiroiban/launchpad/bug-525992 === gary_poster is now known as gary-lunch === deryck[lunch] is now known as deryck === beuno is now known as beuno-lunch === salgado is now known as salgado-lunch === gary-lunch is now known as gary_poster === beuno-lunch is now known as beuno === salgado-lunch is now known as salgado [18:15] * jml is gone [18:42] gary_poster: hi [18:43] lifeless, hi [18:43] saw your question on bug [18:43] haven't gotten around to answering, but [18:43] I was wondering, since I seem to be awake at 5am, if we could talk about that warning [18:43] :-) [18:43] basic answer is two parts: [18:44] - This is typically a packaging issue, not a software issue [18:44] - I'm happy to help. I haven't looked at the bug again to see if there are instructions to dupe but was going to [18:44] ok [18:45] so I appreciate that it may well be a 'what some distro dude did' cause, but.... [18:45] (Also, in regards to bug about packaging, yes, would like to have launchpadlib et al in a PPA) [18:45] on all our products, we're judge by how they deploy, not how they run on our machines :) [18:45] s/judge/judged/ [18:45] I'm glad you're happy to help. [18:45] yup [18:45] that makes me happy, because otherwise I suspect it would bitrot [18:46] now, for reproduction [18:46] it seems to be spotty [18:46] e.g. spiv doesn't see it [18:46] I see it reliably [18:46] james westby sees it occasionally and can't remember where from [18:46] typical for these things :-/ [18:46] When it first occured I dug down and looks at the setuptools code in question - ugh :( [18:46] often means there's either a package installed on your system causing an interaction [18:47] or an old package that left stuff around on an upgrade or deletion [18:47] yeah, ugh :-/ [18:47] ok, so one posibility is a stale pth file or similar ? [18:47] uhm [18:48] perhaps makign sys.path a smarter object in site.py would be a good way to test? [18:48] get a traceback on every change made to it [18:48] meh, maybe [18:48] well, we don't know whats triggering [18:49] yeah I know [18:49] It's just...there's probably going to be a lot of noise [18:49] oh, sure. [18:49] but I don't have a better idea. thinking, [18:49] but we can filter for the path we care about in the first instance [18:49] and if that doesn't help go broader [18:49] lemme see the bug, how's that for a crazy idea [18:49] \o/ [18:50] ok got it [18:54] .pyc files from older versions aren't out of the question when packages have been installed and upgraded [18:55] in other locations I mean [18:58] heh, yeah, these are always fun. "import pkg_resources" is the trigger, with lazr.restfulclient not doing anything else. this isn't lazr.restfulclient's problem (and possibly not even the problem in the packaging) but your system + setuptools (+ our Ubuntu python dances? not sure). [18:58] lifeless, if it were on my system, I'd put a pdb where that UserWarning is raised. /usr/lib/python2.6/dist-packages/pkg_resources.py:2334 . Then I'd see if I could see what's going on. [18:58] There's a high probability that the problem has already occurred at that point. Your site.py idea might help then, though...there might be a nicer place to hook into the process in site.py. looking there, one sec [18:59] Maybe instrumenting site.addpackage? [18:59] And if that isn't informative enough, site.addsitedir [18:59] * gary_poster wonders if python -v would help [19:00] nah, doesn't look like it [19:01] * gary_poster goes back to regularly scheduled programming until he is pinged again [19:02] -v gives http://pastebin.com/PgizKBdY [19:02] for the curious [19:02] this is interesting [19:02] # /usr/lib/pymodules/python2.6/launchpadlib/__init__.pyc has bad mtime [19:02] but may be from me fiddling [19:03] moving the file doesn't fix it, its noise [19:04] running feed-pqm as root doesn't magically fix it either [19:04] actually I lie [19:04] it did [19:04] for the next run. [19:05] so a) definitely something stale and b) I can't tell you what, now. [19:06] ok [19:06] um [19:07] so the problem has gone away, entirely, after running as root? Or it did not occur when you ran as root? [19:08] occured as root, and is now gone [19:08] I'm doing a find for modified files [19:08] ok [21:11] weird. I just did a merge of a branch from an MP and the changes are dramatically different from the diff in the MP. [21:17] ah [21:17] my branch wasn't up2date :) === Ursinha is now known as Ursinha-afk === james_w` is now known as james_w [22:32] someone intimate with launchpad lib needs to check if much of bzrlib.plugins.launchpad.lp_api is still needed === salgado is now known as salgado-afk === matsubara is now known as matsubara-afk [22:39] lifeless: LAUNCHPAD_API_URLS can probably be dropped, you can now pass 'edge' etc. to the factory methods [22:39] it's taking something that isn't a plain string though [22:39] I don't know that the chmod that is in login() is in launchpadlib yet? [22:41] the rest looks like it is still needed [22:48] thanks