/srv/irclogs.ubuntu.com/2010/05/21/#launchpad-dev.txt

* thumper has lunch calling01:49
lifelessgary_poster: ping, if you're around03:03
lifelessgary_poster: I'd like https://bugs.edge.launchpad.net/launchpadlib/+bug/552419 to be addressed by squelching the warning; we can do it in bzrlib but really doing it in launchpad would be closer to the source03:04
mupBug #552419: Multiple module import warning from ubuntu-bug <apport-bug> <i386> <lucid> <launchpadlib :Invalid> <python-launchpadlib (Ubuntu):Confirmed> <https://launchpad.net/bugs/552419>03:04
thumperhmm...03:36
thumperwhere to general launchpad bugs live?03:36
thumperas in cross app ones?03:37
thumperah ha03:38
thumperlaunchpad-web03:38
=== Ursinha is now known as Ursinha-afk
=== henninge_ is now known as henninge
=== g4tsu_ is now known as g4tsu
adeuringgood morning08:28
mrevellMorning!08:53
leonardrlifeless, i woke up very early--if you're around and want to talk about web service things, let me know09:49
lifelesshey09:50
lifelessI'd love to know enough to get useful traces of behaviour for you09:50
lifelessI like some of the things you're talking about09:50
leonardrtraces of behavior == my plan?09:51
lifelesstraces of behavior == what requests are made by launchpadlib when do something that uses API's09:51
lifelesss/when do/when I do/09:51
leonardrah09:52
lifelesslike you pasted in your email09:52
lifelessyou know precisely whats going on and going wrong09:52
leonardrsure09:52
leonardrthe simplest way to do it is:09:52
leonardrimport httplib209:52
leonardrhttplib2.debuglevel = 109:52
lifelesshow would you feel about that being in the docstring for launchpadlib/__init__.py ?09:52
leonardrsounds good09:52
lifelesswould you like me to propose a merge for that ?09:53
lifelessor do you want to JFDI ? :)09:53
leonardrsure. i can also put timing information in lazr.restfulclient that's controlled by httplib2.debuglevel or a separate debuglevel09:53
leonardrhttplib2's debuglevel won't give your elapsed time09:53
lifelessok09:53
lifelessis there a PPA with latest (launchpadlib and deps) ?09:54
lifelessmaybe not nightlies, but something moving faster than lucid :)09:54
bigjoolswe have this great search facility for PPAs09:55
bigjools*cough* :)09:55
lifelessbigjools: we do?09:55
bigjoolshttps://edge.launchpad.net/ubuntu/+ppas/09:55
bigjoolsyou can also vist the source package page and it lists PPAs with the same source09:56
lifelessbigjools: so, I visited https://launchpad.net/ - no links towards PPAs there09:56
leonardri don't have a ppa--i know pretty much nothing about building debs. if i can get a script from james_w or something i can run it as part of my release process09:56
lifelessleonardr: we can hook you up with a script, for sure.09:56
lifelessbigjools: the problem here is that I want social data, not machine data.09:56
lifelessbigjools: teasing you about discoverability aside.09:57
bigjoolslifeless: they're linked to distros, but it seems the link from the Ubuntu page has disappeared :/09:57
lifelessbigjools: also, https://edge.launchpad.net/+search?field.text=launchpadlib  is pretty useless (global search)09:57
bigjoolsoh wait, top right of https://edge.launchpad.net/ubuntu/09:57
lifelessbigjools: it would be nice for global search to say 'this package is in X PPAS [-> for more]'09:57
bigjoolsindeed09:58
lifelessneed a bug ?09:58
bigjoolsI think PPAs should be linked from the front page too09:58
bigjoolsyeah, file it on foundations09:58
bigjoolsthe latter on registry I think?09:58
lifelesswhat is the latter?09:58
bigjools[09:58:23] <bigjools> I think PPAs should be linked from the front page too09:59
bigjoolsI wonder what mrevell thinks about that one actually?09:59
* mrevell reads up09:59
mrevellbigjools, Yeah, I agree. We get complaints that Ubuntu isn't always that easy to find from the LP home page, which is kinda related.10:00
mrevellbigjools, What do you have in mind, other than the link we have already?10:00
lifelessmrevell: on the home page?10:01
bigjoolsmrevell: PPAs should generally be more prominent I think10:01
mrevelllifeless, In the list below "Launchpad is a software collaboration platform that provides:" ... that you didn't see it suggest it's a failure...10:01
lifelessmrevell: https://launchpad.net/10:02
lifelessmrevell: where is that text on that page ?10:02
bigjoolsI don't see it either10:02
mrevellbigjools, We need to redesign the home page generally, I think, so that it's easier to get stuck into some work, as well as just a promotion for LP.10:02
mrevelllifeless, bigjools: Damnit, I'm looking at the "not logged in" version.10:02
* mrevell logs in10:03
lifelessmrevell: perhaps you have a super secret launchpad instance ?10:03
bigjoolsmrevell: yeah, the front page has sucked for years :(10:03
mrevelllifeless, bigjools: I have a call just now but I'd love to talk about this some more later.10:03
lifelessmrevell: bug 58370710:03
mupBug #583707: show a link to PPAs on the launchpad home page <Launchpad Registry:New> <https://launchpad.net/bugs/583707>10:03
lifelessleonardr: I'll file a couple of small bugs for you re what we just chatted about10:04
mrevellthanks lifeless10:04
leonardrlifeless, great10:04
lifelessleonardr: I think that generally things are heading in a good direction; I appreciate that you have a gargatuan challenge; I see and get your concerns about manpower and API management.10:05
bigjoolslifeless: anyway did you find the PPA using the search?10:05
lifelessbigjools: no10:05
bigjoolshas10:05
lifelessI've been filing bugs10:05
lifelessLP from NZ man.10:05
lifelessbigjools: are you saying there is one ?10:05
bigjoolshttps://edge.launchpad.net/ubuntu/+source/python-launchpadlib10:06
bigjoolsopen the expander at the bottom10:06
leonardrlifeless: supposedly i have another person coming on to the team soon, but even so we don't really have any option but to put a facade on top of what already exists in launchpad10:06
lifelessleonardr: well, thats certainly true while your separate team is totally responsible for the web api10:06
lifelessleonardr: I'm not suggesting that that should change, just noting that its not the entire space of ways-to-structure-the-problem10:07
leonardragreed10:07
lifelessbigjools: so there are some ppas. That doesn't tell me which are reliable, and which track what leonardr does ;)10:07
leonardrthe main alternative is to delegate the work to the launchpad team as a whole. that's what we did for the initial implementation, and it worked out okay10:09
leonardrcertainly it was the solution that got us a working web service the fastest10:09
leonardrbut, apart from any quibbles i have about the way it was done, i don't think it's reasonable to be able to call on a certain amount of work from everyone on the launchpad team whenever we need to make a web service change. and not all the changes can be rolled out gradually as the different teams schedule work for it10:10
lifelessI kind of feel that your role should be an empowerment one10:11
bigjoolslifeless: those ones are sorted by karma10:11
lifelessto help the team scale out10:11
bigjoolsI think we suck at packaging lplib10:11
lifelessbigjools: which is interesting but doesn't tell me tiether of the things that I need to know.10:11
lifelessleonardr: bug 583710 about some docs10:12
mupBug #583710: debugging docs <launchpadlib :New> <https://launchpad.net/bugs/583710>10:12
leonardri would like to take on that role as well. the web service needs an 'audit' to make it self-consistent, but after doing that audit i may be able to step back a bit10:12
lifelessin that each team appears keen to garden their part of the API already10:12
bigjoolslifeless: we'll be adding PPA download stats and ratings as part of the software centre work10:13
lifelessbigjools: what I want is the answer to '<project lead>, what PPA do you recommend?'10:13
leonardrbut in that role i would like to say, here's a new tool and how to adapt your part of launchpad to use it, rather than direct the creation of entire alternate apis. even distributed over the launchpad team, alternate apis are a lot more work for everyone10:14
bigjoolslifeless: what did <project lead> say? :)10:14
lifelessbigjools: he said 'hook me up with daily buidls'10:14
bigjoolsheh10:14
lifelessleonardr: yup, if everything else is equal that is obviously massively preferrable. And stays preferrable even when many things aren't equal.10:15
lifelesshttps://bugs.edge.launchpad.net/launchpadlib/+bug/583712 for PPA packaging10:17
mupBug #583712: nightly / release builds in a PPA? <launchpadlib :New> <https://launchpad.net/bugs/583712>10:17
lifelessits actually pretty easy, but you'll want oh, 3-4 hours set aside to come up the learning curve.10:17
leonardrall right10:19
lifelessleonardr: or someone may take pity and give you a little script to put in your tree and run to do an upload. However I think its advantageous to you to know how to get releases 'up there'.10:28
lifelessOn the u1 team, they take responsibility for packaging all the way to Ubuntu for their products.10:28
leonardreven if i get a little script i'll need to hack the script for lazr.restfulclient10:28
leonardrwhere most of the code lives10:28
lifelessI think this is a good model, for all that it has some overhead, it makes understanding how the distro hangs together (and thus what the crazy folk over there want :P) easier.10:29
lifelessnaturally.10:29
deryckMorning, all.11:05
=== mrevell is now known as mrevell-lunch
=== mrevell-lunch is now known as mrevell
lifelessleonardr: around? (Pdb) mp.all_comments[-1]12:40
lifeless*** ValueError: Collection slices must have a nonnegative start point.12:40
lifelessleonardr: is that by design or by time ?12:40
lifelessif the latter, where should I file the bug.12:41
leonardrhmm12:41
lifelessI want to 'show the user the most recent review comment'12:41
leonardrit's by time in that we don't have any fundamental desire to not have that feature12:41
lifelessmp is a branch_merge_proposal12:41
leonardrbut, making it work in general is pretty difficult12:42
lifelesssure, but lets go step by step;)12:42
lifelessis the all_comments collection sorted by anything at the moment /12:42
lifelessnothing seems to indicate any which way, that I can see12:43
leonardri don't know, that's up to the database implementation12:43
lifelessok12:43
lifelessso separate concerns: is -1 useful for what I need, and how to make -1 work at all, assuming it would be useful.12:43
leonardrright12:44
leonardrso let's assume -1 is what you want12:44
leonardrfirst step is to find out how big the collection is12:44
lifelesswhy ?12:44
lifelessactually, where should I record this ? lazr.restful ?12:44
leonardrthat depends on the answer to 'why?', actually12:45
lifelessheh, ok.12:45
leonardrwe could fix this on the server side or the client side12:45
lifeless[or both?]12:45
leonardrbut file it against lazr.restfulclient, because there will definitely be a client component12:45
leonardrlet me start over12:46
leonardrin the easy case, all the comments will fit on one page12:46
leonardrso, the client will get /mp/comments and get back a collection that has no next_link12:46
leonardrthen it can just give you the last one12:47
lifelessso what we see as a colletion is a series of pages [in the VM sense] with a forward pointer ?12:47
leonardryes12:47
leonardryou can simulate this with [x for x in mp.comments][-1], since it's unlikely a merge proposal will have a huge number of comments12:47
lifelessright12:48
lifelessI'm just doing 'is there a better way' analysis12:48
leonardryeah12:48
lifelesswell, you're doing it.12:48
lifelessI'm joe schmo asking silly questions :)12:48
leonardrlet's consider what happens when there are too many items to fit on one page12:48
leonardrlet me say something and then we can optimize it12:49
leonardrwe can make a request for /mp/comments. we'll get back a document full of useless comments, but we will also get back the total size of the collection12:49
leonardrlet's say it's 78012:49
lifelessthats pages or items ?12:49
leonardritems12:49
leonardrwe can then calculate 780-1=779 on the client side12:50
leonardrand make a second request for /mp/comments?ws.start=77912:50
leonardrthat will give us -112:50
leonardrthat will work without changing the server, but it requires two requests, one of which will send a lot of useless data12:50
lifelesssure12:51
lifelessserver side could accept -1 though, right? and give us a 'last page'12:51
leonardryes, that's what i was thinking of12:51
leonardrthe client could make a request to /mp/comments?ws.start=-1 and get back a collection of one item12:52
leonardrone request, less bandwidth12:52
lifelessor perhaps, assume negative iteration, and give the last page-count items12:53
leonardrthat would make ws.start mean different things--a page if the number was negative, an offset if the number was positive12:53
lifelessRTT dominates bandwidth I'd expect here. Clearly tunable later though, which ever way12:53
lifelessleonardr: ah, I see what you mean.12:54
lifelessleonardr: I was thinking the vector would invert for a -value12:54
lifelessso it would mean the same thing12:54
lifelessgo 'X' away from the origin12:54
lifelessif we consider the collection a circle12:54
lifelessconsider this:12:54
leonardrok12:54
lifelesscollection[-1]; collection[-2]; ....12:55
lifelesswouldn't want to send us all the things we have every time.12:55
lifelessOTOH the client could cater for that and supply absolute values after the first one12:55
lifelessas well as a ws.stop12:55
* lifeless guesses at parameter names12:55
leonardrws.size12:55
leonardrit's the same as we use internally, i think12:55
leonardrin launchpad12:55
james_wwell, if lazr.restful sees -1 and sends the last page lazr.restfulclient would still handle that wouldn't it?12:57
james_wyou don't get one element if you do collection[0]12:57
james_wor at least, could handle it12:57
lifelessleonardr: that seems to be pretty easy actually ;). Anything else I should capture ?12:58
leonardryes, but i still think it would be inconsistent for lazr.restful to see -1 and send the last *page*12:58
lifelessjames_w: I would think so, yes.12:58
james_wwhy?12:58
leonardrif you do collection[0] lazr.restful gets a page _starting_ at 012:58
* lifeless waves the I have a bug open flag12:58
leonardrlifeless: what we have now is fine12:58
lifelessleonardr: thanks! brain dump completed, your ticket number is bug 583761, have a nice day.12:59
leonardri guess lazr.restfulclient could get the largest possible page that _includes_ the item you want12:59
mupBug #583761: collection[-1] raises an error : *** ValueError: Collection slices must have a nonnegative start point. <lazr.restfulclient:New> <https://launchpad.net/bugs/583761>12:59
leonardrbut without other changes, that wouldn't buy us anything. we don't cache collections client-side because we have no idea of a max-age for the cache and no algorithms for invalidating the cache12:59
=== matsubara-afk is now known as matsubara
leonardrthe most efficient thing to do when given a single-item slice is to make a request for that specific item--anything else uses more bandwidth without reducing the number of http requests13:00
leonardr(at least i'm pretty sure that's how it works)13:00
lifelessok13:01
lifelessso [0] only gets one item as well ?13:01
james_wno [0] gets a page13:01
james_wand that page will be reused if you then to [1] I believe13:02
lifelesshang on, thats inconsistent13:02
lifelesswith not caching collections client-side13:02
james_wactually, that's a bit general13:03
leonardrlet me find out what actually happens13:03
james_wif you do [X] then it gets a page13:03
james_wbut [X+1] will not re-use that page and do a new GET13:03
leonardrso the current behavior uses more bandwidth than it should, given how many http requests it makes13:04
james_whowever, there is a special case, if on the first request you get a representation then that is stored, and any request from that page will re-use it13:04
leonardrjames_w, do you have a line number handy?13:04
james_w722 resource.py13:04
james_whttp://paste.ubuntu.com/437272/13:05
leonardrso we only keep one 'page' (the first page) in memory13:05
leonardrthis has all the problems of client-side caching13:06
leonardran app could run for 3 days, contain inaccurate data, and the client would never know13:06
leonardrbut in most ordinary cases it's not a big deal13:06
james_wyes13:07
leonardrhowever, if we start getting serious about this, caching the 'last' page or any other pages of the collection, it would become a bigger deal13:07
lifelessI really need to track /usr/lib/pymodules/python2.6/lazr/uri/__init__.py:19: UserWarning: Module launchpadlib was already imported from /usr/lib/pymodules/python2.6/launchpadlib/__init__.py, but /usr/lib/pymodules/python2.6 is being added to sys.path13:09
lifeless  import pkg_resources13:09
lifelessit is tres annoying13:09
leonardrit may make sense to limit our caching of the first 'page' to the results of named operations, which are very expensive, always deliver a single 'page' of results, and are represented as ad hoc collections rather than as things you can get to by dereferencing attributes of launchpad13:09
lifeless down13:09
leonardrlifeless, gary might be able to help, if james_w can't13:09
lifelessoh, he has mail ;)13:10
james_wI have NFI, I've briefly looked previously13:10
lifelessjames_w: do you see it ?13:10
james_wthe message is kind of silly anyway13:10
james_wyeah13:10
lifelessspiv doesn't13:10
james_wcan't remember what from though13:10
lifelesshe's on lucid too13:10
lifelessjames_w: try hydrazine, bughugger, pqm ?13:11
lifelesslp-propose13:11
james_wdon't think it was them13:11
leonardrso, summary: once we have some standards for client-side cache, it would make sense to have lazr.restfulclient ask for a whole page when you try [-1]. until that point, it makes sense to have lazr.restfulclient ask for a page just big enough to handle whatever slice it's given13:11
lifelesscould you try 'bzr lp-propose' somewhere ?13:11
lifelesshmmm13:12
lifelessthat was unexpected13:12
lifeless  File "/usr/lib/pymodules/python2.6/lazr/restfulclient/resource.py", line 860, in __getitem__13:12
lifeless    raise KeyError(key)13:12
lifelessKeyError: u''13:12
lifelessbzr bug, not lp - don't bother looking for it13:13
matsubarabac, ping13:15
james_wlifeless: I don't get it from lp-propose13:16
lifelessinteresting13:16
lifelesswhat if you run from source?13:16
james_wI am running from source13:17
lifelessok13:17
lifelessso its not the packaged status13:17
lifelessand its not the python code itself, as lp-propose does it for me13:17
lifeless:!echo $PYTHONPATH13:18
lifeless/home/robertc/lib/python:/home/robertc/local/lib/python2.6/dist-packages/13:18
lifelessanyhow, getting close to 1am, sleeeeeeep13:22
=== Ursinha-afk is now known as Ursinha
deryckBjornT, hi.  Did we ever support +bugs-text for a person context in the past?14:05
BjornTderyck: hmm. can't remember. if we don't today, then probably not.14:06
deryckBjornT, yeah, this is my impression, just on glancing at the code.  That we don't and likely never did.14:07
didrockshey,15:27
didrocksI got some issues when trying to branch: $ bzr branch lp:~ubuntu-branches/debian/sid/evince/sid/ debian15:27
didrocksbzr: ERROR: Invalid http response for https://xmlrpc.edge.launchpad.net/bazaar/: Unable to handle http code 503: Service Unavailable15:27
didrocks(for the last few minutes)15:27
didrocksah, this time it's ok, temporary issue so :)15:28
=== deryck is now known as deryck[lunch]
adiroibanhenninge: merged and pushed15:43
adiroibanlp:~adiroiban/launchpad/bug-52599215:43
=== gary_poster is now known as gary-lunch
=== deryck[lunch] is now known as deryck
=== beuno is now known as beuno-lunch
=== salgado is now known as salgado-lunch
=== gary-lunch is now known as gary_poster
=== beuno-lunch is now known as beuno
=== salgado-lunch is now known as salgado
* jml is gone18:15
lifelessgary_poster: hi18:42
gary_posterlifeless, hi18:43
gary_postersaw your question on bug18:43
gary_posterhaven't gotten around to answering, but18:43
lifelessI was wondering, since I seem to be awake at 5am, if we could talk about that warning18:43
gary_poster:-)18:43
gary_posterbasic answer is two parts:18:43
gary_poster- This is typically a packaging issue, not a software issue18:44
gary_poster- I'm happy to help.  I haven't looked at the bug again to see if there are instructions to dupe but was going to18:44
lifelessok18:44
lifelessso I appreciate that it may well be a 'what some distro dude did' cause, but....18:45
gary_poster(Also, in regards to bug about packaging, yes, would like to have launchpadlib et al in a PPA)18:45
lifelesson all our products, we're judge by how they deploy, not how they run on our machines :)18:45
lifelesss/judge/judged/18:45
lifelessI'm glad you're happy to help.18:45
gary_posteryup18:45
lifelessthat makes me happy, because otherwise I suspect it would bitrot18:45
lifelessnow, for reproduction18:46
lifelessit seems to be spotty18:46
lifelesse.g. spiv doesn't see it18:46
lifelessI see it reliably18:46
lifelessjames westby sees it occasionally and can't remember where from18:46
gary_postertypical for these things :-/18:46
lifelessWhen it first occured I dug down and looks at the setuptools code in question - ugh  :(18:46
gary_posteroften means there's either a package installed on your system causing an interaction18:46
gary_posteror an old package that left stuff around on an upgrade or deletion18:47
gary_posteryeah, ugh :-/18:47
lifelessok, so one posibility is a stale pth file or similar ?18:47
lifelessuhm18:47
lifelessperhaps makign sys.path a smarter object in site.py would be a good way to test?18:48
lifelessget a traceback on every change made to it18:48
gary_postermeh, maybe18:48
lifelesswell, we don't know whats triggering18:48
gary_posteryeah I know18:49
gary_posterIt's just...there's probably going to be a lot of noise18:49
lifelessoh, sure.18:49
gary_posterbut I don't have a better idea.  thinking,18:49
lifelessbut we can filter for the path we care about in the first instance18:49
lifelessand if that doesn't help go broader18:49
gary_posterlemme see the bug, how's that for a crazy idea18:49
lifeless\o/18:49
gary_posterok got it18:50
james_w.pyc files from older versions aren't out of the question when packages have been installed and upgraded18:54
james_win other locations I mean18:55
gary_posterheh, yeah, these are always fun.  "import pkg_resources" is the trigger, with lazr.restfulclient not doing anything else.  this isn't lazr.restfulclient's problem (and possibly not even the problem in the packaging) but your system + setuptools (+ our Ubuntu python dances?  not sure).18:58
gary_posterlifeless, if it were on my system, I'd put a pdb where that UserWarning is raised.  /usr/lib/python2.6/dist-packages/pkg_resources.py:2334 .  Then I'd see if I could see what's going on.18:58
gary_posterThere's a high probability that the problem has already occurred at that point.  Your site.py idea might help then, though...there might be a nicer place to hook into the process in site.py.  looking there, one sec18:58
gary_posterMaybe instrumenting site.addpackage?18:59
gary_posterAnd if that isn't informative enough, site.addsitedir18:59
* gary_poster wonders if python -v would help18:59
gary_posternah, doesn't look like it19:00
* gary_poster goes back to regularly scheduled programming until he is pinged again19:01
lifeless-v gives http://pastebin.com/PgizKBdY19:02
lifelessfor the curious19:02
lifelessthis is interesting19:02
lifeless# /usr/lib/pymodules/python2.6/launchpadlib/__init__.pyc has bad mtime19:02
lifelessbut may be from me fiddling19:02
lifelessmoving the file doesn't fix it, its noise19:03
lifelessrunning feed-pqm as root doesn't magically fix it either19:04
lifelessactually I lie19:04
lifelessit did19:04
lifelessfor the next run.19:04
lifelessso a) definitely something stale and b) I can't tell you what, now.19:05
gary_posterok19:06
gary_posterum19:06
gary_posterso the problem has gone away, entirely, after running as root?  Or it did not occur when you ran as root?19:07
lifelessoccured as root, and is now gone19:08
lifelessI'm doing a find for modified files19:08
gary_posterok19:08
cody-somervilleweird. I just did a merge of a branch from an MP and the changes are dramatically different from the diff in the MP.21:11
cody-somervilleah21:17
cody-somervillemy branch wasn't up2date :)21:17
=== Ursinha is now known as Ursinha-afk
=== james_w` is now known as james_w
lifelesssomeone intimate with launchpad lib needs to check if much of bzrlib.plugins.launchpad.lp_api is still needed22:32
=== salgado is now known as salgado-afk
=== matsubara is now known as matsubara-afk
james_wlifeless: LAUNCHPAD_API_URLS can probably be dropped, you can now pass 'edge' etc. to the factory methods22:39
james_wit's taking something that isn't a plain string though22:39
james_wI don't know that the chmod that is in login() is in launchpadlib yet?22:39
james_wthe rest looks like it is still needed22:41
lifelessthanks22:48

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!