/srv/irclogs.ubuntu.com/2010/08/25/#launchpad-dev.txt

lifelessenums seems to be our canned answer for this00:00
mwhudsoni think it's a big case of WHUI00:01
lifelessWHUI ?00:01
mwhudsonWe Haven't Used It00:01
mwhudsona SteveA-ism00:01
lifelessgoogle was ... un helpful00:01
mwhudsonsomething like KarmaAction is in the db so we can tweak the constants to adjust the way karma is allocated without landing code changes00:02
lifelessmwhudson: so, we're talking IRequest00:02
mwhudsonbut of course we don't00:02
lifelessetc00:02
mwhudsonstub will know more00:02
mwhudsonlifeless: i'm actually replying to your email00:02
lifelessmwhudson: cool00:02
lifelessmwhudson: the big question, is an unknown00:02
lifelessmwhudson: is it shallow enough I should plunge on and doit00:03
lifelessmwhudson: or should I thread-locals-it.00:03
mwhudsonlifeless: what's 'it' ?00:03
wgrantlifeless: 'karmacategories' in http://bazaar.launchpad.net/~wgrant/launchpad/bootstrap-db-from-scratch/annotate/head:/utilities/bootstrap-lp-db is the data in question.00:04
lifelessmwhudson: it is00:04
lifelessmwhudson: making scripts have an IRequest always, so that when they do sql it is logged in my new code00:05
lifelessthe second it00:05
lifelessis00:05
lifelessthe new code : change requesttimeline to be a threadslocal thing00:05
mwhudsonlifeless: ok, my email is nearly done00:08
mwhudson    # This is a convenient hack to set up a zope interaction, before we get00:10
mwhudson    # the proper API for having a principal / user running in scripts.00:10
mwhudson    # The script will have full permissions because of the00:10
mwhudson    # PermissiveSecurityPolicy set up in script.zcml.00:10
mwhudsonha ha00:10
mwhudsoni wonder when that was written00:10
* mwhudson bets on 200500:10
* mwhudson wins00:11
mwhudsontimestamp: Tue 2005-04-12 09:37:50 +000000:12
mwhudsonfrom the arch days00:12
mwhudsonsteve.alexander@canonical.com/launchpad--devel--0--patch-36800:12
mwhudsonlifeless: ok, mail sent00:17
lifelessthanks00:27
lifelessmwhudson: followup sent btw00:37
mwhudsonlifeless: i replied one more time, happy to talk about it in irc now :-)00:46
mwhudsonalthough i don't think there's much need00:46
* mtaylor thinks you're both wrong and obviously everything should be re-written in google go00:50
* mtaylor falls on the floor laughing00:50
* mtaylor is obviously in an odd mood00:51
StevenKmwhudson: Patches welcome00:51
StevenKDoh00:51
StevenKIRC tab fail :-(00:52
lifelessmwhudson: I think I'm good.01:02
lifelessmwhudson: I guess that under setupInteractionByEmail(ANONYMOUS) in script base01:02
lifelessmwhudson: I'll add something??? that sets up a participationwithannotations ?01:03
mwhudsonlifeless: setupInteractionByEmail takes a participation as an argument01:08
lifelessmwhudson: yeah01:16
lifelessactually though01:16
lifelessset_request_started is where scripts expect to do stuff01:17
lifelessso *it* needs to check and see if there is a participiation...ICanHasAnnotations, and if not setone up01:17
lifelesswe still need to unify these two things01:17
lifelessI like your approach, but I'm not sure we don't actually want - eventually - scripts to say they are in requests via participations rather than set_request_started01:18
mwhudsoni admit i don't really know what set_request_started is about01:19
mwhudsonlifeless: in the particular case of checkwatches, it does it's one interaction management01:23
mwhudson-'01:24
lifelessmwhudson: kindof-management02:05
lifelessmwhudson: but yes02:05
mwhudsonlifeless: it looks like it probably suffers locally from the kind of confusion it would be good to clear up globally02:06
lifelesswith_interaction looks like exactly what needs clearing up02:07
lifelessvis-a-vis transactions, security & context02:08
lifelessalso @statement_logging is just bong02:08
mwhudsoni'm not sure i'm correct or expressing this clearly02:10
mwhudsonbut i wonder if there's a bit of a tension between things that don't care at all about participations (like most scripts and tests) and the few things that do (like checkwatches)02:10
lifelessmost scripts do work on behalf of someone02:11
mwhudsonalso, i wonder if thinking about how you'd like stuff bundled up in an oops report is a good guideline towards how long your participations should be current for02:11
lifelessthe work doesn't just 'appear'02:11
lifelessI think that bundling point is exactly on the spot02:12
lifelessits certainly how I think abou tit02:12
lifelessexcuse me; brain flagging food needed02:13
mwhudsonso for example, each job you process in a job running script should have it's own participation02:13
* mwhudson reads errorlog.py, is surprised to find it reads db statements out of the request, realizes it's because lifeless' branch is merged in02:16
lifelessmwhudson: huh, no.02:21
lifelessmwhudson: oh righ, locally perhaps :)02:21
lifelessanyhow, our webapp adapter is essentially tracking units of work02:21
lifelesscalled 'requests'02:21
lifelessI think that this is fine and sensible, even for scripts, but what isn't fine or sensible is having this separate to the object representing the work - the IRequest02:22
wgrantARGH.02:23
* wgrant curses whoever decided that doctest log levels should be specified in the test registration, and should not be overridable within the test itself.02:24
lifelesswgrant: welcome to doctests02:24
wgrantBaaah.02:27
* StevenK finishes QAing gina02:28
StevenKThat was ... fun02:28
wgrantIt's even more fun when you have to do it locally, because there are no configs for that.02:29
StevenKI have mawson for that sort of thing02:29
* wgrant fixes build logging.02:32
wgrantBut buildd-slavescanner.txt seems to want me dead.02:33
StevenKwgrant: Not just you, I suspect.02:34
wgrantRarely have a found a doctest so slow and seemingly so fragiley malevolent.02:35
lifelessmwhudson: does my point about 'on behalf of' make sense?02:35
lifelesshmm02:35
mwhudsonlifeless: yes, but i'm not sure how relevant it is02:35
StevenKwgrant: I see your buildd-slavescanner.txt and raise you gina.txt02:36
wgrantStevenK: True. That one is really really slow.02:36
lifelessmwhudson: thats interesting; I thought it was the heart of the issue02:36
mwhudsonlifeless: scripts use the PermissiveSecurityPolicy by default, so in some sense at least the current principal doesn't really matter02:36
lifelessmwhudson: I think the PSP is essentially undesirable02:37
mwhudsonlifeless: maybe02:37
lifelessif:02:37
lifeless - we started scripts with an anonymous participation with a stubbish request02:38
lifeless - and the regular sec policy02:38
lifeless - and they called login() as soon as they identified the work they were doing02:38
lifelesswould we need the PSP at all ?02:38
mwhudsonlifeless: well02:39
mwhudsonthere's stuff like IBranch['updateScannedDetails'] that the scanner calls02:39
* mwhudson pauses, backtracks02:39
mwhudsonlifeless: i'm not sure this is really a good example, but there's a branchChanged method on branches02:41
mwhudsonthis is called by codehosting to record the format & tip of a branch02:41
mwhudsonno this is a really bad exmple02:41
mwhudsonlifeless: basically the point i'm trying to make is that i have this feeling that many scripts call 'internal' apis02:42
mwhudsonthat we wouldn't want the user to call via the webservice api say02:42
mwhudsonfor example, the stuff the build manager calls to record that a build has finished02:43
lifelessso thats a great example02:43
lifelessthere is a nonce02:43
lifelesswhich is security sensitive02:43
wgrantIt's not a nonce, and it's not security sensitive.02:44
wgrantBut OK.02:44
lifelesswgrant: if we want to allow the buiild slaves to push results, it becomes security sensitive02:44
lifelesswgrant: and I think it was julian who called it a nonce.02:44
lifelessthere is this thing02:44
mwhudson:)02:44
lifelessif you don't have it, we would not believe a claim that <data> is the result of a build02:45
lifelessif you do, we can believe that.02:45
wgrant(That's one explanation for its existence, but I don't think it's correct. Nobody really knows.)02:45
lifelesswgrant: it was added in to support slaves pushing back02:45
lifelesswgrant: I know this because I was tolk thats why02:45
lifelessits a WHUI case, but one we should.02:46
lifelessanyhow02:46
lifeless*IF* you imagine that we submit build results via the API02:46
lifelessI imagine we'd check something like02:46
lifelesssource ip address (are you a build slave)02:46
lifelessand02:46
lifeless(do you have the right nonce)02:46
lifelessif you have those two things, you can say a build is finished, if you don't, you can't.02:47
lifeless*noone* except the dispatcher can read the nonce02:47
lifeless(this is ideally, not describing what we have today)02:47
lifelessmwhudson: anyhow, I think it fits fairly well; finishing a build is conceptually a request from the builder02:48
lifelessmwhudson: garbo tasks *don't* fit well unless we have a celebrity with the right permissions02:49
mwhudsonlifeless: i guess where this leads to is that, yes, we could replace the use PSP in scripts with something else02:49
lifelessbut, coincidentally, thats exactly what we do do for the DB; I don't see why we shouldn't do it higher up too.02:49
mwhudsonbut i don't think you could easily replace it with the LaunchpadSecurityPolicy02:49
mwhudsonbecause that's all based around principals that are Persons02:49
lifelessmwhudson: I think a good mental exercise is to ask 'what would it take to make script X an API client02:50
lifelesswe probably need to get pgbouncer installed at some point02:50
lifelessbut even then, it would be nice to have less sources of idle connections02:50
mwhudsonlifeless: well funnily enough i did that fairly recently02:51
lifelessmwhudson: and what did it entail ? :)02:51
mwhudsoni changed code imports to do all their communication with the db via the internal xml-rpc server02:51
mwhudsonlifeless: calling removeSecurityProxy a lot :(02:51
lifelessmwhudson: thats kindof cheating02:51
mwhudsonyep02:51
lifelessmwhudson: can we do better ?02:52
mwhudsonnot even kindof02:52
mwhudsonlifeless: i don't konw02:52
mwhudsonlifeless: i wrote some mails about this this a while back, lemme hunt02:52
mwhudsonlifeless: does saying "Message-ID: <4B8C8089.1030105@canonical.com>" help you with your mail setup?02:53
lifelesshahaha02:54
mwhudsonlifeless: or http://www.mail-archive.com/launchpad-dev@lists.launchpad.net/msg02733.html02:55
lifelessmwhudson: subjects are normally enough02:55
mwhudson"using PermissiveSecurityPolicy when serving private xmlrpc requests"02:56
lifelessthanks02:57
mwhudsonmutable global state aaaaaaaaaaaaaaaa02:58
lifelessmwhudson: what just  bit you ?02:58
lifelessalso, you know we have a database, right ?02:58
mwhudsonlifeless: the thing i refer to in the first mail in that thread02:58
mwhudsonit's not really possible to use a different interaction class for a given request02:59
mwhudsoninteraction class == security policy btw02:59
lifelessso03:00
lifelessI think I'm fairly happy with saying:03:01
lifeless - PSP is almost certainly covering bugs and security holes03:01
lifeless - it divides our code arbitrarily and makes moving code out of web requests into backend systems hard and fragile03:01
lifeless - I don't see, and haven't seen a case for PSP existing other than 'its how we made stuff work way back when'03:02
mwhudsonmaybe a special principal that LaunchpadSecurityPolicy does something different with would be ok03:02
mwhudsonor special class of principals03:02
lifelessmwhudson: I don't see why impersonation isn't totally sufficient03:02
lifelesshave a privileged version of login()03:03
wgrantYou also need a superpowered principal.03:03
lifelessgrant script principles access to that03:03
wgrantSince lots of operations shouldn't even be possible for ~admins.03:03
mwhudsonlifeless: you said earlier that "<lifeless> mwhudson: anyhow, I think it fits fairly well; finishing a build is conceptually a request from the builder"03:03
lifelessmwhudson: I did03:04
mwhudsonlifeless: by builder did you mean 'person who uploaded the source package' ?03:04
thumper:(03:04
thumperit appears that facets are still used03:04
lifelessmwhudson: no, I meant the build slave03:04
thumper:((03:04
lifelessmwhudson: the one that builds03:04
mwhudsonlifeless: the build slave isn't a Person03:04
mwhudsonand Persons are the only sort of principal we really have today03:05
lifelessmwhudson: we have celebrities for this; we might want something better.03:05
lifeless(I dislike celebrities hugely)03:05
lifelessbut, they are square, and the hole is square.03:05
mwhudsoneww03:05
lifelessmwhudson: we have a celeb for the software centre agent, for instance.03:06
lifelesswhich is doing *exactly* this sort of thing03:06
mwhudsonyes, i guess so03:06
mwhudsondoesn't mean it's not horrible though03:06
lifelesssure03:06
lifelessI agree03:06
lifelessI'm happy though, to trade two, pervasive, icky things, for one pervasive icky thing and a clear concept for work-on-behalf-of.03:07
lifelessand then we can look at the remaining icky thing.03:07
mwhudsonhang on03:08
mwhudsontwo pervasive icky things?03:08
mwhudsonone is PSP03:08
mwhudsonwhat's the other?03:08
lifelesscelebrities03:08
mwhudsonah ok03:08
mwhudsoni think i misread you then03:08
* wgrant despairs at buildd-manager logging priorities.03:23
wgrantA build failed? CRITICAL! I can't communicate with a builder? Debug.03:23
=== Edwin is now known as Guest62855
spmwgrant: ho hum.04:37
spmwgrant: actually - Not being able to communicate with a Builder is perhaps info at best. outputting a critical on network blips would be a complete pita; and has been a problem with soyuz.04:40
spmfor services of this nature, the best I can describe: if a human *MUST* intervene, it's critical. if they don't have to, s/w can recover on it's own? it's error or lower.04:41
mwhudsonThere is no blueprint named "" in kubuntu, or krunch-desktop-plan isn't valid dependency of that blueprint.04:43
* mwhudson hearts the blueprints code04:43
thumpermwhudson: you forgot your sarcasterisk04:46
mwhudsontis true04:47
spmthumper: i dunno... in this case I don't think it was needed. the bright flashy neon lit sign with ***sarcasm ahead*** and awoooogah "sarcasm warning" horn, were a bit of a giveaway. ???04:49
lifelessmwhudson: do you think its ok to have Participation support annotations ?05:07
mwhudsonlifeless: probably05:07
mwhudsoni didn't realise in my first mail that Participation was a launchpad thing05:08
wgrantspm: I'm thinking of making communication errors like that a warning, disabled builders errors, and nothing critical.05:14
wgrantEverything that was previously critical could only be a warning at most.05:16
spmsweet05:16
mwhudsondoes anyone know by what mechanism the doctests in lp.registry.browser.tests get run?05:19
wgrantmwhudson: Not test_views?05:21
wgrantThat instantiates a LayeredDocFileSuite.05:21
mwhudsonwgrant: ah yes, thanks05:21
mwhudsonoh yes doctests, how do i hate thee, let me count the ways05:23
* mwhudson stares at this one and thinks about converting it to a unit test05:23
wgrantWho is your victim today?05:24
mwhudsonwgrant: part of vocabularies.txt05:28
wgrantyay.05:30
thumperlifeless: remember we were talking about the project cloud the other day?05:42
thumperlifeless: this seems to be a much more performant (and relevant) query: select product.name, count(*) as commits, count(distinct(revision_author)) as author_count, max(revision_date) as last_commit from revisioncache, product where revisioncache.product = product.id and not revisioncache.private group by product.name order by count(*) desc limit 50005:44
thumpernot sure which value should be the size though...05:45
thumpercommit count or author count05:45
thumpersuggestions anyone?05:46
lifelesscommits per author ?05:46
wgrantSome combination of commit and author counts seems best.05:47
lifelesswe don't want just kde, gnome etc showing up05:47
lifelessand they are biased to large commit counts & authors, but their normalised contributions should be much smaller05:47
wgrantBut KDE and GNOME are not projects, hopefully.05:48
lifelessshrug05:48
lifelessif you want to be picky05:48
wgrantThe projects within GNOME and KDE should not be overwhelmingly active.05:48
lifelesswgrant: I'm 99.9999999% sure you know what I am talking about.05:49
thumperwe have size and colour to use05:50
thumperperhaps size is based on number of commits05:50
thumperand darkess grouped on committer numbers05:50
lifelessif your metrics are highly correlated05:51
lifelessthen this will just mean small=dark big=light (or vice versa05:51
thumperyeah... mostly05:52
lifelessand thus it would be simler to just have one figure you calculate05:52
lifelessand show small=dark, big=light05:52
thumperalthough not always the case05:52
lifelessOTOH, if they are not highly correlated, it may look fugly.05:52
lifeless:)05:52
thumperopenerp-hr-payroll-cr          |     568 |           11 | 2010-08-06 21:37:17.62905:52
thumper mplayer                        |     103 |           11 | 2010-08-07 18:23:31.78605:52
thumper ubuntu-seeds                   |      18 |           10 | 2010-08-07 03:31:38.47705:52
thumpercommits is second05:52
thumpercount is third05:52
thumperauthor count that is05:53
lifelesspersonally, I don't think folk try to get stats out of the cloud05:53
thumperno, they don't05:53
lifelesswearing my colourblind-critic hat05:53
thumperperhaps just don't bother with shade :)05:53
lifelessI'd really rather keep it simple05:53
* thumper nods05:53
thumperok, just size based on commit count in the last 30 days05:53
thumperagreed?05:54
lifelessperhaps size based on commit count/author count05:54
lifelessto let small but prolific show up05:54
thumperah, ok05:54
lifelessperhaps thats a bad idea; I don't know.05:54
* thumper runs to guitar lesson :)05:55
lifelessciao05:55
mwhudsoni wonder how many times people view code.launchpad.net06:09
mwhudsonin the 3.0 design it's not easy to get to06:10
lifelessmwhudson: project group clouds can die too06:10
mwhudsonlifeless: what are they?06:11
lifelesserm, I may have the wrong context06:11
lifelessthumper said when we were tlaking on th ephone06:11
lifelessthat the global cloud is just worst06:11
lifelessthat smaller ones also have trouble from time to time06:11
mwhudsoni didn't realize we had smaller clouds06:18
wgrantThe only other clouds I know of are the bug tag ones.06:18
StevenKArgh, why does PQM hate me06:24
=== almaisan-away is now known as al-maisan
StevenKAre we in testfix?06:26
=== jtv is now known as jtv_
=== jtv_ is now known as jtv
spm* fyi * about to stab the buildbot master, have a new hardy-slave built and want to ensure it gets picked up07:22
noodles775Morning07:29
spmheya noodles07:32
noodles775Hi spm07:32
spm* fyi * buildbot master appears to be happy again; new hardy-slave picked up. we return to your regular unscheduled building.07:34
lifelessgrah detoxing from caffeine headache :(07:37
lifelessalso hate hate hate untested code07:37
spmlifeless: ?? isn't the cure for a caffiene headache to have more caffeine? If you keep this up eventually the headache WILL go away; Of course you'll also be dead, but that's considered a mere side effect07:43
nigelb'mere side effect', heh08:05
adeuringgood morning08:25
jtvhi adeuring08:43
adeuringhi jtv08:44
jtvIs anyone else getting what looks like missing CSS on edge?08:44
adeuringone some pages, yes08:44
noodles775jtv: yep.08:44
noodles775jtv: have you let a losa know?08:45
jtvAnd it's on r11435… I think it was on 11430 an hour or so ago08:45
jtvI'm just finding out.08:45
spmargh. not again!?!?!08:45
jtvspm: see for yourself08:46
spmso I do08:46
jtvthen it's not just the rest of us08:46
jtvI _think_ it upgraded from 11340 to 11345 just now.08:47
adeuringThe missing CSS lets me spot new details of the pages. I did not know that we have a "progress bar" for configuration on the main project pages and that https://edge.launchpad.net/launchpad is only 75% configured08:48
jtvspm: is this something you can do anything about?08:49
jtvOr at least, does anyone know what causes this?08:50
spmlaunchpad-rev-11415 to launchpad-rev-1143508:50
jtvahhh I see you're fixing stuff already08:50
spmshould be in the edge restore email to the error list08:50
jtvthanks for the fast reaction08:50
spm:-)08:50
jtvyay!  CSS!08:52
jtvActually in some ways I kind of liked our new, back-to-basics look.08:52
spmadeuring: nice find for the silver lining there! ;-)08:53
adeuringyeah ;)08:53
* jtv wonders if that phrase is taken as a name for a cloud computing-related infrastructure project08:54
wgrantHm, still broken for me.08:54
spmahh crap. I need to do the FE's as well. ta.08:54
jtvHi mrevell, thanks for the email08:55
mrevellHu08:55
mrevellOr should I say, Hi?08:55
jtvI think Hi is better.08:56
mrevelljtv, My pleasure, I'm sorry for the delay.08:56
mrevell:)08:56
jtvnp…  I'm hitting something hard and serrated with my ongoing feature work though, so I many not get back to it today.08:56
spmright that should be fuixed?08:56
jtvspm: it's fixed again for me08:57
jtvOn to the next one… I have a MP in "updating diff" state more than an hour after the last change was pushed: https://code.edge.launchpad.net/~mwhudson/launchpad/move-SpecificationDepCandidatesVocabulary/+merge/3361108:58
wgrantLooks good. Thanks spm.08:58
wgrantbigjools: Morning09:02
spmjtv: gah. lookin'09:02
jtvwe're sure getting our money's worth out of Steve this evening.09:02
spm....09:02
bigjoolswgrant: g'day09:04
spmyeah. the m-p jobs task has gone gaga; killin'09:04
wgrantbigjools: Do you have time to talk about ddebs?09:06
bigjoolswgrant: at some point but not just now09:06
wgrantbigjools: Sure.09:06
bigjoolshow long are you around?09:06
spmjtv: that seems to be processing again. and fwiw, it apepars to be all mwhudson's stuff that caused the problem.09:06
StevenKHaha09:07
jtvspm: otp… thanks09:07
spm (accusation based on no scientific evidence, beyond his branches in the follow 'is working' log)09:07
wgrantbigjools: Four or five hours, probably.09:07
bigjoolsok09:07
jmladeuring, the 75% thing is a known bug that I'm told registry will fix any moment now10:24
=== Guest17053 is now known as jelmer
=== jelmer is now known as Guest61071
=== Guest61071 is now known as jelmer
jtvjml: ISTR you mentioned a TAL macro a few years ago that would turn a bunch of fragments into a neat "a, b, and c"—style list.  Can't seem to find it now.11:38
jmljtv, otp11:38
wgrantjtv: I don't know that there's a TALES expression for it, but there is canonical.launchpad.helpers.english_list11:44
jtvwgrant: thanks, that's the one I was thinking of—I thought it was TAL so no wonder I didn't find it!11:45
deryckMorning, all.12:02
jtvhi deryck!12:02
jtvjam: people are getting eager for that BranchRevision weight-loss program we worked on in Prague.  :)12:03
jtvmrevell: prototype for the translations help-bubble changes at lp:~jtv/launchpad/bug-517700 — playing with the real thing is probably more useful than me describing it in detail.  Still some rough edges, I think.  I'm EOD, but would appreciate feedback later!12:06
mrevellThanks jtv, I shall take a look at this next.12:06
=== danilo_ is now known as danilos
=== danilos is now known as danilo
=== danilo is now known as danilos
=== mrevell is now known as mrevell-lunch
jtvSee you tomorrow, folks!12:36
=== matsubara-afk is now known as matsubara
salgadobigjools, do you have a minute to talk about the removal of the security upload policy?13:29
=== mrevell-lunch is now known as mrevell
adeuringbigjools: ...and another question: ProxiedLibraryFileAlias.http_url ensures that the returned URL does not start with "api.lp.net". The reason seems to be bug 354373, which I don't really understand. I have at present the opposite problem: I _need_ a webservice URL for ProxiedLFAs, see bug 620458.13:35
_mup_Bug #354373: [API] build.build_log_url and build.upload_log_url provide wrong URLs <api> <Soyuz:Fix Released by cprov> <https://launchpad.net/bugs/354373>13:35
_mup_Bug #620458: cannot access attachments of private bugs any more <httplib2:Unknown> <Launchpad Bugs:Triaged by adeuring> <https://launchpad.net/bugs/620458>13:35
adeuringbigjools: so, I could either write a variant of ProxiedLFA.http_url which does not enforce the usage of IWebBrowserOriginatingRequest. like "default_http_url", or I could change the behaviour of http_url so that the current request is always used and add a property like web_browser_http_url which has the currnet behaviour of http_url.13:39
adeuringbut: why is this overriding necessary?13:39
bigjoolsadeuring: why do you need webservice URLs for librarian files?13:50
adeuringbigjools: so that lplib scripts can access private data13:50
adeuringbigjools: see bug 62045813:51
_mup_Bug #620458: cannot access attachments of private bugs any more <httplib2:Unknown> <Launchpad Bugs:Triaged by adeuring> <https://launchpad.net/bugs/620458>13:51
bigjoolsok13:51
bigjoolssounds fair enough - I think we overrode it because it was breaking something else though13:51
adeuringbigjools: OK, so, changing the behaviour of ProxiedLFA.http_url, keeping the current behaviour in something like web_brwoser_http_url and using this in the affected code would be OK for you?13:55
bigjoolsadeuring: I can't think of all the ramifications right now, but as long as you don't have to change any of the soyuz tests to make them work then it sounds fine.  I'd check with Gary though to see if he has any thoughts.13:56
adeuringOK, gary_poster: ^^^13:57
wgrantIt's not so bad any more, since api.launchpad.net doesn't require auth.13:58
wgrantHowever, some API clients will still want webapp URLs, so they can serve up links to private files.13:58
wgrantSo we really want both :/13:58
adeuringwgrant: You can meanwhile access private files via the webservice13:59
wgrantadeuring: Not if I'm serving links to web clients.14:00
adeuringwgrant: ?14:00
wgrantIf I use an API client to create a web page, I need to serve webapp URLs, since my users aren't authenticated to the API host.14:00
adeuringwgrant: ah, right!14:01
salgadobigjools, did you see my ping earlier about removing the security upload policy?14:05
bigjoolssalgado: yes, sorry, I am dealing with other things before getting around to you14:05
bigjoolsbut there's a lull, so fire away14:05
deryckgmb, can I get an "amen!" to my changes here:  https://help.launchpad.net/Bugs/ImportFormat ?14:06
salgadobigjools, soyuz-set-of-uploads.txt depends on that policy, and I've tried changing it to use another policy that accepts the same kinds of uploads but it fails and leaves me with no clue as to why14:06
* bigjools checks14:07
salgadobigjools, line 32614:08
gmbderyck, Amen, brother.14:08
gmbLooks good.14:08
bigjoolssalgado: I see14:08
bigjoolswhat's the error?14:08
deryckexcellent.  thanks, gmb14:08
gmbnp14:08
salgadobigjools, Failed upload(s): ['unstable_1.0-1'] instead of the rejected exception14:09
salgadothat's when I use the 'buildd' policy14:10
bigjoolssalgado: what does the next output say (for read_email())14:11
salgadoNone14:11
bigjoolsawesome14:11
gary_posteradeuring: Please confirm if I understand the situation correctly.  http_url was a url friendly to the webservice.  It has changed recently to be a url friendly to the browser.  This is problematic for a number of reasons, many of which go under the category of "backwards compatibility".  You propose to reinstate the previous behavior and create a new attribute named "browser_url" or something similar.  That's my 14:11
gary_posterright?14:11
adeuringgary_poster: yes.14:11
adeuringgary_poster: the alternatvie would be to add something like "default_http_url"14:12
adeuringwhich looks a bit odd to me14:12
wgrantThat's not my understanding -- ProxiedLibraryFileAlias has returned a browser-friendly URL for 18 months. Bug attachments just started using it a couple of weeks ago.14:12
jmljelmer, I'm warming up to the idea of a testr integration branch.14:12
jmljelmer, lack of incremental output is hurting me.14:12
adeuringwgrant: well, yes. But http_url is not vey specific to ProxiedLFA14:13
gary_posterwgrant, ok, thanks for clarification.14:13
gary_posteradeuring, wgrant, I'm in favor of using the webservice versioning for this.  1.0 and beta should keep the current behavior, whatever it is, since that appears to not be breaking anything and wgrant says it has been stable.14:14
jelmerjml: Yeah, that's particularly annoying with a project as large as lp.14:14
gary_posterI like http_url for webservice and browser_url for browser for the devel service, but there's an obvious downside of surprising migration (it's easier to know to migrate when a attribute disappears than when it subtly changes meaning).14:15
adeuringgary_poster: OK... what about leaving http_url as it is and adding web_url and api_url?14:16
salgadobigjools, how about I remove that test and add a unit test to AbstractUploadPolicy.setDistroSeriesAndPocket(), which is what raises that exception shown in the email message?14:16
wgrantThe failure mode here is probably just that private files become inaccessible. So it's not that bad.14:16
bigjoolssalgado: +1, that doctest needs to die in flames14:17
deryckadeuring, hi.  Can we get a card into WIP on the Kanban board for that attachment work you're doing?14:17
adeuringderyck: sure14:17
deryckadeuring, thanks!14:17
gary_posteradeuring: http_url will effectively be alias for web_url in your proposal?14:18
adeuringgary_poster: no necessarily. web_url and api_url should enforce the hostnames code.lp.net and api.lp.net, repsectively14:18
gary_posterSo what is the value of http_url then?  Why would I use it instead of web_ or api_?14:19
salgadobigjools, cool.  however, there's also a big chunk starting at line 606 for testing staged uploads to the security pocket.  I know there are other tests for staged uploads, so maybe I can just nuke that?14:19
salgadolib/lp/archiveuploader/tests/test_buildduploads.py has those tests for staged uploads14:20
adeuringgary_poster: well... I'm trying to find a way to cop out from changing soyuz code while having somewhat same property names ;)14:20
adeuringActually, I could simply add api_url -- that's all I need14:21
adeurings/same/sane/14:22
gary_posteradeuring: heh, ok fair enough. :-)  from this conversation, http_url seems poorly defined and unclearly named though.  I'd prefer you add api_url and web_url, and make a note in http_url that that users should cuse api_url and web_url instead, and http_url may be removed in a future version of the webservice.  Maybe that's too aggressive...14:24
gary_posterThat's my preference, but I would be OK with only adding api_url and putting a bug in against the webservice about this problem, so that when leonardr and benji start trying to clean up the webservice generally this is one of the issues they consider tackling.14:24
adeuringgary_poster: good proposal; I'll go for it.14:25
gary_posteradeuring: cool, thank you!14:26
leonardrgary, adeuring, are you aware of rockstar's work on this? has he completed the work and that's cuasing the problem?14:26
gary_posterleonardr: I have no knowledge of this :-/14:26
adeuringleonardr: no, maybe his work will fix my problem, no idea.14:26
adeuringleonardr: the code I'm talking about is from r816614:26
leonardrgary, adeuring, at the epic rockstar started working on a 'web_link' that was like 'self_link' except it pointed to the object on the website14:27
leonardrbut given that revision number i imagine you're not talking about something added to lazr.restful14:28
gary_posterleonardr: Is that pertinent to the library files?14:28
adeuringleonardr: right, its about lp code itself. and as gary says, about library files14:28
adeuringProxiedKFA, more specifically14:28
adeuringProxiedLFA14:29
leonardrso, the library files used to have an http_url that used whatever host the reuqest came from?14:29
adeuringleonardr: yes, and that points _not_ to the webservice14:29
adeuringmaking access to the files from a webservice client impossible from private files14:30
adeurings/from/for/14:30
leonardrok, i see14:30
gary_posteradeuring, just a warning, we're about to have a team call, so will be away for just a moment14:30
adeuringok14:31
leonardrin that case, you can check for request.version to see which version of the web service is in use, and change behavior based on that14:31
leonardri don't have an opinion on what you should implement, i just wanted to make sure this wasn't overlapping rockstar's work14:31
rockstarleonardr, I should get back to fixing that one day.14:32
=== jelmer_ is now known as Guest3182
=== Guest3182 is now known as jelmer
=== jelmer is now known as Guest48768
gmbAaaaaaaaa15:33
gmbSo.15:33
gmbWhy would assertRaises() in a test case *not* catch the exception that I'm asserting the callable raises?15:34
jmlhmm15:38
salgadobigjools, did you my msg earlier about the staged-upload test on soyuz-set-of-uploads.txt?15:38
jmlgmb, I'm thinking about that.15:38
jmlgmb, I'm fairly sure the answer is that you are doing it wrong.15:38
gmbjml, Specifically, the exception is zope.security.interfaces.Unauthorized15:38
gmbAnd the code is:15:39
gmb        self.assertRaises(15:39
gmb            Unauthorized, self.bug_tracker.resetWatches,15:39
gmb            "Unprivileged users should not be allowed to reset a "15:39
gmb            "tracker's watches.")15:39
jmlgmb, ahh, I know this one :)15:39
gmbOh goodie.15:39
gmbDo share.15:39
jmlgmb, it's getattr(self.bug_tracker, 'resetWatches') that's raising the Unauthorized15:39
bigjoolssalgado: sorry missed that, looking now15:39
jmlgmb, rather than the actual method call.15:40
gmbjml, Ah, because it's launchpad.Admin'd.15:40
gmbso an unpriv'd user can't get at the method, let alone call it.15:40
jmlgmb, and because zope security works on attribute access.15:40
jmlgmb, exactly.15:40
gmbD'oh. So obvious.15:40
gmbjml, Thanks.15:40
jmlself.assertRaises(Unauthorized, getattr, self.bug_tracker, 'resetWatches') has worked well for me in the past.15:40
jml(although arguably that's a custom assertion method / matcher waiting to happen)15:41
jmlgmb, np.15:41
bigjoolssalgado: I think we can nuke the test15:42
=== Guest48768 is now known as jelmer
salgadobigjools, cool, the problem now is that the test hangs after I removed that section.  I'll see if I can find out where/why15:50
bigjoolssalgado: argh.  That test is a nightmare.15:51
salgadobigjools, btw, would you like to have a look at the other branch which replaces the can_upload_* attributes with a single enum?  jtv has approved it, but I thought you might want to have a look anyway?15:52
bigjoolssalgado: I can but I'm not sure when!15:52
* bigjools is too busy :(15:52
salgadobigjools, maybe jelmer or StevenK can have a look?  or if you think it's not necessary, I've already got jtv's approval anyway15:55
salgadobtw, it's publish-distro.py that hangs15:55
bigjoolssalgado: don't block on landing it, we can look later.  jelmer may be very interested anyway as he's changing the upload processor a bit at the moment.15:56
salgadook, cool15:58
=== matsubara is now known as matsubara-lunch
jml"testr run failing" doesn't do what I meant16:38
deryckgmb, the an MP I approved for a "scratch" branch of yours.  Can that be landed?16:39
gmbderyck, Er. Hang on, I don't remember that.16:46
gmbBlimey, that was a while back.16:46
deryckJudging by the diff I wonder if you merged it in another branch?16:46
gmbderyck, Ah, I think that first bit was to do with the fix for the initial_message problem.16:47
gmbHrm.16:47
gmbderyck, I'll do some digging and find out what's landed and what's not.16:47
gmbI suspect that diff is a lie.16:48
deryckok, cool.  Thanks!16:48
gmbderyck, Yes, there's some lying going on. Well, not lying, but basically the diff is against the ancestor revision of the scratch branch; when I merge devel it conflicts with what's already landed. I'll clean it up and submit it.16:51
deryckgmb, ok,cool.16:56
=== matsubara-lunch is now known as matsubara
jelmersalgado: I see you're having fun with huge interdependent soyuz doctests17:20
=== beuno is now known as beuno-lunch
salgadojelmer, yes!  it's been such a long time since it last happened that I'd almost forgotten how much fun they can be17:30
=== Ursinha is now known as Ursinha-lunch
jelmersalgado: :-)17:38
=== benji is now known as benji-lunch
=== al-maisan is now known as almaisan-away
=== salgado is now known as salgado-lunch
=== beuno-lunch is now known as beuno
leonardri'd like to talk to someone who understands zope permissions well, maybe gary, or salgado-lunch once he returns from lunch18:35
gary_posterleonardr: benji-lunch would be a good choice too.  I better go get some lunch because I have a call in 24 min :-/18:36
gary_posterotherwise I should be available 3:30 or 418:36
leonardrok18:36
leonardri'll just explain the problem18:36
gary_posterok18:36
leonardri've created a security policy for IOAuthAccessToken that basically says:18:36
leonardrif you're trying to look at this oauthaccesstoken through the website, the old rules apply: it has to be your token, or you have to be an admin18:36
=== benji-lunch is now known as benji
leonardrif you're trying to look at this oauthaccesstoken through the web service itself, the rules are more restrictive.18:38
leonardryou can only look at your own token, and your request must itself be signed by an oauthaccesstoken that has the GRANT_PERMISSIONS access level18:39
=== Ursinha-lunch is now known as Ursinha
leonardrthis works fine for prohibiting writes to the token, and it also keeps the token from showing up in lists in the web service (since you don't have launchpad.View on the token)18:40
leonardrbut, you can still guess the url and get the token data that way18:40
leonardrso i added this bit to oauth.zcml18:40
leonardr      <require18:40
leonardr          permission="launchpad.View"18:40
leonardr          interface="canonical.launchpad.interfaces.IOAuthAccessToken"/>18:40
leonardrand that protects the objects themselves18:42
leonardrhowever, there's a catch-22: to determine whether the request is signed by an appropriate OAuthAccessToken, you need to be able to look at an OAuthAccessToken object18:42
leonardrthat's where i'm stuck18:43
gary_posterleonardr: which component needs to be able to look at an OAuthAccessTokenObject?18:44
gary_posterleonardr: a mediator is a typical pattern for this18:44
gary_postermediator rips off security proxy and does what needs to be done and returns answer18:44
leonardrgary: well, right now, the code that signs the _outgoing_ request needs to be able to look at it. the request isn't even being made18:44
gary_posterwould mediator work in context?18:45
* gary_poster really should get some food18:45
leonardrgo ahead18:45
jmlgary_poster, stay hungry, the TL meeting will be shorter for it :)18:45
leonardri'll try some stuff18:45
gary_posterjml :-)18:45
leonardrgary: two well-placed removeSecurityProxy calls solved the problem19:11
jmlg'night all.19:15
gary_posterleonardr: great.19:16
bigjoolsnn jml19:17
=== salgado-lunch is now known as salgado
leonardrbenji, got another problem with my permissions. the 'view' permission seems to work correctly, but the 'edit' permisison check is failing without my code ever being called20:40
leonardrlet me know what kind of details will help20:40
* benji scrolls back to get context.20:41
leonardrbenji: basically i updated the AuthorizationBase subclass for OAuthToken objects20:42
leonardrso that you can only modify them from the web service under certain circumstances20:42
leonardrmy code is running when it comes to _viewing_ objects through the web service20:42
leonardrbut when i try to modify one, i get Unauthorized, and the code from security.py never runs20:43
leonardrsetattr(context, self.name, value) raises an exception20:44
benjileonardr: What is the security checker for the object in question?  Also, I'm trying to figure out how your AuthorizationBase subclass tied into zope.security.  I've not touched the LP-specific security stuff any yet.20:47
leonardrbenji, i believe the seucirty checker is canonical.launchpad.webapp.authorization.LaunchpadSecurityPolicy20:49
leonardrbenji: NEVER MIND. i brought this problem on myself20:50
benjileonardr: I'd put some breakpoints in one or two methods of LaunchpadSecurityPolicy and then execute your setattr; tracing through what happens should...20:51
leonardrthere is a real problem, but i understand why this is happening20:51
benji:)20:51
leonardrbenji: the real problem is in webapp/authorization.py, _checkRequiredAccessLevel20:53
leonardran AccessLevel of GRANT_PERMISSIONS doesn't have the ability to 'write'20:54
leonardri want a situation where GRANT_PERMISSIONS has the ability to 'write', but only to OAuthACcessToken objects20:54
benjimakes sense20:55
leonardri have no clue how to do this. i can use the zcml guards to attach an AuthorizationBase subclass to OAuthAccessToken20:56
leonardri guess i could change AuthorizationBase to explicitly forbid writes if the AccessLevel is GRANT_PERMISSIONS, but that seems hacky20:57
leonardri think salgado might have some insight into this20:57
lifeless.21:17
salgadoleonardr, maybe, but I'd need more context21:55
leonardrsalgado: so, take a look at LaunchpadSecurityPolicy._checkRequiredAccessLevel21:56
leonardrthis code says "no matter what permissions the principal has, if the access level is not high enough, access denied"21:56
leonardri would like GRANT_PERMISSIONS to be considered a 'read' access level for everything _except_ oauth access tokens21:57
leonardri implemented permissions to this effect (you can only write to an oauth access token if you are using GRANT_PERMISSIONS)21:58
leonardrbut since GRANT_PERMISSIONS is considered a 'read' access level globally, you never get to use those permissions21:58
leonardrthe only thing i can think of is to make GRANT_PERMISSIONS a 'write' access level, and special-case the superclass of all write-permission checkers so that GRANT_PERMISSIONS does _not_ give you any write permisson22:01
leonardror, give up and just make GRANT_PERMISSIONS a 'write' access level22:01
benjileonardr: it wouldn't seem too bad for GRANT_PERMISSIONS to have write access; after all if something has GRANT_PERMISSIONS then they could just give themselves write access, right?22:03
leonardrbenji: yes, the idea is more to make sure that a GRANT_PERMISSIONS script doesn't suffer feature creep and become a do-all-sorts-of-things script22:03
benjimmm22:04
benjigiant ascii art warning perhaps?  :P22:04
leonardrif we could determine when to print that warning, we could just deny access :P22:05
mwhudsonmorning22:06
salgadoleonardr, what about forcing all tokens with permission==GRANT_PERMISSIONS to be scoped to OAuthToken?  that way the client would have whatever access_level is defined in GRANT_PERMISSIONS for OAuthToken but read-only access for everything else22:08
leonardrsalgado: that's a good idea, but i'm pretty sure scoped tokens don't work and never did work22:11
leonardrbut, it's possible the internals work and the interface was never completed22:12
salgadoI think that's the case, but even if it doesn't work it should be easy to fix it22:13
leonardrok, i will look into this tomorrow22:13
mwhudsonec2 land is blowing up for me22:57
mwhudsonException AttributeError: "'SmartSSHClientMedium' object has no attribute '_ssh_connection'" in <bound method SmartSSHClientMedium.__del__ of SmartSSHClientMedium(bzr+ssh://mwhudson@bazaar.launchpad.net/)> ignored22:57
mwhudsonregular bzr operations work fine though22:57
mwhudsonany ideas anyone?22:58
mwhudsonsinzui: did you get my mail about menus?22:59
mwhudsonhm, probably not23:01
mwhudsonsinzui: https://lists.launchpad.net/launchpad-dev/msg04367.html23:01
* sinzui looks23:01
sinzuimwhudson, I did not see this reply23:02
=== Ursinha is now known as Ursinha-afk
mwhudsonsinzui: yeah, i screwed up my mail server config somehow23:04
sinzuiWell I will reply shortly23:04
mwhudsonit's not a very deep reply, mostly a series of questions....23:04
mwhudsoncool23:04
=== matsubara is now known as matsubara-afk
rockstarwallyworld, do you have something like this in ~/.ssh/config http://pastebin.ubuntu.com/483652/23:17
wallyworldi'll look23:17
lifelesssinzui: https://edge.launchpad.net/landscape/+milestone/later23:28
lifeless At least 81 queries issued in 11.15 seconds23:28
lifelesssinzui: seems a bit healthier23:28
lifeless(and I'm seeing the private bugs)23:29
sinzuiyep23:31
wgrantlifeless: I wouldn't call it healthy -- there are still massive scaling issues.23:34
wgrantTakes 1.1s here.23:34
wgrant69 queries.23:34
lifelesswgrant: I wouldn't call it healthy either23:35
lifelesswgrant: 20, constant, would be healthy.23:35
lifelesswgrant: but healthier, eys.23:35
wgrantI wonder where the extra 10s comes from.23:35
wgrantSure not those 12 queries.23:35
lifelessOOPS-1698EA2488 may tell us23:36
wgrantDo you want an OOPS from mine as well to compare?23:37
rockstarwallyworld, I think you're looking for /etc/apache2/sites-available/local-launchpad23:46
rockstarwallyworld, do you have bazaar.launchpad.dev in your /etc/hosts ?23:48
wallyworldyep - 127.0.0.9923:49
=== Guest83593 is now known as jelmer

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!