[00:00] <lifeless> enums seems to be our canned answer for this
[00:01] <mwhudson> i think it's a big case of WHUI
[00:01] <lifeless> WHUI ?
[00:01] <mwhudson> We Haven't Used It
[00:01] <mwhudson> a SteveA-ism
[00:01] <lifeless> google was ... un helpful
[00:02] <mwhudson> something like KarmaAction is in the db so we can tweak the constants to adjust the way karma is allocated without landing code changes
[00:02] <lifeless> mwhudson: so, we're talking IRequest
[00:02] <mwhudson> but of course we don't
[00:02] <lifeless> etc
[00:02] <mwhudson> stub will know more
[00:02] <mwhudson> lifeless: i'm actually replying to your email
[00:02] <lifeless> mwhudson: cool
[00:02] <lifeless> mwhudson: the big question, is an unknown
[00:03] <lifeless> mwhudson: is it shallow enough I should plunge on and doit
[00:03] <lifeless> mwhudson: or should I thread-locals-it.
[00:03] <mwhudson> lifeless: what's 'it' ?
[00:04] <wgrant> lifeless: 'karmacategories' in http://bazaar.launchpad.net/~wgrant/launchpad/bootstrap-db-from-scratch/annotate/head:/utilities/bootstrap-lp-db is the data in question.
[00:04] <lifeless> mwhudson: it is
[00:05] <lifeless> mwhudson: making scripts have an IRequest always, so that when they do sql it is logged in my new code
[00:05] <lifeless> the second it
[00:05] <lifeless> is
[00:05] <lifeless> the new code : change requesttimeline to be a threadslocal thing
[00:08] <mwhudson> lifeless: ok, my email is nearly done
[00:10] <mwhudson>     # This is a convenient hack to set up a zope interaction, before we get
[00:10] <mwhudson>     # the proper API for having a principal / user running in scripts.
[00:10] <mwhudson>     # The script will have full permissions because of the
[00:10] <mwhudson>     # PermissiveSecurityPolicy set up in script.zcml.
[00:10] <mwhudson> ha ha
[00:10] <mwhudson> i wonder when that was written
[00:10]  * mwhudson bets on 2005
[00:11]  * mwhudson wins
[00:12] <mwhudson> timestamp: Tue 2005-04-12 09:37:50 +0000
[00:12] <mwhudson> from the arch days
[00:12] <mwhudson> steve.alexander@canonical.com/launchpad--devel--0--patch-368
[00:17] <mwhudson> lifeless: ok, mail sent
[00:27] <lifeless> thanks
[00:37] <lifeless> mwhudson: followup sent btw
[00:46] <mwhudson> lifeless: i replied one more time, happy to talk about it in irc now :-)
[00:46] <mwhudson> although i don't think there's much need
[00:50]  * mtaylor thinks you're both wrong and obviously everything should be re-written in google go
[00:50]  * mtaylor falls on the floor laughing
[00:51]  * mtaylor is obviously in an odd mood
[00:51] <StevenK> mwhudson: Patches welcome
[00:51] <StevenK> Doh
[00:52] <StevenK> IRC tab fail :-(
[01:02] <lifeless> mwhudson: I think I'm good.
[01:02] <lifeless> mwhudson: I guess that under setupInteractionByEmail(ANONYMOUS) in script base
[01:03] <lifeless> mwhudson: I'll add something??? that sets up a participationwithannotations ?
[01:08] <mwhudson> lifeless: setupInteractionByEmail takes a participation as an argument
[01:16] <lifeless> mwhudson: yeah
[01:16] <lifeless> actually though
[01:17] <lifeless> set_request_started is where scripts expect to do stuff
[01:17] <lifeless> so *it* needs to check and see if there is a participiation...ICanHasAnnotations, and if not setone up
[01:17] <lifeless> we still need to unify these two things
[01:18] <lifeless> I like your approach, but I'm not sure we don't actually want - eventually - scripts to say they are in requests via participations rather than set_request_started
[01:19] <mwhudson> i admit i don't really know what set_request_started is about
[01:23] <mwhudson> lifeless: in the particular case of checkwatches, it does it's one interaction management
[01:24] <mwhudson> -'
[02:05] <lifeless> mwhudson: kindof-management
[02:05] <lifeless> mwhudson: but yes
[02:06] <mwhudson> lifeless: it looks like it probably suffers locally from the kind of confusion it would be good to clear up globally
[02:07] <lifeless> with_interaction looks like exactly what needs clearing up
[02:08] <lifeless> vis-a-vis transactions, security & context
[02:08] <lifeless> also @statement_logging is just bong
[02:10] <mwhudson> i'm not sure i'm correct or expressing this clearly
[02:10] <mwhudson> but i wonder if there's a bit of a tension between things that don't care at all about participations (like most scripts and tests) and the few things that do (like checkwatches)
[02:11] <lifeless> most scripts do work on behalf of someone
[02:11] <mwhudson> also, i wonder if thinking about how you'd like stuff bundled up in an oops report is a good guideline towards how long your participations should be current for
[02:11] <lifeless> the work doesn't just 'appear'
[02:12] <lifeless> I think that bundling point is exactly on the spot
[02:12] <lifeless> its certainly how I think abou tit
[02:13] <lifeless> excuse me; brain flagging food needed
[02:13] <mwhudson> so for example, each job you process in a job running script should have it's own participation
[02:16]  * mwhudson reads errorlog.py, is surprised to find it reads db statements out of the request, realizes it's because lifeless' branch is merged in
[02:21] <lifeless> mwhudson: huh, no.
[02:21] <lifeless> mwhudson: oh righ, locally perhaps :)
[02:21] <lifeless> anyhow, our webapp adapter is essentially tracking units of work
[02:21] <lifeless> called 'requests'
[02:22] <lifeless> I think that this is fine and sensible, even for scripts, but what isn't fine or sensible is having this separate to the object representing the work - the IRequest
[02:23] <wgrant> ARGH.
[02:24]  * wgrant curses whoever decided that doctest log levels should be specified in the test registration, and should not be overridable within the test itself.
[02:24] <lifeless> wgrant: welcome to doctests
[02:27] <wgrant> Baaah.
[02:28]  * StevenK finishes QAing gina
[02:28] <StevenK> That was ... fun
[02:29] <wgrant> It's even more fun when you have to do it locally, because there are no configs for that.
[02:29] <StevenK> I have mawson for that sort of thing
[02:32]  * wgrant fixes build logging.
[02:33] <wgrant> But buildd-slavescanner.txt seems to want me dead.
[02:34] <StevenK> wgrant: Not just you, I suspect.
[02:35] <wgrant> Rarely have a found a doctest so slow and seemingly so fragiley malevolent.
[02:35] <lifeless> mwhudson: does my point about 'on behalf of' make sense?
[02:35] <lifeless> hmm
[02:35] <mwhudson> lifeless: yes, but i'm not sure how relevant it is
[02:36] <StevenK> wgrant: I see your buildd-slavescanner.txt and raise you gina.txt
[02:36] <wgrant> StevenK: True. That one is really really slow.
[02:36] <lifeless> mwhudson: thats interesting; I thought it was the heart of the issue
[02:36] <mwhudson> lifeless: scripts use the PermissiveSecurityPolicy by default, so in some sense at least the current principal doesn't really matter
[02:37] <lifeless> mwhudson: I think the PSP is essentially undesirable
[02:37] <mwhudson> lifeless: maybe
[02:37] <lifeless> if:
[02:38] <lifeless>  - we started scripts with an anonymous participation with a stubbish request
[02:38] <lifeless>  - and the regular sec policy
[02:38] <lifeless>  - and they called login() as soon as they identified the work they were doing
[02:38] <lifeless> would we need the PSP at all ?
[02:39] <mwhudson> lifeless: well
[02:39] <mwhudson> there's stuff like IBranch['updateScannedDetails'] that the scanner calls
[02:39]  * mwhudson pauses, backtracks
[02:41] <mwhudson> lifeless: i'm not sure this is really a good example, but there's a branchChanged method on branches
[02:41] <mwhudson> this is called by codehosting to record the format & tip of a branch
[02:41] <mwhudson> no this is a really bad exmple
[02:42] <mwhudson> lifeless: basically the point i'm trying to make is that i have this feeling that many scripts call 'internal' apis
[02:42] <mwhudson> that we wouldn't want the user to call via the webservice api say
[02:43] <mwhudson> for example, the stuff the build manager calls to record that a build has finished
[02:43] <lifeless> so thats a great example
[02:43] <lifeless> there is a nonce
[02:43] <lifeless> which is security sensitive
[02:44] <wgrant> It's not a nonce, and it's not security sensitive.
[02:44] <wgrant> But OK.
[02:44] <lifeless> wgrant: if we want to allow the buiild slaves to push results, it becomes security sensitive
[02:44] <lifeless> wgrant: and I think it was julian who called it a nonce.
[02:44] <lifeless> there is this thing
[02:44] <mwhudson> :)
[02:45] <lifeless> if you don't have it, we would not believe a claim that <data> is the result of a build
[02:45] <lifeless> if you do, we can believe that.
[02:45] <wgrant> (That's one explanation for its existence, but I don't think it's correct. Nobody really knows.)
[02:45] <lifeless> wgrant: it was added in to support slaves pushing back
[02:45] <lifeless> wgrant: I know this because I was tolk thats why
[02:46] <lifeless> its a WHUI case, but one we should.
[02:46] <lifeless> anyhow
[02:46] <lifeless> *IF* you imagine that we submit build results via the API
[02:46] <lifeless> I imagine we'd check something like
[02:46] <lifeless> source ip address (are you a build slave)
[02:46] <lifeless> and
[02:46] <lifeless> (do you have the right nonce)
[02:47] <lifeless> if you have those two things, you can say a build is finished, if you don't, you can't.
[02:47] <lifeless> *noone* except the dispatcher can read the nonce
[02:47] <lifeless> (this is ideally, not describing what we have today)
[02:48] <lifeless> mwhudson: anyhow, I think it fits fairly well; finishing a build is conceptually a request from the builder
[02:49] <lifeless> mwhudson: garbo tasks *don't* fit well unless we have a celebrity with the right permissions
[02:49] <mwhudson> lifeless: i guess where this leads to is that, yes, we could replace the use PSP in scripts with something else
[02:49] <lifeless> but, coincidentally, thats exactly what we do do for the DB; I don't see why we shouldn't do it higher up too.
[02:49] <mwhudson> but i don't think you could easily replace it with the LaunchpadSecurityPolicy
[02:49] <mwhudson> because that's all based around principals that are Persons
[02:50] <lifeless> mwhudson: I think a good mental exercise is to ask 'what would it take to make script X an API client
[02:50] <lifeless> we probably need to get pgbouncer installed at some point
[02:50] <lifeless> but even then, it would be nice to have less sources of idle connections
[02:51] <mwhudson> lifeless: well funnily enough i did that fairly recently
[02:51] <lifeless> mwhudson: and what did it entail ? :)
[02:51] <mwhudson> i changed code imports to do all their communication with the db via the internal xml-rpc server
[02:51] <mwhudson> lifeless: calling removeSecurityProxy a lot :(
[02:51] <lifeless> mwhudson: thats kindof cheating
[02:51] <mwhudson> yep
[02:52] <lifeless> mwhudson: can we do better ?
[02:52] <mwhudson> not even kindof
[02:52] <mwhudson> lifeless: i don't konw
[02:52] <mwhudson> lifeless: i wrote some mails about this this a while back, lemme hunt
[02:53] <mwhudson> lifeless: does saying "Message-ID: <4B8C8089.1030105@canonical.com>" help you with your mail setup?
[02:54] <lifeless> hahaha
[02:55] <mwhudson> lifeless: or http://www.mail-archive.com/launchpad-dev@lists.launchpad.net/msg02733.html
[02:55] <lifeless> mwhudson: subjects are normally enough
[02:56] <mwhudson> "using PermissiveSecurityPolicy when serving private xmlrpc requests"
[02:57] <lifeless> thanks
[02:58] <mwhudson> mutable global state aaaaaaaaaaaaaaaa
[02:58] <lifeless> mwhudson: what just  bit you ?
[02:58] <lifeless> also, you know we have a database, right ?
[02:58] <mwhudson> lifeless: the thing i refer to in the first mail in that thread
[02:59] <mwhudson> it's not really possible to use a different interaction class for a given request
[02:59] <mwhudson> interaction class == security policy btw
[03:00] <lifeless> so
[03:01] <lifeless> I think I'm fairly happy with saying:
[03:01] <lifeless>  - PSP is almost certainly covering bugs and security holes
[03:01] <lifeless>  - it divides our code arbitrarily and makes moving code out of web requests into backend systems hard and fragile
[03:02] <lifeless>  - I don't see, and haven't seen a case for PSP existing other than 'its how we made stuff work way back when'
[03:02] <mwhudson> maybe a special principal that LaunchpadSecurityPolicy does something different with would be ok
[03:02] <mwhudson> or special class of principals
[03:02] <lifeless> mwhudson: I don't see why impersonation isn't totally sufficient
[03:03] <lifeless> have a privileged version of login()
[03:03] <wgrant> You also need a superpowered principal.
[03:03] <lifeless> grant script principles access to that
[03:03] <wgrant> Since lots of operations shouldn't even be possible for ~admins.
[03:03] <mwhudson> lifeless: you said earlier that "<lifeless> mwhudson: anyhow, I think it fits fairly well; finishing a build is conceptually a request from the builder"
[03:04] <lifeless> mwhudson: I did
[03:04] <mwhudson> lifeless: by builder did you mean 'person who uploaded the source package' ?
[03:04] <thumper> :(
[03:04] <thumper> it appears that facets are still used
[03:04] <lifeless> mwhudson: no, I meant the build slave
[03:04] <thumper> :((
[03:04] <lifeless> mwhudson: the one that builds
[03:04] <mwhudson> lifeless: the build slave isn't a Person
[03:05] <mwhudson> and Persons are the only sort of principal we really have today
[03:05] <lifeless> mwhudson: we have celebrities for this; we might want something better.
[03:05] <lifeless> (I dislike celebrities hugely)
[03:05] <lifeless> but, they are square, and the hole is square.
[03:05] <mwhudson> eww
[03:06] <lifeless> mwhudson: we have a celeb for the software centre agent, for instance.
[03:06] <lifeless> which is doing *exactly* this sort of thing
[03:06] <mwhudson> yes, i guess so
[03:06] <mwhudson> doesn't mean it's not horrible though
[03:06] <lifeless> sure
[03:06] <lifeless> I agree
[03:07] <lifeless> I'm happy though, to trade two, pervasive, icky things, for one pervasive icky thing and a clear concept for work-on-behalf-of.
[03:07] <lifeless> and then we can look at the remaining icky thing.
[03:08] <mwhudson> hang on
[03:08] <mwhudson> two pervasive icky things?
[03:08] <mwhudson> one is PSP
[03:08] <mwhudson> what's the other?
[03:08] <lifeless> celebrities
[03:08] <mwhudson> ah ok
[03:08] <mwhudson> i think i misread you then
[03:23]  * wgrant despairs at buildd-manager logging priorities.
[03:23] <wgrant> A build failed? CRITICAL! I can't communicate with a builder? Debug.
[04:37] <spm> wgrant: ho hum.
[04:40] <spm> wgrant: actually - Not being able to communicate with a Builder is perhaps info at best. outputting a critical on network blips would be a complete pita; and has been a problem with soyuz.
[04:41] <spm> for services of this nature, the best I can describe: if a human *MUST* intervene, it's critical. if they don't have to, s/w can recover on it's own? it's error or lower.
[04:43] <mwhudson> There is no blueprint named "" in kubuntu, or krunch-desktop-plan isn't valid dependency of that blueprint.
[04:43]  * mwhudson hearts the blueprints code
[04:46] <thumper> mwhudson: you forgot your sarcasterisk
[04:47] <mwhudson> tis true
[04:49] <spm> thumper: i dunno... in this case I don't think it was needed. the bright flashy neon lit sign with ***sarcasm ahead*** and awoooogah "sarcasm warning" horn, were a bit of a giveaway. ???
[05:07] <lifeless> mwhudson: do you think its ok to have Participation support annotations ?
[05:07] <mwhudson> lifeless: probably
[05:08] <mwhudson> i didn't realise in my first mail that Participation was a launchpad thing
[05:14] <wgrant> spm: I'm thinking of making communication errors like that a warning, disabled builders errors, and nothing critical.
[05:16] <wgrant> Everything that was previously critical could only be a warning at most.
[05:16] <spm> sweet
[05:19] <mwhudson> does anyone know by what mechanism the doctests in lp.registry.browser.tests get run?
[05:21] <wgrant> mwhudson: Not test_views?
[05:21] <wgrant> That instantiates a LayeredDocFileSuite.
[05:21] <mwhudson> wgrant: ah yes, thanks
[05:23] <mwhudson> oh yes doctests, how do i hate thee, let me count the ways
[05:23]  * mwhudson stares at this one and thinks about converting it to a unit test
[05:24] <wgrant> Who is your victim today?
[05:28] <mwhudson> wgrant: part of vocabularies.txt
[05:30] <wgrant> yay.
[05:42] <thumper> lifeless: remember we were talking about the project cloud the other day?
[05:44] <thumper> lifeless: this seems to be a much more performant (and relevant) query: select product.name, count(*) as commits, count(distinct(revision_author)) as author_count, max(revision_date) as last_commit from revisioncache, product where revisioncache.product = product.id and not revisioncache.private group by product.name order by count(*) desc limit 500
[05:45] <thumper> not sure which value should be the size though...
[05:45] <thumper> commit count or author count
[05:46] <thumper> suggestions anyone?
[05:46] <lifeless> commits per author ?
[05:47] <wgrant> Some combination of commit and author counts seems best.
[05:47] <lifeless> we don't want just kde, gnome etc showing up
[05:47] <lifeless> and they are biased to large commit counts & authors, but their normalised contributions should be much smaller
[05:48] <wgrant> But KDE and GNOME are not projects, hopefully.
[05:48] <lifeless> shrug
[05:48] <lifeless> if you want to be picky
[05:48] <wgrant> The projects within GNOME and KDE should not be overwhelmingly active.
[05:49] <lifeless> wgrant: I'm 99.9999999% sure you know what I am talking about.
[05:50] <thumper> we have size and colour to use
[05:50] <thumper> perhaps size is based on number of commits
[05:50] <thumper> and darkess grouped on committer numbers
[05:51] <lifeless> if your metrics are highly correlated
[05:51] <lifeless> then this will just mean small=dark big=light (or vice versa
[05:52] <thumper> yeah... mostly
[05:52] <lifeless> and thus it would be simler to just have one figure you calculate
[05:52] <lifeless> and show small=dark, big=light
[05:52] <thumper> although not always the case
[05:52] <lifeless> OTOH, if they are not highly correlated, it may look fugly.
[05:52] <lifeless> :)
[05:52] <thumper> openerp-hr-payroll-cr          |     568 |           11 | 2010-08-06 21:37:17.629
[05:52] <thumper>  mplayer                        |     103 |           11 | 2010-08-07 18:23:31.786
[05:52] <thumper>  ubuntu-seeds                   |      18 |           10 | 2010-08-07 03:31:38.477
[05:52] <thumper> commits is second
[05:52] <thumper> count is third
[05:53] <thumper> author count that is
[05:53] <lifeless> personally, I don't think folk try to get stats out of the cloud
[05:53] <thumper> no, they don't
[05:53] <lifeless> wearing my colourblind-critic hat
[05:53] <thumper> perhaps just don't bother with shade :)
[05:53] <lifeless> I'd really rather keep it simple
[05:53]  * thumper nods
[05:53] <thumper> ok, just size based on commit count in the last 30 days
[05:54] <thumper> agreed?
[05:54] <lifeless> perhaps size based on commit count/author count
[05:54] <lifeless> to let small but prolific show up
[05:54] <thumper> ah, ok
[05:54] <lifeless> perhaps thats a bad idea; I don't know.
[05:55]  * thumper runs to guitar lesson :)
[05:55] <lifeless> ciao
[06:09] <mwhudson> i wonder how many times people view code.launchpad.net
[06:10] <mwhudson> in the 3.0 design it's not easy to get to
[06:10] <lifeless> mwhudson: project group clouds can die too
[06:11] <mwhudson> lifeless: what are they?
[06:11] <lifeless> erm, I may have the wrong context
[06:11] <lifeless> thumper said when we were tlaking on th ephone
[06:11] <lifeless> that the global cloud is just worst
[06:11] <lifeless> that smaller ones also have trouble from time to time
[06:18] <mwhudson> i didn't realize we had smaller clouds
[06:18] <wgrant> The only other clouds I know of are the bug tag ones.
[06:24] <StevenK> Argh, why does PQM hate me
[06:26] <StevenK> Are we in testfix?
[07:22] <spm> * fyi * about to stab the buildbot master, have a new hardy-slave built and want to ensure it gets picked up
[07:29] <noodles775> Morning
[07:32] <spm> heya noodles
[07:32] <noodles775> Hi spm
[07:34] <spm> * fyi * buildbot master appears to be happy again; new hardy-slave picked up. we return to your regular unscheduled building.
[07:37] <lifeless> grah detoxing from caffeine headache :(
[07:37] <lifeless> also hate hate hate untested code
[07:43] <spm> lifeless: ?? isn't the cure for a caffiene headache to have more caffeine? If you keep this up eventually the headache WILL go away; Of course you'll also be dead, but that's considered a mere side effect
[08:05] <nigelb> 'mere side effect', heh
[08:25] <adeuring> good morning
[08:43] <jtv> hi adeuring
[08:44] <adeuring> hi jtv
[08:44] <jtv> Is anyone else getting what looks like missing CSS on edge?
[08:44] <adeuring> one some pages, yes
[08:44] <noodles775> jtv: yep.
[08:45] <noodles775> jtv: have you let a losa know?
[08:45] <jtv> And it's on r11435… I think it was on 11430 an hour or so ago
[08:45] <jtv> I'm just finding out.
[08:45] <spm> argh. not again!?!?!
[08:46] <jtv> spm: see for yourself
[08:46] <spm> so I do
[08:46] <jtv> then it's not just the rest of us
[08:47] <jtv> I _think_ it upgraded from 11340 to 11345 just now.
[08:48] <adeuring> The missing CSS lets me spot new details of the pages. I did not know that we have a "progress bar" for configuration on the main project pages and that https://edge.launchpad.net/launchpad is only 75% configured
[08:49] <jtv> spm: is this something you can do anything about?
[08:50] <jtv> Or at least, does anyone know what causes this?
[08:50] <spm> launchpad-rev-11415 to launchpad-rev-11435
[08:50] <jtv> ahhh I see you're fixing stuff already
[08:50] <spm> should be in the edge restore email to the error list
[08:50] <jtv> thanks for the fast reaction
[08:50] <spm> :-)
[08:52] <jtv> yay!  CSS!
[08:52] <jtv> Actually in some ways I kind of liked our new, back-to-basics look.
[08:53] <spm> adeuring: nice find for the silver lining there! ;-)
[08:53] <adeuring> yeah ;)
[08:54]  * jtv wonders if that phrase is taken as a name for a cloud computing-related infrastructure project
[08:54] <wgrant> Hm, still broken for me.
[08:54] <spm> ahh crap. I need to do the FE's as well. ta.
[08:55] <jtv> Hi mrevell, thanks for the email
[08:55] <mrevell> Hu
[08:55] <mrevell> Or should I say, Hi?
[08:56] <jtv> I think Hi is better.
[08:56] <mrevell> jtv, My pleasure, I'm sorry for the delay.
[08:56] <mrevell> :)
[08:56] <jtv> np…  I'm hitting something hard and serrated with my ongoing feature work though, so I many not get back to it today.
[08:56] <spm> right that should be fuixed?
[08:57] <jtv> spm: it's fixed again for me
[08:58] <jtv> On to the next one… I have a MP in "updating diff" state more than an hour after the last change was pushed: https://code.edge.launchpad.net/~mwhudson/launchpad/move-SpecificationDepCandidatesVocabulary/+merge/33611
[08:58] <wgrant> Looks good. Thanks spm.
[09:02] <wgrant> bigjools: Morning
[09:02] <spm> jtv: gah. lookin'
[09:02] <jtv> we're sure getting our money's worth out of Steve this evening.
[09:02] <spm> ....
[09:04] <bigjools> wgrant: g'day
[09:04] <spm> yeah. the m-p jobs task has gone gaga; killin'
[09:06] <wgrant> bigjools: Do you have time to talk about ddebs?
[09:06] <bigjools> wgrant: at some point but not just now
[09:06] <wgrant> bigjools: Sure.
[09:06] <bigjools> how long are you around?
[09:06] <spm> jtv: that seems to be processing again. and fwiw, it apepars to be all mwhudson's stuff that caused the problem.
[09:07] <StevenK> Haha
[09:07] <jtv> spm: otp… thanks
[09:07] <spm>  (accusation based on no scientific evidence, beyond his branches in the follow 'is working' log)
[09:07] <wgrant> bigjools: Four or five hours, probably.
[09:07] <bigjools> ok
[10:24] <jml> adeuring, the 75% thing is a known bug that I'm told registry will fix any moment now
[11:38] <jtv> jml: ISTR you mentioned a TAL macro a few years ago that would turn a bunch of fragments into a neat "a, b, and c"—style list.  Can't seem to find it now.
[11:38] <jml> jtv, otp
[11:44] <wgrant> jtv: I don't know that there's a TALES expression for it, but there is canonical.launchpad.helpers.english_list
[11:45] <jtv> wgrant: thanks, that's the one I was thinking of—I thought it was TAL so no wonder I didn't find it!
[12:02] <deryck> Morning, all.
[12:02] <jtv> hi deryck!
[12:03] <jtv> jam: people are getting eager for that BranchRevision weight-loss program we worked on in Prague.  :)
[12:06] <jtv> mrevell: prototype for the translations help-bubble changes at lp:~jtv/launchpad/bug-517700 — playing with the real thing is probably more useful than me describing it in detail.  Still some rough edges, I think.  I'm EOD, but would appreciate feedback later!
[12:06] <mrevell> Thanks jtv, I shall take a look at this next.
[12:36] <jtv> See you tomorrow, folks!
[13:29] <salgado> bigjools, do you have a minute to talk about the removal of the security upload policy?
[13:35] <adeuring> bigjools: ...and another question: ProxiedLibraryFileAlias.http_url ensures that the returned URL does not start with "api.lp.net". The reason seems to be bug 354373, which I don't really understand. I have at present the opposite problem: I _need_ a webservice URL for ProxiedLFAs, see bug 620458.
[13:35] <_mup_> Bug #354373: [API] build.build_log_url and build.upload_log_url provide wrong URLs <api> <Soyuz:Fix Released by cprov> <https://launchpad.net/bugs/354373>
[13:35] <_mup_> Bug #620458: cannot access attachments of private bugs any more <httplib2:Unknown> <Launchpad Bugs:Triaged by adeuring> <https://launchpad.net/bugs/620458>
[13:39] <adeuring> bigjools: so, I could either write a variant of ProxiedLFA.http_url which does not enforce the usage of IWebBrowserOriginatingRequest. like "default_http_url", or I could change the behaviour of http_url so that the current request is always used and add a property like web_browser_http_url which has the currnet behaviour of http_url.
[13:39] <adeuring> but: why is this overriding necessary?
[13:50] <bigjools> adeuring: why do you need webservice URLs for librarian files?
[13:50] <adeuring> bigjools: so that lplib scripts can access private data
[13:51] <adeuring> bigjools: see bug 620458
[13:51] <_mup_> Bug #620458: cannot access attachments of private bugs any more <httplib2:Unknown> <Launchpad Bugs:Triaged by adeuring> <https://launchpad.net/bugs/620458>
[13:51] <bigjools> ok
[13:51] <bigjools> sounds fair enough - I think we overrode it because it was breaking something else though
[13:55] <adeuring> bigjools: OK, so, changing the behaviour of ProxiedLFA.http_url, keeping the current behaviour in something like web_brwoser_http_url and using this in the affected code would be OK for you?
[13:56] <bigjools> adeuring: I can't think of all the ramifications right now, but as long as you don't have to change any of the soyuz tests to make them work then it sounds fine.  I'd check with Gary though to see if he has any thoughts.
[13:57] <adeuring> OK, gary_poster: ^^^
[13:58] <wgrant> It's not so bad any more, since api.launchpad.net doesn't require auth.
[13:58] <wgrant> However, some API clients will still want webapp URLs, so they can serve up links to private files.
[13:58] <wgrant> So we really want both :/
[13:59] <adeuring> wgrant: You can meanwhile access private files via the webservice
[14:00] <wgrant> adeuring: Not if I'm serving links to web clients.
[14:00] <adeuring> wgrant: ?
[14:00] <wgrant> If I use an API client to create a web page, I need to serve webapp URLs, since my users aren't authenticated to the API host.
[14:01] <adeuring> wgrant: ah, right!
[14:05] <salgado> bigjools, did you see my ping earlier about removing the security upload policy?
[14:05] <bigjools> salgado: yes, sorry, I am dealing with other things before getting around to you
[14:05] <bigjools> but there's a lull, so fire away
[14:06] <deryck> gmb, can I get an "amen!" to my changes here:  https://help.launchpad.net/Bugs/ImportFormat ?
[14:06] <salgado> bigjools, soyuz-set-of-uploads.txt depends on that policy, and I've tried changing it to use another policy that accepts the same kinds of uploads but it fails and leaves me with no clue as to why
[14:07]  * bigjools checks
[14:08] <salgado> bigjools, line 326
[14:08] <gmb> deryck, Amen, brother.
[14:08] <gmb> Looks good.
[14:08] <bigjools> salgado: I see
[14:08] <bigjools> what's the error?
[14:08] <deryck> excellent.  thanks, gmb
[14:08] <gmb> np
[14:09] <salgado> bigjools, Failed upload(s): ['unstable_1.0-1'] instead of the rejected exception
[14:10] <salgado> that's when I use the 'buildd' policy
[14:11] <bigjools> salgado: what does the next output say (for read_email())
[14:11] <salgado> None
[14:11] <bigjools> awesome
[14:11] <gary_poster> adeuring: Please confirm if I understand the situation correctly.  http_url was a url friendly to the webservice.  It has changed recently to be a url friendly to the browser.  This is problematic for a number of reasons, many of which go under the category of "backwards compatibility".  You propose to reinstate the previous behavior and create a new attribute named "browser_url" or something similar.  That's my 
[14:11] <gary_poster> right?
[14:11] <adeuring> gary_poster: yes.
[14:12] <adeuring> gary_poster: the alternatvie would be to add something like "default_http_url"
[14:12] <adeuring> which looks a bit odd to me
[14:12] <wgrant> That's not my understanding -- ProxiedLibraryFileAlias has returned a browser-friendly URL for 18 months. Bug attachments just started using it a couple of weeks ago.
[14:12] <jml> jelmer, I'm warming up to the idea of a testr integration branch.
[14:12] <jml> jelmer, lack of incremental output is hurting me.
[14:13] <adeuring> wgrant: well, yes. But http_url is not vey specific to ProxiedLFA
[14:13] <gary_poster> wgrant, ok, thanks for clarification.
[14:14] <gary_poster> adeuring, wgrant, I'm in favor of using the webservice versioning for this.  1.0 and beta should keep the current behavior, whatever it is, since that appears to not be breaking anything and wgrant says it has been stable.
[14:14] <jelmer> jml: Yeah, that's particularly annoying with a project as large as lp.
[14:15] <gary_poster> I like http_url for webservice and browser_url for browser for the devel service, but there's an obvious downside of surprising migration (it's easier to know to migrate when a attribute disappears than when it subtly changes meaning).
[14:16] <adeuring> gary_poster: OK... what about leaving http_url as it is and adding web_url and api_url?
[14:16] <salgado> bigjools, how about I remove that test and add a unit test to AbstractUploadPolicy.setDistroSeriesAndPocket(), which is what raises that exception shown in the email message?
[14:16] <wgrant> The failure mode here is probably just that private files become inaccessible. So it's not that bad.
[14:17] <bigjools> salgado: +1, that doctest needs to die in flames
[14:17] <deryck> adeuring, hi.  Can we get a card into WIP on the Kanban board for that attachment work you're doing?
[14:17] <adeuring> deryck: sure
[14:17] <deryck> adeuring, thanks!
[14:18] <gary_poster> adeuring: http_url will effectively be alias for web_url in your proposal?
[14:18] <adeuring> gary_poster: no necessarily. web_url and api_url should enforce the hostnames code.lp.net and api.lp.net, repsectively
[14:19] <gary_poster> So what is the value of http_url then?  Why would I use it instead of web_ or api_?
[14:19] <salgado> bigjools, cool.  however, there's also a big chunk starting at line 606 for testing staged uploads to the security pocket.  I know there are other tests for staged uploads, so maybe I can just nuke that?
[14:20] <salgado> lib/lp/archiveuploader/tests/test_buildduploads.py has those tests for staged uploads
[14:20] <adeuring> gary_poster: well... I'm trying to find a way to cop out from changing soyuz code while having somewhat same property names ;)
[14:21] <adeuring> Actually, I could simply add api_url -- that's all I need
[14:22] <adeuring> s/same/sane/
[14:24] <gary_poster> adeuring: heh, ok fair enough. :-)  from this conversation, http_url seems poorly defined and unclearly named though.  I'd prefer you add api_url and web_url, and make a note in http_url that that users should cuse api_url and web_url instead, and http_url may be removed in a future version of the webservice.  Maybe that's too aggressive...
[14:24] <gary_poster> That's my preference, but I would be OK with only adding api_url and putting a bug in against the webservice about this problem, so that when leonardr and benji start trying to clean up the webservice generally this is one of the issues they consider tackling.
[14:25] <adeuring> gary_poster: good proposal; I'll go for it.
[14:26] <gary_poster> adeuring: cool, thank you!
[14:26] <leonardr> gary, adeuring, are you aware of rockstar's work on this? has he completed the work and that's cuasing the problem?
[14:26] <gary_poster> leonardr: I have no knowledge of this :-/
[14:26] <adeuring> leonardr: no, maybe his work will fix my problem, no idea.
[14:26] <adeuring> leonardr: the code I'm talking about is from r8166
[14:27] <leonardr> gary, adeuring, at the epic rockstar started working on a 'web_link' that was like 'self_link' except it pointed to the object on the website
[14:28] <leonardr> but given that revision number i imagine you're not talking about something added to lazr.restful
[14:28] <gary_poster> leonardr: Is that pertinent to the library files?
[14:28] <adeuring> leonardr: right, its about lp code itself. and as gary says, about library files
[14:28] <adeuring> ProxiedKFA, more specifically
[14:29] <adeuring> ProxiedLFA
[14:29] <leonardr> so, the library files used to have an http_url that used whatever host the reuqest came from?
[14:29] <adeuring> leonardr: yes, and that points _not_ to the webservice
[14:30] <adeuring> making access to the files from a webservice client impossible from private files
[14:30] <adeuring> s/from/for/
[14:30] <leonardr> ok, i see
[14:30] <gary_poster> adeuring, just a warning, we're about to have a team call, so will be away for just a moment
[14:31] <adeuring> ok
[14:31] <leonardr> in that case, you can check for request.version to see which version of the web service is in use, and change behavior based on that
[14:31] <leonardr> i don't have an opinion on what you should implement, i just wanted to make sure this wasn't overlapping rockstar's work
[14:32] <rockstar> leonardr, I should get back to fixing that one day.
[15:33] <gmb> Aaaaaaaaa
[15:33] <gmb> So.
[15:34] <gmb> Why would assertRaises() in a test case *not* catch the exception that I'm asserting the callable raises?
[15:38] <jml> hmm
[15:38] <salgado> bigjools, did you my msg earlier about the staged-upload test on soyuz-set-of-uploads.txt?
[15:38] <jml> gmb, I'm thinking about that.
[15:38] <jml> gmb, I'm fairly sure the answer is that you are doing it wrong.
[15:38] <gmb> jml, Specifically, the exception is zope.security.interfaces.Unauthorized
[15:39] <gmb> And the code is:
[15:39] <gmb>         self.assertRaises(
[15:39] <gmb>             Unauthorized, self.bug_tracker.resetWatches,
[15:39] <gmb>             "Unprivileged users should not be allowed to reset a "
[15:39] <gmb>             "tracker's watches.")
[15:39] <jml> gmb, ahh, I know this one :)
[15:39] <gmb> Oh goodie.
[15:39] <gmb> Do share.
[15:39] <jml> gmb, it's getattr(self.bug_tracker, 'resetWatches') that's raising the Unauthorized
[15:39] <bigjools> salgado: sorry missed that, looking now
[15:40] <jml> gmb, rather than the actual method call.
[15:40] <gmb> jml, Ah, because it's launchpad.Admin'd.
[15:40] <gmb> so an unpriv'd user can't get at the method, let alone call it.
[15:40] <jml> gmb, and because zope security works on attribute access.
[15:40] <jml> gmb, exactly.
[15:40] <gmb> D'oh. So obvious.
[15:40] <gmb> jml, Thanks.
[15:40] <jml> self.assertRaises(Unauthorized, getattr, self.bug_tracker, 'resetWatches') has worked well for me in the past.
[15:41] <jml> (although arguably that's a custom assertion method / matcher waiting to happen)
[15:41] <jml> gmb, np.
[15:42] <bigjools> salgado: I think we can nuke the test
[15:50] <salgado> bigjools, cool, the problem now is that the test hangs after I removed that section.  I'll see if I can find out where/why
[15:51] <bigjools> salgado: argh.  That test is a nightmare.
[15:52] <salgado> bigjools, btw, would you like to have a look at the other branch which replaces the can_upload_* attributes with a single enum?  jtv has approved it, but I thought you might want to have a look anyway?
[15:52] <bigjools> salgado: I can but I'm not sure when!
[15:52]  * bigjools is too busy :(
[15:55] <salgado> bigjools, maybe jelmer or StevenK can have a look?  or if you think it's not necessary, I've already got jtv's approval anyway
[15:55] <salgado> btw, it's publish-distro.py that hangs
[15:56] <bigjools> salgado: don't block on landing it, we can look later.  jelmer may be very interested anyway as he's changing the upload processor a bit at the moment.
[15:58] <salgado> ok, cool
[16:38] <jml> "testr run failing" doesn't do what I meant
[16:39] <deryck> gmb, the an MP I approved for a "scratch" branch of yours.  Can that be landed?
[16:46] <gmb> deryck, Er. Hang on, I don't remember that.
[16:46] <gmb> Blimey, that was a while back.
[16:46] <deryck> Judging by the diff I wonder if you merged it in another branch?
[16:47] <gmb> deryck, Ah, I think that first bit was to do with the fix for the initial_message problem.
[16:47] <gmb> Hrm.
[16:47] <gmb> deryck, I'll do some digging and find out what's landed and what's not.
[16:48] <gmb> I suspect that diff is a lie.
[16:48] <deryck> ok, cool.  Thanks!
[16:51] <gmb> deryck, Yes, there's some lying going on. Well, not lying, but basically the diff is against the ancestor revision of the scratch branch; when I merge devel it conflicts with what's already landed. I'll clean it up and submit it.
[16:56] <deryck> gmb, ok,cool.
[17:20] <jelmer> salgado: I see you're having fun with huge interdependent soyuz doctests
[17:30] <salgado> jelmer, yes!  it's been such a long time since it last happened that I'd almost forgotten how much fun they can be
[17:38] <jelmer> salgado: :-)
[18:35] <leonardr> i'd like to talk to someone who understands zope permissions well, maybe gary, or salgado-lunch once he returns from lunch
[18:36] <gary_poster> leonardr: benji-lunch would be a good choice too.  I better go get some lunch because I have a call in 24 min :-/
[18:36] <gary_poster> otherwise I should be available 3:30 or 4
[18:36] <leonardr> ok
[18:36] <leonardr> i'll just explain the problem
[18:36] <gary_poster> ok
[18:36] <leonardr> i've created a security policy for IOAuthAccessToken that basically says:
[18:36] <leonardr> if you're trying to look at this oauthaccesstoken through the website, the old rules apply: it has to be your token, or you have to be an admin
[18:38] <leonardr> if you're trying to look at this oauthaccesstoken through the web service itself, the rules are more restrictive.
[18:39] <leonardr> you can only look at your own token, and your request must itself be signed by an oauthaccesstoken that has the GRANT_PERMISSIONS access level
[18:40] <leonardr> this works fine for prohibiting writes to the token, and it also keeps the token from showing up in lists in the web service (since you don't have launchpad.View on the token)
[18:40] <leonardr> but, you can still guess the url and get the token data that way
[18:40] <leonardr> so i added this bit to oauth.zcml
[18:40] <leonardr>       <require
[18:40] <leonardr>           permission="launchpad.View"
[18:40] <leonardr>           interface="canonical.launchpad.interfaces.IOAuthAccessToken"/>
[18:42] <leonardr> and that protects the objects themselves
[18:42] <leonardr> however, there's a catch-22: to determine whether the request is signed by an appropriate OAuthAccessToken, you need to be able to look at an OAuthAccessToken object
[18:43] <leonardr> that's where i'm stuck
[18:44] <gary_poster> leonardr: which component needs to be able to look at an OAuthAccessTokenObject?
[18:44] <gary_poster> leonardr: a mediator is a typical pattern for this
[18:44] <gary_poster> mediator rips off security proxy and does what needs to be done and returns answer
[18:44] <leonardr> gary: well, right now, the code that signs the _outgoing_ request needs to be able to look at it. the request isn't even being made
[18:45] <gary_poster> would mediator work in context?
[18:45]  * gary_poster really should get some food
[18:45] <leonardr> go ahead
[18:45] <jml> gary_poster, stay hungry, the TL meeting will be shorter for it :)
[18:45] <leonardr> i'll try some stuff
[18:45] <gary_poster> jml :-)
[19:11] <leonardr> gary: two well-placed removeSecurityProxy calls solved the problem
[19:15] <jml> g'night all.
[19:16] <gary_poster> leonardr: great.
[19:17] <bigjools> nn jml
[20:40] <leonardr> benji, got another problem with my permissions. the 'view' permission seems to work correctly, but the 'edit' permisison check is failing without my code ever being called
[20:40] <leonardr> let me know what kind of details will help
[20:41]  * benji scrolls back to get context.
[20:42] <leonardr> benji: basically i updated the AuthorizationBase subclass for OAuthToken objects
[20:42] <leonardr> so that you can only modify them from the web service under certain circumstances
[20:42] <leonardr> my code is running when it comes to _viewing_ objects through the web service
[20:43] <leonardr> but when i try to modify one, i get Unauthorized, and the code from security.py never runs
[20:44] <leonardr> setattr(context, self.name, value) raises an exception
[20:47] <benji> leonardr: What is the security checker for the object in question?  Also, I'm trying to figure out how your AuthorizationBase subclass tied into zope.security.  I've not touched the LP-specific security stuff any yet.
[20:49] <leonardr> benji, i believe the seucirty checker is canonical.launchpad.webapp.authorization.LaunchpadSecurityPolicy
[20:50] <leonardr> benji: NEVER MIND. i brought this problem on myself
[20:51] <benji> leonardr: I'd put some breakpoints in one or two methods of LaunchpadSecurityPolicy and then execute your setattr; tracing through what happens should...
[20:51] <leonardr> there is a real problem, but i understand why this is happening
[20:51] <benji> :)
[20:53] <leonardr> benji: the real problem is in webapp/authorization.py, _checkRequiredAccessLevel
[20:54] <leonardr> an AccessLevel of GRANT_PERMISSIONS doesn't have the ability to 'write'
[20:54] <leonardr> i want a situation where GRANT_PERMISSIONS has the ability to 'write', but only to OAuthACcessToken objects
[20:55] <benji> makes sense
[20:56] <leonardr> i have no clue how to do this. i can use the zcml guards to attach an AuthorizationBase subclass to OAuthAccessToken
[20:57] <leonardr> i guess i could change AuthorizationBase to explicitly forbid writes if the AccessLevel is GRANT_PERMISSIONS, but that seems hacky
[20:57] <leonardr> i think salgado might have some insight into this
[21:17] <lifeless> .
[21:55] <salgado> leonardr, maybe, but I'd need more context
[21:56] <leonardr> salgado: so, take a look at LaunchpadSecurityPolicy._checkRequiredAccessLevel
[21:56] <leonardr> this code says "no matter what permissions the principal has, if the access level is not high enough, access denied"
[21:57] <leonardr> i would like GRANT_PERMISSIONS to be considered a 'read' access level for everything _except_ oauth access tokens
[21:58] <leonardr> i implemented permissions to this effect (you can only write to an oauth access token if you are using GRANT_PERMISSIONS)
[21:58] <leonardr> but since GRANT_PERMISSIONS is considered a 'read' access level globally, you never get to use those permissions
[22:01] <leonardr> the only thing i can think of is to make GRANT_PERMISSIONS a 'write' access level, and special-case the superclass of all write-permission checkers so that GRANT_PERMISSIONS does _not_ give you any write permisson
[22:01] <leonardr> or, give up and just make GRANT_PERMISSIONS a 'write' access level
[22:03] <benji> leonardr: it wouldn't seem too bad for GRANT_PERMISSIONS to have write access; after all if something has GRANT_PERMISSIONS then they could just give themselves write access, right?
[22:03] <leonardr> benji: yes, the idea is more to make sure that a GRANT_PERMISSIONS script doesn't suffer feature creep and become a do-all-sorts-of-things script
[22:04] <benji> mmm
[22:04] <benji> giant ascii art warning perhaps?  :P
[22:05] <leonardr> if we could determine when to print that warning, we could just deny access :P
[22:06] <mwhudson> morning
[22:08] <salgado> leonardr, what about forcing all tokens with permission==GRANT_PERMISSIONS to be scoped to OAuthToken?  that way the client would have whatever access_level is defined in GRANT_PERMISSIONS for OAuthToken but read-only access for everything else
[22:11] <leonardr> salgado: that's a good idea, but i'm pretty sure scoped tokens don't work and never did work
[22:12] <leonardr> but, it's possible the internals work and the interface was never completed
[22:13] <salgado> I think that's the case, but even if it doesn't work it should be easy to fix it
[22:13] <leonardr> ok, i will look into this tomorrow
[22:57] <mwhudson> ec2 land is blowing up for me
[22:57] <mwhudson> Exception AttributeError: "'SmartSSHClientMedium' object has no attribute '_ssh_connection'" in <bound method SmartSSHClientMedium.__del__ of SmartSSHClientMedium(bzr+ssh://mwhudson@bazaar.launchpad.net/)> ignored
[22:57] <mwhudson> regular bzr operations work fine though
[22:58] <mwhudson> any ideas anyone?
[22:59] <mwhudson> sinzui: did you get my mail about menus?
[23:01] <mwhudson> hm, probably not
[23:01] <mwhudson> sinzui: https://lists.launchpad.net/launchpad-dev/msg04367.html
[23:01]  * sinzui looks
[23:02] <sinzui> mwhudson, I did not see this reply
[23:04] <mwhudson> sinzui: yeah, i screwed up my mail server config somehow
[23:04] <sinzui> Well I will reply shortly
[23:04] <mwhudson> it's not a very deep reply, mostly a series of questions....
[23:04] <mwhudson> cool
[23:17] <rockstar> wallyworld, do you have something like this in ~/.ssh/config http://pastebin.ubuntu.com/483652/
[23:17] <wallyworld> i'll look
[23:28] <lifeless> sinzui: https://edge.launchpad.net/landscape/+milestone/later
[23:28] <lifeless>  At least 81 queries issued in 11.15 seconds
[23:28] <lifeless> sinzui: seems a bit healthier
[23:29] <lifeless> (and I'm seeing the private bugs)
[23:31] <sinzui> yep
[23:34] <wgrant> lifeless: I wouldn't call it healthy -- there are still massive scaling issues.
[23:34] <wgrant> Takes 1.1s here.
[23:34] <wgrant> 69 queries.
[23:35] <lifeless> wgrant: I wouldn't call it healthy either
[23:35] <lifeless> wgrant: 20, constant, would be healthy.
[23:35] <lifeless> wgrant: but healthier, eys.
[23:35] <wgrant> I wonder where the extra 10s comes from.
[23:35] <wgrant> Sure not those 12 queries.
[23:36] <lifeless> OOPS-1698EA2488 may tell us
[23:37] <wgrant> Do you want an OOPS from mine as well to compare?
[23:46] <rockstar> wallyworld, I think you're looking for /etc/apache2/sites-available/local-launchpad
[23:48] <rockstar> wallyworld, do you have bazaar.launchpad.dev in your /etc/hosts ?
[23:49] <wallyworld> yep - 127.0.0.99