[00:00] enums seems to be our canned answer for this [00:01] i think it's a big case of WHUI [00:01] WHUI ? [00:01] We Haven't Used It [00:01] a SteveA-ism [00:01] google was ... un helpful [00:02] something like KarmaAction is in the db so we can tweak the constants to adjust the way karma is allocated without landing code changes [00:02] mwhudson: so, we're talking IRequest [00:02] but of course we don't [00:02] etc [00:02] stub will know more [00:02] lifeless: i'm actually replying to your email [00:02] mwhudson: cool [00:02] mwhudson: the big question, is an unknown [00:03] mwhudson: is it shallow enough I should plunge on and doit [00:03] mwhudson: or should I thread-locals-it. [00:03] lifeless: what's 'it' ? [00:04] lifeless: 'karmacategories' in http://bazaar.launchpad.net/~wgrant/launchpad/bootstrap-db-from-scratch/annotate/head:/utilities/bootstrap-lp-db is the data in question. [00:04] mwhudson: it is [00:05] mwhudson: making scripts have an IRequest always, so that when they do sql it is logged in my new code [00:05] the second it [00:05] is [00:05] the new code : change requesttimeline to be a threadslocal thing [00:08] lifeless: ok, my email is nearly done [00:10] # This is a convenient hack to set up a zope interaction, before we get [00:10] # the proper API for having a principal / user running in scripts. [00:10] # The script will have full permissions because of the [00:10] # PermissiveSecurityPolicy set up in script.zcml. [00:10] ha ha [00:10] i wonder when that was written [00:10] * mwhudson bets on 2005 [00:11] * mwhudson wins [00:12] timestamp: Tue 2005-04-12 09:37:50 +0000 [00:12] from the arch days [00:12] steve.alexander@canonical.com/launchpad--devel--0--patch-368 [00:17] lifeless: ok, mail sent [00:27] thanks [00:37] mwhudson: followup sent btw [00:46] lifeless: i replied one more time, happy to talk about it in irc now :-) [00:46] although i don't think there's much need [00:50] * mtaylor thinks you're both wrong and obviously everything should be re-written in google go [00:50] * mtaylor falls on the floor laughing [00:51] * mtaylor is obviously in an odd mood [00:51] mwhudson: Patches welcome [00:51] Doh [00:52] IRC tab fail :-( [01:02] mwhudson: I think I'm good. [01:02] mwhudson: I guess that under setupInteractionByEmail(ANONYMOUS) in script base [01:03] mwhudson: I'll add something??? that sets up a participationwithannotations ? [01:08] lifeless: setupInteractionByEmail takes a participation as an argument [01:16] mwhudson: yeah [01:16] actually though [01:17] set_request_started is where scripts expect to do stuff [01:17] so *it* needs to check and see if there is a participiation...ICanHasAnnotations, and if not setone up [01:17] we still need to unify these two things [01:18] I like your approach, but I'm not sure we don't actually want - eventually - scripts to say they are in requests via participations rather than set_request_started [01:19] i admit i don't really know what set_request_started is about [01:23] lifeless: in the particular case of checkwatches, it does it's one interaction management [01:24] -' [02:05] mwhudson: kindof-management [02:05] mwhudson: but yes [02:06] lifeless: it looks like it probably suffers locally from the kind of confusion it would be good to clear up globally [02:07] with_interaction looks like exactly what needs clearing up [02:08] vis-a-vis transactions, security & context [02:08] also @statement_logging is just bong [02:10] i'm not sure i'm correct or expressing this clearly [02:10] but i wonder if there's a bit of a tension between things that don't care at all about participations (like most scripts and tests) and the few things that do (like checkwatches) [02:11] most scripts do work on behalf of someone [02:11] also, i wonder if thinking about how you'd like stuff bundled up in an oops report is a good guideline towards how long your participations should be current for [02:11] the work doesn't just 'appear' [02:12] I think that bundling point is exactly on the spot [02:12] its certainly how I think abou tit [02:13] excuse me; brain flagging food needed [02:13] so for example, each job you process in a job running script should have it's own participation [02:16] * mwhudson reads errorlog.py, is surprised to find it reads db statements out of the request, realizes it's because lifeless' branch is merged in [02:21] mwhudson: huh, no. [02:21] mwhudson: oh righ, locally perhaps :) [02:21] anyhow, our webapp adapter is essentially tracking units of work [02:21] called 'requests' [02:22] I think that this is fine and sensible, even for scripts, but what isn't fine or sensible is having this separate to the object representing the work - the IRequest [02:23] ARGH. [02:24] * wgrant curses whoever decided that doctest log levels should be specified in the test registration, and should not be overridable within the test itself. [02:24] wgrant: welcome to doctests [02:27] Baaah. [02:28] * StevenK finishes QAing gina [02:28] That was ... fun [02:29] It's even more fun when you have to do it locally, because there are no configs for that. [02:29] I have mawson for that sort of thing [02:32] * wgrant fixes build logging. [02:33] But buildd-slavescanner.txt seems to want me dead. [02:34] wgrant: Not just you, I suspect. [02:35] Rarely have a found a doctest so slow and seemingly so fragiley malevolent. [02:35] mwhudson: does my point about 'on behalf of' make sense? [02:35] hmm [02:35] lifeless: yes, but i'm not sure how relevant it is [02:36] wgrant: I see your buildd-slavescanner.txt and raise you gina.txt [02:36] StevenK: True. That one is really really slow. [02:36] mwhudson: thats interesting; I thought it was the heart of the issue [02:36] lifeless: scripts use the PermissiveSecurityPolicy by default, so in some sense at least the current principal doesn't really matter [02:37] mwhudson: I think the PSP is essentially undesirable [02:37] lifeless: maybe [02:37] if: [02:38] - we started scripts with an anonymous participation with a stubbish request [02:38] - and the regular sec policy [02:38] - and they called login() as soon as they identified the work they were doing [02:38] would we need the PSP at all ? [02:39] lifeless: well [02:39] there's stuff like IBranch['updateScannedDetails'] that the scanner calls [02:39] * mwhudson pauses, backtracks [02:41] lifeless: i'm not sure this is really a good example, but there's a branchChanged method on branches [02:41] this is called by codehosting to record the format & tip of a branch [02:41] no this is a really bad exmple [02:42] lifeless: basically the point i'm trying to make is that i have this feeling that many scripts call 'internal' apis [02:42] that we wouldn't want the user to call via the webservice api say [02:43] for example, the stuff the build manager calls to record that a build has finished [02:43] so thats a great example [02:43] there is a nonce [02:43] which is security sensitive [02:44] It's not a nonce, and it's not security sensitive. [02:44] But OK. [02:44] wgrant: if we want to allow the buiild slaves to push results, it becomes security sensitive [02:44] wgrant: and I think it was julian who called it a nonce. [02:44] there is this thing [02:44] :) [02:45] if you don't have it, we would not believe a claim that is the result of a build [02:45] if you do, we can believe that. [02:45] (That's one explanation for its existence, but I don't think it's correct. Nobody really knows.) [02:45] wgrant: it was added in to support slaves pushing back [02:45] wgrant: I know this because I was tolk thats why [02:46] its a WHUI case, but one we should. [02:46] anyhow [02:46] *IF* you imagine that we submit build results via the API [02:46] I imagine we'd check something like [02:46] source ip address (are you a build slave) [02:46] and [02:46] (do you have the right nonce) [02:47] if you have those two things, you can say a build is finished, if you don't, you can't. [02:47] *noone* except the dispatcher can read the nonce [02:47] (this is ideally, not describing what we have today) [02:48] mwhudson: anyhow, I think it fits fairly well; finishing a build is conceptually a request from the builder [02:49] mwhudson: garbo tasks *don't* fit well unless we have a celebrity with the right permissions [02:49] lifeless: i guess where this leads to is that, yes, we could replace the use PSP in scripts with something else [02:49] but, coincidentally, thats exactly what we do do for the DB; I don't see why we shouldn't do it higher up too. [02:49] but i don't think you could easily replace it with the LaunchpadSecurityPolicy [02:49] because that's all based around principals that are Persons [02:50] mwhudson: I think a good mental exercise is to ask 'what would it take to make script X an API client [02:50] we probably need to get pgbouncer installed at some point [02:50] but even then, it would be nice to have less sources of idle connections [02:51] lifeless: well funnily enough i did that fairly recently [02:51] mwhudson: and what did it entail ? :) [02:51] i changed code imports to do all their communication with the db via the internal xml-rpc server [02:51] lifeless: calling removeSecurityProxy a lot :( [02:51] mwhudson: thats kindof cheating [02:51] yep [02:52] mwhudson: can we do better ? [02:52] not even kindof [02:52] lifeless: i don't konw [02:52] lifeless: i wrote some mails about this this a while back, lemme hunt [02:53] lifeless: does saying "Message-ID: <4B8C8089.1030105@canonical.com>" help you with your mail setup? [02:54] hahaha [02:55] lifeless: or http://www.mail-archive.com/launchpad-dev@lists.launchpad.net/msg02733.html [02:55] mwhudson: subjects are normally enough [02:56] "using PermissiveSecurityPolicy when serving private xmlrpc requests" [02:57] thanks [02:58] mutable global state aaaaaaaaaaaaaaaa [02:58] mwhudson: what just bit you ? [02:58] also, you know we have a database, right ? [02:58] lifeless: the thing i refer to in the first mail in that thread [02:59] it's not really possible to use a different interaction class for a given request [02:59] interaction class == security policy btw [03:00] so [03:01] I think I'm fairly happy with saying: [03:01] - PSP is almost certainly covering bugs and security holes [03:01] - it divides our code arbitrarily and makes moving code out of web requests into backend systems hard and fragile [03:02] - I don't see, and haven't seen a case for PSP existing other than 'its how we made stuff work way back when' [03:02] maybe a special principal that LaunchpadSecurityPolicy does something different with would be ok [03:02] or special class of principals [03:02] mwhudson: I don't see why impersonation isn't totally sufficient [03:03] have a privileged version of login() [03:03] You also need a superpowered principal. [03:03] grant script principles access to that [03:03] Since lots of operations shouldn't even be possible for ~admins. [03:03] lifeless: you said earlier that "<lifeless> mwhudson: anyhow, I think it fits fairly well; finishing a build is conceptually a request from the builder" [03:04] mwhudson: I did [03:04] lifeless: by builder did you mean 'person who uploaded the source package' ? [03:04] :( [03:04] it appears that facets are still used [03:04] mwhudson: no, I meant the build slave [03:04] :(( [03:04] mwhudson: the one that builds [03:04] lifeless: the build slave isn't a Person [03:05] and Persons are the only sort of principal we really have today [03:05] mwhudson: we have celebrities for this; we might want something better. [03:05] (I dislike celebrities hugely) [03:05] but, they are square, and the hole is square. [03:05] eww [03:06] mwhudson: we have a celeb for the software centre agent, for instance. [03:06] which is doing *exactly* this sort of thing [03:06] yes, i guess so [03:06] doesn't mean it's not horrible though [03:06] sure [03:06] I agree [03:07] I'm happy though, to trade two, pervasive, icky things, for one pervasive icky thing and a clear concept for work-on-behalf-of. [03:07] and then we can look at the remaining icky thing. [03:08] hang on [03:08] two pervasive icky things? [03:08] one is PSP [03:08] what's the other? [03:08] celebrities [03:08] ah ok [03:08] i think i misread you then [03:23] * wgrant despairs at buildd-manager logging priorities. [03:23] A build failed? CRITICAL! I can't communicate with a builder? Debug. === Edwin is now known as Guest62855 [04:37] wgrant: ho hum. [04:40] wgrant: actually - Not being able to communicate with a Builder is perhaps info at best. outputting a critical on network blips would be a complete pita; and has been a problem with soyuz. [04:41] for services of this nature, the best I can describe: if a human *MUST* intervene, it's critical. if they don't have to, s/w can recover on it's own? it's error or lower. [04:43] There is no blueprint named "" in kubuntu, or krunch-desktop-plan isn't valid dependency of that blueprint. [04:43] * mwhudson hearts the blueprints code [04:46] mwhudson: you forgot your sarcasterisk [04:47] tis true [04:49] thumper: i dunno... in this case I don't think it was needed. the bright flashy neon lit sign with ***sarcasm ahead*** and awoooogah "sarcasm warning" horn, were a bit of a giveaway. ??? [05:07] mwhudson: do you think its ok to have Participation support annotations ? [05:07] lifeless: probably [05:08] i didn't realise in my first mail that Participation was a launchpad thing [05:14] spm: I'm thinking of making communication errors like that a warning, disabled builders errors, and nothing critical. [05:16] Everything that was previously critical could only be a warning at most. [05:16] sweet [05:19] does anyone know by what mechanism the doctests in lp.registry.browser.tests get run? [05:21] mwhudson: Not test_views? [05:21] That instantiates a LayeredDocFileSuite. [05:21] wgrant: ah yes, thanks [05:23] oh yes doctests, how do i hate thee, let me count the ways [05:23] * mwhudson stares at this one and thinks about converting it to a unit test [05:24] Who is your victim today? [05:28] wgrant: part of vocabularies.txt [05:30] yay. [05:42] lifeless: remember we were talking about the project cloud the other day? [05:44] lifeless: this seems to be a much more performant (and relevant) query: select product.name, count(*) as commits, count(distinct(revision_author)) as author_count, max(revision_date) as last_commit from revisioncache, product where revisioncache.product = product.id and not revisioncache.private group by product.name order by count(*) desc limit 500 [05:45] not sure which value should be the size though... [05:45] commit count or author count [05:46] suggestions anyone? [05:46] commits per author ? [05:47] Some combination of commit and author counts seems best. [05:47] we don't want just kde, gnome etc showing up [05:47] and they are biased to large commit counts & authors, but their normalised contributions should be much smaller [05:48] But KDE and GNOME are not projects, hopefully. [05:48] shrug [05:48] if you want to be picky [05:48] The projects within GNOME and KDE should not be overwhelmingly active. [05:49] wgrant: I'm 99.9999999% sure you know what I am talking about. [05:50] we have size and colour to use [05:50] perhaps size is based on number of commits [05:50] and darkess grouped on committer numbers [05:51] if your metrics are highly correlated [05:51] then this will just mean small=dark big=light (or vice versa [05:52] yeah... mostly [05:52] and thus it would be simler to just have one figure you calculate [05:52] and show small=dark, big=light [05:52] although not always the case [05:52] OTOH, if they are not highly correlated, it may look fugly. [05:52] :) [05:52] openerp-hr-payroll-cr | 568 | 11 | 2010-08-06 21:37:17.629 [05:52] mplayer | 103 | 11 | 2010-08-07 18:23:31.786 [05:52] ubuntu-seeds | 18 | 10 | 2010-08-07 03:31:38.477 [05:52] commits is second [05:52] count is third [05:53] author count that is [05:53] personally, I don't think folk try to get stats out of the cloud [05:53] no, they don't [05:53] wearing my colourblind-critic hat [05:53] perhaps just don't bother with shade :) [05:53] I'd really rather keep it simple [05:53] * thumper nods [05:53] ok, just size based on commit count in the last 30 days [05:54] agreed? [05:54] perhaps size based on commit count/author count [05:54] to let small but prolific show up [05:54] ah, ok [05:54] perhaps thats a bad idea; I don't know. [05:55] * thumper runs to guitar lesson :) [05:55] ciao [06:09] i wonder how many times people view code.launchpad.net [06:10] in the 3.0 design it's not easy to get to [06:10] mwhudson: project group clouds can die too [06:11] lifeless: what are they? [06:11] erm, I may have the wrong context [06:11] thumper said when we were tlaking on th ephone [06:11] that the global cloud is just worst [06:11] that smaller ones also have trouble from time to time [06:18] i didn't realize we had smaller clouds [06:18] The only other clouds I know of are the bug tag ones. [06:24] Argh, why does PQM hate me === almaisan-away is now known as al-maisan [06:26] Are we in testfix? === jtv is now known as jtv_ === jtv_ is now known as jtv [07:22] * fyi * about to stab the buildbot master, have a new hardy-slave built and want to ensure it gets picked up [07:29] Morning [07:32] heya noodles [07:32] Hi spm [07:34] * fyi * buildbot master appears to be happy again; new hardy-slave picked up. we return to your regular unscheduled building. [07:37] grah detoxing from caffeine headache :( [07:37] also hate hate hate untested code [07:43] lifeless: ?? isn't the cure for a caffiene headache to have more caffeine? If you keep this up eventually the headache WILL go away; Of course you'll also be dead, but that's considered a mere side effect [08:05] 'mere side effect', heh [08:25] good morning [08:43] hi adeuring [08:44] hi jtv [08:44] Is anyone else getting what looks like missing CSS on edge? [08:44] one some pages, yes [08:44] jtv: yep. [08:45] jtv: have you let a losa know? [08:45] And it's on r11435… I think it was on 11430 an hour or so ago [08:45] I'm just finding out. [08:45] argh. not again!?!?! [08:46] spm: see for yourself [08:46] so I do [08:46] then it's not just the rest of us [08:47] I _think_ it upgraded from 11340 to 11345 just now. [08:48] The missing CSS lets me spot new details of the pages. I did not know that we have a "progress bar" for configuration on the main project pages and that https://edge.launchpad.net/launchpad is only 75% configured [08:49] spm: is this something you can do anything about? [08:50] Or at least, does anyone know what causes this? [08:50] launchpad-rev-11415 to launchpad-rev-11435 [08:50] ahhh I see you're fixing stuff already [08:50] should be in the edge restore email to the error list [08:50] thanks for the fast reaction [08:50] :-) [08:52] yay! CSS! [08:52] Actually in some ways I kind of liked our new, back-to-basics look. [08:53] adeuring: nice find for the silver lining there! ;-) [08:53] yeah ;) [08:54] * jtv wonders if that phrase is taken as a name for a cloud computing-related infrastructure project [08:54] Hm, still broken for me. [08:54] ahh crap. I need to do the FE's as well. ta. [08:55] Hi mrevell, thanks for the email [08:55] Hu [08:55] Or should I say, Hi? [08:56] I think Hi is better. [08:56] jtv, My pleasure, I'm sorry for the delay. [08:56] :) [08:56] np… I'm hitting something hard and serrated with my ongoing feature work though, so I many not get back to it today. [08:56] right that should be fuixed? [08:57] spm: it's fixed again for me [08:58] On to the next one… I have a MP in "updating diff" state more than an hour after the last change was pushed: https://code.edge.launchpad.net/~mwhudson/launchpad/move-SpecificationDepCandidatesVocabulary/+merge/33611 [08:58] Looks good. Thanks spm. [09:02] bigjools: Morning [09:02] jtv: gah. lookin' [09:02] we're sure getting our money's worth out of Steve this evening. [09:02] .... [09:04] wgrant: g'day [09:04] yeah. the m-p jobs task has gone gaga; killin' [09:06] bigjools: Do you have time to talk about ddebs? [09:06] wgrant: at some point but not just now [09:06] bigjools: Sure. [09:06] how long are you around? [09:06] jtv: that seems to be processing again. and fwiw, it apepars to be all mwhudson's stuff that caused the problem. [09:07] Haha [09:07] spm: otp… thanks [09:07] (accusation based on no scientific evidence, beyond his branches in the follow 'is working' log) [09:07] bigjools: Four or five hours, probably. [09:07] ok [10:24] adeuring, the 75% thing is a known bug that I'm told registry will fix any moment now === Guest17053 is now known as jelmer === jelmer is now known as Guest61071 === Guest61071 is now known as jelmer [11:38] jml: ISTR you mentioned a TAL macro a few years ago that would turn a bunch of fragments into a neat "a, b, and c"—style list. Can't seem to find it now. [11:38] jtv, otp [11:44] jtv: I don't know that there's a TALES expression for it, but there is canonical.launchpad.helpers.english_list [11:45] wgrant: thanks, that's the one I was thinking of—I thought it was TAL so no wonder I didn't find it! [12:02] Morning, all. [12:02] hi deryck! [12:03] jam: people are getting eager for that BranchRevision weight-loss program we worked on in Prague. :) [12:06] mrevell: prototype for the translations help-bubble changes at lp:~jtv/launchpad/bug-517700 — playing with the real thing is probably more useful than me describing it in detail. Still some rough edges, I think. I'm EOD, but would appreciate feedback later! [12:06] Thanks jtv, I shall take a look at this next. === danilo_ is now known as danilos === danilos is now known as danilo === danilo is now known as danilos === mrevell is now known as mrevell-lunch [12:36] See you tomorrow, folks! === matsubara-afk is now known as matsubara [13:29] bigjools, do you have a minute to talk about the removal of the security upload policy? === mrevell-lunch is now known as mrevell [13:35] bigjools: ...and another question: ProxiedLibraryFileAlias.http_url ensures that the returned URL does not start with "api.lp.net". The reason seems to be bug 354373, which I don't really understand. I have at present the opposite problem: I _need_ a webservice URL for ProxiedLFAs, see bug 620458. [13:35] <_mup_> Bug #354373: [API] build.build_log_url and build.upload_log_url provide wrong URLs [13:35] <_mup_> Bug #620458: cannot access attachments of private bugs any more [13:39] bigjools: so, I could either write a variant of ProxiedLFA.http_url which does not enforce the usage of IWebBrowserOriginatingRequest. like "default_http_url", or I could change the behaviour of http_url so that the current request is always used and add a property like web_browser_http_url which has the currnet behaviour of http_url. [13:39] but: why is this overriding necessary? [13:50] adeuring: why do you need webservice URLs for librarian files? [13:50] bigjools: so that lplib scripts can access private data [13:51] bigjools: see bug 620458 [13:51] <_mup_> Bug #620458: cannot access attachments of private bugs any more [13:51] ok [13:51] sounds fair enough - I think we overrode it because it was breaking something else though [13:55] bigjools: OK, so, changing the behaviour of ProxiedLFA.http_url, keeping the current behaviour in something like web_brwoser_http_url and using this in the affected code would be OK for you? [13:56] adeuring: I can't think of all the ramifications right now, but as long as you don't have to change any of the soyuz tests to make them work then it sounds fine. I'd check with Gary though to see if he has any thoughts. [13:57] OK, gary_poster: ^^^ [13:58] It's not so bad any more, since api.launchpad.net doesn't require auth. [13:58] However, some API clients will still want webapp URLs, so they can serve up links to private files. [13:58] So we really want both :/ [13:59] wgrant: You can meanwhile access private files via the webservice [14:00] adeuring: Not if I'm serving links to web clients. [14:00] wgrant: ? [14:00] If I use an API client to create a web page, I need to serve webapp URLs, since my users aren't authenticated to the API host. [14:01] wgrant: ah, right! [14:05] bigjools, did you see my ping earlier about removing the security upload policy? [14:05] salgado: yes, sorry, I am dealing with other things before getting around to you [14:05] but there's a lull, so fire away [14:06] gmb, can I get an "amen!" to my changes here: https://help.launchpad.net/Bugs/ImportFormat ? [14:06] bigjools, soyuz-set-of-uploads.txt depends on that policy, and I've tried changing it to use another policy that accepts the same kinds of uploads but it fails and leaves me with no clue as to why [14:07] * bigjools checks [14:08] bigjools, line 326 [14:08] deryck, Amen, brother. [14:08] Looks good. [14:08] salgado: I see [14:08] what's the error? [14:08] excellent. thanks, gmb [14:08] np [14:09] bigjools, Failed upload(s): ['unstable_1.0-1'] instead of the rejected exception [14:10] that's when I use the 'buildd' policy [14:11] salgado: what does the next output say (for read_email()) [14:11] None [14:11] awesome [14:11]