/srv/irclogs.ubuntu.com/2011/12/21/#launchpad-dev.txt

poolieso if we do 100:00
pooliewe'll be able to switch to new versions at any point00:00
pooliethe deployment will be fast00:00
pooliebut there will be an open issue of garbage collection00:01
poolieof two aspects00:01
poolieold trees, and old processes00:01
pooliewith the constraint that the tree shouldn't be deleted until the processes running from it have exited00:01
poolieand the possible complication that some processes may never exit00:01
lifelesswell, I have asked that we have a signal to send those processes telling them to exit after the next verb; other than hostile attacks, they will always exit in that situation00:02
poolieright, but this may take a long time00:02
poolieand it may break some clients00:03
poolieso adding that thing will help with gc of processes00:04
pooliebut it won't make it fast enough that we want to do it within the deployment loop itself00:05
lifelessI know it will break old clients, but folk do upgrade bzr fairly regularly00:06
poolieok00:07
pooliewell, that exists in bzr now00:07
pooliei don't think calling it right away is necessarily a good idea00:07
lifelessso, you want to make it easier to rev the LP bzr service00:08
lifelesswhich is good00:08
lifelessRight now, its nodowntime but manual request00:09
pooliefsvo 'nodowntime'00:09
lifelesswhat do you mean?00:09
poolieit drops connections at present?00:10
poolieanyhow00:11
poolieyes, i want to be able to land changes00:11
poolieso...00:13
lifelessso you can today00:14
lifelessif you want to improve the deploy process, cool. I think you have a better handle on the constraints.00:14
pooliebut what's running will be uncontrolled vs what's in the tree00:14
pooliewhich is undesirable00:15
lifelessI can't parse that00:15
pooliethe changes won't necessarily run until a restart is requested?00:15
poolieso it seems like you're saying this is not actually a blocker to landing new bzr changes?00:16
pooliewe can do a new deployment, which will cause it to start a new version of bzr, but keep running the current frontend?00:17
lifelessa ndt request for codehosting will be slow and a bit manual, but when it completes everything is running the deployed revision00:17
lifelessbecause its slow and manual, its not included in the 'nodowntime' alias.00:17
lifelessbut if you have a change for the service, its a stock standard deploy request away.00:18
poolieok, so there's no reason to think this bug is a blocker to doing other changes?00:19
poolieit's just something that would be nice to do?00:19
lifelessits very important to do as part of getting all LP running the same codebase all the time00:20
lifelessthats an operational efficiency priority00:20
pooliei see it's critical00:23
pooliei guess i'm wondering if it is super-critical, or if it blocks anything else00:24
poolieand also, what would actually have to be done to close it00:24
lifelessIt adds friction to the testing of the forking daemon00:24
lifelessbecause it takes so long to deploy a change (e.g. to stop using it)00:24
lifelessto close it, we need to have deploys be time bounded and not need admin intervention; they don't have to be super fast, just admin free.00:25
pooliethat seems to imply the tree will get automatically deleted hours later?00:28
pooliehm00:29
lifelessYes, when it reaches the front of the gc queue.00:29
poolieis there any kind of technical gc queue at the moment or is that just done by losas?00:30
lifelessits the last stage of the deploy scripr00:30
lifelessI would be happy if codehosting deploys took 2 hours - unsupervised, totally automatic00:31
poolieok, so the deploy script could pause for a long time waiting for the process to complete00:31
poolieas long as it could eg poll something to see when it was done with that tree00:31
lifelessideally shorter, but the key thing is the totally automatic aspect, which we don't have today.00:31
lifelesswe tried to get it with an haproxy timeout, but that broke bzr (per the not-auto-reconnecting thing), and when we set a higher timeout, we found a set of clients that never go idle.00:32
lifelessthose are the CI servers.00:32
lifelesslots of short verbs00:32
poolie..00:33
lifelessthus the two-pronged approach: tell the backend to shutdown as soon as it can, and tell bzr clients to reconnect if the channel goes away between verbs00:33
poolieso perhaps this requires no more code changes, only changes to the deployment scripts?00:33
poolie1- tell haproxy to stop using that ssh server00:33
poolie2- send sighup to the bzr processes00:33
lifelesshas the client side reconnect been SRU'd yet ?00:34
lifelesspoolie: does sighup tell the bzr process to 'exit as soon as idle' ?00:34
poolie3- start a new ssh server and hook it into haproxy00:34
lifelesspoolie: we have a signal for the front end already, thats stops it listening00:34
poolielifeless, i can't SRU it until it has been well tested in trunk00:34
poolieso there is a chicken and egg00:35
lifelesspoolie: so 1) is signal the frontend, and already happens. As does 3 - just adding in step 2 is probably enough, when we're happy enough with the disconnect reliability.00:35
poolieyes, the bzr side of this bug means the server will drop the connection after the current request00:36
poolieok so all we need is a deployment script change to have it sighup the bzr processes?00:36
poolieor, perhaps the ssh server should do that when it is itself told to shut down00:37
lifelessyes, thats where I would put it in. I wouldn't want to do that until we think a majority of users will be running disconnect-ready clients though00:37
lifelesswe could do it on staging00:38
lifelessas a way to ease into breaking the chicken-egg thing00:38
pooliestaging is not actually used00:38
poolieso, this is why i saw a way out of letting the backend continue running for a few hours00:38
pooliethat will let most clients finish their business00:38
lifelessso thats the current status quo00:39
lifelessif we let them finish, we're not actually learning more about the safety of the code00:39
pooliemm00:41
poolieso, i don't know00:41
pooliei don't want to just leave it stuck here00:42
pooliewe can do some manual tests against staging but i am not sure that will really give enough assurance00:42
pooliein some ways i think the current status quo is a better design00:42
poolieto just not disconnect people unless we have to00:42
pooliewhy rely on the reconnect working when you can just avoid the issue00:43
lifelesswell, this is one of the reasons http has a connection:close header00:44
lifelessperhaps we should add the ability to signal that to bzr, to make it more reliable.00:44
lifelessThat said, we *know* there are clients out there that will stay connected for weeks.00:44
poolieright00:45
poolieso they're going to either have to upgrade bzrlib, or cope with some disconnections00:45
lifelessThat is clearly unsustainable, and there is a certain elegance in having one solution.00:45
lifelessyes, we intend to communicate with them once the code is in all supported bzr series, and say 'please upgrade, you can use a point release if you want'00:45
poolieso, again, a chicken and egg00:47
lifelessI have to go, sorry.00:47
poolienp00:47
gary_posterpoolie, hi.  bzr has been doing something a bit weird for me.  I have a LP repo initialized with bzr init-repo --no-trees, and a lightweight checkout of local branches.  For weeks or maybe even months now, (almost?) every time I switch the sandbox to local devel pull from LP's devel I get this: http://pastebin.ubuntu.com/776983/ .  I then do break-lock and update and I'm fine, but it's kind of annoying. This is with00:59
gary_poster bzr 2.4.2. Do you have any ideas or advice?00:59
pooliehi, please file a bug, with a traceback and reproduction instructions, and try 2.501:00
pooliei don't recall an existing bug about that01:00
gary_posterpoolie ack will do.  thanks01:02
gary_poster(will do tomorrow01:02
gary_poster) :-)01:02
gary_posterbye01:02
pooliethanks01:02
lifelesspoolie: back03:30
poolielifeless, i don't want to ruin your vacation03:42
wallyworld_what's the preferred style for multi-line comments? # or '''03:44
pooliefor a comment, definitely #03:45
wallyworld_ok, thanks. makes it less readbale though03:47
wallyworld_i saw that guido approves of ''' :-)03:48
lifelessguido also is quoted as saying 'what was I thinking' (about other things, but shows his approval is no  guarantee...03:58
pooliewallyworld_, i guess you know a ''' is not a comment?04:14
poolieit's a string04:14
wallyworld_poolie: yes04:14
poolieso apart from the specific use of docstrings, putting comments in them feels a bit weird04:14
wallyworld_but it serves the purpose :-)04:14
wallyworld_stub: hi there. if you had a moment, now or later, i'd love to get a 2nd opinion on some sql. i think it will be efficient but i'd like to check04:59
stubwallyworld_: sure05:05
wallyworld_stub: thanks. here's something to look at. i can explain whatever is unclear. https://pastebin.canonical.com/57479/05:05
wallyworld_stub: the code runs in the security adaptor05:06
wgrantOh god.05:07
wallyworld_stub:  the sql is obstinately correct because the unit tests pass (the ones written so far). it's the performance aspects i'm interested in05:07
wallyworld_wgrant: what's wrong?05:08
wgrantThis is crazy :)05:08
wallyworld_what is?05:08
wgrantLet's not put an expensive global bug search into person traversals..05:08
wallyworld_wgrant: it's only used in the api - for views, it's pre-cached05:09
stubI'm tending to agree with that, but looking at the SQL at the moment05:09
wgrantwallyworld_: Or for anything under a person.05:09
wgranteg. a PPA, a branch.05:09
stubWhat is 'user_bug_filter'?05:09
wgrantHaving pillar-wide visibility would render this obsolete.05:09
wgrantWhich means we won't be causing more timeouts.05:09
wallyworld_stub: that's an existing filter implemented in BugTask used to return the bugtasks a user can see05:09
wgrantWhich would be a good thing.05:09
stubwallyworld_: Have an example?05:10
wallyworld_stub: https://pastebin.canonical.com/57480/05:11
wgrantIt probably won't be *that* bad most of the time, because of the lack of other constraints.05:12
wgrantHowever.05:12
wgrantIt is a slippery and unnecessary slope, and will be slow in some situation.05:12
wgrant+s05:12
wallyworld_wgrant: that's the idea, and remember, it's only for when the API is used05:13
wgrantNo.05:13
wgrantIt's for any traversal through a person.05:13
wgrantAnd "only for when the API is used" is particularly bad.05:13
wallyworld_for the common use case, where someone opens a bugtask view, we pre-cache05:13
wgrantBecause this adds probably >1s per request.05:13
wgrantAnd API makes lots of requests.05:13
wallyworld_another common case is when someone navigates to a private team page05:14
wallyworld_ah, but yes, any traversal through a private will trigger it05:14
lifelesswallyworld_: whats this for ?05:15
wallyworld_lifeless: it's in the security adaptor for limitedView permission on private teams05:16
lifelesswallyworld_: ah, person visibility ?05:16
lifelesswallyworld_: cool05:16
lifelessyou don't need the bug its, just exists05:16
lifelessso you can tune this to be more simple still05:16
lifelesswallyworld_: this only runs when the user doesn't have direct access to the team right ? (non-admin, non-member)05:16
wallyworld_lifeless: yes, these checks occur last in the security adaptor05:17
wallyworld_the least expensive ones are run first05:17
lifelesscool05:17
lifelessand \o/05:17
lifelesswgrant: this doesn't add 1s per request, it will add (amount to be checked) per request *by someone that cannot see the team today*05:18
wallyworld_lifeless: this will allow someone to click through a private team in the bug subscription portlet and see the limited details of the team for example05:18
wgrantAnd it scales for every single reference ever.05:18
wgrantIt's silly and unnecessary.05:18
wgrantOvercomplicated, confusing, and slow.05:18
lifelesswgrant: linear probing increase on the query planner, stats-based linear search of log-time checks from there05:18
wgrantlifeless: + 100 other queries for the other person foreign keys.05:19
stubThat clause is certainly a killer. Materializing a list of all public bug ids is a sequential scan of the bug table05:19
wallyworld_it's not confusing to me, or complicated. slow perhaps, which is why i've asked for advice05:19
wgrantwallyworld_: The query isn't.05:19
wgrantThe concept is.05:19
wallyworld_not to me05:19
wallyworld_it's what we are basing social private teams on05:19
lifelesswgrant: not all fk's will lead to disclosure (libraryfilealias owner for instance, or message owner)05:20
wallyworld_stub: you mean the bug filter clause i pasted second05:20
wallyworld_?05:20
wgrantlifeless: No, but most have to, in the proposed model.05:20
lifelessmany will yes05:20
wgrantSo, we are going to be in a nice situation where you can see a private team when it's linked to something, except when you can't.05:21
lifelesswe may want to build a fast data structure for it in time, but this is a reasonable approximation, especially given that its an uncommon-path anyway.05:21
wgrantAnd it's impossible to say who can see a private team.05:21
stubwallyworld_: Yes. The SQL you have will certainly need optimizing for use in a security adapter. Technically, it could be called an unlimited number of times on a page (once per private team)05:21
wgrantIt's exactly the opposite of the disclosure project.05:21
wallyworld_lifeless: wgrant: but isn't the issue here that even if a private team is on a fk, the logged in user may not be able to see that object, so the team's existent should not be shown in that case05:21
stubRealistically, once or maybe twice05:21
wgrantIt's making disclosure of private teams entirely opaque.05:22
lifelesswgrant: its making it *possible* to use them sensibly05:22
wallyworld_stub: what we do now, for is pre-cache the permissions for pages which require this, so the security adaptor is not called05:22
wgrantlifeless: There are simpler ways.05:22
lifelesswgrant: such as?05:22
wgrantlifeless: Pillar-wide scope.05:22
wgrantTo subscribe a private team to a bug, it has to be able to be disclosed to that project.05:23
wgrantSo visibility is simply based on whether you can see a pillar in which the private team is involved.05:23
wgrantIt makes it easy to see what is visible where.05:23
wgrantAnd it makes it difficult to make mistkaes.05:23
wallyworld_stub: for example, on the bug page, the subscribers portlet which may have many private teams listed, all the permissions are cached05:24
wgrantAnd it is cheap to query, and very understandable.05:24
wgrant(it doesn't make sense until we have private projects, however)05:24
lifelessand it doesn't address things like ayatana05:24
wgrantOh?05:25
lifelessprivate list of folk working on the project, public bugs, do public triage.05:25
lifelessas a for instance.05:25
wgrantThe team membership is still private.05:25
lifelessright05:25
wgrantVisible only to members/admins/etc.05:25
wallyworld_does our model even support the simple determination of which pillars a private team is involved with yet? we would need something like the TeamParticipation table, no?05:25
wgrantBut the existence of the team is not.05:25
wgrantLimitedView is existence, not membership.05:25
lifelessbut hten the next team over, which is totally private, but has access to private ayatana things, what is their safeguard for disclosure ?05:26
wgrantI posit that LimitedView should be granted pillar-wide.05:26
wgrantHuh?05:26
wgrantConfused.05:26
lifelesslets talk in budapest, what wallyworld_ is doing is what we agreed made sense on the call a week or two back05:26
wgrantMy plan was to talk in Budapest; I didn't expect this particular case to arise so quickly.05:26
lifelessor we can take this offline now and beat it to death, but I don't want my holiday to die a death of a thousand cuts05:26
stubWhat we have is surprisingly quick in any case for the example personid (243656)05:27
wallyworld_stub: is person 243656 some random person?05:28
stubhttp://paste.ubuntu.com/777089/05:28
stubwallyworld_: It was the id from your second pastebin :)05:28
wgrant243656 is a dev person05:28
wallyworld_stub: ah, ok. that was from a unit test05:28
lifelessstub: try with me or you05:29
stublets try it with a user that is a member of some teams then05:29
wallyworld_or a user that can see a bug that a team is subscribed to but is not a member of the team05:30
stub1.2 seconds for me05:30
lifeless7.5 seconds for me05:30
stubSo lets see if we can optimize this.05:30
wallyworld_stub: lifeless also mentioned the bug_id bits could be optimised out05:30
stubyer05:30
lifelesshttp://paste.ubuntu.com/777092/05:31
lifelessoh, I fail ad adjusting05:31
lifelesswallyworld_: hang on, there are two persons involved05:32
wallyworld_yes, the private team we are checking the visbility of and the person who wants to see05:32
lifelessthere is the person being *viewed* and the person *viewing*05:32
lifelessstubs test only has one id05:32
lifelessthe pathological case is a private team that the person doesn't know about05:32
wallyworld_ah, didn;t notice that05:33
lifelessthe bug visiiblity code is 'which bugs can this person see'05:33
lifelessbut we want 'are there mutually visible bugs'05:33
lifelessI think you'll need to start from ground up05:33
wallyworld_yes, the team being check has it's id in the TEamParticiaption query near the top05:34
lifelessyes, but thats still wrong05:34
lifelessthere are *two* team expansions to do05:34
* lifeless stops to think05:34
wallyworld_lifeless: the query checks directly subscribed to and assignee05:34
wallyworld_i think the query is correct05:34
lifelessthe private team is visible if:05:35
lifeless - it has any interaction with a truely public object05:35
lifeless - there is any interaction with a visible object05:35
lifelessso the one half should be 'users visible objects' the other half 'interactions with objects'05:36
wallyworld_the current implementation is for a subset of such rules, as per the comments in the python in the pastebin05:36
wallyworld_just the things we can do quickly05:36
lifelessand we can short circuit a bunch of that by saying 'mutual ineractions'05:36
lifelesswallyworld_: yes, I saw that05:36
lifelessso the blueprint check is one sided because they are only public05:37
wallyworld_i included that check to hook into the team participation query05:38
lifelessthe select bug_id. .. would probably be better phrased as two separate things05:38
lifelessa) bug subscription or assignee to public bugs05:38
lifelessb) subscription or assignee to a mutually visible private bug05:38
wallyworld_b) it not quite right i don't think05:39
stubBetter https://pastebin.canonical.com/57481/05:40
wallyworld_just because a private team can see a private bug doesn't mean i can see that private team05:40
* wallyworld_ looks05:40
lifelesswallyworld_: I know - there are two conditions: interact * and * mutual visibile05:41
wallyworld_lifeless: not sure i agree with mutually visible05:41
lifelesswallyworld_: so, if the private team interacts with something I cannot see, I cannot see the team, right ?05:42
wallyworld_lifeless: yes05:42
lifelessthats why mutually visible05:42
wallyworld_i think i misunderstood you05:43
lifelessif the private team does not interact with it, I can't see the team.05:43
stubI believe in an exhaustive test suite and the query is right when it passes and is fast enough :)05:43
lifelessAnd if I can't see the bug I can't see the team05:43
lifelessand they cannot interact with something they cannot see05:43
wallyworld_that 2nd point is irrelevant if we query for the things the team inetracts with, no?05:44
wallyworld_since the result set will by definition contains the things they can see05:44
lifelessdepends on wgrants implementation plans for folk that are kicked out of a project05:44
wgrantThat's one bit I hadn't worked out yet.05:45
lifelessanyhow, 'visible to user + interacted with by the team' is sufficient for now05:45
wallyworld_stub: so, if i redo it based on your query, how do we want to ensure it's fast enough in practice?05:45
lifelesswallyworld_: a doubly nested for loop on staging05:45
lifelessfor person in person for team in teams05:45
lifelessstub: is that query rephrased at all, or just new constants plugged in ?05:46
stubit is rephrased somewhat05:46
stubhttps://pastebin.canonical.com/57482/ is even more reworked05:47
wallyworld_lifeless: so you'd prefer this to land behind a feature flag? so we can test on staging and leave the flag off if we are not happy?05:47
stubJust mild tweaks though05:47
lifelesswallyworld_: always ;)05:47
wallyworld_ok05:47
lifelesswallyworld_: safety nets make radical change easier05:47
lifelesswallyworld_: this is radical05:47
stubSo where we want to know 'EXISTS', don't generate more results than absolutely necessary05:48
wallyworld_stub: so if i land this, and it's up on qas, can you help me test it for performance before fflag is activated?05:48
stubpg will shortcut a simple query in an EXISTS, but I don't know if it is smart enough for a UNION query so I optimized those bits. Also, UNION ALL will be faster than UNION, again PG might have optimized that for us but better safe than sorry05:49
lifelessstub: cool, I was just checking with me an dwanted to know whether to fiddle with c&p from your query05:49
stubwallyworld_: If it is up on qas, you don't need me - you can get query times for a page request directly05:49
lifelesswallyworld_: I'm pretty sure you will want to unwrap the user_bug_visibility filter and partition the checks05:49
lifelesswallyworld_: IMBW but splitting on unbalanced data often makes pg happier05:50
wallyworld_lifeless: not sure i understand what you mean by partition the checks05:50
stubMy updated query will involve changing the user_bug_visibility in any case - changing the single EXISTS to two EXISTS05:51
stuboic05:51
wallyworld_stub: with the testing question, i will and can do my own tests. lifeless mentioned the use of a double for loop, hence my question about additional tests05:51
lifelesswallyworld_: rather than checking 'all interacted with' against 'all visible'05:51
stubhe means 'WHERE Bug.id IN (SELECT id FROM Bug WHERE private IS FALSE) OR Bug.id IN (SELECT id FROM Bug WHERE private IS TRUE AND (...)'05:52
lifelesswallyworld_: do two checks with - 'interacted with public' || 'interacted with private-visible'05:52
wallyworld_ah, ok05:52
lifelessnote that one half of that is dramatically cheaper05:52
wallyworld_and it may short circuit the check?05:52
lifelessdon't need to know what the viewer did if there are any public interactions05:52
wallyworld_sorry if i've stirred up a hornet's nest with this. i really appreciate your and stub's help05:53
lifelessmay want a CTE of the bug subscriptions, may not.05:56
wallyworld_CTE?05:56
lifelesscommon table expression - the WITH stuff05:57
wallyworld_ah, ok05:57
huwshimihmmm... make run is failing with "mkdir: cannot create directory `/var/tmp/bazaar.launchpad.dev/mirrors': Permission denied"06:04
lifelessanyhow, stub has a fast query now.06:04
stubit still needs work06:04
lifelessremember that disclosure brings in a new model for 'can view'06:04
stubWorth getting it as fast as possible as there will be outliers that cause grief06:05
lifelessso there are diminishing returns in optimising this today: a) its used in the may-fail-anyway case, and b) the underpinnings are going to be reworked soon06:05
lifelessstub: sure, not meaning you shoul stop06:05
lifelessjust be aware that we'll need to do it all again in 3 weeks or so :)06:05
stubyer06:06
stubjust looking at my last query plan on how much was never executed - short circuits are great, but they won't always happen06:06
wallyworld_huwshimi: sudo remove that dir should fix it i think06:08
huwshimiwallyworld_: Thanks, I'll give it a go06:08
huwshimiwallyworld_: It doesn't exist06:09
wallyworld_lifeless: stub: doing this now fixes the current 404 on prod when users try and click through to private teams, plus picks up the work curtis just did this week on the page templating to take account of limitedView permissions06:09
huwshimiwallyworld_: Oh, the parent does, I'll remove that06:09
wallyworld_huwshimi: remove the parent06:09
wallyworld_beat me to it :-)06:09
stubI haven't been following the frequent and lengthy discussions so am trusting a sane rationale for all this was produced :)06:10
huwshimiwallyworld_: Perfect, thankyou06:10
wallyworld_huwshimi: np. it's apache that writes it out as root i think06:10
huwshimisilly apache06:10
wallyworld_stub: lifeless and curtis agree on it all. it's a step along the way to a better solution and a fairly radical change to allow some private team details to be known. stakeholders seem pleased that we are doing it06:12
stubwallyworld_: So if a team has subscribed to a public bug, or subscribed to a private bug that has been made public, that team is always visible?06:22
huwshimiA simple CSS review: https://code.launchpad.net/~huwshimi/launchpad/tag-link-colour-907132/+merge/8649906:25
wallyworld_stub: yes. if a user can see a bug that a team is subscribed to (it may be a private bug), that team has limitedView visibility for the user06:27
wgrantNote that LimitedView doesn't provide access to the member listing.06:28
wallyworld_just the name, displayname etc06:28
stubYes, and if that is a public bug then it is visible to everybody06:28
wallyworld_stub: yes06:28
wallyworld_since the team has placed itself in a public role06:28
stubOr someone twiddled the privacy setting on a bug they are subscribed to :)06:29
stubNot necessarily a member of the team06:29
wallyworld_stub: and we are implementing code to warn the user who is placing the team in such a role when this happens06:29
wgrantExactly.06:29
wgrantThis is why it gets very messy.06:29
wgrantAnyone can disclose the team.06:29
wallyworld_huwshimi: with the new screenshot, it looks like "None" is lighter than the other tags?06:31
huwshimiwallyworld_: Yeah, it's not a link06:31
wallyworld_huwshimi: my (perhaps invalid) opinion is that the label should be different from the text content06:32
wallyworld_colour06:32
wallyworld_sure, links need to be different as well06:33
huwshimiwallyworld_: I think that's valid, but as you say the links should be different06:33
huwshimiwallyworld_: I guess this branch is just about link colours06:33
wallyworld_yes. and i don;t like the grey personally06:33
wgrantShould the tag links perhaps look like tags?06:33
wgrantlike status/importance do06:33
wallyworld_grey to me means disabled06:34
stubwallyworld_: While you are there, can you quickly twiddle the code that generates https://pastebin.canonical.com/57480/ to say 'IS FALSE' rather than '= FALSE' to give us maximum chance of being optimized.06:34
huwshimiwgrant: I think we should do something with our tags, but if we do we need to do it everywhere06:34
wallyworld_stub: sure06:34
stubwallyworld_: And the UNION to a UNION ALL06:35
wgranthuwshimi: I agree, but we changed status and importance in only one place...06:35
wallyworld_stub: this will be in the existing bug filter code. so i'll fiddle the sql and let the existing tests validate correctness06:35
stubwallyworld_: Yes. Doing my suggestion will be a win for everything.06:36
stubwallyworld_: I'd like to change it to two EXISTS rather than one using UNION ALL, but that might have fallout.06:36
wallyworld_stub: you mean performance regressions?06:37
wallyworld_for existing bug queries06:37
stubwallyworld_: The changes I suggested will not cause regressions. The change I'd like to do might, so I won't get you to make a quick hack right now :)06:37
stub(just do the IS FALSE and UNION ALL)06:37
huwshimiwgrant: Yeah, we need to fix that06:37
wgranthuwshimi: They look all old and crap everywhere else :(06:39
wallyworld_stub: ok. i'll also split the other query as suggested earlier, and add you as a reviewer for the mp to check it?06:39
stubSure06:39
wallyworld_thanks :-) i may not get it done tonight, have other stuff on06:39
wallyworld_gotta finish cobverting the existing doc test to unit tests06:40
wallyworld_huwshimi: so were you going to dick with the colours a bit?06:40
huwshimiwallyworld_: Well there are outstanding issues that will affect what other changes would be made. E.g. bug 89473406:42
_mup_Bug #894734: "Assignee: None" and "Tags: None" clutter bug listings <bug-columns> <Launchpad itself:Triaged> < https://launchpad.net/bugs/894734 >06:42
huwshimiwallyworld_: And all the other bugs about content not being linked06:43
wallyworld_huwshimi: and you are sure about the yucky grey?06:43
huwshimiwallyworld_: Well we could look at having a saturated version of the blue06:44
wallyworld_huwshimi: my opinion is that that would be better, since grey to me signifies disabled, and is used elsewhere in lp to mean that (eg bug abd branch auto linking in text content)06:44
stubwallyworld_: https://pastebin.canonical.com/57483/ is what I end up with anyway. I could split up the bug check into 'subscribed to public bugs', 'subscribed to visible bugs' etc. but that would involve bypassing user_bug_filter. I thought this would be bad, as that stanza will be evolving as bug privacy evolves.06:45
wallyworld_stub: yes, that's why i wanted to use that method to get the user bug filter06:46
wallyworld_stub: not looking too closely, that query took 0.5, the last one took 0.3.06:46
huwshimiwallyworld_: I'm about to finish up for the day (and wander over the road for the Big Bash), but I'll have a play and probably update things tomorrow.06:47
stubwallyworld_: That was using incorrect parameters I think06:47
stuboh no06:48
stubyer - https://pastebin.canonical.com/57482/ does seem quicker, both in the cost estimate and the actual query time06:49
wallyworld_huwshimi: ok. i wnt to the big bash last night. saw warnie and liz. bloody excellent. have fun06:49
wallyworld_stub: ok. i'll go with the quicker one for now. i have to run up and make dinner - my wife came home from a procedure at hospital today. but i'll check irc again after dinner06:50
wallyworld_thanks so much for your help06:50
huwshimiwallyworld_: Yeah, should be good06:51
stubwallyworld_: np. I'm done anyway - need to go play with replication06:51
jtv1Any reviewers in the house?  https://code.launchpad.net/~jtv/launchpad/lint-lpserve/+merge/8649808:58
stubjtv1: done09:03
jtv1thanks09:03
=== almaisan-away is now known as al-maisan
=== jtv1 is now known as jtv
gmbDoes anyone have any idea why I'd get his error http://pastebin.ubuntu.com/777225/ whilst running cronscripts/scan_branches.py against a local codehosting instance?10:10
gmbAh, hang on.10:12
gmbBecause I'm an ijit, that's way.10:12
wgrantgmb: was /var/tmp/bazaar.launchpad.dev/mirrors 700 or so?10:13
wgrantThat's been happening for me recently.10:13
gmbwgrant, Hmm, lemme check. I'm using dev.launchpad.net/Running/RemoteAccess and the Apache config for bazaar-internal looks a bit weird...10:15
gmbNo, /mirrors is rw-rw-r--10:15
wgrant/var/tmp/bazaar.launchpad.dev is sensible too?10:16
wgrantApache likes to create them as root with a restricted umask :/10:16
gmbYep, that's sane.10:17
wgrantHuh10:17
wgrantI guess the apache error log is the next step.10:17
gmbYep.10:17
gmbHaha10:19
gmb[client 172.16.15.154] client denied by server configuration: /var/tmp/bazaar.launchpad.dev/mirrors/00/00/00/4d/.bzr/branch-format10:19
gmbThe clue's in the brackets.10:19
gmbThe RemoteAccess document needs a rewrite :)10:19
wgrantAh10:19
wgrantI keep my container's /etc/hosts pointing at 127.0.0.8810:20
gmbWise.10:20
wgrantOnly the host's gets the real IP.10:20
gmbAs it should be.10:21
gmbI'll update the page.10:21
wgrantGreat.10:22
gmbArgh. Can anyone remember francis's non-netstat way of finding out what's hogging a port?10:50
StevenKfuser?10:50
StevenKfuser <port number>/tcp10:51
StevenKOr lsof10:51
gmblsof!10:51
gmbThat's it.10:51
gmbThanks.10:51
gmbTwisted: I hate you, die in a fire.10:51
StevenKgmb: I tend to reach for fuser first, then netstat, then lsof. But whatever works.10:51
bigjoolsmy third successive day of doing maintenance rotation. FML10:51
StevenKbigjools: I shall think of you while I play Half-Life.10:52
bigjoolsgmb: Twisted is good, except its logging10:52
gmbStevenK, Ah, cool. I went netstat->lsof, but I'll try your ordering next time.10:52
bigjoolsStevenK: you still play Half Life?10:52
StevenKbigjools: I haven't played it before. It hit 75% off on Steam today, so I grabbed all of it.10:53
bigjoolsStevenK: it's *awesome* :)10:54
bigjoolsI wasted my life on it 10+ years ago10:54
StevenKHaha10:54
bigjoolsif you go online with it, prepare to end yours too10:54
StevenKNow for the Quake Collection to hit 75% off. Which, amusingly enough, doesn't include Quake 4.10:55
StevenKbigjools: The first Portal is 50% off, so it's probably only a few quid.10:56
bigjoolsStevenK: cool, I might get it on the PS310:57
stub$3.75 vs ???11:07
StevenKstub: It's $10 normally11:11
stubIt was on special. $10 I think got you both 1+211:11
stubyesterday I think, maybe still is.11:12
stubI'm backlogged about a year with games, so no need to pay full price :)11:12
StevenKHaha11:14
jmlHello, we'd appreciate a review of https://code.launchpad.net/~james-w/launchpad/bpph-binary-file-urls/+merge/8647011:15
* bigjools will look11:18
jmlbigjools: thanks.11:22
jmlgmb: the great thing about Twisted is that it holds your puny hatred in contempt, dwarfed as it is by its own vast loathing.11:24
jmlgmb: try getting that with Django!11:25
bigjoolsjml, james_w: needs fixing, but minor stuff11:31
* gmb wonders if he can fix bug 808930 buy just adding another commit()....11:32
_mup_Bug #808930: Timeout running branch scanner job <escalated> <linaro> <oops> <Launchpad itself:In Progress by gmb> < https://launchpad.net/bugs/808930 >11:32
gmb(The answer is "probably." Whether it's the _right_ fix or not is debatable)11:33
gmb"Buy" just adding another commit? Seriously Graham, _that's_ the way you're spelling it?11:34
wgrantgmb: Howso?11:36
gmbwgrant, Well, first, s/probably/possibly. We're suffering death by SQL (in at least one case) with a big for loop that holds the txn open whilst it adds new revisions.11:37
gmbNow, adding a commit at the end of that loop may obviate the problem.11:38
gmbBut as a solution, it stinks.11:38
wgrantIs this creating new Revisions?11:38
wgrantOr BranchRevisions?11:38
gmbThis is new Revisions.11:38
wgrantIt doesn't do a massive manually constructed INSERT?11:41
gmbNope.11:41
wgrantHuh.11:41
gmbLots of revision = Revision(...)11:41
wgrantHow did that ever work :/11:41
wgrantAwesome!11:41
gmbwgrant, There's a lovely comment to the effect of "this is bad, but, eh, it works."11:41
wgrantSo, this is the issue I thought you'd hit first.11:41
wgrantDidn't think it was quite *that* bad, though.11:41
gmb:)11:42
wgrantThere's already a mass-insertion method for branchrevision.11:42
wgrantAnd binarypackagepublishinghistory11:42
gmbAh, cool.11:42
gmbI shall crib appropriately.11:43
wgrantAnd possibly bugnotificationrecipient and co.11:43
wgrantIt's ugly, but it works.11:43
gmbHey, I'm not pretty, but I do okay.11:43
gmb"Ugly but works" works for me.11:43
wgrantYou may well need a looptuner, since inserting 200000 revisions in a single statement might not be too helpful, but we'll see...11:43
gmbLoopTuner was one of the things I was thinking of.11:43
rick_h__morning11:56
cjwatsonI'll get my pending QA done at some point over the next couple of days, BTW - currently on holiday but should be able to squeeze it in to unblock people12:00
cjwatsonmind you I'm not sure if anyone's planning another ftpmaster deployment this year :-)12:01
nigelbok, I have to admit, githubs "Merge" button is really really nice for code review.12:59
nigelbI know that tarmac does something equaivalent for us, but I'd really really like to see something like that on LP :-)13:00
rick_h__nigelb: :)13:05
rick_h__nigelb: subsume reviewboard!13:05
nigelbheh13:06
rick_h__even though I'm not a django fan13:06
nigelbrick_h__: The other day I probably complained a litle too hard about sqlalchmeny and michael bates replied to me :P13:07
rick_h__beyer?13:07
nigelbbah, beyer13:07
rick_h__nigelb: he's really awesome at being open/communicating13:07
rick_h__and a super genius13:07
rick_h__but cool, he help you out then?13:07
nigelbWell, eventually, it wasn't sqlalchemy that was giving me either13:08
nigelbSomething else threw an sqlalechemy error :)13:08
rick_h__orly?13:08
rick_h__now I'm curious, what was throwing an SA exception at you?13:08
nigelbYeah. Flask-Admin.13:08
rick_h__ah, gotcha13:09
rick_h__yea, I've not used flask other htan hello world to kind of look at it13:09
nigelbI'm coding exclusively in Flask for work.13:09
nigelbIts harder, but its much more flexible than a full framework.13:09
rick_h__yea, I liked what I saw13:09
rick_h__and if I had a small app I'd use it13:10
nigelbMy boss wrote something like login.launchpad.net in Flask, except its an OAuth provider and not OpenID.13:11
rick_h__cool, check out https://github.com/bbangert/velruse sometime13:12
rick_h__another "to check out" sometime item13:12
* nigelb looks13:12
nigelbInteresting!13:13
rick_h__yea, I really love the idea13:13
rick_h__I was pitching it at my last job as a way to centralize auth through our apps13:13
nigelbwe have a centralized auth through our app. And have a client lib for our apps :-)13:15
rick_h__cool13:16
rick_h__yea, we had a ton of small little apps and used ldap internally, but for external users thought it might make sense.13:16
rick_h__we'd have needed to find a way to hack an ldap provider into there somehow, but would have been cool13:16
nigelbthat's the one thing I like about almost all Canonical apps. Everything uses openID.13:17
rick_h__right13:17
rick_h__took a little bit, but thank goodness13:17
StevenKnigelb: How much data?13:25
nigelbStevenK: Every single person row.13:25
StevenKArgh13:25
nigelb:D13:25
StevenKYes, write a garbo job.13:25
StevenKnigelb: lib/lp/scripts/garbo.py. Create a new class with a descriptive name, copy method *names* from the other classes.13:26
StevenKnigelb: What the methods return should be self-explanatory. If you want more help from me, you'll have to wait until it's not 0030.13:27
nigelbStevenK: Well, I originally, wanted to poke you tomorrow and work along asking you questions.13:27
StevenKI suppose you can do that.13:28
nigelbI'll make some time before I start work to finish it off.13:28
nigelbI haven't worked on LP in a while :)13:28
james_wbigjools, how do I know want to compare the list of urls to?14:00
bigjoolsjames_w: the stuff you created in the factory15:21
bigjoolsmake a list out of it and compare to the listified results15:22
james_wbigjools, do the same url generation against them?15:22
bigjoolsjames_w: well I'd construct an expected URL15:22
james_wbigjools, using ProxiedLibraryFileAlias?15:23
bigjoolsyup15:23
james_wok15:24
bigjoolsjames_w: or even just file.htt-purl15:24
bigjoolserr15:24
bigjoolsfile.http_url15:24
james_wexcept that won't match as it's not proxied?15:24
bigjoolsthere's nothing to proxy15:24
bigjoolsunless it's private15:24
james_wbigjools,15:34
james_wAssertionError: !=:15:34
james_wreference = ['http://localhost:41286/93/filename-100190']15:34
james_wactual = [u'http://launchpad.dev/~person-name-100146/+archive/unique-from-factory-py-line2685-100148/+files/filename-100190']15:34
james_w: ['http://localhost:41286/93/filename-100190'] != [u'http://launchpad.dev/~person-name-100146/+archive/unique-from-factory-py-line2685-100148/+files/filename-100190']15:34
james_wso it looks like they are proxied?15:35
james_warchive = self.factory.makeArchive(private=False)15:35
bigjoolsjames_w: meh, my bad then.15:36
bigjoolsthe redirect happens in the appserver15:37
rick_h__deryck: can you review this please? https://code.launchpad.net/~rharding/launchpad/sort_buttons_907376/+merge/8657916:06
james_wbigjools, https://code.launchpad.net/~james-w/launchpad/bpph-binary-file-urls/+merge/8647016:09
bigjoolschecking16:11
bigjoolsjames_w: perfect, thanks. Do you need me to land it? I can't recall if you can do that16:12
james_wbigjools, I had big problems doing so last time, so if you could throw it at ec2 it would be much appreciated16:13
bigjoolssure thing16:13
bigjoolsjames_w: can you set a commit message16:13
james_wthanks16:13
james_woh yeah16:13
james_wdone16:14
=== deryck is now known as deryck[lunch]
=== al-maisan is now known as almaisan-away
=== deryck[lunch] is now known as deryck
rick_h__deryck: ping, have a sec to peek at https://code.launchpad.net/~rharding/launchpad/sort_buttons_907376/+merge/8657917:54
deryckrick_h__, sure17:56
rick_h__deryck: ty17:56
deryckrick_h__, so it feels like the kind of thing we should test since we keep breaking because of it.18:02
deryckrick_h__, but it seems hard to test, too.18:02
rick_h__deryck: right, because you have to find/replace all cases whereever you're catching the history event18:02
rick_h__the only *good* way would be to wrap history event access entirely with our own api18:02
rick_h__deryck: I searched before, but missed this case because it was in the .pt and not in the .js18:03
rick_h__deryck: I'd definitely prefer to have this stuff in a .js file and the .pt only do a single "setup()" kind of call, but didn't want to change this that much18:03
deryckrick_h__, yeah.  so let's do this change as it is now.  and can you file a high bug about needing to wrap history handling to prevent this happening again?18:03
rick_h__deryck: sure thing18:03
deryckrick_h__, awsome, thanks!18:04
deryckgary_poster, so about deploy…. I can hold for a bit to see if you guys can qa benji's work, if that's cool.19:46
deryckgary_poster, or were you thinking this is something to wait for tomorrow on?19:47
gary_posterbenji, see ops:-/19:47
gary_posterI mean deryck, see ops :-/19:47
deryckgary_poster, gotchas, I see now.  thanks.19:47
cjwatsonDoes anyone mind if I set up /srv/launchpad.net/ubuntu-archive/gnupg-home/ on mawson so that I can QA https://code.launchpad.net/~cjwatson/launchpad/sign-installer/+merge/85670 ?  Much like the setup on production only with a dogfood key21:25
cjwatsony'know what, I'm pretty certain nobody cares so I'll JFDI and we can remove it later if it breaks somebody21:27
* cjwatson wonders what "The calling script may set GNUPGHOME ..." is supposed to mean in cronscripts/publishing/distro-parts/ubuntu/publish-distro.d/10-sign-releases given that it proceeds to unconditionally override that variable21:30
cjwatsonmaybe the right answer is ln -s $HOME/.gnupg /srv/launchpad.net/ubuntu-archive/gnupg-home21:32
cjwatsonany objections to upgrading mawson to >= r14564?21:34
cjwatsonOK, nobody else logged in, so upgrading mawson21:54
pooliehi all21:57

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!