/srv/irclogs.ubuntu.com/2011/11/08/#launchpad-dev.txt

huwshimilifeless: Morning00:02
huwshimilifeless: I started to take a look at the caching00:02
huwshimilifeless: Is this what you meant? https://code.launchpad.net/~huwshimi/launchpad/avatars-everywhere-712894/+merge/8143000:02
huwshimi(I've pushed changes which will be in the diff)00:03
huwshimilifeless: Is there more that I need to do?00:03
huwshimilifeless: You'll have to bear with me, as I really don't know what I'm doing here :)00:05
lifelessheh, sure00:07
lifelesshuwshimi: thats pretty close00:07
lifelessI'll give a longer answer in the review00:08
huwshimilifeless: Thanks :)00:08
lifelessdone, -> optometrist, bbiab00:13
wgrant%#$@00:54
wgrantdie, postgres, die00:54
lifeless?00:54
StevenKWhat about it?00:54
wgrantThe planner is making stupid decisions.00:55
* wgrant pastes.00:55
wgrantlifeless: http://paste.ubuntu.com/731544/00:57
wgrantFirst plan, for a single policy.00:57
wgrantVery sensible.00:58
wgrantStraight double nested loop down three indices.00:58
wgrantSecond one, also a single policy but not specifying the ID directly (this is how I initially tried it): terrible terrible plan doing a manual person sort afterwards. It should have known it would get at most one row, but perhaps the policy ID I hardcoded is in the stats so it gets special treatment.00:59
wgrantThird one, hardcoding the three IDs.00:59
wgrantEven worse plan.00:59
wgrant(these are Ubuntu's three access policies, on real data)00:59
lifelesswgrant: a request for these pastes in future00:59
lifelesswgrant: please wrap the queries.00:59
wgrantlifeless: I normally do, but was too grumpy this time.01:00
wgrantI tried a CTE, but it doesn't work. Even if it returns just the one row.01:00
wgrantI guess because it plans beforehand.01:00
wgrantIt's probably seeing that the 97822 policy is special and planning accordingly, I guess.01:01
lifelessso id=97822 == policy=1 ?01:02
wgrantBut if it knows it's going to dominate, then why does it ignore it when I specify 3 IDs...01:02
wgrantAh, yes, sorry.01:02
wgrant97822 == distro 1, policy 101:02
wgrantWhich is Ubuntu private non-security.01:02
lifelessis policy an enum ?01:02
wgrantYes.01:02
wgrantIt may be better to explode it here.01:03
wgrantBut only because the planner is being obtuse.01:03
wgrantIt's perfectly feasible to do the same simple nested loop down three indices, just with an ANY on the apg filter.01:03
lifelesslook at the estimates01:03
lifelessits estimating 2, then 4501:03
lifeless2nd plan01:04
wgrantYeah.01:04
lifelessfirst one its estimating 57K rows - and considering 1.2M01:04
lifelessbut the actual execution is fast01:04
lifelessthis is on df ?01:05
wgrantYes.01:05
wgrantI have the queries to set up the data in a minute or two, if you want.01:05
lifelessthe second plan has a lower cost estimate01:06
lifelessand the third lowest of all01:06
wgrantYeah, because the row count estimates are off.01:06
lifelessso the interesting question is why01:06
lifelesshave you analyzed ?01:06
wgrantBy a factor of 3.01:06
wgrantEr, 3 orders of magnitude.01:06
wgrantYes.01:06
wgrantVACUUM ANALYZE, ANALYZE, etc.01:06
wgrantSeveral times.01:06
wgrantNo change after the initial one.01:06
lifelessscale 1000 ?01:06
lifelessor the 100 df is running ?01:06
wgrant10001:06
lifelesscare to try01:07
wgrantI guess.01:07
wgrantWorth a try.01:07
lifelessits a thing to rule out01:07
lifelessso the first one is potentially a bad plan too01:09
lifelessbut we expect the data sizes to stay modest01:09
lifelesswhat does it do if you say offset 500 to it ?01:09
lifeless[curiosity]01:09
wgrantI'm not sure that's important, but sure.01:09
wgrant680ms, so it's pretty linear.01:10
wgrantAs expected from the plan.01:10
wgrantDoing a sort-key-based offset is very fast, also as expected.01:12
wgrantThat plan is perfect for the main disclosure management view.01:12
wgrant(unless the set of observers is small)01:13
wgrantperson_sorting_idx is only 57MB, and presumably very hot, so it may even be fast enough when there aren't many.01:15
wgrantI suspect the first batch is actually slower than the rest.01:16
wgrantBecause it's going to have to walk through all the usernames that start with digits.01:17
wgrantRoughly none of which are going to be relevant, because sensible people don't generally have usernames that start with digits.01:17
StevenKUPDATE person set name = '8wgrant' where name = 'wgrant'; commit;01:17
wgrantI actually mean displayname.01:18
wgrantBut yes.01:18
lifelessrofl01:31
lifelesshttps://github.com/juuso/BozoCrack01:31
wgrantHeh01:35
wgrantIt is a useful strategy sometimes.01:35
wgrantlifeless: So, any query suggestions?01:40
lifelesssorry was sidetracked01:41
lifelessuhm01:41
lifeless(distribution, policy) is unique01:41
lifelessbut id is what shows up on the other tables01:42
wgrantYes.01:42
lifelesswhere are your data generating scripts?01:42
lifeless(and do they work on temp tables ;))01:42
wgrantThey are temp tables, yes.01:44
lifelesswgrant: try this http://paste.ubuntu.com/731583/01:44
wgrantJust dropped the fks and it all works.01:44
lifelessbah01:44
lifelessnot limit 001:44
lifelessoffset 001:44
wgranthttp://paste.ubuntu.com/731585/ <- forgive the awful code to create artifact-specific permissions. There's probably a better way to do it, but everything else I tried was unbelievably slow on DF.01:45
wgrantlifeless: Erm, explicit offset 0?01:50
wgrantIsn't that redundant?01:50
wgrant(also, that's still 3.5s)01:50
lifelessoptimisation barrier01:51
wgrantI think that's the problem.01:51
wgrantI suspect it chooses the index-scan-only plan because the ID it's looking for dominates the stats.01:52
wgrantExcept not.01:52
lifelesswhat do you mean by 'dropped the fks and it all works'02:06
wgrantThe original table definitions had foreign keys to person/bug/product/etc.02:06
wgrantBut temp tables can't have fks to permanent tables.02:06
wgrantSo I dropped the constraints.02:06
lifelessoh right, of course ;)02:06
lifelessI presumed that (bugsummary experience most recently ;P)02:07
lifelesshah02:08
lifelessTime: 27666.227 ms02:08
wgrantOh?02:09
lifelessfirst run of second plan02:09
wgrantAh02:09
lifeless SELECT DISTINCT ON (person_sort_key(person.displayname, person.name)) person.id FROM accesspolicygrant JOIN accesspolicyuse ON accesspolicyuse.id = accesspolicygrant.policy JOIN person ON person.id = accesspolicygrant.person WHERE accesspolicyuse.distribution = 1 AND accesspolicyuse.id=97822 ORDER BY person_sort_key(person.displayname, person.name) LIMIT 75;02:10
lifeless id02:10
lifeless----02:10
lifeless(0 rows)02:10
wgrantYour IDs will be different :)02:10
lifelessah, doh02:10
wgrantSELECT id, policy FROM accesspolicyuse WHERE distribution=1;02:11
lifelesswe'll need to iterate on abels code02:11
lifelessto paginate on a function sort02:11
wgrantYes.02:12
wgrantI had already looked at that a little.02:12
lifelessdid you look at nuking the functional index?02:12
wgrantJust using eg. person.name for these queries? Yes, didn't help.02:13
huwshimiwgrant: Is this what you meant for fixing that XSS problem? http://paste.ubuntu.com/731598/02:14
wgranthuwshimi: !!! someone did it right the first time. Yes, that's what I was meaning.02:15
huwshimiwgrant: Hah! It might look right, but it provides a mangled output02:15
wgrantOh?02:16
wgrantAh02:16
wgrantYou may need to return .escapedtext, depending on how you're using it.02:17
huwshimiwgrant: It returns this: http://paste.ubuntu.com/731600/02:17
wgrantIf it's in a TALES formatter, it probably will need .escapedtext.02:17
huwshimiwgrant: Ah02:17
wgrantYeah.02:17
wgrantSorry, forgot to mention that.02:17
huwshimiwgrant: That's fine :)02:18
huwshimiwgrant: Perfect, working now :)02:19
wgrantGreat!02:19
lifelesswgrant: .name wouldn';t help here, but for abels work...02:21
lifeless registrant                         |          1 |         4 |          002:22
lifelesswe might want to drop that02:22
wgrant?02:22
lifelesstotally unused colum on person02:23
wgrants/drop/fix/02:23
lifelessalso language02:23
lifeless'meh'02:24
lifelesswgrant: can you paste me02:28
lifelessSELECT attname, most_common_vals, most_common_freqs FROM pg_stats WHERE tablename='accesspolicyuse';02:28
lifelessactually, nvm02:29
wgranthttp://paste.ubuntu.com/731614/ anyway.02:29
lifelessthats very interesting02:31
lifelessI wonder why distribution is blank02:31
wgrantDistributions are extremely rare.02:31
wgrant40 vs 3300002:31
lifeless SELECT attname, most_common_vals, most_common_freqs FROM pg_stats WHERE tablename='accesspolicyuse' and attname='distribution';02:32
lifeless   attname    |                                         most_common_vals                                          |                                                                                                                                                                                                           most_common_freqs02:32
lifeless--------------+---------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------02:32
lifeless distribution | {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35} | {3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,3.05157e-05,302:32
wgrantEven a stats target of 1000 might not catch a distribution.02:32
lifeless...02:32
lifelesse.g. its not blank for me02:32
wgrantYou probably still have a huge target.02:32
* wgrant tries.02:32
lifeless105 apu entries with no product02:33
lifeless98K where it is02:33
lifeless0.1%02:33
lifeless1-0.001^1000 chance of not seeing one02:33
lifelessIIRC02:34
wgrantWTF02:34
wgrantNow product is empty.02:34
wgrantdistribution has lots.02:34
lifeless1000 is the stats it keeps, not the rows it consults02:36
lifelessto determine most common it has to consult more rows than it keeps02:36
wgrantOh.02:36
wgrantStill doesn't explain it.02:36
lifelesswe're looking at the rows it keeps, and if that was random over the population we could estimate by frequency02:36
lifelessbut this is explicitly the most common values per attribute02:37
lifelessso yeah, I dunno02:37
wgrantAnd on APU it's 3 tuples for each non-NULL value.02:37
lifelesstell me something, why does your script generate 98K policy uses02:39
lifelessis that because all products get security ?02:39
wgrantIt currently just creates public, private, security for every pillar.02:40
wgrantUntil we have a sensible way to manage comment editing, we probably have to allow private bugs for all projects.02:40
wgrant        4 / 1346  RootObject:+login02:44
lifeless  EXPLAIN ANALYZE SELECT person.id FROM person WHERE id IN (SELECT person FROM accesspolicygrant where policy IN (SELECT id from accesspolicyuse where distribution=1 and policy=1)) ORDER BY person_sort_key(person.displayname, person.name) LIMIT 75;02:47
lifeless780ms02:47
lifelesswgrant: I think your first plan is a fluke the more I look at it02:50
wgrantOh?02:50
lifelesswgrant: the indices are not in the same order02:50
wgrantIt may be a fluke, but the others are crap :)02:50
lifelessone is person sort02:50
lifelessthe other is person02:51
wgrantYes.02:51
lifelessit can't dual iterate02:51
wgrantNo.02:51
lifelessit can?02:52
wgrantBut it happens that (in this case) the coverage is good enough that it's not a problem.02:52
wgrantIt can't, AFAIK.02:52
lifelessdouble negs :)02:52
lifelessso02:52
lifelessI think a better question isn't 'why are the other plans slow'02:52
lifelessits 'what do we want to actually query'02:52
lifelessthus the query above02:53
lifelessor02:53
lifelessa less extreme02:54
lifelessEXPLAIN ANALYZE SELECT person.id FROM person WHERE id IN (SELECT person FROM accesspolicygrant, accesspolicyuse WHERE accesspolicygrant.policy=accesspolicyuse.id AND  distribution=1 and accesspolicyuse.policy=1) ORDER BY person_sort_key(person.displayname, person.name) LIMIT 75;02:54
wgrantWhich is 1180ms02:54
lifelessinterestingly02:57
lifelessyou realise that you are querying outside of both your partial indices02:57
wgrantYes.02:58
wgrantI added a new non-partial one.02:58
lifelessis NULL -> 4ms02:58
wgrantWhich would possibly be better done by making another one non-partial, but perhaps not.02:58
lifelessis not NULL -> 2000ms02:58
wgrantOn what?02:58
lifeless EXPLAIN ANALYZE SELECT DISTINCT ON (person_sort_key(person.displayname, person.name)) person.id FROM accesspolicygrant JOIN accesspolicyuse ON accesspolicyuse.id = accesspolicygrant.policy JOIN person ON person.id = accesspolicygrant.person WHERE accesspolicyuse.distribution = 1 AND accesspolicyuse.policy = 1 AND artifact is NOT NULL ORDER BY person_sort_key(person.displayname, person.name) LIMIT 75;02:58
wgrantartifact?02:58
wgrantWell, yes.02:59
wgrantThere are far fewer where artifact is NOT NULL02:59
wgrantThat's not so much the indices as the volume of data.02:59
wgrantYour thing now has to sort a total of 1 person.02:59
wgrantWhich is not a significant challenge.02:59
lifelessdifferent plans02:59
lifelessnot unsurprising03:00
lifelessok so to rephrase the problem03:01
lifeless'there are 57K people with explicit access within Ubuntu and we want to sort them by fields only on Person'03:01
wgrantYes.03:01
lifelessby definition we need to either read *all people* in sort order filtering against the set of grants - which will for random names mean we read ~ all people03:02
lifelessor we need to read all grants (57K) and then cherry pick people03:02
wgrantYes.03:02
lifelesswhich is much more selective and what its doing03:02
lifelessso why is it slow?03:02
wgrantI don't know. I wouldn't expect materialising 57k rows from such a narrow table to be slow at all.03:03
lifeless               ->  Nested Loop  (cost=3.39..66.16 rows=13 width=25) (actual time=14.369..1910.529 rows=57946 loops=1)03:03
wgrantWhich plan is this?03:03
lifelessEXPLAIN ANALYZE SELECT DISTINCT ON (person_sort_key(person.displayname, person.name)) person.id FROM accesspolicygrant JOIN accesspolicyuse ON accesspolicyuse.id = accesspolicygrant.policy JOIN person ON person.id = accesspolicygrant.person WHERE accesspolicyuse.distribution = 1 AND accesspolicyuse.policy = 1 AND artifact is NOT NULL ORDER BY person_sort_key(person.displayname, person.name) LIMIT 75;03:03
lifeless                     03:03
lifelessbut any of them start to look like this03:03
wgranteparse03:03
lifeless explain analyze SELECT DISTINCT ON (person_sort_key(person.displayname, person.name)) person.id FROM accesspolicygrant JOIN accesspolicyuse ON accesspolicyuse.id = accesspolicygrant.policy JOIN person ON person.id = accesspolicygrant.person WHERE accesspolicyuse.distribution = 1 AND accesspolicyuse.id=98234 ORDER BY person_sort_key(person.displayname, person.name) LIMIT 75;03:04
lifeless                      03:04
lifelessis weird03:04
wgrantWhat about it is weird? The plan?03:04
lifelessyes03:05
wgrantHmm.03:05
wgrantI guess the non-indexed sort of 57k people is not going to be terribly fast, because person_sort_key is PL/Python.03:06
wgrantBut I'm pretty sure it happened with a straight string sort too.03:06
lifelessI'll just reply to your mail then test ;)03:07
lifelessI think materialising the sort column may well help03:07
wgrantDid anything useful come from the discussion this morning?03:07
lifelessyes, we cleared things up03:07
wgrantFSVO03:07
wgrantAnd for now.03:07
StevenKHah03:07
StevenKDoes that mean I can continue with my branch to hide private teams in subteam of?03:08
wgrantANd probably by trebling the scope of the project.03:08
lifelessStevenK: yes, the confusion was that that page shows super and sub03:08
lifelessStevenK: see bug 405277 which really covers the visibility rules03:08
_mup_Bug #405277: Private teams are not able to join other teams (public or private) <disclosure> <lp-registry> <privacy> <teams> <Launchpad itself:Triaged> < https://launchpad.net/bugs/405277 >03:08
StevenKwgrant: If it's a death march until March, what difference does December next year make? :-(03:09
lifelessStevenK: you probably also want to hide proposed memberships from private teams, unless the private team has signed off already.03:09
StevenKI really want to finish this branch that kills strict_component from {check,verify}Upload since it's close to done, but wgrant is ignoring me. :-P03:10
wgrantBah, that could have been said here rather than in PM.03:10
StevenKYou were busy here :-P03:11
wgrantStevenK: I don't understand what you're saying.03:11
StevenKSo, there is two things03:11
StevenKOne test failure is test_checkUpload_without_strict_component. I'm concerned that is a buggy test since strict_component is dead and is possibily relying on behaviour that is removed.03:12
StevenKHm. It's expecting checkUpload() to pass, but it now raises NoRightsForComponent.03:14
StevenKs/raises/returns/03:14
StevenKSince returning exceptions is sexy or something03:14
wgrantIt's no longer a useful or possible test.03:15
wgrantSo it should be removed.03:16
StevenKThe other test failure is from the upload processor, which is not rejecting a package that should be.03:17
StevenKI think it is because checkUpload() is returning an exception and we do not reject because the package is NEW.03:17
wgrantpastebin the failure03:18
wgrantAnd diff03:18
StevenKAssertionError: !=:03:18
StevenKreference = "Not permitted to upload to the RELEASE pocket in a series in the 'CURRENT' state."03:18
StevenKactual = ''03:18
wgrantAh, indeed.03:18
wgrantSo, we need to catch the no privileges exception.03:18
wgrantWell, "catch"03:19
wgrantCheck for it.03:19
StevenKYes.03:19
wgrantIf checkUpload has failed due to a lack of permissions, then do the NEW check.03:19
StevenKThis strikes me as ugly03:19
wgrantWelcome to Soyuz.03:19
StevenKwgrant: Any exception from checkUpload is lack of perms?03:21
wgrantNo.03:21
wgranteg. the pocket error that you quoted03:21
StevenKNoRightsForArchive() == lack of permissions03:21
StevenKNoRightsForComponent() == lack of permissions03:22
wgrantNoRightsForArchive is a lie and should probably die, or be moved into NascentUpload.03:22
wgrantNoRightsForComponent is stupid and too specific.03:22
StevenKCannotUploadToPPA is arguably permissions03:23
StevenKHowever, I agree ArchiveDisabled is not.03:23
wgrantIt's not a matter for agreement.03:23
StevenKSo, how should this code be structured?03:24
StevenKAnd do you still need to see the diff?03:24
wgrantI would probably rework verifyUpload to raise a single exception when there are insufficient upload rights.03:27
wgrantExcept perhaps preserving NoRightsForArchive, but implementing it in a non-buggy fashion.03:27
wgrantNoRightsForComponent has no right to exist.03:27
StevenKIsn't it used for, say, partner?03:28
wgrantPartner doesn't need to behave well.03:29
wgrantAs long as it vaguely works for 6 months.03:29
StevenKHm, partner is a seperate Archive anyway03:30
wgrantIn some ways.03:30
StevenKIt isn't Archive.id 1, is what I mean03:30
StevenKWhich means it has its own ArchivePermission entries03:30
wgrantYes.03:30
wgrantBut that's almost all it means.03:30
StevenKNoRightsForComponent is only checked due to checkArchivePermission(person, component), is that check pointless too?03:32
wgrantThat would be cheating.03:33
StevenKI don't really want to rewrite verifyUpload() in this branch03:35
wgrantIt's only like 10 lines.03:35
StevenKIt's currently 30, so that sounds like an improvment03:35
StevenKI'm tempted to say 'prove it' :-P03:36
lifelesswgrant: ok, the call03:40
lifelesswgrant: so the two bits we were getting stuck/confused on were:03:40
lifeless Person:+index shows sub and super; this was just confusion. super should not be shown. sub should always be shown.03:40
lifelessStevenK: ^03:40
lifelessand bug subscriptions should always be shown too03:40
lifelessno spying!03:41
lifelesswallyworld_: ^03:41
wgrantThat was my understanding.03:41
wallyworld_lifeless: so we don't *ever* want to hide private subscribers?03:42
lifelesssuper should not be shown because the teams a person are in should not be disclosed just because you know the person -> visibility is not granted to non-participants of a Person against their memberships.03:42
lifelesssubteams should always be shown (if you can see the team they are a member of) because the subteam could otherwise give you an invisible project owner. e.g. public project, public owning team, no members visible. But really there is a private team lurking inside the public team.03:43
lifelesswe don't want that-> we show the existence of the private team.03:43
lifelesswallyworld_: the rule is, if you can see the bug, you can see the name, displayname, url, of its subscribers (including structural subs)03:44
wallyworld_ok. but if you click on the link, it will 404?03:44
wgrantlifeless: aaaaaaa03:44
lifelesswallyworld_: there are a few drivers for this. One is the guiding principle that interacting with a public object discloses private objects.03:44
wgrantlifeless: We're changing privileges based on admin status?03:44
lifelesswallyworld_: it will give a 200 and a little details.03:44
lifelesswallyworld_: which will require separate work.03:45
wallyworld_ok,so the normal +index view with a bunch of stuff not rendered03:45
lifelesswallyworld_: the second is that if anyone replies to the bug via mail, and they are subscribed via a private team, they are very likely to disclose the private team anyhow due to the footer we include, which 'Reply' in many modern mail clients includes.03:45
lifelesswallyworld_: yes03:45
wgrantlifeless: How do you propose that Person:+index knows if you have partial visibility to it?03:45
lifelesswgrant: theres a something like 10 clause EXISTS union we need to write.03:46
lifelesswgrant: AFAICT it should be fast.03:46
wgrantI was about to say that that was not an answer.03:46
wallyworld_only 10 clauses? surely we could try for 20 :-P03:47
wgrantWe would have to join bugsubscription->bug->accesspolicyartifact->accesspolicyuse->accesspolicygrant->teamparticipation, and that's just for one part of it.03:47
lifelesswgrant: its not trivial unfeasible.03:48
wgrantIt is, however, trivial silly.03:48
lifelessI have been thinking about this for a year or so03:48
lifelessand I disagree with you :)03:49
wgrantSo.03:49
lifeless903:49
lifelessbah03:49
wgrantSay that one day in future we decide that being a monolithic webapp is actually a stupid idea, and SOA is cool.03:49
lifelesshow about that huh03:49
lifeless\o/03:50
wgrantHow does that work now?03:50
wgrantWe call out to 50 services?03:50
lifelessin parallel, yes03:50
wgrant...03:50
StevenKLike fun that scales03:50
lifeless(and cancel the calls when one answers)03:50
wgrantThis also puts the LP services in a strongly privileged position.03:50
lifelessStevenK: it actually scales very very well03:50
lifelessStevenK: because each service can cache answers to these questions when appropriate03:51
lifelessStevenK: 'can user X see object Y' is a -stock- query any service that owns data has to be able to answer efficiently03:51
StevenKlifeless: But 50 parallel calls? Times 50 users?03:51
lifelessStevenK: where do you get times 50 users ?03:51
* wallyworld_ gets out the popcorn03:51
StevenKOut of the air03:51
lifelessStevenK: so, this is the sequence:03:52
lifelesstraversal03:52
lifelessoh this is private03:52
lifelesscheck rules03:52
lifeless-> denied03:52
lifeless-> full03:52
lifeless-> restricted03:52
lifelesswhere denied is a failure to meet full or restricted03:52
lifelessfull depends on the object, say for Person, it is granted by: is_member or is_owner03:52
lifelessrestricted is granted by 'common ground', so we need to find *a* object where:03:53
lifelessuser.expand_through_team_participation() can see && object subscribed-to, owns, etc.03:53
lifelessso one call to the team membership service, to get all the teams of user03:54
lifelessthats fast, we know it is.03:54
lifelessand then for each data store that might hold an object the Person we are looking at owns [note we don't need TP expansion of it]03:54
lifelesswe ask for exists(discloses=Person, viewer=TP_expanded_user)03:55
lifelesswill it be dead trivial to write? Probably not. Is it beyond our current schema, or an SOA - not at all.03:56
lifelessits also very cachable if we need to.03:56
lifelessthough that might be terrifyingly large.03:56
lifelesswallyworld_: StevenK: I've commented on both your bugs.04:11
wallyworld_thanks04:11
wallyworld_nice to see it clarified :-)04:11
lifelesshopefully it makes sense04:11
wallyworld_i'll read soon. don't want to context switch right at the minute04:12
lifelessif it doesn't, I'm certain sinzui and I are in sync (we were very close to as it turned out), and I can clarify further as needed.04:12
lifelessthats fine04:12
lifelesstis EOD here, so I'm going to be in and out for a hiwle04:12
wallyworld_i'll grok before tomorrow's standup04:12
StevenKwgrant: I still can't see how verifyUpload() can be only ten lines :-(04:12
wgrant20, maybe.04:13
StevenKwgrant: If it makes it easier and simplier, then I'm for it, I just can't envision where to start ripping out bits04:14
lifelesswgrant: http://www.postgresonline.com/journal/archives/209-Manually-setting-table-column-statistics.html04:15
lifelesswgrant: may help04:15
lifelesswgrant: e.g. upgrade to pg 904:15
StevenKOh sure, because the upgrade to 8.4 went so well :-P04:15
wgrantWe should really do that eventually.04:15
lifelessstub and I began discussing logistics for it yesterday, with fdt nearly done04:15
lifelesswgrant:  EXPLAIN ANALYZE SELECT person.id FROM person WHERE id IN (SELECT person FROM accesspolicygrant, accesspolicyuse WHERE accesspolicygrant.policy=accesspolicyuse.id AND  distribution=1 and accesspolicyuse.policy=1) ORDER BY person.displayname LIMIT 75;04:18
lifelessTime: 351.454 ms04:18
lifelesswgrant: compare the plans04:19
lifeless               ->  Index Scan using person_pkey on person  (cost=0.00..6.39 rows=1 width=14) (actual time=0.012..0.013 rows=1 loops=17313)04:19
lifelesswgrant: its a bit variable, but as low as 200ms04:21
lifelesswgrant: I suspect staging is getting a hammering just now ;)04:21
lifelesswgrant: smaller sort too04:22
stubIt will be faster if you ORDER BY lower(displayname)04:22
stubOr at least, it might be able to use an index then04:22
wgrantlifeless: Hm, does that win for any reason beyond the DISTINCT being done first, and not using person_sort_key?04:23
wgrantThe DISTINCT only cuts it by a factor of 4.04:23
wgrantThe rest is pretty similar, apart from the sort key.04:23
stubORDER BY person_sort_key(displayname, name) might be good too04:23
lifelessstub: thats twice as slow04:23
lifelessstub: functional index04:23
lifelessstub: wgrant can fill you in on the gory details :)04:24
stubThere used to be bugs with functional indexes, where the function would be called unnecessarily. Not sure if that has been fixed yet.04:24
stubBut if the index is usable, lower() should be fast since we have an index on lower(displayname) and the functional index bug does not occur for magic internal functions like lower04:25
lifelessstub: its not using the index04:26
lifelesshttp://paste.ubuntu.com/731585/04:26
lifelesswill get you the datatructure in temp tables, to play with04:26
lifelesshttp://paste.ubuntu.com/731544/ is the problem wgrant put to me04:26
stubWith lower() it is not using the index?04:26
wgrantIt can't really use the index.04:26
lifelessit could04:27
wgrantExcept for that one special case.04:27
wgrantWell.04:27
wgrantIt could.04:27
lifelessload it all up and walk04:27
lifelessbut its going to be inefficient04:27
wgrantBut it would have to walk the whole thing.04:27
wgrantRight.04:27
wgrantLike the first plan I provided.04:27
wgrantWhich lifeless may have just linked.04:27
lifelessbecause we're likely to get a wide spread04:27
wgrantYes04:27
lifelessonly special cases will walk successfully04:27
wgrantFor Ubuntu it works very well.04:27
wgrantFor everyone else, probably not so much.04:27
lifelesswe're pulling back 17K / 1437K peoples04:28
lifeless<2%04:28
stubBut anyway, if there is slowness with a person_sort_key query, it might be due to PG bugs and try a lower instead as there is an index on that too.04:28
lifelessindex walks will hit data pages, so an index walk becomes roughly equivalent to a table scan at this size of random-spread data04:28
lifelessstub: yeah, got that04:28
lifelessstub: I think there is (because the function is called per row, and there are [depending on query structure] 56K or 17K rows,.. and python is slow04:29
lifelessI might put a extra column on person and give it a spin, I'm curious04:29
wgrantI'm not quite sure why it's done in Python.04:29
wgrantPossibly because why not.04:30
wgrantIn 2005.04:30
lifelessbecause we didn't comprehend the cost of functional indices when they were introduced.04:30
wgrantRight.04:30
stubIts designed to be used to order results, so slowness in theory only matters on insert04:30
stubCause ordering would hit the index04:30
lifelessstub: and in sorting04:30
wgrantOrdering can hit the index.04:30
wgrantBut it's only effective if you're sorting *lots* of people.04:30
lifelessordering by (foo) requires foo to be calculated04:30
lifelesswgrant: other way around04:30
lifelessits only useful if you're doing an index walk04:30
lifelesswhich requires the index to -very very- closely match the query04:31
wgrantRight, which is only efficient if you're wanting lots of those people...04:31
wgrantYes.04:31
wgrantIsn't that what I said?04:31
lifelessintersecting sets04:31
lifelesslots of people04:31
lifelessor subsections of the index that are well defined can do it too04:31
wgrantWalking the index is only efficient if you want a significant chunk of its values.04:31
lifelessor a small subsection04:34
lifelesswith deep indices04:34
lifelesswe can go around this all day :)04:34
mwhudsonheh are you still talking about query plans?  i've reinstalled my machine and restored ~ from backup since you started on that04:34
wgrantOK, "walking a portion of an index is only efficient if you want a significant chunk of the values from that portion"04:34
lifelessyes, though really we should qualify that to 'in pg' - because other systems can walk on key prefixes ('loose scans')04:35
StevenKmwhudson: Why?04:38
lifelessqastaging may timeout for you for a little bit04:38
huwshimilifeless: If you happen to get a chance to have another look at that avatar branch it would be much appreciated.04:39
lifelesshuwshimi: is thursday ok ? I have tomorrow off for baby stuff, and doubt my ability to page it in properly tonight04:39
mwhudsonStevenK: mixture of really strange acpi bugs and wanting the disk space win7 was taking up04:39
huwshimilifeless: Yeah sure, no problems at all04:39
lifelesshuwshimi: perhaps you can get the OCR (wgrant?) to review it, and take my comments into consideration.04:39
huwshimilifeless: Sure, thanks :)04:40
wgrantI started at 6am, so I was actually about to wander off. But if it's short...04:40
lifelessmwhudson: my new intel 320 ssd is coming ... :)04:40
* wgrant looks.04:40
lifelessmwhudson: I got the 300GB model04:40
huwshimiwgrant: It's not a problem. Another time is fine04:40
mwhudsonlifeless: yeah, that's a bit tempting, but the 140 gigs i now have will take me a while to fill :)04:42
mwhudsoni don't keep photos or much music on this machine anyway04:42
lifelessmwhudson: funny win7 stry04:42
lifelessmwhudson: my desktop came with win7 - its a 16GB ram machine04:42
mwhudson(about 1/3 of my disk space consumption is in ~/VirtualBox\ VMs/ ...)04:42
lifelessmwhudson: I kept win7 for games and just-in-cases04:42
lifelessmwhudson: gave it, I think 100GB or something.04:43
lifelessmwhudson: imagine my surprise when it creates a 16GB swapfile.04:43
StevenKHah04:43
lifelesswgrant: stub: 40 seconds to do:04:43
lifelessselect count(distinct person_sort_key(person.displayname, person.name)) from person;04:43
wgrantlifeless: Swapfile or hiberfile?04:43
lifelesswgrant: pagefile.sys04:44
wgrantHuh04:44
wgrantOdd04:44
lifelessno, unchanged since NT 3.504:44
lifelessits daft, because more ram == more slowness from page file and less need04:44
wgrantI thought it by default used a dynamically scaling one.04:45
lifelessyes, thats the default size heuristic04:45
StevenKLinux still wants swap with a large amount of RAM04:45
lifelessStevenK: lots of sites that care about perf disable it these days for linux.04:45
StevenKSure, the kernel wants it, doesn't mean it gets it. :-)04:45
lifelessStevenK: you still get VM backed shared pages etc; but there is precious little use for swap per se04:46
stublaunchpad_prod_3=# explain select count(distinct(person_sort_key(displayname,name))) from person;04:46
stub                              QUERY PLAN04:46
stub-----------------------------------------------------------------------04:46
stub Aggregate  (cost=51296.04..51296.30 rows=1 width=21)04:46
stub   ->  Seq Scan on person  (cost=0.00..47617.83 rows=1471283 width=21)04:46
stub(2 rows)04:46
stubTime: 534.701 ms04:46
mwhudsonlifeless: hah04:46
lifelessstub: garh what04:46
mwhudsonlifeless: that might have happened to me, i guess04:46
mwhudson(8 gigs in this machine)04:46
stublifeless: Warm index04:47
lifelessmwhudson: http://support.microsoft.com/kb/31448204:47
StevenKI thought Windows had an option to twiddle for no swap04:47
lifelessThe default paging file size is equal to 1.5 times the total RAM. However, this default configuration may not be optimal in all cases04:47
wgrantStevenK: It does.04:47
lifelessso, yes, not 16. 24GB04:47
mwhudsonrofl04:47
stublifeless: actually, not hitting the index. just warm and gobs of ram I guess to hold the immutable result of the function :)04:47
wgrantStevenK: Hm?04:48
wgrantEr.04:48
wgrantstub: That was an EXPLAIN.04:48
stubahaha04:48
wgrantstub: Not an EXPLAIN ANALYZE...04:48
lifelesshah yes04:48
lifelessECOFFEE04:48
wgrantSo how was it 500ms.04:48
wgrant1000?04:48
wgrantBut surely not on such a trivial query.04:48
wgrantI mean, what planning is there to do...04:48
mwhudsondoes pg 9.1 work for lp dev?04:49
lifelessmwhudson: maybe ;)04:49
wgrantmwhudson: Unlikely.04:49
wgrantBut possibly.04:49
stubmwhudson: probably04:49
stubmwhudson: Might need some tweaking of the various setup scripts04:49
lifelessperson update still going... probably should have chunked it04:49
* mwhudson chuckles04:49
mwhudsoni guess i'll find out sooner or later04:49
lifelessat least its not waiting for anything04:49
wgrantWell, yes, if you want to tweak stuff then of course it will work :P04:49
stublifeless: So the python method is about 3x slower than the builtin lower()04:49
lifelessstub: yes :)04:50
lifelessstub: which will be slower than an actual column04:50
lifelessI'm making one on qastaging's Person table.04:50
lifelessjust because04:50
stublifeless: I'm thinking of whitelist for scripts we know have long running transactions, so we need to add them explicitly and open bugs.04:51
wgrantstub: Hmm, but we often have to kill them early.04:51
lifelessstub: so there are three cases I think04:51
wgrantBecause eg. karma sticks around for 10 minutes after we kill it.04:52
lifelessstub: readonly safe to kill04:52
lifelessstub: readonly unsafe (mutates disk unrecoverably)04:52
lifelessstub: readwrite unsafe (mutates disk unrecoverably)04:52
lifelessfirst case should be readonly/readwrite safe to kill (atomic, no external changes)04:52
stublifeless: safe to kill, but should we do it without making note that this task needs to be fixed?04:52
lifelessstub: I like making a note04:53
lifelessstub: I worry that you mean 'have the deploy abort when it finds a new one'04:53
stubwgrant: But it won't stick around for 10 minutes if the full-update.py terminates the backend04:53
stublifeless: if we find a new one, we should abort and investigate.04:53
wgrantstub: Are you sure?04:54
lifelessstub: why do you say that? we know what code we have...04:54
wgrantpg_cancel_backend isn't effective. I thought we tried pg_terminate_backend once too...04:54
stublifeless: sticking our head in the sand, crossing our fingers and full steam ahead will *normally* work04:54
stublifeless:  I mean defining EVIL_USERS = [], similar to fragile users, where we list the badly behaved db users along with a bug number. These users we ignore their long running transactions and let preflight.py terminate things.04:55
lifelesswow, person is sparse on id04:55
stuberm... full-update.py terminate things04:55
mwhudsonlifeless: all those shipit accounts that got deleted?04:56
lifelessmwhudson: p'rhaps04:56
StevenKlifeless: Out of interest, did you see my analysis on bug 618399?04:56
_mup_Bug #618399: DistroSeries:+nominations timeouts <lp-blueprints> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/618399 >04:56
lifelessStevenK: yes, sounded plausible04:56
wgrantThey actually mostly didn't get deleted.04:56
wgrantThere's still a tonne around.04:56
lifelesswgrant: check a histogram of person ids on 100K intervals04:56
lifelessstub: what about WARNING in the script (so losas see it) if something isn't in EVIL, but not aborting.04:57
stublifeless: we used to be at >10million rows04:57
stublifeless: Yes, so we proceed into downtime hoping everything is working fine.04:57
stubI'd rather not hope. I'd rather we paused and thought about things a little bit.04:57
wgrantstub: But the highest person ID is only like 3.6m04:57
wgrantAnd account ID is 4.8m.04:57
StevenKDROP TABLE Account;04:58
stubCould have sworn we got to 10million. Might be thinking of account?04:58
wgrant2.2m people no longer exist.04:58
stubGraphs will have the answers if anyone wants to wait for them to load ;)04:58
wgrantI suspect lots of the remaining ones shouldn't either.04:58
wgrantAccount retains 4.6m of its 4.8m rows :/04:59
wgrantI suspect 3.4m of those are unused.04:59
wgrantNow that shipit is deceased.04:59
StevenKHow can we tell, and can we just delete them?05:02
lifelessstub: so you're placing a premium on us having *unidentified* code that is both (bad) and (harmful to interrupt).05:02
lifelessstub: I'm placing a premium on failing to do a FDT being disruptive to velocity and our squad leads having identified (harmful to interrupt) cases already.05:03
lifelessstub: how can we better refine the expected outcome of e.g. me being wrong and something horribly harmful and bad coming along05:04
lifelessstub: one way would be to list everything not fragile as evil, going from the list of db users05:09
lifelessanything we think interrupts could be harmful to wouldn't be evil.05:09
lifeless(and would cause a stop)05:09
lifelessbut this seems, to me, to be equivalent to fragile05:09
stublifeless: I think the rare occurrence of delaying a FDT a day or two (without any downtime) is worth us not guessing about what is safe to terminate or not. We have done a lot of FDT now, and identified two processes that are a problem.05:16
stubIf it is not rare that some new is badly behaved or an existing process starts misbehaving, then we have a different problem.05:18
stubWhy has this process changed? Why is this new process behaving badly? Has the system become unstable enough that we shouldn't risk further changes?05:21
stubHave the assumptions we used when designing fastdowntime deploys become invalid?05:22
lifelesswgrant: 200ms is as fast as I can get it tonight; stub may do better05:25
stubI haven't seen the query yet05:26
lifelessstub: the pastebins ;)05:26
stubI don't see the query we are trying to optimize in the pastebin I have.05:26
lifelessstub: separately: functional index - 35s, cached sort column, 8s - count distinct over person (1.4M rows) - hot index05:27
stuboic. found it05:27
lifelessstub: two pastebins: http://paste.ubuntu.com/731585/ and http://paste.ubuntu.com/731544/05:28
lifelessbah, yes05:28
stubMy first question is why on earth would we want to do this query? Being able to hide from security reports by changing your name to match someone elses doesn't seem a good idea.05:29
lifelessstub: they can't do that :) - name is unique05:29
lifelessstub: its the url component05:29
stubSo the distinct on is redundant05:30
lifelessyes, but wgrants first query needs the distinct to eliminate 40K duplicate rows05:30
lifelessmy rephrased one focused on person only finds the 17K interesting rows inthe first place05:30
lifelessthis is the fastest I have (so far):05:31
lifelessEXPLAIN ANALYZE SELECT person.id FROM person WHERE id IN (SELECT distinct person FROM accesspolicygrant, accesspolicyuse WHERE accesspolicygrant.policy=accesspolicyuse.id AND  distribution=1 and accesspolicyuse.policy=1) ORDER BY person.temp_sort_key LIMIT 75;05:31
lifelesswhere temp_sort_key is just a materialised person_sort_key05:31
lifelessit jumps around but hovers aorund 300ms05:31
lifelessstub: i'm thinking about the other stuff; need to cook dinner now05:32
stubyup05:32
stubBah. setup pastebin missing a bracket05:34
stubOr gnome ate it05:37
stubI can't see any obvious ways of improving those queries more. I suspect we don't care enough to materialize the person_sort_key and should just use displayname or lower(displayname)05:48
stub(although lower(displayname) won't give unique ordering, blergh)05:49
stubEXPLAIN ANALYZE SELECT person.id FROM person WHERE id IN (SELECT DISTINCT ON (person) person FROM accesspolicygrant, accesspolicyuse WHERE accesspolicygrant.policy=accesspolicyuse.id AND  distribution=1 and accesspolicyuse.policy=1) ORDER BY lower(displayname),name LIMIT 75;05:50
stubSeems a little faster actually... probably cache artifacts.05:50
stubyer - performs the same as using temp_sort_key05:52
jtvWho can help out with an openid-and-account-merging problem?  https://answers.launchpad.net/launchpad/+question/17753905:56
jtvCome on people, not everybody at once please.  :)06:02
jtvstub, lifeless: any ideas what to do about that one?06:03
lifelessjtv: they need to merge them separately in SSO AIUI06:59
wgrantstub: I was initially doing DISTINCT ON (Person.name). But that wasn't fast, so I tried using the other indexed values that we normally use (person_sort_key). Those aren't real queries, but they're close :)07:32
stubwgrant: yer. surprised the standard join version comes in at twice lifeless' version though. But selecting 55k rows, distilling & ordering looks like it takes 2-300ms.07:39
wgrantstub: Yeah :/07:39
lifelessI was surprised at the slow too07:44
adeuringgood morning08:33
mrevellHi09:11
allenapHello.09:14
=== gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugtasks: 275
mrevellhiw bigjools11:19
mrevell or rather, "hi"11:19
bigjoolsmrevell: yello11:20
bigjoolsok, how much do you know about copying packages in PPAs?11:21
mrevellbigjools, Relatively little.11:21
bigjoolsmrevell: ok I'll show you an example page11:21
bigjoolsmrevell: https://launchpad.net/~launchpad/+archive/ppa/+copy-packages11:22
* mrevell looks11:22
mrevellRighto.11:22
bigjoolsmrevell: so at the moment, people click the boxes and the buttons and it tries to do all the copying inside a webapp request11:23
mrevellRight, I see.11:23
bigjoolsthis frequently times out11:23
mrevellI can imagine :)11:23
bigjoolsbecause there's a million checks to make when copying11:23
bigjoolswe now have an infrastructure to do copying in the background in a job11:24
bigjoolsdone as part of derived distros11:24
bigjoolsso I'd like to switch over to using that11:24
bigjoolsbut there are two problems:11:24
bigjools1. some people still want to do "instant" copying11:25
bigjools2. if we do the background copying, we have no way of providing feedback if it fails11:25
bigjoolsso my question is, do you feel up to providing some input on that?11:25
mrevellI certainly do.11:25
mrevellI'd like to ask some questions, first.11:26
bigjoolssure11:26
mrevellwrt the first point: how important do you judge this need to be? We can look at it in more detail etc but what's your feeling right now?11:26
bigjoolssort of.  people are used to it, it would be hard to take away. I think wgrant had some strong opinions?11:27
maxbAlso, there are scripts which invoke API methods to copy packages, and may legitimately (IMO) assume that a successful method return indicates a successful copy11:28
bigjoolsah yes, I forgot about that11:28
maxbCopying a single source package generally does not time out11:28
mrevellbigjools, Would it be naive of me to ask if we can just hand the job to the longpoll infrastructure and then have a live status box on the page?11:29
bigjoolsmaxb: there's a new API call for background copying11:29
maxbJust so long as the synchronous one doesn't go away11:29
bigjoolsit won't11:29
mrevellAs for instant copying, I can imagine it might be fairly straightforward to have a UI that allowed for instant copying of a single package11:29
mrevellbut explained it'd be a batch job for anything more.11:30
bigjoolsmrevell: longpoll would complement this for sure, but we'd need it to work without as well11:30
mrevellHuw can help us design that.11:30
mrevellbigjools, Forgive me for asking, but why does it need to work without it?11:30
bigjoolsmrevell: basically lifeless is insistent that we don't depend on rabbit11:30
bigjoolsI sort of agree11:31
mrevellOkay, fair enough.11:31
mrevellbigjools, So, we could we email people if there's a failure?11:31
bigjoolsit's an option, but one that I dislike11:31
mrevellWhy?11:32
bigjoolsemail indicates you have a UI failure11:32
bigjoolswe all get too much email already11:32
wgrantSorry, back.11:33
mrevellOkay, fair points; particularly the second about getting too much email already.11:33
bigjoolsnow it *could* be justified in that we already get email notifications for uploads anyway11:33
wgrantSo, email is probably wrong.11:33
wgrantWe should provide an API method to poll.11:33
mrevellbigjools, This is something we could fix with a personal dashboard.11:33
wgrantAnd longpoll for the normal case.11:33
bigjoolswgrant: longpoll11:33
wgrantRight.11:33
bigjoolswe already have API to see if a package is present11:34
wgrantWe should identify syncSource callers and ask known ones to migrate.11:34
=== matsubara-afk is now known as matsubara
mrevellbigjools, There'd be a notifications area. We could even push out a Twitter notification. "@bigjools Your package copy failed. See pad.lv/fail1234 for more detail."11:34
bigjoolswell we can keep syncSource11:34
wgrantOf course, there's no reason that reasonably sized copies can't be done in the webapp.11:34
wgrantIt's just that our schema and code are terrible.11:34
mrevellwgrant, Do we have any idea of what is reasonable?11:34
wgrantMostly the code is terrible.11:34
wgrantmrevell: Not really, no.11:34
bigjoolsmrevell: I think these are all great - there are Rabbit → Twitter etc. gateways11:35
wgrantmrevell: Nobody has spent much time optimising the copier.11:35
mrevellAre we safe to say that, generally, a single package copy will work in the webapp?11:35
mrevellbigjools, But we need a solution that doesn't rely on Rabbit, right?11:35
bigjoolshowever much it's optimised, it'll always fail given too many packages11:35
mrevellAnd is a single copy the primary use case?11:36
bigjoolsmrevell: for throw-away notifications I think it's fine. what's important is that the *only* way doesn't involve rabbit11:36
bigjoolssingle copy can fail actually11:36
StevenKmrevell: Generally. And then the kernel team tries to copy 'linux' and it constantly times out.11:36
mrevellbigjools, So, we need a notification area on a personal dashboard.11:36
bigjoolsthat --^11:36
lifelessbigjools: mrevell: a bit of nuance on rabbit11:36
mrevellBlimey, okay11:36
bigjoolsmrevell: or a notification area on the PPA page11:36
lifelessI'm saying we can't depend on it not being rebooted etc11:36
bigjoolsmrevell: which is better in this case as others will want to see the copy11:37
lifelessI think its fine for something *we can accept occasional loss on* to be rabbit only11:37
lifelessbut we need to explicitly make that choice.11:37
bigjoolslifeless: right.  Like twitter notifications11:37
lifelessright11:37
mrevellbigjools, The beginnings of a wall for the PPA, effectively11:37
lifelessthe reality is that such loss will be extremely rare11:37
lifelessbecause a graceful reboot will keep persistent rabbit messages11:38
bigjoolsmrevell: I was thinking of a notification area in the PPA page that shows in-progress and failed copy jobs11:38
lifelessits only if ackee shits itself :) that we'll lose in-flight messages.11:38
mrevellThanks for the clarification lifeless.11:38
bigjoolsand you can ack the failed ones which makes them go away11:38
mrevellbigjools, Sounds ideal.11:38
mrevellbigjools, Let's involve Huw.11:38
mrevellbigjools, Is the Twitter thing realistic?11:39
mrevellbigjools, Within the scope of a maintenance task, I mean.11:39
bigjoolsmrevell: excellent - if he can design something that'd be fab.  I am away for 2 weeks from Thurs, but I can send an email describing what we need11:39
lifelessso you could, for instance, do rabbit only notifications, and if we have a hardware failure on rabbit, we lose some11:39
bigjoolsmrevell: sorta, depends on how hard it is11:39
StevenKI don't like the idea of Twitter11:39
mrevellbigjools, Please do. Huw can get something done this week.11:39
bigjoolswe also want an RSS feed based on Rabbit notifications11:39
lifelessmrevell: twitter - I'll need to talk to elmo about how we should interact with twitter, but in principle doable11:39
mrevellStevenK, Or identi.ca. I'm looking for a way to do push notifications that doesn't involve email.11:39
bigjoolsthere's a million things that soyuz does that we need notifications for11:40
lifelessmrevell: (firewall aspects basically)11:40
mrevelllifeless, Sounds good. I'm not totally sold on it but "Why doesn't LP send me a message on Twitter when X happens?" seems to come up every now and then.11:40
StevenKmrevell: I don't like it for the concept, not the use of Twitter.11:40
lifelessmrevell: I'm not arguing for or against, just letting you know :)11:40
mrevellStevenK, Not even as a back-up to a notifications area on the PPA npage?11:40
lifelessmrevell: wearing my TA hat I want a scalable secure pub-sub to the internet facility for all of LP, which would let folk external to lp do interesting things like twitter notifications11:41
mrevelllifeless, Yeah, thanks :) I mean, if there are mountains to be moved to get Twitter notifications, then I'd really rather spend to time ... and research ... looking at whether we really want to do this.#11:41
lifelessmrevell: but we haven't built that11:41
lifelessmrevell: I have a design in mind though, FWIW :)11:41
mrevelllifeless, Right, precisely. I'd rather do it properly than just tack something on for this one use case.11:41
bigjoolslifeless: Rabbit has all sorts of useful gateways :)11:41
StevenKmrevell: I seriously want notifications about stuff I'm doing within Launchpad to stay within Launchpad.11:41
mrevellStevenK, There's no reason we couldn't make it opt-in.11:42
bigjoolsyes there is the privacy issue11:42
lifelessbigjools: its a likely component, I'm actually thinking pshb as the primary protocol, rabbit can be used to shove notifications around internally, and pingthesemanticweb.com externally; I have a rough idea for handling private objects even.11:42
mrevellLike I say, this is something that people ask for but that I havne't necessarily given all that much thought.11:42
StevenKThe fact that I *despise* Twitter/identi.ca might be shining through ...11:43
mrevellHowever, I'm generally in favour of Launchpad pushing notifications out to where people want to get them, rather than forcing them to come to LP.11:43
bigjools#luddite11:43
mrevellStevenK, In which case, don't opt-in11:43
mrevellbigjools, Haha!11:43
mrevellStevenK, I think it's right not to force people to use it.11:43
bigjools+1 t o push11:44
wgrantDoes Twitter do private tweets?11:46
wgrantWe have lots of private notifications :)11:46
bigjoolsDMs11:47
mrevellwgrant, There's direct messaging but ... whether we consider that private or not is another matter.11:47
bigjoolsok thanks for the input everyone11:54
stubI think it would be awesome if people could choose from a variety of notification methods, making email addresses in Launchpad optional.12:17
stubemail/jabber/twitter/identi.ca probably covers the bulk of our userbase.12:18
lifelessfacebook.12:20
lifelesswould cover nearly all :P12:20
allenapjelmer: Can you help me with mardy in #launchpad. He's having problems doing a git import, http://launchpadlibrarian.net/84700393/mardy-webcredentials-libaccounts-glib-trunk.log12:31
allenap?12:31
bigjoolswgrant: did you ever look at the debtags bug?12:39
jelmerallenap: looking now12:45
allenapjelmer: Ta, thanks.12:46
stublifeless: Facebook can be accessed via email12:46
wgrantbigjools: I mostly ignored it.12:50
bigjoolswgrant: I'm giving it an initial analysis - I'm a little confused as to the aims of the bug12:50
bigjoolsbug 5741812:50
_mup_Bug #57418: Support debtags in Packages.gz <escalated> <feature> <lp-soyuz> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/57418 >12:50
bigjoolsseems like we need another import :/12:52
wgrantbigjools: Well, yes, we need to work out what they want. I strongly suspect that using Debian's DB directly will be unacceptable, so we'll need to DBify it or have a custom Ubuntu one.12:52
bigjoolsindeed12:52
wgrant(because we have lots of local package)12:52
wgrant+s12:52
bigjoolswhich brings the question of merging upstream's12:52
wgrantAnd that is the stage at which I play the midnight card and disappear :)12:54
bigjoolsheh12:54
wgrantFor it is a mess.12:54
bigjoolsit's an escalated mess12:54
wgrantVery similar to P-a-s.12:54
bigjoolshence I am trying to clear it up12:54
bigjoolsI am not doing another P-a-s12:54
wgrantYeah, but escalated Soyuz issues are all your problem now :)12:54
wgrantHeh.12:54
wgrantWell.12:54
wgrantIt'd be easy to.12:54
bigjoolsthat's the problem12:55
bigjoolsit's soooooo tempting12:55
wgrantAs is sleep.12:57
wgrantSo I shall leave you to temptation.12:57
bigjoolsjml: are you involved with the request to support debtags?13:04
bigjoolswgrant: good night13:04
jmlbigjools: yes. I asked Francis when we could expect it and whether there were any plans / specs in case we wanted to do the work.13:05
bigjoolsjml: I am doing some analysis13:06
bigjoolsjml: I wondered if the requirement was to use Debian's debtags database or to start a new one for Ubuntu?13:07
bigjoolsit's not specified13:07
jmlbigjools: cool. If you need requirements from us, mvo & james_w are the best to ask.13:07
jmlbigjools: I *think* we'd need to start a new one13:07
jmlbigjools: but mvo would know for sure.13:07
bigjoolsjml: ok cool - do you guys have an IRC channel of your own?13:07
jmlbigjools: oh boy, do we ever! #ubuntu-app-devel, #software-center on Freenode13:08
bigjoolsok I'll pop in after lunch13:08
bigjoolscheers13:08
jmlnp13:08
sinzuimrevell, Did you ever send StevenK the text for the team PPA emails?14:31
mrevellStevenK, I did, yes, this morning.14:32
sinzuifab14:32
nigelbGmail thinks every mail bigjools writes is very important :P14:32
sinzuimrevell, did you see the email I sent to the list about open/closed teams. Maybe Dan can help solve the language issue?14:32
mrevellsinzui, I did; I'm hoping to reply later. Dan'll be back tomorrow.14:34
sinzuithanks14:34
bigjoolsnigelb: gmail is clever14:35
=== matsubara is now known as matsubara-lunch
nigelbbigjools: :)14:43
sinzuijcsackett, ping14:48
jcsackettsinzui: pong.14:48
sinzuijcsackett, do you have a few minutes to mumble?14:48
jcsackettof course.14:48
* jcsackett fires up mumble.14:48
deryckabentley, adeuring -- you guys see any reason we shouldn't turn on new listings for ourselves now?14:57
adeuringderyck: sounds fine14:58
abentleyderyck: we might get timeouts, and we can't turn them off.14:58
deryckabentley, are you opposed to trying, seeing what it's like, and if arise, disabling the flag?14:59
abentleyderyck: no.15:00
deryckgreat, thanks!15:00
abentleyderyck: Just be sure to disable wallyworld's stuff, if it's enabled for us.15:00
mrevellhey abentley, I just need to grab a coffee; time ran away with me. Five minutes.15:00
deryckabentley, ok, will do.  I'll make the rules match qastaging.15:01
abentleymrevell: no problem.15:01
* bigjools just deleted a bugtask.. Awesome15:03
abentleyderyck: dynamic bug listings are currently disabled on qastaging for me.15:08
deryckabentley, ok.  I think you need to be in the demo group.  I'll look here in a sec.15:08
abentleyderyck: Also, the rule for disabling ajax.batch_navigator.enabled is set for orange squad, when it should be set for custom-buglisting-demo15:10
deryckabentley, ok.  And I don't need that rule on lpnet if it's not turned on at all on lpnet, right?15:11
abentleyderyck: right.15:11
deryckok, cool.15:11
deryckabentley, I added us back to the demo team.  Things seem to work ok, until I can change the flag for the other feature to the demo team.15:16
abentleyderyck: great.  Thanks.15:17
=== matsubara-lunch is now known as matsubara
=== gmb is now known as graham
=== graham is now known as gmb
=== gmb changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 275
mrevellNight all18:03
brycehderyck, quick question (raised by one of our upstreams) - do you know if the bugzilla comment importer is enabled for GNOME?18:21
deryckbryceh, hmmm, I don't actually know.  I recall it being raised some time ago about wanting to get it working....18:21
deryckbryceh, I may have dropped the ball on getting it enabled.18:22
brycehderyck, ok thanks18:22
deryckbryceh, I think it would be better for one of the maintenance squads to check into it... perhaps yellow, which has gmb on it, since he knows a lot about that area.18:23
deryckbryceh, I think that's why I dropped it before, just caught in feature work like I am now.18:24
brycehderyck, ok good to know.  Yeah at this point it's not a request, just question about the status of it18:24
deryckok, cool.18:24
=== deryck is now known as deryck[lunch]
=== deryck[lunch] is now known as deryck
=== matsubara is now known as matsubara-afk
deryckabentley, you around still?22:36
bachi lifeless, the test i added to the branch you reviewed yesterday is causing other worker tests that run after it to fail.  can you see anything i have bungled?  http://pastebin.ubuntu.com/732539/22:42
lifelessbac: getMirrorFailureForException is a helper, not a test itself22:56
lifelessbac: what is the test that fails (and does it fail when run on its own ?)22:56
lifelessbac: one possibility is that the test isn't twisted safe - the call to worker.mirror() may be assuming synchronous behaviour, which isn't a guarantee in any twisted environment.22:57
lifelessbac: if that is the case, the helper will need to be reworked to use deferred and so forth22:58
lifeless:(22:58
abentleyderyck: kinda.22:58
deryckabentley, one question.... I assume I need to call change_fields with the right config for my integration work.  true?22:59
abentleyderyck: Right.23:00
deryckabentley, awesome, thanks!  Have a good night.23:00
abentleyderyck: The list of fields is around browser/bugtask.py:224123:00
deryckabentley, thanks23:01
abentleyderyck: kp23:02
abentleyderyck: np23:02

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!