wgrantNo CREATE TABLE IF NOT EXISTS in 8.400:19
wgrantAdded in 9.1 :(00:19
wgrantlifeless_: the temp journal flush is very slow because it calls a function for each row. Fixing it to do a bulk insert is still fairly slow, because of the bugsummaryjournal FKs and TRUNCATE being slower than DELETE for small tables like this.00:32
StevenKwgrant: I'm questioning if my IBug leak fix is really QAable :-/00:33
wgrantStevenK: Did the test suite pass?00:33
wgrant(yes, it did => qa-ok)00:33
wgrantOr untestable00:33
StevenKIt's on qas, so both ec2 and buildbot did00:34
lifeless_wgrant: the flush shouldn't block journal inserts though00:48
wgrantlifeless_: The flush is called as part of the trigger.00:48
wgranttemp journal flushes to journal within the trigger00:48
wgrantThen garbo rolls up the journal00:48
lifeless_ah, skimmed over the word temp00:50
lifeless_thanks for debugging00:50
=== lifeless_ is now known as lifeless
lifelessnot bad - 386 Exceptions02:15
wgrantlifeless: 100 more than normal :(02:49
wgranthwdb crap, probably02:49
lifelessalso 50 from oops-tools thyself02:50
lifelesstouch 500.html -> -5002:50
wallyworld_wgrant: what's the timeframe for the landing of the code to mirror data into the new access policy schema?02:53
wgrantwallyworld_: I expect the trigger will land in a few hours (just waiting for a branch to be merged from stable to db-devel)02:53
wgrantwallyworld_: But that doesn't include the sampledata updates.02:54
wgrant(and we need to create AccessPolicies on prod and dev before the triggers will do much)02:54
wallyworld_ok. i have a couple of options to do next. i'll wait for this stuff02:54
lifelessjust don't use sampledata02:54
lifelessits deprecated anyhow02:54
wallyworld_i wasn't going to use sample data02:55
lifelessah cool, I was misled :P02:55
wgrantlifeless: We have to update the sampledata02:55
wgrantOr none of the private bugs will be valid.02:55
lifelesswgrant: how many tests use private bug sampledata?02:55
wgrantProbably quite enough :(02:55
lifelessmight be worth a check02:56
wgrantsampledata is also still critical for test performance, unfortunately.02:56
wallyworld_lifeless: above, i was talking about waiting for the triggers to land, not sample data02:56
wgrantwallyworld_: https://code.launchpad.net/~wgrant/launchpad/bug-mirror-access-policies/+merge/94722 is the MP, and it has tests which use normal factory methods.02:57
lifelesswallyworld_: yes, I was misled by wgrant bringing in sampledata to the convo02:59
wallyworld_np :-)02:59
wgrantlifeless: Well, for most purposes this isn't useful without the sampledata updates.02:59
wgrantPerhaps for wallyworld it is :)02:59
lifelesswgrant: I would love it if you check and see what the effort to delete the private bugs would be from the sampledata03:01
wgrantlifeless: I have a sampledata vs. factory-ng battle experiment planned for next week.03:02
lifelessyou had me at battle03:04
wgrantlifeless: Well, the current factory is reasonably awkward and extremely slow, so I'm going to attempt to improve it.03:07
wallyworld_lifeless: there needs to be some tal for the basic page chrome. the other tal that's there was from an earlier branch where john just put in sample data as a placeholder. it's totally replaced at runtime. i still see some value in retaining that03:13
lifelesswallyworld_: so, I'd love us to get rid of the duplication03:38
lifelesswallyworld_: how is up to you :)03:38
StevenKlifeless: Your DSL is still misbehaving03:39
lifelessStevenK: I sispect so03:41
StevenKwgrant: O hai, Mr OCR.04:34
wgranthow is it bad04:36
wgrantI ran it over THE WHOLE TREE two months ago04:36
StevenKTell me about it04:37
StevenKI only thought to because Aaron added a whole of bunch of 'from x import (\nsingle module,\n)'04:37
wgrantOne of them is mine :(04:45
StevenKI also didn't notice the problematic codehosting import04:45
StevenKIn the diff04:45
wgrantMost of those are now handled.04:46
wgrantby a # FIRST comment.04:46
* StevenK notes one is wrong and reverts it04:46
wgrantWhat's wrong with it, beyond the superfluous canonical import?04:47
StevenKIt's been moved to the wrong place.04:47
wgrantIt shouldn't exist in the first place.04:47
wgrantThere's a few others around too.04:47
wgrantI've listed in them in my review that I'm about to finish.04:47
wgrantBut in general anything importing anything from canonical is wrong, unless it's using __file__ to find icing.04:48
StevenKlib/lp/services/librarianserver/testing/server.py makes use of canonical.__file__04:48
wgrantAh, it's possible they all do.04:49
wgrantBut it seems unlikely.04:49
wgrantIf they do still need them, fix the formatter to know that canonical is local.04:49
wgrantReviewed, anyway.04:49
StevenKI've already pushed the change that does04:49
wgrantLook at how it uses canonical.__file__04:49
wgrant'tis stupid.04:49
wgrantPls fix.04:49
StevenKTo do what instead?04:50
wgrantIt uses canonical.__file__ so it can get the root of the tree.04:50
wgrantPossibly use lp.__file__ instead.04:50
wgrantMost other cases are probably similar.04:51
wgrantMight as well wipe them all out now.04:51
wgrantcanonical.launchpad.__file__ is usually valid, though, as it's used to find icing.04:51
StevenKwgrant: What do you think about http://pastebin.ubuntu.com/860097/ ?05:10
wgrantStevenK: Looks reasonable.05:14
StevenKwgrant: Recommendation for the use of canonical.__file__ in lib/lp/services/config/doc/canonical-config.txt?05:16
wgrantStevenK: s/canonical/lp/g05:17
StevenKsteven@liquified:~/launchpad/lp-branches/format-imports-cleanup% bzr grep canonical.__file__ | wc -l05:21
stubwgrant, StevenK: I think config.root gets you the root better05:25
wgrantstub: It's testing config.root.05:26
stuboh :)05:26
wgrantI was going to say the same thing until I actually read the test.05:26
wgrantit works05:39
StevenKWhich bit?05:40
wgrantBug searches without any joins at all.05:43
wgrant(including for security checks, which were the big remaining pain)05:43
lifelesswgrant: interesting05:45
wgrantStill waiting for DF to finish regenerating the data, but the plan looks good.05:48
lifelesswhats the schema for that ?05:55
lifelesswgrant: ^05:57
wgrantInclude AccessPolicyArtifact.policy and AccessArtifactGrant.grantee in arrays on BugTaskFlat. Not feasible to index, but a whole lot faster in a scan than joining against BugSubscription.05:58
wgrantBecause we can calculate the policies and grantees to match against once at the start of the query.05:58
lifelessI'm very glad you're spending time trying different htings05:58
wgrantWell, that's better than expected.06:03
wgrant Seq Scan on bugtaskflat  (cost=0.00..958352572.67 rows=193926 width=4) (actual time=0.020..970.982 rows=91164 loops=1)06:03
wgrant Total runtime: 1026.971 ms06:03
wgrantSo even completely unindexed, as long as it's hot it only takes a second to do a full scan.06:03
wgrantI think that is a reasonable next victory.06:03
wgrant'cause this table should be pretty damn hot.06:03
stubGIN and GIST indexes support array operations. Landscape are using them.06:04
wgrantstub: Yeah, but it's not too useful here.06:04
stubNot sure if they are the operators you need though (IS IN etc.)06:04
wgrantIt's possible we can get something out of BitmapAnding06:04
wgrantBut in general this shouldn't be too bad.06:04
wgrantBecause the arrays are all small06:04
wgrantNormally only one policy, and <5 grantees.06:05
wgrantThe index may provide significant benefit if you're unable to see most of the bugs, eg. in a private project.06:05
wgrantBut we'll see.06:05
stubStill pulling up 91k rows, which is a lot even if half the estimate06:06
wgrant(the above is a COUNT(*) of a default Ubuntu bug search, with bugs visible to me)06:06
wgrantSo it's meant to be 91k rows06:06
stubAnd better than what we already have for sure :)06:06
wgrantOh yes.06:06
wgrantAnd it's still treating the CTEs as correlated.06:06
wgrantIt's not meant to do that...06:06
wgrantAh, no it's not, just the top-level.06:07
=== stub1 is now known as stub
wgrantneed more rams06:24
wgrantBah, forgot to include tags in this rebuild.06:26
wgrantBut ideally need stats for tags06:29
wgrantWhich means forcing them into a tsvector AFAIK :(06:29
wgrantAnyway, this will hopefully be fast enough for now, and we can easily garbo up a new fact table online later.06:32
wgrantHmm, under-selective FTI is slow, even with GIN. Possibly just that DF can't really fit the full indexes and tables in cache.06:34
* wgrant declares the experiment a great success and shall propose it tomorrow.07:12
lifelesswgrant: if you have a test script I can run it on asuka07:13
wgrantlifeless: Are you around tomorrow?07:17
wgrantlifeless: Great, will throw stuff at you then.07:29
lifelessStevenK: perhaps 'make' should format imports08:51
wgrantThat's a bad, bad precedent I think.08:51
adeuringgood morning08:55
wgrantLint could, though.08:55
wgrantMorning adeuring.08:55
lifelesswgrant: also, please to try doubling our data set with the flatbugtask perf tests08:59
lifelesswgrant: [estimate behaviour if we sit on it for a while after deploying]08:59
wgrantbut OK08:59
wgrantstill need to set up proper indexes and such tomorrow.08:59
wgrantBut even with no indices it's largely better than we have now.09:00
wgrantEven on dogfood.09:00
wgrantWhich can't fit the full table and indices in RAM09:00
wgrant(well, it could if there was nothing else running, but there is)09:00
lifelessIf you can honestly state you're not worried about behaviour with our dataset doubled, I will take your word on it.09:01
lifelessMy point is that we need to make /some/ thought and assessment about it.09:01
wgrantA lot of sort and tag/fti filter cases still degrade to an in-memory sort, so yeah, I'm a little worried.09:01
wgrantBut I've looked at it a bit.09:02
wgrantfti often turns into a BitmapAnd, which drops order, so when the result is large it gets bad.09:02
wgrantBut it's still not more than 1.3s hot, even when it's sorting the whole set.09:02
wgrantI'm going to look tomorrow morning at throwing tags into the first iteration.09:03
wgrantBut I don't have much hope because array stats aren't implemented.09:03
wgrantAnd stats matter for Ubuntu's tags.09:03
wgrantAlthough it may not be so bad to always do a full scan for tag searches if they're inline.09:04
nigelbMornin' mrevell09:04
wgrantStill, it no longer feels like bug search performance is entirely hopeless.09:05
danhgaloha vera09:10
=== almaisan-away is now known as al-maisan
=== danhg_ is now known as danhg
salgadoStevenK, hey, do you know when you might have some time to finish the review of https://code.launchpad.net/~linaro-infrastructure/launchpad/workitems-migration-script/+merge/93883?  I'm happy to bug somebody else if you won't have time for it soon12:37
mabacI've proposed  the Work items UI if anyone is up for reviewing that: https://code.launchpad.net/~linaro-infrastructure/launchpad/workitems-widget/+merge/94790 :)12:53
=== al-maisan is now known as almaisan-away
=== almaisan-away is now known as al-maisan
deryckadeuring, abentley, rick_h_ -- I'm stuck on the phone with my bank. Can we hold standup until this completes please?14:31
adeuringderyck: np14:31
rick_h_deryck: gotcha14:31
abentleyderyck: sure thing.14:31
deryckthanks, guys.  sorry.14:32
czajkowskilifeless: when you're about can you perhaps have a look over https://bugs.launchpad.net/launchpad/+bug/942523  please ?14:43
_mup_Bug #942523: [Wish] Add a section Trophees in launchpad profile with the Trophee linked to Launchpad & Ubuntu Community <accomplishments> <launchpad> <ubuntu> <Launchpad itself:New> < https://launchpad.net/bugs/942523 >14:43
deryckadeuring, abentley, rick_h_ -- sorry, almost done. let's hangout in 5.14:45
abentleyderyck: ack14:46
deryckadeuring, abentley, rick_h_  -- https://plus.google.com/hangouts/extras/talk.google.com/orange-standup14:52
sinzuijcsackett, do you have time to mumble15:18
czajkowskijtv: are you about?15:30
jtvczajkowski: I am15:30
czajkowskijtv: I dont fully understand the bug here. https://bugs.launchpad.net/launchpad/+bug/94264515:31
_mup_Bug #942645: Credit strings show up untranslated for kcm-gtk <Launchpad itself:New> < https://launchpad.net/bugs/942645 >15:31
jtvczajkowski: you know what "translation credit strings" are?15:31
czajkowskijtv: yes15:32
jtvLaunchpad pretends that it has translations for those, but actually those are generated on the fly.15:32
czajkowskiah ok15:33
jtvFor some reason, in this case, the Launchpad UI lapses in its pretense that it has translations for these strings.15:33
jtvIf “lapses” is the right word.15:34
jtvArgh.  Look at the computed percentages here!  That ought to be fixed: https://translations.launchpad.net/ubuntu/precise/+source/kcm-gtk/+pots/kcm-gtk/de/+details15:35
jtvczajkowski: if memory serves, we had some mechanism for preventing this — in which case this needs investigation for which unfortunately I do not have time.  But it's also possible that this was never solved, in which case there's probably a duplicate bug.15:36
czajkowskijtv: nods no worries shall investigate further15:37
czajkowskithank you for the help15:37
=== al-maisan is now known as almaisan-away
=== salgado is now known as salgado-lunch
=== danhg_ is now known as danhg
abentleyadeuring: I just noticed your lazr.jobrunner branch is lp:~adeuring/launchpad/lazr.jobrunner, not lp:~adeuring/lazr.jobrunner/foo.  Intentional?16:06
adeuringabentley: usual sloppyness...16:06
abentleyadeuring: Oh, okay.16:07
jcsackettsinzui: changes to enum branch pushed.16:23
sinzuijcsackett, r=me16:27
jcsackettsinzui: thanks.16:27
abentleyjcsackett: do you have a sec?16:48
jcsackettabentley: sure.16:48
jcsackettwhat's up?16:48
abentleyYou're a contributor to ec2 land, and I'm trying to harmonize it with lp-land.16:48
abentleyjcsackett: A difference is that ec2 land updates the commit message.  Do you find that useful?16:49
jcsackettabentley: you mean the bit about setting the [r=] tags?16:49
jcsackettor the commit message option you can pass it?16:49
abentleyjcsackett: I mean that it sets the [r=] tags etc on the merge proposal.16:50
jcsackettabentley: i find that useful, yes, in that i would never remember to set those tags myself.16:50
jcsackettabentley: i thought lp-land does the same thing, though ...16:50
abentleyjcsackett: You wouldn't have to set them yourself.16:51
abentleyjcsackett: It would already use them for submitting the proposal.16:51
abentleyjcsackett: It's just whether it rewrites them onto the merge proposal.16:51
jcsackettabentley: oh! no, i do not find that useful. i actually find it kind of odd that the MP gets tagged with that stuff outside the actual landing.16:52
jcsackettand if you have an error with an ec2 land and run it again, i have noticed you can end up with two sets of the [r=] tags. which is vexing.16:53
abentleyjcsackett: Really?  It was supposed to avoid duplication, but I guess it's buggy.16:53
abentleyjcsackett: Okay, I'll leave that functionality out.16:53
jcsackettabentley: ok.16:54
abentleyjcsackett: thanks.16:54
jcsackettabentley: you're welcome.16:54
=== deryck is now known as deryck[lunch]
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
lifelessczajkowski: hi; crazy bug yes.17:55
czajkowskilifeless: one for you then :)17:56
lifelessczajkowski: I'd just mark it low triaged17:58
czajkowskilifeless: ah so it may happen ?17:59
lifelessczajkowski: if someone were to add support, with appropriate security, and maintain it, we'd (probably) have no problem doing it, though we wouldn't do it ourselves17:59
lifelessprobability 0 that someone will17:59
lifelessmrevell as product manager can say 'no way never' if he likes :)18:00
lifelessI get to set the bar we need to reach when we do something, but for /what/ we do, thats up to gentle suasion!18:01
czajkowskigrand job18:01
=== deryck[lunch] is now known as deryck
lifelessczajkowski: you should never be concerned about triaging wrongly: the worst that can happen is someone can use the [formal] escalation process to request the thing be prioritised18:13
lifelessczajkowski: if its wrong in a simpler manner (e.g. is a regression / isn't or whatever, one of your peers will probably just fix it up)18:13
czajkowskilifeless: true, but with that I jst wondered would there be further discussion on the topic given the amount of blogging about the topic and other mails18:13
lifelessoh there probably will be18:13
* lifeless sadfaces18:14
james_wwould someone help us out and do a release of oops_datedir_repo please?18:23
james_wI don't want to have to depend on the branch in another oops module18:23
lifelesssinzui: hey18:24
sinzuihi lifeless18:25
lifelesssinzui: I am sure you will know this18:25
lifelesssinzui: why do I get messages 'foo has deactivated registry from bar' every day18:25
lifelessjames_w: sure18:25
lifelesssinzui: is it 'merge' or 'delete' ?18:25
sinzuiI was thinking about that this morning18:25
sinzuiIt is delete18:26
james_wlifeless, thanks18:26
lifelessI need to write code; this week has had no code writing and I'm getting so very very deprived18:26
lifelesssinzui: so for delete, I'm guessing that that person wasn't involved at all, but the notification happens automagically18:27
sinzuiThere is some step I want to avoid in the merge process when merging with ~registry. I do not know what case is.18:27
sinzuilifeless, they do, but something is out of order...18:27
lifelesssinzui: separately, there was a bug we found where an admin approved a membership but the notification claimed someone else did18:27
lifelesssinzui: I wonder if it could be the same thing18:27
sinzuiWe remove super and subteams when deleting. by the point of the merge...there should be no memberships that could trigger that email18:28
lifelesswell, a component of the same confusion18:28
lifelesssinzui: there is an owner18:28
lifelesssinzui: and presumably a creator. I wonder if we have fallback code18:28
lifelesscrazy cracy fallback code18:28
sinzuilifeless, the person we see in the email is the one that initiated the merge job that performs the delete18:29
lifelesssinzui: ok, so 'foo deleted team bar' is what the mail should say (and it should perhaps go to the old members only)18:29
sinzuilifeless, No18:29
* lifeless doth not understand18:30
sinzuiThere is a step missing. Only the owner user doing the action should get an email...but that is not the email we are seeing18:30
sinzuiI think the act of merge/delete queue *2* jobs18:30
sinzui1st job queues the membership changes which is the email we are seeing.18:31
lifelesssinzui: wouldn't it be confusing for the members to have the team disappear? That they were part of ?18:31
sinzui2nd job queue the merge. The merge happens first, so that the email go out and report ~registry was removed when the team was never a member18:32
lifelesssinzui: ah, I see, you're saying that yes the members get a notification (perhaps 'removed from team' which is a little misleading); and separately we purge it18:32
sinzuilifeless, I looked at the staging memberships of the people reported in the email. They are the team owners and the team is not related to ~registry. So the email really is about ~fnord was deactivated from ~pting18:34
lifelesssinzui: unless registry gets put in fpting first18:35
sinzuilifeless, this is the *very odd* part merge will not take place is any memberships exist. I assume team participation are indeed complete, so the emails should be impossible to send18:35
lifelessjames_w: done18:37
james_wthanks lifeless18:37
lifelessjames_w: can you do me a favour and close off any fix-committed bugs?18:37
james_wzero open bugs18:38
james_wplus moving LP to pymongo's bson and getting rid of the compatibility code is probably a good idea, given that the tests in datedir and amqp aren't checking against both implementations18:39
james_wone inconsistency I've found is that pymongo refuses to serialize things it doesn't know about18:39
james_wso you can't just put arbitrary objects in the oops18:40
james_wI don't know what the expected behaviour is there18:40
lifelessthat would be nice18:41
lifelesswe had a nasty bug where we tossed something bson didn't understand in18:41
lifelessand it just skipped it18:41
lifelessthen oops-tools barfed18:41
james_wskipped is clearly the wrong thing to do18:42
james_wI assumed it did repr or something18:42
lifelesssinzui: so, I think we should move all of _merge into the job18:49
lifelesssinzui: there could be a lot of work there, may need multiple transactions to avoid concurrency issues, and there is no UI except for 'yay I did it' or 'boo cannot be done'18:50
lifelesssinzui: (and, I think we should notify about deletes, but thats an orthogonal discussion)18:52
james_wthe options seem to be to repr every value, or try and bsonify every value individually and repr if it fails18:52
james_w(we're trying to add task *args and **kwargs to the oops, and we don't know the types up front)18:53
lifelessjames_w: I thought you said pymongo errors ?18:53
james_wit does18:53
james_wand I don't want it to18:53
lifelessjames_w: I think that is great18:53
james_wI want to put the data in without caring about the type in this case18:53
lifelesssafe_unicode them all ?18:54
lifelessI suspect that (with the exception of collections) is probably going to DWIM for you18:54
james_wwhere does safe_unicode live?18:55
james_wgood idea, /me switches18:57
=== almaisan-away is now known as al-maisan
rick_h_lifeless: ping19:22
rick_h_howdy, I'm trying to figure out your comment in https://code.launchpad.net/~rharding/launchpad/watch_jsbuild/+merge/9437919:22
rick_h_about both the make target and buidlout template?19:22
rick_h_can you elaborate for the buildout impaired what I've done wrong/correct way to fix?19:23
derycklifeless, hi.  I'll get in line behind rick_h_, but would love to get your review of a preload attrs for buglistings branch.19:23
rick_h_hah, my faster typing pays off19:24
deryckI let you win. ;)19:25
rick_h_I feel a monty python skit involving a black knight coming on19:25
lifelessderyck: url ?19:29
lifeless(OTP :P)19:30
derycklifeless, https://code.launchpad.net/~deryck/launchpad/preload-tags-for-buglistings-901122/+merge/95023 (and thanks!)19:30
lifelessrick_h_: I made that comment on the initial diff19:31
rick_h_lifeless: ok, cool19:31
lifelessrick_h_: it looks sane now, but perhaps s/foo.py.in/foo.in/19:31
rick_h_lifeless: ok19:31
lifeless(e.g. I don't think the language is relevant for an executable interface)19:32
rick_h_right, ok nop19:32
rick_h_np that is19:32
=== al-maisan is now known as almaisan-away
lifelessderyck: reviewed21:42
lifelesswith 2 comments (second coming in a sec)21:43
=== matsubara is now known as matsubara-afk
derycklifeless, ok, thanks, man.21:45
jcsackettsinzui: i will be a few minutes late to standup.21:52
thumperhi people21:53
thumperis launchpad still using windmill?21:53
lifelessdied in a fire21:56
james_wsomeone should put the smackdown on windmill on -tech21:57
derycklifeless, so I've got the last branch up for review, but you don't have to review it now.  I'll add the scaling test in that final branch.21:59
derycklifeless, but if you're curious: https://code.launchpad.net/~deryck/launchpad/adapt-badges-listing-item-901122/+merge/9506321:59
deryckthumper, we still have windmill tests lying around, with the hope they'll get ported to our new yuixhr framework.22:00
deryckthumper, but they haven't been running for some time.22:00
deryckif I ever need LOC credit for something I'm working on, I plan to delete them then. ;)22:01
thumperdo what is the new magic?22:03
deryckyeah, just plain ol' yui run headless with html5-browser.22:06
deryckI'm out guys.  catch you all later.22:06
sinzuiwgrant, https://code.launchpad.net/~sinzui/launchpad/mailman-archive-022:51

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!