/srv/irclogs.ubuntu.com/2011/02/10/#launchpad-dev.txt

mwhudsonugh, i'm sure apis are supposed to work in read-only mode...00:08
wgrantmwhudson: I'm seeing distribution['ubuntu'] failing.00:09
wgrantI never have before.00:09
wgrantAre you seeing something similar?00:09
james_w503s?00:09
wgrantYeah.00:10
james_wI've definitely seen them before00:10
wgrantFor some things, sure.00:10
james_wI've not tried to correlate it to anything00:10
wgrantBut never this.00:10
wgrantUgh.00:13
mwhudsonwee!00:14
wgrantc.l.d.oauth tries to use the master store.00:14
mwhudsoni wonder what's happening00:14
wgrantBut that hasn't changed lately.00:14
mwhudsonthe response is being generated by launchpad, it seems?00:15
=== mbarnett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: abentley | https://code.launchpad.net/launchpad-project/+activereviews
mwhudsonso there should be oopses00:15
mwhudsonmaybe?00:15
wgrantmwhudson: Unlikely.00:16
wgrantThat probably won't generate an OOPS.00:16
wgrantBut I can see the issue locally.00:16
mwhudsonah ok00:16
wgranthttp://paste.ubuntu.com/565226/00:16
wgrantThat's with login_anonymously00:17
gary_posterwgrant: looking00:34
wgrantgary_poster: Thanks. Although it's less critical now, since the rollout failed so we'll probably have to cowboy.00:35
gary_posterwgrant, ah :-/00:35
gary_posterwgrant, qa instructions already exist for the first: last para of description of https://code.launchpad.net/~gary/launchpad/bug548/+merge/4885000:37
gary_posterlooking at the other one...00:37
gary_posterI also did the same for the other one, wgrant (https://code.launchpad.net/~gary/launchpad/bug713392/+merge/48939)00:38
wgrantgary_poster: Ah, sorry. Just about nobody does, so I stopped checking there a while ago.00:38
gary_posterwgrant, np, understood :-) .  I'm not consistent either, but I try to do that00:39
=== Ursinha is now known as Ursinha-afk
thumperso...01:21
thumperwhat's the status with writing new windmill tests?01:21
thumperwe are supposed to?01:21
pooliespm, i sent bug mail to lp during the rollout and it does not yet seem to have any effect....01:40
poolienm, they have come through01:52
* thumper nipping out to what the girls' first touch game of the season02:12
LPCIBotProject devel build (429): STILL FAILING in 6 hr 6 min: https://hudson.wedontsleep.org/job/devel/429/02:15
LPCIBot* Launchpad Patch Queue Manager: [r=jtv][bug=714820] Ignore malformed responses when probing remote02:15
LPCIBotBugzilla instances for XML-RPC capabilities.02:15
LPCIBot* Launchpad Patch Queue Manager: [r=adeuring][bug=181368] Gut BinaryPackageFilePublishing view.02:15
LPCIBot* Launchpad Patch Queue Manager: [r=jtv][bug=715659] Use zope.exceptions to format logged exceptions02:15
LPCIBotso that __traceback_info__ is reported.02:15
LPCIBot* Launchpad Patch Queue Manager: [r=leonardr][bug=715116] Add a big informational notice on the PPA02:15
LPCIBotpages if publishing is disabled.02:15
LPCIBot* Launchpad Patch Queue Manager: [r=lifeless][bug=713234] Remove duplicated queries issued when02:15
LPCIBotcalling ArchiveView.latest_updates02:15
lifelesshttps://code.launchpad.net/~lifeless/launchpad/bug-704446/+merge/4916702:36
pooliehi lifeless02:51
lifelesshi poolie02:52
pooliedo you have any idea whether https://bugs.launchpad.net/launchpad/+bug/716175 would be shallow?02:54
_mup_Bug #716175: api calls fail while launchpad is readonly <Launchpad itself:Triaged> < https://launchpad.net/bugs/716175 >02:54
lifelesspoolie: I suspect its not, but without an oops can't really confirm (could simulate locally I guess)03:03
lifelesspoolie: but I'm not hugely interested in fixing readonly mode, I don't think its a particularly good use of engineering time03:04
lifelesspoolie: I'm going to reply to jam about loggerhead and try and clear up the confusion03:05
lifelesspoolie: however I have a few fires to deal with first today.03:05
lifelesshuwshimi: ping03:14
huwshimilifeless: Hey03:14
lifelesshuwshimi: contrast this03:14
lifelesshttp://bazaar.launchpad.net/~jameinel/loggerhead/merge_pqm_updates/files03:14
lifelessand03:14
lifelesshttp://bazaar.launchpad.net/~jameinel/loggerhead/merge_pqm_updates/changes03:14
lifelessnote the former has a 'help' tab03:14
lifelessits in the lp loggerhead branch only03:14
lifelessI think it was added during the drive to have 'help everywhere' in LP03:14
lifelessbut if you click on it03:14
lifelessit takes you to a sad panda wiki page03:15
lifelessI'm seeking a second opinion on just discarding that noddy tab03:15
wgrantlifeless: It's easy to replicate locally.03:15
wgrantI pastebinned the traceback earlier.03:15
wgrantI don't see how it could have ever worked, but it apparently did.d03:15
lifelesswgrant: I was not connected to irc :)03:15
wgrantRight.03:15
lifelesswgrant: feel free to annotate the bug ;)03:15
poolielifeless, yeah, much better to just never go into  readonly mode03:18
pooliehearty +1 from me to removing the help tab03:18
huwshimilifeless: Help that is not helpful is useless. It appears to me that that help is not really helpful. You could either remove the tab or provide some useful help.03:18
pooliethose little 'help' drawers were such an awful idea03:19
poolielifeless, in that case i might bump 716175 down to low03:19
lifelesspoolie: I have done so03:19
huwshimiUseless help just results in what I like to call "help page rage".03:20
huwshimiI once worked on a massive application where all the help documentation for every form field was "This is the Foo field". The documentation guy didn't understand why this was a bad thing.03:23
pooliehuwshimi, that's what we had03:24
pooliei cannot think of a better way to train people to never click on things labelled Help03:24
huwshimipoolie: Ouch.03:24
poolieif we'd kept the same mechanism but only used it when we had something important to say, it would have been half decent03:24
pooliebut better to just put it inline if it's really important and you can't think of a way to make it obvious03:25
huwshimiIf there is a bug that has been fixed by another bug do I change it to "Fix released"?03:27
poolieyes, or mark it as dupe, or as invalid03:28
pooliewhatever you think best expresses the situation or is easiest03:28
huwshimipoolie: Cheers03:28
lifelessabentley: are you still OCR?03:35
thumperlifeless: somewhat late for abentley don't you think?03:37
=== thumper changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
lifelessyes :)03:37
* thumper still feels strangely protective03:38
poolie:)03:39
huwshimiDoes anyone know what research might be alluded to in this comment? https://bugs.launchpad.net/launchpad/+bug/490058/comments/703:42
_mup_Bug #490058: Launchpad shouldn't present itself as a database <lp-web> <Launchpad itself:Triaged> < https://launchpad.net/bugs/490058 >03:42
poolieshe has done some user testing sessions03:42
pooliemm03:42
pooliei don't think the title of that bug is very accurate03:43
poolieit should be something more like "Launchpad home page should show cool stuff that's happening"03:43
poolie"not presented as a database" would be more like getting away from table-type views03:43
huwshimipoolie: Hmm.. I should try and get her findings03:46
poolieshe'd be a good person for you to know03:47
wgrantThere were a couple of surveys last year too.03:47
wgrantI never saw anything from them; were they even distributed internally?03:47
pooliemrevell posted something, but not very prominently03:47
pooliemm03:47
spmfyi. am doing the stabby dance on the builtbot master to remove the living dead problem it's caused.03:48
poolieit's kind of connected to the timeline view concept03:48
thumperw00t: Committed revision 12345.04:00
thumpernice number04:00
LPCIBotProject db-devel build (354): STILL FAILING in 5 hr 20 min: https://hudson.wedontsleep.org/job/db-devel/354/04:00
LPCIBotLaunchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 1234804:00
LPCIBotincluded.04:00
huwshimiUgh, what am I doing wrong. I updated download-cache and did an update of devel and merged it into my branch and now getting this error: http://paste.ubuntu.com/565245/ . I know there was an update to YUI that removed a whole bunch of files. Is it possible there's something else I need to update?04:54
thumperhuwshimi: try a "make clean build"04:58
huwshimithumper: Trying but really weird things are going on05:02
huwshimithumper: That's fixed it. Thanks mate05:08
huwshimiwgrant: Did you end up finding any of those facet images?05:14
wgranthuwshimi: They should be in the history of lib/canonical/launchpad/icing.05:16
wgrantRemoved in the last 18 months.05:16
huwshimiwgrant: Cheers05:17
huwshimiwgrant: Would you use bzr explorer or something to find these files?05:35
wgranthuwshimi: Try checking the directory around r10000 or so. Just create a new branch and revert to then.05:40
huwshimiwgrant: ah right ok05:40
lifelessthumper: ping05:44
lifelessthumper: I know its paste your EOD, but perhaps you could mentor Ian's review on https://code.launchpad.net/~lifeless/launchpad/bug-704446/+merge/4916705:45
wallyworldlifeless: that sql bind params review - you approved it but you also need to claim it so it can be landed https://code.launchpad.net/~wallyworld/launchpad/profile-sql-bind-values/+merge/4887205:54
wallyworldpretty please :-)05:54
lifelessroger05:56
wgranthuwshimi: Did you find the images?06:32
huwshimiwgrant: yeah got them. I remember then now06:32
wgrantSomeone went through and cleaned up c/l/icing a while back.06:33
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
=== almaisan-away is now known as al-maisan
huwshimiok time to head off07:05
lifelessstub: could I have a review on https://code.launchpad.net/~lifeless/launchpad/bug-704446/+merge/4916707:19
stublifeless: What is the "+                    AND BugMessage.index > 0" for?07:29
stubend of the diff07:29
lifelesstunes the query for 'commented on a bug'07:33
stuboic. 0 is first comment == description07:34
lifelesshttps://bugs.launchpad.net/launchpad/+bug/421901/comments/1007:34
_mup_Bug #421901: Person:+bugs timeouts <lp-bugs> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/421901 >07:34
stubapproved07:36
lifelessthanks07:37
lifelesshmm07:43
lifelessbringing back fti columns seems entirely pointless07:43
lifelessjkakar: is there some way to tell storm /not/ to load a column from the db ?07:43
lifelessstub: can you compare the execution time for these on prod ? http://pastebin.com/raw.php?i=Lib9DXHz07:45
stublifeless: second 11 seconds, first 12.7 seconds07:54
lifelessgrr, really?07:55
lifelessTime: 1355.438 ms on staging07:55
lifeless[qa]staging07:55
stubhttp://paste.ubuntu.com/565267/07:58
stublifeless: qastaging is patched up remember07:59
lifelessstub: this doesn't depend on any db patches07:59
lifelessoh...07:59
lifelesswe haven't indexed BugMessage.index on qastaging yet.08:00
lifelessstub: can I get a plan of that second query, once hot ?08:00
lifelessstub: also please try http://pastebin.com/TeHbFEwD08:02
stubThat was the second query. This is the plan of the first one: http://paste.ubuntu.com/565268/08:02
lifelessseq scan on bugmessage08:03
stuburg... that pastebin includes linenumbers in c&p08:03
lifelesshttp://pastebin.com/raw.php?i=TeHbFEwD08:03
lifelessstub: click on 'raw'08:04
stubI suspect there are a lot of bugs with only a single comment08:05
stubreverse that...08:06
lifelessindeed08:06
stubWe have an index on (bug,index)08:07
stubhttp://pastebin.com/HMY6QqJd btw08:08
lifelesswell, 8 seconds is better than 1208:08
lifelessI think I'll unlink the bug fro mthis branch, its going to need more work08:09
lifelessstub: do you think a separate index on (index,) is needed, or perhaps two indices (bug,) and (index,) and drop the composite index?08:13
stubI'll test a separate index in a tic. Looking at some alternative querys atm.08:13
lifelessthe one in https://bugs.launchpad.net/launchpad/+bug/421901/comments/5 I think you tried already for me, but was the best I found when experimenting last week08:15
_mup_Bug #421901: Person:+bugs timeouts <lp-bugs> <timeout> <Launchpad itself:Triaged by lifeless> < https://launchpad.net/bugs/421901 >08:15
stubdifferent indexes don't seem to help on BugMessage.index (or message,index)08:16
stubThere are 177k messages from that user...08:17
lifelessprobably why it shows up in the timeouts08:18
stubbut I don't see a sane way to execute the query. Traversing from bug -> Message means materializing all the bugtasks with those statuses, traversing from message -> bug means materializing all 177k messages...08:20
lifelessperhaps if owner was on BugMessage08:22
stubSo for that data, I'm not sure if we can beat explain analyze08:22
stubSELECT DISTINCT ON (BugTask.id) BugTask.*08:22
stubFROM BugTask, Bug, BugMessage, Message08:22
stubWHERE08:22
stub    Bug.id = BugTask.bug08:22
stub    AND Bug.id = BugMessage.bug08:22
stub    AND BugMessage.message = Message.id08:22
stub    AND BugTask.status in (10, 15, 20, 21, 22, 25)08:22
stub    AND Bug.duplicateof IS NULL08:22
stub    AND Bug.private = FALSE08:22
stub    AND BugMessage.index > 008:22
stub    AND Message.owner = 93112908:22
stubORDER BY BugTask.id;08:22
stubYer - we would have to denormalize08:23
stubOr we just store a table of commenters on bugs08:23
lifelessdistinct on is just more efficient, right ?08:23
stubit's a little more efficient than DISTINCT BugTask.*08:24
lifelessyeah08:24
stubBut probably not noticebly faster08:24
lifelessstub: how many seconds is that ^08:25
lifelessI'm not actually sure why BugMessage is separate to Message08:26
stub8.7 seconds08:26
lifelesshmm, I had a 6 second one08:26
LPCIBotYippie, build fixed!08:26
LPCIBotProject devel build (430): FIXED in 6 hr 11 min: https://hudson.wedontsleep.org/job/devel/430/08:26
LPCIBot* Launchpad Patch Queue Manager: [r=flacoste][bug=716109] Update Launchpad to r202 of lazr-js,08:26
LPCIBotwhich includes an update to YUI 3.3.08:26
LPCIBot* Launchpad Patch Queue Manager: [r=jtv] [ui=none][bug=316694][incr] Take advantage of web_link when08:26
LPCIBotchanging the URL in the browser bar,08:26
LPCIBotrather than hacking self_link into web_link.08:26
LPCIBot* Launchpad Patch Queue Manager: [r=jtv][bug=705657] Fix mirror-prober.sh wrapper script to export the08:26
LPCIBotcorrect config. (Fix bug 705657)08:26
LPCIBot* Launchpad Patch Queue Manager: [r=allenap][bug=710603] The new direct subscription overlay allows08:26
_mup_Bug #705657: cronscripts using production/launchpad-lazr.conf config file with REPORTIFSEEN prefix <Launchpad itself:In Progress by matsubara> < https://launchpad.net/bugs/705657 >08:26
LPCIBotsubscribing,08:26
LPCIBotunsubscribing and editing subscriptions and is available to the08:27
stubBugMessage contains bug message specific data. Message contains data relevant to all messages (answer comments etc.)08:27
LPCIBot~malone-alpha team.08:27
stubSpambot08:27
lifelesshttp://pastebin.com/raw.php?i=UdFwWx3308:27
stub10s now :-) But the estimated cost is lower, and I think that is what we need to go by here.08:28
lifelessstub: the distinct on is 10 seconds?08:31
lifelessstub: whats the http://pastebin.com/raw.php?i=UdFwWx33 one ?08:31
stubThey are all around the same range in seconds. The distinct on has a cost of 391000, your variant 325000.08:32
lifelessok08:32
lifelessso I'll code up my variant next week08:32
lifelesshow should we go about assessing the right denormalisation (e.g. owner in bugmessage; fold bugmessage and message togeter; have a table listing commentors)08:32
stubThat last one has pretty much the same cost08:33
stubEven with denormalizing, we are going to have similar issues - materializing 170k rows. Lots of messages there.08:34
stubDoing a count of 'messages owner by 931... and index > 0' has a cost of 280k08:36
lifelessstub: if we had (bugtask.id, commentator), and updated it when adding a bugtask ?08:36
stubBug,commentator makes more sense08:36
lifelessstub: we filter on bugtask status, and we search and report rows in bugtask; thats why i thought bugtask would make more sense08:37
lifeless-> what do I have wrong08:37
stubThat user has commented on 43k distinct bugs08:38
thumperstub: is that user lifeless?08:38
stubPeople comment on bugs, not on bugtasks.08:38
lifelessapport08:39
lifelessactually not apport08:39
lifeless!janitor08:39
stubjanitor08:39
stubwho I never wanted as an actual user.08:39
lifelessapport has made the most comment08:40
lifelessand seb128 the next least08:40
lifelesswhat is ~janitor for?08:40
stubI think expiry and stuff08:40
stubits the launchpad system robot anyway08:41
stubso, this query makes no sense for their outliers anyway. Are we tackling the wrong problem?08:41
stubc/their/these08:41
lifelessseb128 is an outlier :P08:42
lifeless[yes, he is, I'm trolling]08:42
lifelessso there is a bug saying that the default Person:+bugs view is useless because it includes these bugs08:42
stubheh... so we can't just turn this off for robots (unless we just make it official and designate seb a robot) - he has almost as many messages in the system as the janitor.08:43
lifelessI know :)08:44
adeuringgood morning08:45
mrevellGuten morgen09:18
=== jtv-afk is now known as jtv
lifelessjml: oh hai09:56
jmllifeless: hello. I got your message.09:57
lifelessgreat09:57
lifelesssince i'm still conscious, and thinking work; do you want to chat now?09:57
jmllifeless: definitely would like to talk, but not now.09:57
lifelessok09:57
Ursinhagood morning10:07
bigjoolsmorning Ursinha, up early again10:19
Ursinhabigjools, trying to make it a habit10:20
bigjoolslifeless: totally loving the render time data at the top of the page10:38
adeuringhenninge: fancy a small review? https://code.launchpad.net/~adeuring/launchpad/bug-702468/+merge/4920510:49
henningema kuckn10:50
=== matsubara-afk is now known as matsubara
henningeI am getting this error when doing "make schema" http://paste.ubuntu.com/565319/11:02
henningeAnybody can tell me what is wrong?11:02
wgranthenninge: Upgrade lp-dev-deps.11:02
henningehm11:02
wgrantYou don't have postgresql-8.4-debversion installed.11:02
henningewgrant: I see, thanks.11:04
stubPlease send more Internets. Thailand has run out.11:04
stubI'm rather surprised the scripts get that far when the debversion package isn't installed.11:05
=== al-maisan is now known as almaisan-away
henningeWhy does it keep these packages back? Anything I might have done? http://paste.ubuntu.com/565321/11:07
* henninge realizes this is probably turning into Ubuntu support.11:07
wgrantdist-upgrade instead.11:08
wgrantupgrade will try to avoid installing new packages.11:08
henningeI only get this on this machine, though. My laptop upgraded fine (using update-manager).11:08
wgrantRight, update-manager does more than just apt-get upgrade.11:08
henningecurrently update-manager gives me the "Not all updates can be installed. Run a partial update to install as many updates as possible." message and offers to "Partial upgrade". Is that a dist-upgrade?11:10
=== Ursinha-afk is now known as Ursinha
wgranthenninge: Hmm, that's not good.11:12
wgrantWhat does apt-get dist-upgrade say? It sounds like it may want to remove stuff, which would be bad.11:12
wgrantOr there is some version skew.11:13
henningehttp://paste.ubuntu.com/565323/11:13
henningeThis looks about right, though.11:13
wgrantThat does.11:13
wgrantDo it.11:13
henningeDoing ...11:13
henninge... done.11:16
henningeNow it wants to reboot, of course. (kernel update)11:17
henningeBut update-manager is quiet about partial updates.11:17
henningewgrant: tanks a lot!11:17
wgrantGreat.11:17
wgrantHopefully make will be happy now.11:17
bigjoolsjtv: does these queries look like they'll do the same thing? http://pastebin.ubuntu.com/565325/11:20
bigjoolss/does/do/11:20
jtvbigjools: looking…11:20
jtvbigjools: what's the %s in the COALESCE stand for?11:22
bigjoolsjtv: False11:22
jtvbigjools: NULL OR TRUE equals TRUE, so no need for that I think.11:23
bigjoolsjtv: see lib/lp/buildmaster/model/buildfarmjob.py -> getBuildsForBuilder()11:23
bigjoolsfor the storm code that generated that11:24
bigjoolsjtv: null equals true in PG?11:25
jtvNo!11:25
bigjools:)11:25
jtvBut unusually for binary operators, OR has the property that (NULL OR TRUE) evaluates as TRUE.11:25
bigjoolsthe second query is missing the coalesce11:26
jtvRather than to NULL, as one might otherwise expect.11:26
bigjoolsthe second is Rob's attempt to optimise the first11:26
bigjoolsI'm not convinced it's right11:26
jtvbigjools: I don't see the "TeamParticipation.person = 2" reflected in the new query.11:27
jtvOh wait, found it11:27
jtvIt's another % in the old query.11:28
bigjoolsyeah11:28
jtvbigjools: but all the nastiness, complexity-wise, is in the question of which foreign keys may be null.11:30
bigjoolsthe nastiness is in the fact that not all build farm jobs will have a related package build11:31
bigjoolsie TTBJs11:31
jtvI get an impression that this won't show jobs for public archives that the user is not an owner of.11:31
jtvNo, that's not right.11:33
jtvBut the other way around?  Will this respect privacy?11:33
jtvSay that Archive.private = TRUE and there's no matching TeamParticipation.11:34
bigjoolslooks wrong11:36
jtvThere'll be no matching (PackageBuild, Archive, TeamParticipation), since those are inner-joined together on the conditions for visibility to the given user.11:36
jtvWhich _sounds_ like what we want,11:36
jtvexcept then that tuple is being outer-joined to BuildFarmJob.11:36
bigjoolsit's making my head explode11:36
jtvWhich means that if there is no (PackageBuild, Archive, TeamParticipation) that proves visibility to the user,11:36
jtvthe outer join will just make up a (PackageBuild, Archive, TeamParticipation) of all nulls.11:37
jtvAnd the WHERE clause doesn't check for that, so AFAICT it'll happily return a BFJ for an archive that the user isn't supposed to see.11:38
jtvThe important thing there is that with an outer join, a join condition is crucially different from a "where" condition in that the latter filters out tuples from the entire join, whereas the former may just result in a tuple of nulls where nothing matches.11:40
bigjoolsthe problem with the original query is that it's doing a seq scan on archive11:40
jtvUse Union.11:40
bigjoolsright11:40
jtvThis is the wide-shallow/narrow-deep pattern that we keep seeing with privacy.11:41
bigjoolsI'll ditch the left joins11:41
jtvJust write out separate cases for public and private archives, then factor for identical parts that you can share.11:41
jtvWhat should happen if there's no PackageBuild?  Should be shown anyway?11:42
bigjoolsyes11:42
bigjoolsthere's no privacy outside of anything that has a packagebuild11:42
jtvSo for public archives you get something like…11:42
jtvSELECT DISTINCT BuildFarmJob.*11:42
jtvLEFT JOIN PackageBuild ON PackageBuild.build_farm_job = BuildFarmJob.id11:43
jtvLEFT JOIN Archive ON Archive.id = PackageBuild.archive11:43
jtvWHERE Archive.private IS FALSE11:44
jtvUNION11:44
jtvSELECT DISTINCT BuildFarmJob.*11:44
jtvLEFT JOIN PackageBuild ON PackageBuild.build_farm_job = BuildFarmJob.id11:44
jtvLEFT JOIN Archive ON Archive.id = PackageBuild.archive11:45
jtvLEFT JOIN TeamParticipation ON TeamParticipation.team = Archive.owner11:45
jtvWHERE Archive.private IS TRUE AND TeamParticipation.person = %s;11:46
jtvIf there are a lot of BFJs without PBs, you could go even further:11:46
jtvSELECT DISTINCT BuildFarmJob.*11:47
jtvLEFT JOIN PackageBuild ON PackageBuild.build_farm_job = BuildFarmJob.id11:47
jtvWHERE PackageBuild.id IS NULL11:47
jtvUNION11:47
jtvSELECT DISTINCT BuildFarmJob.*11:47
jtvFROM BuildFarmJob  -- I've been forgetting these lines, haven't I?11:48
jtvJOIN PackageBuild ON PackageBuild.build_farm_job = BuildFarmJob.id11:48
bigjoolsI think you need to paste11:48
jtvJOIN Archive ON Archive.id = PackageBuild.archive11:48
jtvOK, OK, I'll paste. :)11:48
bigjoolsI can't follow this here11:49
jtvbigjools: https://pastebin.canonical.com/43093/11:54
jtvAhhh, there's the LIMIT/OFFSET too isn't it?11:55
jtvNeed to work that in as well.11:55
jtvWe can use that to solve the remaining optimization problems.  Hang on.11:55
bigjoolsjtv: don't worry about that, it's the batchnav11:56
jtvDoesn't mean we shouldn't worry about it.  It's actually most likely the single biggest factor in this problem.11:56
bigjoolsthe problem is that a slow query is issued twice, once to count, the next time with the limit/offset11:56
jtvBut part of the problem is that slicing happens all the way at the end, and the query planner currently has no way to push that down into the nodes for preliminary optimization.11:57
jtvHence the seq scan on Archive.11:57
bigjoolswe need ordering as well11:57
jtvYes.11:58
bigjoolsand, is the distinct honoured across unions?11:58
bigjoolsor did you guarantee they don't intersect?11:58
jtvYes, I required that "Archive.private = TRUE" in the last query so that makes this a partitioning.11:58
jtvNo overlap.11:58
bigjoolsjtv: the queries are also missing a BuildFarmJob.builder = XXX12:04
jtvah12:04
bigjoolsbut I shall experiment and see what else I can do!12:04
jtvbigjools: I'm not done either :)12:04
deryckMorning, folks.12:05
jtvhi deryck12:05
bigjoolshowdy deryck12:05
bigjoolsjtv: explain analyze on your new query shows a seq scan on archive still12:09
jtvbigjools: not entirely surprising.12:09
bigjoolsbecause there's no index on private12:09
jtvWell that's why I said index it.  :-)12:10
bigjools:)12:10
jtvbigjools: somewhat elaborate attempt here… https://pastebin.canonical.com/43094/12:13
bigjoolsjtv: why order each inner query and then again in the outer?12:14
jtvbigjools: to shortcut their execution when the limit is reached.12:14
bigjoolsorder and limit in fact12:14
bigjoolsok12:14
bigjoolsI don't know how we can control that in the batchnav12:14
jtvThe backend may be smart enough to do that for itself, but not sure.12:14
bigjoolsone can try12:15
jtvI've seen 8.4 backends push the ORDER BY down into sub-plans, so maybe we can leave that out.12:15
bigjoolswell I can't do it in our current infrastructure anyway12:15
jtvOK, I'll drop that bit.12:16
jtvBTW I've been assuming that by far most archives are public… is that correct?12:16
bigjoolsyep12:17
bigjools3 seq scans in that query12:18
bigjoolsbuildfarmjob, which is ok12:18
bigjoolspackagebuild12:18
jtv2 on Archive?12:18
bigjoolsand archive12:18
jtvOh12:18
bigjoolsthe good news is that it seems to return the right number of rows :)12:18
jtvWe could try the TDD approach: write the absolute minimum that satisfies our tests...  SELECT %(expected_number_of_rows)d12:19
jtvIs this less elaborate version any worse?  https://pastebin.canonical.com/43095/12:19
bigjoolsassuming the tests are good, yes12:19
bigjoolsjtv: same sort of performane12:21
bigjoolsperformance12:21
bigjoolssame seq scans12:21
bigjoolswe definitely need an index on archive.private12:22
bigjoolsit would help elsewhere12:22
jtvOr in this last version of the query, one on (private, owner)12:23
bigjoolsyeah12:23
jtvbigjools: can you see which of the two parts the packagebuild query is coming from?12:25
bigjoolsthe seq scan?12:25
jtvSorry, yes.12:26
bigjoolsthe second I think12:26
bigjoolsthere's a hash cond for archive.id above it12:27
jtvCould you paste me the plan?12:27
bigjoolsso prob need an index on packagebuild.archive12:27
bigjoolswhich query? :)12:28
bigjoolsthey're all similar but ...12:28
jtvhttps://pastebin.canonical.com/43095/12:29
jtvQuite possible that we need that index, yes, or on (archive, buildfarmjob) or vice versa.12:30
jtvThere's a patch now for hypothetical indexes in postgres.  That'd be cool to have.12:30
bigjoolshttp://pastebin.ubuntu.com/565348/12:30
bigjoolsyes, you mentioned that, it would be very useful12:31
jtvDoesn't create real indexes, but lets you say "EXPLAIN HYPOTHETICAL" and it'll explain your query as it would do it if the hypothetical indexes were real.12:31
bigjoolsso you do CREATE INDEX HYPOTHETICAL ... ?12:31
bigjoolshmmm so packagebuild already has indexes on archive and BFJ12:32
jtvYes, like that12:35
jtvA unified index may help.12:35
jtvSo one on both columns at the same time.12:35
bigjoolsis that hypothetical index stuff packaged?12:35
jtvNo12:35
jtvIt's a patch.12:35
bigjoolsurgh12:35
jtvBut there's a PostgreSQL eXtension Network coming.12:36
bigjoolsso, I am seeing the original query run quicker than your new ones right now12:36
jtvheh12:36
bigjoolsI suspect it's cause it only has one seq scan12:36
jtvThe seq scan on archive doesn't look at all costly.12:36
bigjoolsrows=2359612:37
jtvbigjools: oh!  And have you tried indexing on the sort order?12:39
bigjoolsyeah that adds 2 seconds alone, good idea12:39
jtvIndexing slows it down?12:39
bigjoolsremoving the ORDER BY speeds it up by 2 seconds12:39
bigjoolsI'll add the index on dogfood and see what happens12:40
=== almaisan-away is now known as al-maisan
jtvAnother big cost seems to be the PackageBuild→BFJ join.12:43
* jtv slowly figures out that bigjools wasn't being sarcastic with the "that adds 2 seconds, good idea" :)12:44
bigjoolsheh12:44
bigjoolsjtv: actually date_finished is already indexed12:47
jtvbigjools: but (date_finished, id)?12:47
bigjoolsno, but how often will date_finished need a tie breaker to ID?12:48
jtvIf the sort order is completely covered, that may open up new plans.12:48
bigjoolsindex on private makes bugger all difference12:48
jtv:(12:48
jtvWell, scanning Archive wasn't that costly.12:49
bigjoolsno, I mean there's still a seq scan12:49
bigjoolswhy is it scanning :/12:49
jtvA seq scan is the most efficient thing to do unless it's clear that only a surprisingly small percentage of the table will actually be needed.12:50
jtvThink disk seeks: you can read quite a lot of data in a seq scan in the same time that it might take to seek to the next table record that an index identifies as matching.12:50
bigjoolsindeed12:51
jtv(Bearing in mind that the index is in key order, not tuple storage order, so likely to be very random access patterns)12:51
jtvThat's also why we have bitmap scans.12:52
jtvbigjools: hang on, I just stumbled onto a major simplification12:54
bigjoolsadding an index on (date_finished, id) makes no difference :/12:54
=== matsubara is now known as matsubara-brb
jtvbigjools: boo13:01
jtvBut here's another version.  I'd guess it was slower, but knowing trumps guessing: https://pastebin.canonical.com/43096/13:01
bigjoolsso far, a lot slower ....13:04
jtvbigjools: that's fine; it's just a stepping stone to https://pastebin.canonical.com/43097/13:08
jtvOh, that needs a DISTINCT obviously13:08
jtvLike so: https://pastebin.canonical.com/43098/13:09
bigjoolsoooooo quick13:09
jtvThen presumably it's incorrect in some way, but still nice to see a bit of speed.  :-)13:09
bigjoolsnope, looks fine!13:09
jtvI suspect the UNION is a very effective optimization barrier.13:10
bigjoolsreturns correct number of rows13:10
bigjoolsdown to 2 seconds from 913:10
jtvThis makes it possible for the backend to materialize "archives this user is an owner of," which just has to be fast because so few records will be involved.13:10
bigjoolsit's quicksorting in memory now13:10
bigjoolswas using external merge on disk last time13:10
jtvThe joins can't have helped.13:11
bigjools"Disk: 142912kB" - yes :)13:11
jtvThe key to this is that the Archive seq scan is actually relatively cheap.13:11
bigjoolsit is13:11
bigjoolsactual time=0.011..21.49413:11
jtvSo I'm materializing one, as the first part of the union!13:11
jtvThat's even cheaper than previously, for some reason.13:12
bigjoolsbizarro13:12
jtvThere was just no way to get around a seq scan on archive, unless we managed to cram enormous amounts of selectivity into the PBs first—and even then.13:12
jtvI suspect that selectivity just isn't there.13:13
jtvWhich can't have helped the size of the ultimate sort buffers either, for what it's worth.13:13
jtvAlso, the PackageBuild join was very slow, and an outer join didn't look very different from an inner one, so I hoisted it out of the subquery.13:14
jtvThat way it gets done only once.13:14
bigjoolsthanks jtv, I'll Stormify this and run the tests against it13:14
bigjoolsnow, I need food13:14
jtvOK, glad I could help.13:14
bigjoolsjtv: you certainly did13:16
jtv:)13:16
gary_posterbo hoo, no reviewer13:22
gary_posterboo, even13:22
=== matsubara-brb is now known as matsubara
=== lionel__ is now known as lionel
=== jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: jcsackett | https://code.launchpad.net/launchpad-project/+activereviews
jcsackettgary_poster: looking for a review for https://code.launchpad.net/~gary/launchpad/bug713382/+merge/49145 ?13:32
gary_posteryes, jcsackett ! :-)13:32
* jcsackett is looking at it now.13:33
gary_posterthank you13:35
jcsackettgary_poster: in bug.py, what's the reason for working with "temp_recipients" instead of on "recipients"?13:43
jcsackettnm, reread proposal.13:43
gary_poster:-) l13:43
gary_posterk13:43
* jcsackett gets more coffee while reviewing.13:43
gary_posterm13:43
gary_postern13:43
deryckhenninge, we see you connect, but never hear you14:00
jcsackettgary: r=me. i've requested follow up from sinzui. let me know if that becomes a blocker and i'll hunt him or someone else down.14:06
jcsackettgary_poster ^14:06
sinzuiI can review it now14:07
jcsackettah, i missed that you had logged on, sinzui. :-)14:07
gary_posterjcsackett: many thanks.  will be back in afternoon14:13
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
abentleyjcsackett, thanks for your review.  You'll see that all the items I've put up for review build toward the goal of merging translations using a job.14:27
jcsackettabentley: i am indeed noticing that trend.14:27
abentleyIf you have any questions, I'm happy to answer 'em here.14:28
henningeadeuring: Two things about your branch.14:34
adeuringhenninge: yes?14:34
henninge1. I am pretty sure this is only supposed to work one way.14:34
henningei.e. first "upstream" translation overwrites "Ubuntu" translation but not the other way round.14:35
adeuringhenninge: right, but this is guarded by share_with_other_side14:35
adeuringthis flag is always true for product and projectgroups14:36
adeuringand must be expicitly enabled for SPs14:36
adeuringso I think we don't need extra guards14:36
henningehm14:37
henningeWhere is the flag "always true"?14:37
henningeIn the policy?14:37
adeuringhenninge: right14:38
henningeok, I see what you mean but AFAICT this is not the what you want.14:44
henninges/the//14:44
henningeadeuring: ^14:44
adeuringhenninge: well, I think we want upstream translations propagated to ubuntu, and that's done by share_with_other_side == True for upstreams14:45
adeuringOr am I missing something?14:46
LPCIBotYippie, build fixed!14:48
LPCIBotProject db-devel build (355): FIXED in 6 hr 1 min: https://hudson.wedontsleep.org/job/db-devel/355/14:48
henningeadeuring: no, they are only to propagate if the Ubuntu translation is either empty or identical to the upstream one.14:50
henningeadeuring: if they are different, they stay different.14:51
adeuringhenninge: well, the bug says "In the old model, providing the first upstream translation would have overwritten the Ubuntu translation, too. "14:52
henningeadeuring: yes, that is the exception to the rule I just described and which is missing from the code.14:52
henningeand which you implemented correctly.14:52
henningeBut it does not work the other way round.14:53
henningeThe exception, I mean.14:53
adeuringhenninge: ah, so you think thatg we should restrict this feature explicitly for the flow upstream -> ubuntu?14:53
henningeexactly! ;-)14:53
adeuringeven if sharing the other way round is enabled?14:54
henningeyes14:54
henningeThe reasoning for this exception goes like this:14:54
henningeWe import templates and translations from upstream, but there are no (or little ) translations yet.14:55
henningeThen Ubuntu translators go to work and start translating.14:55
henningeThe upstream translators get going, too, and after a while we get updated upstream translations.14:55
henningeWithout this exception, these upstream translations would not show up in Ubuntu.14:56
henningeWe have been getting bad press about how "Ubuntu ignores upstream translations214:56
henninge".14:56
henningeBut really, Ubuntu translations should be fed back to upstream anyway, but that is not something we can solve by code.14:57
adeuringright. But how likely is the situation that the other "sharing variant" does something wrong? I'd prefer to not make the code even more complex ;)14:57
adeuringI mean, somebody must explicitly enable this sort of "sharing flow"14:58
henningeadeuring: It needs to retain the old behavior, though. It's documented and the translators are used to it.14:58
henningeWhat do you mean by "enable"?14:59
adeuringwell, ok, I'll add a sort of diode then14:59
bigjoolsjtv: so I converted your SQL into STORM syntax.  I think I might be about to faint that I got it right first time and the tests all pass.14:59
henningeadeuring: please do.14:59
jtvbigjools: congratulations!14:59
adeuringhenninge: and what is the other issue?14:59
bigjoolsjtv: http://pastebin.ubuntu.com/565404/14:59
henningeadeuring: secondly, I would expect a test or two for this specific behavior.14:59
henningeadeuring: you adapted some tests but I guess they just failed because the incumbent message happened to be empty.15:00
adeuringhenninge: hmmm, I think the test method names indicate that the incumbent message is supposed to be empty, right?15:01
adeuringbut nevenr mind, i can add another test ;)15:01
henningeadeuring: hm, you may be right15:01
henningebut then perhaps a test is missing for when it is not?15:02
adeuringok, i'll check it15:02
henningeadeuring: thanks.15:02
henningeadeuring: I fear you will run into some work with the test when you add the "diode" because I think it is designed highly symmetrical atm.15:04
henningebut it would be good to show the assymmteric behavior in the test.15:04
benjideryck: do you know of any docs about how to use LP.client.Launchpad() to interact with the web service from JavaScript?15:05
deryckbenji, on call, will respond just a minute15:05
benjithanks15:05
LPCIBotProject devel build (431): FAILURE in 5 hr 33 min: https://hudson.wedontsleep.org/job/devel/431/15:16
jcsackettabentley: i'm looking at your last branch now. i'm not familiar with lazr configs--the purpose of your additions is just proper error capturing when the job runs?15:16
abentleyjcsackett, no, the configuration glues it all together.  Nothing could be run without it.15:17
abentleyjcsackett, It specifies the database user to use, the JobSource to use, the error handling, etc.15:17
jcsackettabentley: ah, i see. i had only looked at the first two config changes.15:18
abentleyjcsackett, lazr-js is a layered configuration system, so those first two overlay the last one.15:18
jcsackettabentley: dig. thanks.15:18
deryckbenji, hi, back now.15:29
deryckhmmm, so lp_client docs basically>15:29
deryckbenji, I don't think we have any actually.  I've always looked at the client code itself, or followed examples in other js code.15:30
derycksorry15:30
=== matsubara is now known as matsubara-lunch
benjideryck: that's what I suspected; I'm writing the start of some docs that will hopefully fill the hole15:32
deryckah nice15:32
jcsackettabentley: all branches are r=me; sinzui has already followed up on several.15:35
jcsackettreally nice work.15:35
abentleyjcsackett, thanks.15:35
deryckabentley or rockstar -- has the minus icon on bug pages to unlink a branch been ajax before now?  Or it always went to a confirmation page?15:38
* deryck is QAing new lazr-js and not sure15:38
rockstarderyck, on bug pages? No idea.  gmb did that work.15:38
abentleyderyck, unsure.15:38
deryckah ok. gmb ^^ ?15:39
sinzuiabentley: I think everything is cleared to land now15:39
* gmb reads scrollback15:39
abentleysinzui, great, thanks.15:39
gmbderyck: I *think* it's always gone to a confirmation page.15:39
gmbYet another we-only-did-half-the-work-before-moving-on.15:40
deryckgmb, ok, cool.  thanks.15:40
jcsackettderyck: assuming you're concerned about a branch that recently hit, i can confirm that as of last monday unlinking went to a page, not ajax.15:44
deryckjcsackett, indeed that is my concern.  Thanks!15:44
=== al-maisan is now known as almaisan-away
deryckheh, oops.  Somewhere a link from qastaging pointed to lpnet.  And now there is a dumb project on launchpad.15:52
deryckor a new team rather.15:53
deryckheh, or a project that thinks it's a team.  gah!15:53
derycksinzui, can I or you delete https://launchpad.net/deryck-test-qa-team/, or do we need an admin for that?15:54
sinzuideryck: you cannot see the delete link?15:54
sinzuideryck: oh sorry, that is a project. the cannot be deleted. you can rename it and deactivate it15:55
derycksinzui, ok, thanks.  I'll just deactivate it.  Sorry about that.15:56
deryckmaybe I hand entered a URL and didn't realize I forgot the qastaging bit.15:59
abentleysinzui, I want to create a Job every time a Packaging is created, to merge the translations between the sourcepackage and the productseries.16:01
=== allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: jcsackett, allenap | https://code.launchpad.net/launchpad-project/+activereviews
abentleysinzui, Packaging doesn't have a constructor, but I'm thinking of adding one, and having it call notify.  Make sense?16:02
sinzuiyes it does16:02
abentleysinzui, cool, thanks.16:02
=== salgado is now known as salgado-lunch
abentleysinzui, are Packagings ever modified, or just created and deleted?16:04
sinzui abentley never modified16:05
abentleysinzui, great.16:06
* sinzui wonders though if changing a ds packaging workflow calls delete?16:06
* sinzui lookse16:06
sinzuiabentley: definitely not modified. We create, and occasionally delete. Packaging acts as a historical record16:10
abentleyGood to know.16:10
jtvjcsackett: got an MP for you!  Are you free?  https://code.launchpad.net/~jtv/launchpad/bug-711077/+merge/4924216:14
jcsackettjtv: in the middle of investigating a test failure. can look in half an hour?16:14
jtvjcsackett: if you like, though I could ask allenap first16:14
jcsackettjtv: if he's free now, sure. if not, i'll be happy to take a look at it soon. :-)16:15
allenapjtv: I'm up for a review.16:15
jtvgreat!16:15
jtvsee mp                                                                   ^^^^16:15
=== deryck is now known as deryck[lunch]
=== beuno is now known as beuno-lunch
jtvbigjools: it seems the parallel a-f is on qastaging… want to walk me through the Q/A process?16:30
bigjoolsjtv: I'm in the middle of something, happy to do so later16:30
jtvOK16:30
bigjoolsjtv: passing tests were some sort of weird miracle earlier, they don't actually pass and my storm expression won't compile16:31
bigjoolsI am starting to really detest storm syntax16:31
jtvbigjools: it can be annoying, yes.  Anything there I can help with?16:32
bigjoolsjtv: maybe, one sec16:32
jtvDebugging the things isn't fun either.16:32
bigjoolstell me about it16:32
bigjoolsjtv: http://pastebin.ubuntu.com/565450/16:36
bigjoolswon't compile,  it doesn't like the PackageBuild.archiveID or PackageBuild.archive16:36
jtvbigjools: have you tried PackageBuild.id.is_in(Select(…))?16:36
bigjoolsjeez16:36
bigjoolsok16:36
bigjoolsit would be PackageBuild.archive.is_in of course16:37
bigjoolsAttributeError: 'Reference' object has no attribute 'is_in'16:38
jtvGrrrr16:38
jtvTry PackageBuild.archive_id.is_in or PackageBuild.archiveID.is_in16:38
bigjoolsthis ID vs reference stuff is seriously irritating16:38
jtvThis stuff is way too hard.16:39
jtvYes16:39
bigjoolsthat too16:39
bigjoolsI suspect the _id will do it, I was using ID before16:39
jtvgrrrr16:39
bigjoolsyay it worked16:39
jtv\o/16:39
bigjoolsthanks jtv16:40
jtvnp16:40
jtvyou were blocking my lane, had to help :)16:40
bigjoolsI think I knew about this, it's clearly been a while since I wrote storm16:40
jtvBut you keep hoping someone will fix this stuff while you're away, don't you?16:40
bigjoolsnow that part is fixed, I get the problem somewhere else16:43
=== salgado-lunch is now known as salgado
jtvThat's progress.16:43
jtvWhat is it now?16:43
bigjoolssame compile error about a Reference - but no indication of which one16:43
jtvAh yes that one16:43
jtvit's always fun.16:43
bigjoolsFSVO16:44
jtvI do TLAs, not ETLAs16:44
bigjoolsha16:44
bigjoolsOMG NFW BMX16:44
jtvBMX?  WTF!?16:44
jtvAnyway, with the Reference one at least you know you're missing a _id or ID suffix on a reference attribute.  It's not that you might have mis-spelled one.16:45
jtv(That comes later.)16:45
bigjoolsyes16:46
bigjoolsshockingly, my first guess found it16:49
jtvthe gods are toying with you16:49
bigjoolsso is dogfood16:55
jtv"…they play with us for their sport"16:56
bigjoolsdogfood is doofgod backwards.16:58
bigjoolsmawson should be aergia it's so bloody slow16:59
jtvaegria?17:01
bigjoolsaergia17:02
jtvAh, not familiar with her17:02
bigjoolsdarn, this query is bust17:02
jtvWhat now?17:02
bigjoolsit's returning the wrong stuff - yet the tests pass17:03
LPCIBotProject devel build (432): STILL FAILING in 1 hr 53 min: https://hudson.wedontsleep.org/job/devel/432/17:09
LPCIBot* Launchpad Patch Queue Manager: [r=thumper][no-qa] Mechanical move of17:09
LPCIBotcanonical.launchpad.windmill.testing to lp.testing.windmill.17:09
LPCIBot* Launchpad Patch Queue Manager: [r=allenap][bug=528367] Added alt tag to remove icons.17:09
=== beuno-lunch is now known as beuno
LPCIBotProject db-devel build (356): FAILURE in 2 hr 27 min: https://hudson.wedontsleep.org/job/db-devel/356/17:15
LPCIBotLaunchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 1235717:15
LPCIBotincluded.17:15
bigjoolsjtv: hurray at last.17:16
jtvWhat was the problem?17:17
bigjoolsthe history page on dogfood finally loads for adare17:17
bigjoolsjtv: pebkac17:17
bigjoolsjtv: where's your branch that you need testing?17:29
jtvbigjools: lp:~jtv/launchpad/bug-181368-view17:31
jtvThat's against devel though.17:31
bigjools'sok17:31
jtv(In fact it's already _in_ devel)17:31
bigjoolsI'm going to pocket copy a package to create publications in lots of places (it has 6 arch binaries)17:32
lifelessmoin17:34
lifelessbigjools: glad you like it17:35
bigjoolslifeless: saves a bit of ctrl-u/page-down17:35
lifelessI described it to elliot as 'tastefully IN YOUR FACE'17:35
bigjoolsjtv: ok, I've copied acl in maverick to maverick-proposed, with binaries.  See https://dogfood.launchpad.net/ubuntu/+source/acl/+publishinghistory17:36
* bigjools runs df publisher17:36
jtvbigjools: I'm not sure what to look out for though17:36
bigjoolsjtv: I'll send you a log file when it's done.  You'll see that source go from pending to published17:36
allenapjtv: I'm not going to finish your review today. Do you mind if I complete it in the morning?17:38
jtvallenap: then maybe I should ask jcsackett to give it a go after all.17:38
jtvallenap: or were you already partway through it?17:38
bigjoolsjtv: also this publisher may not finish within the 20 minutes I have left today17:38
jtvbigjools: ah… and this is df innit?17:39
bigjoolsjtv: yeah I limited it to just the suite we want, but it will still take some time17:39
bigjoolsa full run takes *hours* :/17:39
allenapjcsackett: I am partway through it, but not that far, so I'm happy for him to take it if he is too. I have been pulled away by non-work things.17:39
jtvOh well, have to check up tomorrow then.17:39
bigjoolsjtv: ah it finished17:40
jtvbigjools: so it seems to have17:40
jtvSo how do we know it did something sensible?17:40
jmllifeless: good morning.17:40
lifelessjml: hi hi17:41
jcsackettjtv: you want to wait on allenap, or shall i take a look at it now?17:41
jmllifeless: now or in the next hour would be a good time for a call.17:41
bigjoolsjtv: I am checking the Packages files17:41
lifelessok, I'll ping you in a few minutes17:41
bigjoolsjtv: how many does it kick off in parallel BTW?17:41
jtvjcsackett: I'd appreciate a look—I'm pretty much EOD, but I'm slightly ahead of allenap timezone-wise.17:41
jtvbigjools: one for each architecture (including "source")17:42
jcsackettjtv: okay, i'll take a look and ask any questions in the comments.17:42
bigjoolsjtv: so 617:42
jtvbigjools: but that's the parallel run.  This branch is a different change.17:42
jmllifeless: ok.17:42
bigjoolsit is?17:42
jtvbigjools: yes, circumventing an awkward view.17:42
=== deryck[lunch] is now known as deryck
bigjoolsseemed quicker :)17:42
jtvGood!17:43
jtvbigjools: the parallel run is still waiting for the RT17:43
bigjoolsjtv: only 4 of the binaries got published, I don't know if that's right or not17:44
bigjoolsbut the ones that did look ok17:44
jtvbigjools: hmmm for proper Q/A we'd have to know whether that was right.17:44
bigjoolsyes17:44
bigjoolscould be PaS17:45
jtvPhotoAcoustic Spectroscopy?17:45
bigjoolsah I should try an arch indep, that'll not matter17:45
bigjoolsPackage Arch Specific17:46
bigjoolsit's a last-ditch override file that the publisher uses17:46
jcsackettjtv: while you're still here, i note in the diff you're adding some columns, but this isn't targeted to db-devel; i assume it's not introducting change to the db, but don't really understand why not.17:47
jtvjcsackett: adding columns!?17:47
jtvjcsackett: sounds as if something went wrong with the MP… just a mo'17:48
jcsackettjtv: several adds of IntCol (branch_id), but not replacing similar statements. i may be misinterpreting.17:48
jtvjcsackett: oh, those aren't columns—they're attributes that allow me to read the FK attributes directly17:49
jtvPretty standard stuff.17:49
jcsackettjtv: ah. i figured it was something like that, but wasn't sure. and the use of "Col" in the declaration raised alarms. :-)17:49
jtvSeems I have to go help with a little accident here.17:50
bigjoolsjtv: so a tentative ok, but please ask wgrant to double check since this really needs more eyes to verify it before we unleash on the distro17:50
jtvbigjools: ack17:51
=== matsubara-lunch is now known as matsubara
=== jtv is now known as jtv-afk
bigjoolsjtv-afk: so meh, I forgot to notice that copy-package only copied some of the binaries, so it seems perfectly ok.17:54
lifelessjml: ping18:00
lifelessjml: skype me18:02
lifelessderyck: question for you, why are BugMessage and Message separate tables?18:03
derycklifeless, I literally have no idea. :-)  I asked once before and didn't understand the response.  I think....18:04
derycklifeless, it was written in a time when we imagined different kinds of messages, of which bug message was one type.18:04
lifelessthere are 640 shared messages18:04
jmllifeless: k18:04
lifelessthat is, one message on several bugs.18:04
deryckhmmm, I don't know why that would be.18:05
lifelessderyck: I think we could make a bunch of queries faster if we deep sixed the sharing and just copied the message when that happens18:05
lifelessderyck: email control interface controlling many bugs?18:05
lifelessderyck: in case its not obvious I mean a db migration too :)18:05
derycklifeless, perhaps.  but I thought you could only affect multiple bugtasks, not bugs, from email18:05
derycklifeless, yes, I think moving all that across to copies, not shared messages, is best18:06
jcsackettwhat's the difference between using IStore and Store.of? they seem to result in the same thing, but i'm not certain.18:12
deryckjcsackett, so IStore is what we provide to help manage connecting to the db, instead of just straight import Store from Storm.18:19
deryckjcsackett, see lib/canonical/launchpad/doc/storm.txt18:20
jcsackettderyck: thanks.18:20
deryckjcsackett, actually, I meant to point you at:  lib/canonical/launchpad/doc/db-policy.txt18:20
deryckwell, either is probably ok ;)18:20
* jcsackett changes which file he is opening...18:20
lifelessjml: http://gems.github.com/list.html18:22
jcsacketttoday has been a very busy review day...18:26
=== jcsackett_ is now known as jcsackett
jcsackettsinzui: can i get you to look at something?19:09
sinzuiyes19:09
jcsacketti'm having trouble understanding the "why" of a bit of publisher code: https://pastebin.canonical.com/43132/19:09
jcsackettit looks like it's trying to rebuild web service based requests, but i don't understand why--commenting out that code doesn't seem to hurt webservice functions.19:10
jcsackett(this is the actual culprit for the api.launchpad.net pillar aliases redirecting outside of the api)19:10
sinzuijcsackett: I confess I do not understand it19:12
lifelessjcsackett: btw19:12
lifelessjcsackett: see my last comment on https://code.launchpad.net/~jtv/launchpad/bug-711077/+merge/49242 - thats something to bear in mind19:12
sinzuimy head may be filled with straw today. I can struggling to refactor what I think is a simple test19:12
jcsackettlifeless: good thought.19:13
jcsackettsinzui: yeah, after a morning filled with the largest review queue i've seen, i'm feeling pretty muddled.19:13
jcsackett(and since you follow up on them all, i'm not surprised you feel the same).19:13
jcsackettlifeless, any chance you know what's up in this bit? https://pastebin.canonical.com/43132/19:14
lifelessother than the gratuitious pass?19:14
jcsacketthm, you can ignore that--left over from when i commented out a bunch of code to see what changed.19:15
* jcsackett edits paste.19:15
lifelesswhat file is that in ?19:15
lifelessjcsackett: have you been introduced to bzr's annotate functionality ?19:15
jcsackettlifeless: i've seen annotate used once, and heard much of it. i do not, as yet, know how to do anything useful with it. :-P19:16
jcsackettfile is canonical.launchpad.webapp.publisher19:16
jcsackettit seems to rebuilding webservice requests (and in the process losing the api layer)19:16
lifelessso the comment says:19:17
lifeless - no rootsite19:17
lifeless - handling shipit19:17
lifelessblame says the outer if was done in rev 3564.1.4319:18
lifelessand the inner one in 7675.289.2 / 3 / 419:18
lifelessbzr log -r 7675.289.2..7675.289.4 tells me that bug-376990 was at issue19:19
lifeless'canonical_url() needs to return browser URLs when generating XHTML representations'19:19
jcsackettyeah.19:19
jcsackettthat seems to be the inverse of the problem leonardr pointed out to me.19:20
lifelessso, it wants to get canonical urls for browser links some of the time19:20
jcsackettso, this only happens when canonical_url is called for pillars that we've gotten by alias in the webservice. i think i can just change a call up the pipe to pass the request along.19:20
lifelessthis also tells us how to test it19:20
lifelessinline bug commenting19:20
lifelessbut we have a stale comment or something, because the block *above* the comment was changed by tim in rev 3758.1.119:22
bachi sinzui19:22
lifelessjcsackett: so, what are the symptoms?19:22
sinzuihi bac19:23
jcsackettlifeless: bug 71599219:23
_mup_Bug #715992: Pillar aliases redirect outside of webservice on webservice calls <Launchpad itself:In Progress by jcsackett> < https://launchpad.net/bugs/715992 >19:23
lifelesswow, we are digging up some chestnuts19:24
lifelessthis should be critical otherwise its a priority inversion19:24
jcsackettthis came up because it needs to be whacked to deal with bug 62309919:25
_mup_Bug #623099: AttributeError filing a bug using the API <lp-foundations> <oops> <Launchpad itself:In Progress by jcsackett> < https://launchpad.net/bugs/623099 >19:25
lifelessyes19:25
lifelessagreed; I'm just saying bump it to critical19:25
jcsackettah, dig.19:25
jcsackettdone.19:25
lifelessbecause its otherwise holding a critical hostage, which wouldn't make sense19:25
lifelessso19:26
lifelesslooks to me like you need to pass a request in19:26
lifeless'    If there is no request available, the protocol, host and port are taken19:26
lifeless    from the root_url given in launchpad.conf.19:26
lifeless'19:26
lifelessas leonardr says19:27
lifeless'Possibly there's a method that needs to have the current request passed into it.'19:27
lifelessjcsackett: whats the backtrace leading into this call of canonical_url ?19:27
jcsackettyeah, i think i've found it; i just hit this block of code and was wondering what on earth it was doing.19:28
jcsackettbit of a rabbit hole, really.19:28
lifelessits guessing at the url the client used19:28
lifelessand the default is to guess 'web browser request on main site'19:28
lifelessworth enhancing the comment19:28
lifelessand adding a :param request: up in the docstring (subsuming the 'if there is no request available' prose.)19:29
lifeless?19:29
jcsackettlifeless: yeah, i'm cleaning up the comments.19:31
jcsackettlifeless: thanks.19:34
benjideryck: I'd love any feedback or additions you might have to my first draft of "how to access webservice APIs via JavaScript": https://dev.launchpad.net/yellow/JavaScriptAPIAccessDraft19:53
deryckbenji, ok, taking a look now19:54
jcsackettany chance i can get a review on https://code.launchpad.net/~jcsackett/launchpad/api-pillar-redirects-715992/+merge/49279?19:57
deryckbenji, yeah, it's good.  Do you have plans to fill in more about PATCHPlugin?19:58
benjideryck: I hope to.  I have to figure out how to use it first. :)19:58
deryckbenji, you can look at how the inline editor widgets and commenting use it.19:59
benjiMy plan is to put in everything I know so people can get a better start.  I spent way too much time figuring out what little I know thus far.19:59
benjicool, thanks for the pointer19:59
deryckbenji, I can add to this too.  But I'll need pings to remind me about it.19:59
benjiyou can count on me20:00
deryckbenji, our use of the js client as evolved over time, and we're probably named_post and named_get heavy, when we should favor patch with patch_field set true.20:00
benjiinteresting20:01
deryckand then setting accept to xhtml, we get back the html for the part we patched. instead of the full resource.20:01
deryckwell, unless we need the entire thing.  it depends on use, I guess.  but it seems more common to do attribute patching.20:02
benjican the client understand xhtml?  why does the server not return partial JSON?20:02
deryckbenji, well xhtml can be created into a node easily in YUI.  it's just a string, I think, not try xhtml.20:03
derycks/try/true/20:03
deryckso...  var stuff = result_of_my_patch(); var my_html = Y.Node.create(stuff);20:03
benjihmm, I don't understand.  Is that for when we only patch one attribute at a time?20:04
deryckbenji, yes.  which is actually common.  And the attribute will be a FK to some other object on the backend.20:05
benjiah, ok that makes sense20:05
deryckthat pseudo code might not be exactly how we do it either.  Just going off sketch from memory.20:05
sinzuibenji: js replies on the the DOM's engine to construct elements You can create xhtml or html using DOM nodes, but what is rendered could be the other :( I have also see cases where the code tries to place a list in a paragraph, but what is rendered is a paragraph, a list, and another paragraph20:07
sinzuis/replies/relies/20:07
benjiinteresting20:08
thumpermorning20:13
abentleyjcsackett, I have a sequel for you: https://code.launchpad.net/~abentley/launchpad/job-on-new-packaging/+merge/4928520:18
* jcsackett looks20:27
thumperderyck: ping20:42
thumperderyck: do you know most about windmill tests?20:42
deryckthumper, I guess I'm your only help where windmill is concerned ;)20:42
thumperI have a question about lines like: start_url = (windmill.settings['TEST_URL'] + 'bugs/%d' % bug.id)20:42
thumperwindmill.settings['TEST_URL'] is what?20:43
thumperand why?20:43
thumperand why does only the code windmill tests use it?20:43
thumperand lib/lp/testing/windmill/lpuser.py:3520:43
=== almaisan-away is now known as al-maisan
deryckthumper, I actually don't know what that is.20:44
thumperderyck: I think it may have to do with that wallyworld did, to do with parallelizing the test suite20:44
deryckthumper, only code tests made use of it, and I'm not sure why.20:44
thumperand not using hard coded ports20:44
deryckyeah, that would be my guess.  but that's not the way he originally mentioned about not hard coding ports.20:45
thumperderyck: I have a day of doing windmill today with three branches needing some form of windmill testing20:45
thumperderyck: I was thinking of a base class helper method20:45
thumperderyck: that uses canonical_url(obj, force_local=True)20:45
thumperderyck: and appending that to the base test url20:45
thumperderyck: there are too many uses of this already20:46
deryckthumper, why do you need windmill tests?  What are you wanting to test?20:46
thumperderyck: mumble?20:46
derycksure20:46
Ronnieis there a python-launchpadlib that can run without compability problems on 8.04?20:46
RonnieSWAT ^20:46
jcsackettabentley: r=me.20:50
abentleyjcsackett, thanks.20:50
=== al-maisan is now known as almaisan-away
=== matsubara is now known as matsubara-afk
deryckwallyworld, windmill.settings['TEST_URL']21:03
deryckwallyworld, thumper -- client.open(url='%s/bugs/1' % BugsWindmillLayer.base_url)21:05
_mup_Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I21:05
derycklater on, everyone21:12
wallyworldthumper: http://people.canonical.com/~ianb/request-build-error.png21:25
Ronnielifeless: wgrant ^^21:26
lifelessRonnie: how far up are you wanting me to look ?21:26
Ronniea few lines: (21:46:52) Ronnie: is there a python-launchpadlib that can run without compability problems on 8.04?21:27
lifelessthe one we ship in 8.0.4 ?21:27
lifelessblah, 8.0421:27
lifelesschecking rmadison21:27
lifelessno, nothing before karmic21:28
lifelessdapper is EOL in a few months21:28
lifelessso no plans to chane this21:28
lifelessRonnie: why?21:28
Ronniethe ubuntu-nl server still runs on 8.04. we want in one case to communicate with launchpad (creating bug reports)21:29
lifelessif thats in the canonical datacentre, they are migrating them all to lucid21:29
lifelessjust tee that up, it should be queued already21:29
Ronnielifeless: its an own managed server21:30
lifelessah21:30
Ronniehoe flexible are loco's if we use canonical data centre>21:31
Ronniecan we deploy our apps ourself, are there limitations, whats the connection speed in europe etc21:31
lifelesstheres a range of configs, really up to IS what and where21:32
lifelessin terms of capabilities and performance, pretty darn good; the dc is in London21:32
lifelessthat said21:32
lifelessuhm, you should be able to get launchpadlib on 8.04, it will just require some work21:32
Ronnielifeless: currently were thinking about staticcaly linking21:34
lifelessRonnie: well, its pytho n:)21:35
lifelessyou can probably backport the karmic set of packages fairly easily21:36
beunoso, is it known that the page to reqeust a merge proposal is slightly broken?21:36
beunothe "Other" option shows up as: "Other:'>"21:37
beunowhich I assume is a broken template21:37
thumperbeuno: yeah, I saw that today too... file a bug?21:37
lifelessthere is a bug open21:37
lifelessits critical21:37
Ronnielifeless: again about the hosting on canonical servers. how would such thing look like. if we have problems can we fix it ourself or do we have to ask canonical-isd, and what are there response times?21:37
lifelesssinzui is looking at it21:37
beunocool21:38
lifelessRonnie: I'm a poor source of info for this; #canonical-sysadmins is I think the channel for contact21:38
=== salgado is now known as salgado-afk
Ronnielifeless: spel error on channel?21:39
wgrant#canonical-sysadmin21:39
wgrantNo s.21:39
Ronniethx wgrant21:40
sinzuibeuno: the fix is waiting for the rollout21:43
sinzuiabentley: I had a few remarks about your last branch. Maybe you have good reason to not do as I expected21:47
abentleysinzui, no, mostly copying bad ideas from existing code.21:47
jami'm seeing an odd typo on the merge proposal page. https://bugs.launchpad.net/launchpad/+bug/71671321:52
_mup_Bug #716713: typo in propose page <Launchpad itself:New> < https://launchpad.net/bugs/716713 >21:52
jamSpecifically, the "Other: "line is rendered as "Other: '>"21:52
jamLooking at the HTML it looks like invalid html21:52
lifelesssinzui: whats the bug number, so we can dup jams bug :)21:52
jamah, beuno, you beat me to it21:52
sinzuilifeless: beuno jam: bug 714527 address the issue on the recipe and Mp pages21:54
_mup_Bug #714527: owned widget structured strings render as bits of quoted html <qa-ok> <regression> <Launchpad itself:Fix Committed by sinzui> < https://launchpad.net/bugs/714527 >21:54
abentleysinzui, thanks for your review.  I've pushed changes.  Diff still updating, though.22:01
sinzuiabentley: I approved it.22:02
abentleysinzui, I saw, I just wanted to close the loop.22:03
=== jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
sinzuiwgrant: mumble?22:33
huwshimisinzui: Can I get you to review the colour unification stuff I've done? Not now as it sounds like you've had a long day, but when you have a chance?22:52
sinzuihuwshimi: I sure can22:52
huwshimisinzui: Thanks heaps.22:52
sinzuiI think I have done part of it already since I looked at it the other day22:52
huwshimisinzui: OK great. Would be good if you could have a quick look at the css if you haven't already.22:54
huwshimijust putting together the mp22:54
huwshimisinzui: For whenever you can get to it: https://code.launchpad.net/~huwshimi/launchpad/unified-colours-710446/+merge/4931622:59
sinzuithanks. I will wait for the diff to generate22:59
* thumper is hacking the windmill base test case class23:00
thumperinteresting.... windmill hanging on start at http://code.launchpad.dev:8085/windmill-serv/start.html23:02
thumpersometimes only...23:03
* thumper wanders into the kitchen in search of food23:06
huwshimisinzui: There's 60 or 70 lines removed from style.css which is nice23:06
LPCIBotYippie, build fixed!23:09
LPCIBotProject db-devel build (357): FIXED in 5 hr 43 min: https://hudson.wedontsleep.org/job/db-devel/357/23:09
lifelesswgrant: the linkification of post-filing messages is nice23:11
lifelesswgrant: I'm really glad you jumped on that23:11
huwshimisinzui: I removed that line. All ok?23:37
sinzuiyes23:37
huwshimisinzui: Thanks :)23:38

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!