benjilifeless: you probably figured it out already, but there has to be a 1:1 mapping from values to tokens (and back)00:39
lifelessbenji: yeah00:47
lifelessI had a query that wasn't DISTINCT00:47
lifelesseasy fixed00:47
lifelessbenji: a better error would list the token and titles that were duplicated00:48
lifelesswgrant: if you're interested02:11
lifeless 54 SELECT DISTINCT DistributionSourcePackageCache.archive, DistributionSourcePackageCache.binpkgdesc ...  DistributionSourcePackageCache.archive IN (%s, %s) AND BinaryPackageName.name = %s ORDER BY name:02:11
lifeless   GET: 54 Robots: 2  Local: 4002:11
lifeless     54 https://launchpad.net/ubuntu/+search (Distribution:+search)02:11
lifeless       OOPS-1704C1791, OOPS-1704C1793, OOPS-1704C554, OOPS-1704C773, OOPS-1704C77502:11
lifelesswgrant: e.g. highest oopser yesterday02:11
rockstarthumper, I just got your winge email about lazr-js.  What's up?02:30
* rockstar has no problems02:30
thumperrockstar: no worky02:30
rockstarthumper, did you do a make?02:30
thumperI pulled trunk02:30
thumperwent make02:30
thumpersaid I had bad versions02:30
thumperwent to download-cache02:30
thumperdid bzr up02:30
thumperwent back02:30
thumperdid make02:31
thumpersaid invalid version for setuptools02:31
thumperedited version.cfg02:31
rockstarthumper, what about download-cache 'bzr up'02:31
thumperdid make02:31
thumperbut examples don't work02:31
rockstarthumper, do you get any javascript errors?02:31
thumpernot obviously02:32
thumperCSS seems broken02:32
rockstarthumper, probably because your make didn't actually work.02:32
rockstarthumper, could you file a bug with the output of make and the like?03:03
rockstarthumper, I'm concerned that my 'make' works and yours doesn't.03:03
thumperrockstar: bit busy right now trying to get ian connected to my network again03:03
rockstarthumper, no hurry.  I'm EOD'd, but I'd like to make it work for you tomorrow if I can.03:04
rockstarMight be nice for wallyworld to be able to see the demos...03:04
=== Ursinha-afk is now known as Ursinha
lifelessanyone seen this ?04:13
lifeless  File "/home/robertc/launchpad/lp-branches/working/lib/canonical/lazr/pidfile.py", line 83, in get_pid04:13
lifeless    raise ValueError("Invalid PID %s" % repr(pid))04:13
lifelessValueError: Invalid PID ''04:13
lifelessnvm, found the files04:16
=== Ursinha is now known as Ursinha-afk
wgrantnoodles775: It looks to me like the distroseriesdifference model doesn't allow for multiple parent series.08:23
wgrantThat seems like a pretty critical feature.08:24
wgrantIs it in the spec?08:24
lifelesswgrant: pretty sure its not08:24
wgrantHow's that going to work, then?08:24
wgrantSurely Linaro's N release is going to want to inherit from Linaro M, and then merge from Ubuntu N.08:25
lifelessI suspect it will be done thusly:08:25
lifelessL-N lcones from L-M, with differences, and then inherits U-N08:25
noodles775wgrant: I've not heard of it before it was mentioned in that email, but I wasn't involved in the initial discussions.08:27
wgrantAh, maybe I should read this thread, then.08:28
* wgrant looks.08:28
noodles775hrm, maybe it was on https://dev.launchpad.net/LEP/DerivativeDistributions/UserTestingRound2 ...08:28
* noodles775 checks08:29
noodles775wgrant: yes, it was mentioned by Loic (see Mock-up 2 for Loic on the above page).08:29
wgrantAnd cjwatson as well.08:30
wgrantI should probably read the whole thing.08:30
wgrantAlthough Loïc refers to a parent distribution, not series.08:31
adeuringgood morning09:00
wgrantA friend just ran into a bit of a strange PPA key generator case.09:30
wgrantHe created his first PPA a while ago, but never uploaded anything.09:31
wgrantThen created a new one just now, and uploaded something to it.09:31
wgrantNow, of course it had no key preset... so it's now going to get one of its own generated.09:31
wgrantHow often does the key generator run?09:43
noodles775wgrant: currently every 20mins09:45
wgrantYeah, it just appeared now.09:45
lifelessany reason that can't run on-demand ?09:49
* noodles775 remembers cprov requesting hardware entropy.09:49
noodles775But not sure.09:50
lifelesswell, on-demand would have no higher or lower demand on entropy09:50
lifelesson average (and if there is enough to backlog after 20 minutes, there is enough to backlog for 20 minutes09:50
wgrantI thought it already did have a hardware RNG.09:51
wgrantIt's particularly bad, since the PPA will be published unsigned.09:52
wgrantThen you have to upload something again to get it signed.09:52
wgrantlifeless: But yes, the Referer check is stupid.09:59
wgrantI proposed it as a quick solution.09:59
wgrantI hoped that the user backslash would be enough to convince Foundations to fix it properly.09:59
wgrantBut apparently not.09:59
lifelessthumper: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1704EB174310:08
wgrantSo... user pages have been broken for five days now.10:09
wgrantOr 6. One of them.10:10
lifelesslosa ping10:11
=== jtv is now known as jtv-afk
lifelesslosa ping?10:40
mthaddonlifeless: sorry, missed the last one10:41
mthaddonlifeless: here now... :)10:41
lifelessthats ok :)10:41
lifelessI didn't want to interrupt steve if he'd EOD'd.10:41
lifelesstwo tings10:41
lifelessthere is a CP for the home page maps problem in LP10:41
lifelessthis one actually has the code10:41
lifelesssecondly, staging - has it successfully restored yet ?10:42
lifelessI'm a little worried about staging, monday PQM freezes and folk will be qaing like mad... we hope.10:42
mthaddonhmm, there was a stale lockfile from the last broken restore, I've removed and it'll kick off very soon10:43
mthaddonstarting now...10:45
lifelessmthaddon: thanks; can you please let uhm, someone, know about the CP ?10:49
mthaddonlifeless: I'm doing it now10:50
lifelessperhaps get a dev to tickle launchpadstatus on identi.ca or something if it works10:50
lifelessmthaddon: cool cool, not rushing you10:50
lifelesshanding-off, I have to sleep, early wakeup tomorrow10:50
mthaddonit's all scripted, so there's no rushing to do :)10:50
lifeless(I fail at sleep)11:42
lifelessSpamapS: https://lp-oops.canonical.com/oops.py/?oopsid=1704S20 btw11:42
lifelessthats your slow page11:42
lifelessfor some reason its counting every open bug.11:43
lifelessclearly daft.11:43
lifelessSpamapS: https://bugs.edge.launchpad.net/malone/+bug/62797411:46
_mup_Bug #627974: MaloneApplication:CollectionResource is slow/times out <timeout> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/627974>11:46
deryckMorning, all.12:06
=== Ursinha-afk is now known as Ursinha
=== matsubara-afk is now known as matsubara
=== Ursinha is now known as Ursinha-afk
=== jtv1 is now known as jtv
=== Ursinha-afk is now known as Ursinha
jcsackettmorning, all.13:52
jcsackettquick question for anyone who might have an answer: is it unexpected that a test should fail on EC2 but work just fine when you run that test locally?13:53
salgadojcsackett, yes, but you might try to run that test by itself on EC2, to find out whether it's test-fails-on-EC2 problem or test-fails-when-running-full-testsuite problem14:01
salgadoif you're brave enough to run the full testsuite locally and the test passes, then it's likely to be something related to EC2 or one of those intermittent failures we have once in a while14:02
nigelbum, anyone around I could talk to about the map issue? Do you folks think its because of the sensor parameter?14:09
nigelbIt would be nice to have a way to check if its that.14:09
benjiwgrant: you around?14:10
Ursinhanigelb, thanks! :)14:14
wgrantbenji: Hi.14:15
nigelbUrsinha: :)14:15
benjiwgrant: hey; I hear that you have a launchpadlib script that I should take a look at14:16
wgrantbenji: It mostly works (albeit very slowly) when running on a lazr.restfulclient with retry support.14:17
wgrantBut http://qa.ubuntuwire.org/ftbfs/source/ has the code.14:17
marsjcsackett, instead of running the entire suite locally you may want to try just running the part of the suite that contains the failing test.  If lp.registry.openid.foo failed, try running all the tests in lp.registry.openid14:20
marsjcsackett, it is a fast-running way to see that you did not encounter a test isolation failure14:21
marsjcsackett, while you are doing that you can also fire off an EC2 run or two to see if the failure is intermittent:  ec2 test -o '-vv lp.registry.foo.bar'14:22
jcsackettmars: i've already run the single test locally; it passed.14:24
jcsacketti'm about to fire off EC214:25
jcsackettmars: i believe it may be an isolation issue; if i run the entire bugs set of tests the one in question fails.14:26
marsjcsackett, if it fails locally when running that one suite then I wouldn't bother re-running on EC214:27
jcsackettmars, good to know.14:27
jcsackettthanks, mars.14:28
benjiwgrant: we'd like for you to try you the new lazr.restfulclient (0.10.0) and launchpadlib (1.6.5) and the devel version of the web service; using those versions will be much faster for you and be easier on the servers14:50
wgrantbenji: Is there a PPA for those two?14:50
benjimmm, I don't think so; let me see14:50
=== Ursinha is now known as Ursinha-afk
benjiwgrant: sorry, there isn't14:51
wgrantbenji: OK. It's about to hit midnight, so I'll look at installing those tomorrow.14:58
wgrantbenji: The script shouldn't need any changes?14:58
benjiwgrant: it shouldn't14:59
wgrantbenji: Great. Thanks.15:02
gmb Hurrah for tests that should never have passed and yet somehow did.15:04
SpamapSlifeless: :) thanks, I've subscribed to that bug report. :)15:11
marsrockstar, ping, have a moment for a question about the bzr pipeline mail you sent to the list last week?15:21
allenapIs anyone here an ace with zope.component?15:46
benjiallenap: what's up?15:47
allenapbenji: Hi. I'm trying to land lp:~allenap/launchpad/cache-experiment-roll-out, but I keep getting adaption errors when it runs in ec2. I can't replicate them locally.15:48
allenapbenji: The adapters are registered normally with some zcml, but I've also added registered them with the global site manager at import time.15:49
allenaps/added /15:49
benjiallenap: can you point me at the test results?15:50
allenapbenji: http://pastebin.ubuntu.com/486797/15:51
benjiallenap: I hava a suspicion, but it's taking forever for bzr to download the branch so I can confirm the suspicion16:02
allenapbenji: Shall I send you a bundle?16:04
benjiallenap: a diff would be good16:06
allenapbenji: http://paste.ubuntu.com/486805/16:07
adeuringgary_poster: can I ask you again for help about this failure: https://pastebin.canonical.com/36340/ for this diff https://pastebin.canonical.com/36336/ ?16:10
* gary_poster whimpers16:10
gary_postertrying to help noodles with an immediate problem while trying to get longer term fix done in background cycles16:11
benjiallenap: I /suspect/ that the IPropertyCache adapter (get_default_cache perhaps) is returning None, which means "I couldn't adapt"16:11
gary_posterbut will look adeuring16:11
adeuringgary_poster: thanks116:11
benjiof course, I don't know why that would happen in EC2 and not in dev16:11
allenapbenji: Yeah, it's very odd.16:12
gary_posteradeuring: help me page back in Robert's comments, please--hat did he say?16:13
benjiyou might add an assertion to get_default_cache that the return value is non-None and run just one of the failing tests in EC2 to see if my suspicion is correct16:14
adeuringgary_poster: that we should try a completely different approach for the current issue, ie.e, complete aoid this code, for unrelated reasons.16:14
adeuringgary_poster: problem is: his approach -- allow access to the restricted librarian for certain DC machines -- is refused by the admins16:15
adeuringdue to security concerns16:15
allenapbenji: Okay, I'll give that a go.16:17
adeuringgary_poster: so, to get a fix for bug 620458, i am back to the implementation as sketched by francis16:17
_mup_Bug #620458: cannot access attachments of private bugs any more <httplib2:Unknown> <Launchpad Bugs:Triaged by adeuring> <https://launchpad.net/bugs/620458>16:17
gary_posteradeuring: oh, LOSAs or James or somebody weighed in and said no?16:17
adeuringgary_poster: it was james16:18
gary_posterthen here we are, yes, got it. :-(16:18
gary_posterRobert's concerns as to load were very valid16:18
adeuringgary_poster: sure...16:18
gary_posterbut we don't have another answer now, I know16:18
=== matsubara is now known as matsubara-lunch
gary_posterFWIW, adeuring...16:19
gary_posterZope/WSGI allows us to pass file objects back to serve16:20
adeuringsounds interesting16:20
gary_posternot immediately useful, maybe...but yeah16:20
gary_posterif we had the files to serve16:20
gary_posterthe Zope could just give the open files back as a response (there's probably some incantation)16:21
adeuringgary_poster: well, the current implementationn even creates a temp file16:21
gary_posterand then the thread would be back open for business.16:21
gary_posterright, but it blocks on downloading the temp file, right?16:21
gary_posterwell s/right/good/ :-)16:22
adeuringgary_poster: yes. But: We have a file-like object there16:22
gary_posterI meant16:22
gary_poster"good, but it blocks on downloading the temp file, right?"16:22
gary_posterso adeuring, just so I understand, this is an oauth sort of problem, right?16:23
adeuringgary_poster: do you my test failures or the whole issue?16:23
gary_posteradeuring, the whole issue16:24
adeuringgary_poster: yes16:24
gary_posteryour test failures, AIUI, are about the fact that something somewhere in the vicinity of canonical_url does not perform as we expect16:24
adeuringgary_poster: exactly16:24
gary_posteradeuring: does your code work correctly in the real world?16:26
gary_posterif you know what I mean?16:27
adeuringgary_poster: well, it works launchpad.dev16:27
gary_posterright, ok16:27
gary_posteradeuring: There, of course, are tons of pieces of context that are not set up, between what you have here in the test, and what you have in live usage.  If this were my problem, I would put a pdb in getUrl, and walk through it once when it worked (launchpad.dev) and once when it didn't (tests).  If you want, give me the branch name, and I'll try to do that comparison for you.16:32
gary_poster(To be clear, I'm saying I don't know what context you need.  Sorry. :-/ )16:33
adeuringgary_poster: ok, just a second16:33
adeuringgary_poster: lp:~adeuring/launchpad/bug-620458-private-bugattachments-api-access16:36
gary_posteron it, adeuring.  What are instructions for testing live?16:40
=== salgado is now known as salgado-lunch
adeuringgary_poster: (1) add an attachment to a bug, (2) make that bug private, (3) use the sample code from pitti in bug 62045816:42
_mup_Bug #620458: cannot access attachments of private bugs any more <httplib2:Unknown> <Launchpad Bugs:Triaged by adeuring> <https://launchpad.net/bugs/620458>16:43
adeuringgary_poster: in short: lp.bugs[1].attachments[0].open()16:44
=== Ursinha-afk is now known as Ursinha
lifelessgary_poster: r375 of storm trunk has the fix we discussed17:11
gary_posterlifeless, awesome, thanks for update.  May make a temporary release then , when I get to it.17:12
lifelessgary_poster: It should be cherrypickable; and the workaround that is in launchpad/lazr for the sqlobject variant should be backed out17:12
lifelessI suspect a cherrypick on 0.17 will work better than a snapshot of trunk; there are two bugs in 0.17; one is fixed in trunk, but it exposes the other... the first one resultsin more queries, the second in an internal error.17:13
lifelessyou're welcome to try however :P just hoping to save you some time.17:13
=== benji is now known as benji-lunch
lifelessleonardr: ^17:14
gary_posterack lifeless, thanks17:16
=== beuno is now known as beuno-lunch
dobeyis the entire change history of a branch_merge_proposal? like all status changes and such (not votes/comments)17:38
=== matsubara-lunch is now known as matsubara
=== salgado-lunch is now known as salgado
=== benji-lunch is now known as benji
=== deryck is now known as deryck[lunch]
=== gary_poster_ is now known as gary_poster
cr3lifeless: if you had to choose between running zope or django for the standalone instance of the results tracker, what would you do?18:41
=== beuno-lunch is now known as beuno
=== deryck[lunch] is now known as deryck
dobeyleonardr: ping19:03
dobeyor anyone else i guess19:05
dobeyshould launchpadlib throw HTTPError when it gets a 301 redirect for an api call?19:05
leonardrdobey: no. what are you doing? does launchpad.me work?19:06
dobeyleonardr: getSeries() on a distribution is giving me a redirect and HTTPError19:07
leonardrdobey: do you know if this is a recent change, or did you just start using getSeries?19:08
dobeyi just started using it. i'm trying to get a valid series link so i can pass it in to the source_package_recipe.requestBuild() method, since it doesn't accept u'maverick' as valid19:09
leonardrdobey: give me the code you use to invoke getSeries so i can see if the problem has been fixed in a more recent version19:10
dobeyleonardr: do you need the whole script, or just the launchpad bits after authentication?19:11
leonardrdobey: the latter19:12
dobeyleonardr: http://pastebin.ubuntu.com/486896/19:13
gary_posteradeuring: I believe the biggest difference between live QA and automated testing is because the WebserviceTestRequest has a different behavior in getRootURL than the real webservice request.  The real request has the behavior in line 1332 of c/l/w/servers.py, while the test request does not include that method for some reason .19:14
gary_posterhttp://pastebin.ubuntu.com/486893/ gets closer (sorry, it includes your changes because I did this in a not very smart way).  See lib/canonical/launchpad/webapp/servers.py for the most important change, and line 135 .  Can I toss this to you for a bit more digging on your side?19:14
leonardrdobey: can you tell me the value of self.series?19:14
leonardrdobey: ok, i can reproduce it. will you file a bug against lazr.restfulclient? it might be a problem in httplib2 but we'll start there19:15
dobeyleonardr: does distribution.series[u'maverick'] give the (ideal) same result?19:16
leonardrdobey: well, [x for x in distribution.series if x.name=='maverick'] will work19:16
leonardrbut the [] operator isn't supported on arbitrary collections right now19:17
lifelessbryceh: don't put the [] stuff in the launchpad commit message19:32
lifelessbryceh: ec2land does it for you19:32
brycehlifeless, why not?  does it hurt anything?19:39
brycehlifeless, from WorkingWithDbDevel it seems to suggest including it in the commit message19:39
dobeyleonardr: bug #62826719:39
_mup_Bug #628267: distribution.getSeries() raises HTTPError on redirect <lazr.restfulclient:New> <https://launchpad.net/bugs/628267>19:39
cr3lifeless: did you notice my question above? personally, I'm leaning towards django, not by preference but to keep things lean for now19:40
lifelessbryceh: yes, it hurts19:43
lifelessbryceh: you'll have them twice19:43
lifelessbryceh: they have to be in what pqm receives, which is -not- what goes in the lp commit message; its what ec2 land outputs for you19:43
lifelesscr3: uhm, I don't know. Maybe neither?19:44
cr3lifeless: what did you have in mind?19:44
brycehlifeless, sheesh19:44
lifelesscr3: trolling you successfully19:44
lifelessbryceh: please do improve the wiki page to prevent other folk having the same misunderstanding19:45
benjicr3: if you feel like being experimental as well as lean, you might like bobo: http://bobo.digicool.com/19:45
lifelessrockstar: ^ also tarmac will need to know the logic ec2land uses to do commit messages/approval recording etc; presumably via a plugin (and as folk set this from the commandline, I guess we need extra data when the branch is set to 'Queued'19:46
lifelesscr3: our best of breed set is (djanog, zope);19:46
lifelesscr3: frankly I think there is a bit of a hole in the middle; djano gives too little support, zope ---to much---19:47
rockstarlifeless, maybe I'm missing context there, but that sentence didn't make a lot of sense to me.19:47
lifelessrockstar: to use tarmac to land launchpad branches, preserving the [foo=bar] in the commit messages which is used for QA workflow, we'll need to make sure that the 'ec2 land' process still handshakes properly with the thing doing the commits.19:48
cr3lifeless: agreed, I'll work towards that19:48
lifelesscr3: either django or zope would make sense to me here I guess19:48
rockstarlifeless, oh, yeah.  I kinda see what you're saying.  I suspect Tarmac does most of that already (you can get it to format your commit message for you)19:49
lifelessrockstar: the tricky bit will be things like ec2 land --incr19:49
lifelessrockstar: and --no-qa19:49
lifelessrockstar: we may need to change how ec2 land does it for pqm, in advance, if we want to make migration easy19:49
cr3lifeless: a combination of both works too, manage.py here, storm there and interfaces everywhere. something like that19:50
dobeylifeless: what does ec2land do for approval recording?19:51
rockstarlifeless, hm.  I think we just teach ec2 how to set a commit message then.19:51
lifelessdobey: generates the [r=foo] prefixes launchpad use by convention (and that PQM checks against a regex)19:52
lifelessrockstar: something :)19:52
lifelessrockstar: just alerting you is all19:52
rockstarlifeless, when we move to Tarmac, we won't want to be setting "Approved" on the branches, and deferring that instead to having ec2 do it on a successful test run.19:52
lifelessrockstar: I can't quite parse the nested clauses there.19:53
lifelessrockstar: can you rephrase?19:53
dobeylifeless: ah. so, tarmac automatically harvests the list of reviewers and shoves them in a revision property on the new revision, called 'reviewers' :)19:53
rockstarTarmac is pretty dumb (by design).  We could have it grow some brains, but I think having our tools know how Tarmac likes it would be better.19:54
rockstardobey, yeah, kinda like what we already do with the commit message formatter.  :)19:54
lifelessrockstar: pqm is too :P19:54
brycehdoes it only generate [r=xxx] or does it also include the [bug=nnn] and [ui=none] stuff?19:54
lifelessbryceh: it generates it all19:54
* bryceh grumbles about crufty wiki pages19:54
rockstarbryceh, allenap was the last to touch the Tarmac commit message formatter, but I think it'll do bug= as well.19:54
brycehthis is like the 3rd thing WorkingWithDbDevel has misled19:55
bryceher, me on19:55
lifelesslets try my oops branch again19:55
brycehI've fixed WorkingWithReviews - it was mentioning that you shouldn't use [r=xx] but only in a footnotey way19:56
dobeywell, we've moved away from the [r=] bit in Ubuntu One stuff, for branches we're landing with tarmac, since it stores the metadata in revision properties19:57
lifelessprobably we want a dedicated page for landing-stuff19:57
lifelessdobey: sure; the main thing is toolchain friction - need to update a bunch of analysis tools, and in the past folk have said they really like it being visible in trunk with just 'bzr log'19:58
dobeyyeah, the bugs are currently the only thing visible there19:59
dobeybut we could (should) fix bzr to make it much easier to see additional revision properties19:59
adeuring1gary_poster: thanks!20:02
lifelessbryceh: ah so the key thing is 'do not use -s'20:03
lifelessI'll make that edit20:03
lifelessbryceh: also you don't need to add options, you just target your branch to db-devel and it all Just Works.20:04
brycehlifeless, yeah you'll notice I corrected that already20:05
lifelessbryceh: I just used bigger scissors20:05
brycehas a db-devel newbie I found that page full of snakes20:05
gary_posteradeuring1: welcome. :-) please feel free to toss it back after you've tackled it a bit more.20:06
lifelessI can imagine20:06
lifelessbryceh: anyhow, I've just deleted all the stale info around the ectest/land bits20:06
bryceher actually DatabaseSchemaChangesProcess is the one full of snakes20:07
lifelessthe wiki is full of schnaaaaake20:07
brycehdescribes a cases that are actually special cases the ordinary newbie doesn't actually need to care about20:07
brycehe.g. all the emphasis about doing sampledata updates, security.cfg changes, etc.20:08
brycehlifeless, yeah that looks better20:09
lifelessbryceh: partly this is cruft we've accumulated/rtolerated20:11
lifelessbryceh: you do need to do security.py very often20:11
brycehlifeless, in my case I did it but it wasn't needed.  The problem with the docs is it is vague about indicating *when* it should/shouldn't be done (perhaps a link to an appropriate doc about what the file is/does?)20:12
lifelessgood idea :)20:12
lifelessthe README in the schema is probably appropriate20:12
lifelessits visible on bazaar.lp.net/...20:12
lifelessbac: is the review meeting today?20:19
lifelessbac: if so, how many hours away20:19
brycehis https://staging.launchpad.net/ supposed to say 'Code Update In Progress' right now?20:20
lifelessusually does about this time20:21
lifelesshowever, unless a losa restarted it, its bust20:21
lifelesslosa ping20:21
mbarnetthello lifeless20:22
lifelesshi mbarnett20:24
lifelessso staging failed overnight20:24
lifeless(my overnight)20:24
lifelessslony again20:24
lifelessstub isn't around for 6-7 hours20:24
mbarnettlifeless: i am pretty sure tom kicked it off again a couple horus ago20:24
mbarnettlet me check20:25
lifelessstaging is down right now; do we have any way to bring it up or kill-db-clients-harder and start it again.20:25
lifelessI saw chatted in lp-code 2 hours back20:25
mbarnettlifeless: hmm, it looks like a db error that we are actually waiting on stub for.20:33
lifelessmbarnett: the slony timeout20:33
lifelessmbarnett: thats what I said above, isn't it ?20:34
lifelessor is it something differnet?20:34
mbarnettthe slony timeout seems to be the current offender20:35
lifelessMy understanding, totally limited, is that connected clients break slony.20:37
lifelessthat *looks* to my limited understanding to be that happening.20:37
derycklifeless, hi.  Some timeout work questions, if you don't mind.20:42
lifelessI'm all eyes20:42
deryckso I see 2-3 bugs that all seem to come back to adding notifications.  and I'm trying small steps to improving then breaking that out....20:43
derycktrying to just get rid of a flush to start with :-)  See:  http://pastebin.ubuntu.com/486931/20:43
derycknow obviously, this has the downside of doing severl inserts, compared to the original way20:43
lifelessif its not in an inner loop it shouldn't be an issue20:43
deryckah, so that flush I removed is probably not an issue then?20:44
lifelessI don't think it will be20:44
deryckah, ok20:44
lifelessflushes have the following overhead:20:44
lifeless - a cache walk to detect dirty objects20:44
lifeless - DB update/insert calls per modified object to clean them up20:45
lifelessa flush with no modified objects then has a cachewalk only overhead (costly, but not fatal)20:45
lifelessa flush with many modified objects pending behind it is costly, but not because its a flush, because there is a lot of work to do.20:45
lifelessnote that storm, when you do a select, will often flush automatically anyway.20:46
deryckok, that makes sense and is useful to understand.20:46
lifelessthe previous single INSERT INTO is probably maximally efficient.20:46
lifelesswhats the bug report that got you digging into this code ?20:46
deryckand that was my other question, is there a storm way to get that single insert?  i.e. without manually building the sql statement?20:47
lifelesslets have a quick poke at expr.py20:47
lifelesssee the class Insert20:48
lifelessthe docstring on it is fairly clear to me:20:48
deryckok, cool.  it wasn't to me.20:48
lifelesshmm, Insert isn't good enough20:48
lifelesslooking more20:48
lifelessah yes it is, it uses itervalues() which confuses me20:50
lifelessI wonder how it guarantees iteration order20:50
lifelessderyck: https://bugs.edge.launchpad.net/storm/+bug/41155620:50
_mup_Bug #411556: Storm should support multi-row inserts <Storm:New> <https://launchpad.net/bugs/411556>20:50
deryckah, looking at the bug....20:51
=== Ursinha is now known as Ursinha-afk
lifelessderyck: so what bug/oopses are you looking at ?20:52
derycklet me get numbers, I started with 2 or 3 and just poked and ended up where I was at.20:52
=== jtv is now known as jtv-zzz
lifelessderyck: yeah, I don't doubt that we have a few key inflection points for many issues20:54
deryckbug 618403 and bug 611115 and something else that led me here... maybe the one about addComment20:54
_mup_Bug #618403: BugTask:+editstatus-page timing out in ~4% of requests <timeout> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/618403>20:54
lifelessderyck: I'm just curious ;)20:54
_mup_Bug #611115: timeout: bug notifications are calculated in-request <timeout> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/611115>20:54
lifelesshah, I recognise the chap that filed those.20:54
lifelessterrible fellow20:54
lifelesslets see if we can get an oops for the first one20:55
deryckyeah, todays edge OOPS report had that one20:55
lifelessdid it, looking20:55
derycktrying to get on myself.... slow browser20:56
lifelessI think this is fixed20:57
lifelessah no20:57
deryckI think the actual timeout on max(bug.heat) is misleading.20:57
deryckjust before is the call to Bug.addChange, which many of these I looked at had in common20:58
lifelessjust colating20:59
lifelessso I've put my normal stuff-from-an-oops in the tbug21:00
lifelesswhat I've done is:21:00
lifeless - linked the url21:00
lifeless - the query parameters that are needed to make sense of whats happening (if not private)21:00
lifeless - the broad times21:00
lifelessand the sql queries that really stand out21:00
lifelessnote the *300* email lookups - they will be hurting more than 1.6 seconds; I'd estimate 2 or more seconds21:01
lifelessas for the 8 second heat lookup, lets see how it does on staging quickly21:01
lifelesswheeee its slow21:01
lifelessstill waiting21:01
lifelessTime: 14447.014 ms21:02
deryckyeah, i guess it is the max heat select.  I could have sworn the ones I looked at today that was misleading.21:02
lifelessderyck: I don't think the 8 second heat lookup is misleading at all :)21:02
deryckright :-)21:02
lifelessnow, its faster the second time, but I could swear we had this the other day21:02
lifelesslpmain_staging=> select bug.heat from bug, bugtask where bugtask.bug = bug.id and bugtask.distribution = 1 order by bug.heat desc limit 1;21:03
lifeless heat21:03
lifeless 1106221:03
lifeless(1 row)21:03
lifelessTime: 58.710 ms21:03
lifelessthat should fix it nicely.21:03
deryckeasy peasy then :-)21:04
lifelessshould be yes21:04
deryckexcept it makes me nervous.21:04
lifelesshow so ?21:05
deryckI remember us moving to limit 1 before for speed, but then someone changed back to max() saying it was faster.21:05
deryckso now limit 1 is suddenly faster again?21:05
lifelessso this is db query plan hell21:05
lifelesslets check whether this exact query went through that transition21:05
lifelessannotate time!21:05
deryckheh, blame FTW!21:06
lifelessbugtarget.py seems to be the file21:07
lifeless7675.778.1 is relevant21:08
lifeless10580 perhaps21:08
lifelessand 1042821:09
lifelessrunning with -p now21:09
lifelessyeah, 06-25 it was switched21:10
lifelessr=me in fact21:10
lifelessbug 61564421:10
_mup_Bug #615644: BugTask:+distrotask timeout on HEAT lookup <qa-ok> <timeout> <Launchpad Bugs:Fix Released by gmb> <https://launchpad.net/bugs/615644>21:10
lifelessthe patch didn't completely change it one way or the other21:11
deryckright, I thought I recalled this.21:11
lifelessit left projectgroups alone, or they were the current way already or something21:11
deryckyeah, in was inconsistent in the first iteration.21:11
deryckI imagine oversight, not design :-)21:12
lifelessif we used a query builder for this21:12
lifelessrather than complete literals, it would mean only the builder needs changing21:12
lifelessrather than multiple similar clauses21:12
lifelessso I've looked in the whole history21:13
lifelessyes, this has flip flopped21:13
lifelesslets look at that old bug21:13
lifelessand see what context it was searching on21:13
lifelessit may be that distros want one, and products another21:13
lifelessok, so it was a different case stub analysed21:14
lifelessand it still runs fast21:14
deryckstill runs fast meaning the limit runs as fast for the case stub tried as for the case we're at now?21:15
deryckrather than the max he suggested?21:15
lifelessthe current code runs fast for the case that bug 615644 examined21:16
_mup_Bug #615644: BugTask:+distrotask timeout on HEAT lookup <qa-ok> <timeout> <Launchpad Bugs:Fix Released by gmb> <https://launchpad.net/bugs/615644>21:16
lifelessgmb, quite reasonably changed the other similar queries at the same time.21:16
lifelesssadly it looks like the different queries go through the planner differently21:16
lifelesslpmain_staging=> SELECT max(Bug.heat) FROM Bug, BugTask, DistroSeries21:17
lifelesslpmain_staging-> WHERE BugTask.bug = Bug.id21:17
lifelesslpmain_staging->     AND BugTask.distroseries = DistroSeries.id21:17
lifelesslpmain_staging->     AND DistroSeries.distribution=3;21:17
lifeless max21:17
lifeless(1 row)21:17
lifelessTime: 58.259 ms21:17
lifelessthis is likely related to how many bugs are on distribution 321:17
lifeless3 is debian21:18
lifeless1 is ubuntu21:18
cody-somervillehmm... I thought there was a bot that would give a link if I pasted an oops into the channel?21:18
lifelessderyck: so there are two variations here:21:19
lifeless - different query with distro series21:19
lifeless - different distro21:19
lifelesslets try changing 1 to 321:19
deryckyeah, I did.  longer but not horrible.  ~200 ms21:19
deryckwell, not horrible on the order of 8000 :-)21:19
lifelessand the limit approach there is 10ms21:20
lifelesslets try the limit approach for the qyery from the 06 bug21:20
lifelessbingo thats slow21:20
lifeless- no rows21:21
derycklifeless, which query is this?21:21
lifelessI'm trying permutations21:22
deryckI wonder if adding the condition heat not null makes it faster?21:22
lifelesswe have: MAX or LIMIT21:22
lifelesswe have distro 1 and distro 321:22
lifelesswe have with distroseries and without21:22
cody-somervilleDoes anyone know why calling setOwner on a branch via web API OOPses even though its successful?21:22
lifelesscody-somerville: web API ?21:22
lifeless  Module canonical.launchpad.webapp.publisher, line 691, in _handle_next_object21:23
lifeless    raise NotFound(self.context, name)21:23
lifelesscody-somerville: please file a bug21:23
lifelessNotFound: Object: <lp.registry.model.personproduct.PersonProduct object at 0x2ab2b0e7dd90>, name: u'omsk'21:23
lifelessderyck: no change for me adding not null there21:23
lifelesswith the MAX case21:24
derycklifeless, I meant for the limit case, sorry.21:24
lifelessI'm going to fill in this table:21:26
lifelessMAX/LIMIT 1/3 D/DS21:26
lifelessif you're on staging, and wanted to do all the LIMIT ones21:26
lifelessI'll do all the MAX ones21:26
lifelessMAX/LIMIT 1/3 D/DS   TIME21:27
cody-somervillelifeless, Filed bug #62835221:27
deryckah.... checking that distroseries is not null seems to speed up the original offending query using limit 1.21:27
_mup_Bug #628352: Calling setOwner on a branch via web API OOPses even though operation is successful <oem-services> <Launchpad itself:New> <https://launchpad.net/bugs/628352>21:27
lifelessMAX            1      D      2000ms21:28
lifelessMAX            3      D      150ms21:28
lifelessMAX            1      DS    196ms21:28
lifelessMAX            3      DS    2ms21:28
lifelesscody-somerville: I don't know what a 'web API' is21:29
lifelessderyck: adding that in21:29
derycklifeless, the distro series not null condition?21:29
cody-somervillelifeless, http://en.wikipedia.org/wiki/Web_API21:30
lifelessderyck: can you paste the query, so I get it right21:30
lifelesscody-somerville: ugh21:30
lifelesscody-somerville: please say 'LP API'  and you will avoid all confusion21:31
lifelessor just "API"21:31
lifelessat least round these parts21:31
deryckconfirming query, lifeless ...21:31
derycklifeless, slow first, fast second:21:33
deryckSELECT Bug.heat FROM Bug, Bugtask, DistroSeries21:33
deryck    WHERE Bugtask.bug = Bug.id AND21:33
deryck    Bugtask.distroseries = DistroSeries.id AND21:33
deryck    DistroSeries.distribution = 3 ORDER BY Bug.heat21:33
deryck    DESC LIMIT 1;21:33
deryckSELECT Bug.heat FROM Bug, Bugtask, DistroSeries21:33
deryck    WHERE Bugtask.bug = Bug.id AND21:33
deryck    Bugtask.distroseries IS NOT NULL AND21:33
deryck    Bugtask.distroseries = DistroSeries.id AND21:33
deryck    DistroSeries.distribution = 3 ORDER BY Bug.heat21:33
deryck    DESC LIMIT 1;21:33
lifeless(because inner joins imply not null anyhow ..21:34
lifelessdoing the limit cases in the table21:35
derycklifeless, I really appreciate this help, but I need to leave now.  Passed EOD and kids are arriving home.21:36
deryckSorry if I distracted you from other work21:36
lifelessderyck: this sort of stuff is my job :)21:36
lifelessMAX/LIMIT 1/3 D/DS/DSNN  TIME21:36
lifelessMAX            1      D                 2000ms21:36
lifelessMAX            3      D                 150ms21:36
lifelessMAX            1      DS               196ms21:36
lifelessMAX            3      DS               2ms21:36
lifelessLIMIT          3       DSNN         5000ms21:36
lifelessLIMIT          3       DSNN         2ms21:36
lifelessLIMIT          1       DSNN         200ms21:36
lifelessLIMIT          1       DS             2ms21:37
lifelessits missing two rows - LIMIT 'D'21:37
lifelessadding them in a sec, I'll put this in the bug21:37
lifelesscatch you later21:37
deryckok, thanks!  I'll build on this work tomorrow, if you don't beat me to it :-)21:37
derycklater on21:38
lifelessanyone from registry still around ?22:02
jcsackettlifeless, yes. what's up?22:06
lifelessI had a question22:06
lifelessah yes22:06
lifelessis that on your kanban board for moving to a job ?22:06
lifelessbug 10408822:07
_mup_Bug #104088: Time-out problem at merging accounts <merge-deactivate> <timeout> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/104088>22:07
jcsackettlifeless, yeah it's on our OOPS backlog.22:07
lifelessits the 3rd slowest page in the candidate-timeout-reports22:07
jcsackettif you like i can reference it tomorrow at our standup; i'm wrapping up work on some stuff today, and may be able to take a look at it tomorrow.22:08
lifelesswell, its possibly longwinded; I was just cirious22:08
lifelesswhatever work item selection process you are following; keep following it :)22:08
lifelessoh sodium22:13
lifelessthou art the very esssence of a reactive metal22:13
* rockstar afks22:51
=== matsubara is now known as matsubara-afk
jcsackettso if i have a testing situation that is coming up b/c of the way bugs are being cached breaking test isolation, is there a way in a test to clear out the cache?22:52
lifelesswhat sort of cache22:53
lifelessmemcache? cachedproperty? storm cache?22:53
lifelessjcsackett: ^22:56
cr3lifeless: in your refactoring of launchpad, are you considering splitting the database into multiple databases per pillar for example?22:58
lifelesscr3: yes to splitting the database, ENotDesigned to anything further22:58
lifelessand splitting may mean ' one big db using a different tech'22:59
lifelessthere are all sorts of options22:59
cr3lifeless: fun!22:59
lifelessI need the breathing space to get into it though22:59
lifelessget page performance up and the DB will have a lot more headroom22:59
lifelesswe can relax a bit and take time to fix stuff up22:59
jcsackettlifeless: storm, i believe.23:02
jcsacketti'm working off an unclaimed not entirely clear XXX.23:02
lifelessgimme details23:03
lifeless<- data monster23:03
jcsackettlifeless: this is the test case: https://pastebin.canonical.com/36595/23:04
lifelessahha, I wrote this :)23:04
jcsackettif i run that test, it's case, or the file it's in by itself, it passes.23:04
jcsackettif i run the bugs.tests module or more, it fails, with one more query than expected.23:05
jcsackettthis is true both locally and on ec2.23:05
lifelesslet me have a go23:05
lifelessso there are two possibilities23:05
jcsackettlifeless: i suspect it will pass for you. if i run this on devel there's no problem. i think, given the behavior, it's an isolation problem. but it's one that cropped up b/c of my changes.23:06
* jcsackett realizes that might not be what you meant by "have a go"23:06
lifelessone is that the fuzz I permitting in the test is permitting it to pass when bug 619017 triggers, and that when it doesn't you've caussed a scaled bug23:06
_mup_Bug #619017: __storm_loaded__ called on empty object <Storm:Fix Committed by therve> <https://launchpad.net/bugs/619017>23:06
lifelessthe other possibility is that as you say its purely storm caches getting in the way23:07
lifelessso, to reset the storm cache23:07
lifelessyou can do IMasterStore(Bug).flush(); IMasterStore(Bug).reset()23:07
lifelessor something like that23:07
lifelessat the top23:07
lifelesstest_testcase (I think) does this23:08
lifelessits a bit of a heavy hammer, but likely needed23:08
lifeless0.18 should fix this, but 0.18 has a different showstopper bug in it23:08
lifelessjcsackett: what I'd like you to do though, is to compare the two queries23:08
lifelessdrop the 24 to 20 or so23:08
lifelessand run it in the it-fails manner23:08
lifelessthat will get you a query list23:08
lifelessshove that in a buffer somewhere23:08
lifelessand compare it to the query list you get when it fails without modifications23:09
lifelessjcsackett: by have a go I meant run bugs.tests23:09
jcsacketti can do that, lifeless.23:09
jcsackettprobably won't have a comparison for a while though.23:12
lifelessjcsackett: ok, well I'm going to pop out for a chore or two23:14
jcsackettlifeless: i'll be EODing shortly. i'll ping you tomorrow if i have anything informative.23:14
lifelessI'll be delighted to help figure out whats going on when I get back. I suspect a diff will be reasonably helpful though.23:14
lifelessjcsackett: ok.23:15
jcsackettthanks for the help. :-)23:15
jcsackett(and who knows, i may still be poking at this)23:15
lifelessif it is an isolation issue, we'll definitely want to fix it systematically - I'm adding *lots* of these tests.23:15
lifelessand I hope others will too.23:16
lifelessits a great way to prevent scaling surprises23:16
jcsackettlifeless, i hear that.23:17
lifelessone place in particular that may be causing the queries23:17
lifelessis Person._init23:17
=== Ursinha-afk is now known as Ursinha
lifelesslosa ping23:24
lifelesslpoops is telling me23:24
lifelessException Value:23:24
lifeless[Errno 13] Permission denied: '/x/launchpad.net-logs/production/mizuho/2010-09-01'23:24
lifelesson https://lp-oops.canonical.com/oops.py/?oopsid=1705A202623:24
mbarnettlet me take a look23:25
mbarnetthmm, that's a bad path23:26
* mbarnett lies23:26
mbarnettwell, it is.  there is no mizuho directory there23:27
mbarnettwow, i am failing this evening23:27
mbarnettthere is.  checking permissions.23:27
* lifeless sniggers23:27
mbarnetttry again23:31
mbarnettlifeless: that time i got that it didn't match any oopses..23:34
lifelessmbarnett: I got OSError at /oops.py/23:34
lifeless[Errno 2] No such file or directory: 'lp_publish'23:34
mbarnetthmm, reloading it got me that as well.23:35
mbarnett[Wed Sep 01 23:18:25 2010] [error] [client] Unauthorized access attempt for 'https://lp-oops.canonical.com/oops.py/?oopsid=1705A2026' by 'https://login.ubuntu.com/+id/kPbPBDC' (teams: [])23:37
mbarnettlooks like there might be a bug in the lp-oops auth code...23:37
lifelesswould not surprise me23:38
mbarnetti wonder who would handle that..23:38
lifelessmatsubara, but hes a) afk and b) swamped with merge workflow stuff23:39
mbarnettheh, yeah.  i pinged him anywhoo.23:40
lifelessI saw :P23:41
* mbarnett goes to file a bug 23:41
lifelessok, bbs23:41
=== salgado is now known as salgado-afk
thumperlifeless: how do I profile a page?23:51
thumperlifeless: the new cloud is in place23:51
thumperlifeless: but the queries still seem a little slow to me23:51
thumperbut I don't know which bit is slow23:51

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!