/srv/irclogs.ubuntu.com/2010/08/30/#launchpad-dev.txt

lifelesserm00:04
lifeless  File "/var/launchpad/tmp/eggs/setuptools-0.6c11-py2.6.egg/setuptools/package_index.py", line 475, in fetch_distribution00:04
lifelessAttributeError: 'NoneType' object has no attribute 'clone'00:04
lifelessin devel ?!00:04
lifelessDownloading file:///var/launchpad/test/download-cache/dist/setuptools-0.6c11-py2.6.egg00:04
lifeless(on ec2)00:04
deryckHi, all.00:22
lifelesshiya00:30
deryckAnyone else have make blowing up over not having zc.buildout 1.5.1?00:31
lifelessyes00:32
lifelessjust hit me on ec200:32
lifelessas I was about to land the fix for Bug.attachments timing out all over the place00:32
deryckI can't figure out how to fix it.  Rolled back bootstap.py to earlier version in the branch I'm in just to get hacking again.00:33
lifelessderyck: while you are here00:33
derycksure00:33
lifelessdo you know why BugAttachment.message doesn't know its own index ?00:33
lifelessIIndexedMessage seems pretty expensive to generate00:34
deryckI don't.  Meant to look into that when I saw your earlier email, and never got to it with things happening Thurs/Fri.00:34
lifelessso I figure gary must have no landed 1.5.1 into the cache00:34
lifelessi wonder if its on lp or pypi00:35
lifelessderyck: I suspect messages can be shared between bugs00:35
deryckyeah, I wondered that too.  But didn't know if he was rolling his own.00:35
lifelessbut there must be a BugMessage or something that does the actual join00:35
lifelessand thus we could have an index in that00:35
deryckyeah, there is.00:35
lifelesschange bugattachment to go via BugMessage00:36
lifelessand get the index from that00:36
deryckright00:36
* deryck is looking at code....00:37
lifelessshould I file a wishlist bug for this? I have Bug.attachments down to 23 queries - constant -00:37
lifelessbut, the way it does that is to use self.indexed_messages00:37
lifelessderyck: ok, there is  1.5.1 on pypi.org, and it seems to be working.00:39
lifelessderyck: I'm committing it to the download cache.00:39
deryckawesome!00:39
mwhudsonsomething funny is going on00:39
mwhudsonbecause versions.cfg still says 1.5.0 for buildout00:40
derycklifeless, yeah, file a bug on that.  I'll see if I can get work on that for my performance Tuesday work.00:40
* deryck plans to do performance Tuesday this Tuesday again00:40
lifelessderyck: I'd leave it for a little while, my stop gap should get us through to the 5sec 99% barrier00:40
deryckah, ok00:40
deryckcool then00:40
lifeless\o/ about perf on tuesday though00:40
lifelessthere are plenty of other timeouts in bugs code00:41
lifelessmwhudson: well, thats interesting.00:41
lifelessmwhudson: still, committed. we'll see what happens.00:41
mwhudsonoh00:41
mwhudsonhmm00:42
mwhudsonbootstrap.py must be connecting to the internet to find out that 1.5.1 exists00:42
mwhudsonthis would explain why codebrowse restarts were taking 5 minutes00:42
lifelessderyck: https://bugs.edge.launchpad.net/malone/+bugs?field.tag=timeout00:42
derycklifeless, yeah, we're not getting to put the attention on timeouts that I had hoped.  But after this cycle, I'll make sure we're getting the work in.00:42
lifelessderyck: Oh, I wasn't meaning to criticise; I was offering that as a palette00:43
deryckyes, I appreciate the tag list!  I didn't take it as criticism.  Was just saying we haven't put the attention to the bugs as I had hoped to.00:43
mwhudsoni bet this affects edge rollouts too00:44
lifelesshttps://bugs.edge.launchpad.net/malone/+bug/121363 looks pretty shallow00:44
_mup_Bug #121363: Order 'most recently closed' on 'Bugtask.id DESC' instead of 'BugTask.id' <timeout> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/121363>00:44
mwhudsonspm: hi, if you have time, can you tell if the last edge rollout took longer than you'd have expected?00:44
spmmwhudson: actually it's looking like it brokee rather nicely. I'm still chasing.00:44
mwhudsonspm: nice00:45
lifelessit probably broke due to this issue00:51
lifelesswhich I have just addressed00:51
lifelessin a fairly shallow way00:51
wgrantHowever. it still shouldn't be looking anywhere else... What's changed lately? Just buildout 1.5.0?00:53
lifelessbootstrap.py00:53
mwhudsoni guess this is one for that operatic team lead person00:53
lifelessspm: also - bzr: ERROR: Could not acquire lock "LockDir(file:///home/warthogs/archives/rocketfuel-built/launchpad/.bzr/branch/lock)":01:13
lifelessspm: is really annoying me :)01:13
spm./ignore lifeless01:14
spmgah. no wait wrong ignore.01:14
lifelessspm: hahaonlyjoking?01:28
spmlifeless: I can't see your comments anymore. so no idea if you asked if I'm joking or not. maybe.01:57
lifelessspm: :P01:58
MTecknologylifeless: This is what I'm doing now - http://staging.profarius.com/01:58
lifelessMTecknology: so the artwork for launchpad is not open01:59
MTecknologyIt just popped into my head that we're going that and I realized I'd better check it out..01:59
MTecknologys/going/doing/02:00
lifelessI'm just looking for the reference02:00
lifelessah yes, LICENSE in the tree:02:00
lifelessThe image and icon files in Launchpad are copyright Canonical, and02:00
lifelessunlike the source code they are not licensed under the AGPLv3.02:00
lifelessCanonical grants you the right to use them for testing and development02:00
lifelesspurposes only, but not to use them in production (commercially or02:00
lifelessnon-commercially).02:00
lifeless             02:00
lifelessThe Launchpad name and logo are trademarks of Canonical, and may not02:01
lifelessbe used without the prior written permission of Canonical.02:01
MTecknologySo I should definitely remove it..02:01
spivI would think (but IANAL) that if you are using the LP logo specifically to refer to the launchpad.net instance then you're on the right side of trademark law.02:01
wgrantspiv: Copyright-wise that's less clear, though.02:01
spivwgrant: indeed02:01
lifelessspiv: trademark + copyright are needed though02:02
ajmitchcan't login.ubuntu.com be used instead of LP?02:02
lifelessMTecknology: is it general openid you want to refer to02:02
MTecknologywgrant: has there ever been anything clear about legal issues?02:02
lifelessMTecknology: or lp specifically ?02:02
wgrantMTecknology: Frequently.02:02
MTecknologyit uses launchpad teams02:02
wgrantBut not when it comes to LP.02:02
wgrantI mean, the tree isn't even distributable...02:02
lifelessMTecknology: so, I suggest you email.. damn folk are on leave. Thumper. Email him.02:03
MTecknologywill do02:03
lifelessask for the ok. He'll bounce to jml, jml is awl for 2 weeks.02:03
lifelessso it will queue there02:03
lifelessand then get bounced to legal. :P02:03
lifelessthumper: ^ :P02:03
MTecknology:p02:03
MTecknologyI'll make a nice pretty email for him to read :)02:03
MTecknologyI never enjoy legal stuff - I kinda wish the whole world was just under the BSD license :P02:04
lifelessthat has some appeal02:05
MTecknologyI don't think society as is could handle that though :(02:05
MTecknologyI've kinda been fighting the same with with the light-drupal-theme - I want it made for wide distribution - but I need to avoid license and trademark issues02:06
wgrantMTecknology: I would use Ubuntu SSO for that.02:09
wgrantIt's more correct, and the trademark and copyright issues are clearer.02:10
MTecknologyhm?02:10
lifelesswgrant: doesn't have lp team memberships02:10
MTecknologywgrant: for the theme - I mean images like this - http://s.ubuntu.ru/header.png02:10
wgrantlifeless: Doesn't it?02:10
lifelesswgrant: AFAIK, no.02:10
MTecknologyand specific color schemes02:10
wgrantlifeless: They're the same code and DB, but with a different theme.02:10
wgrantlifeless, MTecknology: login.ubuntu.com does send team memberships.02:17
wgrantI just checked.02:17
lifelesscool02:17
lifelessI dunno if its /meant/ to - its hardly separate from LP :P02:18
wgrantIt's meant to.02:18
wgrantAFAIK nobody thought about how the split was meant to work...02:18
wgrantBecause it's still hanging off the LP DB, and probably will always have to.02:18
lifelesswgrant: I wasn't part of that work, can't comment at all.02:19
lifelessbut its really a bit of a bastard child atm02:19
MTecknologylol02:20
wgrantIt tries to be separate from LP.02:20
wgrantBut everything revolves around it having knowledge of LP team memberships.02:20
MTecknologyIs there any particular reason it was broken out?02:23
lifelesssinzui: hi02:23
lifelessMTecknology: to make it usable by U102:23
sinzuihi lifeless02:23
lifelesssinzui: I am planning on nagging spm about the maps CP02:23
MTecknologyU1?02:23
lifelesssinzui: he's dealing with falling buildings atm02:23
sinzuiI was just about to do that02:23
lifelessMTecknology: ubuntuone02:23
MTecknologyoh02:23
sinzuilifeless, okay, that saves me pinging him02:24
lifelessMTecknology: its an (important) branding exercise02:24
lifelessbut its not really split out ATM02:24
thumperMTecknology: isn't it more ubuntu single signon than a launchpad login?02:35
MTecknologythumper: This image has been there since before the ubuntu login02:35
thumperhmm02:36
* thumper shrugs02:36
MTecknologylp:drupal-{openid,launchpad,teams}02:37
lifelessspm: please ping when I can get another profile03:02
thumpertesting bug 60000003:02
_mup_Bug #600000: missing dependency on Bazaar <hitchhiker (Ubuntu):Fix Released> <hitchhiker (Debian):New> <https://launchpad.net/bugs/600000>03:02
spmlifeless: just kicking this imports thing for thumper; then you're next?03:05
thumperta spm03:05
lifelessspm: thanks03:12
spmlifeless: oki; about to do the magic on staging. brb...03:12
lifelesswow https://bugs.launchpad.net/ubuntu/+bug/1/+index is a mess03:20
_mup_Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:In Progress by sabdfl> <ubuntu-express (Ubuntu):Invali03:20
lifelessspm: so I waiting for you to say 'go'03:20
spmlifeless: so am I. KaBoom.03:20
spmstaging appears to be having FireTruck issues03:21
lifelessits going03:21
lifelessoh03:21
spmAn error occurred when trying to install zc.buildout. Look above this message for any errors that were output by easy_install.03:21
mwhudsonspm: big, red doesn't stop at lights?03:21
spmmake: *** [bin/buildout] Error 103:21
spm^^ not my idea of 'going' :-)03:21
lifelessspm: >< - update the source code cache03:21
mwhudsonspm: this'll be the 'connecting to the internet' problem maybe?03:21
lifelessI put a new buildout in there this morning to work around it03:21
spmugh. it didn't STOP!!!!! ARGH!03:21
lifelessspm: is it currently running with profilig on?03:22
lifelessspm: cause I don't care if the code is old03:22
spmsomethings changed in the recent week or so that stops staging/edge from shutting down. *some* of the time.03:22
spmlifeless: doubt it; it's been running since Aug 27.03:22
lifelessgrah03:22
lifelessok03:22
spmie. we'd have faceplanted on disk space by now.03:22
* lifeless emotes kill -9 @ spm03:22
spmI try the regular kill first. the think piece of silk wrapped around the 25 kg sledgehammer. if that doesn't work I take the slik off, and use the -9.03:23
spms/think/thin/03:23
spmright. silk comes off.03:24
spmwe used to have an incredibly aggressive. Try X, fail, try kill, fail, try kill -9 shutdown sequence; that seems to have been removed :-((((03:24
spmmwhudson: I don't think so: Link to http://pypi.python.org/simple/zc.buildout/ ***BLOCKED*** by --allow-hosts03:25
lifelessspm: thats the thing mwhudson is referring to03:26
mwhudsonwell no, that sounds better03:26
lifelessspm: you need to update the dist cache03:26
spmthe internet problem was a timeout; we had last week03:26
mwhudsonthat's being blocked at the application level, not by the firewall03:26
spmhttps://pastebin.canonical.com/36438/03:26
lifelessspm: 11:05 < lifeless>   File "/var/launchpad/tmp/eggs/setuptools-0.6c11-py2.6.egg/setuptools/package_index.py", line 475, in fetch_distribution03:27
lifeless11:05 < lifeless> AttributeError: 'NoneType' object has no attribute 'clone'03:27
lifeless11:05 < lifeless> Downloading file:///var/launchpad/test/download-cache/dist/setuptools-0.6c11-py2.6.egg03:27
lifeless11:05 < lifeless> in devel ?!03:27
lifeless11:05 < lifeless> (on ec2)03:27
spmso I assume tehre's a patch floating around somewhere to ensure we have the lastest hotness?03:27
lifelessspm: is what I was asking this morning. Its the same - AttributeError: 'NoneType' object has no attribute 'clone' - is the magic bit to note, where it has blown well up.03:27
lifelessspm: yes, *update the dist cache*.03:27
spmthat sounds... broken somewhere. why is our automatic updates not getting this automatically?03:28
lifelesssomething like cd lp-sourcedeps/download-cache; bzr update03:28
mwhudsoni would like to understand why it's looking for 1.5.1 though03:28
lifelessspm: appears to be a skew between bootstrap.py and versions.cfg03:28
lifelessor something like that03:28
spmgah. sorry - germanium is trying to eat several hundred Gb of disk asap. bbs... maybe.03:29
lifelesswhats germanium do03:30
spmppa03:30
mwhudsonit's ppa.launchpad.net i think03:30
spmlooks like the leaky /tmp crap buglet we saw a week or so ago. G just has more / space. so took longer to manifest.03:31
lifelessah so a simple rm *03:31
spmwell.... o; but that's the idea. a little more targetted; but.03:32
spmwell *no*. typo win.03:32
lifelessdid you just rm -rf / ? Please say yes....03:32
spmI could try; but it won't work well :-)03:32
spmdoing a verify if /tmp is in fact te problem. it *looks* like it; but I want a little more verification before I start rm'ing away; then I'll prolly start with something like: find /tmp -maxdepth 1 -name 'tmp*' -type d -mtime +7 -delete03:34
spmactually no. dir's; so -print0 | xargs -0 rm -rf03:34
lifeless-type f03:35
lifelessthen -type d :P03:35
spm... | xargs -rn rm -r <== better :-)03:35
spm-0r, argh03:36
lifeless-exec { rm \$1 }; ?03:36
lifelessactually, probably wants a -cut too03:36
wgrantspm: Not my script again? :P03:37
wgrant(killing germanium)03:37
spmlifeless: for this many folders, -exec would be horrible. fork hell.03:37
lifelessspm: clone ftw!03:38
spmwgrant: "yes", even if not your fault; it now is.03:38
lifelessspm: (actually its exec hell, the fork is cheap)03:38
spmwhatever :-)03:38
spmok. 15 mins of trying to get a du summary; I call /tmp/tmp* is the problem and move into rm mode03:39
spmwhee. some srsly old poppy uplods too.03:40
lifelessmwhudson: its odd, bootstrap.py was last change in 1141903:40
lifelesson tuesday03:40
lifelessoh oop, thats my branch03:41
lifelessno, devel agrees03:41
lifelessah, 11452 changes versions.cfg03:42
lifelessbut that doesn't change... ah I wonder if lazr.restful 0.11.3 wants the newer zc/buildout ?03:42
mwhudsonoh03:43
mwhudsonmaybe03:43
lifelessor launchpadlib03:43
mwhudsonthat should lead to a conflict though i think03:43
spmlifeless: (vaguely related aside) - did you ever submit a branch to the prod/staging configs that had the profiler in, but disabled?03:46
lifelessspm: pretty sure03:48
wgrantlifeless: It isn't just because the buildout 1.5.0 has new logic to update, and 1.5.1 was only just released?03:48
wgrantEr, English fail there, but you get the picture.03:48
mwhudsonwgrant: no, because this happens on machines that can't connect to the internet03:48
wgrantmwhudson: And it hasn't been happening since the update?03:49
mwhudsonto buildout?03:49
mwhudsonno, we don't think so03:49
wgrantYeah.03:49
wgrant:(03:49
lifelessspm: ah, its in my branch lp:~lifeless/lp-production-configs/timeouts03:50
spmlifeless: hrm. doesn't look like it. not an active proposal; and we've only had one other change in the past month. Can I pester/nag/trouble/bother you to get that sooner than later? ;-)03:50
spmsnap, ftw.03:50
spmmerge propose away!03:50
lifelessspm: the other change was this branch too, I thought03:51
lifelesshttps://code.edge.launchpad.net/~lifeless/lp-production-configs/timeouts/+merge/3404203:52
spmlifeless: no; the other recent change only has librarian oops changes.03:52
spmta03:52
spmheya poolie03:53
pooliehi there spm!03:53
lifelessspm: so, does this mean we're ready to profile ?:)03:53
spmlifeless: ha you wish. staging is still very much firetrucked. still getting the ambulance round.03:54
lifelessshall we call the cops, go for the trifecta03:54
spmwell, my awesomely wonderful (and delightfully sarcastic) wife is making me a ham sandwich... so........03:55
mwhudsonlifeless: doesn't look like anything depends on buildbot 1.5.1 so it's a mystery (tm)03:59
spmlifeless: approved; pls submit whenever you're ready.04:01
lifelessspm: I've no idea how to do that anymore; it pages out so quickly (and its nuts to use pqm for this)04:02
lifelessspm: perhaps you could submit it ?04:02
spm:-)04:02
spmsure04:03
lifelessthanks!04:03
spmfwiw, we do via pqm; more as a gatekeeper than anything04:03
lifelessI love pqm/tarmac systems (I should shouldn't I :P) but they come with a cost, for benefits. if the benefits aren't worth the cost, it's not useful to use it :)04:04
lifelessspm: so, you're having lunch? Should I pop my head back in in 20 minutes or something for the staging profile ?04:04
spmlifeless: not really... Ill be eating lunch shortly; not breaking; if you ken the difference.04:05
lifelessoch aye04:05
spmtoo many eggs on the boil to afk yet04:05
lifelessso staging - have you tried updating the download cache?04:05
spmnot yet; eed to unfudge edge 1st04:08
pooliehi lifeless (realize lp's eggs are on fire) did anything change with feature flags?04:08
pooliei may go back to it say wednesday04:09
pooliehaven't read much mail ypet04:09
lifelesspoolie: sinzui has a patch that uses them to control google maps04:09
lifelesswhich are currently stuffed due to a google side problem04:09
pooliesweet04:09
* poolie spoke too soon04:09
pooliebut i'm glad to see some takeup04:09
lifelessit should get CP'd todayish :P04:09
lifelessspm: edge almost certainly has the same problem. have you tried updating the download cache :P)04:10
spmlifeless: that would emphatically be Doing It Wrongâ„¢04:10
spmmanual intervention on every server? yuk.04:10
lifelessspm: well, doesn't your deploy script do that ?04:11
spmhrm. from the same state; the question is why is that state borked.04:11
lifelessspm: kick off a new deploy, done :P04:11
lifelessspm: so we don't know the root cause, but the package I put in the download cache about 4 hours ago seems to work.04:12
spmahh. I see. edge borked is the failed stop on edge4; it never updated to latest shiny.04:13
lifelessright04:13
lifelessI strongly suspect, if the pastebin you gave is what the error was, that kicking off a new one, which should as a matter of course grab the latest download cache, should work.04:14
spmedge hasn't (yet) exhibited) that error...04:16
spmhas special errors all of it's very own04:16
lifelessshow n tell ?04:17
spmper above; failed to stop and never got updated; so when I stabbed and started it; it came up as the wrong revno.04:18
lifelessah04:19
spmwe have a check for that you see :-)04:19
lifelessspm: so what do you think is wrong?04:25
spmlifeless: with edge in this case? it's a reversion on an old bug faict. have just logged bug#62657704:28
_mup_Bug #626577: app servers not shutting down (again) <canonical-losa-lp> <Launchpad Foundations:New> <https://launchpad.net/bugs/626577>04:28
spmactually - that's a moderately deadly one for continuous rollouts btw. if we need manual intervention to just stop/start servers; that project is pretty much dead before started.04:29
lifelesswe will fix04:30
lifelessthe more we do something the more we'll improve it04:30
spmOh i know that; was just saying :-)04:30
spmstill the root cause of that bug has never been fixed. what actually causes the daemons to lock like that and not shutdown correctly.04:31
lifelessspm: did you get a per-thread stack trace?04:31
spmway back when; probably. we spent a fair bit of effort ages ago on it. given we had a (hack) fix; it was deemed not worth the time and effort to keep chasing further.04:32
spmargh. so staging full udpates died on a replication fail.04:33
lifelesswas it the new table again ?04:37
lifelessspm: anyhow04:37
lifelessspm: I don't care about updates, I just want running + profiling :)04:38
lifelessspm: can we do that, and you fiddle with updates after ?04:38
spmwell, tbh, I'd really rather not. otherwise I'm just duplicating the same work and somewhat wasting time messing around with something that needs fixing at the root anyway04:38
lifelessspm: ok04:39
lifelessspm: I'll go do some other stuff and check back with you in 20-3004:39
spmI'd hazard at least part of an educated guess that the pqm spam is all realted to stagings borked ness04:40
lifelessspm: the db-devel update is due to sodium crashing mid-update04:40
lifelessits left stale bzr locks04:40
lifelessat a WAG04:40
spmyeah - but those files get auto synced around. so a partial update gets synced around....04:41
lifelessouchies04:41
spmright; and we have a dir that's missing stuff. so I call 1+1 = maybe 2 :-)04:41
StevenK1 + 1 = 1.8 ish?04:41
persiacloser to e04:41
spm2.1, you could average04:41
lifelessspm: so how goes it ?05:03
spmwell we have successful rf builds again.05:05
spmright. next regular staging code update is in ~ 5 mins. see how that goes as an automated thing.05:09
thumper:((05:11
* thumper wishes jelmer lived in NZ or AU05:12
persiaWhat do you have against the northern hemisphere?  We've a few active timezones on this side of the globe.05:12
persia(some east of some AU timezones)05:13
lifelesspersia: thats only because .au is silly-wide05:13
* persia is reminded of Negativland's "Time Zones"05:15
mwhudsonthe name of that ci tool that's written in java *still* trips me up05:16
persiaIn terms of increased highlight count?05:17
mwhudsonno05:18
mwhudsonjust reading emails05:18
mwhudson"meet hudson" <blink>05:19
persiaAh, so yes, but with a wider definition of "highlight" to be more a cognitive filter than any technical notifaction system :)05:19
mwhudsonright05:19
thumperpersia: it is more that I want jelmer around RIGHT NOW05:22
persiaI figured, but just felt you were unfairly discriminating against landmasses north of the equator.05:23
* ajmitch thinks it's a fairly reasonable wish05:25
lifelessthumper: what do you want from jelmer ?05:27
spmlifeless: mwhudson: https://pastebin.canonical.com/36440/ <== that's getting zotted at source, so to speak. if the staging restore/build can't get those files; we have a bigger problem up front.05:28
lifelessspm: its in the download cache now05:29
lifelessspm: I don't know how staging updates work05:29
lifelessbut if they are not updating the download cache, its a fail.05:30
lifelessif they , that file is available.05:30
thumperlifeless: I've emailed him about a git import failure05:30
lifelessspm: can you please confirm that zc.buildout-1.5.1.tar.gz exists on disk on staging05:30
lifelessoh joy05:36
lifelessec2 hates my branch, but it works locallyt.05:37
lifelesswhat could it be? ... storm05:37
spmlifeless: so. cm.py (rf built) runs on sodium. 'builds' the tree; that gets pulled on sourcherry, which every half hour does a staging 'restore'. pushes that where it needs to go.05:38
lifelessspm: ok, so lets check on sodium05:39
spmdbupdates are done once 1 week; superset of the normal code only updates05:39
lifelessdoes it have a download-cache there05:39
lifelessgrah this is nuts05:44
lifelessany advice on debugging a 'works locally, fails on ec2' issue ?05:44
lifelessI've zapped my storm local bugfixes, running whatever the dist facility made05:44
mwhudsonlifeless: how is it failing on ec2?05:45
lifelesstwo ways05:45
lifelessone05:45
mwhudsonare we using lucid ec2 images yet?05:45
lifelessmy queries-are-constant test fails05:45
lifelessthe check for constant finds the second run did 4 less queries05:45
lifelessthe second thing is a check in bug.txt that something takes 2 queries fails - it takes 3 (which is probably appropriate)05:46
lifelessFile "lib/lp/bugs/tests/../doc/bug.txt", line 1133, in bug.txt05:46
lifelessFailed example:05:46
lifeless   len(CursorWrapper.last_executed_sql) - queries05:46
lifelessDifferences (ndiff with -expected +actual):05:46
lifeless   - 205:46
lifeless   + 305:46
mwhudsonlifeless: the only thing that leaps to mind is an isolation failure, some test is leaving something with a __del__ that does queries or something behind05:46
mwhudsonbut that seems a bit mental05:47
lifelessso they do definitely both pass in isolation05:47
mwhudsonlifeless: have you tried running not just your test but a few around that?05:47
lifelessI guess I'll run a .. yes05:47
mwhudsonsounds horrible though05:47
lifelessfortunately subunit streams give me a nice log05:47
mwhudsonyeah05:47
lifelessI'm fully merged with devel too05:48
lifelessfor kicks05:48
mwhudsoni guess the failure of your new test doesn't include the statement logs?05:48
lifelessit does05:48
lifelessit doesn't include the statement log for the *first* attempt05:48
mwhudsonah05:48
lifelessthats on my todo to fix (use a currying approach)05:48
lifelessget a matcher which builds a second matcher for you05:48
mwhudsonis it the same number of statements locally the second time05:48
lifelessand then diffs the statements05:48
mwhudson>05:48
mwhudson?05:48
mwhudsonthat'd be awesome05:48
lifelessyes, locally its the same both times05:49
mwhudsonso it really is the first time doing fewer queries?05:49
mwhudsonthan locally05:49
mwhudsonthat's very odd05:49
lifelessno, the first time is doing more05:49
* mwhudson rubs eyes, learns to read05:49
lifelessbut I can't imagine a flush being batched up05:49
lifelesssorry, crossed wires05:50
lifeless*that* count change is in bug.txt05:50
lifelesswhich I didn't really touch at all.05:50
lifelessthis is my one:05:50
lifelessMatcher: HasQueryCount(Equals(23))05:50
lifelessDifference: queries do not match: 23 != 1905:50
lifelessthe 23 is the count from the first API call05:50
lifeless   self.assertThat(collector, HasQueryCount(Equals(with_2_count)))05:50
mwhudsonand locally both are 23?05:50
lifelessis what generates that05:50
lifelessI think so. Locally both are < 24.05:50
lifelessI'll see if it can go lower, I don't think it can.05:51
mwhudsonso you have the 19 queries that were executed the second time?05:51
lifelessyes05:51
lifelessI can pastebin05:52
mwhudsoni guess you could try diffing that to a local run05:52
lifelesstrue05:52
lifeless23 is the local count05:54
lifelessso ec2 is missing05:54
spmright. versions on db-stable and sourcherry/sodium now match. re-running code staging restore atm. pray.05:54
* spm afks for school run & lunch05:55
lifelesssorry, not missing05:55
lifelessthey are different05:55
lifelessthe 19 we get are the same for the OAuthNonce dance05:55
lifelessthen we get a05:55
lifeless6, 6, 'launchpad-main-master', 'SELECT 1 FROM Person WHERE Person.id = %s')05:55
lifelesswhich I don't see locally05:55
lifelessmaybe as many as 8 queries05:56
lifelessselect 1 from looks like an updated/changed .exists()05:58
lifelessmwhudson: I propose to make the constant check a < check05:58
lifelessmwhudson: that is, it can get lower, it can't get higher.05:58
mwhudsonyay for AP falling over06:00
* mwhudson has to run away anyway06:00
mwhudsonlifeless: good luck06:00
lifelessmwhudson: do I have your ok for minor hit-it-hard-tweaks06:00
lifelesslike06:00
lifeless-        self.assertThat(collector, HasQueryCount(Equals(with_2_count)))06:00
lifeless+        self.assertThat(collector, HasQueryCount(MatchesAny(06:00
lifeless+            Equals(with_2_count),06:00
lifeless+            LessThan(with_2_count))))06:00
lifelessI'm pretty sure this is a storm version skew issue or something like that06:01
lifelessI've seen the storm cache cause all sorts of stuff (and been filing bugs and mps because of it)06:01
lifelessah yes, its definitely storm.06:13
lifelessbug 61901706:13
_mup_Bug #619017: __storm_loaded__ called on empty object <Storm:Fix Committed by therve> <https://launchpad.net/bugs/619017>06:13
lifelessit causes spurious person queries when initialising cached objects06:13
lifelessI think its causing both06:14
lifelessbecause getMessageChunks issues a Person lookup too06:14
lifelesscan probably rewrite that to do a single query instead, but the bug will still shoot us.06:15
stublifeless: How are you using a different version of Storm?06:28
lifelessstub: I was running a version with cache fixes06:28
lifelesspending 0.18 being released06:28
stubRight. So any reason not to add that to buildout?06:28
lifelessyes, the fix provokes another bug06:29
lifelessthat bug is more severe - it stops things dead if triggered, and gustavo is working on the fix06:29
lifelesshe gave me a 'give this a shot' version06:29
lifelessbut untested, I didn't want to land that for all devs, better to wait for 0.1806:29
stubRight. And getting exact query counts isn't going to work with version skew.06:29
lifelessanyhow, the query count variation is entirely explained by Person._init lookups06:30
lifelessstub: the tests I write set an upper bound and then check for consistency, so they should not be very sensitive to storm versions, unless the version has brain damage of some sort06:30
lifeless(which, sadly, 0.16 and 0.17 did, in different ways)06:30
stublifeless: Consider putting a key -> query count mapping in an external file. If we get lots of these tests, we might want to update the counts in bulk as part of a Storm update. Maybe YAGNI.06:45
stubHmm... a test fixture that automatically does the count check, unless the magic wand is waved and it records the count check...06:46
lifelessstub: that might be nice; need to find a way to say 'this is the bit to track07:40
lifeless'07:40
lifelessstub: that said07:40
lifelessstub: if we have a storm update that wants to raise query counts across the board, I'll just send it back with 'no thanks' written in technicolour across its flanks07:40
lifelessstub: something that lowers query counts won't break the tests.07:41
lifelessnononono test fix nonononoonono07:41
lifelessgrah07:42
lifelesssinzui: poolie: sadly there is an API skew between devel and production for this07:43
lifelesshttps://lpbuildbot.canonical.com/builders/prod_lp/builds/91/steps/shell_7/logs/stdio07:43
lifelessspm: whats the blessed way to rollback an lp-prod change (and how did this get so far before being picked up ?)07:45
poolielifeless: "this" being?07:45
lifelesssinzui used the flags API07:45
spmeek. probably via a cowboy and/or CP vs actually rolling back. where either would essentially revert the change.07:45
lifelessbut you changed the meaning of something - look at the backtrace in the link I posted - between the last release and this release, so his CP to use flags to disable gmaps fails horribly07:46
spmnegative patch if you ken.07:46
spmlifeless: so. um. don't try and rollout that CP? I was just about to...07:46
lifelessspm: hell no07:46
lifeless83 failures 1 errors07:47
* spm searches that response for possible subtleties...07:47
spm\o/07:47
lifelessput it this way; users profile pages would be - more- broken.07:47
spmlifeless: can you rm that from the requested CP list? actually - it shouldn't be on the CP list until such time as it passes and lands....07:48
lifelessspm: I've marked it as snafued07:50
lifelesswhich sinzui will see07:50
spmcoolio; ta07:50
spmjust don't want an accidental rollout by the wrong one07:50
lifelessnaturally.07:52
lifelessactually, we can do this much more quickly.07:52
lifelesscan I get you to be remote fingers ?07:52
spm<ding> yes master, what is your will.07:53
lifelesson praseodymium07:53
spmprasé to it's mates. fwiw.07:53
lifelessfirstly, check that the archives/... production-devel and production-stable are up to date with LP - I presume PQM still uses local disk as authoritative ?07:53
spmshould do...07:54
lifelessjust a 'bzr missing lp:~launchpad-pqm/launchpad/production-stable' should be fine07:54
lifeless(and -devel, in -devel)07:54
lifelessspm: then, in production-devel07:56
lifelessdoes it have a working tree - are there files present? If no, do 'bzr checkout .'07:56
lifelessfinally, 'bzr merge . -r -2..-1'07:57
spm~/archives/rocketfuel/launchpad/production-devel <== Branches are up to date.07:57
lifelessthis should affect only files in lib/lp/registry07:57
spmum. actually. I may have done that wrong. pebkac...07:57
lifelessspecifically, browser/__init__, person, team, tests/mailinglistviews, tests/person-views, 3x stories and 2xtemplates07:58
spmproduction-stable You are missing 5 revision(s): 9649 - 965307:58
spmprod-devel is fine07:58
lifeless-stable doesn't actually matter for this07:59
lifelessthe failure was on -devel07:59
spmoh. and stable is CP's. that only gets updated locally with a full rollout.07:59
lifelessI wasn't sure when we started (i was reading just ahead of what I typed)07:59
spm:-)07:59
lifelessok, so in -devel07:59
lifelessis there a tree of files?07:59
lifelessand shit08:00
spmyup08:00
spmto both.08:00
lifelessbzr update08:00
lifelessthen08:00
spm(bzr add shit)08:00
lifelessor08:00
lifelessbzr st08:00
lifelessshould be empty08:00
spm$ bzr st ==> working tree is out of date, run 'bzr update'08:00
lifelessrun bzr update08:01
lifeless:P08:01
spmjust ensuring we're all on the same page :-)08:01
lifelessits good08:01
spmUpdated to revision 9655.08:01
lifelessbzr st08:01
lifelessshould be empty08:01
spm+108:01
lifelessbzr merge . -r -2..-108:01
lifelessbzr diff -r -208:01
lifelessshould be empty08:01
spmbzr merge --preview . -r -2..-1 <== nothing. good enough?08:02
lifeless#bzr commit -m '[rs=lifeless][rs=spm]08:02
lifelessspm: huh, no, merge --preview is doing to be totally confused here.08:02
lifelessspm: bad fingers!08:03
spmactually - before I make the change; can we step back a bit and get an explanation of what we're doing? reverting a change I gather?08:03
lifelessyes08:03
lifelessthe reverse patch08:03
lifelesswhich is done by 'bzr merge . -r -1..-2', and confirmed by checking that there is no difference against the second commit back.08:04
spmOh I see.08:04
StevenKBut you said -r -2..-1 earlier?08:04
lifelessyes08:04
lifelessthat was a teset08:04
lifelessI failed it.08:04
lifelessthe diff step would have caught it regardless08:04
spmwhat I meant in this case was to (via pqm) submit a reversal patch; not fiddle around with the core players directly.08:05
lifelessspm: I can do that if you prefer, its rather pointless IMO : the last commit back is known-good08:05
spmmainly as we have so many moving parts; it's really easy to miss one; whereas going via the normal steps, we achieve bliss.08:06
lifelessspm: PQM *wasn't used properly* or we wouldn't have the branch broken.08:06
spmheh08:06
lifelessspm: if PQM was running the test suite, not running in brain-damaged mode, this wouldn't have happened.08:06
lifelessits up to you : we can be in test fix for another few hours08:06
lifelessor we can be out in 5 minutes.08:06
spmOh. I think i see. we had a branch submitted directly to prod-devel that hadn't actually passed the testing?08:07
lifelessthat or this branch passes randomly 50% of the time.08:07
* spm eods in 55 mins. the former option is looking appealling >:)08:07
lifelessI think submitted without testing is what happened.08:07
spmfair enough08:07
lifelesstested in devel, and in dbdevel and db-stable, but not vs prod-devel in ec208:08
spmright08:08
spmoki, making magic fingers.08:08
lifelessso,08:08
lifelessbzr merge . -r -1..-208:08
lifelesswill back out the last commit08:08
lifelessbzr diff -r -208:08
spmprod-devel, right.08:08
lifelessshould be empty08:08
lifelessack08:08
lifelessif that diff is empty the tip commit's contents are gone.08:09
spmum. hang a sec. bzr merge . -r -2..-1 is what you had earlier; vs -1..-2. does it matter?08:09
lifeless#and we can bzr commit -m '[rs=lifeless][rs=spm] Back out08:09
lifelessyes, I had it wrong before (and the diff would have show n that)08:09
lifelesswe want 'from tip to the one before'08:09
lifelessrather than 'from the one before to tip', which already exists in the branch.08:09
lifelessa --preview here shold in fact work - try if you like08:10
lifeless bzr merge --preview . -r -1..-208:10
lifelessand we can bzr commit -m '[rs=lifeless][rs=spm][rollback=9655] Back out broken Google maps API fix due to feature flags api differences with production-devel vs devel.'08:11
spmlifeless: for verification: https://pastebin.canonical.com/36441/08:11
lifelessspm: that looks right. Is bzr diff -r -2 empty ?08:12
spmyup08:12
lifelessbzr commit -m '[rs=lifeless][rs=spm][rollback=9655] Back out broken Google maps API fix due to feature flags api differences with production-devel vs devel.'08:12
spmci'ing...08:12
lifelessand bzr push08:12
spmUsing saved push location: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/production-devel/08:13
spmPushed up to revision 9656.08:13
spmOh I see what you've done. rather then "uncommit/revert" to get back to X-good. We deliberately roll the reverse patch. Nice! Very clean.08:14
lifelessthank you _very much_08:14
* spm adds that nice littel trick to his 'bzr notes'08:14
lifelessuncommit *count* run into trouble if some branch somewhere had pulled in the intervening time08:15
lifelessand then would need a --overwrite to fix up etc etc08:15
spmright08:15
lifelessso, two las tthings08:15
lifelessproduction-devel buildbot slave is dead08:15
lifelessplease shoot, corpse in the river.08:15
spmle-sigh. again?08:15
lifelesshttps://lpbuildbot.canonical.com/buildslaves08:16
poolielifeless: i don't think the test_addmember failure is related to anything i did08:16
lifelesspoolie: I don't think you've done anything wrong, but I am pretty positive that production's feature flag module treats *something* differently to devels08:17
lifelesspoolie: or sinzui's branch wouldn't have passed buildbot for devel->stable, db-devel->db-stable, but blown up on production-devel08:17
pooliethat's quite possible08:18
lifelesspoolie: I think it would be nice to harmonise them, either by cping your further flags work to production, or something.08:18
poolieit's a bit opaque which version is running where08:18
poolieobviously it is in the bzr branches08:18
lifelesslp:~launchpad-pqm/launchpad/production-stable is whats deployed08:18
lifelessyes, it would be nice to have this a lot clearer.08:18
lifelessspm: secondly, I can has profile ?08:19
spmlifeless: prod-lp kicked off a new ec2 instance just a few mins ago. so not so much wedged as 'still instansiating"08:19
lifelessspm: ok cool., I thought they were real machines now - mea culpa08:19
spmand now running apparently. no, just the lucid ones08:19
spmlp, db_lp and prod_lp are still ec208:20
lifelessI'll try to remember. Thanks :)08:20
lifelessspm: so that leaves just getting a profile from staging :)08:20
spmis it back yet? I checked about an hour ago and staging was still restoring (successfully tho)08:20
lifelessfront page comes up08:21
spmit LIVES!08:21
spmso the 'fix' was simply ensure the latest shiny was loaded for sourcherry to try and rollout; other problems around access etc went away. fwiw.08:22
lifelessyeah.08:22
lifelessfragile sucks08:22
spmi think my first test was a little too early and hence a few versions behind.08:22
* lifeless loads more ammo in the just-one-dep-style-please gun08:22
lifelessspm: ok, so enable-profiling-and-restart time? pl-pl-please ?08:23
spmlifeless: go for it, live.08:26
lifelessthanks08:28
lifelessgot it08:28
spmrevert?08:28
lifelessplease08:28
lifelesswe'll know once it rsyncs if its good or not, but no point slowing up the system in the mean time08:28
spmrestarting....08:29
lifelessOOPS-1703S29808:29
spmoh gah. sodium seems to still be borked.08:29
poolielifeless: so production-stable's tip is changed pretty much atomically with it actually being deployed?08:30
StevenKspm: Borked how?08:30
spm"hardware"08:30
StevenKspm: Bleh, still :-(08:30
spmnormally it recovers on it's own; but seems to not be coming back.08:31
spmheh, yeah. I believe it'shad just about everything replaced and still dies.08:31
StevenKPerhaps it's wet, sodium reacts badly to water. :-P08:31
lifelesspoolie: AIUI08:31
spmI made that joke /fosty response at the joke theif08:31
StevenKHaha08:32
lifelessStevenK: elmo says that they've replaced/reassembled the entire thing08:32
lifelessits going to be totally replaced, its queued to do so.08:32
StevenKlifeless: IE, new name, new everything?08:32
lifelessit'll still be sodium I think :)08:32
lifelessbut new chassis & guts08:32
spmit'll still be devpad; maybe not sodium.08:32
lifelesstrue08:33
lifelessI live in hope, its a cool name08:33
spmaiui, nafallo gets naming rights on new boxes. so....08:33
spmwhich explains some of the more .... well lets just say: thank $deity for ssh tab completion on names08:33
wgrantThe original armel builders had nicely obscure names. The new ones aren't so good :(08:35
StevenKI thanked Nafallo for those. He took it as a compliment.08:35
wgrantHeh.08:35
poolieStevenK: your hudson url gives 'connection timed out' for me08:35
StevenKpoolie: Let me check, I think I'm a muppet08:36
StevenKpoolie: Should work now08:36
lifelessStevenK: are you wearing the muppet hat?08:37
lifelessStevenK: also congrats08:37
pooliealso, does it really need to be private?08:37
wgrantDoes canonical.com support Unicode subdomains?08:37
poolieistm the readonly mode could be public08:37
wgrantNafallo could have looots of fun with that :P08:38
spmlifeless: sodium should be back08:38
lifelessspm: can you make the rsync magic magically happen ?08:38
spmonly by magik08:38
StevenKpoolie: I'm paying for the box currently, so I'd like to limit the number of people that can fiddle for the moment08:39
spmlifeless: syncin'....08:39
lifelessspm: would that be the magic bus ?08:39
spm#42, yes.08:39
poolieStevenK: logging in sends me back to the default apache "it works" page08:40
spmlifeless: should be there now08:40
lifelessthanks08:40
lifelesspulling08:40
spm /srv/launchpad.net-logs/staging/asuka/08:40
pooliei guess because of going back to http not https y08:40
StevenKpoolie: Right.08:40
poolieyou probably just want a redirect there08:40
lifelessspm: /profiling/ :P08:40
StevenKpoolie: I just put one in, you were probably too fast08:40
spmyeah, that too.08:40
pooliei would have expected to see some history for previous builds?08:41
poolieStevenK: istm that allowing anonymous readonly access wouldn't cost you very much?08:41
pooliei'm not suggesting allowing people to start new builds08:41
lifelessooh shiny this looks like one I may be able to stop on hard08:41
pooliejust to see if the previous ones wokred08:42
lifelessspm: also you can probably clear out > 3 day old profiling things automatically (rephrase - we need to :P)08:42
StevenKpoolie: Hmmm, I can do that08:42
pooliejust an idea08:42
lifelessspm: as we're going to have on-demand profiling soonish.08:42
spmgah. I thought I'd done something like that; maybe just manually...08:42
StevenKpoolie: And history should be there, drill down into the jobs08:42
lifelessspm: I know you did manually the other week08:43
wgrantStevenK: What's this thing?08:43
spmrm 2008-09-16_2* <== no.08:43
StevenKwgrant: A hudson install08:43
StevenKwgrant: Have a look: https://hudson.wedontsleep.org/08:43
lifelessStevenK: have you hooked up junitxml test reports ?08:44
StevenKlifeless: Yup08:44
lifelessawesome08:44
poolieStevenK: hm it wasn't there before08:45
lifelessyeah, thats shiny08:45
lifelessbah, that failure isn'y very useful though :(08:45
lifelesshttps://hudson.wedontsleep.org/job/db-devel/4/testReport/junit/lp.codehosting.puller.tests.test_worker/TestWorkerProgressReporting/test_network/08:45
StevenKI don't understand the test failures :_/08:46
wgrantStevenK: Shiny.08:46
* StevenK kicks apache until his redirect works08:46
poolieStevenK: is this supposed to eventually supersede buildbot?08:47
StevenKpoolie: I'd like for that to be the plan08:48
StevenKI suspect others would too08:48
lifelesshmm, something funky in the test times08:48
lifelessEMUSTLOOKATTHATSOMEDAY08:48
StevenKIt doesn't currently use ec2 to build since I suspect it would bankrupt me within 2 or 3 days08:49
lifelessget a UEC account08:49
poolielifeless, the dupefinder on edge seems to be giving me pretty weird results recently08:49
poolieis that affected by your search changes or is it no more weird than usual?08:49
lifelessmy search changes were all about the dupefinder08:49
lifelessit used to | all the terms08:49
StevenKlifeless: Er, but it isn't really work?08:49
poolieshall i file a bug or is this known or wontfix?08:49
lifelessthis performed terribly08:49
lifelessStevenK: its very much work :)08:50
poolieyes, i remember08:50
lifelesspoolie: if you can quantify whats going on, please do file a bug.08:50
lifelesspoolie: however, until we replace the search engine, I don't think there is much we can do sensibly.08:50
StevenKlifeless: It's hosted privately, it's been developed privately during spare time; doesn't sound like work to me08:50
lifelessthe performance curve is terrible.; perhaps we could estimate the size and use more broad searches when it won't blow up08:51
* StevenK is sad his mail hasn't had any replies yet08:51
lifelessStevenK: thats all true; what I mean is that what you are doing is of benefit to the team; and canonical as a sponsor of the team should - dare I say would - be happy to help08:51
lifelessspm: man, MailingListApplication:MailingListAPIView must get hammered08:52
lifelessspm: every time, its like 90% of the profiles.08:52
StevenKlifeless: So, what do you think the next step should be?08:52
spmlifeless: unknown08:52
lifelessStevenK: I think getting it to the point that you're fairly confident a failure is genuine is important.08:53
StevenKlifeless: Agreed08:53
StevenKI've had one db-devel build pass08:53
lifelessStevenK: at that point, workflow changes to make it what the pqm buildbot thing does would probably be conceivable (for testfix mode stuff)08:53
StevenKlifeless: So it should remain where it is for the time being while we work out kinks?08:54
lifelessStevenK: its what I'd do08:54
lifelessmoving it into prod while its in dev would just add friction to your ability to tweak and fiddle.08:54
StevenKOh, clearly08:54
StevenKlifeless: But should we investigate using UEC as executors in parallel to that?08:55
lifelesssure08:55
lifelessI expect gary and or maris will respond08:55
lifelessI will definitely reply tomorrow if noone else has08:55
StevenKYes, I'm looking forward to that08:55
StevenKThe testsuite didn't like running with only 512MiB of RAM08:56
poolielifeless: so you did change it to matching N-1 terms?08:56
lifelessyes08:56
lifelessvs 1 :P08:56
lifelesspoolie: specifically it searches for a match in any of the N-1 subsets, and scores across all of them08:57
lifelessso the more detailed you are, the less results you'll get08:57
lifelessbut, unlike before, you can trim too-many-results by being more detailed08:57
pooliethat's true08:58
pooliebut not a very obvious use of this dialog08:58
lifelessyes08:59
lifelessI'd have liked to have kept it just as it was08:59
lifelessand hope to restore that when we overhaul search08:59
poolienot complaining08:59
lifelessI know - just expounding09:00
lifelesshah!09:00
lifelessis_empty cheap09:00
lifeless22% of bug one is in that 'cheap' query09:01
lifelessjtv: ^09:01
jtv22% of Microsoft's market share?09:01
lifelessno09:01
lifelessof rendering bug 109:01
_mup_Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:In Progress by sabdfl> <ubuntu-express (Ubuntu):Invali09:01
pooliehttps://bugs.edge.launchpad.net/malone/+bug/626656 fwiw09:01
jtvThank you mup, we had that one perma-cached09:01
jtvlifeless: that's astounding09:01
lifelessjtv: not to me09:02
_mup_Bug #626656: dupefinder now over-tight <Launchpad Bugs:New> <https://launchpad.net/bugs/626656>09:02
poolieand fwiw it did in fact fail to find my dupe when used in the usual way09:02
lifelesspoolie: thanks; an open question is whether the usual way should be changed09:02
lifelesscertainly for apport bugs I expect the current behaviour to be ok09:03
poolieyeah, arguably we should not just restore the old behaviour but instead reconsider the story of bug filing09:03
pooliehm09:03
poolieapport bug duping seems problematic in different ways09:03
adeuringgood morning09:05
pooliehi abel09:05
jtvhi adeuring09:05
adeuringhi jtv!09:05
lifelessjtv: I think its easy for folk to under-estimate the impact of repeated small queries09:06
=== almaisan-away is now known as al-maisan
lifelessjtv: its more work for the db - scales N rather than log(N); its more friction up and down the call stack.09:07
lifelessjtv: in short, it adds up - a lot-.09:07
jtvlifeless: otp09:07
lifelessjtv: de nada, catch you another time.09:07
mrevellMorning09:08
=== StevenK changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.09 | PQM is OPEN | firefighting: - | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews
StevenK... I think it's week 309:08
jtvhi mrevell, otp but your help bubble is up on edge.09:13
poolielifeless: oh, random thought from somewhere on the highway09:22
poolieis it at all possible to emit (perhaps shrunken) versions of sql queries into the html as its rendered09:23
pooliein comments, obviously, and of course only for some users09:23
poolieor will the template/view layering mean that we don't have a good view of when the query was run, or what caused it?09:23
lifelessyes, but it would probably cripple things right now09:23
pooliein what sense?09:24
lifeless(because template rendering is already a slow spot, and many pages do silly-count numbers of queries.09:24
poolieah right09:24
lifelessso we'd be adding a lookup into a slow code path, that would be exercised a lot.09:24
pooliesure09:24
poolieperhaps it could be turned on by a cookie or query parameter09:25
pooliebut it might still be too much09:25
poolieand the implementation may not be trivial09:25
jtvlifeless: off phone… about the is_empty queries: sounds like a Foo, count(*) query in disguise.09:25
pooliebecause it can't insert comments just anywhere09:25
jtvpoolie: you want to be able to trace back queries to the TAL that necessitates them?09:26
poolieyes09:27
poolieperhaps people already have tools to do this or can almost always guess correctly?09:27
jtvI would like that too.  But getting pretty deep into the critical path.  What if we had a way of inserting HTML comments with the current query count, so we could couple them to the oops report?  Not great, but relatively low impact.09:28
poolieistm on the way to implementing that, you'd want a thing to tell TAL "emit this comment as soon as is convenient and legal"09:29
pooliemaybe there is such a thing already09:29
lifelessjtv: I haven't checked the code yet, but I'm pretty sure its a if block guarding an 'expensive' query09:30
lifelessin a loop09:30
jtvlifeless: so a Foo, EXISTS(…) query in disguise.09:30
lifelessjtv: well, I think it just wants to be a single Foo query09:31
lifelessI'm saying the exists checking is totally unnecessary09:31
lifelesspoolie: I love your idea; please do wishlist it in lp-foundations09:31
lifelesspoolie: I was merely commenting on the pragmaticness of it today ;)09:31
jtvlifeless: it's not blocking anything _except_ an expensive query?  Or is it a case of "this entire piece of UI shouldn't be displayed"?09:32
mrevellthanks jtv09:32
mrevelljtv, I'll land a branch today updating the help content.09:32
jtvmrevell: yup, you can do that now!  Maybe we'll want things like "user hasn't done any translation _recently_" or (a bit harder) "user hasn't done any translation since I last updated this text."09:34
lifelessjtv: bug 607935, feel free to dig09:40
_mup_Bug #607935: timeout on bugtask:+index <timeout> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/607935>09:40
lifeless149 calls to bug.isSubscribed and bug.isSubscribedToDupes09:41
jtvlifeless: I'm a bit sick today, so any effort will be sporadic09:41
lifeless148 to the second09:41
lifelessfor team in self.user.teams_participated_in:09:41
lifeless ^ warning, this may loop a lot09:41
poolielifeless:  https://bugs.edge.launchpad.net/launchpad-foundations/+bug/626673 fwiw09:41
lifelessif bug.isSubscribed(team) or bug.isSubscribedToDupes(team)):09:41
_mup_Bug #626673: want sql statements interleaved in html comments <Launchpad Foundations:New> <https://launchpad.net/bugs/626673>09:41
lifelesspoolie: thanks!09:41
pooliemaybe i should have a selfimposed wip limit for wishes09:42
lifelessjtv: so, this is yet another example of what I've been going on about : looks cheap, but actually, its a good fifth of the page, and a simple set intersection query, done once, can return the info needed09:42
pooliebtv so do you think that would have ever helped you, had it existed at the time?09:42
lifelessmaybe09:43
lifelessit would provide some hint09:43
lifelessbut what I've seen a lot of you'd just get many queries smooshed at the top of the page09:43
lifeless(and in fact, I want us to head to that: no queries mid-page)09:43
poolieright09:43
pooliei think being told that is still useful09:44
lifelessalso, hah! this template does this:09:45
lifelesss/template/browser:09:45
lifelessself.many_bugtasks = len(self.bugtasks) >= 1009:45
jtvthat's an old favorite…09:45
lifelessoh nvm that one is actually cheap, I guess I'm tired ;)09:45
lifelessI was going to say that count() is about as expensive as the full query09:45
lifelessand thus to be avoided like the plague09:46
wgrantlifeless: Isn't it often even more expensive?09:47
wgrantGiven that the full query is batched.09:47
wgrantNormally.09:47
lifelesswgrant: well you're comparing across layers there09:48
lifelessbatching is something the API or UI exposes09:48
lifelesscount() is ~= to getting the last row09:48
lifelessbut, its worse09:48
lifelessbatching in the API makes iteration O(N^2)09:49
lifelesswhich reminds me, file a bug to turn it off09:49
lifelesshttps://bugs.edge.launchpad.net/launchpad-foundations/+bug/626680 if you're interested09:52
_mup_Bug #626680: iteration in LP API's is O(N^2) due to batching <Launchpad Foundations:New> <https://launchpad.net/bugs/626680>09:52
jtvlifeless: is the problem that that subscription line you quoted earlier, the "if bug.isSubscribed" etc. one, should be cached across many bugtasks on a page?09:53
lifelessjtv: no, its looking for the intersection of 'which of this bug& its dupes am or or my teams subscribed to'09:56
lifelessso someone in a lot of teams (e.g. me) will cause 2 queries per team: one for subscribed to the bug, one for subcribed to a dupe.09:57
lifelessits building a list of unsubcribe links09:57
lifelessso that when you get mail you can click on the link09:57
lifelessand in the bug you see 'unsubcribe <team-that-is-subscribed> from this bug'09:57
lifelesswe can halve the query count with a trivial helper to query bug.isSubcribed and bug.isSubscribedToDupes in one query09:58
lifelessand we can make it constant by doing a query that will return the team objects that are in the users participations && subscribed to the bug or a dupe.09:59
lifelessthat will make the OOPS report a lot easier to read, for the next iteration (and according to the report, take 1.5 seconds off of the page, for me, *for all bug pages I look at*.09:59
jtvlifeless: for cases like this one, perhaps we should get used to exposing methods that compose collections, Storm queries etc.  One of our slowest pages could benefit handsomely from a helper that makes it easy to prejoin the icons for a listof persons or products.10:03
jtvIf we keep optimizing these things at the call sites, we give up some schema flexibility.10:04
lifelessthis is why I want a separate layer10:05
lifelessI like generic collection10:06
lifelessI don't think its enough, nor a fit for this sort of thing (yet)10:06
lifelesswell, it might fit this10:07
lifelessbut its a little weird starting with team (the result you want), filtering by your-participations, then by your-bug-subscriptions. Perhaps its not.10:07
* jtv is too used to that to question it10:08
lifelesswhats missing from collections is getting multiple types back10:09
jtvHow is that missing?10:09
jtvThat's built in.10:09
jtvcollection.find(Foo, Count(Foo.bar), PrejoinedExtra)10:10
* jtv steps back outside w/book10:10
lifelessjtv: I don't know what PrejoinedExtra is10:11
lifelessjtv: but perhaps you could look at Person.all_members_prepopulated and show me how to Collectionise it10:11
lifeless*keeping the elegance* of collections.10:12
jtvlifeless: PrejoinedExtra is a class that I just made up.10:14
jtvlifeless: for each of those left joins, you'd refine the collection using joinOuter.    I'd also keep a list of "things I want from this query" in the collection, and extend that for each item you added.10:20
jtvI'm not sure there's a truly elegant solution for the variable result columns because it's not a very elegant way to pose the problem.10:20
lifelessjtv: well it also needs to populate the cached attributes10:21
lifelessjtv: I'd like to have a reusable solution to this10:22
jtvOne way to do it might be to pass in a series of classes you want, and use that to base the query construction on, and return that same list.  Bit harsh where you need aliases, of course.  For the counts and exists etc. I think you'd want to stuff the aggregates into the result objects somewhere rather than return them.10:23
jtvSorry, I'm being imprecise.10:23
lifelessthats ok10:23
lifelessthis doesn't have to be solved today10:23
jtvYou return a result set whose columns match the classes that you passed on10:23
jtvpassed in10:23
jtvThe aggregates and other things that aren't easily identified as classes probably shouldn't stand on their own; you could have a delegating "pseudo-model" object, e.g. BugWithTasks, that holds cached data so it's similar to a Bug with lots of @cachedpropertys except the caching goes away at the end of the request.10:25
lifelesswell10:27
lifelessThis comes back to separated storage and logic10:27
lifelesswhich I want, I don't think that pushing caching out to a separate place is a sane approach10:27
lifelessI think that pushing storage out to a separate place may work10:27
jtvlifeless: sorry, I just noticed I'm no shape to go in depth right now10:29
lifelessno rush10:29
lifelessgo take some aspirin and lie down10:30
jtvy10:30
deryckMorning, all.11:56
=== al-maisan is now known as almaisan-away
=== almaisan-away is now known as al-maisan
=== matsubara is now known as matsubara-lunch
Ursinhasinzui, hi, should I add qa-bad to bug 624981?16:46
sinzuiNo, the code never arrives16:46
sinzuiarrived16:46
sinzuiUrsinha, and the code we are using is not on edge or staging, though I landed it first16:47
sinzuiUrsinha, so the bug was tagged for testing, but there is no sever to test on16:47
=== beuno is now known as beuno-lunch
=== benji is now known as benji-lunch
Ursinhasinzui, I see17:09
=== Ursinha is now known as Ursinha-lunch
=== matsubara-lunch is now known as matsubara
=== deryck is now known as deryck[lunch]
=== beuno-lunch is now known as beuno
=== abentley_ is now known as abentley
=== benji-lunch is now known as benji
=== deryck[lunch] is now known as deryck
=== al-maisan is now known as almaisan-away
lifelessmoin moin18:57
* lifeless wonders when the next edge update is19:00
deryckhi lifeless19:01
lifelesshi deryck19:01
lifelessderyck: you might like rev 11472 of devel/stable19:01
=== lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Performance Tuesday | Week 3 of 10.09 | PQM is OPEN | firefighting: - | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews
derycklifeless, I saw the commit message, but haven't looked at what's in the rev yet.19:02
lifelessderyck: its a -little- ugly due to some lazr.restful limits19:03
lifelessderyck: but function, very functional19:03
* deryck takes a break from test fixing to look19:04
* benji wonders if anyone has looked at just how much spam gets dumped into http://pastebin.ubuntu.com19:07
derycklifeless, yeah, that is nice.  I really like the DecoratedResultSet pattern that we've got now.19:08
lifelesscool19:14
=== Ursinha-lunch is now known as Ursinha
lifelessderyck: leonardr: either of you know what pageid api/1.0/bugs?assignee=xxx would have ?19:31
leonardrlifeless, probably something like IBugSet:assignee19:32
leonardror IMaloneApplication:assignee19:32
lifelessIBugSet has no exported() decorators19:33
lifelessleonardr: does that mean its not exported?19:33
deryckThat translates to searchTasks call.  So wouldn't it be a product.searchTasks or distro19:34
SpamapSlifeless: seems like it would be a really really fast query.. unless there's no index on the assignee column for bugs. ;)19:34
lifelessSpamapS: lets get data19:34
lifelessSpamapS: just figuring out where the code is19:34
leonardrlifeless: yeah, it's not exported, IMaloneApplication is /bugs/19:34
lifelessso MaloneApplication:+bugs perhaps19:35
lifelesswhich on thursday (last day the ppr ran without sodium trashing it)19:36
lifeless256 hits19:36
lifeless1744 total seconds,19:36
lifeless99% in 16.97 seconds, 6.82 mean, 3.38 stddev19:36
SpamapSlifeless: I think one problem is, its actually the global bugs list.. assignee=clint-fewbar isn't the right way, is it?19:37
lifelessmean 33 sql statements19:37
lifelesslooks like its mainly sql time19:37
lifelessSpamapS: no reason for that not to be fast19:37
SpamapSlifeless: agreed, :)19:37
lifelessSpamapS: Ubuntu is ~50% of bugs anyway, so you wouldn't save jack by filtering by it19:37
lifelesslets see if it has oopsed19:38
lifelessif it has we'll have some nice data, if it hasn't we can get a profile from staging.19:38
lifelessit is, for reference, in my hitlist already: https://devpad.canonical.com/~stub/ppr/lpnet/daily_2010-08-25_2010-08-26/timeout-candidates.html19:38
lifelessabout half way down19:39
lifelesselmo: around ?19:40
lifelessSpamapS: so now I'm grepping our oopses reports (slow page diagnostics)19:43
lifelesswhich will take a while19:43
lifelessleonardr: so, I'd really really love to get oops ids on apis when ++oops++ is used19:44
lifelessleonardr: do you have any ideas about a tasteful way to do that19:44
leonardrlifeless: can you quickly run down how it works on the website?19:44
lifelessleonardr: sure19:44
lifeless++oops++ is triggered by traversal, a match anything adapter19:45
elmolifeless: vageuly19:45
lifelessit sets a global variable that says to the oops code19:45
lifelesselmo: is there an ETA on sodium - its dying so much that many cron based things we use regularly are not completing19:45
elmolifeless: it's been body swapped already19:46
lifelessleonardr: so when ++oops++ is traversed it sets a glad in errorreport.py19:46
lifelesselmo: !19:46
elmolifeless: that didn't take - the latest theory is that the disks are fucked in interesting ways that is causing the kernel to crash19:46
lifelesselmo: -ah-19:46
elmolifeless: we're going to force an fsck of the disks and if that fails, just give up and replace the box wholesale19:46
lifelesselmo: sorry for nagging then.19:46
lifelessSpamapS: ok, I've got an OOPS from a few days back19:47
lifelessleonardr: s/glad/flag/19:47
lifelessleonardr: at the end of the request this causes two things: oops report written to disk, and, the oops number put in the comment region in the main template.19:48
leonardrlifeless, this only happens if there's an oops, right?19:48
lifelessleonardr: these may actually be the 'same' thing - that is the comment region triggers evaluation of the 'should I write an OOPS code' (when no exception has occured)19:48
leonardrno oops, ++oops++ does nothing?19:49
lifelessleonardr: no, ++oops++ makes a 'user requested oops' - its generated regardless, so you get get operational info on not-quite-crashing-but-bad pages19:49
lifelessthere is however no big traceback unless an actual crash did occur (in which case ++oops++ has no effect on what happens)19:49
elmolifeless: no problem19:49
lifelessleonardr: so, for API's something equivalent would be to somehow (not necessarily ++oops++) tell the api code that at the end of generating all its stuff, it should generate an oops19:50
lifelessthe oops goes to disk19:50
lifelessand the oops id gets put somewhere. like an http header, or in the outermost json dict, or something19:50
leonardrok19:50
leonardrhow does it generate an oops? just by raising an exception?19:51
lifelessit calls into the errorlog code19:51
lifelesslook at maybe_record_user_requested_oops19:51
lifelessessentially we'd want API's to call that function19:52
lifelessand store the oops id it returns *somewhere*19:52
lifelessSpamapS: so https://lp-oops.canonical.com/oops.py/?oopsid=1701M1562 is *a search* that timed out on that api19:53
lifelessactually no, its a regular webapp request19:53
lifelessdigging more19:53
SpamapSlifeless: its very consistent at 11 seconds..19:58
SpamapSlifeless: so I doubt it will time out.19:58
SpamapSuntil something else slows everything down that is19:58
leonardrlifeless: so, oops ids are already sent out in the X-Oops-ID header (don't remember the exact name)20:01
leonardrX-Lazr-Oopsid20:01
leonardrthat's handled by the WebServiceExceptionView20:02
leonardrit would be pretty easy to make lazr.restful check for an incoming X-Lazr-User-Triggered-Oops header or something, and raise an exception at the end20:03
lifelessleonardr: so the point of user triggered oops is that they don't have an exception20:03
lifelessyou still get the response20:04
lifelessand you get the oops20:04
lifelessI appreciate this is probably more work :P20:04
lifelessleonardr: here is my use case20:04
leonardrsince maybe_record_user_requested_oops is launchpad code, we either need to move it out of launchpad or put a hook in the configuration object20:04
lifelesssomeone says 'api X is slow', I say 'in your browser, do XXX and look at YYY and tell me the OOPSID'20:05
lifelesstell you what, I'll file a bug on lp-foundations about it20:05
lifelessits possibly easy, but isn't -trivial-20:06
lifelessA hook approach sounds sensible to me (though there is already IRequestEnd which is what errorlog is hooked into; is that perhaps not enough?20:06
leonardri don't know about IRequestEnd. at the very least we need a hook so that lazr.restful can set the magic variable that ++oops++ sets20:07
lifelessso thats set at the start20:07
lifelessits currently done via a traversal adapter20:07
leonardrif you want it to be doable in the browser then we need something like ++oops++ to go in the url20:07
lifelessbrowser would be ideal20:07
lifelessits just so much easier for people to adhoc stuff with20:08
lifelessah, I have already20:09
lifelessbug 60695220:09
_mup_Bug #606952: ++oops++ should work on api urls <performance> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/606952>20:09
lifelessor martin has20:09
lifelessSpamapS: so the fallback position is staging.20:10
lifelessSpamapS: staging is a) slower b) less resources c) lower timeout.20:10
lifelessI guarantee it will break for us20:10
lifelessonce it comes up after the code update ><20:11
lifelessmatsubara: ping20:11
matsubarahi lifeless20:11
lifelessmatsubara: thanks for working on https://bugs.edge.launchpad.net/launchpad-foundations/+bug/60618420:11
_mup_Bug #606184: API Pageid for collections is 'scopedcollection:collectionresource'  which does not mention the origin page id <qa-needstesting> <Launchpad Foundations:Fix Committed by matsubara> <https://launchpad.net/bugs/606184>20:11
lifelessmatsubara: did you see my follow up there ? :)20:11
matsubaralifeless, np, I saw your comment there and will follow up later20:11
lifelessok kk20:12
matsubaralifeless, but I guess, the fix will have what you want because it builds on benji's work to include the named operation for webservice oopses20:12
matsubara(unless I misunderstood what you wanted)20:12
leonardrlifeless: bottom line, it's a small to medium sized project. unless you just want to be able to tell people to use their browsers, there's also a client-side component20:12
lifelessleonardr: just browser would be fine in the first iteration20:13
lifelessleonardr: we're very read-heavy, and GETs are fine in the browser20:13
lifelessmatsubara: well I wanted instead of ScopedCollection:CollectionResource, IMaloneApplication:searchTasks20:14
lifelessmatsubara: as I understand it you've added in the *type* of the collection, which is nice to have, but doesn't actually help pinning down the code to look at.20:14
lifelessmatsubara: my goal is that the pageid should be a reliable key for grouping on.20:15
leonardrlifeless: in that case the main challenge is figuring out which code goes in lazr.restful and which in launchpad. i would like to do something simple like make the /++oops++/ traversal apply to the api vhost and set this magic variable20:15
lifelessleonardr: that would be lovely20:16
leonardrso that lazr.restful thinks it's working normally, but the code it's running inside does something different with the data20:16
matsubaralifeless, take a look here: https://devpad.canonical.com/~lpqateam/edge-oops.html and you'll see how the pageid looks like now with my fix (search for scopedcollection)20:16
matsubaralifeless, my understanding was that you wanted engineers looking at the oops summary to be able to pinpoint if the CollectionResource triggering the error was under their domain and then act to fix it20:16
matsubaralifeless, my fix also builds on top of benji's work which includes the named operation (if any) to the page id20:17
lifelessmatsubara: Thats a necessary condition, but not sufficient :)20:17
lifelessmatsubara: I think what you've done is a great improvement.20:17
lifelessBut I'm greedy, I want more.20:17
matsubaraso, it'd look like: ScopedCollection:CollectionResource:#bug-attachment-resource:searchTasks (of course this is a made up pageid)20:17
lifelessmatsubara: so this one -  (ScopedCollection:CollectionResource:#bug_attachment-page-resource)20:18
lifelessis actually IBug:attachments20:18
lifelessif an engineer looks at the IBug interface at the exported 'attachments' collection, they are looking at the right place.20:18
lifelessthats what I'd like to achieve.20:19
lifelessthe scopedcollection:collectionresource stuff is, AFAICT, not relevant20:19
lifelesslosa ping: is the staging update looking normal - they normally finish by 14 past20:21
elmolifeless: staging got upgraded to lucid today20:21
lifelesselmo: \o/20:21
elmo(just as a data point)20:21
elmoit may be fucking with the updates20:21
lifelesselmo: thank you (both for that its upgraded and letting me know)20:21
lifelesselmo: it wouldn't surprise me20:21
matsubaralifeless, I see. thanks for the feedback. I guess I'll have to make another patch to accomplish that. Would you file another bug please?20:21
lifelesselmo: the only signal I have for this is https://staging.launchpad.net/successful-updates.txt AFAIK20:22
lifelessmatsubara: certainly20:22
matsubarathanks lifeless20:23
mbarnettlifeless: let me take a look20:24
* mbarnett checks to see if it finisehd in the mean time as it doesn't seem to be running20:24
mbarnettnope20:25
lifelessmbarnett: ok, so its still going, thats fine.20:26
mbarnettlifeless: it isn't20:26
lifelessmbarnett: is it doing a db restore or just code update?20:26
mbarnettlifeless: last i see is an error in the logs20:26
lifelessmbarnett: oh, its fallen over?20:26
lifelesscan I see it?20:26
mbarnettlifeless: here is the tail of the log...20:27
mbarnetthttps://pastebin.canonical.com/36480/20:27
mbarnettlet me know if you would like to see more20:27
lifelessmatsubara: see bug 62702720:28
_mup_Bug #627027: further improvements to API collection page ids <Launchpad Foundations:New> <https://launchpad.net/bugs/627027>20:28
lifelessah slony, I love to hate you20:28
lifelessmbarnett: AIUI all clients have to be kicked off before the upgrade can happen, but there are staging sso clients connected so it would fail.20:29
lifelessmbarnett: those sso appservers may have failed to shutdown as per spm's bug filed yesterday20:30
matsubarathanks lifeless20:30
lifelessmbarnett: so I'd be inclined to check if thats the case, if so get a thread dump and attach to the bug, then nuke them, then kick the staging restore to go again20:30
lifelesss/restore/update/20:31
lifelessmbarnett: of course, you probably are already doing all that and I'm just annoying :)20:31
* lifeless goes for breakfast so he won't be annoying 20:31
mbarnettheh, i have not initiated any of it, so i will get on it momentarily20:33
lifelessnow, where was I20:40
lifelessSpamapS: so, we'll look into your api performance question when staging comes up20:41
lifelessSpamapS: bugs that would have made it easier to look into have been tickled.20:41
lifelessbut for now, I'm going to make BugTask:+index faster20:42
lifelessit does *300* team subscription lookups on every page for me.20:42
mbarnettlifeless: heya, do you have that bug #?20:43
lifelesssec, I'll grab it20:43
mbarnetti am having parsing problems from my weekend mail!20:43
mbarnettthanks20:43
lifeless62657720:44
lifeless(I went to launchpad-project and searched for shutdown)20:44
=== matsubara is now known as matsubara-afk
mbarnettthx20:45
mbarnetthmm, i wonder if this is the same issue..  lp appservers are a bit differerent than the sso client20:46
lifelessoh?20:47
lifelessI thought the sso account was for the sso appservers, which are a fork of lp ?20:47
lifelessHappy to be told I'm wrong :)20:47
SpamapSlifeless: doh20:47
mbarnettsso clients are running their own package, no longer out of lp trunk20:47
SpamapSlifeless: glad I could help tickle some things. :)20:47
lifelessSpamapS: :P20:47
lifelessmbarnett: ah yes, so it would depend if they have got the same issue merged into them20:48
SpamapSNow I just need to figureo ut why jquery/chrome are not happy w/ the response coming back from the API calls..20:48
lifelessmbarnett: have they failed to shutdown though ?20:48
mbarnettyeah, the sso stuff runs right out of apache, so i think they don't20:48
lifelessmbarnett: really? wow, thats very different. And it still talks to the LP db ?20:49
mbarnettthey have their own sso database20:49
mbarnettthat part of is replicated back into lp20:49
lifelessmbarnett: are teams replicated to their DB ?20:49
mbarnettthat i am not sure20:49
lifeless(because sso.ubuntu.com still serves team data)20:49
mbarnettbut if you notice20:50
mbarnettthe sso_staging connections are to the sso_staging database20:50
lifelessyeah20:50
lifelessslonik takes a lock on all replication sets before doing any schema changes20:50
lifelessits a ... feature20:50
mbarnetti do see 1 read only thread on the lpmain db though20:51
lifelessthat would block it20:51
mbarnettlet me see if that is still there20:51
mbarnettif not, i'll fire back off the staging restore20:51
mbarnettmaybe someone connected at just a very bad moment20:51
mbarnettafter the kill switch but before the next step..20:52
lifelessmbarnett: its just a regular update, not a db restore today though, isn't it ? (with incremental db schema changes done automatically)20:52
lifelessSpamapS: can you please file a bug - IMaloneApplication:searchTasks is slow, include the url you used.20:53
mbarnettlifeless: the db update failed over the weekend.. stub made some recommendations for a code update then another full update20:53
lifelessah ok20:54
mbarnettso, i believe this to be the full update20:54
lifelessok, I shall be patient :)20:54
mbarnettok, i am kiling the read only idle connection20:55
mbarnettand fire it back off20:55
mbarnettdead20:56
mbarnetttrying again20:56
SpamapSlifeless: Definitely, I have added it to my todo list, I won't get to it for a couple of hours (have to run to an appointment).20:58
mbarnettand, we are off.20:58
mbarnettlifeless: this will of course take a while.20:58
lifelessmbarnett: of course; thank you!21:17
lifelessSpamapS: thats ok, we won't fix it for a couple of months.21:17
rexbronhey22:51
rexbroncould any one comment on why the run command is not supported in recipes on launchpad?22:52
lifelessbecause it can run arbitrary code22:52
lifelessrecipes are evaluated in many contexts, only some of which are secured22:52
rexbronI thought that because of this recipes were run in vms, same as builds22:53
lifelesssource package creation runs in vms22:54
mwhudsonbuilding the source package from the debianized source tree is done in vms22:54
rexbronhttp://blog.launchpad.net/cool-new-stuff/launchpads-build-farm-improvements22:54
lifelessany bugs or registry folk around22:55
lifelesssinzui: perhaps you are here?22:55
rexbronlifeless: basically, I need to nest and move the debian folder in the the upstream src from the debian branch22:56
rexbronmerging complains of no common ancestor22:56
rexbronand nesting was suggested as a work around22:56
lifelessplease file a bug with tasks on udd and bzr-builder22:57
lifelesswith enough details to reproduce22:57
lifelesscertainly we need to meet your needs, but the run command is not how we'd like to do that22:57
rexbronlifeless: basically, In an ideal world, you could merge just a directory, ignoring the rest23:00
lifelessspiv has been working on that23:00
lifelessI believe you can with the lastest builder; it might not be rolled out yet23:00
rexbronlifeless: rolled out to edge?23:02
lifelessrexbron: production23:02
lifelessthere is no 'edge' for anything other than the appservers.23:02
rexbronahh23:02
lifelessWe're going to get rid of 'edge' for appservers too, but start rolling out much more often (without downtime) in the near future23:02
lifelesslosa ping - staging update; just want to make sure it hasn't fallen over and died again.23:03
rexbronhmm... so what are my options at this point?23:04
rexbronlifeless: ^23:07
lifelessfile the bug as I requested23:07
lifelessudd is a high priority for the bzr team23:07
rexbronudd? I'm not familar with that term?23:07
lifelesslaunchpad.net/udd23:08
lifelessfile a bug there23:08
lifelessand also add a task to bzr-builder23:08
lifelessjkakar: around ?23:20
rexbronhttps://bugs.edge.launchpad.net/udd/+bug/62711923:21
_mup_Bug #627119: Can not merge branches that have no common ancestor <Ubuntu Distributed Development:New> <https://launchpad.net/bugs/627119>23:21
jelmerthumper: hi23:26
thumperjelmer: morning23:26
jelmerthumper: did you see my conversation with Peter? It looks like the chicken repo works now.23:27
thumperyep saw that23:27
thumperawesome23:27
thumperthanks for following up23:27
jelmerthumper: I was wondering if it would still be possible to do a CP (and perhaps also include the newer bzr-svn ?)23:27
thumperjelmer: we are in week 3 now23:28
thumperjelmer: and it isn't really critical23:28
lifelessthumper: what does that mean23:28
thumperlifeless: what are your thoughts on this?23:28
jelmerthumper: ok, fair enough23:28
thumperlifeless: what I'm saying is that we aren't that far from a real release23:28
thumperjelmer: does it need the bzr 2.2 updates too?23:28
lifelessthumper: I'm torn23:29
thumperlifeless: me too23:29
lifelesson the one hand I love getting things out of the way - low kanban limits23:29
thumperobviously we want working stuff out ASAP23:29
lifelesson the other hand, a release is soon23:30
lifelessthumper: flip a coin?23:30
thumperis the overhead of chasing the change onto production worth the gain of releasing it 8 or 9 days earlier?23:31
mwhudsonwait until release, spend the time that would have been spent chasing reducing the overhead?23:32
lifelessI like that suggestion23:32
lifelessif we have a figure in mind - say we reckon it will take 5 hours to do a CP for all of this.23:32
lifelessspend those 5 hours making the next one < 5 hours23:33
mwhudsongary_poster: did we find out why bootstrap.py was connecting to the internet?23:33
jelmerthumper: I don't know of any people other than Peter are seriously affected by these issues.23:35
thumperjelmer: given that he has been waiting months, another 8 days won't kill him :)23:35
thumperI know that isn't a great reason23:35
thumperbut...23:35
* thumper shrugs23:36
lifelesspersonally, I'd say this.23:39
* lifeless says it23:39
lifelessjelmer: if you want to do the work, I'll +1 a production change for it. But it goes by the book and with due caution.23:39
jelmerlifeless: Thanks, I'll pass. :-)23:41
lifelessthumper: one thing not covered in our analysis here: doing CP's reduces the size of the 'release'.23:42
thumperlifeless: one of things that makes it a little less clear for me is the dependancy on the 2.2 change23:42
thumperthere was a big change to get that in23:43
thumperwhich caused changes in a lot of tests23:43
jelmerlifeless: Thumper and I discussed a CP earlier, but admittedly that was much much earlier in the cycle (first few days of week 1 I believe).23:43
thumperif the tests for the new plugin uses these changes, then it won't apply cleanly23:43
thumperthat is my fear/concern23:43
lifeless2.2 was tricky23:45
lifelessyes23:45
lifelessI share the concern - thus the due caution comment23:45
* thumper afk for a few minutes for coffee and planning23:47
jelmerhi rexbron23:54

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!