[00:04] <lifeless> erm
[00:04] <lifeless>   File "/var/launchpad/tmp/eggs/setuptools-0.6c11-py2.6.egg/setuptools/package_index.py", line 475, in fetch_distribution
[00:04] <lifeless> AttributeError: 'NoneType' object has no attribute 'clone'
[00:04] <lifeless> in devel ?!
[00:04] <lifeless> Downloading file:///var/launchpad/test/download-cache/dist/setuptools-0.6c11-py2.6.egg
[00:04] <lifeless> (on ec2)
[00:22] <deryck> Hi, all.
[00:30] <lifeless> hiya
[00:31] <deryck> Anyone else have make blowing up over not having zc.buildout 1.5.1?
[00:32] <lifeless> yes
[00:32] <lifeless> just hit me on ec2
[00:32] <lifeless> as I was about to land the fix for Bug.attachments timing out all over the place
[00:33] <deryck> I can't figure out how to fix it.  Rolled back bootstap.py to earlier version in the branch I'm in just to get hacking again.
[00:33] <lifeless> deryck: while you are here
[00:33] <deryck> sure
[00:33] <lifeless> do you know why BugAttachment.message doesn't know its own index ?
[00:34] <lifeless> IIndexedMessage seems pretty expensive to generate
[00:34] <deryck> I don't.  Meant to look into that when I saw your earlier email, and never got to it with things happening Thurs/Fri.
[00:34] <lifeless> so I figure gary must have no landed 1.5.1 into the cache
[00:35] <lifeless> i wonder if its on lp or pypi
[00:35] <lifeless> deryck: I suspect messages can be shared between bugs
[00:35] <deryck> yeah, I wondered that too.  But didn't know if he was rolling his own.
[00:35] <lifeless> but there must be a BugMessage or something that does the actual join
[00:35] <lifeless> and thus we could have an index in that
[00:35] <deryck> yeah, there is.
[00:36] <lifeless> change bugattachment to go via BugMessage
[00:36] <lifeless> and get the index from that
[00:36] <deryck> right
[00:37]  * deryck is looking at code....
[00:37] <lifeless> should I file a wishlist bug for this? I have Bug.attachments down to 23 queries - constant -
[00:37] <lifeless> but, the way it does that is to use self.indexed_messages
[00:39] <lifeless> deryck: ok, there is  1.5.1 on pypi.org, and it seems to be working.
[00:39] <lifeless> deryck: I'm committing it to the download cache.
[00:39] <deryck> awesome!
[00:39] <mwhudson> something funny is going on
[00:40] <mwhudson> because versions.cfg still says 1.5.0 for buildout
[00:40] <deryck> lifeless, yeah, file a bug on that.  I'll see if I can get work on that for my performance Tuesday work.
[00:40]  * deryck plans to do performance Tuesday this Tuesday again
[00:40] <lifeless> deryck: I'd leave it for a little while, my stop gap should get us through to the 5sec 99% barrier
[00:40] <deryck> ah, ok
[00:40] <deryck> cool then
[00:40] <lifeless> \o/ about perf on tuesday though
[00:41] <lifeless> there are plenty of other timeouts in bugs code
[00:41] <lifeless> mwhudson: well, thats interesting.
[00:41] <lifeless> mwhudson: still, committed. we'll see what happens.
[00:41] <mwhudson> oh
[00:42] <mwhudson> hmm
[00:42] <mwhudson> bootstrap.py must be connecting to the internet to find out that 1.5.1 exists
[00:42] <mwhudson> this would explain why codebrowse restarts were taking 5 minutes
[00:42] <lifeless> deryck: https://bugs.edge.launchpad.net/malone/+bugs?field.tag=timeout
[00:42] <deryck> lifeless, yeah, we're not getting to put the attention on timeouts that I had hoped.  But after this cycle, I'll make sure we're getting the work in.
[00:43] <lifeless> deryck: Oh, I wasn't meaning to criticise; I was offering that as a palette
[00:43] <deryck> yes, I appreciate the tag list!  I didn't take it as criticism.  Was just saying we haven't put the attention to the bugs as I had hoped to.
[00:44] <mwhudson> i bet this affects edge rollouts too
[00:44] <lifeless> https://bugs.edge.launchpad.net/malone/+bug/121363 looks pretty shallow
[00:44] <_mup_> Bug #121363: Order 'most recently closed' on 'Bugtask.id DESC' instead of 'BugTask.id' <timeout> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/121363>
[00:44] <mwhudson> spm: hi, if you have time, can you tell if the last edge rollout took longer than you'd have expected?
[00:44] <spm> mwhudson: actually it's looking like it brokee rather nicely. I'm still chasing.
[00:45] <mwhudson> spm: nice
[00:51] <lifeless> it probably broke due to this issue
[00:51] <lifeless> which I have just addressed
[00:51] <lifeless> in a fairly shallow way
[00:53] <wgrant> However. it still shouldn't be looking anywhere else... What's changed lately? Just buildout 1.5.0?
[00:53] <lifeless> bootstrap.py
[00:53] <mwhudson> i guess this is one for that operatic team lead person
[01:13] <lifeless> spm: also - bzr: ERROR: Could not acquire lock "LockDir(file:///home/warthogs/archives/rocketfuel-built/launchpad/.bzr/branch/lock)":
[01:13] <lifeless> spm: is really annoying me :)
[01:14] <spm> ./ignore lifeless
[01:14] <spm> gah. no wait wrong ignore.
[01:28] <lifeless> spm: hahaonlyjoking?
[01:57] <spm> lifeless: I can't see your comments anymore. so no idea if you asked if I'm joking or not. maybe.
[01:58] <lifeless> spm: :P
[01:58] <MTecknology> lifeless: This is what I'm doing now - http://staging.profarius.com/
[01:59] <lifeless> MTecknology: so the artwork for launchpad is not open
[01:59] <MTecknology> It just popped into my head that we're going that and I realized I'd better check it out..
[02:00] <MTecknology> s/going/doing/
[02:00] <lifeless> I'm just looking for the reference
[02:00] <lifeless> ah yes, LICENSE in the tree:
[02:00] <lifeless> The image and icon files in Launchpad are copyright Canonical, and
[02:00] <lifeless> unlike the source code they are not licensed under the AGPLv3.
[02:00] <lifeless> Canonical grants you the right to use them for testing and development
[02:00] <lifeless> purposes only, but not to use them in production (commercially or
[02:00] <lifeless> non-commercially).
[02:00] <lifeless>              
[02:01] <lifeless> The Launchpad name and logo are trademarks of Canonical, and may not
[02:01] <lifeless> be used without the prior written permission of Canonical.
[02:01] <MTecknology> So I should definitely remove it..
[02:01] <spiv> I would think (but IANAL) that if you are using the LP logo specifically to refer to the launchpad.net instance then you're on the right side of trademark law.
[02:01] <wgrant> spiv: Copyright-wise that's less clear, though.
[02:01] <spiv> wgrant: indeed
[02:02] <lifeless> spiv: trademark + copyright are needed though
[02:02] <ajmitch> can't login.ubuntu.com be used instead of LP?
[02:02] <lifeless> MTecknology: is it general openid you want to refer to
[02:02] <MTecknology> wgrant: has there ever been anything clear about legal issues?
[02:02] <lifeless> MTecknology: or lp specifically ?
[02:02] <wgrant> MTecknology: Frequently.
[02:02] <MTecknology> it uses launchpad teams
[02:02] <wgrant> But not when it comes to LP.
[02:02] <wgrant> I mean, the tree isn't even distributable...
[02:03] <lifeless> MTecknology: so, I suggest you email.. damn folk are on leave. Thumper. Email him.
[02:03] <MTecknology> will do
[02:03] <lifeless> ask for the ok. He'll bounce to jml, jml is awl for 2 weeks.
[02:03] <lifeless> so it will queue there
[02:03] <lifeless> and then get bounced to legal. :P
[02:03] <lifeless> thumper: ^ :P
[02:03] <MTecknology> :p
[02:03] <MTecknology> I'll make a nice pretty email for him to read :)
[02:04] <MTecknology> I never enjoy legal stuff - I kinda wish the whole world was just under the BSD license :P
[02:05] <lifeless> that has some appeal
[02:05] <MTecknology> I don't think society as is could handle that though :(
[02:06] <MTecknology> I've kinda been fighting the same with with the light-drupal-theme - I want it made for wide distribution - but I need to avoid license and trademark issues
[02:09] <wgrant> MTecknology: I would use Ubuntu SSO for that.
[02:10] <wgrant> It's more correct, and the trademark and copyright issues are clearer.
[02:10] <MTecknology> hm?
[02:10] <lifeless> wgrant: doesn't have lp team memberships
[02:10] <MTecknology> wgrant: for the theme - I mean images like this - http://s.ubuntu.ru/header.png
[02:10] <wgrant> lifeless: Doesn't it?
[02:10] <lifeless> wgrant: AFAIK, no.
[02:10] <MTecknology> and specific color schemes
[02:10] <wgrant> lifeless: They're the same code and DB, but with a different theme.
[02:17] <wgrant> lifeless, MTecknology: login.ubuntu.com does send team memberships.
[02:17] <wgrant> I just checked.
[02:17] <lifeless> cool
[02:18] <lifeless> I dunno if its /meant/ to - its hardly separate from LP :P
[02:18] <wgrant> It's meant to.
[02:18] <wgrant> AFAIK nobody thought about how the split was meant to work...
[02:18] <wgrant> Because it's still hanging off the LP DB, and probably will always have to.
[02:19] <lifeless> wgrant: I wasn't part of that work, can't comment at all.
[02:19] <lifeless> but its really a bit of a bastard child atm
[02:20] <MTecknology> lol
[02:20] <wgrant> It tries to be separate from LP.
[02:20] <wgrant> But everything revolves around it having knowledge of LP team memberships.
[02:23] <MTecknology> Is there any particular reason it was broken out?
[02:23] <lifeless> sinzui: hi
[02:23] <lifeless> MTecknology: to make it usable by U1
[02:23] <sinzui> hi lifeless
[02:23] <lifeless> sinzui: I am planning on nagging spm about the maps CP
[02:23] <MTecknology> U1?
[02:23] <lifeless> sinzui: he's dealing with falling buildings atm
[02:23] <sinzui> I was just about to do that
[02:23] <lifeless> MTecknology: ubuntuone
[02:23] <MTecknology> oh
[02:24] <sinzui> lifeless, okay, that saves me pinging him
[02:24] <lifeless> MTecknology: its an (important) branding exercise
[02:24] <lifeless> but its not really split out ATM
[02:35] <thumper> MTecknology: isn't it more ubuntu single signon than a launchpad login?
[02:35] <MTecknology> thumper: This image has been there since before the ubuntu login
[02:36] <thumper> hmm
[02:36]  * thumper shrugs
[02:37] <MTecknology> lp:drupal-{openid,launchpad,teams}
[03:02] <lifeless> spm: please ping when I can get another profile
[03:02] <thumper> testing bug 600000
[03:02] <_mup_> Bug #600000: missing dependency on Bazaar <hitchhiker (Ubuntu):Fix Released> <hitchhiker (Debian):New> <https://launchpad.net/bugs/600000>
[03:05] <spm> lifeless: just kicking this imports thing for thumper; then you're next?
[03:05] <thumper> ta spm
[03:12] <lifeless> spm: thanks
[03:12] <spm> lifeless: oki; about to do the magic on staging. brb...
[03:20] <lifeless> wow https://bugs.launchpad.net/ubuntu/+bug/1/+index is a mess
[03:20] <_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:In Progress by sabdfl> <ubuntu-express (Ubuntu):Invali
[03:20] <lifeless> spm: so I waiting for you to say 'go'
[03:20] <spm> lifeless: so am I. KaBoom.
[03:21] <spm> staging appears to be having FireTruck issues
[03:21] <lifeless> its going
[03:21] <lifeless> oh
[03:21] <spm> An error occurred when trying to install zc.buildout. Look above this message for any errors that were output by easy_install.
[03:21] <mwhudson> spm: big, red doesn't stop at lights?
[03:21] <spm> make: *** [bin/buildout] Error 1
[03:21] <spm> ^^ not my idea of 'going' :-)
[03:21] <lifeless> spm: >< - update the source code cache
[03:21] <mwhudson> spm: this'll be the 'connecting to the internet' problem maybe?
[03:21] <lifeless> I put a new buildout in there this morning to work around it
[03:21] <spm> ugh. it didn't STOP!!!!! ARGH!
[03:22] <lifeless> spm: is it currently running with profilig on?
[03:22] <lifeless> spm: cause I don't care if the code is old
[03:22] <spm> somethings changed in the recent week or so that stops staging/edge from shutting down. *some* of the time.
[03:22] <spm> lifeless: doubt it; it's been running since Aug 27.
[03:22] <lifeless> grah
[03:22] <lifeless> ok
[03:22] <spm> ie. we'd have faceplanted on disk space by now.
[03:22]  * lifeless emotes kill -9 @ spm
[03:23] <spm> I try the regular kill first. the think piece of silk wrapped around the 25 kg sledgehammer. if that doesn't work I take the slik off, and use the -9.
[03:23] <spm> s/think/thin/
[03:24] <spm> right. silk comes off.
[03:24] <spm> we used to have an incredibly aggressive. Try X, fail, try kill, fail, try kill -9 shutdown sequence; that seems to have been removed :-((((
[03:25] <spm> mwhudson: I don't think so: Link to http://pypi.python.org/simple/zc.buildout/ ***BLOCKED*** by --allow-hosts
[03:26] <lifeless> spm: thats the thing mwhudson is referring to
[03:26] <mwhudson> well no, that sounds better
[03:26] <lifeless> spm: you need to update the dist cache
[03:26] <spm> the internet problem was a timeout; we had last week
[03:26] <mwhudson> that's being blocked at the application level, not by the firewall
[03:26] <spm> https://pastebin.canonical.com/36438/
[03:27] <lifeless> spm: 11:05 < lifeless>   File "/var/launchpad/tmp/eggs/setuptools-0.6c11-py2.6.egg/setuptools/package_index.py", line 475, in fetch_distribution
[03:27] <lifeless> 11:05 < lifeless> AttributeError: 'NoneType' object has no attribute 'clone'
[03:27] <lifeless> 11:05 < lifeless> Downloading file:///var/launchpad/test/download-cache/dist/setuptools-0.6c11-py2.6.egg
[03:27] <lifeless> 11:05 < lifeless> in devel ?!
[03:27] <lifeless> 11:05 < lifeless> (on ec2)
[03:27] <spm> so I assume tehre's a patch floating around somewhere to ensure we have the lastest hotness?
[03:27] <lifeless> spm: is what I was asking this morning. Its the same - AttributeError: 'NoneType' object has no attribute 'clone' - is the magic bit to note, where it has blown well up.
[03:27] <lifeless> spm: yes, *update the dist cache*.
[03:28] <spm> that sounds... broken somewhere. why is our automatic updates not getting this automatically?
[03:28] <lifeless> something like cd lp-sourcedeps/download-cache; bzr update
[03:28] <mwhudson> i would like to understand why it's looking for 1.5.1 though
[03:28] <lifeless> spm: appears to be a skew between bootstrap.py and versions.cfg
[03:28] <lifeless> or something like that
[03:29] <spm> gah. sorry - germanium is trying to eat several hundred Gb of disk asap. bbs... maybe.
[03:30] <lifeless> whats germanium do
[03:30] <spm> ppa
[03:30] <mwhudson> it's ppa.launchpad.net i think
[03:31] <spm> looks like the leaky /tmp crap buglet we saw a week or so ago. G just has more / space. so took longer to manifest.
[03:31] <lifeless> ah so a simple rm *
[03:32] <spm> well.... o; but that's the idea. a little more targetted; but.
[03:32] <spm> well *no*. typo win.
[03:32] <lifeless> did you just rm -rf / ? Please say yes....
[03:32] <spm> I could try; but it won't work well :-)
[03:34] <spm> doing a verify if /tmp is in fact te problem. it *looks* like it; but I want a little more verification before I start rm'ing away; then I'll prolly start with something like: find /tmp -maxdepth 1 -name 'tmp*' -type d -mtime +7 -delete
[03:34] <spm> actually no. dir's; so -print0 | xargs -0 rm -rf
[03:35] <lifeless> -type f
[03:35] <lifeless> then -type d :P
[03:35] <spm> ... | xargs -rn rm -r <== better :-)
[03:36] <spm> -0r, argh
[03:36] <lifeless> -exec { rm \$1 }; ?
[03:36] <lifeless> actually, probably wants a -cut too
[03:37] <wgrant> spm: Not my script again? :P
[03:37] <wgrant> (killing germanium)
[03:37] <spm> lifeless: for this many folders, -exec would be horrible. fork hell.
[03:38] <lifeless> spm: clone ftw!
[03:38] <spm> wgrant: "yes", even if not your fault; it now is.
[03:38] <lifeless> spm: (actually its exec hell, the fork is cheap)
[03:38] <spm> whatever :-)
[03:39] <spm> ok. 15 mins of trying to get a du summary; I call /tmp/tmp* is the problem and move into rm mode
[03:40] <spm> whee. some srsly old poppy uplods too.
[03:40] <lifeless> mwhudson: its odd, bootstrap.py was last change in 11419
[03:40] <lifeless> on tuesday
[03:41] <lifeless> oh oop, thats my branch
[03:41] <lifeless> no, devel agrees
[03:42] <lifeless> ah, 11452 changes versions.cfg
[03:42] <lifeless> but that doesn't change... ah I wonder if lazr.restful 0.11.3 wants the newer zc/buildout ?
[03:43] <mwhudson> oh
[03:43] <mwhudson> maybe
[03:43] <lifeless> or launchpadlib
[03:43] <mwhudson> that should lead to a conflict though i think
[03:46] <spm> lifeless: (vaguely related aside) - did you ever submit a branch to the prod/staging configs that had the profiler in, but disabled?
[03:48] <lifeless> spm: pretty sure
[03:48] <wgrant> lifeless: It isn't just because the buildout 1.5.0 has new logic to update, and 1.5.1 was only just released?
[03:48] <wgrant> Er, English fail there, but you get the picture.
[03:48] <mwhudson> wgrant: no, because this happens on machines that can't connect to the internet
[03:49] <wgrant> mwhudson: And it hasn't been happening since the update?
[03:49] <mwhudson> to buildout?
[03:49] <mwhudson> no, we don't think so
[03:49] <wgrant> Yeah.
[03:49] <wgrant> :(
[03:50] <lifeless> spm: ah, its in my branch lp:~lifeless/lp-production-configs/timeouts
[03:50] <spm> lifeless: hrm. doesn't look like it. not an active proposal; and we've only had one other change in the past month. Can I pester/nag/trouble/bother you to get that sooner than later? ;-)
[03:50] <spm> snap, ftw.
[03:50] <spm> merge propose away!
[03:51] <lifeless> spm: the other change was this branch too, I thought
[03:52] <lifeless> https://code.edge.launchpad.net/~lifeless/lp-production-configs/timeouts/+merge/34042
[03:52] <spm> lifeless: no; the other recent change only has librarian oops changes.
[03:52] <spm> ta
[03:53] <spm> heya poolie
[03:53] <poolie> hi there spm!
[03:53] <lifeless> spm: so, does this mean we're ready to profile ?:)
[03:54] <spm> lifeless: ha you wish. staging is still very much firetrucked. still getting the ambulance round.
[03:54] <lifeless> shall we call the cops, go for the trifecta
[03:55] <spm> well, my awesomely wonderful (and delightfully sarcastic) wife is making me a ham sandwich... so........
[03:59] <mwhudson> lifeless: doesn't look like anything depends on buildbot 1.5.1 so it's a mystery (tm)
[04:01] <spm> lifeless: approved; pls submit whenever you're ready.
[04:02] <lifeless> spm: I've no idea how to do that anymore; it pages out so quickly (and its nuts to use pqm for this)
[04:02] <lifeless> spm: perhaps you could submit it ?
[04:02] <spm> :-)
[04:03] <spm> sure
[04:03] <lifeless> thanks!
[04:03] <spm> fwiw, we do via pqm; more as a gatekeeper than anything
[04:04] <lifeless> I love pqm/tarmac systems (I should shouldn't I :P) but they come with a cost, for benefits. if the benefits aren't worth the cost, it's not useful to use it :)
[04:04] <lifeless> spm: so, you're having lunch? Should I pop my head back in in 20 minutes or something for the staging profile ?
[04:05] <spm> lifeless: not really... Ill be eating lunch shortly; not breaking; if you ken the difference.
[04:05] <lifeless> och aye
[04:05] <spm> too many eggs on the boil to afk yet
[04:05] <lifeless> so staging - have you tried updating the download cache?
[04:08] <spm> not yet; eed to unfudge edge 1st
[04:08] <poolie> hi lifeless (realize lp's eggs are on fire) did anything change with feature flags?
[04:09] <poolie> i may go back to it say wednesday
[04:09] <poolie> haven't read much mail ypet
[04:09] <lifeless> poolie: sinzui has a patch that uses them to control google maps
[04:09] <lifeless> which are currently stuffed due to a google side problem
[04:09] <poolie> sweet
[04:09]  * poolie spoke too soon
[04:09] <poolie> but i'm glad to see some takeup
[04:09] <lifeless> it should get CP'd todayish :P
[04:10] <lifeless> spm: edge almost certainly has the same problem. have you tried updating the download cache :P)
[04:10] <spm> lifeless: that would emphatically be Doing It Wrong™
[04:10] <spm> manual intervention on every server? yuk.
[04:11] <lifeless> spm: well, doesn't your deploy script do that ?
[04:11] <spm> hrm. from the same state; the question is why is that state borked.
[04:11] <lifeless> spm: kick off a new deploy, done :P
[04:12] <lifeless> spm: so we don't know the root cause, but the package I put in the download cache about 4 hours ago seems to work.
[04:13] <spm> ahh. I see. edge borked is the failed stop on edge4; it never updated to latest shiny.
[04:13] <lifeless> right
[04:14] <lifeless> I strongly suspect, if the pastebin you gave is what the error was, that kicking off a new one, which should as a matter of course grab the latest download cache, should work.
[04:16] <spm> edge hasn't (yet) exhibited) that error...
[04:16] <spm> has special errors all of it's very own
[04:17] <lifeless> show n tell ?
[04:18] <spm> per above; failed to stop and never got updated; so when I stabbed and started it; it came up as the wrong revno.
[04:19] <lifeless> ah
[04:19] <spm> we have a check for that you see :-)
[04:25] <lifeless> spm: so what do you think is wrong?
[04:28] <spm> lifeless: with edge in this case? it's a reversion on an old bug faict. have just logged bug#626577
[04:28] <_mup_> Bug #626577: app servers not shutting down (again) <canonical-losa-lp> <Launchpad Foundations:New> <https://launchpad.net/bugs/626577>
[04:29] <spm> actually - that's a moderately deadly one for continuous rollouts btw. if we need manual intervention to just stop/start servers; that project is pretty much dead before started.
[04:30] <lifeless> we will fix
[04:30] <lifeless> the more we do something the more we'll improve it
[04:30] <spm> Oh i know that; was just saying :-)
[04:31] <spm> still the root cause of that bug has never been fixed. what actually causes the daemons to lock like that and not shutdown correctly.
[04:31] <lifeless> spm: did you get a per-thread stack trace?
[04:32] <spm> way back when; probably. we spent a fair bit of effort ages ago on it. given we had a (hack) fix; it was deemed not worth the time and effort to keep chasing further.
[04:33] <spm> argh. so staging full udpates died on a replication fail.
[04:37] <lifeless> was it the new table again ?
[04:37] <lifeless> spm: anyhow
[04:38] <lifeless> spm: I don't care about updates, I just want running + profiling :)
[04:38] <lifeless> spm: can we do that, and you fiddle with updates after ?
[04:38] <spm> well, tbh, I'd really rather not. otherwise I'm just duplicating the same work and somewhat wasting time messing around with something that needs fixing at the root anyway
[04:39] <lifeless> spm: ok
[04:39] <lifeless> spm: I'll go do some other stuff and check back with you in 20-30
[04:40] <spm> I'd hazard at least part of an educated guess that the pqm spam is all realted to stagings borked ness
[04:40] <lifeless> spm: the db-devel update is due to sodium crashing mid-update
[04:40] <lifeless> its left stale bzr locks
[04:40] <lifeless> at a WAG
[04:41] <spm> yeah - but those files get auto synced around. so a partial update gets synced around....
[04:41] <lifeless> ouchies
[04:41] <spm> right; and we have a dir that's missing stuff. so I call 1+1 = maybe 2 :-)
[04:41] <StevenK> 1 + 1 = 1.8 ish?
[04:41] <persia> closer to e
[04:41] <spm> 2.1, you could average
[05:03] <lifeless> spm: so how goes it ?
[05:05] <spm> well we have successful rf builds again.
[05:09] <spm> right. next regular staging code update is in ~ 5 mins. see how that goes as an automated thing.
[05:11] <thumper> :((
[05:12]  * thumper wishes jelmer lived in NZ or AU
[05:12] <persia> What do you have against the northern hemisphere?  We've a few active timezones on this side of the globe.
[05:13] <persia> (some east of some AU timezones)
[05:13] <lifeless> persia: thats only because .au is silly-wide
[05:15]  * persia is reminded of Negativland's "Time Zones"
[05:16] <mwhudson> the name of that ci tool that's written in java *still* trips me up
[05:17] <persia> In terms of increased highlight count?
[05:18] <mwhudson> no
[05:18] <mwhudson> just reading emails
[05:19] <mwhudson> "meet hudson" <blink>
[05:19] <persia> Ah, so yes, but with a wider definition of "highlight" to be more a cognitive filter than any technical notifaction system :)
[05:19] <mwhudson> right
[05:22] <thumper> persia: it is more that I want jelmer around RIGHT NOW
[05:23] <persia> I figured, but just felt you were unfairly discriminating against landmasses north of the equator.
[05:25]  * ajmitch thinks it's a fairly reasonable wish
[05:27] <lifeless> thumper: what do you want from jelmer ?
[05:28] <spm> lifeless: mwhudson: https://pastebin.canonical.com/36440/ <== that's getting zotted at source, so to speak. if the staging restore/build can't get those files; we have a bigger problem up front.
[05:29] <lifeless> spm: its in the download cache now
[05:29] <lifeless> spm: I don't know how staging updates work
[05:30] <lifeless> but if they are not updating the download cache, its a fail.
[05:30] <lifeless> if they , that file is available.
[05:30] <thumper> lifeless: I've emailed him about a git import failure
[05:30] <lifeless> spm: can you please confirm that zc.buildout-1.5.1.tar.gz exists on disk on staging
[05:36] <lifeless> oh joy
[05:37] <lifeless> ec2 hates my branch, but it works locallyt.
[05:37] <lifeless> what could it be? ... storm
[05:38] <spm> lifeless: so. cm.py (rf built) runs on sodium. 'builds' the tree; that gets pulled on sourcherry, which every half hour does a staging 'restore'. pushes that where it needs to go.
[05:39] <lifeless> spm: ok, so lets check on sodium
[05:39] <spm> dbupdates are done once 1 week; superset of the normal code only updates
[05:39] <lifeless> does it have a download-cache there
[05:44] <lifeless> grah this is nuts
[05:44] <lifeless> any advice on debugging a 'works locally, fails on ec2' issue ?
[05:44] <lifeless> I've zapped my storm local bugfixes, running whatever the dist facility made
[05:45] <mwhudson> lifeless: how is it failing on ec2?
[05:45] <lifeless> two ways
[05:45] <lifeless> one
[05:45] <mwhudson> are we using lucid ec2 images yet?
[05:45] <lifeless> my queries-are-constant test fails
[05:45] <lifeless> the check for constant finds the second run did 4 less queries
[05:46] <lifeless> the second thing is a check in bug.txt that something takes 2 queries fails - it takes 3 (which is probably appropriate)
[05:46] <lifeless> File "lib/lp/bugs/tests/../doc/bug.txt", line 1133, in bug.txt
[05:46] <lifeless> Failed example:
[05:46] <lifeless>    len(CursorWrapper.last_executed_sql) - queries
[05:46] <lifeless> Differences (ndiff with -expected +actual):
[05:46] <lifeless>    - 2
[05:46] <lifeless>    + 3
[05:46] <mwhudson> lifeless: the only thing that leaps to mind is an isolation failure, some test is leaving something with a __del__ that does queries or something behind
[05:47] <mwhudson> but that seems a bit mental
[05:47] <lifeless> so they do definitely both pass in isolation
[05:47] <mwhudson> lifeless: have you tried running not just your test but a few around that?
[05:47] <lifeless> I guess I'll run a .. yes
[05:47] <mwhudson> sounds horrible though
[05:47] <lifeless> fortunately subunit streams give me a nice log
[05:47] <mwhudson> yeah
[05:48] <lifeless> I'm fully merged with devel too
[05:48] <lifeless> for kicks
[05:48] <mwhudson> i guess the failure of your new test doesn't include the statement logs?
[05:48] <lifeless> it does
[05:48] <lifeless> it doesn't include the statement log for the *first* attempt
[05:48] <mwhudson> ah
[05:48] <lifeless> thats on my todo to fix (use a currying approach)
[05:48] <lifeless> get a matcher which builds a second matcher for you
[05:48] <mwhudson> is it the same number of statements locally the second time
[05:48] <lifeless> and then diffs the statements
[05:48] <mwhudson> >
[05:48] <mwhudson> ?
[05:48] <mwhudson> that'd be awesome
[05:49] <lifeless> yes, locally its the same both times
[05:49] <mwhudson> so it really is the first time doing fewer queries?
[05:49] <mwhudson> than locally
[05:49] <mwhudson> that's very odd
[05:49] <lifeless> no, the first time is doing more
[05:49]  * mwhudson rubs eyes, learns to read
[05:49] <lifeless> but I can't imagine a flush being batched up
[05:50] <lifeless> sorry, crossed wires
[05:50] <lifeless> *that* count change is in bug.txt
[05:50] <lifeless> which I didn't really touch at all.
[05:50] <lifeless> this is my one:
[05:50] <lifeless> Matcher: HasQueryCount(Equals(23))
[05:50] <lifeless> Difference: queries do not match: 23 != 19
[05:50] <lifeless> the 23 is the count from the first API call
[05:50] <lifeless>    self.assertThat(collector, HasQueryCount(Equals(with_2_count)))
[05:50] <mwhudson> and locally both are 23?
[05:50] <lifeless> is what generates that
[05:50] <lifeless> I think so. Locally both are < 24.
[05:51] <lifeless> I'll see if it can go lower, I don't think it can.
[05:51] <mwhudson> so you have the 19 queries that were executed the second time?
[05:51] <lifeless> yes
[05:52] <lifeless> I can pastebin
[05:52] <mwhudson> i guess you could try diffing that to a local run
[05:52] <lifeless> true
[05:54] <lifeless> 23 is the local count
[05:54] <lifeless> so ec2 is missing
[05:54] <spm> right. versions on db-stable and sourcherry/sodium now match. re-running code staging restore atm. pray.
[05:55]  * spm afks for school run & lunch
[05:55] <lifeless> sorry, not missing
[05:55] <lifeless> they are different
[05:55] <lifeless> the 19 we get are the same for the OAuthNonce dance
[05:55] <lifeless> then we get a
[05:55] <lifeless> 6, 6, 'launchpad-main-master', 'SELECT 1 FROM Person WHERE Person.id = %s')
[05:55] <lifeless> which I don't see locally
[05:56] <lifeless> maybe as many as 8 queries
[05:58] <lifeless> select 1 from looks like an updated/changed .exists()
[05:58] <lifeless> mwhudson: I propose to make the constant check a < check
[05:58] <lifeless> mwhudson: that is, it can get lower, it can't get higher.
[06:00] <mwhudson> yay for AP falling over
[06:00]  * mwhudson has to run away anyway
[06:00] <mwhudson> lifeless: good luck
[06:00] <lifeless> mwhudson: do I have your ok for minor hit-it-hard-tweaks
[06:00] <lifeless> like
[06:00] <lifeless> -        self.assertThat(collector, HasQueryCount(Equals(with_2_count)))
[06:00] <lifeless> +        self.assertThat(collector, HasQueryCount(MatchesAny(
[06:00] <lifeless> +            Equals(with_2_count),
[06:00] <lifeless> +            LessThan(with_2_count))))
[06:01] <lifeless> I'm pretty sure this is a storm version skew issue or something like that
[06:01] <lifeless> I've seen the storm cache cause all sorts of stuff (and been filing bugs and mps because of it)
[06:13] <lifeless> ah yes, its definitely storm.
[06:13] <lifeless> bug 619017
[06:13] <_mup_> Bug #619017: __storm_loaded__ called on empty object <Storm:Fix Committed by therve> <https://launchpad.net/bugs/619017>
[06:13] <lifeless> it causes spurious person queries when initialising cached objects
[06:14] <lifeless> I think its causing both
[06:14] <lifeless> because getMessageChunks issues a Person lookup too
[06:15] <lifeless> can probably rewrite that to do a single query instead, but the bug will still shoot us.
[06:28] <stub> lifeless: How are you using a different version of Storm?
[06:28] <lifeless> stub: I was running a version with cache fixes
[06:28] <lifeless> pending 0.18 being released
[06:28] <stub> Right. So any reason not to add that to buildout?
[06:29] <lifeless> yes, the fix provokes another bug
[06:29] <lifeless> that bug is more severe - it stops things dead if triggered, and gustavo is working on the fix
[06:29] <lifeless> he gave me a 'give this a shot' version
[06:29] <lifeless> but untested, I didn't want to land that for all devs, better to wait for 0.18
[06:29] <stub> Right. And getting exact query counts isn't going to work with version skew.
[06:30] <lifeless> anyhow, the query count variation is entirely explained by Person._init lookups
[06:30] <lifeless> stub: the tests I write set an upper bound and then check for consistency, so they should not be very sensitive to storm versions, unless the version has brain damage of some sort
[06:30] <lifeless> (which, sadly, 0.16 and 0.17 did, in different ways)
[06:45] <stub> lifeless: Consider putting a key -> query count mapping in an external file. If we get lots of these tests, we might want to update the counts in bulk as part of a Storm update. Maybe YAGNI.
[06:46] <stub> Hmm... a test fixture that automatically does the count check, unless the magic wand is waved and it records the count check...
[07:40] <lifeless> stub: that might be nice; need to find a way to say 'this is the bit to track
[07:40] <lifeless> '
[07:40] <lifeless> stub: that said
[07:40] <lifeless> stub: if we have a storm update that wants to raise query counts across the board, I'll just send it back with 'no thanks' written in technicolour across its flanks
[07:41] <lifeless> stub: something that lowers query counts won't break the tests.
[07:41] <lifeless> nononono test fix nonononoonono
[07:42] <lifeless> grah
[07:43] <lifeless> sinzui: poolie: sadly there is an API skew between devel and production for this
[07:43] <lifeless> https://lpbuildbot.canonical.com/builders/prod_lp/builds/91/steps/shell_7/logs/stdio
[07:45] <lifeless> spm: whats the blessed way to rollback an lp-prod change (and how did this get so far before being picked up ?)
[07:45] <poolie> lifeless: "this" being?
[07:45] <lifeless> sinzui used the flags API
[07:45] <spm> eek. probably via a cowboy and/or CP vs actually rolling back. where either would essentially revert the change.
[07:46] <lifeless> but you changed the meaning of something - look at the backtrace in the link I posted - between the last release and this release, so his CP to use flags to disable gmaps fails horribly
[07:46] <spm> negative patch if you ken.
[07:46] <spm> lifeless: so. um. don't try and rollout that CP? I was just about to...
[07:46] <lifeless> spm: hell no
[07:47] <lifeless> 83 failures 1 errors
[07:47]  * spm searches that response for possible subtleties...
[07:47] <spm> \o/
[07:47] <lifeless> put it this way; users profile pages would be - more- broken.
[07:48] <spm> lifeless: can you rm that from the requested CP list? actually - it shouldn't be on the CP list until such time as it passes and lands....
[07:50] <lifeless> spm: I've marked it as snafued
[07:50] <lifeless> which sinzui will see
[07:50] <spm> coolio; ta
[07:50] <spm> just don't want an accidental rollout by the wrong one
[07:52] <lifeless> naturally.
[07:52] <lifeless> actually, we can do this much more quickly.
[07:52] <lifeless> can I get you to be remote fingers ?
 yes master, what is your will.
[07:53] <lifeless> on praseodymium
[07:53] <spm> prasé to it's mates. fwiw.
[07:53] <lifeless> firstly, check that the archives/... production-devel and production-stable are up to date with LP - I presume PQM still uses local disk as authoritative ?
[07:54] <spm> should do...
[07:54] <lifeless> just a 'bzr missing lp:~launchpad-pqm/launchpad/production-stable' should be fine
[07:54] <lifeless> (and -devel, in -devel)
[07:56] <lifeless> spm: then, in production-devel
[07:56] <lifeless> does it have a working tree - are there files present? If no, do 'bzr checkout .'
[07:57] <lifeless> finally, 'bzr merge . -r -2..-1'
[07:57] <spm> ~/archives/rocketfuel/launchpad/production-devel <== Branches are up to date.
[07:57] <lifeless> this should affect only files in lib/lp/registry
[07:57] <spm> um. actually. I may have done that wrong. pebkac...
[07:58] <lifeless> specifically, browser/__init__, person, team, tests/mailinglistviews, tests/person-views, 3x stories and 2xtemplates
[07:58] <spm> production-stable You are missing 5 revision(s): 9649 - 9653
[07:58] <spm> prod-devel is fine
[07:59] <lifeless> -stable doesn't actually matter for this
[07:59] <lifeless> the failure was on -devel
[07:59] <spm> oh. and stable is CP's. that only gets updated locally with a full rollout.
[07:59] <lifeless> I wasn't sure when we started (i was reading just ahead of what I typed)
[07:59] <spm> :-)
[07:59] <lifeless> ok, so in -devel
[07:59] <lifeless> is there a tree of files?
[08:00] <lifeless> and shit
[08:00] <spm> yup
[08:00] <spm> to both.
[08:00] <lifeless> bzr update
[08:00] <lifeless> then
[08:00] <spm> (bzr add shit)
[08:00] <lifeless> or
[08:00] <lifeless> bzr st
[08:00] <lifeless> should be empty
[08:00] <spm> $ bzr st ==> working tree is out of date, run 'bzr update'
[08:01] <lifeless> run bzr update
[08:01] <lifeless> :P
[08:01] <spm> just ensuring we're all on the same page :-)
[08:01] <lifeless> its good
[08:01] <spm> Updated to revision 9655.
[08:01] <lifeless> bzr st
[08:01] <lifeless> should be empty
[08:01] <spm> +1
[08:01] <lifeless> bzr merge . -r -2..-1
[08:01] <lifeless> bzr diff -r -2
[08:01] <lifeless> should be empty
[08:02] <spm> bzr merge --preview . -r -2..-1 <== nothing. good enough?
[08:02] <lifeless> #bzr commit -m '[rs=lifeless][rs=spm]
[08:02] <lifeless> spm: huh, no, merge --preview is doing to be totally confused here.
[08:03] <lifeless> spm: bad fingers!
[08:03] <spm> actually - before I make the change; can we step back a bit and get an explanation of what we're doing? reverting a change I gather?
[08:03] <lifeless> yes
[08:03] <lifeless> the reverse patch
[08:04] <lifeless> which is done by 'bzr merge . -r -1..-2', and confirmed by checking that there is no difference against the second commit back.
[08:04] <spm> Oh I see.
[08:04] <StevenK> But you said -r -2..-1 earlier?
[08:04] <lifeless> yes
[08:04] <lifeless> that was a teset
[08:04] <lifeless> I failed it.
[08:04] <lifeless> the diff step would have caught it regardless
[08:05] <spm> what I meant in this case was to (via pqm) submit a reversal patch; not fiddle around with the core players directly.
[08:05] <lifeless> spm: I can do that if you prefer, its rather pointless IMO : the last commit back is known-good
[08:06] <spm> mainly as we have so many moving parts; it's really easy to miss one; whereas going via the normal steps, we achieve bliss.
[08:06] <lifeless> spm: PQM *wasn't used properly* or we wouldn't have the branch broken.
[08:06] <spm> heh
[08:06] <lifeless> spm: if PQM was running the test suite, not running in brain-damaged mode, this wouldn't have happened.
[08:06] <lifeless> its up to you : we can be in test fix for another few hours
[08:06] <lifeless> or we can be out in 5 minutes.
[08:07] <spm> Oh. I think i see. we had a branch submitted directly to prod-devel that hadn't actually passed the testing?
[08:07] <lifeless> that or this branch passes randomly 50% of the time.
[08:07]  * spm eods in 55 mins. the former option is looking appealling >:)
[08:07] <lifeless> I think submitted without testing is what happened.
[08:07] <spm> fair enough
[08:08] <lifeless> tested in devel, and in dbdevel and db-stable, but not vs prod-devel in ec2
[08:08] <spm> right
[08:08] <spm> oki, making magic fingers.
[08:08] <lifeless> so,
[08:08] <lifeless> bzr merge . -r -1..-2
[08:08] <lifeless> will back out the last commit
[08:08] <lifeless> bzr diff -r -2
[08:08] <spm> prod-devel, right.
[08:08] <lifeless> should be empty
[08:08] <lifeless> ack
[08:09] <lifeless> if that diff is empty the tip commit's contents are gone.
[08:09] <spm> um. hang a sec. bzr merge . -r -2..-1 is what you had earlier; vs -1..-2. does it matter?
[08:09] <lifeless> #and we can bzr commit -m '[rs=lifeless][rs=spm] Back out
[08:09] <lifeless> yes, I had it wrong before (and the diff would have show n that)
[08:09] <lifeless> we want 'from tip to the one before'
[08:09] <lifeless> rather than 'from the one before to tip', which already exists in the branch.
[08:10] <lifeless> a --preview here shold in fact work - try if you like
[08:10] <lifeless>  bzr merge --preview . -r -1..-2
[08:11] <lifeless> and we can bzr commit -m '[rs=lifeless][rs=spm][rollback=9655] Back out broken Google maps API fix due to feature flags api differences with production-devel vs devel.'
[08:11] <spm> lifeless: for verification: https://pastebin.canonical.com/36441/
[08:12] <lifeless> spm: that looks right. Is bzr diff -r -2 empty ?
[08:12] <spm> yup
[08:12] <lifeless> bzr commit -m '[rs=lifeless][rs=spm][rollback=9655] Back out broken Google maps API fix due to feature flags api differences with production-devel vs devel.'
[08:12] <spm> ci'ing...
[08:12] <lifeless> and bzr push
[08:13] <spm> Using saved push location: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/production-devel/
[08:13] <spm> Pushed up to revision 9656.
[08:14] <spm> Oh I see what you've done. rather then "uncommit/revert" to get back to X-good. We deliberately roll the reverse patch. Nice! Very clean.
[08:14] <lifeless> thank you _very much_
[08:14]  * spm adds that nice littel trick to his 'bzr notes'
[08:15] <lifeless> uncommit *count* run into trouble if some branch somewhere had pulled in the intervening time
[08:15] <lifeless> and then would need a --overwrite to fix up etc etc
[08:15] <spm> right
[08:15] <lifeless> so, two las tthings
[08:15] <lifeless> production-devel buildbot slave is dead
[08:15] <lifeless> please shoot, corpse in the river.
[08:15] <spm> le-sigh. again?
[08:16] <lifeless> https://lpbuildbot.canonical.com/buildslaves
[08:16] <poolie> lifeless: i don't think the test_addmember failure is related to anything i did
[08:17] <lifeless> poolie: I don't think you've done anything wrong, but I am pretty positive that production's feature flag module treats *something* differently to devels
[08:17] <lifeless> poolie: or sinzui's branch wouldn't have passed buildbot for devel->stable, db-devel->db-stable, but blown up on production-devel
[08:18] <poolie> that's quite possible
[08:18] <lifeless> poolie: I think it would be nice to harmonise them, either by cping your further flags work to production, or something.
[08:18] <poolie> it's a bit opaque which version is running where
[08:18] <poolie> obviously it is in the bzr branches
[08:18] <lifeless> lp:~launchpad-pqm/launchpad/production-stable is whats deployed
[08:18] <lifeless> yes, it would be nice to have this a lot clearer.
[08:19] <lifeless> spm: secondly, I can has profile ?
[08:19] <spm> lifeless: prod-lp kicked off a new ec2 instance just a few mins ago. so not so much wedged as 'still instansiating"
[08:19] <lifeless> spm: ok cool., I thought they were real machines now - mea culpa
[08:19] <spm> and now running apparently. no, just the lucid ones
[08:20] <spm> lp, db_lp and prod_lp are still ec2
[08:20] <lifeless> I'll try to remember. Thanks :)
[08:20] <lifeless> spm: so that leaves just getting a profile from staging :)
[08:20] <spm> is it back yet? I checked about an hour ago and staging was still restoring (successfully tho)
[08:21] <lifeless> front page comes up
[08:21] <spm> it LIVES!
[08:22] <spm> so the 'fix' was simply ensure the latest shiny was loaded for sourcherry to try and rollout; other problems around access etc went away. fwiw.
[08:22] <lifeless> yeah.
[08:22] <lifeless> fragile sucks
[08:22] <spm> i think my first test was a little too early and hence a few versions behind.
[08:22]  * lifeless loads more ammo in the just-one-dep-style-please gun
[08:23] <lifeless> spm: ok, so enable-profiling-and-restart time? pl-pl-please ?
[08:26] <spm> lifeless: go for it, live.
[08:28] <lifeless> thanks
[08:28] <lifeless> got it
[08:28] <spm> revert?
[08:28] <lifeless> please
[08:28] <lifeless> we'll know once it rsyncs if its good or not, but no point slowing up the system in the mean time
[08:29] <spm> restarting....
[08:29] <lifeless> OOPS-1703S298
[08:29] <spm> oh gah. sodium seems to still be borked.
[08:30] <poolie> lifeless: so production-stable's tip is changed pretty much atomically with it actually being deployed?
[08:30] <StevenK> spm: Borked how?
[08:30] <spm> "hardware"
[08:30] <StevenK> spm: Bleh, still :-(
[08:31] <spm> normally it recovers on it's own; but seems to not be coming back.
[08:31] <spm> heh, yeah. I believe it'shad just about everything replaced and still dies.
[08:31] <StevenK> Perhaps it's wet, sodium reacts badly to water. :-P
[08:31] <lifeless> poolie: AIUI
[08:31] <spm> I made that joke /fosty response at the joke theif
[08:32] <StevenK> Haha
[08:32] <lifeless> StevenK: elmo says that they've replaced/reassembled the entire thing
[08:32] <lifeless> its going to be totally replaced, its queued to do so.
[08:32] <StevenK> lifeless: IE, new name, new everything?
[08:32] <lifeless> it'll still be sodium I think :)
[08:32] <lifeless> but new chassis & guts
[08:32] <spm> it'll still be devpad; maybe not sodium.
[08:33] <lifeless> true
[08:33] <lifeless> I live in hope, its a cool name
[08:33] <spm> aiui, nafallo gets naming rights on new boxes. so....
[08:33] <spm> which explains some of the more .... well lets just say: thank $deity for ssh tab completion on names
[08:35] <wgrant> The original armel builders had nicely obscure names. The new ones aren't so good :(
[08:35] <StevenK> I thanked Nafallo for those. He took it as a compliment.
[08:35] <wgrant> Heh.
[08:35] <poolie> StevenK: your hudson url gives 'connection timed out' for me
[08:36] <StevenK> poolie: Let me check, I think I'm a muppet
[08:36] <StevenK> poolie: Should work now
[08:37] <lifeless> StevenK: are you wearing the muppet hat?
[08:37] <lifeless> StevenK: also congrats
[08:37] <poolie> also, does it really need to be private?
[08:37] <wgrant> Does canonical.com support Unicode subdomains?
[08:37] <poolie> istm the readonly mode could be public
[08:38] <wgrant> Nafallo could have looots of fun with that :P
[08:38] <spm> lifeless: sodium should be back
[08:38] <lifeless> spm: can you make the rsync magic magically happen ?
[08:38] <spm> only by magik
[08:39] <StevenK> poolie: I'm paying for the box currently, so I'd like to limit the number of people that can fiddle for the moment
[08:39] <spm> lifeless: syncin'....
[08:39] <lifeless> spm: would that be the magic bus ?
[08:39] <spm> #42, yes.
[08:40] <poolie> StevenK: logging in sends me back to the default apache "it works" page
[08:40] <spm> lifeless: should be there now
[08:40] <lifeless> thanks
[08:40] <lifeless> pulling
[08:40] <spm>  /srv/launchpad.net-logs/staging/asuka/
[08:40] <poolie> i guess because of going back to http not https y
[08:40] <StevenK> poolie: Right.
[08:40] <poolie> you probably just want a redirect there
[08:40] <lifeless> spm: /profiling/ :P
[08:40] <StevenK> poolie: I just put one in, you were probably too fast
[08:40] <spm> yeah, that too.
[08:41] <poolie> i would have expected to see some history for previous builds?
[08:41] <poolie> StevenK: istm that allowing anonymous readonly access wouldn't cost you very much?
[08:41] <poolie> i'm not suggesting allowing people to start new builds
[08:41] <lifeless> ooh shiny this looks like one I may be able to stop on hard
[08:42] <poolie> just to see if the previous ones wokred
[08:42] <lifeless> spm: also you can probably clear out > 3 day old profiling things automatically (rephrase - we need to :P)
[08:42] <StevenK> poolie: Hmmm, I can do that
[08:42] <poolie> just an idea
[08:42] <lifeless> spm: as we're going to have on-demand profiling soonish.
[08:42] <spm> gah. I thought I'd done something like that; maybe just manually...
[08:42] <StevenK> poolie: And history should be there, drill down into the jobs
[08:43] <lifeless> spm: I know you did manually the other week
[08:43] <wgrant> StevenK: What's this thing?
[08:43] <spm> rm 2008-09-16_2* <== no.
[08:43] <StevenK> wgrant: A hudson install
[08:43] <StevenK> wgrant: Have a look: https://hudson.wedontsleep.org/
[08:44] <lifeless> StevenK: have you hooked up junitxml test reports ?
[08:44] <StevenK> lifeless: Yup
[08:44] <lifeless> awesome
[08:45] <poolie> StevenK: hm it wasn't there before
[08:45] <lifeless> yeah, thats shiny
[08:45] <lifeless> bah, that failure isn'y very useful though :(
[08:45] <lifeless> https://hudson.wedontsleep.org/job/db-devel/4/testReport/junit/lp.codehosting.puller.tests.test_worker/TestWorkerProgressReporting/test_network/
[08:46] <StevenK> I don't understand the test failures :_/
[08:46] <wgrant> StevenK: Shiny.
[08:46]  * StevenK kicks apache until his redirect works
[08:47] <poolie> StevenK: is this supposed to eventually supersede buildbot?
[08:48] <StevenK> poolie: I'd like for that to be the plan
[08:48] <StevenK> I suspect others would too
[08:48] <lifeless> hmm, something funky in the test times
[08:48] <lifeless> EMUSTLOOKATTHATSOMEDAY
[08:49] <StevenK> It doesn't currently use ec2 to build since I suspect it would bankrupt me within 2 or 3 days
[08:49] <lifeless> get a UEC account
[08:49] <poolie> lifeless, the dupefinder on edge seems to be giving me pretty weird results recently
[08:49] <poolie> is that affected by your search changes or is it no more weird than usual?
[08:49] <lifeless> my search changes were all about the dupefinder
[08:49] <lifeless> it used to | all the terms
[08:49] <StevenK> lifeless: Er, but it isn't really work?
[08:49] <poolie> shall i file a bug or is this known or wontfix?
[08:49] <lifeless> this performed terribly
[08:50] <lifeless> StevenK: its very much work :)
[08:50] <poolie> yes, i remember
[08:50] <lifeless> poolie: if you can quantify whats going on, please do file a bug.
[08:50] <lifeless> poolie: however, until we replace the search engine, I don't think there is much we can do sensibly.
[08:50] <StevenK> lifeless: It's hosted privately, it's been developed privately during spare time; doesn't sound like work to me
[08:51] <lifeless> the performance curve is terrible.; perhaps we could estimate the size and use more broad searches when it won't blow up
[08:51]  * StevenK is sad his mail hasn't had any replies yet
[08:51] <lifeless> StevenK: thats all true; what I mean is that what you are doing is of benefit to the team; and canonical as a sponsor of the team should - dare I say would - be happy to help
[08:52] <lifeless> spm: man, MailingListApplication:MailingListAPIView must get hammered
[08:52] <lifeless> spm: every time, its like 90% of the profiles.
[08:52] <StevenK> lifeless: So, what do you think the next step should be?
[08:52] <spm> lifeless: unknown
[08:53] <lifeless> StevenK: I think getting it to the point that you're fairly confident a failure is genuine is important.
[08:53] <StevenK> lifeless: Agreed
[08:53] <StevenK> I've had one db-devel build pass
[08:53] <lifeless> StevenK: at that point, workflow changes to make it what the pqm buildbot thing does would probably be conceivable (for testfix mode stuff)
[08:54] <StevenK> lifeless: So it should remain where it is for the time being while we work out kinks?
[08:54] <lifeless> StevenK: its what I'd do
[08:54] <lifeless> moving it into prod while its in dev would just add friction to your ability to tweak and fiddle.
[08:54] <StevenK> Oh, clearly
[08:55] <StevenK> lifeless: But should we investigate using UEC as executors in parallel to that?
[08:55] <lifeless> sure
[08:55] <lifeless> I expect gary and or maris will respond
[08:55] <lifeless> I will definitely reply tomorrow if noone else has
[08:55] <StevenK> Yes, I'm looking forward to that
[08:56] <StevenK> The testsuite didn't like running with only 512MiB of RAM
[08:56] <poolie> lifeless: so you did change it to matching N-1 terms?
[08:56] <lifeless> yes
[08:56] <lifeless> vs 1 :P
[08:57] <lifeless> poolie: specifically it searches for a match in any of the N-1 subsets, and scores across all of them
[08:57] <lifeless> so the more detailed you are, the less results you'll get
[08:57] <lifeless> but, unlike before, you can trim too-many-results by being more detailed
[08:58] <poolie> that's true
[08:58] <poolie> but not a very obvious use of this dialog
[08:59] <lifeless> yes
[08:59] <lifeless> I'd have liked to have kept it just as it was
[08:59] <lifeless> and hope to restore that when we overhaul search
[08:59] <poolie> not complaining
[09:00] <lifeless> I know - just expounding
[09:00] <lifeless> hah!
[09:00] <lifeless> is_empty cheap
[09:01] <lifeless> 22% of bug one is in that 'cheap' query
[09:01] <lifeless> jtv: ^
[09:01] <jtv> 22% of Microsoft's market share?
[09:01] <lifeless> no
[09:01] <lifeless> of rendering bug 1
[09:01] <_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:In Progress by sabdfl> <ubuntu-express (Ubuntu):Invali
[09:01] <poolie> https://bugs.edge.launchpad.net/malone/+bug/626656 fwiw
[09:01] <jtv> Thank you mup, we had that one perma-cached
[09:01] <jtv> lifeless: that's astounding
[09:02] <lifeless> jtv: not to me
[09:02] <_mup_> Bug #626656: dupefinder now over-tight <Launchpad Bugs:New> <https://launchpad.net/bugs/626656>
[09:02] <poolie> and fwiw it did in fact fail to find my dupe when used in the usual way
[09:02] <lifeless> poolie: thanks; an open question is whether the usual way should be changed
[09:03] <lifeless> certainly for apport bugs I expect the current behaviour to be ok
[09:03] <poolie> yeah, arguably we should not just restore the old behaviour but instead reconsider the story of bug filing
[09:03] <poolie> hm
[09:03] <poolie> apport bug duping seems problematic in different ways
[09:05] <adeuring> good morning
[09:05] <poolie> hi abel
[09:05] <jtv> hi adeuring
[09:05] <adeuring> hi jtv!
[09:06] <lifeless> jtv: I think its easy for folk to under-estimate the impact of repeated small queries
[09:07] <lifeless> jtv: its more work for the db - scales N rather than log(N); its more friction up and down the call stack.
[09:07] <lifeless> jtv: in short, it adds up - a lot-.
[09:07] <jtv> lifeless: otp
[09:07] <lifeless> jtv: de nada, catch you another time.
[09:08] <mrevell> Morning
[09:08] <StevenK> ... I think it's week 3
[09:13] <jtv> hi mrevell, otp but your help bubble is up on edge.
[09:22] <poolie> lifeless: oh, random thought from somewhere on the highway
[09:23] <poolie> is it at all possible to emit (perhaps shrunken) versions of sql queries into the html as its rendered
[09:23] <poolie> in comments, obviously, and of course only for some users
[09:23] <poolie> or will the template/view layering mean that we don't have a good view of when the query was run, or what caused it?
[09:23] <lifeless> yes, but it would probably cripple things right now
[09:24] <poolie> in what sense?
[09:24] <lifeless> (because template rendering is already a slow spot, and many pages do silly-count numbers of queries.
[09:24] <poolie> ah right
[09:24] <lifeless> so we'd be adding a lookup into a slow code path, that would be exercised a lot.
[09:24] <poolie> sure
[09:25] <poolie> perhaps it could be turned on by a cookie or query parameter
[09:25] <poolie> but it might still be too much
[09:25] <poolie> and the implementation may not be trivial
[09:25] <jtv> lifeless: off phone… about the is_empty queries: sounds like a Foo, count(*) query in disguise.
[09:25] <poolie> because it can't insert comments just anywhere
[09:26] <jtv> poolie: you want to be able to trace back queries to the TAL that necessitates them?
[09:27] <poolie> yes
[09:27] <poolie> perhaps people already have tools to do this or can almost always guess correctly?
[09:28] <jtv> I would like that too.  But getting pretty deep into the critical path.  What if we had a way of inserting HTML comments with the current query count, so we could couple them to the oops report?  Not great, but relatively low impact.
[09:29] <poolie> istm on the way to implementing that, you'd want a thing to tell TAL "emit this comment as soon as is convenient and legal"
[09:29] <poolie> maybe there is such a thing already
[09:30] <lifeless> jtv: I haven't checked the code yet, but I'm pretty sure its a if block guarding an 'expensive' query
[09:30] <lifeless> in a loop
[09:30] <jtv> lifeless: so a Foo, EXISTS(…) query in disguise.
[09:31] <lifeless> jtv: well, I think it just wants to be a single Foo query
[09:31] <lifeless> I'm saying the exists checking is totally unnecessary
[09:31] <lifeless> poolie: I love your idea; please do wishlist it in lp-foundations
[09:31] <lifeless> poolie: I was merely commenting on the pragmaticness of it today ;)
[09:32] <jtv> lifeless: it's not blocking anything _except_ an expensive query?  Or is it a case of "this entire piece of UI shouldn't be displayed"?
[09:32] <mrevell> thanks jtv
[09:32] <mrevell> jtv, I'll land a branch today updating the help content.
[09:34] <jtv> mrevell: yup, you can do that now!  Maybe we'll want things like "user hasn't done any translation _recently_" or (a bit harder) "user hasn't done any translation since I last updated this text."
[09:40] <lifeless> jtv: bug 607935, feel free to dig
[09:40] <_mup_> Bug #607935: timeout on bugtask:+index <timeout> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/607935>
[09:41] <lifeless> 149 calls to bug.isSubscribed and bug.isSubscribedToDupes
[09:41] <jtv> lifeless: I'm a bit sick today, so any effort will be sporadic
[09:41] <lifeless> 148 to the second
[09:41] <lifeless> for team in self.user.teams_participated_in:
[09:41] <lifeless>  ^ warning, this may loop a lot
[09:41] <poolie> lifeless:  https://bugs.edge.launchpad.net/launchpad-foundations/+bug/626673 fwiw
[09:41] <lifeless> if bug.isSubscribed(team) or bug.isSubscribedToDupes(team)):
[09:41] <_mup_> Bug #626673: want sql statements interleaved in html comments <Launchpad Foundations:New> <https://launchpad.net/bugs/626673>
[09:41] <lifeless> poolie: thanks!
[09:42] <poolie> maybe i should have a selfimposed wip limit for wishes
[09:42] <lifeless> jtv: so, this is yet another example of what I've been going on about : looks cheap, but actually, its a good fifth of the page, and a simple set intersection query, done once, can return the info needed
[09:42] <poolie> btv so do you think that would have ever helped you, had it existed at the time?
[09:43] <lifeless> maybe
[09:43] <lifeless> it would provide some hint
[09:43] <lifeless> but what I've seen a lot of you'd just get many queries smooshed at the top of the page
[09:43] <lifeless> (and in fact, I want us to head to that: no queries mid-page)
[09:43] <poolie> right
[09:44] <poolie> i think being told that is still useful
[09:45] <lifeless> also, hah! this template does this:
[09:45] <lifeless> s/template/browser:
[09:45] <lifeless> self.many_bugtasks = len(self.bugtasks) >= 10
[09:45] <jtv> that's an old favorite…
[09:45] <lifeless> oh nvm that one is actually cheap, I guess I'm tired ;)
[09:45] <lifeless> I was going to say that count() is about as expensive as the full query
[09:46] <lifeless> and thus to be avoided like the plague
[09:47] <wgrant> lifeless: Isn't it often even more expensive?
[09:47] <wgrant> Given that the full query is batched.
[09:47] <wgrant> Normally.
[09:48] <lifeless> wgrant: well you're comparing across layers there
[09:48] <lifeless> batching is something the API or UI exposes
[09:48] <lifeless> count() is ~= to getting the last row
[09:48] <lifeless> but, its worse
[09:49] <lifeless> batching in the API makes iteration O(N^2)
[09:49] <lifeless> which reminds me, file a bug to turn it off
[09:52] <lifeless> https://bugs.edge.launchpad.net/launchpad-foundations/+bug/626680 if you're interested
[09:52] <_mup_> Bug #626680: iteration in LP API's is O(N^2) due to batching <Launchpad Foundations:New> <https://launchpad.net/bugs/626680>
[09:53] <jtv> lifeless: is the problem that that subscription line you quoted earlier, the "if bug.isSubscribed" etc. one, should be cached across many bugtasks on a page?
[09:56] <lifeless> jtv: no, its looking for the intersection of 'which of this bug& its dupes am or or my teams subscribed to'
[09:57] <lifeless> so someone in a lot of teams (e.g. me) will cause 2 queries per team: one for subscribed to the bug, one for subcribed to a dupe.
[09:57] <lifeless> its building a list of unsubcribe links
[09:57] <lifeless> so that when you get mail you can click on the link
[09:57] <lifeless> and in the bug you see 'unsubcribe <team-that-is-subscribed> from this bug'
[09:58] <lifeless> we can halve the query count with a trivial helper to query bug.isSubcribed and bug.isSubscribedToDupes in one query
[09:59] <lifeless> and we can make it constant by doing a query that will return the team objects that are in the users participations && subscribed to the bug or a dupe.
[09:59] <lifeless> that will make the OOPS report a lot easier to read, for the next iteration (and according to the report, take 1.5 seconds off of the page, for me, *for all bug pages I look at*.
[10:03] <jtv> lifeless: for cases like this one, perhaps we should get used to exposing methods that compose collections, Storm queries etc.  One of our slowest pages could benefit handsomely from a helper that makes it easy to prejoin the icons for a listof persons or products.
[10:04] <jtv> If we keep optimizing these things at the call sites, we give up some schema flexibility.
[10:05] <lifeless> this is why I want a separate layer
[10:06] <lifeless> I like generic collection
[10:06] <lifeless> I don't think its enough, nor a fit for this sort of thing (yet)
[10:07] <lifeless> well, it might fit this
[10:07] <lifeless> but its a little weird starting with team (the result you want), filtering by your-participations, then by your-bug-subscriptions. Perhaps its not.
[10:08]  * jtv is too used to that to question it
[10:09] <lifeless> whats missing from collections is getting multiple types back
[10:09] <jtv> How is that missing?
[10:09] <jtv> That's built in.
[10:10] <jtv> collection.find(Foo, Count(Foo.bar), PrejoinedExtra)
[10:10]  * jtv steps back outside w/book
[10:11] <lifeless> jtv: I don't know what PrejoinedExtra is
[10:11] <lifeless> jtv: but perhaps you could look at Person.all_members_prepopulated and show me how to Collectionise it
[10:12] <lifeless> *keeping the elegance* of collections.
[10:14] <jtv> lifeless: PrejoinedExtra is a class that I just made up.
[10:20] <jtv> lifeless: for each of those left joins, you'd refine the collection using joinOuter.    I'd also keep a list of "things I want from this query" in the collection, and extend that for each item you added.
[10:20] <jtv> I'm not sure there's a truly elegant solution for the variable result columns because it's not a very elegant way to pose the problem.
[10:21] <lifeless> jtv: well it also needs to populate the cached attributes
[10:22] <lifeless> jtv: I'd like to have a reusable solution to this
[10:23] <jtv> One way to do it might be to pass in a series of classes you want, and use that to base the query construction on, and return that same list.  Bit harsh where you need aliases, of course.  For the counts and exists etc. I think you'd want to stuff the aggregates into the result objects somewhere rather than return them.
[10:23] <jtv> Sorry, I'm being imprecise.
[10:23] <lifeless> thats ok
[10:23] <lifeless> this doesn't have to be solved today
[10:23] <jtv> You return a result set whose columns match the classes that you passed on
[10:23] <jtv> passed in
[10:25] <jtv> The aggregates and other things that aren't easily identified as classes probably shouldn't stand on their own; you could have a delegating "pseudo-model" object, e.g. BugWithTasks, that holds cached data so it's similar to a Bug with lots of @cachedpropertys except the caching goes away at the end of the request.
[10:27] <lifeless> well
[10:27] <lifeless> This comes back to separated storage and logic
[10:27] <lifeless> which I want, I don't think that pushing caching out to a separate place is a sane approach
[10:27] <lifeless> I think that pushing storage out to a separate place may work
[10:29] <jtv> lifeless: sorry, I just noticed I'm no shape to go in depth right now
[10:29] <lifeless> no rush
[10:30] <lifeless> go take some aspirin and lie down
[10:30] <jtv> y
[11:56] <deryck> Morning, all.
[16:46] <Ursinha> sinzui, hi, should I add qa-bad to bug 624981?
[16:46] <sinzui> No, the code never arrives
[16:46] <sinzui> arrived
[16:47] <sinzui> Ursinha, and the code we are using is not on edge or staging, though I landed it first
[16:47] <sinzui> Ursinha, so the bug was tagged for testing, but there is no sever to test on
[17:09] <Ursinha> sinzui, I see
[18:57] <lifeless> moin moin
[19:00]  * lifeless wonders when the next edge update is
[19:01] <deryck> hi lifeless
[19:01] <lifeless> hi deryck
[19:01] <lifeless> deryck: you might like rev 11472 of devel/stable
[19:02] <deryck> lifeless, I saw the commit message, but haven't looked at what's in the rev yet.
[19:03] <lifeless> deryck: its a -little- ugly due to some lazr.restful limits
[19:03] <lifeless> deryck: but function, very functional
[19:04]  * deryck takes a break from test fixing to look
[19:07]  * benji wonders if anyone has looked at just how much spam gets dumped into http://pastebin.ubuntu.com
[19:08] <deryck> lifeless, yeah, that is nice.  I really like the DecoratedResultSet pattern that we've got now.
[19:14] <lifeless> cool
[19:31] <lifeless> deryck: leonardr: either of you know what pageid api/1.0/bugs?assignee=xxx would have ?
[19:32] <leonardr> lifeless, probably something like IBugSet:assignee
[19:32] <leonardr> or IMaloneApplication:assignee
[19:33] <lifeless> IBugSet has no exported() decorators
[19:33] <lifeless> leonardr: does that mean its not exported?
[19:34] <deryck> That translates to searchTasks call.  So wouldn't it be a product.searchTasks or distro
[19:34] <SpamapS> lifeless: seems like it would be a really really fast query.. unless there's no index on the assignee column for bugs. ;)
[19:34] <lifeless> SpamapS: lets get data
[19:34] <lifeless> SpamapS: just figuring out where the code is
[19:34] <leonardr> lifeless: yeah, it's not exported, IMaloneApplication is /bugs/
[19:35] <lifeless> so MaloneApplication:+bugs perhaps
[19:36] <lifeless> which on thursday (last day the ppr ran without sodium trashing it)
[19:36] <lifeless> 256 hits
[19:36] <lifeless> 1744 total seconds,
[19:36] <lifeless> 99% in 16.97 seconds, 6.82 mean, 3.38 stddev
[19:37] <SpamapS> lifeless: I think one problem is, its actually the global bugs list.. assignee=clint-fewbar isn't the right way, is it?
[19:37] <lifeless> mean 33 sql statements
[19:37] <lifeless> looks like its mainly sql time
[19:37] <lifeless> SpamapS: no reason for that not to be fast
[19:37] <SpamapS> lifeless: agreed, :)
[19:37] <lifeless> SpamapS: Ubuntu is ~50% of bugs anyway, so you wouldn't save jack by filtering by it
[19:38] <lifeless> lets see if it has oopsed
[19:38] <lifeless> if it has we'll have some nice data, if it hasn't we can get a profile from staging.
[19:38] <lifeless> it is, for reference, in my hitlist already: https://devpad.canonical.com/~stub/ppr/lpnet/daily_2010-08-25_2010-08-26/timeout-candidates.html
[19:39] <lifeless> about half way down
[19:40] <lifeless> elmo: around ?
[19:43] <lifeless> SpamapS: so now I'm grepping our oopses reports (slow page diagnostics)
[19:43] <lifeless> which will take a while
[19:44] <lifeless> leonardr: so, I'd really really love to get oops ids on apis when ++oops++ is used
[19:44] <lifeless> leonardr: do you have any ideas about a tasteful way to do that
[19:44] <leonardr> lifeless: can you quickly run down how it works on the website?
[19:44] <lifeless> leonardr: sure
[19:45] <lifeless> ++oops++ is triggered by traversal, a match anything adapter
[19:45] <elmo> lifeless: vageuly
[19:45] <lifeless> it sets a global variable that says to the oops code
[19:45] <lifeless> elmo: is there an ETA on sodium - its dying so much that many cron based things we use regularly are not completing
[19:46] <elmo> lifeless: it's been body swapped already
[19:46] <lifeless> leonardr: so when ++oops++ is traversed it sets a glad in errorreport.py
[19:46] <lifeless> elmo: !
[19:46] <elmo> lifeless: that didn't take - the latest theory is that the disks are fucked in interesting ways that is causing the kernel to crash
[19:46] <lifeless> elmo: -ah-
[19:46] <elmo> lifeless: we're going to force an fsck of the disks and if that fails, just give up and replace the box wholesale
[19:46] <lifeless> elmo: sorry for nagging then.
[19:47] <lifeless> SpamapS: ok, I've got an OOPS from a few days back
[19:47] <lifeless> leonardr: s/glad/flag/
[19:48] <lifeless> leonardr: at the end of the request this causes two things: oops report written to disk, and, the oops number put in the comment region in the main template.
[19:48] <leonardr> lifeless, this only happens if there's an oops, right?
[19:48] <lifeless> leonardr: these may actually be the 'same' thing - that is the comment region triggers evaluation of the 'should I write an OOPS code' (when no exception has occured)
[19:49] <leonardr> no oops, ++oops++ does nothing?
[19:49] <lifeless> leonardr: no, ++oops++ makes a 'user requested oops' - its generated regardless, so you get get operational info on not-quite-crashing-but-bad pages
[19:49] <lifeless> there is however no big traceback unless an actual crash did occur (in which case ++oops++ has no effect on what happens)
[19:49] <elmo> lifeless: no problem
[19:50] <lifeless> leonardr: so, for API's something equivalent would be to somehow (not necessarily ++oops++) tell the api code that at the end of generating all its stuff, it should generate an oops
[19:50] <lifeless> the oops goes to disk
[19:50] <lifeless> and the oops id gets put somewhere. like an http header, or in the outermost json dict, or something
[19:50] <leonardr> ok
[19:51] <leonardr> how does it generate an oops? just by raising an exception?
[19:51] <lifeless> it calls into the errorlog code
[19:51] <lifeless> look at maybe_record_user_requested_oops
[19:52] <lifeless> essentially we'd want API's to call that function
[19:52] <lifeless> and store the oops id it returns *somewhere*
[19:53] <lifeless> SpamapS: so https://lp-oops.canonical.com/oops.py/?oopsid=1701M1562 is *a search* that timed out on that api
[19:53] <lifeless> actually no, its a regular webapp request
[19:53] <lifeless> digging more
[19:58] <SpamapS> lifeless: its very consistent at 11 seconds..
[19:58] <SpamapS> lifeless: so I doubt it will time out.
[19:58] <SpamapS> until something else slows everything down that is
[20:01] <leonardr> lifeless: so, oops ids are already sent out in the X-Oops-ID header (don't remember the exact name)
[20:01] <leonardr> X-Lazr-Oopsid
[20:02] <leonardr> that's handled by the WebServiceExceptionView
[20:03] <leonardr> it would be pretty easy to make lazr.restful check for an incoming X-Lazr-User-Triggered-Oops header or something, and raise an exception at the end
[20:03] <lifeless> leonardr: so the point of user triggered oops is that they don't have an exception
[20:04] <lifeless> you still get the response
[20:04] <lifeless> and you get the oops
[20:04] <lifeless> I appreciate this is probably more work :P
[20:04] <lifeless> leonardr: here is my use case
[20:04] <leonardr> since maybe_record_user_requested_oops is launchpad code, we either need to move it out of launchpad or put a hook in the configuration object
[20:05] <lifeless> someone says 'api X is slow', I say 'in your browser, do XXX and look at YYY and tell me the OOPSID'
[20:05] <lifeless> tell you what, I'll file a bug on lp-foundations about it
[20:06] <lifeless> its possibly easy, but isn't -trivial-
[20:06] <lifeless> A hook approach sounds sensible to me (though there is already IRequestEnd which is what errorlog is hooked into; is that perhaps not enough?
[20:07] <leonardr> i don't know about IRequestEnd. at the very least we need a hook so that lazr.restful can set the magic variable that ++oops++ sets
[20:07] <lifeless> so thats set at the start
[20:07] <lifeless> its currently done via a traversal adapter
[20:07] <leonardr> if you want it to be doable in the browser then we need something like ++oops++ to go in the url
[20:07] <lifeless> browser would be ideal
[20:08] <lifeless> its just so much easier for people to adhoc stuff with
[20:09] <lifeless> ah, I have already
[20:09] <lifeless> bug 606952
[20:09] <_mup_> Bug #606952: ++oops++ should work on api urls <performance> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/606952>
[20:09] <lifeless> or martin has
[20:10] <lifeless> SpamapS: so the fallback position is staging.
[20:10] <lifeless> SpamapS: staging is a) slower b) less resources c) lower timeout.
[20:10] <lifeless> I guarantee it will break for us
[20:11] <lifeless> once it comes up after the code update ><
[20:11] <lifeless> matsubara: ping
[20:11] <matsubara> hi lifeless
[20:11] <lifeless> matsubara: thanks for working on https://bugs.edge.launchpad.net/launchpad-foundations/+bug/606184
[20:11] <_mup_> Bug #606184: API Pageid for collections is 'scopedcollection:collectionresource'  which does not mention the origin page id <qa-needstesting> <Launchpad Foundations:Fix Committed by matsubara> <https://launchpad.net/bugs/606184>
[20:11] <lifeless> matsubara: did you see my follow up there ? :)
[20:11] <matsubara> lifeless, np, I saw your comment there and will follow up later
[20:12] <lifeless> ok kk
[20:12] <matsubara> lifeless, but I guess, the fix will have what you want because it builds on benji's work to include the named operation for webservice oopses
[20:12] <matsubara> (unless I misunderstood what you wanted)
[20:12] <leonardr> lifeless: bottom line, it's a small to medium sized project. unless you just want to be able to tell people to use their browsers, there's also a client-side component
[20:13] <lifeless> leonardr: just browser would be fine in the first iteration
[20:13] <lifeless> leonardr: we're very read-heavy, and GETs are fine in the browser
[20:14] <lifeless> matsubara: well I wanted instead of ScopedCollection:CollectionResource, IMaloneApplication:searchTasks
[20:14] <lifeless> matsubara: as I understand it you've added in the *type* of the collection, which is nice to have, but doesn't actually help pinning down the code to look at.
[20:15] <lifeless> matsubara: my goal is that the pageid should be a reliable key for grouping on.
[20:15] <leonardr> lifeless: in that case the main challenge is figuring out which code goes in lazr.restful and which in launchpad. i would like to do something simple like make the /++oops++/ traversal apply to the api vhost and set this magic variable
[20:16] <lifeless> leonardr: that would be lovely
[20:16] <leonardr> so that lazr.restful thinks it's working normally, but the code it's running inside does something different with the data
[20:16] <matsubara> lifeless, take a look here: https://devpad.canonical.com/~lpqateam/edge-oops.html and you'll see how the pageid looks like now with my fix (search for scopedcollection)
[20:16] <matsubara> lifeless, my understanding was that you wanted engineers looking at the oops summary to be able to pinpoint if the CollectionResource triggering the error was under their domain and then act to fix it
[20:17] <matsubara> lifeless, my fix also builds on top of benji's work which includes the named operation (if any) to the page id
[20:17] <lifeless> matsubara: Thats a necessary condition, but not sufficient :)
[20:17] <lifeless> matsubara: I think what you've done is a great improvement.
[20:17] <lifeless> But I'm greedy, I want more.
[20:17] <matsubara> so, it'd look like: ScopedCollection:CollectionResource:#bug-attachment-resource:searchTasks (of course this is a made up pageid)
[20:18] <lifeless> matsubara: so this one -  (ScopedCollection:CollectionResource:#bug_attachment-page-resource)
[20:18] <lifeless> is actually IBug:attachments
[20:18] <lifeless> if an engineer looks at the IBug interface at the exported 'attachments' collection, they are looking at the right place.
[20:19] <lifeless> thats what I'd like to achieve.
[20:19] <lifeless> the scopedcollection:collectionresource stuff is, AFAICT, not relevant
[20:21] <lifeless> losa ping: is the staging update looking normal - they normally finish by 14 past
[20:21] <elmo> lifeless: staging got upgraded to lucid today
[20:21] <lifeless> elmo: \o/
[20:21] <elmo> (just as a data point)
[20:21] <elmo> it may be fucking with the updates
[20:21] <lifeless> elmo: thank you (both for that its upgraded and letting me know)
[20:21] <lifeless> elmo: it wouldn't surprise me
[20:21] <matsubara> lifeless, I see. thanks for the feedback. I guess I'll have to make another patch to accomplish that. Would you file another bug please?
[20:22] <lifeless> elmo: the only signal I have for this is https://staging.launchpad.net/successful-updates.txt AFAIK
[20:22] <lifeless> matsubara: certainly
[20:23] <matsubara> thanks lifeless
[20:24] <mbarnett> lifeless: let me take a look
[20:24]  * mbarnett checks to see if it finisehd in the mean time as it doesn't seem to be running
[20:25] <mbarnett> nope
[20:26] <lifeless> mbarnett: ok, so its still going, thats fine.
[20:26] <mbarnett> lifeless: it isn't
[20:26] <lifeless> mbarnett: is it doing a db restore or just code update?
[20:26] <mbarnett> lifeless: last i see is an error in the logs
[20:26] <lifeless> mbarnett: oh, its fallen over?
[20:26] <lifeless> can I see it?
[20:27] <mbarnett> lifeless: here is the tail of the log...
[20:27] <mbarnett> https://pastebin.canonical.com/36480/
[20:27] <mbarnett> let me know if you would like to see more
[20:28] <lifeless> matsubara: see bug 627027
[20:28] <_mup_> Bug #627027: further improvements to API collection page ids <Launchpad Foundations:New> <https://launchpad.net/bugs/627027>
[20:28] <lifeless> ah slony, I love to hate you
[20:29] <lifeless> mbarnett: AIUI all clients have to be kicked off before the upgrade can happen, but there are staging sso clients connected so it would fail.
[20:30] <lifeless> mbarnett: those sso appservers may have failed to shutdown as per spm's bug filed yesterday
[20:30] <matsubara> thanks lifeless
[20:30] <lifeless> mbarnett: so I'd be inclined to check if thats the case, if so get a thread dump and attach to the bug, then nuke them, then kick the staging restore to go again
[20:31] <lifeless> s/restore/update/
[20:31] <lifeless> mbarnett: of course, you probably are already doing all that and I'm just annoying :)
[20:31]  * lifeless goes for breakfast so he won't be annoying 
[20:33] <mbarnett> heh, i have not initiated any of it, so i will get on it momentarily
[20:40] <lifeless> now, where was I
[20:41] <lifeless> SpamapS: so, we'll look into your api performance question when staging comes up
[20:41] <lifeless> SpamapS: bugs that would have made it easier to look into have been tickled.
[20:42] <lifeless> but for now, I'm going to make BugTask:+index faster
[20:42] <lifeless> it does *300* team subscription lookups on every page for me.
[20:43] <mbarnett> lifeless: heya, do you have that bug #?
[20:43] <lifeless> sec, I'll grab it
[20:43] <mbarnett> i am having parsing problems from my weekend mail!
[20:43] <mbarnett> thanks
[20:44] <lifeless> 626577
[20:44] <lifeless> (I went to launchpad-project and searched for shutdown)
[20:45] <mbarnett> thx
[20:46] <mbarnett> hmm, i wonder if this is the same issue..  lp appservers are a bit differerent than the sso client
[20:47] <lifeless> oh?
[20:47] <lifeless> I thought the sso account was for the sso appservers, which are a fork of lp ?
[20:47] <lifeless> Happy to be told I'm wrong :)
[20:47] <SpamapS> lifeless: doh
[20:47] <mbarnett> sso clients are running their own package, no longer out of lp trunk
[20:47] <SpamapS> lifeless: glad I could help tickle some things. :)
[20:47] <lifeless> SpamapS: :P
[20:48] <lifeless> mbarnett: ah yes, so it would depend if they have got the same issue merged into them
[20:48] <SpamapS> Now I just need to figureo ut why jquery/chrome are not happy w/ the response coming back from the API calls..
[20:48] <lifeless> mbarnett: have they failed to shutdown though ?
[20:48] <mbarnett> yeah, the sso stuff runs right out of apache, so i think they don't
[20:49] <lifeless> mbarnett: really? wow, thats very different. And it still talks to the LP db ?
[20:49] <mbarnett> they have their own sso database
[20:49] <mbarnett> that part of is replicated back into lp
[20:49] <lifeless> mbarnett: are teams replicated to their DB ?
[20:49] <mbarnett> that i am not sure
[20:49] <lifeless> (because sso.ubuntu.com still serves team data)
[20:50] <mbarnett> but if you notice
[20:50] <mbarnett> the sso_staging connections are to the sso_staging database
[20:50] <lifeless> yeah
[20:50] <lifeless> slonik takes a lock on all replication sets before doing any schema changes
[20:50] <lifeless> its a ... feature
[20:51] <mbarnett> i do see 1 read only thread on the lpmain db though
[20:51] <lifeless> that would block it
[20:51] <mbarnett> let me see if that is still there
[20:51] <mbarnett> if not, i'll fire back off the staging restore
[20:51] <mbarnett> maybe someone connected at just a very bad moment
[20:52] <mbarnett> after the kill switch but before the next step..
[20:52] <lifeless> mbarnett: its just a regular update, not a db restore today though, isn't it ? (with incremental db schema changes done automatically)
[20:53] <lifeless> SpamapS: can you please file a bug - IMaloneApplication:searchTasks is slow, include the url you used.
[20:53] <mbarnett> lifeless: the db update failed over the weekend.. stub made some recommendations for a code update then another full update
[20:54] <lifeless> ah ok
[20:54] <mbarnett> so, i believe this to be the full update
[20:54] <lifeless> ok, I shall be patient :)
[20:55] <mbarnett> ok, i am kiling the read only idle connection
[20:55] <mbarnett> and fire it back off
[20:56] <mbarnett> dead
[20:56] <mbarnett> trying again
[20:58] <SpamapS> lifeless: Definitely, I have added it to my todo list, I won't get to it for a couple of hours (have to run to an appointment).
[20:58] <mbarnett> and, we are off.
[20:58] <mbarnett> lifeless: this will of course take a while.
[21:17] <lifeless> mbarnett: of course; thank you!
[21:17] <lifeless> SpamapS: thats ok, we won't fix it for a couple of months.
[22:51] <rexbron> hey
[22:52] <rexbron> could any one comment on why the run command is not supported in recipes on launchpad?
[22:52] <lifeless> because it can run arbitrary code
[22:52] <lifeless> recipes are evaluated in many contexts, only some of which are secured
[22:53] <rexbron> I thought that because of this recipes were run in vms, same as builds
[22:54] <lifeless> source package creation runs in vms
[22:54] <mwhudson> building the source package from the debianized source tree is done in vms
[22:54] <rexbron> http://blog.launchpad.net/cool-new-stuff/launchpads-build-farm-improvements
[22:55] <lifeless> any bugs or registry folk around
[22:55] <lifeless> sinzui: perhaps you are here?
[22:56] <rexbron> lifeless: basically, I need to nest and move the debian folder in the the upstream src from the debian branch
[22:56] <rexbron> merging complains of no common ancestor
[22:56] <rexbron> and nesting was suggested as a work around
[22:57] <lifeless> please file a bug with tasks on udd and bzr-builder
[22:57] <lifeless> with enough details to reproduce
[22:57] <lifeless> certainly we need to meet your needs, but the run command is not how we'd like to do that
[23:00] <rexbron> lifeless: basically, In an ideal world, you could merge just a directory, ignoring the rest
[23:00] <lifeless> spiv has been working on that
[23:00] <lifeless> I believe you can with the lastest builder; it might not be rolled out yet
[23:02] <rexbron> lifeless: rolled out to edge?
[23:02] <lifeless> rexbron: production
[23:02] <lifeless> there is no 'edge' for anything other than the appservers.
[23:02] <rexbron> ahh
[23:02] <lifeless> We're going to get rid of 'edge' for appservers too, but start rolling out much more often (without downtime) in the near future
[23:03] <lifeless> losa ping - staging update; just want to make sure it hasn't fallen over and died again.
[23:04] <rexbron> hmm... so what are my options at this point?
[23:07] <rexbron> lifeless: ^
[23:07] <lifeless> file the bug as I requested
[23:07] <lifeless> udd is a high priority for the bzr team
[23:07] <rexbron> udd? I'm not familar with that term?
[23:08] <lifeless> launchpad.net/udd
[23:08] <lifeless> file a bug there
[23:08] <lifeless> and also add a task to bzr-builder
[23:20] <lifeless> jkakar: around ?
[23:21] <rexbron> https://bugs.edge.launchpad.net/udd/+bug/627119
[23:21] <_mup_> Bug #627119: Can not merge branches that have no common ancestor <Ubuntu Distributed Development:New> <https://launchpad.net/bugs/627119>
[23:26] <jelmer> thumper: hi
[23:26] <thumper> jelmer: morning
[23:27] <jelmer> thumper: did you see my conversation with Peter? It looks like the chicken repo works now.
[23:27] <thumper> yep saw that
[23:27] <thumper> awesome
[23:27] <thumper> thanks for following up
[23:27] <jelmer> thumper: I was wondering if it would still be possible to do a CP (and perhaps also include the newer bzr-svn ?)
[23:28] <thumper> jelmer: we are in week 3 now
[23:28] <thumper> jelmer: and it isn't really critical
[23:28] <lifeless> thumper: what does that mean
[23:28] <thumper> lifeless: what are your thoughts on this?
[23:28] <jelmer> thumper: ok, fair enough
[23:28] <thumper> lifeless: what I'm saying is that we aren't that far from a real release
[23:28] <thumper> jelmer: does it need the bzr 2.2 updates too?
[23:29] <lifeless> thumper: I'm torn
[23:29] <thumper> lifeless: me too
[23:29] <lifeless> on the one hand I love getting things out of the way - low kanban limits
[23:29] <thumper> obviously we want working stuff out ASAP
[23:30] <lifeless> on the other hand, a release is soon
[23:30] <lifeless> thumper: flip a coin?
[23:31] <thumper> is the overhead of chasing the change onto production worth the gain of releasing it 8 or 9 days earlier?
[23:32] <mwhudson> wait until release, spend the time that would have been spent chasing reducing the overhead?
[23:32] <lifeless> I like that suggestion
[23:32] <lifeless> if we have a figure in mind - say we reckon it will take 5 hours to do a CP for all of this.
[23:33] <lifeless> spend those 5 hours making the next one < 5 hours
[23:33] <mwhudson> gary_poster: did we find out why bootstrap.py was connecting to the internet?
[23:35] <jelmer> thumper: I don't know of any people other than Peter are seriously affected by these issues.
[23:35] <thumper> jelmer: given that he has been waiting months, another 8 days won't kill him :)
[23:35] <thumper> I know that isn't a great reason
[23:35] <thumper> but...
[23:36]  * thumper shrugs
[23:39] <lifeless> personally, I'd say this.
[23:39]  * lifeless says it
[23:39] <lifeless> jelmer: if you want to do the work, I'll +1 a production change for it. But it goes by the book and with due caution.
[23:41] <jelmer> lifeless: Thanks, I'll pass. :-)
[23:42] <lifeless> thumper: one thing not covered in our analysis here: doing CP's reduces the size of the 'release'.
[23:42] <thumper> lifeless: one of things that makes it a little less clear for me is the dependancy on the 2.2 change
[23:43] <thumper> there was a big change to get that in
[23:43] <thumper> which caused changes in a lot of tests
[23:43] <jelmer> lifeless: Thumper and I discussed a CP earlier, but admittedly that was much much earlier in the cycle (first few days of week 1 I believe).
[23:43] <thumper> if the tests for the new plugin uses these changes, then it won't apply cleanly
[23:43] <thumper> that is my fear/concern
[23:45] <lifeless> 2.2 was tricky
[23:45] <lifeless> yes
[23:45] <lifeless> I share the concern - thus the due caution comment
[23:47]  * thumper afk for a few minutes for coffee and planning
[23:54] <jelmer> hi rexbron