[00:35] <michaelh1> Speaking of me,
[00:36] <michaelh1> is staging up at the moment?  The last entry on successful-updates is from four hours ago
[00:39] <lifeless> michaelh1: see my question to spm just before the reference to you :P
[00:40] <michaelh1> Thanks.
[00:40]  * spm quickly unignores lifeless, reads, adds the ognore back and crawls into a depper hole.
[00:40] <michaelh1> A different question: is there a way to edit and/or delete bug messages in LP?  If not, can I edit them through the API?
[00:40] <lifeless> ognore eh?
[00:40] <spm> and adds extra typos for maximal confusions
[00:40] <lifeless> michaelh1: no
[00:41] <lifeless> michaelh1: they can be hidden I believe, in extremis
[00:41] <michaelh1> No to both the UI and the API?
[00:41] <lifeless> and the DB
[00:41] <spm> edited for super awesomely special cases as well; but *really* needs to be special.
[00:42] <spm> wich is another way of saying I can probably count on one hand the numbers of times it's been done.
[00:42] <michaelh1> spm: nope, I don't count for that.  We're using messages to add meta data to a bug.  The problem is when the meta data is wrong...  tags are inappropriate.
[00:43] <wgrant> Why are tags inappropriate?
[00:43] <michaelh1> wgrant: we're using tickets to also track the upstream status of a bzr revision
[00:44] <michaelh1> wgrant: the revisions are fully qualified as in lp:gcc-linaro/4.5,revno=1234 and there are up to 20 entries per bug
[00:44] <wgrant> michaelh1: You could keep them in the description.
[00:45] <michaelh1> wgrant: I'd like to have a timestamp and author that is easy to review.  Descriptions not a bad idea though...
[00:51] <lifeless> michaelh1: why are tags inappropriate?
[00:51] <michaelh1> lifeless: see the reply to wgrant above
[00:51] <lifeless> michaelh1: I asked this before and you linked to a wiki page which didn't help me understand your design constraints or needs at all
[00:52] <lifeless> so, I'm open to the idea of building a generic lookaside store, but I want the use cases pinned down very carefully
[00:53] <lifeless> since there are so many ways such a thing could go wrong
[00:58] <michaelh1> lifeless: yip.  I'll try and describe it better over the next few weeks
[01:05] <lifeless> michaelh1: some specifics - please look at https://dev.launchpad.net/ArchitectureGuide for my big picture concerns
[01:05] <lifeless> michaelh1: and for this thing - do we need:
[01:06] <lifeless>  - search facilities (e.g. metadata values, modifiers, modification times)
[01:06] <lifeless>  - logs/journalling
[01:06] <lifeless>  - display UI in web pages
[01:06] <lifeless>  - would this be better served by specific integration-information e.g. 'patches to forward upstream'.
[01:16] <poolie> gary_poster: you could link that lep from the relevant bug
[01:16] <poolie> hi lifeless
[01:16] <lifeless> hi poolie
[01:17] <gary_poster> poolie, I'm trying to link all sub-bugs to LEP, but yeah, you are right, there is that big one, I forgot, thanks
[01:17] <gary_poster> Currently fighting moin syntax though :-P
[01:18] <lifeless> spm: so, in all seriousness - what is the status of staging
[01:18] <spm> gah. sorry distracted. looking...
[01:19]  * lifeless offers spm some eye for the eyeballs
[01:19] <spm> it's building an importd atm; so "soon". I'd suggest counted in smller minutes. maybe 15-30?
[01:20] <lifeless> gary_poster: oh you're still around
[01:21] <lifeless> excellent. I need help.
[01:21] <gary_poster> :-P
[01:21] <lifeless> can you do voice? just so that I can have the code full screen
[01:21] <gary_poster> I'm here because I didn't have time to get the stuff I needed to get done today, done
[01:21] <lifeless> L<
[01:21] <lifeless> :<
[01:21] <gary_poster> :-)
[01:21] <lifeless> Let me sketch my issue, and you can decide if mercy is appropriate
[01:21] <lifeless> I have a patch, lp:~lifeless/launchpad/oops
[01:21] <gary_poster> ok looking
[01:22] <lifeless> which replaces the request_statements list with a Timeline object
[01:22] <lifeless> its failing one test. xx-opstats.txt line 186
[01:22] <gary_poster> and trunk doesn't, I assume?
[01:23] <lifeless> right
[01:23] <gary_poster> ok
[01:23] <lifeless> I've thrown a bunch of print statements at timeout and soft timeout setting code
[01:23] <lifeless> and patched SoftTimeoutView in an attempt to understand it
[01:23] <lifeless> but I'm getting less clue not more
[01:24] <lifeless> http://pastebin.com/xBnLuiPR has my 'wth' edits
[01:24] <lifeless> One of the things confusing me is that AFAICT the test should *never have passed*
[01:24] <lifeless> the page it requests does not DB access, and only DB access can set a hard timeout
[01:24] <gary_poster> heh, one of those
[01:25] <lifeless> I added DB access in, but that didn't work
[01:25] <lifeless> setting the soft time out to 200ms seems to work
[01:25] <lifeless> 20 doesn't reliably
[01:25] <gary_poster> have to get boys out of bath, but back in a bit
[01:25] <lifeless> I'd be ok with a 'does not blow up in the next devs face' tweak
[01:25] <lifeless> kk
[01:25] <lifeless> will rabbit on a bit
[01:26] <lifeless> Ideally I'd actually fix it
[01:26] <lifeless> one thing i'm sure is happening is this:
[01:26] <lifeless>  the db code I addedfire, a timeout is raised
[01:27] <lifeless> but the end request hook sees the request as having no oops id
[01:27] <lifeless> and the oops code that puts the oops id into the request isn't running until *after* the end request hook has set a soft request oops id
[01:33] <gary_poster> same request?
[01:34] <lifeless> pretty sure
[01:34] <gary_poster> fly by--still putting kids to bed
[01:34] <lifeless> will check
[01:42] <lifeless>     + I (<class 'storm.exceptions.TimeoutError'>, TimeoutError(), <traceback object at 0xf1a16c8>)
[01:42] <lifeless>     + raising request= <canonical.launchpad.webapp.servers.LaunchpadBrowserRequest instance URL=http://localhost> 280053312 (<class 'storm.exceptions.TimeoutError'>, TimeoutError(), <traceback object at 0xf194b48>)
[01:42] <lifeless>     + G
[01:42] <lifeless>     +             UPDATE SessionData SET last_accessed = CURRENT_TIMESTAMP
[01:42] <lifeless>     + end-hook-set-soft <canonical.launchpad.webapp.servers.LaunchpadBrowserRequest instance URL=http://localhost> 280053312
[01:42] <lifeless>     + raising request= <canonical.launchpad.webapp.servers.LaunchpadBrowserRequest instance URL=http://localhost> 280053312 (<class 'canonical.launchpad.webapp.errorlog.SoftRequestTimeout'>, SoftRequestTimeout(None,), None)
[01:42] <lifeless>     + save fool
[01:42] <lifeless>     + raised-soft <canonical.launchpad.webapp.servers.LaunchpadBrowserRequest instance URL=http://localhost> 280053312
[01:42] <lifeless> save fool is where the oops is set on the reuqest
[01:42] <lifeless> so - same request
[01:42] <lifeless> and raising is being called into
[01:49] <thumper> hmm... lunch is caling
[01:49] <lifeless> so _makeErrorReport is barfing
[01:49] <lifeless> and its silently swalloed
[01:49] <lifeless> gary_poster: thanks
[01:50] <gary_poster> lifeless, oh solved
[01:50] <gary_poster> awesome
[01:50] <lifeless> gary_poster: well not solved
[01:50] <lifeless> but progress
[01:50] <gary_poster> I had actually built a branch and was proceeding to start a test
[01:50] <gary_poster> great
[01:51] <lifeless> gary_poster: its failing to call sys.exc_info() ><
[01:51] <gary_poster> heh
[01:51] <lifeless> s/call/print the exc_info/
[01:51] <lifeless> I suspect a securityProxy stabbing me
[01:51] <lifeless> or something
[01:51] <lifeless>     + 'NoneType' object has no attribute 'microseconds'
[01:51] <lifeless> *blink*
[01:53] <lifeless> gary_poster: is there something that one can raise, which *will* escape the publication machinery and give a sensible diagnostic in pagetests?
[01:54] <gary_poster> do not understand "escape the publication machinery"
[01:55] <gary_poster> (IOW, the goal)
[01:56] <gary_poster> lifeless ^^^ (and I need to get off the computer in 4 min for mental health :-) )
[01:56] <lifeless> gary_poster: I was thinking of ways to make this easier to debug in future
[01:57] <lifeless> gary_poster: thank you very much for helping me escape my own mental trap
[01:57] <gary_poster> heh, I'm not sure what I did to help, but I'm glad I did
[01:57] <lifeless> gary_poster: I think I'm good now. May brainstorm later about how this might have been easier to diagnose.
[01:57] <lifeless> gary_poster: you were patient; asked good questions, and sympathised.
[01:58] <lifeless> combination let me take enough of a step back to apply my own awesome skillz :>
[01:59] <lifeless> now to find this ztrace log thing
[02:00] <gary_poster> :-) cool.
[02:00] <gary_poster> have a good day
[02:09] <spm> lifeless: if you haven't noticed; staging lives1
[02:11] <lifeless> spm: \i/
[02:11] <lifeless> spm: erm
[02:11] <lifeless> spm: something is wrong
[02:11] <spm> oh?
[02:12] <lifeless> is https://staging.launchpad.net/successful-updates.txt meant to match with the revision in the footer of staging ?
[02:12] <spm> shuold
[02:13] <lifeless> spm: ...
[02:13] <spm> so I see
[02:14] <spm> is def revno 9738 in the live tree.
[02:15]  * lifeless is skeptical
[02:15] <spm> bzr-version-info.py is 9710 tho. how... curious
[02:15] <spm> launchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ bzr revno
[02:15] <spm> 9738
[02:15] <spm> ^^ bug in bzr then? :-P
[02:16] <lifeless> bug in the lp build thingymajig
[02:16] <lifeless> file a bug with the details please? lp-foundations
[02:17] <spm> wait a sec... something's not right.
[02:18] <spm> https://pastebin.canonical.com/36696/ <== build date vs date
[02:18] <spm> and the rev id for that matter
[02:18] <lifeless> I agree
[02:19] <lifeless> something is wrong :)
[02:19] <lifeless> run generate_version_info
[02:19] <lifeless> see what happens
[02:19] <spm> heh, was kinda hoping that was "Oh That's *X*"
[02:19] <lifeless> gary_poster: a) thanks again; its all sorted and I'm much happier. b) what happened to your mental health :)
[02:20] <gary_poster> lifeless: (a) yay! (b) it appears to be hosed ;-)
[02:20] <lifeless> gary_poster: ah, welcome to my world... we can be insane together :)
[02:20] <gary_poster> lol :-)
[02:20] <spm> um. find . -name 'generate_version*' <== nothing found. ????
[02:21] <lifeless> generate_version_info
[02:21] <lifeless> hmm
[02:21] <lifeless> grepping
[02:22] <lifeless> gary_poster: you may like knowing that the fix to my issue also means that every sql statement *attempted* on 'LaunchpadDatabase' will be logged.
[02:22] <gary_poster> that sounds cool
[02:22] <lifeless> changing the confusion that existed around attempted-but-not-executed
[02:23] <gary_poster> right
[02:23] <lifeless> for now I'm logging them with a duration of 999999
[02:23] <gary_poster> heh
[02:23] <gary_poster> that'll stand out, I hope
[02:23] <lifeless> can dig into the oops stack to see if we can do '-' or something in future
[02:23] <gary_poster> sure
[02:24] <lifeless> spm: scripts/update-bzr-version-info.sh
[02:25] <spm> *blink*
[02:25] <spm> launchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ scripts/update-bzr-version-info.sh
[02:25] <spm> Skipping bzr-version-info.py update; already at revno 9710
[02:25] <lifeless> bzr 2.2
[02:25] <lifeless> bet you its a bug
[02:26] <lifeless> please file; launchpad-foundations + bzr tasks; losa tag, critical (it will really mess us up if that doesn't right itself)
[02:26] <spm> bzr --version ==> Bazaar (bzr) 2.2.0
[02:26] <thumper> wallyworld_: best tales docs I've found http://www.owlfish.com/software/simpleTAL/tal-guide.html
[02:27] <thumper> lifeless: that's that all about?
[02:27] <lifeless> spiv: can you please assist ^ >
[02:27] <lifeless> ?
[02:27] <lifeless> thumper: the thing that updates the revision in the bottom right of staging and edge isn't (updating)
[02:27] <thumper> ah
[02:27] <thumper> ok
[02:28] <lifeless> which suggests, very strongly, a bzr bug
[02:30] <poolie> this twitter oauth thing strongly reminds me of the "we must have unique per app tokens" launchpad used to have
[02:31] <poolie> i think we've now seen the light, fortunately
[02:31] <wgrant> For anonymous access, sure.
[02:32] <poolie> yes
[02:32] <poolie> this was proposed for a long time as a reason we couldn't allow anonymous access
[02:33] <wgrant> Indeed.
[02:33] <wgrant> But Twitter's situation is different, as it's authenticated.
[02:36] <spm> lifeless: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/629212
[02:36] <_mup_> Bug #629212: staging update is showing incorrect version in the html/page footer <canonical-losa-lp> <Launchpad Foundations:New> <https://launchpad.net/bugs/629212>
[02:36] <lifeless> we can have multiple apps per token, its just a config + trust thing
[02:37] <lifeless> spiv: if you could look at bug 629212 that would be awesome
[02:37] <_mup_> Bug #629212: staging update is showing incorrect version in the html/page footer <canonical-losa-lp> <Launchpad Foundations:New> <https://launchpad.net/bugs/629212>
[02:37] <lifeless> spiv: I realise its an interupt; but we're hitting release-week on Monday, and thats already pretty harsh for lp :(
[02:40] <poolie> lifeless: he's out atm (or he was)
[02:43] <lifeless> poolie: oh sure, it can wait for later today
[02:43] <lifeless> poolie: I just wanted to be clear about it all
[02:46] <lifeless> ok, thats pretty impressive, I admit it: 2455  OOPS-1706L64    ProductSeries:+bugs
[02:46] <lifeless> the first number is sql queries
[02:51] <wgrant> You should have seen Soyuz a year or so ago.
[02:51] <wgrant> It was better than that :)
[02:52] <lifeless> wgrant: without timing out ?
[02:53] <wgrant> lifeless: ... just.
[02:54] <lifeless> foods time
[03:19] <mwhudson> lifeless: well, that's why we have such beefy db servers :-)
[03:24] <lifeless> mwhudson: ><
[03:29] <lifeless> wtf
[03:29] <lifeless> we still query AccountPassword?
[03:30] <wgrant> Yup.
[03:30] <wgrant> It's still needed for basic auth.
[03:30] <wgrant> But I don't know why it's queried every time.
[03:31] <lifeless> we don't support basic auth anymore do we? I mean, I know its *enabled*, but *support* is different
[03:31] <wgrant> Right.
[03:31] <wgrant> It's enabled and dangerous.
[03:31] <wgrant> So should be deleted.
[03:31] <lifeless> propose a patch
[03:31] <lifeless> DoIt
[03:31] <lifeless> make it controlled by a feature flag
[03:31] <lifeless> if after the rollout folk are screaming
[03:32] <lifeless> we can add a rule to enable it
[03:32] <wgrant> Ooh, good idea.
[03:32] <lifeless> and transition the remaining user(s)
[03:32] <lifeless> if noone screams by 10.10, we nuke the code completely.
[03:35] <wgrant> (and the table)
[03:35] <lifeless> spm: ping
[03:35] <spm> yo
[03:35] <lifeless> spm: on staging
[03:35] <lifeless> I have some evidence suggesting it *is running 9710*
[03:35] <lifeless> spm: I'd like to dig deeper
[03:35] <lifeless> here is my evident
[03:36] <lifeless> my patch 11483 in stable preloads is_valid_person for https://staging.launchpad.net/ubuntu/+assignments
[03:36] <stub> We still have AccountPassword to support the test OpenId provider the testsuite uses. Dropping it is part of the roadmap I drafted yesterday.
[03:36] <lifeless> but
[03:36] <lifeless> stub: awesome
[03:36] <lifeless> spm: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1707S19
[03:37] <lifeless> shows 191 SELECT ValidPersonCache.id FROM ValidPersonCache WHERE ValidPersonCache.id = %s LIMIT 1 calls
[03:37] <lifeless> spm: on edge, when I checked yesterday, the page didn't have that
[03:37] <wgrant> stub: I guess it really just needs a place to enter an email address?
[03:37] <wgrant> And maybe a button to fail auth, to make testing easier.
[03:38] <stub> wgrant: Yes, and a box to override the OpenId Identity to test some edge cases.
[03:38] <wgrant> Since that's difficult at the moment.
[03:38] <lifeless> spm: db-stable 9738 is past 11483
[03:38] <wgrant> Ah, true.
[03:38] <lifeless> spm: so, in theory, it should not be issuing those queryies
[03:38] <lifeless> spm: so I think that staging right now is still useless
[03:39] <spm> mmmmmmmmmmm
[03:39] <spm> frown evem
[03:39] <lifeless> spm: "grep 'def _specification_sort' lib/lp/blueprints/model/specification.py"
[03:40] <lifeless> spm: if that finds a function, lets cross check that whats showing on the web *is running from where you think its running*
[03:40] <lifeless> spm: if it doesn't, then what we're seeing on the web is consistent with the source code you're looking at, and we can look at bzr etc
[03:42] <spm>  was going to ask do you have a way to confirm existance... ta
[03:42] <spm> lifeless: most curious. zero results found. ??
[03:42] <lifeless> then we're not running 9738
[03:42] <lifeless> ok
[03:42] <spm> orsum
[03:42] <spm> bzr missing; perhaps?
[03:42] <lifeless> bzr revision-info --tree
[03:42] <lifeless> bzr revision-info
[03:42] <lifeless> run both please
[03:42] <spm> oki
[03:43] <lifeless> bug 629212
[03:43] <_mup_> Bug #629212: staging update is showing incorrect version in the html/page footer <canonical-losa-lp> <Launchpad Foundations:New> <https://launchpad.net/bugs/629212>
[03:43] <spm> wow
[03:43] <spm> launchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ bzr revision-info --tree
[03:43] <spm> 9710 launchpad@pqm.canonical.com-20100827184922-1f7jszjyrqw4dasv
[03:43] <spm> launchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ bzr revision-info
[03:43] <spm> 9738 launchpad@pqm.canonical.com-20100902161314-dy8vhr3l8epgpo2s
[03:43] <lifeless> spm: ok, the tree is out of date
[03:43] <lifeless> I'll update 629212 with that
[03:43] <lifeless> spm: please run 'bzr update'
[03:43] <lifeless> and make run, or however you kick things off
[03:44] <spm> i wonder if this'll break staging rather spectacularly - the DB would have been set via 9710....
[03:44] <lifeless> spm: almost certainly
[03:44] <spm> as we just rsync the code around. whee.
[03:44] <lifeless> well
[03:44] <lifeless> where does this code start?
[03:44] <lifeless> whats the source of the rsync
[03:44] <lifeless> and if you say sodium, el crasho, I'm going to cry.
[03:45]  * spm hates to see a grown man cry and says nothing
[03:45] <lifeless> ok, so you probably just need to run 'bzr update' on sodium
[03:45] <spm> launchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ bzr update
[03:45] <spm> bzr: ERROR: This tree contains left-over files from a failed operation.
[03:45] <spm>     Please examine /srv/staging.launchpad.net/staging/launchpad/.bzr/checkout/limbo to see if it contains any files you wish to
[03:45] <spm>     keep, and delete it when you are done.
[03:45] <lifeless> and kick the whole thing off again
[03:45]  * spm puts head in hands and tears up a little
[03:46] <lifeless> sodium will be like that too
[03:46] <lifeless> lets fix at root
[03:46] <spm> yeah, that's where I'm heading
[03:48] <spm> damn. sodium is really futzed with that tree.
[03:48] <lifeless> bug updated
[03:48] <lifeless> whats the sodium rt ? I cannae see it
[03:49] <spm> the h/w fail one? not sure if it even has one tbh
[03:50] <lifeless> it had one
[03:50] <lifeless> it may have been closed after the chassis was replaced.
[03:51] <spm> have moved the existing db-stable to *.busticated; doing a brand new branch et al to be sure to be sure
[03:52] <lifeless> thanks
[03:52] <lifeless> sorry for the interrupt
[03:58] <lifeless> spm: I've updated the bug and the bodyswap RT ticket
[03:59] <lifeless> stub: sodium has less CPU and RAM than it did, its why the PPR may be running into swap when it wasn't before.
[03:59] <spm> oh no, is fine; and yeah that sounds about right.
[04:01] <lifeless> hah
[04:01] <lifeless> rt fail
[04:01] <lifeless> it mades a new ticket
[04:01] <lifeless> 41200 ><
[04:01] <lifeless> spiv: unping
[04:09] <stub> lifeless: ahh
[04:10] <lifeless> stub: I only found out reading closed rt tickets looking for the sodium thing
[04:10] <lifeless> spm: it would be nice to share those things btw, just saying
[04:11] <lifeless> (and I know it wasn't up to you :>)
[04:13] <lifeless> http://ayende.com/Blog/archive/2010/08/31/it-really-happened-legacy-programmers-tales.aspx?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+AyendeRahien+(Ayende+%40+Rahien)
[04:17] <stub> Is sourcepackage one word or two in the Launchpad vocabulary? I can never remember. I think I even asked this yesterday...
[04:17] <lifeless> one I think
[04:19] <stub> BinaryPackageBuild thinks it is two, pretty much everything else thinks its one. But BPB is new and perhaps the new way forward?
[04:20] <wgrant> stub: Two.
[04:20] <wgrant> What says it's one?
[04:20] <lifeless> spm: Segmentation fault in cron still waorries me
[04:24] <stub> wgrant: hyphenation of database column names... haven't looked for capitalisation in Python source code.
[04:25] <stub> but hyphenation can be wonky from when we didn't underscore between words.
[04:25] <wgrant> stub: Pretty much everything except BPB predates the hyphenation.
[04:25] <wgrant> Exactly.
[04:25] <stub> Ta.
[04:25] <wgrant> It's been capitalised SourcePackage since like 2005.
[04:30] <spm`> pqm@devpad:/code/rocketfuel-built/db-stable$ bzr revision-info --tree
[04:30] <spm`> 9739 launchpad@pqm.canonical.com-20100902175034-3o81ksb8qmulx82x
[04:30] <spm`> pqm@devpad:/code/rocketfuel-built/db-stable$ bzr revision-info
[04:30] <spm`> 9739 launchpad@pqm.canonical.com-20100902175034-3o81ksb8qmulx82x
[04:30] <spm`> lifeless: ^^
[04:30] <lifeless> spm`: \o/
[04:31] <lifeless> spm`: time for the acid test
[04:31] <spm`> :-)
[05:05] <thumper> :(
[05:05] <thumper> 'lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout' spurious test failure
[05:10] <spm`> lifeless: right. timing worked against us a bit there; have stabbed the (still doing it wrong) restore;  manually ensureed we have the latest pony; and the next restore in .... 4 minutes, should push out 9739
[05:10] <lifeless> spm`: if my fingers cross any harder they will break
[05:10] <spm`> :-)
[05:10] <lifeless> stub: did you remember where our new storm base base class is ?
[05:11] <stub> lifeless: I don't think anyone has created it yet.
[05:12] <lifeless> stub: is it ok to use storm.base.Storm ?
[05:12] <lifeless> stub: on completely new things?
[05:13] <lifeless> (TimeLimitedToken for the librarian )
[05:13] <stub> I use storm.locals.Storm, but yet. Better to create a three line subclass in the LP tree though and use it. I don't think anyone knows where it should live though.
[05:13] <stub> c/yet/yes/
[05:13] <lifeless> stub: I want to get this branch in shape for Abel
[05:14] <lifeless> if we're already using Storm directly, anywhere, we'll need to do a mass migration, so its no harder.
[05:14] <stub> We already use Storm directly elsewhere.
[05:14] <lifeless> thanks
[05:18] <lifeless> time to rebuild my session db
[05:18] <lifeless> stub: will you want a sql patch for the tokens
[05:18] <lifeless> stub: or just grab the table defn I put in session.sql
[05:24] <lifeless> spm`: staging mail came through, but staging is down.
[05:24] <lifeless> spm`: is that normal, monsieur backtick ?
[05:24] <spm`> yeah. I manually rekicked it.
[05:24] <lifeless> spm`: just a minute ago ?
[05:24] <spm`> about 10ish
[05:24] <spm`> 15 I kicked; it should have restarted abot 10 ago
[05:25] <spm`> and isn't. sigh.
[05:25] <lifeless> spm`: I saw mail 6 minutes or so ago
[05:26] <spm`> yeah. something funkies up
[05:26] <stub> lifeless: I need to manually apply things, so no need for a database patch. Needs a note on 'unusual deployment' on the rollout page, because things will explode if I forget.
[05:27] <stub> lifeless: Staging will explode too if it lands without work, so I guess I should make the changes to prod and staging just before you land.
[05:31] <spm`> wowo. something really blew up.
[05:31] <spm`> Fri Sep 3 04:22:22 UTC 2010 Applying database updates and permissions to DB
[05:31] <spm`> Traceback (most recent call last):
[05:31] <spm`>   File "./upgrade.py", line 13, in <module>
[05:31] <spm`>     import _pythonpath # Sort PYTHONPATH
[05:31] <spm`> ImportError: No module named _pythonpath
[05:31] <spm`> stub: ^^ staging restore just now
[05:31] <lifeless> buildout
[05:32] <stub> Yay, it isn't me!
[05:32] <spm`> although this looks bad too: urllib2.URLError: <urlopen error [Errno 2] No such file or directory: '/srv/staging.launchpad.net/staging/launchpad/download-cache/dist/setuptools-0.6c11-py2.6.egg'>
[05:32] <spm`> ha
[05:32] <lifeless> whats the staging librarian url ?
[05:34] <lifeless>         ZopeXMLConfigurationError: File "/home/robertc/launchpad/lp-branches/working/lib/canonical/shipit/browser/configure.zcml", line 7.4-12.49
[05:34] <lifeless>         ImportError: No module named cachedproperty
[05:34] <lifeless> *hate hate hate stab stab stab*
[05:37] <spm`> lifeless: librarian.staging.launchpad.net ? or did you mean something else...
[05:37] <wgrant> lifeless: Split split split.
[05:37] <wgrant> spm`: Er, really?
[05:38] <wgrant> I really hope not.
[05:38] <lifeless> spm`: the staging equivalent to launchpadlibrarian.net
[05:39] <spm`> oh that one. right. staging.launchpadlibrarian.net
[05:39] <wgrant> (phew)
[05:39] <spm`> you may begin to see a pattern here... :-)
[05:40] <lifeless> thumper: can you please put rt 41202 to pri 89
[05:41] <thumper> lifeless: possibly, trying to login
[05:41] <lifeless> where do shipit bugs get filed?
[05:42] <lifeless> like 'we can't rollout until this is fixed' style shipit bugs
[05:42] <wgrant> shipit
[05:42] <wgrant> But they're normally fixed by LP...
[05:42] <lifeless> yeah
[05:42] <lifeless> I have a sketchy patch
[05:42] <thumper> lifeless: done
[05:42] <wgrant> But it can't be broken, can it?
[05:42] <wgrant> ec2 tests it.
[05:42] <wgrant> So does buildbot.
[05:43] <lifeless> wgrant: guess what
[05:43] <lifeless> wgrant: go on guess
[05:43] <lifeless> wgrant: I dares you
[05:45] <wgrant> My guess would be python version differences, but that doesn't seem relevant here.
[05:45] <wgrant> Or buildbot wasn't update-sourcecoding.
[05:45] <wgrant> Or prasé isn't.
[05:45]  * spm` notes the 'prasé' and smiles :-)
[05:46] <lifeless> wgrant: zcml barfs in includes that don't affect the core test suite don't break it.
[05:46] <lifeless> wgrant: or something ~= to that
[05:46] <wgrant> lifeless: O_o
[05:46] <lifeless> wgrant: anyhow, I dunno.
[05:47] <lifeless> maybe allenap cheated and didn't ec2land, in which case its going to go boom and we'll be in testfix in, oh, 2.5 hours
[05:47] <thumper> ec2 giving WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
[05:47] <thumper> anyone else seeing this?
[05:47] <thumper> is there a reason?
[05:48] <lifeless> wgrant: https://bugs.edge.launchpad.net/shipit/+bug/629259 for your pain
[05:48] <_mup_> Bug #629259: cachedproperty was imported by shipit <canonical-losa-lp> <ShipIt:Confirmed> <https://launchpad.net/bugs/629259>
[05:49] <mwhudson> lifeless: have you run utilities/update-sourcecode?
[05:49] <lifeless> mwhudson: I'm checking that now, but pqm barfed at me with the same error earlier
[05:49] <mwhudson> ah
[05:49] <lifeless> mwhudson: on a production config change
[05:50] <mwhudson> oh there's probably some config-manager branch that needs to change
[05:50] <mwhudson> maybe
[05:50] <lifeless> yes
[05:50] <lifeless> I suspect so now having checked
[05:50] <lifeless> sourcecode has been udpated
[05:50] <wgrant> thumper: That just means you're lucky.
[05:51] <wgrant> thumper: You've got an instance reusing an old IP.
[05:51] <mwhudson> the config-manager config is in the lp-production-configs branch iirc...
[05:51] <thumper> :(
[05:51] <lifeless> mwhudson: yes, had a lovely race condition there last night
[05:51] <thumper> wgrant: three times in a row
[05:51] <thumper> must be extra lucky*
[05:51] <lifeless> actually its a bug
[05:51] <lifeless> if you're using ec2 land/test
[05:51] <mwhudson> thumper: rm ~/.ec2/known_hosts
[05:52] <lifeless> that is meant to zap stuff and set it up for you
[05:52] <wgrant> Ah, right.
[05:52] <thumper> lifeless: using ec2 test
[05:53] <mwhudson> ec2 test should probably use the ssh -RogerMeHarder options to not get it to check keys at all
[05:53] <mwhudson> if it has them even
[05:54] <lifeless> mwhudson: every now and then your uk heritage shines through ;)
[05:54] <thumper> so I can fix this by removing the known_hosts
[05:54] <thumper> ?
[05:55] <mwhudson> thumper: yes
[05:55] <mwhudson> until next time
[05:55] <mwhudson> lifeless: >:)
[05:55]  * thumper sighs
[06:01] <lifeless> spm`: when is the next edge update ?
[06:01] <lifeless> spm`: can we abort it ? tip of stable is not safe to rollout.
[06:07] <lifeless> I really dislike this Robert Collins(bugnumnber)...
[06:07] <lifeless> in emails
[06:08] <wgrant> As in the fake bug email address?
[06:08] <lifeless> yes
[06:09] <wgrant> Yes :(
[06:12] <lifeless> ok -> airport. bbiab
[07:02] <spm`> lifeless: about one hour, and yes.
[07:37]  * StevenK tries to work out what Zope is telling him
[07:37] <wgrant> StevenK: What's it complaining about?
[07:40] <StevenK> wgrant: ComponentLookupError
[07:42]  * StevenK grumbles at zcml
[07:59] <lifeless> spm`: can we stop it ?
[07:59] <spm`> lifeless: it is: https://wiki.canonical.com/InformationInfrastructure/OSA/LaunchpadProductionStatus#Edge%20Updates
[07:59] <lifeless> spm`: thanks
[07:59] <spm`> np.
[07:59] <spm`> I should relaly restart bip to get rid of that '`'
[08:00] <lifeless> I'll put the rollback into pqm in a few minutes
[08:00] <spm> coolio
[08:01] <lifeless> spm: hows staging?
[08:03] <spm> still unwell I assume, haven't chased tbh.
[08:38] <spm> lifeless: revno 9741 should be being rolled out in about 5 mins. (fingers crossed etc etc)
[08:39] <spm> on staging, as in.
[08:46] <lifeless> spm: great, thanks
[08:46] <adeuring> good morning
[08:56] <lifeless> spm: bugger
[08:56] <lifeless> spm: can you stamp on pqm for me
[08:56] <spm> wich part?
[08:56] <lifeless> the job I just mistargeted to prod-devel
[08:56] <spm> oops
[08:58] <spm> lifeless: lucky you
[08:58] <StevenK> Is there some way I can cleanup my lp-sourcedeps/eggs directory?
[08:58] <spm> Commit message [[rs=deryck][ui=none] Rollback rev 11491.] does not match commit_re
[08:59] <lifeless> spm: thanks
[09:02] <lifeless> hopefully this one will get through
[09:02] <lifeless> adeuring: I have a rollback landing
[09:03] <adeuring> lifeless: I've seen your mails (but haven't had yet enough coffee)
[09:04] <lifeless> adeuring: thats cool
[09:04] <lifeless> adeuring: I"m starting to update bugs and stuff now that I'm back home from the airport
[09:04] <lifeless> adeuring: simply letting you know so that we don't roll it back twice :P
[09:04] <adeuring> lifeless: ok. so, how shall we proceed regarding the need to get the retracers working again?
[09:05] <lifeless> the firewall port has been opened
[09:05] <lifeless> adeuring: if lp is serving the restricted librarian urls out now, it should be working
[09:05] <adeuring> lifeless: right
[09:05] <lifeless> adeuring: have your coffee; catch up on mails
[09:06] <adeuring> ;)
[09:06] <lifeless> that will give me time to update bugs to make the plan as clear as I can
 lifeless: I've seen your mails (but haven't had yet enough coffee) <== Man after my own heart. I too find dealing with lifeless and his emails to be much smoother post coffee. ;-)
[09:21] <allenap> lifeless: Have I broken something?
[09:22] <lifeless> allenap: I'm not sure
[09:22] <allenap> :)
[09:22] <lifeless> allenap: have you landed the shipit change?
[09:22] <allenap> lifeless: Yeah, that landed over a week ago I think.
[09:23] <lifeless> allenap: for cachedproperty use?
[09:23] <stub> Gah. I'm always bitching about other people adding stdout noise to the test suite, and I just notice some from one of my recent landings :-P
[09:23] <allenap> lifeless: Yep.
[09:23] <lifeless> allenap: ah, but your lp branch only landed last night?
[09:23] <allenap> lifeless: Yeah, but versions.cfg only got bumped in my branch too.
[09:24] <lifeless> allenap: so landing a production config change barfed this morning on shipit zcml
[09:24] <allenap> lifeless: Sorry, utilities/sourcedeps.conf
[09:25] <lifeless> allenap: because that doesn't use versions.cfg, we have to land branches that cause incompatibilities in lockstep, not way apart.
[09:25] <lifeless> allenap: it may be fixed now, I'll toss a test change at it.
[09:26] <allenap> lifeless: Was shipit tip pulled into production?
[09:27] <lifeless> allenap: yes, thats how the older stuff works
[09:27] <allenap> lifeless: Gah, I didn't realise that could happen.
[09:27] <lifeless> allenap: not live production, but the mechanism by which production config changes are made
[09:28] <lifeless> allenap: its a hangover, it should be changed.
[09:28] <allenap> lifeless: Okay. I'm sorry about that. I meant to land the two branches close to one another, then realised there were problems with the Launchpad part, and thought it would be okay - because of sourcedeps.conf - to leave shipit as is.
[09:29] <lifeless> allenap: no worries, now I know what went on it makes sense. I'm checking now that there isn't a persisting problem
[09:29] <wgrant> Why doesn't production use sourcedeps.conf?
[09:29] <allenap> lifeless: Cool, thanks for sorting this out while I slept unaware :)
[09:29] <lifeless> wgrant: pqm landing production-configs uses config-manager
[09:29] <lifeless> allenap: I think your branch landing has sorted it
[09:29] <wgrant> lifeless: Yes, but why?
[09:30] <lifeless> wgrant: 20:30 < lifeless> allenap: its a hangover, it should be changed.
[09:31] <wgrant> lifeless: Your clock is wrong.
[09:31] <wgrant> But, ah.
[09:32] <StevenK> wgrant: Pedant.
[09:32] <lifeless> oh foo, I can't land stuff till codehosting is back.
[09:32]  * lifeless files a bug for redundandcy
[09:33] <StevenK> A redundant bug?
[09:42] <jkakar> I think the diff on the merge proposal page would be more readable if it used a fixed-width font.
[09:43] <wgrant> jkakar: It does.
[09:44] <wgrant> jkakar: Are you using edge, and also don't have the UbuntuBeta font installed?
[09:45] <jkakar> wgrant: I'm using edge and I have the UbuntuBeta font installed.
[09:46] <wgrant> Hmm.
[09:47] <jkakar> The diff is shown using the UbuntuBeta font.
[09:47] <wgrant> Not for me :(
[09:48] <wgrant> Maybe my UbuntuBeta is old.
[09:48] <wgrant> Hm, no.
[09:50] <lifeless> thumper filed a bug for this on launchpad-web today, I think.
[09:50] <lifeless> or someone did.
[09:50] <allenap> +1 for diffs in cursive.
[09:50] <StevenK> Argh, no
[09:50] <StevenK> allenap: Evil!
[09:50] <wgrant> lifeless: There is a bug for it, yes.
[09:50] <wgrant> Bug 629181
[09:50] <_mup_> Bug #629181: code review diffs are in a proportional font on edge <launchpad-web:New> <https://launchpad.net/bugs/629181>
[09:51] <wgrant> Hence the question about whether the font was installed.
[09:51] <jkakar> allenap: p:first-letter { font-size: 200% }
[09:51] <lifeless> wgrant: ah, it was you ;p
[09:51] <jkakar> Might as well make the diff look like an old manuscript. :)
[09:51] <allenap> jkakar: Oh, oh, that's so brilliant.
[09:51] <wgrant> lifeless: It wasn't me.
[09:51] <wgrant> I just saw it fly past.
[09:52] <lifeless> oh, someone.
[09:52] <StevenK> lifeless: mwhudson
[09:53] <lifeless> ah yes thanks
[10:02] <lifeless> adeuring: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/395960
[10:02] <_mup_> Bug #395960: proxying user supplied files via the launchpad appserver domain has security and performance issues <librarian> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/395960>
[10:03] <lifeless> adeuring: thats the bug we'll be fixing
[10:04] <adeuring> right
[10:09] <lifeless> adeuring: codehosting is back if you want to pull the private-librarian branch
[10:09]  * adeuring is looking
[10:10] <lifeless> adeuring: there are instructions on https://code.edge.launchpad.net/~lifeless/launchpad/private-librarian/+merge/31020 about getting going with it
[10:31] <lifeless> adeuring: hey, so I've updated the bug and the mp and chased a few things around
[10:31] <adeuring> lifeless: ok
[10:32] <lifeless> adeuring: so I'm going to work on this monday, tuesday etc
[10:32] <adeuring> cool
[10:32] <lifeless> adeuring: if you'd like to collaborate that would be awesome
[10:32] <lifeless> adeuring: I understand deryck is giving you a slab of time to do this, if you're interested.
[10:32] <adeuring> lifeless: sure, I am
[10:32] <lifeless> awesome
[10:33] <lifeless> so, its heading towards late for me; I'd love to give you a brain scan to get my internal state on it
[10:33] <lifeless> but failing that, perhaps you can poke around and ask / discuss to get some info before I crash?
[10:33] <adeuring> lifeless: thanks :) Are the scanners yet internet ready?
[10:34] <lifeless> adeuring: the scanners?
[10:34] <lifeless> oh
[10:34] <adeuring> lifeless: for brain scans
[10:34] <lifeless> for the scan. No. Thats why I can't
[10:34] <lifeless> I have them in the doorways at the house; instant backup
[10:34] <adeuring> lifeless: cool. Anyway, I'm reading the MP, but I'm not that fast ;)
[10:45] <michaelh1> Hey, is the API up at the moment?  I'm getting strange errors using launchpadlib...
[10:55] <lifeless> hmm, its not perf tuesday anymore ;P
[10:55] <lifeless> bah
[10:58] <seb128> hi
[10:58] <seb128> is there any known bug with untagging bugs with the launchpablib api recently?
[10:58] <seb128> the retracers are running but seems untagging is not working
[10:59] <seb128> though the recent retracer crashed on a
[10:59] <seb128> lazr.restfulclient.errors.HTTPError: HTTP Error 500: Internal Server Error
[10:59] <seb128> ...
[10:59] <seb128>       Sorry, you can't upload or download files from Launchpad at the moment,
[10:59] <seb128>       because we're doing feng shui in the server room.  Normal service will
[10:59] <seb128>       resume within an hour.
[10:59] <lifeless> hmm, thats a librarian related error
[10:59] <seb128> I guess that's different from the untagging
[10:59] <seb128> it failed to add the stracktrace
[11:00] <lifeless> mthaddon: sorry for the interrupt; have both librarians been behaving ?
[11:00] <lifeless> seb128: yeah, thats the upload process
[11:00] <mthaddon> not aware of any librarian problems
[11:01] <seb128> ok, I've restarted the retracer
[11:01] <seb128> let's see if that was a one time issue
[11:21] <lifeless> adeuring: well I hope it made sense ;)
[11:21] <adeuring> lifeless: the discussion in the mp? sure
[11:36] <lifeless> adeuring: gnight
[11:37] <lifeless> adeuring: I'll probably poke at this in the weekend; please do push up any stuff you do on top of it and mention in the MP or something
[11:39] <adeuring> lifeless: ok, will do. nice weekend!
[11:42] <noodles775> Night lifeless
[11:49] <jtv> Argh.  This familiar to anyone?  It doesn't seem related to anything I did in my branch.
[11:49] <jtv> ZopeXMLConfigurationError: File "/home/jtv/canonical/lp-branches/bug-618393/ftesting.zcml", line 17.4-17.35
[11:49] <jtv>     ZopeXMLConfigurationError: File "/home/jtv/canonical/lp-branches/bug-618393/lib/canonical/configure.zcml", line 147.4-148.42
[11:49] <jtv>     ZopeXMLConfigurationError: File "/home/jtv/canonical/lp-branches/bug-618393/lib/canonical/shipit/configure.zcml", line 7.4-8.28
[11:49] <jtv>     ZopeXMLConfigurationError: File "/home/jtv/canonical/lp-branches/bug-618393/lib/canonical/shipit/browser/configure.zcml", line 7.4-12.49
[11:49] <jtv>     ImportError: cannot import name cachedproperty
[11:57] <deryck> Morning, all.
[11:57] <noodles775> jtv: bug 629259 ?
[11:57] <_mup_> Bug #629259: cachedproperty was imported by shipit <canonical-losa-lp> <ShipIt:Fix Released> <https://launchpad.net/bugs/629259>
[11:57] <jtv> noodles775: ah thanks!
[11:57] <jtv> hi deryck
[11:57] <seb128> re
[11:58] <seb128> retracer crahed again the same way
[11:58] <seb128> lazr.restfulclient.errors.HTTPError: HTTP Error 500: Internal Server Error
[11:58] <deryck> hi seb128.
[11:58] <seb128>       Sorry, you can't upload or download files from Launchpad at the moment,
[11:58] <seb128>       because we're doing feng shui in the server room.  Normal service will
[11:58] <seb128>       resume within an hour.
[11:58] <seb128> hey deryck
[11:58] <seb128> deryck, seems we progress, the retracers have access to the crashes now
[11:59] <jtv> noodles775: I wonder why suddenly this started affecting me around the time I renamed my branch directory…
[11:59] <seb128> they just get an error when trying to upload the retraced stacktrace
[11:59] <deryck> seb128, ok, let me check on that.
[12:04] <jtv> noodles775: ah, a devel merge was also needed.
[17:11] <mars> beuno, something interesting for you and the design group: http://uxmovement.com/design-articles/faster-with-top-aligned-labels
[17:13] <beuno> mars, based on research, that's how I defined this: https://wiki.canonical.com/UserExperienceDesign/WebGuidelines/Forms
[17:14] <beuno> not that I'm allowed to do design anymore (?)
[17:14] <mars> beuno, that's awesome
[17:14] <mars> the page I mean :)
[17:15] <beuno> heh
[17:15] <beuno> (apologies for all you lurkers not being able to see that page)
[17:15] <beuno> mars, there's a bunch of stuff there: https://wiki.canonical.com/UserExperienceDesign/WebGuidelines/
[17:15] <beuno> I don't think anyone has picked up that work
[21:44] <lifeless> gary_poster: timeline has landed
[21:44] <gary_poster> great, lifeless
[21:44] <gary_poster> that will be nice to see
[21:44] <lifeless> gary_poster: do you know of anything other than (sql, email, memcache) in production that can block atm ?
[21:44] <gary_poster> do the oops tools need a parallel update?
[21:45] <lifeless> gary_poster: it would be nice to teach them to do more with the info and look at a format change; for now I worked in the current constraints as expressed in the LP code
[21:45] <lifeless> so the database column is now category
[21:45] <gary_poster> fair enough
[21:45] <lifeless> 'launchpad-master' will be SQL-launchpad-master
[21:45] <lifeless> so the existing group-and-report-longest should still work
[21:46] <gary_poster> blocks:I'm told that librarian blocks already, and I do seem to recall this.
[21:46] <lifeless> ah yes, I knew that one but forgot. Thanks!
[21:46] <lifeless> I'm looking at doing a tiny patch to add categories and reporting for all of these,
[21:46] <lifeless> .
[21:47] <lifeless> and propose it for RC so we get better reporting during the next cycle
[21:47] <gary_poster> before I forget, lifeless: in regards to your "simplify OOPS emails" email from a while ago, that fits in line with things matsubara and Ursinha had discussed.  In a couple of weeks I'm planning to start focusing them + mars on that.  We'll be in touch soon with our thoughts so we can all be aligned.
[21:48] <lifeless> gary_poster: great, thanks for letting me know.
[21:48] <gary_poster> of course
[21:59] <lifeless> http://queue.acm.org/detail.cfm?id=1854041
[21:59] <lifeless> slightly tweaked version of his blog post
[22:19] <mars> heya lifeless, have a sec for a testrepository question?
[22:19] <lifeless> shoot
[22:19] <mars> ok
[22:19] <mars> I was looking at using testrepository with the Zope testrunner
[22:20] <mars> and thinking about how zope can take test IDs in the form 'bin/test id1 id2 id3'
[22:20] <thumper> lifeless, any quake damage where you are?
[22:20] <lifeless> thumper: rangiora; haven't gone around the town looking but not expecting to find much : old riverbed, not reclaimed land
[22:21] <lifeless> chc turned to jelly after the first shockwave passed through
[22:21] <thumper> I've seen some pictures
[22:21] <lifeless> thumper: my place seems fine so far, haven't gone into the weather to audit the outside yet.
[22:21] <thumper> personally I slept through it
[22:21] <lifeless> the press has some pretty stunning ones
[22:21] <thumper> cool
[22:22] <mars> lifeless, so testrepository has an IDFILE and test_id_option... ok, just thought of something
[22:23] <lifeless> mars: the zope test runner should support --load-list
[22:23] <lifeless> mars: jml patched it up
[22:23] <lifeless> mars: naively, I'd expect the .testr.conf for launchpad to work with it unmodified
[22:23] <mars> lifeless, hasn't landed in upstream.  I have the latest
[22:23] <lifeless> ah
[22:24] <lifeless> maybe zope.testing is ahead of zope itself?
[22:25] <mars> zope.testing + zope.testrunner
[22:25] <mars> (now)
[22:25] <mars> actually
[22:25] <mars> zope.testing + zope.testrunner + zc.recipe.testrunner
[22:26] <lifeless> ok
[22:26] <lifeless> anyhow, --load-list is terribly simple.
[22:26] <lifeless> best thing to do would be to get that upstream
[22:26] <lifeless> it is 'load all the tests. Perform a set intersection.'
[22:26] <mars> lifeless, this should work: deoesn't though:
[22:26] <mars> [DEFAULT]
[22:27] <mars> test_command=./bin/test --subunit $IDOPTION
[22:27] <mars> test_id_option= $IDLIST
[22:27] <lifeless> what dos it do ?
[22:27] <mars> prints the string 'IDLIST'
[22:27] <mars> $ testr run --failing
[22:27] <lifeless> there are a couple of testr bugs in this area
[22:27] <mars> 'IDLIST'
[22:27] <lifeless> up
[22:27] <lifeless> uhm
[22:27] <lifeless> try test_command=./bin/test --subunit $IDLIST
[22:29] <mars> ok, that sort of works, but it looks like it only accepts one module or test
[22:29] <mars> bin/test that is
[22:29] <mars> ok, so upstream is still the best option