=== spiv_ is now known as spiv [00:35] Speaking of me, [00:36] is staging up at the moment? The last entry on successful-updates is from four hours ago [00:39] michaelh1: see my question to spm just before the reference to you :P [00:40] Thanks. [00:40] * spm quickly unignores lifeless, reads, adds the ognore back and crawls into a depper hole. [00:40] A different question: is there a way to edit and/or delete bug messages in LP? If not, can I edit them through the API? [00:40] ognore eh? [00:40] and adds extra typos for maximal confusions [00:40] michaelh1: no [00:41] michaelh1: they can be hidden I believe, in extremis [00:41] No to both the UI and the API? [00:41] and the DB [00:41] edited for super awesomely special cases as well; but *really* needs to be special. [00:42] wich is another way of saying I can probably count on one hand the numbers of times it's been done. [00:42] spm: nope, I don't count for that. We're using messages to add meta data to a bug. The problem is when the meta data is wrong... tags are inappropriate. [00:43] Why are tags inappropriate? [00:43] wgrant: we're using tickets to also track the upstream status of a bzr revision [00:44] wgrant: the revisions are fully qualified as in lp:gcc-linaro/4.5,revno=1234 and there are up to 20 entries per bug [00:44] michaelh1: You could keep them in the description. [00:45] wgrant: I'd like to have a timestamp and author that is easy to review. Descriptions not a bad idea though... [00:51] michaelh1: why are tags inappropriate? [00:51] lifeless: see the reply to wgrant above [00:51] michaelh1: I asked this before and you linked to a wiki page which didn't help me understand your design constraints or needs at all [00:52] so, I'm open to the idea of building a generic lookaside store, but I want the use cases pinned down very carefully [00:53] since there are so many ways such a thing could go wrong [00:58] lifeless: yip. I'll try and describe it better over the next few weeks [01:05] michaelh1: some specifics - please look at https://dev.launchpad.net/ArchitectureGuide for my big picture concerns [01:05] michaelh1: and for this thing - do we need: [01:06] - search facilities (e.g. metadata values, modifiers, modification times) [01:06] - logs/journalling [01:06] - display UI in web pages [01:06] - would this be better served by specific integration-information e.g. 'patches to forward upstream'. [01:16] gary_poster: you could link that lep from the relevant bug [01:16] hi lifeless [01:16] hi poolie [01:17] poolie, I'm trying to link all sub-bugs to LEP, but yeah, you are right, there is that big one, I forgot, thanks [01:17] Currently fighting moin syntax though :-P [01:18] spm: so, in all seriousness - what is the status of staging [01:18] gah. sorry distracted. looking... [01:19] * lifeless offers spm some eye for the eyeballs [01:19] it's building an importd atm; so "soon". I'd suggest counted in smller minutes. maybe 15-30? [01:20] gary_poster: oh you're still around [01:21] excellent. I need help. [01:21] :-P [01:21] can you do voice? just so that I can have the code full screen [01:21] I'm here because I didn't have time to get the stuff I needed to get done today, done [01:21] L< [01:21] :< [01:21] :-) [01:21] Let me sketch my issue, and you can decide if mercy is appropriate [01:21] I have a patch, lp:~lifeless/launchpad/oops [01:21] ok looking [01:22] which replaces the request_statements list with a Timeline object [01:22] its failing one test. xx-opstats.txt line 186 [01:22] and trunk doesn't, I assume? [01:23] right [01:23] ok [01:23] I've thrown a bunch of print statements at timeout and soft timeout setting code [01:23] and patched SoftTimeoutView in an attempt to understand it [01:23] but I'm getting less clue not more [01:24] http://pastebin.com/xBnLuiPR has my 'wth' edits [01:24] One of the things confusing me is that AFAICT the test should *never have passed* [01:24] the page it requests does not DB access, and only DB access can set a hard timeout [01:24] heh, one of those [01:25] I added DB access in, but that didn't work [01:25] setting the soft time out to 200ms seems to work [01:25] 20 doesn't reliably [01:25] have to get boys out of bath, but back in a bit [01:25] I'd be ok with a 'does not blow up in the next devs face' tweak [01:25] kk [01:25] will rabbit on a bit [01:26] Ideally I'd actually fix it [01:26] one thing i'm sure is happening is this: [01:26] the db code I addedfire, a timeout is raised [01:27] but the end request hook sees the request as having no oops id [01:27] and the oops code that puts the oops id into the request isn't running until *after* the end request hook has set a soft request oops id [01:33] same request? [01:34] pretty sure [01:34] fly by--still putting kids to bed [01:34] will check [01:42] + I (, TimeoutError(), ) [01:42] + raising request= 280053312 (, TimeoutError(), ) [01:42] + G [01:42] + UPDATE SessionData SET last_accessed = CURRENT_TIMESTAMP [01:42] + end-hook-set-soft 280053312 [01:42] + raising request= 280053312 (, SoftRequestTimeout(None,), None) [01:42] + save fool [01:42] + raised-soft 280053312 [01:42] save fool is where the oops is set on the reuqest [01:42] so - same request [01:42] and raising is being called into [01:49] hmm... lunch is caling [01:49] so _makeErrorReport is barfing [01:49] and its silently swalloed [01:49] gary_poster: thanks [01:50] lifeless, oh solved [01:50] awesome [01:50] gary_poster: well not solved [01:50] but progress [01:50] I had actually built a branch and was proceeding to start a test [01:50] great [01:51] gary_poster: its failing to call sys.exc_info() >< [01:51] heh [01:51] s/call/print the exc_info/ [01:51] I suspect a securityProxy stabbing me [01:51] or something [01:51] + 'NoneType' object has no attribute 'microseconds' [01:51] *blink* [01:53] gary_poster: is there something that one can raise, which *will* escape the publication machinery and give a sensible diagnostic in pagetests? [01:54] do not understand "escape the publication machinery" [01:55] (IOW, the goal) [01:56] lifeless ^^^ (and I need to get off the computer in 4 min for mental health :-) ) [01:56] gary_poster: I was thinking of ways to make this easier to debug in future [01:57] gary_poster: thank you very much for helping me escape my own mental trap [01:57] heh, I'm not sure what I did to help, but I'm glad I did [01:57] gary_poster: I think I'm good now. May brainstorm later about how this might have been easier to diagnose. [01:57] gary_poster: you were patient; asked good questions, and sympathised. [01:58] combination let me take enough of a step back to apply my own awesome skillz :> [01:59] now to find this ztrace log thing [02:00] :-) cool. [02:00] have a good day [02:09] lifeless: if you haven't noticed; staging lives1 [02:11] spm: \i/ [02:11] spm: erm [02:11] spm: something is wrong [02:11] oh? [02:12] is https://staging.launchpad.net/successful-updates.txt meant to match with the revision in the footer of staging ? [02:12] shuold [02:13] spm: ... [02:13] so I see [02:14] is def revno 9738 in the live tree. [02:15] * lifeless is skeptical [02:15] bzr-version-info.py is 9710 tho. how... curious [02:15] launchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ bzr revno [02:15] 9738 [02:15] ^^ bug in bzr then? :-P [02:16] bug in the lp build thingymajig [02:16] file a bug with the details please? lp-foundations [02:17] wait a sec... something's not right. [02:18] https://pastebin.canonical.com/36696/ <== build date vs date [02:18] and the rev id for that matter [02:18] I agree [02:19] something is wrong :) [02:19] run generate_version_info [02:19] see what happens [02:19] heh, was kinda hoping that was "Oh That's *X*" [02:19] gary_poster: a) thanks again; its all sorted and I'm much happier. b) what happened to your mental health :) [02:20] lifeless: (a) yay! (b) it appears to be hosed ;-) [02:20] gary_poster: ah, welcome to my world... we can be insane together :) [02:20] lol :-) [02:20] um. find . -name 'generate_version*' <== nothing found. ???? [02:21] generate_version_info [02:21] hmm [02:21] grepping [02:22] gary_poster: you may like knowing that the fix to my issue also means that every sql statement *attempted* on 'LaunchpadDatabase' will be logged. [02:22] that sounds cool [02:22] changing the confusion that existed around attempted-but-not-executed [02:23] right [02:23] for now I'm logging them with a duration of 999999 [02:23] heh [02:23] that'll stand out, I hope [02:23] can dig into the oops stack to see if we can do '-' or something in future [02:23] sure [02:24] spm: scripts/update-bzr-version-info.sh [02:25] *blink* [02:25] launchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ scripts/update-bzr-version-info.sh [02:25] Skipping bzr-version-info.py update; already at revno 9710 [02:25] bzr 2.2 [02:25] bet you its a bug [02:26] please file; launchpad-foundations + bzr tasks; losa tag, critical (it will really mess us up if that doesn't right itself) [02:26] bzr --version ==> Bazaar (bzr) 2.2.0 [02:26] wallyworld_: best tales docs I've found http://www.owlfish.com/software/simpleTAL/tal-guide.html [02:27] lifeless: that's that all about? [02:27] spiv: can you please assist ^ > [02:27] ? [02:27] thumper: the thing that updates the revision in the bottom right of staging and edge isn't (updating) [02:27] ah [02:27] ok [02:28] which suggests, very strongly, a bzr bug [02:30] this twitter oauth thing strongly reminds me of the "we must have unique per app tokens" launchpad used to have [02:31] i think we've now seen the light, fortunately [02:31] For anonymous access, sure. [02:32] yes [02:32] this was proposed for a long time as a reason we couldn't allow anonymous access [02:33] Indeed. [02:33] But Twitter's situation is different, as it's authenticated. [02:36] lifeless: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/629212 [02:36] <_mup_> Bug #629212: staging update is showing incorrect version in the html/page footer [02:36] we can have multiple apps per token, its just a config + trust thing [02:37] spiv: if you could look at bug 629212 that would be awesome [02:37] <_mup_> Bug #629212: staging update is showing incorrect version in the html/page footer [02:37] spiv: I realise its an interupt; but we're hitting release-week on Monday, and thats already pretty harsh for lp :( [02:40] lifeless: he's out atm (or he was) [02:43] poolie: oh sure, it can wait for later today [02:43] poolie: I just wanted to be clear about it all [02:46] ok, thats pretty impressive, I admit it: 2455 OOPS-1706L64 ProductSeries:+bugs [02:46] the first number is sql queries [02:51] You should have seen Soyuz a year or so ago. [02:51] It was better than that :) [02:52] wgrant: without timing out ? [02:53] lifeless: ... just. [02:54] foods time [03:19] lifeless: well, that's why we have such beefy db servers :-) [03:24] mwhudson: >< [03:29] wtf [03:29] we still query AccountPassword? [03:30] Yup. [03:30] It's still needed for basic auth. [03:30] But I don't know why it's queried every time. [03:31] we don't support basic auth anymore do we? I mean, I know its *enabled*, but *support* is different [03:31] Right. [03:31] It's enabled and dangerous. [03:31] So should be deleted. [03:31] propose a patch [03:31] DoIt [03:31] make it controlled by a feature flag [03:31] if after the rollout folk are screaming [03:32] we can add a rule to enable it [03:32] Ooh, good idea. [03:32] and transition the remaining user(s) [03:32] if noone screams by 10.10, we nuke the code completely. [03:35] (and the table) [03:35] spm: ping [03:35] yo [03:35] spm: on staging [03:35] I have some evidence suggesting it *is running 9710* [03:35] spm: I'd like to dig deeper [03:35] here is my evident [03:36] my patch 11483 in stable preloads is_valid_person for https://staging.launchpad.net/ubuntu/+assignments [03:36] We still have AccountPassword to support the test OpenId provider the testsuite uses. Dropping it is part of the roadmap I drafted yesterday. [03:36] but [03:36] stub: awesome [03:36] spm: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1707S19 [03:37] shows 191 SELECT ValidPersonCache.id FROM ValidPersonCache WHERE ValidPersonCache.id = %s LIMIT 1 calls [03:37] spm: on edge, when I checked yesterday, the page didn't have that [03:37] stub: I guess it really just needs a place to enter an email address? [03:37] And maybe a button to fail auth, to make testing easier. [03:38] wgrant: Yes, and a box to override the OpenId Identity to test some edge cases. [03:38] Since that's difficult at the moment. [03:38] spm: db-stable 9738 is past 11483 [03:38] Ah, true. [03:38] spm: so, in theory, it should not be issuing those queryies [03:38] spm: so I think that staging right now is still useless [03:39] mmmmmmmmmmm [03:39] frown evem [03:39] spm: "grep 'def _specification_sort' lib/lp/blueprints/model/specification.py" [03:40] spm: if that finds a function, lets cross check that whats showing on the web *is running from where you think its running* [03:40] spm: if it doesn't, then what we're seeing on the web is consistent with the source code you're looking at, and we can look at bzr etc [03:42] was going to ask do you have a way to confirm existance... ta [03:42] lifeless: most curious. zero results found. ?? [03:42] then we're not running 9738 [03:42] ok [03:42] orsum [03:42] bzr missing; perhaps? [03:42] bzr revision-info --tree [03:42] bzr revision-info [03:42] run both please [03:42] oki [03:43] bug 629212 [03:43] <_mup_> Bug #629212: staging update is showing incorrect version in the html/page footer [03:43] wow [03:43] launchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ bzr revision-info --tree [03:43] 9710 launchpad@pqm.canonical.com-20100827184922-1f7jszjyrqw4dasv [03:43] launchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ bzr revision-info [03:43] 9738 launchpad@pqm.canonical.com-20100902161314-dy8vhr3l8epgpo2s [03:43] spm: ok, the tree is out of date [03:43] I'll update 629212 with that [03:43] spm: please run 'bzr update' [03:43] and make run, or however you kick things off [03:44] i wonder if this'll break staging rather spectacularly - the DB would have been set via 9710.... [03:44] spm: almost certainly [03:44] as we just rsync the code around. whee. [03:44] well [03:44] where does this code start? [03:44] whats the source of the rsync [03:44] and if you say sodium, el crasho, I'm going to cry. [03:45] * spm hates to see a grown man cry and says nothing [03:45] ok, so you probably just need to run 'bzr update' on sodium [03:45] launchpad@asuka:/srv/staging.launchpad.net/staging/launchpad$ bzr update [03:45] bzr: ERROR: This tree contains left-over files from a failed operation. [03:45] Please examine /srv/staging.launchpad.net/staging/launchpad/.bzr/checkout/limbo to see if it contains any files you wish to [03:45] keep, and delete it when you are done. [03:45] and kick the whole thing off again [03:45] * spm puts head in hands and tears up a little [03:46] sodium will be like that too [03:46] lets fix at root [03:46] yeah, that's where I'm heading [03:48] damn. sodium is really futzed with that tree. [03:48] bug updated [03:48] whats the sodium rt ? I cannae see it [03:49] the h/w fail one? not sure if it even has one tbh [03:50] it had one [03:50] it may have been closed after the chassis was replaced. [03:51] have moved the existing db-stable to *.busticated; doing a brand new branch et al to be sure to be sure [03:52] thanks [03:52] sorry for the interrupt [03:58] spm: I've updated the bug and the bodyswap RT ticket [03:59] stub: sodium has less CPU and RAM than it did, its why the PPR may be running into swap when it wasn't before. [03:59] oh no, is fine; and yeah that sounds about right. [04:01] hah [04:01] rt fail [04:01] it mades a new ticket [04:01] 41200 >< [04:01] spiv: unping [04:09] lifeless: ahh [04:10] stub: I only found out reading closed rt tickets looking for the sodium thing [04:10] spm: it would be nice to share those things btw, just saying [04:11] (and I know it wasn't up to you :>) [04:13] http://ayende.com/Blog/archive/2010/08/31/it-really-happened-legacy-programmers-tales.aspx?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+AyendeRahien+(Ayende+%40+Rahien) [04:17] Is sourcepackage one word or two in the Launchpad vocabulary? I can never remember. I think I even asked this yesterday... [04:17] one I think [04:19] BinaryPackageBuild thinks it is two, pretty much everything else thinks its one. But BPB is new and perhaps the new way forward? [04:20] stub: Two. [04:20] What says it's one? [04:20] spm: Segmentation fault in cron still waorries me [04:24] wgrant: hyphenation of database column names... haven't looked for capitalisation in Python source code. [04:25] but hyphenation can be wonky from when we didn't underscore between words. [04:25] stub: Pretty much everything except BPB predates the hyphenation. [04:25] Exactly. [04:25] Ta. [04:25] It's been capitalised SourcePackage since like 2005. [04:30] pqm@devpad:/code/rocketfuel-built/db-stable$ bzr revision-info --tree [04:30] 9739 launchpad@pqm.canonical.com-20100902175034-3o81ksb8qmulx82x [04:30] pqm@devpad:/code/rocketfuel-built/db-stable$ bzr revision-info [04:30] 9739 launchpad@pqm.canonical.com-20100902175034-3o81ksb8qmulx82x [04:30] lifeless: ^^ [04:30] spm`: \o/ [04:31] spm`: time for the acid test [04:31] :-) [05:05] :( [05:05] 'lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout' spurious test failure [05:10] lifeless: right. timing worked against us a bit there; have stabbed the (still doing it wrong) restore; manually ensureed we have the latest pony; and the next restore in .... 4 minutes, should push out 9739 [05:10] spm`: if my fingers cross any harder they will break [05:10] :-) [05:10] stub: did you remember where our new storm base base class is ? [05:11] lifeless: I don't think anyone has created it yet. [05:12] stub: is it ok to use storm.base.Storm ? [05:12] stub: on completely new things? [05:13] (TimeLimitedToken for the librarian ) [05:13] I use storm.locals.Storm, but yet. Better to create a three line subclass in the LP tree though and use it. I don't think anyone knows where it should live though. [05:13] c/yet/yes/ [05:13] stub: I want to get this branch in shape for Abel [05:14] if we're already using Storm directly, anywhere, we'll need to do a mass migration, so its no harder. [05:14] We already use Storm directly elsewhere. [05:14] thanks === almaisan` is now known as al-maisan [05:18] time to rebuild my session db [05:18] stub: will you want a sql patch for the tokens [05:18] stub: or just grab the table defn I put in session.sql [05:24] spm`: staging mail came through, but staging is down. [05:24] spm`: is that normal, monsieur backtick ? [05:24] yeah. I manually rekicked it. [05:24] spm`: just a minute ago ? [05:24] about 10ish [05:24] 15 I kicked; it should have restarted abot 10 ago [05:25] and isn't. sigh. [05:25] spm`: I saw mail 6 minutes or so ago [05:26] yeah. something funkies up [05:26] lifeless: I need to manually apply things, so no need for a database patch. Needs a note on 'unusual deployment' on the rollout page, because things will explode if I forget. [05:27] lifeless: Staging will explode too if it lands without work, so I guess I should make the changes to prod and staging just before you land. [05:31] wowo. something really blew up. [05:31] Fri Sep 3 04:22:22 UTC 2010 Applying database updates and permissions to DB [05:31] Traceback (most recent call last): [05:31] File "./upgrade.py", line 13, in [05:31] import _pythonpath # Sort PYTHONPATH [05:31] ImportError: No module named _pythonpath [05:31] stub: ^^ staging restore just now [05:31] buildout [05:32] Yay, it isn't me! [05:32] although this looks bad too: urllib2.URLError: [05:32] ha [05:32] whats the staging librarian url ? [05:34] ZopeXMLConfigurationError: File "/home/robertc/launchpad/lp-branches/working/lib/canonical/shipit/browser/configure.zcml", line 7.4-12.49 [05:34] ImportError: No module named cachedproperty [05:34] *hate hate hate stab stab stab* [05:37] lifeless: librarian.staging.launchpad.net ? or did you mean something else... [05:37] lifeless: Split split split. [05:37] spm`: Er, really? [05:38] I really hope not. [05:38] spm`: the staging equivalent to launchpadlibrarian.net [05:39] oh that one. right. staging.launchpadlibrarian.net [05:39] (phew) [05:39] you may begin to see a pattern here... :-) [05:40] thumper: can you please put rt 41202 to pri 89 [05:41] lifeless: possibly, trying to login [05:41] where do shipit bugs get filed? [05:42] like 'we can't rollout until this is fixed' style shipit bugs [05:42] shipit [05:42] But they're normally fixed by LP... [05:42]