[00:04] erm [00:04] File "/var/launchpad/tmp/eggs/setuptools-0.6c11-py2.6.egg/setuptools/package_index.py", line 475, in fetch_distribution [00:04] AttributeError: 'NoneType' object has no attribute 'clone' [00:04] in devel ?! [00:04] Downloading file:///var/launchpad/test/download-cache/dist/setuptools-0.6c11-py2.6.egg [00:04] (on ec2) [00:22] Hi, all. [00:30] hiya [00:31] Anyone else have make blowing up over not having zc.buildout 1.5.1? [00:32] yes [00:32] just hit me on ec2 [00:32] as I was about to land the fix for Bug.attachments timing out all over the place [00:33] I can't figure out how to fix it. Rolled back bootstap.py to earlier version in the branch I'm in just to get hacking again. [00:33] deryck: while you are here [00:33] sure [00:33] do you know why BugAttachment.message doesn't know its own index ? [00:34] IIndexedMessage seems pretty expensive to generate [00:34] I don't. Meant to look into that when I saw your earlier email, and never got to it with things happening Thurs/Fri. [00:34] so I figure gary must have no landed 1.5.1 into the cache [00:35] i wonder if its on lp or pypi [00:35] deryck: I suspect messages can be shared between bugs [00:35] yeah, I wondered that too. But didn't know if he was rolling his own. [00:35] but there must be a BugMessage or something that does the actual join [00:35] and thus we could have an index in that [00:35] yeah, there is. [00:36] change bugattachment to go via BugMessage [00:36] and get the index from that [00:36] right [00:37] * deryck is looking at code.... [00:37] should I file a wishlist bug for this? I have Bug.attachments down to 23 queries - constant - [00:37] but, the way it does that is to use self.indexed_messages [00:39] deryck: ok, there is 1.5.1 on pypi.org, and it seems to be working. [00:39] deryck: I'm committing it to the download cache. [00:39] awesome! [00:39] something funny is going on [00:40] because versions.cfg still says 1.5.0 for buildout [00:40] lifeless, yeah, file a bug on that. I'll see if I can get work on that for my performance Tuesday work. [00:40] * deryck plans to do performance Tuesday this Tuesday again [00:40] deryck: I'd leave it for a little while, my stop gap should get us through to the 5sec 99% barrier [00:40] ah, ok [00:40] cool then [00:40] \o/ about perf on tuesday though [00:41] there are plenty of other timeouts in bugs code [00:41] mwhudson: well, thats interesting. [00:41] mwhudson: still, committed. we'll see what happens. [00:41] oh [00:42] hmm [00:42] bootstrap.py must be connecting to the internet to find out that 1.5.1 exists [00:42] this would explain why codebrowse restarts were taking 5 minutes [00:42] deryck: https://bugs.edge.launchpad.net/malone/+bugs?field.tag=timeout [00:42] lifeless, yeah, we're not getting to put the attention on timeouts that I had hoped. But after this cycle, I'll make sure we're getting the work in. [00:43] deryck: Oh, I wasn't meaning to criticise; I was offering that as a palette [00:43] yes, I appreciate the tag list! I didn't take it as criticism. Was just saying we haven't put the attention to the bugs as I had hoped to. [00:44] i bet this affects edge rollouts too [00:44] https://bugs.edge.launchpad.net/malone/+bug/121363 looks pretty shallow [00:44] <_mup_> Bug #121363: Order 'most recently closed' on 'Bugtask.id DESC' instead of 'BugTask.id' [00:44] spm: hi, if you have time, can you tell if the last edge rollout took longer than you'd have expected? [00:44] mwhudson: actually it's looking like it brokee rather nicely. I'm still chasing. [00:45] spm: nice [00:51] it probably broke due to this issue [00:51] which I have just addressed [00:51] in a fairly shallow way [00:53] However. it still shouldn't be looking anywhere else... What's changed lately? Just buildout 1.5.0? [00:53] bootstrap.py [00:53] i guess this is one for that operatic team lead person [01:13] spm: also - bzr: ERROR: Could not acquire lock "LockDir(file:///home/warthogs/archives/rocketfuel-built/launchpad/.bzr/branch/lock)": [01:13] spm: is really annoying me :) [01:14] ./ignore lifeless [01:14] gah. no wait wrong ignore. [01:28] spm: hahaonlyjoking? [01:57] lifeless: I can't see your comments anymore. so no idea if you asked if I'm joking or not. maybe. [01:58] spm: :P [01:58] lifeless: This is what I'm doing now - http://staging.profarius.com/ [01:59] MTecknology: so the artwork for launchpad is not open [01:59] It just popped into my head that we're going that and I realized I'd better check it out.. [02:00] s/going/doing/ [02:00] I'm just looking for the reference [02:00] ah yes, LICENSE in the tree: [02:00] The image and icon files in Launchpad are copyright Canonical, and [02:00] unlike the source code they are not licensed under the AGPLv3. [02:00] Canonical grants you the right to use them for testing and development [02:00] purposes only, but not to use them in production (commercially or [02:00] non-commercially). [02:00] [02:01] The Launchpad name and logo are trademarks of Canonical, and may not [02:01] be used without the prior written permission of Canonical. [02:01] So I should definitely remove it.. [02:01] I would think (but IANAL) that if you are using the LP logo specifically to refer to the launchpad.net instance then you're on the right side of trademark law. [02:01] spiv: Copyright-wise that's less clear, though. [02:01] wgrant: indeed [02:02] spiv: trademark + copyright are needed though [02:02] can't login.ubuntu.com be used instead of LP? [02:02] MTecknology: is it general openid you want to refer to [02:02] wgrant: has there ever been anything clear about legal issues? [02:02] MTecknology: or lp specifically ? [02:02] MTecknology: Frequently. [02:02] it uses launchpad teams [02:02] But not when it comes to LP. [02:02] I mean, the tree isn't even distributable... [02:03] MTecknology: so, I suggest you email.. damn folk are on leave. Thumper. Email him. [02:03] will do [02:03] ask for the ok. He'll bounce to jml, jml is awl for 2 weeks. [02:03] so it will queue there [02:03] and then get bounced to legal. :P [02:03] thumper: ^ :P [02:03] :p [02:03] I'll make a nice pretty email for him to read :) [02:04] I never enjoy legal stuff - I kinda wish the whole world was just under the BSD license :P [02:05] that has some appeal [02:05] I don't think society as is could handle that though :( [02:06] I've kinda been fighting the same with with the light-drupal-theme - I want it made for wide distribution - but I need to avoid license and trademark issues [02:09] MTecknology: I would use Ubuntu SSO for that. [02:10] It's more correct, and the trademark and copyright issues are clearer. [02:10] hm? [02:10] wgrant: doesn't have lp team memberships [02:10] wgrant: for the theme - I mean images like this - http://s.ubuntu.ru/header.png [02:10] lifeless: Doesn't it? [02:10] wgrant: AFAIK, no. [02:10] and specific color schemes [02:10] lifeless: They're the same code and DB, but with a different theme. [02:17] lifeless, MTecknology: login.ubuntu.com does send team memberships. [02:17] I just checked. [02:17] cool [02:18] I dunno if its /meant/ to - its hardly separate from LP :P [02:18] It's meant to. [02:18] AFAIK nobody thought about how the split was meant to work... [02:18] Because it's still hanging off the LP DB, and probably will always have to. [02:19] wgrant: I wasn't part of that work, can't comment at all. [02:19] but its really a bit of a bastard child atm [02:20] lol [02:20] It tries to be separate from LP. [02:20] But everything revolves around it having knowledge of LP team memberships. [02:23] Is there any particular reason it was broken out? [02:23] sinzui: hi [02:23] MTecknology: to make it usable by U1 [02:23] hi lifeless [02:23] sinzui: I am planning on nagging spm about the maps CP [02:23] U1? [02:23] sinzui: he's dealing with falling buildings atm [02:23] I was just about to do that [02:23] MTecknology: ubuntuone [02:23] oh [02:24] lifeless, okay, that saves me pinging him [02:24] MTecknology: its an (important) branding exercise [02:24] but its not really split out ATM [02:35] MTecknology: isn't it more ubuntu single signon than a launchpad login? [02:35] thumper: This image has been there since before the ubuntu login [02:36] hmm [02:36] * thumper shrugs [02:37] lp:drupal-{openid,launchpad,teams} [03:02] spm: please ping when I can get another profile [03:02] testing bug 600000 [03:02] <_mup_> Bug #600000: missing dependency on Bazaar [03:05] lifeless: just kicking this imports thing for thumper; then you're next? [03:05] ta spm [03:12] spm: thanks [03:12] lifeless: oki; about to do the magic on staging. brb... [03:20] wow https://bugs.launchpad.net/ubuntu/+bug/1/+index is a mess [03:20] <_mup_> Bug #1: Microsoft has a majority market share

spm: so I waiting for you to say 'go' [03:20] lifeless: so am I. KaBoom. [03:21] staging appears to be having FireTruck issues [03:21] its going [03:21] oh [03:21] An error occurred when trying to install zc.buildout. Look above this message for any errors that were output by easy_install. [03:21] spm: big, red doesn't stop at lights? [03:21] make: *** [bin/buildout] Error 1 [03:21] ^^ not my idea of 'going' :-) [03:21] spm: >< - update the source code cache [03:21] spm: this'll be the 'connecting to the internet' problem maybe? [03:21] I put a new buildout in there this morning to work around it [03:21] ugh. it didn't STOP!!!!! ARGH! [03:22] spm: is it currently running with profilig on? [03:22] spm: cause I don't care if the code is old [03:22] somethings changed in the recent week or so that stops staging/edge from shutting down. *some* of the time. [03:22] lifeless: doubt it; it's been running since Aug 27. [03:22] grah [03:22] ok [03:22] ie. we'd have faceplanted on disk space by now. [03:22] * lifeless emotes kill -9 @ spm [03:23] I try the regular kill first. the think piece of silk wrapped around the 25 kg sledgehammer. if that doesn't work I take the slik off, and use the -9. [03:23] s/think/thin/ [03:24] right. silk comes off. [03:24] we used to have an incredibly aggressive. Try X, fail, try kill, fail, try kill -9 shutdown sequence; that seems to have been removed :-(((( [03:25] mwhudson: I don't think so: Link to http://pypi.python.org/simple/zc.buildout/ ***BLOCKED*** by --allow-hosts [03:26] spm: thats the thing mwhudson is referring to [03:26] well no, that sounds better [03:26] spm: you need to update the dist cache [03:26] the internet problem was a timeout; we had last week [03:26] that's being blocked at the application level, not by the firewall [03:26] https://pastebin.canonical.com/36438/ [03:27] spm: 11:05 < lifeless> File "/var/launchpad/tmp/eggs/setuptools-0.6c11-py2.6.egg/setuptools/package_index.py", line 475, in fetch_distribution [03:27] 11:05 < lifeless> AttributeError: 'NoneType' object has no attribute 'clone' [03:27] 11:05 < lifeless> Downloading file:///var/launchpad/test/download-cache/dist/setuptools-0.6c11-py2.6.egg [03:27] 11:05 < lifeless> in devel ?! [03:27] 11:05 < lifeless> (on ec2) [03:27] so I assume tehre's a patch floating around somewhere to ensure we have the lastest hotness? [03:27] spm: is what I was asking this morning. Its the same - AttributeError: 'NoneType' object has no attribute 'clone' - is the magic bit to note, where it has blown well up. [03:27] spm: yes, *update the dist cache*. [03:28] that sounds... broken somewhere. why is our automatic updates not getting this automatically? [03:28] something like cd lp-sourcedeps/download-cache; bzr update [03:28] i would like to understand why it's looking for 1.5.1 though [03:28] spm: appears to be a skew between bootstrap.py and versions.cfg [03:28] or something like that [03:29] gah. sorry - germanium is trying to eat several hundred Gb of disk asap. bbs... maybe. [03:30] whats germanium do [03:30] ppa [03:30] it's ppa.launchpad.net i think [03:31] looks like the leaky /tmp crap buglet we saw a week or so ago. G just has more / space. so took longer to manifest. [03:31] ah so a simple rm * [03:32] well.... o; but that's the idea. a little more targetted; but. [03:32] well *no*. typo win. [03:32] did you just rm -rf / ? Please say yes.... [03:32] I could try; but it won't work well :-) [03:34] doing a verify if /tmp is in fact te problem. it *looks* like it; but I want a little more verification before I start rm'ing away; then I'll prolly start with something like: find /tmp -maxdepth 1 -name 'tmp*' -type d -mtime +7 -delete [03:34] actually no. dir's; so -print0 | xargs -0 rm -rf [03:35] -type f [03:35] then -type d :P [03:35] ... | xargs -rn rm -r <== better :-) [03:36] -0r, argh [03:36] -exec { rm \$1 }; ? [03:36] actually, probably wants a -cut too [03:37] spm: Not my script again? :P [03:37] (killing germanium) [03:37] lifeless: for this many folders, -exec would be horrible. fork hell. [03:38] spm: clone ftw! [03:38] wgrant: "yes", even if not your fault; it now is. [03:38] spm: (actually its exec hell, the fork is cheap) [03:38] whatever :-) [03:39] ok. 15 mins of trying to get a du summary; I call /tmp/tmp* is the problem and move into rm mode [03:40] whee. some srsly old poppy uplods too. [03:40] mwhudson: its odd, bootstrap.py was last change in 11419 [03:40] on tuesday [03:41] oh oop, thats my branch [03:41] no, devel agrees [03:42] ah, 11452 changes versions.cfg [03:42] but that doesn't change... ah I wonder if lazr.restful 0.11.3 wants the newer zc/buildout ? [03:43] oh [03:43] maybe [03:43] or launchpadlib [03:43] that should lead to a conflict though i think [03:46] lifeless: (vaguely related aside) - did you ever submit a branch to the prod/staging configs that had the profiler in, but disabled? [03:48] spm: pretty sure [03:48] lifeless: It isn't just because the buildout 1.5.0 has new logic to update, and 1.5.1 was only just released? [03:48] Er, English fail there, but you get the picture. [03:48] wgrant: no, because this happens on machines that can't connect to the internet [03:49] mwhudson: And it hasn't been happening since the update? [03:49] to buildout? [03:49] no, we don't think so [03:49] Yeah. [03:49] :( [03:50] spm: ah, its in my branch lp:~lifeless/lp-production-configs/timeouts [03:50] lifeless: hrm. doesn't look like it. not an active proposal; and we've only had one other change in the past month. Can I pester/nag/trouble/bother you to get that sooner than later? ;-) [03:50] snap, ftw. [03:50] merge propose away! [03:51] spm: the other change was this branch too, I thought [03:52] https://code.edge.launchpad.net/~lifeless/lp-production-configs/timeouts/+merge/34042 [03:52] lifeless: no; the other recent change only has librarian oops changes. [03:52] ta [03:53] heya poolie [03:53] hi there spm! [03:53] spm: so, does this mean we're ready to profile ?:) [03:54] lifeless: ha you wish. staging is still very much firetrucked. still getting the ambulance round. [03:54] shall we call the cops, go for the trifecta [03:55] well, my awesomely wonderful (and delightfully sarcastic) wife is making me a ham sandwich... so........ [03:59] lifeless: doesn't look like anything depends on buildbot 1.5.1 so it's a mystery (tm) [04:01] lifeless: approved; pls submit whenever you're ready. [04:02] spm: I've no idea how to do that anymore; it pages out so quickly (and its nuts to use pqm for this) [04:02] spm: perhaps you could submit it ? [04:02] :-) [04:03] sure [04:03] thanks! [04:03] fwiw, we do via pqm; more as a gatekeeper than anything [04:04] I love pqm/tarmac systems (I should shouldn't I :P) but they come with a cost, for benefits. if the benefits aren't worth the cost, it's not useful to use it :) [04:04] spm: so, you're having lunch? Should I pop my head back in in 20 minutes or something for the staging profile ? [04:05] lifeless: not really... Ill be eating lunch shortly; not breaking; if you ken the difference. [04:05] och aye [04:05] too many eggs on the boil to afk yet [04:05] so staging - have you tried updating the download cache? [04:08] not yet; eed to unfudge edge 1st [04:08] hi lifeless (realize lp's eggs are on fire) did anything change with feature flags? [04:09] i may go back to it say wednesday [04:09] haven't read much mail ypet [04:09] poolie: sinzui has a patch that uses them to control google maps [04:09] which are currently stuffed due to a google side problem [04:09] sweet [04:09] * poolie spoke too soon [04:09] but i'm glad to see some takeup [04:09] it should get CP'd todayish :P [04:10] spm: edge almost certainly has the same problem. have you tried updating the download cache :P) [04:10] lifeless: that would emphatically be Doing It Wrong™ [04:10] manual intervention on every server? yuk. [04:11] spm: well, doesn't your deploy script do that ? [04:11] hrm. from the same state; the question is why is that state borked. [04:11] spm: kick off a new deploy, done :P [04:12] spm: so we don't know the root cause, but the package I put in the download cache about 4 hours ago seems to work. [04:13] ahh. I see. edge borked is the failed stop on edge4; it never updated to latest shiny. [04:13] right [04:14] I strongly suspect, if the pastebin you gave is what the error was, that kicking off a new one, which should as a matter of course grab the latest download cache, should work. [04:16] edge hasn't (yet) exhibited) that error... [04:16] has special errors all of it's very own [04:17] show n tell ? [04:18] per above; failed to stop and never got updated; so when I stabbed and started it; it came up as the wrong revno. [04:19] ah [04:19] we have a check for that you see :-) [04:25] spm: so what do you think is wrong? [04:28] lifeless: with edge in this case? it's a reversion on an old bug faict. have just logged bug#626577 [04:28] <_mup_> Bug #626577: app servers not shutting down (again) [04:29] actually - that's a moderately deadly one for continuous rollouts btw. if we need manual intervention to just stop/start servers; that project is pretty much dead before started. [04:30] we will fix [04:30] the more we do something the more we'll improve it [04:30] Oh i know that; was just saying :-) [04:31] still the root cause of that bug has never been fixed. what actually causes the daemons to lock like that and not shutdown correctly. [04:31] spm: did you get a per-thread stack trace? [04:32] way back when; probably. we spent a fair bit of effort ages ago on it. given we had a (hack) fix; it was deemed not worth the time and effort to keep chasing further. [04:33] argh. so staging full udpates died on a replication fail. [04:37] was it the new table again ? [04:37] spm: anyhow [04:38] spm: I don't care about updates, I just want running + profiling :) [04:38] spm: can we do that, and you fiddle with updates after ? [04:38] well, tbh, I'd really rather not. otherwise I'm just duplicating the same work and somewhat wasting time messing around with something that needs fixing at the root anyway [04:39] spm: ok [04:39] spm: I'll go do some other stuff and check back with you in 20-30 [04:40] I'd hazard at least part of an educated guess that the pqm spam is all realted to stagings borked ness [04:40] spm: the db-devel update is due to sodium crashing mid-update [04:40] its left stale bzr locks [04:40] at a WAG [04:41] yeah - but those files get auto synced around. so a partial update gets synced around.... [04:41] ouchies [04:41] right; and we have a dir that's missing stuff. so I call 1+1 = maybe 2 :-) [04:41] 1 + 1 = 1.8 ish? [04:41] closer to e [04:41] 2.1, you could average [05:03] spm: so how goes it ? [05:05] well we have successful rf builds again. [05:09] right. next regular staging code update is in ~ 5 mins. see how that goes as an automated thing. [05:11] :(( [05:12] * thumper wishes jelmer lived in NZ or AU [05:12] What do you have against the northern hemisphere? We've a few active timezones on this side of the globe. [05:13] (some east of some AU timezones) [05:13] persia: thats only because .au is silly-wide [05:15] * persia is reminded of Negativland's "Time Zones" [05:16] the name of that ci tool that's written in java *still* trips me up [05:17] In terms of increased highlight count? [05:18] no [05:18] just reading emails [05:19] "meet hudson" [05:19] Ah, so yes, but with a wider definition of "highlight" to be more a cognitive filter than any technical notifaction system :) [05:19] right [05:22] persia: it is more that I want jelmer around RIGHT NOW [05:23] I figured, but just felt you were unfairly discriminating against landmasses north of the equator. [05:25] * ajmitch thinks it's a fairly reasonable wish [05:27] thumper: what do you want from jelmer ? [05:28] lifeless: mwhudson: https://pastebin.canonical.com/36440/ <== that's getting zotted at source, so to speak. if the staging restore/build can't get those files; we have a bigger problem up front. [05:29] spm: its in the download cache now [05:29] spm: I don't know how staging updates work [05:30] but if they are not updating the download cache, its a fail. [05:30] if they , that file is available. [05:30] lifeless: I've emailed him about a git import failure [05:30] spm: can you please confirm that zc.buildout-1.5.1.tar.gz exists on disk on staging [05:36] oh joy [05:37] ec2 hates my branch, but it works locallyt. [05:37] what could it be? ... storm [05:38] lifeless: so. cm.py (rf built) runs on sodium. 'builds' the tree; that gets pulled on sourcherry, which every half hour does a staging 'restore'. pushes that where it needs to go. [05:39] spm: ok, so lets check on sodium [05:39] dbupdates are done once 1 week; superset of the normal code only updates [05:39] does it have a download-cache there [05:44] grah this is nuts [05:44] any advice on debugging a 'works locally, fails on ec2' issue ? [05:44] I've zapped my storm local bugfixes, running whatever the dist facility made [05:45] lifeless: how is it failing on ec2? [05:45] two ways [05:45] one [05:45] are we using lucid ec2 images yet? [05:45] my queries-are-constant test fails [05:45] the check for constant finds the second run did 4 less queries [05:46] the second thing is a check in bug.txt that something takes 2 queries fails - it takes 3 (which is probably appropriate) [05:46] File "lib/lp/bugs/tests/../doc/bug.txt", line 1133, in bug.txt [05:46] Failed example: [05:46] len(CursorWrapper.last_executed_sql) - queries [05:46] Differences (ndiff with -expected +actual): [05:46] - 2 [05:46] + 3 [05:46] lifeless: the only thing that leaps to mind is an isolation failure, some test is leaving something with a __del__ that does queries or something behind [05:47] but that seems a bit mental [05:47] so they do definitely both pass in isolation [05:47] lifeless: have you tried running not just your test but a few around that? [05:47] I guess I'll run a .. yes [05:47] sounds horrible though [05:47]