[00:18] StevenK: howdy, how goes the convoy branch? [00:24] rick_h: I've not worked on it. If this critical regression branch gets finished off, I'll toss it at ec2 today. [00:25] StevenK: awesome, ok. wanted to check. I've got deryck starting to reivew my JS test branches so was curious [00:27] Hahahawefwefw [00:28] quick! someone get wgrant some water before he passes out! [00:29] I just discovered why Product:+index is so terrible. [00:30] ProductPackagesPortletView always calculates linkage suggestions, even if it's not going to show them. [00:30] And that's an expensive FTI query. [00:31] Is there a way to merge pipes, or push changes back into the parent? I accidentally created a pipe when I had uncommitted changes and now want these changes to be in the parent pipe. [00:32] huwshimi: They're just normal branches with handy aliases. [00:32] You should be able to say 'bzr merge :next' [00:32] That sort of thing. [00:33] wgrant: Ah awesome, that makes sense [00:36] * wallyworld punches wireless keyboard and mouse that just died [00:37] wallyworld: wired > * [00:37] i think it's the receiver. bollocks [00:38] * StevenK loves his wireless keyboard and mouse [00:38] mine is a logitech which should be better than a no-name brand you would think [00:39] I think they're due for replacement. I've played enough first person shooters that the labels for WASD are either completly or just about worn off [00:39] hah [00:39] That's why real keyboards don't have labels :) [00:39] Well, that and Dvorak. [00:39] i've never tried a Dvorak keyboard [00:40] vim + dvorak scares me too much [00:40] though I do love my buckling spring unicomp :) [00:41] wallyworld: I do hate that the two major choices for keyboard/mouse are Microsoft and Logitech [00:41] bah [00:41] http://elitekeyboards.com/ http://www.pckeyboard.com/ [00:41] yeah. i've have to buy another one today. can't not have one :-( [00:42] there's your real keyboards [00:42] rick_h: How much do you want to bet they don't ship to .au? [00:42] wow. the pckeyboard one even has a clitoris [00:43] wallyworld: yea, but no middle scroll mouse button :( guess it's a trademark thingy they can't do [00:43] * wgrant hates non-mechanical switches now. [00:43] StevenK: ah, ok got me there. [00:43] wgrant: you tried a topre yet? I can't get my wallet out to go that route [00:43] rick_h: No, but they intrigue me. [00:44] * StevenK prefers keyboards that are quiet [00:44] But they feel terrible :/ [00:44] * wallyworld prefers keyboard that work [00:44] wallyworld: You seem to be typing okay. [00:44] wgrant: do they? it's one of the few switches I've used [00:44] StevenK: check out the cherry brown [00:44] StevenK: on my laptop keyboard [00:44] those are pretty quiet switches [00:45] i'll have to undock for today [00:45] Googling for Cherry Brown isn't so helpful [00:45] and whenever i undock, it kernel panics [00:45] rick_h: I was replying to StevenK's quiet keyboards thing. [00:45] wallyworld: Handy. [00:45] yeah [00:45] well here goes..... [00:46] I believe mine has Cherry Browns. [00:46] They're certainly not quiet. [00:46] wgrant: oic [00:46] But they're a bit quieter than Blues :) [00:46] Bwaha [00:46] That sounds like a panic [00:46] wgrant: yea, I've got two of the leopolds, one in brown and one in blue [00:46] wgrant: So you prefer keyboards that sound like a typewriter factory? [00:46] from the elite keyboards place [00:46] StevenK: Yes, because they are so nice to type on. [00:47] wgrant: ++ [00:47] I used to prefer my ThinkPad keyboard to every other keyboard I'd tried. [00:47] But now it feels terrible compared to this :/ [00:48] rick_h: $54USD to ship a keyboard from elitekeyboards [00:48] StevenK: ouch [00:49] pccasegear has some reasonable keyboards. [00:49] They're Australian. [00:50] http://www.pccasegear.com/index.php?main_page=index&cPath=113_1277 [00:50] All four of them are corded :-) [00:50] cool, the leopolds are two that I have used. wgrant you used cherry reds? another one I'm interseted in [00:51] Why would you want a wireless keyboard :/ [00:51] You have to have wires to your monitor, and your keyboard is right in front of your monitor... [00:51] rick_h: I just have a Das Keyboard Ultimate S [00:51] Which has Browns AFAIK [00:51] wgrant: ah, gotcha [00:52] I've never tried reds. [00:52] heh, tonight at the local coders meet up is keyboard night. Brought out the leopold blue, brown, happy hacker, and unicomp [00:52] I need help... [00:52] wgrant: My keyboard isn't [00:52] I have a keyboard drawer [00:53] I used an actual Model M for a while. [00:53] * StevenK wonders why calling this view is returning "" [00:54] wgrant: my unicorn is one of those split model M's. Like a model M and a MS Natural had babies, but they for for like $1k on ebay :/ [00:54] is bug 732510 f-r really? [00:54] <_mup_> Bug #732510: poppy-sftp should connect to the database with a unique user ID

< https://launchpad.net/bugs/732510 > [00:58] wallyworld_: That was a nice panic? [00:58] * StevenK puts together a deployment [00:59] StevenK: i went and got a drink before i rebooted [01:11] lifeless: No [01:11] lifeless: Needs downtime. [01:21] wgrant: paramses?! [01:23] lifeless: Well, it's a sequence of sets of parameters :( [01:23] Yes, it's terrible, but params is already used for the singular. [01:35] rules, searches, clauses, conditions, param_sets, ... [01:35] I dunno, but accepting it as terrible seems undesirable :) [01:36] lifeless: It's a sequence of BugTaskSearchParams objects. [01:36] yes [01:37] which all get unioned together [01:37] not anded [01:37] this is worth calling out perhaps [01:37] Yes [01:37] -> alternatives [01:37] -> alternate_parameters [01:40] alternatives it is [01:46] wgrant: bug 929241 [01:46] <_mup_> Bug #929241: ProductPackagesPortletView calculates suggestions when it knows it won't show them < https://launchpad.net/bugs/929241 > [01:46] wgrant: why can't that just be the first line of setUpFields ? [01:54] lifeless: Because then setUpWidgets complains that there aren't any fields. [01:55] ah. thanks [01:55] If I avoid setUpFields, I have to also avoid the rest of the form code. [01:57] yea [02:02] WidgetInputError('name', u'Name', LaunchpadValidationError(u'unique-from-test-team-py-line529-100007 is already in use by another person or team.')) [02:02] I call shenanigans [02:12] maybe i was dreaming but i recall hearing about a C API for launchpad. [02:12] if it exists, where can i download it? [02:12] I'm not aware of one. [03:09] lifeless: the docs say any alter table patch cannot be applied live. is this true even for "alter table xxxx drop constraint...." patches? [03:10] Doesn't that take a full write lock on the table? [03:10] no idea [03:10] wallyworld_: It should still be fine to apply during FDT [03:10] but i would have thought a drop would be virtually instant [03:10] It still needs a lock [03:11] sure, but a very short one [03:11] which hopefully we could tolerate? [03:11] wallyworld_: Applied live refers to "Do not needs us to go down, can be done live" [03:11] I think we should tolerate it. [03:11] Although it will cause timeouts. [03:11] Ah, except slony. [03:11] So no, it requires a full lock across the cluster. [03:12] Which is FDT only [03:12] If we weren't using slony, I think it would be acceptable, as it would only cause things to time out for a few seconds. [03:12] Yes. [03:12] ok, will redo patch nr [03:12] nr? [03:13] wallyworld_: yes, it is true. [03:13] nr is an (i thought) common abbreviation for number [03:13] wallyworld_: Right, but this doesn't involve a number change. [03:13] The docs in allocated.txt are out of date. [03:13] wgrant: my patch nr ends in -1 [03:13] wallyworld_: the reason it says cannot be applied live is because it cannot be applied live. [03:13] 'tis irrelevant nowadays. [03:14] so do i make the patch nr end in -0? [03:14] wallyworld_: in a busy cluster with e.g. backups, we might wait up to 24 hours to get a lock. [03:14] wallyworld_: It no longer matters. [03:14] wallyworld_: so even without slony, it would be unsafe to apply such a patch live. [03:14] lifeless: It's safe if you are careful. [03:14] ok, will mark the mp as needs review then [03:14] wgrant: FSVO [03:14] wgrant: its not trivially repeatedly safe [03:14] * StevenK stops looking at this visibility branch for a bit, since it is making him very very frustrated. [03:15] lifeless: It involves less downtime than fastdowntime (up to ~9s of timeouts, rather than 90s of outage) [03:15] * wallyworld_ taps fingers waiting for diff [03:15] wgrant: the goal of the process is to make it trivially and repeatedly safe to evolve the schema; having complex rules that only db gurus can get right is not a good way to achieve this. [03:15] But because of slony we can't do it. [03:15] Mmm [03:15] True [03:15] wgrant: more than 9 seconds, even without slony. not unless and until we have sorted a bunch of other stuff out. [03:16] wgrant: (other stuff being e.g. backups - anywhere in the cluster. Cronscripts with locks. Etc. [03:16] lifeless: Assuming that you are smart and don't apply when there are long transactions. [03:16] (read locks) [03:16] Which is what I meant by "careful" [03:16] wgrant: hard to automate -well- because you cannot tell what /will/ be long when you start, only what *is* long already. [03:16] lifeless: Lies. [03:17] crontab disablement is handy. [03:17] wgrant: e.g. the fdt kick everyone off logic would still be needed. [03:17] Nope. [03:17] hurry up diff!! [03:17] You set a global flag which tells longrunning jobs to GTFO [03:17] s/GTFO/not start/ [03:17] wgrant: I agree that in an ideal world it could be made moderately reliable, but I won't go further than that :0 [03:17] Wait for any stragglers to go away. [03:17] Apply patch, which waits for contending requests to go away [03:17] Done [03:18] wgrant: modulo bugs, modulo stale connections, modulo another sysadmin doing something ... [03:18] Yes. [03:18] wgrant: the patch during that period breaks the world, of course. [03:19] wgrant: taking everything out as fdt does, ignoring cluster sync, we can get less than that 9 second downtime window, and do it reliably (ignoring cluster sync because this is a post-slony discussion) [03:19] lifeless: It breaks things that use that table for up to $REQUEST_TIMEOUT. [03:19] Rather than breaking the entire application for $SLONY_TIME [03:19] wgrant: and it also pauses all other incoming queries for that same window [03:19] Isn't that [03:19] "breaks things"? [03:20] That was my definition of "breaks things". [03:20] Oh, you mean it will hang threads and therefore cause queueing? [03:20] True. [03:20] yes [03:20] knock on effects [03:20] rather than 'LP is down, come back soon' folk see 'LP is slow with no notice' [03:20] this is undesirable [03:21] Mmm [03:21] Slow for 5 seconds is better than down ever. [03:21] s/5/10/ [03:21] debatable; there is no reason FDT can't be that fast *anyhow*, so ... [03:22] given a less risky and highly repeatable process, with similar limits on performance, why would you want a more risky process? [03:23] wgrant: are you working on that new loggerhead bug right now? should I delay releasing? [03:24] lifeless: I'm 30s from pushing [03:24] Just found another couple [03:24] cool [03:28] * wgrant stomps on the branch scanner's face [03:28] Hard [03:28] That's okay, it has another one [03:29] Unfortunately. [03:29] wallyworld_: this ajax submit thing is very slow [03:29] Or is it broken... [03:29] wasn't slow for me before [03:29] I'm leaning towards broken at this point. [03:30] Oh [03:30] It's the longpoll bug [03:30] Had too many MPs open. [03:31] rotfl [03:31] cat just fell off the desk [03:31] meow [03:31] I have pillows behind my LCD [03:31] wgrant: what's the longpoll bug? [03:31] nice big one stacked two deep; both cats can lie up there [03:31] wallyworld_: Web browsers suck [03:31] * wallyworld_ hates cats [03:31] wallyworld_: They try to avoid DoSing, by limiting the number of connections per hostname [03:32] the white one stretched out, arched back, and slid off the edge of the pillows, and the desk, and fell onto my minitower case :) [03:32] ah :-) [03:32] lifeless: https://code.launchpad.net/~wgrant/loggerhead/bug-929275/+merge/92194 [03:32] Finally [03:32] is that a request for a code review ? [03:33] s/request/fishing attempt/ [03:33] 31 from paste import httpexceptions [03:33] 32 +from paste.httpexceptions import HTTPNotFound [03:33] Fail [03:34] bah [03:34] wgrant: so, I wonder, would a wsgi middleware to map NoSuchId and NoSuchFile -> 404 be a good idea, or hide to many legit issues [03:34] lifeless: I don't think that's a good idea. [03:34] right now, for instance, the cod eyou've added can mask some classes of file system error [03:35] because they just trigger NoSuchFile [03:35] I used NoSuchId and NoSuchRevision [03:35] Aren't those pretty specific? [03:35] oh hmm, [03:35] I misread slightly [03:35] so yes, that should be pretty narrow [03:36] anyhow, fine at first glance, I'll let StevenK who seems to be in a reviewing mood do it ;) [03:36] StevenK: FIxed [03:36] lifeless: I think that could mask legitimate bugs. [03:36] And encouraging people to write terrible code is not what Loggerhead needs more of :) [03:41] wgrant: r=me [03:42] StevenK: Thanks. [03:42] * wgrant lands. [03:47] lifeless: Should I upgrade Launchpad's? [03:47] Or do you have stuff planned? [03:52] wgrant: just release stuff from toshio, so please go ahead [03:52] k [03:54] WidgetInputError('name', u'Name', LaunchpadValidationError(u'unique-from-test-team-py-line529-100007 is already in use by another person or team.')) [03:54] LIES! [04:05] lifeless: the GIL contention between xmlrpc-private and app seems to be growing :( [04:05] we see lots of https://lp-oops.canonical.com/oops/?oopsid=OOPS-1e8d134f2de3a8101bf09a55c20c00ae now [04:07] 4 seconds getting feature flags? [04:07] Precisely. [04:07] ugh. [04:07] ugh ugh ugh ugh ugh guh [04:07] It's not unexpected. [04:08] doesn't make it /nice/ :) [04:08] so, we need to figure an appropriate ratio and then manually split the cluster, I guess [04:08] Yeah [04:08] I haven't seen anything suggesting haproxy knows how to count across two clusters yet [04:08] I don't think so, no. [04:08] that, or we teach lp how to server both from the same port [04:09] But we can do the rebalancing entirely in haproxy [04:09] which is nice [04:09] I considered that, but that is scary. [04:10] it could be a pgbouncer limit being hit [04:10] Unlikely [04:10] first query [04:10] Hm, really? [04:10] https://lp-oops.canonical.com/oops/?oopsid=OOPS-1e8d134f2de3a8101bf09a55c20c00ae#statementlog [04:10] Oh, xmlrpc, so no auth [04:11] I've never seen that on a webapp request, though. [04:11] in main servers we get the ff early too, to determine query timeout [04:11] should be the first query everywhere [04:11] I don't think we get it before auth. [04:11] * wgrant checks. [04:11] webapp threads are probably never idle [04:12] fsvo idle [04:12] SQL-main-slave SELECT getlocalnodeid() [04:12] is the first for webapp [04:12] or more importantly, the number of webapp errors probably dwarfs the number that would match this... [04:12] Right. [04:12] which is on -slave [04:12] Mmm, true. [04:12] so, two theories; whats the cheapest thing to do [04:13] Graph pgbouncer connections, which UI has been trying to do this week? :) [04:13] At least that's what #webops has looked like. [04:13] excellent [04:13] I've only glanced, though. [04:13] as a starting point that makes sense [04:13] I wonder if pgbouncer logs enough that we can short circuit that [04:13] I'll check when these started [04:13] Because we hit the FD limit last week [04:14] when we moved session to go through pgbouncer. [04:14] e.g. we've just added X appservers (have we?) -> more connections used [04:14] No, they've not been added yet [04:14] Also someone needs to add the new prefixes to oops-tools once they are hactive [04:14] At least I hope not [04:14] Since there's still queue depth issues [04:17] Last I heard we were still waiting for more RAM and the extra CPU in one appserver [04:17] HMmm, you may be right. [04:17] On the pgbouncer thing [04:17] Unfortunately I started pruning properly again yesterday, so we don't have full records. [04:17] But I can't see too many similar OOPSes before the 31st. [04:26] I think we should investigate pgbouncer [04:26] 4 seconds in the GIL is pretty extreme, we were seeing that when we had 20 threads running on private-xmlrpc full-tilt, or something crazy like it [04:27] its more plausible to me that this is a queued connection in pgbouncer [04:27] at least on the data we have [04:32] grrr, wtf, quoting *everything*. sob. [04:35] What's quoting everything? [04:35] handlebars [04:35] shouldCompileTo("{{awesome}}", {awesome: "&\"'`\\<>"}, '&"'`\\<>' [04:35] its quoting " and ' ` [04:36] some might say this is overkill [04:36] "' need quoting [04:36] ` not really [04:36] why does ' need quoting in a template expansion ? [04:45] lifeless: Attributes [04:45] Some misguided folk use ' to delimit XML attributes. [04:45] It's perfectly valid, if a little discouraged and quite rare. [04:52] nah [04:52] ah [05:01] hi [05:01] is it a regression, or a known problem, that attaching a patch to a bug no longer shows up in the comment timeline? [05:03] oh well, https://bugs.launchpad.net/launchpad/+bug/929313 [05:04] <_mup_> Bug #929313: patches aren't mentioned in the bug comment where they were attached < https://launchpad.net/bugs/929313 > [05:05] I'm not sure it is either [05:05] o/ [05:07] wgrant: Can haz help? With http://pastebin.ubuntu.com/834814/ and running the new feature flag test I get WidgetInputError('name', u'Name', LaunchpadValidationError(u'unique-from-test-team-py-line529-100007 is already in use by another person or team.')) [05:09] StevenK: Looking [05:14] StevenK: I'm not sure that's a problem. [05:14] StevenK: (Pdb) view.request.response._headers [05:14] {'location': ['http://launchpad.dev/~unique-from-test-team-py-line529-100007']} [05:14] It created it anyway, and redirected. [05:14] it's possible that you're not meant to render the form in that case. [05:18] So I'm not supposed to call view()? [05:18] IIRC redirects have no body -> no need to render ever [05:19] I'm just a bit unclear what I should do in that case. [05:19] Just call c_i_v() and then check the team exists and is private? [05:21] StevenK: I believe so. [05:22] AssertionError: !=: [05:22] reference = [05:22] actual = [05:22] :-( [05:25] StevenK: Also, lint. [05:25] Trailing whitespace everywhere! [05:26] If everywhere is 3 places [05:26] Anyway, fixed. [05:26] It's bright red. [05:26] So yes, everywhere. [05:27] * StevenK isn't sure why visibility isn't set [06:45] ImportError: No module named pqm.pqm_submit [06:49] Anyone know where that is supposed to be pulled in from? I don't see this bzr plugin in sourcecode. [06:49] stub: The bzr-pqm package [06:50] Ta [06:50] Are you on precise? [06:51] Which is installed [06:51] Oneiric [06:52] Hmm [06:52] Possibly 2.6 vs 2.7 [06:53] My system bzr is happy with the pqm plugin. Not sure how Launchpad's bzrlib is supposed to load the system plugin. [06:54] Could be right about 2.7 - system bzr uses 2.7 and lp uses 2.6 [07:05] Nah [07:05] So how is the pqm plugin from the package supposed to get loaded when bzrlib.plugins is an egg? [07:06] The system path is shadowed [07:06] bzrlib.plugins is magical. [07:06] It can pull stuff in from lots of places. [07:06] I don't know how, though. [07:15] stub: Will postgres plan through a view as if it wasn't there? [07:15] Yes [07:15] It gets expanded inline and the whole thing planned [07:15] Right. Thanks. [07:21] stub: You're not really concerned with say, planner changes between 8.4 and 9.1? [07:23] StevenK: Usually things get better with a few regressions. [07:36] webops ping: Need the pqm pre-commit hook tuned [07:37] Or maybe just removed is better no [07:37] w [07:37] Need 'Makefile' added to the list of exceptions along with *.sql and fti.py [07:38] stub: Why? [07:40] stub: We've never had problems with database/schema/Makefile changes accidentally landing on devel, so I don't see any point in restricting them. [07:50] sorry... on the phoe [07:50] Had to change Makefile rules for the newly labotimized fti.py invocation [07:51] lobotomised even [07:51] stub: *-0.sql and fti.py are the only things that are *forbidden* [07:51] I think I should give up typing today [07:51] Everything else is allowed. [07:51] Ahh... so my fti.py changes are the trigger [07:51] Yep [07:52] webops ping: Please remove the precommit hook then [07:52] Heh [08:00] stub: where would I find this? [08:00] ah, think I've found it [08:02] stub: is this for /home/pqm/archives/rocketfuel/launchpad/devel or /home/pqm/archives/rocketfuel/launchpad/db-devel or both? [08:05] precommit_hook=[ -z "$(bzr status -S database/schema/ | grep -e '$-0\.sql\|/fti.py$$')" ] [08:05] this one? [08:10] hloeung: that is the one [08:11] hloeung: It is no longer helpful, and a hindrance for my branch [08:14] lifeless: Do you really think it is worth keeping memcache around for just the blog info? [08:22] stub: ok, commented out [08:22] Ta [08:26] stub: well its also used as a cheap temp store for some migrations [08:27] stub: I nearly wrote 'and move the blog to something else' before I remember that [08:27] StevenK: I expect us to get fucked over in short order by pg 9.1, but the really traumatic stuff I think was in 8.4 [08:28]