rick_hStevenK: howdy, how goes the convoy branch?00:18
StevenKrick_h: I've not worked on it. If this critical regression branch gets finished off, I'll toss it at ec2 today.00:24
rick_hStevenK: awesome, ok. wanted to check. I've got deryck starting to reivew my JS test branches so was curious00:25
rick_hquick! someone get wgrant some water before he passes out!00:28
wgrantI just discovered why Product:+index is so terrible.00:29
wgrantProductPackagesPortletView always calculates linkage suggestions, even if it's not going to show them.00:30
wgrantAnd that's an expensive FTI query.00:30
huwshimiIs there a way to merge pipes, or push changes back into the parent? I accidentally created a pipe when I had uncommitted changes and now want these changes to be in the parent pipe.00:31
wgranthuwshimi: They're just normal branches with handy aliases.00:32
wgrantYou should be able to say 'bzr merge :next'00:32
wgrantThat sort of thing.00:32
huwshimiwgrant: Ah awesome, that makes sense00:33
* wallyworld punches wireless keyboard and mouse that just died00:36
rick_hwallyworld: wired > *00:37
wallyworldi think it's the receiver. bollocks00:37
* StevenK loves his wireless keyboard and mouse00:38
wallyworldmine is a logitech which should be better than a no-name brand you would think00:38
StevenKI think they're due for replacement. I've played enough first person shooters that the labels for WASD are either completly or just about worn off00:39
wgrantThat's why real keyboards don't have labels :)00:39
wgrantWell, that and Dvorak.00:39
wallyworldi've never tried a Dvorak keyboard00:39
rick_hvim + dvorak scares me too much00:40
rick_hthough I do love my buckling spring unicomp :)00:40
StevenKwallyworld: I do hate that the two major choices for keyboard/mouse are Microsoft and Logitech00:41
rick_hhttp://elitekeyboards.com/ http://www.pckeyboard.com/00:41
wallyworldyeah. i've have to buy another one today. can't not have one :-(00:41
rick_hthere's your real keyboards00:42
StevenKrick_h: How much do you want to bet they don't ship to .au?00:42
wallyworldwow. the pckeyboard one even has a clitoris00:42
rick_hwallyworld: yea, but no middle scroll mouse button :( guess it's a trademark thingy they can't do00:43
* wgrant hates non-mechanical switches now.00:43
rick_hStevenK: ah, ok got me there.00:43
rick_hwgrant: you tried a topre yet? I can't get my wallet out to go that route00:43
wgrantrick_h: No, but they intrigue me.00:43
* StevenK prefers keyboards that are quiet00:44
wgrantBut they feel terrible :/00:44
* wallyworld prefers keyboard that work00:44
StevenKwallyworld: You seem to be typing okay.00:44
rick_hwgrant: do they? it's one of the few switches I've used00:44
rick_hStevenK: check out the cherry brown00:44
wallyworldStevenK: on my laptop keyboard00:44
rick_hthose are pretty quiet switches00:44
wallyworldi'll have to undock for today00:45
StevenKGoogling for Cherry Brown isn't so helpful00:45
wallyworldand whenever i undock, it kernel panics00:45
wgrantrick_h: I was replying to StevenK's quiet keyboards thing.00:45
StevenKwallyworld: Handy.00:45
wallyworldwell here goes.....00:45
wgrantI believe mine has Cherry Browns.00:46
wgrantThey're certainly not quiet.00:46
rick_hwgrant: oic00:46
wgrantBut they're a bit quieter than Blues :)00:46
StevenKThat sounds like a panic00:46
rick_hwgrant: yea, I've got two of the leopolds, one in brown and one in blue00:46
StevenKwgrant: So you prefer keyboards that sound like a typewriter factory?00:46
rick_hfrom the elite keyboards place00:46
wgrantStevenK: Yes, because they are so nice to type on.00:46
rick_hwgrant: ++00:47
wgrantI used to prefer my ThinkPad keyboard to every other keyboard I'd tried.00:47
wgrantBut now it feels terrible compared to this :/00:47
StevenKrick_h: $54USD to ship a keyboard from elitekeyboards00:48
rick_hStevenK: ouch00:48
wgrantpccasegear has some reasonable keyboards.00:49
wgrantThey're Australian.00:49
StevenKAll four of them are corded :-)00:50
rick_hcool, the leopolds are two that I have used. wgrant you used cherry reds? another one I'm interseted in00:50
wgrantWhy would you want a wireless keyboard :/00:51
wgrantYou have to have wires to your monitor, and your keyboard is right in front of your monitor...00:51
wgrantrick_h: I just have a Das Keyboard Ultimate S00:51
wgrantWhich has Browns AFAIK00:51
rick_hwgrant: ah, gotcha00:51
wgrantI've never tried reds.00:52
rick_hheh, tonight at the local coders meet up is keyboard night. Brought out the leopold blue, brown, happy hacker, and unicomp00:52
rick_hI need help...00:52
StevenKwgrant: My keyboard isn't00:52
StevenKI have a keyboard drawer00:52
wgrantI used an actual Model M for a while.00:53
* StevenK wonders why calling this view is returning ""00:53
rick_hwgrant: my unicorn is one of those split model M's. Like a model M and a MS Natural had babies, but they for for like $1k on ebay :/00:54
lifelessis bug 732510 f-r really?00:54
_mup_Bug #732510: poppy-sftp should connect to the database with a unique user ID <canonical-losa-lp> <dbuser> <poppy> <qa-ok> <tech-debt> <trivial> <Launchpad itself:Fix Committed by stub> < https://launchpad.net/bugs/732510 >00:54
StevenKwallyworld_: That was a nice panic?00:58
* StevenK puts together a deployment00:58
wallyworld_StevenK: i went and got a drink before i rebooted00:59
wgrantlifeless: No01:11
wgrantlifeless: Needs downtime.01:11
lifelesswgrant: paramses?!01:21
wgrantlifeless: Well, it's a sequence of sets of parameters :(01:23
wgrantYes, it's terrible, but params is already used for the singular.01:23
lifelessrules, searches, clauses, conditions, param_sets, ...01:35
lifelessI dunno, but accepting it as terrible seems undesirable :)01:35
wgrantlifeless: It's a sequence of BugTaskSearchParams objects.01:36
lifelesswhich all get unioned together01:37
lifelessnot anded01:37
lifelessthis is worth calling out perhaps01:37
lifeless-> alternatives01:37
lifeless-> alternate_parameters01:37
wgrantalternatives it is01:40
lifelesswgrant: bug 92924101:46
_mup_Bug #929241: ProductPackagesPortletView calculates suggestions when it knows it won't show them <Launchpad itself:In Progress by wgrant> < https://launchpad.net/bugs/929241 >01:46
lifelesswgrant: why can't that just be the first line of setUpFields ?01:46
wgrantlifeless: Because then setUpWidgets complains that there aren't any fields.01:54
lifelessah. thanks01:55
wgrantIf I avoid setUpFields, I have to also avoid the rest of the form code.01:55
StevenKWidgetInputError('name', u'Name', LaunchpadValidationError(u'unique-from-test-team-py-line529-100007 is already in use by another person or team.'))02:02
StevenKI call shenanigans02:02
ayanmaybe i was dreaming but i recall hearing about a C API for launchpad.02:12
ayanif it exists, where can i download it?02:12
wgrantI'm not aware of one.02:12
wallyworld_lifeless: the docs say any alter table patch cannot be applied live. is this true even for "alter table xxxx drop constraint...." patches?03:09
StevenKDoesn't that take a full write lock on the table?03:10
wallyworld_no idea03:10
StevenKwallyworld_: It should still be fine to apply during FDT03:10
wallyworld_but i would have thought a drop would be virtually instant03:10
StevenKIt still needs a lock03:10
wallyworld_sure, but a very short one03:11
wallyworld_which hopefully we could tolerate?03:11
StevenKwallyworld_: Applied live refers to "Do not needs us to go down, can be done live"03:11
wgrantI think we should tolerate it.03:11
wgrantAlthough it will cause timeouts.03:11
wgrantAh, except slony.03:11
wgrantSo no, it requires a full lock across the cluster.03:11
StevenKWhich is FDT only03:12
wgrantIf we weren't using slony, I think it would be acceptable, as it would only cause things to time out for a few seconds.03:12
wallyworld_ok, will redo patch nr03:12
lifelesswallyworld_: yes, it is true.03:13
wallyworld_nr is an (i thought) common abbreviation for number03:13
wgrantwallyworld_: Right, but this doesn't involve a number change.03:13
wgrantThe docs in allocated.txt are out of date.03:13
wallyworld_wgrant: my patch nr ends in -103:13
lifelesswallyworld_: the reason it says cannot be applied live is because it cannot be applied live.03:13
wgrant'tis irrelevant nowadays.03:13
wallyworld_so do i make the patch nr end in -0?03:14
lifelesswallyworld_: in a busy cluster with e.g. backups, we might wait up to 24 hours to get a lock.03:14
wgrantwallyworld_: It no longer matters.03:14
lifelesswallyworld_: so even without slony, it would be unsafe to apply such a patch live.03:14
wgrantlifeless: It's safe if you are careful.03:14
wallyworld_ok, will mark the mp as needs review then03:14
lifelesswgrant: FSVO03:14
lifelesswgrant: its not trivially repeatedly safe03:14
* StevenK stops looking at this visibility branch for a bit, since it is making him very very frustrated.03:14
wgrantlifeless: It involves less downtime than fastdowntime (up to ~9s of timeouts, rather than 90s of outage)03:15
* wallyworld_ taps fingers waiting for diff03:15
lifelesswgrant: the goal of the process is to make it trivially and repeatedly safe to evolve the schema; having complex rules that only db gurus can get right is not a good way to achieve this.03:15
wgrantBut because of slony we can't do it.03:15
lifelesswgrant: more than 9 seconds, even without slony. not unless and until we have sorted a bunch of other stuff out.03:15
lifelesswgrant: (other stuff being e.g. backups - anywhere in the cluster. Cronscripts with locks. Etc.03:16
wgrantlifeless: Assuming that you are smart and don't apply when there are long transactions.03:16
lifeless(read locks)03:16
wgrantWhich is what I meant by "careful"03:16
lifelesswgrant: hard to automate -well- because you cannot tell what /will/ be long when you start, only what *is* long already.03:16
wgrantlifeless: Lies.03:16
wgrantcrontab disablement is handy.03:17
lifelesswgrant: e.g. the fdt kick everyone off logic would still be needed.03:17
wallyworld_hurry up diff!!03:17
wgrantYou set a global flag which tells longrunning jobs to GTFO03:17
wgrants/GTFO/not start/03:17
lifelesswgrant: I agree that in an ideal world it could be made moderately reliable, but I won't go further than that :003:17
wgrantWait for any stragglers to go away.03:17
wgrantApply patch, which waits for contending requests to go away03:17
lifelesswgrant: modulo bugs, modulo stale connections, modulo another sysadmin doing something ...03:18
lifelesswgrant: the patch during that period breaks the world, of course.03:18
lifelesswgrant: taking everything out as fdt does, ignoring cluster sync, we can get less than that 9 second downtime window, and do it reliably (ignoring cluster sync because this is a post-slony discussion)03:19
wgrantlifeless: It breaks things that use that table for up to $REQUEST_TIMEOUT.03:19
wgrantRather than breaking the entire application for $SLONY_TIME03:19
lifelesswgrant: and it also pauses all other incoming queries for that same window03:19
wgrantIsn't that03:19
wgrant"breaks things"?03:19
wgrantThat was my definition of "breaks things".03:20
wgrantOh, you mean it will hang threads and therefore cause queueing?03:20
lifelessknock on effects03:20
lifelessrather than 'LP is down, come back soon' folk see 'LP is slow with no notice'03:20
lifelessthis is undesirable03:20
wgrantSlow for 5 seconds is better than down ever.03:21
lifelessdebatable; there is no reason FDT can't be that fast *anyhow*, so ...03:21
lifelessgiven a less risky and highly repeatable process, with similar limits on performance, why would you want a more risky process?03:22
lifelesswgrant: are you working on that new loggerhead bug right now? should I delay releasing?03:23
wgrantlifeless: I'm 30s from pushing03:24
wgrantJust found another couple03:24
* wgrant stomps on the branch scanner's face03:28
StevenKThat's okay, it has another one03:28
wgrantwallyworld_: this ajax submit thing is very slow03:29
wgrantOr is it broken...03:29
wallyworld_wasn't slow for me before03:29
wgrantI'm leaning towards broken at this point.03:29
wgrantIt's the longpoll bug03:30
wgrantHad too many MPs open.03:30
lifelesscat just fell off the desk03:31
lifelessI have pillows behind my LCD03:31
wallyworld_wgrant: what's the longpoll bug?03:31
lifelessnice big one stacked two deep; both cats can lie up there03:31
wgrantwallyworld_: Web browsers suck03:31
* wallyworld_ hates cats03:31
wgrantwallyworld_: They try to avoid DoSing, by limiting the number of connections per hostname03:31
lifelessthe white one stretched out, arched back, and slid off the edge of the pillows, and the desk, and fell onto my minitower case :)03:32
wallyworld_ah :-)03:32
wgrantlifeless: https://code.launchpad.net/~wgrant/loggerhead/bug-929275/+merge/9219403:32
lifelessis that a request for a code review ?03:32
wgrants/request/fishing attempt/03:33
StevenK31from paste import httpexceptions03:33
StevenK32+from paste.httpexceptions import HTTPNotFound03:33
lifelesswgrant: so, I wonder, would a wsgi middleware to map NoSuchId and NoSuchFile -> 404 be a good idea, or hide to many legit issues03:34
wgrantlifeless: I don't think that's a good idea.03:34
lifelessright now, for instance, the cod eyou've added can mask some classes of file system error03:34
lifelessbecause they just trigger NoSuchFile03:35
wgrantI used NoSuchId and NoSuchRevision03:35
wgrantAren't those pretty specific?03:35
lifelessoh hmm,03:35
lifelessI misread slightly03:35
lifelessso yes, that should be pretty narrow03:35
lifelessanyhow, fine at first glance, I'll let StevenK who seems to be in a reviewing mood do it ;)03:36
wgrantStevenK: FIxed03:36
wgrantlifeless: I think that could mask legitimate bugs.03:36
wgrantAnd encouraging people to write terrible code is not what Loggerhead needs more of :)03:36
StevenKwgrant: r=me03:41
wgrantStevenK: Thanks.03:42
* wgrant lands.03:42
wgrantlifeless: Should I upgrade Launchpad's?03:47
wgrantOr do you have stuff planned?03:47
lifelesswgrant: just release stuff from toshio, so please go ahead03:52
StevenKWidgetInputError('name', u'Name', LaunchpadValidationError(u'unique-from-test-team-py-line529-100007 is already in use by another person or team.'))03:54
wgrantlifeless: the GIL contention between xmlrpc-private and app seems to be growing :(04:05
wgrantwe see lots of https://lp-oops.canonical.com/oops/?oopsid=OOPS-1e8d134f2de3a8101bf09a55c20c00ae now04:05
lifeless4 seconds getting feature flags?04:07
lifelessugh ugh ugh ugh ugh guh04:07
wgrantIt's not unexpected.04:07
lifelessdoesn't make it /nice/ :)04:08
lifelessso, we need to figure an appropriate ratio and then manually split the cluster, I guess04:08
lifelessI haven't seen anything suggesting haproxy knows how to count across two clusters yet04:08
wgrantI don't think so, no.04:08
lifelessthat, or we teach lp how to server both from the same port04:08
wgrantBut we can do the rebalancing entirely in haproxy04:09
wgrantwhich is nice04:09
wgrantI considered that, but that is scary.04:09
lifelessit could be a pgbouncer limit being hit04:10
lifelessfirst query04:10
wgrantHm, really?04:10
wgrantOh, xmlrpc, so no auth04:10
wgrantI've never seen that on a webapp request, though.04:11
lifelessin main servers we get the ff early too, to determine query timeout04:11
lifelessshould be the first query everywhere04:11
wgrantI don't think we get it before auth.04:11
* wgrant checks.04:11
lifelesswebapp threads are probably never idle04:11
lifelessfsvo idle04:12
wgrantSQL-main-slave SELECT getlocalnodeid()04:12
wgrantis the first for webapp04:12
lifelessor more importantly, the number of webapp errors probably dwarfs the number that would match this...04:12
lifelesswhich is on -slave04:12
wgrantMmm, true.04:12
lifelessso, two theories; whats the cheapest thing to do04:12
wgrantGraph pgbouncer connections, which UI has been trying to do this week? :)04:13
wgrantAt least that's what #webops has looked like.04:13
wgrantI've only glanced, though.04:13
lifelessas a starting point that makes sense04:13
lifelessI wonder if pgbouncer logs enough that we can short circuit that04:13
wgrantI'll check when these started04:13
wgrantBecause we hit the FD limit last week04:13
wgrantwhen we moved session to go through pgbouncer.04:14
lifelesse.g. we've just added X appservers (have we?) -> more connections used04:14
wgrantNo, they've not been added yet04:14
lifelessAlso someone needs to add the new prefixes to oops-tools once they are hactive04:14
wgrantAt least I hope not04:14
wgrantSince there's still queue depth issues04:14
StevenKLast I heard we were still waiting for more RAM and the extra CPU in one appserver04:17
wgrantHMmm, you may be right.04:17
wgrantOn the pgbouncer thing04:17
wgrantUnfortunately I started pruning properly again yesterday, so we don't have full records.04:17
wgrantBut I can't see too many similar OOPSes before the 31st.04:17
lifelessI think we should investigate pgbouncer04:26
lifeless4 seconds in the GIL is pretty extreme, we were seeing that when we had 20 threads running on private-xmlrpc full-tilt, or something crazy like it04:26
lifelessits more plausible to me that this is a queued connection in pgbouncer04:27
lifelessat least on the data we have04:27
lifelessgrrr, wtf, quoting *everything*. sob.04:32
wgrantWhat's quoting everything?04:35
lifelessshouldCompileTo("{{awesome}}", {awesome: "&\"'`\\<>"}, '&amp;&quot;&#x27;&#x60;\\&lt;&gt;'04:35
lifelessits quoting " and ' `04:35
lifelesssome might say this is overkill04:36
wgrant"' need quoting04:36
wgrant` not really04:36
lifelesswhy does ' need quoting in a template expansion ?04:36
wgrantlifeless: Attributes04:45
wgrantSome misguided folk use ' to delimit XML attributes.04:45
wgrantIt's perfectly valid, if a little discouraged and quite rare.04:45
poolieis it a regression, or a known problem, that attaching a patch to a bug no longer shows up in the comment timeline?05:01
poolieoh well, https://bugs.launchpad.net/launchpad/+bug/92931305:03
_mup_Bug #929313: patches aren't mentioned in the bug comment where they were attached <Launchpad itself:New> < https://launchpad.net/bugs/929313 >05:04
lifelessI'm not sure it is either05:05
StevenKwgrant: Can haz help? With http://pastebin.ubuntu.com/834814/ and running the new feature flag test I get WidgetInputError('name', u'Name', LaunchpadValidationError(u'unique-from-test-team-py-line529-100007 is already in use by another person or team.'))05:07
wgrantStevenK: Looking05:09
wgrantStevenK: I'm not sure that's a problem.05:14
wgrantStevenK: (Pdb) view.request.response._headers05:14
wgrant{'location': ['http://launchpad.dev/~unique-from-test-team-py-line529-100007']}05:14
wgrantIt created it anyway, and redirected.05:14
wgrantit's possible that you're not meant to render the form in that case.05:14
StevenKSo I'm not supposed to call view()?05:18
lifelessIIRC redirects have no body -> no need to render ever05:18
StevenKI'm just a bit unclear what I should do in that case.05:19
StevenKJust call c_i_v() and then check the team exists and is private?05:19
wgrantStevenK: I believe so.05:21
StevenKAssertionError: !=:05:22
StevenKreference = <DBItem PersonVisibility.PRIVATE, (30) Private>05:22
StevenKactual = <DBItem PersonVisibility.PUBLIC, (1) Public>05:22
wgrantStevenK: Also, lint.05:25
wgrantTrailing whitespace everywhere!05:25
StevenKIf everywhere is 3 places05:26
StevenKAnyway, fixed.05:26
wgrantIt's bright red.05:26
wgrantSo yes, everywhere.05:26
* StevenK isn't sure why visibility isn't set05:27
stubImportError: No module named pqm.pqm_submit06:45
stubAnyone know where that is supposed to be pulled in from? I don't see this bzr plugin in sourcecode.06:49
wgrantstub: The bzr-pqm package06:49
wgrantAre you on precise?06:50
stubWhich is installed06:51
wgrantPossibly 2.6 vs 2.706:52
stubMy system bzr is happy with the pqm plugin. Not sure how Launchpad's bzrlib is supposed to load the system plugin.06:53
stubCould be right about 2.7 - system bzr uses 2.7 and lp uses 2.606:54
stubSo how is the pqm plugin from the package supposed to get loaded when bzrlib.plugins is an egg?07:05
stubThe system path is shadowed07:06
wgrantbzrlib.plugins is magical.07:06
wgrantIt can pull stuff in from lots of places.07:06
wgrantI don't know how, though.07:06
wgrantstub: Will postgres plan through a view as if it wasn't there?07:15
stubIt gets expanded inline and the whole thing planned07:15
wgrantRight. Thanks.07:15
StevenKstub: You're not really concerned with say, planner changes between 8.4 and 9.1?07:21
stubStevenK: Usually things get better with a few regressions.07:23
stubwebops ping: Need the pqm pre-commit hook tuned07:36
stubOr maybe just removed is better no07:37
stubNeed 'Makefile' added to the list of exceptions along with *.sql and fti.py07:37
wgrantstub: Why?07:38
wgrantstub: We've never had problems with database/schema/Makefile changes accidentally landing on devel, so I don't see any point in restricting them.07:40
stubsorry... on the phoe07:50
stubHad to change Makefile rules for the newly labotimized fti.py invocation07:50
stublobotomised  even07:51
wgrantstub: *-0.sql and fti.py are the only things that are *forbidden*07:51
stubI think I should give up typing today07:51
wgrantEverything else is allowed.07:51
stubAhh... so my fti.py changes are the trigger07:51
stubwebops ping: Please remove the precommit hook then07:52
hloeungstub: where would I find this?08:00
hloeungah, think I've found it08:00
hloeungstub: is this for /home/pqm/archives/rocketfuel/launchpad/devel or /home/pqm/archives/rocketfuel/launchpad/db-devel or both?08:02
hloeungprecommit_hook=[ -z "$(bzr status -S database/schema/ | grep -e '\(-0\.sql\|/fti.py\)$')" ]08:05
hloeungthis one?08:05
stubhloeung: that is the one08:10
stubhloeung: It is no longer helpful, and a hindrance for my branch08:11
stublifeless: Do you really think it is worth keeping memcache around for just the blog info?08:14
hloeungstub: ok, commented out08:22
lifelessstub: well its also used as a cheap temp store for some migrations08:26
lifelessstub: I nearly wrote 'and move the blog to something else' before I remember that08:27
lifelessStevenK: I expect us to get fucked over in short order by pg 9.1, but the really traumatic stuff I think was in 8.408:27
lifelessStevenK: when a bunch of old implicit optimisations got derailed by CTE AFAICT08:28
mthaddonwgrant: I'm not sure I see the logic of "we've never had a problem, so let's remove the prevention of a problem"08:51
wgrantmthaddon: For some things, sure.08:53
mthaddoncan we change it rather than removing it altogether?08:54
wgrantmthaddon: The particular case I was arguing against was a suggestion to *add* a new restriction that would not have stopped any previous problems.08:54
wgrantBut in general I think that restriction does more harm than good.08:54
adeuringgood morning09:04
=== almaisan-away is now known as al-maisan
wgrantrick_h: JS on qastaging seems to be pretty broken.10:19
wgrantLPJS is not defined10:19
wgrantrick_h: At least on +register-merge10:20
=== matsubara-afk is now known as matsubara
rick_hwgrant: looking11:47
rick_hwgrant: @#$#@ see it. Putting up a mp right now. Very stupid11:52
StevenKrick_h: I got annoyed with the regression branch so much I pushed convoy to the PPA and put up a small MP for convoy.11:53
StevenKrick_h: I also tossed combo-url at ec2, which failed with codehosting failures.11:54
rick_hStevenK: or wgrant can you review: https://code.launchpad.net/~rharding/launchpad/graph_lpjs_928500/+merge/9225711:54
rick_hvery small stupid me change that's got qastaging broken JS11:54
rick_hStevenK: ok, thanks for the heads up. Want to shoot me the failed tests and I can peek at them today?11:55
StevenKrick_h: Forwarded. Not sure if it's my fault.12:05
=== rick_h changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rick_h* | Firefighting: - | Critical bugtasks: 4*10^2
rick_hadeuring: ping, can you peek at the MP please? https://code.launchpad.net/~rharding/launchpad/graph_lpjs_928500/+merge/9225712:23
adeuringrick_h: sure12:23
adeuringrick_h: r=me12:41
rick_hadeuring: thanks12:42
=== Ursinha` is now known as Ursinha
lifelessbah, whats that thing where you can't sleep?13:48
lifelesshah yes13:55
lifelesshey, so I've got the db hardware quote moving again13:55
lifelessshould have updated options soon and will rope flacoste into it when he is back on deck13:56
lifelessstub: herb is proposing SSD, perhaps with the same RAM we have today, or even a reduction. Your thoughts?13:56
stubSSDs are great if you pick the right hardware (and a disaster waiting to happen if you pick the wrong ones, such as the ones with volatile write caches on board). But not sure if they help our performance since we see little disk activity overall.13:57
=== jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rick_h*, jcsackett | Firefighting: - | Critical bugtasks: 4*10^2
stubNot sure what RAID setup we would need - need to scan the section in the PG performance tomb.14:00
stubMaybe a single RAID5 would be fine offsetting cost, or maybe we would still need two channels or an expensive RAID0+1 setup.14:00
lifelessstub: we do have a number of cold-cache bugs open14:00
lifelessstub: I know we see little absolute disk IO, but when we do its all concentrated AFAICT :)14:01
abentleygmb: I'm taking on branchscanner-timeout-bug-808930.  Can we chat?14:04
gmbabentley: Sure, can you give me a little while to get to a point where I can switch? Say 14:30 UTC?14:06
abentleygmb: That's stand-up time.  15:00?14:07
gmbabentley: Sure, 15:00 would be fine.14:07
abentleygmb: great.14:07
flacostegood night lifeless ;-)14:07
lifelessflacoste: heh I wish.14:10
lifelessflacoste: we finally got cynthia to sleep properly, fingers crossed. And now *I* have insomnia.14:11
flacostelifeless: that's so not unusual!14:11
lifelessstub: whats your feeling about RAM usage with SSD's? Could we keep it at todays levels, even with more appservers; or could we even reduce it?14:11
lifelessstub: and yeah, please do scan the PG performance tomb and get back to me14:12
lifelessflacoste: \i/14:12
stublifeless: I'd need to do research or benchmarking. Spinning plates of rust could end up faster if we can afford more RAM14:12
lifelessso, herb is going to get options on SSD w/128GB of RAM14:12
stublifeless: similarly, you are right that we might be able to get away with less RAM with SSDs (although SSD access is still slower than cache)14:13
lifelessI think we need to advise on the raid setup for those options?14:13
herbmy intention would be raid 1014:13
stubYes. There have been several threads on the mailing lists.14:13
lifelessherb: oh hai!14:13
herbeven with SSD. :)14:13
stubherb: over hardware raid 5? For paranoia or throughput reasons?14:14
* lifeless guesses at less rewrites-of-sectors14:14
lifeless(+ performance)14:14
herbstub: both14:14
lifelessherb: ok, since thats about as good as it gets, I don't think we need to ask you for a specific config :)14:15
stubherb: And one or two channels? We have two partitions, but that might be unnecessary since we are not queuing up writes for the right bit of rust to be in the right point in space/time.14:15
herbmy intention was a single channel SAS. even with SSD RAID 10 we won't be able to saturate the channel (nor could with with 16 or more spindles)14:16
lifelessstub: the controller would still have a limited tagging queue, which can still saturate; but I doubt we would have an issue given our current write load14:16
herbbut I'm happy to take input on that.14:17
stubelmo has the same book I havem but I'll trawl the mailing lists for the more resent threads. Only major thing I recall is many drives cheat by having write caches that fail for disaster recovery (including 'enterprise' devices)14:17
lifelessherb: we could add another channel later, right ?14:17
herbin any case, let's get some order of magnitude pricing. it might show that SSD is a non-starter and RAM is the way to go.14:18
stubI suspect it is unnecessary14:18
lifelessherb: if we can add another channel later, I think its totaly fine to start with one; SSD is -very- nice these days :)14:18
herblifeless: indeed we could.14:18
lifelessso, we all seem agreed - lets defer the question14:19
stubI think we should save money on RAM and buy me a coffee machine. Much better performance improvement.14:19
lifelessuse your frequent flyer points :)14:19
herbI find your claim to be suspect.14:19
lifelessherb: lol :) Also, I can point you at some hair raising queries/schema bits, which no amount of hardware can compensate for :)14:20
stubKLM, Air France, Emirates.... don't think I've earned a real FF point in ages...14:20
stubAnyway, off for dinner at 9:20pm because some of us have this non-sleeping thing from the other end.14:22
stubherb: Consider a combination of SSD and spinning metal. No point wasting SSD on logs and dumps.14:26
herbstub: good call. will do.14:27
deryckabentley, adeuring, rick_h -- https://plus.google.com/hangouts/extras/talk.google.com/orange-standup14:30
lifelessstub: herb: May I suggest the san rather than spinning metal for those offcuts ?14:31
lifelessprobably faster than spinning metal ...14:31
abentleygmb: mum-ble or han-gout?15:01
gmbabentley: Hangout works for me - besides, I can't guarantee that Mumble will. Bear with me a sec...15:01
* gmb waits for firefox to get its act together15:03
flacosteadeuring, deryck: are we done with bug 829074? or is there some other branches that need to land?15:13
_mup_Bug #829074: Show bugs that are not known to affect "official" upstream <bugs> <escalated> <qa-ok> <Launchpad itself:Fix Released by adeuring> < https://launchpad.net/bugs/829074 >15:13
deryckflacoste, on call with adeuring now.  explain shortly.15:15
flacostederyck: ok15:15
adeuringflacoste: there is a minimal fix in place: you can filter by upstream target, if a packaging link exists15:21
adeuringflacoste: i am working on a more comprehensive fix:15:22
adeuringunder the label "bug 2325"15:22
_mup_Bug #2325: Distro CVE report: permission denied! <lp-bugs> <Launchpad itself:Fix Released by kiko> < https://launchpad.net/bugs/2325 >15:22
adeuringthe idea is to porvide an option to select arbitrary upstream targets15:22
adeuringwell, not completely arbitrary, but any product or other source pachkage15:23
adeuringerm, i meant bug 23254515:23
_mup_Bug #232545: resolved_upstream list does not do product / source package matching <lp-bugs> <ubuntu-upstream-relations> <Launchpad itself:In Progress by adeuring> < https://launchpad.net/bugs/232545 >15:24
flacosteadeuring: is the comprehensive fix requested by the stakeholders also?15:24
jcsackettanyone else unable to use ec2 commands b/c of pqm issue? did i miss an email about something i need to change?15:24
adeuringthe bug report was a bit vague...15:24
lifelessjcsackett: I am, abentley is/has fixed it.15:24
jcsackettlifeless: ah fantastic. thanks.15:24
lifelessjcsackett: I haven't tried again since, so i don't know where teh fix has gotten up to.15:25
abentleylifeless: mur?15:25
lifelessabentley: the pqm-submit config stacks patch15:25
lifelessabentley: (IIRC the various bits correctly) - a few days ago.15:25
abentleylifeless: Yes, I've fixed the lp-land command.  Does that also fix ec2 land?15:26
lifelessabentley: oh, I thought it did/would :)15:26
lifelessabentley: IIRC the error in ec2 land happens when it invokes bzr lp-land to get the signed email for later delivery15:26
lifelessjcsackett: can you pastebin the transcript showing hte error?15:27
jcsackettlifeless: sure, one moment.15:27
flacosteadeuring: might be worth talking to brian and bryce about it15:28
abentleylifeless: Oh, I didn't think it shelled out to lp-land.15:28
lifelessit might be the thing rick has fixed15:28
flacosteadeuring: just to make sure we actually implement what they need15:28
jcsackettlifeless: http://pastebin.ubuntu.com/835337/15:28
lifelessin which case pulling trunk will fix it15:28
adeuringflacoste: ok15:28
lifelessjcsackett: ah, thats not what I saw a few days back15:29
abentleyjcsackett: That looks unrelated to the issue I fixed.15:29
jcsackettlifeless, abentley: well, darn.15:29
lifelessjcsackett: that looks like what stub experienced15:29
jcsackettlots of different issues cropping up here, huh? :-P15:29
lifelessjcsackett: system plugin path isn't on your bzr plugins path (is all I know)15:29
abentleyjcsackett: Possibly pqm-submit isn't installed on the box in question.15:29
lifelessjcsackett: stub filed a bug15:29
flacosteadeuring: from the launchpadlib pseudo-code that Bryce shows in the bug report, i think it was fine to make this work for linked_product only15:30
flacosteadeuring: best to validate with them directly15:30
lifelessjcsackett: look in latest-bugs for LP15:30
jcsackettpqm-submit is installed, abentley. i was hoping that was the issue too, since it was easily resolved. :-p15:30
adeuringflacoste: ok15:30
jcsackettlifeless: ok, thanks.15:30
jcsackettlifeless: yup, i'm experiencing the same thing. thanks for pointing me to the bug report.15:33
sinzuidanhg, I have 3 draft posts on blog.launchpad.net. Do you want to read them?15:41
deryckadeuring, abentley, rick_h -- I'm starting on the open questions now. for real. :)15:49
rick_hderyck: excellent15:49
deryckallenap or gmb -- if an external tracker is a Trac bug tracker, and we're syncing comments, shouldn't the status also update based on upstream bug?16:04
allenapderyck: Yes, it ought to be.16:04
deryckthere's no mapping required between trac and us, right?16:04
=== al-maisan is now known as almaisan-away
deryckah, not updated in awhile. resetting watch it is.16:08
deryckbigjools or jelmer -- I've got a question about failed recipe build I don't know how to answer. Can I assign to one of you, or you point me to who? :)16:13
rick_hderyck: link? jelmer was just helping someone in #lp and curious if it's the same16:14
deryckrick_h, https://answers.launchpad.net/launchpad/+question/18619316:14
rick_hderyck: ok, different one, nvm16:15
deryckrick_h, thanks anyway :)16:15
bigjoolsderyck: I really have NFI what's going on there.16:19
bigjoolsother than something wants more memory than it can get16:19
bigjoolsderyck: however, it's Java, so all bets are off, we've had this before with java builds16:20
jelmerderyck: that just sounds like the issue with java using too much memory on the buildds16:22
lifelessadeuring: that launchpadlib bug is a dupe of a server side bug, i forget the # though.16:22
lifelessadeuring: you may want to mark it as such to avoid duplicate analysis16:23
adeuringlifeless: right16:23
deryckbigjools, jelmer -- ok thanks guys.  I'll find the bug and point the user at it.16:23
lifelessadeuring: its almost certainly fallout from the materialised INCOMPLETE enums16:23
lifelessadeuring: (but you know that :P)16:23
jelmerbigjools: bug 693524 might be related16:23
_mup_Bug #693524: Daily builds of Java packages fail: "Could not reserve enough space for object heap" <recipe> <Launchpad Auto Build System:Triaged> < https://launchpad.net/bugs/693524 >16:23
adeuringyeah ;)16:23
=== fjlacoste is now known as flacoste
deryckabentley, adeuring, rick_h -- questions are all caught up.  answered or assigned them all.16:42
abentleyderyck: win!16:42
deryckadeuring, have you done interrupts today? ;)16:45
adeuringderyck: working on roject review16:45
deryckadeuring, awesome!16:45
deryckadeuring, I shall quit nagging you now. :)16:45
deryckadeuring, was about to ratchet it up to public internet shaming, but all is well now. ;)16:46
deryckI had alreading registered adeuringsucksatinterruptduties.com16:46
=== deryck is now known as deryck[lunch]
mabacrick_h, thanks for reviewing https://code.launchpad.net/~linaro-infrastructure/launchpad/workitems-model-classes/+merge/92174 . could you have a look at my update and let me know if I'm on the right track?17:33
rick_hmabac: will do, sec17:34
mabacrick_h, absolutely no rush. :) thanks!17:38
abentleylifeless: What's the best way to examine an OOPS that occurred on a dev instance (launchpad.dev) ?18:00
lifelessabentley: my preferred way is to glue oops-tools into the rabbit instance and have them come up in the normal web UI18:00
lifelessany oopses before you set that up will have been tossed by rabbit (as the exchange would have had no queue attached to it)18:01
lifelesson https://dev.launchpad.net/QA/OopsToolsSetup there is a 'deploying locally' section18:01
abentleylifeless: I have an oops in /var/tmp/codehosting.test/2012-02-0918:01
lifelessabentley: ah, interesting - must have been running without rabbit active18:02
lifelessin which csae the simplest thing is dump-bson18:02
lifelesssorry, bsondump18:02
lifelesswhich is in utilities/bsondump, or if you have a buildout of oops-amqp in bin/bsondump of it18:02
lifelesse.g. bsondump $path18:02
abentleylifeless: cool.18:03
abentleylifeless: utilities/bsondump says "No module named bson", but "bin/py utilities/bsondump" says "'module' object has no attribute 'decode_all'"18:04
lifelessyou can load that into the web UI if you want using datedir2amqp from python-oops-datedir2amqp18:04
lifelessabentley: argh, that blows.18:05
lifelessabentley: I know the one in oops-amqp works well18:05
lifelesserm, wrong project :<18:06
lifelessabentley: quick start:18:06
lifelessbranch python-oops-datedir-repo18:06
lifelesslink in the eggs and download-cache from your LP work area18:06
abentleylifeless: I installed python-bson with synaptic, and it works with that.18:07
lifeless./bootstrap ...18:07
lifelessabentley: ok cool18:07
=== deryck[lunch] is now known as deryck
rick_hmabac: hey, sorry, was going to ping you in irc but you stepped out18:17
rick_hmabac: replied to your MP, the model definition is different with stormbase18:17
mabacrick_h, sorry. I'm going in and out of sessions at Linaro Connect18:17
mabacrick_h, thanks. I'll check in a bit18:17
rick_hmabac: I hooked you up with an example in the blueprints directory there with you18:17
rick_hlet me know if that doesn't help18:18
mabacrick_h, awesome. thank you!18:22
rick_hmabac: thank you for working on it18:23
mabacmy pleasure :)18:23
abentleygmb: The transaction killer is not related to bug 808930 AFAICT.  The jobs are killed by the job-running infrastructure, because they run too long.  It's not length-of-transaction; it's length-of-job, which is tougher to optimize, because you'll be lucky if you only have O(n) complexity.19:09
_mup_Bug #808930: Timeout running branch scanner job <escalated> <linaro> <oops> <Launchpad itself:Triaged> < https://launchpad.net/bugs/808930 >19:09
abentleyAnd with O(n) complexity, it's always possible to time out by increasing n.19:10
bachi, abentley, i've been looking around and cannot figure out the story for nested branches with bzr.  is it in 2.5?  can you point me somewhere to find out about them?19:27
abentleybac: Nested trees?  Not implemented yet AFAIK.19:28
bacabentley: ok.  i'd heard they were19:28
abentleybac: You don't mean colocated branches, do you?19:28
bacnested, as in the equivalent of svn externs19:29
abentleybac: jelmer is planning to implement them, but he was nowhere near finished at the thunderdome.19:29
bacabentley: ok, thanks19:30
=== matsubara is now known as matsubara-afk
=== almaisan-away is now known as al-maisan
sinzuiwallyworld_, wgrant, jcsackett, StevenK, I think this summarises my day reading the voucer/commercial subscription code: http://people.canonical.com/~curtis/wtfpm.jpg22:04
maxbI shall have to show that picture to my colleagues tomorrow :-)22:08
StevenKsinzui: http://pastebin.ubuntu.com/835839/22:14
jcsackettsinzui, wallyworld: bug 929352 is what has hit me.22:17
_mup_Bug #929352: ec2 land unable to find pqm plugin <Launchpad itself:Confirmed> < https://launchpad.net/bugs/929352 >22:17
sinzuijcsackett, set the env before calling the script BZR_PLUGINS_AT=gtk@/path/to/your/plugins/pqm c22:19
jcsackettsinzui: thanks!22:19
sinzui^ that is how I test plugin changes in the branch I am hacking on22:19
sinzuijcsackett, set the env before calling the script BZR_PLUGINS_AT=pqm@/path/to/your/plugins/pqm c22:20
wgrantabentley: Around?22:21
abentleywgrant: yes.22:21
wgrantabentley: I think your bzr plugin loading changes break ec2 land22:22
wgrantIt can't load the pqm plugin22:22
abentleywgrant: You're talking about the site-customize changes?22:23
=== al-maisan is now known as almaisan-away
wgrantabentley: Yes. Just finding exactly what it is now.22:25
abentleywgrant: wouldn't have thought that ec2* ran Launpad in-process.22:26
wgrantabentley: bin/ec2 is a buildout script22:26
wgrantSo it uses lp_sitecustomize22:26
wgrantSo loading lp.codehosting breaks everything.22:33
wgrantabentley: Can we avoid that, or should I revert the whole rev?22:33
abentleywgrant: I don't think we should avoid that.  If we do, we get weirdness where LoomBranch isn't treated as a subclass of Branch.22:34
wgrantabentley: I suspect we want to rip that out of lp.codehosting, then.22:35
wgrantlp_sitecustomize is imported extremely early, and pulls in no other big bit of lP.22:35
wgrantJust a few isolated bits from lp.services.22:35
abentleywgrant: I don't know what you're saying would be ripped out of lp.codehosting.22:36
abentleywgrant: I'm sure the fact that it's loading bzr plugins is related to the fact that pqm-submit isn't loading.22:36
wgrantYeah, but separately I don't think lp_sitecustomize should be importing codehosting.22:37
abentleywgrant: Okay, but the main efffect of importing codehosting is initializing the plugins.22:38
wgrantabentley: True22:40
wgrantAnyway, I think we have little choice but to revert this.22:40
wgrantAs it breaks the dev toolchain.22:40
abentleywgrant: Doesn't break *my* dev toolchain, but I guess you should.22:41
wgrantabentley: Have you rerun bin/buildout?22:41
abentleywgrant: I don't use ec2.22:41
sinzuiwallyworld_, http://www.youtube.com/watch?v=YWMVVpOsjko22:59
sinzuiIts from 2005 I think22:59
wallyworld_sinzui: thanks, looking now :-)23:00
james_womg buildout is annoying23:02
james_w"no you can't have that"23:03
james_wError: Picked: distribute = 0.6.2423:03
james_wok, add to to versions.cfg23:03
james_wError: There is a version conflict.23:03
james_wWe already have: distribute 0.6.16dev-r023:03
james_wok, so why didn't you pick that one?23:04
james_wchange versions.cfg23:04
james_wError: Couldn't find a distribution for 'distribute==0.6.16dev-r0'.23:04
james_wbut you said you already had it!23:04
wgrantjames_w: That's what you get for daring to use a packaged Python.23:08
wgrantUpstream doesn't like distributions much.23:08
james_wwe're considering migrating to buildout though23:09
james_wbecause it's possible that other alternative suck more23:10
wgrantI think buildout sucks less than virtualenv(+pip)23:11
james_wI find that combination easier to use23:11
wgrantIn some respects.23:12
james_wbut I don't like the "oh, you have that version already installed? well I'm going to try installing it anyway? oh look, it's already in the download cache! well, I'll try downloading it anyway just to make sure"23:12
james_whttps://code.launchpad.net/~james-w/python-oops-amqp/bson-compat/+merge/92389 is what I was trying to do23:18
=== wallyworld changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld | Firefighting: - | Critical bugtasks: 4*10^2
mabacjames_w, hi. looks to me that you need to have a little cider break ;)23:47
mabacrick_h, thanks for being patient. I have pushed another change to lp:~linaro-infrastructure/launchpad/workitems-model-classes/ and hope that will work better23:48
james_wmabac, heh, yeah, heading out curling now :-)23:50
mabacjames_w, sounds dangerous ;) take care!23:51
james_wI will!23:52

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!