lifelessI wonder if my with patch will be past buildbot in time for the deploy00:41
* lifeless has a cunning plan to add as many things as possible00:41
wgrantlifeless: Will we have an mthaddon in time?00:41
lifelessalso a good question00:42
wgrantI was assuming not, otherwise I would have been landing stuff more aggressively today.00:42
lifelessstaging takes 20 minutes00:42
lifelessif he shows up at ~8 as normal00:42
lifelessthen yes00:42
lifelessotherwise no00:42
wgrantHmm, the docs say the tree should be prepped 2.5 hours before :)00:43
lifelessand we have such a tree00:43
wgrantBut I guess you could try to get another pushed out, and just use the old one if it's not there in time.00:43
lifelessas long as we don't bork it we're fine00:43
lifelessthe 2.5 hours - if you are reading the losa docs - is time for 2 staging attempts + discussion00:43
wgrantstaging as in staging, not staging, right. I was wondering why you were talking about staging, and how you'd managed to get it to update in 20 minutes.00:44
* wgrant lunches.00:46
StevenKBut it isn't even midday yet?00:46
StevenKAnother day, another firefox update.00:47
thumperStevenK: it is past midday01:07
thumperwell past :)01:07
StevenKthumper: Timezone fail :-)01:08
wgrantStevenK: Shh.01:08
StevenKwgrant: Do you have dinner at 4pm, too?01:09
wgrantStevenK: No :(01:09
wgrantthumper: Is the EnumChoiceWidget suitable for the bugtask table too?01:11
thumperwgrant: it should be, perhaps with a little tweaking01:12
thumperwgrant: in the same way the InlineEditPickerWidget should be for selecting a person01:12
wgrantthumper: Why does one have InlineEdit and the other not?01:12
thumperwgrant: because I never got around to renaming it, and that is what it was originally called01:13
wgrantAh, good. Was hoping I wasn't missing some difference.01:13
thumperI'm just fixing a few tests on my blueprint-magic branch01:16
thumperwhich widgetizes the blueprint page as a proof of concept01:16
thumperor I should say01:16
thumperanother proof of use01:16
StevenKAwww. Here I was hoping it removed blueprints.01:16
thumperI like blueprints01:17
StevenKBlueprints are annoying01:17
thumperpersonally I think merging blueprints and bugs is wrong01:17
* thumper runs blueprint-magic through ec201:44
wgrantlifeless: what benefit does colocation provide us?02:46
lifelesswgrant: we need to support the protocol: more metadata, streaming fetch of N branches at onces, multiple heads etc02:55
lifelesswgrant: + [possibly] get rid of stacking and massively simplify things02:55
wgrantI guess.02:55
StevenKGet rid of stacking?02:56
lifelesswgrant: think about ideal loom behaviour02:57
lifelesswgrant: pushing 200 vim patches == pain; pushing 1 collection of branches == nice02:57
lifelessStevenK: stacking is the source of many bugs and slowdowns in bzr02:58
StevenKI like it for LP development03:00
lifelessStevenK: you like the performance03:03
lifelessStevenK: if it was faster than it is now, would you really whine?03:04
StevenKlifeless: TBH, with stacking I don't mind bzr push performance, and I'm happy about the disk space win for crowberry. If it was faster without losing the win for crowberry, that would be awesome.03:05
lifelessit has the potential to be smaller03:08
lifelesswe don't delete branches03:08
lifelessand the minimum size for a stacked branch is the size of one inventory - which doesn't compress well03:09
lifelessif all those branches were combined, the incremental overhead per branch could be a lot lower03:09
lifelessthe question is whether the baseline overhead would be more or less03:09
StevenKBut we can't do that today, right?03:09
lifelessits a nontrivial discussion03:10
lifelessand we have other fish to fry03:10
wgrantlifeless: Why does it need a full inventory? Because it starts a new compression group?03:11
wgrant(my knowledge of 2a and above is sorely lacking)03:12
StevenKAnd above?03:12
StevenKThere's another format after 2a?03:12
wgrantdevelopment-subtree, for one. But it's not exactly very different.03:13
lifelesswgrant: it has to be able to generate a delta03:19
lifelesswgrant: for anything in it03:19
StevenKSigh. Minutes after I say I'm happy with push performance, I'm stuck waiting for it.03:19
wgrantlifeless: And it can't use just a delta on top of the stacked-on CHK tree?03:20
wgrantI should probably read how CHK actually works :)03:20
lifelesswgrant: fetch operations are single repo always03:20
lifelesswgrant: consider: client A, servers B and C with a firewall between B and C03:20
lifelesswgrant: if sftp to B and C worked but bzr+ssh didn't it would be unpleasant03:21
lifelesswgrant: so the way we did it is to say that a repository must:03:21
lifeless - for any rev R it has:03:21
lifeless   - be able to return the content of the texts in R [in a repo specific format - e.g. fulltext, delta against some ancestor, whatever]03:22
lifeless   - be able to describe the content of R as a delta against the immediate ancestors of R on all sides03:22
lifeless - on pushes the server says "I am missing the parents of revisions X,Y,Z"03:23
wgrantOh, right.03:24
lifelesswe could, in theory, have a partial CHK tree for a given rev03:25
lifelessso far we haven't implemented that03:25
lifelesshmm, new timeout03:25
wgrantWhat's the bad query?03:26
lifelessdunno yet03:26
wgrantAh, there.03:27
StevenKwgrant: https://code.launchpad.net/~stevenk/launchpad/derive-common-ancestor/+merge/52796 -- given it's our work, I'm not asking for a reviewer, but look it over?03:27
wgrantlifeless: Ouch, 9s in 3 repeated queries.03:28
wgrantRather one triplicated query.03:28
wgrantIt seems to be exactly the same query.03:29
wgrant(looking at https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1894B1679, queries 20, 40 and 41)03:29
lifelessand there is AND SourcePackagePublishingHistory.status IN (2)03:30
lifelesson q 1903:30
wgrantSo there is.03:30
lifelessso maybe 4 calls03:30
wgrantI would normally say that two of them are probably TAL guarding the display of the third... but these are full queries.03:31
wgrantMight try to get tracebacks from DF.03:31
wgrantOr go TAL-diving, but that is more tedious.03:31
lifelessI'm doing some houseworky stuff atm03:32
lifelessbut if you wanted to identify the call sites, that would be awesome03:32
wgrantSure, just trying not to collide with you.03:32
StevenKWe can't get full tracebacks from qas?03:33
wgrantStevenK: I have a patch to give OOPSes a traceback for every query,.03:33
wgrantBut it makes them unparseable, so it's not really usable on qas.03:33
wgrantSo I pretend that Julian doesn't exist and use it on mawson.03:34
* StevenK ponders ringing Subaru03:34
StevenKThey've had my car for 5 and a half hours now. Surely they're done servicing it.03:35
StevenKwgrant: Can haz opinion on MP? Or is it on your list?03:36
wgrantStevenK: Looking.03:36
wgrantmawson will take about forever to update.03:37
StevenKJust one or two eons03:37
wgrant+                'derived': dervied_changelog,03:37
StevenKAll instances fixed, thanks.03:38
wgrantYou want to factory out the madness into something like get_ancestry(SPR)03:38
wgrantOtherwise that looks good.03:39
StevenKI do? I figured _updateBaseVersion() was self-contained enough.03:39
wgrantYou're duplicating the set(Changelog(spr.changelog.read()).versions)03:40
wgrantAlso, what does debian.changelog do if the changelog is unparsable?03:41
StevenKReturns an empty list03:42
StevenKWhich is fine by me03:42
wgrantIt's not going to raise exceptions in any case?03:43
StevenKwgrant: I'm happy to write a test for that case.03:43
wgrantThat would be handy.03:43
StevenKwgrant: get_ancestry in DSD or SPR?03:44
wgrantStevenK: SPR is big enough already.03:44
wgrantKeep it in DSD until we need it elsewhere, I think.03:44
StevenKwgrant: http://pastebin.ubuntu.com/578173/03:54
wgrantStevenK: Looks reasonable.03:55
StevenKNo manual entry for subunit-stats03:58
StevenKSubunit has to be the most poorly documented set of scripts ever03:59
lifelessStevenK: I give you perl04:04
lifelessStevenK: seriously, subunit-stats --help.04:04
StevenKhelp2man, kthxbye04:04
lifelesspatches appreciated kthxdeal04:04
wgrantlifeless: So, I've found the sources of the queries.04:05
wgrantlifeless: They are very fast on DF when caches are hot.04:06
wgrantWell, not very fast, but <200ms.04:06
StevenKBut very fast on DF is still 2 seconds.04:06
wgrantlifeless: SourcePackage.summary and SourcePackage.published_by_pocket. A couple of calls to each.04:07
wgrantCan you get a plan from a staging?04:07
wgrantlifeless: Thanks for the review.04:09
wgrantI hope next week that the OOPS counts will be low enough that I can sensibly go through and tear out the old OOPS reporting stuff.04:09
wgrantAnd then clean up the exception handling.04:09
lifelessI'm using query 11 from https://lp-oops.canonical.com/oops.py/?oopsid=1894B1679#statementlog04:10
wgrantNow that we know (hopefully) everywhere it's needed.04:10
lifelessnote that its not a hold/cold issue because its consistnetly slow in the oops04:10
wgrantIndeed, I noticed that.04:11
lifelesssadly, tis fast on qas04:11
lifelesswant me to check staging ?04:11
wgrantWorth a try, I guess :/04:11
lifelessoh but04:12
lifelessthere is also the deserialiation overhead04:12
lifelesssame results on staging04:12
lifelessno, not that04:13
lifeless72 rows04:13
lifelesstagged it dba04:14
lifelesswe need to start capturing db hostnames04:14
lifelessstill, we should fix04:15
lifelessno need to do 3 lookps04:15
=== MTecknology is now known as MTeck-InPain
StevenKHuzzah, I have my car back05:02
lifelessstub: hi05:23
stublifeless: https://dev.launchpad.net/Database/LivePatching05:23
lifelessstub: I saw - looking good05:23
lifelessstub: I am drafting a 'ReliabileDBDeployments' LEP too05:23
lifelessstub: which will frame whatever work we need to invest in this05:24
stubOk. Do you want to incorporate what I put together?05:24
wgrantI was very pleased to see LivePatching this morning.05:24
lifelessstub: I think the are complementary - the L-P page is how and implementation strategies05:25
lifelessstub: the LEP will be what, goals, constraints, requirements05:25
stubOk. I'll update that wiki document if I think of anything new or get feedback then.05:25
lifelessstub: we have some queries running slow on prod slaves, but fast on qastaging/staging05:26
lifelessstub: two so far that I know of : the duplicate bug detection FTI queries (the ones you said we can't do realtime) and the one in https://bugs.launchpad.net/launchpad/+bug/73239805:27
_mup_Bug #732398: SourcePackage:+index timeout <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/732398 >05:27
stubwgrant: So I'm worried the extra overhead (sometimes needing 3x as many db patches, extra code to support 'old' and 'new' schemas) could deter devs. You disagree and think you would make use of the process?05:27
wgrantstub: We can push things out more quickly and without hideous amounts of downtime.05:27
stubSounds like we need to pull some RAM out of prod....05:27
wgrantWhy wouldn't people make use of it, even if it slightly more cumbersome?:05:28
wgrant(I note that the page doesn't define what a light patch is, though.05:28
lifelessstub: if you could get an explain analyze on all three db's for the query in https://bugs.launchpad.net/launchpad/+bug/732398/comments/1 - that would be awesome05:28
_mup_Bug #732398: SourcePackage:+index timeout <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/732398 >05:28
stubwgrant: I'm just a born devil's advocate. There is more overhead in this process, and I'm interested in if the extra overhead will overcome the desire to get stuff rolled out 'now' rather than 'next cycle'05:29
lifelessbah, brb05:29
lifelessam I back?05:30
stublifeless: your back05:30
wgrantYou're not gone.05:30
lifelessso - can has analyze ?05:30
lifelessstub: and then, I'd like to talk fti briefly, possibly voice, possibly here05:31
wgrantInterestingly enough, as soon as I said "you're not gone", freenode lagged for 30s.05:31
lifelessstub: (short story, I want to know what I'm missing on the query - it's plan and behaviour on staging seem totally fine)05:31
lifelesswgrant: had to bounce wifi to stop openid trashing all my open tabs when I restarted chromium05:32
lifelessand I had to restart chromium becuase it had forgotten about a popup window which was permanently stuck inthe foreground05:32
cody-somervilleSoftware sucks :(05:33
lifeless(not a browser window, right mouse context window)05:33
stubCold, that query ran in 500ms or less on all prod servers05:34
lifelessstub: argh05:34
lifelessstub: so, we have a diagnostic challenge05:34
wgrantBecause it took 3s hot.05:34
stublifeless: Somehow get more information about locks05:34
lifelesswgrant: lets be precise05:34
lifelesswgrant: our timelime which records query serialisation, queuing, deserialistion and upcasting to objects and any time given to another worker thread, showed 3 seconds.05:35
stubInterestingly, the fastest one (launchpad_prod_1) had a slightly different plan05:36
stubSorry - launchpad_prod_205:36
lifelessperhaps we should capture the plan for any query over 1 second05:36
lifelessinto the timeline05:36
lifelessthere is already a per thread timeline bug05:38
lifelessbug 24355405:38
_mup_Bug #243554: oops report should record information about the running environment <lp-foundations> <oops-infrastructure> <Launchpad itself:Triaged> <OOPS Tools:Triaged> < https://launchpad.net/bugs/243554 >05:38
* lifeless retitles05:39
lifelessstub: so the same query is run three times in that page05:40
stubIn this case, I think the plan is a red herring and just an artifact of different statistics. The costs of the different parts of the plans are close enough to identical.05:40
lifelessstub: https://launchpad.net/ubuntu/lucid/+source/chromium-browser/+index - and its timing out now05:40
lifelesstrigging an lpnet sync05:41
stubJust reran the previous query - slowest was 318ms05:42
wgrantIs that with or without status IN (2)?05:42
lifelessstub: would lock contention explain the same query being slow 3 times in a row ?05:43
stubWell... maybe.05:44
lifeless4th row05:44
lifeless3 calls to it, average time 2911ms05:44
stubIf it is a slow process like the publisher it could be locking rows in the same set returned by the slow query in different transactions05:44
lifelessok, so lets see the times for these oopses05:45
stubOh... 3 queries in one transaction, no - not lock contention05:45
lifelessthey are spread over the day05:45
stublocks shouldn't be blocking selects anyway.05:47
lifelesstime of day for the oopses: 1937 1420 1937 0846 1201 1335 1258 105605:48
lifelesswgrant: what time range does the publisher run in ?05:48
wgrantlifeless: Primary archive? Normally 03-4005:48
lifelessok, not that then05:49
wgrantBut it should release locks a good 10-15 minutes before it finishes.05:49
stubHigh replication load possibly - look for corresponding lag spikes05:49
lifelessall the oopses have exactly the same pattern05:49
lifelessstub: where is the replication lag graph ?05:49
* stub is looking for it05:49
lifelessstub: and do we have one right now ?05:49
lifelesshttps://launchpad.net/ubuntu/lucid/+source/chromium-browser/+index is the page I hit to generate an oops05:50
stubNot lagged atm.05:50
lifelessstub: then thats likely not it, cause its still timing out on prod :>05:51
stubGraph is here anyway: https://lpstats.canonical.com/graphs/ProductionDBReplicationLag/05:51
wgrantSeems to time out on the master too.05:51
stubSo the same query with different parameters from the bug is still not having any problems.05:52
stubAnyone have the actual query currently timing out handy yet?05:52
lifelessstub: the one I linked is the one timing out05:53
lifeless3.7 seconds in prod05:54
stubWhen I run it, slowest 536ms.05:55
* stub checks the pastebin matches05:55
lifelessok thats strange05:56
lifelessstub: try this05:56
lifelessstub: run it, *not explain*, and press 'end'05:56
lifelessmake sure you have \timing on05:56
stubThe query is returning 5k rows and building 10k objects - would this be appserver time hidden from our metrics?05:56
lifelesshidden from an explain analyze anyhow05:57
stubThat will give crap results atm.... I'll open some local psql shells.05:57
stuberm... remote shells05:57
stubOoh... perhaps there are some silly large text fields in there?05:58
wgrantSPR is pretty fat.05:58
wgrantBecause of SPR.copyright.05:58
wgrantThere shouldn't be that many rows, though :/05:59
wgrantI'd expect a few dozen at most.05:59
lifelesscount(*) says 72 rows for me05:59
lifelessdoing select count(*) from (SE...) as _tmp;05:59
lifelessstub: how are you measuring the 5K ?06:00
stubSorry - I was looking at the estimate, not the actual returned count.06:00
wgrantI was scared that our kernel developers were even more insane than I thought.06:00
lifelesswgrant: can you put the functions into the bug ?06:01
wgrantlifeless: SUre.06:02
lifelesstheory is the analyse is not showing us the cost of a file processing of the rows in some fashion06:03
stubIt certainly is taking a lot longer to get the results to the client than it is to get the query plan.06:04
lifelessone mystery solved06:05
lifelessanother item for the db performance tips page06:05
stubJust seeing bug queries causing large temporary files in the logs06:06
wgrantOh hah.06:06
wgrantI guess linux might have an enormous copyright file.06:06
wgrantAs well as a few uploads.06:07
wgrantWe should really do away with that column.06:09
wgrantstub: Can you see how big a column is across an entire table?06:09
stubI'm about to do that.06:09
stubmax(length(foo)) stuff06:10
wgrantIt could well be a few times the size of the rest of the table.06:10
wgrantAs a bonus, the data is not used by anything yet.06:10
lifelessI selected into a temp table06:11
lifeless\dt+ foo206:11
lifeless                    List of relations06:11
lifeless   Schema   | Name | Type  | Owner | Size  | Description06:11
lifeless pg_temp_30 | foo2 | table | ro    | 96 kB |06:12
lifelessits only 96kB apparently06:12
wgrantDid you select all the SPR changelogs into a temp table?06:12
wgrantOr is that the result of the problematic select?06:12
lifelessthats the result of the problematic select06:13
wgrantI find that difficult to believe.06:13
wgrantBut I guess it's possible.06:13
lifelesswith id as id2, component as c2 etc etc06:13
lifelesslaunchpad_qastaging=> select count(*) from foo2;06:13
lifeless count06:13
lifeless    7206:13
stub1.1 mb is the largest text field in the current slow query06:14
stub(the one from the last oops)06:14
* stub checks the entire table.06:14
lifelessstub: whats the sum of that field ?06:14
wgrantDF is being a bit slow at summing all the lengths.06:15
wgrantOuch. Which SPR is that?06:15
lifelessstub: whats the length() function return ?06:16
stubThat is the sum of copywrite... so multiple06:16
lifelessI got noddy small values06:16
wgrant2.5GB of linux copyright files on DF.06:16
lifelessbut the text is big06:16
stubsum of the length? 8446605006:16
lifelessI mean hte function06:16
lifelessselect max(length(changelog_entry)) from foo2;06:16
lifeless max06:16
lifeless 341806:16
lifelesswhat does the 3418 mean ?06:17
lifelessis that pages? sectors?06:17
stubIts bytes06:17
stubSorry - characters06:17
wgrantYou mean changelog?06:17
lifelesswgrant: changelog is an int06:17
wgrantchangelog_entry is different.06:17
wgrantcopyright, not changelog06:17
stubSo UTF-8 might theoretically be 4x as many bytes?06:17
wgrantchangelog_entry is always going to be small; it's only the latest.06:18
stubI've checked all text fields on the table. The problem is copyright as you suspected.06:18
lifelessselect max(length(copyright)) from foo2;06:18
lifeless   max06:18
lifeless 112621406:18
lifeless(1 row)06:18
lifelessok, confusion sorted06:18
=== Ursinha is now known as Ursinha-afk
lifelessdropping copyright fixes it06:19
lifelesswhat do we have this in there for ?06:19
wgrantNothing at all.06:19
wgrantIt's populated, but not used.06:19
lifelessdrop the column definition from launchpad ?06:19
wgrantWe could possibly drop it from the class immediately, and just set it on upload, and then migrate it out of the DB later.06:19
lifelessthat will stop storm querying it06:19
stubSo yeah, that column needs to be split into a separate table. Code only fix might be to remove that field from the main Storm class, and have a separate Storm class with the extra column and use that only where necessary.06:20
wgrantstub: s/table/librarian/, I think.06:20
lifelessquery can show all the rows in 300ms with that field removed.06:20
stubwgrant: If there is no need for it to be in the DB, sure.06:20
lifelesswgrant: want to do this one?06:21
stubwgrant: If it is being shown inline on the page, I guess that is a performance call (pull it from the librarian and render it will be slower than from the db... unless it is ajax, and then search engines won't see it)06:21
wgrantlifeless: I'll drop the column from the Storm definition now.06:21
lifelesswgrant: cool06:21
lifelessstub: its not used at all06:21
lifelessstub: premature optimisation years ago06:21
wgrantstub: It may plausibly be displayed in the page.06:21
wgrantBut it isn't yet.06:21
wgrantAnd if someone wants to, they can damn well pull it from the librarian.06:21
lifelessiframe embedding ftw06:22
wgrantAnd if someone wants to display multiple on a single page, they are probably wrong.06:22
wgrantSo we don't have to consider that case.06:22
stubSo asuming ascii text, and Python decoding it into Unicode, that is 324MB of RAM needed before it even gets past psycopg2. This sort of thing will certainly be driving our memory footprint.06:23
lifelessor they can just hand out librarian urls06:23
wgrantI wonder if the uploader can update SPR. I guess I'm about to find out.06:23
stubGiven Python won't clean up, and that is per thread.06:23
lifelessstub: ok, next one up is fti06:24
wgrantI've been concerned about SPR.copyright for well over a year now, but never had the data to confirm that it was a problem.06:24
lifelesswgrant: data is wonderful isn't it06:24
stubI prefer a bucket of sand06:24
stubnah... nah... fti... can't hear you... nah... nah06:25
jtvstub: I just put up a security.py cleanup for your review that also gets it to run under 3 seconds.06:25
stubSo the query I looked at the other day had 32 lines of multiple boolean operations going on. I'm surprised it doesn't timeout just generating the plan for that, let alone get around to running the query06:25
stubjtv: Ta.06:26
lifelessstub: the plan is simple :)06:26
lifelessstub: its an 800ms query on qastaging06:26
jtvstub: Just testing if you really couldn't hear anyone.  :)  https://code.launchpad.net/~jtv/launchpad/faster-security/+merge/5280406:26
lifelessstub: http://pastebin.com/7DuF11Z806:28
* stub finds the bug 72617506:31
_mup_Bug #726175: DistributionSourcePackage:+filebug Timeout trying to file bug due to FTI timeout <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/726175 >06:31
stubAnd 3.6 seconds on production06:33
lifelessstub: doing the same show-the-results test?06:34
stubBy accident, yes....06:34
* stub waits for his terminal to return to normal06:34
lifelessso 3.6 is << 1506:34
=== MTeck-InPain is now known as MTecknology
stubSo width isn't the problem here - no insane targetnamecaches or statusexplanation06:38
lifelessstub: right, and it performs tolerably interactively06:39
stub65% of the index is data, which is a little bloated but not bad (launchpad_prod_2 here)06:40
lifelessusing package git not installed failed to install upgrade ErrorMessage: subprocess installed post-installation script returned error exit status 1 as a search term the search completes right now on prod06:40
lifelessat /ubuntu/+filebug06:41
lifelesshuwshimi: hey06:41
lifelesshuwshimi: I have a challenge for you06:41
lifelesshuwshimi: when an ajax/api request is made, I'd like to put the time in the top right - you know where - into a journal of items06:42
stubThe limit isn't helping at all - all the rows are going to be pulled to calculate the rank so things can be ordered.06:42
lifelessit would be nice to set a cap on the relevance but I don't htink our tsearch setup is ready for that06:45
huwshimilifeless: so you'd like to figure out how to get the time for the roundtrip?06:45
lifelesshuwshimi: and glue it all together :)06:46
lifelesshuwshimi: its not urgent06:46
lifelesshuwshimi: but its the next step in visibility to devs of perofrmance06:46
stubhttp://paste.ubuntu.com/578204/ is the plan I'm looking at btw.06:46
lifelessstub: yeah, thats what i see too06:46
lifelessstub: except mine is a little faster06:47
lifeless Sort  (cost=41.08..41.09 rows=1 width=650) (actual time=554.699..554.699 rows=1 loops=1)06:47
stublifeless: So wtf is the query doing for the first 1.4 seconds?06:47
stubThe fti stuff starts 1412 ms in.06:47
lifelessstub: search me06:48
stubOh.... remember I mentioned those big temporary tables in the pg logs?06:49
lifelessstub: I think you may have *meant* to, but I don't remember a discussion06:49
lifelessstub: or you did tell me and I've lost it ;)06:49
wgrantHe mentioned them.06:49
huwshimilifeless: I'm not sure about YUI, but with other ajax frameworks you can add hooks to every ajax event. And then it's just a matter of recording the time the ajax event fires and then comparing it to the time when the ajax event completes. I'll take a look some time06:49
stub(13:06:15) stub: Just seeing bug queries causing large temporary files in the logs06:49
wgrantGnargh gina.06:50
lifelessstub: ah thaks06:50
StevenKwgrant: ?06:50
lifelessstub: you think this is it ?06:50
huwshimilifeless: The one thing I've learnt though is that everything is an order of magnitude harder with YUI than other frameworks :)06:50
lifelesshuwshimi: what framework should we be using?06:51
wgrantStevenK: I'm removing SPR.copyright.06:51
jtvstub: I may be missing the point since I just jumped in, but all it's got at the point where the 1.4 seconds are lost is 1 tuple06:51
jtv(See all the way at the bottom of the plan)06:51
huwshimilifeless: Do you really want to go into that? :)06:51
StevenKwgrant: Why? I think some people want it.06:51
lifelesshuwshimi: sure, why not?06:51
wgrantStevenK: I'm removing it from the class for now. It will be moved into the librarian later.06:51
lifelessstub: launchpad_qastaging=> SELECT BugTask. ... assignee |  bug   | bugwatch | date_assigned | date_closed | date_confirmed | date_fix_committed | date_fix_released | date_incomplete | date_inprogress | date_left_closed | date_left_new | date_triaged |        datecreated         | distribution | distroseries |   id   | importance | milestone |  owner  | product | productseries | sourcepackagename | status | statusexplanation | t06:52
lifeless          | 715778 |          |               |             |                |                    |                   |                 |                 |                  |               |              | 2011-02-09 14:06:41.463232 |              |              | 836597 |          5 |           | 3439478 |   24742 |               |                   |     10 |                   | Linaro GCC06:52
lifeless(1 row)06:52
lifelessTime: 861.816 ms06:52
lifelessjtv: thats a relative offset, not absolute06:52
lifelessjtv: because its in the nested loop06:52
lifelessjtv: so its at 1413.054 + 0.040 and takes 0.04206:53
jtvlifeless: it's in the same nested loop where the other part starts at 1.4 seconds06:53
jtvDon't they both count from the start of the loop?06:53
huwshimilifeless: OK, I think there are a bunch of issues with YUI. Its development is really slow and can not keep up with the pace of other frameworks. With the way things are going at Yahoo I also worry that there will get to a point where it may stop being developed by Yahoo at all.06:54
lifelessor at least, I've only been able to make sense from plans if I assume the answer is no06:54
huwshimilifeless: The community around YUI is tiny and I don't think a serious community effort would be made to keep it going (I could be wrong and by the community taking it over it might help drastically)06:55
* jtv retouches his fading formerly-yellow note saying "read actual documentation"06:55
lifelesshuwshimi: so thats a bunch of risks06:55
huwshimilifeless: All of this means that there are a lot of half baked features in YUI.06:55
lifelessnot really an argument for where to aim *at*06:56
stubjtv: From the plan, it seems we start looking at the fti index at 1.4 seconds in. Before that, the only thing that was done was an index scan on bugtask_product__bug__key that completed less than 1ms in.06:56
huwshimilifeless: Some things people seem to really like. Like the testing framework06:56
stub(looking at the start times in actual time=)06:56
stubI can't see it being a temporary file issue since the plan is reporting memory sorting.06:57
huwshimilifeless: But for many things it seems that you write more code and spend more time getting around YUI's drawbacks than you should.06:57
jtvstub: so… some weird combinatorial behaviour in the planner for those booleans?  Try removing one and see if the time halves.  :)06:58
huwshimilifeless: Also, I'm not a YUI expert, from discussions with sinzui, he has very similar feelings about a lot of this and he might be a better person to talk to.06:58
stubIt might be spending 1.4seconds working out wtf that obscene boolean calculates down to...06:58
stubI can time that...06:58
huwshimilifeless: Another side of YUI having such a small community is that the plugin ecosystem is tiny.06:59
lifelessstub: that last line is a per-found-bug inner loop06:59
lifelessstub: it can /only/ execute after the fti starts spitting out rows06:59
huwshimilifeless: We would save a lot of development time if we didn't have to reinvent the wheel for a lot of stuff.06:59
lifelesshuwshimi: so, there is a slightly larger discussion to have if we decide that a different framework would be better07:00
huwshimilifeless: And most of the plugins that do exist were developed a number of years ago and are not maintained or don't work with new version of YUI07:01
lifelesshuwshimi: but I want to be clear that it is a discussion we can have if you want07:01
huwshimilifeless: I would be *very* open to that discussion.07:01
lifelesshuwshimi: then I suggest you start it up :) - talk to sinzui, talk to rockstar07:02
lifelesshuwshimi: take it to the canonical-rhinos list if those two folk are agreeable07:02
lifelessfeel free to cc me07:02
huwshimilifeless: my concern is that we have a lot of existing YUI dependant code07:02
lifelesshuwshimi: we had a lot of slow code 6 months ago07:02
huwshimilifeless: Haha07:03
lifelesshuwshimi: but we're down to 2.7seconds for our 99th percentile request time07:03
lifelesshuwshimi: so big things can be don07:06
lifelesshuwshimi: and we've much less yui code than slow code07:06
LPCIBotProject windmill build #32: FAILURE in 1 hr 8 min: https://hudson.wedontsleep.org/job/windmill/32/07:07
wgrantHeh. It hasn't failed in more than a week, and it fails while we are discussing its demise...07:07
wgrantAlthough it looks like it could be a real failure.07:07
huwshimilifeless: Is there a particular reason you're bringing this up (I'm just kind of surprised that you've raised the topic)?07:07
lifelesshuwshimi: you whinged07:08
huwshimilifeless: Haha ok07:08
lifelesshuwshimi: my job can be summarised as:07:08
lifeless - figure out what makes delivering on our goals hard for our devs07:08
lifeless - and arrange for it to be fixed07:08
lifelessstub: to make sure we are looking at the same thing07:10
huwshimilifeless: I've very aware that just because I have opinions on (or prefer) things it doesn't mean that I'm right. I wouldn't want to duplicate a bunch of effort in rewriting/training etc. without it being worth it.07:11
lifelesshuwshimi: indeed, so thats part of the discussion that you need to have07:11
huwshimilifeless: As we're moving to a more javascript heavy version of the site this is probably a good time to discuss it07:13
stublifeless: So the difference between the qastaging query plan and the production query plan is the qastaging query plan starts doing the fti index scan 230ms in, and the production query plan starts doing the fti index scan 1412ms in. That seems to account for most of the difference.07:14
lifelessstub: ok07:15
lifelessstub: how do we do we figure out the 15000seconds case ?07:15
stubSo on #postgres, seems unanimous you see this when PG is waiting for disk07:16
lifelessstub: -> #postgres then07:20
wgrantThe test suite breaks if you have debian-keyring installed.07:36
wgrantBecause dpkg-source then has keys to verify some of the packages in the gina test archive.07:36
wgrantIt by default allows an unverifiable signature, but refuses to unpack a bad one.07:36
lifeless\o/ the with query works07:43
lifelesswgrant: https://bugs.launchpad.net/launchpad/+bug/22193807:44
_mup_Bug #221938: Email interface crashes when an attachment file name contains a slash <email> <lp-bugs> <oops> <qa-needstesting> <Launchpad itself:Fix Committed by thumper> < https://launchpad.net/bugs/221938 >07:44
wgrantlifeless: Currently fixing Windmill breakage (real bug, but it won't affect any production data).07:44
lifelessbah, no wally07:44
=== mthaddon changed the topic of #launchpad-dev to: Launchpad down/read-only from 09:00-10:30UTC for a code update | https://dev.launchpad.net/ | firefighting: - | On call reviewer: abentley | https://code.launchpad.net/launchpad-project/+activereviews
wgrantec2 mail fails if your default bzr email address isn't @canonical.com.08:10
StevenKMy instances mail me at my debian.org address just fine08:11
wgrantYou're probably not using the canonical.com SMTP server, though.08:11
StevenKI am, and my from address is @ubuntu.com08:11
StevenKSo it's fine08:11
wgrantIt stopped working for me today, and I changed my default email address last night...08:12
StevenKwgrant: Happy to share headers if it will help.08:13
wgrantStevenK: bazaar.conf would be handy.08:14
StevenKHmm, when did smtp.c.c start being youngberry ...08:17
stubdebian.org might be special cased given the background of our admins...08:17
StevenKwgrant: https://pastebin.canonical.com/44501/ is the relevant bit08:18
wgrantStevenK: Your default bzr email address looks a lot like ubuntu.com08:19
StevenKWhich is what I said my From address was ...08:19
wgrantOh, mail you *at* your debian.org address.08:19
StevenKwgrant: You can't talk via smtp.c.c if your From address isn't canonical.com or ubuntu.com. If you want to use something else as your From address, use a different SMTP server.08:20
wgrantStevenK: Sure. But my email address is @canonical.com for my branches... ec2 must not use locations.conf, or it uses some other path.08:21
StevenKI didn't think your bazaar config was copied to the instances ...08:22
wgrantIt does some evil stuff.08:23
wgrantNot copying the file directly, but some of its values.08:23
lifelesswgrant: what are you fighting08:28
wgrantlifeless: Hm?08:28
lifelesswhats up08:29
wgrantBug #73244208:29
_mup_Bug #732442: disable_existing_builds compares series name to display name <recipe> <regression> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/732442 >08:29
wgrantGiven up for now, will ask wallyworld tomorrow.08:29
lifelessstub: hey, so the analyze might have helped ?08:29
lifelesswgrant: oh right08:29
lifelesswgrant: ;(08:29
wgrantI say it's qa-ok, although that is sort of cheating.08:30
wgrantlifeless: https://code.launchpad.net/~wgrant/launchpad/unuse-spr-copyright/+merge/52808 may interest you.08:31
lifelessstub: how often do we do full backups ?08:38
stublifeless: daily dumps08:41
stublifeless: No, analyze didn't change anything.08:41
stublifeless: So waiting a while, I still see the initial query with a high startup time and a query immediately after a lower startup time. So I think we must be seeing the effects of shuffling data between disk cache and the pg shared memory area.08:45
lifelessstub: thats plausible08:46
stublifeless: Currently, the shared memory area is set to 5GB on all the production boxes, which is the high side of best practice and when people start seeing degraded performance.08:46
lifelessstub: can we change this during this downtime ?08:46
lifelessstub: ah08:46
lifelessstub: how can we test to see whether we would suffer08:46
stublifeless: I'm considering bumping it up to 7GB on one of the slaves. Even though we will be going 'too high' in most peoples opinions, we do have an unusual load so best practice might not apply here.08:47
stubWe have run with 8GB before, no particular ill effects, so I don't think it will hurt. And we can then check out differences in performance between the two slaves.08:47
stubBut still, even 8GB will be lower than the hotset of data.08:48
lifelesswhats it set to on staging?08:48
stubprobably about 3...08:48
* stub checks08:48
stubOf course, it could be counter intuitive and lowering the value might work better ;)08:49
stub2GB on staging08:49
adeuringgood morning08:49
wgrantHm, no.08:49
wgrantstupid ec2.08:49
=== almaisan-away is now known as al-maisan
jammorning all09:00
wgrantMorning jam.09:01
jamman, living in a country where you don't speak the language causes all sorts of web confusion09:02
jamevery site wants to default me to Dutch09:02
jamstupid geoip :)09:02
jammaybe google is just on crack. Because I manage to get to the "Settings" page, set my language as English, looks ok09:05
jamgo back to account settings, and everything is back in dutch09:06
bigjoolsyeah that's annoying when travelling to sprints09:06
jamsigning out and back in again seemed to help in the end09:09
jambut yeha09:09
jamwgrant: are you watching the upgrade? I'll be happy to start testing/monitoring once things are up09:09
wgrantjam: I'm watching.09:10
* wgrant opens the graph.09:10
=== adeuring1 is now known as adeuring
jamwgrant: looks like bzr-sftp still isn't wanting to shut down cleanly. getting the "cannot shutdown reactor that isn't running"09:27
jamMaybe that is the code that wants to always cleanly shut down, waiting for the last connection to exit09:28
wgrantjam: There were 5 connections remaining when it was forcibly killed.09:28
jamwgrant: k, where do you get this info? Maybe I'm in the wrong channels?09:28
wgrantjam: #launchpad-ops is where it happens, and you seem to be there.09:30
wgrantBut I checked the connection count myself.09:30
jamhow did you check it?09:30
jamYeah, I'm there, but I haven't been following as much as I should :)09:30
wgrantjam: Certainly machines can see bazaar.launchpad.net:8022.09:32
jamwgrant: "certain" machines ?09:33
wgrantUm, yes, that.09:33
jamwgrant: devpad doesn't appear to be one of them09:33
jamwgrant: do you know the rsync module for the crowberry-sftp-log subdir?09:38
wgrantjam: I did... let me check.09:39
wgrantCould be sftp-logs09:39
jamcrowberry::sftp-logs/ => unknown module09:39
wgrant20:39:26 < wgrant> logs-sftp09:40
jamyeah, just found that myself09:41
henningeHi adeuring!09:43
adeuringhi henninge09:44
wgrantjam: Looking OK?09:44
henningeDid you change anything on the branch since you last pushed it?09:44
henningeadeuring: ^09:44
jamwgrant: so far, haven't gotten to the end. the sftp service started slightly before the forking service, and was already getting connection requsetts09:44
adeuringhenninge: I just merged devel yesterday evening. nothing else yet09:44
henningeyes, I saw that09:45
wgrantjam: We do have slight ordering issues in both directions :(09:45
henningeadeuring: You used "sharing status" throughout while I had been talking about "sharing details".09:45
jamwgrant: as long as both are up when we consider the site 'live' I'm not really worried :)09:45
wgrantResponse times are still fine from here.09:46
henningeadeuring: any particular reason for that naming?09:46
wgrantNo graphs yet.09:46
adeuringhenninge: no particular reason. I'll change it to details09:46
henningeadeuring: I can do that.09:46
adeuringhenninge: ok09:46
henningeadeuring: I finished the dummy template.09:47
adeuringhenninge: sounds great09:47
henningeadeuring: you will have to add the conditions and such from the view.09:47
adeuringhenninge: ok09:47
henningeadeuring: I will push that when I am done with the renaming.09:48
=== allenap changed the topic of #launchpad-dev to: Launchpad down/read-only from 09:00-10:30UTC for a code update | https://dev.launchpad.net/ | firefighting: - | On call reviewer: allenap | https://code.launchpad.net/launchpad-project/+activereviews
* allenap assumes abentley is no longer reviewing.09:48
StevenKIn a slightly related topic, is Firefox history editable?09:51
=== Ursinha-afk is now known as Ursinha
StevenK(I have a whole bunch of edge URLs there that need to die. Slowly.)09:51
StevenKallenap: https://code.launchpad.net/~stevenk/launchpad/derive-common-ancestor/+merge/52796 (and thanks!)09:52
allenapStevenK: Got it.09:52
StevenKallenap: afk for dinner, if you have questions queue them up here and I'll answer them when I can.09:53
allenapStevenK: Cool.09:54
jamwgrant: is it just that you're machine is allowed through the firewall to :8022?09:57
wgrantjam: I guess it must be. I presumed carob could too, but I guess not.09:58
jamwgrant: well 'w3m http://bazaar.launchpad.net:8022' didn't do much for me09:58
jamcould be a proxy issue09:58
wgrantOh, it's certainly not going to work externally :)09:58
jamno, from carob09:59
jambut I can do wget just  fine09:59
jamso good enough, I guess09:59
jam134 conn09:59
jamstill says "unavailable" though, which is odd09:59
jamI though spiv had a fix for that09:59
jamwgrant: I'm a bit surprised about IP addresses that have 10+ connections active10:00
jam(wget | sort)10:00
lifelessjam: spiv couldn't reproduce it10:01
jamjust jumped to 246 conns...10:03
jamwgrant: but I'm not seeing any failures, yet10:05
jmlhttps://code.launchpad.net/~jml/launchpad/what-is-in-the-web-ui/+merge/52594 up for review10:05
jamnote that conns includes ones that aren't authenticated, IIRC10:05
wgrantjam: 246? That's not good.10:05
jamnot many access failures in the log, though10:08
jamwgrant: 25910:09
jambut no failures in the other logs10:09
henningeadeuring: I will have to wait for the roll-out to finish before I can push my changes. :(10:09
adeuringhenninge: no problem10:09
wgranthenninge: Hm? It should all be back now.10:09
henningeoh, already?10:09
wgranthenninge: Codehosting has been back for like 25 minutes.10:10
wgrantjam: How are your connection times? Up to 8s here ;/10:10
henningeoh, I just didn't try. thanks wgrant10:10
henningeadeuring: pushed10:12
adeuringhenninge: I'll look10:12
henningeadeuring: "pull" is the right term ;-)10:13
=== mthaddon changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: allenap | https://code.launchpad.net/launchpad-project/+activereviews
jamwgrant: https://code.launchpad.net/~jameinel/lp-production-configs/disable-forking/+merge/5281810:48
wgrantjam: LOSAs review those.10:49
jamwgrant: https://lpstats.canonical.com/graphs/CodehostingPerformance/20110310/20110311/nocache/10:52
jamthe 900s spike is going to kill the graph for a long time to come... :(10:52
wgrantHeh, yes.10:52
StevenKallenap: Thank you for the review!11:05
allenapStevenK: You're welcome :)11:05
StevenKallenap: How did you find the trailing whitespace?11:07
allenapStevenK: I load the diff into my editor, and I've set it to show trailing whitespace as big red blocks. I can't miss it :)11:07
StevenKallenap: Is your editor vim?11:08
allenapStevenK: The other one.11:08
=== henninge_ is now known as henninge
=== al-maisan is now known as almaisan-away
jamI'm trying to set up lp on a new virtual host, but it is giving me failures during "make-lp-user".11:48
jamany ideas?11:48
wgrantWell then.11:50
wgrantDid you use rocketfuel-setup?11:51
jamand then tweaked after for being in a VM11:52
jambut not much changed there11:52
wgrantYou ran launchpad-database-setup?11:52
wgrantmake schema should have failed without it, but it's possible you did enough manually to unbreak it.11:52
wgrantThere should be "Launchpad configuration" bits at the end of postgresql.conf.11:53
leonardrmy internet connection is up and down rightn ow11:53
jamwgrant: running it directly11:54
jamI did very little manually11:54
jambut I didn't see a Launchpad configuration bit at the end.11:54
jamand I did have earlier problems connecting to postgres11:55
wgrantjam: Try running it again, I guess.11:55
wgrantIt is meant to change the default search path.11:55
jamyeah, running it now, but have to wait for "make schema" to finish11:55
jamwgrant: looks like that did it. Thanks!11:56
wgrantjam: Great.11:57
wgrant2011-03-10 09:40:34+0000 [SSHChannel session (0) on SSHService ssh-connection on ProtocolWrapper,36,] Forking returned pid: 18870, path: /tmp/lp-forking-service-child-kwWa4B11:59
wgrantechan, but yeah.11:59
jamwgrant: that was the first death?12:00
wgrantjam: It's the first one that's still alive at the end.12:01
jamwgrant: if you grep around there, you can see what user it was12:01
wgrantIndeed, but that was less than a minute after the service started...12:02
jamwgrant: yay, even have codehosting serving on 5022. Though I wonder if there is a way to do that without hacking the source code.12:03
jamso I don't have to worry about accidentally committing that12:03
wgrantjam: You could possibly create a config overlay and run it with that.12:04
jamwgrant: wouldn't integrate well with 'make run_codehosting' I imagine12:04
wgrantjam: 'LPCONFIG=mycustomconfig make run_codehosting' should work.12:05
deryckMorning, all.12:06
wgrantleonardr: Hi.12:31
leonardrwgrant, hey12:32
wgrantleonardr: What is the recommended way to use a keyringed launchpadlib from a cron job?12:32
leonardrwgrant: you need to store the credential in an unencrypted file, and pass it in as credentials_file12:33
leonardrsee section 5 of https://lists.launchpad.net/launchpad-users/msg06239.html12:33
LPCIBotProject windmill build #33: STILL FAILING in 1 hr 10 min: https://hudson.wedontsleep.org/job/windmill/33/12:53
vilaHey all, what the update frequency for the downloads stats ?13:19
vilaas in https://launchpad.net/bzr/+download for example13:20
=== almaisan-away is now known as al-maisan
bachi mrevell13:34
mrevellhi bac13:35
=== mrevell is now known as mrevell-lunch
jamanyone else here use Empathy? I'm trying to use it, since it integrates with the OS (and is the suggested default)13:52
jambut it only really shows about 10-20 lines of actual content per page13:52
jamwith all the extra formatting13:52
jamIs there a way to make all the bubbles smaller?13:52
deryckhenninge, adeuring -- ping for standup14:00
wgrantbigjools: Hi.14:01
=== jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: allenap, jcsackett | https://code.launchpad.net/launchpad-project/+activereviews
jmlallenap: thanks for the review. I had posted some updates as you were doing it (to docs, mostly) and have now fixed the zcml problem14:05
allenapjml: Cool, I'll have a look.14:06
allenapjml: Did you consider parsing the zcml directly?14:08
jmlallenap: to what end?14:08
jmlallenap: I mean, yes I did, but then I thought that all I'd be doing is constructing objects much like the adapters that Zope provides in the first place.14:09
allenapjml: Because it might end up being less hacky... I haven't given it much thought, but wondered if you had.14:09
jmlallenap: it might fix some of the hackiness in format_page_adapter, but not much else. It would come at a cost of parsing ZCML myself14:10
allenapjml: I guess the only thing you might avoid is filtering through the mro to find the real view.14:10
jmlallenap: yeah, exactly.14:10
allenapjml: Yeah, and processing includes. Okay. I don't think the mro thing is too bad actually.14:11
jmlanother approach would be overriding the browser:page handler14:11
jmlbut I'm not sure that would be less hacky, since our custom handler is specified in zcml also14:11
jmlso I'd have to find that ZCML and somehow exclude it.14:12
jmlalso, Launchpad has 462 different types of page.14:13
=== mrevell-lunch is now known as mrevell
deryckhenninge, I'm wondering about this "deactivate translation imports on a per package basis" card.  Is there a bug for this?14:20
leonardrallenap or jcsackett, could you take a look at https://code.launchpad.net/~leonardr/lazr.restful/operation-must-be-versioned/+merge/52858?14:24
jtvAnd I have one that's particularly relevant to allenap: https://code.launchpad.net/~jtv/launchpad/bug-730460-job-class/+merge/5285714:24
jcsackettallenap: i'll take leonardr, you take jtv?14:24
allenapSounds good.14:24
jcsackettleonardr: this the one that's been causing you and sinzui so much pain?14:25
henningederyck: hm ...14:26
henningederyck: no, there is not.14:26
henningederyck: It came up in a discussion and is something that still needs to be done before the feature can really go live.14:27
deryckhenninge, why do we need per-package disabling?14:27
henningederyck: but it should be quite simple14:27
henningewe just need to find the place in the code where it needs to be done.14:27
deryckhenninge, can't you already disable by changing the template name?14:27
henningederyck: once a sourcepackage has translation sharing set up, imports from package uploads must stop.14:28
henningederyck: it should only import the template from then on.14:28
henningederyck: I don't know what you mean about changing thetemplate name.14:28
henningederyck: I will file a bug and explain there.14:28
deryckhenninge, ah!  I get you now.  I misunderstood.14:29
deryckhenninge, yes, please file a bug and tag it with the story.14:29
jmlwgrant: I thought you said PQM was way faster now.14:35
henningederyck: bug 73261214:39
deryckhenninge, thanks!14:39
henningederyck: updated card14:39
deryckhenninge, thanks, again! :-)14:40
leonardrjcsackett: no, i'm only working a half day today so i'm not even touching that one14:40
jcsackettleonardr: sounds wise. :-)14:41
leonardroops, forgot to push the latest version14:42
leonardrit's pushed now14:42
deryckabentley, bug 719521 is fix released, yes?14:44
jcsackettleonardr: i gather this branch is part of helping us not randomly bork the 1.0 version of the webservice?14:44
leonardrjcsackett: exactly14:45
abentleyderyck: I don't think so.14:45
deryckabentley, no?  The card is in done-done.14:45
abentleyderyck: I can move it back if you'd like :-)14:45
deryckabentley, heh. well, I don't know. :-)  What's left to do?14:45
abentleyderyck: the code is in place, but the configs need updating and the cron script needs to be configured before unlinking will actually split translations.14:46
deryckabentley, ok, the cron script is the same for linking, right?14:47
abentleyderyck: right.14:47
jcsackettleonardr: r=me.14:47
deryckabentley, ok, so I will close the bug as fix released.  Since the coding is done.  We have a card for the configs.  and the cronscript is an RT away.14:47
leonardrjcsackett: just in time!14:48
abentleyderyck: from an end-user perspective, no fix is released, so we could get pushback.14:48
jcsackettreplate that with ":-)"14:48
abentleyderyck: but of course it's your call.14:49
deryckabentley, thanks, and I can live with that risk.  also, bug 696009 is fix released, too?14:49
jtvI suppose jml was thinking låünchpäd14:50
abentleyderyck: yes.14:50
deryckabentley, great, many thanks for the chat.14:50
jmljtv: huh?14:50
abentleyderyck: np14:50
jtvjml: wasn't that you?  About sticking umlauts on the name?14:51
jmljtv: oh, right.14:51
=== al-maisan is now known as almaisan-away
jtv"It's not bad, for a two-umlaut site"14:51
=== almaisan-away is now known as al-maisan
deryckabentley, sorry, one more.  bug 706005 is released?14:52
abentleyjtv: lauñçħpad?14:52
jtvFunny how most of the diacritics I can think to stick on there are actually more or less appropriate…  the first "a" really is like "å," the "n" really sounds like "ñ," and so on.14:53
jtvabentley: what's the "h-like" letter?14:53
abentleyderyck: yes, but not deployed, like 719521.14:53
jtvAh, forgot one… Łåüñčħpäđ14:54
deryckabentley, gotcha. thanks.14:54
abentleyjtv: No idea, I've just seen it in names.14:54
jtvYou clearly hang out at cooler clubs than I do.14:55
abentleyderyck: should I be using YUI attributes or normal Javascript object properties by default?14:56
jtvallenap: I'm going to have to call it a day… can we continue the review offline?14:56
jtvAnd by "we" I mean you.14:56
allenapjtv: Sure, no worries. Have a good evening :)14:57
jtvThanks. :)14:57
deryckabentley, YUI attrs.  Since it makes it obvious when you're doing get and set on an attr.14:58
deryckand not silently adding something that didn't previously exist.14:58
abentleyderyck: coming from python, the risk of silently adding something that didn't previously exist doesn't seem very serious.15:00
deryckabentley, yeah, maybe it's not in javascript, too.  but behavior is not always clear.  do I check for undefined or null if I want to be sure, for example?  Using YUI attrs offers a consistent API, among other benefits.15:02
deryckhenninge, adeuring -- I filed bug 732633 about your work, and made it team assigned.  please link branches there.15:06
jmlwgrant: never mind, I'm behind an lp-production-configs branch.15:06
deryckgah.  can't save two cards with the same bug id again.  I thought this was fixed.15:08
deryckadeuring, because of bug ^^ your card lacks the bug link.  sorry.15:08
deryckabentley, and I made bug 732639 for the js work if you want to link branches there.15:12
abentleyderyck: okay.15:13
bigjoolsWOA, stop adding tests to Lanchpad.  "13373 tests run..."  :-)15:19
rvbasinzui: when we were talking about modifying distroseries (to clean up this registrant/owner) thing, you said I had to options: migrate the field owner to registrant or add a registrant field that would return the content of the owner field.15:21
rvbasinzui: At first I thought the most clean way to do this was to migrate the field but after looking at the code I'm not so sure because this object would then differ from most of the others ... any thought on this?15:21
sinzuirvba: firstly, distroseries and productseries, being series should be the same in this case, I suspect milestones and releases should be the same. 1, the real owner is always the project/distro owner. The creator is the registrant. and has no power. So all four object use owner when we mean registrant15:23
sinzui^ well the 'firstly' and '1' never got answered in that sentence15:25
rvbasinzui: should I be waiting to a secondly or a 2. then :-) ?15:35
sinzuisorry, did I miss a message? I had display issues so I left the channel for a moment15:35
sinzuirvba I think the distroseries is is larger than you started with. You are looking at a group of objects  that could have different owners in 2005, but, by 2007, the project/distro became the owner in the permissions rules, so we resused the owner field as the registrant15:37
rvbasinzui: so I guess your advice is to fix the interface then15:38
rvbacreating a registrant field returning the content of owner15:38
rvbasinzui: or do you think I should engage in refactoring the schemas to create a proper registrant field15:39
sinzuirvba: storm does not require that the field name be the same as the db column. Several objects differ. The simply fix might be to rename the owner => registrant in the interface and model for the objects, and ensure that column=owner15:39
* sinzui looks for example15:40
rvbasinzui: I get it, seems like a very simple fix15:40
sinzuiyes. I might have done it years ago if I had this conversation.15:40
rvbaonce this is done, it's really nothing to migrate the data properly and rename the field in the database for consistency15:41
rvbawell, I guess :-)15:42
sinzuigreat, the models look good they all specify dbname="owner". rvba: I think you can change the interface and models attribute names owner => registrant of distroseries, productseries, and productrelease15:44
abentleyhenninge: do you expect the sharing details page to control translation permissions or translation group?15:44
rvbasinzui: all right ... so I think it's most clean to migrate the db column as well15:45
sinzuirvba: So you will need to update all the callsites and templates, but since this information is not very important, you will not need to change many.15:45
henningeabentley: we need to set the permission to "Closed" if the project is unmaintained.15:46
sinzuirvba: you certainly can change the column name15:46
rvbasinzui: ok, that is pretty much what I did for the distributions so I think I'll manage thx15:46
abentleyhenninge: how would we do that from the sharing details page?15:46
rvbasinzui: you'll get to be the reviewer on this though :)15:47
henningeabentley: actually, thinking about it, I am not sure that we need to do it through ajax.15:49
rvbasinzui:thanks a lot15:49
henningeabentley: it's just that projects are created with "OPEN" permissions by default and that needs to be changed if the project is not using Launchpad.15:49
sinzuiI am happy to help15:50
abentleyhenninge: so if a project is unmaintained, then it's owned by Registry Admins, and the logged-in user probably can't change the setting.15:50
henningeabentley: so it's "if product.translation_usage != LAUNCHPAD: permission =CLOSED"15:50
henningeso it's probably something that we need to change in the product creation itself.15:51
sinzuihas leonardr been about? I really hope not15:51
abentleyhenninge: oh dear.  You add the card :-P15:51
henningesinzui: did anybody ever answer your question about reviewing translation imports?15:52
henningeabentley: gee, thanks ;)15:52
sinzuiabentley: henninge about the previous conversation. Users often ask for the project back or ask me to change settings once they realise that giving the project to ~registry excludes them from configuring it...15:53
henningemaybe we can do it in a way that it is owned by the creating user until all is done?15:54
sinzuiabentley: henninge: I proposed they we let anyone or trusted users be permitted to configure unconfigured projects or those owned by ~registry. The conversation went into audit logs and new showing permissions in the UI. So that was way out of scope. I stopped working on the bug15:55
jamwell, I at least managed to track down the cause of one of my critical failures today (bug #732481). Always funny when releasing code X triggers a bug in code Y.15:55
jamanyway, EOD for now. Maybe I'll be back on to finish it later tonight.15:56
sinzuihenninge: as to my question, no, it was not answered. I am trying to answer the translations questions, but I am very slow at providing answers15:56
jammthaddon: if you are still around, can you post what you did to make HAPROXY use a GET request instead of a HEAD request, we probably want to revert it once we get bug #732481 fixed.15:56
henningesinzui: ok, let me see if I can shed some light on that15:56
abentleyhenninge: If we did that, it would cover the case where the upstream project is new.15:57
henningeabentley: yes, that is the one I am mostly thinking of. In the other case they'd need the help of the project's owner.15:58
abentleyhenninge: which might be ~registry.15:58
sinzuiabentley: henninge: we did want project registration to be guided, take the user through all the configuration screens. Such a process should make the ~registry the owner at the end15:58
henningeabentley: in that case we'd have to make the current user the owner temporarily.15:59
henningebut how to know when "temporarily" ends?16:00
abentleyhenninge: I don't know.16:00
abentleyhenninge: we should be able to write a script to ensure all the ~registry-owned projects already have the right settings.16:01
bigjoolsgary_poster: if a tarball in dists is not mentioned in versions.cfg, I can completely remove it right?  CherryPy has a zip file but is not in versions.cfg.16:15
gary_posterbigjools, that *should* be true, but fwiw, versions.cfg is about enforcing the versions, not about enforcing their presence or absence.  One sec, there is an easy way to doublecheck...16:16
gary_posterbigjools, there may be a nicer way, but ./bin/buildout -vvv | grep 'Getting required' will get you a list of everything that buildout wants.  you could grep the same output for CherryPy if you like16:18
bigjoolsexcellent, thanks16:19
gary_posterand fwiw, no, we don't use it16:19
gary_poster(having just tasted my own medicine)16:19
bigjoolsI was gonna leave 2 versions of everything "just in case", but bugger it, I am going to be ruthless.16:20
bigjoolsbzr is used for a reason16:20
gary_poster:-) being ruthless will require being careful, but I admire the idea16:20
gary_posterand now, just after a rollout, is a good time to do it16:21
bigjoolsas long as both db-devel and trunk build locally, I'm happy16:21
bigjoolsgary_poster: here's a weird one - Jinja 2.2 is required in versions.cfg but it doesn't exist16:22
bigjoolsJinja2 even16:23
bigjoolsoh bugger16:23
bigjoolsyes it does :)16:23
gary_posterbigjools, as I said, versions.cfg is just about enforcing version numbers.  presence or absence is a dependency tree based on setup.py declarations.  So it sounds like you can remove it from versions.cfg.  buildout will complain if it doesn16:23
gary_posteroh well there you go :-)16:24
bigjools2.4.1 and 2.5.5 are there though, I wonder why they're not used16:24
gary_postersomeone was unduly optimistic, I'm guessing :-)16:24
=== deryck is now known as deryck[lunch]
bigjoolsvery likekly :)16:25
bigjoolsand other words that are spelled right16:26
sinzuihenninge: should ~rosetta-admins really own translation teams? I see from many teams listed on the +subscribe page that I own https://launchpad.net/gedit-plugins/+subscribe16:32
sinzuihenninge: When I see odd teams like this, they are often because someone made ~registry own a team, which is a no-no. So I either delete the team or make someone else an owner.16:33
henningesinzui: no, I don't think we should be the owner.16:34
henningethey should be owned by someone from that team16:34
sinzuiokay, I will put that on my todo list16:35
sinzuithank your16:35
sinzuithank you!16:35
jmlwho has filed the most bugs that are still open? http://paste.ubuntu.com/578421/ (yeah, I'm easily distractable today)16:36
sinzuibefore I look, I think it is mpt16:37
henningesinzui: you win16:37
LPCIBotProject windmill build #34: STILL FAILING in 1 hr 9 min: https://hudson.wedontsleep.org/job/windmill/34/16:37
=== al-maisan is now known as almaisan-away
rvbasinzui: """mars 08 16:46:20 <sinzui> It is not possible BTW to change a series owner. We removed the field from edit forms""" ... but the /ubuntu/hoary/+reassign is still there (and tested) ... is there something I don't understand or is this wrong?16:49
bigjoolsjcsackett, the difference is that the upload processor is the lp_queue user and the ftp server lp_upload16:53
maxbSo basically we are questioning what exactly getUtility(IGPGHandler).getVerifiedSignatureResilient() verifies16:53
jcsackettbut you earlier had the gpg conf copied from one to the other, right? so our simplest solution is hosed? :-P16:53
bigjoolsjcsackett: lp_upload had no gpg.conf at all16:54
bigjoolswe copied lp_queue's16:54
bigjoolsmaxb: yes16:54
jcsackettokay, i don't see anything obvious in getVerififiedSignature. there's a fair bit of cruft in there XXX notice-wise, but nothing obviously weird.17:01
bigjoolsin lib/lp/archiveuploader/dscfile.py look for its usage in there - it's no different to the ftp server17:02
* bigjools scratches head17:02
adeuringabentley: how can I figure out if a transtion sharing job is pending for a source package?17:02
bigjoolssinzui: can you think of anything that would make IGPGHandler.getVerifiedSignatureResilient return an error in one account but not another?17:03
maxbbigjools: It couldn't possibly be simply a difference in where stderr points in the two processes, could it?17:03
bigjoolsmaxb: maybe, why would that have an affect?17:04
maxbThe only problem here is an unnecessary warning, right?17:04
maxbCould that just be harmlessly disappearing into an ignored logfile in the upload processor?17:04
abentleyadeuring: You'll need to write a new query.17:05
adeuringabentley: ok... so, other question: where are these jobs created? (I need a straing point to look around ;)17:05
adeuring...starting point..17:05
bigjoolsmaxb: the stderr goes to the log in both, AFAIK17:06
abentleyadeuring: Are you sure you don't want me to point out similar queries instead?17:06
adeuringabentley: well, that would be fine too :)17:06
maxbhrm. So I tried running a verify locally in an interactive python interpreter and got none of that output17:07
bigjoolsmaxb: the pastebin output is from dput though17:08
abentleyadeuring: see registry.model.packagingjob.PackagingJob.iterReady() and translations.model.translationpackagingjob.TranslationPackagingJob.iterReady for inspiration.17:08
bigjoolson the server side, it needs to see getVerifiedSignatureResilient throwing exceptions to return errors down the ftp session17:08
adeuringabentley: thanks17:09
maxbbigjools: Oh, I've completely misunderstood the issue :-)17:09
bigjoolsmaxb: yeah :)17:09
abentleyadeuring: But instead of checking whether the jobs are ready_jobs, you want to check whether their status is "pending".17:09
abentleyadeuring: Actually, it's JobStatus.WAITING17:10
maxbbigjools: gah, ok. So we need to trap whatever the exception is that is supposedly bubbling out of PoppyFileWriter.close() and breaking things?17:11
bigjoolsmaxb: yup.17:11
abentleyadeuring: you probably also want to look for running jobs.17:11
maxbAnd that's not being logged anywhere useful at all currently?17:11
bigjoolstrying to get to the logs right now, they're a bit out of date...17:12
sinzuijcsackett: bigjools: the keyserver issue I was looking at was that that keys from keyserver.ubuntu.com were taking 24 hours to get to canonical's keyserver17:12
jcsackettdig. not related.17:12
sinzuirvba, mars was confused at the time. remember that the owner field on series has no power, users think it does so they want to change it. It is really the registrant. Some users think the registrant has power, but it does not. We should never let users edit the historical record of who registered it.17:15
=== beuno is now known as beuno-lunch
rvbasinzui: I'm sorry "mars = march" in french, this was a quote from what you told me yesterday.17:16
sinzuirvba: just in case you ask, the driver or a series is called a release manager. That user can edit the series and can create milestones. The project owner delegates power over the series to the RM so that the user/team can accomplish their task17:16
rvbasinzui: so for the distroseries, productseries, and productrelease it should not be possible to reassign them right?17:17
rvbasinzui: all right ... I'll have a few tests to refactor then ;-) ...17:18
rvbaI meant I'll have _quite_ a few tests to refactor17:19
sinzuirvba: do we have tests that show changing the owner of a series?17:23
rvbasinzui: ./lib/lp/registry/stories/distroseries/xx-reassign-distroseries.txt17:24
* sinzui looks17:25
sinzuirvba: that ancient test can be deleted17:25
rvbasinzui: so I figured if series don't have an owner ;-)17:26
sinzuirvba: a story test is a user acceptance test. The narrative is written from the user perspective and show a simple path to accomplish a task. That test does not state who the user use, what is task is, show how he accomplishes it, and how he knows he is done. When you see stories like this, or testing error conditions, there is a good chance you can delete instead of refactor17:28
jmlallenap: could you please take a look at https://code.launchpad.net/~jml/launchpad/reported-by-me-121646/+merge/5114817:31
jmlallenap: would appreciate someone familiar w/ bugs taking a look at it.17:31
rvbasinzui: I'll have to go someplace now ... but I think I'm ok to continue, I've made the structural changes (interface, model, sql patch) and then I correct things bit by bit as tests fails. Good to know that I can delete tests also ;-).17:31
sinzuiHave a good evening17:32
rvbathanks a lot for your support. I think I'll have something for you to review when you log in tomorrow.17:34
=== deryck[lunch] is now known as deryck
bigjoolssinzui: hi, did you get my email about the ftp server bug?17:39
sinzuibigjools: I did and I subscribed to the bug17:54
bigjoolssinzui: great, not sure out of you and tim who's best suited to fix it17:55
sinzuiwe will talk about it in a few hours17:55
bigjoolssomething is arsed up with the logging which causes gpg verification to fail17:55
bigjoolsjml might be able to help17:55
=== beuno-lunch is now known as beuno
jmlI don't help, I *strategize*19:16
lifelessthats very strategic of you19:16
jmllifeless: hi19:17
jmlthere are a bunch of critical bugs for loggerhead that are fix committed19:17
jmlwhat has to happen to get them fix released?19:18
lifelessI'm not sure19:18
lifelesswe could do what we do with lp19:18
lifelessand say 'once lp has deployed it, we're done'19:18
lifelessencourage the packagers of loggerhead to package trunk19:19
lifelessor we could do what we do with e.g. lazr.restful19:19
lifelessand cut releases regularly19:19
jmllifeless: we have a regular release for lazr.restful?19:21
lifelessjml: we make a release when we fix something19:21
jmloh right.19:21
Ronnieif someone files a private bug to a project where im administrator of, why cant i view the bug?19:22
lifelessRonnie: are you the bug supervisor as well ?19:22
Ronnielifeless: how can i see that?19:22
lifelessits on the front page for the project IIRC19:23
Ronnielifeless: the project-group where im admin of, is maintainer and driver19:24
lifelessRonnie: whats the project group and project in question19:24
jcsackettsinzui: i think you and i got irritated by pqm at the same time.19:24
Ronnieand bug: https://api.staging.launchpad.net/1.0/bugs/72892019:25
Ronniethe bug is created with a script were testing19:25
sinzuijcsackett: have you submitted a fix19:27
lifelessRonnie: https://bugs.staging.launchpad.net/ubuntu-nl-artwork19:28
jcsackettsinzui: not yet submitted, but https://code.launchpad.net/~jcsackett/launchpad/resolve-conflicts has no text conflicts and the conflicted test file passes.19:28
lifelessRonnie: see the bug supervisor is a team, not you19:28
jcsacketti'm not 100% certain it's a good resolution, as i wasn't entirley certain what some of the changes were doing.19:29
lifelessRonnie: have you added yourself to the team ?19:29
Ronniehttps://staging.launchpad.net/~ubuntu-nl-artwork/+members (name=Ronnie)19:29
sinzuijcsackett: if db-devel does not conflict with stable, someone post a fix without says so19:29
Ronnieim also the owner of the team19:30
jmlI'm fixing that too19:30
lifelessRonnie: I know it shows you as an admin, but https://staging.launchpad.net/~ronnie.vd.c/+participation19:30
jcsackettsinzui: no, there were conflicts when i merged stable into a checkout of db-devel.19:30
jmljcsackett: I'll look19:30
jcsackettjml: you mean pqm conflicts?19:30
lifelessRonnie: you may not actually be in the team19:30
jmljcsackett: yeah. I'll take a look at your branch.19:30
lifelessRonnie: can other folk in that team see the bug ?19:30
jcsackettjml: sinzui: okay. it's the garbo file i'm not sure i cleaned up right, but again, test_garbo passes.19:30
jmlI got distracted while make schema was running19:31
jmljcsackett: yeah. I'll check to see if your fixes match mine19:31
jcsackettjml: cool.19:31
jmljcsackett: did you just union the changes?19:31
jcsackettjml: yeah, i just whacked all the conflict gunk.19:31
jcsackettmost naive fix possible, but tests pass, so...19:31
jmljcsackett: ok. that's what I did. And, as you say, the tests pass.19:31
sinzuijcsackett: sorry for misunderstanding. I am struggling to concentrate this week. Too many issues in my head19:31
jmljcsackett: go ahead and land it, r=jml19:32
jcsackettjml: dig.19:32
jcsackettjml: skip ec2?19:32
jmljcsackett: I would.19:32
jmljcsackett: thanks for resolving it, those mails are a pain.19:32
sinzuijcsackett: yes, submit to pqm19:33
sinzuiokay, I will now fix leonardr's bugtask branch19:34
sinzui\o/ no critical bugs19:36
* sinzui opens the champaign.19:36
lifelesssinzui: heh, not yet :)19:36
Ronnielifeless: another member of that team cant access the bug19:37
lifelessthen its probably your script thats at fault19:38
lifelessthe rules for visibility on private bugs are:19:38
lifeless - if you are subscribed, you can see it19:38
lifeless - you aren't, you can't.19:38
lifelessthe default for private bugs filed through the web ui is to subcribe the security team or failing that the bug maintenance team19:39
Ronnienew_bug = launchpad.bugs.createBug(title=mail['Subject'], description=message, private=True, target=forum_project, tags=tags)19:39
Ronnielifeless: thats the line that creates the bug19:40
sinzuilifeless: Ronnie: has the bug supervisor changed? Lp really does not let you change it once set. Changing the bug supervisor will not change the permission on existing bugs. eg, the old supervisor still has access, and the new one does not19:40
sinzuiThis scenario is a common way users shoot themselves in the foot19:40
Ronniesinzui: i didnt change the team settings for months now19:41
Ronnieand were testing the script today19:41
lifelessso first step19:41
lifelessI'l file a bug for you manually19:41
lifelesssee if you can see it19:41
lifelessbug 72892119:42
Ronnielifeless: nope, private19:42
Ronnieow wait, i used the api link, moment19:43
Ronnielifeless: i can see the bug you made19:43
lifelessso its the api made bug not having subscribers19:43
sinzuireally! surely such an issue would have been reported years ago19:48
* sinzui looks19:48
sinzuiRonnie: lifeless bug 398846 says it is low. I think that is insane since you can make it private at the same time19:50
sinzuilifeless: High at least, do you think it is critical19:50
sinzuiI see I glanced at the problem from a permission's sperpective19:51
Ronnieis there some way to create the bug non private, add a bug supervisor and then make the bug private?19:53
lifelessRonnie: the usercode running the script can subscribe the security team19:55
lifelessI've commented on the bug19:57
lifelessyou can probably get the security team from the api19:58
Ronnielifeless: ill try19:58
allenapjcsackett: Do you need a mentored review for jml's branch from earlier?20:04
jcsackettallenap: no, i'm a graduate. :-)20:05
allenapjcsackett: Woohoo :)20:05
jmlallenap: it's already in EC2, but I would appreciate a quick glance if you've got the time20:07
allenapjml: Sure.20:07
jmlallenap: mostly because I'm not familiar with searchTasks and maybe there are booby traps.20:07
lifelessjml: jcsackett: before you land that patch20:11
lifelessare you aware you are going to cause timeouts ?20:11
allenaplifeless: The beauty of that portlet is that few people will notice ;)20:11
lifelessallenap: /everyone/ will notice20:12
jmllifeless: no, I'm not.20:12
lifelessallenap: and we should care deeply about it20:12
jmllifeless: why is it going to cause timeouts?20:13
lifelessjml: because 'reported by me' is an expensive query that times out regularly on Person:+bugs20:13
allenaplifeless: I'm being silly. But in this case, the portlet with links is rendered in the initial request, and the stats are calculated and slotted into place after the fact. So, only the people who read the numbers will notice. But, with my serious hat on, yes, we should care.20:13
lifelessallenap: folk will notice, and think less of LP; folk will notice and ask in #launchpad; we'll have added another bug to the critical pile we're trying to reduce.20:14
jmllifeless: the search never times out20:14
jmllifeless: at least, not for me.20:14
lifelessbug 42190120:15
lifelessjml: you're adding the overhead of the search to the overhead of the bug counts portlet20:15
jmllifeless: it's a different query20:16
Ronnielifeless: that worked:    new_bug.subscribe(person=project.owner_link)20:18
jcsackett\o/ pqm success.20:19
lifelessRonnie: cool20:19
Ronniethx all20:19
lifelessjml: reported by in ubuntu is up to 4 seconds of overhead.20:20
jmlwell that was a huge waste of time20:20
lifelessjml: you've added 14 bugs in ubuntu, so for you the query will be fast20:21
lifelessat the other send of the scale, keybuk has filed nearly 2000 bugs in ubuntu20:21
lifelessjml: I may be wrong20:21
lifelessjml: we will need to qa carefully though - particularly by taking over canonical-scott on qastaging and making sure that the portlet doesn't time out20:22
lifeless bug 71107120:22
lifelessis the existing bug about the portlet timing out20:22
jmltoo late. I've cancelled the branch. I've got too much on at the moment to go through that for what was an opportunistic bug fix.20:22
lifelessjml: I understand20:23
lifelessjml: to help you assess things in future - if you add work to a page, assume its nontrivial.20:23
lifelessjml: that includes adding links to person objects we don't already show, aggregate and non aggregate stats, tables, and branding20:24
lifelessjcsackett: ^ when reviewing - you need to also think about the performance impact of a change : not /really really deeply/ - but just : 'has performance been considered? If not, what could go wrong?'20:26
jcsackettlifeless: yeah, i saw searchTasks and thought about it, but didn't think that was an expensive query.20:27
jcsacketti'll remember next time.20:27
lifelessjcsackett: 4 seconds worst case isn't /terrible/20:27
lifelessjcsackett: but the portlet in the distro context is 10 seconds already.20:27
allenapjml: Is it worth leaving the link in, without the stats?20:27
lifeless10+4 ...20:27
* jcsackett nods.20:27
jcsackettyeah, that's hitting our timeout threshold.20:27
jcsackettnot to mention sort of sucking. :-P20:27
lifelessits a shame that we have things like the portlet which are already so slow20:28
lifelesswe're making progress though20:28
jmlallenap: for me, it would be an incremental improvement over having to type my nick in on the advanced search page every time I want to do it20:29
allenaplifeless: The non-personal bug counts could be cached quite readily, and refreshed out of band.20:30
jmlallenap: but visually, it would look like a bug20:30
jmlallenap: like we forgot to add the count.20:30
lifelessallenap: indeed, as long as we inject them rather than doing on-miss20:30
lifelessallenap: -but- we can make it massively faster just using aggregate queries20:30
lifelessallenap: see what I did for series + tag counts20:30
allenapjml: Yeah, that's a good point.20:30
lifelessallenap: we should make the miss case faster before going out of band: its more efficient for when we do need out of band / denormalisation in the future20:31
jmllifeless: when we lower the timeout, do we automatically add exceptions for the pages that are still over it?20:31
lifelessjml: if we have things go completely out of whack20:32
lifelessjml: we don't default to do doing that20:32
jmllifeless: hmm ok.20:32
jmllifeless: the kernel team remonstrated with me about how our timeout lowering is blocking them from doing critical things20:33
lifelesspossibly https://bugs.launchpad.net/launchpad/+bug/732398 ?20:34
lifelesswhich wgrant is landing a branch for today20:35
jmllifeless: maybe. they are mostly from scripts they run, I gather.20:35
lifelessif they have a particular thing that has /stopped/ working they shuld pop into #launchpad and we'll see what we can do20:35
jmllifeless: they feel quite strongly that it is wrong to stop a fraction of users from doing their work so we can lower our timeouts.20:36
lifelessthat isn't the intention, and if they come talk to us when a particular thing stops working, we'll look and see whats up20:38
lifelessyesterdays oops report shows 531 timeouts20:38
lifelessand 6.3Million non-monitoring web pages served20:38
lifelessthats 0.009% failure rate20:39
jmllifeless: that includes API calls?20:40
jmllifeless: hmm. so either they are very unlucky or were speaking from stale data.20:41
lifelessthe page ids with # in them20:41
lifelessare api calls20:41
lifelessjml: they need to come communicate when the problem is happening. its /cheap/ for us to workaround *if* its a simple 'this is on the edge'20:42
lifelessjml: in the past things have been taking 30+ seconds to complete.20:42
jmllifeless: OK. I'll pass that on.20:42
lifelessjml: which -regretfully- there is no sensible way for us to work around20:42
Ronnielifeless: another question, which characters are allowed for tags?20:42
jmllifeless: they are working on a two week cadence now20:42
lifelessjml: I hold the failure rate below 0.05%20:43
jmllifeless: if they do come to us for help and you aren't around, is it clear what we should do?20:43
lifelessRonnie: lowercase20:43
Ronnielowercase and dash20:43
lifelessjml: the CHR should evaluate the problem and decide what to do20:44
lifelessjml: I don't think there is a cut and dried precanned answer20:44
lifelessjml: in previous discussions with e.g. andy, they have not considered the costs of us leaving the timeout high20:44
jmllifeless: ok. so that's a "no", at least for the moment.20:44
lifelessjml: I have pretty huge overlap with the bulk of the kernel team - them being most us based AFAIK20:45
jmllifeless: yeah. it's understandable. having the timeout high is a cost borne by the commons20:45
lifelessjml: and by then - slower limits = choppier queuing = more latency on their own requests20:45
jmllifeless: yeah, tragedy of the commons.20:48
lifelessjml: I've said in previous mails to the list that I'm happy for any dev to request a timeout exception - and for losas to Just Do Them20:49
lifelessjml: all I ask is that it be <= 20 seconds, and there must be a critical timeout bug associated with it.20:50
jmllifeless: ok, thanks.20:51
lifelessI'm happy for losas to add them off their own bat as well, of course20:51
jmllifeless: fwiw, apw reported that the site feels faster in the morning.20:51
lifelessits very much a 'look after the site' kindof thing - exactly what ops should be doing20:51
lifelessjml: naturally, we deployed nearly a weeks worth of timeout fixes20:51
lifelessjml: its the downtick on http://webnumbr.com/.join(launchpad-oops-bugs.all,launchpad-timeout-bugs.all,launchpad-critical-bugs.all)20:52
jmllifeless: not quite what I meant. During mornings in general.20:52
jmllifeless: before the US awakes.20:52
lifelessthats probably in dc queuing20:52
jmllifeless: that was my guess.20:52
lifelesswe are out of the terrible phase there20:52
lifelessbut still have a ways to go20:52
jml(insofar as anecdata needs explanations)20:52
lifelesswe're bringing up 50 more appserver instances20:52
lifelessprobably not next week, as we're still a losa short20:53
lifelessmaybe even two20:53
jcsackettjml: i officially love the word "anecdata"20:53
jmljcsackett: it's one of my favourites.20:54
lifelessthumper: Revision 12563 can not be deployed: needstesting20:55
thumperlifeless: which one is that?20:55
lifelessStrip any path information off email attachments when storing in the librarian20:56
thumperlifeless: I thought I marked that as untestable20:56
thumperhttps://bugs.launchpad.net/launchpad/+bugs?field.tag=qa-needstesting doesn't show it20:56
lifelessthumper: ah, just recently20:57
* lifeless refreshes the report20:58
thumperlifeless: well... in the last half hour20:58
lifelessthumper: yeah, I'm jus tseeing if we can fix things for oem20:58
thumper9:17 according to the email20:58
sinzuithumper: abentley: I am looking at https://answers.launchpad.net/launchpad/+question/148614 . I have seen this before. I think the svn repo is corrupt and we cannot complete the import. I do not see this kind of problem described in https://help.launchpad.net/VcsImportRequests20:59
thumpersinzui: oh god21:00
thumpersinzui: get them to request another import21:01
thumpersinzui: as this one uses CSCVS21:01
thumpersinzui: bzr-svn is much more robust21:01
thumperthat'll most likely fix it21:01
sinzuioh, that's right. I was surpised to see cscvs but did not think getting a new one was the right thing to do21:01
sinzuithanks thumper21:01
wgrantlifeless: Does garbo-hourly run regularly on qas?21:07
lifelesswgrant: we did start it running, I don't know if its still running21:08
lifelesswgrant: check with losa21:08
abentleysinzui: you can simulate getting a new one using bzr-svn on your local machine, if you like.21:09
sinzuiabentley: good to know21:09
wgrantlifeless: Can you do a quick 'SELECT COUNT(*) FROM sourcepackagerelease WHERE changelog IS NULL' on qas, so we can check?21:10
lifelessnot exactly quick.21:10
wgrantThis seems like it deserves an incident-level response...21:15
LPCIBotProject windmill build #35: STILL FAILING in 1 hr 14 min: https://hudson.wedontsleep.org/job/windmill/35/21:15
wgrantParticularly since we can't fix it today.21:15
wgrantBecause ELOSA.21:15
lifelesswgrant: I agree21:16
wgrantThis should have been a drop-everything situation 8 hours ago :/21:16
wgrantI realise that we have no EU maintenance teams, but...21:16
lifelessbenji: I filled out the deploy report21:18
=== bac` is now known as bac
jmlhuwshimi: hi21:55
huwshimijml: Hey there. Sorry I got distracted :(21:55
jmlhuwshimi: np21:56
jmlhuwshimi: skype?21:56
huwshimijml: yes21:56
jmlhuwshimi: http://pastebin.ubuntu.com/578349/21:59
LPCIBotProject windmill build #36: STILL FAILING in 48 min: https://hudson.wedontsleep.org/job/windmill/36/22:04
wgrantwallyworld: Hi.22:04
wgrantwallyworld: Bug #73244222:05
wgrantThat's the Windmill failure.22:05
wgrantIt's a real bug, but doesn't affect production data.22:05
wgrantwallyworld: I looked at it last night, but it's non-trivial.22:06
wallyworldwgrant: thanks. that was on my todo list to follow up. i'll fix it.22:07
wallyworldgood that the production issue is fixed22:07
thumperwallyworld: hi, mumble?22:17
wallyworldthumper: ok22:17
wallyworldthumper: https://code.launchpad.net/~wallyworld/launchpad/inline-recipe-distro-series-edit22:33
wallyworldthumper: https://code.launchpad.net/~wallyworld/launchpad/inline-recipe-distro-series-edit/+merge/5294022:34
wallyworldthumper: http://people.canonical.com/~ianb/distroseries-checkbox.png22:36
=== jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: allenap | https://code.launchpad.net/launchpad-project/+activereviews
wallyworldthumper: https://code.launchpad.net/~wallyworld/launchpad/inline-multicheckbox-widget/+merge/5294322:45
=== lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews
wallyworldthumper: http://people.canonical.com/~ianb/distroseries-popup.png22:57
lifelesswallyworld: does it need a popup?22:58
lifelesswallyworld: why not just edit in-place?22:58
wallyworldlifeless: because you need to show all the choices22:59
wallyworldlifeless: the html only shows the selected ones22:59
thumperlifeless: just a design decision22:59
lifelesswallyworld: hmm, I guess I'm thinking down the track22:59
thumperlifeless: we aren't using this pattern anywhere else23:00
lifelesswallyworld: e.g. youtubes 'save this video' ui23:00
thumperlifeless: if we decide it is worth a change, we can look at it then23:00
lifelessthumper: of course23:00
thumperlifeless: it should be implemented as a widget, so easy to fix23:00
lifelessbenji: I'm volunteering to be the next 'rm'23:04
lifelesselmo: yes, thats my plan23:07
lifelessgary_poster: with_ is deployed and working.23:16
lifelessgary_poster: we're likely to change the api to make storm upstream happy, but changing a few callsites will be easy enough23:16
thumperlunch is calling23:25

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!