/srv/irclogs.ubuntu.com/2011/09/08/#launchpad-dev.txt

wallyworld_jelmer: i've just noticed the same thing, don't know what causs it :-(00:06
jelmerI can reproduce it on both of my machines00:06
wallyworld_i've just upgraded to oneiric00:06
jelmermy machines are both on oneiric00:06
jelmerstrangely enough the server logs indicate that the client disconnected suddenly00:06
mwhudsonssl error?00:08
jelmermwhudson: yup00:08
mwhudsonwhy do we use ssl on a db connection to localhost? :)00:08
cjwatsonit isn't trying to use SSLv2, is it?00:08
wallyworld_and why has it started failing now? what's changed?00:08
cjwatsonwell, actually, that was turned off in natty00:08
cjwatsonbut OpenSSL 1.0.0 in Oneiric turned off various other rarely-used bits of SSL00:09
cjwatsonthings like MD2 hashing00:09
cjwatsonmight be worth checking that00:09
wallyworld_might explain why lucid hasn't any problems00:10
jelmercjwatson: that's an interesting thought00:11
jelmercjwatson: that's just libssl0.9.8?00:13
cjwatsonjelmer: well, it's libssl1.0.0 now00:16
cjwatsonthere's only a handful of libssl0.9.8 reverse-deps left, and none that I think have anything to do with Launchpad00:16
jelmerit seems unlikely that's the culprit00:21
jelmerI have libssl1.0.0 1.0.0d-2ubuntu2, which was uploaded on aug 15; the breakage started at most two or three days ago00:21
cjwatsonwell, it could have been some dependency switching over to it (they're different sonames), but yes, two or three days ago does make that a bit unlikely00:22
jelmerhmm00:25
jelmerI'll have a closer look tomorrow, thanks for the hints00:26
jelmerwallyworld_: if you do stumble upon a solution.. please let me know :)00:26
StevenKjelmer: Can you explain https://code.launchpad.net/~loggerhead-team/loggerhead/1.18/+merge/74532 ? It's full of conflict markers.00:26
wallyworld_jelmer: will do. i'm currently looking at something else though00:26
jelmerStevenK: argh, fail. I'll set it back to wip for now00:29
jelmerthe trunk *packaging* branch seemed to merge 1.18 just fine for some reason, looks like 1.18 and trunk don't :(00:30
wallyworld_jelmer: i replaced host with hostnossl in pg_hba.conf and it works for now00:43
wgrantOdd, it's working fine for me.00:44
wgrantBut it's in a container.00:45
wallyworld_wgrant: i think lucid is ok00:45
wgrantContainer is up to date, host is not... maybe will upgrade the host too.00:45
wgrantNo, this is an oneiric container.00:45
wallyworld_something seems to have changed wrt ssl that has broken things, so if you upgrade, be careful00:45
wallyworld_gah. my "fix" didn't work a second time00:46
* wgrant upgrades the host.00:47
* StevenK stares at Thunderbird01:04
StevenK"Canonical has 21524 new messages."01:04
StevenKOMGWTFBBQ?!01:04
wallyworld_StevenK: that's in the staging mailbox?01:16
StevenKwallyworld_: No, my Canonical mailbox01:17
wallyworld_oh. that's a shitload of messages then01:17
StevenKIt was telling lies, but it was still a little shocking01:18
G21.5k new messages, ouch01:27
=== wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 253 - 0:[########]:256
LPCIBotProject db-devel build #849: FAILURE in 4 hr 12 min: https://lpci.wedontsleep.org/job/db-devel/849/02:28
wgrantErk02:29
wgrantAh.02:30
wgrantlegit failure.02:30
wgrantMerge issue.02:30
* wgrant enfixorates.02:30
nigelbOh. I may have gotten a branch reviewed but forgotten to ask to land04:20
StevenKnigelb: Which?04:20
nigelbhttps://code.launchpad.net/~nigelbabu/launchpad/font-fixes-787798/+merge/7449204:20
StevenKI wonder if that's totally pointless to run through ec204:21
nigelbIt is.04:22
poolieo/ nigel, stevenk04:22
nigelbMorning poolie!04:22
StevenKO hai04:22
nigelbOooh, the Ubuntu font thing landed. It looks good.04:23
jtvAnd while we're at it: hi Nigel, hi antipodeans!04:23
nigelbhaha04:23
StevenKYeah, we deployed early AU morning04:23
nigelb\o/04:24
jtvIf I may interrupt briefly… is anyone willing to review this soyuzzy branch?  Julian said he would yesterday but evidently he didn't get around to it, and I've got other work building on it: https://code.launchpad.net/~jtv/launchpad/bug-832647/+merge/7438004:25
StevenK% df -h /home | tail -n 104:25
StevenK/dev/mapper/sys-home   46G   42G  2.1G  96% /home04:25
StevenKHmmmm04:25
nigelbwhat's /dev/mapper? lvm?04:26
StevenKYeah, LVM04:27
StevenKWell, strictly, it's device-mapper, so could be LVM or something else than makes use it04:27
poolieStevenK: you're almost done!04:28
StevenKHah04:28
StevenKI still have 75G unallocated in the VG, so I could just grow /home ...04:29
nigelbHow hard is bug 88545. Is it even possible to do for someone without acces to the db?04:32
_mup_Bug #88545: Abolish the "statusexplanation" database field <lp-bugs> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/88545 >04:32
StevenKIt's a DB patch04:34
poolienigelb: you need to make a patch that changes the db but you don't need access to the db to write it04:34
StevenKBut it may take postgres more than 2.5 seconds to perform ...04:34
poolieprobably the first thing is to just grep for statusexplanation and see if anything uses it at all04:34
poolieeg to remove it from the activity log04:34
nigelbAh04:34
nigelbdo I need db-devel setup?04:35
nigelb(that seems to be the branch where all db stuff happens)04:35
StevenKSo to remove it from the activity log, no, that can probably just land.04:36
StevenKYou want to make sure the model and interface no longer reference it04:36
StevenKAnd then you write *just* a db patch to ALTER TABLE DROP COLUMN04:36
nigelbOkay :)04:36
nigelbGraduating to the bugs not marked easy or trivial :D04:37
StevenKThe drop column might perform poorly -- I suggest you talk to stub when he surfaces04:37
nigelbYeah, I think so as well.04:37
StevenKBut the first two steps should be easy enough04:37
pooliei think you get a badge for fixing an mpt bug too :)04:37
poolieif you haven't already04:38
* StevenK fixed a few at the Thunderdome04:38
StevenK'a moment ago' is my fault04:38
pooliei think i have too04:38
nigelbI've fixed way too many mpt bugs already04:39
wgrantStevenK: column dropping is cheap.04:39
wgrantStevenK: The rows don't get rewritten, the column just gets ignored.04:39
wgrantThe next time the row moves, it will be shrunk.04:39
StevenKDoes VACUUM clean it up?04:40
wgrantPossibly not entirely.04:40
wgrantBut it's not important.04:40
wgrantIt only takes a bit of disk space, and it's not going to be a truly massive column.04:41
StevenKIt's still used, so that has to go.04:41
wgrantYes.04:42
jtvStill want a reviewer for a soyuzzy branch!  Blocking me a little and reviewer didn't get around to me yesterday: https://code.launchpad.net/~jtv/launchpad/bug-832647/+merge/7438004:43
wgrantjtv: Interesting tactic to teach the dominator about deletion...04:45
wgrant(looking)04:45
jtvwgrant: "interesting" as in "cleverly unexpected," or "interesting" as in "oh my God what were you thinking"?  :-)04:46
jtvThanks btw.04:46
wgrantA combination.04:46
wgrantIn the Ubuntu case I'd prefer a crash over a deletion.04:46
jtvCrashing shouldn't be hard.04:47
jtvAnyway, that's for later: I left Ubuntu domination in a state where you don't get deletions.04:48
wgrantI'd prefer it was for now. Domination has always been very fragile -- I've spent today diagnosing issues surrounding it, with a production incident caused in part by a bad refactoring of the dominator that I made and fixed last year.04:49
wgrantWith a big change like this, I'd really prefer you stick an assertion in there so breakage doesn't go unnoticed.04:50
jtvWhat would I assert though?  distro_name != 'ubuntu' or pubs[0].sourcepackagerelease.version == pubs[0].sourcepackagerelease.version?04:51
wgrantI don't know. But gina and archivepublisher currently use dominator to do different things, so they should probably run it in different modes.04:52
wgrantThe dominator should never delete anything in a Soyuz-native archive.04:52
wgrantOnly for gina.04:52
wgrantIt will cease to be Ubuntu-specific this year.04:52
jtvWait, now I'm completely confused.  What are you saying is currently Ubuntu-specific?04:53
jtvNot gina, obviously,04:53
jtvand not the dominator, obviously,04:53
jtvso what will cease to be Ubuntu-specific?04:53
wgrantWhen I said "In the Ubuntu case", I really meant "In the Soyuz-native case"04:53
wgrantCurrently Soyuz-native archives are Ubuntu-specific, but they will cease to be soon.04:53
jtvOkay, to prevent further confusion, could you re-state with that mistake fixed?04:54
jtvGina will cease to be specific to Soyuz-native archives this year?04:54
wgrantgina is by definition not used for Soyuz-native archives.04:55
jtvSo what will cease to be specific to Soyuz-native archives this year?04:55
wgrantNothing. Soyuz-native archives will cease to be specific *to Ubuntu*.04:55
wgrantI said before that stuff was specific to Ubuntu.04:55
wgrantBut I really meant specific to Soyuz-native archives.04:55
wgrantIn particular, the dominator should never call requestDeletion on a publication in a Soyuz-native archive.04:56
wgrantIt should instead crash, as that case doesn't make sense.04:56
jtvWell the algorithm I've got here (the part that gina bypasses, so that's the "different mode" you referred to above) is such that the deletion case just can't happen.  It's the what-do-we-do-now answer—the one you wanted, I should add—for the newly possible case in gina only.04:58
jtvHow we may extend it later to make that case possible for the publisher's domination is a later concern.04:58
wgrantSure, it shouldn't be able to happen.04:58
wgrantBut I want the code to ensure that it can't.04:58
wgrantBecause nature finds a way.04:59
wgrantWe've been bitten before by things like supersede() working on a deleted publication. Other code breaks, calls methods in bad ways, and they do things that nobody ever expected.04:59
jtvOkay, how would the assertion check for a Soyuz-native archive?04:59
wgrantdominatePackage will need to know in what mode it is being called.05:00
wgrantBy _dominatePublications or that new thing.05:00
wgrantBecause it should never call requestDeletion when it's called by _dominatePublications.05:01
wgrantNo, I don't know what to call the argument :(05:01
jtvI don't think it makes any sense to pass a mode argument just to enable an assertion that, essentially, the caller also passed the expected value for another argument.05:02
wgrantI guess.05:03
wgrantanyway. dominate_imported_source_packages looks like it's only going to dominate sources that are still named in the source archive's indices.05:03
wgrantSo it won't actually catch deletions, will it?05:03
wgrantBecause they'll no longer be in the indices at all.05:03
jtvThat's for my follow-up branch.05:03
wgrantAh.05:03
jtvI've already got most of the code here; it works out to be quite elegant.05:04
wgrantIndeed, this has turned out pretty well, I think.05:04
jtvEspecially in the way it lets us handle migration.05:04
wgrantYes. So I guess the first run will sort out everything that's not deleted... I think that looks OK.05:04
jtvThe new code iterates over all package names that are published in the series/archive/pocket, and then get()s the list from gina's src_map.05:05
wgrantAh. I was thinking you'd have to union, but of course this is done after everything is imported.05:06
jtv*In the transitional code*, if it's not in there, it finds the latest version (reusing otherwise useful bits of the dominator) and passes that as live_versions.05:06
wgrantSo the DB set is fine, indeed.05:06
wgrantYup.05:06
jtv*Once transition is complete*, I'll remove that code and then the get() will have [] as a default value.05:06
jtvThe really great thing about it is that it requires no code of any real complexity or intimacy, except stuff from the dominator that we can reuse.  Some of it is private, but who cares for explicitly temporary code?05:07
wgrantYep.05:07
StevenKOh, RARGH05:08
StevenKnigelb: Still here?05:08
wgrantI would appreciate a comment in dominatePackage stating that deletion should never happen in native domination, as the live publication is always first. But you have me convinced that the assertion is not required.05:08
StevenKThank you for forgetting my session totally, Firefox05:11
jtvwgrant: I agree with the idea of a comment, but this policy choice falls above dominatePackage's pay grade.  It should be in _dominatePublications, where the choice is made that leads to this.05:12
wgrantI know it's not really appropriate for dominatePackage, but with a conventional understanding of domination the code makes no sense.05:13
jtvwgrant: I'm in the process of writing docstrings, with a comment in _dominatePublications pointing out how the mechanism enforces the policy.05:20
wgrantjtv: Thanks.05:21
wgrantApproved.05:22
jtv\o/05:22
jtvThanks05:22
jtvNow if only ec2 would work for me.  Perhaps I should give it another try.05:22
wgrantWorked fine for me this morning.05:23
StevenKStill confused :-/05:24
StevenKBut Eventbug helped a little05:24
nigelbStevenK: yeah05:31
StevenKnigelb: Compounded by testfix!05:31
StevenKBut your branch has landed05:31
nigelbhaha05:31
nigelbI have great luck!05:31
nigelb\o/05:32
* StevenK ponders asking for buildbot to be set on fire.05:32
nigelbStevenK: need help burning it? :D05:37
StevenKHaha05:38
StevenKI need help with JS, I'm so confused05:38
wallyworld_StevenK: i need to go to soccer soon, but can i help with js?05:40
StevenKwallyworld_: I'm trying to figure out what JS is fired when I hit a link05:41
wallyworld_StevenK: normally you would register an onclick handler05:41
StevenK<a class="sprite modify remove js-action" href="#" style="">unsubscribe from this bug</a>05:41
StevenKNo onclick05:41
wallyworld_StevenK: there is one set up - but it's done in js whe nthe page loads05:42
wallyworld_the href="#" means that an onclick wil be used05:42
nigelbBut doesn't that mean there's no fallback in case of disabled JS?05:42
wallyworld_probably, but we require js for lp now05:43
StevenKcewrapper.fire(Event.getEvent(e, el, compat || false === facade));05:43
* StevenK blinks05:43
wallyworld_StevenK: try looking in bugtask_index.js or bug_subscription_portlet.js05:43
wallyworld_i think one or both of those have methods which run in the onload callback to set up the links05:44
StevenKlib/lp/bugs/javascript/subscription.js looks very related05:45
StevenKBut I have no idea how to find the onclick handler function for the link so I can trace it05:45
wallyworld_StevenK: it's bug_subscription_portlet.js. see it's make_link() method to set the attributes on the link as you posted above05:45
wallyworld_nigelb: the link does have a fallback - the js sets the href="#"05:46
wallyworld_StevenK: let me have a quick look05:46
nigelbwallyworld_: Ah! :)05:46
wallyworld_StevenK: search for link.on('click'05:47
wallyworld_there's a few of those for the mute link, the subscribe link etc05:47
StevenKwallyworld_: The page I'm on is https://bugs.launchpad.dev/product-name-989062/+bug/16/+subscriptions05:47
_mup_Bug #16: "Swedish" and "Swedish (Sweden)" should be the same language <lp-translations> <Launchpad itself:Fix Released by daf> < https://launchpad.net/bugs/16 >05:47
* wallyworld_ looks05:48
StevenKI know that doesn't help you much, but it isn't the bug page05:48
wgrantOh, that's not the subscriptions portlet.05:48
StevenKExactly05:48
StevenKAnd Firebug's script tab isn't being very handy about pointing out where the JS is hiding05:49
wallyworld_StevenK: i don't think you would normally get to that page except by url hacking05:50
wallyworld_but it is the subscriptions portlet i think, put not rendered in a portlet05:50
StevenKHuh? I followed a link to get to that page!05:50
StevenKThe "Edit bug mail" link05:51
wallyworld_StevenK: oh ok. anyway see the bug-subsctipion-list.pt05:51
wallyworld_it loads the bug_subscription_list.js05:52
wallyworld_which is the same js as used for the subscription portlet05:52
wallyworld_StevenK: i have to run to take the kid to soccer but will check back in 30 minutes or so05:52
StevenKOh god, all of the JS is in launchpad.js, isn't it06:03
StevenKAll 100,000 lines of it06:04
lifelessisn't that a build product?06:04
nigelbStevenK: yup06:04
StevenKYes, but it's the JS the browser sees06:04
StevenKWhereas the JS we edit is split out06:04
nigelbIts fun using firebug.06:04
nigelbStevenK: found it yet? :)06:23
nigelbwgrant / StevenK: Do you both have HE tunnels?06:29
wgrantWe use AARNet's tunnel broker.06:31
wgrantIt's a little closer :)06:31
nigelbheh06:36
StevenKI'm quite happy with it06:40
* ajmitch wonders when lp & *.ubuntu.com can be accessible on v606:40
wallyworld_StevenK: how's the js stuff?06:41
StevenKwallyworld_: Crappy06:41
StevenKGiving up for today06:41
wallyworld_need any more help?06:41
wallyworld_ok06:41
StevenKwallyworld_: I might grab you inappropiately after the stand-up06:41
wallyworld_oooh. promises, promises06:41
nigelbBWAHAHA06:41
nigelbShould I or should I not screenshot.06:42
jtvnigelb: the way I see it, this could be your retirement money.  And then again, it could be your reason for not reaching retirement age.06:50
nigelbjtv: I would much rather prefer using it in a presentation. So, I'm leaning to the second option.06:51
jtv“Not screenshot”?06:52
nigelbScreenshot and use in a presentation06:52
nigelbwgrant: do you have a minute to clear up a confusion in the ipv6 wiki page?06:52
nigelbIt says "gksudo gedit /etc/networking/interfaces", in lucid its /etc/network/interfaces. On oneiric do you have networking instead?06:53
jtvIIRC it's always been /etc/networking/interfaces06:54
jtv(and by "always" I mean "going back to Debian a decade ago"—I don't know much more than that really)06:54
wgrantIt's always been /etc/network/interfaces06:54
wgrantAs long as I've been using Debian, which is like 2002.06:55
jtvAh.  Tab completion has spoiled me.06:55
jtvAnd somehow admins who don't seem to mind this keep refusing to set up tab password completion.  It's not fair.06:55
wgrantHeh06:55
nigelbok, so the wiki is lying.06:56
jtvBTW my having been using Debian for longer than wgrant comes as a bit of a shock.06:56
jtvMy using Debian for longer than wgrant can walk, that I would have believed.  But this…06:56
nigelbwell, he should be old enough to read before he uses a computer...06:56
jtvGood point.06:57
wgrantI sadly used Windows until 2001 :(06:58
nigelbNo buffer space available?06:58
nigelbGAH. No ipv6 yet for me.06:58
nigelbwgrant: I used Windows until 2007. Windows Vista for a year too (yeah, *stab* *stab* *stab*)06:59
nigelbLunch time! Laters folks.07:04
wgrantHmm, no stub yet :(07:13
wgrantMorning stub.07:16
stubMorning07:16
wgrantstub: We snuck a librarian deployment through without downtime earlier, so everywhere is 13891 except germanium's poppy and cocoplum.07:17
wgrantHopefully that is all the fixes we need.07:17
stubIt is all the fixes we know we need in any way :-)07:18
wgrantRight.07:18
stubSo if mthaddon has time when he gets here or spm is free, we can kick the tires.07:19
wgrantstub: My zopeless destruction branches get rejected by the devel database/schema check :(07:20
stubWith luck you can land it on devel later today. Tests pass?07:20
wgrantYup, all three went through ec2 overnight.07:20
wgrantAre we going to lift the restriction?07:21
wgrantSince otherwise we're going to be doing a lot of manual merging.07:21
jtvwgrant: I just realized we never did make Gina create Published publication records.  Still all Pending.  So need to change the code for that as well, _and_ update the database.07:21
stubI like having it as a safety net, but I don't see the point of having be so vigorously enforced.07:21
wgrantjtv: We do, but it's not critical.07:22
wgrantjtv: Since your new dominator method finds everything that's active.07:22
wgrantNot just published.07:22
jtvwgrant: errrr07:22
jtvI stuck to the logic that was already there, which looked for published.07:22
wgrant+ self._composeActiveSourcePubsCondition(distroseries, pocket))07:23
wgrantIs that active, or published?07:23
jtvIt's published.07:23
wgrantHeh.07:23
jtvThe dominator tends to say "active" when it means "published."  :(07:23
jtvBut let me just double-check, because we had the confusion in gina before.07:24
wgrantstub: So, I guess we should actually try it tonight...07:24
jtvYup, published.  Not active.07:24
stubwgrant: now if I can sucker hloeung into doing it07:24
* hloeung logs off... right.... now.... 07:25
rvbaStevenK: Hi, I'd like to ask you a question about the creator field of spph.07:28
rvbaIt was introduced by a branch you created to fix bug #684101.07:29
_mup_Bug #684101: Add an ancestor column to SPPH <derivation> <lp-soyuz> <qa-untestable> <Launchpad itself:Fix Released by stevenk> < https://launchpad.net/bugs/684101 >07:29
wgrantrvba: It and ancestor were never used.07:29
rvbaIndeed, Julian an I took a look at the db and it's None. (besides, copyTo sets it to None)07:30
rvbawgrant: StevenK I'd like to use it for real to track copies of spph. Any objection?07:31
wgrantThat was the intention of creator, so do it!07:32
wgrantIt was added alongside ancestor mostly because it was convenient.07:32
wgrantAnd ancestor then became redundant; we examine changelogs instead.07:32
rvbaOkay great, this will save me a db patch!07:32
rvbawgrant: Thanks.07:33
wgrantrvba: Not that that is a problem.07:33
lifelessstub: wgrant: landing schema changes on devel? I'd suggest needing a [schema] tag in the commit, but perhaps that wouldn't actually help.07:33
wgrantlifeless: Possibly, yes.07:33
wgrantlifeless: If you're not watching #-ops, we're about to do the first test.07:33
lifelessI'm not here :)07:33
lifelesssafety net off, free falling :)07:34
lifelesswgrant: someone should tweet :)07:35
wgrantOnce we're confirmed that it's going ahead, yep.07:35
stublifeless: something like that, yer. Although I hate overloading the commit message with metadata.07:35
lifelesswill need a pqm patch if we want it, but htats not as scary as some folk think :)07:37
adeuringgood morning07:38
LPCIBotYippie, build fixed!07:42
LPCIBotProject db-devel build #850: FIXED in 4 hr 36 min: https://lpci.wedontsleep.org/job/db-devel/850/07:42
=== rvba changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rvba* (jtv) | Critical bugs: 253 - 0:[########]:256
rvbaFirst day as OCR, allenap is off today so jtv has kindly accepted to be my mentor today.07:45
jtv“Soft target.  Fire away!”07:46
rvba:)07:46
wgrantjtv: That is a very long XXX comment!07:50
wgrantjtv: Are you going to land this before we do the code+data fixes for Pending->Published?07:51
jtvwgrant: probably, unless it turns out to be inconvenient w.r.t. how we update the legacy data.07:52
jtvAs long as Gina doesn't get to observe the Pending→Published change in an unnatural order, it shouldn't make much difference AFAICS.07:53
wgrantIndeed.07:53
jtvAbout the worst I can imagine for that scenario is intermediate, but more-or-less-accurate, supersededby links.07:53
wgrantjtv: Why are you using _sortPackages instead of sorted(..., cmp=generalization.compare), as you did in dominatePackage?07:54
wgrantYou should only have SPPHs for a single name, so _sortPackages isn't needed.07:54
jtvJust because it gets a little less intimate with the data.07:54
jtvMatter of taste, really, no more.07:55
wgrantk07:55
wgrantApprovalated.07:56
jtvGreat, thanks.  We're really going through them at a clip now.07:56
jtvWhich is good, because I once again gave Julian a "done by the end of this week" estimate.07:56
jtvHmm I just realized though: with these changes, the new source file I added to gina ought to be called dominate.py, not retire.py07:57
wgrantProbably.07:58
jtvAnd done.07:59
* mpt awards nigelb his "fixed an MPT bug" badge: (╯°^°)╯︵{FAMPTB}08:22
mrevellheh08:25
=== almaisan-away is now known as al-maisan
nigelbhaha08:31
nigelbI should get that on a t-shirt.08:31
nigelbNext UDS I attend, I'll do that :D08:31
Grvba: welcome to reviewing :)08:34
rvbaG: ;)08:35
Ghere have a look at this patch that removes all the code.... ;)08:35
nigelbG: That sounds like StevenK's branches :P08:38
nigelbstub: Hi08:40
Gnigelb: haha sounds about right :)08:40
nigelbstub: I was asked to run bug 88545 by you. Specifically the bit that requires a db patch08:41
_mup_Bug #88545: Abolish the "statusexplanation" database field <lp-bugs> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/88545 >08:41
stubnigelb: I can't look right now - deployment stuff underway. Should be able to check it out within the hour though.08:42
nigelbstub: sounds good08:42
cjwatson[6~[6~[6~[6~[5~/wg 6208:53
cjwatsonGAH08:53
nigelbheh08:53
nigelbGAH. Bad thing about working from home - telemarketeers knocking on the door.09:14
wgrantTelemarketers... on the door?09:15
nigelbWell, wrong word.09:15
Gwgrant: I was just going to make that point09:15
nigelbThe door-to-door salesmen09:15
Gbut it was pretty obvious that we all knew what the other Nigel meant :)09:15
ajmitchparasites?09:15
nigelbI guess you could call them that.09:16
Gone advantage of living up a 100m driveway and all that, about the only door-to-door thing we get is various churches/religous ones, and the occasional one that does things like door-to-door sales of aerial photography etc09:19
bigjoolsSame here!09:24
bigjoolslast time I had some religious dudes show up, I squirted them with "holy water" and brandished garlic09:25
lifelessttp://hawkersnotice.com/09:26
nigelbbigjools: heh09:27
nigelblifeless: Its not linked! thedoghousediaries.com/?p=300309:27
nigelbOooh,that is good stuff.09:28
nigelbAmazon has "11.6 seconds mean time between deployments"... what in the world.09:29
lifelesslots of focused services09:29
nigelbIs that where the service oriented architecture is leading Launchpad to?09:30
lifelessyes09:30
nigelbexcellent!09:30
lifelessThe goal is that all commits to trunks are deployed on their own, without batching, and that unrelated things are separate services with narrow contracts09:31
stubOr it just means serialized deployments to lots and lots of hosts...09:32
nigelbunrelated things?09:32
lifelessnigelb: librarian vs codehosting vs group management vs build farm vs ...09:33
nigelbahhh, right09:34
nigelbHow does lp deploy now? I know its batched etc. But is it a click to deploy?09:35
lifelessnot quite but close09:35
jtvAny reviewers in the house?  https://code.launchpad.net/~jtv/launchpad/bug-844577/+merge/74568  ← bigjools maybe, since you didn't get around to my bigger branch yesterday?09:38
nigelbThe reason I ask is, would making each commit deployed on their own cause more work than now?09:39
bigjoolsjtv: sorry, gotta sort out a prod issue09:39
jtvAh of course, sorry,09:39
jtv—kept that in a different context09:39
jtvAnyone else?  https://code.launchpad.net/~jtv/launchpad/bug-844577/+merge/7456809:40
lifelessnigelb: it always will09:41
lifelessnigelb: the question is, does the improved accuracy in dealing with fallout and omgs gain more than the cost.09:41
nigelbOkay, yeah. That's a good point.09:42
nigelbAlso, fail whale for Launchpad would be nice ;)09:43
ajmitchLP shouldn't need it :)09:43
lifelessnigelb: patch it up09:43
lifelessnigelb: I'd love a nicer OOPS page09:43
ajmitcha rocket exploding?09:43
lifelessnigelb: and OOPS for ajax requests09:43
nigelblifeless: Oh, OOPS for javascript?09:44
nigelbHave you seen this -> errorception.com09:44
lifelessnigelb: ajax get OOPS ids in the header of the reply09:44
nigelbhttp://errorception.com <-- clickable.09:44
lifelessnigelb: but the error is shown nastily to the user, not tastefully09:44
nigelbThat would be nice to improve, yes.09:45
nigelbajmitch: Rocket exploding OOPS page would be nice.09:45
ajmitchnigelb: that'd just be an artwork change09:46
lifelessnigelb: u1 has a form that can be POSTed to by javascript that fails.09:46
lifelessnigelb: we just need the same for LP, rather than a 3rd party service.09:46
nigelblifeless: I'm guessing the code for the u1 page isn't open source?09:46
lifelessnigelb: we can't use a 3rd party service without some very careful legal stuff, because we have non-public data in js contexts.09:46
lifelessnigelb: no, but the js side of it is visible :) its a django stack not zope09:47
nigelbEvery page has a form that javascript posts to?09:47
nigelbIn caes of errors..09:48
lifelessno there is one form for all errors09:48
nigelbAh.09:48
nigelbhrm, how do I force an error from u1.09:48
ajmitchthis openid thing is puzzling me now09:48
lifelessask rockstar about it :)09:48
nigelbDoes u1 web team have a different channel or all in one channel.09:49
nigelbI shall poke rockstar at a more favorable time, when he's awake.09:49
lifelesswell, not on freenode :)09:49
nigelbGAH.09:49
nigelboh. I haven't looked at Ubuntu one website in a while.09:49
nigelbIts...wow09:50
nigelblifeless: Nope, my firebug skills don't show me anything. Need to ask rockstar.09:54
bigjoolswgrant: we're getting rabbitmq 2.6.0 in oneiric \o/09:56
bigjoolsit's got HA clustering09:56
wgrantbigjools: Oneiric? Nice.09:56
wgrantWell, yes.09:56
wgrantIt has active-active.09:56
wgrantIt requires some restrictions, though.09:57
bigjoolsI'd like to say it was my doing, but it's a requirement for OpenStack09:57
wgrantbigjools: It's not quite a trivial update, as the PID handling has changed. But it's like 5 lines of initscript that need adjusting.09:57
wgrantAhh.09:57
wgrantbigjools: I guess I should update the plugins for 2.6.0 at some point, then.09:58
bigjoolswgrant: yes please ;)09:58
nigelbwe don't yet use rabbit heavily, do we?09:59
nigelbI thought the only place I saw it was the new sync stuff09:59
wgrantnigelb: We don't use it at all yet.09:59
wgrantThe first use is MP diff updating.09:59
nigelbOH YES09:59
nigelbPLEASE09:59
wgrantIt is implemented.09:59
wgrantJust not deployed.09:59
nigelbAh. That will make a lot of people very happy.10:00
bigjoolscoming soon to a Launchpad near you10:01
wgrant"soon"10:01
bigjoolswe start work on it next week10:01
bigjools2 weeks will be enough10:01
nigelbOh, btw. What does the tag bug-columns mean?10:01
wgrantnigelb: That's the next feature.10:01
wgrantnigelb: Customisable bug listings.10:01
wgrantnigelb: So you can eg. show assignee in search results.10:02
=== al-maisan is now known as almaisan-away
nigelb\o/10:02
ajmitchthat sounds useful10:02
bigjoolsonce that's done I want to replace crontabs with rabbit queues10:02
wgrantbigjools: Yes.10:02
wgrantJob runner daemons.10:02
nigelbbigjools: YES!10:02
wgrantConsuming immediately from rabbit.10:02
wgrantIt will be glorious.10:02
wgrantGLORIOUS10:02
bigjoolsyes and yes10:02
ajmitchrabbit allows for scheduling at a specific time, or will you just run tasks as available?10:02
bigjoolswhich is something I was doing 18 years ago in a former job, but meh10:02
stubchaos! glorious CHAOS!10:02
nigelbajmitch: as available.10:03
bigjoolsit's to reduce latency and load spiking10:03
ajmitchso karma updates would happen during the day rather than all at once? :)10:03
bigjoolswe need a way to specify cumulative updates though10:03
bigjoolsajmitch: we have many, many cron jobs. one at a time :)10:04
ajmitcheveryone knows that LP is all about the karma :)10:04
nigelbkim0 tweeted this the other day "Ask a sysadmin about something and he'll tell you there's a cron for that"10:05
nigelbI can totally believe Launchpad having many, many, many crons.10:06
lifeless> 20010:07
nigelbAchivement Unlocked - I just rocketfuel-branch'd a branch starting with destroy.10:07
nigelbajmitch: Personally, I think we should have badges as well.10:07
ajmitchhah10:07
nigelbLike what stackoverflow does.10:07
ajmitchnigelb: certainly, I've collected a few on askubuntu10:08
bigjoolsI think we need a penis icon that gets longer and longer10:08
nigelbheh10:08
ajmitchbigjools: where would you fit it in the UI?10:08
lifelessajmitch: tell users they need multiple monitors.10:09
ajmitchheh10:09
bigjoolsyou don't need to, you just lie about its size10:09
nigelbI should suggest the badges idea at some point. But there stuff I'd like to see before that.10:10
nigelbLike instant gratification for buying commercial subscriptions10:10
lifelessyes10:10
ajmitchthen you have people doing things for the sake of badges, like commenting on 50 bugs10:10
lifelessfwiw I think gamification has to be done ----very----- carefully10:10
lifelesspathological incentives are terrible.10:10
lifelesswe already have badges for team memberships after all10:11
lifelessand have seen folk making farms of teams with badges10:11
nigelbajmitch: Right. It needs to be done carefully. There already are people joining teams for bade collection..10:11
nigelberr, what lifeless said.10:11
nigelbThere are also people creating blueprints and answering questionst for karma :/10:13
* ajmitch has got no further with being able to log in to launchpad.dev, sadly :)10:14
bigjoolsget any reviews yet rvba?10:22
rvbabigjools: nothing yet.10:22
nigelbHrm, is there a reason why on my ~ page only bugs assigned to me *and* In Progress are shows?10:46
nigelb*shown10:46
nigelbIdeally, I'd like to see everything that's valid and not closed.10:47
wgrantIt's the "Working on" section.10:59
wgrant/~/+assignedbugs shows everything assigned.11:00
nigelbaha11:00
bigjoolswgrant: ok finished reading that page11:12
bigjoolsfinally11:12
bigjoolswgrant: interesting analysis, although I am not sure why you marked two of the rows as "bad" in the last query output.11:13
nigelbbigjools: You'll love this - "BCCI refuse sponsorship offer from Branson-We can't have VIRGIN written on our shirts, when we're getting screwed in every match in England"11:15
bigjools:D11:16
=== ccxCZ_ is now known as ccxCZ
wgrantbigjools: Those are the two that expired the 53 binaries.11:19
bigjoolshow do you know?11:19
wgrantJust above that I have a list of the LFAs with their expiry dates.11:20
wgrantLet me open that page.11:20
bigjoolsok11:20
bigjoolsthere are some leaps of logic that are hard to follow11:20
bigjoolslet's go through this11:21
bigjoolsmight be easier on a call11:21
bigjoolss/might/will be/11:21
* wgrant finds headset.11:22
bigjoolsskype is better, mumble gets on my nerves11:22
bigjoolsor voip for purists11:23
wgrantI only have mumble set up... let's see if Skype likes oneiric.11:23
stubnigelb: So what did you need to know about https://bugs.launchpad.net/launchpad/+bug/88545 ?11:24
_mup_Bug #88545: Abolish the "statusexplanation" database field <lp-bugs> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/88545 >11:24
stubnigelb: It needs to be done in two steps. First is purging that column from our code. Once that is landed and deployed everywhere, we then land a db patch to drop the actual column.11:25
nigelbstub: StevenK asked me to check with you about the expensiveness of the db patch11:26
stubDropping the column is not expensive.11:26
nigelbExcellent, I'll go ahead with a destroy branch11:27
stubPG doesn't rewrite the table or anything - the column is just flagged 'removed' and new rows won't store the data but old rows keep the crap until we recompact the table.11:27
nigelbthat's the bit that'll be expensive?11:28
stubits the bit we don't bother with, as it will happen eventually with PG upgrades and similar.11:30
nigelbcool, ok :)11:30
nigelbOh, and how does one write a db patch?11:30
stubWe do it if we really need the space or if things are performing badly because of the waste11:30
wgrantbigjools: Skype seems to work.11:31
wgrantbigjools: Ready when you are.11:31
bigjoolsok11:31
bigjoolsopne mo11:31
bigjoolsone11:31
bigjoolsjust looking at jtv's patch now!!!11:31
stubnigelb: There are guides on the wiki. You branch from lp:launchpad/db-devel and add a .sql script to database/schema similar to the others in there. The file name requires a number which someone will need to allocate for you.11:31
nigelbstub: okay, I'll worry about it when I land the code changes :)11:32
=== almaisan-away is now known as al-maisan
stubnigelb: Cool. I don't think we need to worry about waiting for deployment between landing the code changes on launchpad/devel and the db changes on launchpad/db-devel. We do need to wait for those code changes to have been merged into db-devel though.11:36
=== jtv1 is now known as jtv
nigelbok!11:41
StevenKnigelb: IE, it will hit db-devel pretty much after it hits stable11:42
nigelbStevenK: yeah, I understand now :)11:51
* nigelb is waiting for EOD to start working on this.11:52
rvbastub: Hi, when you get a chance, could you please have a look at this mp: https://code.launchpad.net/~rvb/launchpad/sync-bug-827608-spph-creator/+merge/7457111:55
rvbaIt's only huge because of the changes in database/sampledata11:55
rvbastub: I'm wondering if CONCURRENTLY should be used for the index creation.11:56
jtvStill no reviewers in the house?  Need one for https://code.launchpad.net/~jtv/launchpad/bug-844577/+merge/7456812:13
jtvrvba: I won't be sticking around much longer though.12:13
jtvjcsackett: are you reviewing today?12:13
bacjtv: i can look at it in an hour or so if jcsackett isn't around today12:14
jtvbac: that'd be great, thanks.  I sort of promised this stuff would finally be done this week, and I'd be ecstatic to be able to land it tomorrow.12:14
rvbajtv: I'll let jcsackett handle the reviews when you're gone.12:15
bacjtv: well let's go for ecstasy then12:15
jtvrvba: OK, better take me off the topic now though.  :)12:15
jtvbac: an oft-heard phrase in Amsterdam, actually.12:15
jtvSort of like "let's go for a beer."12:15
bacrvba: great that you're reviewing now!12:15
rvbabac: Well, I must say this first reviewing session has been quiet ;)12:17
=== rvba changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: rvba* (?) | Critical bugs: 253 - 0:[########]:256
bacrvba: that happens.  much preferred to the days when we always had a huge backlog12:18
rvbajtv: I'll find someone to mentor my review if anything comes up before jcsackett takes over.12:19
jtvOK12:19
rvbajtv: Thanks a lot for your support.12:19
jtvrvba: well you haven't needed any!12:20
rvbajtv: Indeed but I appreciate the gesture anyway ;)12:20
jtvYou and your Gallic chivalry.  :)12:21
rvbahehe12:21
bigjools\o\12:25
bigjoolswe need a smiley for a Gallic shrug12:26
rvbaOk, that was a Gallic shug then ;)12:26
bigjoolsnot intentionally :)12:27
jtvI suppose a Gallic shrug would be something like " M "12:31
bigjoolsthat just looks like someone doing air quotes12:31
jtvI mean M12:31
jtvbut I quoted it.12:31
bigjools^O^12:31
jtvThe shruggie was meant to be the bit between the quotes.12:32
jtvThe ^O^ does have a certain gastonic quality about it.12:37
bigjoolsjtv: it could also be someone turning into a bat12:41
jtvhmm yes, happens all the time so people would get confused12:42
* bac begins jtv's review12:54
jtvthanks bac!12:54
* bac sees mention of Gina and has instant regret12:54
nigelbhaha12:55
rvbabac: This review was a trap!12:55
nigelbNicely played ;)12:55
stubrvba: No CONCURRENTLY in DB patches ('make schema' will fail in any case). We add that in manually if we apply the patch live.12:59
rvbastub: Okay.12:59
jmlI'm starting to work on something that's going to need a db patch.13:00
jmlShould I still start from db-stable?13:00
stubrvba: Its not 2010 either, so the copyright in the patch is incorrect.13:01
rvbastub: Arg ... sorry about that.13:01
rvbaFixed.13:02
stubrvba: Do you want me to allocate a patch number or are you happy doing that yourself?13:02
rvbastub: I'm happy to do it ... could you just point me out the right direction?13:03
stublp:~launchpad/+junk/dbpatches13:03
rvbaOkay.13:04
wgrantjml: db-stable is probably. But unless something goes really wrong, we'll be merging db-stable into devel for the first time in 2 months tomorrow.13:06
wgrantdb-stable is probably best, that is.13:06
jmlwgrant: thanks. are there any differences to what I should do now to what I would have done two years ago?13:07
wgrantjml: DB patches and code must be in separate branches.13:07
jmlI looked on the wiki, but couldn't find anything.13:07
bacjtv: r=bac13:07
wgrantjml: We must be able to deploy the DB patch without the code.13:07
jmlwgrant: and the code without the db patch?13:07
jtvThanks again bac13:07
wgrantjml: That's not really defined yet.13:07
wgrantjml: But at the moment, no, the code usually depends on the DB patch.13:08
wgrantSo you land DB patch to db-devel, then code to db-devel separately.13:08
wgrantWe apply the patches, then merge db-stable to devel, and deploy the code.13:08
wgrantjml: It's maintained by buildout, so it should start out empty.13:34
jmlhmm.13:34
wgrantmkdir ~/launchpad/lp-sourcedeps/yui and symlink it into devel, I guess.13:34
wgrantI wonder if the upgrade code was dropped at some point.13:35
jmlOK. I think it's actually created by buildout. So my *real* question is, why does link-external-sourcecode link it13:35
wgrantBut it's only like 2 months old...13:35
nigelbAH, rf-get is what got it for me.13:35
wgrantjml: Ah, so it probably only ends up in lp-sourcedeps in new installations.13:35
wgrantjml: Same reason we share eggs/: caching for speed.13:35
wgrantAlthough yui isn't too expensive yet.13:35
jmlOh, I see. And it can't live in eggs/ because it's not Python.13:35
=== jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett, rvba* (?) | Critical bugs: 253 - 0:[########]:256
rvbajcsackett: Hi, I think I'll leave you the floor since I'm mentor less atm ;)13:35
jcsackettrvba: sounds fine. :-)13:35
=== rvba changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett | Critical bugs: 253 - 0:[########]:256
nigelbyay, EOD. Now to work on the destroy branch.13:36
jcsackettrvba: how did your first day of review go?13:36
rvbajcsackett: 0 reviews asked, 0 reviews done ;)13:36
* jcsackett laughs13:36
jcsackettyeah, that happens from time to time.13:36
rvbaThe only branches up for review are mine.13:37
jcsackettrvba: soon, you will have a day where 17 reviews hit the queue at the same time. frequently as you're heading to lunch. :-)13:37
rvbahehe13:37
nigelbLike wating for a bus? ;)13:37
rvbajcsackett: one of my branches seems huge but really this is only because of the mecanical modifications to database/sampledata13:41
jcsackettrvba: looking at it now. thought it was already reviewed but i see that was just db approval.13:42
rvbaTrue.13:42
jcsackettthe other one looks fine, assuming of course its prereq is alright. :-)13:42
rvbaThe big one is the prereq.13:43
wgrantjml: is it behaving now?13:43
jmlwgrant: yep13:43
wgrantGreat.13:43
gary_posterhenninge, none afaik (sorry, was doing car stuff)13:43
jmlunittest2?13:43
jmlreally?13:43
wgrantjml: I think that's for Chameleon.13:44
wgrantChameleon seems to be a bit special in far too many ways.13:45
jmlOK.13:45
wgrantDon't worry, we're not using it :)13:45
wgrantit == unittest213:45
jmlgood good :)13:46
nigelbwhat's Chameleon?13:46
wgrantA templating engine.13:46
wgrantImplements TAL and some other stuff, I believe.13:46
wgrantWe use it on a page or two.13:46
gary_posterjml, why it would be bad if we were?  It's stdlib 2.7, yeah?  Don't know much beyond that13:46
jmlgary_poster: prejudice.13:47
gary_posterjml :-P13:47
nigelbwgrant: A page or two? Isn't tal everywhere?13:47
nigelbOr was that irony I failed to catch :P13:47
gary_posternigelb, Chameleon is an optimized engine13:47
gary_posterwe use a legacy engine everywhere13:47
wgrantWe use another TAL interpreter for the rest of the site.13:47
wgrantRight.13:47
gary_posterthere's a LEP to switch the rest13:47
nigelbAHHHH.13:47
gary_posterIt is supposed to save 10-20% processing across the board, IIRC.  It's been awhile since I did the research and tests13:48
bigjoolswhat we lose in render time, we gain in cronspam!13:51
nigelbI was just about to ask why we didn't do that.13:52
gary_poster:-P13:52
gary_posterbigjools, there's an RT for that.13:53
bigjoolsWe're just like an alternative Apple app store13:54
gary_poster:-)13:54
jcsackettrvba: code in the big branch looks fine. one curiousity though. why remove the comments on columns like potemplate? i don't see anything wrong with doing so, just curious if that was cleanup or somehow within the scope of your changes.13:54
gary_posternigelb, because it is non-trivial work.  See Gotchas and schedule at bottom of https://dev.launchpad.net/LEP/Chameleon That analysis itself has been overtaken by events since it was written.13:57
gary_posterFor instance, on the bright side, maybe some of the Gotchas are no longer a problem because the newer version of Chameleon has greater compatibility.  On the not-so-bright side, we would need to rewrite the memcache tal plugin again, or decide conclusively to rip it out as failed; and do a reanalysis of What Breaks Now.13:57
nigelbfun.13:58
gary_posteryeah13:59
henningegary_poster: ok, I'll mark the bug as qa-ok then. (Sorry, was otp)14:01
rvbajcsackett: no, that's not within the scope of my branch, let me have a look at that.14:01
henningeor rather qa-untestable?14:02
jcsackettrvba: like i said, it's not a big deal. the comments all basically are "this is the $var" where $var is the name of the column *anyway*.14:02
gary_posterhenninge, np, I am about to go on phone, but I was going to try and put a branch in junk for the heck of it as a smoketest.  If you can do that it would be great; if not I'll do it after I get off call14:02
henningegary_poster: yeah, I'll do that.14:02
jcsackettso removing them has not materially changed the documentation. :-P14:02
gary_posterthank you very mucj henninge14:03
henningenp ;)14:03
jmlCan someone please give me a patch number?14:10
jmlhttps://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess#Patch%20ids says I need one before continuing.14:11
rvbajml: I'm just modifing the patch number file.  If you give me a comment I'll do it for you.14:12
jmlrvba: what sort of comment?14:12
rvbajml: like what the patch does.14:12
jmlrvba: "Add tables for a package symbol database"14:13
rvbajml: Okay sir, here is your patch number: 2208-83-1 ;)14:13
jmlrvba: thank you.14:13
rvbanp14:13
rvbajcsackett: I've fixed the comment removal. That was due to an unfortunate manipulation.14:16
jcsackettrvba: ah, cool stuff.14:16
rvbajcsackett: BTW I also fixed the patch number, it was 2208-82-0 and I changed it to 2208-82-114:16
jcsackettrvba: ok.14:16
jcsackettrvba: looks good, branch is r=me.14:17
rvba(the reason I did that is explained in the last paragraph of http://bazaar.launchpad.net/~launchpad/+junk/dbpatches/view/head:/allocated.txt)14:17
rvbajcsackett: cool, thanks.14:17
abentleygary_poster: Do you know what buildout does with binaries?  I've added Celery to buildout, but I can't find celeryd anywhere.14:19
nigelbTo delete a db field, I need to remove all instances that the field is being used, right?14:20
gary_posterabentley, on call, but look in egg in EGG-INFO.14:20
gary_posterabentley, you can also put a binary in bin.  There are a couple of ways, but most standard is to use entry_points in setup.py14:26
gary_posteryou can see examples there now14:27
abentleygary_poster: It seems to have entry points already: http://pastebin.ubuntu.com/685317/14:27
henningeShould code hosting on qastaging be expected to behave differently from production?14:28
gary_posterabentley, yes those are if you install it in a Python.  Use what I told you for those entrypoints14:28
gary_posteron call again14:28
abentleyhenninge: yes.14:29
abentleyhenninge: codehosting on qastaging has only a small subset of branches physically present.14:29
henningeabentley: i saw this when pushing http://paste.ubuntu.com/685307/14:29
henningeah, I see14:29
henningeabentley: but what about new branches14:30
henningeI had to push twice.14:30
henningealso, the branchscanner is either very slow or not running.14:30
abentleyhenninge: I believe it's very slow.14:30
abentleyhenninge: I wouldn't expect the first push to error like that.14:31
wgrantIt runs on staging, but I don't believe it runs regularly on qastaging.14:31
wgrantThat first error was because it's trying to stack on a branch that isn't on qastaging.14:31
wgrantThe subsequent push will see that the bzrdir exists, so not try to stack again.14:31
henningethat seems to make sens14:32
henningee14:32
henningePushing to +junk went fine btw.14:32
abentleyhenninge: I can't reproduce your issue: http://paste.ubuntu.com/685321/14:32
wgrant+junk doesn't stack.14:33
wgrantSo that's unsurprising.14:33
abentleyhenninge: was there a previous push attempt?14:33
abentleywgrant, henninge: true.  It would be complaining about the missing development focus branch.14:33
wgrantExactly.14:33
henningeabentley: pushing to +junk worked fine for me, too.14:34
henningeabentley: my bzr complains about lp://staging/ urls. What might be missing/old?14:34
abentleyhenninge: bzrtools development focus is always present on (qa)staging.  Can you push as a branch of that instead?14:35
abentleyhenninge: I don't know.  What's the complaint?14:35
henninge"not a valid url"14:35
henningeI will try bzrtools14:35
abentleyhenninge: And it's complaining about qastaging, not staging?14:36
abentleyhenninge: I mean, I can understand if it doesn't grok lp://qastaging, because that's newish.14:36
abentleyhenninge: But lp://staging has been supported for years.14:36
henningeyes, that works.14:37
henningeabentley: and bzrtools works flawlessly, too. ;)14:39
henningethanks abentley, wgrant14:40
abentleyhenninge: qastaging support was introduced to trunk in January, and should be in bzr 2.4.14:40
henningeabentley: ah, I am running 2.3.414:40
henningethat's probably stock natty.14:41
henningeabentley: there are two ppas in my config (deactivated): /bzr/ubuntu and /bzr/ppa/ubuntu14:44
henningeabentley: which one should I use?14:44
henningeor both14:44
henninge?14:44
abentleyhenninge: bzr/ppa14:45
henningeabentley: thanks, now I am on 2.4, too14:48
abentleyhenninge: np14:49
=== al-maisan is now known as almaisan-away
=== almaisan-away is now known as al-maisan
henningeHi bac! How far along are you with QA for bug 838957? Need any help?14:56
_mup_Bug #838957: AssertionError: Search by component requires a context with a distribution or distroseries. <oops> <qa-needstesting> <Launchpad itself:Fix Committed by bac> < https://launchpad.net/bugs/838957 >14:56
bachenninge: i'm doing it now, thanks14:56
henningebac: awesome, thank you ;)14:56
bacshould just take a sec14:56
bachenninge: it's gary you need to harass!14:56
bac:)14:56
henningebac: I am done with that ;)14:57
* bac hurries...14:57
henninge"Don't rush the miracle man or you'll get the wrong miracle."14:57
bachenninge: done!14:58
henningewgrant: is the buildd-manager in nodowntime?14:58
henningebac: great, thanks14:58
henninge;-)14:58
=== dpm is now known as dpm_
flacostemrevell: we should announce that we are doing fastdowntime to our users15:31
flacostesince they might encounter a 5 minutes outage weekdays at around 8:30 UTC15:32
bacn15:33
mrevellflacoste, Cheers, I'll get a blog post etc together.15:33
mrevellbac, Hey15:33
mrevellbac, got a question for you15:33
bachi mrevell15:33
bigjoolscjwatson: how would you feel about automatically cleaning out the archive and librarian of all binaries as soon as a series is marked obsolete?15:34
flacostemrevell: thanks! can you take care of sending the news to the stakeholders list also?15:34
mrevellflacoste, Sure!15:34
flacostethanks15:34
stubflacoste, mrevell: I put an announcement via launchpadstatus already15:37
stuboh... just for tomorrow.15:37
mrevellThanks stub.15:37
flacostemrevell: it's probably important to point out that until we fix bug 844631, this will manifest itself through a bunch of OOPSes during the outage15:38
_mup_Bug #844631: Launchpad should return 503 error pages when database is unavailable <fastdowntime-later> <Launchpad itself:Triaged> < https://launchpad.net/bugs/844631 >15:38
mrevellGood point, thanks.15:41
=== al-maisan is now known as almaisan-away
=== bdmurray_ is now known as bdmurray
cjwatsonbigjools: it needs a grace period so that we can make sure that old-releases has something current15:50
bigjoolsok15:51
cjwatson(IMO)15:51
bigjoolscjwatson: is that not something you'd check before obsoleting it?15:51
cjwatsonit doesn't get moved to old-releases until it's obsolete15:51
bigjoolsah, faire enough.15:51
cjwatsonAFAIK anyway15:51
bigjoolsI'm just trying to work out how to phrase a bug which captures the requirements15:52
bigjoolsI want to automate as much as possible - sounds like we need an extra Big Red Button15:52
bigjoolscjwatson: are old sources captured on old-releases? And do you still need all the old versions?15:53
cjwatson(a) yes (b) which old versions?15:54
bigjoolsthe superseded ones15:55
bigjoolsie can we make this like binaries15:56
stubbigjools: I sometimes wonder if the Librarian should have an 'external' field which we can use to point to stuff no longer stored in the Librarian.16:07
bigjoolsstub: yes, we desperately need offline storage.  It's a bit nuts keeping *everything* in there.16:07
elmoblink16:08
elmohow is that not just "make it someone else's problem"?16:08
elmoif you want it online but still available, someone has to store it and make it available16:08
bigjoolsIf you want to keep adding disk space in the librarian then I have no problem filling it up16:09
stubIt very much is making it someone elses problem. I'm just wondering if it is appropriate sometimes (eg. of we are going to store archives for ever anyway on disk in an accessible format, no real point duplicating that storage in the Libraraian).16:10
gary_postersinzui, in a fresh oneiric server, when I try to import html5browser I get a "RuntimeError: Gtk couldn't be initialized" error in gi/overrides/Gtk.py.  I can provide a traceback if that helps.  I verified that various seemingly obvious gtk packages were installed, and they were.  Is there something else relatively obvious I can try?16:13
sinzuigary_poster, thank you for reminding me. wgrant reported this as well. I think there is an undocumented dep16:14
* sinzui investigates16:14
gary_postersinzui, oh ok.  thank you16:15
sinzuigary_poster, can you paste the traceback?16:17
gary_postersinzui http://pastebin.ubuntu.com/685391/16:19
sinzuigary_poster, fab, we did the same thing and it works for me. Maybe I can publish the correct libs in a few minutes16:20
gary_posterok thanks sinzui16:20
=== deryck is now known as deryck[lunch]
sinzuigary_poster, I think the dep that marries python to gir is missing. Do you have python-gobject 2.90.3 installed16:27
* gary_poster looks16:28
gary_postersinzui, -gobject-2 @  2.28.6-6svn116:29
gary_postersinzui, oh and -gobject 2.90.3-1svn116:29
gary_posterboth installed16:29
sinzuigary_poster, that would be the problem  -gobject-2 is for obsolete libs16:30
sinzuiof dear. -gobject 2.90.3-1svn1 is the one I thought was missing16:30
gary_postersinzui, so they would conflict?  Should I try to install -gobject-2?16:30
gary_posterI mean uninstall16:30
gary_postersorry16:30
sinzuigary_poster, they do not, and I happen to have it installed. give it a try16:31
gary_postersinzui, "python-gobject depends on python-gobject-2 "16:31
gary_poster(and I aborted the attempt to uninstall)16:32
sinzuiah. python-webkit is a gtk2 app so it does want that lib16:32
sinzuigary_poster, can you try both of these command, are they the same TB: http://pastebin.ubuntu.com/685415/16:41
* gary_poster tries16:41
gary_postersinzui, yes, same TB16:42
gary_poster(for both)16:42
sinzuithank you that helps. This is a python-gi-gtk lib issue. Mine works by accident because I hack with that combination for other projects16:43
gary_posterah ok16:43
mtaylormorning sinzui ... I got a weird one for you16:48
mtaylorone of our devs (~annegentle) has launchpad openid returning a different localid than annegentle. apparently this happened with adamg a little while ago and the working hypothesis is that it is related to having had an account get renamed16:49
mtaylorsinzui: any idea who the right person to ping about this would be?16:50
sinzuimtaylor, This is often caused by a disconnect between the person's login.ubuntu.com emails and launchpad.net emails16:51
mtaylorsinzui: is that something that can be sorted out by the person themselves?16:51
sinzuiThis was caused several times in the past by profile merges, but I think that is fixed given how rare it happens. I think this commonly happens when a user add/removed emails from wither site without knowing they are working with two sites16:52
sinzuimtaylor, I honestly do not know. I think the person should testing logging in to login.ubuntu.com to verify the registered emails, then do that with Lp and add the missing addresses to Lp16:53
sinzuimtaylor, gary_poster had some experience with this in the past. I believe the a DBA was employed on several occasions to fix the ids in either of the sites16:55
mtaylorsinzui: gotcha16:55
gary_posterhello16:55
mtaylorhey gary_poster !16:55
gary_poster:-)16:56
jeblairi've asked annegentle to verify the email addrs are in sync16:56
gary_posterSo, lemme see if I can find the bug.  If it's the same problem, the proper fix is for ISD to make some changes that have been planned for a long time, but there's a recipe for fixing it on the admin side.16:57
jeblairif it helps:16:57
gary_posterSo first let's verify that we're talking about the same thing.  1 sec, looking16:57
jeblairhttps://launchpad.net/~annegentle says the local_id is https://login.launchpad.net/+id/6sDWhGL16:57
jeblairbut when she logs in, we get passed https://login.launchpad.net/+id/xQHRTxt16:57
jeblairand she did say that she has had some kind of account rename (i don't know if it was executed as a merge)16:58
gary_posterOK, this is the root bug... https://bugs.launchpad.net/launchpad/+bug/64482416:59
_mup_Bug #644824: User created a second Launchpad account; now gets mixed success/denied to various services <canonical-losa-lp> <defect> <lp-foundations> <Canonical SSO provider:Confirmed> <Launchpad itself:Fix Released by stub> < https://launchpad.net/bugs/644824 >16:59
* gary_poster is scanning to try and find remediation...17:00
gary_postermtaylor, comment #31 says that ISD can delete the bad openid account.  contacting them might be your best bet.  https://answers.launchpad.net/canonical-identity-provider/+question/15013517:03
gary_posterAlternatively we can ask losas to make a deletion on the LP side, per comment 12 of that bug.17:04
mtaylorgary_poster: so is the best way to do that to file a question against canonical-identity-provider?17:04
gary_postermtaylor, that would be the better long term soluton, assuming anne does not want to maintain two Canonical openids.  I will also ask the losas to delete the tokens on the LP side, which should alleviate the problem for now (she will have to log in again, using the openid she actually wants to maintain)17:06
gary_posterah, right, there's a process for this now...17:07
=== deryck[lunch] is now known as deryck
sinzuigary_poster, do your have libgirepository-1.0-1? I wonder if the -dev version also need to be installed, if so python-gobject-2-dev may also need installation17:11
gary_postersinzui, I have it installed, but not the -dev17:12
gary_postershould I try it?17:12
sinzuigary_poster, I think I could be smarter about this investigation. paste or email the results of this so that I know exactly what is different about our packages:17:13
sinzuidpkg --get-selections > installed-software17:13
bacmrevell: still around?17:14
mtaylorgary_poster: ok. I'm confused now ... which thing should I be doing now. filing the question?17:18
gary_postermtaylor, I'll do it for you, and ping you with it.17:18
mtaylorgary_poster: awesome. you are amazing17:18
gary_postersinzui, sorry it took a sec, paste isn't set up yet and had some kind of problem17:24
gary_posterbut sent17:24
nigelbwgrant: you around?17:24
nigelbwgrant: Nevermind :)17:27
sinzuigary_poster, can you install gir1.2-gtk-2.017:34
sinzuioh, maybe I am the one without it17:34
sinzuiokay I have it and I am sure webkit is a gtk2 app17:35
gary_postersinzui, ok trying17:35
gary_postersinzui "Setting up gir1.2-gtk-2.0 (2.24.5-0ubuntu7)" but same traceback when I try to import html5browser17:37
sinzuiI will continue looking a the diff17:37
sinzuigary_poster, gir1.2-webkit-1.017:38
gary_posterk17:38
* sinzui doubts that is correct, but if it is, he will be enlighten17:38
=== beuno is now known as beuno-lunch
gary_postersinzui, same problem (and that had three dependencies)17:40
sinzuiokay :( I have python-gtk2-dev  python-gobject-dev python-gobject-2-dev too17:44
sinzuiThey have binaries in them to17:44
gary_postersinzui, k trying17:45
gary_posterwhoo!  That'll bring in the neighbors...17:46
sinzuiyuck17:46
gary_poster59 dependencies17:46
sinzuiI see only two other cases like in on the net and they were solved by installing pygobject, which is python-gobject :(17:46
gary_postermtaylor, losas have sql request on LP side, and I filed https://answers.launchpad.net/canonical-identity-provider/+question/170575 for you on CIP side.17:48
gary_posterI will ping when sql request has been run17:48
mtaylorgary_poster: thanks man!17:49
gary_postersinzui, !! still same error.  you want me to be paranoid and do something crazy like restart or something?  That seems crazy, but...17:49
gary_posterwelcome mtaylor17:49
sinzuigary_poster, no restart. removed those dev packages.17:50
gary_posterok17:50
sinzuiI have started a upgrade to oneiric on another machine and I will look at the diff some more17:50
gary_postersinzui, should I also remove gir1.2-gtk-2.0 and gir1.2-webkit-1.017:52
gary_posteror leave 'em around till later?17:52
gary_postersinzui, /me going to take lunch in a sec17:52
gary_posteroh...17:52
* gary_poster has call soon :-P17:53
sinzuigary_poster, take your time. I will continue to work on this17:53
gary_posterok thanks sinzui17:53
sinzuigary_poster, Gtk is a GUI lib, as webkit is a gui app. you do not have an x server up18:01
bacsinzui: is it possible to get the openid identifier for a user now?  it used to be in the +rdf but isn't now18:09
gary_postersinzui, duh!  sorry, I should have thought of that.  xvfb it is18:09
sinzuibac it is not, but it is a trivial bug fix to remove the constraint in the template18:09
sinzuithe bug is already reported18:09
bacsinzui: ok.  i needs it now, though.  :(18:10
sinzuigary_poster, Sorry for being so slow. I always test html5browser with18:10
sinzuiXvfb :23 -ac -screen 0 1024x768x24 &18:10
sinzuienv -i DISPLAY=:23  ./bin/py18:10
sinzuiI should have read my own notes before asking you to tell me what is installed18:10
gary_poster:-) thanks again18:11
nigelbrockstar: Hi, around?18:33
rockstarnigelb, I am.18:33
nigelbrockstar: lifeless suggested I talk to you about how U1 web deals with javascript oops for potentially duplicating it in LP :)18:34
rockstarnigelb, short answer: we don't.18:34
rockstar:)18:34
nigelbOh.18:34
rockstarWe've disabled jsoops for the time being, because it got in the way more than it helped.18:35
rockstarnigelb, however, we're going to refactor it as soon as I'm done with my current project, which should release soon.18:35
nigelbOuch, okay18:35
=== beuno-lunch is now known as beuno
nigelbrockstar: Any pointers on what's the "right way" to do it?18:36
rockstarnigelb, so, the very basics of it are this:18:37
rockstarwindow.onerrer = function() { new Image().src = '/jsoops/endpoint' };18:38
gary_postermtaylor, please ask anne to try logging in again18:38
rockstarnigelb, of course, the onerror function has message, url, and line arguments.18:38
nigelbokay18:39
nigelbBut Image?18:39
mtaylorgary_poster: will do18:40
nigelbrockstar: I'm curious, why the Image()? Does the jsoops load and image to show the user?18:42
rockstarnigelb, it's just a neater way to make a GET request than XMLHttpRequest18:43
rockstarYou never add the image to the DOM.18:43
rockstarAlso, you don't have to worry about the domain there either.18:44
nigelbOooh. Interesting!18:44
rockstarnigelb, so, the endpoint itself will has ?message=...&url=...&line=... so you just make sure the endpoint knows what to do.18:45
nigelbOh, so these gets logged by the endpoint for future usage?18:45
nigelbusage/triager/debugging/fixing18:45
rockstarnigelb, Be advised, however, that YUI will step all over that once it gets loaded, so you'll have to do something a bit more YUI-ish after that.18:45
rockstarnigelb, exactly.18:45
rockstarnigelb, so we do Y.one(Y.config.win).on('error', function(e) { new Image()... }); once YUI gets loaded.18:46
* rockstar feels another blogpost coming on.18:47
nigelbrockstar: Nice. But we'd be doing a GET to write data in this case right?18:47
nigelbrockstar: WRITE IT! :)18:47
rockstarnigelb, not necessarily.  You could parse logs after the fact, since you usually don't want to write on GET (and Launchpad specifically forbids it).18:48
rockstarYou don't even have to have an actual endpoint.  You could just parse Apache's logs.18:48
nigelbOh, right.18:49
nigelbYeah.18:49
rockstarnigelb, the Ubuntu One web team is a bit more fast and lose when it comes to errors though.  It's often enough to see a graph of errors, rather than the actual errors.18:53
rockstar(at least in the client)18:54
nigelbrockstar: Oh, then you debug based on their numbers? (rising, decreasing)18:54
rockstarnigelb, yeah, and usually it's pretty simple to find, especially if you get the url right.18:55
rockstarMost of the time it's an unsupported browser, and the error was thrown right before the user got the "non-javascript" experience.18:55
nigelbrockstar: ooh, right. But this at least seems fairly trivial to do.18:57
nigelbmay be its my ignorance though :)18:57
rockstarnigelb, yeah, it's not difficult, no.18:58
rockstarnigelb, but you get lots of edge cases (also referred to as "what users do") that make things less trivial.18:58
nigelbI'll run it by lifeless or stub and ask for more comments before I start writing anything :)18:58
rockstarFor instance, you don't want the error to break the rest of the page.18:58
bacsinzui: is this the bug you referenced? https://bugs.launchpad.net/launchpad/+bug/50173118:59
_mup_Bug #501731: XRDS for launchpad users missing local id <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/501731 >18:59
rockstarnigelb, also, remember that in a few weeks, U1 is re-addressing this, so we might want to knowledge share a bit.19:00
lifelessnigelb: we'd generate a custom oops on GET, its not modifying the resource in the usual rfc2616 sense, so would be OK.19:00
nigelblifeless: So, what rockstar proposed sounds good?19:01
lifelessyes, thats why I pointed you at him :)19:01
sinzuibac no. I misunderstood you question. I thought you wanted the openid url which is shown only to yourself on your profile page19:02
* rockstar writes down in his diary "Dear diary, today lifeless approved of something I was doing. It made me feel all warm inside!"19:02
sinzuibac: there is no bug/request to leak the id19:02
nigelbrockstar: haha19:02
nigelblifeless: so, having a stub URL with nothing, on javascript error, do a GET request to that endpoint, look at logs in apache?19:04
lifelessnigelb: s/look at logs in apache/create a custom oops/19:05
nigelblifeless: hrm, so the stub URL should grab it and push it somewhere?19:06
nigelbcustom logs?19:06
nigelblike what wsgi-oops does?19:06
lifelessit will translate between whatever format the javascript is producing and an oops report19:06
lifelessprobably wanting a custom oops.Config() without the in-appserver-hooks installed, because we're not reporting a problem *in this appserver*.19:07
lifelessbut don't take this bit literally: thats detail to figure out when doing it19:08
nigelblifeless: I think I need to look at how LP does oops. I have not much clue :)19:08
lifelesslib/canonical/launchpad/webapp/errorlog.py - allow a couple of hours to read it and the supporting libraries and generally play around getting a feel for whats under the hood19:09
nigelbokay, doing that over the weekend19:09
nigelbrockstar: when you modify the jsoops for u1, I'd be intersted in details :)19:15
lifelessnigelb: Oh, one further tweak: I suggest a dedicated plain-ol-simplewsgi daemon for doing js oops collection19:19
lifelessnigelb: it has no reason to be in either the LP or u1 services; we can map the submission url to the daemon via apache19:20
rockstarlifeless, I would most certainly agree.19:22
nigelblifeless: ah, this is part of the service oriented architecture bit?19:24
lifelessnigelb: no more than anything else19:25
lifelessnigelb: also no less so than anything else19:25
* nigelb couldn't parse that :P19:26
lifelessnigelb: everything we do gets an soa assessment now19:26
nigelbAH!19:26
nigelblifeless: so I can just write something in paste? ;)19:26
lifelessnigelb: yes, or even simpler :)19:27
nigelbOr just a wsgi app, right.19:27
lifelessnigelb: lp:~lifeless/+junk/gpgverify might have some ideas; I *think* I pushed my simpleserver version19:27
nigelbI'm going through error.py19:27
lifelessnot that19:28
nigelbI'll add it to my reading list after that :)19:28
lifelesserrorlog.py19:28
nigelbgah19:28
lifelessah, I'm still using web.py there19:29
lifelessah! its the gpgfixtures test keyserver I used simpleserver19:29
lifelesshttp://bazaar.launchpad.net/~canonical-launchpad-branches/python-gpgfixtures/trunk/view/head:/gpgfixtures/keyserver.py19:30
nigelb*click*19:31
lifelessthats a microservice I was about ready to get the LP test suite to use when I went on leave ;)19:31
lifelesswgrant was mumbling about finishing the migration off19:32
lifelessokies, I have to run; ciao!19:32
nigelbLaters!19:32
nigelbThanks :)19:33
jeblairgary_poster, mtaylor: annegentle is still showing up with the old id, though her launchpad page now lists _no_ openid info19:55
gary_posterjeblair, she needs to log out and log back in.19:56
jeblairthat was after a browser restart...19:57
gary_posterjeblair, IIRC, and I'm pretty sure I do, the session cookie will be retained after a browser restart.  If she didn't log out and in again, there will be no change.19:57
jeblairgary_poster: ok. i'll let her know.20:00
gary_posterjeblair, I've also subscribed her to the question I filed for her against canonical identity provider (https://answers.launchpad.net/canonical-identity-provider/+question/170575) so if there is any reply there, she should know.  You or others can subscribe to replies as well, if desired.20:02
jeblairgary_poster: thanks!20:04
gary_posterwelcome20:04
benjijcsackett: do you have a minute to review lp:~benji/lazr.restful/error-on-multiple-ws.op-fields ?20:19
jcsackettbenji: sure thing.20:20
jcsackettbenji: i don't actually see an MP for that branch.20:21
sinzuijcsackett, I am reviewing your branch now. I am going to to run it to see it20:21
benjijcsackett: heh, you are correct; one second20:21
jcsackettsinzui: ok. sorry i forgot to post screenshots. :-)20:21
salgadocan somebody with the needed rights see the contents of a private branch on bazaar.launchpad.net/...?20:23
benjijcsackett: here you go: https://code.launchpad.net/~benji/lazr.restful/error-on-multiple-ws.op-fields/+merge/7468220:25
jcsackettbenji: i'm not getting a diff generated, any chance you can throw the diff to me as a paste?20:36
benjijcsackett: http://paste.ubuntu.com/685561/20:36
jcsackettthanks.20:36
jcsackettbenji: looks good to me. r=me.20:38
benjithanks!20:39
salgadodoes launchpad the web app have access to the contents of bzr branches?21:17
=== salgado is now known as salgado-afk
=== jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 253 - 0:[########]:256
nigelbAnyone awake? :)22:07
nigelbI'm removing a field completely.22:07
nigelbIs it okay to remove statusexplanation = StringCol(dbName='statusexplanation', default=None)22:07
nigelbfrom the model?22:07
mwhudsonif nothing references it, sure22:16
nigelbI'm removing everything else that references it as well.22:17
reedhello folks22:18
reedis there a way to download the mailing list archives from launchpad? (like pipermail does)22:18
nigelbgrar, postgres error. /me postpones for tomorrow.22:23
reedargh https://bugs.edge.launchpad.net/launchpad/+bug/58841122:37
_mup_Bug #588411: downloading of mailing list archives <feature> <lp-registry> <ml-archive-sucks> <Launchpad itself:Triaged> < https://launchpad.net/bugs/588411 >22:37
sinzuireed, there is not22:37
reedsinzui, very unfortunate22:38
reedI'll update the bug: that's a very important feature to collect metrics on the mailing lists22:38
sinzuireed,  you can ask a question at answers.launchpad.net/launchpad. An admin can get you a copy of the archive22:38
reedI need them on an ongoing basis, like every week or month22:39
sinzuireed, mailing lists are not a priority now. I can update the bug with detail about the scope of the effort for community members implement22:39
reedI can add a comment on the bug too if you want22:40
reedI understand it's not a priority for you at the moment though... I hope you'll reconsider :)22:40
reedsinzui, if there is a way to get the past archives as a one off, I may setup a workaround to gather new messages locally and process the stats from the local archive22:43
sinzuireed, I believe other people have done that. As I said above, and admin can get you a copy of the archive22:45
reedthanks22:46
nigelbAchivement unlocked.22:50
nigelbRemoved a field successfully. Or so I think.22:50
pooliehi all22:54
poolieway to go nigelb :)22:54
nigelbMorning poolie!22:54
wgrantsinzui: Sorry, can't be at the standup this morning. I can probably talk in an hour or so if you want.23:13
sinzuiokay. I will check back later23:14
nigelbwgrant: Is psycopg2.OperationalError: fe_sendauth: no password supplied related tosomething you and jtv were talking on the list?23:16
nigelb(for tests)23:16
StevenKsinzui: 35. http://paste.ubuntu.com/674281/23:17
wgrantnigelb: Yes, that same thing.23:17
wgrantnigelb: Wait, no, not jtv.23:17
wgrantnigelb: My other email to the list.23:17
wgrantNot the reply to jtv.23:17
nigelbwgrant: Okay :) Updating my setup23:19
nigelbwgrant: Wait, but I'm not on ipv6 :/23:20
nigelbStevenK: I join your ranks with a destroy- branch ;)23:21
nigelb+8/-58  not much though23:21
wgrantnigelb: You have loopback IPv6.23:22
wgrantWhat does your /etc/hosts look like?23:22
nigelbAh23:22
nigelb::1     localhost ip6-localhost ip6-loopback23:22
wgrantThat is the problematic bit.23:23
wgrantApply the fix, and all should be good.23:23
* nigelb runs l-d-s again23:23
wgrantAnyway, I need to wander off properly for a while.23:23
nigelb:)23:24
nigelbthanks!23:24
nigelb(I should sleep)23:24
wgrantwgrant@carob:~/logs$ find */2011-09-07 -type f | wc -l23:40
wgrant522623:41
wgrantwgrant@carob:~/logs$ find */2011-09-08 -type f | wc -l23:41
wgrant7243023:41
wgrantfastdowntime appears to have worked...23:41
lifelessthats more than expected ;)23:43
lifeless2minutes at 900rpm => 1800 failures23:43
wgrantThey were mostly appserver OOPSes.23:43
wgrantI calculated there were about 60000 extra appserver OOPSes, from the OOPS IDs at the time.23:43
wgrantAnd yes, it is slightly high.23:43
wgrantEr.23:44
wgrant900rpm?23:44
wgrantConsidering the appservers serve up to 14M requests a day (normally around 9-10M), 900 per minute is a little low :)23:46
lifelessEONLEAVE.23:47
lifelesswe may need to revisit capacity again shortly.23:47
lifelessanyhow, 3600 <<< 6000023:47
wgrant>>> (10000000 / 86400) * 13223:48
wgrant1518023:48
wgrantAnd given that the responses were immediate, API scripts would have made lots of requests.23:48
lifeless132?23:49
lifelessseconds down ?23:49
wgrantMaybe it was 127, but yes.23:49
wgrantRoughly.23:49
wgrant13223:49
lifeless1736 rpm per appserver machine :)23:50
lifelessbut yes, our rate is getting pretty high23:50
wgrantWe had very odd spikes on the 18th and 20th23:50
lifelessare we starting to queue in haproxy again ?23:51
wgrantBut given that they apply to both web and API traffic, I'm inclined to think it was something else.23:51
wgrantWe've had 5 queue depth incidents in the last week.23:51
wgrantSo yes.23:51
lifelessmeep.23:51
wgrantThey all cleared quickly, and only once did it exceed 20.23:51
wgrantBut it is happening increasingly often.23:51
wgrantSo yeah, we may want to expand a bit soon.23:51
lifelesswe need to check the PPR and see if there has been a systematic increase (5% would be significant) in page render time23:51
wgrantLet's see how appserver load is...23:52
lifelessif not, cross check on capacity23:52
lifelessand then look at adding a core + memory to one of the appservers.23:52
wgrantLet's see..23:52
lifelesswe *should* be running at N+1 at all times: hitting a queue warning -ever- is a Big Deal.23:52
StevenKlifeless: I was wondering how you were thinking of scaling -- via a 5th machine or making the 4 beefier23:53
wgrantWe can scale these machines up further.23:53
lifelessStevenK: elmo wants to consolidate in the DC23:53
wgrantThey're not full AFAIK.23:53
lifelessStevenK: we have 4 because 2*dc + redundancy within the DC23:53
lifelessStevenK: but thats the old (february) strategy; now we're moving DC's we're looking at something different e.g. full N+1 deploys in both DC's and one DC running hot-idle23:54
lifelessits been 2 weeks since I caught up with elmo on this, I don't know the current status.23:54
wgrantlifeless: We've had two days in the last month where average render time is 30% higher than the norm.23:54
lifelesswgrant: was the sql time likewise higher ?23:54
wgrantBut apart from that it's only marginally higher.23:54
lifelessor load spikes on the appservers correlatimg with that ?23:55
wgrantThe graphs don't show SQL time.23:55
lifelesswgrant: check the daily PPR report to see that.23:55
lifeless(for the day in question)23:55
wgrantYeah.23:55
wgrantThe first spike is mainly API with a bit of the rest, the second translations with a bit of the rest.23:55
lifelessbbiab23:56
wgrantThe SQL increase was only 10% of the total increase.23:58
wgrantThe first load spike was, interestingly, the day in between the two >13M request spikes.23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!