[01:40] <lifeless> elmo: 'we haven't used it'
[01:47] <wgrant> Er, yes, forgot to actually expand it, sorry.
[02:06] <wgrant> errrm
[02:08] <wgrant> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-2107AP52 is a worry
[02:09] <wgrant> Erk erk erk.
[02:09] <wgrant> BugTaskStatus is a lie.
[02:09] <wgrant> 13/14 are INCOMPLET_WITH(OUT)_RESPONSE
[02:09] <wgrant> But they're not in BugTaskStatus
[02:09] <wgrant> Only in BugTaskStatusSearch
[02:10] <wgrant> Fortunately the migrator is buggy and crashing, so not too many bugs are broken.
[03:25] <lifeless> meep
[03:25] <lifeless> (erm, what do you mean)
[03:26] <wgrant> lifeless: It looks like only Product:+series is actually crashing due to this enum strangeness, but it's still pretty horrible.
[03:27] <wgrant> Bug #871076
[03:27] <lifeless> thanks, have commented
[03:28] <wgrant> Have you?
[03:28] <lifeless> email
[03:28] <wgrant> Ah
[03:30] <lifeless> so the incomplete patch teachs enumvariable about multiple schemas
[03:31] <lifeless> so we can do evolutions like this
[03:31] <lifeless> on load things not found in the first schema are looked up in the second etc
[03:32] <wgrant> Ah
[03:32] <lifeless> I haven't checked the view in question specifically but it sounds like its grouping by and then manually making a dict of the types expected
[03:32] <wgrant> Right, it manually does BugTaskStatus.items[somevaluefromthedb]
[03:32] <lifeless> one way would be to change that to use search
[03:33] <lifeless> another would be to generate a mapping dict manually (or better yet teach the column now to do it)
[03:33] <lifeless> s/now/how/
[03:34] <lifeless> BugTask._status.load(somevaluefromthedb, from_db=True) or some such, -might- do it already.
[03:34] <lifeless> but I suspect the code is currently too coupled to being a descriptor rather than mapping + description
[03:37] <lifeless> .
[20:20] <wgrant> Morning.
[20:24] <jelmer> g'morning wgrant
[20:32] <mwhudson> good morning
[20:32] <wgrant> Morning jelmer, mwhudson.
[20:40] <lifeless> morning early-grant
[20:42] <wgrant> Hi lifeless.
[20:48]  * mwhudson forgets things
[20:49] <mwhudson> can i land changes to the config branch with pqm or does a losa have to do that?
[20:49] <lifeless> pqm
[20:49] <mwhudson> cool
[20:50] <wgrant> https://dev.launchpad.net/WorkingWithProductionConfigs
[20:51] <lifeless> much of that should probably be in-tree in a README
[20:51] <lifeless> (or at least, a link to that wiki page)
[20:51] <wgrant> mwhudson: It's been deployed *everywhere*?
[20:51] <mwhudson> wgrant: https://devpad.canonical.com/~wgrant/production-revnos says so
[20:52] <wgrant> Great.
[20:53] <wgrant> Just wanted to be sure, as this week is a bit special.
[20:53] <mwhudson> ah, release week & all that?
[20:53] <wgrant> Yes.
[20:53] <wgrant> and we need to deploy a fix to cocoplum before oneiric freezes for good.
[20:53] <wgrant> And hope that nothing else is broken.
[20:54] <mwhudson> it only took 12 days from (re-)landing to deployment
[20:54] <mwhudson> felt longer
[20:54] <mwhudson> how did we cope in the old deploy once a month world?
[20:54] <wgrant> Yeah.
[20:54] <wgrant> The librarian took a little while to get HA.
[20:54] <wgrant> But it's pretty much done now.
[20:54] <wgrant> Finally.
[20:54] <lifeless> \o/
[20:54] <mwhudson> cool
[20:54] <wgrant> Just need to add graceful shutdown.
[20:54] <wgrant> But everything is through haproxy, which is the hard bit.
[20:55] <wgrant> Which leaves poppy :(
[20:56] <wgrant> Hmm, I wonder what happens if we just push the FTP control connection through haproxy, but leave the data connection to go directly.
[20:56] <wgrant> I suppose that would work.
[20:56] <wgrant> And be trivial.
[20:57] <lifeless> wgrant: thats what I've been telling you :)
[20:57] <lifeless> wgrant: as a first iteration
[20:58] <lifeless> second iteration will need more thought
[20:58] <wgrant> Well, I'd been missing the fact that we don't care about the data connection.
[20:58] <wgrant> Because normally I'm doing this sort of thing to firewall off or NAT the real host :)
[20:58] <lifeless> for that you need frox
[20:58] <lifeless> and we may need it for the second iteration
[21:00] <lifeless> I haven't put a plan together for the details there yet, because theres no point planning when we're as far out as we are from execution
[21:00] <mwhudson> jelmer: do you have a plan for converting existing mirror branches to bzr-code-import branches? (just curious)
[21:01] <jelmer> mwhudson: I was actually just looking at the numbers
[21:01] <mwhudson> jelmer: i have this old memory that says "roughly 3k mirrored branches"
[21:01] <lifeless> that was gnome wasn't it ?
[21:02] <mwhudson> possibly
[21:02] <jelmer> mwhudson: it's more like 2k
[21:02] <lifeless> we had a separate machine doing imports of gnome, and jelmer registered all of it one day
[21:02] <mwhudson> ah no, that was a lot more than 3k
[21:02] <jelmer> and there's about 300 remote branches
[21:03] <jelmer> lifeless: IIRC that was all code imports though, not mirrors
[21:03] <mwhudson> jelmer: i did some log grepping once and worked out that about 17 out of however many mirror puller runs actually imported anything in a week
[21:03] <cjwatson> wgrant: so I have a short while now if you want me to QA r14123; although I'll need help with dogfood setup
[21:04] <wgrant> cjwatson: I just updated it.
[21:04] <wgrant> Takes ~forever.
[21:04] <cjwatson> Will take or has taken?
[21:04] <wgrant> Has taken.
[21:05]  * wgrant dirties and publishes a PPA to start.
[21:05] <jelmer> mwhudson, lifeless: I don't have plans to work on converting all remaining mirror/remote branches to import branches, and am mainly focussing on bfbia at the moment. Do you think we should worry about converting the existing mirror/remote branches ?
[21:05] <mwhudson> jelmer: nah
[21:06] <mwhudson> bfbia seems more important
[21:07] <wgrant> cjwatson: Do we perhaps want to rsync the real primary archive's dists across?
[21:08] <lifeless> jelmer: mwhudson: what would conversion entail ?
[21:08] <mwhudson> i guess a bunch of fiddling -- db work to create a code import for each branch and change the type, an rsync script to prepare the staging area on escudero
[21:08] <lifeless> jelmer: mwhudson: I'm a -huge- fan of finishing things off, so unless its particularly hard, yes, I think we should finish the work.
[21:09] <wgrant> mwhudson: We don't actually need to populate the staging area, do we?
[21:09] <lifeless> mwhudson: won't the staging area auto-populate, same as a new import ?
[21:09] <wgrant> All we lose is a bit of pear time by not doing it.
[21:09] <mwhudson> wgrant: well no, but that would imply a reimport of everything
[21:09] <lifeless> mwhudson: known as a fetch :P
[21:09] <mwhudson> i guess it depends if any of the branches being mirrored are large
[21:09] <wgrant> Not cheap, but meh.
[21:09] <cjwatson> wgrant: I thought there was a plausible distroseries somewhere already with a forked hello package in it
[21:10] <mwhudson> mysql were using mirrored branches at one point... are they still?
[21:10] <wgrant> cjwatson: DF was erased a couple of weeks back.
[21:10] <mwhudson> (should assess disk space impact on escudero too i guess)
[21:10] <cjwatson> wgrant: what would rsyncing the real dists across achieve?  if we actually tried to modify any of those distroseries, surely LP would clobber them anyway
[21:10] <cjwatson> wgrant: oh, Translation-* custom uploads I suppose
[21:10] <wgrant> cjwatson: Well, we don't have ddtp files.
[21:10] <wgrant> Right.
[21:10] <cjwatson> yeah, that might be an OK plan
[21:10] <wgrant> But we can easily forge those, I guess.
[21:11] <cjwatson> real data wouldn't hurt
[21:11] <cjwatson> although I think the DS with include_long_descriptions=False was querulous or something?
[21:11] <jelmer> wgrant, mwhudson: some of the existing mirror branches have data but the location they mirror no longer exists. I think we should make sure that data is kept around.
[21:11] <lifeless> the risk of not completing the transition is that we have (yet another) ambiguous and stubby code area
[21:12] <wgrant> cjwatson: Right, but that's gone now.
[21:12] <wgrant> I'll have to upload something new.
[21:12] <lifeless> mwhudson: jelmer: can the imports *read* from b.l.n ?
[21:12] <wgrant> Or, well, copy it across.
[21:12] <wgrant> That would work too.
[21:12] <wgrant> But first I need to work out why the PPA publisher is trying to publish every PPA...
[21:12] <mwhudson> lifeless: probably
[21:12] <mwhudson> lifeless: not sure though, they don't currently
[21:13] <lifeless> so one way you could solve this is to clone from b.l.n to escudero if the branch is populated and no staging area exists
[21:13] <jelmer> lifeless: at the moment they only read from the data dir on escudero AFAIK
[21:13] <lifeless> given all imports are public
[21:13] <jelmer> are imports necessarily public?
[21:13] <lifeless> today, yes.
[21:14] <lifeless> (mwhudson will shout out if I'm wrong :P)
[21:15] <jelmer> hmmok, I've been working on the assumption that they aren't necessarily public (which should only be a good thing for the future)
[21:15] <mwhudson> i'm not sure imports are necessarily public
[21:16] <mwhudson> i think they probably *are* though
[21:16] <mwhudson> there was a bug about them at some point
[21:16] <wgrant> 2011-10-09 21:16:18 DEBUG   Writing Index file for maverick/main/i18n
[21:16] <wgrant> Let's see if it actually did anything...
[21:16] <mwhudson> yay bug search
[21:16] <wgrant> ppa:launchpad/ppa, this is.
[21:17] <wgrant> No index there, which sounds right.
[21:18] <cjwatson> That's a buglet - it shouldn't log that until it's sure it's actually going to do so.
[21:19] <cjwatson> Not fatal though
[21:19] <lifeless>  private | count
[21:19] <lifeless> ---------+-------
[21:19] <lifeless>  f       |  5593
[21:19] <lifeless>  t       |     5
[21:19] <lifeless> there are 5 private imports.
[21:20] <lifeless> mwhudson: jelmer: however, as the import *details* are public (IIRC) this is a bit strange.
[21:20] <wgrant> cjwatson: Bah, mawson can't connect to rsync on (rsync.)archive.ubuntu.com, and cocoplum's rsyncd forbids it.
[21:20]  * wgrant creates fake files.
[21:20] <mwhudson> lifeless: how so?  if the branch is private, how are you going to see the details?
[21:21] <lifeless> mwhudson: only if the codeimport, which is directly traversable inherits the privacy rule
[21:21] <mwhudson> lifeless: are you talking about api access?
[21:22] <lifeless> mwhudson: well, maybe its changed, but imports used to have their own url
[21:22] <lifeless> anyhow
[21:22] <lifeless> all 5 have public source urls
[21:22] <lifeless> I believe the only reason they are private is they were created on private-by-default projects
[21:22] <mwhudson> lifeless: that changed a _long_ time ago
[21:22] <mwhudson> yeah, i agree
[21:22] <jelmer> lifeless: how many private mirror branches?
[21:23] <lifeless> jelmer: give me the query to id mirror branches and I'll tell you
[21:23] <mwhudson> lifeless: branch_type = ... 2?
[21:24] <mwhudson> yes, 2
[21:25] <lifeless>  private | count
[21:25] <lifeless> ---------+-------
[21:25] <lifeless>  f       |  2112
[21:25] <lifeless> (1 row)
[21:25] <lifeless> (0)
[21:25] <mwhudson> \o/
[21:26] <lifeless> need to make sure we don't permit by-id direct-apache access to the importds (would permit public imports of random private branches), but other than that, letting importds read from b.l.n makes a lot of sense to me anyway
[21:26] <lifeless> e.g. to remove the staging area for imports that don't need it
[21:26] <mwhudson> unless things have changed a bunch, we don't provide by id access to the importds
[21:27] <lifeless> right, and shouldn't
[21:27] <mwhudson> it's locked down to crowberry itself & loganberry by ip address
[21:27] <wgrant> It's forbidden in Apache, and the opener should forbid it too.
[21:27] <lifeless> I was just saying we need to keep it closed :)
[21:28] <lifeless> I'd like to remove the importd non-xmlrpc-api code from the codebase
[21:29] <wgrant> We can finally revoke DB access from the importds once we have a rabbit OOPS transport :)
[21:30] <mwhudson> yes, that was a disappointing discovery
[21:30] <lifeless> mwhudson: what that ?
[21:30] <mwhudson> i did a bunch of work so we could revoke db access from the importds
[21:30] <mwhudson> and then oops-prune fell over :-)
[21:36] <cjwatson> wgrant: well, let me know if there's something I can actually do to help, or whether I'll just blunder around and get in the way :-)
[21:36] <wgrant> cjwatson: Well, I know mostly what I'm doing here, so QA on dogfood mostly involves sitting around and watching bzr st bring it into 50% iowait.
[21:37] <cjwatson> OK
[21:37]  * wgrant is beginning to think that there may actually be something wrong with mawson.
[21:38] <wgrant> 2011-10-09 21:37:40 DEBUG   Publishing custom translations_multiverse_20110922.tar.gz to ubuntu/oneiric
[21:38] <wgrant> We should now have some files.
[22:00] <cjwatson> wgrant: OK, well, I'm off to bed; I see publish-distro is running, so if it looks like it's all gone wrong in the next couple of hours, feel free to SMS me (number's in the directory)
[22:01] <wgrant> cjwatson: I'll hopefully sort it out if stuff does go wrong :)
[22:01] <wgrant> Night, and thanks for fixing this quickly!
[22:02] <cjwatson> The knowledge that it will be worse not to fix it quickly is a wonderful incentive.
[22:02] <wgrant> That's true.
[22:10] <lifeless> wgrant: ians work may be a candidate
[22:10] <lifeless> wgrant: the queries are consulting both private and transitively_private
[22:11] <wgrant> lifeless: Yes, I noticed that.
[22:11] <lifeless> not to mention that its buggy
[22:12] <wgrant> All attempts to implement sensible privacy on top of non-sensible privacy have so far been buggy.
[22:12] <wgrant> So of course it is buggy.
[22:12] <lifeless> branches that are only transitively private and owned by the viewer won't be shown
[22:12] <lifeless> well, specifically buggy
[22:13] <mwhudson> yeah, i was wondering about that, surely all checks should be for transitively_private ?
[22:14] <lifeless> yes
[22:22] <wgrant> cjwatson: Get Translation files ...
[22:22] <wgrant> [  0%] Getting: dists/oneiric/main/i18n/Translation-en.bz2ok
[22:22] <wgrant> [  0%] Getting: dists/oneiric/multiverse/i18n/Translation-bg.bz2ok
[22:23] <wgrant> cjwatson: debmirror is even happy with the output.
[22:25] <cjwatson> awesome!
[22:26] <cjwatson> looks fine visually too.
[22:26] <wgrant> Yep.
[22:27] <wgrant> Hm. LP derived distros are going to be published at derived.archive.ubuntu.com? :/
[22:27] <wgrant> That seems a bit odd.
[22:27]  * cjwatson is uncomfortable with that.
[22:27] <wgrant> Yes.
[22:27] <wgrant> Just a bit.
[22:28]  * wgrant will convince Julian that it's a terrible idea tonight.
[22:28] <elmo> wgrant: reference?
[22:28] <elmo> nm, got it
[22:28] <wgrant> In the DD RT.
[22:28]  * wgrant just closed it.
[22:28] <wgrant> eg. https://rt.admin.canonical.com/Ticket/Display.html?id=48314
[22:29]  * elmo reopens
[22:29] <lifeless> wgrant: you closed the url, or the ticket ?
[22:29] <elmo> well, not that
[22:29] <wgrant> Er.
[22:29] <wgrant> I mean I closed the tab.
[22:29] <wgrant> So didn't have the URL handy.
[22:30] <lifeless> ma
[22:30] <lifeless> man
[22:31] <lifeless> i get disproportionately annoyed by 'N time has passed, whats up' in bugs.
[22:31] <wgrant> Well, in their defense the bzr upgrade is taking its time :)
[22:31] <wallyworld_> lifeless: just saw your comments about transitively_private. all queries are supposed to look only at transitively_private. if any still look at private, that's a bug and needs to be fixed. do you have an example?
[22:31] <elmo> lifeless: paste large chunks of some random dude's biography as a reply
[22:31] <lifeless> wallyworld_: the OOPS referenced in #launchpad
[22:31] <wgrant> elmo: Thanks.
[22:31] <elmo> "what's up?  well, let me tell you..."
[22:31] <wgrant> elmo: I didn't see this name until today either.
[22:31] <lifeless> wgrant: I meant in bug 294159
[22:32] <wgrant> I wonder where it was discussed.
[22:32] <lifeless> (naming - me neither)
[22:32] <wgrant> "not", I guess.
[22:32] <wallyworld_> thanks. i'll look. and fix it. it was meant to be s/private/transitively_private
[22:32] <wgrant> Let's not make the PPA mistake again, either.
[22:32] <wgrant> lifeless: Oh, a different one.
[22:32] <wgrant> lifeless: I see.
[22:32] <elmo> upload?  yeah
[22:33] <wgrant> elmo: I mean having stuff under launchpad.net.
[22:33] <wgrant> And that too.
[22:33] <wallyworld_> lifeless: bollocks. can you paste to oops num. i had been disconnected from #launchpad for some reason
[22:33] <lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=2108CP126
[22:33] <wgrant> wallyworld_: https://pastebin.canonical.com/54081 is a query
[22:34] <wallyworld_> thanks
[22:40] <wallyworld_> lifeless: why is the oops shown as being linked to an old bug 638924?
[22:42] <lifeless> because that correlation generates many false positives
[22:42] <wgrant> StevenK: Around yet?
[22:43] <wallyworld_> ok. i'll raise a bug to fix the query column. the timeout is a separate issue which won't be affected by using the correct column
[22:43] <wgrant> wallyworld_: Why not?
[22:43] <wgrant> wallyworld_: It'll have to query two indices, separately.
[22:44] <wgrant> It may very well negatively influence the plan.
[22:44] <wallyworld_> perhaps. but sad if that's the case. poor reflection on postgres
[22:45] <wgrant> Only slightly.
[22:45] <wallyworld_> a decent db should be able to efficiently query on two indexed boolean columns
[22:46] <wgrant> On large, skewed tables, where the query was optimised for a single one?
[22:47] <wallyworld_> i'm not familar with postgres's query optimisation so am not sure
[22:51] <wgrant> Let's delete Launchpad and start again.
[22:57] <wgrant> Changes that will be made are A) destroying bug heat, B) making bug subscription queries not terrible.
[22:57] <wgrant> lifeless: can you explanalyse 'SELECT Bug.heat FROM Bug, Bugtask, Product WHERE Bugtask.bug = Bug.id AND Bugtask.product = Product.id AND Product.project IS NOT NULL AND Product.project = 82 ORDER BY Bug.heat DESC LIMIT 1' on qastaging?
[22:58] <lifeless> wallyworld_: so, querying on two unrelated indices requires either A) a hash join on the indices or B) a bitmap join on the indices
[22:58] <lifeless> wallyworld_: A) is done by generating the hash during processing, and B) requires reading every row in the index
[22:59] <lifeless> wallyworld_: the only other form of join I'm aware of that could benefit is nested loops, and I've never seen postgresql do that within one table
[22:59] <lifeless> wallyworld_: lastly, for any DB, using more indices (usually) implies more potential disk IO (and usually random at that), which the planner will avoid as its costly
[23:00] <lifeless> wallyworld_: so yes, using two separate fields that are index separately is both more costly (in theory) and a candidate for causing performance issues
[23:01] <wallyworld_> sure, but surely it could narrow the results by using each index sequentially. there are lots of queries we do which use more than one indexed column
[23:03] <lifeless> yes, but check the plans - pg picks one index and runs with it
[23:03] <lifeless> wallyworld_: some reasons for this are that indices can't tell you liveness of rows - you have to consult the table itself, and that cross-index statistics are (AFAIK) not well defined
[23:04] <wallyworld_> ok
[23:04] <lifeless> wallyworld_: mostly though, its the liveness I suspect: finding the candidate rows from the most selective index means you'll be reading the actual rows to do that; and checking a field in those rows is usually about as cheap or much cheaper than paging in index pages from a separate index
[23:05] <lifeless> now, the math says that in very large tables with lopsided data (which we have) that the CPU time will become more expensive than the time to grab a second index and refine, but I suspect its so marginal that the planner code doesn't even permit it
[23:06] <lifeless> if someone were to do a N-index without-consulting-the-rows filter, that would likely be more useful
[23:07] <wallyworld_> so you seem to be implying we could run into issues for *any* query which filters on more than one indexed column
[23:07] <lifeless> wallyworld_: we do
[23:07] <wallyworld_> which seems absurd!
[23:07] <lifeless> wallyworld_: its particularly bad on very big tables (like BPPH and SPPH)
[23:07] <wallyworld_> that a db can't handle that scenario
[23:08] <lifeless> wallyworld_: for precision: queries which filter on more than one column, where the most selective index is not very selective, in big tables, will have issues
[23:08] <lifeless> wallyworld_: the db provides tools like N-column indices to handle such scenarioes
[23:08] <wallyworld_> sure
[23:09] <lifeless> wallyworld_: bitmap filters across multiple indices are great for hot-indices in moderate size tables
[23:09] <wallyworld_> in this case, private|transitively_private = true is very selective
[23:09] <wgrant> But that's two indices.
[23:09] <wallyworld_> yes, and to me the db should be good at that, but it seems not so
[23:10] <lifeless> wallyworld_: profiling in db's is the same as in regular code: unless you measure, you can't predict reliably.
[23:11] <lifeless> wallyworld_: saying 'should' here is something I guess I object to: if someone writes the code to model the costs reliably, from table stats, then they can write the executor, and away we go.
[23:12] <lifeless> wallyworld_: but in the absence of a model, we can't even say *hypothetically* that it should be good at it.
[23:14] <wallyworld_> sure. i have no knowledge of how postgres' stats are calculated or how we have parameterised the analysis engine
[23:15] <wallyworld_> past experience with oracle != postgres clearly
[23:15] <lifeless> wallyworld_: we have 450K branches in the system; the indices are 10MB each, we have 22K private branches - hugely lopsided - so in principle the performance question is, is 'how expensive is consulting 22K rows of a cold index'
[23:15] <lifeless> wallyworld_: oracle itself is also sensitive and needs continual review and tuning, different precise rules but similar issues, from my experience
[23:16] <lifeless> bottom line is, you can't assume anything is fast until its been tested - both for python and for postgresql; things will always surprise us
[23:16] <wallyworld_> yes
[23:18] <wallyworld_> i know how lopsided the private branches numbers are. you wouldn't expect looking at "only" 22k rows from a cold index would be too bad, but you never can tell
[23:18] <lifeless> fwiw - doing the query wgrant asked about above on bugs
[23:18] <lifeless> which chose a nested loop plan, examining 27K bugs
[23:19] <lifeless> took 25 seconds
[23:19] <lifeless> you're looking at 1-2ms per row when IO is involved
[23:19] <lifeless> so I expect that looking at 22k rows in a cold index to be moderately painful at best.
[23:47] <wgrant> lifeless: Why do we have an haproxy status page on each service, rather than just closing the listener?
[23:51] <wgrant> Seems like it would be much easier if processes just closed their listener and died when there were no connections left.
[23:51] <wgrant> Rather than the haproxy dance in appserver initscripts, and adding an HTTP listener to every service.