[00:00] <LPCIBot> Yippie, build fixed!
[00:00] <LPCIBot> Project db-devel build (381): FIXED in 5 hr 47 min: https://hudson.wedontsleep.org/job/db-devel/381/
[00:24] <SpamapS> lifeless: re the cassandra stuff, https://launchpad.net/~cassandra-ubuntu
[00:25] <SpamapS> lifeless: stable releases has 0.7.0 .. I'm working on 0.7.2
[00:25] <SpamapS> lifeless: and yes, it installs fine on lucid and maverick
[00:25] <lifeless> \o
[00:25] <lifeless> I'll have to see about getting it into cat
[00:30] <lifeless> wgrant: are you around?
[00:30] <wgrant> lifeless: Yes.
[00:31] <lifeless> wgrant: https://code.launchpad.net/~lifeless/launchpad/rabbit/+merge/50488
[00:31] <lifeless> if you're interested
[00:32] <lifeless> I'd like another set of eyeballs before I land it
[00:39] <wgrant> lifeless: I spent a while looking for the definition of BRANCH_NICK
[00:39] <wgrant> Then I noticed the docstring.
[00:39] <wgrant> Er, comment.
[00:39] <wgrant> os_exec's name sucks.
[00:40] <wgrant> This daemonisation stuff looks far more crackful than required.
[00:40] <wgrant> What's wrong with subprocess?
[00:41] <lifeless> I don't know yet
[00:41] <lifeless> I was handed some working code that was run from Makefile and totally unsuitable for parallel testing etc
[00:41] <wgrant> Ah.
[00:41] <lifeless> I've done the minimal migration
[00:41] <lifeless> theres more to go
[00:41] <wgrant> Maybe the rabbit binaries don't daemonise and set PIDs and stuff.
[00:42] <lifeless> I don't know which bits matter, and which don't.
[00:42] <wgrant> It is awful and you need to clean it up, but yes, get it in the tree.
[00:43] <wgrant> It looks like it will mostly work, but it's fairly insane.
[00:44]  * wgrant considers destroying Zopeless.
[00:45] <lifeless> heh
[00:45] <lifeless> go look at the original
[00:45] <lifeless> :)
[00:49] <lifeless> ec2 away
[00:49] <lifeless> hopefully all the deps are already presen
[00:49] <lifeless> t
[00:52] <lifeless> I'm seriously considering having the production rabbit present on every node
[00:53] <lifeless> it would make reasoning about many failure modes a lot easier
[01:03] <wgrant> Hmm.
[01:03] <wgrant> lifeless: Not just on the DB servers?
[01:06] <wgrant> lifeless: Hmm, ~canonical can't see U1?
[01:07] <wgrant> :(
[01:20]  * wgrant pokes staging in the eye.
[01:47] <wgrant> Subclasses overriding private attributes... grr.
[01:57] <wgrant> There is minified JS in our bzr history wtf wtf wtf
[03:27] <lifeless> wgrant: probably not on the db servers at all
[03:27] <wgrant> lifeless: Hmmm.
[03:28] <lifeless> wgrant: why would it be on the db servers?
[03:28] <wgrant> lifeless: I like keeping my critical data stores on machines that are meant to keep critical data stores.
[03:29] <lifeless> rabbit sits in a spectrum
[03:29] <lifeless> if (say) devpad croaks
[03:29] <lifeless> we don't care if in-progress OOPSes are lost
[03:30] <lifeless> all our machines have RAID level redundancy atm
[03:30] <lifeless> the db servers aren't big because they have more redundancy, they are big because they are central load points
[03:31] <wgrant> But the DB servers stay up.
[03:31] <lifeless> so do the appservers
[03:31] <wgrant> And if they go down then we are screwed anyway.
[03:31] <lifeless> we can fail over to a slave
[03:32] <lifeless> with rabbit by default the queued messages on a down load will return when the node returns
[03:32] <lifeless> so it needs to be a pretty critical boom moment to permanently lose them
[03:32] <lifeless> and whatever machine we bring another queue processor up on would establish new queues
[03:33] <wgrant> Appservers having the only copy of any data is scary.
[03:33] <lifeless> they have that all the time at the moment
[03:33] <wgrant> Do they?
[03:34] <lifeless> in progress requests
[03:34] <wgrant> The persistent data stores are on four other machines, not appservers.
[03:34] <lifeless> rabbit isn't a persistent data sttore
[03:34] <wgrant> It persists across a request boundary, so it is persistent.
[03:35] <lifeless> by that token so is an oops report
[03:35] <lifeless> which means we have 5 persistent data stores on the app servers
[03:35] <lifeless> (I disagree with that definition btw)
[03:36] <wgrant> I don't want a taken-out appserver to reverse actions taken by a completed request.
[03:36] <wgrant> Losing queue items does that.
[03:36] <lifeless> I think you need to learn a little  more about rabbit
[03:36] <wgrant> 14:32:14 < lifeless> with rabbit by default the queued messages on a down load will return when the node returns
[03:37] <lifeless> right
[03:37] <lifeless> we wouldn't have queues on the appservers
[03:37] <lifeless> we'd have a rabbit on them
[03:37] <lifeless> if something got buffered there, it means the network is down
[03:37] <lifeless> the only use case for that is oops forwarding atm
[03:38] <lifeless> separately
[03:38] <wgrant> So each queue has a master node that will not be on an appserver?
[03:38] <lifeless> no, dropping a queues items won't reverse actions from a completed request
[03:38] <lifeless> those actions clearly wouldn't have happened if that was possible
[03:38] <lifeless> in rabbit, a queue lives on a node
[03:39] <lifeless> exchanges forward messages to queues based on bindings
[03:39] <wgrant> Right.
[03:39] <lifeless> using oops as an example
[03:39] <lifeless> I'd put the queue of oopses to process on a local rabbit where the oops site is
[03:40] <lifeless> I'd make it entirely ephemeral because I don't care if we lose a few in the event of failure
[03:40] <lifeless> (I care if we lost tonnes, but not a few)
[03:40] <wgrant> Oh, right, it lets you configure the reliability? It's been ~a year since I last looked at it in depth.
[03:40] <lifeless> each message
[03:40] <lifeless> and each queue
[03:41] <lifeless> you can be transactional
[03:41] <lifeless> or you can be fire and forget
[03:41] <lifeless> we're going to have a bunch of differing requirements
[03:41] <lifeless> and how we deal with failures of various sorts is going to require case by case thought.
[03:41] <lifeless> simply saying 'transactional persistent, done' would be a recipe for later pain.
[03:41] <wgrant> Indeed.
[03:43] <lifeless> anyhow
[03:43] <lifeless> it doesn't have 2pc
[03:43] <lifeless> AFAICT
[03:44] <wgrant> :(
[03:44] <lifeless> https://dev.rabbitmq.com/wiki/AckingSchemes#section_1 kindof talks about it
[03:44] <lifeless> but hte images are fucked
[03:44] <wgrant> As fucked as some of our import webs?
[03:44] <lifeless> it looks like you build it on top of the characteristics
[04:09] <lifeless> righto, amqp 1.0 does 2pc - but IIRC 1.0 kindof foundered
[04:09] <lifeless> as a ill performing mess
[04:09] <wgrant> Heh.
[04:09] <lifeless> best you can do with 0.8 is queue things and not release until the db is committed
[04:09] <lifeless> (thats what pgamqp does)
[04:31] <lifeless> ok, wow time ;)
[05:09] <StevenK> lifeless: Hah
[05:10] <StevenK> lifeless: 3rd 85 yesterday
[06:02] <lifeless> cool
[06:38] <lifeless> grrr, rabbit failed on ec2
[06:38] <lifeless> is it easy to get a shell into an ec2 instance while its running tests ?
[06:45] <wgrant> lifeless: It gives you the ssh command near the end of the output.
[07:10] <StevenK> lifeless: ssh -v ec2test@<ec2 ip or hostname>
[08:06] <lifeless> next weekend I guess
[11:06]  * jml is around fo reals
[11:17] <lifeless> hai
[11:17] <lifeless> and bai
[18:05] <lifeless> jml: I am on that bridge ;) - I have a patch where I'm trying to figure out what package to put the subunitstream model class in
[19:49] <lifeless> grrr
[19:49] <lifeless> dict.update() shouldn't be forbidden ... zopeSecurityProxy fail
[19:59] <thumper> morning lifeless
[19:59]  * thumper needs coffee 
[20:00]  * thumper wanders off to find said coffee
[20:00] <lifeless> hi thumper
[20:52] <jelmer> 'morning thumper, lifeless
[20:52] <thumper> hi jelmer
[20:52] <jml> lifeless: uhh, I see.
[20:53] <jml> lifeless: there's a relatively easy way to fix the {}.update thing.
[20:54] <jml> lifeless: I'd suggest "testdb" if you can't think of anything better.
[20:55] <jml> lifeless: you could add 'checker.BasicTypes.update({dict: checker.NoProxy})' to lp_sitecustomize, which would completely disable security proxy checking for dicts. I don't think that would be a bad thing.
[20:59] <lifeless> jml: I actually think the current structure of <domain>/<layer> is really annoying; I'd much rather we had <layer>/<domain>
[20:59] <lifeless> jml: wouldn't that let things escape securityproxying? if an unproxied object is in the dict
[21:26] <lifeless> wgrant: hey
[21:26] <lifeless> wgrant: when you are around; can more than one distro share a distroseries row?
[21:28] <StevenK> lifeless: I strongly doubt it.
[21:28] <lifeless> StevenK: ok cool
[21:29] <thumper> StevenK, wallyworld_: standup
[21:29] <thumper> ?
[21:29] <StevenK> thumper: Aye, coming
[21:38] <lifeless> is there a helper for sql strings like Or for storm expressions?
[21:38] <lifeless> I have a list of sql fragments
[21:39] <lifeless> and want on length 1 -> foo[0]
[21:39] <lifeless> on length 2+ -> "(" + " OR ".join(foo) + ")"
[21:48] <thumper> StevenK: https://code.launchpad.net/~thumper/launchpad/choosing-recipe-name/+merge/50530
[21:59] <wgrant> lifeless: I hope not, because that would break everything.
[22:01] <jml> lifeless: oh yes, I think you're right.
[22:01] <jml> lifeless: I don't find our "current" thing annoying. What I find annoying is that we haven't fully switched to it.
[22:08] <jml> lifeless: however, if you wanted to change, and were actually serious about finishing that change and could guarantee that we wouldn't have a mutiny, then I wouldn't mind a <layer>/<domain> approach. They seem equally preferable to me.
[22:12] <wallyworld_> thumper: sorry, was having breakfast
[22:14] <thumper> wallyworld_: ok
[22:16] <thumper> rachel's kindle just arrived \o/
[22:18] <wallyworld_> you're just happy now cause you don't like sharing
[22:19] <lifeless> jml: I'm waiting for the squad culture to bed down a bit
[22:19] <lifeless> jml: I think it will become obvious to others, the more they work cross <domain> that the current structure is a nuisance
[22:19] <wgrant> Hmm.
[22:20] <lifeless> things like the buildmaster are good as they are
[22:20] <wgrant> lifeless: Except it should be in services.
[22:20] <lifeless> that is, they actually are components
[22:20] <lifeless> wgrant: sure, or services should die in a fire and promote its contents
[22:20] <wgrant> The LP_DB* envvars make me sad.
[22:21] <lifeless> excuse the hyperbole
[22:21] <lifeless> wgrant: yes
[22:21] <wgrant> But I guess I should still support them :(
[22:21] <wgrant> Apart from that Zopeless is just about gone.
[22:21] <lifeless> or find and remove their uses
[22:21] <lifeless> wgrant: nice
[22:21] <lifeless> oh frell
[22:21] <lifeless> I have the rabbitmq code in my bug-636158 branch
[22:21] <lifeless> ><
[22:21] <wgrant> Hah.
[22:21] <wgrant> You are good at that.
[22:22] <mwhudson> the problem with "finding the uses" is that they may well be in initscripts and so on that aren't in the tree
[22:22] <wgrant> mwhudson: Exactly.
[22:22] <wgrant> And it's easy enough to support them in a much less OMG MUST KILL way than they are now.
[22:22] <wgrant> So I will.
[22:23] <wgrant> There is also a set of options that some scripts take that do the same thing :/
[22:24] <lifeless> mwhudson: losado grep
[22:25] <wgrant> Also, shipit needs to die.
[22:25] <lifeless> thats feature level work
[22:25] <wgrant> Yes :(
[22:25] <lifeless> it needs some apis
[22:25] <lifeless> launchpadlib fixed to be usable in appservers
[22:25] <lifeless> then its doable
[22:26] <wgrant> Sure. I wasn't saying it was doable. Just that it pisses me off when I have test failures because bzr grep doesn't catch stuff outside the tree.
[22:28] <wgrant> Why did staging not try to restore over the weekend?
[22:30] <jml> wgrant: +1
[22:31] <lifeless> I need a reviewer
[22:31] <lifeless> http://bazaar.launchpad.net/~lifeless/launchpad/bug-636158/revision/12413
[22:31] <jml> pisses me off even more when I have deployment failures because of the same.
[22:32] <lifeless> this is a refactoring
[22:32] <lifeless> it has some stuff I'm not entirely happy with, but I don't konw if there are existing facilities in tree, or if I should make some.
[22:37] <wgrant> lifeless: My first comment is that the method should probably be on IPublishingSet, not IDistroSeriesSet.
[22:38] <lifeless> why ?
[22:38] <wgrant> Because all this publishing-related stuff on DS bloats DS massively.
[22:38] <wgrant> It doesn't belong there.
[22:38] <lifeless> mmm, thats fair enough, but why does it belon on publishing set ?
[22:39] <lifeless> why would someone look there rather than distro/distroseries ?
[22:39] <wgrant> Because in the end you are asking for the latest publication.
[22:39] <lifeless> thats an implementation detail
[22:39] <lifeless> isn't it ?
[22:39] <wgrant> No.
[22:39] <wgrant> I don't think so.
[22:40] <lifeless> theres a related oddness
[22:40] <wgrant> There's more than one.
[22:40] <lifeless> distro.getCurrentSourceReleases != distro.development_series.getCurrentSourceReleases
[22:40] <lifeless> and it appears to be deliberate
[22:40] <lifeless> but I'm not sure users will understand
[22:41] <wgrant> I didn't know that both existed.
[22:41] <lifeless> indeed
[22:41] <wgrant> Distribution's finds the latest in any series?
[22:41] <lifeless> there would be much less duplcicate code if they didn't both exist
[22:41] <lifeless> correct
[22:42] <wgrant> That's braindead.
[22:42] <wgrant> But Bugs uses it.
[22:42] <lifeless> indeed
[22:42] <lifeless> indeed
[22:42] <wgrant> I don't recall anywhere else that does.
[22:42] <lifeless> I'm in correctness-preserving refactoring at the moment
[22:42] <lifeless> so I am not going to fiddle
[22:42] <wgrant> And I filed a bug on the Bugs behaviour a few years ago.
[22:42] <wgrant> :(
[22:42] <lifeless> but - it makes me care less about the duplication I have that I was worried about
[22:43] <lifeless> because we can just delete the method in a separate pass
[22:44] <wgrant> Right.
[22:44] <wgrant> Make sure you file a bug.
[22:45] <lifeless> where is the bug you filed ?
[22:45] <lifeless> (because that would be the one to use)
[22:46] <wgrant> Bug #279513
[22:46] <_mup_> Bug #279513: Distribution source package tooltip in bugtask table shows most recent SPP <lp-bugs> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/279513 >
[22:46] <wgrant> Not as long ago as I thought.
[22:48] <lifeless> so
[22:48] <lifeless> it could show most highest version
[22:48] <wgrant> Or latest in the dev series.
[22:48] <lifeless> right
[22:48] <lifeless> which I think is better and simpler
[22:48] <wgrant> Definitely.
[22:49] <thumper> hmm... it seems that Matcher and Mismatch aren't in the __all__ for testtools.matchers
[22:49] <thumper> I'm guessing they should be
[22:50] <jml> yeah, they should.
[22:50] <jml> they might even be in trunk.
[22:50] <lifeless> They were skipped deliberately
[22:50] <lifeless> because from testtools.matchers import * shouldn't import the base classes
[22:50] <lifeless> this was perhaps a mistake
[22:51] <lifeless> erm
[22:51] <lifeless> 400 Bad Request Bad Request Your browser sent a request that this server could not understand. Apache/2.2.14 (Ubuntu) Server at bugs.launchpad.net Port 443
[22:51] <lifeless> wtf
[22:52] <wgrant> What was the request? I've seen that on bad hostnames, but that's about it.
[22:52] <lifeless> commenting on that bug
[22:53] <lifeless> jml: as you are around
[22:53] <lifeless> jml: testtools
[22:53] <lifeless> I reviewed and so on; what did you think
[22:53] <jml> lifeless: I haven't had a chance to look
[22:54] <lifeless> kk
[22:59] <lifeless> ok, I can has review please: https://code.launchpad.net/~lifeless/launchpad/bug-636158/+merge/50534
[23:00] <lifeless> thumper: ^ if you have the time, would be awesome
[23:01]  * thumper is writing some groovey matchers
[23:02]  * thumper enlunchenates
[23:16] <jelmer_> lesigh
[23:16] <jelmer_> thumper: I almost looked up enlunchenates in my English->Dutch dictionary
[23:19] <lifeless> Lol
[23:29] <lifeless> grrr
[23:30] <lifeless> so milestones pretend to be series related
[23:30] <lifeless> but they are really pillar related
[23:30] <lifeless> modelling fail
[23:30] <jml> lifeless: sinzui will tell you a story about that one.
[23:30] <lifeless> we need to say 'series in ... or milestone in ...'
[23:30] <lifeless> rather than 'series in... group by series, milestone'
[23:31] <lifeless> and our bug task search compiler is byzantine
[23:31] <jml> what's the sanest way to synchronously download a small thing via the web in Python?
[23:31] <lifeless> bzrlib.transport ?
[23:31] <lifeless> its a sane way
[23:32] <lifeless> but perhaps not /the/ sanest way
[23:32] <lifeless> jml: whats the story?
[23:32] <jml> lifeless: fetching a very small json dict
[23:32] <jml> bzrlib.transport is fine
[23:33] <lifeless> jml: I meant whats the story that sinzui would tell me
[23:34] <jml> lifeless: oh, I forget the details. I think for 3.0 he & beuno wanted to do a big refactoring of series/milestone relationship and got blocked.
[23:34] <jml> lifeless: not many people find "series" intuitive.
[23:36] <wallyworld_> thumper: can you ping me when you're back?
[23:36] <wgrant> lifeless: They made releases sane on top of milestones, but milestones atill aren't completely sane.
[23:36] <wgrant> lifeless: Also, milestones do have a series...
[23:37] <wgrant> But they also have a product, possibly because some old data doesn't have a series, which could be fixed :)
[23:47] <jml> how do we feel about this output?
[23:47] <jml> http://paste.ubuntu.com/569855/
[23:49] <lifeless> jml: interesting
[23:49] <lifeless> jml: would be cool to put that in aws-status
[23:49] <jml> lifeless: it's somewhat specific to Launchpad ec2test.
[23:50] <lifeless> why is it specifc?
[23:50] <wgrant> jml: How does it do that? SSH in?
[23:51] <jml> lifeless: it could be made more general, but right now it looks for special information about what the test run is and whether or not it's successful.
[23:51] <jml> wgrant: no, just fetches a JSON file that remote.py creates & keeps up-to-date.
[23:51] <wgrant> Ah, handy.
[23:52] <thumper> wallyworld_: ping
[23:52] <lifeless> jml: so, I know the bug you're looking at
[23:52] <wallyworld_> thumper: mumble?
[23:52] <thumper> wallyworld_: ok
[23:52] <lifeless> jml: and I suspect this won't meet the goal of it, though IMBW
[23:52] <lifeless> jml: won't this fail if e.g. your ip has changed?
[23:52] <lifeless> and isn't the goal of the bug to find things that have gone awol ?
[23:54] <lifeless> oh man
[23:54] <lifeless> bug task bug summaries are so broken right now
[23:54] <lifeless> pop quiz
[23:54] <jml> lifeless: if you don't have permission to hit those web pages, then yes, it will fail.
[23:54] <jml> lifeless: it can still show instances and uptimes though.
[23:55] <lifeless> should a bug be counted once per distroseries for aggregation into milestones ?
[23:55] <jml> which will be good enough for that case.
[23:55] <lifeless> jml: I would like this stuff to live outside the lp tree
[23:55] <jml> lifeless: too late?
[23:55] <lifeless> I think its great you're improving it
[23:55] <lifeless> jml: its never too late
[23:58] <wgrant> What's the process for shipit branches?