[00:00] Yippie, build fixed! [00:00] Project db-devel build (381): FIXED in 5 hr 47 min: https://hudson.wedontsleep.org/job/db-devel/381/ [00:24] lifeless: re the cassandra stuff, https://launchpad.net/~cassandra-ubuntu [00:25] lifeless: stable releases has 0.7.0 .. I'm working on 0.7.2 [00:25] lifeless: and yes, it installs fine on lucid and maverick [00:25] \o [00:25] I'll have to see about getting it into cat [00:30] wgrant: are you around? [00:30] lifeless: Yes. === lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews [00:31] wgrant: https://code.launchpad.net/~lifeless/launchpad/rabbit/+merge/50488 [00:31] if you're interested [00:32] I'd like another set of eyeballs before I land it [00:39] lifeless: I spent a while looking for the definition of BRANCH_NICK [00:39] Then I noticed the docstring. [00:39] Er, comment. [00:39] os_exec's name sucks. [00:40] This daemonisation stuff looks far more crackful than required. [00:40] What's wrong with subprocess? [00:41] I don't know yet [00:41] I was handed some working code that was run from Makefile and totally unsuitable for parallel testing etc [00:41] Ah. [00:41] I've done the minimal migration [00:41] theres more to go [00:41] Maybe the rabbit binaries don't daemonise and set PIDs and stuff. [00:42] I don't know which bits matter, and which don't. [00:42] It is awful and you need to clean it up, but yes, get it in the tree. [00:43] It looks like it will mostly work, but it's fairly insane. [00:44] * wgrant considers destroying Zopeless. [00:45] heh [00:45] go look at the original [00:45] :) [00:49] ec2 away [00:49] hopefully all the deps are already presen [00:49] t [00:52] I'm seriously considering having the production rabbit present on every node [00:53] it would make reasoning about many failure modes a lot easier [01:03] Hmm. [01:03] lifeless: Not just on the DB servers? [01:06] lifeless: Hmm, ~canonical can't see U1? [01:07] :( [01:20] * wgrant pokes staging in the eye. [01:47] Subclasses overriding private attributes... grr. [01:57] There is minified JS in our bzr history wtf wtf wtf [03:27] wgrant: probably not on the db servers at all [03:27] lifeless: Hmmm. [03:28] wgrant: why would it be on the db servers? [03:28] lifeless: I like keeping my critical data stores on machines that are meant to keep critical data stores. [03:29] rabbit sits in a spectrum [03:29] if (say) devpad croaks [03:29] we don't care if in-progress OOPSes are lost [03:30] all our machines have RAID level redundancy atm [03:30] the db servers aren't big because they have more redundancy, they are big because they are central load points [03:31] But the DB servers stay up. [03:31] so do the appservers [03:31] And if they go down then we are screwed anyway. [03:31] we can fail over to a slave [03:32] with rabbit by default the queued messages on a down load will return when the node returns [03:32] so it needs to be a pretty critical boom moment to permanently lose them [03:32] and whatever machine we bring another queue processor up on would establish new queues [03:33] Appservers having the only copy of any data is scary. [03:33] they have that all the time at the moment [03:33] Do they? [03:34] in progress requests [03:34] The persistent data stores are on four other machines, not appservers. [03:34] rabbit isn't a persistent data sttore [03:34] It persists across a request boundary, so it is persistent. [03:35] by that token so is an oops report [03:35] which means we have 5 persistent data stores on the app servers [03:35] (I disagree with that definition btw) [03:36] I don't want a taken-out appserver to reverse actions taken by a completed request. [03:36] Losing queue items does that. [03:36] I think you need to learn a little more about rabbit [03:36] 14:32:14 < lifeless> with rabbit by default the queued messages on a down load will return when the node returns [03:37] right [03:37] we wouldn't have queues on the appservers [03:37] we'd have a rabbit on them [03:37] if something got buffered there, it means the network is down [03:37] the only use case for that is oops forwarding atm [03:38] separately [03:38] So each queue has a master node that will not be on an appserver? [03:38] no, dropping a queues items won't reverse actions from a completed request [03:38] those actions clearly wouldn't have happened if that was possible [03:38] in rabbit, a queue lives on a node [03:39] exchanges forward messages to queues based on bindings [03:39] Right. [03:39] using oops as an example [03:39] I'd put the queue of oopses to process on a local rabbit where the oops site is [03:40] I'd make it entirely ephemeral because I don't care if we lose a few in the event of failure [03:40] (I care if we lost tonnes, but not a few) [03:40] Oh, right, it lets you configure the reliability? It's been ~a year since I last looked at it in depth. [03:40] each message [03:40] and each queue [03:41] you can be transactional [03:41] or you can be fire and forget [03:41] we're going to have a bunch of differing requirements [03:41] and how we deal with failures of various sorts is going to require case by case thought. [03:41] simply saying 'transactional persistent, done' would be a recipe for later pain. [03:41] Indeed. [03:43] anyhow [03:43] it doesn't have 2pc [03:43] AFAICT [03:44] :( [03:44] https://dev.rabbitmq.com/wiki/AckingSchemes#section_1 kindof talks about it [03:44] but hte images are fucked [03:44] As fucked as some of our import webs? [03:44] it looks like you build it on top of the characteristics [04:09] righto, amqp 1.0 does 2pc - but IIRC 1.0 kindof foundered [04:09] as a ill performing mess [04:09] Heh. [04:09] best you can do with 0.8 is queue things and not release until the db is committed [04:09] (thats what pgamqp does) [04:31] ok, wow time ;) [05:09] lifeless: Hah [05:10] lifeless: 3rd 85 yesterday [06:02] cool [06:38] grrr, rabbit failed on ec2 [06:38] is it easy to get a shell into an ec2 instance while its running tests ? [06:45] lifeless: It gives you the ssh command near the end of the output. [07:10] lifeless: ssh -v ec2test@ [08:06] next weekend I guess [11:06] * jml is around fo reals [11:17] hai [11:17] and bai [18:05] jml: I am on that bridge ;) - I have a patch where I'm trying to figure out what package to put the subunitstream model class in [19:49] grrr [19:49] dict.update() shouldn't be forbidden ... zopeSecurityProxy fail [19:59] morning lifeless [19:59] * thumper needs coffee [20:00] * thumper wanders off to find said coffee [20:00] hi thumper [20:52] 'morning thumper, lifeless [20:52] hi jelmer [20:52] lifeless: uhh, I see. [20:53] lifeless: there's a relatively easy way to fix the {}.update thing. [20:54] lifeless: I'd suggest "testdb" if you can't think of anything better. [20:55] lifeless: you could add 'checker.BasicTypes.update({dict: checker.NoProxy})' to lp_sitecustomize, which would completely disable security proxy checking for dicts. I don't think that would be a bad thing. [20:59] jml: I actually think the current structure of / is really annoying; I'd much rather we had / [20:59] jml: wouldn't that let things escape securityproxying? if an unproxied object is in the dict [21:26] wgrant: hey [21:26] wgrant: when you are around; can more than one distro share a distroseries row? [21:28] lifeless: I strongly doubt it. [21:28] StevenK: ok cool [21:29] StevenK, wallyworld_: standup [21:29] ? [21:29] thumper: Aye, coming [21:38] is there a helper for sql strings like Or for storm expressions? [21:38] I have a list of sql fragments [21:39] and want on length 1 -> foo[0] [21:39] on length 2+ -> "(" + " OR ".join(foo) + ")" [21:48] StevenK: https://code.launchpad.net/~thumper/launchpad/choosing-recipe-name/+merge/50530 [21:59] lifeless: I hope not, because that would break everything. [22:01] lifeless: oh yes, I think you're right. [22:01] lifeless: I don't find our "current" thing annoying. What I find annoying is that we haven't fully switched to it. [22:08] lifeless: however, if you wanted to change, and were actually serious about finishing that change and could guarantee that we wouldn't have a mutiny, then I wouldn't mind a / approach. They seem equally preferable to me. [22:12] thumper: sorry, was having breakfast [22:14] wallyworld_: ok [22:16] rachel's kindle just arrived \o/ [22:18] you're just happy now cause you don't like sharing [22:19] jml: I'm waiting for the squad culture to bed down a bit [22:19] jml: I think it will become obvious to others, the more they work cross that the current structure is a nuisance [22:19] Hmm. [22:20] things like the buildmaster are good as they are [22:20] lifeless: Except it should be in services. [22:20] that is, they actually are components [22:20] wgrant: sure, or services should die in a fire and promote its contents [22:20] The LP_DB* envvars make me sad. [22:21] excuse the hyperbole [22:21] wgrant: yes [22:21] But I guess I should still support them :( [22:21] Apart from that Zopeless is just about gone. [22:21] or find and remove their uses [22:21] wgrant: nice [22:21] oh frell [22:21] I have the rabbitmq code in my bug-636158 branch [22:21] >< [22:21] Hah. [22:21] You are good at that. [22:22] the problem with "finding the uses" is that they may well be in initscripts and so on that aren't in the tree [22:22] mwhudson: Exactly. [22:22] And it's easy enough to support them in a much less OMG MUST KILL way than they are now. [22:22] So I will. [22:23] There is also a set of options that some scripts take that do the same thing :/ [22:24] mwhudson: losado grep [22:25] Also, shipit needs to die. [22:25] thats feature level work [22:25] Yes :( [22:25] it needs some apis [22:25] launchpadlib fixed to be usable in appservers [22:25] then its doable [22:26] Sure. I wasn't saying it was doable. Just that it pisses me off when I have test failures because bzr grep doesn't catch stuff outside the tree. [22:28] Why did staging not try to restore over the weekend? [22:30] wgrant: +1 [22:31] I need a reviewer [22:31] http://bazaar.launchpad.net/~lifeless/launchpad/bug-636158/revision/12413 [22:31] pisses me off even more when I have deployment failures because of the same. [22:32] this is a refactoring [22:32] it has some stuff I'm not entirely happy with, but I don't konw if there are existing facilities in tree, or if I should make some. [22:37] lifeless: My first comment is that the method should probably be on IPublishingSet, not IDistroSeriesSet. [22:38] why ? [22:38] Because all this publishing-related stuff on DS bloats DS massively. [22:38] It doesn't belong there. [22:38] mmm, thats fair enough, but why does it belon on publishing set ? [22:39] why would someone look there rather than distro/distroseries ? [22:39] Because in the end you are asking for the latest publication. [22:39] thats an implementation detail [22:39] isn't it ? [22:39] No. [22:39] I don't think so. [22:40] theres a related oddness [22:40] There's more than one. [22:40] distro.getCurrentSourceReleases != distro.development_series.getCurrentSourceReleases [22:40] and it appears to be deliberate [22:40] but I'm not sure users will understand [22:41] I didn't know that both existed. [22:41] indeed [22:41] Distribution's finds the latest in any series? [22:41] there would be much less duplcicate code if they didn't both exist [22:41] correct [22:42] That's braindead. [22:42] But Bugs uses it. [22:42] indeed [22:42] indeed [22:42] I don't recall anywhere else that does. [22:42] I'm in correctness-preserving refactoring at the moment [22:42] so I am not going to fiddle [22:42] And I filed a bug on the Bugs behaviour a few years ago. [22:42] :( [22:42] but - it makes me care less about the duplication I have that I was worried about [22:43] because we can just delete the method in a separate pass [22:44] Right. [22:44] Make sure you file a bug. [22:45] where is the bug you filed ? [22:45] (because that would be the one to use) [22:46] Bug #279513 [22:46] <_mup_> Bug #279513: Distribution source package tooltip in bugtask table shows most recent SPP < https://launchpad.net/bugs/279513 > [22:46] Not as long ago as I thought. [22:48] so [22:48] it could show most highest version [22:48] Or latest in the dev series. [22:48] right [22:48] which I think is better and simpler [22:48] Definitely. [22:49] hmm... it seems that Matcher and Mismatch aren't in the __all__ for testtools.matchers [22:49] I'm guessing they should be [22:50] yeah, they should. [22:50] they might even be in trunk. [22:50] They were skipped deliberately [22:50] because from testtools.matchers import * shouldn't import the base classes [22:50] this was perhaps a mistake [22:51] erm [22:51] 400 Bad Request Bad Request Your browser sent a request that this server could not understand. Apache/2.2.14 (Ubuntu) Server at bugs.launchpad.net Port 443 [22:51] wtf [22:52] What was the request? I've seen that on bad hostnames, but that's about it. [22:52] commenting on that bug [22:53] jml: as you are around [22:53] jml: testtools [22:53] I reviewed and so on; what did you think [22:53] lifeless: I haven't had a chance to look [22:54] kk [22:59] ok, I can has review please: https://code.launchpad.net/~lifeless/launchpad/bug-636158/+merge/50534 [23:00] thumper: ^ if you have the time, would be awesome [23:01] * thumper is writing some groovey matchers [23:02] * thumper enlunchenates [23:16] lesigh [23:16] thumper: I almost looked up enlunchenates in my English->Dutch dictionary [23:19] Lol [23:29] grrr [23:30] so milestones pretend to be series related [23:30] but they are really pillar related [23:30] modelling fail [23:30] lifeless: sinzui will tell you a story about that one. [23:30] we need to say 'series in ... or milestone in ...' [23:30] rather than 'series in... group by series, milestone' [23:31] and our bug task search compiler is byzantine [23:31] what's the sanest way to synchronously download a small thing via the web in Python? [23:31] bzrlib.transport ? [23:31] its a sane way [23:32] but perhaps not /the/ sanest way [23:32] jml: whats the story? [23:32] lifeless: fetching a very small json dict [23:32] bzrlib.transport is fine [23:33] jml: I meant whats the story that sinzui would tell me [23:34] lifeless: oh, I forget the details. I think for 3.0 he & beuno wanted to do a big refactoring of series/milestone relationship and got blocked. [23:34] lifeless: not many people find "series" intuitive. [23:36] thumper: can you ping me when you're back? [23:36] lifeless: They made releases sane on top of milestones, but milestones atill aren't completely sane. [23:36] lifeless: Also, milestones do have a series... [23:37] But they also have a product, possibly because some old data doesn't have a series, which could be fixed :) [23:47] how do we feel about this output? [23:47] http://paste.ubuntu.com/569855/ [23:49] jml: interesting [23:49] jml: would be cool to put that in aws-status [23:49] lifeless: it's somewhat specific to Launchpad ec2test. [23:50] why is it specifc? [23:50] jml: How does it do that? SSH in? [23:51] lifeless: it could be made more general, but right now it looks for special information about what the test run is and whether or not it's successful. [23:51] wgrant: no, just fetches a JSON file that remote.py creates & keeps up-to-date. [23:51] Ah, handy. [23:52] wallyworld_: ping [23:52] jml: so, I know the bug you're looking at [23:52] thumper: mumble? [23:52] wallyworld_: ok [23:52] jml: and I suspect this won't meet the goal of it, though IMBW [23:52] jml: won't this fail if e.g. your ip has changed? [23:52] and isn't the goal of the bug to find things that have gone awol ? [23:54] oh man [23:54] bug task bug summaries are so broken right now [23:54] pop quiz [23:54] lifeless: if you don't have permission to hit those web pages, then yes, it will fail. [23:54] lifeless: it can still show instances and uptimes though. [23:55] should a bug be counted once per distroseries for aggregation into milestones ? [23:55] which will be good enough for that case. [23:55] jml: I would like this stuff to live outside the lp tree [23:55] lifeless: too late? [23:55] I think its great you're improving it [23:55] jml: its never too late [23:58] What's the process for shipit branches?