[09:07] heya TheMue [09:24] fwereade: Hi [09:24] fwereade: Phew, it's raining cats and dogs here. [09:25] TheMue, it's pretty hot here [09:25] TheMue, we're just approaching the too-damn-hot point of the year [09:25] TheMue, give me a few days and I'll be begging for a decent rainstorm :) [09:26] * TheMue dcc's fwereade some rain. [09:26] TheMue, :) [09:27] TheMue, so how's it going? I hope the format 2 stuff isn't too much of a hassle -- I feel like I maybe should have done it myself, but I got caught up in the relations and I felt it was getting pretty important [09:29] fwereade: I'll do firewall first, then format 2. [09:29] TheMue, ah, excellent [09:30] TheMue, are we going with the security groups style or are we doing it properly this time? [09:31] fwereade: I'm talking about todays firewall.py in state. security and auth will be handled later. [09:31] TheMue, ah cool [09:31] fwereade: firewall is used by the PA. [09:32] fwereade: So I'm moving it out of state to cmd. [09:32] * fwereade suddenly gets suspicious [09:32] * fwereade goes to read code a mo [09:33] TheMue, doesn't implementing that presuppose the security groups approach? [09:33] fwereade: As far as I've seen yet not. [09:34] fwereade: But I've just started. [09:34] TheMue, it seems to me that if the PA is going to use it, then we're assuming that the PA will remain responsible for opening/closing ports [09:35] TheMue, a proper solution using firewalls on the units surely shouldn't involve the PA at all? [09:36] fwereade: Sorry, don't know. [09:36] TheMue, blast, wish niemeyer was on [09:37] fwereade: So how would your solution look like? [09:38] TheMue, unit agent messing with iptables, rather than PA messing with the provider [09:38] TheMue, we've certainly talked about our use of security groups being a serious problem, and about the need for a cross-provider firewall solution [09:39] fwereade: Pls go on ... [09:40] TheMue, but it would not necessarily be *irrational* for us to go with the tried, tested, known-working-at-small-scale solution (given the time constraints that are starting to wear at me slightly) [09:41] TheMue, the problems with security groups are (1) aws is really not designed to handle what we're doing with them and (2) the solution only works for aws [09:41] TheMue, (2) is not important wrt our critical short-term goals [09:42] TheMue, but disregarding (1) feels like the sort of decision that we should get some sort of consensus on before writing code that presupposes it [09:42] TheMue, s/presupposes it/presupposes that approach/ [09:43] fwereade: That are worries, ok, but how would a proper solution look like? [09:44] TheMue, I'm afraid I don't have a clear idea of the *precise* problem with our use of security groups... just that we're not meant to use any, and an apocryphal amazon engineer was said to look somewhat horrified by the prospect :) [09:44] TheMue, I think it comes down to the *unit* agents watching the ports that should be open in their container and taking charge of it themselves [09:45] TheMue, we'd still need *some* security groups, but probably just 2: one for PA machines and one for everything else [09:45] TheMue, make sense? [09:46] fwereade: Yep, so far understandable. === wrtp is now known as rogpeppe [09:46] rogpeppe: Hey, you are not here. ;) [09:46] TheMue: that's right. i'm an invisible ghost. [09:47] TheMue: i've been given special dispensation :-) [09:47] rogpeppe: Ah, ok, then it's ok. [09:47] TheMue, that's pretty much it... [09:47] rogpeppe, heyhey [09:47] fwereade: yo! [09:47] rogpeppe, are you aware of any official preference as to how we implement firewalling this time round? [09:47] i seem to remember we've got a meeting scheduled in 13 minutes, so i thought i'd try and turn up for it... [09:47] (maybe i've got it wrong though!) [09:48] rogpeppe, btw, finished To Hold Infinity, very enjoyable [09:48] fwereade: cool, glad you enjoyed it. am enjoying wwz, in a slightly grim kinda way [09:48] hmm, firewalling [09:49] until we containerise everything, i think the current approach is probably the only one [09:49] rogpeppe, enjoying Axiomatic too, fun to have a more thinky, less experiencey read once in a while [09:49] rogpeppe, ah, expand please? I don't see the issue [09:50] rogpeppe, after all everything is containerised already... in a sense... which feels like the appropriate sense for this context [09:50] fwereade: how do we firewall without making use of ec2's facilities? [09:50] rogpeppe, iptables? [09:51] fwereade: can't anything get around that? [09:51] rogpeppe, I have always presumed that it works as advertised, but I can't point to anything proving that [09:52] rogpeppe, and I'm not saying we don't use security groups at all -- we have to -- but we know that using one per machine is a problem [09:52] rogpeppe, I just don't know whether it's the sort of problem we want to fix now, or the sort of problem we leave for 13.04 [09:53] fwereade: am i right about the meeting, BTW? [09:53] rogpeppe, er, I have no idea... I had a vague feeling it was weds, but maybe I missed another change [09:54] rogpeppe, but davecheney is on, and that may lend support to your theory ;p [09:55] dammit, it's an hour later [09:55] bugger, my dispensation is invalid [09:55] fwereade: iptables are manipulatable by root, and the charms run as root. [09:55] fwereade: we need to talk to niemeyer about this [09:56] fwereade, TheMue: well, gotta go. will miss the meeting, i think. have fun, and post any interesting/relevant conversations to juju-dev, where i will see 'em and sneakily read 'em... [09:57] rogpeppe, yeah, indeed -- I'm not even sure I have a strong position on this, I just feel it's something we should get niemeyer's input on before we implement code that supposes either way [09:57] rogpeppe, enjoy the holiday :) [09:57] rogpeppe: OK, have fun. [09:59] TheMue, I think that either way you can certainly implement something that keeps an eye on both sets of conditions, and emits events when ports should actually open or close [09:59] fwereade: we could cache groups, because we're unlikely to have too many configurations of ports. [09:59] fwereade: which might mitigate the issue [09:59] fwereade: That's what firewall does today. [10:00] TheMue: ah, it must've changed since i last looked [10:00] TheMue: i thought there was one group for each machine [10:00] anyway, gotta go [10:00] rogpeppe: The firewall.py does not very much. It's only used by the PA. [10:01] TheMue, where does it do that? [10:01] TheMue, I don't see anything that shares groups in there [10:02] fwereade: I didn't say anything about groups. I meant watcing the ports. [10:02] TheMue, if anything does that, it's in the individual provider's open_port/close_port methods [10:02] TheMue, ah got you [10:03] TheMue, all I'd suggest then is to make sure that the thing that watches an individual machine remains distinct from the thing that watches all machines [10:05] TheMue, do I appear to be approximately sane there? [10:05] fwereade: I'll keep it in mind. I'm not yet deep enough in it. Just started the porting and as a prerequisite the watcher for the exposed flag. [10:06] TheMue, cool [10:06] * fwereade starts to wonder whether he's right about it being up to the UA... maybe the MA would be better... [10:07] fwereade: You've got more insight than me. I sometimes miss an architecture graphics where the components, their responsibilities and roles and how they communicate are visible. [10:07] * TheMue is a very visual being. [10:07] TheMue, I think the issue there is that the responsibilities in python are not necessarily as they should be [10:08] TheMue, eg theMA being responsible for the first download of the charm, and the UA being responsible for subsequent ones [10:08] fwereade: OK, then two graphics: todays implementation and wanted implementation [10:09] TheMue, the first one is of limited value and the second one is subject to change as we figure out *how* we should be doing things... [10:09] *should* [10:09] TheMue, hopefully without succumbing to second-system effect [10:11] fwereade: That's a problem of working remote. I've used whiteboards a lot for a discussion of how something is and how it should change. [10:11] fwereade: My intention is now first class diagram [10:11] s/now/no/ [10:34] moin. [10:54] Aram, heyhey [11:03] Aram: Moin [11:05] fwereade: TheMue: had a little bit of fun yestarday: http://play.golang.org/p/D-qPq8uIw3 [11:14] Aram, haha, nice [11:18] Aram: *lol* [13:08] Hmm, seems it's time for a topology watcher. [14:05] fwereade: Any experiences with the size of topologies in large installations? [14:06] TheMue, all I know is that yaml was too big for the 2k deployment, json makes it small enough for that with room to spare [14:06] fwereade: I'm asking because topology watchers keep an old one in memory and pass it and a new one to the using callbacks/watcher users. [14:07] TheMue, IIRC max ZK node size is 1MB, so order of that, I guess [14:07] fwereade: I would store it already parsed, so there should be not whitespace problem. [14:08] TheMue, it shouldn't be an overwhelming load though [14:08] fwereade: ok [14:08] TheMue, however you way want to look at recent topology watchers in go, which don't keep a whole topology around [14:08] TheMue, they just keep the bits they're interested in [14:09] fwereade: WHich ones you're talking about? Most I've seen so far watch simple nodes. [14:09] TheMue, MachinesWatcher and MachineUnitsWatcher [14:10] fwereade: Also the event of change always forces me to at least read one complete node. [14:10] TheMue, also ServiceRelationsWatcher, new in review today [14:10] TheMue, yeah, you always read the whole new topology [14:10] TheMue, no reason to keep unit info around when all you care about is relations for one service [14:10] fwereade: Thx, will take a look. I need it for the ServiceUnitsWatcher. [14:11] TheMue, cool [14:11] TheMue, a suggestion, don't know if it applies: [14:12] * TheMue listens [14:12] TheMue, when doing the ServiceRelationsWatcher, it was very convenient to add (*Service)relationsFromTopology(t *topology) and use it both in Relations and the watcher [14:13] TheMue, haven't looked at MW or MUW to see whether they'd benefit from similar [14:13] fwereade: OK, will look, it sounds good. [14:14] TheMue, it may be that the code to extract the stuff we care about is small enough not to bother in those cases and maybe in yours [14:15] fwereade: Huh, the last sentence is difficult for me to understand. [14:15] TheMue, sorry [14:16] TheMue, I'm saying that getting a []*Relation from a service and a topology is enough work to make it worth factoring out [14:16] TheMue, but getting a []*Unit from a service and a topology may be trivial enough that it's better to duplicate the code [14:17] TheMue, similar may apply to MW and MUW [14:17] fwereade: OK, understand, I will see how much it is. [15:19] Hellos! [15:27] aujuju: Is juju specific to ubuntu OS on EC2 [closed] [15:31] niemeyer: Hello to the far west. [15:35] TheMue: Hi :) [15:35] TheMue: How's been the weekend? [15:37] niemeyer: Fine, a but support for my brother in law, he is building a house, and sitting on the couch on Sunday, it rained cats and dogs. [15:37] niemeyer: And your travel to SFO? [15:37] TheMue: Hah :) [15:38] TheMue: The trip was quite fine [16:17] Hmm.. so it seems that Go's behavior on redirections has changed somehow.. lpad seems broken :( [16:17] * niemeyer investigates [16:27] niemeyer, heyhey [16:28] niemeyer, TheMue: please confirm that it is not safe to select on a send to a channel that might be closed [16:28] fwereade: It is actually safe [16:29] niemeyer, really? oh, cool [16:29] fwereade: It depends a bit on what you mean by that, though [16:29] fwereade: Oh, wait.. *send*.. hmm [16:29] niemeyer, select {dodgy <- event: blah; <-t.Dying()} [16:29] niemeyer, select {dodgy <- event: blah; <-t.Dying():} [16:30] fwereade: No, that's not ok, sorry for the misinfo [16:30] niemeyer, no worries :) [16:32] fwereade: It's considered a bad practice (hence why it blows up) because it's a clear statement that the life time of the channel is messed up. [16:32] niemeyer, that was what I thought [16:32] hi niemeyer, how's SF? [16:32] niemeyer, and I'm pretty sure I'm in a situation where I can just leave the channel alone without ever closing it anyway :) [16:33] Aram: Pretty nice, sunny.. had a good time with Andrew yesterday as well [16:33] niemeyer: nice. [16:33] fwereade: That's a possible answer [16:33] niemeyer: did you see my silly paste entry? http://play.golang.org/p/D-qPq8uIw3 [16:34] Aram: Yeah, that was awesome :) [16:35] robbiew: ping [16:35] niemeyer: I could have made it an actual animated PNG, but animated PNGs don't work in webkit browsers yet. [16:35] niemeyer: pong [16:35] robbiew: Heya [16:35] robbiew: Do we have a meeting today? [16:36] niemeyer: heh...as usual, I have no idea...checking [16:36] Aram: Surprisingly short [16:36] robbiew: Cool.. I better find out a good way to call out of the hotel if so [16:36] niemeyer: no meeting [16:37] robbiew: Super, thanks for checking [17:28] gn all [18:08] niemeyer, btw, I have to go again in a sec, but I meant to ask: [18:08] niemeyer, are we planning to replicate the security-group firewalling for 12.10? [18:09] fwereade: Yeah [18:09] fwereade: Should be easy, and gets us parity [18:09] fwereade: We can then fix it another way later [18:09] fwereade: But, [18:09] fwereade: We should try to make the implementation sensible, so that we can reuse bits [18:09] niemeyer, yep, I approve (despite emotionally wanting to Do It Right ;)) [18:09] fwereade: I've been talking to Frank about that [18:10] fwereade: He's working on the firewall port watcher stuff [18:10] niemeyer, excellent, I realised I didn't know what plan we were following when he mentioned it this morning [18:10] fwereade: That we have under state/firewall.py in Python [18:10] fwereade: But with some twists.. the Python version assumes it knows about a provider and what not [18:10] fwereade: The Go version will be a normal watcher [18:10] niemeyer, yeah, I presume we'll just be outputting changes [18:10] niemeyer, perfect [18:11] fwereade: Exactly [18:11] niemeyer, I would guess two levels of watchers so we can reuse the inner one when it becomes the MA (UA???)'s responsibility? [18:12] fwereade: Yeah, we actually already have one in the unit [18:12] fwereade: So this is adding the second one, on Machine [18:12] niemeyer, ah, nice [18:12] fwereade: WatchPorts [18:12] fwereade: I think we'll use the exact same thing when we move [18:12] fwereade: The difference is that the machine agent will call Machine.WatchPorts, rather than the provisioning [18:13] niemeyer, perfect :) [19:27] mramm: looking for me? [19:33] niemeyer: somethins intriguing is happening... compare this: http://bazaar.launchpad.net/~gophers/juju-core/trunk/view/head:/mstate/state.go#L56 with this: https://codereview.appspot.com/6304099/diff2/9002:18002/mstate/state.go [19:33] the machine function [19:33] is different :) [19:33] how can this be? [19:33] the AllMachines function is the same though, and both have been altered in the same commit. [19:34] Aram: Why should they be the same, just so I get the context? [19:35] niemeyer: because I submitted what's on codereview, and what's in launchpad seems an earlier version. [19:36] Aram: Ah, it's actually not [19:37] Aram: https://codereview.appspot.com/6330045/ [19:37] interesting. [19:38] why the removal of that branch? [19:38] Aram: The new error will look like "can't get machine 42: not found", which is fine [19:38] Aram: I had to touch that logic due to the NotFound renaming [19:38] Aram: (ErrNotFound now) [19:38] yes, yes. [19:39] Aram: But rather than replacing it, I just dropped and allowed the underlying error to go through as per the message above [19:39] well yes, that was my initial version as well. [19:39] Aram: Not really [19:39] Aram: your initial version was the opposite.. any error would lead to "not found" [19:40] right. [19:41] niemeyer: anyway, thanks for clearing the confusion. [19:41] Aram: np, and sorry for the trouble.. I wanted to ask for your review on it too, but at the same time didn't want to leave trunk broken [19:41] of course [20:01] niemeyer: first piece of the puzzle: https://codereview.appspot.com/6341050 [20:01] Aram: Awesome, thanks! [20:12] niemeyer: the diff on codereview is always done against lp:juju-core? can't I do it against some other branch I have? [20:13] Aram: You can, with -req [20:13] Aram: It only allows trees rather than graphs, but it works [20:14] strange, that's what I did, lbox propose -cr -wip -req="lp:~aramh/juju-core/mstate-charm-basic" [20:14] Aram: -req has to be used at propose time [20:14] but it generated this: https://codereview.appspot.com/6325057 which is wrong because it should only be two lines [20:14] Aram: After the merge proposal is created, it doesn't work anymore [20:14] Aram: (because Launchpad doesn't allow changing it) [20:14] can I delete a merge proposal and do it again from the same branch? [20:14] Aram: Yeah [20:14] Aram: That works fine [20:15] ok, thanks [20:17] np [20:33] Okay, lpad works again.. I'll go out for finding some food, and will be back to work on reviews [22:50] morning davecheney [22:51] morning Aram [22:51] hows it going ? [22:51] great [22:56] niemeyer: I believe three pieces of the puzzle should be in the queue now [22:56] Aram: Super, thanks! [22:56] davecheney: Heya [22:57] howdy lads === Aram2 is now known as Aram