[00:06] <_mup_> ensemble/expose-provision-machines-reexpose r296 committed by jim.baker@canonical.com [00:06] <_mup_> Mocked version of verifying connection closed terminates topology watch [00:09] <_mup_> ensemble/expose-provision-machines-reexpose r297 committed by jim.baker@canonical.com [00:09] <_mup_> Removed need for mocking [00:24] <_mup_> ensemble/expose-provision-machines-reexpose r298 committed by jim.baker@canonical.com [00:24] <_mup_> Test clean up/comments; also now test what happens when the client is disconnected before the watch is even started [00:42] <_mup_> ensemble/expose-provision-machines-reexpose r299 committed by jim.baker@canonical.com [00:42] <_mup_> Comment [00:43] <_mup_> ensemble/expose-provision-machines-reexpose r300 committed by jim.baker@canonical.com [00:43] <_mup_> Removed unused import [00:49] <_mup_> Bug #812619 was filed: Re-exposing a service does not work with the provisioning agent < https://launchpad.net/bugs/812619 > === daker_ is now known as daker [12:00] does anyone know anything abut the config parsing in ensemble? [12:31] Morning all [12:31] fwereade: I'm sure someone knows about it :-) [12:31] fwereade: What are you looking for? [12:32] niemeyer: I've written a test for the orchestra schema, and I can't get both that and the ec2schema tests to pass at the same time [12:32] niemeyer: the problem is that we get an error for every schema we check [12:33] niemeyer: and we don't necessarily get the one I'm testing for [12:33] niemeyer: (that's controlled by the order in which we specify the provider schemas in the OneOf) [12:34] niemeyer: am I missing something obvious? :) [12:34] fwereade: Hard to tell without knowing what the error is [12:35] niemeyer: if I'm missing (say) orchestra-server, then we get an exception from coerce() [12:35] niemeyer: quite rightly [12:35] fwereade: Can you please paste the error and the code somewhere? [12:35] niemeyer: but we also get an exception when we attempt to coerce() to the EC2 schema, because we don't have (say) control-bucket [12:36] http://paste.ubuntu.com/647260/ [12:36] fwereade: Well, we shouldn't attempt to coerce an orchestra config to an EC2 schema [12:36] niemeyer: agreed [12:37] fwereade: They have to be separate schemas [12:37] niemeyer: hm, ok, so we just do a first pass to determine the type [12:38] niemeyer: and then try to apply the actual schema we've detected [12:38] niemeyer: it's not intractable, but I didn't want to lay heavily into that section of code without checking with someone [12:39] fwereade: Hmmm.. just a sec [12:39] niemeyer: no rush, I was just about to have lunch anyway [12:40] niemeyer: and I've got some unrelated issue with my VMs that I need to sort out before I can be confident it's worth proposing a merge [12:41] niemeyer: (I have the cobbler communication tested, and I can boot machines, but they don't actually manage to install) [12:41] niemeyer: talk later? [12:44] fwereade: I see what you mean [12:44] fwereade: This is a shortcoming in our schema library [12:44] fwereade: Let me ponder for a moment on the issue and we catch up later [12:45] fwereade: You're not blocked on this, right? [12:45] fwereade: I think you're doing the right thing, FWIW.. it's just the schema library itself that should be fixed [13:29] hey guys, I have a question for you :) [13:29] got ensemble module on Mac but when I execute it says [13:29] ImportError: No module named zookeeper [13:29] that I assume is not the txzookeper python library [13:29] so which one should I look to import? [13:30] zookeeper itself? [13:36] ello? :) [13:50] lynxman: Hey [13:50] lynxman: Yeah, that's the C extension that is provided by zookeeper itself [13:51] lynxman: txzookeeper is a wrapper on top of that which "Twistifies" (ugh) the library [13:52] niemeyer: that was my reading of the issue, but I thought it was worth checking ;) [13:53] niemeyer: and no, I'm not blocked yet, but I should be in a little while [13:54] niemeyer: I'll let you know when it bcomes urgent :) [13:59] niemeyer: cool, so I'll just create the python extensions for that :) [14:11] fwereade: Thanks :) [14:11] lynxman: Super :) [14:50] fwereade: howdy! do you think I should do something like this?: http://paste.ubuntu.com/647347/ [14:58] RoAkSoAx: sounds sensible to me [14:58] RoAkSoAx: will make it easy to pull out whatever's common [14:59] RoAkSoAx: btw, you remember we got everything running on my machine [14:59] RoAkSoAx: seems we didn't quite, was wondering if you had a few minutes to point me in the right direction [15:00] RoAkSoAx: I can launch a machine but the install doesn't complete [15:01] fwereade: you need to change the IP of the proxy [15:02] fwereade: in the preseed [15:04] fwereade: that's what comes to my mind [15:05] RoAkSoAx: the first errors I see are a couple of "debconf-copydb: cannot open question file"s [15:06] RoAkSoAx: but it keeps doing stuff after that [15:07] RoAkSoAx: it starts to look fatal around "cannot stat /etc/default/puppet", while setting up ubuntu-orchestra-client [15:19] niemeyer: just FYI, it's done https://trac.macports.org/ticket/30237 [15:20] fwereade: remove orchestra-client from the preseed [15:20] fwereade: or extra packages [15:20] that is from the preseed *inside* the cobbler VM [15:20] /var/lib/cobbler/kickstarts/ensemble.preseed I think [15:26] RoAkSoAx: thanks: it took me an embarrassingly long time peering at the kickstarts dir on the host system before the penny dropped there [15:28] lynxman, awesome [15:29] lynxman, does the live check stuff mean it will auto pull the latest? [15:29] hmm. probably not with the checksums [15:31] hazmat: nope, we'll have to update manually for now [15:32] hazmat: macports is like BSD ports, it requires static versions, but once it's there updating it is easier [15:33] <_mup_> ensemble/trunk r276 committed by kapil.thangavelu@canonical.com [15:33] <_mup_> merge deploy-tolerates-broken-formulas-bug-810765 [r=fwereade,jimbaker][f=810765] [15:33] <_mup_> Ensemble deploy will behave nicely when formulas in a repository have broken metadatas, by ignoring them. [15:36] <_mup_> ensemble/trunk-merge r270 committed by kapil.thangavelu@canonical.com [15:36] <_mup_> merge trunk [15:41] <_mup_> ensemble/security-node-policy-def r276 committed by kapil.thangavelu@canonical.com [15:41] <_mup_> address review comments, inline owner_ace func, additional doc comments on rule signature. [15:47] fwereade: heh it happens [16:05] lynxman: Woot! [16:08] niemeyer: \o/ [16:08] niemeyer: as soon as it's live mac developers will be able to use and deploy ensemble as well [16:08] (client side of course) [16:08] lynxman: Please keep us updated.. quite curious to see how that'll go [16:09] niemeyer: will do :) [16:10] Alright.. I'll step out for lunch [16:10] Just off of a meeting [16:10] Oh [16:10] Btw, [16:11] For those who didn't receive it, we have a developer (all formula and core are welcome) meeting today at 17:30UTC for sync up [16:11] (voice) [16:11] In Google+ [16:11] Please be here if you're interested [16:12] * niemeyer => lunch [16:14] <_mup_> Bug #810808 was filed: Formula install hook never completes < https://launchpad.net/bugs/810808 > [16:33] niemeyer: how does one attend this meeting? [16:34] niemeyer: is it irc only? [16:34] niemeyer: i see "(voice)" and "In Google+", but I don't see how to join [16:36] kirkland: I think someone has to start it which hasn't happened yet as far as I can see [16:38] bcsaller: kthx [16:39] <_mup_> Bug #812996 was filed: Deploy should support config options < https://launchpad.net/bugs/812996 > [16:43] bcsaller, it looks like you have a conflict against trunk in that merge proposal [16:47] kirkland, are you on google plus? [16:47] hazmat: sadly, yes [16:47] hangouts are cool...but setting them up sucks [16:48] unless you create a damn circle for each individual friend...or always hangout with the same group [16:48] You don't have to "hangout" .. you can just video chat with somebody [16:49] SpamapS: where's the fun in that? [16:49] but if I want to video chat with 2 people...I need to hangout, right? [16:49] Probably [16:49] if those two people are in my circles, i should be able to just select them and hang...but you can't [16:50] you have to invite them..and hope you have the right address for them that they've used with google + [16:50] Beta crap [16:50] your worlds... will collide [16:50] [16:50] I guess I'm getting what I paid for [16:50] lol [16:52] robbiew: you're paying with your own soul [16:53] fwereade, niemeyer, jimbaker .. we're all hanging out in hangout if your interested [16:54] hazmat, i've been following this discussion about the hangout - but is there a URL for it? [16:54] jimbaker, you have to be logged into google plus, and it should appear there [16:54] let me check it again... i just joined yesterday [16:56] hazmat, looked around the page, just see something about creating a hangout [16:59] jimbaker, i'll send another link [16:59] jimbaker, what's your icon.. there are lots on google + [17:00] hazmat, my icon? ;) [17:00] jimbaker, its hard to figure out which of the many is you [17:01] hazmat, i guess i'm the one who has not yet associated a picture w/ my account yet [17:04] jimbaker, there are lots of those.. do you have profile info filled out? [17:04] jimbaker, actually easier might just be sending me your gmail address via priv msg [17:05] hazmat, i just added two bits - my picture, and employment at canonical [17:10] btw, my desktop doesn't have audio input [17:11] i will have to switch over to my laptop instead [17:14] <_mup_> ensemble/trunk r277 committed by kapil.thangavelu@canonical.com [17:14] <_mup_> merge security-node-policy-def [r=niemeyer,bcsaller][f=810510] [17:14] <_mup_> A security policy implementation that returns ACLs for zk nodes by path. === serue_ is now known as hallyn [17:20] hazmat: Oh, wasn't it 17:30? [17:20] I'll join [17:22] Hmm [17:22] bcsaller, fwereade, jimbaker, m_3: ping? [17:22] niemeyer, hi [17:22] niemeyer: ready when you are [17:22] niemeyer: heyhey [17:22] niemeyer: we're all hanging out [17:22] fwereade: Where? [17:23] I don't have any invitations here [17:23] niemeyer: google plus somewhere... not sure who set it up [17:23] fwereade: Someone has to invite me [17:23] fwereade: and bcsaller/jimbaker, I suppose [17:24] niemeyer: done, I think [17:24] hazmat is the person who set up the hangout [17:25] i think the easiest way is for every person who wants to join the meeting to pm hazmat their gmail address, assuming they have one [17:26] fwereade: No, nothing here [17:27] heh, bizarre [17:27] ney niemeyer... we're "hanging" [17:27] niemeyer: just tried again, just in case [17:28] hazmat: how do i get in? [17:28] it's a shame niemeyer's not there to enjoy our cool party [17:28] :p [17:28] Ah, got from bcsaller now [17:28] I think ben set it up... it appeared in my stream [17:28] just now, yeah [17:29] kirkland: We have our own party now! :-) [17:29] bummer [17:30] hazmat: we switched from that hangout to a new one [17:32] SpamapS: ping [17:32] robbiew: SpamapS: you guys coming? [17:32] Where do I go? [17:33] looks like i have a problem joining from my laptop... will try again [17:33] can someone send invite to gandelman.a@gmail.com plzz? [17:33] * robbiew has no delusions about the lack of value he will provide to the discussion [17:34] hazmat: ping? [17:34] hazmat: Where are you? [17:35] niemeyer, networking hicckup [17:36] Sorry guys, I want to be there, but I don't know where there is [17:36] SpamapS: invited [17:37] adam_g: invited [17:38] m_3: I have no evidence of any invitations. :p [17:38] SpamapS: not in your stream? [17:38] it shows up on the stream [17:38] go to ben's profile [17:39] well hidden [17:42] fwereade: I can hear you [17:43] niemeyer: looks like it was actually foretelling the near future [17:43] SpamapS: bad audio [17:43] trying to rejoin [17:46] not looking good :/ [18:02] SpamapS: bad audio again [18:10] my net connection has basically failed [18:15] fwereade: still arou nd? [18:15] RoAkSoAx: yep [18:15] RoAkSoAx: what can I do for you? [18:16] fwereade: want you to review something real quick [18:16] RoAkSoAx: sure :) [18:16] fwereade: http://paste.ubuntu.com/647512/ [18:17] fwereade: so from OrchestraSaveState __init__ I'm instancing OrchestraStorage which basically obtains the connection to the storage [18:17] fwereade: so my concerns are, instancing this : # FIXME: Should this be done like this? [18:17] self._storage = OrchestraStorage(self) [18:17] and in OrchestraStorage [18:17] obtain the config like this: [18:17] self.config = provider._provider.config [18:17] RoAkSoAx: I think that should be a self._provider.get_file_storage [18:18] RoAkSoAx: in general I think if we can do things via the provider we should [18:18] fwereade: ok so I keep the get_file_storage function on the MachineProvider then [18:18] RoAkSoAx: I think so [18:18] fwereade: alright, cool, thanks! [18:19] RoAkSoAx: a pleasure :) === daker is now known as daker_ [18:37] is this error message meaningful to anyone? i haven't seen it before: [18:37] http://paste.ubuntu.com/647521/ [18:39] whoa that does look confusing [18:39] are the hooks copied to some tmp dir? [18:40] * SpamapS prepares an upload of ensemble that runs the test suite during build [18:41] SpamapS: the file in /tmp is some template script generated by ensemble [18:42] http://paste.ubuntu.com/647525/ [18:49] Interesting.. here are the failures when running the test suite ina clean chroot... [18:49] http://paste.ubuntu.com/647530/ [18:50] exceptions.ValueError: Could not find AWS_ACCESS_KEY_ID [18:53] adam_g: the kill not finding the pid just means the hook exited without removing its pid file [18:53] just an unclean or abnormal exit for some reason? [18:54] hopefully [18:55] what happens in non debug-hooks mode? [18:55] same [18:56] or at least i think.. let me check once more [19:12] <_mup_> ensemble/expose-provider-ec2 r293 committed by jim.baker@canonical.com [19:12] <_mup_> Merged trunk [19:20] <_mup_> ensemble/expose-provider-ec2 r294 committed by jim.baker@canonical.com [19:20] <_mup_> Merged upstream expose-provision-machines-reexpose and resolved conflicts [19:25] <_mup_> ensemble/expose-provider-ec2 r295 committed by jim.baker@canonical.com [19:25] <_mup_> Default to NotExposed for watched_units [19:27] <_mup_> Bug #813112 was filed: test suite cannot run without an ssh key < https://launchpad.net/bugs/813112 > [19:28] Alright! [19:28] That was a long call [19:37] fwereade: ping [19:40] good to see the firewall actually opening/closing ports in response to the use of ensemble expose/ensemble unexpose [19:42] niemeyer, the only issue with dividing up the branches is that there's no ec2 provider implementation in expose-provision-machines; this doesn't come into place until the branch i'm currently working on, expose-provider-ec2 [19:43] jimbaker: Dividing which branches in this case? [19:43] the two branches i just mentioned, expose-provision-machines and expose-provider-ec2 [19:44] given that MachineProvider is a subclass of object, we don't currently have a way of defaulting say an empty implementation [19:44] expose-provision-machines works fine with the dummy provider of course [19:45] jimbaker: Sorry, ECONTEXT [19:45] jimbaker: What's the issue you're trying to address? [19:45] niemeyer, the MachineProvider implementation is required to implement open_port, close_port, get_opened_ports [19:46] this is done with the dummy provider [19:46] jimbaker: Yes, the implementation is required.. for the implementation? :-) [19:46] and is changed with the ec2 provider in expose-provider-ec2 [19:47] niemeyer, maybe i should have a branch called transitional that adds these empty methods, possibly throwing some sort of error like a provider error? [19:47] alternatively, this can be merged as a group into trunk [19:47] jimbaker: Probably not, if that's not required [19:47] jimbaker: Having just the dummy implementation, and then adding the ec2 one in a follow up sounds ok [19:47] niemeyer, sounds good [19:47] jimbaker: Unless you're detecting a problem with doing it that I'm still missing? [19:48] niemeyer, the problem is that if you run expose-provision-machines against an ec2 provider, it will fail, due to the lack of implementation [19:49] so i think the cleanest way of addressing this is to just merge these branches as a group [19:50] once they are all approved [19:52] niemeyer, the other thing is that once in trunk, this will force some formula writers and presumably all formula deployers to consider the firewall [19:52] niemeyer, so this will be an important change to communicate on my part :) [19:53] jimbaker: That's doable too [19:54] jimbaker: Yes, that sounds reasonable, on both counts [19:54] jimbaker: We should also discuss the change [19:54] niemeyer, cool [19:54] jimbaker: To see if we have to introduce a migration path or not [19:55] niemeyer, my feeling is that in discussing the migration, it would be best to combine the one line + comments change for the wordpress example formula with the doc changes (mostly from draft to main) [19:56] that would be the branch after the one i'm currently working on [19:56] jimbaker: Hmm.. the example formulas don't really worry me.. what we have to discuss with SpamapS, m_3, adam_g, negronjl, and lynxman is the fact that their formulas will stop working [19:56] niemeyer, after expose-provider-ec2 and expose-migration (the branch just discussed), there's a firewall solution in place for ec2 [19:57] Well.. not really stop working per se [19:57] niemeyer, correct [19:57] They'll just not open the firewall [19:57] Which doesn't sound like a big issue, but we have to talk [19:57] jimbaker: Can you please bring up that conversation in the mailing list? [19:57] niemeyer, right, they can always manually open the firewall using their favorite tool [19:57] Yep [19:58] niemeyer, i will do the mailing list discussion [19:58] or at least kick it off ;) [19:58] jimbaker: Thanks [19:59] niemeyer, then after all this dust settles, i can then work on supporting unexpose and expose hooks. but that's more for completeness than what i would imagine is a common use case [19:59] * hazmat goes into review mode [20:01] hazmat? [20:02] from my point of view guys, change what needs to be changed in ensemble and let me know what changed. I'll adapt the formulas to match the work. [20:03] in other words...don't worry about my formulas. Just let me know what changed and I'll adapt them to work. [20:03] * negronjl is out to a meeting [20:04] negronjl, it boils down to: open-port PORT[/PROTOCOL] in a hook script; ensemble expose SERVICE_TO_BE_EXPOSED [20:04] ensemble does the rest of the work [20:04] hazmat: Looking at that can isn't fun [20:06] niemeyer, back [20:10] when using add-unit, is formula uploaded from locall repository to new unit or is it fetched from bootstrap node such that all units are running the same revision of a formula? [20:17] jimbaker, niemeyer: +1 on what negronjl said (different answer later when things are in production tho!) [20:20] I wonder if there is some sort of kernel level event service that can be tapped to find out what ports a process is using. [20:20] adam_g: dunno, never tried to add-unit without upgrading changes first... big issue though [20:20] like inotify but for sockets instead of files [20:22] SpamapS: doesn't netstat have a mode where it continues watching sockets? [20:24] m_3: --continuous .. that polls tho [20:25] m_3: i'm trying to add new units that have modified configuration. is this doable? [20:25] configuration? [20:25] do we have config settings in trunk yet? [20:26] I really want to start mucking with that [20:27] i need each unit deployed to have a zone # assigned to it, and i'd like to be able to decide which zone a member joins when i deploy [20:27] adam_g: I've only been able to do that by adding the config info to an interface and letting another related service manage it [20:28] i was thinking, in the meantime, of just hard-coding and updating it each time i deploy a new unit, but that doesn't work if they all share the same revision [20:30] other service can do 'relation-list' on a relation-joined and then send out new info across the board [20:30] can peer solve this? [20:31] i'd like to decide which zones new units join, not the formula [20:31] yeah, that's just totally config [20:33] we'll get a 'deploy --config local.yaml' but I don't know if that'll be an option on add-unit [20:37] i think another way would be to deploy the service once for each zone, and add-units to that services/zone [20:37] zone_manager service, or just curl it from somewhere... bad, worse [20:38] separate formulas? [20:39] deploy --config zone1.yaml swift-storage swift-zone1 [20:39] deploy --config zone2.yaml swift-storage swift-zone2 [20:39] add-unit swift-zone1 [20:39] absolutely if that's in there yet [20:39] not yet, im just using some hax for now. [20:40] that's probably the best... I've mocked up some config-get calls already [20:41] we'll have to see what bcsaller did with add-unit and --config [20:42] I didn't, you configure the service, not the units [20:42] so in adam_g's previous example, will 'add-unit swift-zone1' pick up the config? [20:43] (that came from a local yaml?) [20:45] add-unit, no [20:46] oops, was scrolled back [20:49] I'm confused then... I'll dig trhough the branch to see what's up [20:49] m_3: service options are available available in all units and are the same [20:50] all units of a given service [21:01] <_mup_> ensemble/expose-provider-ec2 r296 committed by jim.baker@canonical.com [21:01] <_mup_> Doc strings, comments, test on EC2 error when accessing instances [21:08] <_mup_> ensemble/expose-provider-ec2 r297 committed by jim.baker@canonical.com [21:08] <_mup_> Docstrings for test_securitygroup [21:12] <_mup_> ensemble/expose-provider-ec2 r298 committed by jim.baker@canonical.com [21:12] <_mup_> PEP8 [21:35] Extremely sleepy right now.. [21:35] Will step out and lay down a bit [21:44] * negronjl is back [21:53] <_mup_> ensemble/expose-provider-ec2 r299 committed by jim.baker@canonical.com [21:53] <_mup_> Test for EC2 errors in security group ops [21:56] <_mup_> ensemble/expose-provider-ec2 r300 committed by jim.baker@canonical.com [21:56] <_mup_> PEP8/PyFlakes [22:01] adam_g, formulas are uploaded to formula storage (s3) and downloaded from there for each deployed unit [22:02] adam_g, environment's don't span ec2 regions [22:02] we'll need to work out some sort of capability for that in the future [22:03] SpamapS, service config is almost there, but not yet fully in trunk, last bits are in review [22:05] * SpamapS will continue meditating in the corner until said review is complete [22:06] SpamapS, i'm going to be looking at the lxc stuff and your branch next week, had a long chat with niemeyer about it an hr ago. [22:08] hazmat: I'm at your disposal if you have questions [22:11] SpamapS, i definitely will, thanks [22:28] ok, i just proposed a branch that pulls all the pieces together for working with the firewall [22:29] in terms of doing this on ec2, that is [22:35] jimbaker: do you have a rough estimate of when ensemble will default to all ports closed behavior? [22:35] m_3, the branch i just proposed implements exactly that [22:36] m_3, it might be worthwhile checking it out so you can see the impact - but i think it's going to be super minimal [22:36] ok, I'll keep and eye out [22:37] m_3, re "when", only when it gets through the review process of course ;) - but the only big dependency has been approved by niemeyer and is awaiting a review by hazmat [22:37] (that would be expose-provision-machines) [22:39] thanks! [22:43] * hazmat digs back into reviews