=== serge_ is now known as serge_afk [08:01] Morning everyone [08:06] hi kim0 [08:08] TeTeT: Hey o/ Morning [09:30] hi all [09:32] hey :) [09:35] hi kim0 TeTeT [09:37] hi koolhead11 [14:25] fwereade, how's the spring location? [14:39] Hey there [14:40] Part of the team is sprinting in Ensemble this week [14:41] So apologies if conversations feel a bit confusing, in advance [14:41] Feel free to join in, though :) [14:42] :) [15:04] RoAkSoAx, so... the preseed file that we were starting with (which i believe is incomplete at this point) is orchestra/./provisioning-server/etc/orchestra/ubuntu-orchestra-client.seed [15:06] smoser: ok. That's not in the archives yet I presume [15:07] so we'll have to build that from source [15:09] smoser: would it be possible for you to do a small howto of the things you were doing so we can work that from here? [15:09] well, its not exactly what we want anyway [15:09] right. thats what i'll work on. [15:09] getting you a running system and doc on how its working [15:10] smoser: cool, I'm setting up my own cobbler /orchestra server here [15:10] RoAkSoAx, http://paste.ubuntu.com/641956/ is an example of a late_command [15:11] <_mup_> ensemble/expose-provision-machines r281 committed by jim.baker@canonical.com [15:11] <_mup_> Stubs for directly testing open_close_ports_on_machine [15:12] that is generated with ./update-latecmd oneiric-amd64-preseed.cfg update-latecmd from lp:~smoser/+junk/cobbler-devenv/ [15:12] <_mup_> ensemble/expose-provision-machines r282 committed by jim.baker@canonical.com [15:12] <_mup_> Merged trunk [15:13] smoser: ok cool [15:21] niemeyer: do you have any slides from any ensemble presentations you might have done? I'm making a collection. [15:21] RoAkSoAx, http://paste.ubuntu.com/641966/ would be how i think preseed might look for us. [15:22] smoser: right, all those variables are provided by cobbler, orchestra, or ensemble? [15:23] i'm not sure about EXTRA_PACKAGES [15:24] but the idea is that we will start hte installer from ensemble and pass it ks_args to populate ENSEMBLE_LATE_COMMAND [15:24] smoser: ahh I see, so those are ks_args then [15:29] Hi guys: Regarding ensemble and standard AMIs ( Bug: 791501 ). I have two questions: Is it released in the ppa already and, how do I specify the AMI/size ? [15:37] <_mup_> Bug #808854 was filed: need a skeleton for a cobbler provider < https://launchpad.net/bugs/808854 > [15:37] negronjl, as i understand it, the ppa should be built against trunk; bug 791501 is marked released and the code is in there [15:37] <_mup_> Bug #791501: Ensemble should use standard amis and the ppa < https://launchpad.net/bugs/791501 > [15:38] jimbaker': thx. How do I specify AMI/size for ensemble to use ? [15:38] re the way to specify it: default-instance-type and default-ami [15:38] (i was on vacation last week, so i haven't had a chance to use it) [15:39] hazmat, sorry I missed you, I kinda forgot IRC existed this morning :) [15:39] indeed, that's it, ensemble is configured entirely through cloud-init with the new branch, it no longer uses pre-built images [15:39] jimbaker': thx [15:39] so any standard ubuntu ami (natty or latter due to cloud-init fixes) [15:39] er. 10.10 should work [15:40] hazmat: Is there a way to dynamically specify the AMI/size when deploying a formula with ensemble? [15:40] fwereade, no worries, just looking over the orchestra skeleton branch [15:40] hazmat: ie: ensemble deploy [15:40] negronjl, not atm [15:41] hazmat, negronjl - that would be something that the stack concept should be handle [15:41] hazmat: then I would use the default-instance-type and default-ami in the ~/.ensemble/environment.yaml file ? [15:41] able to handle [15:41] negronjl, correct [15:41] negronjl, yeah.. at the moment's that the only expose way of configuring them [15:41] hazmat it's smal, but I bet it's got some sort of problem :p [15:42] thx hazmat and jimbaker' [15:42] negronjl, tests ;-) [15:43] hazmat: I'll be testing today. I'll report back === jimbaker` is now known as jimbaker [15:44] hazmat: do we have to add a default-ami line to environment.yaml? [15:44] hazmat: I've been having problems with the new AMIs in r268 [15:51] m_3, hmm are you on oneric or natty? [15:52] m_3, it should default to the natty image [15:52] m_3, what sort of issues are you seeing? [15:55] hazmat: the ami (ami-06ad526f) is coming up, but any ensemble command stays in "...will wait" [15:56] hazmat: let me try on natty locally first... then I'll ping you [15:59] m_3, i just verified it, i do see the wait problem, the agents on the bootstrap machine are running though.. i'm investigating [16:01] o/ [16:09] Daviey: o/ [16:12] hazmat: same behavior for natty and lucid (don't know about oneiric yet)... both default to same ami(^^) [16:12] m_3, i'm testing a patch right now [16:12] hazmat: cool... I'll grab coffee... lemme know if I can help [16:19] <_mup_> ensemble/test-using-system-zookeeper r261 committed by gustavo@niemeyer.net [16:19] <_mup_> Minor styling changes from review. [16:19] fwereade: Can you please merge this and see if it works for you: lp:~niemeyer/ensemble/test-using-system-zookeeper [16:19] hey kim0 [16:19] fwereade: That's just SpamapS' branch with stylistic fixes [16:19] fwereade: If it does I'll commit to trunk [16:20] niemeyer, jimbaker could you guys look at this trunk trivial diff, it looks like i missed a shelved commit when merging the sans-ami branch.. https://pastebin.canonical.com/49564/ [16:21] hazmat, looking [16:21] m_3, that diff is the fix for the issue your seeing [16:21] niemeyer, works fine for me [16:22] hazmat: +1 [16:23] hazmat: I'll apply it locally and give it a spin [16:26] <_mup_> ensemble/trunk r269 committed by gustavo@niemeyer.net [16:26] <_mup_> Merged branch test-using-system-zookeeper from Clint [a=spamaps] [r=hazmat,niemeyer] [16:26] <_mup_> This enables tests to be run using the system installed ZooKeeper. [16:26] <_mup_> Note: I've applied irrelevant style changes to the original branch. [16:26] hazmat, looks good to me too [16:29] smoser: any luck? [16:29] hazmat: that worked... thanks! [16:29] jcastro: I don't have any useful slides myself.. the talk from last week was in pt_BR, and the ones before are out-of-date and in a crazy format [16:30] no. working on it right now. [16:30] ok, I'm organizing a bunch now [16:30] will focus on gettin gthat to you. [16:30] PSA: If anyone has ensemble slides send them my way [16:30] jcastro: bcsaller, Nick Barcet, and Davie Duffey had good ones, though [16:30] m_3, yeah.. i missed that one, it should have been part of the sans-ami merge. glad its working for you. [16:30] I've got Nick's already, I'll check with the other 2, thanks! [16:31] <_mup_> ensemble/trunk r270 committed by kapil.thangavelu@canonical.com [16:31] <_mup_> [trivial] Fix bootstrap initialization with ensemble ppa installation [r=jimbaker,niemeyer] [16:31] <_mup_> This was a missing commit from the sans-ami merge. It allows for any ensemble-admin cli within [16:31] <_mup_> the path to be used instead of hardcoding it to /usr/local. [16:31] jcastro: David's was not specific to Ensemble, though [16:31] jcastro: Not sure if it'd be useful [16:31] jcastro: It had an awesome video cut out from a screencast [16:31] smoser: thanks! [16:31] jcastro: Which I found handy to reuse [16:31] niemeyer: even raw material will be useful [16:37] does anyone know if that hadoop formula can be used today, as is [16:37] thinking about coming up with some cool demo for it [16:38] kim0: let's coordinate... I'm working on similar demos [16:40] m_3: sure thing [16:41] well basically the simple plan for me was for every working formula, I'd blog an article about it and perhaps a screencast. And I'll try to push to the interested communities [16:41] m_3: ofc there's enough cool demos for everyone :) if you think you'll be doing lots of those, then perhaps let's coordinate with bugs? [16:41] Folks, we'll attempt to run through the normal procedure here in the sprint [16:42] kim0: we should certainly cover http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/ [16:42] m_3: btw, where do you blog, I'd like to add you to cloud.ubuntu.com [16:42] including branches, merge proposals, reviews, etc [16:43] kim0: but the single/multi in one tutorial... it'd be great if we could get him to repoint to an ensemble-based demo as "the" getting-started [16:43] as one exception, we'd like to run with one review for the bits which are specific to Orchestra support [16:43] (rather than changing existing logic) [16:43] m_3: I hope the ensemble formula is gonna make it way much simpler than this :D [16:43] So that we can run a tighter loop here [16:43] E.g. for stuff like this: https://code.launchpad.net/~fwereade/ensemble/cobbler-integration/+merge/67568 [16:44] m_3: probably once we have a dead simple article demo'ing that, we can point him, he'll surely add that [16:44] hazmat, bcsaller, jimbaker: Is that alright? [16:44] kim0: blog.markmims.com, but it's still under construction atm [16:44] niemeyer, +1 [16:44] m_3: once you have a cloud tag, give me the rss I should add [16:45] kim0: dude, it'd totally be simpler for that getting started guide [16:45] hazmat, bcsaller, jimbaker: Even then, you should feel free to come over and review the logic, before or after merging [16:45] m_3: so the formula is ready? do we have like a readme of what needs to be done to get to crunching map/red jobs ? [16:45] We'll be happy to address any points [16:45] kim0: I want some more interesting ones too though [16:46] m_3: is it a good idea to create a spreadsheet now to track cool demos we can think of ? [16:47] I was thinking more like, as each formula is done, I'd figure out a cool demo for it [16:47] but if you have ideas, let's write'em down [16:48] kim0: nope... those forms not ready yet... I'll devote some time today to test juan's formulas out and add a readme [16:48] kim0: spreadsheet yes, definitely [16:48] m_3: that would be awesome :) [16:48] m_3: cool then .. creating it [16:48] fwereade: Re. XML-RPC on Twisted: http://twistedmatrix.com/documents/current/web/howto/xmlrpc.html === koolhead11 is now known as koolhead11|afk [16:50] niemeyer, that sounds good [16:51] Cool, thanks [16:51] Is twisted really needed for this? [16:51] Here's a spreadsheet to track cool demo ideas for formulas that are being written http://j.mp/peTAKG [16:51] m_3: jcastro ^ [17:07] Daviey: It's not needed for anything in theory, and I hope we manage to simplify things quite a bit in the near future [17:07] Daviey: But we do have to use Twisted right now for implementing that [17:07] Daviey: Not using Twisted for XML-RPC blocks every other activity in the system [17:12] niemeyer: it's like 4 xmlrpc calls? [17:12] blocking for <0.05s is a big deal? [17:16] Daviey: It's a remote call, and it stops every other activity in the system.. if it takes longer for whatever reason, nothing else works. [17:16] Daviey: This is the price for using Twisted.. I very much want to get rid of it, but we can't be in the middle [17:23] We're stepping out for lunch [17:25] have fun [17:50] <_mup_> ensemble/security-principal-def r269 committed by kapil.thangavelu@canonical.com [17:50] <_mup_> security principals for ensemble [18:16] <_mup_> ensemble/security-principal-def r270 committed by kapil.thangavelu@canonical.com [18:16] <_mup_> principal identity token database. [18:19] <_mup_> Bug #808935 was filed: ensemble needs principals and way to store/retrieve principal identity tokens < https://launchpad.net/bugs/808935 > [18:58] guys: running ensemble r270 from ppa. I have modified my environment.yaml by adding default-instance-type: m1.large and default-ami: ami-4eab5127 (oneiric). I then run ensemble bootstrap and I get: http://pastebin.ubuntu.com/642099/ [18:58] any help would be appreciated. [19:00] a simple search for ami-06ad526f finds it at: /usr/share/pyshared/ensemble/tests/test_utils.py in case that makes a difference. [19:01] negronjl: hazmat would be the right person to help you [19:01] niemeyer: thx. [19:01] hazmat: ping [19:01] negronjl, looking [19:01] hazmat: Danke [19:01] hazmat: thx [19:03] negronjl, that paste is to the environments.yaml? the instance launched is not the ami specified? [19:03] I'm not following. [19:03] negronjl, i'm trying to understand what the problem is [19:03] <_mup_> ensemble/expose-provision-machines r283 committed by jim.baker@canonical.com [19:03] <_mup_> Explicit direct testing of corner cases in open_close_ports_on_machine [19:03] hazmat: Here is what I did. [19:04] hazmat: I am running natty btw. [19:04] hazmat: I modified my environment.yaml file by adding the default-instance-type and default-ami values [19:04] hazmat: after I saved the file, I ran "ensemble bootstrap" [19:04] hazmat: ...and got the error I pasted. [19:05] hazmat: that's all I have done so far. [19:05] negronjl, ah.. i see i missed that at the bottom of the pastebin, i thought it was just the environment.yaml content [19:06] hazmat: no worries. I appreciate the help :) [19:06] fwereade, RoAkSoAx: For reference, this is the cobbler/remote.py file which contains the server-side XML-RPC API: http://j.mp/cobbler-remote [19:10] negronjl, the setting is 'default-image-id' [19:11] hazmat: ahh.. that may be it....let me change/try again. [19:11] hazmat: heh....look at that...it seems to be working so far. [19:11] hazmat: thx for the help [19:11] negronjl, cool [19:11] np [19:16] negronjl, it looks like we led you astray on irc re the param name, but fwiw it is documented correctly here https://ensemble.ubuntu.com/docs/provider-configuration-ec2.html [19:16] hazmat: cool.. no worries. I'm just glad it's working now. Thanks for the help [19:21] <_mup_> Bug #808969 was filed: determine the instance type at deploy < https://launchpad.net/bugs/808969 > [19:34] <_mup_> ensemble/expose-provision-machines r284 committed by jim.baker@canonical.com [19:34] <_mup_> Comments/docstrings [19:41] <_mup_> ensemble/expose-provision-machines r285 committed by jim.baker@canonical.com [19:41] <_mup_> Test some corner cases [19:59] <_mup_> ensemble/expose-provision-machines r286 committed by jim.baker@canonical.com [19:59] <_mup_> Cleanup [19:59] <_mup_> ensemble/expose-provision-machines r287 committed by jim.baker@canonical.com [19:59] <_mup_> Merged trunk [20:02] hazmat: As a heads up, we're attempting some refactoring work on the EC2 provider [20:02] niemeyer, great [20:03] niemeyer, based on what clint already did? [20:03] <_mup_> ensemble/expose-provision-machines r288 committed by jim.baker@canonical.com [20:03] <_mup_> PEP8 [20:03] hazmat: The first few branches may not be entirely obvious, because it's just untangling things [20:03] SpamapS, not sure if those branches are pushed re refactoring [20:03] niemeyer, sounds good [20:04] hazmat: We hope to move some bits to a common place til the end of the day [20:04] hazmat: No, it's not based on Clint's work.. we're hacking the EC2 provider itself to extract logic out of it [20:04] hazmat: Hopefully the bootstrap logic inside it will get very thin [20:05] niemeyer, sounds good [20:12] hazmat: SpamapS work will be most useful to us once get past that initial phase, because it has very good hints about how to structure the work with relation to the cobbler interaction [20:12] <_mup_> ensemble/expose-provision-machines r289 committed by jim.baker@canonical.com [20:12] <_mup_> Properly guard against topology state changes [20:15] niemeyer, sounds good about ec2 refactoring - will be useful for the ec2 firewall mgmt piece i have queued up [20:15] jimbaker: Likely [20:18] <_mup_> ensemble/expose-provision-machines r290 committed by jim.baker@canonical.com [20:18] <_mup_> Removed debugging statements [20:30] <_mup_> Bug #809005 was filed: inconvenient EC2Operation inheritance < https://launchpad.net/bugs/809005 > [20:38] <_mup_> Bug #809007 was filed: Modify provisioning agent so that port changes are made against a machine < https://launchpad.net/bugs/809007 > [20:49] hazmat: ping [20:53] <_mup_> ensemble/expose-status r265 committed by jim.baker@canonical.com [20:53] <_mup_> Merged trunk & resolved conflict [21:04] <_mup_> ensemble/expose-status r266 committed by jim.baker@canonical.com [21:04] <_mup_> Fix open-ports output in ensemble status to report [] if service is exposed but the service unit has no open ports [21:12] niemeyer, pong [21:12] hazmat: Hye [21:13] hazmat: Curious about a test problem here [21:13] niemeyer, shoot [21:13] hazmat: I suspect it may be caused by a left-over/unfinished branch, but not sure [21:13] niemeyer, what's the failure? [21:13] hazmat: test_watch_user_callback_invocation_delays_node_watch [21:13] hazmat: ensemble/state/tests/test_relation.py [21:13] hmm [21:13] hazmat: See the assertions at the end of the test? [21:14] niemeyer, seems to work fine on trunk [21:14] * hazmat looks over the test [21:14] hazmat: That "state: connected" string doesn't seem to be anywhere [21:14] hazmat: I suspect you may have a local uncommitted change in your txzookeeper branch [21:14] niemeyer, update your txzk [21:14] hazmat: Already did [21:14] hmm [21:14] hazmat: let me confirm I don't have paths mixed up [21:15] hazmat: fwereade sees the same issue, FWIW [21:15] niemeyer, hmm my python path is pointing to a txzk branch.. [21:15] hazmat: Uh oh [21:15] :) [21:15] yeah.. i was trying to make sure ensemble trunk ran with the branch before i merged.. investigating [21:15] <_mup_> ensemble/trunk r272 committed by jim.baker@canonical.com [21:15] <_mup_> merged expose-status [r=niemeyer,hazmat][f=767414] [21:15] <_mup_> Modifies "ensemble status" to report whether a service is exposed, and [21:15] <_mup_> if so, the open ports for each of its service units, if any. [21:16] hazmat: Well, that's exactly what is happening! ensemble trunk runs with the branch before you merged. 8-) === daker_ is now known as daker [21:17] niemeyer, what revno do you have for txzk [21:17] niemeyer, it works fine for me with rev 42 of trunk [21:18] on txzk [21:18] hmmm [21:19] hazmat: Well, something funky just happened [21:20] hazmat: I checked it moments ago, on revision 42, and ClientEvent wasn't changed [21:20] hazmat, niemeyer - i saw this problem this morning, it went away with the update to txzkookeeper trunk [21:20] hazmat: I've updated back to 40 to test, and now updated to 42, and it is there [21:20] hazmat: Super weird [21:20] niemeyer, trial doesn't clear out stale pyc files, not generally an issue.. [21:20] hazmat: Let me check with fwereade's branch before he touches it [21:21] k [21:22] hazmat: No game.. he's using the package, so the issue is expected [21:22] hazmat: Just so you don't think I'm crazy, this is from my history: [21:22] ..eeper-bzr/trunk]% bzr update [21:22] Tree is up to date at revision 42 of branch bzr+ssh://bazaar.launchpad.net/~ensemble/txzookeeper/trunk [21:23] hazmat: I could reproduce the bug after that.. [21:23] niemeyer, i believe you.. although they have some good rum in miami i hear ;-) [21:23] perhaps the universal answer confused the issue [21:23] * hazmat looks for a towel [21:25] i just cleaned up pyc files in trunk txzk and ensemble, our test suite still works fine for me [21:25] hazmat: :-) [21:25] hazmat: So, we need to fix that package.. will look at that [21:26] jimbaker: Sweet, thanks for confirming [21:29] <_mup_> Bug #809030 was filed: [ReOpen] Ensemble should use standard amis and the ppa < https://launchpad.net/bugs/809030 > [21:29] hazmat: Did you enter the date by hand? [21:29] niemeyer, ? what date [21:30] hazmat: The changelog one in txzookeeper [21:30] niemeyer, i did [21:30] <_mup_> ensemble/trunk r43 committed by gustavo@niemeyer.net [21:30] <_mup_> Add seconds to the changelog date. (/me looks at hazmat) [21:30] niemeyer, it looked like the other lines, but it still doesn't like it [21:30] hazmat: "date -R" is what you want [21:30] hazmat: hi, could we get https://bugs.launchpad.net/ensemble/+bug/791501 reopened? the fix doesn't appear to be working for us ... [21:30] niemeyer, when you have a chance, my branch debug-log-relation-settings-changes has been waiting for a review for 2.5 weeks (i made all changes that hazmat requested) [21:30] <_mup_> Bug #791501: Ensemble should use standard amis and the ppa < https://launchpad.net/bugs/791501 > [21:30] hazmat: it seems that LP has recently disabled the ability to reopen bugs in projects :-) [21:31] hazmat: in the mean time, we've filed https://bugs.launchpad.net/ensemble/+bug/809030 as a placeholder [21:31] <_mup_> Bug #809030: [ReOpen] Ensemble should use standard amis and the ppa < https://launchpad.net/bugs/809030 > [21:31] jimbaker: Ok.. it will wait for at least another one, sorry about that [21:31] kirkland`, k, i'll try it out now [21:31] niemeyer, np. this should be a pretty useful branch for formula developers, so just wanted to get merged in :) [21:32] jimbaker: Yeah, I know.. I still want to check it out for sure, but it's also a low priority feature.. this week we really have to get orchestra cranking [21:32] niemeyer, ok [21:36] niemeyer, fwereade re the ec2op refactoring, i feel like i'm missing the point about what value this brings [21:36] could one of you elaborate on where you going [21:36] hazmat: Hehe :-) [21:36] hazmat: I anticipated that [21:36] niemeyer, all i see is [21:36] 56 - d = self.ec2.describe_instances(instance_id) [21:36] 57 + d = self._provider.ec2.describe_instances(instance_id) [21:37] which is a little unclear as far refactoring towards orchestra [21:37] hazmat: We need higher bandwidth [21:37] or others [21:37] hazmat: Yeah, I know [21:37] hazmat: Which is why I've briefed you [21:38] niemeyer, skype | google+ ? [21:38] hazmat: Google+ [21:38] Anyone else want to join? === kirkland` is now known as kirkland [21:41] hazmat: try "what" out now? changing the state of the bug? [21:41] hazmat: or launching ami's? [21:42] kirkland, launching amis [21:42] kirkland, going to do this meeting, and i'll check back in on it [21:42] negronjl: what version of ensemble do you test the ami launching? [21:43] hazmat: let me check with negronjl to see which version he tested [21:43] kirkland, hazmat: latest ensemble ppa ( r270 ) [21:43] hazmat: which rXXX do you want us to test, specifically? [21:43] kirkland, hazmat: 0.5+270-0ensemble1~natty1 [21:44] hazmat: also, could you check that we're configuring it correctly? it's possible that we have a bad configuration [21:44] kirkland, un momento, doing a chat with niemeyer [21:45] hazmat: k [21:47] kirkland, hazmat: the config that I am using: http://pastebin.ubuntu.com/642192/ [22:07] hazmat: Oh, btw, before you leave, would you be able to have a quick look at the branch? [22:07] hazmat: Should be pretty much a trivial now that you have the context [22:09] hazmat: Btw, txzookeeper package updated.. [22:09] hazmat: It was the missing seconds indeed [22:14] niemeyer, sure [22:14] re quick look at the branch [22:19] niemeyer, done [22:23] kirkland, negronjl it's working for me.. using the same config.. all m1.large oneiric instances.. http://ec2-174-129-90-179.compute-1.amazonaws.com/wp-admin/install.php [22:23] well same param names and same ami and machine size [22:24] kirkland, negronjl what problems are you seeing? your on the latest rev, so that should all be good [22:28] niemeyer, thanks re seconds [22:28] * hazmat wanders off to dinner [22:30] hazmat: i am unable to deploy any formulas at all. [22:35] hazmat: no worries [22:36] negronjl: What problem are you seeing there? [22:36] niemeyer: I don't see anything at all.. nothing happens. [22:36] ensemble deploy [22:36] then ensemble status [22:36] and status is always null [22:36] hey guys, stumbling in here late but it i can confirm 0.5+270-0ensemble1~natty1 doesn't deploy [22:37] it bootstraps the first node fine [22:37] ensemble ssh hangs as well [22:37] but deploying nodes does not use the new userdata [22:37] adam_g: new userdata? [22:38] one sec [22:39] niemeyer: the userdata that cloud-init uses to install and bootstrap ensemble on the stock ubuntu AMIs [22:40] from ensemble boostrap: http://paste.ubuntu.com/642216/ [22:40] from ensemble deploy: http://paste.ubuntu.com/642217/ [22:40] adam_g: Hmm [22:41] it shouldn't be doing the bzr stuff there [22:41] i believe hazmat committed something last week to produce userdata from the first pastebinit, to allow ensemble to bootstrap stock AMIs [22:41] but 'deploy' is still producing it in the way pre-commit [22:42] Ok, understood [22:42] adam_g, are you using an existing environment perhaps? [22:42] hmm [22:42] hazmat: MACHINE_ID=1.. probably not [22:43] hazmat: perhaps i am? http://paste.ubuntu.com/642225/ [22:43] the "bzr up" line isn't in trunk (rev 270) anymore is why i ask [22:44] hazmat: im using the enviornmnet ensemble originally generated, with a new default-image-id [22:44] ill wipe my environment and try again.. strange, though, 'ensemble bootstrap' brought up the first node as expected [22:44] adam_g, but you've shutdown and bootstrap since then? [22:44] hazmat: yup. bootstrap'd for the first time this week after upgrade [22:45] bootstrap on existing environment is a no-op.. the environment has to be shutdown and then bootstrap again to get a fresh install [22:45] negronjl, perhaps the same applies for you? i was able to shutdown/bootstrap/deploy formulas/add-relations with trunk (r270) [22:46] using the ppas for the ensemble install on the environment nodes [22:46] hazmat: I did: [22:46] hazmat: ensemble shutdown [22:46] hazmat: updated environment.yaml [22:46] hazmat: ensemble bootstrap [22:46] hazmat: ensemble deploy [22:46] hazmat: waited a while [22:47] hazmat: ensemble ssh < machine > [22:47] hazmant: nothing [22:48] negronjl, if you ec2-describe-instances can you login into the machine directly? [22:48] i'm curious if the machine and unit agent are running [22:48] negronjl: can you pastebinit /var/lib/cloud/instance/scripts/runcmd? [22:48] hazmat: bootstrapping now [22:49] hazmat: i see the same as juan. i can ssh directly, there is nothing installed or running [22:49] hazmat: waiting for bootstrap to complete [22:49] * hazmat checks the ppa [22:50] hazmat: do you want me to ssh into the bootstrap instance or try to deploy and then check that ? [22:50] negronjl, the service unit instance (post deploy) [22:50] hazmat: ok. waiting [22:50] hazmat: Please note txzookeeper was _just_ updated.. if they were using the PPA, it was out of sync with Ensemble trunk [22:52] looks like the current ppa rev is 272 (updating) [22:53] hazmat: upgrading ensemble to latest ppa [22:53] hazmat: the bootstrap process on the previous attempt timed out. [22:53] i'm running through the same steps with the latest [22:53] negronjl, the bootstrap is waiting on the remote end, so i'm not clear what its timing out on [22:54] negronjl, you mean the command just hangs? [22:54] that would imply a problem connecting to ec2 [22:54] hazmat: ensemble returned a time out error ( took longer than 30 seconds ) [22:54] hazmat: bootstrapping now [22:55] hazmat: bootstrapped. should I deploy something now ? [22:55] negronjl, yes [22:55] hazmat: trying to deploy tomcat6. waiting [22:56] negronjl, i've been using the example formulas as a known working base set [22:56] hazmat: that's ok. If this one doesn't provide definitive results, I can try one of the included ones. [22:57] "2011-07-11 18:56:36,260 ERROR SSH forwarding error: @@@@@@@" [22:57] strange [22:58] hazmat: I now have an instance. What should I do now ? [22:58] negronjl, does it show up in ensemble status? [22:58] hazmat: it does [22:58] hazmat: after the upgrade. it seems to be working now. [22:59] hazmat: let me do some testing .... hold on [22:59] negronjl, k [22:59] negronjl, i'm going to grab dinner but i'll be back shortly [22:59] hazmat: ok [23:00] brb [23:08] back [23:10] negronjl, how'd it go? [23:10] hazmat: so far, it seems good. but, I am having issues with aws/ec2. two of my instances are still pending. [23:13] after upgrade, cloud-init stuff looks consistent across boostrap + deploy'd nodes. [23:14] i too ran into a timed out instance, but it may be ec2 related. it also took an unusually long time to provision the instance [23:14] ditto that. ec2 is having issues [23:17] hazmat: it's all working well for me now. [23:17] negronjl, great [23:18] hazmat: thx for the patience and help :) [23:18] negronjl, np === franciscosouza_ is now known as franciscosouza [23:40] adam_g, afaics your bug and patch are against an older version of the codebase [23:40] adam_g, is your local ensemble from the ppa? [23:41] ensemble doesn't do bzr up anymore.. it either does the bzr checkout, or defaults to installing from ppa on the environment machines. it looks like your local installation might be out of date [23:42] hazmat: yeah, that was last week before you committed the sans-AMI stuff [23:43] adam_g, ah.. gotcha [23:43] adam_g, are you still having issues running with oneric images? [23:45] hazmat: nope, that last upgrade ~30mins ago seems to fix the issues i was seeing [23:45] adam_g, great [23:47] <_mup_> Bug #809070 was filed: ensemble should allow execution of formula specific/defined hooks < https://launchpad.net/bugs/809070 >