[02:48] <_mup_> ensemble/fix-functional-testing r335 committed by jim.baker@canonical.com [02:48] <_mup_> More robust ftests [02:49] <_mup_> ensemble/fix-functional-testing r336 committed by jim.baker@canonical.com [02:49] <_mup_> Merged trunk [10:28] actually don't worry :) [11:56] fwereade: Hey! [11:56] niemeyer: heyhey! [11:56] niemeyer: how's life? [11:56] fwereade: Awaken [11:57] niemeyer: sorry, did I miss something important? === ehw is now known as ehw|asdf [11:58] fwereade: Hmmm.. why do you ask? [11:58] niemeyer: ah, I wasn't sure whether to read "Awaken" as an imperative [11:58] Erm.. [11:59] fwereade: I don't think I even know how to interpret that way ;-) [11:59] niemeyer: I'd see "awaken" as a verb and "awake" as an adjective === ehw|asdf is now known as ehw [12:00] niemeyer: now I think of it, "awoken" would be a ...er, a participle? I can't actually remember much grammar [12:01] fwereade: Ohhh.. got it [12:01] niemeyer: computer languages are easier ;) [12:01] fwereade: Yeah.. see, it was my grammar being bad :) [12:01] niemeyer: no worries, I take things far too literally ;) [12:37] niemeyer: did we agree it would be ok to delete bin/ensemble-make-image and debian/ec2-build? [12:39] fwereade: We did not.. it sounds fine to me, but we should check with hazmat first as he's created and used these the most [12:40] niemeyer: I'll assume they should stay for now then, all it costs me to keep them is a couple of global constants [12:40] fwereade, delete away [12:41] hazmat: ah, sweet :D [12:41] and g'morning to all [12:42] hazmat: and a good morning to you [12:42] i finally gave up on screen, and switched out to using tmux, its very nice [12:57] hazmat: Morning, and welcome to tmux [13:02] jimbaker, i noticed that the unit test in py 2.7 has some very nice builtin layer support at module and class level [13:06] niemeyer, one tmux question i wondered about, and just curious if you knew, is it possible to reload the conf file for an existing session? [13:11] found it.. source-file ~/.tmux.conf [13:11] awesomeness [13:16] hazmat: Hmm [13:16] hazmat: Good question [13:16] hazmat: Ah, you found it, cool [13:16] hazmat: I haven't used that yet [13:17] niemeyer, makes testing new settings with tmux rather easy also find some nice tools from the tmux home page.. this one in particular fits my usage pretty nicely.. https://github.com/aziz/tmuxinator [13:18] hazmat: True.. checking [13:18] also in the same vein but with more support for pane splitting https://github.com/remiprev/teamocil [13:19] hazmat: Oh, neat [13:19] hazmat: I create new tabs as I go [13:20] niemeyer, i normally do.. but crashes are frequent enough.. that having some auto-restore functionality helps me retain context, i'm trying to put in a cron job that uses ps-util to sniff all the process hierarchy for the windows and stores them in the yaml file, so upgrade instability isn't as bad.. [13:21] it sort of like that software you mentioned a while back.. that has this awesome resume after crash system to take you back to exactly what you where doing b4 the crash ;-) [13:25] hazmat: Wow, so freezing a running tmux? [13:26] hazmat: Yeah.. Cinelerra :-) [13:27] niemeyer, its not really freezing, its just recording the running programs to restart them.. i'm using an emacs per window, and the emacs is auto-saving its open files list, so on startup i'll get back my open buffers... if the program didn't support resumption, then the context is loss.. [13:30] hazmat: I see.. quite neat even then [14:28] hey, hazmat, can I delete those 2 scripts in trunk as a trivial? [14:29] fwereade, sounds good [14:29] hazmat: cheers [15:02] hazmat, thanks, i will take a look at that [15:03] jimbaker, yeah.. i'm not sure if its useful given how twisted is doing the test running and we need the reactor support [15:03] maybe [15:03] hazmat, generally it should just mix together [15:03] so twisted trial basically is managing the test runner, which is one component that can be customized in pyunit [15:04] * jimbaker now has a much deeper appreciation for how trial works [16:15] awesome.. I just managed to shutdown my jenkins environment, taking all the data with it... [16:15] So, this is a good chance to answer the "how are we protecting peoples' data?" question. :) [16:15] I did make an EBS snapshot of said data.. but there's basically no way to tell ensemble to re-gain that environment. [16:21] Project's in Launchpad have name, display name, title, summary and description (!) [16:21] Projects [16:22] SpamapS: Indeed, which is why I'm advocating for a while that we call this command destroy-environment rather than shutdown [16:24] <_mup_> Bug #838215 was filed: "shutdown" must be renamed to "destroy-environment" < https://launchpad.net/bugs/838215 > [16:25] sounds like a good plan [16:26] lengthy command, matches the internal api, no mistaking this is going to do something extreme [16:26] well i don't think the rename would have helped SpamapS [16:26] also parallels destroy-service [16:27] the rename sounds fine [16:27] addressing the underlying issue of volume management is probably a larger task [16:27] maybe ;-) [16:27] well it's an interesting question of what would destroy that ;) [16:28] maybe have a command that forces you to read what it asks. to destroy everything, please sum 10 + 32. like the gmail plugin to avoid inadvertent emails [16:37] the rename would not have helped me, no [16:37] but +1 for renaming it [16:38] jimbaker: haha.. true, I did put 'echo y | ensemble shutdown' in my test scripts [16:38] the difference is I was running my test script on my machine, instead of on the jenkins machine, so it shutdown my default environment [16:38] which I had selected as the jenkins env because I was tired of adding -e jenkins [16:39] Flat out.. [16:39] help me recover the data easily [16:39] Right now what I have to do is start a new raw instance, mount the snapshot, and rsync it back to a new bootstrapped/deployed env [16:39] SpamapS, yeah i figured as much. command histories, scripts, automation - all good, until they turn bad on oneself [16:40] one good thing is that shutdown now tells you which environment you will be deleting. a small detail for sure [16:42] Honestly, there's nothing good about this. :-P [16:42] Thought... [16:42] snapshot all volumes before shutdown. [16:43] Unless you configure the environment not to. [16:43] Luckily most of what I had done is captured in a bzr tree. :) [16:56] <_mup_> Bug #838238 was filed: No matter what version of ensemble you have on client, deployed instances get the one from the PPA < https://launchpad.net/bugs/838238 > === foods is now known as adam_g [17:10] SpamapS, hazmat: I think the rename would have helped only in the sense that there's no return out of a destroy-environment [17:11] Makes the outcome more obvious [17:11] niemeyer: I intended to destroy the environment I had just created. [17:11] niemeyer: the jenkins service running inside it was collateral damage. [17:12] SpamapS: The jenkins was part of the environment.. [17:12] Because they happened to be using the same s3 control bucket and name. [17:12] SpamapS: It's a bit like saying.. I wanted to format my disk, I just didn't want to lose that one file [17:13] It picked up the .ensemble/environments.yaml from my home dir automatically.. accidentally. [17:13] This is why I filed the bug request for a "freeze" or "lock" command. [17:13] let me control changes to an environment so automated scripts can't screw it up [17:14] SpamapS: We could change destroy-environment so it asks [17:14] It does ask [17:14] and I echo'd y in [17:14] SpamapS: Well.. :-) [17:14] because this was an automated script [17:14] SpamapS: Sorry about that then :-) [17:14] I'm not blaming ensemble, I'm suggesting that this is the first in a long line of "WTF!" questions that will arrive here when ensemble usage begins in earnest. [17:16] SpamapS: I understand, and while I share your pain, I think it was an operator issue on that one case [17:16] SpamapS: rm -rf / does bad things too [17:16] I agree 100% [17:16] lol.. this is going to be fun [17:17] I'm going to /part whenever this question comes up. :) [17:17] I burn easily, the flames will be intense. ;) [17:18] SpamapS: Do you have a recommended solution for the problem? [17:18] Its so easy to roll out on ensemble.. it shouldn't be so easy to destroy it. I'd actually be in favor of making shutdown/destroy environment a disabled command until you mark an environment as "ephemeral" or something like that. [17:18] But then that will happen in automation too.. hrm. [17:19] SpamapS: Exactly what I was writing [17:19] Ahh, maybe you have to --ephemeral at boot... [17:19] bootstrap rather.. [17:19] And then the env name/controlbucket/etc are all generated. [17:19] SpamapS: What happens if --ephemeral wasn't used? [17:19] No shutdown [17:19] no destroy env [17:19] pull it apart one piece at a time [17:20] SpamapS: Ugh.. [17:20] I'm shooting from the hip here.. haven't given it much thought.. [17:21] but it feels like we're playing just a little fast and loose with data when we could put a simple padlock on it. [17:21] Maybe the padlock is to just make it clear that the default environment is not to be used in automation. [17:22] SpamapS: I'm fine with adding yet another lock command, if you think that'd help [17:23] SpamapS: Hmm [17:23] SpamapS: There's another option.. disabling automation of destroy-environment.. [17:23] SpamapS: This sounds better, actually [17:23] if not isatty .. say no [17:23] SpamapS: Except if we want to test things in an automated way! :-) [17:24] SpamapS: --i-am-sure-about-that-do-it-right-now-damn-it [17:24] Yeah, mdadm had some options like that [17:25] I'm just envisioning netflix admins running their test automation script and accidentally shutting down.. everything. [17:25] It almost seems more sane to just get rid of it. Keep track of your services, destroy them one by one. Terminate the machines one by one.. [17:26] For my automated test script, I actually did have a shutdown equivilent for this very reason early on.. [17:26] SpamapS: We created this option because we need it, very foten [17:26] often [17:26] I kept track of all my machine ids and service names and destroyed/terminated them one by one. [17:27] SpamapS: Making it painful at all times to protect the few that will automate the "yes I am sure" measure as you've done is not a good solution IMO [17:28] niemeyer, so regarding the bridge it look like libvirt does a nat to it by default, with some forward rules, those aren't compatible though with a network per environment.. only one nat active at a time [17:29] hazmat: That sounds fine for now.. [17:30] hazmat: I wouldn't like to get in that situation just because we were not careful to namespace, but we don't have to waste lots of cycles on the problem [17:30] niemeyer, it also means that we're just going to be piggy backing on the libvirt default network. [17:30] i'm looking into the libvirt network options a bit more [17:31] i've got the network abstraction on libvirt done, so i'd like to keep the isolation if possible [17:33] hazmat: Yeah, just wrapping what's there sounds very neat indeed [17:34] niemeyer, well if we're just using the default, there's not much point to wrapping, as we don't need to change anything [17:34] outside of namespacing the containers correctly [17:34] but effectively the network setup is a no-op [17:35] hazmat: I meant just reusing what's there [17:35] hazmat: As you described [17:36] <_mup_> ensemble/local-network r341 committed by kapil.thangavelu@canonical.com [17:36] <_mup_> a network abstraction for starting/stopping/defining libvirt networks.. not going to use but i wanted to capture it. [17:46] sorry I disappeared.. my unity/wifi/oneiric melted down [17:47] niemeyer: We're packing a self destruct into ensemble. Its useful, for sure. However, I wonder if we should just leave it out for non-developers. :-P [17:49] SpamapS: For now I think we're good.. if you a) Have the proper AWS keys; b) Have the proper environments.yaml file; c) Type destroy-environment; d) Confirm with y... I think it's fine to destroy it. [17:49] SpamapS: I'll pay a beer to everyone who does that by mistake, so I owe you one [17:50] I think you're ok, because I basically packed a grenade w/ a pulled pin in my script. [17:51] I may actually have a better suggestion, which is to make it easier to "create" environments in an automated fashion... rather than having to type them into the yaml file. [17:52] That way people won't be tempted to just re-use the same env over and over. [17:52] SpamapS: yeah, stacks ftw [17:53] well yes stacks.. but even more isolation.. just being able to say --config-dir=/tmp/ensemble.asdfy431k would make it so I can be more careful in my automated scripts not to use a static environments.yaml ever [17:53] I mean ultimately the responsibility comes down on the automator to be careful. :) [17:55] But if I have to touch files in the home dir for automation.. thats a dangerous game. [18:02] SpamapS: We explicitly try to avoid that [18:03] SpamapS: The information in a local admin's laptop should be only the necessary to reach the environment [18:03] SpamapS: The authoritative version of the configuration should live in the environment itself, with HA etc [18:04] SpamapS: So ultimately the issue is that there's too little in the local env yaml [18:04] Ok, new problem then. I want to put safeties in my automated scripts.. something like.. if this env already exists, STOP. [18:04] hm, does status give an error if its not bootstrapped? [18:04] * SpamapS checks [18:05] hm, just a generic "1" [18:05] could be lots of reasons for that.. no way to be sure it means not bootstrapped other than grepping.. :P [18:06] SpamapS: Agreed.. we should do better on rcs [18:07] SpamapS: and also on helper commands [18:07] SpamapS: is-bootrapped or whatever [18:08] Well for now, I'll grep for that exact message.. hopefully that is enough to prevent another damnit moment. [18:08] <_mup_> ensemble/rename-shutdown-command r337 committed by jim.baker@canonical.com [18:08] <_mup_> Code changes [18:11] SpamapS: Hmm.. [18:11] SpamapS: But bootstrap shouldn't really cause any issues [18:11] SpamapS: If you do it again on a bootstrapped env, that is [18:11] SpamapS: This would be a major problem [18:11] SpamapS: Rather than grepping for logs that may change, I suggest just trying it again [18:12] Right, my test script is basically bootstrap, deploy deploy relate relate , shutdown ... [18:12] I want to make *SURE* that I am the one doing the bootstrapping, that it wasn't already done [18:13] I'd prefer to be able to use a generated yaml file for this.. but I can't set the config dir, and I don't want to append yaml to the environments.yaml .. [18:13] SpamapS: Just if [ $? -ne 0 ]; then FAIL fi; after bootstrap? [18:15] I run with set -e [18:15] existing env is not an error in bootstrap [18:23] SpamapS: Oops.. that's certainly a bug [18:39] SpamapS, define $HOME in your test script if you want better isolation [18:39] SpamapS, yeah.. the return code from the ensemble cli are suspect [18:39] i filed a bug a while back regrading [18:40] actually jim did.. bug 697093 [18:40] <_mup_> Bug #697093: Ensemble command should return nonzero status code for errors < https://launchpad.net/bugs/697093 > [18:50] <_mup_> ensemble/rename-shutdown-command r338 committed by jim.baker@canonical.com [18:50] <_mup_> Doc changes [18:51] <_mup_> ensemble/rename-shutdown-command r339 committed by jim.baker@canonical.com [18:51] <_mup_> Add test for destroying a specified default env [19:24] i think i will keep the network abstraction, its still pretty useful [19:24] even against libvirt default for exposing attributes and starting if its not started [19:42] <_mup_> Bug #838330 was filed: Bootstrap command should error on existing < https://launchpad.net/bugs/838330 > [19:55] bcsaller1, the other important parallel piece of work is doing the lxc integration with a serviceunitdeployment subclass [19:55] bcsaller1, wiring in the deployment class selection based on machine provider type later [19:56] hazmat: I'll look at it but I might not be able to start that part today, we'll see [19:56] bcsaller1, fair enough.. i'm pushing through the last of the lxc local provider work, should have all the branches in review for tomorrow, i can start on it tomorrow [19:57] its the next step [19:58] bcsaller1, lost my irc connection for a moment [19:59] hazmat: didn't miss anything [20:10] bcsaller1, one think re lxc-lib, i'd like to create a container without running the customize script.. it doesn't seem like that's possible atm [20:10] s/think/thing [20:11] hmm [20:11] i'm trying to prime the deb-cache doing bootstrap so ops like add unit are fast.. but their async anyways.. so maybe that's just a bad assumption [20:12] yeah.. nevermind re lxc lib.. it makes more sense to just let the agent deal with the lag on the container creation [20:13] ok... for ensembel cloud-config, you should probably for your sanity put something like: [20:13] output: [20:14] smoser, output: ? [20:14] all: "| tee /var/log/cloud-init-output.log" [20:14] smoser, i thought cloud-init had log file output builtin now ? [20:15] yes, it does log to /var/log/cloud-init.log [20:15] but this will capture output of it [20:15] and tee it [20:15] cool [20:16] output: [20:16] all: [ "| tee /var/log/cloud-init-output.log", "&1" ] [20:16] that would help me debug some [20:47] i think maybe http://paste.ubuntu.com/679208/ [20:48] hmm... better with tee -a [21:06] <_mup_> ensemble/local-ubuntu-provider r341 committed by kapil.thangavelu@canonical.com [21:06] <_mup_> local provider skeleton [21:16] ok. my first ensemble merge proposal! [21:16] https://code.launchpad.net/~smoser/ensemble/cloud-init-output-log/+merge/73596 [21:16] smoser: Woohay! [21:40] <_mup_> ensemble/local-network r342 committed by kapil.thangavelu@canonical.com [21:40] <_mup_> expose network attributes from libvirt to local provider [21:48] <_mup_> ensemble/local-provider r343 committed by kapil.thangavelu@canonical.com [21:48] <_mup_> local provider that bootstraps and shutsdown [21:51] smoser: hey, I was just reading cloud-init's code and I noticed that you're not passing -y to add-apt-repository [21:53] smoser: I believe this may be causing the issue with getting stuck on console... http://paste.ubuntu.com/679240/ [22:23] <_mup_> ensemble/managed-agent r343 committed by kapil.thangavelu@canonical.com [22:23] <_mup_> managed machine agent [22:51] hrm.. bug 819329 is actually pretty tricky [22:51] <_mup_> Bug #819329: Tests depend on AWS_ACCESS_KEY_ID being set < https://launchpad.net/bugs/819329 > [22:51] txaws is raising an error during a constructor because it can't find this environment variable... [22:57] the conundrum is, there is no sane default.. [22:58] So do we set the env var in the tests? Do we shove it into the config during the tests, potentially bypassing code? Hrm. [22:58] SpamapS, its a unit test, you can shove any value there [22:58] SpamapS, it should never actually use it [22:58] SpamapS, it can also be fixed by ensuring its in the environment config of whatever test is using it [23:00] so the real issue is that txaws needs to be mocked? [23:01] since we're not "testing txaws" [23:02] hazmat: if txaws isn't going to be mocked, then we need to test both having it and not having it in the env, so we know when that behavior changes.. [23:03] SpamapS, we're using txaws but we don't ever interact externally during the unit tests [23:03] any external interaction is typically mocked [23:12] hazmat: so the simple workaround is just to set them to dummy values before running the test suite. :-P [23:13] SpamapS: there is a change_environment function on all the test classes that will set env for the duration of the test === bcsaller1 is now known as bcsaller [23:14] mmk. So to work around the problem, we can certainly just dump AWS_ACCESS_KEY_ID in .. but shouldn't we also *verify* that if its not there, an error is raised? [23:16] if the provider is ec2 for a given environment it should assert the values it depends on at runtime, yes [23:16] Thats the frustrating/confusing part.. I don't see where ec2 was even asserted. :P [23:23] SpamapS: that test is using write_sample_config which includes a default value of ec2, when it looks for the machine provider it triggers the issue you see. I agree, this is not clear [23:42] hrm.. tests don't seem to run cleanly for me anyway.. :-P [23:46] weird.. just running './test' should work pretty well, right? [23:46] I'm getting *piles* of errors [23:50] SpamapS, assuming you've got txzk and zk yes [23:50] if I try to run say, ensemble.environment tests.. I get these fails: [23:50] http://paste.ubuntu.com/679281/ [23:51] Looks like it lost the HOME override [23:51] ok n/m on that.. I misunderstood change_environment