[02:48] <_mup_> ensemble/fix-functional-testing r335 committed by jim.baker@canonical.com
[02:48] <_mup_> More robust ftests
[02:49] <_mup_> ensemble/fix-functional-testing r336 committed by jim.baker@canonical.com
[02:49] <_mup_> Merged trunk
[10:28] <fwereade> actually don't worry :)
[11:56] <niemeyer> fwereade: Hey!
[11:56] <fwereade> niemeyer: heyhey!
[11:56] <fwereade> niemeyer: how's life?
[11:56] <niemeyer> fwereade: Awaken
[11:57] <fwereade> niemeyer: sorry, did I miss something important?
[11:58] <niemeyer> fwereade: Hmmm.. why do you ask?
[11:58] <fwereade> niemeyer: ah, I wasn't sure whether to read "Awaken" as an imperative
[11:58] <niemeyer> Erm..
[11:59] <niemeyer> fwereade: I don't think I even know how to interpret that way ;-)
[11:59] <fwereade> niemeyer: I'd see "awaken" as a verb and "awake" as an adjective
[12:00] <fwereade> niemeyer: now I think of it, "awoken" would be a ...er, a participle? I can't actually remember much grammar
[12:01] <niemeyer> fwereade: Ohhh.. got it
[12:01] <fwereade> niemeyer: computer languages are easier ;)
[12:01] <niemeyer> fwereade: Yeah.. see, it was my grammar being bad :)
[12:01] <fwereade> niemeyer: no worries, I take things far too literally ;)
[12:37] <fwereade> niemeyer: did we agree it would be ok to delete bin/ensemble-make-image and debian/ec2-build?
[12:39] <niemeyer> fwereade: We did not.. it sounds fine to me, but we should check with hazmat first as he's created and used these the most
[12:40] <fwereade> niemeyer: I'll assume they should stay for now then, all it costs me to keep them is a couple of global constants
[12:40] <hazmat> fwereade, delete away
[12:41] <fwereade> hazmat: ah, sweet :D
[12:41] <hazmat> and g'morning to all
[12:42] <fwereade> hazmat: and a good morning to you
[12:42] <hazmat> i finally gave up on screen, and switched out to using tmux, its very nice
[12:57] <niemeyer> hazmat: Morning, and welcome to tmux
[13:02] <hazmat> jimbaker, i noticed that the unit test in py 2.7 has some very nice builtin layer support at module and class level
[13:06] <hazmat> niemeyer, one tmux question i wondered about, and just curious if you knew, is it possible to reload the conf file for an existing session?
[13:11] <hazmat> found it.. source-file ~/.tmux.conf
[13:11] <hazmat> awesomeness
[13:16] <niemeyer> hazmat: Hmm
[13:16] <niemeyer> hazmat: Good question
[13:16] <niemeyer> hazmat: Ah, you found it, cool
[13:16] <niemeyer> hazmat: I haven't used that yet
[13:17] <hazmat> niemeyer, makes testing new settings with tmux rather easy also find some nice tools from the tmux home page.. this one in particular fits my usage pretty nicely.. https://github.com/aziz/tmuxinator
[13:18] <niemeyer> hazmat: True.. checking
[13:18] <hazmat> also in the same vein but with more support for pane splitting https://github.com/remiprev/teamocil
[13:19] <niemeyer> hazmat: Oh, neat
[13:19] <niemeyer> hazmat: I create new tabs as I go
[13:20] <hazmat> niemeyer, i normally do.. but crashes are frequent enough.. that having some auto-restore functionality helps me retain context, i'm trying to put in a cron job that uses ps-util to sniff all the process hierarchy for the windows and stores them in the yaml file, so upgrade instability isn't as bad..
[13:21] <hazmat> it sort of like that software you mentioned a while back.. that has this awesome resume after crash system to take you back to exactly what you where doing b4 the crash  ;-)
[13:25] <niemeyer> hazmat: Wow, so freezing a running tmux?
[13:26] <niemeyer> hazmat: Yeah.. Cinelerra :-)
[13:27] <hazmat> niemeyer, its not really freezing, its just recording the running programs to restart them.. i'm using an emacs per window, and the emacs is auto-saving its open files list, so on startup i'll get back  my open buffers... if the program didn't support resumption, then the context is loss..
[13:30] <niemeyer> hazmat: I see.. quite neat even then
[14:28] <fwereade> hey, hazmat, can I delete those 2 scripts in trunk as a trivial?
[14:29] <hazmat> fwereade, sounds good
[14:29] <fwereade> hazmat: cheers
[15:02] <jimbaker> hazmat, thanks, i will take a look at that
[15:03] <hazmat> jimbaker, yeah.. i'm not sure if its useful given how twisted is doing the test running and we need the reactor support
[15:03] <hazmat> maybe
[15:03] <jimbaker> hazmat, generally it should just mix together
[15:03] <jimbaker> so twisted trial basically is managing the test runner, which is one component that can be customized in pyunit
[15:04]  * jimbaker now has a much deeper appreciation for how trial works
[16:15] <SpamapS> awesome.. I just managed to shutdown my jenkins environment, taking all the data with it...
[16:15] <SpamapS> So, this is a good chance to answer the "how are we protecting peoples' data?" question. :)
[16:15] <SpamapS> I did make an EBS snapshot of said data.. but there's basically no way to tell ensemble to re-gain that environment.
[16:21] <niemeyer> Project's in Launchpad have name, display name, title, summary and description (!)
[16:21] <niemeyer> Projects
[16:22] <niemeyer> SpamapS: Indeed, which is why I'm advocating for a while that we call this command destroy-environment rather than shutdown
[16:24] <_mup_> Bug #838215 was filed: "shutdown" must be renamed to "destroy-environment" <Ensemble:Confirmed for jimbaker> < https://launchpad.net/bugs/838215 >
[16:25] <jimbaker> sounds like a good plan
[16:26] <jimbaker> lengthy command, matches the internal api, no mistaking this is going to do something extreme
[16:26] <hazmat> well i don't think the rename would have helped SpamapS 
[16:26] <jimbaker> also parallels destroy-service
[16:27] <hazmat> the rename sounds fine
[16:27] <hazmat> addressing the underlying issue of volume management is probably a larger task
[16:27] <hazmat> maybe ;-)
[16:27] <jimbaker> well it's an interesting question of what would destroy that ;)
[16:28] <jimbaker> maybe have a command that forces you to read what it asks. to destroy everything, please sum 10 + 32. like the gmail plugin to avoid inadvertent emails
[16:37] <SpamapS> the rename would not have helped me, no
[16:37] <SpamapS> but +1 for renaming it
[16:38] <SpamapS> jimbaker: haha.. true, I did put 'echo y | ensemble shutdown' in my test scripts
[16:38] <SpamapS> the difference is I was running my test script on my machine, instead of on the jenkins machine, so it shutdown my default environment
[16:38] <SpamapS> which I had selected as the jenkins env because I was tired of adding -e jenkins
[16:39] <SpamapS> Flat out..
[16:39] <SpamapS> help me recover the data easily
[16:39] <SpamapS> Right now what I have to do is start a new raw instance, mount the snapshot, and rsync it back to a new bootstrapped/deployed env
[16:39] <jimbaker> SpamapS, yeah i figured as much. command histories, scripts, automation - all good, until they turn bad on oneself
[16:40] <jimbaker> one good thing is that shutdown now tells you which environment you will be deleting. a small detail for sure
[16:42] <SpamapS> Honestly, there's nothing good about this. :-P
[16:42] <SpamapS> Thought...
[16:42] <SpamapS> snapshot all volumes before shutdown.
[16:43] <SpamapS> Unless you configure the environment not to.
[16:43] <SpamapS> Luckily most of what I had done is captured in a bzr tree. :)
[16:56] <_mup_> Bug #838238 was filed: No matter what version of ensemble you have on client, deployed instances get the one from the PPA <Ensemble:New> < https://launchpad.net/bugs/838238 >
[17:10] <niemeyer> SpamapS, hazmat: I think the rename would have helped only in the sense that there's no return out of a destroy-environment
[17:11] <niemeyer> Makes the outcome more obvious
[17:11] <SpamapS> niemeyer: I intended to destroy the environment I had just created.
[17:11] <SpamapS> niemeyer: the jenkins service running inside it was collateral damage.
[17:12] <niemeyer> SpamapS: The jenkins was part of the environment..
[17:12] <SpamapS> Because they happened to be using the same s3 control bucket and name.
[17:12] <niemeyer> SpamapS: It's a bit like saying.. I wanted to format my disk, I just didn't want to lose that one file
[17:13] <SpamapS> It picked up the .ensemble/environments.yaml from my home dir automatically.. accidentally.
[17:13] <SpamapS> This is why I filed the bug request for a "freeze" or "lock" command.
[17:13] <SpamapS> let me control changes to an environment so automated scripts can't screw it up
[17:14] <niemeyer> SpamapS: We could change destroy-environment so it asks
[17:14] <SpamapS> It does ask
[17:14] <SpamapS> and I echo'd y in
[17:14] <niemeyer> SpamapS: Well.. :-)
[17:14] <SpamapS> because this was an automated script
[17:14] <niemeyer> SpamapS: Sorry about that then :-)
[17:14] <SpamapS> I'm not blaming ensemble, I'm suggesting that this is the first in a long line of "WTF!" questions that will arrive here when ensemble usage begins in earnest.
[17:16] <niemeyer> SpamapS: I understand, and while I share your pain, I think it was an operator issue on that one case 
[17:16] <niemeyer> SpamapS: rm -rf / does bad things too
[17:16] <SpamapS> I agree 100%
[17:16] <SpamapS> lol.. this is going to be fun
[17:17] <SpamapS> I'm going to /part whenever this question comes up. :)
[17:17] <SpamapS> I burn easily, the flames will be intense. ;)
[17:18] <niemeyer> SpamapS: Do you have a recommended solution for the problem?
[17:18] <SpamapS> Its so easy to roll out on ensemble.. it shouldn't be so easy to destroy it. I'd actually be in favor of making shutdown/destroy environment a disabled command until you mark an environment as "ephemeral" or something like that.
[17:18] <SpamapS> But then that will happen in automation too.. hrm.
[17:19] <niemeyer> SpamapS: Exactly what I was writing
[17:19] <SpamapS> Ahh, maybe you have to --ephemeral at boot...
[17:19] <SpamapS> bootstrap rather..
[17:19] <SpamapS> And then the env name/controlbucket/etc are all generated.
[17:19] <niemeyer> SpamapS: What happens if --ephemeral wasn't used?
[17:19] <SpamapS> No shutdown
[17:19] <SpamapS> no destroy env
[17:19] <SpamapS> pull it apart one piece at a time
[17:20] <niemeyer> SpamapS: Ugh..
[17:20] <SpamapS> I'm shooting from the hip here.. haven't given it much thought..
[17:21] <SpamapS> but it feels like we're playing just a little fast and loose with data when we could put a simple padlock on it.
[17:21] <SpamapS> Maybe the padlock is to just make it clear that the default environment is not to be used in automation.
[17:22] <niemeyer> SpamapS: I'm fine with adding yet another lock command, if you think that'd help
[17:23] <niemeyer> SpamapS: Hmm
[17:23] <niemeyer> SpamapS: There's another option.. disabling automation of destroy-environment..
[17:23] <niemeyer> SpamapS: This sounds better, actually
[17:23] <SpamapS> if not isatty .. say no
[17:23] <niemeyer> SpamapS: Except if we want to test things in an automated way! :-)
[17:24] <niemeyer> SpamapS: --i-am-sure-about-that-do-it-right-now-damn-it
[17:24] <SpamapS> Yeah, mdadm had some options like that
[17:25] <SpamapS> I'm just envisioning netflix admins running their test automation script and accidentally shutting down.. everything.
[17:25] <SpamapS> It almost seems more sane to just get rid of it. Keep track of your services, destroy them one by one. Terminate the machines one by one..
[17:26] <SpamapS> For my automated test script, I actually did have a shutdown equivilent for this very reason early on..
[17:26] <niemeyer> SpamapS: We created this option because we need it, very foten
[17:26] <niemeyer> often
[17:26] <SpamapS> I kept track of all my machine ids and service names and destroyed/terminated them one by one.
[17:27] <niemeyer> SpamapS: Making it painful at all times to protect the few that will automate the "yes I am sure" measure as you've done is not a good solution IMO
[17:28] <hazmat> niemeyer, so regarding the bridge it look like libvirt does a nat to it by default, with some forward rules, those aren't compatible though with a network per environment.. only one nat active at a time
[17:29] <niemeyer> hazmat: That sounds fine for now..
[17:30] <niemeyer> hazmat: I wouldn't like to get in that situation just because we were not careful to namespace, but we don't have to waste lots of cycles on the problem
[17:30] <hazmat> niemeyer, it also means that we're just going to be piggy backing on the libvirt default network.
[17:30] <hazmat> i'm looking into the libvirt network options a bit more
[17:31] <hazmat> i've got the network abstraction on libvirt done, so i'd like to keep the isolation if possible
[17:33] <niemeyer> hazmat: Yeah, just wrapping what's there sounds very neat indeed
[17:34] <hazmat> niemeyer, well if we're just using the default, there's not much point to wrapping, as we don't need to change anything
[17:34] <hazmat> outside of namespacing the containers correctly
[17:34] <hazmat> but effectively the network setup is a no-op
[17:35] <niemeyer> hazmat: I meant just reusing what's there
[17:35] <niemeyer> hazmat: As you described
[17:36] <_mup_> ensemble/local-network r341 committed by kapil.thangavelu@canonical.com
[17:36] <_mup_> a network abstraction for starting/stopping/defining libvirt networks.. not going to use but i wanted to capture it.
[17:46] <SpamapS> sorry I disappeared.. my unity/wifi/oneiric melted down
[17:47] <SpamapS> niemeyer: We're packing a self destruct into ensemble. Its useful, for sure. However, I wonder if we should just leave it out for non-developers. :-P
[17:49] <niemeyer> SpamapS: For now I think we're good.. if you a) Have the proper AWS keys; b) Have the proper environments.yaml file; c) Type destroy-environment; d) Confirm with y...  I think it's fine to destroy it.
[17:49] <niemeyer> SpamapS: I'll pay a beer to everyone who does that by mistake, so I owe you one
[17:50] <SpamapS> I think you're ok, because I basically packed a grenade w/ a pulled pin in my script.
[17:51] <SpamapS> I may actually have a better suggestion, which is to make it easier to "create" environments in an automated fashion... rather than having to type them into the yaml file.
[17:52] <SpamapS> That way people won't be tempted to just re-use the same env over and over.
[17:52] <niemeyer> SpamapS: yeah, stacks ftw
[17:53] <SpamapS> well yes stacks.. but even more isolation.. just being able to say --config-dir=/tmp/ensemble.asdfy431k would make it so I can be more careful in my automated scripts not to use a static environments.yaml ever
[17:53] <SpamapS> I mean ultimately the responsibility comes down on the automator to be careful. :)
[17:55] <SpamapS> But if I have to touch files in the home dir for automation.. thats a dangerous game.
[18:02] <niemeyer> SpamapS: We explicitly try to avoid that
[18:03] <niemeyer> SpamapS: The information in a local admin's laptop should be only the necessary to reach the environment
[18:03] <niemeyer> SpamapS: The authoritative version of the configuration should live in the environment itself, with HA etc
[18:04] <niemeyer> SpamapS: So ultimately the issue is that there's too little in the local env yaml
[18:04] <SpamapS> Ok, new problem then. I want to put safeties in my automated scripts.. something like.. if this env already exists, STOP.
[18:04] <SpamapS> hm, does status give an error if its not bootstrapped?
[18:04]  * SpamapS checks
[18:05] <SpamapS> hm, just a generic "1"
[18:05] <SpamapS> could be lots of reasons for that.. no way to be sure it means not bootstrapped other than grepping.. :P
[18:06] <niemeyer> SpamapS: Agreed.. we should do better on rcs
[18:07] <niemeyer> SpamapS: and also on helper commands
[18:07] <niemeyer> SpamapS: is-bootrapped or whatever
[18:08] <SpamapS> Well for now, I'll grep for that exact message.. hopefully that is enough to prevent another damnit moment.
[18:08] <_mup_> ensemble/rename-shutdown-command r337 committed by jim.baker@canonical.com
[18:08] <_mup_> Code changes
[18:11] <niemeyer> SpamapS: Hmm..
[18:11] <niemeyer> SpamapS: But bootstrap shouldn't really cause any issues
[18:11] <niemeyer> SpamapS: If you do it again on a bootstrapped env, that is
[18:11] <niemeyer> SpamapS: This would be a major problem
[18:11] <niemeyer> SpamapS: Rather than grepping for logs that may change, I suggest just trying it again
[18:12] <SpamapS> Right, my test script is basically  bootstrap, deploy deploy relate relate , shutdown ... 
[18:12] <SpamapS> I want to make *SURE* that I am the one doing the bootstrapping, that it wasn't already done
[18:13] <SpamapS> I'd prefer to be able to use a generated yaml file for this.. but I can't set the config dir, and I don't want to append yaml to the environments.yaml ..
[18:13] <niemeyer> SpamapS: Just if [ $? -ne 0 ]; then FAIL fi; after bootstrap?
[18:15] <SpamapS> I run with set -e
[18:15] <SpamapS> existing env is not an error in bootstrap
[18:23] <niemeyer> SpamapS: Oops.. that's certainly a bug
[18:39] <hazmat> SpamapS, define $HOME in your test script if you want better isolation
[18:39] <hazmat> SpamapS, yeah.. the return code from the ensemble cli are suspect
[18:39] <hazmat> i filed a bug a while back regrading
[18:40] <hazmat> actually jim did.. bug 697093
[18:40] <_mup_> Bug #697093: Ensemble command should return nonzero status code for errors <cli> <Ensemble:New> < https://launchpad.net/bugs/697093 >
[18:50] <_mup_> ensemble/rename-shutdown-command r338 committed by jim.baker@canonical.com
[18:50] <_mup_> Doc changes
[18:51] <_mup_> ensemble/rename-shutdown-command r339 committed by jim.baker@canonical.com
[18:51] <_mup_> Add test for destroying a specified default env
[19:24] <hazmat> i think i will keep the network abstraction, its still pretty useful
[19:24] <hazmat> even against libvirt default for exposing attributes and starting if its not started
[19:42] <_mup_> Bug #838330 was filed: Bootstrap command should error on existing <Ensemble:New> < https://launchpad.net/bugs/838330 >
[19:55] <hazmat> bcsaller1, the other important parallel piece of work is doing the lxc integration with a serviceunitdeployment subclass
[19:55] <hazmat> bcsaller1, wiring in the deployment class selection based on machine provider type later
[19:56] <bcsaller1> hazmat: I'll look at it but I might not be able to start that part today, we'll see 
[19:56] <hazmat> bcsaller1, fair enough.. i'm pushing through the last of the lxc local provider work, should have all the branches in review for tomorrow, i can start on it tomorrow
[19:57] <hazmat> its the next step
[19:58] <hazmat> bcsaller1, lost my irc connection for a moment
[19:59] <bcsaller1> hazmat: didn't miss anything
[20:10] <hazmat> bcsaller1, one think re lxc-lib, i'd like to create a container without running the customize script.. it doesn't seem like that's possible atm
[20:10] <hazmat> s/think/thing
[20:11] <hazmat> hmm
[20:11] <hazmat> i'm trying to prime the deb-cache doing bootstrap so ops like add unit are fast.. but their async anyways.. so maybe that's just a bad assumption
[20:12] <hazmat> yeah.. nevermind re lxc lib.. it makes more sense to just let the agent deal with the lag on the container creation
[20:13] <smoser> ok... for ensembel cloud-config, you should probably for your sanity put something like:
[20:13] <smoser> output:
[20:14] <hazmat> smoser, output: ?
[20:14] <smoser>  all: "| tee /var/log/cloud-init-output.log"
[20:14] <hazmat> smoser, i thought cloud-init had log file output builtin now ?
[20:15] <smoser> yes, it does log to /var/log/cloud-init.log
[20:15] <smoser> but this will capture output of it
[20:15] <smoser> and tee it
[20:15] <hazmat> cool
[20:16] <smoser> output:
[20:16] <smoser>  all: [ "| tee /var/log/cloud-init-output.log", "&1" ]
[20:16] <smoser> that would help me debug some
[20:47] <smoser> i think maybe http://paste.ubuntu.com/679208/
[20:48] <smoser> hmm... better with tee -a
[21:06] <_mup_> ensemble/local-ubuntu-provider r341 committed by kapil.thangavelu@canonical.com
[21:06] <_mup_> local provider skeleton
[21:16] <smoser> ok. my first ensemble merge proposal!
[21:16] <smoser> https://code.launchpad.net/~smoser/ensemble/cloud-init-output-log/+merge/73596
[21:16] <niemeyer> smoser: Woohay!
[21:40] <_mup_> ensemble/local-network r342 committed by kapil.thangavelu@canonical.com
[21:40] <_mup_> expose network attributes from libvirt to local provider
[21:48] <_mup_> ensemble/local-provider r343 committed by kapil.thangavelu@canonical.com
[21:48] <_mup_> local provider that bootstraps and shutsdown
[21:51] <SpamapS> smoser: hey, I was just reading cloud-init's code and I noticed that you're not passing -y to add-apt-repository
[21:53] <SpamapS> smoser: I believe this may be causing the issue with getting stuck on console... http://paste.ubuntu.com/679240/
[22:23] <_mup_> ensemble/managed-agent r343 committed by kapil.thangavelu@canonical.com
[22:23] <_mup_> managed machine agent
[22:51] <SpamapS> hrm.. bug 819329 is actually pretty tricky
[22:51] <_mup_> Bug #819329: Tests depend on AWS_ACCESS_KEY_ID being set <Ensemble:Confirmed> <ensemble (Ubuntu):Confirmed> < https://launchpad.net/bugs/819329 >
[22:51] <SpamapS> txaws is raising an error during a constructor because it can't find this environment variable...
[22:57] <SpamapS> the conundrum is, there is no sane default..
[22:58] <SpamapS> So do we set the env var in the tests? Do we shove it into the config during the tests, potentially bypassing code? Hrm.
[22:58] <hazmat> SpamapS, its a unit test, you can shove any value there
[22:58] <hazmat> SpamapS, it should never actually use it
[22:58] <hazmat> SpamapS, it can also be fixed by ensuring its in the environment config of whatever test is using it
[23:00] <SpamapS> so the real issue is that txaws needs to be mocked?
[23:01] <SpamapS> since we're not "testing txaws"
[23:02] <SpamapS> hazmat: if txaws isn't going to be mocked, then we need to test both having it and not having it in the env, so we know when that behavior changes.. 
[23:03] <hazmat> SpamapS, we're using txaws but we don't ever interact externally during the unit tests
[23:03] <hazmat> any external interaction is typically mocked
[23:12] <SpamapS> hazmat: so the simple workaround is just to set them to dummy values before running the test suite. :-P
[23:13] <bcsaller1> SpamapS: there is a change_environment function on all the test classes that will set env for the duration of the test
[23:14] <SpamapS> mmk. So to work around the problem, we can certainly just dump AWS_ACCESS_KEY_ID in .. but shouldn't we also *verify* that if its not there, an error is raised?
[23:16] <bcsaller> if the provider is ec2 for a given environment it should assert the values it depends on at runtime, yes
[23:16] <SpamapS> Thats the frustrating/confusing part.. I don't see where ec2 was even asserted. :P
[23:23] <bcsaller> SpamapS: that test is using write_sample_config which includes a default value of ec2, when it looks for the machine provider it triggers the issue you see. I agree, this is not clear
[23:42] <SpamapS> hrm.. tests don't seem to run cleanly for me anyway.. :-P
[23:46] <SpamapS> weird.. just running './test' should work pretty well, right?
[23:46] <SpamapS> I'm getting *piles* of errors
[23:50] <hazmat> SpamapS, assuming you've got txzk and zk yes
[23:50] <SpamapS> if I try to run say, ensemble.environment tests.. I get these fails:
[23:50] <SpamapS> http://paste.ubuntu.com/679281/
[23:51] <SpamapS> Looks like it lost the HOME override
[23:51] <SpamapS> ok n/m on that.. I misunderstood change_environment