/srv/irclogs.ubuntu.com/2011/08/31/#ubuntu-ensemble.txt

_mup_ensemble/fix-functional-testing r335 committed by jim.baker@canonical.com02:48
_mup_More robust ftests02:48
_mup_ensemble/fix-functional-testing r336 committed by jim.baker@canonical.com02:49
_mup_Merged trunk02:49
fwereadeactually don't worry :)10:28
niemeyerfwereade: Hey!11:56
fwereadeniemeyer: heyhey!11:56
fwereadeniemeyer: how's life?11:56
niemeyerfwereade: Awaken11:56
fwereadeniemeyer: sorry, did I miss something important?11:57
=== ehw is now known as ehw|asdf
niemeyerfwereade: Hmmm.. why do you ask?11:58
fwereadeniemeyer: ah, I wasn't sure whether to read "Awaken" as an imperative11:58
niemeyerErm..11:58
niemeyerfwereade: I don't think I even know how to interpret that way ;-)11:59
fwereadeniemeyer: I'd see "awaken" as a verb and "awake" as an adjective11:59
=== ehw|asdf is now known as ehw
fwereadeniemeyer: now I think of it, "awoken" would be a ...er, a participle? I can't actually remember much grammar12:00
niemeyerfwereade: Ohhh.. got it12:01
fwereadeniemeyer: computer languages are easier ;)12:01
niemeyerfwereade: Yeah.. see, it was my grammar being bad :)12:01
fwereadeniemeyer: no worries, I take things far too literally ;)12:01
fwereadeniemeyer: did we agree it would be ok to delete bin/ensemble-make-image and debian/ec2-build?12:37
niemeyerfwereade: We did not.. it sounds fine to me, but we should check with hazmat first as he's created and used these the most12:39
fwereadeniemeyer: I'll assume they should stay for now then, all it costs me to keep them is a couple of global constants12:40
hazmatfwereade, delete away12:40
fwereadehazmat: ah, sweet :D12:41
hazmatand g'morning to all12:41
fwereadehazmat: and a good morning to you12:42
hazmati finally gave up on screen, and switched out to using tmux, its very nice12:42
niemeyerhazmat: Morning, and welcome to tmux12:57
hazmatjimbaker, i noticed that the unit test in py 2.7 has some very nice builtin layer support at module and class level13:02
hazmatniemeyer, one tmux question i wondered about, and just curious if you knew, is it possible to reload the conf file for an existing session?13:06
hazmatfound it.. source-file ~/.tmux.conf13:11
hazmatawesomeness13:11
niemeyerhazmat: Hmm13:16
niemeyerhazmat: Good question13:16
niemeyerhazmat: Ah, you found it, cool13:16
niemeyerhazmat: I haven't used that yet13:16
hazmatniemeyer, makes testing new settings with tmux rather easy also find some nice tools from the tmux home page.. this one in particular fits my usage pretty nicely.. https://github.com/aziz/tmuxinator13:17
niemeyerhazmat: True.. checking13:18
hazmatalso in the same vein but with more support for pane splitting https://github.com/remiprev/teamocil13:18
niemeyerhazmat: Oh, neat13:19
niemeyerhazmat: I create new tabs as I go13:19
hazmatniemeyer, i normally do.. but crashes are frequent enough.. that having some auto-restore functionality helps me retain context, i'm trying to put in a cron job that uses ps-util to sniff all the process hierarchy for the windows and stores them in the yaml file, so upgrade instability isn't as bad..13:20
hazmatit sort of like that software you mentioned a while back.. that has this awesome resume after crash system to take you back to exactly what you where doing b4 the crash  ;-)13:21
niemeyerhazmat: Wow, so freezing a running tmux?13:25
niemeyerhazmat: Yeah.. Cinelerra :-)13:26
hazmatniemeyer, its not really freezing, its just recording the running programs to restart them.. i'm using an emacs per window, and the emacs is auto-saving its open files list, so on startup i'll get back  my open buffers... if the program didn't support resumption, then the context is loss..13:27
niemeyerhazmat: I see.. quite neat even then13:30
fwereadehey, hazmat, can I delete those 2 scripts in trunk as a trivial?14:28
hazmatfwereade, sounds good14:29
fwereadehazmat: cheers14:29
jimbakerhazmat, thanks, i will take a look at that15:02
hazmatjimbaker, yeah.. i'm not sure if its useful given how twisted is doing the test running and we need the reactor support15:03
hazmatmaybe15:03
jimbakerhazmat, generally it should just mix together15:03
jimbakerso twisted trial basically is managing the test runner, which is one component that can be customized in pyunit15:03
* jimbaker now has a much deeper appreciation for how trial works15:04
SpamapSawesome.. I just managed to shutdown my jenkins environment, taking all the data with it...16:15
SpamapSSo, this is a good chance to answer the "how are we protecting peoples' data?" question. :)16:15
SpamapSI did make an EBS snapshot of said data.. but there's basically no way to tell ensemble to re-gain that environment.16:15
niemeyerProject's in Launchpad have name, display name, title, summary and description (!)16:21
niemeyerProjects16:21
niemeyerSpamapS: Indeed, which is why I'm advocating for a while that we call this command destroy-environment rather than shutdown16:22
_mup_Bug #838215 was filed: "shutdown" must be renamed to "destroy-environment" <Ensemble:Confirmed for jimbaker> < https://launchpad.net/bugs/838215 >16:24
jimbakersounds like a good plan16:25
jimbakerlengthy command, matches the internal api, no mistaking this is going to do something extreme16:26
hazmatwell i don't think the rename would have helped SpamapS 16:26
jimbakeralso parallels destroy-service16:26
hazmatthe rename sounds fine16:27
hazmataddressing the underlying issue of volume management is probably a larger task16:27
hazmatmaybe ;-)16:27
jimbakerwell it's an interesting question of what would destroy that ;)16:27
jimbakermaybe have a command that forces you to read what it asks. to destroy everything, please sum 10 + 32. like the gmail plugin to avoid inadvertent emails16:28
SpamapSthe rename would not have helped me, no16:37
SpamapSbut +1 for renaming it16:37
SpamapSjimbaker: haha.. true, I did put 'echo y | ensemble shutdown' in my test scripts16:38
SpamapSthe difference is I was running my test script on my machine, instead of on the jenkins machine, so it shutdown my default environment16:38
SpamapSwhich I had selected as the jenkins env because I was tired of adding -e jenkins16:38
SpamapSFlat out..16:39
SpamapShelp me recover the data easily16:39
SpamapSRight now what I have to do is start a new raw instance, mount the snapshot, and rsync it back to a new bootstrapped/deployed env16:39
jimbakerSpamapS, yeah i figured as much. command histories, scripts, automation - all good, until they turn bad on oneself16:39
jimbakerone good thing is that shutdown now tells you which environment you will be deleting. a small detail for sure16:40
SpamapSHonestly, there's nothing good about this. :-P16:42
SpamapSThought...16:42
SpamapSsnapshot all volumes before shutdown.16:42
SpamapSUnless you configure the environment not to.16:43
SpamapSLuckily most of what I had done is captured in a bzr tree. :)16:43
_mup_Bug #838238 was filed: No matter what version of ensemble you have on client, deployed instances get the one from the PPA <Ensemble:New> < https://launchpad.net/bugs/838238 >16:56
=== foods is now known as adam_g
niemeyerSpamapS, hazmat: I think the rename would have helped only in the sense that there's no return out of a destroy-environment17:10
niemeyerMakes the outcome more obvious17:11
SpamapSniemeyer: I intended to destroy the environment I had just created.17:11
SpamapSniemeyer: the jenkins service running inside it was collateral damage.17:11
niemeyerSpamapS: The jenkins was part of the environment..17:12
SpamapSBecause they happened to be using the same s3 control bucket and name.17:12
niemeyerSpamapS: It's a bit like saying.. I wanted to format my disk, I just didn't want to lose that one file17:12
SpamapSIt picked up the .ensemble/environments.yaml from my home dir automatically.. accidentally.17:13
SpamapSThis is why I filed the bug request for a "freeze" or "lock" command.17:13
SpamapSlet me control changes to an environment so automated scripts can't screw it up17:13
niemeyerSpamapS: We could change destroy-environment so it asks17:14
SpamapSIt does ask17:14
SpamapSand I echo'd y in17:14
niemeyerSpamapS: Well.. :-)17:14
SpamapSbecause this was an automated script17:14
niemeyerSpamapS: Sorry about that then :-)17:14
SpamapSI'm not blaming ensemble, I'm suggesting that this is the first in a long line of "WTF!" questions that will arrive here when ensemble usage begins in earnest.17:14
niemeyerSpamapS: I understand, and while I share your pain, I think it was an operator issue on that one case 17:16
niemeyerSpamapS: rm -rf / does bad things too17:16
SpamapSI agree 100%17:16
SpamapSlol.. this is going to be fun17:16
SpamapSI'm going to /part whenever this question comes up. :)17:17
SpamapSI burn easily, the flames will be intense. ;)17:17
niemeyerSpamapS: Do you have a recommended solution for the problem?17:18
SpamapSIts so easy to roll out on ensemble.. it shouldn't be so easy to destroy it. I'd actually be in favor of making shutdown/destroy environment a disabled command until you mark an environment as "ephemeral" or something like that.17:18
SpamapSBut then that will happen in automation too.. hrm.17:18
niemeyerSpamapS: Exactly what I was writing17:19
SpamapSAhh, maybe you have to --ephemeral at boot...17:19
SpamapSbootstrap rather..17:19
SpamapSAnd then the env name/controlbucket/etc are all generated.17:19
niemeyerSpamapS: What happens if --ephemeral wasn't used?17:19
SpamapSNo shutdown17:19
SpamapSno destroy env17:19
SpamapSpull it apart one piece at a time17:19
niemeyerSpamapS: Ugh..17:20
SpamapSI'm shooting from the hip here.. haven't given it much thought..17:20
SpamapSbut it feels like we're playing just a little fast and loose with data when we could put a simple padlock on it.17:21
SpamapSMaybe the padlock is to just make it clear that the default environment is not to be used in automation.17:21
niemeyerSpamapS: I'm fine with adding yet another lock command, if you think that'd help17:22
niemeyerSpamapS: Hmm17:23
niemeyerSpamapS: There's another option.. disabling automation of destroy-environment..17:23
niemeyerSpamapS: This sounds better, actually17:23
SpamapSif not isatty .. say no17:23
niemeyerSpamapS: Except if we want to test things in an automated way! :-)17:23
niemeyerSpamapS: --i-am-sure-about-that-do-it-right-now-damn-it17:24
SpamapSYeah, mdadm had some options like that17:24
SpamapSI'm just envisioning netflix admins running their test automation script and accidentally shutting down.. everything.17:25
SpamapSIt almost seems more sane to just get rid of it. Keep track of your services, destroy them one by one. Terminate the machines one by one..17:25
SpamapSFor my automated test script, I actually did have a shutdown equivilent for this very reason early on..17:26
niemeyerSpamapS: We created this option because we need it, very foten17:26
niemeyeroften17:26
SpamapSI kept track of all my machine ids and service names and destroyed/terminated them one by one.17:26
niemeyerSpamapS: Making it painful at all times to protect the few that will automate the "yes I am sure" measure as you've done is not a good solution IMO17:27
hazmatniemeyer, so regarding the bridge it look like libvirt does a nat to it by default, with some forward rules, those aren't compatible though with a network per environment.. only one nat active at a time17:28
niemeyerhazmat: That sounds fine for now..17:29
niemeyerhazmat: I wouldn't like to get in that situation just because we were not careful to namespace, but we don't have to waste lots of cycles on the problem17:30
hazmatniemeyer, it also means that we're just going to be piggy backing on the libvirt default network.17:30
hazmati'm looking into the libvirt network options a bit more17:30
hazmati've got the network abstraction on libvirt done, so i'd like to keep the isolation if possible17:31
niemeyerhazmat: Yeah, just wrapping what's there sounds very neat indeed17:33
hazmatniemeyer, well if we're just using the default, there's not much point to wrapping, as we don't need to change anything17:34
hazmatoutside of namespacing the containers correctly17:34
hazmatbut effectively the network setup is a no-op17:34
niemeyerhazmat: I meant just reusing what's there17:35
niemeyerhazmat: As you described17:35
_mup_ensemble/local-network r341 committed by kapil.thangavelu@canonical.com17:36
_mup_a network abstraction for starting/stopping/defining libvirt networks.. not going to use but i wanted to capture it.17:36
SpamapSsorry I disappeared.. my unity/wifi/oneiric melted down17:46
SpamapSniemeyer: We're packing a self destruct into ensemble. Its useful, for sure. However, I wonder if we should just leave it out for non-developers. :-P17:47
niemeyerSpamapS: For now I think we're good.. if you a) Have the proper AWS keys; b) Have the proper environments.yaml file; c) Type destroy-environment; d) Confirm with y...  I think it's fine to destroy it.17:49
niemeyerSpamapS: I'll pay a beer to everyone who does that by mistake, so I owe you one17:49
SpamapSI think you're ok, because I basically packed a grenade w/ a pulled pin in my script.17:50
SpamapSI may actually have a better suggestion, which is to make it easier to "create" environments in an automated fashion... rather than having to type them into the yaml file.17:51
SpamapSThat way people won't be tempted to just re-use the same env over and over.17:52
niemeyerSpamapS: yeah, stacks ftw17:52
SpamapSwell yes stacks.. but even more isolation.. just being able to say --config-dir=/tmp/ensemble.asdfy431k would make it so I can be more careful in my automated scripts not to use a static environments.yaml ever17:53
SpamapSI mean ultimately the responsibility comes down on the automator to be careful. :)17:53
SpamapSBut if I have to touch files in the home dir for automation.. thats a dangerous game.17:55
niemeyerSpamapS: We explicitly try to avoid that18:02
niemeyerSpamapS: The information in a local admin's laptop should be only the necessary to reach the environment18:03
niemeyerSpamapS: The authoritative version of the configuration should live in the environment itself, with HA etc18:03
niemeyerSpamapS: So ultimately the issue is that there's too little in the local env yaml18:04
SpamapSOk, new problem then. I want to put safeties in my automated scripts.. something like.. if this env already exists, STOP.18:04
SpamapShm, does status give an error if its not bootstrapped?18:04
* SpamapS checks18:04
SpamapShm, just a generic "1"18:05
SpamapScould be lots of reasons for that.. no way to be sure it means not bootstrapped other than grepping.. :P18:05
niemeyerSpamapS: Agreed.. we should do better on rcs18:06
niemeyerSpamapS: and also on helper commands18:07
niemeyerSpamapS: is-bootrapped or whatever18:07
SpamapSWell for now, I'll grep for that exact message.. hopefully that is enough to prevent another damnit moment.18:08
_mup_ensemble/rename-shutdown-command r337 committed by jim.baker@canonical.com18:08
_mup_Code changes18:08
niemeyerSpamapS: Hmm..18:11
niemeyerSpamapS: But bootstrap shouldn't really cause any issues18:11
niemeyerSpamapS: If you do it again on a bootstrapped env, that is18:11
niemeyerSpamapS: This would be a major problem18:11
niemeyerSpamapS: Rather than grepping for logs that may change, I suggest just trying it again18:11
SpamapSRight, my test script is basically  bootstrap, deploy deploy relate relate , shutdown ... 18:12
SpamapSI want to make *SURE* that I am the one doing the bootstrapping, that it wasn't already done18:12
SpamapSI'd prefer to be able to use a generated yaml file for this.. but I can't set the config dir, and I don't want to append yaml to the environments.yaml ..18:13
niemeyerSpamapS: Just if [ $? -ne 0 ]; then FAIL fi; after bootstrap?18:13
SpamapSI run with set -e18:15
SpamapSexisting env is not an error in bootstrap18:15
niemeyerSpamapS: Oops.. that's certainly a bug18:23
hazmatSpamapS, define $HOME in your test script if you want better isolation18:39
hazmatSpamapS, yeah.. the return code from the ensemble cli are suspect18:39
hazmati filed a bug a while back regrading18:39
hazmatactually jim did.. bug 69709318:40
_mup_Bug #697093: Ensemble command should return nonzero status code for errors <cli> <Ensemble:New> < https://launchpad.net/bugs/697093 >18:40
_mup_ensemble/rename-shutdown-command r338 committed by jim.baker@canonical.com18:50
_mup_Doc changes18:50
_mup_ensemble/rename-shutdown-command r339 committed by jim.baker@canonical.com18:51
_mup_Add test for destroying a specified default env18:51
hazmati think i will keep the network abstraction, its still pretty useful19:24
hazmateven against libvirt default for exposing attributes and starting if its not started19:24
_mup_Bug #838330 was filed: Bootstrap command should error on existing <Ensemble:New> < https://launchpad.net/bugs/838330 >19:42
hazmatbcsaller1, the other important parallel piece of work is doing the lxc integration with a serviceunitdeployment subclass19:55
hazmatbcsaller1, wiring in the deployment class selection based on machine provider type later19:55
bcsaller1hazmat: I'll look at it but I might not be able to start that part today, we'll see 19:56
hazmatbcsaller1, fair enough.. i'm pushing through the last of the lxc local provider work, should have all the branches in review for tomorrow, i can start on it tomorrow19:56
hazmatits the next step19:57
hazmatbcsaller1, lost my irc connection for a moment19:58
bcsaller1hazmat: didn't miss anything19:59
hazmatbcsaller1, one think re lxc-lib, i'd like to create a container without running the customize script.. it doesn't seem like that's possible atm20:10
hazmats/think/thing20:10
hazmathmm20:11
hazmati'm trying to prime the deb-cache doing bootstrap so ops like add unit are fast.. but their async anyways.. so maybe that's just a bad assumption20:11
hazmatyeah.. nevermind re lxc lib.. it makes more sense to just let the agent deal with the lag on the container creation20:12
smoserok... for ensembel cloud-config, you should probably for your sanity put something like:20:13
smoseroutput:20:13
hazmatsmoser, output: ?20:14
smoser all: "| tee /var/log/cloud-init-output.log"20:14
hazmatsmoser, i thought cloud-init had log file output builtin now ?20:14
smoseryes, it does log to /var/log/cloud-init.log20:15
smoserbut this will capture output of it20:15
smoserand tee it20:15
hazmatcool20:15
smoseroutput:20:16
smoser all: [ "| tee /var/log/cloud-init-output.log", "&1" ]20:16
smoserthat would help me debug some20:16
smoseri think maybe http://paste.ubuntu.com/679208/20:47
smoserhmm... better with tee -a20:48
_mup_ensemble/local-ubuntu-provider r341 committed by kapil.thangavelu@canonical.com21:06
_mup_local provider skeleton21:06
smoserok. my first ensemble merge proposal!21:16
smoserhttps://code.launchpad.net/~smoser/ensemble/cloud-init-output-log/+merge/7359621:16
niemeyersmoser: Woohay!21:16
_mup_ensemble/local-network r342 committed by kapil.thangavelu@canonical.com21:40
_mup_expose network attributes from libvirt to local provider21:40
_mup_ensemble/local-provider r343 committed by kapil.thangavelu@canonical.com21:48
_mup_local provider that bootstraps and shutsdown21:48
SpamapSsmoser: hey, I was just reading cloud-init's code and I noticed that you're not passing -y to add-apt-repository21:51
SpamapSsmoser: I believe this may be causing the issue with getting stuck on console... http://paste.ubuntu.com/679240/21:53
_mup_ensemble/managed-agent r343 committed by kapil.thangavelu@canonical.com22:23
_mup_managed machine agent22:23
SpamapShrm.. bug 819329 is actually pretty tricky22:51
_mup_Bug #819329: Tests depend on AWS_ACCESS_KEY_ID being set <Ensemble:Confirmed> <ensemble (Ubuntu):Confirmed> < https://launchpad.net/bugs/819329 >22:51
SpamapStxaws is raising an error during a constructor because it can't find this environment variable...22:51
SpamapSthe conundrum is, there is no sane default..22:57
SpamapSSo do we set the env var in the tests? Do we shove it into the config during the tests, potentially bypassing code? Hrm.22:58
hazmatSpamapS, its a unit test, you can shove any value there22:58
hazmatSpamapS, it should never actually use it22:58
hazmatSpamapS, it can also be fixed by ensuring its in the environment config of whatever test is using it22:58
SpamapSso the real issue is that txaws needs to be mocked?23:00
SpamapSsince we're not "testing txaws"23:01
SpamapShazmat: if txaws isn't going to be mocked, then we need to test both having it and not having it in the env, so we know when that behavior changes.. 23:02
hazmatSpamapS, we're using txaws but we don't ever interact externally during the unit tests23:03
hazmatany external interaction is typically mocked23:03
SpamapShazmat: so the simple workaround is just to set them to dummy values before running the test suite. :-P23:12
bcsaller1SpamapS: there is a change_environment function on all the test classes that will set env for the duration of the test23:13
=== bcsaller1 is now known as bcsaller
SpamapSmmk. So to work around the problem, we can certainly just dump AWS_ACCESS_KEY_ID in .. but shouldn't we also *verify* that if its not there, an error is raised?23:14
bcsallerif the provider is ec2 for a given environment it should assert the values it depends on at runtime, yes23:16
SpamapSThats the frustrating/confusing part.. I don't see where ec2 was even asserted. :P23:16
bcsallerSpamapS: that test is using write_sample_config which includes a default value of ec2, when it looks for the machine provider it triggers the issue you see. I agree, this is not clear23:23
SpamapShrm.. tests don't seem to run cleanly for me anyway.. :-P23:42
SpamapSweird.. just running './test' should work pretty well, right?23:46
SpamapSI'm getting *piles* of errors23:46
hazmatSpamapS, assuming you've got txzk and zk yes23:50
SpamapSif I try to run say, ensemble.environment tests.. I get these fails:23:50
SpamapShttp://paste.ubuntu.com/679281/23:50
SpamapSLooks like it lost the HOME override23:51
SpamapSok n/m on that.. I misunderstood change_environment23:51

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!