_mup_ | ensemble/fix-functional-testing r335 committed by jim.baker@canonical.com | 02:48 |
---|---|---|
_mup_ | More robust ftests | 02:48 |
_mup_ | ensemble/fix-functional-testing r336 committed by jim.baker@canonical.com | 02:49 |
_mup_ | Merged trunk | 02:49 |
fwereade | actually don't worry :) | 10:28 |
niemeyer | fwereade: Hey! | 11:56 |
fwereade | niemeyer: heyhey! | 11:56 |
fwereade | niemeyer: how's life? | 11:56 |
niemeyer | fwereade: Awaken | 11:56 |
fwereade | niemeyer: sorry, did I miss something important? | 11:57 |
=== ehw is now known as ehw|asdf | ||
niemeyer | fwereade: Hmmm.. why do you ask? | 11:58 |
fwereade | niemeyer: ah, I wasn't sure whether to read "Awaken" as an imperative | 11:58 |
niemeyer | Erm.. | 11:58 |
niemeyer | fwereade: I don't think I even know how to interpret that way ;-) | 11:59 |
fwereade | niemeyer: I'd see "awaken" as a verb and "awake" as an adjective | 11:59 |
=== ehw|asdf is now known as ehw | ||
fwereade | niemeyer: now I think of it, "awoken" would be a ...er, a participle? I can't actually remember much grammar | 12:00 |
niemeyer | fwereade: Ohhh.. got it | 12:01 |
fwereade | niemeyer: computer languages are easier ;) | 12:01 |
niemeyer | fwereade: Yeah.. see, it was my grammar being bad :) | 12:01 |
fwereade | niemeyer: no worries, I take things far too literally ;) | 12:01 |
fwereade | niemeyer: did we agree it would be ok to delete bin/ensemble-make-image and debian/ec2-build? | 12:37 |
niemeyer | fwereade: We did not.. it sounds fine to me, but we should check with hazmat first as he's created and used these the most | 12:39 |
fwereade | niemeyer: I'll assume they should stay for now then, all it costs me to keep them is a couple of global constants | 12:40 |
hazmat | fwereade, delete away | 12:40 |
fwereade | hazmat: ah, sweet :D | 12:41 |
hazmat | and g'morning to all | 12:41 |
fwereade | hazmat: and a good morning to you | 12:42 |
hazmat | i finally gave up on screen, and switched out to using tmux, its very nice | 12:42 |
niemeyer | hazmat: Morning, and welcome to tmux | 12:57 |
hazmat | jimbaker, i noticed that the unit test in py 2.7 has some very nice builtin layer support at module and class level | 13:02 |
hazmat | niemeyer, one tmux question i wondered about, and just curious if you knew, is it possible to reload the conf file for an existing session? | 13:06 |
hazmat | found it.. source-file ~/.tmux.conf | 13:11 |
hazmat | awesomeness | 13:11 |
niemeyer | hazmat: Hmm | 13:16 |
niemeyer | hazmat: Good question | 13:16 |
niemeyer | hazmat: Ah, you found it, cool | 13:16 |
niemeyer | hazmat: I haven't used that yet | 13:16 |
hazmat | niemeyer, makes testing new settings with tmux rather easy also find some nice tools from the tmux home page.. this one in particular fits my usage pretty nicely.. https://github.com/aziz/tmuxinator | 13:17 |
niemeyer | hazmat: True.. checking | 13:18 |
hazmat | also in the same vein but with more support for pane splitting https://github.com/remiprev/teamocil | 13:18 |
niemeyer | hazmat: Oh, neat | 13:19 |
niemeyer | hazmat: I create new tabs as I go | 13:19 |
hazmat | niemeyer, i normally do.. but crashes are frequent enough.. that having some auto-restore functionality helps me retain context, i'm trying to put in a cron job that uses ps-util to sniff all the process hierarchy for the windows and stores them in the yaml file, so upgrade instability isn't as bad.. | 13:20 |
hazmat | it sort of like that software you mentioned a while back.. that has this awesome resume after crash system to take you back to exactly what you where doing b4 the crash ;-) | 13:21 |
niemeyer | hazmat: Wow, so freezing a running tmux? | 13:25 |
niemeyer | hazmat: Yeah.. Cinelerra :-) | 13:26 |
hazmat | niemeyer, its not really freezing, its just recording the running programs to restart them.. i'm using an emacs per window, and the emacs is auto-saving its open files list, so on startup i'll get back my open buffers... if the program didn't support resumption, then the context is loss.. | 13:27 |
niemeyer | hazmat: I see.. quite neat even then | 13:30 |
fwereade | hey, hazmat, can I delete those 2 scripts in trunk as a trivial? | 14:28 |
hazmat | fwereade, sounds good | 14:29 |
fwereade | hazmat: cheers | 14:29 |
jimbaker | hazmat, thanks, i will take a look at that | 15:02 |
hazmat | jimbaker, yeah.. i'm not sure if its useful given how twisted is doing the test running and we need the reactor support | 15:03 |
hazmat | maybe | 15:03 |
jimbaker | hazmat, generally it should just mix together | 15:03 |
jimbaker | so twisted trial basically is managing the test runner, which is one component that can be customized in pyunit | 15:03 |
* jimbaker now has a much deeper appreciation for how trial works | 15:04 | |
SpamapS | awesome.. I just managed to shutdown my jenkins environment, taking all the data with it... | 16:15 |
SpamapS | So, this is a good chance to answer the "how are we protecting peoples' data?" question. :) | 16:15 |
SpamapS | I did make an EBS snapshot of said data.. but there's basically no way to tell ensemble to re-gain that environment. | 16:15 |
niemeyer | Project's in Launchpad have name, display name, title, summary and description (!) | 16:21 |
niemeyer | Projects | 16:21 |
niemeyer | SpamapS: Indeed, which is why I'm advocating for a while that we call this command destroy-environment rather than shutdown | 16:22 |
_mup_ | Bug #838215 was filed: "shutdown" must be renamed to "destroy-environment" <Ensemble:Confirmed for jimbaker> < https://launchpad.net/bugs/838215 > | 16:24 |
jimbaker | sounds like a good plan | 16:25 |
jimbaker | lengthy command, matches the internal api, no mistaking this is going to do something extreme | 16:26 |
hazmat | well i don't think the rename would have helped SpamapS | 16:26 |
jimbaker | also parallels destroy-service | 16:26 |
hazmat | the rename sounds fine | 16:27 |
hazmat | addressing the underlying issue of volume management is probably a larger task | 16:27 |
hazmat | maybe ;-) | 16:27 |
jimbaker | well it's an interesting question of what would destroy that ;) | 16:27 |
jimbaker | maybe have a command that forces you to read what it asks. to destroy everything, please sum 10 + 32. like the gmail plugin to avoid inadvertent emails | 16:28 |
SpamapS | the rename would not have helped me, no | 16:37 |
SpamapS | but +1 for renaming it | 16:37 |
SpamapS | jimbaker: haha.. true, I did put 'echo y | ensemble shutdown' in my test scripts | 16:38 |
SpamapS | the difference is I was running my test script on my machine, instead of on the jenkins machine, so it shutdown my default environment | 16:38 |
SpamapS | which I had selected as the jenkins env because I was tired of adding -e jenkins | 16:38 |
SpamapS | Flat out.. | 16:39 |
SpamapS | help me recover the data easily | 16:39 |
SpamapS | Right now what I have to do is start a new raw instance, mount the snapshot, and rsync it back to a new bootstrapped/deployed env | 16:39 |
jimbaker | SpamapS, yeah i figured as much. command histories, scripts, automation - all good, until they turn bad on oneself | 16:39 |
jimbaker | one good thing is that shutdown now tells you which environment you will be deleting. a small detail for sure | 16:40 |
SpamapS | Honestly, there's nothing good about this. :-P | 16:42 |
SpamapS | Thought... | 16:42 |
SpamapS | snapshot all volumes before shutdown. | 16:42 |
SpamapS | Unless you configure the environment not to. | 16:43 |
SpamapS | Luckily most of what I had done is captured in a bzr tree. :) | 16:43 |
_mup_ | Bug #838238 was filed: No matter what version of ensemble you have on client, deployed instances get the one from the PPA <Ensemble:New> < https://launchpad.net/bugs/838238 > | 16:56 |
=== foods is now known as adam_g | ||
niemeyer | SpamapS, hazmat: I think the rename would have helped only in the sense that there's no return out of a destroy-environment | 17:10 |
niemeyer | Makes the outcome more obvious | 17:11 |
SpamapS | niemeyer: I intended to destroy the environment I had just created. | 17:11 |
SpamapS | niemeyer: the jenkins service running inside it was collateral damage. | 17:11 |
niemeyer | SpamapS: The jenkins was part of the environment.. | 17:12 |
SpamapS | Because they happened to be using the same s3 control bucket and name. | 17:12 |
niemeyer | SpamapS: It's a bit like saying.. I wanted to format my disk, I just didn't want to lose that one file | 17:12 |
SpamapS | It picked up the .ensemble/environments.yaml from my home dir automatically.. accidentally. | 17:13 |
SpamapS | This is why I filed the bug request for a "freeze" or "lock" command. | 17:13 |
SpamapS | let me control changes to an environment so automated scripts can't screw it up | 17:13 |
niemeyer | SpamapS: We could change destroy-environment so it asks | 17:14 |
SpamapS | It does ask | 17:14 |
SpamapS | and I echo'd y in | 17:14 |
niemeyer | SpamapS: Well.. :-) | 17:14 |
SpamapS | because this was an automated script | 17:14 |
niemeyer | SpamapS: Sorry about that then :-) | 17:14 |
SpamapS | I'm not blaming ensemble, I'm suggesting that this is the first in a long line of "WTF!" questions that will arrive here when ensemble usage begins in earnest. | 17:14 |
niemeyer | SpamapS: I understand, and while I share your pain, I think it was an operator issue on that one case | 17:16 |
niemeyer | SpamapS: rm -rf / does bad things too | 17:16 |
SpamapS | I agree 100% | 17:16 |
SpamapS | lol.. this is going to be fun | 17:16 |
SpamapS | I'm going to /part whenever this question comes up. :) | 17:17 |
SpamapS | I burn easily, the flames will be intense. ;) | 17:17 |
niemeyer | SpamapS: Do you have a recommended solution for the problem? | 17:18 |
SpamapS | Its so easy to roll out on ensemble.. it shouldn't be so easy to destroy it. I'd actually be in favor of making shutdown/destroy environment a disabled command until you mark an environment as "ephemeral" or something like that. | 17:18 |
SpamapS | But then that will happen in automation too.. hrm. | 17:18 |
niemeyer | SpamapS: Exactly what I was writing | 17:19 |
SpamapS | Ahh, maybe you have to --ephemeral at boot... | 17:19 |
SpamapS | bootstrap rather.. | 17:19 |
SpamapS | And then the env name/controlbucket/etc are all generated. | 17:19 |
niemeyer | SpamapS: What happens if --ephemeral wasn't used? | 17:19 |
SpamapS | No shutdown | 17:19 |
SpamapS | no destroy env | 17:19 |
SpamapS | pull it apart one piece at a time | 17:19 |
niemeyer | SpamapS: Ugh.. | 17:20 |
SpamapS | I'm shooting from the hip here.. haven't given it much thought.. | 17:20 |
SpamapS | but it feels like we're playing just a little fast and loose with data when we could put a simple padlock on it. | 17:21 |
SpamapS | Maybe the padlock is to just make it clear that the default environment is not to be used in automation. | 17:21 |
niemeyer | SpamapS: I'm fine with adding yet another lock command, if you think that'd help | 17:22 |
niemeyer | SpamapS: Hmm | 17:23 |
niemeyer | SpamapS: There's another option.. disabling automation of destroy-environment.. | 17:23 |
niemeyer | SpamapS: This sounds better, actually | 17:23 |
SpamapS | if not isatty .. say no | 17:23 |
niemeyer | SpamapS: Except if we want to test things in an automated way! :-) | 17:23 |
niemeyer | SpamapS: --i-am-sure-about-that-do-it-right-now-damn-it | 17:24 |
SpamapS | Yeah, mdadm had some options like that | 17:24 |
SpamapS | I'm just envisioning netflix admins running their test automation script and accidentally shutting down.. everything. | 17:25 |
SpamapS | It almost seems more sane to just get rid of it. Keep track of your services, destroy them one by one. Terminate the machines one by one.. | 17:25 |
SpamapS | For my automated test script, I actually did have a shutdown equivilent for this very reason early on.. | 17:26 |
niemeyer | SpamapS: We created this option because we need it, very foten | 17:26 |
niemeyer | often | 17:26 |
SpamapS | I kept track of all my machine ids and service names and destroyed/terminated them one by one. | 17:26 |
niemeyer | SpamapS: Making it painful at all times to protect the few that will automate the "yes I am sure" measure as you've done is not a good solution IMO | 17:27 |
hazmat | niemeyer, so regarding the bridge it look like libvirt does a nat to it by default, with some forward rules, those aren't compatible though with a network per environment.. only one nat active at a time | 17:28 |
niemeyer | hazmat: That sounds fine for now.. | 17:29 |
niemeyer | hazmat: I wouldn't like to get in that situation just because we were not careful to namespace, but we don't have to waste lots of cycles on the problem | 17:30 |
hazmat | niemeyer, it also means that we're just going to be piggy backing on the libvirt default network. | 17:30 |
hazmat | i'm looking into the libvirt network options a bit more | 17:30 |
hazmat | i've got the network abstraction on libvirt done, so i'd like to keep the isolation if possible | 17:31 |
niemeyer | hazmat: Yeah, just wrapping what's there sounds very neat indeed | 17:33 |
hazmat | niemeyer, well if we're just using the default, there's not much point to wrapping, as we don't need to change anything | 17:34 |
hazmat | outside of namespacing the containers correctly | 17:34 |
hazmat | but effectively the network setup is a no-op | 17:34 |
niemeyer | hazmat: I meant just reusing what's there | 17:35 |
niemeyer | hazmat: As you described | 17:35 |
_mup_ | ensemble/local-network r341 committed by kapil.thangavelu@canonical.com | 17:36 |
_mup_ | a network abstraction for starting/stopping/defining libvirt networks.. not going to use but i wanted to capture it. | 17:36 |
SpamapS | sorry I disappeared.. my unity/wifi/oneiric melted down | 17:46 |
SpamapS | niemeyer: We're packing a self destruct into ensemble. Its useful, for sure. However, I wonder if we should just leave it out for non-developers. :-P | 17:47 |
niemeyer | SpamapS: For now I think we're good.. if you a) Have the proper AWS keys; b) Have the proper environments.yaml file; c) Type destroy-environment; d) Confirm with y... I think it's fine to destroy it. | 17:49 |
niemeyer | SpamapS: I'll pay a beer to everyone who does that by mistake, so I owe you one | 17:49 |
SpamapS | I think you're ok, because I basically packed a grenade w/ a pulled pin in my script. | 17:50 |
SpamapS | I may actually have a better suggestion, which is to make it easier to "create" environments in an automated fashion... rather than having to type them into the yaml file. | 17:51 |
SpamapS | That way people won't be tempted to just re-use the same env over and over. | 17:52 |
niemeyer | SpamapS: yeah, stacks ftw | 17:52 |
SpamapS | well yes stacks.. but even more isolation.. just being able to say --config-dir=/tmp/ensemble.asdfy431k would make it so I can be more careful in my automated scripts not to use a static environments.yaml ever | 17:53 |
SpamapS | I mean ultimately the responsibility comes down on the automator to be careful. :) | 17:53 |
SpamapS | But if I have to touch files in the home dir for automation.. thats a dangerous game. | 17:55 |
niemeyer | SpamapS: We explicitly try to avoid that | 18:02 |
niemeyer | SpamapS: The information in a local admin's laptop should be only the necessary to reach the environment | 18:03 |
niemeyer | SpamapS: The authoritative version of the configuration should live in the environment itself, with HA etc | 18:03 |
niemeyer | SpamapS: So ultimately the issue is that there's too little in the local env yaml | 18:04 |
SpamapS | Ok, new problem then. I want to put safeties in my automated scripts.. something like.. if this env already exists, STOP. | 18:04 |
SpamapS | hm, does status give an error if its not bootstrapped? | 18:04 |
* SpamapS checks | 18:04 | |
SpamapS | hm, just a generic "1" | 18:05 |
SpamapS | could be lots of reasons for that.. no way to be sure it means not bootstrapped other than grepping.. :P | 18:05 |
niemeyer | SpamapS: Agreed.. we should do better on rcs | 18:06 |
niemeyer | SpamapS: and also on helper commands | 18:07 |
niemeyer | SpamapS: is-bootrapped or whatever | 18:07 |
SpamapS | Well for now, I'll grep for that exact message.. hopefully that is enough to prevent another damnit moment. | 18:08 |
_mup_ | ensemble/rename-shutdown-command r337 committed by jim.baker@canonical.com | 18:08 |
_mup_ | Code changes | 18:08 |
niemeyer | SpamapS: Hmm.. | 18:11 |
niemeyer | SpamapS: But bootstrap shouldn't really cause any issues | 18:11 |
niemeyer | SpamapS: If you do it again on a bootstrapped env, that is | 18:11 |
niemeyer | SpamapS: This would be a major problem | 18:11 |
niemeyer | SpamapS: Rather than grepping for logs that may change, I suggest just trying it again | 18:11 |
SpamapS | Right, my test script is basically bootstrap, deploy deploy relate relate , shutdown ... | 18:12 |
SpamapS | I want to make *SURE* that I am the one doing the bootstrapping, that it wasn't already done | 18:12 |
SpamapS | I'd prefer to be able to use a generated yaml file for this.. but I can't set the config dir, and I don't want to append yaml to the environments.yaml .. | 18:13 |
niemeyer | SpamapS: Just if [ $? -ne 0 ]; then FAIL fi; after bootstrap? | 18:13 |
SpamapS | I run with set -e | 18:15 |
SpamapS | existing env is not an error in bootstrap | 18:15 |
niemeyer | SpamapS: Oops.. that's certainly a bug | 18:23 |
hazmat | SpamapS, define $HOME in your test script if you want better isolation | 18:39 |
hazmat | SpamapS, yeah.. the return code from the ensemble cli are suspect | 18:39 |
hazmat | i filed a bug a while back regrading | 18:39 |
hazmat | actually jim did.. bug 697093 | 18:40 |
_mup_ | Bug #697093: Ensemble command should return nonzero status code for errors <cli> <Ensemble:New> < https://launchpad.net/bugs/697093 > | 18:40 |
_mup_ | ensemble/rename-shutdown-command r338 committed by jim.baker@canonical.com | 18:50 |
_mup_ | Doc changes | 18:50 |
_mup_ | ensemble/rename-shutdown-command r339 committed by jim.baker@canonical.com | 18:51 |
_mup_ | Add test for destroying a specified default env | 18:51 |
hazmat | i think i will keep the network abstraction, its still pretty useful | 19:24 |
hazmat | even against libvirt default for exposing attributes and starting if its not started | 19:24 |
_mup_ | Bug #838330 was filed: Bootstrap command should error on existing <Ensemble:New> < https://launchpad.net/bugs/838330 > | 19:42 |
hazmat | bcsaller1, the other important parallel piece of work is doing the lxc integration with a serviceunitdeployment subclass | 19:55 |
hazmat | bcsaller1, wiring in the deployment class selection based on machine provider type later | 19:55 |
bcsaller1 | hazmat: I'll look at it but I might not be able to start that part today, we'll see | 19:56 |
hazmat | bcsaller1, fair enough.. i'm pushing through the last of the lxc local provider work, should have all the branches in review for tomorrow, i can start on it tomorrow | 19:56 |
hazmat | its the next step | 19:57 |
hazmat | bcsaller1, lost my irc connection for a moment | 19:58 |
bcsaller1 | hazmat: didn't miss anything | 19:59 |
hazmat | bcsaller1, one think re lxc-lib, i'd like to create a container without running the customize script.. it doesn't seem like that's possible atm | 20:10 |
hazmat | s/think/thing | 20:10 |
hazmat | hmm | 20:11 |
hazmat | i'm trying to prime the deb-cache doing bootstrap so ops like add unit are fast.. but their async anyways.. so maybe that's just a bad assumption | 20:11 |
hazmat | yeah.. nevermind re lxc lib.. it makes more sense to just let the agent deal with the lag on the container creation | 20:12 |
smoser | ok... for ensembel cloud-config, you should probably for your sanity put something like: | 20:13 |
smoser | output: | 20:13 |
hazmat | smoser, output: ? | 20:14 |
smoser | all: "| tee /var/log/cloud-init-output.log" | 20:14 |
hazmat | smoser, i thought cloud-init had log file output builtin now ? | 20:14 |
smoser | yes, it does log to /var/log/cloud-init.log | 20:15 |
smoser | but this will capture output of it | 20:15 |
smoser | and tee it | 20:15 |
hazmat | cool | 20:15 |
smoser | output: | 20:16 |
smoser | all: [ "| tee /var/log/cloud-init-output.log", "&1" ] | 20:16 |
smoser | that would help me debug some | 20:16 |
smoser | i think maybe http://paste.ubuntu.com/679208/ | 20:47 |
smoser | hmm... better with tee -a | 20:48 |
_mup_ | ensemble/local-ubuntu-provider r341 committed by kapil.thangavelu@canonical.com | 21:06 |
_mup_ | local provider skeleton | 21:06 |
smoser | ok. my first ensemble merge proposal! | 21:16 |
smoser | https://code.launchpad.net/~smoser/ensemble/cloud-init-output-log/+merge/73596 | 21:16 |
niemeyer | smoser: Woohay! | 21:16 |
_mup_ | ensemble/local-network r342 committed by kapil.thangavelu@canonical.com | 21:40 |
_mup_ | expose network attributes from libvirt to local provider | 21:40 |
_mup_ | ensemble/local-provider r343 committed by kapil.thangavelu@canonical.com | 21:48 |
_mup_ | local provider that bootstraps and shutsdown | 21:48 |
SpamapS | smoser: hey, I was just reading cloud-init's code and I noticed that you're not passing -y to add-apt-repository | 21:51 |
SpamapS | smoser: I believe this may be causing the issue with getting stuck on console... http://paste.ubuntu.com/679240/ | 21:53 |
_mup_ | ensemble/managed-agent r343 committed by kapil.thangavelu@canonical.com | 22:23 |
_mup_ | managed machine agent | 22:23 |
SpamapS | hrm.. bug 819329 is actually pretty tricky | 22:51 |
_mup_ | Bug #819329: Tests depend on AWS_ACCESS_KEY_ID being set <Ensemble:Confirmed> <ensemble (Ubuntu):Confirmed> < https://launchpad.net/bugs/819329 > | 22:51 |
SpamapS | txaws is raising an error during a constructor because it can't find this environment variable... | 22:51 |
SpamapS | the conundrum is, there is no sane default.. | 22:57 |
SpamapS | So do we set the env var in the tests? Do we shove it into the config during the tests, potentially bypassing code? Hrm. | 22:58 |
hazmat | SpamapS, its a unit test, you can shove any value there | 22:58 |
hazmat | SpamapS, it should never actually use it | 22:58 |
hazmat | SpamapS, it can also be fixed by ensuring its in the environment config of whatever test is using it | 22:58 |
SpamapS | so the real issue is that txaws needs to be mocked? | 23:00 |
SpamapS | since we're not "testing txaws" | 23:01 |
SpamapS | hazmat: if txaws isn't going to be mocked, then we need to test both having it and not having it in the env, so we know when that behavior changes.. | 23:02 |
hazmat | SpamapS, we're using txaws but we don't ever interact externally during the unit tests | 23:03 |
hazmat | any external interaction is typically mocked | 23:03 |
SpamapS | hazmat: so the simple workaround is just to set them to dummy values before running the test suite. :-P | 23:12 |
bcsaller1 | SpamapS: there is a change_environment function on all the test classes that will set env for the duration of the test | 23:13 |
=== bcsaller1 is now known as bcsaller | ||
SpamapS | mmk. So to work around the problem, we can certainly just dump AWS_ACCESS_KEY_ID in .. but shouldn't we also *verify* that if its not there, an error is raised? | 23:14 |
bcsaller | if the provider is ec2 for a given environment it should assert the values it depends on at runtime, yes | 23:16 |
SpamapS | Thats the frustrating/confusing part.. I don't see where ec2 was even asserted. :P | 23:16 |
bcsaller | SpamapS: that test is using write_sample_config which includes a default value of ec2, when it looks for the machine provider it triggers the issue you see. I agree, this is not clear | 23:23 |
SpamapS | hrm.. tests don't seem to run cleanly for me anyway.. :-P | 23:42 |
SpamapS | weird.. just running './test' should work pretty well, right? | 23:46 |
SpamapS | I'm getting *piles* of errors | 23:46 |
hazmat | SpamapS, assuming you've got txzk and zk yes | 23:50 |
SpamapS | if I try to run say, ensemble.environment tests.. I get these fails: | 23:50 |
SpamapS | http://paste.ubuntu.com/679281/ | 23:50 |
SpamapS | Looks like it lost the HOME override | 23:51 |
SpamapS | ok n/m on that.. I misunderstood change_environment | 23:51 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!