[00:46] adam_g: If it gets super custom per node.. then ensemble becomes a bit pointless. [00:48] adam_g: whatever is creating these one-off nodes (I'm assuming this is orchestra not ec2) should just make sure each node has the block device at the specified location.. maybe make a symlink. [00:52] SpamapS: i wouldnt consider the existence of a specific blockdev to be super custom. :) but yeah, im thinking it should be left up to cobbler/orchestra/admin. even if that is the case, though, and we can declare something similar to m1.large with specific storage layout, it would be useful to be able to direct add-unit to a specific machine type/class [00:55] yeah, --placement=available is pretty inadequate for this.. we probably need a better one. [02:27] adam_g: Yeah, I agree with SpamapS.. we need to find a way to scale that.. if the admin has to put info per node, it becomes harder to manage [02:27] adam_g: I understand your point, though.. we can dig a bit deeper to find a way to manage that need without introducing individual entries per node in the config [08:33] Hi, I'm having a hard time on finding information about whether it is possible to override default-image-id somehow per service-unit/formula or something. I.e. I wan't to use small instances for everything but when it comes to db-nodes I'd like'd them to be high-mem instances. My google-fu isn't strong enough :) [08:35] Whiz: there's an open bug for that.. [08:35] SpamapS, ok. so it was my google-fu :D thank you [08:36] Whiz: you can set it in the environments.yaml just before you deploy right now.. and it will be "the image id" for that service as you add units [08:36] there are actually two bugs.. tho they might need to be merged.. [08:37] bug 830995 and bug 808969 [08:37] <_mup_> Bug #830995: default-instance-type and default-image-id are inadequate < https://launchpad.net/bugs/830995 > [08:37] <_mup_> Bug #808969: determine the instance type at deploy < https://launchpad.net/bugs/808969 > [08:39] okay :) [08:39] thank you [11:59] morning folks === daker_ is now known as daker [12:42] hazmat: ping, whenever you're around :) [13:07] lynxman, pong [13:08] * hazmat contemplates an oneiric upgrade [13:14] niemeyer, how's oneiric been treating you? [13:14] hazmat: Haven't been using it yet, but was just pondering yesterday about how it _might_ treat me :) [13:15] niemeyer, i'm considering taking the plunge this morning, although bug 818830 concerns me [13:15] <_mup_> Bug #818830: [Sandy Bridge] serious power regression from kernel 3.0.0-6 to 3.0.0-7 (rc6 disabled) <3.0> < https://launchpad.net/bugs/818830 > [13:16] <_mup_> ensemble/stack-crack r328 committed by kapil.thangavelu@canonical.com [13:16] <_mup_> merge trunk [13:17] hazmat: Looks like there's a trivial way to fix it? [13:17] niemeyer, doesn't work for everyone [13:18] Ah, gotcha [13:18] <_mup_> ensemble/local-ubuntu-provider r330 committed by kapil.thangavelu@canonical.com [13:18] <_mup_> refactor get class path utility function into managed zk method [13:18] * hazmat commits work in progress and pushes pre upgrade [13:20] hazmat: Good choice ;-) [13:22] <_mup_> txzookeeper/errors-with-path r44 committed by kapil.foss@gmail.com [13:22] <_mup_> zk exceptions w/ path information [13:25] niemeyer, apparently i scored in the race to get a touchpad as well :-) [13:25] took a week to get a confirmation email of my order out of hp, which is sad [13:25] hazmat: Neat [13:26] hazmat: I'm curious about these machines [13:27] niemeyer, their dual core qualcomms, 1.2 ghz.. overclockable to 1.5ghz... seem reasonably capable, there's a couple of bounties out to get android on them, wifi only + usb. even if nothing else they should make decent xmas gifts [13:33] hazmat: They are neat Ubuntu machines it seems :) [13:37] niemeyer, mee+go+ubuntu :-) [13:51] niemeyer, update-manager failed [13:52] hazmat: Ugh [13:52] hazmat: Let me know how it goes [13:53] hazmat: Depending on your results I could get "brave" ;-) [13:53] niemeyer, looks like it rolled back properly, i think i might wait till the official beta on sept 1st [13:54] hazmat: Oh, seriously? [13:54] hazmat: That's quite amazing on itself [13:54] hazmat: Well.. depending on where it failed [13:54] niemeyer, it looks like it adjusted packages sources, updated, calculated the upgrade, failed to do so, and restored the sources [13:55] hazmat: Ah, ok [13:55] hazmat: That's less impressive [13:55] niemeyer, do you still use smart for dist-upgrades? [13:56] hazmat: I know mvogt is looking at doing magic for rolling back for quite a while [13:56] hazmat: Like, _real_ rollbacks :) [13:56] hazmat: I never used Smart for dist-upgrades [13:56] hazmat: update-manager is not just apt.. it's full of hand-made fixes [14:17] ugh.. post earthquake.. hurricane.. just got a call from the power company to expect multi-day outages [14:17] hazmat: damn [14:20] http://t.co/2fNBu7M [14:21] heh [14:25] hazmat: Good time for a sprint! [14:26] Daviey: LOL [14:32] fwereade: howdy!! [14:33] heya RoAkSoAx [14:33] how's life? [14:33] fwereade: pretty good, testing ensemble/orchestra on real HW [14:33] RoAkSoAx: sweet [14:33] RoAkSoAx: all that's missing in trunk is the unicode-key thing now [14:34] fwereade: alrgith, great then [14:34] RoAkSoAx: not mped yet though, haven't actually finished it yet [14:34] RoAkSoAx: ...and damn, I need to be off a little early today and might not quite get there [14:34] fwereade: that's fine [14:34] fwereade: at least it is just one more to go [14:35] RoAkSoAx: *might* be able to polish it off this evening but don't count on it :) [14:35] fwereade: no worries [14:35] I'm testing with cobbler-compelte-fixes now [14:35] fwereade: but have a few things that might already sound familiar [14:35] RoAkSoAx: oh ah? [14:36] fwereade: 1. bootstrap doesn't check if there's ssh keys apparently, so when trying to do status it fails due to not having a ssh key in the zk node. 2. which is related to 1, is throwing and exception when it should probably throw just an error message: http://pastebin.ubuntu.com/675290/ 3. When we bootstrap/deploy, we need to check that power management is configured in the system, cause it is bootstrapping/depoying without actually checking [14:37] 1/2: is that just saying that bootstrap fails if you don't have a public key in ~/.ssh? [14:37] fwereade: 3 (cont'd) which means that enmsemble will think it started a machine but since cobbler doesn't have power management configured nothing really happened [14:38] 3: hm, I guess that's set in the system dict somewhere, I can just filter on that I presume? [14:38] hazmat: ping? [14:38] fwereade: 1/2. not exactlyu. it bootstraps correctly, but since there are no keys on ~/.ssh then, we cannot connect to the zk (I think there's a bug report already) [14:39] fwereade: 3. yes, if I have time later today I'll do it myself so don';t worry just yet ;) [14:39] RoAkSoAx: yeah, it rings a bell [14:39] RoAkSoAx: cool [14:39] RoAkSoAx: nothing showstopping then :) [14:39] lynxman, pong [14:40] fwereade: not really, just wanted to make you aware in case you stumbled on them [14:40] hazmat: I have an issue with Ensemble + Openstack [14:40] s/on/upon/ [14:40] lynxman, details? [14:40] hazmat: I can bootstrap machines but it looks that no matter how I try the ssh keys are not copied over, the console output logs don't help either :/ [14:40] fwereade: speaking of which ^^ [14:41] hazmat: tried to a) specify public key with authorized-keys variable in environments.yaml b) create default id_rsa and let Ensemble choose it [14:41] RoAkSoAx: hmm :) [14:41] lynxman, i had that problem as well.. i worked around with specifying a euca key pair [14:41] hazmat: in both cases Ensemble bootstraps successfully but then it complains about incorrect SSH key, even ssh'ing manually says that the key is not valid [14:41] RoAkSoAx: ...or maybe a different issue [14:41] hazmat: cool, any details in how to do so in environments.yaml ? [14:42] fwereade: yeah but related as there was a similar bug report anyways, will test and see what happens ;) [14:42] lynxman, its a new option on that branch, from the paste its under ec2-key-name http://pastebin.ubuntu.com/673083/ [14:42] hazmat: when I tried that ensemble complained at bootstrap that no SSH was found :) [14:42] * lynxman comes full circle [14:42] lynxman, effectively euca-add-keypair and then use the name [14:42] hazmat: it's when you suggested to use authorized-keys [14:43] hazmat: I'll try again [14:43] lynxman, its not authorized keys from your local machine, its the provider managed keys [14:43] hazmat: I know :) [14:43] hazmat: and that's the one I specified [14:43] hmm [14:43] fwereade: nothing to worry from our side though [14:43] hazmat: as a matter of fact copy/pasted the key name from euca-describe-keys [14:44] hazmat: I'll quickly produce a pastebin so you can see the issue [14:45] lynxman, sounds good, just trying to figure out how to debug it [14:47] jcastro: excess flood? wow [14:52] hazmat: it worked perfectly now *shrugs* thanks :) [14:52] lynxman, magic :-) [14:52] lynxman, yeah... i had some transient issues with keys as well, i'm not even entirely sure the ec2-key-name has any purpose [14:52] hazmat: looks like my version of the line before wasn't loved by Ensemble [14:52] hazmat: or yeah, transient issues as you say :) [14:53] lynxman, i think it might be cloud-init related (delay in key installation) but its hard to say without more investigation [14:54] <_mup_> ensemble/go r2 committed by gustavo@niemeyer.net [14:54] <_mup_> Merged go-schema branch into go [r=fwereade,jimbaker] [14:54] <_mup_> Ported schema verification and coercion logic from Python. [15:00] http://goneat.org/lp/ensemble/go/schema [15:06] niemeyer: very neat :) === nijaba_afk is now known as nijaba [15:25] I need to skip out a bit early today -- happy weekends all :) [15:27] fwereade: Cheers man! [15:28] fwereade: see ya my frioend [15:29] fwereade, have a good one [15:29] niemeyer, 503 on http://goneat.org/lp/ensemble/go/schema [15:31] hazmat: Try again [15:31] hazmat: I was updating it so that searching would work too [15:33] hazmat: http://goneat.org/search?q=FieldMap [15:34] niemeyer, looks good [15:36] On that note, I'm stepping out for lunch.. I hope to nail down Formulas this afternoon, given that we have a clean Eureka review queue (!) [15:38] ugh.. that can't be allowed to continue ;-) [15:38] niemeyer, have a good one [15:38] hazmat: Thanks! :-) === daker is now known as daker_ [16:05] $ ./unit2machine.py master-jenkins/0 [16:05] ec2-107-20-64-136.compute-1.amazonaws.com [16:05] :-D [16:06] * SpamapS is in ur tezts, writing new codez [16:08] :-) [16:17] this looks pretty interesting http://wiki.perfkit.org/index.php?title=Main_Page [16:17] a nice gui to profiling tools [16:17] ala osx instruments [16:19] hm.. how does ensemble work on python 2.6 if we use argparse? [16:19] is it just an external module? [16:22] SpamapS, correct [16:23] so using argparse won't cause my code to be 2.7 only? [16:23] there are some small differences in error messages iirc, so that breaks a couple of tests [16:23] SpamapS, that's right [16:24] SpamapS, i was running the ensemble test suite for a while on os x (don't tell anyone), it worked just fine except for a few trivial issues [16:25] SpamapS, it would certainly be useful to have a jenkins slave testing the client code, as packaged by macports [16:27] um, can a formula request to run on oneiric specifically ? [16:27] jimbaker: yeah we should do that. I'm sure we can find an old mac mini to use as a slave. :) [16:27] kim0: not yet no [16:27] okie .. ami-id is env.yaml then for now [16:31] kim0: yeah, thats being discussed in a couple of bugs tho [16:42] SpamapS, python 2.6 and ensemble work, there was one minor incompatibility recently that jim fixed [16:47] We'll find out for sure.. running lots of examples on lucid in my test script.. :) [16:59] SpamapS: I do lots of ensemble cli on lucid VMs... couple of warnings lately, but it's generally worked [17:00] never spun up lucid amis tho [17:10] jamespage: go back to your vacation! ;) [17:24] hazmat: ping [17:24] niemeyer, pong [17:24] hazmat: Can I get a quick review on this simplification: http://paste.ubuntu.com/675388/ [17:26] niemeyer, hmm.. doesn't the existing use of resphaper inline to the formula need to be removed as well if its directly in the schema [17:27] hazmat: How do you mean? [17:27] oh.. nm. its all in the schema [17:27] Right [17:27] hazmat: The only logical change is removing the OneOf [17:27] niemeyer, +1 [17:28] hazmat: Thanks! [17:28] np [17:30] <_mup_> ensemble/trunk r332 committed by gustavo@niemeyer.net [17:30] <_mup_> Minor simplification in metadata schema [r=hazmat] [17:30] <_mup_> The OneOf was redundant since the interface reshaper/expander also [17:30] <_mup_> takes care of a complete interface. [17:30] <_mup_> Also took the chance for improving naming and docs a bit. [17:36] Me and Ale teaching our future kids: http://www.smbc-comics.com/index.php?db=comics&id=2348#comic [17:47] cute [17:51] if not formula_id.count(":") >= 1: [17:51] That's an awesome way to say == 0 :-) [17:58] hmm.. is there a way to add an SSH key to an ensemble environment currently? [18:00] SpamapS, ? there already is one [18:00] Yeah, so say I bootstrapped from my laptop... [18:00] SpamapS, the authorized key specified ( or default ssh key found if not specified) is installed on all the machines on the env [18:00] then I want to hand off maintenance to Bob [18:01] I'm not going to give Bob my key. :) [18:01] Needs to be a way to add/remove keys. [18:01] SpamapS, atm no. we haven't really tackled multi-user management.. better would be to specify a shared key when creating the env, but your right we should probably support changing [18:01] better, but not realistic [18:02] s/should probably/must/ IMO [18:02] indeed [18:02] It shouldn't be too hard really [18:02] ensemble is root.. so it can just twiddle the keys as needed [18:02] SpamapS, its straight forward, its just a question of handling multiple users.. i think we'll might start walking down a rest api road for multi-user support [18:02] I will file a bug. [18:03] sounds good [18:03] hazmat: that will mitigate some of it, but you still need to be able to ssh to the machines. [18:03] true [18:05] <_mup_> Bug #834930 was filed: Need a way to manage SSH keys in an ensemble environment. < https://launchpad.net/bugs/834930 > [18:05] <_mup_> ensemble/go-formulas r3 committed by gustavo@niemeyer.net [18:06] <_mup_> Implemented ParseId. [18:06] has the behavior of ensemble debug-hooks changed? The hooks are being executed ( apparently ) but, I don't see anything happening in the debug-hooks windows [18:07] negronjl: There was a minor change by kirkland to use byobu, but this shouldn't have affected anything [18:07] negronjl: Which hook are you trying to debug? [18:08] niemeyer: mysql-admin-relation-joined/changed on cloudfoundry-server and cf-mysql formulas [18:08] negronjl: Ok, this should definitely be working [18:08] SpamapS, m_3: Any comments? [18:08] niemeyer: I bootstrapped, deployed both formulas, debug-hooks on both and then add-relation.....and I get nothing on the debug-hooks .... it's stuck on "bash" [18:09] negronjl: Yeah, this scenario sounds fine [18:09] btw. I deployed....waited for them to be successfully deployed ( started ), then debug-hooks on them... [18:09] negronjl: Does the relation get succesfully established afterwards? [18:09] negronjl: According to status [18:10] niemyer: according to status yes but, none of the changes get applied. [18:10] negronjl: Hmm [18:10] negronjl: So the hooks do not run either [18:11] ? [18:11] negronjl: ? [18:11] niemeyer: not consistenly. they apparently run on one of the two deployed formulas but not on both of them. [18:11] niemeyer: this doesn't always/nor never happens....it's all erratic. [18:12] negronjl: Very supect.. [18:12] jimbaker: Are you able to investigate this? [18:12] jimbaker [18:15] negronjl, do you have a debug-log output [18:15] I am writing up exactly what I am doing ( down to the environment.yaml file ) so I can pass it on.... [18:15] negronjl, awesome [18:16] negronjl, i can't think of anything directly related to debug-hooks which has changed recently outside of the byobu changes [18:17] hallyn: negronjl: hmm, I'm here, if there's any issue to look at, wrt to the byobu/debug-hooks changes [18:17] hazmat: yeah... I tend to agree plus, I don't see byobu breaking debug-hooks either.... I am just hoping that I am missing something here and a fresh set of eyes on this issue can just clear it all up for me :) [18:44] negronjl: send your config... haven't seen this, but I was on a pretty stale version until yesterday [18:45] m_3: I will... I am also documenting everything ( environment.yaml, steps, debug-log, formula.log, etc. ) so I can give you a complete picture on what's going on and the necessary resources for you to reproduce if you can . [18:49] * SpamapS starts another test run .. 25th time is the charm! [19:01] m_3, are you looking into this with negronjl? just ping me if you need some help from the core perspective [19:01] jimbaker: will do [19:15] m_3, hazmat, kirkland, niemeyer, jimbaker, bcsaller, anyone :) : Here is more or less my issue with as much documentation as I can remember. Please let me know if I missed anything here. [19:15] m_3, hazmat, kirkland, niemeyer, jimbaker, bcsaller, anyone :) : Here is more or less my issue with as much documentation as I can remember. Please let me know if I missed anything here: It would help if I paste it: http://paste.ubuntu.com/675480/ [19:16] negronjl: Sweeet [19:16] negronjl: Thanks! [19:16] Now, who's looking at this? It won't help us if everyone jumps into it :) [19:16] niemeyer: Please let me know if there is anything at all I can do to help solve this :) [19:17] negronjl: Will do, thanks again [19:17] hazmat: Are you interested in looking? [19:17] sure [19:18] * negronjl going to grab a quick byte while you guys look at this.... [19:18] * negronjl will be back in a few minutes. [19:18] negronjl, cool, i should have some more info soon [19:18] hazmat: thx [19:19] hazmat: Thanks! [19:36] Hrm.. I need shutdown --yes-really-I-want-that [19:37] SpamapS: +1 I so want a '-y' option to shutdown [19:37] currently I'm walking through and just torching all services/machines now because it doesn't exist [19:38] python experts.. halp [19:38] http://paste.ubuntu.com/675490/ [19:39] if my eyes don't deceive me, those two things equal [19:39] p type(service) [19:40] [19:40] (Pdb) type(status['services'].keys()[2]) [19:40] [19:40] I thought it might be a __str__ method on the state object [19:40] wordpess [19:40] spelling [19:40] No I'm actually just parsing ensemble status via subprocess and yaml.load() ;) [19:41] wordpess ? hahahahahaha [19:41] thats what I get for copy/pasting [19:41] I can see why they didn't go with that name [19:41] instead of using a durn variable [19:41] well wordpuke was taken [19:42] by a slam poet? [19:42] slampress [19:42] heh [19:42] negronjl, is that cloud foundry formula the latest, on my deploy it fails to start [19:42] although i don't see anything obvious in the error log [19:44] odd it seems like its hanging on config-changed [19:44] but there isn't a config-change hook here [19:45] but it never finishes the transition to start, and stays installed, and resolved doesn't help :-( [19:46] i've got two debug windows up though [19:47] hazmat: heck of a long install [19:47] indeed [19:48] still its a bug that's not in debug-log output.. [19:49] <_mup_> ensemble/go-formulas r4 committed by gustavo@niemeyer.net [19:49] <_mup_> The schema package now consistently outputs values of type schema.M [19:49] <_mup_> for maps and schema.L for lists. [19:51] heh.. well at least my tests legitimately failed for a good reason [19:52] service names can only be like 16 chars long w/ the example formulas [19:52] * negronjl is back [19:52] hazmat: I'll make sure you get the latest one....committing, pushing now [19:53] hazmat: cloudfoundry-server and cf-mysql both updated. [19:54] negronjl, thanks, redeploying [19:54] hazmat: make sure you deploy on oneiric and not natty. I put the ami and type in the environment file there. [19:56] negronjl, indeed i'm using those, i don't see any new revs on cf-mysql.. latest revno is 2 [19:57] so I get joined and two changes firing [19:57] walking through the logic [19:57] hazmat: I pushed revno 3 [19:59] negronjl, to where? https://pastebin.canonical.com/51828/ [19:59] hazmat: cf-mysql is here: bzr+ssh://bazaar.launchpad.net/~negronjl/%2Bjunk/cf-mysql/ [20:00] hazmat: cloudfoundry-server is here: bzr+ssh://bazaar.launchpad.net/~canonical-sig/%2Bjunk/cloudfoundry-server/ [20:00] negronjl, cool [20:08] hmm [20:08] hazmat: hmm...doesn't sound so good :) [20:10] negronjl, so it looks like debug-hooks isn't working, i haven't seen a hook go by yet, ( ic new hooks being executed while sitting in the tmux session), and it seems to active be preventing successful transition recording.. quite odd.. [20:11] * hazmat baselines against trunk [20:11] and the example formulas [20:11] hazmat: ahah!! I finally got someone to see it too ....YEAH!!! I am not crazy :) [20:12] negronjl, indeed, not i'm going to see if this also occurs with the example formulas and take it from there [20:12] s/not/ [20:16] <_mup_> ensemble/go-formulas r5 committed by gustavo@niemeyer.net [20:16] <_mup_> Fixed and tested schema coercing of nil values in several cases. [20:18] I'm running default amis (already had other services running) and saw debug-hooks fire between cf and cf-mysql [20:20] install failed though [20:20] m_3: install fails on natty....works only on oneiric [20:20] (exited cleanly but the service wasn't up) [20:21] <_mup_> ensemble/go-formulas r6 committed by gustavo@niemeyer.net [20:21] <_mup_> Ported formula id parsing and interface schema normalization. [20:25] seems like the easiest way to debug is to setup an ssh into python interpreter on the agents [20:25] * hazmat goes back to trying it from the cli [20:26] hazmat: FWIW. I deploy both formulas ( cloudfoundry-server and cf-mysql ), ensemble ssh to both of them and tail -f formula.log... [20:26] hazmat: then I add-relation etc. At least I get to see more or less what's going on [20:26] <_mup_> Bug #835037 was filed: Go port must handle interface schemas < https://launchpad.net/bugs/835037 > [20:29] negronjl, indeed that's a good thing to do, i'm just focusing on debug-hook functionality and forensics there is really about getting into the runtime state in the process to try and see what's going wrong.. before getting a local failing test case i can bisect against.. i have some suspicions but evidence is better [20:30] hazmat: indeed :) [20:31] <_mup_> ensemble/simplify-iface-schema r333 committed by gustavo@niemeyer.net [20:31] <_mup_> Simplify the interface schema further. [20:32] hazmat++ [20:33] reproduced against example formulas [20:33] <_mup_> Bug #835044 was filed: Interface schema handling is too complex < https://launchpad.net/bugs/835044 > [20:33] hazmat**2 [20:36] niemeyer, so i deployed mysql, and then did an immediate debug-hooks.. install hook is still running, completes and then executes start transition, config-change hook tries to execute.. no new debug-hook window in tmux session.. and it looks like the debug hook watcher is still running.. [20:37] unit agent log.. https://pastebin.canonical.com/51830/ [20:39] hook.pid never shows up in the debug-hook dir [20:40] so creating the new window seems to have the problem [20:40] hazmat: Yeah, looks stuck in config-changed [20:40] hazmat: What's the deal there? [20:40] niemeyer, its not really config-changed, its any hook, config-changed doesn't exist on these [20:41] hazmat: Ok, but I supposed you've hooked in with debug [20:41] niemeyer, the debug hook script that executes something into tmux and waits for it, the launch in tmux portion is failing [20:41] and the script just waits around forever [20:41] hazmat, niemeyer: unrelated question... config-change seems to be called when anything changes on the instances. Is there a way to ( from inside the config-change hook ) figure out which hook changed or some indication of what changed? This way I know more or less what needs to be updated . [20:41] hazmat: Any idea of how's it failing? [20:42] niemeyer, still looking bbiam [20:42] negronjl: Not yet.. we've discussed a few ideas around that [20:42] negronjl: But still have to evolve a bit more [20:42] niemeyer: cool...thx [20:43] negronjl: np [20:45] niemeyer, what's this line do exec > $ENSEMBLE_DEBUG/debug.log >&1 [20:46] its just execing the rest of the script and sending stdout/sterr to the debug.log file? [20:47] hazmat: That's right [20:47] niemeyer, hmm.. executing the script by hand works.. creates the new tmux window [20:48] Hmmm [20:51] hazmat: Is the log empty? [20:51] niemeyer, it is [20:51] Feels like it hasn't run [20:51] i'm adding a bunch of echoes to it, and going to run it again [20:51] niemeyer, its running, its in the process list [20:51] when the hook 'hangs' [20:51] Cool, that's good at least [20:53] niemeyer, found it.. unit-name isn't being exported to debug hook [20:53] hazmat: Woah? [20:53] niemeyer, tmux command doesn't get captured as error (there is a set -e at the top), and then goes onto wait for hook.pid [20:54] tmux fails to start the new window because of the invalid session name [20:54] hazmat: set -e is the opposite.. it forces everything to error out [20:54] ah [20:55] niemeyer, i got it wrong about the env var i think.. i was using another window in the session, and its env variables got exported overwriting the original debug hook ones [20:55] Hmm, ok [20:56] * hazmat goes back to drawing board [20:58] hmm tmux versions went from 1.3 in natty to 1.5 in oneiric [21:01] Nice bump.. I hope they didn't introduce the screen bug on the way [21:01] * niemeyer looks at the release notes [21:01] yeah. was just checking those out [21:06] niemeyer, this interesting comparison of screen 2 tmux.. makes why 2 use tmux pretty clear.. http://www.techrepublic.com/blog/opensource/is-tmux-the-gnu-screen-killer/1901 [21:07] hazmat: Can't see anything obvious at least in the changelog [21:07] i'm going to try backing the byobu work to see [21:08] hazmat: What about the echos? [21:08] niemeyer, yeah.. me neither.. lots of couple of new session starting option [21:08] niemeyer, just executing the debug hook script with echoes inserted to figure out the failure point [21:09] i'm going to back out to a higher level and just try it sans the byobu work [21:10] one more try on the cli b4 that [21:11] hazmat: Note that I've screwed up when merging the byobu work.. [21:11] niemeyer, yeah.. i saw the two commits [21:11] hazmat: Cool [21:14] niemeyer, this is the line that locks up it looks like -> http://www.techrepublic.com/blog/opensource/is-tmux-the-gnu-screen-killer/1901 [21:14] whoops.. copy paste error [21:14] tmux new-session -d -s $ENSEMBLE_UNIT_NAME 2> /dev/null || true [21:15] hazmat: Oh my.. not again :-) [21:15] hazmat: Ok.. what happens if you run two tmux with the same name on Oneiric? [21:15] hazmat: Using that line [21:16] This would also explain why sometimes it happens and sometimes it doesn't, since it depends on who wins the race to get the session created [21:16] niemeyer, root@domU-12-31-39-05-75-E1:/tmp# tmux new-session -d -s wordpress/0 [21:16] duplicate session: wordpress/0 [21:17] hazmat: Phew.. alright, that's good [21:17] hazmat: No locks or anything [21:17] ? [21:17] niemeyer, but there's serveral processes in the process listing that are on that same command [21:17] its definitely locked for some of them [21:17] hazmat: What's the behavior if you use a new name [21:17] ? [21:18] hazmat: That "-d" is from detach.. it should never stay [21:18] hmmm.. i should probably try it outside of tmux ;-) [21:18] hazmat: Yeah, sounds like an idea :) [21:19] hmm. on login via ssh ERROR: [/home/ubuntu/.byobu] is not writable by the current user [21:21] hazmat: That sounds like a hit [21:22] hazmat: Let's try to roll byobu back [21:22] i'm not entirely convinced, but i'm not sure what else to try [21:22] its wierd i can execute the debug hook new-session command from the cli and get the duplicate output, but it hangs if i execute the script on that line [21:23] * hazmat tries the rollback [21:23] hazmat: If you execute in which line? [21:23] niemeyer, the tmux new-session -d -s unit_name 2 line [21:24] hazmat: Sorry, I'm not sure I understand [21:24] hazmat: You say it hangs if you execute the script on that line [21:24] hazmat: How are the working and the non-working executions different/ [21:24] ? [21:24] niemeyer, i can execute "tmux new-session -d -s unit_name" on the cli, works fine.. but the debug-hook hangs on that line [21:25] hazmat: Ok.. are you able to reproduce a hanging tmux somehow? [21:25] hazmat: Or is it just the outcome of the debug-hooks context itself that gets there? [21:26] niemeyer, its a new ssh shell executing it, and it is reproducible.. executing the debug-hook script hangs [21:27] this is with some previous context of the debug-hook session already being active [21:27] hazmat: Ok [21:27] anyways.. going to rollback byobu and see if that helps [21:27] hazmat: If you get rid of ~/.tmux.conf, does it hang? [21:27] niemeyer, checking [21:27] hazmat: "Getting rid" is just removing the tmux.conf.. [21:28] hazmat: If byobu is at fault, the hanging script should be free as in speech [21:28] niemeyer, still hangs [21:28] Ok, that's not it then [21:28] yup [21:28]