/srv/irclogs.ubuntu.com/2011/08/26/#ubuntu-ensemble.txt

SpamapSadam_g: If it gets super custom per node.. then ensemble becomes a bit pointless.00:46
SpamapSadam_g: whatever is creating these one-off nodes (I'm assuming this is orchestra not ec2) should just make sure each node has the block device at the specified location.. maybe make a symlink.00:48
adam_gSpamapS: i wouldnt consider the existence of a specific blockdev to be super custom. :) but yeah, im thinking it should be left up to cobbler/orchestra/admin.  even if that is the case, though, and we can declare something similar to m1.large with specific storage layout, it would be useful to be able to direct add-unit to a specific machine type/class 00:52
SpamapSyeah, --placement=available is pretty inadequate for this.. we probably need a better one.00:55
niemeyeradam_g: Yeah, I agree with SpamapS.. we need to find a way to scale that.. if the admin has to put info per node, it becomes harder to manage02:27
niemeyeradam_g: I understand your point, though.. we can dig a bit deeper to find a way to manage that need without introducing individual entries per node in the config02:27
WhizHi, I'm having a hard time on finding information about whether it is possible to override default-image-id somehow per service-unit/formula or something. I.e. I wan't to use small instances for everything but when it comes to db-nodes I'd like'd them to be high-mem instances. My google-fu isn't strong enough :)08:33
SpamapSWhiz: there's an open bug for that..08:35
WhizSpamapS, ok. so it was my google-fu :D thank you08:35
SpamapSWhiz: you can set it in the environments.yaml just before you deploy right now.. and it will be "the image id" for that service as you add units08:36
SpamapSthere are actually two bugs.. tho they might need to be merged..08:36
SpamapSbug 830995 and bug 80896908:37
_mup_Bug #830995: default-instance-type and default-image-id are inadequate <Ensemble:New> < https://launchpad.net/bugs/830995 >08:37
_mup_Bug #808969: determine the instance type at deploy <Ensemble:Confirmed> < https://launchpad.net/bugs/808969 >08:37
Whizokay :)08:39
Whizthank you08:39
hazmatmorning folks11:59
=== daker_ is now known as daker
lynxmanhazmat: ping, whenever you're around :)12:42
hazmatlynxman, pong13:07
* hazmat contemplates an oneiric upgrade13:08
hazmatniemeyer, how's oneiric been treating you?13:14
niemeyerhazmat: Haven't been using it yet, but was just pondering yesterday about how it _might_ treat me :)13:14
hazmatniemeyer, i'm considering taking the plunge this morning, although bug 818830 concerns me13:15
_mup_Bug #818830: [Sandy Bridge] serious power regression from kernel 3.0.0-6 to 3.0.0-7 (rc6 disabled) <3.0> <kernel> <oneiric> <power> <linux (Ubuntu):Triaged> < https://launchpad.net/bugs/818830 >13:15
_mup_ensemble/stack-crack r328 committed by kapil.thangavelu@canonical.com13:16
_mup_merge trunk13:16
niemeyerhazmat: Looks like there's a trivial way to fix it?13:17
hazmatniemeyer, doesn't work for everyone13:17
niemeyerAh, gotcha13:18
_mup_ensemble/local-ubuntu-provider r330 committed by kapil.thangavelu@canonical.com13:18
_mup_refactor get class path utility function into managed zk method13:18
* hazmat commits work in progress and pushes pre upgrade13:18
niemeyerhazmat: Good choice ;-)13:20
_mup_txzookeeper/errors-with-path r44 committed by kapil.foss@gmail.com13:22
_mup_zk exceptions w/ path information13:22
hazmatniemeyer, apparently i scored in the race to get a touchpad as well :-)13:25
hazmattook a week to get a confirmation email of my order out of hp, which is sad13:25
niemeyerhazmat: Neat13:25
niemeyerhazmat: I'm curious about these machines13:26
hazmatniemeyer, their dual core qualcomms, 1.2 ghz.. overclockable to 1.5ghz... seem reasonably capable, there's a couple of bounties out to get android on them, wifi only + usb. even if nothing else they should make decent xmas gifts13:27
niemeyerhazmat: They are neat Ubuntu machines it seems :)13:33
hazmatniemeyer, mee+go+ubuntu :-)13:37
hazmatniemeyer, update-manager failed13:51
niemeyerhazmat: Ugh13:52
niemeyerhazmat: Let me know how it goes13:52
niemeyerhazmat: Depending on your results I could get "brave" ;-)13:53
hazmatniemeyer, looks like it rolled back properly, i think i might wait till the official beta on sept 1st13:53
niemeyerhazmat: Oh, seriously?13:54
niemeyerhazmat: That's quite amazing on itself13:54
niemeyerhazmat: Well.. depending on where it failed13:54
hazmatniemeyer, it looks like it adjusted packages sources, updated, calculated the upgrade, failed to do so, and restored the sources13:54
niemeyerhazmat: Ah, ok13:55
niemeyerhazmat: That's less impressive13:55
hazmatniemeyer, do you still use smart for dist-upgrades?13:55
niemeyerhazmat: I know mvogt is looking at doing magic for rolling back for quite a while13:56
niemeyerhazmat: Like, _real_ rollbacks :)13:56
niemeyerhazmat: I never used Smart for dist-upgrades13:56
niemeyerhazmat: update-manager is not just apt.. it's full of hand-made fixes13:56
hazmatugh.. post earthquake.. hurricane.. just got a call from the power company to expect multi-day outages14:17
robbiewhazmat: damn14:17
Davieyhttp://t.co/2fNBu7M14:20
robbiewheh14:21
niemeyerhazmat: Good time for a sprint!14:25
niemeyerDaviey: LOL14:26
RoAkSoAxfwereade: howdy!!14:32
fwereadeheya RoAkSoAx14:33
fwereadehow's life?14:33
RoAkSoAxfwereade: pretty good, testing ensemble/orchestra on real HW14:33
fwereadeRoAkSoAx: sweet14:33
fwereadeRoAkSoAx: all that's missing in trunk is the unicode-key thing now14:33
RoAkSoAxfwereade: alrgith, great then14:34
fwereadeRoAkSoAx: not mped yet though, haven't actually finished it yet14:34
fwereadeRoAkSoAx: ...and damn, I need to be off a little early today and might not quite get there14:34
RoAkSoAxfwereade: that's fine14:34
RoAkSoAxfwereade: at least it is just one more to go14:34
fwereadeRoAkSoAx: *might* be able to polish it off this evening but don't count on it :)14:35
RoAkSoAxfwereade: no worries14:35
RoAkSoAxI'm testing with cobbler-compelte-fixes now14:35
RoAkSoAxfwereade: but have a few things that might already sound familiar14:35
fwereadeRoAkSoAx: oh ah?14:35
RoAkSoAxfwereade: 1. bootstrap doesn't check if there's ssh keys apparently, so when trying to do status it fails due to not having a ssh key in the zk node. 2. which is related to 1, is throwing and exception when it should probably throw just an error message: http://pastebin.ubuntu.com/675290/ 3. When we bootstrap/deploy, we need to check that power management is configured in the system, cause it is bootstrapping/depoying without actually checking14:36
fwereade1/2: is that just saying that bootstrap fails if you don't have a public key in ~/.ssh?14:37
RoAkSoAxfwereade: 3 (cont'd) which means that enmsemble will think it started a machine but since cobbler doesn't have power management configured nothing really happened14:37
fwereade3: hm, I guess that's set in the system dict somewhere, I can just filter on that I presume?14:38
lynxmanhazmat: ping?14:38
RoAkSoAxfwereade: 1/2. not exactlyu. it bootstraps correctly, but since there are no keys on ~/.ssh then, we cannot connect to the zk (I think there's a bug report already)14:38
RoAkSoAxfwereade: 3. yes, if I have time later today I'll do it myself so don';t worry just yet ;)14:39
fwereadeRoAkSoAx: yeah, it rings a bell14:39
fwereadeRoAkSoAx: cool14:39
fwereadeRoAkSoAx: nothing showstopping then :)14:39
hazmatlynxman, pong14:39
RoAkSoAxfwereade: not really, just wanted to make you aware in case you stumbled on them14:40
lynxmanhazmat: I have an issue with Ensemble + Openstack14:40
RoAkSoAxs/on/upon/14:40
hazmatlynxman, details?14:40
lynxmanhazmat: I can bootstrap machines but it looks that no matter how I try the ssh keys are not copied over, the console output logs don't help either :/14:40
RoAkSoAxfwereade: speaking of which ^^14:40
lynxmanhazmat: tried to a) specify public key with authorized-keys variable in environments.yaml b) create default id_rsa and let Ensemble choose it14:41
fwereadeRoAkSoAx: hmm :)14:41
hazmatlynxman, i had that problem as well.. i worked around with specifying a euca key pair14:41
lynxmanhazmat: in both cases Ensemble bootstraps successfully but then it complains about incorrect SSH key, even ssh'ing manually says that the key is not valid14:41
fwereadeRoAkSoAx: ...or maybe a different issue14:41
lynxmanhazmat: cool, any details in how to do so in environments.yaml ?14:41
RoAkSoAxfwereade: yeah but related as there was a similar bug report anyways, will test and see what happens ;)14:42
hazmatlynxman, its a new option on that branch,  from the paste its under ec2-key-name http://pastebin.ubuntu.com/673083/14:42
lynxmanhazmat: when I tried that ensemble complained at bootstrap that no SSH was found :)14:42
* lynxman comes full circle14:42
hazmatlynxman, effectively euca-add-keypair and then use the name14:42
lynxmanhazmat: it's when you suggested to use authorized-keys14:42
lynxmanhazmat: I'll try again14:43
hazmatlynxman, its not authorized keys from your local machine, its the provider managed keys14:43
lynxmanhazmat: I know :)14:43
lynxmanhazmat: and that's the one I specified14:43
hazmathmm14:43
RoAkSoAxfwereade: nothing to worry from our side though14:43
lynxmanhazmat: as a matter of fact copy/pasted the key name from euca-describe-keys14:43
lynxmanhazmat: I'll quickly produce a pastebin so you can see the issue14:44
hazmatlynxman, sounds good, just trying to figure out how to debug it14:45
lynxmanjcastro: excess flood? wow14:47
lynxmanhazmat: it worked perfectly now *shrugs* thanks :)14:52
hazmatlynxman, magic :-)14:52
hazmatlynxman, yeah... i had some transient issues with keys as well, i'm not even entirely sure the ec2-key-name has any purpose14:52
lynxmanhazmat: looks like my version of the line before wasn't loved by Ensemble14:52
lynxmanhazmat: or yeah, transient issues as you say :)14:52
hazmatlynxman, i think it might be cloud-init related (delay in key installation) but its hard to say without more investigation14:53
_mup_ensemble/go r2 committed by gustavo@niemeyer.net14:54
_mup_Merged go-schema branch into go [r=fwereade,jimbaker]14:54
_mup_Ported schema verification and coercion logic from Python.14:54
niemeyerhttp://goneat.org/lp/ensemble/go/schema15:00
fwereadeniemeyer: very neat :)15:06
=== nijaba_afk is now known as nijaba
fwereadeI need to skip out a bit early today -- happy weekends all :)15:25
niemeyerfwereade: Cheers man!15:27
RoAkSoAxfwereade: see ya my frioend15:28
hazmatfwereade, have a good one15:29
hazmatniemeyer, 503 on http://goneat.org/lp/ensemble/go/schema15:29
niemeyerhazmat: Try again15:31
niemeyerhazmat: I was updating it so that searching would work too15:31
niemeyerhazmat: http://goneat.org/search?q=FieldMap15:33
hazmatniemeyer, looks good15:34
niemeyerOn that note, I'm stepping out for lunch.. I hope to nail down Formulas this afternoon, given that we have a clean Eureka review queue (!)15:36
hazmatugh.. that can't be allowed to continue ;-)15:38
hazmatniemeyer, have a good one15:38
niemeyerhazmat: Thanks! :-)15:38
=== daker is now known as daker_
SpamapS$ ./unit2machine.py master-jenkins/016:05
SpamapSec2-107-20-64-136.compute-1.amazonaws.com16:05
SpamapS:-D16:05
* SpamapS is in ur tezts, writing new codez16:06
hazmat:-)16:08
hazmatthis looks pretty interesting http://wiki.perfkit.org/index.php?title=Main_Page16:17
hazmata nice gui to profiling tools16:17
hazmatala osx instruments16:17
SpamapShm.. how does ensemble work on python 2.6 if we use argparse?16:19
SpamapSis it just an external module?16:19
jimbakerSpamapS, correct16:22
SpamapSso using argparse won't cause my code to be 2.7 only?16:23
jimbakerthere are some small differences in error messages iirc, so that breaks a couple of tests16:23
jimbakerSpamapS, that's right16:23
jimbakerSpamapS, i was running the ensemble test suite for a while on os x (don't tell anyone), it worked just fine except for a few trivial issues16:24
jimbakerSpamapS, it would certainly be useful to have a jenkins slave testing the client code, as packaged by macports16:25
kim0um, can a formula request to run on oneiric specifically ?16:27
SpamapSjimbaker: yeah we should do that. I'm sure we can find an old mac mini to use as a slave. :)16:27
SpamapSkim0: not yet no16:27
kim0okie .. ami-id is env.yaml then for now16:27
SpamapSkim0: yeah, thats being discussed in a couple of bugs tho16:31
hazmatSpamapS, python 2.6 and ensemble work, there was one minor incompatibility recently that jim fixed16:42
SpamapSWe'll find out for sure.. running lots of examples on lucid in my test script.. :)16:47
m_3SpamapS: I do lots of ensemble cli on lucid VMs... couple of warnings lately, but it's generally worked16:59
m_3never spun up lucid amis tho17:00
SpamapSjamespage: go back to your vacation! ;)17:10
niemeyerhazmat: ping17:24
hazmatniemeyer, pong17:24
niemeyerhazmat: Can I get a quick review on this simplification: http://paste.ubuntu.com/675388/17:24
hazmatniemeyer, hmm.. doesn't the existing use of resphaper inline to the formula need to be removed as well if its directly in the schema17:26
niemeyerhazmat: How do you mean?17:27
hazmatoh.. nm. its all in the schema17:27
niemeyerRight17:27
niemeyerhazmat: The only logical  change is removing the OneOf17:27
hazmatniemeyer, +117:27
niemeyerhazmat: Thanks!17:28
hazmatnp17:28
_mup_ensemble/trunk r332 committed by gustavo@niemeyer.net17:30
_mup_Minor simplification in metadata schema [r=hazmat]17:30
_mup_The OneOf was redundant since the interface reshaper/expander also17:30
_mup_takes care of a complete interface.17:30
_mup_Also took the chance for improving naming and docs a bit.17:30
niemeyerMe and Ale teaching our future kids: http://www.smbc-comics.com/index.php?db=comics&id=2348#comic17:36
hazmatcute17:47
niemeyerif not formula_id.count(":") >= 1:17:51
niemeyerThat's an awesome way to say == 0 :-)17:51
SpamapShmm.. is there a way to add an SSH key to an ensemble environment currently?17:58
hazmatSpamapS, ? there already is one18:00
SpamapSYeah, so say I bootstrapped from my laptop...18:00
hazmatSpamapS, the authorized key specified ( or default ssh key found if not specified) is installed on all the machines on the env18:00
SpamapSthen I want to hand off maintenance to Bob18:00
SpamapSI'm not going to give Bob my key. :)18:01
SpamapSNeeds to be a way to add/remove keys.18:01
hazmatSpamapS, atm no. we haven't really tackled multi-user management.. better would be to specify a shared key when creating the env, but your right we should probably support changing18:01
SpamapSbetter, but not realistic18:01
SpamapSs/should probably/must/ IMO18:02
hazmatindeed18:02
SpamapSIt shouldn't be too hard really18:02
SpamapSensemble is root.. so it can just twiddle the keys as needed18:02
hazmatSpamapS, its straight forward, its just a question of handling multiple users.. i think we'll might start walking down a rest api road for multi-user support18:02
SpamapSI will file a bug.18:02
hazmatsounds good18:03
SpamapShazmat: that will mitigate some of it, but you still need to be able to ssh to the machines.18:03
hazmattrue18:03
_mup_Bug #834930 was filed: Need a way to manage SSH keys in an ensemble environment. <Ensemble:New> < https://launchpad.net/bugs/834930 >18:05
_mup_ensemble/go-formulas r3 committed by gustavo@niemeyer.net18:05
_mup_Implemented ParseId.18:06
negronjlhas the behavior of ensemble debug-hooks <blah> changed?  The hooks are being executed ( apparently ) but, I don't see anything happening in the debug-hooks windows 18:06
niemeyernegronjl: There was a minor change by kirkland to use byobu, but this shouldn't have affected anything18:07
niemeyernegronjl: Which hook are you trying to debug?18:07
negronjlniemeyer: mysql-admin-relation-joined/changed on cloudfoundry-server and cf-mysql formulas18:08
niemeyernegronjl: Ok, this should definitely be working18:08
niemeyerSpamapS, m_3: Any comments?18:08
negronjlniemeyer:  I bootstrapped, deployed both formulas, debug-hooks on both and then add-relation.....and I get nothing on the debug-hooks .... it's stuck on "bash"18:08
niemeyernegronjl: Yeah, this scenario sounds fine18:09
negronjlbtw. I deployed....waited for them to be successfully deployed ( started ), then debug-hooks on them...18:09
niemeyernegronjl: Does the relation get succesfully established afterwards?18:09
niemeyernegronjl: According to status18:09
negronjlniemyer:  according to status yes but, none of the changes get applied.18:10
niemeyernegronjl: Hmm18:10
niemeyernegronjl: So the hooks do not run either18:10
niemeyer?18:11
niemeyernegronjl: ?18:11
negronjlniemeyer:  not consistenly.  they apparently run on one of the two deployed formulas but not on both of them.18:11
negronjlniemeyer:  this doesn't always/nor never happens....it's all erratic.18:11
niemeyernegronjl: Very supect..18:12
niemeyerjimbaker: Are you able to investigate this?18:12
niemeyerjimbaker18:12
hazmatnegronjl, do you have a debug-log output18:15
negronjlI am writing up exactly what I am doing ( down to the environment.yaml file ) so I can pass it on....18:15
hazmatnegronjl, awesome18:15
hazmatnegronjl, i can't think of anything directly related to debug-hooks which has changed recently outside of the byobu changes18:16
kirklandhallyn: negronjl: hmm, I'm here, if there's any issue to look at, wrt to the byobu/debug-hooks changes18:17
negronjlhazmat:  yeah... I tend to agree plus, I don't see byobu breaking debug-hooks either.... I am just hoping that I am missing something here and a fresh set of eyes on this issue can just clear it all up for me :)18:17
m_3negronjl: send your config... haven't seen this, but I was on a pretty stale version until yesterday18:44
negronjlm_3:  I will... I am also documenting everything ( environment.yaml, steps, debug-log, formula.log, etc. ) so I can give you a complete picture on what's going on and the necessary resources for you to reproduce if you can .18:45
* SpamapS starts another test run .. 25th time is the charm!18:49
jimbakerm_3, are you looking into this with negronjl? just ping me if you need some help from the core perspective19:01
m_3jimbaker: will do19:01
negronjlm_3, hazmat, kirkland, niemeyer, jimbaker, bcsaller, anyone :) :  Here is more or less my issue with as much documentation as I can remember.  Please let me know if I missed anything here.19:15
negronjlm_3, hazmat, kirkland, niemeyer, jimbaker, bcsaller, anyone :) :  Here is more or less my issue with as much documentation as I can remember.  Please let me know if I missed anything here:  It would help if I paste it: http://paste.ubuntu.com/675480/19:15
niemeyernegronjl: Sweeet19:16
niemeyernegronjl: Thanks!19:16
niemeyerNow, who's looking at this?  It won't help us if everyone jumps into it :)19:16
negronjlniemeyer:  Please let me know if there is anything at all I can do to help solve this :)19:16
niemeyernegronjl: Will do, thanks again19:17
niemeyerhazmat: Are you interested in looking?19:17
hazmatsure19:17
* negronjl going to grab a quick byte while you guys look at this....19:18
* negronjl will be back in a few minutes.19:18
hazmatnegronjl, cool, i should  have some more info soon19:18
negronjlhazmat: thx19:18
niemeyerhazmat: Thanks!19:19
SpamapSHrm.. I need shutdown --yes-really-I-want-that19:36
m_3SpamapS: +1  I so want a '-y' option to shutdown19:37
SpamapScurrently I'm walking through and just torching all services/machines now because it doesn't exist19:37
SpamapSpython experts.. halp19:38
SpamapShttp://paste.ubuntu.com/675490/19:38
SpamapSif my eyes don't deceive me, those two things equal19:39
bcsallerp type(service)19:39
SpamapS<type 'str'>19:40
SpamapS(Pdb) type(status['services'].keys()[2])19:40
SpamapS<type 'str'>19:40
bcsallerI thought it might be a __str__ method on the state object19:40
bcsallerwordpess19:40
bcsallerspelling19:40
SpamapSNo I'm actually just parsing ensemble status via subprocess and yaml.load() ;)19:40
SpamapSwordpess ? hahahahahaha19:41
SpamapSthats what I get for copy/pasting19:41
bcsallerI can see why they didn't go with that name19:41
SpamapSinstead of using a durn variable19:41
SpamapSwell wordpuke was taken19:41
bcsallerby a slam poet?19:42
m_3slampress19:42
bcsallerheh19:42
hazmatnegronjl, is that cloud foundry formula the latest, on my deploy it fails to start19:42
hazmatalthough i don't see anything obvious in the error log19:42
hazmatodd it seems like its hanging on config-changed19:44
hazmatbut there isn't a config-change hook here19:44
hazmatbut it never finishes the transition to start, and stays installed, and resolved doesn't help :-(19:45
hazmati've got two debug windows up though19:46
m_3hazmat: heck of a long install19:47
hazmatindeed19:47
hazmatstill its a bug that's not in debug-log output..19:48
_mup_ensemble/go-formulas r4 committed by gustavo@niemeyer.net19:49
_mup_The schema package now consistently outputs values of type schema.M19:49
_mup_for maps and schema.L for lists.19:49
SpamapSheh.. well at least my tests legitimately failed for a good reason19:51
SpamapSservice names can only be like 16 chars long w/ the example formulas19:52
* negronjl is back19:52
negronjlhazmat:  I'll make sure you get the latest one....committing, pushing now19:52
negronjlhazmat: cloudfoundry-server and cf-mysql both updated.19:53
hazmatnegronjl, thanks, redeploying19:54
negronjlhazmat:  make sure you deploy on oneiric and not natty.  I put the ami and type in the environment file there.19:54
hazmatnegronjl, indeed i'm using those, i don't see any new revs on cf-mysql.. latest revno is 219:56
m_3so I get joined and two changes firing19:57
m_3walking through the logic19:57
negronjlhazmat:  I pushed revno 319:57
hazmatnegronjl, to where? https://pastebin.canonical.com/51828/19:59
negronjlhazmat: cf-mysql is here: bzr+ssh://bazaar.launchpad.net/~negronjl/%2Bjunk/cf-mysql/19:59
negronjlhazmat: cloudfoundry-server is here: bzr+ssh://bazaar.launchpad.net/~canonical-sig/%2Bjunk/cloudfoundry-server/20:00
hazmatnegronjl, cool20:00
hazmathmm20:08
negronjlhazmat: hmm...doesn't sound so good :)20:08
hazmatnegronjl, so it looks like debug-hooks isn't working, i haven't seen a hook go by yet, ( ic new hooks being executed while sitting in the tmux session), and it seems to active be preventing successful transition recording.. quite odd.. 20:10
* hazmat baselines against trunk20:11
hazmatand the example formulas20:11
negronjlhazmat:  ahah!!  I finally got someone to see it too ....YEAH!!!  I am not crazy :)20:11
hazmatnegronjl, indeed, not i'm going to see if this also occurs with the example formulas and take it from there20:12
hazmats/not/20:12
_mup_ensemble/go-formulas r5 committed by gustavo@niemeyer.net20:16
_mup_Fixed and tested schema coercing of nil values in several cases.20:16
m_3I'm running default amis (already had other services running) and saw debug-hooks fire between cf and cf-mysql20:18
m_3install failed though20:20
negronjlm_3:  install fails on natty....works only on oneiric20:20
m_3(exited cleanly but the service wasn't up)20:20
_mup_ensemble/go-formulas r6 committed by gustavo@niemeyer.net20:21
_mup_Ported formula id parsing and interface schema normalization.20:21
hazmatseems like the easiest way to debug is to setup an ssh into python interpreter on the agents20:25
* hazmat goes back to trying it from the cli20:25
negronjlhazmat: FWIW.  I deploy both formulas ( cloudfoundry-server and cf-mysql ), ensemble ssh to both of them and tail -f formula.log... 20:26
negronjlhazmat: then I add-relation etc.  At least I get to see more or less what's going on 20:26
_mup_Bug #835037 was filed: Go port must handle interface schemas <Ensemble:New> < https://launchpad.net/bugs/835037 >20:26
hazmatnegronjl, indeed that's a good thing to do, i'm just focusing on debug-hook functionality and forensics there is really about getting into the runtime state in the process to try and see what's going wrong.. before getting a local failing test case i can bisect against.. i have some suspicions but evidence is better20:29
negronjlhazmat: indeed :)20:30
_mup_ensemble/simplify-iface-schema r333 committed by gustavo@niemeyer.net20:31
_mup_Simplify the interface schema further.20:31
niemeyerhazmat++20:32
hazmatreproduced against example formulas20:33
_mup_Bug #835044 was filed: Interface schema handling is too complex <Ensemble:In Progress by niemeyer> < https://launchpad.net/bugs/835044 >20:33
niemeyerhazmat**220:33
hazmatniemeyer, so i deployed mysql, and then did an immediate debug-hooks.. install hook is still running, completes and then executes start transition, config-change hook tries to execute.. no new debug-hook window in tmux session.. and it looks like the debug hook watcher is still running..20:36
hazmatunit agent log.. https://pastebin.canonical.com/51830/20:37
hazmathook.pid never shows up in the debug-hook dir20:39
hazmatso creating the new window seems to have the problem20:40
niemeyerhazmat: Yeah, looks stuck in config-changed20:40
niemeyerhazmat: What's the deal there?20:40
hazmatniemeyer, its not really config-changed, its any hook, config-changed doesn't exist on these20:40
niemeyerhazmat: Ok, but I supposed you've hooked in with debug20:41
hazmatniemeyer, the debug hook script that executes something into tmux and waits for it, the launch in tmux portion is failing20:41
hazmatand the script just waits around forever20:41
negronjlhazmat, niemeyer:  unrelated question... config-change seems to be called when anything changes on the instances.   Is there a way to ( from inside the config-change hook ) figure out which hook changed or some indication of what changed?  This way I know more or less what needs to be updated .20:41
niemeyerhazmat: Any idea of how's it failing?20:41
hazmatniemeyer, still looking bbiam20:42
niemeyernegronjl: Not yet.. we've discussed a few ideas around that20:42
niemeyernegronjl: But still have to evolve a bit more20:42
negronjlniemeyer: cool...thx20:42
niemeyernegronjl: np20:43
hazmatniemeyer, what's this line do exec > $ENSEMBLE_DEBUG/debug.log >&120:45
hazmatits just execing the rest of the script and sending stdout/sterr to the debug.log file?20:46
niemeyerhazmat: That's right20:47
hazmatniemeyer, hmm.. executing the script by hand works.. creates the new tmux window20:47
niemeyerHmmm20:48
niemeyerhazmat: Is the log empty?20:51
hazmatniemeyer, it is20:51
niemeyerFeels like it hasn't run20:51
hazmati'm adding a bunch of echoes to it, and going to run it again20:51
hazmatniemeyer, its running, its in the process list20:51
hazmatwhen the hook 'hangs'20:51
niemeyerCool, that's good at least20:51
hazmatniemeyer, found it.. unit-name isn't being exported to debug hook20:53
niemeyerhazmat: Woah?20:53
hazmatniemeyer, tmux command doesn't get captured as error (there is a set -e at the top), and then goes onto wait for hook.pid20:53
hazmattmux fails to start the new window because of the invalid session name20:54
niemeyerhazmat: set -e is the opposite.. it forces everything to error out20:54
hazmatah20:54
hazmatniemeyer, i got it wrong about the env var i think.. i was using another window in the session, and its env variables got exported overwriting the original debug hook ones20:55
niemeyerHmm, ok20:55
* hazmat goes back to drawing board20:56
hazmathmm tmux versions went from 1.3 in natty to 1.5 in oneiric20:58
niemeyerNice bump.. I hope they didn't introduce the screen bug on the way21:01
* niemeyer looks at the release notes21:01
hazmatyeah. was just checking those out21:01
hazmatniemeyer, this interesting comparison of screen 2 tmux.. makes why 2 use tmux pretty clear.. http://www.techrepublic.com/blog/opensource/is-tmux-the-gnu-screen-killer/190121:06
niemeyerhazmat: Can't see anything obvious at least in the changelog21:07
hazmati'm going to try backing the byobu work to see21:07
niemeyerhazmat: What about the echos?21:08
hazmatniemeyer, yeah.. me neither.. lots of couple of new session starting option21:08
hazmatniemeyer, just executing the debug hook script with echoes inserted to figure out the failure point21:08
hazmati'm going to back out to a higher level and just try it sans the byobu work21:09
hazmatone more try on the cli b4 that21:10
niemeyerhazmat: Note that I've screwed up when merging the byobu work..21:11
hazmatniemeyer, yeah.. i saw the two commits21:11
niemeyerhazmat: Cool21:11
hazmatniemeyer, this is the line that locks up it looks like -> http://www.techrepublic.com/blog/opensource/is-tmux-the-gnu-screen-killer/190121:14
hazmatwhoops.. copy paste error21:14
hazmattmux new-session -d -s $ENSEMBLE_UNIT_NAME 2> /dev/null || true21:14
niemeyerhazmat: Oh my.. not again :-)21:15
niemeyerhazmat: Ok.. what happens if you run two tmux with the same name on Oneiric?21:15
niemeyerhazmat: Using that line21:15
niemeyerThis would also explain why sometimes it happens and sometimes it doesn't, since it depends on who wins the race to get the session created21:16
hazmatniemeyer, root@domU-12-31-39-05-75-E1:/tmp# tmux new-session -d -s wordpress/0 21:16
hazmatduplicate session: wordpress/021:16
niemeyerhazmat: Phew.. alright, that's good21:17
niemeyerhazmat: No locks or anything21:17
niemeyer?21:17
hazmatniemeyer,  but there's serveral processes in the process listing that are on that same command21:17
hazmatits definitely locked for some of them21:17
niemeyerhazmat: What's the behavior if you use a new name21:17
niemeyer?21:17
niemeyerhazmat: That "-d" is from detach.. it should never stay21:18
hazmathmmm.. i should probably try it outside of tmux ;-)21:18
niemeyerhazmat: Yeah, sounds like an idea :)21:18
hazmathmm. on login via ssh ERROR: [/home/ubuntu/.byobu] is not writable by the current user21:19
niemeyerhazmat: That sounds like a hit21:21
niemeyerhazmat: Let's try to roll byobu back21:22
hazmati'm not entirely convinced, but i'm not sure what else to try21:22
hazmatits wierd i can execute the debug hook new-session command from the cli and get the duplicate output, but it hangs if i execute the script on that line21:22
* hazmat tries the rollback21:23
niemeyerhazmat: If you execute in which line?21:23
hazmatniemeyer, the tmux new-session -d -s unit_name 2   line21:23
niemeyerhazmat: Sorry, I'm not sure I understand21:24
niemeyerhazmat: You say it hangs if you execute the script on that line21:24
niemeyerhazmat: How are the working and the non-working executions different/21:24
niemeyer?21:24
hazmatniemeyer, i can execute "tmux new-session -d -s unit_name" on the cli, works fine.. but the debug-hook hangs on that line21:24
niemeyerhazmat: Ok.. are you able to reproduce a hanging tmux somehow?21:25
niemeyerhazmat: Or is it just the outcome of the debug-hooks context itself that gets there?21:25
hazmatniemeyer, its a new ssh shell executing it, and it is reproducible.. executing the debug-hook script hangs21:26
hazmatthis is with some previous context of the debug-hook session already being active21:27
niemeyerhazmat: Ok21:27
hazmatanyways.. going to rollback byobu and see if that helps21:27
niemeyerhazmat: If you get rid of ~/.tmux.conf, does it hang?21:27
hazmatniemeyer, checking21:27
niemeyerhazmat: "Getting rid" is just removing the tmux.conf..21:27
niemeyerhazmat: If byobu is at fault, the hanging script should be free as in speech21:28
hazmatniemeyer, still hangs21:28
niemeyerOk, that's not it then21:28
hazmatyup21:28
niemeyerhazmat: Can you strace that?21:28
hazmatsuer21:29
niemeyerhazmat: strace f21:29
niemeyerhazmat: strace -f21:29
hazmatsitting in epoll21:29
niemeyerhazmat: On what fds?21:31
hazmatniemeyer, hold on grabbing a trace output for pastebin21:31
hazmatniemeyer, https://pastebin.canonical.com/51834/21:33
niemeyerhazmat: Super21:33
niemeyerhazmat: 23695 connect(7, {sa_family=AF_FILE, path="/tmp/tmux-0/default"}, 21) = 021:35
niemeyerhazmat: That's what it's waiting on21:35
hazmatsomethings wonkey.. i started killing off the new-session processes that are hanging now i'm a tmux refresh loop21:37
niemeyerhazmat: I'm connecting some dots here21:38
negronjlhazmat, niemeyer: fwiw....I think byobu is installed and enabled by default on oneiric.. not sure how it could affect what's going on but, worth mentioning21:39
hazmatniemeyer, so the byobu session was still active actually in the existing session after the conf file21:39
niemeyerhazmat: Hmm21:39
niemeyerhazmat: Makes sense21:39
niemeyernegronjl: Cool.. we still have to source the file I think, but from hazmat's comment we were actually still with it21:40
niemeyerhazmat: I'm starting to suspect this may actually be a new internal race too21:40
niemeyerhazmat: Worth testing without byobu21:40
hazmatniemeyer, in progress21:40
niemeyerhazmat: If it happens again, try to see what are the initial tmux processes that are active 21:42
niemeyerhazmat: It looks like a lock is being left acquired21:42
niemeyerhazmat: I have a workaround in mind, but we should probably try to understand a bit better what's going on before that 21:45
niemeyerhazmat: I read it wrong.. it's not waiting for a lock.. it's waiting for data21:48
niemeyerMakes it yet more strange21:49
hazmathmm. ah i see what negronjl was saying.. oneiric defaults you into a byobu screen shell21:55
niemeyerhazmat: How do you mean?21:58
hazmatssh ec2-instance.. -> byobu shell21:58
hazmatec2 is being slow today..21:59
hazmatat instance creation for me21:59
_mup_ensemble/fix-functional-testing r332 committed by jim.baker@canonical.com22:00
_mup_Fixes for a variety of bitrot in ftests22:00
niemeyerhazmat: Wow, really?22:00
niemeyerhazmat: So we get straight into a screen?22:00
hazmatniemeyer, yeah.22:01
hazmaton debug-hooks into a new machine "To launch in a nested session, run: byobu" is at the top22:01
niemeyerhazmat: Oh my22:01
niemeyerhazmat: Well, wait.. but that's because we're within a tmux session, right?22:02
hazmatit is in that case22:02
niemeyerhazmat: What does CTRL-A-C do?22:02
hazmatnew window22:02
niemeyerhazmat: Does it create something into the internal tmux, or in an outside screen?22:03
hazmatits in tmux22:03
hazmati think22:03
niemeyerhazmat: Ok, so there must be no outside screen22:03
niemeyerhazmat: Or tmux22:03
niemeyerhazmat: Isn't that simply reflecting the fact we're reading the byobu conf?22:03
hazmatniemeyer, we're using the old conf, not sourcing byobu22:04
hazmatfor tmux22:04
niemeyerhazmat: So where is byobu being loaded in? I don't get it..22:05
niemeyerkirkland: ping22:05
negronjlniemeyer:  byobu is being loaded from $HOME/.profile22:05
hazmathmm. there both tmux sessions22:05
niemeyernegronjl: Aha22:05
niemeyerOk22:05
niemeyerhazmat: Hm?22:05
hazmatoh.. i'm on different machines nevermind22:06
hazmatokay debug-hooks goes into tmux with the proper config22:06
hazmatand still hangs22:07
hazmathttps://pastebin.canonical.com/51835/22:07
niemeyerWOah22:08
niemeyerSO, how come22:09
niemeyerhazmat: Is that the initial state?22:10
hazmatniemeyer, that's debug-hooks into a new machine, service unit finishes install, and tries to execute config-changed22:11
hazmatotherwise clean22:11
niemeyerhazmat: I don't understand how there are two pts's there22:11
niemeyerhazmat: Ah, I get it I think22:16
niemeyerhazmat: What does tmux list-sessions show?22:17
hazmatmysql/0: 2 windows (created Fri Aug 26 22:01:05 2011) [168x36] (attached)22:17
niemeyerhazmat: The pstree has one tmux22:20
niemeyerhazmat: The ps auxw has three..22:20
niemeyerhazmat: Sorry, my mistake22:21
hazmatnegronjl, looks like byobu is activated by  /etc/profile.d/Z98-byobu.sh 22:21
niemeyerhazmat: Seriously, there are two sessions somehow :/22:21
kirklandhazmat: right, that's the package-enabled way22:21
negronjlhazmat:  you're right....sorry for the misinformation.  I checked MY box and not the oneiric instance.22:21
kirklandhazmat: there's also a per-user method22:21
niemeyerhazmat: Can you get a list of files that each of those tmux new-session processes are using?22:22
niemeyerhazmat: lsof -p 163422:23
niemeyerhazmat: lsof -p 366722:23
niemeyerhazmat?22:29
hazmatniemeyer, 1634 -> https://pastebin.canonical.com/51837/ 3667 -> https://pastebin.canonical.com/51836/   tmux-attach / 1636 -> https://pastebin.canonical.com/51838/22:29
niemeyerhazmat: Is there more than one /tmp/tmux-* dir?22:29
hazmatwhat's that command to install someones launchpad key on a machine?22:30
hazmatniemeyer, one tmux dir 22:31
hazmatssh-import-lp-id22:32
hazmatniemeyer, ubuntu@ec2-174-129-118-179.compute-1.amazonaws.com22:33
niemeyerhazmat: Cheers!22:35
niemeyerkirkland: This is starting to feel a bit too intrusive22:40
niemeyerubuntu@ip-10-46-54-97:/tmp$ sudo su -22:40
niemeyerTo launch in a nested session, run: byobu22:40
niemeyerroot@ip-10-46-54-97:~#22:40
kirklandniemeyer: right, do you want a nested session, or not?22:41
niemeyerkirkland: I want to run sudo22:41
kirklandniemeyer: then run sudo22:41
niemeyerkirkland: I did.. and the system tells me completely unrelated facts all the time22:41
kirklandniemeyer: what are "unrelated facts"22:42
niemeyerkirkland: I've just pasted above22:42
SpamapS/usr/lib/pymodules/python2.6/ensemble/providers/ec2/files.py:8: DeprecationWarning: the sha module is deprecated; use the hashlib module instead22:43
SpamapSIs there a way to suppress that?22:43
kirklandniemeyer: the fact you're speaking of being, "To launch in a nested session, run: byobu"22:43
kirkland?22:43
niemeyerkirkland: Getting output from "sudo su -" every time telling me to run something else is a bit obnoxious22:43
niemeyerSpamapS: Ugh.. yeah, we should follow the advice :)22:44
niemeyerhazmat: Was that a clean run?22:44
niemeyerhazmat: There are 4 different tmp directories in /tmp22:44
niemeyerhazmat: With hook data, that is22:44
hazmatniemeyer, outside of me killing processes yes ;-)22:44
niemeyerOh man :)22:44
hazmatniemeyer, its pretty clean though, those are the original processes competing22:45
SpamapSa good, clean kill22:45
kirklandniemeyer: that's a one liner to comment out or remove, file a bug and i'll fix it22:45
kirkland./usr/bin/byobu-launcher:                       printf "$(gettext 'To launch in a nested session, run: byobu')\n"22:45
kirklandniemeyer: i put that there at the request of users, however22:45
niemeyerkirkland: Probably of users of byobu.. I'm concerned about people that are used to using sudo22:46
kirklandniemeyer: does it do that if you do "sudo true" ?22:47
kirklandniemeyer: in my experience, "no, it does not";  it does that when it establishes a new login shell, a la 'su -'22:47
niemeyerkirkland: No, that doesn't load the env22:47
kirklandniemeyer: right22:47
kirklandniemeyer: fine.  if you file a bug at https://bugs.launchpad.net/ubuntu/+source/byobu/+filebug, I'll silence that now22:48
niemeyerkirkland: Cheers!22:48
kirklandniemeyer: cheers22:48
kirklandniemeyer: paste me the bug number and i'll commit and push22:48
niemeyerkirkland: #83513022:50
_mup_Bug #835130: byobu is too verbose <byobu (Ubuntu):New> < https://launchpad.net/bugs/835130 >22:50
kirklandniemeyer: Committed revision 1622.                                                                                                                     22:51
kirklandniemeyer: thanks for playing ;-)22:51
niemeyerkirkland: Interestingly, users seem to have asked the feature because it was even more disruptive to the workflow22:51
niemeyerkirkland: #74764922:51
_mup_Bug #747649: opening a nested session from byobu to byobu should not ask, but rather should remind the user... <byobu (Ubuntu):Fix Released by kirkland> < https://launchpad.net/bugs/747649 >22:51
niemeyerkirkland: This kind of tool should really not kick in before the user decides to use, IMO22:52
kirklandniemeyer: it's silently exiting now, as of this commit22:55
niemeyerkirkland: Have you seen this one:22:57
niemeyerERROR: [/home/ubuntu/.byobu] is not writable by the current user22:58
niemeyerubuntu@ip-10-46-54-97:~$ 22:58
kirklandniemeyer: yes22:58
niemeyerkirkland: Supa22:58
kirklandniemeyer: this happens if you sudo byobu22:58
kirklandniemeyer: or run byobu while under sudo22:59
kirklandniemeyer: the byobu directory gets created under the root user22:59
kirklandniemeyer: rather than the $SUDO_USER22:59
niemeyerkirkland: Yeah, I understand why it happened22:59
niemeyerNow I just have to understand tmux23:00
kirklandniemeyer: let me see if I can come up with a suitable fix to that right quick ....23:01
niemeyerhazmat: I have little doubts that this is a race introduced in tmux23:04
niemeyerhazmat: Oh ho ho23:05
niemeyerhazmat: 23:06
niemeyer# tmux new-session -d -s mysql/0 < /dev/null 2> /dev/null23:06
niemeyer^C^\Quit (core dumped)23:06
* SpamapS wonders if core is feeling vulnerable and lonely after being dumped..23:07
hazmatniemeyer, hmm23:08
hazmatso what are options..23:08
hazmatSpamapS, probably just lazing around getting fat ;-)23:08
niemeyerhazmat: 1) Fixing the bug in tmux; 2) Changing the logic so it doesn't hit the bug23:09
hazmatniemeyer, the race seems odd, there are seconds between them23:09
niemeyerhazmat: It's not even a race23:09
niemeyerhazmat: Plain bug :(23:09
hazmatniemeyer, back to screen :-)23:10
kirklandSpamapS: probably relieved, stomachs feeling much better23:10
niemeyerhazmat: :-)23:10
niemeyerhazmat: Let's use dash instead23:10
niemeyerkirkland: You're in oneiric right?23:11
kirklandniemeyer: yup23:11
niemeyerkirkland: Would you mind to help us with a quick test23:11
kirklandniemeyer: i've got a fix for your ownerships issue;  file another bug and i'll commit23:11
kirklandniemeyer: sure, whaddaya need?23:12
niemeyerkirkland: tmux new-session -d -s foo23:12
niemeyerkirkland: Run that a couple of times in different non-nested terminals23:12
kirklandNicke: k23:12
kirklandkirkland@x201:~$ tmux new-session -d -s foo23:12
kirklandkirkland@x201:~$ tmux new-session -d -s foo23:12
kirklandduplicate session: foo23:12
niemeyerkirkland: Should blow up the second time with duplicated session error23:12
niemeyerCool23:12
niemeyerkirkland: Now, one more23:12
niemeyerkirkland: tmux new-session -d -s foo 2> /dev/null23:12
kirklandkirkland@x201:~$ tmux new-session -d -s foo 2> /dev/null23:13
kirkland......23:13
kirkland<hangs>23:13
niemeyerMan.. I'm getting out of this software business23:13
niemeyerkirkland: Thanks!23:13
kirklandniemeyer: np23:13
kirklandniemeyer: checking my $SUDO_USER fix with the security team23:15
niemeyerkirkland: Thanks man23:15
hazmatniemeyer, http://sourceforge.net/mailarchive/forum.php?thread_name=20110826161855.GC13865%40smoon&forum_name=tmux-users23:15
niemeyerhazmat: WOAH!23:17
niemeyerhazmat: Sadly, the workaround there doesn't work23:18
kirklandniemeyer: security team says: <kees> run sudo with -H :)23:18
niemeyerkirkland: Heh23:18
kirklandniemeyer: i'm negotiating with him ;-)23:19
niemeyerhazmat: Let's make sure we don't do anything absurd with tmux like redirecting its output to /dev/null23:21
niemeyerhazmat: tmux new-session -d -s mysql/0 2>&1 | cat > /dev/null23:23
niemeyer!!!23:23
niemeyernegronjl: So.. there's a bug in tmux that we're hitting.  We should have a solution in a moment23:24
negronjlniemeyer:  no worries.  I've been monitoring this :)23:25
kirklandNicke: do you have a bug for the $SUDO_USER issue?23:39
kirklandniemeyer: ^23:39
niemeyerkirkland: Nope, haven't created one for that23:53
SpamapShrm.. debug-log seems to not be as useful as it once was23:53
kirklandniemeyer: 83515223:53
kirklandniemeyer: i created it for you23:53
niemeyerkirkland: Thanks!23:54
hazmatniemeyer, yeah.. works for me23:55
niemeyerhazmat, kirkland, negronjl: http://paste.ubuntu.com/67563123:59
hazmatniemeyer, http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=60944423:59
hazmatniemeyer, that looks fine minus all the formula/* stuff23:59
kirkland+tmux new-session -d -s {unit_name} 2>&1 | cat > /dev/null || true23:59
kirklandwowzer23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!