[00:07] SpamapS: that makes alot of sense [00:08] zirpu: i've always used whatever the current 15inch MBP is and been very happy the last 4 or so years [00:11] hm. i'm not a mac fan. but i have had mbp's at various companies. [00:12] i still hate case-insensitive filesystem with a flaming irrational passion of a sloth. [00:12] weird, why doesn't hp cloud recommend using python-novaclient ? :-P [00:12] * SpamapS reluctantly installs fog [00:12] MiGhTilY i say! [00:12] SpamapS: you can [00:12] SpamapS: i use it [00:13] imbrandon: it wants a username [00:13] SpamapS: its in the docs to setup either, but they are ruby fans [00:13] SpamapS: yea one sec its like ur email:tenneid [00:13] or something [00:13] one sec [00:14] export OS_TENANT_NAME="clint.byrum@canonical.com-tenant1" [00:14] that? [00:14] SpamapS: it appeared the admin node was checking for all those nodes. Is there not a hard requirement for at least 10 nodes? And can you point me to reference for that hack please? [00:14] no [00:14] burnbrighter: there's no reference to that hack [00:14] burnbrighter: because it is.. the suck ;) [00:14] burnbrighter: but for testing w/ < 10 boxes... [00:14] gotcha [00:15] burnbrighter: basically you edit ~/.juju/environments.yaml and add 'placement: local' to the environment .. [00:15] burnbrighter: but, this sends *EVERYTHING* to node 0 [00:15] interesting [00:15] burnbrighter: and it will install things in parallel, which will break things (apt-get can't run in parallel for instance) [00:15] burnbrighter: so you kind of have to set it, install anything you want on node 0 one at a time, and then un-set it. [00:16] anyway, I'm late.. gotta run [00:16] but, how would you do, say, just 5 nodes for example? [00:16] anyways, thnx [00:16] burnbrighter: you can squeeze almost everything onto that one node [00:18] ok, thanks [00:18] I'm running this all on esxi / vsphere server, so adding nodes and cloning them wasn't a problem anyways :) [00:46] anyone around to support MAAS with juju? [02:15] m_3: ping [02:15] m_3: i was just thinking ( not looked at the code yet so you might already do this ) but the juju charm [02:16] would it mayeb be a good idea to have it install by "hand" into a dediacated python virtualenv [02:16] so it cant mess up the "host" juju or mess with the versions etc if the [02:17] charms are frozen etc [02:17] e.g. a seperation of the juju charm installation and the juju on the system already if its a subordiante of like an existing webserver/appserver services grp [02:18] just a thought ... i'll peek into the code and see if u are already doing that or not [02:46] you could create a virtual env from a pip freeze and a cache dir easily w/o network access being needed. [02:46] i actually like that idea. [02:47] imbrandon: yeah that's planned [02:47] rockin , just crossed my mind ( btw i do have it as a sub on my webservers so i can call "juju" from hooks :) [02:47] imbrandon: maybe even use that to provide the tmux session you leave up and drive the env with [02:47] shhhh [02:48] yea good idea [02:48] imbrandon: that's sort of turning into a best practice for groups to manage an env.... two environments... and [02:48] but please think of us screen users too, tmux normally gets apt-get purged from me :) [02:49] easier than talking each member of the team through setup on frozen client branch [02:49] yea good call [02:49] imbrandon: ha! [02:49] yeah, was a screen die-hard for years... but I've totally drunk the cool-aide [02:49] man i cant get it to work like screen [02:49] (helps to have the same bindings) [02:49] really? [02:49] if i could i probably would, but there is a few things that i cant make work [02:49] i just took my bindings with me. [02:50] not the bindings [02:50] more about the window setup [02:50] i'm all about full screen. :-) [02:50] and making the shared screens work "right" [02:50] multiuser acts pretty differently, but imo tmux works better for that [02:50] well mine are full but i keep 5 or so windows open [02:50] i haven't used shared screens (yet). [02:51] i.e., all users looking at same screen as I switch around [02:51] m_3: yea thats what i cant get to work [02:51] on screen works perfect [02:51] tmux wont do it [02:51] there's a way to get it to behave like screen tho [02:51] alias tmux=screen [02:51] no no i dont think you get it [02:51] so it's said. :-) [02:51] where each user canbe looking at diff screens [02:51] my screen does like you say is easy in tmux [02:52] but i cant even get tmux to do it [02:52] i WANT that in tmuc [02:52] so client screens should switch when the master screen switches? [02:52] no [02:52] zirpu: that's tmux default, but opposite of screen default [02:52] i flip arround to whatever i want [02:52] and so can anyoone else [02:52] ah [02:53] or we can all goto screen 0 [02:53] and share [02:53] imbrandon: right... I'd have to dig for that... it's possible [02:53] yea if you can help me figure that out i'd be a happy tmux user [02:54] hm. so attach -r puts client in read only mode. [02:54] like i have byobu set 4 windows on login, and if you and me both ssh in we can see each other type if on the same window but i can flip to others and so can you [02:54] but it can't switch screens. weird. [02:54] mysetup^^ [02:54] read-only mode is kind of useless. [02:54] nah [02:54] great for classroom [02:55] well, ok. there's that. [02:55] or logs reads [02:55] for a dedicated log window [02:56] * zirpu does not know how all these kids live in their gui worlds. :-) [02:56] yeah, I use one-way shared screens more than I ever thought I would [02:56] but yea m_3 eveyone keep saying that WHY they swiched to tmux cuz that was hard in screen but to me thats default how screen works not hard and tmux i cant make do it [02:57] so readonly switches screens w/ the master screen. i see what you meant now. [02:57] and i like multi bars at the bottom of screens in byobu but i think i can make tmux do that too [02:57] imbrandon: http://paste.ubuntu.com/1063589/ is my conf... I'll get back to you on users watching separate windows in one session [02:57] m_3: rockin [02:57] ty [02:58] totally does the bars too [02:58] btw, whats the format for .byobu/windows.tmux ? i would need to convert my .byobu/windows file ( screen based ) [02:59] for the default windows on startup [03:00] way different scripting mechanism I think... there're a couple of different schools on that... I use tmuxinator [03:00] m_3: http://paste.ubuntu.com/1063593/ thats my .byobu/windows file on my OS X box [03:00] 'mux blog' or 'mux charmtester'.... can start with 'mux copy juju mynewcharm; mux mynewcharm' [03:01] tmuxinator is a gem so I assume it's good on osx [03:01] likely , i use a TON of ruby on OSX [03:01] as much as most ubuntu users use python :) [03:01] heh [03:02] trying to bring some of it with my to juju/ubuntu , still havent got a good juju-hook capfile balance yet [03:02] but i'll get there [03:02] s/my/me [03:03] imbrandon: equiv is http://paste.ubuntu.com/1063597/ [03:03] i keep wanting to get at cloud-init tho, juju should really provide a way for us to do that, i dont care that it got shot down on the ML [03:03] that'd be 'mux play' [03:04] its that or i just use hax and write a wrapper in jitsu [03:04] heh [03:04] * m_3 off in tweakage-land :) [03:04] hahaha yea i tweek my setups CONSTANTLY [03:04] like i'm never done [03:04] ever [03:04] jitsu is all things unloved and evil... but working :) [03:05] so anything goes [03:05] yea , perfect home for it [03:05] if i have to resort to it [03:05] i'd rather not tho cuz it will likely take a patch to juju to accomplish cleanly [03:05] and that makes me mad to have to resort yo [03:05] ah, right [03:05] to* [03:06] but i certainly will to show the utility of such [03:06] :) [03:06] crap, sorry forgot to reset my away mgs... hope you're not getting to much noise back [03:06] nope [03:06] well i dunno actually [03:06] cool [03:06] i have msgs like that off [03:06] hehe [03:07] joins parts quits aways etc etc [03:07] all prety much off [03:07] right [03:07] too many "dead" channels i idle in that would fill my disk with logs [03:07] if i dident :) [03:08] m_3: see my newest widget ? http://bholtsclaw.github.com/showdown/ [03:09] think i'm gonna make a charm out of the backend stuff i need to host those , so i can have them all central and easy to put all on one page etc etc [03:09] cool [03:10] cuz its like a mix of nodejs and mustash templates and php ( on my.phpcloud.com ) git and github pages [03:10] you should add other keys in there so we can have multiple peeps manage it [03:10] heh in otherwords i got em all spread out [03:10] and freeze the juju branch [03:10] hmmm yea [03:10] good idea [03:10] great idea actuallu [03:11] etc etc all the production best practices we reommend.... (nother test case) [03:11] * imbrandon puts that on the very short-term todo's [03:11] yea , seriously , good call [03:11] perhaps even use lp:charms/juju in the "control" environment... where it gets left open with a tmux serssion any of the team can attach/drive [03:12] (the latters still a maybe... not nec recommended practice yet) [03:12] just really needed it for scale testing [03:12] ok i need a little food and mt dew then to get started on that, i think that will be a good project tonight ... ( i dident wake up till 6pm local so i'll be up all night heh ) [03:12] and sort of need it after the fact for plumber's summit [03:12] ahh right [03:12] cool later man [03:13] yea i'll be back in ~20ish min, just need a lil snack, but i'll br round all night [03:13] and gonna work on just that, got me kinda fired up about the idea [03:13] ;) [03:13] cool.. I'll be in and out [03:13] kk [03:15] yea i like the idea of a tmux session with charms/juju installed on the bootstrap node , then the "team" can just ssh into the bootsrap and drive from a juju window there [03:15] and not need anything but their ssh key on that bootstrap [03:15] no local juju or env.y etc [03:15] ... [03:16] imbrandon: I'm thinking separate envs... [03:16] yea ok fooood then that is gonna become a reality with the widgets and button as a testcase [03:16] summit and summit-control [03:16] ahhh [03:16] yea [03:16] Anyone know about troubleshooting why the mysql charm instance didn't come up in juju/maas? I can reach the node via regular ssh, but agent shows the node is down. [03:16] yea i liky that too [03:16] both would have 'authorized-keys:' in the env with everybody's keys [03:17] right [03:17] this is for openstack-dashboard btw [03:17] burnbrighter: hrm nothing in juju-debug log? [03:17] err juju debug-log [03:17] burnbrighter: not really no... I've seen machines with services running great but the juju aagents report it down [03:17] nothing extraordinary I can see [03:17] rebooting (those ec2 instances) worked to reset the agents [03:18] this is maas [03:18] right [03:18] I've only seen that prob in ec2.... and pretty rare [03:18] restart the agent ? ( or reboot the node ) [03:18] bbiab [03:18] how does one restart the agent (me=noob) [03:18] yeah, first try taking a look in the provisioning agent's log to see that the service was started up correctly [03:18] I did restart [03:19] machines: [03:19] 1: [03:19] agent-state: not-started [03:19] dns-name: maas07 [03:19] instance-id: /MAAS/api/1.0/nodes/node-4ddba634-bfdb-11e1-8909-005056a44c88/ [03:19] instance-state: unknown [03:19] services: [03:19] mysql: [03:19] charm: cs:precise/mysql-2 [03:19] relations: [03:19] shared-db: [03:19] - glance [03:19] - keystone [03:19] - nova-cloud-controller [03:19] - nova-compute [03:19] - nova-volume [03:19] units: [03:19] mysql/1: [03:19] agent-state: pending [03:19] machine: 1 [03:19] public-address: null [03:20] burnbrighter: did it ever say 'started'? [03:20] no [03:20] looking in provision log now.. [03:20] hmmmm... yeah, dig for the maas version of the provisioning-agent log [03:21] 2012-06-27 23:01:35,045:901(0x7f45fbcc2700):ZOO_INFO@zookeeper_init@727: Initiating client connection, host=localhost:2181 sessionTimeout=10000 watcher=0x7f45f8fce6b0 sessionId=0 sessionPasswd= context=0x3352b20 flags=0 [03:21] and then the machine-agent log on that box [03:21] 2012-06-27 23:01:35,045:901(0x7f45f74ca700):ZOO_ERROR@handle_socket_error_msg@1579: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client [03:21] 2012-06-27 23:01:38,381:901(0x7f45f74ca700):ZOO_INFO@check_events@1585: initiated connection to server [127.0.0.1:2181] [03:21] 2012-06-27 23:01:38,403:901(0x7f45f74ca700):ZOO_INFO@check_events@1632: session establishment complete on server [127.0.0.1:2181], sessionId=0x1383109683e0001, negotiated timeout=10000 [03:21] 2012-06-27 23:11:55,399: juju.agents.provision@INFO: Stopping provisioning agent [03:21] then unit-agent there too [03:21] refused...hmmm [03:22] 2012-06-27 23:13:31,767:886(0x7fa12a2f8700):ZOO_INFO@zookeeper_init@727: Initiating client connection, host=localhost:2181 sessionTimeout=10000 watcher=0x7fa1271b56b0 sessionId=0 sessionPasswd= context=0x1db9090 flags=0 [03:22] 2012-06-27 23:13:31,768:886(0x7fa125b00700):ZOO_ERROR@handle_socket_error_msg@1579: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client [03:22] 2012-06-27 23:13:35,105:886(0x7fa125b00700):ZOO_INFO@check_events@1585: initiated connection to server [127.0.0.1:2181] [03:22] 2012-06-27 23:13:35,133:886(0x7fa125b00700):ZOO_INFO@check_events@1632: session establishment complete on server [127.0.0.1:2181], sessionId=0x138311458a10000, negotiated timeout=10000 [03:22] 2012-06-27 23:13:35,216: juju.agents.machine@INFO: Machine agent started id:0 [03:22] zk's down maybe? [03:22] how can I check please? [03:22] ps awux | grep zoo [03:23] on that node? [03:23] netstat -lnp | less [03:23] on the maas server [03:23] ubuntu@maas07:/var/log/juju$ ps awux | grep zoo [03:23] 107 925 0.2 0.8 1835488 33088 ? Ssl 23:13 0:01 /usr/bin/java -cp /etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/zookeeper.jar -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,ROLLINGFILE org.apache.zookee [03:23] erver.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg [03:23] ubuntu 1205 0.0 0.0 9376 932 pts/0 S+ 23:22 0:00 grep --color=auto zoo [03:23] it must be up, right? because other agents are running fine [03:23] burnbrighter: this is where I'm out of my depth wrt maas.... sorry, I'm not sure exactly [03:24] I'll assume that the maas server is set up as the bootstrap node and runs the provistioning agent and zookeeper [03:24] m_3: thanks anyways. this is tricky [03:24] (note the provisioning agent is attempting to connect to localohost:zk [03:25] eh, localhost on its own zookeeper? [03:25] and getting refused for some reason [03:25] um yeah, I see that - connection to localhost 2181 [03:26] right... in std juju, the provisioning agent and zk live together on the bootstrap node [03:26] provisioning agent should be able to subscribe to zk changes and spin up machines accordingly afaik [03:26] I think in maas, and I may be wrong - the bootstrap node becomes the first node bootstrapped [03:27] ah, yeah, that's the question... does the maas server do this or does it actually still use some _other_ bootstrap node per environment [03:28] I know it pulls the ephemeral image from the primary maas node, but where does it get that info? [03:28] from the first bootstrapped node, or the primary maas node? [03:28] ugh [03:29] um... dunno [03:29] I don't know if a maas env is treated as a single juju environment [03:30] lemme dig a bit [03:30] are you on the canonical team? [03:30] this is known stuff though... might try #ubuntu-server or #juju again tomorrow [03:30] yeah, I'm on the server team, but haven't worked with maas yet [03:30] bigjools may know too - but I think he draws the line when it gets deeper in to the juju side [03:31] I think there's a #maas here too... looking [03:31] yeah - that's my primary channel [03:31] ah, see you're already there :) [03:32] but now that I've got the maas side stable, I've been focusing on trying to get the openstack stuff working [03:32] burnbrighter: yeah... that's probably our most complex stack to date on juju [03:33] and the only one I REALLY want working :) [03:33] lol [03:33] the 'not-started' sounds like a lower-level problem than the charm though [03:33] this has been a 3 week endeavor for me [03:33] wow [03:34] both cheez0r and I have both been trying to get it working [03:34] adam_g: you up still? [03:34] burnbrighter: they've got this going regularly in a hw lab [03:35] but I'm wondering if there was special tweaking for the mysql stuff [03:35] or if there are open issues [03:35] dunno [03:35] yeah [03:36] admittedly, I'm not up on the juju stuff yet enough to be dangerous with troubleshooting [03:36] try to bring it up to a 'started' state before adding any relations [03:36] how do you bring up individual nodes in started state? [03:36] I know I've heard of ordering deps in openstack charm relations [03:36] * imbrandon returns [03:37] ju do the deploys [03:37] no add-relation calls until you see them in a 'started' state [03:37] m_3: i thought about that at one time, me and marcoceppi brefly talked about it at UDS etc [03:37] but i decided that hiabu was a better way [03:37] ( pending rename ) [03:38] (there's a jitsu tool to script all of this... 'apt-get install juju-jitsu; jitsu watch -h'... eventually... but for now just do it in your scripts) [03:38] ok, so destroy the individual services, but don't add the relation calls? [03:38] eh, re-add after destroying then wait till they individually come up? [03:38] then add relations? [03:38] burnbrighter: yeah, so at this point mysql hasn't been "started" [03:39] so you should be able to remove-relation's [03:39] then destroy that service [03:39] then terminate that machine [03:39] then deploy the service [03:39] now, wait until it's 'started' [03:39] so first try to remove relations, destroy mysql service - terminate machine is new for me [03:39] then go in and 'add-relation' [03:40] I'll try that - remove all mysql related relations I take it? [03:40] burnbrighter: so I would totally recomend restarteing from scratch... but you don't _really_ have to do that since the service has never been in a started state [03:40] * imbrandon just thought about something from the backlog ... is tmux in ruby ? if so i'd drop screen a moment and deal with "issue" just for that alone [03:40] so the relation hooks on either side have never really fired [03:41] imbrandon: dunno [03:41] does destroying the service basically start things from scratch too? [03:42] burnbrighter: destroying the service _and_ terminating that machine will start that service from scratch [03:42] ie. without destroying the rm -rf "destroy-environment" [03:42] ok [03:42] burnbrighter: but won't require you to bring your whole stack down [03:42] right [03:42] cool, I can try that [03:42] thank you [03:43] I'd recommend that just for sanity's sake as the 'reboot' option... but that's more of a pain with some providers than others [03:43] and if I thought any relation hooks got fired, I'd say... destroy-environment [03:43] but it doesn't look like that here [03:44] you can tell by sshing to that machine and 'mysql -uroot; show databases' or something like that [03:44] should be empty [03:44] yeah, I didn't even see mysql coming up on that node, so that will be a good test [03:44] yup [03:45] not sure exactly how to terminate-machine in maas... that's equiv to a re-install [03:45] yeah - I learned this the hard way - you have to terminate the juju stack before trying to do anything on the maas side [03:46] 'juju terminate-machine ' on most providers totally kill the instance... so you're starting with a freshubuntu server next time [03:46] your maas nodes end up pulling the ephemeral image, but the maas nodes are based on the precise image, I believe [03:47] * m_3 _really_ needs to learn maas... gotta bump that up higher in the queue [03:47] its so very cool. Its just raw [03:48] a little baking and it will make a really nice pie [03:48] cool [03:49] ok, let me try your suggestions [03:49] ok, I'm on UTC+10 this week so I'll be in and out for several hours still [04:04] still not starting... [04:05] http://pastebin.ubuntu.com/1063654/ [04:05] burnbrighter: wow.... hmmmm [04:06] any of that look interesting? [04:06] not really... that's during an expose [04:06] I'm not exposing, just deploying [04:06] expose is a separate step... much like 'add-relation' [04:06] right [04:07] I didn't expose, I just added the service [04:07] just ran "juju deploy mysql [04:07] " [04:07] it's marked as exposed somehow... but it's not surprising that that's barfing if the machine's not coming up [04:08] ok, 'juju deploy mysql' is perfect [04:08] here is exactly what I did: 1. remove relations: [04:08] so now what did you do to throw that machine away? [04:08] juju remove-relation keystone mysql [04:08] juju remove-relation nova-cloud-controller mysql [04:08] juju remove-relation nova-volume mysql [04:08] juju remove-relation nova-compute mysql [04:08] juju remove-relation glance mysql [04:08] then I removed the service ie. juju destroy-service mysql [04:09] then do a status to see the service sitting all alone (no relations) [04:09] did that, yes [04:09] then destroy-service... sounds good [04:09] now for terminate-machine [04:09] then I terminated the node it was running on [04:09] then I went straight in to deploy - [04:09] does that just put the existing server into WOL sleep? [04:10] or does it _rebuild_ it? [04:10] well, there are VMs ;) [04:10] s/there/they are/ [04:10] oops, sorry not following [04:10] ok, so you're deploying maas on a bunch of vms [04:10] yes [04:10] which to this point and not without a lot of headaches was working :) [04:11] ok, so when that machine is terminated, does it destroy that instance _and_ image? [04:11] the headaches were all procedural though [04:11] or does it put it to sleep or just stop it? [04:11] it appears to remove the instance [04:11] from juju [04:11] let me check if the machine is actually down [04:11] it is not [04:12] ah, but it really needs to kill the instance and image and reprovision a new machine [04:12] so my guess is if I "add" it back it will be back in the line up [04:12] but why do I need to do that? [04:12] burnbrighter: so I think that whatever is causing this to show up in a not-started state is written into the image [04:13] it wouldn't be in a pristine copy of the image [04:13] but it's in that actual one [04:13] so, check this - when I asked for mysql to be redeployed - it went to a whole different node - my node 12 and got the same results [04:13] (total guess... but that's what I'm trying to determine by doing a 'terminate-machine') [04:13] oh wow [04:13] ok, so scratch that.... [04:14] mysql: [04:14] charm: cs:precise/mysql-2 [04:14] relations: {} [04:14] units: [04:14] mysql/2: [04:14] agent-state: pending [04:14] machine: 12 [04:14] public-address: null [04:14] 12: [04:14] agent-state: not-started [04:14]