/srv/irclogs.ubuntu.com/2014/04/07/#cloud-init.txt

=== zz_gondoi is now known as gondoi
=== gondoi is now known as zz_gondoi
yann2Hello! I am trying to use cloud-init with FOG and rackspace, but am failing miserably. It seems that my  mime encoded cloudinit script is properly copied to /var/lib/cloud/seed/nocloud-net/user-data  , but it doesnt appear to be run... not sure where I should look to see whats wrong... any idea? the rackspace examples I found on the web didnt work09:55
=== zz_gondoi is now known as gondoi
=== gondoi is now known as zz_gondoi
=== zz_gondoi is now known as gondoi
smoserSpamapS, yeah, i have creds to that private key.14:23
smoserbut it is separate by design from my private key.14:23
smoseri'll sign it and upload14:23
jcliftyann2: Does the script work when it's not mime encoded?14:26
jcliftyann2: Also, there's a log file in /var/log/cloud-init.log14:27
yann2jclift, yes, actually I got it to run using nova client 14:27
jcliftk, so it sounds like the mime encoding is busting it somehow14:28
jcliftyann2: I've not tried using mime encoded version14:28
jcliftyann2: At a guess, you'll probably need to look through /var/log/cloud-init.log to see if there's anything there14:28
yann2trying this now http://pastealacon.com/34262  - I think the problem is that rackspace only tries to get cloud init files from config disks by default and not from /var14:29
yann2jclift, the only error seem to be unrelated and linked to a "ubuntu" user that is not found.. the rest seems to be rackspace stuf14:30
yann22014-04-07 14:27:13,937 - __init__.py[WARNING]: Unhandled non-multipart userdata '   < maybe that14:31
jcliftyann2: Hmmm, I don't use Ubuntu so no idea there14:31
jcliftYeah, that warning could be significant14:31
jcliftyann2: Are you able to put your proper #cloud-config file on a webserver somewhere, and then point to it using an #include?14:32
jcliftyann2: This is what I use as a #cloud-config file: https://forge.gluster.org/glusterfs-rackspace-regression-tester/glusterfs-rackspace-regression-tester/blobs/raw/master/remote_centos6.cfg14:32
yann2There is also this file in /etc/cloud/cloud.cfg.d/90_dpkg.cfg that contains this datasource_list: [ ConfigDrive ]14:32
yann2so I assume my only way to include additional cloud init config would be via the configdrive...14:33
jcliftyann2: I pass that remote_centos6.cfg in to the local VM via config drive, and then cloud-init goes and retrieves the external URLs itself14:33
jcliftyann2: Is the mime-encoded script showing up in /var/lib/cloud/instance/user_data ?14:34
jcliftOr /var/lib/cloud/instance/user_data.i ?14:34
yann2jclift, cf my last paste, I am now trying to see if I can pass the cloud init file directly to the config file, a bit like the nova client does here  http://developer.rackspace.com/blog/using-cloud-init-with-rackspace-cloud.html14:35
jcliftyann2: I'm using cloud-init with Pyrax in rackspace14:37
yann2but otherwise, yeah, when i used to copy files using personality, the file showed up mime encoded at the right place ('/var/lib/cloud/seed/nocloud-net/user-data' ) but I think cloud init is not configured to read it14:37
yann2jclift, and where do you put the cloud init configuration?14:38
jcliftUnfortunately, it's not documented (so I should write up some docs for it to help the next guy)14:38
jcliftyann2: Are you ok with Python?14:39
yann2I can read, yes, but I need ruby14:39
jcliftk.  This might be helpful for reading then.  It's kind of spagetti code atm though... I need to clean it up.  But it works. https://forge.gluster.org/glusterfs-rackspace-regression-tester/glusterfs-rackspace-regression-tester/blobs/master/create_servers.py14:40
jcliftyann2: Line 179 is where the new server instance is created14:40
jcliftyann2: One of the parameters to the call which creates the new server is "userdata".14:41
yann2thanks a lot for your help, I ll read this :)14:41
jcliftyann2: I provide the #cloud-config file to that, which passes it automatically to cloud-init using config drive14:41
jclift(note that I have config_drive=True as another parameter)14:41
jcliftyann2: The #cloud-config file I use, is that "remote-centos6.cfg" thing14:42
jcliftSo, when cloud-init starts up, it gets given that remote-centos6.cfg file.  It reads that, noticing it starts with #include, then goes and gets the URLs on the next two lines.14:42
jcliftThose URL contain what actually needs to be done14:43
jcliftThe first is a proper #cloud-config thing14:43
jcliftThe seconds is a script to execute after the #cloud-config thing has fully run14:43
jclifts/seconds/second/14:43
yann2so you use config_drive as well14:44
yann2and userdata = ci_config... should be very similar to what I m doig...14:44
jcliftyann2: As a thought, when I was getting my head around cloud-init for the first time last week, I spun up the various VM's remotely (like you), then logged into them to try and figure out wtf was happening14:45
jcliftyann2: Yeah, it sounds similar14:45
yann2so ci_config is not mime encoded, nor base64 encoded14:45
jcliftCorrect. Have you logged into a VM, and mounted /dev/xvde to a temp spot to see what's being passed via config_drive?14:46
jcliftyann2: The userdata= argument here can actually be just a newly opened file handle.  You don't even need to read in the contents.14:46
yann2nope, good idea14:46
jcliftyann2: Yeah, mount /dev/xvde to a temp spot, and take a look through it's structure14:47
jcliftyann2: Helps to get a better conceptual understanding of wtf is going on :)14:47
yann2indeed.. its xvdd here, where would my userdata be?14:47
jcliftAhh yeah. xvdd is right14:48
jcliftxvde is the data drive14:48
jcliftSorry14:48
jcliftBeen using xvde a lot over the weekend :)14:48
yann2mhhh14:49
yann2so I have a xvdd with bunch of jsons, and no xvde14:49
jcliftThat just means you're using a VM type without an additional data disk14:49
jclifteg 512 Standard or 1GB Performance14:49
jcliftDon't get hung up on xvde thing.  That was my mistage14:50
jcliftmistake14:50
yann2ok14:50
jcliftMy typing sucketh today ;)14:50
yann2cant see my config copied anywhere on the config disk though14:50
jcliftyann2: Is this similar to what you're seeing? http://fpaste.org/92259/96882152/14:50
yann2yep14:51
yann2is my  user_data supposed to be copied somewhere there?14:52
jcliftyeah14:52
jcliftNew paste: http://fpaste.org/92262/13968823/14:52
jcliftThat shows the same mount, but also shows the user_data file that was pulled in by the userdata= parameter in my Pyrax call14:53
jcliftThat file gets placed into /var/lib/cloud/instance/user_data14:54
jcliftAfter the remote urls are grabbed, the result is /var/lib/cloud/instance/user_data.i, which is then run/processed/whatever14:54
yann2yeah your ci_config variable actually contains the content of the file14:54
jcliftYeah14:54
yann2so this file is a cloud init config file14:54
jcliftYeah14:55
yann2and if i read it right you do not base64encode it nor is it mime encoded14:55
jcliftCorrect14:55
yann2I ll show you my script give me a sec14:55
jcliftI don't base64 or otherwise touch it14:55
jcliftk14:55
yann2http://pastealacon.com/3426314:57
jcliftDoes it work like that?14:58
yann2I dont have a user_data in openstack/latest though14:58
yann2nope14:58
yann2I mean it runs and creates the vm, but no userdata in the config disk14:59
yann2maybe its a question for fog...14:59
jcliftWhat happens if you try "userdata" instead of "user_data" ?14:59
* jclift noticed some inconsistency in naming around user_data and userdata14:59
jcliftSome parts of Nova require "user_data"15:00
jcliftBut Pyrax definitely needs it as userdata15:00
yann2https://github.com/fog/fog/blob/f531f0bdd715e6f0e3008b35535834ad8e854ec1/lib/fog/cloudstack/models/compute/server.rb 15:00
yann2seems to be an alias, lets try it..15:00
yann2still no success :'(15:10
jcliftDamn15:10
jcliftHmmm, maybe time to ask the Fog guys then15:10
yann2yep..15:10
jcliftDo they have an example of working Fog code with userdata you could copy to experiment with?15:10
yann2dont know, I just asked on #ruby-fog15:12
yann2found this, but its old http://pastebin.com/n6fRuF0C  :D 15:15
jcliftyann2: Heh, well "Good Luck" I guess.  I'm not into Ruby... hopefully the Fog guys can help. :)15:18
yann2yeah thanks, I think it's supported by the openstack compute but not the rackspace one...15:19
yann2I ll try my luck with a bug report15:20
yann2opened a bug there https://github.com/fog/fog/issues/2824  let's wait and see15:25
yann2jclift, trying my luck patching fog, wish me luck ;)15:53
jclift:)15:58
jcliftDefinitely good luck!15:58
yann2yeah man, epic skills here :) I ll do a pull request this evening16:00
jclift:)16:11
saucei am using cloud-init for the first time. using it with EC2. i created a very simple user-data #cloud-config file (see here http://pastebin.com/gE8C0M1d).  i am using centos6.4 AMI's from cloudmarket.com that say "with cloud-init installed".  when I do "ec2-run-instances -f myuserdata.yaml", it boots up fine, i can see cloud-init is installed, but my user-data file was not taken into consideration. anyone know why?16:35
smosersauce, what is the version of cloud-init ?16:55
yann2https://github.com/yannh/fog/commit/e335084b200e88263f739a867366afd97fc6b0cb  jclift , for what its worth16:57
yann2testing a bit more before sending my pull request :)16:57
harlowjasmoser so when we switching cloud-init to git? ;)16:57
smoserharlowja, i try to avoid flamewars.16:59
harlowjalol16:59
SpamapSsmoser: cool. I just fetched the key from keyserver.ubuntu.com and it does not have your signature on it.17:04
SpamapSreally any signatures on it17:04
yann2https://github.com/fog/fog/pull/2826/files now just need a nice soul to merge it and I should be good17:05
smoserSpamapS, i can sign it. thats fine. the thing i didn't know is what i should sign.17:06
smoserits a subkey17:06
yann2alright, off for today, thanks a lot for all your help jclift that was immensely useful17:08
yann2have a nice day/evening17:08
smoserSpamapS, i signed now. and put onto keyserver.ubuntu.com . 17:20
=== ctracey|away is now known as ctracey
SpamapSsmoser: sign the main key. The point is to allow downloaders to verify the key without prayer. ;)17:29
smoserSpamapS, yeah. i did, right?17:30
SpamapSsmoser: indeed, well done :)17:38
saucesmoser cloud-init-0.5.15-68.el6_bashton1.noarch17:45
saucesmoser nevermind, cloud-init did run. i tried it with some low level stuff like touch /tmp/file1 and it did.  now i gotta figure out why the puppet module didnt work17:47
saucewhile i am here, whats the best way to add a host to /etc/hosts with cloud-init?17:47
smosersauce, that is old17:50
smoseri really dont know what support there would be in something called that version.17:51
smoserwithout looking. so that i'd be my first guess.17:51
smoserwrt /etc/hosts, you can modify the apprpriate template file with a boothook.17:51
SpamapSsmoser: note that it would be quite useful now if you started either signing the images, or producing SHA256SUMS17:52
SpamapSsmoser: MD5 is broken17:52
smoseri am signing the images.17:53
smoserhttp://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.sjson17:53
smoserthats the preferred way.17:53
SpamapSpreferred by who?17:53
harlowja0.5.15, woah17:54
smosererr.. wait. bad link.17:54
SpamapSI can use jq to grab those sha256sums tho .. so that works17:54
harlowjasauce u are on rhel6?17:54
smoserhttp://download.cirros-cloud.net/streams/v1/net.cirros-cloud:devel:download.sjson17:54
sauce6.4, about 1 year old17:54
harlowjahttp://repos.fedorapeople.org/repos/openstack/cloud-init/ should be newer :)17:54
harlowjaalthough i thought u could install a newer one17:55
harlowja0.7.2 is much better ;)17:55
SpamapSsmoser: color me old fashioned.. but a .asc file for each image would be um.. a bazillion times preferrable to that for scripting. :-P17:55
SpamapSsmoser: but I can work with it. :-P17:55
smoseri agree that having FILE.asc is simpler for scripting.17:56
SpamapSjq has at least made json available to bash17:57
SpamapSsmoser: would be quite helpful if this format actually had a _list_ for the products. :-P17:59
smoserlist ?18:00
smoserits a dict18:00
smoserbecause the products is a key18:00
SpamapSsmoser: so to find latest version, I should still use /version/released ?18:01
SpamapSsmoser: Yeah, but I have to sort the keys to find the latest. :-P18:01
* SpamapS will call the whambulance18:01
SpamapSsmoser: just whining. :)18:01
jcliftsauce: There's modern cloud-init in EPEL isn't there?18:02
saucejclift im just being lazy and using someone elses AMI18:03
saucerather than create my own18:03
smoserSpamapS, http://paste.ubuntu.com/7218123/18:03
jcliftHeh.  Doesn't seem to be working for you in this instance. :(18:03
smoserthat is secure and allows you to easily point it at a mirror if you want.18:03
smoserand '--max=1' does the sorting for you.18:03
harlowjasauce i'd recommend at least upgrading the AMI cloud-init version :)18:04
SpamapS$ sstream-query18:04
SpamapSsstream-query: command not found18:04
SpamapSsmoser: no package suggestion. :(18:04
SpamapSsmoser: also I have a really giant set of Fedora users so I have to be mindful of hard it will be for them to use.18:05
smoser$ dpkg -S `which sstream-query`18:05
smosersimplestreams: /usr/bin/sstream-query18:05
smoserSpamapS, that giant set of users installs random garbage from pip all the time.18:05
smoserso... not being packaged isn't a *huge* thing.18:06
harlowjabad users18:06
SpamapSsmoser: oh we're quite happy to abuse pip :)18:06
smoserso i should get a simpelstreams into pip.18:06
sauceubuntu's official EC2 ami's use: 0.6.3-0ubuntu1.1018:07
SpamapSsmoser: yes. but for now, jq will suffice18:08
smoserjq ?18:08
harlowjahttp://stedolan.github.io/jq/18:10
harlowjathose bash people18:10
harlowjalol18:10
SpamapS--===/o/18:11
harlowjalol18:11
smoserits C ?18:11
SpamapSsmoser: also, you have files that are '.gpg' but they have .asc content in them18:11
SpamapSsmoser: yeah tiny little utility, we use it everywhere to process json18:11
smoserthats nice.18:12
smoserSpamapS, i dont think i started that.18:14
smoserthe little i know came from modelling ubuntu18:14
smoserhttp://cdimage.ubuntu.com/ubuntu-server/daily/current/MD5SUMS.gpg18:14
SpamapSsure18:14
SpamapSthey did it wrong too :)18:14
SpamapSit confuses gpg, you have to tell it that it is ascii armored detached18:15
harlowjai left my ascii armor at home :-/18:16
smoserSpamapS, well that sucks.18:17
smosermy damned ignorance keeps getting in the way.18:17
smoserthe sstreams code only really handles the inline signed anyway.18:17
SpamapSsmoser: Yeah, and the inlined signed is kind of hard to use w/ jq.. as I have to strip off the signature.. :-P18:28
smoserright. so we need to improve jq for inline signed json18:29
SpamapSsmoser: jq example:18:31
SpamapS$ jq '.["products"]["net.cirros-cloud:standard:0.3:i386"]["versions"] | keys | sort | max' net.cirros-cloud\:released\:download.json18:31
SpamapS"20140317"18:31
SpamapSsee how hard that is. :-P18:32
SpamapS$ jq '.["products"]["net.cirros-cloud:standard:0.3:i386"]["versions"]["20140317"]["items"]["disk.img"]["sha256"]' net.cirros-cloud\:released\:download.json18:33
SpamapS"f0803c2d179c8a02d029239d35fc3e752cc81ad3436ea52b757e11685ca7c074"18:33
smosersee how much easier it was with the tool i gave you ?18:34
SpamapSsmoser: is it on pypi yet?18:35
smoserits not. i can put it there, although i've never done that before.18:35
smoserso again with the damned ignorance getting in my way.18:36
SpamapSsmoser: I've got a jq solution. It's not horrible. But if/when you do get it on pypi, please let me know so I can switch.18:39
smoserSpamapS, are you looking to put that into devstack ?18:39
smoseror elsewhere18:39
SpamapSdevstack is dead to me :)18:42
SpamapSbut it should go there too yes18:42
SpamapSsmoser: TripleO devtest yes18:42
SpamapSsmoser: https://review.openstack.org/#/c/83347/18:42
smoserwell, its interesting actually. both of them are.18:44
smoserfrom the "continuous integration" perspective.18:44
smoserin that I can randomly change / break things. 18:45
=== gondoi is now known as zz_gondoi
=== zz_gondoi is now known as gondoi
saucewhere can i get full cloud-init docs other than http://cloudinit.readthedocs.org/en/latest/?  the Modules section is empty, and i feel that the examples don't list everything.19:00
saucei.e. it's not clear how to add a 3rd party apt repo19:01
saucei would even settle for code to look through.  where are the modules stored?19:02
smosersauce, you get docs for the version you'r using in the package you're using.19:08
smoserthe config modules are stored in cloudinit/modules/cc_*19:09
smoser(except they might be different in 0.5.x)19:09
smoserto add a 3rd party apt repo is trivial 19:09
smoserand is shown at http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config.txt#L2119:09
harlowjaSpamapS u say devstack is dead??19:10
harlowjaintersting19:10
harlowjaSpamapS u know u want to use anvil, lol19:16
SpamapSharlowja: I say that devstack is dead _to me_19:23
SpamapSharlowja: we use devtest.. because multi-node matters. :)19:24
harlowjalol19:24
harlowjaya19:24
saucethanks smoser !19:34
saucesmoser you say the docs come with the package, i have no man cloud-init though19:34
smosersauce, in ubuntu they're in /usr/share19:36
smoserin centos or whatever, id' rpm -qa | grep -i doc19:37
saucethanks sir19:42
saucei switched to ubuntu for now. i want to learn cloud-init in ubuntu first, then i'll go to centos19:43
smoseror just stay on a sane OS :)19:48
smoseralthough, to be fair, 2.6.18 was a good kernel.19:49
=== gondoi is now known as zz_gondoi
=== zz_gondoi is now known as gondoi
smosersauce, i was making fun of centos.20:07
saucei know i know20:07
smoser:)20:07
saucewe are working on building a new app at work, and i have the opportunity to choose the linux distro20:07
saucei want to use ubuntu, but that's just the kid in me talking20:08
saucehey smoser, i'm gonna paste somethin in chan, forgive me20:25
sauceapt_sources: - source: deb http://apt.puppetlabs.com $RELEASE main dependencies20:25
sauce   keyid: 4BD6EC30    # GPG key ID published on a key server20:25
sauce   filename: puppetlabs.list20:25
saucei can't get that to work. i troubleshooted down to the "keyid" line. if i comment it out, cloud-init works20:26
smosersauce, indentation is important.20:32
smoseryouc an see that what you're going to get is sane by just doing somethin glike:20:32
smoserpython -c 'import yaml, pprint, sys; pprint.pprint(yaml.load(open(sys.argv[0])))' my.file20:33
saucealso maybe yamllint.com ?20:33
smoserhttp://paste.ubuntu.com/7218786/20:34
smosersure. 20:34
sauceyamllint.com says the yaml is OK but outputs it vastly different than it appears in examples20:36
saucelemme try python20:36
saucei don't think indentation is the problem here20:39
smosersauce, its not that its valid or not.20:44
smoserits that the result is int he right format20:44
saucei hear ya20:44
smoserapt_sources is a list of dictionaries20:44
saucei don't see how i can be messing up the indentation20:44
sauceits just a few spaces, everything is lined up20:44
saucei deleted the line and rewrote it char by char20:44
sauce"filename" works with the same indentation20:45
saucekeyid doesn't20:45
smoserhm.20:45
saucei know right20:45
smoserpastebin /var/log/cloud-init.log ?20:45
sauceabsolutely hang on20:46
saucewelp there ya go: Apr  7 20:42:26 ip-10-30-4-54 [CLOUDINIT] cc_apt_update_upgrade.py[WARNING]: Source Error: deb http://apt.puppetlabs.com precise main dependencies:failed to get key from keyserver.ubuntu.com20:47
smosersauce, its actuall preferable from both security and reliability perspective to insert the full key20:49
saucerunning this again, lemme check20:49
saucesmoser ok i will try that20:50
smoserhttp://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config.txt#L12020:50
smoserhte example there.20:50
smoserits more reliable because you dont depend on the keyserver20:50
smoserand int his case all you're really using it for is a object store to map <short_key_id> to <public key>20:51
saucei gotcha.  i pasted the key block and it validated as yaml. running it now.20:56
saucemany ec2 instances have died at my hand stoday20:57
saucehands today20:57
smosersauce, for playing, you can use lxc or kvm 20:58
smoserlxc you can use quite performantly inside a ec2 instance20:58
smosersee example here: http://ubuntu-smoser.blogspot.com/2013/08/lxc-with-fast-cloning-via-overlayfs-and.html20:58
saucewhile this is running i have another question: can i specify the order that things are run? i.e. i want to install puppetlabs repo, then use the puppet module20:58
saucenice the key worked!!  20:59
sauceand it installed in the right order, that was a 50/50 chance probably right?20:59
smoserwell, no. :)20:59
smoserpuppet module runs after apt_config20:59
smoserand that is by design20:59
smoseryou can specify the order of things20:59
smoserjust define the list in your user-data21:00
smoserhttp://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/config/cloud.cfg21:00
smoserthe easiest thing to do is just re-define the whole list21:00
smosercloud_init_modules runs first, then cloud_config_modules, then cloud_final_modules21:00
smosernd the listst there in order.21:01
smoseryou really should take a look at that blog post, it makes testing this stuff really easy and much faste rthan waiting on instances in amazon 21:01
sauceawesome smoser 21:02
saucejust awesome21:03
smoserthanks.21:06
saucewhy do i even need puppet, if i have cloud-init?21:22
=== harlowja is now known as harlowja_away
=== harlowja_away is now known as harlowja
=== gondoi is now known as zz_gondoi

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!