[13:09] <smoser> rharper, so you're good right now with core, right ?
[13:16] <fekle> hi!
[13:16] <fekle> anyone here who can help me with a little cloudinit issue? :)
[13:19] <fekle> I want to setup a server with Chef, and added the respective configs to my cloud-init file.
[13:19] <fekle> I tested it on ubuntu and centos (in openstack), however it fails on both oss
[13:19] <fekle> with the following error:
[13:19] <fekle> Running module chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_chef.py'>) failed
[13:20] <fekle> I even ran cloud-init in debug mode, but I always only get this one line of log
[13:31] <smoser> fekle, can you paste /var/log/cloud-init.log ?
[13:31] <smoser> ubuntu preferably
[13:38] <fekle> https://pastebin.com/GDRAEygu
[13:38] <fekle> this is my clou dinit
[13:39] <smoser> fekle, fyi, 'pastebinit' command is available on ubuntu.  'pastebinit file'
[13:40] <fekle> oh thanks :) also available on my arch i see
[13:40] <smoser> fekle, if you can get me /var/log/cloud-init.log  and /var/log/cloud-init-output.log it'd help.
[13:40] <smoser> i realize you're probably worried about sensitive info.
[13:41] <fekle> im currently running again, gonna see what i find in the logs
[13:41] <smoser> look for WARN, and there will probably be a stack trace
[13:44] <fekle> well that's the thing, there is not stacktrace, neither in cloud-init.log nor in cloud-init-output.log
[13:45] <fekle> https://pastebin.com/EUqSqyYH
[13:45] <smoser> thats all that is in /var/log/cloud-init.log ?
[13:45] <fekle> that's all I get below the ip/route/interface table thing in the log
[13:46] <fekle> before net device info and route info i only get
[13:46] <fekle> Cloud-init v. 0.7.5 running 'init-local' at Tue, 25 Apr 2017 13:42:23 +0000. Up 3.23 seconds. Cloud-init v. 0.7.5 running 'init' at Tue, 25 Apr 2017 13:42:25 +0000. Up 6.07 seconds.
[13:47] <smoser> this is 14.04 ?
[13:48] <fekle> nope, centos 7.3
[13:48] <fekle> on ubuntu 16.04 we get the exact same error
[13:48] <fekle> haven't tried 14.04
[13:48] <fekle> no stacktrace on 16.04 either ://
[13:48] <fekle> it just fails
[13:49] <fekle> without doing anything
[13:49] <fekle> even better,
[13:49] <fekle> If i go in the vm, delete /var/lib/cloud, and re-run cloud-init -d init && cloud-init -d moudles
[13:49] <fekle> *modules
[13:49] <fekle> it fails again, no stacktrace
[13:50] <fekle> but this time it always creates the correct chef config in /etc/chef
[13:50] <smoser> well, lets focus on 16.04 for now
[13:50] <smoser> you're using the ubuntu images ?
[13:50] <fekle> yep
[13:50] <fekle> official image
[13:50] <smoser> ok. launch one of those. lets focus on that.
[13:51] <fekle> qcow2 from here https://cloud-images.ubuntu.com/xenial/current/
[13:51] <smoser> then, if you can let me in it would be easier. i  understand your hesitancy. i really supect there is *something* in a 16.04 /var/log/cloud-init.log or /var/log/cloud-init-output.log
[13:52] <fekle> well the VMs are in a private network ;) but I can send you the full contents of both log files
[13:53] <smoser> sure.
[13:55] <fekle> wow, you're right
[13:55] <fekle> on ubuntu I get waaaay more logs in /var/log than on centos
[13:55] <fekle> didn't show a difference in openstack console tho
[13:55] <fekle> https://pastebin.com/H8JT2pwC
[13:57] <smoser>  initial_attributes:
[13:58] <smoser> that needs to be a dictionary or not present
[13:58] <fekle> yeah, they're empty, but I had the same erorr with some set
[13:58] <smoser> well, this is the error you're seeing now
[13:58] <fekle> I'll try again
[13:58] <smoser> it should be fine to just remove it
[13:59] <fekle> weird, normally an empty dictionary in yaml should be fine, not?
[14:00] <smoser> $ python -c 'import yaml; print(yaml.load("foo:\n"))'
[14:00] <smoser> {'foo': None}
[14:00] <fekle> here's the full log https://pastebin.com/uaLjiRMC
[14:00] <smoser> ie, you gave it a None type, not a dictionary
[14:00] <smoser> its expecting a dictionary
[14:00] <fekle> and seing a None Type
[14:00] <fekle> got it
[14:01] <smoser> (i'm not suggesting that the fail path there should not be more informative, but thats what happened)
[14:01] <smoser> once it statck traced it wasn't going to go further, so launch another with cleaned up entry there.
[14:02] <fekle> yeah, it's an understandable error, but it could be handled better, ie print a message warning that it's empty, or just ignoring the nulltype
[14:02] <fekle> I'm spawning a new VM with attributes now
[14:03] <rharper> smoser: yes
[14:05] <fekle> now i get a new error: https://pastebin.com/ZQqird0D
[14:06] <fekle> but nice that cloud-init shows stacktraces on ubuntu, the missing stacktraces on centos really make fixing the issue very hard
[14:12] <fekle> @smoser I tried again, but AttributeError: 'UrlResponse' object has no attribute 'encode' is still the error - it's now creating the chef config correctly on the first run tho
[14:13] <smoser> fekle, https://git.launchpad.net/cloud-init/commit/?id=482b2746b59
[14:14] <smoser> thats the bug. it should be fixed in a 16.04 image newer than 4-20.
[14:14] <smoser> sorry i ditn just recognize this right away
[14:14] <fekle> aaah, awesome, thank you :)
[14:14] <fekle> no problem
[14:14] <smoser> can you verify what cloud-init you have in the image ?
[14:14] <smoser> dpkg-query --show cloud-init
[14:15] <smoser> if it is 0.7.9-90-g61eb03fe-0ubuntu1~16.04.1
[14:15] <smoser> then it is not fixed correctly
[14:15] <fekle> cloud-init	0.7.9-48-g1c795b9-0ubuntu1~16.04.1
[14:16] <smoser> yeah. so if you get taht current image http://cloud-images.ubuntu.com/daily/server/xenial/current/ (which is actually http://cloud-images.ubuntu.com/daily/server/xenial/20170424/)
[14:16] <smoser> then it shoudl be good
[14:16] <smoser> ie, 0424 has that fix
[14:16] <fekle> however, we are using ubuntu only for this testing here, everything else is CentOS
[14:16] <fekle> ok, I'll download the newwest one
[14:16] <smoser> well, *thats* your problem :)
[14:16] <fekle> yeah :D
[14:16] <smoser> kidding.
[14:16] <smoser> but yeah, you'll need to somehow get that fix into your centos then
[14:17] <fekle> it is .... often enough, believe me
[14:17] <smoser> i expect a cloud-init release of 0.7.10 very shortly
[14:17] <fekle> yeah, I'll try downloading the newest version of the centos image
[14:17] <smoser> so , that should be released and larsks does a really good job of picking stuff up.
[14:18] <fekle> awesome
[14:32] <fekle> so, just tried with the new image
[14:32] <fekle> now i get another crash when instaling chef
[14:32] <fekle> but this time it seems to be a chef thing, not an cloud-init issue
[14:33] <fekle> https://pastebin.com/7Kd6gHCC
[14:34] <fekle> on Centos it fails too, but no stacktrace, although i think it's the same error
[14:37] <smoser> hm..
[14:42] <smoser> fekle, so it downloads the 'omnibus_url'
[14:42] <smoser> and puts that in a file with execute permission and then tries to execute it
[14:42] <smoser> the default url is
[14:42] <smoser>  https://www.getchef.com/chef/install.sh
[14:43] <smoser> you had '<censored>' that you were using something else.
[14:43] <smoser> so this issue should be recreatable with just:
[14:43] <smoser>  wget <thaturl> -O out
[14:43] <smoser>  chmod 755 out
[14:43] <smoser>  ./out
[14:44] <blackboxsw> smoser, powersj do we have and jenkins set up for building centos/rhel or suse packages?
[14:44] <powersj> no, but something I wanted to go through with you today.
[14:44] <powersj> I've been using an old smoser script to launch a lxd
[14:44] <powersj> for centos7 and wanted to see best way to get that in jenkins to run unit tests and build rpms
[14:45] <powersj> from there we can decide what we do with them or if we just test that things work
[14:45] <blackboxsw> powersj, yeah I feel like it'd be nice to make sure packaging/dependency changes don't break things. I set up an ec2 instance last night to to poke around at things.
[14:45] <fekle> yeah, we are using our own script
[14:45] <powersj> lxd ftw
[14:45] <fekle> for version pinning
[14:45] <fekle> colleague forgot to add a shebang, so it didn't run if you don't directly pipe it into bash :p
[14:46] <blackboxsw> heh
[14:46] <smoser> fekle, right. its more functional to just let the kernel decide
[14:46] <smoser> that way, if you put a binary there, it just works
[14:46] <smoser> or a python script
[14:46] <smoser> i dont expect it to be shell
[14:47] <smoser> blackboxsw, the scripts he speaks of are https://gist.github.com/smoser/19e65095b342e98fd4d6466e4d4fa1ac
[14:48] <blackboxsw> smoser, ahh I missed that in the url collection effort of 2017 ;) thanks for the pointer
[14:48] <smoser> i think we should look into getting a yum / rpm repo of trunk somehow
[14:48] <blackboxsw> smoser, I completely agree w/ that
[14:48] <blackboxsw> simpler for ci on the final product
[14:48] <smoser> blackboxsw, if i just give you my firefox awesomebar history, you'llk have just about the collected bunch of knowledge that i have
[14:49] <smoser> the rest lives in irc logs
[14:50] <rharper> logs4life
[14:51] <rharper> % du -sh ~/.xchat2/xchatlogs/
[14:51] <rharper> 784M	/home/rharper/.xchat2/xchatlogs/
[14:51] <rharper> that's getting beefy
[14:52] <fekle> I had openstack keystone logs with 20+ million lines of logs, due to forgetting to turn of debug messages :p
[14:53] <fekle> loading those logs with lnav is fun
[14:53] <smoser> $ du -hs ~/data/bip/
[14:53] <smoser> 2.7G	/home/smoser/data/bip/
[14:56] <rharper> smoser: nice!
[14:56] <rharper> smoser: is you're gzipped ?
[14:57] <smoser> no.
[14:57] <smoser> the logs on my local system are a measley 490M or something
[14:57] <smoser> and its those that i most often read, i only go to the bip logs if i need to