=== sambetts|afk is now known as sambetts [13:09] rharper, so you're good right now with core, right ? [13:16] hi! [13:16] anyone here who can help me with a little cloudinit issue? :) [13:19] I want to setup a server with Chef, and added the respective configs to my cloud-init file. [13:19] I tested it on ubuntu and centos (in openstack), however it fails on both oss [13:19] with the following error: [13:19] Running module chef () failed [13:20] I even ran cloud-init in debug mode, but I always only get this one line of log [13:31] fekle, can you paste /var/log/cloud-init.log ? [13:31] ubuntu preferably [13:38] https://pastebin.com/GDRAEygu [13:38] this is my clou dinit [13:39] fekle, fyi, 'pastebinit' command is available on ubuntu. 'pastebinit file' [13:40] oh thanks :) also available on my arch i see [13:40] fekle, if you can get me /var/log/cloud-init.log and /var/log/cloud-init-output.log it'd help. [13:40] i realize you're probably worried about sensitive info. [13:41] im currently running again, gonna see what i find in the logs [13:41] look for WARN, and there will probably be a stack trace [13:44] well that's the thing, there is not stacktrace, neither in cloud-init.log nor in cloud-init-output.log [13:45] https://pastebin.com/EUqSqyYH [13:45] thats all that is in /var/log/cloud-init.log ? [13:45] that's all I get below the ip/route/interface table thing in the log [13:46] before net device info and route info i only get [13:46] Cloud-init v. 0.7.5 running 'init-local' at Tue, 25 Apr 2017 13:42:23 +0000. Up 3.23 seconds. Cloud-init v. 0.7.5 running 'init' at Tue, 25 Apr 2017 13:42:25 +0000. Up 6.07 seconds. [13:47] this is 14.04 ? [13:48] nope, centos 7.3 [13:48] on ubuntu 16.04 we get the exact same error [13:48] haven't tried 14.04 [13:48] no stacktrace on 16.04 either :// [13:48] it just fails [13:49] without doing anything [13:49] even better, [13:49] If i go in the vm, delete /var/lib/cloud, and re-run cloud-init -d init && cloud-init -d moudles [13:49] *modules [13:49] it fails again, no stacktrace [13:50] but this time it always creates the correct chef config in /etc/chef [13:50] well, lets focus on 16.04 for now [13:50] you're using the ubuntu images ? [13:50] yep [13:50] official image [13:50] ok. launch one of those. lets focus on that. [13:51] qcow2 from here https://cloud-images.ubuntu.com/xenial/current/ [13:51] then, if you can let me in it would be easier. i understand your hesitancy. i really supect there is *something* in a 16.04 /var/log/cloud-init.log or /var/log/cloud-init-output.log [13:52] well the VMs are in a private network ;) but I can send you the full contents of both log files [13:53] sure. [13:55] wow, you're right [13:55] on ubuntu I get waaaay more logs in /var/log than on centos [13:55] didn't show a difference in openstack console tho [13:55] https://pastebin.com/H8JT2pwC [13:57] initial_attributes: [13:58] that needs to be a dictionary or not present [13:58] yeah, they're empty, but I had the same erorr with some set [13:58] well, this is the error you're seeing now [13:58] I'll try again [13:58] it should be fine to just remove it [13:59] weird, normally an empty dictionary in yaml should be fine, not? [14:00] $ python -c 'import yaml; print(yaml.load("foo:\n"))' [14:00] {'foo': None} [14:00] here's the full log https://pastebin.com/uaLjiRMC [14:00] ie, you gave it a None type, not a dictionary [14:00] its expecting a dictionary [14:00] and seing a None Type [14:00] got it [14:01] (i'm not suggesting that the fail path there should not be more informative, but thats what happened) [14:01] once it statck traced it wasn't going to go further, so launch another with cleaned up entry there. [14:02] yeah, it's an understandable error, but it could be handled better, ie print a message warning that it's empty, or just ignoring the nulltype [14:02] I'm spawning a new VM with attributes now [14:03] smoser: yes [14:05] now i get a new error: https://pastebin.com/ZQqird0D [14:06] but nice that cloud-init shows stacktraces on ubuntu, the missing stacktraces on centos really make fixing the issue very hard [14:12] @smoser I tried again, but AttributeError: 'UrlResponse' object has no attribute 'encode' is still the error - it's now creating the chef config correctly on the first run tho [14:13] fekle, https://git.launchpad.net/cloud-init/commit/?id=482b2746b59 [14:14] thats the bug. it should be fixed in a 16.04 image newer than 4-20. [14:14] sorry i ditn just recognize this right away [14:14] aaah, awesome, thank you :) [14:14] no problem [14:14] can you verify what cloud-init you have in the image ? [14:14] dpkg-query --show cloud-init [14:15] if it is 0.7.9-90-g61eb03fe-0ubuntu1~16.04.1 [14:15] then it is not fixed correctly [14:15] cloud-init 0.7.9-48-g1c795b9-0ubuntu1~16.04.1 [14:16] yeah. so if you get taht current image http://cloud-images.ubuntu.com/daily/server/xenial/current/ (which is actually http://cloud-images.ubuntu.com/daily/server/xenial/20170424/) [14:16] then it shoudl be good [14:16] ie, 0424 has that fix [14:16] however, we are using ubuntu only for this testing here, everything else is CentOS [14:16] ok, I'll download the newwest one [14:16] well, *thats* your problem :) [14:16] yeah :D [14:16] kidding. [14:16] but yeah, you'll need to somehow get that fix into your centos then [14:17] it is .... often enough, believe me [14:17] i expect a cloud-init release of 0.7.10 very shortly [14:17] yeah, I'll try downloading the newest version of the centos image [14:17] so , that should be released and larsks does a really good job of picking stuff up. [14:18] awesome [14:32] so, just tried with the new image [14:32] now i get another crash when instaling chef [14:32] but this time it seems to be a chef thing, not an cloud-init issue [14:33] https://pastebin.com/7Kd6gHCC [14:34] on Centos it fails too, but no stacktrace, although i think it's the same error [14:37] hm.. [14:42] fekle, so it downloads the 'omnibus_url' [14:42] and puts that in a file with execute permission and then tries to execute it [14:42] the default url is [14:42] https://www.getchef.com/chef/install.sh [14:43] you had '' that you were using something else. [14:43] so this issue should be recreatable with just: [14:43] wget -O out [14:43] chmod 755 out [14:43] ./out [14:44] smoser, powersj do we have and jenkins set up for building centos/rhel or suse packages? [14:44] no, but something I wanted to go through with you today. [14:44] I've been using an old smoser script to launch a lxd [14:44] for centos7 and wanted to see best way to get that in jenkins to run unit tests and build rpms [14:45] from there we can decide what we do with them or if we just test that things work [14:45] powersj, yeah I feel like it'd be nice to make sure packaging/dependency changes don't break things. I set up an ec2 instance last night to to poke around at things. [14:45] yeah, we are using our own script [14:45] lxd ftw [14:45] for version pinning [14:45] colleague forgot to add a shebang, so it didn't run if you don't directly pipe it into bash :p [14:46] heh [14:46] fekle, right. its more functional to just let the kernel decide [14:46] that way, if you put a binary there, it just works [14:46] or a python script [14:46] i dont expect it to be shell [14:47] blackboxsw, the scripts he speaks of are https://gist.github.com/smoser/19e65095b342e98fd4d6466e4d4fa1ac [14:48] smoser, ahh I missed that in the url collection effort of 2017 ;) thanks for the pointer [14:48] i think we should look into getting a yum / rpm repo of trunk somehow [14:48] smoser, I completely agree w/ that [14:48] simpler for ci on the final product [14:48] blackboxsw, if i just give you my firefox awesomebar history, you'llk have just about the collected bunch of knowledge that i have [14:49] the rest lives in irc logs [14:50] logs4life [14:51] % du -sh ~/.xchat2/xchatlogs/ [14:51] 784M /home/rharper/.xchat2/xchatlogs/ [14:51] that's getting beefy [14:52] I had openstack keystone logs with 20+ million lines of logs, due to forgetting to turn of debug messages :p [14:53] loading those logs with lnav is fun [14:53] $ du -hs ~/data/bip/ [14:53] 2.7G /home/smoser/data/bip/ [14:56] smoser: nice! [14:56] smoser: is you're gzipped ? [14:57] no. [14:57] the logs on my local system are a measley 490M or something [14:57] and its those that i most often read, i only go to the bip logs if i need to === rangerpbzzzz is now known as rangerpb === shardy is now known as shardy_afk === shardy_afk is now known as shardy === sambetts is now known as sambetts|afk