| === sambetts|afk is now known as sambetts | ||
| smoser | rharper, so you're good right now with core, right ? | 13:09 |
|---|---|---|
| fekle | hi! | 13:16 |
| fekle | anyone here who can help me with a little cloudinit issue? :) | 13:16 |
| fekle | I want to setup a server with Chef, and added the respective configs to my cloud-init file. | 13:19 |
| fekle | I tested it on ubuntu and centos (in openstack), however it fails on both oss | 13:19 |
| fekle | with the following error: | 13:19 |
| fekle | Running module chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_chef.py'>) failed | 13:19 |
| fekle | I even ran cloud-init in debug mode, but I always only get this one line of log | 13:20 |
| smoser | fekle, can you paste /var/log/cloud-init.log ? | 13:31 |
| smoser | ubuntu preferably | 13:31 |
| fekle | https://pastebin.com/GDRAEygu | 13:38 |
| fekle | this is my clou dinit | 13:38 |
| smoser | fekle, fyi, 'pastebinit' command is available on ubuntu. 'pastebinit file' | 13:39 |
| fekle | oh thanks :) also available on my arch i see | 13:40 |
| smoser | fekle, if you can get me /var/log/cloud-init.log and /var/log/cloud-init-output.log it'd help. | 13:40 |
| smoser | i realize you're probably worried about sensitive info. | 13:40 |
| fekle | im currently running again, gonna see what i find in the logs | 13:41 |
| smoser | look for WARN, and there will probably be a stack trace | 13:41 |
| fekle | well that's the thing, there is not stacktrace, neither in cloud-init.log nor in cloud-init-output.log | 13:44 |
| fekle | https://pastebin.com/EUqSqyYH | 13:45 |
| smoser | thats all that is in /var/log/cloud-init.log ? | 13:45 |
| fekle | that's all I get below the ip/route/interface table thing in the log | 13:45 |
| fekle | before net device info and route info i only get | 13:46 |
| fekle | Cloud-init v. 0.7.5 running 'init-local' at Tue, 25 Apr 2017 13:42:23 +0000. Up 3.23 seconds. Cloud-init v. 0.7.5 running 'init' at Tue, 25 Apr 2017 13:42:25 +0000. Up 6.07 seconds. | 13:46 |
| smoser | this is 14.04 ? | 13:47 |
| fekle | nope, centos 7.3 | 13:48 |
| fekle | on ubuntu 16.04 we get the exact same error | 13:48 |
| fekle | haven't tried 14.04 | 13:48 |
| fekle | no stacktrace on 16.04 either :// | 13:48 |
| fekle | it just fails | 13:48 |
| fekle | without doing anything | 13:49 |
| fekle | even better, | 13:49 |
| fekle | If i go in the vm, delete /var/lib/cloud, and re-run cloud-init -d init && cloud-init -d moudles | 13:49 |
| fekle | *modules | 13:49 |
| fekle | it fails again, no stacktrace | 13:49 |
| fekle | but this time it always creates the correct chef config in /etc/chef | 13:50 |
| smoser | well, lets focus on 16.04 for now | 13:50 |
| smoser | you're using the ubuntu images ? | 13:50 |
| fekle | yep | 13:50 |
| fekle | official image | 13:50 |
| smoser | ok. launch one of those. lets focus on that. | 13:50 |
| fekle | qcow2 from here https://cloud-images.ubuntu.com/xenial/current/ | 13:51 |
| smoser | then, if you can let me in it would be easier. i understand your hesitancy. i really supect there is *something* in a 16.04 /var/log/cloud-init.log or /var/log/cloud-init-output.log | 13:51 |
| fekle | well the VMs are in a private network ;) but I can send you the full contents of both log files | 13:52 |
| smoser | sure. | 13:53 |
| fekle | wow, you're right | 13:55 |
| fekle | on ubuntu I get waaaay more logs in /var/log than on centos | 13:55 |
| fekle | didn't show a difference in openstack console tho | 13:55 |
| fekle | https://pastebin.com/H8JT2pwC | 13:55 |
| smoser | initial_attributes: | 13:57 |
| smoser | that needs to be a dictionary or not present | 13:58 |
| fekle | yeah, they're empty, but I had the same erorr with some set | 13:58 |
| smoser | well, this is the error you're seeing now | 13:58 |
| fekle | I'll try again | 13:58 |
| smoser | it should be fine to just remove it | 13:58 |
| fekle | weird, normally an empty dictionary in yaml should be fine, not? | 13:59 |
| smoser | $ python -c 'import yaml; print(yaml.load("foo:\n"))' | 14:00 |
| smoser | {'foo': None} | 14:00 |
| fekle | here's the full log https://pastebin.com/uaLjiRMC | 14:00 |
| smoser | ie, you gave it a None type, not a dictionary | 14:00 |
| smoser | its expecting a dictionary | 14:00 |
| fekle | and seing a None Type | 14:00 |
| fekle | got it | 14:00 |
| smoser | (i'm not suggesting that the fail path there should not be more informative, but thats what happened) | 14:01 |
| smoser | once it statck traced it wasn't going to go further, so launch another with cleaned up entry there. | 14:01 |
| fekle | yeah, it's an understandable error, but it could be handled better, ie print a message warning that it's empty, or just ignoring the nulltype | 14:02 |
| fekle | I'm spawning a new VM with attributes now | 14:02 |
| rharper | smoser: yes | 14:03 |
| fekle | now i get a new error: https://pastebin.com/ZQqird0D | 14:05 |
| fekle | but nice that cloud-init shows stacktraces on ubuntu, the missing stacktraces on centos really make fixing the issue very hard | 14:06 |
| fekle | @smoser I tried again, but AttributeError: 'UrlResponse' object has no attribute 'encode' is still the error - it's now creating the chef config correctly on the first run tho | 14:12 |
| smoser | fekle, https://git.launchpad.net/cloud-init/commit/?id=482b2746b59 | 14:13 |
| smoser | thats the bug. it should be fixed in a 16.04 image newer than 4-20. | 14:14 |
| smoser | sorry i ditn just recognize this right away | 14:14 |
| fekle | aaah, awesome, thank you :) | 14:14 |
| fekle | no problem | 14:14 |
| smoser | can you verify what cloud-init you have in the image ? | 14:14 |
| smoser | dpkg-query --show cloud-init | 14:14 |
| smoser | if it is 0.7.9-90-g61eb03fe-0ubuntu1~16.04.1 | 14:15 |
| smoser | then it is not fixed correctly | 14:15 |
| fekle | cloud-init0.7.9-48-g1c795b9-0ubuntu1~16.04.1 | 14:15 |
| smoser | yeah. so if you get taht current image http://cloud-images.ubuntu.com/daily/server/xenial/current/ (which is actually http://cloud-images.ubuntu.com/daily/server/xenial/20170424/) | 14:16 |
| smoser | then it shoudl be good | 14:16 |
| smoser | ie, 0424 has that fix | 14:16 |
| fekle | however, we are using ubuntu only for this testing here, everything else is CentOS | 14:16 |
| fekle | ok, I'll download the newwest one | 14:16 |
| smoser | well, *thats* your problem :) | 14:16 |
| fekle | yeah :D | 14:16 |
| smoser | kidding. | 14:16 |
| smoser | but yeah, you'll need to somehow get that fix into your centos then | 14:16 |
| fekle | it is .... often enough, believe me | 14:17 |
| smoser | i expect a cloud-init release of 0.7.10 very shortly | 14:17 |
| fekle | yeah, I'll try downloading the newest version of the centos image | 14:17 |
| smoser | so , that should be released and larsks does a really good job of picking stuff up. | 14:17 |
| fekle | awesome | 14:18 |
| fekle | so, just tried with the new image | 14:32 |
| fekle | now i get another crash when instaling chef | 14:32 |
| fekle | but this time it seems to be a chef thing, not an cloud-init issue | 14:32 |
| fekle | https://pastebin.com/7Kd6gHCC | 14:33 |
| fekle | on Centos it fails too, but no stacktrace, although i think it's the same error | 14:34 |
| smoser | hm.. | 14:37 |
| smoser | fekle, so it downloads the 'omnibus_url' | 14:42 |
| smoser | and puts that in a file with execute permission and then tries to execute it | 14:42 |
| smoser | the default url is | 14:42 |
| smoser | https://www.getchef.com/chef/install.sh | 14:42 |
| smoser | you had '<censored>' that you were using something else. | 14:43 |
| smoser | so this issue should be recreatable with just: | 14:43 |
| smoser | wget <thaturl> -O out | 14:43 |
| smoser | chmod 755 out | 14:43 |
| smoser | ./out | 14:43 |
| blackboxsw | smoser, powersj do we have and jenkins set up for building centos/rhel or suse packages? | 14:44 |
| powersj | no, but something I wanted to go through with you today. | 14:44 |
| powersj | I've been using an old smoser script to launch a lxd | 14:44 |
| powersj | for centos7 and wanted to see best way to get that in jenkins to run unit tests and build rpms | 14:44 |
| powersj | from there we can decide what we do with them or if we just test that things work | 14:45 |
| blackboxsw | powersj, yeah I feel like it'd be nice to make sure packaging/dependency changes don't break things. I set up an ec2 instance last night to to poke around at things. | 14:45 |
| fekle | yeah, we are using our own script | 14:45 |
| powersj | lxd ftw | 14:45 |
| fekle | for version pinning | 14:45 |
| fekle | colleague forgot to add a shebang, so it didn't run if you don't directly pipe it into bash :p | 14:45 |
| blackboxsw | heh | 14:46 |
| smoser | fekle, right. its more functional to just let the kernel decide | 14:46 |
| smoser | that way, if you put a binary there, it just works | 14:46 |
| smoser | or a python script | 14:46 |
| smoser | i dont expect it to be shell | 14:46 |
| smoser | blackboxsw, the scripts he speaks of are https://gist.github.com/smoser/19e65095b342e98fd4d6466e4d4fa1ac | 14:47 |
| blackboxsw | smoser, ahh I missed that in the url collection effort of 2017 ;) thanks for the pointer | 14:48 |
| smoser | i think we should look into getting a yum / rpm repo of trunk somehow | 14:48 |
| blackboxsw | smoser, I completely agree w/ that | 14:48 |
| blackboxsw | simpler for ci on the final product | 14:48 |
| smoser | blackboxsw, if i just give you my firefox awesomebar history, you'llk have just about the collected bunch of knowledge that i have | 14:48 |
| smoser | the rest lives in irc logs | 14:49 |
| rharper | logs4life | 14:50 |
| rharper | % du -sh ~/.xchat2/xchatlogs/ | 14:51 |
| rharper | 784M/home/rharper/.xchat2/xchatlogs/ | 14:51 |
| rharper | that's getting beefy | 14:51 |
| fekle | I had openstack keystone logs with 20+ million lines of logs, due to forgetting to turn of debug messages :p | 14:52 |
| fekle | loading those logs with lnav is fun | 14:53 |
| smoser | $ du -hs ~/data/bip/ | 14:53 |
| smoser | 2.7G/home/smoser/data/bip/ | 14:53 |
| rharper | smoser: nice! | 14:56 |
| rharper | smoser: is you're gzipped ? | 14:56 |
| smoser | no. | 14:57 |
| smoser | the logs on my local system are a measley 490M or something | 14:57 |
| smoser | and its those that i most often read, i only go to the bip logs if i need to | 14:57 |
| === rangerpbzzzz is now known as rangerpb | ||
| === shardy is now known as shardy_afk | ||
| === shardy_afk is now known as shardy | ||
| === sambetts is now known as sambetts|afk | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!