| ausfestivus | whats the best way to get some help with a cloud-init problem here? | 07:29 |
|---|---|---|
| meena | ausfestivus: asking a question is a good start. | 07:32 |
| ausfestivus | lol | 07:33 |
| ausfestivus | sure | 07:33 |
| meena | n.b.: most devs (the people with @ in front of their nick) are somewhere in North America… so if us, mere mortals cannot help, you'll have to wait for them to wake up. | 07:34 |
| ausfestivus | im trying to debug why a script is not running when its placed in /var/lib/cloud/scripts/per-once | 07:34 |
| ausfestivus | on ubuntu 18.04 on azure as part of a scale set | 07:35 |
| ausfestivus | the script IS in the above path | 07:35 |
| ausfestivus | root@cptadotfvm000000:/var/lib/cloud/scripts/per-once# lsinstall-ado-agent.sh | 07:35 |
| ausfestivus | root@cptadotfvm000000:/var/lib/cloud/scripts/per-once# lsinstall-ado-agent.sh | 07:36 |
| ausfestivus | apols for the formatting | 07:36 |
| ausfestivus | im following the steps at https://cloudinit.readthedocs.io/en/latest/topics/faq.html#how-can-i-debug-my-user-data | 07:36 |
| ausfestivus | to debug the user data | 07:36 |
| meena | we've recently improved the documentation on per-X, or so i hope… i wonder if it's released on Read The Docs… | 07:37 |
| ausfestivus | when I run the `cloud-init query userdata > user-data.yaml` command, I get a bunch of python gunk. I suspect because the VM has python2 and cloud-init needs python3 | 07:38 |
| meena | ausfestivus: that shouldn't really matter. cloud-init should be using the correct python. | 07:39 |
| meena | ausfestivus: can you pastebin the python gunk somewhere, please? | 07:39 |
| ausfestivus | standby | 07:41 |
| meena | i can't do anything else | 07:41 |
| meena | well, i could sleep, but i don't think it's appreciated at $customer's office. | 07:41 |
| meena | unless you have 20 years seniority… | 07:42 |
| ausfestivus | whats the best way to paste console output in here? | 07:42 |
| ausfestivus | webclient here | 07:42 |
| ausfestivus | ``` | 07:43 |
| meena | ausfestivus: you copy/paste the output to your favourite pastebin… or gist.github.com or whatnot, and then post the link here | 07:43 |
| ausfestivus | ```cptuser@cptadotfvm000000:/tmp$ sudo cloud-init query userdataTraceback (most recent call last): File "/usr/bin/cloud-init", line 11, in <module> load_entry_point('cloud-init==19.4', 'console_scripts', 'cloud-init')() File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 893, in main get_uptime=True, func=functor, args=(name, | 07:43 |
| ausfestivus | args)) File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2558, in log_time ret = func(*args, **kwargs) File "/usr/lib/python3/dist-packages/cloudinit/cmd/query.py", line 124, in handle_args instance_data['userdata'] = util.load_file(user_data_fn) File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1372, in load_file | 07:43 |
| ausfestivus | return decode_binary(contents) File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 146, in decode_binary return blob.decode(encoding)UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte``` | 07:43 |
| ausfestivus | yea okay | 07:43 |
| meena | ausfestivus: not like this… :P | 07:43 |
| ausfestivus | https://gist.github.com/ausfestivus/cdd1635b9340cb2f61ee83eed071f04e | 07:44 |
| ausfestivus | ^^ python gunk | 07:44 |
| meena | ausfestivus: can you run `file /var/lib/cloud/scripts/per-once/install-ado-agent.sh` | 07:47 |
| ausfestivus | ```cptuser@cptadotfvm000000:/tmp$ file /var/lib/cloud/scripts/per-once/install-ado-agent.sh/var/lib/cloud/scripts/per-once/install-ado-agent.sh: Bourne-Again shell script, ASCII text executable``` | 07:48 |
| ausfestivus | the script is fine | 07:48 |
| ausfestivus | if I run it from there it does the job fine | 07:49 |
| meena | ausfestivus: it would be nice to know which script it's complaining about | 07:49 |
| ausfestivus | hence why im trying to get some debug info from cloud-init | 07:49 |
| ausfestivus | hence the python gunk after following the debug steps in the docs | 07:49 |
| meena | oooh, waaait… is it userdata it's complaining about? | 07:50 |
| ausfestivus | it doesnt complain | 07:50 |
| meena | ausfestivus: python-gunking about | 07:50 |
| ausfestivus | oh | 07:50 |
| ausfestivus | `sudo cloud-init query userdat` | 07:50 |
| ausfestivus | `sudo cloud-init query userdata` | 07:50 |
| meena | that works? | 07:51 |
| ausfestivus | no | 07:51 |
| ausfestivus | gives the gunk | 07:51 |
| meena | i assumed from # that you're root. | 07:51 |
| meena | ausfestivus: it's odd, according to docs, gzip compressed userdata should just be decompressed… | 07:54 |
| meena | and your userdata is. | 07:54 |
| meena | here's a cut from /usr/share/mime/magic: | 07:54 |
| meena | [50:application/gzip] | 07:55 |
| meena | >0=^@^B^_<8b> | 07:55 |
| meena | that means, gzip is identified by a magic byte sequence of 0x8b | 07:55 |
| meena | ausfestivus: you can try play around with https://cloudinit.readthedocs.io/en/latest/topics/analyze.html — or read the logs yourself, to figure out if this same behaviour happens on boot, cuz if it does that's a serious bug (or misconfiguration, or misunderstanding) | 07:58 |
| ausfestivus | are you saying thefile in per-once should be mime-encoded? | 08:03 |
| meena | ausfestivus: no, i'm saying that your userdata is gzipped. that's what "python gunk" is saying. and for reasons unclear to me, cloud-init is choking on that, rather than ungzipping it, before trying to decode it as text | 08:11 |
| meena | the only thing i can imagine going wrong is double gzip encoding, or, a bug. | 08:11 |
| ausfestivus | afk dinner | 08:13 |
| ausfestivus | @meena how can I debug this problem further? Any tips? | 09:27 |
| ausfestivus | the debug tools in the FAQ are not working for a reason I cant decipher | 09:28 |
| meena | ausfestivus: pastebin your logfile, and i can help take a look. also pastebin your /var/lib/cloud-init/instance/userdata (or whatever the actual real path is) | 09:29 |
| ausfestivus | you want both log files? | 09:31 |
| meena | sure | 09:31 |
| meena | if you're unsure about your logs, cuz they're a lot, you can cloud-init clean --logs --reboot | 09:35 |
| meena | to get a new set of fresh logs | 09:35 |
| ausfestivus | cloud-init-output.log - https://pastebin.com/0CiR0yFEcloud-init.log - https://pastebin.com/iG6Up8SP | 09:47 |
| ausfestivus | grrr | 09:47 |
| ausfestivus | standby | 09:47 |
| ausfestivus | cloud-init-output.log - https://pastebin.com/0CiR0yFE | 09:48 |
| ausfestivus | cloud-init.log - https://pastebin.com/iG6Up8SP | 09:48 |
| ausfestivus | I have a userdata.txt and a userdata.txt.i both are binary data | 09:48 |
| ausfestivus | err correction | 09:49 |
| ausfestivus | .i isnt | 09:49 |
| ausfestivus | userdata.txt.i https://pastebin.com/qacVGm4p | 09:50 |
| meena | ausfestivus: how many boots is this? did you run clean --logs --reboot? | 10:03 |
| ausfestivus | just one boot, didnt run clean --logs --reboot | 11:16 |
| hkominos | Good morning guys. I wanted to ask the channel if cloud init works with only python3 installed. I am trying to build centos8 with cloud-init but I am facing issues. (even the upstream image of centos8 is showing missing deps for cloud-init) | 11:47 |
| meena | hkominos: the latest version has dropped python 2.7 support | 12:03 |
| meena | hkominos: but, you may wanna describe exactly what problems you're facing… or pastebin some stack-trace or similar. | 12:06 |
| meena | how are you trying to build cloud-init, etc… | 12:06 |
| otubo | rharper: quick question (because we talked about this last time): regarding BZ#1748015, I managed to fix this by adding "ExecStartPost=/usr/bin/systemctl try-restart NetworkManager.service" to cloud-final.service. | 12:21 |
| otubo | (I thought the bot would resolv bugzillas as well :-) - [cloud-init][RHEL7] /etc/resolv.conf lose config after reboot (initial instance is ok) - https://bugzilla.redhat.com/show_bug.cgi?id=1748015 | 12:22 |
| ubot5 | bugzilla.redhat.com bug 1748015 in cloud-init "[cloud-init][RHEL7] /etc/resolv.conf lose config after reboot (initial instance is ok)" [High,Assigned] | 12:22 |
| otubo | oh it does :-D | 12:23 |
| otubo | rharper: do you know if this race is seen in other distros as well? Shall I send a pull-request or just keep it downstream? | 12:23 |
| meena | otubo: i'd open a PR. better safe than sorry | 12:25 |
| meena | i can imagine an insigificant update in systemd to create this race where it wasn't there previously | 12:25 |
| otubo | meena: true that! | 12:26 |
| hkominos | hi meena!. Yes python 2.7 has been dropped (correctly in my ipinion). SO the stock Centos8 image that comes from upstream here https://cloud.centos.org/centos/8/x86_64/images/ has cloud init 18,5 installed. However a dnf update shows for example that 19.2 is available BUThttp://cpaste.info/29lw/. In my case even I have a prebuild 19.4 that I use in | 12:48 |
| hkominos | our centos7 images which works fine but i see even more dependancies missing http://cpaste.info/29lx/. So i am trying to figure out if is a packaging issue or a cloud-init issue. (or other) | 12:48 |
| meena | seems pretty badly packaged | 13:01 |
| hkominos | so 19.2 is hosted upstream but cannot download it. 19.4 is packages and hosted by me. Dnf shows dependancy errros BUT all the binaries are actually visible by rpm -qa . So does that leave me with a badly build centos8 upstream image ? | 13:20 |
| hkominos | I just feel that the 18.5 cloud init might be causing problmes | 13:24 |
| meena | hkominos: you have more than one cloud-init installed? | 13:32 |
| hkominos | no. | 13:32 |
| hkominos | only 18.5 is installed in the upstream image. I am simply unable to upgrade to any other binary | 13:33 |
| meena | i wouldn't exactly call a bunch of python files strewn over your filesystem a "binary" | 13:34 |
| hkominos | ? what do you mean ? | 13:36 |
| meena | hkominos: if cloud-init was a self-contained binary, it might be easier to update. | 13:37 |
| meena | welcome back hkominos. i hope all your sorrows are taken care of, since we last met. | 13:39 |
| hkominos | hahahahaha. | 13:39 |
| hkominos | naaaaaah still things do not work. THat is why I turn to IRC for help | 13:40 |
| meena | hkominos: how did you build 19.4? | 13:42 |
| meena | how was the 19.2 package built? | 13:42 |
| hkominos | of course I did not document all this :P. It looks like I used a spec file for the older 18.5 cloud init and just changed the binary that it pointed to. to build 19.2 | 13:53 |
| hkominos | http://cpaste.info/29m7/ | 13:53 |
| hkominos | like so | 13:53 |
| meena | hkominos: my company proxy doesn't trust cPaste.info | 13:54 |
| hkominos | ok. attempt2 | 13:54 |
| hkominos | https://pastebin.com/hNCwe1Xk | 13:56 |
| hkominos | perhaps ? | 13:56 |
| hkominos | ok. so the spec specifies python requirements | 14:00 |
| hkominos | that I do have. But it cant find them | 14:00 |
| meena | hkominos: have you seen https://github.com/canonical/cloud-init/blob/master/packages/redhat/cloud-init.spec.in ? | 14:03 |
| meena | missing… things… are replaced with: https://github.com/canonical/cloud-init/blob/master/packages/brpm | 14:05 |
| meena | you can call it from the Makefile | 14:05 |
| meena | make rpm DISTRO=redhat | 14:05 |
| hkominos | first time i see this! | 14:35 |
| hkominos | Will get on it right away. thx | 14:41 |
| rharper | otubo: I suspect that if Ubuntu images only had network-manager we might see the same issue | 15:59 |
| rharper | otubo: would appreciate your thoughts on https://github.com/canonical/cloud-init/pull/226 | 15:59 |
| blackboxsw | meena: thanks for being so active | 16:36 |
| blackboxsw | it adds a lot to the cloud-init project | 16:37 |
| hkominos | meena I will work on this tomorrow. But even with the spec file that you provided i get tricky issues. f.e. https://pastebin.com/tBgPrXBK | 16:52 |
| hkominos | Do i need python 2.7 to build cloud init ? | 16:53 |
| rharper | hkominos: for centos7, at this time, 2.7 is required, for python3 in centos8 etc, there's a branch with a modified spec file; that work has not yet landed, but you can get a look at those changes here: https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/368845 | 17:01 |
| hkominos | thx! | 17:03 |
| AnhVoMSFT | rharper: just checking if there is anything else we need for this PR: https://github.com/canonical/cloud-init/pull/54 | 18:55 |
| powersj | AnhVoMSFT, I don't think there is - we need to loop back and get that as part of our next SRU, but Chad can correct me if I am mistaken | 18:57 |
| rharper | AnhVoMSFT: no, powersj is right; we can land that and it would get picked up in the next SRU | 19:25 |
| AnhVoMSFT | thanks rharper powersj | 19:33 |
| meena | blackboxsw: 💜 | 21:13 |
| meena | btw, any of you seen ausfestivus issue? i was able to pin-point the exception. but not the reason why it's happening. | 21:14 |
| ausfestivus | Do you sleep meena? :-p | 21:17 |
| meena | ausfestivus: between 23:00 and hopefully 7… | 21:19 |
| meena | but my baby daughter is still pretty unpredictable. | 21:20 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!