[07:29] whats the best way to get some help with a cloud-init problem here? [07:32] ausfestivus: asking a question is a good start. [07:33] lol [07:33] sure [07:34] n.b.: most devs (the people with @ in front of their nick) are somewhere in North America… so if us, mere mortals cannot help, you'll have to wait for them to wake up. [07:34] im trying to debug why a script is not running when its placed in /var/lib/cloud/scripts/per-once [07:35] on ubuntu 18.04 on azure as part of a scale set [07:35] the script IS in the above path [07:35] root@cptadotfvm000000:/var/lib/cloud/scripts/per-once# lsinstall-ado-agent.sh [07:36] root@cptadotfvm000000:/var/lib/cloud/scripts/per-once# lsinstall-ado-agent.sh [07:36] apols for the formatting [07:36] im following the steps at https://cloudinit.readthedocs.io/en/latest/topics/faq.html#how-can-i-debug-my-user-data [07:36] to debug the user data [07:37] we've recently improved the documentation on per-X, or so i hope… i wonder if it's released on Read The Docs… [07:38] when I run the `cloud-init query userdata > user-data.yaml` command, I get a bunch of python gunk. I suspect because the VM has python2 and cloud-init needs python3 [07:39] ausfestivus: that shouldn't really matter. cloud-init should be using the correct python. [07:39] ausfestivus: can you pastebin the python gunk somewhere, please? [07:41] standby [07:41] i can't do anything else [07:41] well, i could sleep, but i don't think it's appreciated at $customer's office. [07:42] unless you have 20 years seniority… [07:42] whats the best way to paste console output in here? [07:42] webclient here [07:43] ``` [07:43] ausfestivus: you copy/paste the output to your favourite pastebin… or gist.github.com or whatnot, and then post the link here [07:43] ```cptuser@cptadotfvm000000:/tmp$ sudo cloud-init query userdataTraceback (most recent call last): File "/usr/bin/cloud-init", line 11, in load_entry_point('cloud-init==19.4', 'console_scripts', 'cloud-init')() File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 893, in main get_uptime=True, func=functor, args=(name, [07:43] args)) File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2558, in log_time ret = func(*args, **kwargs) File "/usr/lib/python3/dist-packages/cloudinit/cmd/query.py", line 124, in handle_args instance_data['userdata'] = util.load_file(user_data_fn) File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1372, in load_file [07:43] return decode_binary(contents) File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 146, in decode_binary return blob.decode(encoding)UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte``` [07:43] yea okay [07:43] ausfestivus: not like this… :P [07:44] https://gist.github.com/ausfestivus/cdd1635b9340cb2f61ee83eed071f04e [07:44] ^^ python gunk [07:47] ausfestivus: can you run `file /var/lib/cloud/scripts/per-once/install-ado-agent.sh` [07:48] ```cptuser@cptadotfvm000000:/tmp$ file /var/lib/cloud/scripts/per-once/install-ado-agent.sh/var/lib/cloud/scripts/per-once/install-ado-agent.sh: Bourne-Again shell script, ASCII text executable``` [07:48] the script is fine [07:49] if I run it from there it does the job fine [07:49] ausfestivus: it would be nice to know which script it's complaining about [07:49] hence why im trying to get some debug info from cloud-init [07:49] hence the python gunk after following the debug steps in the docs [07:50] oooh, waaait… is it userdata it's complaining about? [07:50] it doesnt complain [07:50] ausfestivus: python-gunking about [07:50] oh [07:50] `sudo cloud-init query userdat` [07:50] `sudo cloud-init query userdata` [07:51] that works? [07:51] no [07:51] gives the gunk [07:51] i assumed from # that you're root. [07:54] ausfestivus: it's odd, according to docs, gzip compressed userdata should just be decompressed… [07:54] and your userdata is. [07:54] here's a cut from /usr/share/mime/magic: [07:55] [50:application/gzip] [07:55] >0=^@^B^_<8b> [07:55] that means, gzip is identified by a magic byte sequence of 0x8b [07:58] ausfestivus: you can try play around with https://cloudinit.readthedocs.io/en/latest/topics/analyze.html — or read the logs yourself, to figure out if this same behaviour happens on boot, cuz if it does that's a serious bug (or misconfiguration, or misunderstanding) [08:03] are you saying thefile in per-once should be mime-encoded? [08:11] ausfestivus: no, i'm saying that your userdata is gzipped. that's what "python gunk" is saying. and for reasons unclear to me, cloud-init is choking on that, rather than ungzipping it, before trying to decode it as text [08:11] the only thing i can imagine going wrong is double gzip encoding, or, a bug. [08:13] afk dinner [09:27] @meena how can I debug this problem further? Any tips? [09:28] the debug tools in the FAQ are not working for a reason I cant decipher [09:29] ausfestivus: pastebin your logfile, and i can help take a look. also pastebin your /var/lib/cloud-init/instance/userdata (or whatever the actual real path is) [09:31] you want both log files? [09:31] sure [09:35] if you're unsure about your logs, cuz they're a lot, you can cloud-init clean --logs --reboot [09:35] to get a new set of fresh logs [09:47] cloud-init-output.log - https://pastebin.com/0CiR0yFEcloud-init.log - https://pastebin.com/iG6Up8SP [09:47] grrr [09:47] standby [09:48] cloud-init-output.log - https://pastebin.com/0CiR0yFE [09:48] cloud-init.log - https://pastebin.com/iG6Up8SP [09:48] I have a userdata.txt and a userdata.txt.i both are binary data [09:49] err correction [09:49] .i isnt [09:50] userdata.txt.i https://pastebin.com/qacVGm4p [10:03] ausfestivus: how many boots is this? did you run clean --logs --reboot? [11:16] just one boot, didnt run clean --logs --reboot [11:47] Good morning guys. I wanted to ask the channel if cloud init works with only python3 installed. I am trying to build centos8 with cloud-init but I am facing issues. (even the upstream image of centos8 is showing missing deps for cloud-init) [12:03] hkominos: the latest version has dropped python 2.7 support [12:06] hkominos: but, you may wanna describe exactly what problems you're facing… or pastebin some stack-trace or similar. [12:06] how are you trying to build cloud-init, etc… [12:21] rharper: quick question (because we talked about this last time): regarding BZ#1748015, I managed to fix this by adding "ExecStartPost=/usr/bin/systemctl try-restart NetworkManager.service" to cloud-final.service. [12:22] (I thought the bot would resolv bugzillas as well :-) - [cloud-init][RHEL7] /etc/resolv.conf lose config after reboot (initial instance is ok) - https://bugzilla.redhat.com/show_bug.cgi?id=1748015 [12:22] bugzilla.redhat.com bug 1748015 in cloud-init "[cloud-init][RHEL7] /etc/resolv.conf lose config after reboot (initial instance is ok)" [High,Assigned] [12:23] oh it does :-D [12:23] rharper: do you know if this race is seen in other distros as well? Shall I send a pull-request or just keep it downstream? [12:25] otubo: i'd open a PR. better safe than sorry [12:25] i can imagine an insigificant update in systemd to create this race where it wasn't there previously [12:26] meena: true that! [12:48] hi meena!. Yes python 2.7 has been dropped (correctly in my ipinion). SO the stock Centos8 image that comes from upstream here https://cloud.centos.org/centos/8/x86_64/images/ has cloud init 18,5 installed. However a dnf update shows for example that 19.2 is available BUThttp://cpaste.info/29lw/. In my case even I have a prebuild 19.4 that I use in [12:48] our centos7 images which works fine but i see even more dependancies missing http://cpaste.info/29lx/. So i am trying to figure out if is a packaging issue or a cloud-init issue. (or other) [13:01] seems pretty badly packaged [13:20] so 19.2 is hosted upstream but cannot download it. 19.4 is packages and hosted by me. Dnf shows dependancy errros BUT all the binaries are actually visible by rpm -qa . So does that leave me with a badly build centos8 upstream image ? [13:24] I just feel that the 18.5 cloud init might be causing problmes [13:32] hkominos: you have more than one cloud-init installed? [13:32] no. [13:33] only 18.5 is installed in the upstream image. I am simply unable to upgrade to any other binary [13:34] i wouldn't exactly call a bunch of python files strewn over your filesystem a "binary" [13:36] ? what do you mean ? [13:37] hkominos: if cloud-init was a self-contained binary, it might be easier to update. [13:39] welcome back hkominos. i hope all your sorrows are taken care of, since we last met. [13:39] hahahahaha. [13:40] naaaaaah still things do not work. THat is why I turn to IRC for help [13:42] hkominos: how did you build 19.4? [13:42] how was the 19.2 package built? [13:53] of course I did not document all this :P. It looks like I used a spec file for the older 18.5 cloud init and just changed the binary that it pointed to. to build 19.2 [13:53] http://cpaste.info/29m7/ [13:53] like so [13:54] hkominos: my company proxy doesn't trust cPaste.info [13:54] ok. attempt2 [13:56] https://pastebin.com/hNCwe1Xk [13:56] perhaps ? [14:00] ok. so the spec specifies python requirements [14:00] that I do have. But it cant find them [14:03] hkominos: have you seen https://github.com/canonical/cloud-init/blob/master/packages/redhat/cloud-init.spec.in ? [14:05] missing… things… are replaced with: https://github.com/canonical/cloud-init/blob/master/packages/brpm [14:05] you can call it from the Makefile [14:05] make rpm DISTRO=redhat [14:35] first time i see this! [14:41] Will get on it right away. thx [15:59] otubo: I suspect that if Ubuntu images only had network-manager we might see the same issue [15:59] otubo: would appreciate your thoughts on https://github.com/canonical/cloud-init/pull/226 [16:36] meena: thanks for being so active [16:37] it adds a lot to the cloud-init project [16:52] meena I will work on this tomorrow. But even with the spec file that you provided i get tricky issues. f.e. https://pastebin.com/tBgPrXBK [16:53] Do i need python 2.7 to build cloud init ? [17:01] hkominos: for centos7, at this time, 2.7 is required, for python3 in centos8 etc, there's a branch with a modified spec file; that work has not yet landed, but you can get a look at those changes here: https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/368845 [17:03] thx! [18:55] rharper: just checking if there is anything else we need for this PR: https://github.com/canonical/cloud-init/pull/54 [18:57] AnhVoMSFT, I don't think there is - we need to loop back and get that as part of our next SRU, but Chad can correct me if I am mistaken [19:25] AnhVoMSFT: no, powersj is right; we can land that and it would get picked up in the next SRU [19:33] thanks rharper powersj [21:13] blackboxsw: 💜 [21:14] btw, any of you seen ausfestivus issue? i was able to pin-point the exception. but not the reason why it's happening. [21:17] Do you sleep meena? :-p [21:19] ausfestivus: between 23:00 and hopefully 7… [21:20] but my baby daughter is still pretty unpredictable.