[07:29] <ausfestivus> whats the best way to get some help with a cloud-init problem here?
[07:32] <meena> ausfestivus: asking a question is a good start.
[07:33] <ausfestivus> lol
[07:33] <ausfestivus> sure
[07:34] <meena> n.b.: most devs (the people with @ in front of their nick) are somewhere in North America… so if us, mere mortals cannot help, you'll have to wait for them to wake up.
[07:34] <ausfestivus> im trying to debug why a script is not running when its placed in /var/lib/cloud/scripts/per-once
[07:35] <ausfestivus> on ubuntu  18.04 on azure as part  of a  scale set
[07:35] <ausfestivus> the script IS in the above path
[07:35] <ausfestivus> root@cptadotfvm000000:/var/lib/cloud/scripts/per-once# lsinstall-ado-agent.sh
[07:36] <ausfestivus> root@cptadotfvm000000:/var/lib/cloud/scripts/per-once# lsinstall-ado-agent.sh
[07:36] <ausfestivus> apols for the formatting
[07:36] <ausfestivus> im following the steps at https://cloudinit.readthedocs.io/en/latest/topics/faq.html#how-can-i-debug-my-user-data
[07:36] <ausfestivus> to debug the user data
[07:37] <meena> we've recently improved the documentation on per-X, or so i hope… i wonder if it's released on Read The Docs…
[07:38] <ausfestivus> when I run the `cloud-init query userdata > user-data.yaml` command, I get a bunch of python gunk. I suspect because the VM has python2 and cloud-init needs python3
[07:39] <meena> ausfestivus: that shouldn't really matter. cloud-init should be using the correct python.
[07:39] <meena> ausfestivus: can you pastebin the python gunk somewhere, please?
[07:41] <ausfestivus> standby
[07:41] <meena> i can't do anything else
[07:41] <meena> well, i could sleep, but i don't think it's appreciated at $customer's office.
[07:42] <meena> unless you have 20 years seniority…
[07:42] <ausfestivus> whats the best way to paste console output in here?
[07:42] <ausfestivus> webclient here
[07:43] <ausfestivus> ```
[07:43] <meena> ausfestivus: you copy/paste the output to your favourite pastebin… or gist.github.com or whatnot, and then post the link here
[07:43] <ausfestivus> ```cptuser@cptadotfvm000000:/tmp$ sudo cloud-init query userdataTraceback (most recent call last):  File "/usr/bin/cloud-init", line 11, in <module>    load_entry_point('cloud-init==19.4', 'console_scripts', 'cloud-init')()  File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 893, in main    get_uptime=True, func=functor, args=(name,
[07:43] <ausfestivus> args))  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2558, in log_time    ret = func(*args, **kwargs)  File "/usr/lib/python3/dist-packages/cloudinit/cmd/query.py", line 124, in handle_args    instance_data['userdata'] = util.load_file(user_data_fn)  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1372, in load_file
[07:43] <ausfestivus> return decode_binary(contents)  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 146, in decode_binary    return blob.decode(encoding)UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte```
[07:43] <ausfestivus> yea okay
[07:43] <meena> ausfestivus: not like this… :P
[07:44] <ausfestivus> https://gist.github.com/ausfestivus/cdd1635b9340cb2f61ee83eed071f04e
[07:44] <ausfestivus> ^^ python  gunk
[07:47] <meena> ausfestivus: can you run `file /var/lib/cloud/scripts/per-once/install-ado-agent.sh`
[07:48] <ausfestivus> ```cptuser@cptadotfvm000000:/tmp$ file /var/lib/cloud/scripts/per-once/install-ado-agent.sh/var/lib/cloud/scripts/per-once/install-ado-agent.sh: Bourne-Again shell script, ASCII text executable```
[07:48] <ausfestivus> the script is fine
[07:49] <ausfestivus> if I run it from there it does the job fine
[07:49] <meena> ausfestivus: it would be nice to know which script it's complaining about
[07:49] <ausfestivus> hence why im  trying  to get some debug info from cloud-init
[07:49] <ausfestivus> hence the python gunk after following the debug steps in the docs
[07:50] <meena> oooh, waaait… is it userdata it's complaining about?
[07:50] <ausfestivus> it doesnt complain
[07:50] <meena> ausfestivus: python-gunking about
[07:50] <ausfestivus> oh
[07:50] <ausfestivus> `sudo cloud-init query userdat`
[07:50] <ausfestivus> `sudo cloud-init query userdata`
[07:51] <meena> that works?
[07:51] <ausfestivus> no
[07:51] <ausfestivus> gives the gunk
[07:51] <meena> i assumed from # that you're root.
[07:54] <meena> ausfestivus: it's odd, according to docs, gzip compressed userdata should just be decompressed…
[07:54] <meena> and your userdata is.
[07:54] <meena> here's a cut from /usr/share/mime/magic:
[07:55] <meena> [50:application/gzip]
[07:55] <meena> >0=^@^B^_<8b>
[07:55] <meena> that means, gzip is identified by a magic byte sequence of 0x8b
[07:58] <meena> ausfestivus: you can try play around with https://cloudinit.readthedocs.io/en/latest/topics/analyze.html — or read the logs yourself, to figure out if this same behaviour happens on boot, cuz if it does that's a serious bug (or misconfiguration, or misunderstanding)
[08:03] <ausfestivus> are you saying  thefile in per-once should be mime-encoded?
[08:11] <meena> ausfestivus: no, i'm saying that your userdata is gzipped. that's what "python gunk" is saying. and for reasons unclear to me, cloud-init is choking on that, rather than ungzipping it, before trying to decode it as text
[08:11] <meena> the only thing i can imagine going wrong is double gzip encoding, or, a bug.
[08:13] <ausfestivus> afk dinner
[09:27] <ausfestivus> @meena how can I debug this problem further? Any tips?
[09:28] <ausfestivus> the debug tools in the FAQ are not working for a reason I cant decipher
[09:29] <meena> ausfestivus: pastebin your logfile, and i can help take a look. also pastebin your /var/lib/cloud-init/instance/userdata (or whatever the actual real path is)
[09:31] <ausfestivus> you want both log files?
[09:31] <meena> sure
[09:35] <meena> if you're unsure about your logs, cuz they're a lot, you can cloud-init clean --logs --reboot
[09:35] <meena> to get a new set of fresh logs
[09:47] <ausfestivus> cloud-init-output.log - https://pastebin.com/0CiR0yFEcloud-init.log - https://pastebin.com/iG6Up8SP
[09:47] <ausfestivus> grrr
[09:47] <ausfestivus> standby
[09:48] <ausfestivus> cloud-init-output.log - https://pastebin.com/0CiR0yFE
[09:48] <ausfestivus> cloud-init.log - https://pastebin.com/iG6Up8SP
[09:48] <ausfestivus> I have a userdata.txt and a userdata.txt.i both are binary data
[09:49] <ausfestivus> err correction
[09:49] <ausfestivus> .i isnt
[09:50] <ausfestivus> userdata.txt.i https://pastebin.com/qacVGm4p
[10:03] <meena> ausfestivus: how many boots is this? did you run clean --logs --reboot?
[11:16] <ausfestivus> just one boot, didnt run clean --logs --reboot
[11:47] <hkominos> Good morning guys. I wanted to ask the channel if cloud init works with only python3 installed. I am trying to build centos8 with cloud-init but I am facing issues. (even the upstream image of centos8 is showing missing deps for cloud-init)
[12:03] <meena> hkominos: the latest version has dropped python 2.7 support
[12:06] <meena> hkominos: but, you may wanna describe exactly what problems you're facing… or pastebin some stack-trace or similar.
[12:06] <meena> how are you trying to build cloud-init, etc…
[12:21] <otubo> rharper: quick question (because we talked about this last time): regarding BZ#1748015, I managed to fix this by adding "ExecStartPost=/usr/bin/systemctl try-restart NetworkManager.service" to cloud-final.service.
[12:22] <otubo> (I thought the bot would resolv bugzillas as well :-) - [cloud-init][RHEL7] /etc/resolv.conf lose config after reboot (initial instance is ok) - https://bugzilla.redhat.com/show_bug.cgi?id=1748015
[12:23] <otubo> oh it does :-D
[12:23] <otubo> rharper: do you know if this race is seen in other distros as well? Shall I send a pull-request or just keep it downstream?
[12:25] <meena> otubo: i'd open a PR. better safe than sorry
[12:25] <meena> i can imagine an insigificant update in systemd to create this race where it wasn't there previously
[12:26] <otubo> meena: true that!
[12:48] <hkominos> hi meena!. Yes python 2.7 has been dropped (correctly in my ipinion). SO the stock Centos8 image that comes from upstream here https://cloud.centos.org/centos/8/x86_64/images/ has cloud init 18,5 installed. However a dnf update shows for example that 19.2 is available BUThttp://cpaste.info/29lw/. In my case even I have a prebuild 19.4 that I use in
[12:48] <hkominos> our centos7 images which works fine but i see even more dependancies missing http://cpaste.info/29lx/. So i am trying to figure out if is a packaging issue or a cloud-init issue. (or other)
[13:01] <meena> seems pretty badly packaged
[13:20] <hkominos> so 19.2 is hosted upstream but cannot download it. 19.4 is packages and hosted by me. Dnf shows dependancy errros BUT all the binaries are actually visible by rpm -qa . So does that leave me with a badly build centos8 upstream image ?
[13:24] <hkominos> I just feel that the 18.5 cloud init might be causing problmes
[13:32] <meena> hkominos: you have more than one cloud-init installed?
[13:32] <hkominos> no.
[13:33] <hkominos> only 18.5 is installed in the upstream image. I am simply unable to upgrade to any other binary
[13:34] <meena> i wouldn't exactly call a bunch of python files strewn over your filesystem a "binary"
[13:36] <hkominos> ? what do you mean ?
[13:37] <meena> hkominos: if cloud-init was a self-contained binary, it might be easier to update.
[13:39] <meena> welcome back hkominos. i hope all your sorrows are taken care of, since we last met.
[13:39] <hkominos> hahahahaha.
[13:40] <hkominos> naaaaaah still things do not work. THat is why I turn to IRC for help
[13:42] <meena> hkominos: how did you build 19.4?
[13:42] <meena> how was the 19.2 package built?
[13:53] <hkominos> of course I did not document all this :P. It looks like I used a spec file for the older 18.5 cloud init and just changed the binary that it pointed to. to build 19.2
[13:53] <hkominos> http://cpaste.info/29m7/
[13:53] <hkominos> like so
[13:54] <meena> hkominos: my company proxy doesn't trust  cPaste.info
[13:54] <hkominos> ok. attempt2
[13:56] <hkominos> https://pastebin.com/hNCwe1Xk
[13:56] <hkominos> perhaps ?
[14:00] <hkominos> ok. so the spec specifies python requirements
[14:00] <hkominos> that I do have. But it cant find them
[14:03] <meena> hkominos: have you seen https://github.com/canonical/cloud-init/blob/master/packages/redhat/cloud-init.spec.in ?
[14:05] <meena> missing… things… are replaced with: https://github.com/canonical/cloud-init/blob/master/packages/brpm
[14:05] <meena> you can call it from the Makefile
[14:05] <meena> make rpm DISTRO=redhat
[14:35] <hkominos> first time i see this!
[14:41] <hkominos> Will get on it right away. thx
[15:59] <rharper> otubo: I suspect that if Ubuntu images only had network-manager we might see the same issue
[15:59] <rharper> otubo: would appreciate your thoughts on https://github.com/canonical/cloud-init/pull/226
[16:36] <blackboxsw> meena: thanks for being so active
[16:37] <blackboxsw> it adds a lot to the cloud-init project
[16:52] <hkominos> meena I will work on this tomorrow. But even with the spec file that you provided i get tricky issues. f.e. https://pastebin.com/tBgPrXBK
[16:53] <hkominos> Do i need python 2.7 to build cloud init ?
[17:01] <rharper> hkominos: for centos7, at this time, 2.7 is required, for python3 in centos8 etc, there's a branch with a modified spec file; that work has not yet landed, but you can get a look at those changes here: https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/368845
[17:03] <hkominos> thx!
[18:55] <AnhVoMSFT> rharper: just checking if there is anything else we need for this PR: https://github.com/canonical/cloud-init/pull/54
[18:57] <powersj> AnhVoMSFT, I don't think there is - we need to loop back and get that as part of our next SRU, but Chad can correct me if I am mistaken
[19:25] <rharper> AnhVoMSFT:  no, powersj is right; we can land that and it would get picked up in the next SRU
[19:33] <AnhVoMSFT> thanks rharper powersj
[21:13] <meena> blackboxsw: 💜
[21:14] <meena> btw, any of you seen ausfestivus issue? i was able to pin-point the exception. but not the reason why it's happening.
[21:17] <ausfestivus> Do you sleep meena? :-p
[21:19] <meena> ausfestivus: between 23:00 and hopefully 7…
[21:20] <meena> but my baby daughter is still pretty unpredictable.