[06:03] <mk88> Hi All we are trying to customize VM on azure platform,each time install custom packages and run scripts on that VM we get different behaviour.
[06:05] <mk88> It seems cloud-init is messing with some of the processes of VM on First-boot. Is there a way that we can check if that VM is properly booted and then we can trigger cloud-init ?
[06:11] <mk88> Anyone there ?
[06:28] <mk88> ?
[07:05] <prologic> is there some better way to debug why my cloud-init setup in vSphere isn't working reliably? About 1 out of 3 times it looks like none of my user-data gets executed at all. not even something as simple as the hostname
[07:06] <prologic> But neither cloud-init.log nor cloud-init-output.log exhibit any obvious errors or failures
[07:06] <prologic> I have the failed vm up and running now, so if anyone wants me to run some commands against it to figure out why it's in sthis state LMK
[07:07] <prologic> I've confirmed for example that the userdadta is there (after the fact) with vmware-rpctool "info-get guestinfo.userdata" | base64 -d
[07:08] <prologic> I basically have a shell script at the moment that destroys, and re-creates the VM from a template with cloud-init userdata and it fails about 1/3 in this weird state
[07:46] <meena> mk88: how, exactly, is cloud-init messing with your setup?
[07:47] <meena> prologic: how does it fail? do you have debugging enabled so the logs are full of… well, debugging info
[07:48] <prologic> it doesn't run any of the user-data at all
[07:48] <prologic> is there a bit of config I can add to /etc/cloud/cloud.cfg to increase verbose/debug logging?
[07:52] <prologic> I'm a bit more interested in debugging this failed VM that didn't run any of the cloud-init userdata though
[07:52] <prologic> So any helpful suggestions there would be nice, happy to poke around with some guidenace, I'm alll out of ideas myself :/
[07:52] <prologic> Like what possible reasons would none of the userdata ever run (at all) without ovbious errors 1 out of 3 times?
[07:53] <prologic> but work perfectly the other 2 times
[08:04] <prologic> e.g: cloud-init status reports "done" and not "error" in this case
[08:16] <mk88> @meena So we have some processes that run on firstboot like db creation and other things hat we need to do. Also we have a requirement to call ansible after the firstboot completes. So we are using the cloud-init to do the ansible invocation.But what is happening on each VM creation we get different behavior though there are no script level
[08:16] <mk88> changes.
[08:24] <meena> prologic: usually, the reason is that the meta data service cannot be reached, often caused by some kind of race between the VM and the service
[08:24] <mk88> @meena so there any way that we can keep it disabled till first boot completes and then enable it later when firstboot completes to make ansible invocation ?
[08:26] <meena> mk88: there probably is, but then you'd run in different trouble
[08:28] <meena> prologic: we usually have an example config file for logging or debugging, it's in /etc/cloud.cfg.d (i think)
[08:29] <meena> in /etc/cloud/cloud.cfg.d/
[08:29] <prologic> meena this is my thoughts too, a 'race" of some kind
[08:29] <prologic> let me see what's in there right now...
[08:29] <prologic> This is based off an Ubuntu 20.04 image
[08:29] <meena> depending on the distro that config is disabled (in FreeBSD for instance)
[08:30] <prologic> I see
[08:30] <meena> i haven't used Linux since the start of the pandemic lol
[08:30] <prologic> I assume you mean 05_logging.cfg ?
[08:30] <prologic> hahah!
[08:30] <meena> yes
[08:31] <prologic> it would appear DEBUG level logging is already enabled here
[08:31] <prologic> if I'm reading thie weird looking .ini config right :)
[08:31] <meena> but is the file enabled in cloud.cfg?
[08:32] <prologic> ahh
[08:32] <meena> there's an include directive in there
[08:32] <meena> or there should be
[08:33] <prologic> hmm
[08:33] <prologic> there is not that I can see
[08:33] <meena> yeah, there's not, it happens automatically
[08:34] <prologic> ahh okay
[08:34] <meena> so, if debug logging is enabled, how come you don't see any logs
[08:34] <prologic> so I should have debug logging
[08:34] <meena> you should
[08:34] <prologic> so what could I look for to determine if we're seeing a race betwene the metadata service and the vm?
[08:35] <meena> maybe syslog has info?
[08:35] <meena> as to why we're not even logging
[08:35] <prologic> I no I see logs, but I don't see anything to explain what I'm seeing
[08:36] <prologic> I basically need some help (I think) in understanding what to look for
[08:36] <prologic> the logs I do see don't explain this weird behaviour
[08:36] <meena> then you have to paste your logs somewhere, and wait until someone more experienced wakes up here to look at them
[08:36] <meena> I'm also just in my phone,while chasing a toddler
[08:37] <meena> so that's quite an impediment
[09:07] <prologic> will the tarball of cloud-init collect-logs be enough?
[09:13] <meena> most likely, given that that is all relevant logs and some other stuff
[09:27] <prologic> kk
[09:27] <prologic> I wish I could figure this out myself :D
[09:28] <prologic> just nothing in there I can identify as :oh!"
=== apollo13_ is now known as apollo13
=== waxfire2 is now known as waxfire
=== falcojr6 is now known as falcojr
=== shalok_ is now known as shalok
[14:24] <smoser> prologic: did you file a bug ?
[15:19] <ajmyyra> btw, who creates the /etc/cloud/cloud.cfg.d/90_dpkg.cfg file in Ubuntu? cloud-init itself or some package script?
[15:19] <ajmyyra> not included in the package atleast, but can't seem to find anything related to it
[15:19] <ajmyyra> besides some test
[15:24] <Odd_Bloke> ajmyyra: It's generated from a debconf prompt, so it's populated on package installation.
[15:24] <Odd_Bloke> (Which happens at image build time in ~all official Ubuntu images. :)
=== powersj_ is now known as powersj
[18:59] <smoser> anyone able to land https://github.com/canonical/cloud-init/pull/837
[18:59] <smoser> i'mhappy with it, but i'm not up to speed on how to adjust user comment or cpy and paste the commit message....
[20:27] <dam_> hi, I am trying to install ubuntu with cloud-init for the first time on a qemu image. The installation fails when running systemd-cat with exit status 3. There is a log in /var/crash but how I do not know how to retrieve it. Is it possible?
[20:32] <Odd_Bloke> dam_: What installation media are you using?
[20:34] <dam_> I use an iso.
[20:35] <dam_> I use the following command line to start installation:
[20:35] <dam_> kvm -no-reboot -m 1024 -smbios type=0,uefi=on -drive file=image.img,format=raw,cache=none,if=virtio -drive file=seed.img,format=raw,cache=none,if=virtio -cdrom ~/ubuntu-cloud-init/ubuntu-20.04.2-live-server-amd64.iso
[20:35] <dam_>  
[20:41] <dam_> I also use curtin to define partitions.
[20:53] <prologic> smoser no not yet, I'm not even sure where/who the bug is with.
[20:53] <prologic> So far it just smells like "racy" behaviour
[20:53] <prologic> destroy/recreate a vm 10 times, and 3 of those times none of the cloud-init userdata gets run at all
[21:26] <Odd_Bloke> dam_: Aha, OK, I think you're hitting an error during a subiquity install?  Or are you specifically passing in cloud-init configuration that isn't working for you?
[21:33] <dam_> I got an error during subiquity install but I do not know why. If I use default config, the install succeed but not when I specify storage.
[21:41] <dam_> Does ubuntu-20.04.2-live-server-amd64.iso supports uefi boot?
[21:44] <Odd_Bloke> dam_: This isn't the best channel to get subiquity support, I'm afraid: cloud-init is only one component of the whole installer; you'll likely have more success in the #ubuntu-server channel.
[21:49] <dam_> OK. I am not familiar with cloud-init, subiquity and the like.
[21:51] <dam_> The documentation of cloud-init mentions that it supports "Bare metal installs". Does it depends on subiquity?
[21:56] <Odd_Bloke> dam_: cloud-init supports running as part of bare metal installs, but it runs inside the booted system: you still need something else to arrange the image for you and pass the configuration into the system.  subiquity does that for Ubuntu installs (so it's more accurate to say that subiquity depends on cloud-init.)
[21:58] <dam_> In the meantime I fixed my issue. I had to create partition for grub so it seems that I was not able to do a uefi install.
[21:58] <dam_> anyway thank you for your support
[22:40] <dave_the_rave> Hello, struggling with a few points in case any one can help pls?
[22:40] <dave_the_rave> 1. write_files is not suitable for creating files in user directories as makes the users home dir owned by root  and runs before create users (on AWS at least) so owner can't be set to the user. Is that right?
[22:40] <dave_the_rave> 2. I'm try to use curl in runcmd, I've set proxy variables but can't make a connection. It works just fine as a regular user. Any ideas?
[22:40] <dave_the_rave> 3. Is there away to make sure a script runs after runcmd is finished?
[22:40] <dave_the_rave> Thanks!
[22:49] <Odd_Bloke> dave_the_rave: 1. is correct, you're running into https://bugs.launchpad.net/cloud-init/+bug/1486113.  See #2 for background on why things are the way they are, and #13 for our recommended workarounds in the meantime.
[22:49] <ubot5> Ubuntu bug 1486113 in cloud-init "write_files runs before users/groups, renders "owner" useless" [Medium,Triaged]
[22:51] <Odd_Bloke> dave_the_rave: For 2., I'm not sure: how are you setting the proxy variables?
[22:56] <dave_the_rave> Odd_Bloke Thanks, that's helpful. /etc/skel is the answer for me. on 2. I was setting them via source /etc/environment and then later in desperation exporting them in the runcmd script but to no avail. I was wondering if there is something that would stop curl working.. I couldn't connect on 443 to github.com.
=== ivo_cavalcante_ is now known as ivo_cavalcante
[23:06] <Odd_Bloke> dave_the_rave: It might be worth replacing your curl command with `env`; that output should go to /var/log/cloud-init-output.log, then you can double-check that your environment variables are getting through.
[23:07] <Odd_Bloke> dave_the_rave: And for 3., what do you mean by "a script"?  Could you just add it as the final runcmd entry?
[23:12] <prologic> Can I link someone (privately) a cloud-init.tar.gz (from cloud-init collect-logs) to help me figure out why I'm having so much trouble getting my userdata to run reliably? If I destroy/re-create my VM, about 1/3 times it fails with no evidence that it ran any of my userdata, the other 2 times it works perfectly. Environment is Packer built image spun up with Terraform on a vSphere hypervisor.