[11:29] <crandon> Odd_Bloke: Hi Again! So I rather recreated the issue locally. Here's the cloud-init.log: https://pastebin.com/33iVf0pb As a reminder: the ntp config from user-data not getting populated to chrony.conf
=== meetingology` is now known as meetingology
=== vrubiolo1 is now known as vrubiolo
=== cpaelzer__ is now known as cpaelzer
[17:22] <blackboxsw> crandon Odd_Bloke is off today. I see you are on cloud-init 18.5 I'm wondering two things, would you be able to run and paste `sudo cloud-init query userdata` be wary if you have secrets/passwords defined in your user data
[17:23] <blackboxsw> also, it may make a difference as a lot has changes since 18.5, we have a copr repo up with some rpms built from tip of upstream cloud-init master here that may assist in confirming if this a bug that has already been resolved https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/
[17:23] <blackboxsw> crandon: ^
[17:23] <blackboxsw> as well
=== vrubiolo1 is now known as vrubiolo
[18:45] <AnhVoMSFT> @rharper what are the potential ramifications of moving mounts to config stage?
[18:45] <AnhVoMSFT> One thing that comes to mind would be modules such as write_files won't be able to use mounts indicated in mounts unless write_files is moved to config stage as well
[18:55] <rharper> AnhVoMSFT: yeah, also, folks can mount up things like /home or whereever write_files might want to send things;
[19:03] <crandon> blackboxsw: Hi! Thanks for taking this over. I checked and the command you've sent prints out the user-data file content.
[19:07] <crandon> blackboxsw: I'm updating now cloud-init to see if the newer version behaves differently.
[19:08] <rharper> crandon: can you confirm if you have /etc/cloud/templates/chrony.conf.*.tmpl ?
[19:09] <AnhVoMSFT> actually in Ubuntu cloud images today (at least the azure ones) mounts module comes after write_files, so that point is moot anyway
[19:10] <rharper> ah, in this centos7 cloud image, I don't see 'ntp' in /etc/cloud/cloud.cfg
[19:10] <crandon> rharper: I do, but: there's no chrony.conf.centos.tmpl (only fedora and rhel, plus some other non relevant ones)
[19:10] <rharper> ah, centos should use the rhel tmpl
[19:11] <rharper> in your paste earlier I don't see cc_ntp   ...   and your user-data has ntp: {}  at min in there?
[19:11] <crandon> Nope, there's no ntp in /etc/cloud/cloud.cfg
[19:12] <crandon> So you're saying the ntp module is not enabled, hence the problem?
[19:12] <rharper> crandon: right, so they've left the ntp config module off by default
[19:12] <rharper> crandon: yes
[19:12] <rharper> you can manually force to run it like:  cloud-init single --name cc_ntp
[19:13] <rharper> assuming you've already supplied ntp:  {} config in your user-data;  if not, you can do it in a separate file,  cloud-init --file my_conf.cfg single --name cc_ntp;  that should run it
[19:13] <crandon> Grrr... why would one do that I wonder... I'm using a cloud image not to have to maintain an image myself. Configuring NTP is something very basic, unless of course the reason is, that they assume, that the VM will sync it's clock with the host, which then can be synced via NTP
[19:14] <crandon> This didn't alter the chrony.conf either: cloud-init single --name cc_ntp
[19:14] <rharper> AnhVoMSFT: right; wont help write files, but it does help with user homedir
[19:14] <rharper> crandon: I think I need to get you more config;  each distro may have a default policy; I though centos preferred chrony; lemme dig up the config
[19:15] <crandon> Ah wait, I updated cloud-init, but didn't reboot yet.
[19:17] <crandon> Hmm "cloud-init single --name cc_ntp" failed.
[19:17] <rharper> looks like there's a bug, chronyd.service not chrony.service
[19:18] <crandon> Yep, that seems to be the issue
[19:18] <rharper> and, if you run in a container, it may not allow you to run;
[19:18] <crandon> But now at least I can see the entries from user-data being populated to the config
[19:18] <rharper> crandon: can you file a bug on the chrony/chronyd  ?
[19:19] <crandon> Sure, but the version is still cloud-init single --name cc_ntp
[19:19] <crandon> Is it still relevant?
[19:19] <rharper> lemme check upstream; one sec
[19:20] <rharper> crandon: yeah, it's still listed as chrony for the service name
[19:21] <rharper> so, bug is relevant;
[19:21] <crandon> Ok, I'll submit a bug. I guess NTP being off by default is something I should check with the RH/CentOS guys, right?
[19:22] <rharper> well, I suspect you're getting systemd-timesyncd by default ; it's a "good enough" one-shot ntp service;
[19:23] <rharper> in this centos7 cloud image, it has systemd-timedatad.service ...   not sure if that's the same;
[19:26] <crandon> Chrony is there by default, I haven't installed it myself also it is enabled by default, while systemd-timedated is not. Regardless, either will require modifying the cloud.cfg to include ntp... Is it possible to modify cloud.cfg via cloud-init :) ?
[19:29] <crandon> The filed bugreport: https://bugs.launchpad.net/cloud-init/+bug/1885952
[19:29] <ubot5> Ubuntu bug 1885952 in cloud-init "Wrong service name used for chrony when restarting service" [Undecided,New]
[19:31] <rharper> crandon: thanks!
[19:33] <crandon> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/chap-kvm_guest_timing_management: "To avoid the problems described above, the Network Time Protocol (NTP) should be configured on the host and the guest virtual machines. "
[19:33] <crandon> So it seems RedHat suggests using NTP in guest eventhough: "KVM avoids these issues by providing guest virtual machines with a paravirtualized clock (kvm-clock)"
[19:35] <crandon> rharper: Well, I guess there isn't much we can do for now. I'll try to submit a bug/feature request to RH to include the NTP module by default. I wonder if they left it out because of this bug...(which would not be nice without raising the bug themselves...)
[19:37] <rharper> right, typically modern hosts and guests use kvmclock and a paravirtual interface to keep the guest clock up-to-date;  however;  ntp in the guest can help with higher resolution data and drift, https://opensource.com/article/17/6/timekeeping-linux-vms
[19:38] <rharper> crandon: I played around with getting ptp bits enabled, but never finished up that work;
[19:44] <crandon> Well, I guess for new, I'll either create a custom image with ntp enabled in cloud.cfg and the service name fixed (or systemd-timesyncd being used), or just configure chrony via ansible after deployment. I guess I'll do the later, otherwise I'll have to maintain the image for the customer, which I'd like to avoid....
[19:45] <crandon> rharper: Thanks for your and also blackboxsw' and Odd_Bloke's help!
[19:45] <rharper> crandon: yw
[19:46] <crandon> A small compliment if you allow, is that there aren't that many IRC communities around, with this level of activeness/support. Thx again.
[19:57] <rharper> crandon: =) Thank you
[20:56] <rharper> paride: cloud_tests lxd bits don't seem to work with lxd 4.0 ... I'm fighting the image export;  is this known ?