[12:44] <akutz> rharper: Turns out the bug with SSH was much more irritating and difficult to find. The first clue was the Cloud-Init logs showed zero evidence that this system had been booted before, even though I knew it had. That meant this was not a boot post CI upgrade but a boot post CI first install. Turns out the Photon team had introduced CI as a dependency of open-vm-tools, so when the person upgraded the open-vm-tools package, CI was installed for the first
[12:44] <akutz>  time. This meant all of the system's previous SSH keys were removed and then regenerated. falcojr, blackboxsw -- I wonder if this is desired? The OVF DS *was* detected, and thus a new IID, iid-dsovf, the default for that DS, was used, and new instance means the per-instance modules run, including the SSH module. I told the team that this is by design. But should it be? Should the SSH module only overwrite keys previously created by Cloud-Init? Thoughts?
[13:29] <falcojr> it's not common to install cloud-init on an already running machine, so this is a bit of an unusual use case. There is a "ssh_deletekeys" key that could be set False in /etc/cloud/cloud.cfg in some kind of postinst to disable the behavior. 
[13:32] <falcojr> the problem with overwriting keys only created by cloud-init is we don't know the source of this current instance. If it originated from older instance (even if cloud-init wasn't installed), there's a good chance we don't want those keys copied over to the newly launched instance. We REALLY don't want to be leaking private keys
[14:34] <rharper> akutz: interesting.  Were there two datasources the OVF and the new one?  If so, then we likely need to handle in ds-identify what to do when we detect both OVF and VMware, which one wins?  There are two scenarios, 1) when there's an existing instance-id (the ds associated with that instance-id should run 2) if both are present and this is firstboot, no instance-id at all.   (2) will require VMware input on which one should be 
[14:34] <rharper> the default if both are present I think  
[14:35] <akutz> This has nothing to do with the VMware DS. It wasn't activated at any time.
[14:36] <akutz> This was a case of pre-upgrade CI wasn't present. Post-upgrade CI was pulled in because of the new dep added to open-vm-tools. The system also apparently had whatever was required to activate the OVF DS, which upon reboot, told CI "Hey, I'm a DS and I have data, put me in coach, I'm ready to play!", causing CI to enter first-boot, first-instance mode. In turn the SSH module was activated and it erased and regenerated the existing system keys.
[14:36] <akutz> FWIW, the Photon team is removing the dep from open-vm-tools for now while they figure out what to do.
[14:37] <akutz> I also said we could "touch /etc/cloud/cloud-init.disabled" or "mkdir -p /etc/cloud/cloud.cfg.d && echo 'ssh_deletekeys: false' >/etc/cloud/cloud.cfg.d/99_disable-sshd-keygen.cfg" prior to the upgrade.
[14:38] <akutz> falcojr: I see your response up above. I told them the same thing -- that CI shouldn't be installed on a running machine.
[14:41] <akutz> Anyway, I basically indicated to them that adding this dep, while it may be required for a new feature in open-vm-tools, isn't an issue that CI should necessarily solve. The good thing is that this dep isn't part of open-vm-tools upstream, but only on Photon. However, I *am* concerned that if the dep is added to open-vm-tools upstream at some point, or by other distros, people could encounter the same situation.
[19:20] <chadxz> hey there, I'm trying to test a custom cloud-init config that I have placed at `/etc/cloud/cloud.cfg.d/custom.cfg`. I booted an AWS AMI, created that file, and am trying to make sure it does what I expect before I save an AMI of this customized instance. Is there documentation around the correct way to test this?