=== cpaelzer__ is now known as cpaelzer
[14:57] <Odd_Bloke> hggdh: If you're still seeing that NFS issue, a bug report would be appreciated. :)
[16:08] <hggdh> Odd_Bloke: yes, still there. I am trying to reproduce it now, cannot open a bug wit customer data
[16:10] <testing1818282> hello everybody!
[16:12] <testing1818282> currently having a problem with could-init on ec2 using a self built rhel 8 ami. sshd.service always fails on first boot because the host keys are not available but succeeds on second try
[16:13] <testing1818282> sadly, this also causes cloud-init to not print the ssh host key keys on console, the space between 'BEGIN/END SSH HOST KEY KEYS' ist just empty. How can I easily make cloud-init to wait a few seconds beforehand?
[16:31] <Odd_Bloke> hggdh: Understood, let us know if we can help. :)
[16:33] <Odd_Bloke> testing1818282: I would expect cloud-init to be generating those SSH host keys in its default configuration; how are you installing cloud-init into your AMI?  Do you have any configuration?
[16:37] <testing1818282> Odd_Bloke thank for answering. I am creating the ami using packer and have a line in the json spec like this: `"sudo dnf --assumeyes install rsync jq perl cloud-utils-growpart"`. It worked like a charm for months, but I try to switch some of my infrastructure to arm based machines and now stumble upon this. Cloud-init is installed and started
[16:37] <testing1818282> inside the ec2 instance and it also successfully generates those keys. I can find them inside /etc/ssh. But I guess they are generated after it tries to print them to the console
[16:40] <testing1818282> ```[  OK  ] Reached target Login Prompts.[FAILED] Failed to start OpenSSH server daemon.See 'systemctl status sshd.service' for details.[  115.600243] cloud-init[1097]: Cloud-init v. 19.4 running 'modules:config' at Mon, 04 Jan 2021 15:36:36 +0000. Up 115.52 seconds.[  OK  ] Started Apply the settings specified in cloud-config.         Starting
[16:40] <testing1818282> Execute cloud user/final scripts...[  OK  ] Started System Logging Service.[  OK  ] Reached target Multi-User System.         Starting Update UTMP about System Runlevel Changes...[  OK  ] Started Update UTMP about System Runlevel Changes.ci-info: +++++++++++++++++++++Authorized keys from /home/ec2-user/.ssh/authorized_keys for user
[16:40] <testing1818282> ec2-user++++++++++++++++++++++ci-info: +---------+-------------------------------------------------+---------+---------------------------------------------+ci-info: | Keytype |                Fingerprint (md5)                | Options |                   Comment                   |ci-info:
[16:40] <testing1818282> +---------+-------------------------------------------------+---------+---------------------------------------------+ci-info: | ssh-rsa | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |    -    | packer_5ff2df88-947a-5d37-29ff-4ca186ea3f79 |ci-info: | ssh-rsa | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |    -    |                   jenkins
[16:40] <testing1818282> |ci-info: +---------+-------------------------------------------------+---------+---------------------------------------------+<14>Jan  4 15:36:36 ec2:<14>Jan  4 15:36:36 ec2: #############################################################<14>Jan  4 15:36:36 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----<14>Jan  4 15:36:36 ec2: -----END SSH HOST KEY
[16:40] <testing1818282> FINGERPRINTS-----<14>Jan  4 15:36:36 ec2: #############################################################-----BEGIN SSH HOST KEY KEYS----------END SSH HOST KEY KEYS-----[  116.110779] cloud-init[1180]: Cloud-init v. 19.4 running 'modules:final' at Mon, 04 Jan 2021 15:36:36 +0000. Up 115.90 seconds.[  116.112793] cloud-init[1180]: Cloud-init v. 19.4
[16:40] <testing1818282> finished at Mon, 04 Jan 2021 15:36:36 +0000. Datasource DataSourceEc2Local.  Up 116.10 seconds[  116.190100] echo[1376]: trying to reload or restart NetworkManager.service[  OK  ] Started Execute cloud user/final scripts.[  OK  ] Reached target Cloud-init target.
[16:40] <testing1818282> ```[  OK  ] Reached target Login Prompts.[FAILED] Failed to start OpenSSH server daemon.See 'systemctl status sshd.service' for details.[  115.600243] cloud-init[1097]: Cloud-init v. 19.4 running 'modules:config' at Mon, 04 Jan 2021 15:36:36 +0000. Up 115.52 seconds.[  OK  ] Started Apply the settings specified in cloud-config.         Starting
[16:40] <testing1818282> Execute cloud user/final scripts...[  OK  ] Started System Logging Service.[  OK  ] Reached target Multi-User System.         Starting Update UTMP about System Runlevel Changes...[  OK  ] Started Update UTMP about System Runlevel Changes.ci-info: +++++++++++++++++++++Authorized keys from /home/ec2-user/.ssh/authorized_keys for user
[16:40] <testing1818282> ec2-user++++++++++++++++++++++ci-info: +---------+-------------------------------------------------+---------+---------------------------------------------+ci-info: | Keytype |                Fingerprint (md5)                | Options |                   Comment                   |ci-info:
[16:40] <testing1818282> +---------+-------------------------------------------------+---------+---------------------------------------------+ci-info: | ssh-rsa | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |    -    | packer_5ff2df88-947a-5d37-29ff-4ca186ea3f79 |ci-info: | ssh-rsa | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |    -    |                   jenkins
[16:40] <testing1818282> |ci-info: +---------+-------------------------------------------------+---------+---------------------------------------------+<14>Jan  4 15:36:36 ec2:<14>Jan  4 15:36:36 ec2: #############################################################<14>Jan  4 15:36:36 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----<14>Jan  4 15:36:36 ec2: -----END SSH HOST KEY
[16:40] <testing1818282> FINGERPRINTS-----<14>Jan  4 15:36:36 ec2: #############################################################-----BEGIN SSH HOST KEY KEYS----------END SSH HOST KEY KEYS-----[  116.110779] cloud-init[1180]: Cloud-init v. 19.4 running 'modules:final' at Mon, 04 Jan 2021 15:36:36 +0000. Up 115.90 seconds.[  116.112793] cloud-init[1180]: Cloud-init v. 19.4
[16:40] <testing1818282> finished at Mon, 04 Jan 2021 15:36:36 +0000. Datasource DataSourceEc2Local.  Up 116.10 seconds[  116.190100] echo[1376]: trying to reload or restart NetworkManager.service[  OK  ] Started Execute cloud user/final scripts.[  OK  ] Reached target Cloud-init target.
[16:40] <testing1818282> ```[  OK  ] Reached target Login Prompts.[FAILED] Failed to start OpenSSH server daemon.See 'systemctl status sshd.service' for details.[  115.600243] cloud-init[1097]: Cloud-init v. 19.4 running 'modules:config' at Mon, 04 Jan 2021 15:36:36 +0000. Up 115.52 seconds.[  OK  ] Started Apply the settings specified in cloud-config.         Starting
[16:40] <testing1818282> Execute cloud user/final scripts...[  OK  ] Started System Logging Service.[  OK  ] Reached target Multi-User System.         Starting Update UTMP about System Runlevel Changes...[  OK  ] Started Update UTMP about System Runlevel Changes.ci-info: +++++++++++++++++++++Authorized keys from /home/ec2-user/.ssh/authorized_keys for user
[16:40] <testing1818282> ec2-user++++++++++++++++++++++ci-info: +---------+-------------------------------------------------+---------+---------------------------------------------+ci-info: | Keytype |                Fingerprint (md5)                | Options |                   Comment                   |ci-info:
[16:40] <testing1818282> +---------+-------------------------------------------------+---------+---------------------------------------------+ci-info: | ssh-rsa | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |    -    | packer_5ff2df88-947a-5d37-29ff-4ca186ea3f79 |ci-info: | ssh-rsa | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |    -    |                   jenkins
[16:40] <testing1818282> |ci-info: +---------+-------------------------------------------------+---------+---------------------------------------------+<14>Jan  4 15:36:36 ec2:<14>Jan  4 15:36:36 ec2: #############################################################<14>Jan  4 15:36:36 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----<14>Jan  4 15:36:36 ec2: -----END SSH HOST KEY
[16:40] <testing1818282> FINGERPRINTS-----<14>Jan  4 15:36:36 ec2: #############################################################-----BEGIN SSH HOST KEY KEYS----------END SSH HOST KEY KEYS-----[  116.110779] cloud-init[1180]: Cloud-init v. 19.4 running 'modules:final' at Mon, 04 Jan 2021 15:36:36 +0000. Up 115.90 seconds.[  116.112793] cloud-init[1180]: Cloud-init v. 19.4
[16:40] <testing1818282> finished at Mon, 04 Jan 2021 15:36:36 +0000. Datasource DataSourceEc2Local.  Up 116.10 seconds[  116.190100] echo[1376]: trying to reload or restart NetworkManager.service[  OK  ] Started Execute cloud user/final scripts.[  OK  ] Reached target Cloud-init target.
[16:41] <powersj> please use https://pastebin.ubuntu.com/ to paste large amounts of text...
[16:41] <testing1818282> I am so confused with the controls of this chat - sorry
[16:44] <testing1818282> https://pastebin.ubuntu.com/p/DrZ84SWsfM/ here is the log - thanks for taking a look
[17:10] <testing1818282> I am very unsure how to solve this, since generating the host keys and printing them are done both during the config stage as runcmd is. So doing a runcmd and waiting does not make sense I think
[17:12] <testing1818282> once ssh is up I can see the ssh_host_*_key.pub files inside /etc/ssh - they look good
[17:14] <Odd_Bloke> testing1818282: Can you pastebin /var/log/cloud-init.log, please?
[17:28] <testing1818282> Odd_Bloke https://pastebin.ubuntu.com/p/dk4hMqsCVX/
[17:34] <Odd_Bloke> testing1818282: Hmm, it looks to me like cloud-init doesn't think it needs to generate the SSH keys; we don't even see a failure message from the generation.  Is the cloud-init package pulled in by your `dnf` installation?  Or is it already present in the base image you're building on?
[17:36] <testing1818282> I think it is already present - I can't find the installation process anywhere
[17:43] <testing1818282> any ideas on what to try next?
[17:45] <testing1818282> It did generate the keys though at 17:18:36.577952053
=== ijohnson is now known as ijohnson|lunch
=== hjensas is now known as hjensas|PTO
=== ijohnson|lunch is now known as ijohnson