| === esv_ is now known as esv | ||
| carbonator | anybody using cloud-init with Hetzner? I'm stuck at how to chpasswd the root-pwd, and how to make the VM aware of the private network so I can install packages | 11:40 |
|---|---|---|
| meena | carbonator: if cloud-init isn't configuring the private network, what does the network config look like? | 11:52 |
| carbonator | meena: currently, the relevant parts are https://pastebin.com/5V9MgUqD | 11:54 |
| meena | no, that's your config | 11:54 |
| meena | i mean, what does Hetzner send as network config? | 11:55 |
| carbonator | it sends 10.0.0.1 as their gateway, the 10.0.0.10 for the VM, and the 10.0.0.2 is the firewall | 11:56 |
| carbonator | it's working fine after cloud-init is done, just not during the deployment | 12:00 |
| meena | carbonator: https://pastebin.com/5V9MgUqD isn't your full config, or? what is actually failing? | 12:04 |
| carbonator | meena: it's just the part about network. I ry to install packages in runcmd too, but that fails because there is no network available. | 12:08 |
| meena | https://github.com/canonical/cloud-init/blob/main/config/cloud.cfg.tmpl | 12:14 |
| meena | https://github.com/canonical/cloud-init/blob/main/config/cloud.cfg.tmpl#L80 | 12:14 |
| meena | this is the set of modules that runs before we have network | 12:15 |
| carbonator | meena: that's my config (with some info as ****) https://pastebin.com/J6Xnfv23 | 12:21 |
| carbonator | install/update in lines 52/53 fails (sorry for the delay, but work keeps me busy) | 12:22 |
| meena | can you show us the log files? I don't understand why the network is not setup properly, that you think you need to set it up yourself | 12:31 |
| carbonator | meena: okay to pm? | 12:46 |
| meena | carbonator: sure | 12:50 |
| minimal | carbonator: Hetzner should be providing the network config to cloud-init | 13:50 |
| carbonator | minimal: it does, and network is available when the VM creation is done, but network is unavailable during while the creation is running (for the records, there is no public IP on the VM, just a private IP) | 13:53 |
| minimal | carbonator: what do you mean by "during while the creation is running"? during the creation of what? | 13:53 |
| carbonator | minimal: I start the creation of the VM via terraform, and that pushes a yaml with the config. after a few minutes, the VM has been created and is then idling | 13:55 |
| minimal | ok, and so what is the problem? | 13:56 |
| carbonator | in the runcmd block, I try to eg update the install, which fails (package_update: true did too) | 13:57 |
| minimal | the cloud-init Hetzner DataSource should configure the network, so why are you configuring it in user-data? | 13:58 |
| carbonator | I was told by them that with a private network, I need to remove hc-utils and create an ifcfg (as in https://docs.hetzner.com/cloud/networks/server-configuration/) | 14:00 |
| minimal | cloud-init does not use hc-utils AFAIK | 14:01 |
| minimal | have you looked at /var/log/cloud-init.log to see what is happening? If debug is enabled then it should show cloud-init talking to the Hetzner metadata server, getting network info from it and configuring the networking | 14:02 |
| carbonator | minimal: went through it with meena | 14:03 |
| minimal | and? what did the logs show? | 14:03 |
| minimal | ok, it seems to me that the problem is that the Hetzner DataSource in cloud-init does use the "network-config" provided by Hetzner's metadata server but it does *not* use the "private-networks" section provided by Hetzner's metadata server | 14:06 |
| minimal | Hetzner wrote the Hetzner DataSource in cloud-init, looks like they have not (yet) updated it to add this functionality | 14:07 |
| carbonator | and there is no workaround until they do I guess? | 14:08 |
| minimal | well what you added in user-data to configure the private stuff should be a valid workaround | 14:09 |
| minimal | so it is failing at the "dnf -y update"? | 14:13 |
| minimal | have you tried does a "curl http://169.254.169.254/hetzner/v1/metadata/private-networks" to see what that returns? | 14:14 |
| meena | from the logs carbonator showed me, it looks like the network is setup correctly | 14:15 |
| carbonator | it returns the correct network info | 14:16 |
| minimal | carbonator: what error is "dnf -y update" failing with? if it is network issue? | 14:16 |
| minimal | s/if /is / | 14:17 |
| carbonator | one sec, pulling the other log | 14:17 |
| carbonator | https://pastebin.com/srma1xCU | 14:18 |
| minimal | that doesn't show any public IP addresses configured - so how will it get to mirrors.rocklinux.org without a public IP address? | 14:20 |
| carbonator | all the traffic goes through the firewall. like I said, that works fine once the VM provision is done | 14:21 |
| minimal | once the VM provision is done or once the firewall provision is done? | 14:22 |
| carbonator | VM. firewall is up and running a week now | 14:22 |
| minimal | that doesn't make sense | 14:23 |
| minimal | try adding a single "ping" call to the runcmd before running "dnf" to see if it can ping the firewall? | 14:23 |
| carbonator | I'll config a bit more, like dumping network config etc during creation and see what I get | 14:24 |
| minimal | another idea - perhaps "systemctl restart NetworkManager" hasn't finished bringing up the interface before you run "dnf"? try adding a "sleep" to delay "dnf" execution? | 14:25 |
| carbonator | oh, good idea, will do :) | 14:25 |
| minimal | well dumping network config would show if its up or not after restart | 14:26 |
| carbonator | meh, work would be more fun if it wasn't for customers. will have to delay debugging to tomorrow | 14:49 |
| minimal | I suspect you'll find it is a race condition | 14:52 |
| minimal | carbonator: Hetzner private network interfaces apparently have a MTU of 1450, so you'll need to set this | 14:59 |
| meena | this tox call makes me really jealous https://github.com/canonical/cloud-init/issues/4248 | 18:00 |
| -ubottu:#cloud-init- Issue 4248 in canonical/cloud-init "tests: ansible galaxy unable to install collections with Traceback" [Open] | 18:00 | |
| meena | time to sit down and write a FreeBSD kernel module | 18:01 |
| === dbungert1 is now known as dbungert | ||
| === Cadey is now known as Xe | ||
| blackboxsw | meena we'll get there :) freebsd kernel module for lxd integration ? | 22:34 |
| meena | blackboxsw: We need a virtio vsock module | 22:35 |
| meena | I should probably look at the HyperV vsock module as a starting point | 22:35 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!