| chaos89 | dont mean to bug but any luck? akutz | 01:22 |
|---|---|---|
| === paride4 is now known as paride | ||
| akutz | chaos89: Sorry, it was a busy day of meetings. | 13:50 |
| akutz | chaos12: Sorry, wrong user name | 13:51 |
| akutz | chaos87: There you are :) I'm sorry, I'm not sure I fully understand what you want to do. | 13:52 |
| chaos87 | haha no problem. Glad you're around today. So here's my usecase: I have a switch without a DHCP server, I want VMs I provision to be given a static IP from a pool defined in a network protocol profile. | 13:53 |
| chaos87 | The old way: using vApp OVF dynamic properties, you could have an IP automatically get allocated from the pool to the property and configure the network interface | 13:53 |
| chaos87 | New way: It is no longer possible using the new vmware datasource to get this information or have IPs set to instance metadata from the pool. So you can't configure networking using the new guestinfo properties | 13:54 |
| akutz | Can you not continue to use the old way? | 13:55 |
| akutz | (not saying we shouldn't figure out the use case using the new DS .. I just want to be clear on whether or not the old way still works) | 13:55 |
| akutz | Does the old way, vApp OVF dynamic properties, leverage the OVF datasource? I'm trying to figure out how exactly the old way actually ends up assigning the network information inside the guest. | 13:56 |
| chaos87 | Not really, sort of. As you're probably aware you can't use the vApp properties and new DS at the same time. Since the properties aren't available as instance metadata you can't configure a "network" block in the cloudinit metadata info because you cant get those vApp properties. There is a hack, which is to use the vmtoolsd cli in a script to pull | 13:57 |
| chaos87 | this information from vApp then configure and run netplan apply | 13:57 |
| chaos87 | Here's an example of the hack: https://www.virtualthoughts.co.uk/2020/03/29/rancher-vsphere-network-protocol-profiles-and-static-ip-addresses-for-k8s-nodes/ | 13:57 |
| chaos87 | I disabled the OVF properties because 1) That is the steps outlined in the old repo (before it was merged upstream) 2) because I don't want to use OVF properties since I want to configure networking in the metadata block | 13:58 |
| chaos87 | I can't use the upstream cloud init yet in 20.04 because the new version of cloud init won't be released in those repos. So until 22.04 we're using the old repo | 13:59 |
| akutz | > As you're probably aware you can't use the vApp properties and new DS at the same time | 14:00 |
| akutz | I was not in fact aware of this. How so? | 14:00 |
| chaos87 | Because they get disabled when applying your patch to the cloud-init config file which sets the new DS as the only datastore | 14:01 |
| chaos87 | Besides I'm not sure using the old DS would matter, I think it was the vmware customization that actually configured the network interface. You can't use vmware customization and cloud init at the same time. I can link you the vmware docs about that | 14:02 |
| chaos87 | https://kb.vmware.com/s/article/54986 | 14:03 |
| akutz | Which patch is setting that as the only DS? Which distro? The DS does not need to necessarily conflict with others. AFAIK only Photon is hardcoding the DS selection to my DS. You could also override that with a kernel param IIRC> | 14:07 |
| chaos87 | https://github.com/vmware-archive/cloud-init-vmware-guestinfo/blob/master/99-DataSourceVMwareGuestInfo.cfg | 14:07 |
| chaos87 | 20.04, can't use upstream because its not released in those repos | 14:07 |
| chaos87 | but I think this is sort of a moot point. I don't think having both would solve my problem | 14:08 |
| akutz | Could you see if Ubuntu Impish works? Because it does not limit the DS. Also, you could simply not apply the above patch. I would be interested in knowing whether both DSs *would* work. | 14:16 |
| akutz | In the meantime, I honestly do not know a whole lot about vApp dynamic properties or the way the current OVF DS uses them. | 14:17 |
| chaos87 | I could try but I don't think it's the OVF DS that uses them. I think it was vmware customization which has to be turned off to use cloud init | 14:17 |
| akutz | I *can* say that the team responsible for the VMware-specific code paths/features in the OVF DS are working to move that logic to the new VMware DS so no matter the transport type, all VMware code paths in cloud-init will have access to the same post-transport features. | 14:18 |
| chaos87 | so even with the OVF DS, I don't think this use case would work | 14:18 |
| akutz | I guess I'm confused. Do you *want* to use cloud-init at all? If not, why don't you just disable it and use the customization you've been using? | 14:19 |
| chaos87 | I want to use cloud-init. We use it for configuring other features that isn't possible with vmware customization. But because of that, now networking will not wokr | 14:19 |
| akutz | Also, I am really sorry if I seem pedantic or troublesome. I admit I do not know a lot about the other code paths / customization options. | 14:19 |
| akutz | Ah... | 14:19 |
| akutz | Heh. Yeah, *that* I know. | 14:20 |
| akutz | Hang on. | 14:20 |
| chaos87 | My first pass idea would be to ask you, in cloud-init (in general) can you template instance metadata in the "metadata" stage when configuring networking? And if so, maybe we could have the new DS grab the ovfProperties using vmtoolsd and place it in the instance metadata? | 14:21 |
| chaos87 | That way I could reference those vApp properties and configure networking. Either that or the dynamic properties would need to work as extra_config in vSphere and have those pulled in as metadata but I imagine that would be a bigger change | 14:21 |
| akutz | So that's part of what the other team I mentioned up above is considering doing. | 14:21 |
| akutz | But hang on. | 14:21 |
| chaos87 | ok! | 14:21 |
| akutz | I actually have a patch to your current issue. We hit the same thing. | 14:21 |
| akutz | chaos87: https://gist.github.com/akutz/55bb2c13d1849fbce4ec98bacbcd3cd5 | 14:23 |
| akutz | See if that helps | 14:23 |
| akutz | We hit this issue when wanting to bootstrap guest networking using GuestOS customization but wanting to apply user data with cloud-init. The project I'm leading is switching to using cloud-init to bootstrap the network, so this is no longer an issue for us. But still, the linked files should be a way you can defer cloud-init in order to use vmtools to configure the networking and have cloud-init run on second boot to handle the user data | 14:25 |
| akutz | As for: | 14:26 |
| akutz | > My first pass idea would be to ask you, in cloud-init (in general) can you template instance metadata in the "metadata" stage when configuring networking? | 14:26 |
| akutz | Jinja templating does not apply at the stage where metadata is consumed. This is because the metadata is in fact what serves as the basis for the templating engine. | 14:27 |
| chaos87 | hmm so after configuring this on a template VM I should be able to clone it and have vApp properties configure networking, but still be able to use the new DS with extra_config for userdata? | 14:27 |
| chaos87 | damn so even that solution wouldn't work. I wouldn't be able to grab the IP information during the metadata stage | 14:27 |
| akutz | You keep saying "new" DS, but you appear to be using the old, out-of-tree DS. | 14:27 |
| akutz | Also, you *can* use templated data to interpolate userdata, with the template pointing to information about the network. See https://cloudinit.readthedocs.io/en/latest/topics/datasources/vmware.html#instance-data-and-lazy-networks | 14:28 |
| chaos87 | sorry, "new" DS as in the not OVF one | 14:29 |
| akutz | And https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html#using-instance-data | 14:29 |
| akutz | >sorry, "new" DS as in the not OVF one | 14:29 |
| akutz | Fair, except to me this isn't new as it's been around since 2018. It only got merged into Cloud-Init proper last month, but it's existed in plenty of images for three years. Not arguing -- just explaining why I had the disconnect. | 14:29 |
| chaos87 | Ya that's fair. | 14:30 |
| chaos87 | Regarding your links. This still would mean you'd have to sort of cobble together a netplan in the userdata steps and run netplan apply which is too bad. But at least its a work around for when that information might be available as instance metadata | 14:30 |
| akutz | I get your point though. | 14:31 |
| akutz | If I could, you would like to: | 14:31 |
| akutz | * Have the get_data function, when getting the metadata, merge the dynamic information into the metadata | 14:31 |
| akutz | * And possibly take some of that and use it to replace variables in the network v2 config already in the metadata | 14:31 |
| akutz | * Where the variables are replaced by the dynamic info from the vApp dynamic keys or whatever | 14:32 |
| akutz | Is that close to what you are saying? Sorry, I am very, very slow. :) | 14:32 |
| chaos87 | exactly!! | 14:34 |
| chaos87 | imo the "fallback" networking config of cloud-init in the DS should try to use the vApp dynamic properties if they exist (theres a specific pattern that you're supposed to follow for the property keys) and configure networking for you if no other network v2 config is provided | 14:36 |
| chaos87 | what you describe is even better though because you get full control over using those keys | 14:40 |
| akutz | I am not sure what you mean by the "vApp dynamic properties." OVF properties are totally up to the vendor/user to define in the OVF. There's no standard for how these are mapped to networking of which I am aware. Not unless you're referring to something specific that VMware does in conjunction with the OVF DS in order to pass in information constructed in such a way the OVF DS knows how to read it. | 14:43 |
| chaos87 | I'm not sure who reads it, whether its the OVF DS or the vmware customization tool. vApp has specific settings in the UI to select what IP allocation scheme you're doing and you can select from the network protocol profile pool to get assigned an IP. | 14:46 |
| chaos87 | https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-DBF79F52-925A-4D0D-893C-49B911D3AC54.html | 14:46 |
| chaos87 | If you format your vApp property value as "{ip:MY-NETWORK-SWITCH}" for example, the value will dynamically get set to an IP from the pool | 14:47 |
| chaos87 | (there's info about this in that blog post I sent you where someone hacks together using the vApp properties with cloud init instead of vmware customization) | 14:48 |
| akutz | I see. So there *is* a specific property group/key pattern you can follow. Interesting. TIL! (and I work for the company!) :) | 14:48 |
| chaos87 | I believe so. I've seen it so far as "guestinfo.interface.0.ip.0.address" for example | 14:48 |
| akutz | Well, *that* is the old way the old Photon DS did it before they switched to my DS. But that was back in 2018. | 14:49 |
| chaos87 | trust me, I'm just as confused as you when it comes to how these are supposed to work. | 14:49 |
| akutz | https://github.com/vmware/photon/blob/8f264e3b418e2201d6edaf92f5572184817df247/SPECS/cloud-init/datasource-guestinfo.patch | 14:50 |
| chaos87 | What you described as a solution seems the most flexible to me | 14:50 |
| akutz | (the old Photon DS. You can see the pattern you mentioned there) | 14:50 |
| chaos87 | ahh I see. Yea so I'd basically like to have a way to do that using cloud-init. I don't mind providing my own networking v2 config if theres a way I can use those properties | 14:50 |
| blackboxsw | falcojr: do you have an update branch button for the Vultr changes. I've +1''d but can't use github UI or git push direct on that branch to rebase against tip of main https://github.com/canonical/cloud-init/pull/987 | 14:55 |
| ubottu | Pull 987 in canonical/cloud-init "Cleanup Vultr support" [Open] | 14:55 |
| akutz | But see, even *those* properties are just made up for that old DS. You don't really want a way to use *those* properties. You want a way to lazily pull in *some* data into the v2 network config set in *some* properties outside of guestinfo.metadata, before the new DS returns that net config to cloud-init for netplan processing | 14:56 |
| falcojr | blackboxsw: no, the submitter has control over whether to allow maintainers to make changes to the branch. It's enabled by default, but I know of one organization (not sure if it was Vultr) that wanted it disabled for their PRs | 14:57 |
| chaos87 | Yea there's just no other way that I know of in vSphere to have properties set dynamically based on the IP pool defined in a network protocol profile. akutz | 14:58 |
| falcojr | blackboxsw: just ask nicely and don't let me merge anything in the meantime ;) | 14:58 |
| akutz | ~note to self -- research this IP pool / network protocol profile thing chaos87 keeps mentioning...~ | 14:58 |
| chaos87 | But yes if there's a way we could use any of the vApp properties in the metadata stage, it doesn't have to be specifically *those* properties. Just any properties that allow this dynamic value | 14:58 |
| chaos87 | https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-D24DBAA0-68BD-49B9-9744-C06AE754972A.html | 14:59 |
| akutz | Ack. Fair. Good feature request. falcojr: Would you prefer we use launchpad to track this enhancement request or something internal at VMware? | 14:59 |
| chaos87 | If it's on launchpad or somewhere public I'd love to watch it and can help with any testing or field questions | 15:01 |
| falcojr | akutz: (haven't followed this whole conversation, but in general) public on Launchpad is preferable though just so people can see/comment, but it's ultimately up to you | 15:06 |
| akutz | Ack. Will do. Thanks! | 15:06 |
| falcojr | blackboxsw: also, on that point I've always thought our merge policy is too strict. I get that non-ff merges can rarely introduce unintended bugs, but in my professional career I've seen that happen maybe twice, and we never release without running through CI first anyway | 15:07 |
| blackboxsw | falcojr: I think we could back that off and cope with it if we start seeing problems. Given our coverage on release testing, and our more strict unittest integration test coverage on PRs I'm ok with that. | 15:09 |
| falcojr | blackboxsw: the bigger issue is probably when somebody submits a PR and CI is broken because of something they didn't do. That's not good, but probably also rare enough that I think the tradeoff is worth it. If we're worried about that, we could also run the same CI job internally on jenkins on every merge to master so we know right away if we need | 15:13 |
| falcojr | to go fix something. Probably still more efficient than the "update branch" babysitting we do right now | 15:13 |
| chaos87 | akutz let me know if you make an issue for it so I can bookmark :) | 16:33 |
| akutz | I will :) | 16:34 |
| chaos87 | For now from what I hear my workarounds may be, 1) disable clout-init's networking steps and rely on vmware customization reading these and configuring networking or 2) use the userdata step to write out a netplan and apply it using the vmtoolsd cli to read these properties | 16:34 |
| chaos87 | I'll give those a shot :) | 16:34 |
| emix | hi everyone! i've spent few days on learning cloudinit and coding a ds server for a public cloud project. however i just cant wrap my head around the datasources. are there any general recommendation of what network-based ds to use if you don't make your own? i've been tempted to write my own but does it really make sense for everyone to do that? | 21:21 |
| emix | i mean nocloud is great for your own stuff but as soon as you need more dynamic metadata it seems there is nothing "generic" ? | 21:21 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!