[17:34] <dch> cloudinit identifies datasource as OpenStack on 23.3 instead of Ec2
[17:34] <dch> this is FreeBSD so there may be other differences
[17:35] <dch> https://www.irccloud.com/pastebin/MWxQ1bGc/cloud.cfg
[17:36] <dch> https://www.irccloud.com/pastebin/8OBlUpcr/ds-identify
[17:36] <dch> at least from command line can I force the datasource? 
[17:37] <dch> this is what I'd expected `datasource_list: ['Ec2']` to do, which is what was previously relied upon
[17:39] <dch> cc meena 
[17:47] <meena> dch: yes, but I think also need, 'None' to terminate the list for good
[17:48] <dch> meena: just tried that, same result
[17:48] <meena> dch: you can also disable the ds-identify service to disable the feature
[17:48] <dch> I just say this "no datasource_list found, using default: MAAS ConfigDrive... etc
[17:49] <meena> and by disable, i mean, delete
[17:49] <meena> lemme get my laptop, might make writing easier
[17:50] <dch> meena: only if you have the time, its not urgent
[17:51] <dch> 22.4-28-gaecdcbf8 was the previous version
[17:53] <meena> actually, it's bedtime, and i have to put kiddo ti bed now, dch, so anywhere between half an hour, and two hours when I'm actually getting back to my laptop
[17:55] <dch> meena: similar story here, how about I keep debugging and you can drop in later to throw some peanuts at me?
[17:55] <dch> or popcorn, I'm easy
[17:56] <meena> we got monkey nuts for the crows 
[18:20] <minimal> dch: this is a physical machine (Supermicro) ?
[18:27] <minimal> dch: a physical machine won't "look" like AWS (i.e. the DMI values) and so it makes sense that Ec2 is not chosen. There was a change made to the OpenStack DS a while to support physical machines (as well as VMs/Cloud) and I expect that is why ds-identify decides "maybe"
[18:29] <minimal> look at the likes of the ec2_identify_platform in ds-identify to see its' logic
[18:50] <dch> minimal: yep, all real tin
[18:50] <dch> this btw works under 22.x so I'm trying to find how to address this
[18:51] <dch> exploring /etc/cloud/ds-identify.cfg to disable it
[19:02] <dch> meena: where do you want me to log some bugs for this? /etc/cloud/ds-identify.cfg should look in %PREFIX%/etc/cloud/ds-identify.cfg IM(H)O but it doesn't 
[19:04] <dch> and even if that is set, I see `DSNAME=Ec2 DSLIST=Ec2 in ds-identify log, but this doesn't appear to change the DS as seen by cloudinit itself
[19:04] <dch> so I shall dig into that next
[19:05] <dch> https://www.irccloud.com/pastebin/elrefaG2/ds-identify
[19:15] <meena> dch: to disable it, you literally just `rm %PREFIX%/etc/rc.d/ds-identify`
[19:15] <dch> meena: thanks, let me try that.
[19:16] <dch> but every time I get pkg update, this will re-appear I guess
[19:16] <dch> . is there any way to disable it by config, rather than chainsaw?
[19:16] <meena> i should probably introduce a variable to disable only that thing, eh?
[19:17] <meena> dch: do you wanna open up a bug?
[19:17] <dch> meena: sure, on cloudinit repo?
[19:17] <dch> mmm
[19:18] <dch> I think the nicest way / easiest way is to add a cloudinit_dsidentify_enable=NO var 
[19:18] <meena> yeah, i don't have access to mail rn, so bugzilla is kinda meh
[19:19] <dch> In the bigger picture I would like to understand why there appears to be no obvious way to hard-wire this, using the parameters that worked happily under 22.x.
[19:19] <meena> dch: lots of code changes since then
[19:20] <dch> indeed I skimmed the changelog
[19:21] <dch> 1 datasources returned maybe: OpenStack
[19:22] <meena> yeah, i wonder if you can say no maybes
[19:23] <dch> so.
[19:23] <dch> that alone isn't sufficient, it insists on picking openstack
[19:52] <dch> I think I need to remove /run/cloud-init/* as well as /var/lib/cloud/* /var/log/cloud* to clean up
[19:59] <dch> ok that looks better
[20:28] <dch> |`->no network data found from DataSourceEc2 @01.04000s +125.16800s`
[20:28] <dch> if I read this right, despite locking Datasource=Ec2 this still takes over 2 minutes to fetch the metadata?
[20:28] <dch> that seems longer than expected
[20:36] <dch> I wonder if this is 2x 60s metadata server timeouts
[20:38] <meena> some services have 5 minutes timeouts
[20:44] <dch> I think the instance just takes ~ 2 minutes before its allowed to connect to the network
[21:25] <dch> meena: did the rc.d/dsidentify only get added in 23.3 ? I think this the case
[21:25] <meena> dch: yes
[21:26] <minimal> dch: so you have something on your local network emulating a Ec2 metadata server?
[21:26] <dch> minimal: yes, although its not actually mine, so I don't get to change it
[21:27] <minimal> so is it providing network config info?
[21:27] <dch> minimal: not network info, because thats pre-injected (as its complicated), but the rest - ssh keys, userdata scripts, the usual
[21:28] <minimal> dch: pre-injected? but you're trying to use the Ec2 DataSource in which case the Ec2 metadata server is expected to provide network config
[21:28] <dch> `network: {config: disabled}` in cloud.cfg prevents that
[21:28] <dch> i.e. the images are deployed with the requisite network config injected during ipxe stage
[21:29] <dch> before OS is started
[21:29] <minimal> dch: so you don't really want a "proper" Ec2 DS then?
[21:29] <dch> the rest is (largely) user configurable
[21:30] <dch> minimal: TBH all I really want is for cloud-config to have a schema for its config files, that is largely consistent and stable across versions
[21:30] <minimal> with a "real" Ec2 datasource cloud-init brings up a temporary network config in order to contact the metadata server to then obtain the "real" network config so that it can configure the OS accordingly
[21:30] <dch> minimal: right, in this case we skip the first bit
[21:30] <minimal> but if you skip that then you don't really want "real" Ec2 then...
[21:31] <dch> its a bit hard for me to introspect it from the outside, but I think ~ 2 minutes is waiting for something (networking, routing idk) to "come up" and let traffic flow
[21:31] <dch> minimal: true, we also have `strict_id: false` for the ec2 config
[21:31] <dch> again, that bit is not my network so it is what it is
[21:31] <minimal> so if you don't want "real" EC2 then why don't use use another DS, like NoCloud-Net?
[21:33] <dch> minimal: did it exist 4-5 years ago? that would be the main reason its done this particular way
[21:34] <dch> I skimmed https://cloudinit.readthedocs.io/en/latest/reference/datasources/nocloud.html and its not clear to me whether the network-fetched metadata like ssh-keys and stuff would still work the same way (from same url etc)
[21:34] <minimal> I didn't think cloud-init supported FreeBSD 405 years ago...
[21:34] <minimal> s/405/4-5/
[21:35] <dch> minimal: you'd be surprised what FreeBSD people hack on when we think nobody's looking
[21:36] <dch> it was first added in 2014 to FreeBSD ports, and works tolerably well apart from the slow drifts of time and config changes
[21:36] <minimal> dch: "is for cloud-config to have a schema for its config files, that is largely consistent and stable across versions" - cloud-init *does* have a schema for cloud-config, indeed the provided user-data is validated during boot. However regarding stable across versions, when things are changed the old version is marked depricated in the schema and still supported for a period of time
[21:37] <dch> I wonder if there's a way for me to validate our configs externally perhaps?
[21:37] <dch> like, in CI when I build the images?
[21:37] <minimal> actually you referred to cloud-init as "files" - cloud-init is one particular type of user-data
[21:37] <dch> true
[21:38] <minimal> network config and meta data are NOT cloud-config
[21:38] <minimal> the cloud-init cli can validate cloud-config files
[21:39] <dch> so the "data" that are fetched from www servers which may or may not be, or end up, as files, ...
[21:39] <dch> do these follow a schema from cloud-config? or is this actually the DataSource provider, that owns this?
[21:40] <minimal> no, the specific cloud-config user data i.e. a YAML file with first line "#cloud-config", not metadata or network config files/documents
[21:40] <minimal> recent releases of cloud-init have added a schema for network v2 config but I think that is a work-in-progress
[21:42] <minimal> aciba90 also recently wrote a server for validating cloud-config docs: https://github.com/aciba90/cloud-config-validator
[21:43] <minimal> dch: cloud-config schema is independent of DataSource
[21:45] <minimal> dch: "network-fetched metadata like ssh-keys and stuff would still work the same way (from same url etc)" - ssh-keys etc are NOT metadata and NoCloud-Net fetches from multiple URLs relative to the same base-url (i.e. meta-data, network-config, vendor-config, and user-data appended to the base url)
[21:48] <minimal> oops, forget "network-config", it's not relevant for NoCloud-Net
[22:11] <dch> ok I think I got to the bottom of all of this
[22:12] <dch> - on FreeBSD, stuff live in /usr/local/etc/cloud/... not /etc/cloud
[22:12] <dch> - ds-identify script doesn't accommodate that yet, so:
[22:12] <dch> thus:
[22:12] <dch> - it looks for the default cloud.cfg in /etc/cloud/cloud.cfg
[22:12] <dch> - fails to find the `datasource_list: ...` parameters
[22:13] <dch> - falls back on OpenStack
[22:13] <dch> if the default cloudinit path is fixed, this seems to work again
[22:19] <dch> meena: catch you tomorrow I'll add an issue for this too with
[23:52] <DarkenedGentlema> I'm having an issue I just can't seam to figure out.  I've got cloud-init installed on a disk image that is booted by Xen (virtualization).  The OS is almalinux8 and it boots as it should however it for some reason cloud-init doesn't run on boot.
[23:52] <DarkenedGentlema> I've used systemctl to enable cloud-init, cloud-init-local, cloud-config, and cloud-final.
[23:54] <DarkenedGentlema> Still wont run.  cloudinit clean doesn't seam to change anything either.
[23:55] <DarkenedGentlema> cloud-init init will actually run as it should.
[23:58] <minimal> nothing in /var/log/cloud-init.log?