[10:02] <ani> hey folks holmanb is the following syntax no longer supported?
[10:02] <ani> datasource_list:
[10:02] <ani>  - Azure
[10:02] <ani> ```datasource_list:
[10:02] <ani>  - Azure```
[15:01] <holmanb> ani: hi there, not sure i understand the question 
[15:03] <holmanb> The python code in cloud-init understands the syntax, but ds-identify has always required that to be on a single line
[15:03] <holmanb> So depends on your use case
[15:04] <holmanb> ds-identify has a simple yaml parser in posix shell that can't handle multi-line lists
[15:07] <ani> So previous to 23.4, that syntax worked but now it does not
[15:09] <ani> So we are getting errors like https://privatebin.corp.redhat.com/?fb8a2678912a221e#7Ko77UsYQGt9uMKvrHUWPp95LMmgA9GUKTS9AcPLHfYa
[15:09] <ani> For a config like
[15:09] <ani> datasource_list:
[15:09] <ani>  - Azure
[15:10] <ani> I could not pin point specific change that introduced the behaviur change
[15:28] <ani> holmanb
[15:36] <holmanb> ani: link doesn't work
[16:42] <ani> holmanb try now https://pastebin.com/49fr54iR
[16:45] <holmanb> ani: thanks
[16:46] <holmanb> looks like the datasource_list maybe contains a list of type dict, not a str
[16:46] <holmanb> is there a stray colon after one of your list items (`- Azure:`) by chance?
[16:47] <holmanb> ani: could you share the cloud.cfg causing this?
[16:52] <ani> holmanb https://pastebin.com/Lz9jahUc
[16:53] <ani> more complete one is https://pastebin.com/jBArHhnE
[17:00] <ani> Looking for data source in: [\\{'datasource_list': None}, 'None'], via packages ['', 'cloudinit.sources'] that matches dependencies ['FILESYSTEM', 'NETWORK']
[17:01] <ani> So its actually a "{'datasource_list': None}"
[17:01] <ani> so parsing is not working very well
[17:02] <blackboxsw> ani, was that a paste the issue is that second paste? line 1, and 2 aren't indented in scope of the datasource configuration, so` datasource:` is an empty dict and `Azure:` is an empty dict
[17:03] <blackboxsw> and apply_network_config isn't defined under `datasource:Azure:` yaml path
[17:05] <ani> I am not sure, I am pasting from out internal bug report raised by someone else. In any case, they are saying that changing line 4 to datasource_list: [ Azure ] makes it work
[17:05] <ani> with 23.4
[17:05] <ani> Previously the same config was working on 23.1.1
[17:08] <holmanb> to my knowledge datasource_list has always required single line configurations for ds-identify
[17:08] <holmanb> was rhel using ds-identify prior to 23.4?
[17:09] <minimal> holmanb: the example in RTD does have a comment: "# If you use datasource_list array, keep array items in a single line."
[17:09] <holmanb> minimal: exactly
[17:09] <ani> That is recent
[17:09] <minimal> "If you use multi line array, ds-identify script won't read array items."
[17:09] <ani> multiline lists  used to work before
[17:09] <ani> and also the config I shared
[17:10] <ani> that is why I am wondering what changed and from which commit
[17:10] <ani> yes rhel was using ds-identify afaik
[17:11] <holmanb> that comment in the templated config is 3 years old
[17:12] <minimal> Also the warning here: https://cloudinit.readthedocs.io/en/latest/reference/base_config_reference.html#datasource-list 
[17:12] <minimal> "This key is unique in that it uses a subset of YAML syntax. It requires that the key and its contents, a list, must share a single line - no newlines."
[17:14] <minimal> blackboxsw: your comment about lines 1 & 2 do confuse me, based on the docs I don't see "datasource:" needing to be indented, isn't it at the same "level" as datasource_list?
[17:14] <ani> yes but I am not sure why it worked before and now it does not
[17:15] <holmanb> ani: that's why I ask if ds-identify is a new addition to rhel
[17:15] <ani> afaik we have not changed anything
[17:15] <blackboxsw> minimal: sorry about that, the comment I had meant should have been 2 & 3. the keys that should have been nested under "datasource:" top-level key... which should be top-level key just like datasource_list:
[17:15] <holmanb> ani: because if ds-identify was added between 23.1.1 and 23.4 it would have "worked" before, but not now
[17:16] <blackboxsw> I can also confirm that the multi-line datasource_list config (as doc'd/warned) is silently ignoring /etc/cloud/cloud.cfg.d  containing `datasource_list:\n- Azure\n`  ... 
[17:17] <ani> ok well ignoring but not failing the way I showed in the pastebin
[17:17] <blackboxsw> and agreed w/ brett/minimal that this behavior in ds-identity has been there a long time as the shell script doesn't parse proper multi-line yaml in this case. 
[17:17] <ani> so I am not sure what is the difference in our case
[17:18] <ani> I guess we caught this isseu because of the failure
[17:18] <ani> if it had ignored it silently we would not see the issue
[17:18] <ani> so maybe it was ignoring the config silently before and now it crashes
[17:19] <holmanb> ani - I'm guessing that this is just a different failure path than before
[17:19] <holmanb> without selecting the datasource, cloud-init would not have actually been running before either
[17:20] <holmanb> or maybe it fell back to datasourcenone automatically before, but that's still not correct behavior
[17:21] <holmanb> minimal: +1 for referencing fresh docs :-)
[17:21] <blackboxsw> and agreed w/ brett/minimal that this behavior in ds-identity has been there a long time as the shell script doesn't parse proper multi-line yaml in this case. and agreed that it's been there since 2017 so it's something in external configuration on the image that is now aware of this condition (by running ds-identify during systemd generator timeframe now).   
[17:21] <holmanb> I think that warning landed in the docs just a week or two ago 
[17:22] <ani> exactly
[17:22] <ani> but you are saying that the behavior was there for a while
[17:22] <blackboxsw> ani yes, if systemd generator wasn't using ds-identify before, cloud-init python code would fallback to packaged datasource_list defaults which would generally be the full list of all supported datasources and the DataSource.get_data for each subclass would be called to see if it could correctly identify any viable datasource.
[17:23] <holmanb> the behavior has existed for as long as ds-identify has been in use - and the comment warning of that misconfiguration has existed for 3 years 
[17:24] <ani> So in summary I think we simply had a bad config that was somehow getting ignored up until now and it now crashes because of it
[17:24] <holmanb> ani: I would suspect that it crashed before as well, but crashed in a different way.
[17:25] <holmanb> ani: but otherwise yes that sounds about right
[17:25] <ani> ok
=== holmanb changed the topic of #cloud-init to:  23.3 Released 08/28 | File bug: https://github.com/canonical/cloud-init/issues/new/choose | PRs: https://git.io/JeVed | release schedule: https://discourse.ubuntu.com/t/2024-cloud-init-release-schedule/41679