[02:08] <mal41> Am I being daft or does "device_aliases" as documented in the disk_setup module just not exist? The only place in the code base where "device_aliases" appears is in the documentation for that module, never - seemingly - in actual code.
[02:09] <mal41> "disk_aliases" seems to exist, but that only applies to dscfg, not user-land config.
[02:09] <mal41> I feel like I've been gaslit by documentation /o\
[15:42] <rharper> mal41: device_aliases is a but under documented as to where the alias comes from;  from what i know, various clouds provide a 'block-device-mapping' in their metadata, Ec2, OpenStack,  (Azure sort of does but differently);  in the ec2 metadata, the block-device-mapping provides an 'ephemeral' alias to whatever block device name is associated (say /dev/sdb);  so the alias in disk_setup is used to refer abstractly to "whatever 
[15:42] <rharper> device calls itself 'ephemeral' according to this clouds metadata";  for ec2 and OpenStack, theres no on-disk configuration (like in /etc or something) which shows this mapping;  On Azure, there are some /dev/disk/cloud/* symlinks pointing to different disk types (azure_root, azure_resource, azure_datadisk);   
[16:11] <paride> blackboxsw_, it seems that travis didn't pick up https://github.com/canonical/cloud-init/pull/942
[16:12] <paride> I can't see it here https://travis-ci.com/github/canonical/cloud-init/pull_requests
[16:12] <blackboxsw_> paride: yeah I was just checking that to make sure we we have credits to run jobs 
[16:12] <blackboxsw_> and there are no warning banners there about inability to build jobs
[16:12] <blackboxsw_> ... as happened in the past
[16:13] <blackboxsw_> I also see that PR #941  did get run a couple of hours before https://app.travis-ci.com/github/canonical/cloud-init/requests
[16:14] <blackboxsw_> well ubottu, those links are not so helpful
[16:17] <blackboxsw_> paride: WDYT about closing and re-opening the PR?
[16:18] <blackboxsw_> it doesn't seem like the build request was received for some reason
[16:18] <paride> blackboxsw_, sure
[16:20] <paride> blackboxsw_, https://github.com/canonical/cloud-init/pull/943/
[16:22] <paride> it doesn't seem to be working :(
[16:24] <blackboxsw_> yeah, I'm trying to remember if we've hit this before... checking settings on the project
[16:34] <mal41> rharper The disk aliases from the metadata side you describe seem to use the key "disk_aliases" (https://github.com/canonical/cloud-init/blob/main/cloudinit/sources/DataSourceAzure.py#L740). In contrast the "device_aliases" as documented (https://github.com/canonical/cloud-init/blob/main/cloudinit/config/cc_disk_setup.py#L19-L23) suggests the user
[16:34] <mal41> (rather than the datasource) can provide their own. As far as I can see there is no code to support this.
[16:35] <paride> blackboxsw_, https://github.com/canonical/cloud-init/pull/941 is also not getting a travis run
[16:35] <paride> it got one but it's not getting re-run after the force-push
[16:45] <rharper> mal41:  you're right; it's not implemented, what does work is using aliases provided by block-device-mapping from metadata service ;   if you have time, please file a bug so we can remove that ;  
[16:50] <mal41> rharper Thanks, appreciate the sanity check. It would have been a nice feature to have, so I may see how much work it would be to support it as documented and file either a bug or PR accordingly. Thanks again for your help. :D
[16:55] <rharper> mal41: sure ;  one challenge with user-supplied values is not being able to control the platform ...  I suppose in some scenarios (non-cloud) one might say {'ssd-disk1': /dev/disk/by-id/*SERIAL*}  or something like that to id the drive (and allow udev to handle the mapping to the kernel device name (sda, xvda, etc ... )  
[17:03] <mal41> I believe it's udev rules that map the devices in some of the clouds (at least in Azure), and as this would only be a convenience mapping I wouldn't expect any functional difference to explicitly referencing the mapped value everywhere the mapping would be checked.
[17:13] <blackboxsw_> paride: hrm was able to trigger main building in travis, but still not seeing PR updates 
[17:20] <blackboxsw_> #startmeeting cloud-init bi-weekly office-hours
[17:20] <meetingology> Meeting started at 17:20:28 UTC.  The chair is blackboxsw_.  Information about MeetBot at https://wiki.ubuntu.com/meetingology
[17:20] <meetingology> Available commands: action, commands, idea, info, link, nick
[17:22] <rharper> mal41: right, udev rules do the mapping (for azure, not sure if any other clouds are doing udev rule mapping as well); I guess I don't understand the use-case you have;  
[17:23] <blackboxsw_> community-notice: Hi cloud-init folks. Bi-weekly office hours meeting has begun. We keep this time slot available to encourage designated periods of time where at least one  upstream cloud-init committer is available with eyes on the channel for any discussions about bugs, features, feature-requests and general support. Any cloud-init discussions are welcomed..
[17:23] <blackboxsw_> #chair rharper
[17:23] <meetingology> Current chairs: blackboxsw_, rharper
[17:24] <blackboxsw_> heya.... since you are around too :)
[17:25] <rharper> =)
[17:25] <blackboxsw_> the goal of the meeting time is an informal "meeting" where any questions and discussions can potentially gain broader audience... and anyone blocked can be unblocked whether it's for PR reviews, feature-work or general bug discussion.
[17:26] <blackboxsw_> anyhow. I've said my piece, we generally also put up high-level discourse post for out-of-band readers that might want to keep up to date on recent commits/changes in cloud-init
[17:26] <blackboxsw_> #link https://discourse.ubuntu.com/t/cloud-init-status-07-13-2021/23257
[17:27] <mal41> rharper Aside from being able to change a device in a single place, it would also be possible to define things like disk_setup and mounts once using aliases, and then have device_aliases overridden by a more instance specific secondary config. In my that would certainly reduce the amount of duplication between otherwise very similar configs which
[17:27] <mal41> is one of the reasons I started looking at it.
[17:27] <mal41> *in my case
[17:27] <blackboxsw_> note: upcoming upstream release is coming in august and we hope to get post-boot hotplug support into that release.
[17:35] <blackboxsw_> rharper: mal41, so that I understand the discussion, is the "bug" that you suggest filing to remove/correct docs on  https://github.com/canonical/cloud-init/blob/main/cloudinit/config/cc_disk_setup.py#L19-L23  to avoid claiming support for device_aliases?
[17:41] <mal41> I would prefer that the bug be to implement that feature as documented - if I understand correctly it seems achievable via couple of small modifications to cc_disk_detup and cc_mounts.
[17:41] <blackboxsw_> paride dunno what happened with Travis CI, but it appears "unstuck" after I ran the CRON CI run on main and is now building the PR941 https://travis-ci.com/github/canonical/cloud-init/builds/232639556. So we might expect a run from your PR shortly and we can merge it
[17:42] <mal41> But if it needs to be a low hanging fruit thing, then yes, removing the actively misleading documentation would be good until such time that a PR is ready and acceptable (of which part could be to revert any doc removal actioned in the interim).
[17:43] <blackboxsw_> mal41: I think it sounds like a reasonable feature request bug if you can state the use-case you are looking for. I don't think we would be averse to adding such a feature, just a question of priortizing it. We certainly can help shepherd in PRs that interested folks submit on this ;)
[17:44] <blackboxsw_> so either way, additional context on your use case in a but would help establish some level of priority and also give us something to point at as others may be looking for the same feature from userdata.
[17:46] <rharper> mal41: right, I understand the usefulness of the aliases, I meant more practically, how does one on a given platform actually predict (or configure) that a specific block device (/dev/sda) is the disk that you really want to call 'my-data-drive' as an alias ;  and note that /dev/sdX and friends are unstable across reboots/kernels/platform changes (some providers may be stable, but it's not guaranteed);  
[17:50] <mal41> Hmm, I don't think this solves (and doesn't seek to solve) that issue - {"device_aliases": {"abc": "/dev/sdb"}, "disk_setup": {"abc": { ...cfg ...} } would suffer the only the same issue as with { "disk_setup": {"/dev/sdb": { ...cfg ...} } currently.
[18:15] <mal41> blackboxsw_ Apologies, I missed your question; the use it would be in my case would be better readability in the config, a single point of definition for devices referenced in disk_setup and mounts, and following from that the ability to override the aliases from a secondary config without needing to completely redefine those sections. The primary
[18:15] <mal41> case for the latter is when moving long-lived disks and instances between hardware in the cloud. i.e. Modify secondary config on OS disk ahead of time. Stop the instance, dettach/reattach disks to match new expectation in secondary config. Start the instance and have cloud-init understand the new disk devices and treat them appropriately without
[18:15] <mal41> the need to change the config that was baked into the instance at original provisioning time.
[18:17] <mal41> I'm certainly not oblivious to that probably being a very rare case, it's just one of those born of the stuff landing on my desk. :P
[18:35] <mal41> I slapped together this very ugly draft just to confirm I'm on the right lines, if you have a moment to give it a glance. It needs tests, real world testing, and clean up, but hopefully it's a not-insane start, ;)  https://github.com/canonical/cloud-init/compare/main...mal:device-aliases?expand=1
[18:35] <blackboxsw_> heh, understood the "stuff on my desk".  Yeah, I'm trying to wrap my head around how the detatch/reattach migration migration would work from cloud-nit perspective. 
[18:36] <blackboxsw_> I'm trying to understand what you mean by ". Modify secondary config on OS disk ahead of time" llike supplementary #cloud-config userdata?
[18:39] <mal41> Yeah, so at the moment in this scenario I write out a smaller config affecting those areas I need to update, then modify the systemd service file to add the "--file /etc/cloud/override.yml" flag to the init call. Then when the system comes back after being stopped, it sees the instance ID has changed (as it's now on different cloud hardware) and
[18:39] <mal41> runs cloud-init but with the modifications injected by the on disk config.
[18:40] <blackboxsw_> hrm, so both "source storage machine" with orig long-lived disks and "target new storage target machine" would have been deployed with the same storage mount aliases and config expectations, and the migration process would be to detach orig disks for source machine, attach to target machine and launch target machine where the config is properly adapted and mount .
[18:43] <mal41> In a sane world yes, but when you're handed VMs that have persistent data on their OS disk which can't be mounted on the new instance, then you start seeing the lovely idea of using a new properly configured instance slowly drift out of the window. :(
[18:43] <mal41> "Can we just move the data?" "Not at this time." ¬_____¬
[18:56] <blackboxsw_> please correct me on my understanding (now that I've looked over your diff) ....
[18:56] <blackboxsw_> mal41: ok, so simply, for "migration reasons" you want to be able to simply support the documented device_aliases config dict as variable replacement values across both cc_disk_setup and cc_mount config modules to avoid repetition in user-data definitions because in your case you generally know what those alias to device path mappings will be on the "new" system 
[19:11] <blackboxsw_> I put that question simply so you can correct me; but I was wondering if YAML anchors and aliases themselves would work for the case of defining repeated path cases within the mounts module in the absence of this feature.    That said, looking over your diff I get the context a bit and your desire to use device_aliases config keys in both mounts and cc_disk_setup.
[19:12] <blackboxsw_> In both of those patch cases, I'd prefer seeing something that'd fallback to the originally behavior if the key wasn't present in device_aliases.
[19:14] <blackboxsw_> what you have written for disk_setup seems to do that.... first a lookup in device_aliases cfg (if unset we can just make that an empty dict {}) and we can avoid the if/else handling.
[19:17] <blackboxsw_> but.... yeah, a good example #cloud-config that you expect you'd use would help clarify your desired flow.
[19:19] <blackboxsw_> ok, let's wrap up office hours for today. Thanks all for the discussions, we can continue device_aliases discussions as needed or as a PR or bug is created for ongoing discusssion. As always, drop in with any questions you might have. Thanks again
[19:19] <blackboxsw_> #endmeeting
[19:19] <meetingology> Meeting ended at 19:19:39 UTC.  Minutes at https://new.ubottu.com/meetingology/logs/cloud-init/2021/cloud-init.2021-07-13-17.20.moin.txt
[19:20] <blackboxsw_> In testing that udev hotplug network branch on rocky linux I found some issues with SELinux blocking the ability of systemd units from creating FIFO queues
[19:21] <blackboxsw_> I'll probably put up something, but we might have to sort the SELinux needs there if our current hotplug approach is going to work in SELinux controlled environments
[19:45] <mal41> Thanks blackboxsw_, I'll do another pass or two over that diff and test and clean it up a bit and can go from there. :)
[20:18] <blackboxsw_> +1 thx mal41