[03:02] <blackboxsw> minor needs fixing on debian/changelog deb lintian warnings. I also think we left some cruft in debian/patches/ubuntu-advantage-revert-tip.patch on bionic that needs a git rm.
=== rowanthorpe1 is now known as rowanthorpe
[16:01] <blackboxsw> falcojr: bionic PR approved https://github.com/canonical/cloud-init/pull/997/files thx for the lints
[16:01] <ubottu> Pull 997 in canonical/cloud-init "Ubuntu/bionic" [Open]
[16:16] <blackboxsw> falcojr: if you can reflect the minor spelling change to d/changelog in focal and hirsute I think we are good on those branches
[16:17] <falcojr> yep, will do
[16:25] <falcojr> focal and hirsute updated
[16:25] <falcojr> blackboxsw: ^
[16:34] <blackboxsw> falcojr: package d/changelogs look good, running through sbuild now.
[16:35] <blackboxsw> will merge and upload to Ubuntu B,F,H -proposed  once that succeedds
[16:36] <blackboxsw> falcojr: one interesting thing I'm seeing on Bionic VM integration tests: our assert on NoCloud is failing: 18:46:44 E       AssertionError: assert 'seed-dir (/var/lib/cloud/seed/nocloud-net,/dev/sr0)' == 'seed-dir (/var/lib/cloud/seed/nocloud-net)' since we detect both /dev/sr0 && the nocloud-dir
[16:36] <blackboxsw> https://jenkins.ubuntu.com/server/view/cloud-init/job/cloud-init-integration-lxd_vm-bionic-daily/18/consoleFull
[16:38] <falcojr> yeah, I've seen that before...didn't realize that affected this test too
[16:39] <blackboxsw> smewhere between Aug 5th and Aug 10th something changed in lxc I bet?
[17:01] <blackboxsw> falcojr: hrm, just realizing we are missing adding the following d/changelog entry 
[17:02] <blackboxsw>   * d/cloud-init.templates: Add VMware datasource support
[17:02] <blackboxsw> and representing python3-netifaces
[17:03] <blackboxsw>   * d/control: Add dependencies on python3-netifaces for vmware ds
[17:03] <falcojr> hmmm, did our upstream snapshot script eat it or something?
[17:06] <blackboxsw> falcojr: I think because we only added the patch directly into ubuntu/focal|bionic|hirsute for cloud-init.templates and didn't create an unreleased debian changelog entry via `dch -i` new-upstream-snapshot won't document that in the merged changelog
[17:06] <andyf> drat, netifaces doesn’t build on illumos. luckily I already formed for my initial work.
[17:06] <andyf> *forked
[17:08] <falcojr> blackboxsw: we did add those changelog entries before though
[17:10] <falcojr> hmmm, I thought so...but the history is showing otherwise :/
[17:10] <blackboxsw> falcojr: if I git checkout upstream/ubuntu/focal and check d/changelog I see no UNRELEASED entry that represents   * d/cloud-init.templates: Add VMware datasource support
[17:10] <blackboxsw> last changelog entry was our previous SRU on focal. 
[17:10] <falcojr> dang
[17:10] <blackboxsw> so we would have needed a separate changelog commit after the VMware datasource addition
[17:11] <blackboxsw> so I've only pushed upstream/ubuntu/bionic  when I recognized this. and we can easily reject our queued SRU uploads because they haven't been accepted
[17:12] <blackboxsw> but, I think we should push your branch one more time with the following additional changelog entries:
[17:12] <blackboxsw> * d/cloud-init.templates: Add VMware datasource support
[17:12] <blackboxsw>   * d/control: Add dependencies on python3-netifaces for vmware ds
[17:12] <blackboxsw> minus whitespace damage... I'm checking to see if any other debian/* files were altered and undocs
[17:12] <blackboxsw> minus whitespace damage... I'm checking to see if any other debian/* files were altered and undocumented
[17:23] <blackboxsw> andyf: "today I learned" about illumos. thx for the reference. And you and trying to get cloud-init to run on the distro or it already does?
[17:24] <blackboxsw> ... and the recent dependency on netifaces is causing breakage for you?
[17:26] <falcojr> blackboxsw: branches are updated. I was accidentally tracking the wrong bionic branch and pushed straight to upstream. It's just the one commit, but you'll probably want to double check I didn't do anything stupid there either
[17:26] <blackboxsw> sry 'bout the typos. I was curious if illumos already has a working forked cloud-init or whether that is a project you are currently working on
[17:26] <blackboxsw> falcojr: I saw a strange force-push on upstream and was wondering
[17:27] <blackboxsw> falcojr: will double check. right
[17:27] <blackboxsw> only direct consumers of upstream/ubuntu/* branches that I know about are our PPA builders, so a force push shouldn't cause any damage
[17:34] <falcojr> blackboxsw: are you seeing the pylint failures due to cryptography's rust dependency?
[17:35] <falcojr> curious why that's only in the pylint job
[17:45] <blackboxsw> bbiab. pulled afk for a quick errand 
[18:04] <blackboxsw> <- back changelogs look good, rebuilding now
[18:09] <blackboxsw> tox -e pylint succeeding for me falcojr 
[18:09] <blackboxsw> though that note reminds me of an issue I thought we saw somewhere else. and generally this noise storm https://github.com/pyca/cryptography/issues/5771
[18:09] <ubottu> Issue 5771 in pyca/cryptography "Dependency on rust removes support for a number of platforms" [Closed]
[18:10] <blackboxsw> s/noise storm/general expression of discontent/ :)
[18:20] <falcojr> heh, yeah, I remember when that happened
[18:20] <blackboxsw> falcojr: ok uploads re-queued for B, F H, merging PRs now
[18:28] <blackboxsw> merge done, syncing ubuntu/* branches to launchpad
[18:28] <blackboxsw> via https://jenkins.ubuntu.com/server/job/cloud-init-github-mirror/
[18:30] <akutz> Is something going on re: VMware DS with which I need to help? I see a lot of activity, but it's unclear to me what the issue is.
[18:34] <blackboxsw> akutz: nah, just my bungling the release a bit. Sorry for the highlights. falcojr and I are uploading a new upstream snapshot into Ubuntu Impish and also queuing uploads for SRU publishing (StableReleaseUpdates) into Bionic Focal and Hirsute.  Our snapshot script didn't include a couple of packaging debian/changelog entries (all on our side for Ubuntu deb packaging) to document that we are enabling VMware for discovery.
[18:35] <blackboxsw> we need our docs/ducks in a row in the package itself if Ubuntu will accept the uploads.
[18:35] <akutz> blackboxsw: Ack. I looked like primarily a packaging issue, but I wanted to reach out and check since I was the ultimate cause for the new DS. Thanks!
[18:36] <falcojr> much more succinct explanation that I was typing up :)
[18:36] <blackboxsw> falcojr: that's a first for me #achievement unlocked
[18:36] <falcojr> blackboxsw: are we ready to ping a vanguard then?
[18:38] <blackboxsw> akutz: if you have access to Ubuntu  Impish cloud images 21.10, one could add an impish-proposed  source to /etc/apt/sources.list and `apt-upgrade; apt install cloud-init; cloud-init clean --logs --reboot` to see if VMwareDatasource is detected. https://cloudinit.readthedocs.io/en/latest/topics/debugging.html#stable-release-updates-sru-testing-for-cloud-init
[18:39] <blackboxsw> falcojr: I think we are ready to ping SRU vanguard in #ubuntu-release. (bdmurray's day)
[18:40] <blackboxsw> akutz: from a distro-side of things Ubuntu Impish is under FeatureFreeze for Impish, so uploads are paused for additional validation (kindof like the process we go through in SRUs for stable releases lke Bionic/Focal/Hirsute). falcojr or I will do a bit of verification testing on impish-proposed and get that upload reviewed and accepted via a feature freeze process. 
[18:41] <akutz> blackboxsw: Do you want me to doc the verification process? Because I already verified it via a PPA update and dpkg-reconfigure. Does this new verification buy anything else other than e2e?
[18:42] <akutz> Also, where are the 21.10 images? Do you mean a daily from http://cdimage.ubuntu.com?
[18:43] <blackboxsw> akutz: noe, if you verified vIa PPA we/your are good. we'll just spot check a view things on Impish to make sure we are still running without regressions. our daily CI runs look reasonably successful (and any failures we've seen  are explainable due to other infrastructure issues).
[18:43] <akutz> Found http://www.cloud-images.ubuntu.com/impish/current/
[18:43] <blackboxsw> right http://www.cloud-images.ubuntu.com/impish/current/
[18:43] <akutz> I'll deploy the image if I can grab a few, spare cycles and verify the ISO
[18:44] <blackboxsw> if you do great, it not no biggie. Just a heads up on timing/context more than anything
[18:44] <akutz> Ack
[18:44] <akutz> It won't hurt. And I should just need to drop the data into the VMX and do nothing else at this point, so if that works, great. If not, well, crap :)
[18:47] <blackboxsw> akutz: I also just uploaded a binary compatible debs we are testing to https://code.launchpad.net/~cloud-init-dev/+archive/ubuntu/proposed
[18:47] <akutz> Ack
[18:48] <blackboxsw> so you could just pull from that PPA and be reasonably certain it's what we will be publishing in Bionic, Focal, Hirsute and Impish
[18:48] <akutz> To be clear, what's in the current Impish image does not include the new without updating from apt?
[18:49] <blackboxsw> akutz: it won't until we pass featurefreeze exception verification. Until we pass verification, 21.3-1-g6803368d-0ubuntu1  will only live in "impish-proposed" pocket instead of "impish-updates"
[18:49] <akutz> Ack
[18:51] <blackboxsw> -> lunch
[18:51] <akutz> What is the default user/pass for cloud image?
[18:53] <akutz> Not sure how I test this if I cannot log into the image in order to update it with CI that has the DS when I need CI to create a user account?
[19:02] <rharper> no default passwords in cloud images (from Ubuntu), akutz; typically we launch with user-data that includes either a password, or more likely we ssh_import_id: lp:XXXX  (or gh:XXXX)  
[19:03] <akutz> Ah. Well, I created a new VM with the provided VMDK, but I cannot get data into CI that way since my DS is not yet activated :)
[19:04] <rharper> heh, that is a bit of a chicken/egg issue 
[19:04] <akutz> The OVA has some fields for initializing data. I can try that.
[19:05] <rharper> not sure how "loop mountable" VMDKs are ... but if it's possible to mount the filesystem, you could chroot and update root pw for console login 
[19:06] <akutz> Nah, since I have to add a new apt repo and do that whole dance, there's no reason I cannot just use the OVA to get into the image.
[19:06] <rharper> I see
[19:07] <akutz> (since the OVA has a field that let's me supply an SSH key)
[19:08] <akutz> > ubuntu@ubuntuguest:~$ 
[19:08] <akutz> I'm in :)
[19:11] <akutz> > cloud-init is already the newest version (21.3-1-g6803368d-0ubuntu1).
[19:11] <akutz> > 0 upgraded, 0 newly installed, 0 to remove and 60 not upgraded.
[19:11] <akutz> I followed https://cloudinit.readthedocs.io/en/latest/topics/debugging.html#stable-release-updates-sru-testing-for-cloud-init and got the above
[19:11] <akutz> Does this mean this image already has the DS? I mean, i guess it's easy enough to check...
[19:12] <akutz> $ sudo find / -name DataSourceVMware.py -type f
[19:12] <akutz> Um, did my post get lost?
[19:12] <akutz> Ah, /usr/lib/python3/dist-packages/cloudinit/sources/DataSourceVMware.py
[19:12] <akutz> It found the file
[19:13] <akutz> Does this mean this image *does* have the VMware DS, but it's just not activated?
[19:13] <akutz> No, I looked, and the VMware DS is in the list of activated DSs
[19:13] <rharper> so look at /etc/cloud/cloud.cfg.d/90-dpkg.cfg ; 
[19:13] <akutz> (just did)
[19:13] <akutz> It's there
[19:13] <rharper> and then in /run/cloud-init/cloud.cfg will include the datasource that was detected ; 
[19:14] <rharper> but if you booted via OVA, won't that enable the OVF Datasource? (I haven't been following the VMware datasource changes)
[19:14] <akutz> But my point is that blackboxsw said I'd need to log in first, update CI to proposed, and then clean cloud-init and start over
[19:14] <akutz> Yeah, I used the OVF DS to get into the thing because I thought the new DS wasn't activated. yet it's very much part of the distro, even though attempting to update cloud-init said no updates necessary
[19:15] <rharper> Well, impish is devel, so I think they've already uploaded cloud-init main to impish ,,, the SRU will apply to  the previous releases ;   so not quite sure what blackboxsw was asking to verify ; 
[19:16] <akutz> >  if you have access to Ubuntu  Impish cloud images 21.10, one could add an impish-proposed  source to /etc/apt/sources.list and `apt-upgrade; apt install cloud-init; cloud-init clean --logs --reboot` to see if VMwareDatasource is detected. https://cloudinit.readthedocs.io/en/latest/topics/debugging.html#stable-release-updates-sru-testing-for-cloud-init
[19:16] <akutz> That's what he said
[19:17] <rharper> for SRU, we typically launch the previous releases, so 21.04  or 20.04 (focal) , then add -proposed pocket and upgrade to the proposed update to cloud-init 
[19:17] <akutz> Do you think he just mistyped 21.10?
[19:25] <rharper> possibly
[19:46] <blackboxsw> sorry back from lunch 
[19:48] <blackboxsw> checking my typos and msg history.... ahh akutz yes per "cloud-init is already the newest version (21.3-1-g6803368d-0ubuntu1)." it looks like pur queued upload into impish-proposed was already accepted and has landed in impish-updates for cloud-init 21.3-1 and yes that means you already have the VMWareDatasource "support" available in the cloud-init deb package.
[19:49] <akutz> Okay, so I *should* be able to take the VMDK, create a VM with it, set the properties on the VM that activate the new DS, and boot, right?
[19:50] <blackboxsw> rharper: akutz yes I thought  we were still stuck in impish-proposed for cloud-init 21.3. But, it was already approved/landed into impish by Canonical foundations team. so the "upgrade" step is unneeded
[19:53] <blackboxsw> akutz: yes, with 21.3 installed in your image. You should be able to either set the appropriate datasource_list: [VMWare] in /etc/cloud/cloud.cfg.d/99_use_vmware.cfg and see VMWare detected via `cloud-id` or cloud-init query v1.platform
[19:54] <akutz> Well, since the DS is already in the dpkg list, I *should* be able to just set its metadata and userdata and boot I should think
[19:55] <blackboxsw> as long as whatever user-data/meta-data isn't getting a positive match for OVF via ds-identify dscheck_OVF you will get to VMWare 
[19:55] <blackboxsw> since DataSourceOVF is sorted before DataSourceVMWare in terms of discovery, environments that match both would be detected as OVF I believe
[19:57] <akutz> Sure. This is why I'm not going to use the OVA, but the VMDK that doesn't have the OVF properties that will trigger the OVF DS :)
[19:58] <blackboxsw> +1
[19:58] <andyf> blackboxsw: yes I have a mostly working fork for illumos that I would like to get upstreamed, probably piece by piece. i already opened a PR for the first bit as a test.
[19:59] <andyf> netifaces cane
[19:59] <rharper> blackboxsw: ah, I see. 
[19:59] <andyf> netifaces came in after I forked (only a week or so ago) and it doesnt build at the moment but I havent investigated yet
[20:01] <andyf> blackboxsw: it’s at https://github.com/omniosorg/cloud-init ; the illumos branch there. Some it is a bit hacky so far but I will be cleaning it up.
[20:02] <blackboxsw> andyf: and just a heads up related to netifaces, It's a recent dependency addtion. and I think upstream will need to talk about whether we need an ongoing dependency on netifaces. VMWware is the first datasource that uses it to handle a couple of default routing corner cases as well as MacOS support I think that are not captured in upstream cloudinit.net and netinfo functions.
[20:03] <blackboxsw> If we can find ways to incorporate the missing/unsupported use-cases that python3-netifaces handles for  VMWare into cloudinit.net (and netinfo), we might be able to avoid this additional pkg dep long-term
[20:03] <blackboxsw> but, I think we need a broader discussion than just on IRC for this
[20:05] <andyf> I have plenty to keep me busy on the port for now. I’ve so far only had successful tests in SmartOS (which is another illumos distro), OmniOS (uses Nocloud) and in Azure. i have lots more testing to do.
[20:07] <gnafu> andyf: Just so you know, that's coming in as black text on a black background in my client (Irssi).  Copying/pasting into a text editor revealed the secret messages ;-).
[20:07] <gnafu> I suppose some folks' clients have a white background, so it's probably readable for them.
[20:08] <akutz> I have a black bg as well.
[20:08] <akutz> I had no idea that the andyf had shared anything I didn't see :)
[20:08] <akutz> Thanks gnafu!
[20:09] <gnafu> :-)
[20:09] <akutz> Yeah, I'm happy to help figure out how to remove the netifaces dependency. I have another colleague at VMware that's also willing to help, and I believe he's working on something right now. Still, I'd like to use this opportunity to figure out if there's a way to move some of the logic into core cloud-init, such as updating the persisted instance metadata with information about a network brought up via DHCP (the primary reason we use netifaces).
[20:20] <akutz> It works:
[20:20] <akutz> 2021-08-24 20:18:32,348 - __init__.py[DEBUG]: Looking for data source in: ['VMware', 'None'], via packages ['', 'cloudinit.sources'] that matches dependencies ['FILESYSTEM']
[20:21] <akutz> # tail /var/run/cloud-init/ds-identify.log 
[20:21] <akutz> ON_NOTFOUND=disabled
[20:21] <akutz> pid=482 ppid=460
[20:21] <akutz> is_container=false
[20:21] <akutz> is_ds_enabled(IBMCloud) = true.
[20:21] <akutz> Running on vmware but rpctool query returned 1: No value found
[20:21] <akutz> is_ds_enabled(IBMCloud) = true.
[20:21] <akutz> ec2 platform is 'Unknown'.
[20:21] <akutz> check for 'VMware' returned found
[20:21] <akutz> Found single datasource: VMware
[20:21] <akutz> [up 8.32s] returning 0
[20:22] <akutz> That's with the VMDK from http://www.cloud-images.ubuntu.com/impish/current/
[20:22] <akutz> and no additional changes needed
[20:23] <blackboxsw> akutz: excellent. and `cloud-init status --long` to check for errors/and discovered cloud from python
[20:23] <akutz> FWIW, if anyone wants to test it locally using Workstation or Fusion, all I did was add the following keys to the VMX file for the VM: guestinfo.userdata = "<base64 cloud-config>", guestinfo.userdata.encoding  = "base64", guestinfo.metadata = "<base64 DS metadata from the VMware DS docs>", guestinfo.metadata.encoding  = "base64"
[20:24] <akutz> blackboxsw: Oh yeah, lots of errors re: the SSH key imports
[20:24] <akutz> My SSH key worked, but cloud-init wasn't happy for some reason.
[20:25] <blackboxsw> interesting, may be worth a bug. some ssh key handling was touched this release. I'd be curious about tracebacks in /var/log/cloud-init.log
[20:31] <akutz> Yep. I got distracted. Will post a gist.
[20:32] <akutz> status: error
[20:32] <akutz> time: Tue, 24 Aug 2021 20:18:41 +0000
[20:32] <akutz> detail:
[20:32] <akutz> ('ssh-import-id', ProcessExecutionError("Unexpected error while running command.\nCommand: ['sudo', '-Hu', 'mandy', 'ssh-import-id', 'None']\nExit code: 1\nReason: -\nStdout: -\nStderr: -"))
[20:32] <akutz> ('ssh-authkey-fingerprints', OSError(5, 'Input/output error'))
[20:32] <akutz> ('keys-to-console', OSError(5, 'Input/output error'))
[20:33] <akutz> blackboxsw: Here's a link to the log and output log - https://gist.github.com/akutz/0944edd37c094030c2b57556add1cbdd
[20:34] <akutz> blackboxsw: just updated the gist to include the userdata and metadata yamls
[20:38] <blackboxsw> akutz: in your example ssh_import_id: None actually is the string "None" not the null that was intended
[20:38] <blackboxsw> so it gets translates on yaml.load to "None" instead of None
[20:38] <blackboxsw> which is why the tracebacks are showing failures to run ['sudo', '-Hu', 'mandy', 'ssh-import-id', 'None']
[20:38] <blackboxsw> I thjink
[20:41] <blackboxsw> can confirm that same failure locally with that userdata "ssh_import_id: None" instead of "ssh_import_id: null" 
[20:49] <falcojr> you can just remove the ssh_import_id key if you're not using it
[21:59] <akutz> Ack, thanks!
[22:06] <akutz> Yeah, works sans any errors with null, thanks again!
[22:38] <blackboxsw> great to hear thx for the extra validation akutz loooks like our Bionic, Focal and Hirsute packages just got accepted into *-proposed too. we'll be able to start our SRU validation process tomorrow on all clouds
[22:38] <akutz> No problem, thanks again blackboxsw!
[22:39] <blackboxsw> ditto