[15:39] <falcojr> beantaxi: Generally mostly business hours for reviews
[15:41] <falcojr> I don't really have a great answer for your other question
[15:58] <hackedhead> so, the docs make a scary note about using hashed passwd's in a cloud-init file, referencing that the example is crackable in mere seconds. But, unless I'm confused, the format used is the exact same format used in /etc/shadow, so they're no riskier (format wise) than revealing /etc/shadow. I was not able to reproduce the crack in seconds (and I tried for a few minutes), but was able to crack
[15:58] <hackedhead> known trivial passwords with the JtR method, using wordlists that I knew would be suitable for my example. So my question: is the _format_ of the passwd body fundamentally unlike /etc/shadow in a subtle way? Or do I just need to not use bad passwords that are going to appear on common wordlists? (Is the example weak because the format is weak? or because the chosen example password is weak
[15:58] <hackedhead> relative to known wordlists/cracking schemes?)
[15:59] <hackedhead> reference: line 66 from the first example here: https://cloudinit.readthedocs.io/en/latest/topics/examples.html
[15:59] <hackedhead> also, if I _do_ trust the connection: is this moot?
[16:03] <minimal> hackedhead: I guess how "unbreakable" such a password is depends on the algo you use - whether sha512, bcrypt, etc
[16:03] <minimal> the cloud-init docs refer to regex (r'\$(1|2a|2y|5|6)(\$.+){2}') is used to determine if a password value should be treated as a hash
[16:04] <minimal> where the $1 or $2a, or $2y, or $5, or $6 refer to supported/recognised hash algos
[16:52] <hackedhead> but that's literally how the OS does it as well.
[16:53] <hackedhead> I mean, the example reference's sha512, which should be quite secure.
[16:54] <hackedhead> but if i pick the password "abc123" and then run it though 4096 rounds of sha512, and then try to crack it with a wordlist that has "abc123" in the first two hundred entries... it's gonna get craked real fast (seconds)
[16:54] <hackedhead> but that is the result of picking a bad password, not _anything_ to with the algo
[16:54] <minimal> hackedhead: different algos have different resistance to bruteforce attacks
[16:54] <hackedhead> if I picked a string of 16 random characters, it'd hold up ust fine.
[16:54] <hackedhead> I understand that, yes
[16:55] <hackedhead> the question remains: am I _incorrect_ in my udnerstanding that the format in the cloudinit file is the same as the one in /etc/shadow?
[16:56] <hackedhead> becasue they sure look identical.
[16:56] <minimal> it still is related to the algo - some algos (BCRYPT? PBKDF) are specifically designed to be "CPU" expensive to do those sort of attacks
[16:56] <hackedhead> sure, agreed
[16:56] <hackedhead> but that's not part of my question.
[16:56] <minimal> yes I believe it is the same - although as I pointed out the regex may mean cloud-init might not treat some algo hashes appropriately
[16:57] <holmanb> @blackboxsw - I followed your recommended suggestion in PR #1203, but it looks like `build-package;sbuild .... cloud-init*.dsc` fails (in the unittests) and I don't know if it's due to a mistake setting up my chroot config or if I made a mistake in the process
[16:57] <hackedhead> minimal: if the format is the same (as we both expet/assume) cloud init doesn't ahve to know anyhting at all about the algos. it just has to drop the text into the apsswd file in the right place
[16:58] <hackedhead> which is actually ery good for cloud init, because it can just be naive.
[16:58] <hackedhead> my concern is that the docs seem to imply cloud-init is doing some _else_ that's weaker, but the warning is really just "don't take you're cloud-init config anywhere you'd not take /etc/shadow"
[16:59] <hackedhead> the "we cracked the example in 12 seconds" is completely misleading, in that case.
[16:59] <hackedhead> because it implies something about the example password, but could get interpreted as implying something about cloud init
[16:59] <minimal> hackedhead: my point was that is you take YESCRYPT as an example I believe all its hashes begin with "$y$" which the above regex not NOT match
[17:00] <minimal> and therefore cloud-init will not treat it as a hash, so I guess it will treat it as a unencrypted "raw" (i.e. plaintext) password
[17:00] <hackedhead> oh i see, vs treating it as plaintext and hashing the text itself.
[17:00] <hackedhead> okay, fair enough
[17:00] <minimal> that's just based on looking at the docs, not at the code
[17:01] <hackedhead> I guess my reaction was to the docs saying "you can put a hash here, but <scary> hashes aren't safe </scary>"  ... but they are as safe as it gets, really.
[17:02] <hackedhead> now, you still don't want them on the wire in the clear, if you can help it.
[17:02] <minimal> I think in general what the docs are "flagging" is that as user-data is potentially accessible to other parties then as you illustrated weak passwords with certain hash algos risk being "broken" by 3rd parties
[17:02] <hackedhead> but they _are_ cryptographic hashses, they're whole deal is to obscure the plaintext.
[17:03] <hackedhead> right. I feel like the warning needs to focus on the wire-access and not on how fast an example password gets cracked.
[17:04] <minimal> you mentioned /etc/shadow - in order to try and crack passwords there someone needs access to a machine, whereas with user-data (i.e. provided via metadata server) they do not need access to the machine so the potential risk is increased
[17:09] <holmanb> hackhead: The wording could be more clear about the risks associated with using passwd, I agree. For reference, the implementation passes the value to `adduser --password <value>` (seee cloudinit/distros/__init__.py)
[17:12] <minimal> holmanb: I guess my point was that regex is in the code and so restricts the recognised hash algos: https://github.com/canonical/cloud-init/blob/main/cloudinit/config/cc_set_passwords.py#L168
[17:13] <falcojr> minimal: that's actually a different implementation for a different module
[17:14] <falcojr> for user definitions, passwd gets passed straight to useradd
[17:15] <minimal> right,, but this can also be used to set user password via chpasswd: list: 
[17:15] <falcojr> correct
[17:16] <minimal> I would have expected both places to act in the same fashion
[17:27] <blackboxsw> holmanb: checking now.
[17:27] <blackboxsw> didn't realize my IRC proxy logged me out
[17:28] <holmanb> blackboxsw: all good
[17:45] <holmanb> minimal: That module suffers from having to disambiguate different data types from one list since it supports encrypted, unencrypted, and random passwords. At first blush, a more explicit schema is one way I might approach improving that module (a dict of lists, perhaps?).
[17:47] <minimal> holmanb: yeah its trying to use "magic" (regex) which always suffers from not being up-to-date with things like new hash algos.
[17:48] <minimal> Then again personally I'd never
[17:48] <minimal> oopps, I only use SSH key access to machines so no worry about passwd hashes
[17:50] <hackedhead> yeah, not trying to magically tell the difference between plain and cipher text would be safer.
[17:50] <hackedhead> thanks everyone, for improving my understanding here.
[17:51] <minimal> yeah, why can't I see my password to be "$6$afrtgyhuj" :-)
[17:54] <blackboxsw> holmanb: when I take your bionic PR I can confirm unittest failure during  sbuild --resolve-alternatives --dist=bionic --arch=amd64  --arch-all ../out/cloud-init_*dsc`  I can also see it locally outside of package build with `quilt push -a; tox -e py3 tests/unittests/sources/test_openstack.py` Its the default: False on Bionic for JSON network config. I'm trying to figure out why that unittest works on upstream/ubuntu/bionic
[17:55] <blackboxsw> minimal:  "$6$afrtgyhuj" == "hunter2" :)
[17:56] <blackboxsw> ... cloudn't resist the meme reference. apologies
[17:59] <blackboxsw> ahh holmanb, your quilt patch update patched the wrong unittest to apply_network_config: True. which is why the unittests are failing during package build.
[18:00] <minimal> blackboxsw: no worries, the ref was lost on me as I didn't use IRC back then lol
[18:02] <blackboxsw> minimal: heh, I assumed as much. It was a bit oblique and dated :)
[18:02] <minimal> its not the dated bit - I've been online for a very long time :-)
[18:03] <blackboxsw> holmanb: I think our Bionic PR should have patched test/unittests/sources/test_openstack.py::test_network_config_from_network_json instead of test_network_config_disabled_by_datasource_config
[18:15] <blackboxsw> holmanb: posted a diff comment to your PR for bionic. not sure why I thought I saw success friday
[18:16] <blackboxsw> then we can merge and kick off CI package build recipes for bionic
[19:16] <holmanb> @blackboxsw: I just pushed the requested changes. Lintian failed, but since the final status is successful I assume that doesn't matter.
[19:34] <blackboxsw> holmanb: running locally here. Thanks for bringing up the lintian aspect of things. What we care about in lintian mostly on older Ubuntu releases is that we don't introduce new errors and or warnings. In this case, we did introduce one new warning `W: cloud-init: spelling-error-in-changelog propery property`. Let's fix that typo in debian/changelog and git rebase squash it into the commit 
[19:34] <blackboxsw> 4006ff2ceb37369dbd097aa70a69090933783200
[19:35] <blackboxsw> we'll have to remember that typo fixup too when we release into Jammy or other releases too, but as long as we watch for lintian warns/errors increasing we'll continue to catch it
[19:37] <blackboxsw> I don't care much about preserving separate commits to d/changelog for typo-changes etc unless there are significant changesets that we find we want to git cherry-pick across ubuntu/* release branches.
[19:37] <blackboxsw> hence the suggestion to squash in this case. But if you prefer otherwise all you
[19:45] <holmanb> ahh, that one got added after this pr I see
[19:45] <holmanb> I can squash
[19:57] <holmanb> err, scratch that I see what you mean
[20:13] <holmanb> @blackboxsw: lintian still complains about s/propery/property/ after squash, not sure why :/
[20:14] <blackboxsw> that's sure funky. checking your pushed PR
[20:14] <holmanb> blackboxsw: just pushed the latest, might need to refresh
[20:15] <holmanb> oh wait if I didn't rerun build-package that may have done it perhaps?
[20:16] <blackboxsw> ok yes I was just going to ask if you rebuilt the source tgz using build-package 
[20:17] <blackboxsw> that'd be it. otherwise some cases might show up during package build too if you didn't git commit before you used build-package as it ignores local uncommitted changes
[20:17] <blackboxsw> I'm waiting on sbuild, but your commit change looks good
[20:17] <holmanb> I don't think I did after the squash. Rerunning sbuild now, I should know momentarily if it's good
[20:18] <holmanb> spelling warning is gone now for me
[20:19] <blackboxsw> which begs the question, what kind of system are you running builds on as you beat me by quite a bit
[20:19] <holmanb> I started the rebuild a little before mentioning it here ;)
[20:19] <holmanb> lets go with that
[20:19] <blackboxsw> may be time for me to buy myself a hardware upgrade. I'm still awaiting sbuild and I think I kicked it off a min or two before I told you about the re-build-pkg
[20:19] <blackboxsw> ahh same
[20:20] <blackboxsw> yep new lintian warning gone
[20:20] <blackboxsw> +1-ing, up for a 2 min hangout?
[20:20] <holmanb> absolutely
[20:54] <blackboxsw> woo hoo holmanb successful bionic build. thanks for the fix https://code.launchpad.net/~cloud-init-dev/+archive/ubuntu/daily/+recipebuild/2945452 
[20:54] <blackboxsw> will kickoff integration tests once that package shows up as published in the daily PPA
[20:55] <holmanb> awesome, thanks!
[22:56] <raidandfade> Howdy folks, hate to drop in but I've been stumped on a cloud-init x openstack issue for a bit now and figured i'd see if any pros can help out - I can't seem to get it to load network_data.json from the metadata service on Ubuntu 18.04 - On ubuntu 20.04, with the same metadata structure, the network_data.json is loaded and implemented into
[22:56] <raidandfade> netplan. I've tracked it down to a single debug log  message "network config provided via network_json" which is logged on 20.04 but not on 18.04, i'm not sure where to go from here.
[22:57] <raidandfade> Both are running the same 21.4 version of cloud-init from their respective apt repositories, i've also checked back to version 20.3 which seems to reproduce the same behavior.
[22:59] <blackboxsw> raidandfadethanks for pining here. That behavior of Ubuntu 18.04 (bionic) is as intended. 
[22:59] <blackboxsw> raidandfade: cloud-init can't change default behavior on stable Ubuntu release based what major features were originally present on 18.04 when it released. This is because  we don't want to surprise people/scripts etc that expect cloud-init  to automatically start rendering network on 18.04 due to network_data.json once we added that feature upstream
[22:59] <blackboxsw> raidandfade: that said, on older/stable releases we expose configuration options that allow someone to "enable" such configuration if desired
[23:00] <raidandfade> Ah, I figured it would be something like that. How do I go about enabling network_data?
[23:00] <blackboxsw> raidandfade: what would be needed on the 18.04 image that you are consuming is the following datasource configuration:
[23:01] <blackboxsw> datasource:                                                                     
[23:01] <blackboxsw>   OpenStack:                                                                    
[23:01] <blackboxsw>     apply_network_config: true   
[23:01] <blackboxsw> somewhere in /etc/cloud/cloud.cfg.d/some-file.cfg on your filesystem
[23:02] <blackboxsw> this will tell 18.04 version of cloud-init to consume network_data.json
[23:02] <raidandfade> Simple enough! I figure I may as well plop this on all of my images to avoid future headache
[23:02] <blackboxsw> as documented here https://cloudinit.readthedocs.io/en/latest/topics/datasources/openstack.html
[23:02] <blackboxsw> rather https://cloudinit.readthedocs.io/en/latest/topics/datasources/openstack.html#configuration
[23:02] <blackboxsw> though docs are sparse on that feature :)
[23:03] <raidandfade> blackboxsw: I appreciate the quick help and response, you've saved me another few hours of headache!
[23:03] <blackboxsw> no worries, that's what we are here for
[23:03] <blackboxsw> it helps others too
[23:03] <raidandfade> Yeah, I looked at the docs pages for the datasource as well, it doesn't really explain that it's required, and "default: True" isnt helping the confusion
[23:03] <raidandfade> Makes sense as for why it's not default:True at all times, but the docs certainly didn't help figure that out
[23:04] <blackboxsw> absolutely. yeah it's magical how older released "don't"  actually have that default :/
[23:04] <blackboxsw> we found it hard to figure out the best way to document that particular feature and whether to actually document older versions in tip of main docs :/
[23:05] <raidandfade> Is there a changelog sort of page that has "This feature is only on OS newer than <x>" that i should be aware of?
[23:08] <blackboxsw> raidandfade: sadly not exactly. From upstream major.minor version perspective we publish general changelogs for the upstream releases at URLs like https://launchpad.net/cloud-init/trunk/21.4  as found from https://launchpad.net/cloud-init/trunk. But individual features that are disabled on stable release isn't somethat that I think we have captured well
[23:09] <blackboxsw> from Ubuntu perspective at Canonical I can say we can try to better represent that and I'll talk to the team about it. But we don't exactly have visibility to whether other distribution downstreams have disabled certain features.
[23:10] <blackboxsw> raidandfade: for what it's worth: you can quickly check in github what few features Ubuntu has disabled on earlier releases by looking for the patches listed in https://github.com/canonical/cloud-init/tree/ubuntu/bionic/debian/patches or https://github.com/canonical/cloud-init/tree/ubuntu/focal/debian/patches
[23:10] <raidandfade> Oh, I think I might've been confused on who decided to make this default false- If it's a decision that the distribution makes then I can surely understand the difficulty of documenting it
[23:11] <blackboxsw> yeah different distibutions have a different opinion of how much feature risk is reasonable to take with the cloud-init package
[23:11] <blackboxsw> that said, as you can see above, we don't carry any "feature disable" patches for focal 20.04 currently for new feature developments.
[23:12] <blackboxsw> because the focal/debian/patches link 404s (because we don't carry any patch delta there)
[23:12] <raidandfade> Yeah, I see that. Convenient
[23:12] <blackboxsw> hope it helps abbit
[23:15] <raidandfade> You've certainly linked a few good tools for me to use, and given some insight on why this behavior exists in the first place. I've still got a couple more images to create (trying to automate our image generation for most LTS distros i.e. ubuntu fedora debian) so I've got my work cut out for me as far as that goes.
[23:16] <raidandfade> Thanks again for the help!
[23:16] <blackboxsw> raidandfade: nice, do feel free to let us know any pain points in that process. We'll try to iron out what we can
[23:17] <raidandfade> So far this was the only major problem, i ran into some other small quirks along the way but most of that would be because of my particular design I imagine (for example, we do accept_ra with a static ipv6 allocation on VMs, and cloud_init assumes that impossible)
[23:17] <raidandfade> Just patched the source tree myself for that one, a simple False -> True