| sassyn | hi | 08:59 |
|---|---|---|
| sassyn | good morning. | 08:59 |
| sassyn | Is there any way to make sure cloud-init init command will recreate the netplan file | 08:59 |
| sassyn | in a custom way? not just have the first eth0 config with DHCP? | 09:00 |
| sassyn | note that I'm using the None DataStore | 09:00 |
| meena | sassyn: what is the None DataStore? you mean NoCloud, or literally, you have no datastore? | 09:18 |
| sassyn | meena: thank you for getting back to me | 09:19 |
| sassyn | I mean this: https://cloudinit.readthedocs.io/en/latest/topics/datasources/fallback.html | 09:19 |
| meena | sassyn: i don't understand how you're supposed to provide a network config with that datasource… | 09:23 |
| sassyn | the thing is I'm importing a image to vmware ... where the image is in the state of cloud-init init ... so once I boot the image , the cloud init is running. | 09:23 |
| sassyn | What I want is to provide cloud init a way to create the netplan with math interface etc.. | 09:24 |
| sassyn | at the moment i have the 50-cloud-init.yaml being created | 09:25 |
| sassyn | with eth0 setup to dhcp | 09:25 |
| sassyn | and with the match mac address | 09:25 |
| sassyn | but my goal is to have it with 3 more interface setup in some default way | 09:26 |
| sassyn | is there anyway to porivde network init data? | 09:30 |
| sassyn | with a fake datastore ... not like NoCloud | 09:31 |
| sassyn | just files on the local file system? let's say in /opt/tmp or something? | 09:31 |
| meena | NoCloud isn't a fake datasource | 09:31 |
| meena | you provide a couple of files with stuff you wanna see by bundling it up as ISO and putting it into the machine; cloud-init identifies the source via label (cidata) and FS (iso9660), and mounts it and uses it | 09:32 |
| sassyn | https://gist.github.com/imduffy15/2d6f3cd46efa2ff68286 | 09:37 |
| sassyn | look here: this developer use None as datastore | 09:38 |
| sassyn | and still have metadata: | 09:38 |
| sassyn | sorry - I'm really a new baby with cloud init | 09:39 |
| sassyn | so if I ask stupid question. I understand the NoCloud method. But I'm tying to avoid it. | 09:39 |
| meena | sassyn: i've never seen userdata_raw: | 09:51 |
| stephan48 | can someone give me a hint on how the handling of the 169.254.169.254 or similar mechanisms work for f.e. Hetzner? how do hosters attach the info about which machine asks for the info so they can answer with only that machines details? do they have a local proxy checking source interface/mac address and attaching it to the request? | 11:04 |
| meena | stephan48: that may well be possible. i don't think we're making any requests with our instance-id; you can curl that IP and get the same info cloud-init gets | 11:09 |
| stephan48 | yes | 11:10 |
| stephan48 | because often we don't know the ID until that request happens(if it is not injected otherwise) | 11:11 |
| stephan48 | what interests me is how the hoster side of things is solved, i think i understand the cloud-init side fair enough for now | 11:11 |
| meena | stephan48: MAC address based discrimination is probably the most sensible, but i honestly wouldn't know how to route that | 14:36 |
| stephan48 | yea | 14:47 |
| stephan48 | hence the guess with a local proxy server | 14:47 |
| stephan48 | maybe even something like VRF to provide a isolated environment between VMs. grabbing the mac address and attaching it to a HTTP request via a header or param is fairly easy | 14:52 |
| stephan48 | you would need something local anyway with APIPA | 14:52 |
| meena | stephan48: we can always look at OpenStack or CloudStack do it | 16:32 |
| stephan48 | true! | 16:32 |
| stephan48 | althrough these are quite complesh projects | 16:32 |
| meena | well, yeah, i didn't wanna say OpenNebula ; cuz that's too simple? | 16:36 |
| stephan48 | will try to dig a bit this evening | 16:37 |
| stephan48 | i guess a lot of these work via metadata partitions | 16:38 |
| stephan48 | and then announcing either the full files or a url with params | 16:38 |
| === ijohnson is now known as ijohnson|lunch | ||
| blackboxsw | falcojr: I just put up https://github.com/cloud-init/ubuntu-sru/pull/181 for softlayer tests. if there is time later this week I may put up a PR to add softlayer to pycloudlib (I have vestiges of it that I wrote while waiting for these tests to run). | 19:18 |
| blackboxsw | I see you have a number of PRs up for SRU verification on ubuntu-sru falcojr, lucasmoura and I should be able to get through review on those today | 19:19 |
| falcojr | sweet, I'll review it soon. Thanks! | 19:19 |
| falcojr | and yeah, Lucas is reviewing them | 19:19 |
| blackboxsw | awesome thanks lucasmoura | 19:19 |
| meena | can anyone help me answer a few cloud-init mysteries? | 19:32 |
| meena | cloud-id is saying ec2 | 19:32 |
| meena | ds-identify is saying: datasource_list: [ Hetzner, None ] | 19:33 |
| meena | so, where does cloud-id get that idea from? | 19:37 |
| rharper | meena: cloud_id reads from instance-data.json; it grabs 'cloud_name', 'region" and 'platform' from the json file, ; see cloudinit/cmd/cloud_id.py and cloudinit/sources/__init__.py:canonical_cloud_id() | 19:42 |
| meena | aah, here we go: we're back to none; because loads of stuff in net/__init__.py is failing… | 19:45 |
| rharper | and the name comes from cloudinit/sources/__init__.py:Datasource.cloud_name property which reads looks looks for 'cloud-name' in the metadata if present, it falls back to self.dsname if they're not present ... | 19:45 |
| meena | rharper: how can i make cloud-init re-read the data? | 19:53 |
| blackboxsw | option 1: the big hammer: cloud-init clean --reboot --logs | 19:54 |
| blackboxsw | option 2: rm /var/lib/cloud/instance/obj.pkl I believe | 19:55 |
| meena | blackboxsw: option 1 doesn't work; and i'm very slowly working on fixing that… | 19:55 |
| rharper | reread metadata ? | 19:55 |
| rharper | or regenerate instance-data.json ? | 19:55 |
| blackboxsw | ohh I was interpreting re-detect the datasource | 19:55 |
| rharper | meena: I think re-reading metadata and then dumping instance-data.json ; I'd likely rm /var/lib/cloud/instance/obj.pkl; and then just re-run: cloud-init init --local (or cloud-init init if it's a network datasource vs. local datasource) | 19:58 |
| meena | blackboxsw: i'm forcing the datasource via /usr/local/etc/cloud/cloud.cfg.d/cloud.cfg | 19:58 |
| meena | datasource_list: [ Hetzner, None ] | 19:58 |
| === ijohnson|lunch is now known as ijohnson | ||
| meena | because a lot of things assume linux, i have to do that, otherwise, i won't even get to the point where hetzner is attempted | 19:59 |
| meena | now, ec2 as fallback (thanks to standard datasource_list) works *somewhat* but not completely | 19:59 |
| meena | and Hetzner does some funky stuff in before trying to fetch the metadata | 20:00 |
| meena | 2020-12-14 20:00:31,325 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceHetzner.DataSourceHetzner'> failed | 20:00 |
| blackboxsw | meena to iterate on get_data: `sudo python3 -c 'from cloudinit.stages import _pkl_load; print(_pkl_load("/var/lib/cloud/instance/obj.pkl").get_data())'` | 20:01 |
| blackboxsw | that would let you keep loading the detected datasource from the cmdline and re-running the "crawl_metadata" method | 20:01 |
| meena | which is why, back in the day, i tried to add a "backdoor" into fetching metadata: does network already work? skip whatever you're doing and fetch | 20:01 |
| meena | blackboxsw: nah, obj.pkl is bad. | 20:01 |
| blackboxsw | that python cmdline assumes though the datasource that is detected on the system is what you are hoping | 20:02 |
| blackboxsw | ahh roger | 20:02 |
| meena | it's either using ec2 or none; the first one works somewhat (it fetches metadata, but not userdata); the latter… well, it doesn't break anything | 20:02 |
| meena | 2020-12-14 19:41:19,602 - subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '169.254.0.1/16', 'broadcast', '169.254.255.255', 'dev', 'vtnet0'] with allowed return codes [0] (shell=False, capture=True) | 20:04 |
| blackboxsw | meena: so ultimately are you thinking you'll be able to get Hetzner detected properly without failure on BSD? Or is that unattainable?\ | 20:04 |
| meena | blackboxsw: in a year or two | 20:04 |
| blackboxsw | so January 2021?:) | 20:04 |
| rharper | nice | 20:04 |
| meena | i really don't get it, cuz ds-identify works flawlessly | 20:04 |
| rharper | meena: that looks like the ephemeral DHCP ... is networking already up when you're trying again ? | 20:05 |
| meena | it clearly identifies Hetzner, and then cloud-init itself is like, well, i dunno what to do with this stuff that ds-identify just told me, i guess i'll just ignore it | 20:05 |
| meena | rharper: yupp | 20:05 |
| meena | i'm in via SSH | 20:05 |
| meena | that was produced by cloud-init init --local btw | 20:06 |
| rharper | I see, Hetzner is special | 20:06 |
| rharper | in get_data, they don't bother checking if networking is up | 20:06 |
| rharper | and call their Ephemeral DHCP a second time | 20:06 |
| blackboxsw | meena: given that hetzner's datasource is kindof derived from EC2 I could see it kina works. I wonder if ubuntu cloud-images actually specify the datasource_list: [Hetzner] (no None item fallback in the datasource list). | 20:07 |
| rharper | meena: so, if you were to modify that locally, I'd drop the with cloudnet.Ephemeral() context manager and call the md, ud code | 20:07 |
| blackboxsw | but as you said, you are hitting linux-assumption issues in Hetzner which result in a Trace somewhere during network setup | 20:07 |
| meena | blackboxsw: lots and lots, hence the net/refactor. | 20:08 |
| rharper | blackboxsw: did we ever implement cloudinit.net.network_is_up() I know I've done something like that in the hotplug branch ... | 20:08 |
| blackboxsw | rharper: checking too. I knew someone referenced it. but it may have just been your branch | 20:08 |
| meena | what's really bothering me tho is that the ssh-keys are provided in meta_data; and when cloud-init first runs without me messing with it, it can retrieve the meta-data (i can later run cloud-init query ds and *see* it) but it doesn't store the ssh-key to the default user | 20:10 |
| rharper | they keys are stored in the datasource object; and then a separate stage pulls keys from datasource to default user | 20:11 |
| meena | https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html#v1-public-ssh-keys ← there's actually no documentation on what to do with those, so, never mind | 20:11 |
| blackboxsw | rharper: there is EphemeralDHCPv4(connectivity_url=<someurl_to_test>) | 20:11 |
| rharper | so you'd need to re-run the user_groups mod ... | 20:11 |
| blackboxsw | which could be provided and would NOOP if already available | 20:12 |
| rharper | blackboxsw: ah, right, that hits a URL before brnging up | 20:12 |
| blackboxsw | yeah | 20:12 |
| rharper | that should work | 20:12 |
| meena | blackboxsw: i added that. | 20:12 |
| meena | is it not used by Hetzner? | 20:12 |
| blackboxsw | though not consumed anywhere at the moment | 20:13 |
| * meena 🤦🏻♀️️ | 20:13 | |
| meena | i added it, and didn't, use it? | 20:13 |
| blackboxsw | I think it was groudwork that ended up being used by another utility function on Azure at one point, but I think it didn't get general reference in other datasources. | 20:13 |
| meena | ef0611a51a98a273cfa37b0daeb3e9d151888b88 | 20:14 |
| meena | so, when i tried to provide a patch to use that, i believe smoser or rharper where against using the md_url as target, because it may not be available at that point | 20:15 |
| rharper | upstream it looks like it needs to pass connectivity_url=<value> | 20:15 |
| rharper | which is true at local time... I preferred a network is up | 20:15 |
| rharper | something like checking if you ahve a default route or an ip on an interface | 20:16 |
| rharper | there's no point in trying a URL with a timeout if networking isn't even up | 20:16 |
| meena | rharper: aye. it was a hack, because so much of net/ was unusable on non-linux | 20:18 |
| blackboxsw | we also now have the availability of distro.networking.is_up(devname) thx to that networking work that meena/Odd_Bloke spent some time one | 20:20 |
| blackboxsw | *on | 20:20 |
| blackboxsw | so we should be able to pre-flight check that from the datasource self.distro.networking.is_up(<fallbackdev>) I think? | 20:21 |
| blackboxsw | and avoid the EphemeralDCHPv4 context manager if we are up | 20:21 |
| blackboxsw | we/the dev is/ | 20:21 |
| meena | weird, still no user-data | 20:22 |
| meena | and i'm pretty, and sure, i provided some… | 20:23 |
| meena | tho, it was literally just overriding system: distro: freebsd | 20:23 |
| meena | how do i override or disable vendor-data? | 20:25 |
| rharper | meena: one other thought; on linux hetzner instance, if you were to refactor the _getadata such that the metadata-crawl method ran outside of _get_data, one could call it directly, see: cloudinit/sources/DataSourceGCE.py ); verify that refactor works on linux and then see where it fails on bsd | 20:25 |
| meena | rharper: *nod *nod | 20:26 |
| rharper | meena: vendor-data: {enabled: False} in user-data | 20:28 |
| meena | rharper: thanks | 20:29 |
| * meena puts that into her default hetzner snippet | 20:29 | |
| meena | whelp, i don't need to override system_info, if i disable vendor-data… | 20:31 |
| meena | rharper: can we document that somewhere? | 20:31 |
| meena | this mentions part of it, https://cloudinit.readthedocs.io/en/latest/topics/vendordata.html#input-formats but uses a different key than you do | 20:31 |
| rharper | meena: well, that's disabling part of vendor-data, vs disabling processing of all of it | 20:35 |
| rharper | it should be documented there for complete disabling, | 20:35 |
| rharper | it mentions that you can and fails to include the bits to show how to do that | 20:35 |
| AU-S2S | I'm trying to add the contrib and non-free repositories to a yaml that will be pasted into a AWS EC2 Launch Template User data. | 20:44 |
| AU-S2S | This is against the latest Debian Buster AMI. My YAML looks like this: | 20:44 |
| AU-S2S | # set up apt to include contrib and non-free | 20:44 |
| AU-S2S | apt: | 20:44 |
| AU-S2S | preserve_sources_list: false | 20:44 |
| AU-S2S | disable_suites: $RELEASE, $RELEASE-updates, backports | 20:44 |
| AU-S2S | sources_list: | | 20:44 |
| AU-S2S | deb http://cdn-aws.deb.debian.org/debian buster main contrib non-free | 20:44 |
| AU-S2S | deb-src http://cdn-aws.deb.debian.org/debian buster main contrib non-free | 20:44 |
| AU-S2S | deb http://cdn-aws.deb.debian.org/debian buster-updates main contrib non-free | 20:44 |
| AU-S2S | deb-src http://cdn-aws.deb.debian.org/debian buster-updates main contrib non-free | 20:44 |
| AU-S2S | deb http://security.debian.org/debian-security buster/updates main contrib non-free | 20:44 |
| blackboxsw | lucasmoura: falcojr another SRU pr https://github.com/cloud-init/ubuntu-sru/pull/183 for maas logs from automated tests | 20:46 |
| AU-S2S | There are no errors in /var/log/cloud-init..log | 20:46 |
| blackboxsw | AU-S2S: what version of cloud-init on that image? cloud-init -v ? | 20:46 |
| AU-S2S | HMM.. stand by. | 20:47 |
| blackboxsw | also in the /var/log/cloud-init.log ( Cloud-init v. XXX | 20:47 |
| AU-S2S | 2020-12-14 19:46:17,329 - util.py[DEBUG]: Cloud-init v. 20.2 running 'init-local' | 20:48 |
| blackboxsw | cool. | 20:48 |
| blackboxsw | AU-S2S: if you have access to the machine that booted, `sudo cloud-init query userdata` it should show that the userdata was seen by the vm properly | 20:50 |
| AU-S2S | I'm telling the launch template that the user data is not encoded (which it's not, just plain text). | 20:50 |
| blackboxsw | also AU-S2S does the first line of your user-data start with #cloud-config? | 20:50 |
| blackboxsw | if not cloud-init will ignore it | 20:50 |
| AU-S2S | #cloud-config | 20:51 |
| AU-S2S | # https://cloudinit.readthedocs.io/ | 20:51 |
| AU-S2S | # https://salsa.debian.org/cloud-team/debian-cloud-images | 20:51 |
| AU-S2S | # Set up config for EC2 | 20:51 |
| AU-S2S | datasource: | 20:51 |
| AU-S2S | Ec2: | 20:51 |
| AU-S2S | timeout : 50 | 20:51 |
| AU-S2S | max_wait : 120 | 20:51 |
| AU-S2S | metadata_urls: | 20:51 |
| AU-S2S | - http://169.254.169.254:80 | 20:51 |
| AU-S2S | - http://instance-data:8773 | 20:51 |
| AU-S2S | :-D | 20:51 |
| AU-S2S | I've run that command before, but I don't remember the output (last week). Give me a bit to re-launch and capture it again. | 20:52 |
| blackboxsw | hrm, that looks like the content of something in /etc/cloud/cloud.cfg or cloud.cfg.d/ not what I'd expect from the commandline via `cloud-init query userdata` | 20:52 |
| blackboxsw | no worries | 20:52 |
| AU-S2S | Yea, that's the top of my yaml. | 20:53 |
| blackboxsw | ok, that said, I don't recall if the user-data can actually set the datasource. so specifying datasource: keys in your user-data YAML may be worthless. | 20:54 |
| blackboxsw | s/worthless/ignored. | 20:54 |
| blackboxsw | yeah it's possible that however the template data is being presented to the vm, maybe there was stray whitespace at the top or a missing header line that causes cloud-init to ignore it without a specific error. | 20:59 |
| AU-S2S | admin@test1-debian:~$ sudo cloud-init query userdata | 21:03 |
| AU-S2S | <snip> | 21:03 |
| AU-S2S | # set up apt to include contrib and non-free | 21:03 |
| AU-S2S | apt: | 21:03 |
| AU-S2S | preserve_sources_list: false | 21:03 |
| AU-S2S | disable_suites: $RELEASE, $RELEASE-updates, backports | 21:03 |
| AU-S2S | sources_list: | | 21:03 |
| AU-S2S | deb http://cdn-aws.deb.debian.org/debian buster main contrib non-free | 21:03 |
| AU-S2S | deb-src http://cdn-aws.deb.debian.org/debian buster main contrib non-free | 21:03 |
| AU-S2S | deb http://cdn-aws.deb.debian.org/debian buster-updates main contrib non-free | 21:03 |
| AU-S2S | deb-src http://cdn-aws.deb.debian.org/debian buster-updates main contrib non-free | 21:03 |
| AU-S2S | deb http://security.debian.org/debian-security buster/updates main contrib non-free | 21:03 |
| AU-S2S | and when I check sources.list | 21:04 |
| AU-S2S | admin@test1-debian:~$ cat /etc/apt/sources.list | 21:04 |
| AU-S2S | deb http://cdn-aws.deb.debian.org/debian buster main | 21:04 |
| AU-S2S | deb-src http://cdn-aws.deb.debian.org/debian buster main | 21:04 |
| AU-S2S | deb http://security.debian.org/debian-security buster/updates main | 21:04 |
| AU-S2S | deb-src http://security.debian.org/debian-security buster/updates main | 21:04 |
| AU-S2S | deb http://cdn-aws.deb.debian.org/debian buster-updates main | 21:04 |
| AU-S2S | deb-src http://cdn-aws.deb.debian.org/debian buster-updates main | 21:04 |
| AU-S2S | deb http://cdn-aws.deb.debian.org/debian buster-backports main | 21:04 |
| AU-S2S | deb-src http://cdn-aws.deb.debian.org/debian buster-backports main | 21:04 |
| AU-S2S | Aha! looks like I have an error in when cc_apt_configure.py tries to convert V2 to V3. But I don't understand what the error is telling me. | 21:14 |
| AU-S2S | "ValueError: Old and New apt format defined with unequal values True vs False @ apt_preserve_sources_list" | 21:18 |
| AU-S2S | Is True/False case sensitive? | 21:18 |
| blackboxsw | it means that you have two keys in your user-data or vendor-data that are suggesting different values for preserve_sources_list 'sudo cloud-init query userdata| grep preserve_sources_list' # do you see two settings, one false and one true? | 21:25 |
| blackboxsw | cloud-init is telling you it can't decide which config value to use (and the older one is a key named "apt_preserve_sources_list" which I think got deprecated in cloud-init 0.7.9 | 21:27 |
| AU-S2S | On the grep - no, just one: admin@test1-debian:/var/log$ sudo cloud-init query userdata| grep preserve_sources_list | 21:27 |
| AU-S2S | preserve_sources_list: false | 21:27 |
| AU-S2S | Let me query the vendor data.. | 21:28 |
| blackboxsw | AU-S2S: I'm guessing as well if you have two different kinds of userdata/vendordata keys managing preserve_source_list you might also have two keys trying to set apt sources up, maybe look for something under apt_sources key too? | 21:31 |
| blackboxsw | the config key "apt_sources" would be the "old" way to setup apt sources in cloud-init (for cloud-init version 0.7.5 I think) | 21:31 |
| AU-S2S | Bingo!! In /etc/cloud/cloud.cfg | 21:34 |
| AU-S2S | # This preverts apt/sources.list to be updated at boot time, which | 21:34 |
| AU-S2S | # may be annoying. | 21:34 |
| AU-S2S | apt_preserve_sources_list: true | 21:34 |
| AU-S2S | Can I change my yaml to use the old variable name? | 21:37 |
| AU-S2S | BOOM!!! :-D | 21:49 |
| AU-S2S | admin@test1-debian:~$ cat /etc/apt/sources.list | 21:49 |
| AU-S2S | deb http://cdn-aws.deb.debian.org/debian buster main contrib non-free | 21:49 |
| AU-S2S | deb-src http://cdn-aws.deb.debian.org/debian buster main contrib non-free | 21:49 |
| AU-S2S | deb http://cdn-aws.deb.debian.org/debian buster-updates main contrib non-free | 21:49 |
| AU-S2S | deb-src http://cdn-aws.deb.debian.org/debian buster-updates main contrib non-free | 21:49 |
| AU-S2S | deb http://security.debian.org/debian-security buster/updates main contrib non-free | 21:49 |
| AU-S2S | deb-src http://security.debian.org/debian-security buster/updates main contrib non-free | 21:49 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!