sassyngood morning.08:59
sassynIs there any way to make sure cloud-init init command will recreate the netplan file08:59
sassynin a custom way? not just have the first eth0 config with DHCP?09:00
sassynnote that I'm using the None DataStore09:00
meenasassyn: what is the None DataStore? you mean NoCloud, or literally, you have no datastore?09:18
sassynmeena: thank you for getting back to me09:19
sassynI mean this: https://cloudinit.readthedocs.io/en/latest/topics/datasources/fallback.html09:19
meenasassyn: i don't understand how you're supposed to provide a network config with that datasource…09:23
sassynthe thing is I'm importing a image to vmware ... where the image is in the state of cloud-init init ... so once I boot the image , the cloud init is  running.09:23
sassynWhat I want is to provide cloud init a way to create the netplan with math interface etc..09:24
sassynat the moment i have the  50-cloud-init.yaml being created09:25
sassynwith eth0 setup to dhcp09:25
sassynand with the match mac address09:25
sassynbut my goal is to have it with 3 more interface setup in some default way09:26
sassynis there anyway to porivde network init data?09:30
sassynwith a fake datastore ... not like NoCloud09:31
sassynjust files on the local file system? let's say in /opt/tmp or something?09:31
meenaNoCloud isn't a fake datasource09:31
meenayou provide a couple of files with stuff you wanna see by bundling it up as ISO and putting it into the machine; cloud-init identifies the source via label (cidata) and FS (iso9660), and mounts it and uses it09:32
sassynlook here: this developer use None as datastore09:38
sassynand still have metadata:09:38
sassynsorry - I'm really a new baby with cloud init09:39
sassynso if I ask stupid question. I understand the NoCloud method. But I'm tying to avoid it.09:39
meenasassyn: i've never seen userdata_raw:09:51
stephan48can someone give me a hint on how the handling of the or similar mechanisms work for f.e. Hetzner? how do hosters attach the info about which machine asks for the info so they can answer with only that machines details? do they have a local proxy checking source interface/mac address and attaching it to the request?11:04
meenastephan48: that may well be possible. i don't think we're making any requests with our instance-id; you can curl that IP and get the same info cloud-init gets11:09
stephan48because often we don't know the ID until that request happens(if it is not injected otherwise)11:11
stephan48what interests me is how the hoster side of things is solved, i think i understand the cloud-init side fair enough for now11:11
meenastephan48: MAC address based discrimination is probably the most sensible, but i honestly wouldn't know how to route that14:36
stephan48hence the guess with a local proxy server14:47
stephan48maybe even something like VRF to provide a isolated environment between VMs. grabbing the mac address and attaching it to a HTTP request via a header or param is fairly easy14:52
stephan48you would need something local anyway with APIPA14:52
meenastephan48: we can always look at OpenStack or CloudStack do it16:32
stephan48althrough these are quite complesh projects16:32
meenawell, yeah, i didn't wanna say OpenNebula ; cuz that's too simple?16:36
stephan48will try to dig a bit this evening16:37
stephan48i guess a lot of these work via metadata partitions16:38
stephan48and then announcing either the full files or a url with params16:38
=== ijohnson is now known as ijohnson|lunch
blackboxswfalcojr: I just put up https://github.com/cloud-init/ubuntu-sru/pull/181 for softlayer tests. if there is time later this week I may put up a PR to add softlayer to pycloudlib (I have vestiges of it that I wrote while waiting for these tests to run).19:18
blackboxswI see you have a number of PRs up for SRU verification on ubuntu-sru falcojr, lucasmoura and I should be able to get through review on those today19:19
falcojrsweet, I'll review it soon. Thanks!19:19
falcojrand yeah, Lucas is reviewing them19:19
blackboxswawesome thanks lucasmoura19:19
meenacan anyone help me answer a few cloud-init mysteries?19:32
meenacloud-id is saying ec219:32
meenads-identify is saying: datasource_list: [ Hetzner, None ]19:33
meenaso, where does cloud-id get that idea from?19:37
rharpermeena: cloud_id reads from instance-data.json;  it grabs 'cloud_name', 'region" and 'platform' from the json file, ;   see cloudinit/cmd/cloud_id.py and cloudinit/sources/__init__.py:canonical_cloud_id()19:42
meenaaah, here we go: we're back to none; because loads of stuff in net/__init__.py is failing…19:45
rharperand the name comes from cloudinit/sources/__init__.py:Datasource.cloud_name property  which reads  looks looks for 'cloud-name' in the metadata   if present, it falls back to self.dsname  if they're not present ...19:45
meenarharper: how can i make cloud-init re-read the data?19:53
blackboxswoption 1: the big hammer: cloud-init clean --reboot --logs19:54
blackboxswoption 2: rm /var/lib/cloud/instance/obj.pkl I believe19:55
meenablackboxsw: option 1 doesn't work; and i'm very slowly working on fixing that…19:55
rharperreread metadata ?19:55
rharperor regenerate instance-data.json ?19:55
blackboxswohh I was interpreting re-detect the datasource19:55
rharpermeena: I think re-reading metadata  and then dumping instance-data.json ;   I'd likely rm /var/lib/cloud/instance/obj.pkl;  and then just re-run:   cloud-init init --local    (or cloud-init init if it's a network datasource vs. local datasource)19:58
meenablackboxsw: i'm forcing the datasource via  /usr/local/etc/cloud/cloud.cfg.d/cloud.cfg19:58
meenadatasource_list: [ Hetzner, None ]19:58
=== ijohnson|lunch is now known as ijohnson
meenabecause a lot of things assume linux, i have to do that, otherwise, i won't even get to the point where hetzner is attempted19:59
meenanow, ec2 as fallback (thanks to standard datasource_list) works *somewhat* but not completely19:59
meenaand Hetzner does some funky stuff in before trying to fetch the metadata20:00
meena2020-12-14 20:00:31,325 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceHetzner.DataSourceHetzner'> failed20:00
blackboxswmeena to iterate on get_data: `sudo python3 -c 'from cloudinit.stages import _pkl_load; print(_pkl_load("/var/lib/cloud/instance/obj.pkl").get_data())'`20:01
blackboxswthat would let you keep loading the detected datasource from the cmdline and re-running the "crawl_metadata" method20:01
meenawhich is why, back in the day, i tried to add a "backdoor" into fetching metadata: does network already work? skip whatever you're doing and fetch20:01
meenablackboxsw: nah, obj.pkl is bad.20:01
blackboxswthat python cmdline assumes though the datasource that is detected on the system is what you are hoping20:02
blackboxswahh roger20:02
meenait's either using ec2 or none; the first one works somewhat (it fetches metadata, but not userdata); the latter… well, it doesn't break anything20:02
meena2020-12-14 19:41:19,602 - subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '', 'broadcast', '', 'dev', 'vtnet0'] with allowed return codes [0] (shell=False, capture=True)20:04
blackboxswmeena: so ultimately are you thinking you'll be able to get Hetzner detected properly without failure on BSD? Or is that unattainable?\20:04
meenablackboxsw: in a year or two20:04
blackboxswso January 2021?:)20:04
meenai really don't get it, cuz ds-identify works flawlessly20:04
rharpermeena: that looks like the ephemeral DHCP ... is networking already up when you're trying again ?20:05
meenait clearly identifies Hetzner, and then cloud-init itself is like, well, i dunno what to do with this stuff that ds-identify just told me, i guess i'll just ignore it20:05
meenarharper: yupp20:05
meenai'm in via SSH20:05
meenathat was produced by cloud-init init --local btw20:06
rharperI see, Hetzner is special20:06
rharperin get_data, they don't bother checking if networking is up20:06
rharperand call their Ephemeral DHCP a second time20:06
blackboxswmeena: given that hetzner's datasource is kindof derived from EC2 I could see it kina works. I wonder if ubuntu cloud-images actually specify the datasource_list: [Hetzner] (no None item fallback in the datasource list).20:07
rharpermeena: so, if you were to modify that locally, I'd drop the with cloudnet.Ephemeral() context manager and call the md, ud  code20:07
blackboxswbut as you said, you are hitting linux-assumption issues  in Hetzner which result in a Trace somewhere during network setup20:07
meenablackboxsw: lots and lots, hence the net/refactor.20:08
rharperblackboxsw: did we ever implement cloudinit.net.network_is_up()   I know I've done something like that in the hotplug branch ...20:08
blackboxswrharper: checking too. I knew someone referenced it. but it may have just been your branch20:08
meenawhat's really bothering me tho is that the ssh-keys are provided in meta_data; and when cloud-init first runs without me messing with it, it can retrieve the meta-data (i can later run cloud-init query ds and *see* it) but it doesn't store the ssh-key to the default user20:10
rharperthey keys are stored in the datasource object; and then a separate stage pulls keys from datasource to default user20:11
meenahttps://cloudinit.readthedocs.io/en/latest/topics/instancedata.html#v1-public-ssh-keys ← there's actually no documentation on what to do with those, so, never mind20:11
blackboxswrharper: there is EphemeralDHCPv4(connectivity_url=<someurl_to_test>)20:11
rharperso you'd need to re-run the user_groups mod ...20:11
blackboxswwhich could be provided and would NOOP if already available20:12
rharperblackboxsw: ah, right, that hits a URL before brnging up20:12
rharperthat should work20:12
meenablackboxsw: i added that.20:12
meenais it not used by Hetzner?20:12
blackboxswthough not consumed anywhere at the moment20:13
* meena 🤦🏻‍♀️️20:13
meenai added it, and didn't, use it?20:13
blackboxswI think it was groudwork that ended up being used by another utility function on Azure at one point, but I think it didn't get general reference in other datasources.20:13
meenaso, when i tried to provide a patch to use that, i believe smoser or rharper where against using the md_url as target, because it may not be available at that point20:15
rharperupstream it looks like it needs to pass connectivity_url=<value>20:15
rharperwhich is true at local time... I preferred a network is up20:15
rharpersomething like checking if you ahve a default route or an ip on an interface20:16
rharperthere's no point in trying a URL with a timeout if networking isn't even up20:16
meenarharper: aye. it was a hack, because so much of net/ was unusable on non-linux20:18
blackboxswwe also now have the availability of distro.networking.is_up(devname) thx to that networking work that meena/Odd_Bloke spent some time one20:20
blackboxswso we should be able to pre-flight check that from the datasource self.distro.networking.is_up(<fallbackdev>)  I think?20:21
blackboxswand avoid the EphemeralDCHPv4 context manager if we are up20:21
blackboxswwe/the dev is/20:21
meenaweird, still no user-data20:22
meenaand i'm pretty, and sure, i provided some…20:23
meenatho, it was literally just overriding system: distro: freebsd20:23
meenahow do i override or disable vendor-data?20:25
rharpermeena: one other thought;  on linux hetzner instance, if you were to refactor the _getadata such that the metadata-crawl method ran outside of _get_data, one could call it directly, see: cloudinit/sources/DataSourceGCE.py );  verify that refactor works on linux and then see where it fails on bsd20:25
meenarharper: *nod *nod20:26
rharpermeena: vendor-data: {enabled: False}  in user-data20:28
meenarharper: thanks20:29
* meena puts that into her default hetzner snippet20:29
meenawhelp, i don't need to override system_info, if i disable vendor-data…20:31
meenarharper: can we document that somewhere?20:31
meenathis mentions part of it, https://cloudinit.readthedocs.io/en/latest/topics/vendordata.html#input-formats but uses a different key than you do20:31
rharpermeena: well, that's disabling part of vendor-data, vs disabling processing of all of it20:35
rharperit should be documented there for complete disabling,20:35
rharperit mentions that you can and fails to include the bits to show how to do that20:35
AU-S2SI'm trying to add the contrib and non-free repositories to a yaml that will be pasted into a AWS EC2 Launch Template User data.20:44
AU-S2SThis is against the latest Debian Buster AMI. My YAML looks like this:20:44
AU-S2S# set up apt to include contrib and non-free20:44
AU-S2S  preserve_sources_list: false20:44
AU-S2S  disable_suites: $RELEASE, $RELEASE-updates, backports20:44
AU-S2S  sources_list: |20:44
AU-S2S    deb http://cdn-aws.deb.debian.org/debian buster main contrib non-free20:44
AU-S2S    deb-src http://cdn-aws.deb.debian.org/debian buster main contrib non-free20:44
AU-S2S    deb http://cdn-aws.deb.debian.org/debian buster-updates main contrib non-free20:44
AU-S2S    deb-src http://cdn-aws.deb.debian.org/debian buster-updates main contrib non-free20:44
AU-S2S    deb http://security.debian.org/debian-security buster/updates main contrib non-free20:44
blackboxswlucasmoura: falcojr another SRU pr https://github.com/cloud-init/ubuntu-sru/pull/183 for maas logs from automated tests20:46
AU-S2SThere are no errors in /var/log/cloud-init..log20:46
blackboxswAU-S2S: what version of cloud-init on that image? cloud-init -v ?20:46
AU-S2SHMM.. stand by.20:47
blackboxswalso in the /var/log/cloud-init.log ( Cloud-init v. XXX20:47
AU-S2S2020-12-14 19:46:17,329 - util.py[DEBUG]: Cloud-init v. 20.2 running 'init-local'20:48
blackboxswAU-S2S: if you have access to the machine that booted, `sudo cloud-init query userdata` it should show that the userdata was seen by the vm properly20:50
AU-S2SI'm telling the launch template that the user data is not encoded (which it's not, just plain text).20:50
blackboxswalso AU-S2S does the first line of your user-data start with #cloud-config?20:50
blackboxswif not cloud-init will ignore it20:50
AU-S2S# https://cloudinit.readthedocs.io/20:51
AU-S2S# https://salsa.debian.org/cloud-team/debian-cloud-images20:51
AU-S2S# Set up config for EC220:51
AU-S2S  Ec2:20:51
AU-S2S    timeout : 5020:51
AU-S2S    max_wait : 12020:51
AU-S2S    metadata_urls:20:51
AU-S2S      -
AU-S2S      - http://instance-data:877320:51
AU-S2SI've run that command before, but I don't remember the output (last week). Give me a bit to re-launch and capture it again.20:52
blackboxswhrm, that looks like the content of something in /etc/cloud/cloud.cfg or cloud.cfg.d/ not what I'd expect from the commandline via `cloud-init query userdata`20:52
blackboxswno worries20:52
AU-S2SYea, that's the top of my yaml.20:53
blackboxswok, that said, I don't recall if the user-data can actually set the datasource. so specifying datasource: keys in your user-data YAML may be worthless.20:54
blackboxswyeah it's possible that however the template data is being presented to the vm, maybe there was stray whitespace at the top or a missing header line that causes cloud-init to ignore it without a specific error.20:59
AU-S2Sadmin@test1-debian:~$ sudo cloud-init query userdata21:03
AU-S2S# set up apt to include contrib and non-free21:03
AU-S2S  preserve_sources_list: false21:03
AU-S2S  disable_suites: $RELEASE, $RELEASE-updates, backports21:03
AU-S2S  sources_list: |21:03
AU-S2S    deb http://cdn-aws.deb.debian.org/debian buster main contrib non-free21:03
AU-S2S    deb-src http://cdn-aws.deb.debian.org/debian buster main contrib non-free21:03
AU-S2S    deb http://cdn-aws.deb.debian.org/debian buster-updates main contrib non-free21:03
AU-S2S    deb-src http://cdn-aws.deb.debian.org/debian buster-updates main contrib non-free21:03
AU-S2S    deb http://security.debian.org/debian-security buster/updates main contrib non-free21:03
AU-S2Sand when I check sources.list21:04
AU-S2Sadmin@test1-debian:~$ cat /etc/apt/sources.list21:04
AU-S2Sdeb http://cdn-aws.deb.debian.org/debian buster main21:04
AU-S2Sdeb-src http://cdn-aws.deb.debian.org/debian buster main21:04
AU-S2Sdeb http://security.debian.org/debian-security buster/updates main21:04
AU-S2Sdeb-src http://security.debian.org/debian-security buster/updates main21:04
AU-S2Sdeb http://cdn-aws.deb.debian.org/debian buster-updates main21:04
AU-S2Sdeb-src http://cdn-aws.deb.debian.org/debian buster-updates main21:04
AU-S2Sdeb http://cdn-aws.deb.debian.org/debian buster-backports main21:04
AU-S2Sdeb-src http://cdn-aws.deb.debian.org/debian buster-backports main21:04
AU-S2SAha! looks like I have an error in when cc_apt_configure.py tries to convert V2 to V3. But I don't understand what the error is telling me.21:14
AU-S2S"ValueError: Old and New apt format defined with unequal values True vs False @ apt_preserve_sources_list"21:18
AU-S2SIs True/False case sensitive?21:18
blackboxswit means that you have two keys in your user-data or vendor-data that are suggesting different values for preserve_sources_list      'sudo cloud-init query userdata| grep  preserve_sources_list'    # do you see two settings, one false and one true?21:25
blackboxswcloud-init is telling you it can't decide which config value to use (and the older one is a key named "apt_preserve_sources_list" which I think got deprecated in cloud-init 0.7.921:27
AU-S2SOn the grep - no, just one: admin@test1-debian:/var/log$ sudo cloud-init query userdata| grep  preserve_sources_list21:27
AU-S2S  preserve_sources_list: false21:27
AU-S2SLet me query the vendor data..21:28
blackboxswAU-S2S: I'm guessing as well if you have two different kinds of userdata/vendordata keys  managing preserve_source_list you might also have two keys trying to set apt sources up, maybe look for something under apt_sources key too?21:31
blackboxswthe config key "apt_sources" would  be the "old" way to setup apt sources in cloud-init (for cloud-init version 0.7.5 I think)21:31
AU-S2SBingo!!  In /etc/cloud/cloud.cfg21:34
AU-S2S# This preverts apt/sources.list to be updated at boot time, which21:34
AU-S2S# may be annoying.21:34
AU-S2Sapt_preserve_sources_list: true21:34
AU-S2SCan I change my yaml to use the old variable name?21:37
AU-S2SBOOM!!!  :-D21:49
AU-S2Sadmin@test1-debian:~$ cat /etc/apt/sources.list21:49
AU-S2Sdeb http://cdn-aws.deb.debian.org/debian buster main contrib non-free21:49
AU-S2Sdeb-src http://cdn-aws.deb.debian.org/debian buster main contrib non-free21:49
AU-S2Sdeb http://cdn-aws.deb.debian.org/debian buster-updates main contrib non-free21:49
AU-S2Sdeb-src http://cdn-aws.deb.debian.org/debian buster-updates main contrib non-free21:49
AU-S2Sdeb http://security.debian.org/debian-security buster/updates main contrib non-free21:49
AU-S2Sdeb-src http://security.debian.org/debian-security buster/updates main contrib non-free21:49

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!