punkgeekHow can I detect network configuration version in cloud init automatically? because I want to write a general script for Linux distros.13:45
Odd_Blokepunkgeek: What do you mean by "network configuration version"?  Could you perhaps describe your problem/requirement in a little more detail?13:51
smoserblackboxsw: i'm not really sure why there was a difference.  I think powersj might have some insight.  Or such knowledge may well be lost in the recesses of past brains.13:56
smoserbut i think that at one point a jenkins had access to one but not the other or something.13:57
punkgeekOdd_Bloke: There are 2 kinds of network type which you can see in here:  https://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v1.html    and  https://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v2.html14:01
punkgeekSome distoros such as ubuntu 16 use the first version but in ubuntu 18 use the second one (netplan)14:02
punkgeekI want to make a script that can detect the OS network type and config it automatically. For example if the Distro support netplan, Use the second network type14:06
Odd_Blokepunkgeek: Have you seen the `cloud-init devel net-convert` command?14:09
punkgeekOdd_Bloke: This python file? https://github.com/delphix/cloud-init/blob/master/cloudinit/cmd/devel/net_convert.py14:12
punkgeekI couldn't understand this python script14:12
Odd_Blokepunkgeek: I guess I'm still not sure I understand where this "script" will run, so I'm guessing at what you're trying to do a little.14:13
smoserpunkgeek: "Some distoros such as ubuntu 16 use the first version but in ubuntu 18 use the second one (netplan)"14:13
smoserthe point of cloud-init is that it can render either network config version into the correct system configuration format.14:13
smoser`cloud-init devel net-convert` will read either version 1 or version 2 network config14:14
smoserand write sysconfig (rhel, suse), ENI (old ubuntu), netplan (new ubuntu)14:14
smosercloud-init *is* the general script14:15
punkgeekAha, So how can I use cloud-init devel net-convert in the iso file?14:16
punkgeekI should use it by runcmd command?14:23
punkgeekHere is a shell script that I used to config static ip in different distros: https://github.com/autovmnet/tools/blob/master/vm_config.sh   But I want to do it by cloud-init iso14:29
Odd_Blokepunkgeek: When you say "cloud-init ISO", what do you mean?14:30
punkgeekSorry i'm amature in cloud-init. Here is my cloud-init script example that i've convert it to iso file by using cloud-localds and then mount it on the vm   https://github.com/autovmnet/tools/blob/master/cloud-init.cfg14:32
Odd_Blokepunkgeek: OK, great; so if you run `cloud-localds`, you should see that it has a `--network-config` option.  If you use that to specify network configuration (in either v1 or v2 formats), then cloud-init will take care of rendering it appropriately for your target distro version.14:35
Odd_Bloke(And no worries, there's a lot of similarly named things, so I'm just making sure we're on the same page so I'm not giving you bad advice!)14:36
punkgeekThank you, As I understand, in the cloud-localds option, I should create two config file, cloud-init.cfg and network.cfg then convert it to iso file like this:   cloud-localds --network-config=network.cfg new.iso cloud-init.cfg14:47
punkgeekAm I right?14:47
Odd_Blokepunkgeek: Yep, I believe that will do it.15:00
punkgeekHow can I do it in a single file?15:02
punkgeekOdd_Bloke: And is there any way to enter both version configuration and then, system detect version automatically and use the currect one?15:04
smoserpunkgeek: it doesn't matter which you pick15:18
smosercloud-init will render either version to the correct result when configuring networking.15:18
smoseryou declare it however you want.15:18
punkgeeksmoser: Thank you, so is it correct to do sth like this?:  create a config.cfg which contain: https://paste.ubuntu.com/p/rXhfRXfWPz/   and then run cloud-localds --network-config=cloud.cfg new.iso15:23
punkgeekhow should it be? Could you please give me an example to do it?15:25
punkgeekShould it seprate the network part and user part? or require anything else?15:26
punkgeekmoser: what about this one? moser>15:33
smoserthats not valid yaml15:35
smoserwell.. it miht be valid yaml, but its not sensical.15:35
smoseryou have 2 top level network kesy.15:35
smoseronly put one of the two versions in .15:35
smoseri recommend following the demo and changing one thing at a time.15:36
smoserall the files for the demo are at https://gist.github.com/smoser/635897f845f7cb56c0a7ac3018a4f47615:36
punkgeekThe point  what  I want to enter both network-config is that because my platform cannot detect the vm distors.15:38
Odd_Blokepunkgeek: cloud-init can detect the VM distro (because it runs within the VM), and it will convert your input to the appropriate format for that distro.15:40
Odd_BlokeSo if it's running on xenial, it will convert v2->ENI, on bionic and later v2->netplan, etc.15:41
punkgeekOdd_Bloke: So is it ok to enter both version like this? https://paste.ubuntu.com/p/NTRXsk3Rx6/15:41
Odd_Blokepunkgeek: No.15:42
Odd_BlokeYou specify your configuration once, and cloud-init converts that configuration to the appropriate format.15:42
punkgeekOdd_Bloke: sorry I couldn't understand, could you give me an example?15:43
Odd_Bloke(In this case, the second `network` key definition overrides the first, so you are actually only defining a v2 config.)15:43
Odd_Blokepunkgeek: https://paste.ubuntu.com/p/VSrFPTzFdz/ <-- that's all you need to specify, cloud-init will take care of converting it to the required format within the VM15:44
punkgeekAha so if i enter this one and the distro does not support netplan such as ubuntu16.04, It will convert to the version one?15:45
dkingI would like some help troubleshooting my network configuration. I see that my network-scripts were set as expected, but they don't seem to be applied. It seems like it fell back to DHCP for some reason. I'm looking through cloud-init.log now, but I'm not sure what I'm looking for yet.15:46
dkingI see this in the logs: "No network config applied. Neither a new instance nor datasource network update on 'System boot' event"15:49
Odd_Blokepunkgeek: Correct!  (In fact it will convert it to ENI, which is the /etc/network/interfaces format used on xenial; v1 is only used internal to cloud-init.  But your broader understanding is correct.)15:50
dkingI got the same when I ran "cloud-init clean -r". I'm new to debugging cloud-init issues, so any help would be appreciated.15:56
smoserdking: well, you can open a bug. but i'd first explain what you're tryhing to do and paste an entire cloud-init.log .15:59
punkgeekOdd_Bloke: Sorry but I've got this error:   https://paste.ubuntu.com/p/B8B6MTvrKP/15:59
smosercloud-init collect-logs will collect a lot of information that people would probably ask you16:00
smoserpunkgeek: drop the top level 'network'16:00
smoseri think newer versions of cloud-init will figure that out, but the bug you're seeing16:01
smoserthe bug you're seeing is that cloud-init looks for a top level 'version' key in its network config, and you have no such key. (because you have a second level network key)16:02
smoserthose files are valid... if you start there you might have more success.16:02
punkgeeksmoser: like that? https://paste.ubuntu.com/p/8VTMTqJhXR/16:02
punkgeekAha sorry16:03
smoserhttps://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1798117 intended to support your input16:04
ubot5Ubuntu bug 1798117 in juju "juju sends "network" top level key to user.network-config in lxd containers" [High,Fix released]16:04
smoserso i'd supsect that the guest that is there does not have that fix.16:04
dkingsmoser: I'm not convinced that it's a bug yet. I'm trying to just get an idea of why things didn't happen as I expected. Here's the cloud-init.log: http://sprunge.us/oqGOws  And here is some context: http://paste.openstack.org/show/794705/16:12
dking...and I forgot to put it in there, but in case it helps, I'm using cloud-init version 18.516:14
meenawhat is delphix and how do i run it myself?16:16
smoserdking: good job collecting info16:18
smoseryou ahve 2 cloud-init runs in the log there16:19
smosermaybe try a 'cloud-init --logs clean'16:19
smoserjust to reduce it to one boot16:19
smoserbut i suspect that what is wrong is that the /etc/sycconfig is 'diry" in some way16:20
smosernot certain. but it surely looks good.16:20
dkingOkay. I'll give that a try. That's probably from where I ran "cloud-init clean -r"16:20
smoserwhere did you get the image ?16:20
dkingOkay, that may be. I had an issue when it first booted caused by the BIOS.16:20
dkingIt's generated by bifrost. This is part of a Kayobe deploy.16:20
smoserif we think that cloud-init rendered the correct sysyconfig network information16:22
smoserthen there are a few things that might be wrong16:22
dkingOkay, I ran that (actually "cloud-init clean --logs" as the other way gave an error), and now the logs are empty.16:22
punkgeeksmoser: Sorry I have this error now; https://paste.ubuntu.com/p/KNhDJHPRQb/16:23
smosera.) cloud-init rendered this network information *after* the system brought up networking (we try to ensure that 'cloud-init init-local' runs before system network bring up . but something might be broken there.16:23
smoserb.) udev rules might not be written correctly (they handle the device renaming... not exactly sure about that for sysconfig though)16:23
smoserc.) other confusing sysconfig data conflicting iwth cloud-init16:24
smoserpunkgeek: you gave it invalid yaml16:25
smoserit can't load it.16:25
smoseryou can test yaml loading using yaml-dump https://gist.github.com/smoser/280dbed3e9582c53c6035525f35814aa16:25
smosersorry, though. i really ahve to get to actual work now.16:26
dkingYes, they appear to be correct to me, though I could have missed something. Is there any way to tell from the logs where things go wrong? The logs are cleaned now. What is the best way to cleanly run it again? Just reboot the machine, or something like "cloud-init init" ?16:26
powersjdking, sudo cloud-init --local; sudo cloud-init init; or a reboot as you suggested. https://cloudinit.readthedocs.io/en/latest/topics/faq.html#how-can-i-re-run-datasource-detection-and-cloud-init16:31
dkingpowersj: Thank you very much16:35
dkingsmoser: Here's the new log: http://sprunge.us/q5KrHr and here is further context, including the error I got when attempting to bring up the interface: http://paste.openstack.org/show/794706/16:40
smoserdking: init doesnt render networking16:43
smoserinit --local16:43
dkingOKay. I'll try that.16:44
dkingsmoser: Here's the new cloud-init.log: http://sprunge.us/Scz96q16:45
smoserdo you maybe need some vlan spuport package in the image ?16:45
smoseri think that maas testing actually uses vlan and centos . so i think this is a supported path16:46
punkgeekI use this script but still does not works:( https://paste.ubuntu.com/p/xh25XGpYR7/16:49
dkingI'm not sure. I've been using the same system to deploy servers for a while, but I did make some changes to configurations before this run, and it's happened on two different machines.16:52
blackboxswfalcojr: and Odd_Bloke https://github.com/canonical/cloud-init/pull/431  daily recipe fix for bionic16:56
dkingIf the image isn't built correctly, or if there's some problem outside of cloud-init, I'll just need to know enough to locate the bug in the other software. And I have been using VLANs before. I think, though, that we had to use a certain version of cloud-init as the older ones couldn't support them. I think it was around 18.2?16:57
blackboxswfalcojr: Odd_Bloke and https://github.com/canonical/cloud-init/pull/433 for xenial17:03
dkingIs there any way to confirm from the cloud-init.log whether the issue I'm seeing in bringing up eht0.81 manually is actually the reason why the network isn't setup by cloud-init?17:05
smoserpunkgeek: you'd have to post a cluod-init log at this point.17:16
smoserdking: there should be some system logs for sysconfig somewhere17:17
smoserbut i'm not familiar enough to know17:17
dkingsmoser: I'll continue trying to resolve that. But what I meant was, is there some place in the cloud-init logs where it would show whether it even attempted to bring up that interface, or if it received an error?17:19
blackboxswdking: I see in your log that "2020-06-12 16:44:41,585 - DataSourceConfigDrive.py[DEBUG]: network config provided via network_json"17:20
blackboxswdking: and you can see cloud-init tries to render this network config in your logs17:20
blackboxsw2020-06-12 16:44:41,598 - stages.py[INFO]: Applying network configuration from ds bringup=False: {'version': 1, 'config': [{'type': 'vlan', 'subnets': [{'dns_nameservers': ['', ''], 'netmask': '', 'routes': [{'gateway': '', 'netmask': '', 'network': ''}], 'type': 'static', 'address': '', 'ipv4': True}], 'name': 'eth0.81', 'vlan_id': '81', 'mac_address':17:21
blackboxsw'ac:1f:6b:d0:4d:94', 'vlan_link': 'eth0'}, {'mtu': '1500', 'type': 'physical', 'subnets': [], 'mac_address': 'ac:1f:6b:d0:4d:94', 'name': 'eth0'}, {'address': '', 'type': 'nameserver'}, {'address': '', 'type': 'nameserver'}]}17:21
blackboxsw2020-06-12 16:44:41,598 - __init__.py[DEBUG]: Selected renderer 'sysconfig' from priority list: None17:21
blackboxswsaying that it is using the proper sysconfig renderer for the network config it writes. (which aligns with you saying the valid config appears to be in /etc/sysconfig)17:21
blackboxswmight want to check timestamps of 2020-06-12 16:44:41,598 versus you system logs that say when networking is brought up17:22
blackboxswto see if that happening in cloud-init 'after' your network was already up (so the updated config wasn't read by your system)17:22
blackboxswyou might be able to look through output of `journalctl --utc -o short-precise -u systemd-networkd`   on your system maybe ?17:25
* blackboxsw was just grasping at straws though.17:27
punkgeeksmose: Here is the log  https://paste.ubuntu.com/p/mZM3XbVKcx/17:31
Odd_Blokeblackboxsw: Shouldn't those PRs be for the daily branch, as discussed?17:46
blackboxswOdd_Bloke: ohh whoops, force of habit. I was thinking we'd do new-upstream-snapshot to refresh patches and then run fix-daily-branch script to do that. I forgot we can just fix daily only17:48
blackboxswright right will fix them17:48
Odd_BlokeNice, thanks!17:48
blackboxswhrm. thinking it through though.17:48
blackboxswso if we only fix ubuntu/daily/xenial we'll lose that next time we run cloud-init's "cherry-pick" as 'fix-daily-branch' does a git reset --hard origin/ubuntu/$release17:49
blackboxswso somehow we might need to keep a log of quilt refreshes applied in ubuntu/daily/$release to 'fix' non-cpick quilt patches.17:51
Odd_BlokeI think fix-daily-branch should abort if it finds state other than what it expects.17:51
blackboxswso we can re-apply them during the next "fix-daily-branch"17:51
blackboxswyeah I think so, I can fold that into a PR I'm working on fix-daily-branch anyway as I ran into an issue with ubuntu/$releases which have no debian/patches/cpick*17:52
blackboxswso an abort and we can sort on next cherry-pick17:52
Odd_BlokeI would suggest it'll be easier to review if you separate them out instead of folding it in.17:52
Odd_BlokeBut other than that, sounds good!17:53
blackboxswok will do.17:53
Odd_BlokeAlso worth noting that the current reset --hard behaviour will just break daily builds again in a way that we already know how to solve, so it's not necessarily the worst situation.17:53
dkingblackboxsw: Thank you very much for the info. I am starting to wonder now if the problem might be NetworkManager. In the past, I think I've had trouble with it using VLANs.17:58
smoserdking: cloud-init doesn't attempt to bring up networking information17:58
smoserit just renders it and lets the OS bring it up17:58
smoserthe goal is that your OS just thinks that was "statically configured" networking that was already there.17:58
blackboxswOdd_Bloke: falcojr https://github.com/canonical/uss-tableflip/pull/5417:59
blackboxswhandle the no debian/patches/*cpick* case on existing ubuntu/xenial or ubuntu/bionic branches (since we had just performed a new-upstream-snapshot which dropped those cpicks)17:59
dkingYep. So, I suppose that if it got the scripts into the right place, then maybe the issue is with something else in the OS.17:59
smoserpunkgeek: you'd need to give /var/log/cloud-init.log . not ust the console log.  but i suspect that the issue is that the nic name is not what you expected. ens19218:00
punkgeeksmoser: can I use it like ens* ?18:01
smoserpunkgeek: probably not reliably. you may be able to for systems that use netplan natively.18:25
smoserbut really what you need to do is specify the mac addresses18:25
smosermac addresses are something you can know from "outside".  they're properties of virtual hardware.  the names that a given OS chooses to give to that virtual hardware are not reliable18:29
blackboxswOdd_Bloke: let's try that again https://github.com/canonical/cloud-init/pull/435 to fix ubuntu/daily/xenial builds18:30
blackboxswand actually looks like I'll have to rework another patch :/ n/m Odd_Bloke    moving which into the subp module also broke another quilt patch18:31
blackboxswas unit tests now fail18:31
blackboxswafter patches are applied18:31
* blackboxsw moves it to wip18:31
Odd_Blokeblackboxsw: Ack, ping when you want eyes on it again.18:34
dkingsmoser and blackboxsw: Thanks for the help. It helped me to narrow things down. The problem does seem to be from using NetworkManager. If I switch to network-scripts, I can start the VLAN. Now, I just have to find a way to get DIB to build using network-scripts instead of NetworkManager.18:37
=== mpontillo_ is now known as mpontillo
Odd_BlokeLooks like Rick didn't get to my doc PR before the trails called to him, so if anyone wants to do a quick review: https://github.com/canonical/cloud-init/pull/43019:34
smoserdking, blackboxsw it probably would be good for cloud-init to document *somewhere* that network manager is not supported at this time.19:37
smoserand would be better if it noticed somehow that your system was using network manager and said unsupported19:38
blackboxsw+1 Odd_Bloke19:38
smoseror alternatively, actually DTRT and rendered network manager.19:38
blackboxswsmoser: I guess https://cloudinit.readthedocs.io/en/latest/topics/network-config.html#network-configuration-outputs is a bit misleading19:39
blackboxswas we specifically call out NetworkManager for netplan backend19:40
portdirecthow can I configure cloud-init to use Netplan when ifupdown is also present on the host?19:41
blackboxswportdirect: by default in Ubuntu focal and later netplan is preferred over /etc/network/interfaces even if both ifupdown and netplan are present19:41
strobelightis there a doc that lists all the recognized keys and describes them?19:42
blackboxswportdirect: on earlier releases of cloud-init you can set the renderer priority order to prefer netplan first19:43
portdirectblackboxsw: im not experiencing this, its prioritizing ifupdown over Netplan for me19:44
portdirecta simple apt-get purge ifupdown gets Netplan working, but this is not viable for my use case :(19:45
blackboxswportdirect: by either:  1. providing the following config in /etc/cloud-config.d/90-override-renderer.cfg in your image: https://paste.ubuntu.com/p/ky7V4RvdXT/19:45
portdirectblackboxsw: trying that now19:47
blackboxswportdirect: or 2 providing cloud-config to override this on VM boot (if you aren't creating your own images)  https://paste.ubuntu.com/p/f68TbMJRdb/19:47
blackboxswstrobelight: do you mean instance data keys?19:48
blackboxswor network renderer keys?19:48
blackboxswI'm missing the context of your question19:48
blackboxswfalcojr: sorry about https://github.com/canonical/cloud-init/pull/431   I should have closed it when Odd_Bloke reminded me we wanted to do this against ubuntu/daily/xenial|bionic only (as that is the whole reason we have those branches now; to avoid patch commit/refresh noise in ubuntu/xenial|bionic proper)(19:50
blackboxswfalcojr: https://github.com/canonical/cloud-init/pull/435 should have more context, but the short of it is that we expect to be dropped into a quilt patch apply failure cli where we need to run quilt push -f, resolve patch apply conflicts manually, quilt refresh to save those updated patches into debian/patches/*.patch19:52
=== Odd_Bloke changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting June 16 16:15 UTC | 20.1 (Feb 18) | 20.2 (Apr 28) | https://bugs.launchpad.net/cloud-init/+filebug
blackboxswfalcojr: also added some more context for you on this ubuntu-advantage-tools pain https://github.com/canonical/cloud-init/pull/43120:02
blackboxswhttps://github.com/canonical/cloud-init/pull/431#issuecomment-643460234 actually20:03
Odd_Blokerharper: smoser: Your input on https://bugs.launchpad.net/cloud-init/+bug/1883122 would be really appreciated. :)20:03
ubot5Ubuntu bug 1883122 in cloud-init "`cloud-init status` should distinguish between "permanently disabled" and "disabled for this boot"" [Undecided,New]20:03
blackboxswthe reason we are in a bit of a pickle on ubuntu-advantage in upstream cloud-init is because the ua-tools project accepted backwards incompatibility of the ua-client CLI  and cloud-init also was allowed to design a new set of config directives without worrying about backwards compatibility of the old shell-based ua-tools CLI versus to new ua-tools python-based CLI. As a result we've carried a bit of a patch20:05
blackboxswmaintenance pain in ubuntu/xenial and ubuntu/bionic branches20:05
* blackboxsw can't wait to release ubuntu-advantage-tools version 25.0 to xenial and bionic so we can drop the ubuntu-advantage revert stuff20:06
portdirectblackboxsw: that worked perfectly, thank you - though I put the over-ride in /etc/cloud/cloud.cfg.d/90-override-renderer.cfg20:06
blackboxswcool portdirect good to hear20:06
dkingNetworkManager is probably fine for most people. I think it's the VLANs. And it may not even be just that. I'm wondering if there could be another solution, like adding a package.20:12
falcojrblackboxsw thanks for the extra context20:18
rharperOdd_Bloke: looking20:18
blackboxswhttps://github.com/canonical/cloud-init/pull/435 and https://github.com/canonical/cloud-init/pull/436 up for daily build recipe fixes20:33
blackboxswwith happy instuctions :)20:33
strobelight@blackboxsw user-data and instance meta data. https://cloudinit.readthedocs.io/en/latest/topics/examples.html shows examples, but is that all-inclusive? https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html instance data seems to be vendor-specific, or deployment specific, for example "availability_zone" means nothing when deploying ubuntu on my virtualbox.21:40
blackboxswstrobelight: yes definitely each platform supports differing values for instance data. so your mileage varies depending on where you are deploying. https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html is the best bet and running `cloud-init query --all` on your platform is the real content you can rely on.22:12
blackboxswstrobelight: also I put together a couple of sample runs on some platforrms a while back: https://hackmd.io/aAJFerDRRNaoL0Q_vDRYBg  it's a bit dated, but there ya go22:13
strobelight@blackboxsw thanks, I'll have a look22:36

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!