[02:03] blackboxsw: merged your branch uploaded to bionic [02:10] thanks smoser [02:13] and /me is out. [09:23] Hello #cloud-init [09:24] I'm looking for a way to disable a specific service to start. Basically my service is not provisioned yet and I want to delay the service start after custom script execution [09:55] maybe with bootcmd and systemctl disable === daniel is now known as Guest77671 === Guest77671 is now known as Odd_Bloke [18:33] I've a silly question, and I'm happy to take doc pointers as an answer. [18:34] I'm in charge of a DataSource. I have something new: "routes" to provide. [18:34] [18:35] prefix and dst should be straightforward. "linklocal" is what indicates if this is an "interface route". [18:35] If I turned those tuples into route(1M/8) commands, e.g.: [18:35] <10.1.2.0/24 10.1.1.1, false> ==> route add 10.1.2.0/24 10.1.1.1 [18:36] <10.51.50.0/24, 10.1.1.5, true> ==> route add -interface 10.51.50.0/24 10.1.1.5 [18:36] (And 10.1.1.5 damned well better be your local IP.) [18:37] How would a DataSource feed such information up to its consumer in cloud-init? [18:50] danmcd: which datasource are you using ? [18:51] SmartOS. I'm trying to modify it to turn the tuples I mention into something nice for cloud-init. [18:51] ah [18:51] http://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v1.html#route [18:51] so smartos has a helper to convert the sdc:nics info [18:51] not sdc:nics. [18:51] sdc:routes [18:51] Which correspond, I think, to the cloud-init concept whose documentation I linked to above. [18:53] there are two places to set routes in the v1 config; there is the top-level type: route which has gateway,network; or each subnet entry under an interface can accept a routes list [18:53] those end up in /etc/network/interfaces as post-{up/down} route commands [18:54] No way to specify interface routes then? [18:54] there is, if you hang the routes under the interface subnets configuration [18:54] Okay. [18:58] Wait.. [18:58] Let me gist something... [18:59] https://gist.github.com/danmcd/15a9bdeadbfb1bfbc018fa293a1af26b [19:00] It's still not obvious whether or not I can add an interface route. [19:01] The pointed examples where I'm shouting.... it's not clear from the documentation that's allowed. [19:01] we don't explicitly add the -interface, those stanzas result in a route add -net gw [19:03] So if I want to add -interface routes, what do I do? [19:04] we don't have a way to add the -interface [19:05] THere's something about outputting scripts... do I just hack that in there? [19:05] you can certainly have a script run that [19:05] Okay. THank you. [19:06] For your time & patience. :) [19:06] np, on the -interface vs. the -net gw ; does that generate a different static route ? [19:07] Yes. [19:07] -interface is essentially: "Hey, ARP these destinations". [19:07] Or "NDP these" if it's IPv6. [19:08] You can't have both with a matching prefix, it's one or the other. [19:09] danmcd: hrm, ok; I need to read more; [19:09] I hear you... [19:09] I suspect we can file a bug to track this is missing; [19:10] Sure. [19:12] I think this might be a BSD thing in route? net-tools on linux route doesn't have a -interface parameter [19:14] -interface originates from 4.3BSD yes... [19:17] danmcd: you might try the syntax you gist'ed and see what route you end up with; the gw value is used to determine which interface to us, the man page mentions this is a BSDism ; so I wonder if that's equivalent ? [19:17] Maybe. [19:17] Finding out locally. Thank you. [19:17] yw [19:25] Yes, BTW, "route add -net -gw " is like "route add -interfaces

" on BSD and illumos. [19:27] blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341851 [19:27] can you give that a review ? [19:39] np smoser === pickle is now known as dhill_ [19:49] cloud-init generates a strange/broken netplan config if the ipv6 gateway is a link local address [19:49] https://paste.ubuntu.com/p/M7Yd4vvSS5/ [19:51] if I remove the routes section, everything works as expected [19:53] this route section also triggers the following warning [19:53] (generate:666): GLib-WARNING **: 18:19:49.916: GError set over the top of a previous GError or uninitialized memory. [19:54] in case of v1, I get [19:55] post-up route add -net :: netmask 0 gw fe80::1%eth0 || true [19:55] which doesn't hur [19:55] t [19:56] Any idea how to disable these extra routes? [19:58] otherwise the newly added Hetzner datasource works in the official bionic images [19:58] (which seem to have a broken protective MBR) [19:58] smoser: thanks for the merge, btw [20:03] AscII: /me is embarassed though [20:03] i stole your credit with our merge tool [20:03] :-( [20:04] i'm apt to stuff a commit message in with your name proper. [20:04] the squash took the top-most author [20:05] oh well [20:05] AscII: can you give the input ? [20:06] AscII: the only good thing about someone steeling your credit is that they get blame if theere are bugs :) [20:06] i'm really sorry that happened though. [20:06] hehe [20:06] no worries [20:08] the input, what was the input config... ie, did cloud-init mangle the input to break things ? or is netplan just doing the wrong thing . [20:08] no mangling. The config is just wrong [20:09] The routes shouldn't be there [20:09] they are not pushed [20:10] net/netplan.py [20:10] fwiw, we got credit right on the debian/changelog for Markus :) [20:10] yeah, blackboxsw saves the day [20:10] hehe [20:11] after breaking it. shame on review-mps tool [20:11] so L114 in net/netplan.py generates routes for the subnet [20:11] always [20:11] no matter what [20:12] hrm if routes is unspecified on the subnet it should be empty list and remain unset right? [20:13] yes, something like that [20:14] net/eni.py does a the same thing for v1 in L361 [20:15] but that results in the no-op 'post-up route add -net :: netmask 0..' [20:17] well, actually this command is wrong as well, but the added '|| true' prevents any problems [20:18] there is a comment above that section in net/eni.py that says: We may at somepoint not want to emit this additional postfix [20:19] ah, that relates to the ||true [20:19] hmm [20:22] AscII: sry I'm still trying to get the rest of the context on this. Do you have /var/log/cloud-init.log or a tarfile produced by 'cloud-init collect-logs' [20:22] I'm specifically wondering about the log message Applying network configuration from... [20:23] sure [20:23] which should contain the netconfig we base rendering on [20:23] which may or may not have the 'routes' incorrectly specified [20:26] https://paste.ubuntu.com/p/RVrBH9YqBb/ [20:28] AscII: ok so ilne 89 in your paste shows that we are feeding in 'incorrect' route information to the netplan generator [20:28] I'm trying to grok where that comes from [20:32] AscII: it looks like it comes from HetznerCloud datasource.metadata['network-config'] [20:32] cat /run/cloud-init/instance-data.json will show you the metadata harvested on the cloud [20:33] under 'ds':'meta-data' keys [20:36] hmm, that file is not present (anymore). Possibly because I corrected the generated netplan yaml [20:39] smoser: I posted comment/question on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341851 [20:40] smoser: simpoir: I finally got ci to like me on https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/341543 turns out you actually have to 'git add' a new module that you've coded up otherwise those silly import errors. ¯\_(ツ)_/¯ [20:40] blackboxsw: ah, crap. My bad. It seems we actually have these routes in the metadata [20:41] AscII: I like to pass the blame around ;) [20:41] Need to check with my team why. I guess some other distro needed it [20:41] good deal. [20:42] blackboxsw: I just saw. I having a second look at it. [20:42] simpoir: I'm on your landscape branch atm [20:42] it will happen today ;() [20:43] 👍 [20:46] well, it looks like rhel/fedora is to blame [20:47] they add the route as gateway [20:47] IPV6_DEFAULTGW=fe80::1%eth0 [20:48] and I think, we couldn't directly pass the interface in the gateway [20:52] since the gateway is not in the subnet (L346) it doesn't get rendered there, but later during routes (L392) [20:53] great [20:58] so, the only way to work around this, would be to render the meta-data with routes to rhel/centos/fedora and without to debian/ubuntu, I guess [20:58] At least for now [22:02] rharper: are routes not picked up on ever reboot? [22:02] papertigers: not following, could you provide some more context ? [22:03] papertigers: are you manually modifying network configuration after cloud-init has generated one for you? [22:03] following up on what danmcd was asking about earlier. It seems only certain things are setup again on reboot. For example I had to run 'cloud-init clean' to get DNS resolvers to change in systemd on this box [22:04] But my newly-parsed-by-DataSourceSmartOS.py "route" doesn't seem to work. [22:04] Here's some json output I got from the logfile, including teased-out to show things clearly:https://gist.github.com/danmcd/7d1ae639d7877e760a47ad83489a342f [22:11] network configuration is rendered per-instance;, so a reboot won't modify it; if you're adding additional configuration manually via a script, that would need to be marked per-always rather than per-instance [22:11] dns resolves should be encoded in /etc/network/interfaces.d/50-cloud-init.cfg [22:11] in which case they survive reboots on the same instance [22:12] danmcd: yeah, the nameserver should be under the loopback interface in the eni file I refernced, resolvconf reads eni config and updates /etc/resolv.conf which is symlink to /run/resolvconf/resolcv.conf maintained by the resolvconf program [22:16] So nameserver can apply with "cloud-init clean" and a reboot. [22:16] I'm wondering how I can tell why my "route" directive isn't being acted upon. [22:16] Sorry if papertigers didn't make that clear. [22:16] (See the gist above.) [22:18] can you paste your /etc/network/interfaces.d/50-cloud-init.cfg ? or if cloud-init collect-logs and share that? [22:18] from the yaml in your gist, it should render this: https://paste.ubuntu.com/p/RDbq9qsFJ9/ [22:25] I dont seem to have a at /etc/network/interfaces.d/50-cloud-init.cfg [22:26] are you booting ubuntu ? [22:26] "cloud-init collect-logs" barfs [22:26] if so , what release? Xenial [22:27] Ubuntu 17.10 [22:28] ah, netplan [22:28] fun [22:28] let me rephrase the output [22:30] so, /etc/netplan/50-cloud-init.yaml is what youll have, the resolvers should show up under systemd-resolve --status; the static routes appear to be dropped; =/ [22:30] danmcd: I'd also be curious about the collect-logs barfs comment, as that script doesn't do much except tar files on the system and run journalctl [22:31] we'll need to refactor the network conversion to put the sdc:routes data under the network interfaces [22:31] ahh but it definitely needs a fix for non-deb systems as it leverages dpkg-query [22:31] it's missing routes... [22:31] So do I need to do any more work in DataSourceSmartOS.py? Or is this a consumer-of-that's problem? [22:32] blackboxsw: hang on... [22:32] blackboxsw: https://gist.github.com/danmcd/577325259016909c33fd8e18f01cb1c2 [22:33] ahh sudo cloud-init collect-logs danmcd [22:33] DOH! [22:33] the instance-data.json is protected root readonly [22:33] silent. [22:33] :) [22:33] danmcd: yes, we need to fix the converter to put the routes under the interface [22:33] Phew! [22:33] danmcd: one sec, lemme paste [22:34] http://paste.ubuntu.com/p/ScdyqwMq6z/ [22:34] I'm a newb in this arena, so thanks for your patience. [22:34] no worries, we'll file a bug and it should be an easy adjustment [22:34] with the change, then the netplan config file looks like this: http://paste.ubuntu.com/p/htRC3hcK5m/ [22:34] which will include the routes [22:35] So what you pasted... that is something I'll have to fix. [22:35] danmcd: I'll file the bug, and we just need to stuff the sdc:routes data in a different location [22:35] sorry danmcd on the silent run of collect-logs, smoser asked me to add a print saying 'Wrote cloud-init.tar.gz' in your cwd. [22:35] the subnet we generate, can accept a 'routes' array [22:35] Since now I hang the routes as their own same-level-as-physical. But according to your pastebin, I'll have to put it inside the physical "object". [22:35] right? [22:36] I forgot to add that output to the collect-logs cmd. But anyway not a problem you care about at the moment [22:36] yeah, under the subnets for the interface [22:36] Okay, so that is my problem. Since I'm fixing sdc:routes anyway. [22:36] BTW, since sdc:routes is independent of interface, I'm going to have to collect them, and then hang them under all interfaces. [22:37] Which shouldn't be a problem,. even for multi-homed? [22:37] yeah, we struggled with the routes [22:37] Okay... [22:37] in a sense a route has to go out *one* interface [22:37] so, for the netplan format, the routes are stuffed under interfaces [22:37] Oh, this is a netplan thing? [22:37] yeah, 17.10+ has netplan [22:37] (Sorry, I come from the illumos TCP/IP stack...) [22:38] Okay, I see. It'll be more challenging, but at least I understand WTF is going on now. You've been very helpful. Thanks for your time & patience. [22:41] np [23:15] if we do this work for netplan, does that break anything below 17.10? [23:16] papertigers: no, v1 renders the global routes correctly already [23:16] 16.04 should work as-is with the config that danmcd posted in his gist [23:17] I'm trying brute-force... putting it in both places. [23:17] Isn't the idea it works in one form and deploys everywhere, though? [23:17] that may cause issues on 16.04 [23:18] yes, cloud-init converts the v1 into internal network state, then renders to the distro's format (eni, netplan or sysconfig) [23:18] rharper: and my question earlier was basically after the yml file or interfaces file gets written out to disk on first boot it doesn't try to set things up again unless you run 'cloud-init clean'? [23:18] So putting "routes" as a peer of "physical" should not be done? [23:18] what we're hitting is that netplan doesn't handle "global" static routes per the design choice that routes should be bound to at least one interface, since they have to egress one anyhow [23:19] papertigers: so on 17.10, after your first boot, run systemd-resolve --status [23:19] That seems to be a problem beyond just DataSourceSmartOS, right? [23:19] you should see your DNS enatries there [23:20] rharper: yeah I know that works. I am saying if the data in sdc:resolvers changes, cloud-init wont attempt to change that unless you run clean. Just want to confirm behavior [23:20] papertigers: cloud-init writes out either an eni on 16.04, or netplan on 17.10 before the "networking" layer starts, then ifupdown or networkd brings the config up [23:20] so you shouldn't need to do anything to have all of your network config applied [23:20] papertigers: yes, we don't yet have dynamic network config changes [23:21] that's something we're actively discussing [23:21] cool, thanks. That would be super useful to us if the discussions is a matter of if and not when [23:22] https://hackmd.io/M1Tae41PQBC7a9qMsurTJw [23:22] that was one of the discussion we've had with the community [23:22] we're very interested in feedback , especially from datasource folks [23:24] papertigers: danmcd: I've got to step our for bit; feel free to leave comments, issues, I'll see them later tonight, and if not then tomorrow [23:25] RIght now, I generate multiple "routes" entries. [23:25] I'll find out the hard way if that's a real problem or not. [23:25] Thanks rharper