| crandon | Hi, I'm trying to boot up a centos 7.8 clou image on a libvirt host passing a cloud-init image. Network config and user config works fine, however ntp config doesn't seem to be set for chrony. I simply added an ntp fragment to the meta-data file with enabled: true, ntp_client: chrony and a servers list. | 12:53 |
|---|---|---|
| crandon | Here's the resulting instance-data.json: https://pastebin.com/10C2rJDf | 12:57 |
| Odd_Bloke | crandon: Can you expand a little on "added an ntp fragment to the meta-data file"? Which file, specifically#? | 13:00 |
| crandon | To the /meta-data file | 13:00 |
| crandon | I wonder if the ntp module needs to be explicitly enabled... | 13:01 |
| Odd_Bloke | Aha, right, in the image/ISO you're creating: meta-data is information from "the cloud" about its view of the instance (e.g. hostname, cloud region), where as user-data is information about what the user wants the instance to do; NTP configuration belongs in user-data. | 13:02 |
| Odd_Bloke | If that's a problem, you could also consider using vendor data. | 13:03 |
| crandon | Ah sh... I thought all network related info should go there (like network-interfaces). | 13:04 |
| Odd_Bloke | Right, "the cloud" (i.e. you, in this case ;) defines network interfaces, so they are meta-data. | 13:06 |
| crandon | Yes, but the same way the cloud defines/provides the ntp service as well (ideally). Anyway, I moved it to user-data and it's still not working. | 13:10 |
| crandon | Odd_Bloke: Here's the user-data as it is seen now from the guest: https://pastebin.com/bRPmyRM5 | 13:18 |
| crandon | I tried to follow the ntp module docs from: https://cloudinit.readthedocs.io/en/latest/topics/modules.html | 13:20 |
| Odd_Bloke | crandon: You need the first line of user-data to be "#cloud-config" for cloud-init to process it. | 13:25 |
| Odd_Bloke | Clouds define and provide all sorts of services, NTP doesn't need special treatment. (If a cloud did want to pass in default NTP configuration, then it could use vendor-data for that; that's cloud-provided default user-data, precisely for this sort of usecase.) | 13:27 |
| crandon | Makes sense. | 13:29 |
| crandon | With regard to the #cloud-init. That's my bad I only copied the relevant parts and below. It's actually there. | 13:29 |
| Odd_Bloke | smoser: rharper: Can either of you give me some background on the specific-to-RHEL hostname setting behaviour? (The context being https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1883649; feel free to comment there! :) | 13:48 |
| ubot5 | Ubuntu bug 1883649 in cloud-init (Ubuntu) "cloud-init on Ubuntu 18.04 does not set FQDN" [Undecided,New] | 13:48 |
| Odd_Bloke | crandon: Just to double-check: it's "#cloud-config" there, right? | 13:48 |
| crandon | I just checkd (on the VM itself) and the first line is: #cloud-config | 13:49 |
| Odd_Bloke | OK, cool. | 13:49 |
| Odd_Bloke | (You wouldn't believe how many times that's the issue, I always like to be absolutely sure. ;) | 13:50 |
| crandon | I don't see any ntp related log entries in /vat/log/cloud-init, but the user data gets copied into /var/lib/cloud/instances/<host>/user-data.txt | 13:50 |
| Odd_Bloke | crandon: Are there any WARN or "Traceback" lines in your log? | 13:51 |
| crandon | Odd_Bloke: Honestly I got the template from some ansible playbook, so I woudn't have notice it for sure... | 13:51 |
| crandon | Nope all DEBUG except for 3 lines of INFO | 13:52 |
| Odd_Bloke | OK, in that case I think you'll need to pastebin the whole thing for me to take a more detailed look at. :) | 13:53 |
| rharper | Odd_Bloke: lemme refresh; | 14:19 |
| Odd_Bloke | Thanks! | 14:28 |
| rharper | Odd_Bloke: I think your suggestion in the bug is the right answer; as to the history of this; it's related to OS documentation; RHEL has insisted on using FQDN, where has Debian has it separate; many software stacks query their hostname (and fqdn) differently; smoser usually reminds me of some strangeness in java's hostname lookup vs other stacks; | 14:32 |
| crandon | Odd_Bloke: This is the complete stuff with sensitive info masked: https://pastebin.com/3iHadMse | 14:43 |
| crandon | Would it help if I'd move it to vendor data? | 14:52 |
| Odd_Bloke | crandon: Apologies, I meant the full cloud-init.log. | 15:00 |
| crandon | Uh, that's going to be tricky as the customer classifies hostnames and even private IPs as confidential... | 15:02 |
| crandon | And I can copy out anything only via clipboard... | 15:02 |
| crandon | Let me see, what I can do and come back to you (I can't promise it's gona happen today). | 15:03 |
| Odd_Bloke | crandon: Fair enough, that's understandable. :) (You could try reproducing this elsewhere; not necessarily an easy task, but would free you from that restriction.) | 15:07 |
| smoser | Odd_Bloke: java stacks (what is that java web framework?) often do stupid things | 15:58 |
| smoser | but other things do the same. | 15:59 |
| smoser | they think for some reason that they can reverse lookup their hostname | 15:59 |
| smoser | to an IP address (a single IP address) | 15:59 |
| smoser | and then tell other things "Hey! you can talk to me at x.y.a.b" | 15:59 |
| smoser | well, that is flawed for so many reasons. | 15:59 |
| smoser | and often ends in the hilarity of "you can talk to me at 127.0.0.1" | 16:00 |
| AnhVoMSFT | @rharper continuing from the mounting issue yesterday, it seems like the nfs mount is declared implicitly (because type is nfs) to be After network-online.target, it means that whatever cloud-init.service does, it won't be able to successfully call mount -a if there is an nfs mount in /etc/fstab, is that correct? | 16:07 |
| AnhVoMSFT | because cloud-init.service is Before network-online.target, this means network-online.target won't be reached until we finish with cloud-init.service, and as such the nfs mount cannot be successfully brought up, or does mount -a not rely on the nfsvolume.mount unit and actually try to mount it individually? | 16:08 |
| blackboxsw | #startmeeting cloud-init status meeting | 16:22 |
| meetingology | Meeting started Tue Jun 30 16:22:42 2020 UTC. The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology. | 16:22 |
| meetingology | Available commands: action commands idea info link nick | 16:22 |
| blackboxsw | community notice: time for another bi-weekly (or semi-monthly if you prefer) cloud-init community status meeting | 16:23 |
| blackboxsw | #chair smoser rharper Odd_Bloke | 16:24 |
| meetingology | Current chairs: Odd_Bloke blackboxsw rharper smoser | 16:24 |
| blackboxsw | welcome to another round of cloud-init upstream updates and discussion. We use this meeting as a time to gather to discuss current development of cloud-init, ask and answer questions, and generally expedite development be unblocking devs. All questions. side-conversations and interruptions are welcome | 16:25 |
| blackboxsw | last meeting minutes are at the link below | 16:25 |
| blackboxsw | #link https://cloud-init.github.io/status-2020-06-16.html#status-2020-06-16 | 16:26 |
| blackboxsw | turns out I didn't update the topic for the next meeting time last session. Let's do that now | 16:26 |
| blackboxsw | +2 weeks from now, same time | 16:26 |
| === blackboxsw changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting July 14 16:15 UTC | 20.1 (Feb 18) | 20.2 (Apr 28) | https://bugs.launchpad.net/cloud-init/+filebug | ||
| blackboxsw | July 14th, same UTC time | 16:26 |
| blackboxsw | now that that's out of the way, we typically cover the following topics. | 16:27 |
| blackboxsw | Previous Actions, Recent Changes, In-progress Development, Community Charter, Office Hours (~30 mins). | 16:27 |
| blackboxsw | additionally today, I'll discuss the current cloud-init SRU | 16:27 |
| blackboxsw | #topic Previous Actions | 16:28 |
| blackboxsw | topic #1. our previous meeting minutes logged two actions: | 16:28 |
| blackboxsw | ACTION: file feature bug about refactoring startup services | 16:28 |
| blackboxsw | I think in further discussion during last meeting, we talked with Odd_Bloke and meena and determined that we can't actually refactor startup services to live in the distro specifically, because these startup service templates actually get determined at cloud-init generator time (before distribution is determined in cloud-init's python code) so trying to specialize startup script content generation in the distro | 16:30 |
| blackboxsw | python classes in cloud-init is too late | 16:30 |
| blackboxsw | so this action is tabled as /wont-fix | 16:30 |
| blackboxsw | that follows as well with the other ACTION: mailing list email requesting comment/concerns about a refactor of startup services | 16:31 |
| * blackboxsw isn't sure how to close out actions in meetingology syntax/cmds | 16:31 | |
| blackboxsw | #topic In-progress Development | 16:31 |
| blackboxsw | The following is the set of commits landed in 'master' of cloud-init upstream repo: found with git log --since 06-20-2020 | 16:33 |
| blackboxsw | #link ACTION: mailing list email requesting comment/concerns about a refactor of startup services | 16:33 |
| blackboxsw | #link https://paste.ubuntu.com/p/fSvwRks86z/ | 16:33 |
| blackboxsw | heh paste error | 16:34 |
| blackboxsw | #topic Recent Changes | 16:34 |
| * blackboxsw sets appropriate topic for this section | 16:34 | |
| blackboxsw | so recently Odd_Bloke and a number of BSD folks (meena igalic etc) have gone through a number of discussions and design regarding a refactor of cloudinit.net functions to a cloudinit.distro.networking module as most network-related functionality is highly distro-dependent | 16:37 |
| blackboxsw | Odd_Bloke: created an overview of this current refactor work and published it to readthedocs | 16:37 |
| blackboxsw | #link https://cloudinit.readthedocs.io/en/latest/topics/hacking.html#ongoing-refactors | 16:37 |
| blackboxsw | This has been a big effort to get organized and started so many thanks for all those paricipating in this discussion, development and reviews. | 16:38 |
| blackboxsw | there are many, functions that need to be refactored from cloudinit.net into the distribution-specialized cloudinit.distro.networking classes. | 16:39 |
| blackboxsw | It is work that can be easily done in parallel and there is a tag used to classify each refactor as a "net-refactor" bug in launchpad | 16:40 |
| blackboxsw | #link https://bugs.launchpad.net/cloud-init/+bugs?field.tag=net-refactor | 16:40 |
| blackboxsw | community notice: we encourage anyone interested in refactoring cloud-init networking functionality to grab and work any of those net-refactor bugs | 16:41 |
| blackboxsw | there are a couple of example PRs up that give a good idea of how to get started | 16:41 |
| blackboxsw | #link https://github.com/canonical/cloud-init/pull/457 | 16:42 |
| blackboxsw | and I can't seem to find the other at the moment. | 16:42 |
| blackboxsw | besides net-refactor content landing, there have been fixes to Hetzner and RbxCloud datasources, redhat's systemd generator templates, Centos copr build fixes to help RPM build runs and Azure datasource logging. Thanks smoser, paride Moustafa and otubo Adam Dobrawy for contributions this round | 16:48 |
| blackboxsw | #topic In-progress Development | 16:49 |
| blackboxsw | Generally the last two weeks have been sunk into upstream testing and validation of cloud-init for SRU (Stable release Update) into Ubuntu Xenial Bionic, Eoan and Focal. | 16:51 |
| blackboxsw | 3 to 5 of us have been on verification tasks on various clouds for all Ubuntu releases targeted and all features which affect ubuntu. | 16:52 |
| blackboxsw | A thousand thanks rharper Odd_Bloke factor lucasmoura and xiaofeng for working through and validating some of these SRU tasks. | 16:52 |
| blackboxsw | Our job is done, and we are awaiting feedback from an automation CI from Canonical solutions QA at the moment which runs through a ton of Openstack networking customer-configurations. It has been in the test queue for a week, and I just saw a successful run from that test harness this morning. That team has told us it looks for 3 successful runs to "pass" so I expect that pass to come shortly as the test runs | 16:54 |
| blackboxsw | are currently inprogress. | 16:54 |
| blackboxsw | as soon as this test passes we will mark the SRU bug verified and the SRU team will publish bits of cloud-init. | 16:55 |
| blackboxsw | #link https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1881018 | 16:55 |
| ubot5 | Ubuntu bug 1881018 in cloud-init (Ubuntu) "sru cloud-init (19.4.33 to 20.2-45) Xenial, Bionic, Eoan and Focal" [Undecided,In progress] | 16:55 |
| blackboxsw | This SRU has taken about 1+ week longer than normal verification because we hadn't SRU'd cloud-init in around 6 months, so there was a lot more content to verify. | 16:56 |
| blackboxsw | Hopefully additional SRUs will be more frequent and less heavy-weight. We are looking into reducing the overhead on this process and will pitch ideas to the cloud-init mailinglist for input | 16:57 |
| blackboxsw | Beyond SRU work, the following other work is in progress: | 16:57 |
| blackboxsw | * net-refactor formerly mentioned | 16:58 |
| blackboxsw | * falcojr into Oracle integration test harness | 16:58 |
| blackboxsw | * extending json schema validation for remaining cloud-config modules for better error reporting around invalid user-data | 16:58 |
| blackboxsw | Long term work: cloud-init standalone daemon to improve startup time by avoiding reloading python across each cloud-int boot stage, initial networking hot-plug support to which datasources could "opt-in" | 17:00 |
| rharper | blackboxsw: =) | 17:01 |
| blackboxsw | I think that about wraps this topic. | 17:01 |
| blackboxsw | yeah rharper, we've got it on our roadmap. We'd love to see that get in this round. | 17:01 |
| blackboxsw | #topic community charter | 17:02 |
| blackboxsw | We have a couple of general themes of features we are working toward as a community this year: | 17:03 |
| blackboxsw | * json schema additions for cloudinit.config.cc_* modules to improve user-facing errors on invalid user-data | 17:04 |
| blackboxsw | * datasource documentation improvements, updates and corrections | 17:04 |
| blackboxsw | * cloudinit.net-refactor work | 17:04 |
| blackboxsw | We encourage any interested developers to grab any of these work items related to these features. | 17:05 |
| blackboxsw | We have two bug tags which enumerate each component of these work streams: | 17:06 |
| blackboxsw | #link https://bugs.launchpad.net/cloud-init/?field.tag=bitesize | 17:07 |
| blackboxsw | #link https://bugs.launchpad.net/cloud-init/+bugs?field.tag=net-refactor | 17:07 |
| blackboxsw | #topic Office Hours (~20 mins) | 17:08 |
| blackboxsw | This 'section' of the meeting is a time where a couple of upstream devs will be available in channel for any discussions, questions, bug work or PR reviews. | 17:08 |
| blackboxsw | I think I spent most of the time typing, but will hit the review queue in the absence of any other discussion | 17:08 |
| blackboxsw | merged https://github.com/canonical/cloud-init/pull/461 | 17:17 |
| blackboxsw | lucasmoura: one minor change request and description update on the PR requested https://github.com/canonical/cloud-init/pull/390#pullrequestreview-440241947 | 17:51 |
| blackboxsw | then we can land this one | 17:51 |
| blackboxsw | ok folks, thanks for checking into the cloud-init status meeting. See you in 2 weeks. | 17:51 |
| blackboxsw | #endmeeting | 17:51 |
| meetingology | Meeting ended Tue Jun 30 17:51:54 2020 UTC. | 17:51 |
| meetingology | Minutes: http://ubottu.com/meetingology/logs/cloud-init/2020/cloud-init.2020-06-30-16.22.moin.txt | 17:51 |
| lucasmoura | blackboxsw, ack | 17:52 |
| Odd_Bloke | blackboxsw: I've added the tests I think your comment called for, please re-review and LMK: https://github.com/canonical/cloud-init/pull/457#discussion_r447879423 | 18:03 |
| blackboxsw | Odd_Bloke: approved , will let you land when CI passes | 18:05 |
| blackboxsw | Odd_Bloke: merged xenial https://github.com/canonical/cloud-init/pull/462 | 18:06 |
| blackboxsw | keeping that daily build building | 18:07 |
| Odd_Bloke | Thanks! | 18:22 |
| AnhVoMSFT | @rharper, in RHEL 7.8 rpc-statd and rpc-statd-notify.service which are required for nfs mounting are marked "After" network-online. mount -a with nfs mount presence can't proceed with it. In Ubuntu those services are marked After network.target, and not network-online. I changed rpc-statd and rpc-statd-notify to be After network.target and that seemed to work. I think we'll advise the | 19:16 |
| AnhVoMSFT | customer to work with Redhat support on this. I think rpc-statd and rpc-statd-notify can be safely started after networkmanager-wait-online and don't have to wait till network-online.target? | 19:16 |
| rharper | AnhVoMSFT: nice; yes; I suspect they could use the same After= network.target that ubuntu does; but your suggestion will also work as well, just slightly later than network.target; | 19:20 |
| Odd_Bloke | Trivial oneline review needed: https://github.com/canonical/cloud-init/pull/465 | 19:34 |
| blackboxsw | done | 20:04 |
| Odd_Bloke | Thanks! | 20:14 |
| smoser | AnhVoMSFT: fwiw, the issue with 'network-online.target' is network-online.target itself. | 20:28 |
| smoser | "network-online.target is a target that actively waits until the nework is "up", where the definition of "up" is defined by the network management software. Usually it indicates a configured, routable IP address of some kind" | 20:28 |
| smoser | cloud-init (and probably rpc-statd-notify and loads of other services) do not require any network services. | 20:29 |
| smoser | its perfectly fine to have no networking and do work | 20:29 |
| smoser | if you said 'after' of network-online.target, and had no network configured, would you expect 'after' to have been accomplished? | 20:29 |
| smoser | and as the doc says "for example network server software should generally not pull this in (since server software generally is happy to accept local connections even before any routable network interface is up), its primary purpose is network client software that cannot operate without network." | 20:31 |
| AnhVoMSFT | @smoser Agreed. I don't think it's a problem with cloud-init, but rather with the nfs-utils package in Redhat and how they declare dependencies. The ones in Fedora and Ubuntu don't have this issue | 21:21 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!