[01:56] blackboxsw: sorry. kind of confused by Odd_Bloke comments there. [01:57] oh. i see. you fixed. [02:08] Hi everyone! I'll be very glad if someone could give me an advise, about a weird issue that I having with a cloud-init and a packer template. When I boot the image the first time, using terraform provider openstack and libvirt for local, sometime the cloud-init doesnt start. I check the systemd and said tha the cloud-init is disable but this services was enabled in the image creation. [02:09] This happends randomly. Sometime the same temple provision the VM without problems. [02:09] *template [02:09] bugbuilder: its probably related to ds-identify. [02:09] but you should file a bug and attach cloud-init collect-logs output [02:10] ideally inboth "good" and "bad" cases. [02:10] yep I checked them and in the case when the cloud.init service is disable I dont have any log. [02:11] I'm thinking to put the datasource for openstack to see if the identifier could fix it [02:11] run cloud-init collect-logs [02:11] anad attach the output [02:11] k [02:11] it gets mroe than /var/log/cloud-init.log [02:13] I looking on them.. Let me read them first and I coming back in a bit thank yoy smoser [02:13] sure.then jsut file a bug. attach and you can ping me with a link here. [02:18] I red dmesh and journal without luck. [02:18] *read [02:18] but I found a problem about the DS as you said [02:19] di_report: [02:19] datasource_list: [ ] [02:19] # reporting not found result. notfound=disabled. [02:20] Do you think if I create thee datsource for openstack this problem could be disapper? [02:21] iits possible that if you put 'datasource_list: [OpenStack, None]' into /etc/cloud/some.cfg that it will fix the issue. [02:21] also which would be for local develoment [02:21] but you can potentially fix the issue for other people "the right way" (tm) [02:21] er... help fix the issue [02:21] by filing a bug [02:21] sure, how I could do that. [02:22] by filing a bug with that info ;) [02:22] lol [02:22] yeah sure but I dindt know where [02:22] is it ubuntu ? [02:22] Suse [02:22] https://bugs.launchpad.net/cloud-init/+filebug [02:24] let me try the fix, I'll come with news and then I'll fill the report [02:24] * smoser has to go afk [02:24] night night [03:29] thanks smoser. yeah I had force pushed the changes. I need to stop doing that so the PR more easily represents the fixes. [03:39] i think force-push to a pr is fine === tds5 is now known as tds [07:28] Hi I am trying to boot a baremetal server in OpenStack with two network, the provisioning is succesful but the node is not reachable. I saw that the instance folder in /var/lib/cloud was not created. [07:29] When i try to boot the same server with only network, provisioning is succesful and the node is reachable too. cloud-init does the network configuration properly. [07:29] Can someone please help me identify the problem with cloud-init or how to debug this issue? [07:33] It seems that the DataSourceConfigDrive is not known to cloud init [08:02] smoser, Hi [08:02] blackboxsw, portdirect Odd_Bloke rharper Hi, Can you please help? [08:56] mkrai: most people on this chan are in US timezones. If you can prepare the following it would help them be efficient when they wake up: [08:56] mkrai: run "cloud-init collect-logs" and upload the result of that command somewhere (this collects the logs as the name implies) [08:57] tribaal, Ok thanks I will do it now [09:04] tribaal, I get error when i run this command "IOError: [Errno 2] No such file or directory: '/var/log/cloud-init-output.log'" [09:04] I have /var/log/cloud-init.log [09:14] that is interesting - perhaps cloud init is not actually run at boot for some reason [09:38] tribaal, http://paste.openstack.org/show/787707/ [09:42] looks like it works fine as far as I can tell [09:43] I'm afraid you'll have to wait for somebody more knowledgeable than myself to come online [09:43] tribaal, np thanks for the help though :) [09:44] blackboxsw, portdirect Odd_Bloke rharper smoser Here are the logs http://paste.openstack.org/show/787707/ [14:25] mkrai__: i think you're hitting [14:25] https://bugs.launchpad.net/cloud-init/+bug/1801364 [14:25] Ubuntu bug 1801364 in cloud-init "persisting OpenStack metadata fails" [Undecided,Fix committed] [14:26] so i would suggest trying https://copr.fedorainfracloud.org/coprs/g/cloud-init/el-testing/ [14:30] Goneri: your branch is again out of synch? [14:31] you gotta rebase against tip [14:32] I'm rebasing it [14:32] thanks meena for the notice [14:43] For the record. I've tested the branch against FreeBSD-10, 11, and 12 yesterday on OpenStack. And it works just fine. I'm in the process to upload all the qcow2, if anyone wants to give them a try. [14:44] it would be great if the branch could be merged. It has been here for a while, and it resolved several long standing problems. [14:59] the PR is here, https://github.com/canonical/cloud-init/pull/61 [15:32] i have tested this on hetzner, and i'm quite happy with what it does, that is: exactly what it promises to do: it configures the network and renames the device. IPv4 only so far, but it's a start [15:39] Hello everyone, I have a few questions about a thing I am trying to do but due to my lack of knowledge I am struggling quite a bit. If you have 5-10 minutes it would help a lot maybe. Thank you [15:42] wolfiepresley: well, try to write your question [15:42] and if someone can answer,t hey will [15:43] Sure :) [15:46] Been using Amazon Linux inside an AWS Workspaces. [15:46] create a bundle and assign that bundle to a new workspace. [15:47] But the whole idea is that the user should not be prompted to enter the password when he executes the command. So I had the idea to edit /etc/sudoers but when the image/workspace is being created it cleans it. [15:48] Wanted to add something like -- testuser ALL=NOPASSWD: /sbin/openvpn [15:49] well, i dont knwo what is doing cleaning. but that is not somethign that cloud-init control. [15:49] cloud-init's general goal is to not require cleaning [15:49] you can absolutely use user-data to make cloud-init re-add the rules you want. [15:50] I was thinking if I can edit the cloud.cfg or cloud.cfg.d add a new file here where at the boot it adds that line to sudoers.d [15:50] or add systemd jobs that did it, or some other way. [15:51] /var/lib/cloud/per-instance/your-script.sh [15:51] chmod 755 that and it should get run once per instance. [15:51] but if your cleaner deletes it, then there is nothing i can do. [15:51] basically... you can't fight a cleaner that applies its own arbitrary rules. [15:52] it could just as well decide to delete things in /etc/cloud/ [15:53] Here is some info from AWS Workspaces custom bundle documentation [15:53] https://docs.aws.amazon.com/workspaces/latest/adminguide/create-custom-bundle.html [15:54] Seems contents from /var/lib/cloud are removed :( [15:54] :q [15:54] well, file a bug. they should not delete /var/lib/cloud/ [15:55] Thing is this happens only when you create the image, if for e.g you have a workspace already built on top of that image/bundle it will not remove its contents. [15:55] all the files in that list have valid reasons to keep them. [15:56] sure. [15:56] /var/log/amazon/ssm [15:57] funny my initials show up there. [15:57] :D [15:57] anyway... you can absolutely work around their custom bundle stuff. and do whatever you want. [15:57] but they can later decide to update their custom bundler and screw you [15:58] but i do consider it a bug to delete [15:58] /var/lib/cloud [15:58] because [15:58] a.) cloud-init should manage that correctly [15:58] (and does) [15:59] b.) there are interfaces with behaviors based on files in that path... so it shoudlnt get cleaned as that breaks cloud-init's promised interfaces. [16:10] I am not experienced with this at all.. actually I started working on this last week and i've been reading through a lot of stuff but it's a whole mess in my head as you may have realised already. Appreciate you taking the time to write this. [16:11] To be honest i thought if I edit the /etc/cloud/cloud.cfg [16:11] openvpn without the password. [16:15] Looks like Travis is broken in master ATM because of tag issues. I'm looking into it. [16:18] OK, I think I've addressed it. [16:19] I believe the 19.4 tag was pointing at a commit in the proposed branch, which changed when it was squashed. [16:19] And the tag needs to be signed, AFAICT, for read-version to be happy with it, so that took a couple of iterations too. [16:20] Odd_Bloke, good to know - is that something we should doc? [16:40] why would cloud-init.service have only an After directive on systemd-networkd-wait-online.service? shouldn't that be a requirement? [16:41] well, systemd-networkd-wait-online.service is obnoxious [16:42] what cloud-init wants to wait for is "all configured networking to be up". [16:42] smoser: but we have no hard requirement for an online network then right? [16:42] but what systemd-networkd-wait-online.service provides is [16:42] "you have network connectivity". [16:42] otherwise what is the mechanism that says "dont do cloud-init things until we are connected" [16:42] ? [16:42] the difference is that if you intentionally have no network connectivity, or you have no network devices [16:43] then you will not be online, but that is sufficient for cloud-init. it does not have a requirement that there be network. [16:43] smoser: ok so this is basically by design then, not to have a hard requirement on network connectivity? [16:43] ok [16:43] that answers my question [16:43] is it a weird thing and un-cloud-init-like to make that a requirement? [16:44] i don't see _why_ that would be a bad thing to modify, but curious on second opinions [16:44] the use case that I think is completely valid is [16:44] - take ubuntu cloud image [16:44] - add a data disk [16:44] - add a NoCloud datasource [16:45] - userdata says to do a bunch of stuff (compile maybe) and put output on datadisk [16:45] no network needed, nor desired. [16:45] yeah makes sense [16:45] if you change that to be "you must have network connectivity", then it doesnt work. [16:45] because at the moment, for instance, cloud-init does *not* require systemd-networkd.service to be up and running [16:46] i really thikn that what cloud-init wants is what systemd-networkd-wait-online.service should provide [16:46] and i think i have a use-case where we absolutely need networking to be up and running before running cloud-init [16:46] what does "online" even mean? (AOL sound-bite has played ? ) [16:46] smoser: able to make network requests..? i think [16:47] on one network interface ? on all of them ? [16:47] ipv4 ipv6 ? [16:47] its not really straight forward [16:47] right i see where you're going with that [16:47] basically i'm working on an issue right now where systemd-networkd.service isn't up and running, and therefore we're not able to contact the cloud fabric. that's an issue (i don't know when that wouldn't be an issue for azure at least..?) [16:47] i am remembering things kind of fuzzy.... [16:48] but what cloud-init *wants* is "All configured network interfaces are up." [16:48] smoser: where do you see that *wants* directive? [16:49] oh. thats more of a goal. [16:49] not a reall 'Wants' [16:49] but cloud-init will continue even if network interfaces aren't up, correct? [16:50] you mean if they failed ? [16:50] no not even a failure. just never started [16:51] for instance, cloud-init will continue on it's way even if systemd-networkd.service isn't started [16:51] cloud-init.service should not run until networking is up. [16:51] smoser: i don't see a directive in cloud-init.service that enforces that? [16:51] well,. in ubuntu, cloud-init gets that to happen via netplan apply [16:51] at the local time frame i think [16:51] this is kind of fuzzy, and probasbly isnt perfect [16:52] and i think that is why we have After=systemd-networkd-wait-online.service [16:52] smoser: what runs netplan apply? [16:52] smoser: yeah but After isn't a requirement, only timing [16:54] cloudinit/net/netplan.py: will call netplan generate [16:54] hi I have a question about cloud-init [16:54] sorry... i have to go. i think i've probably not been terribly helpful [16:55] smoser: you've definitely been very helpful! thanks for the assistance [16:55] I am using a VPS from a cloud provider on debian 10, and I found that the network configuration by default is /etc/network/interfaces.d/50-cloud-init.cfg [16:56] cloud init confuses me, I want to set up my networking to support multiple ips [16:56] is it correct to think cloud init is not relevant to me, and I can remove it's configuration file and create my own [17:18] Odd_Bloke: i approved, but that still does nothing in github's eyes: https://github.com/canonical/cloud-init/pull/120 also, your branch needs a rebase [17:24] meena: that UI shows a nice grey ✅ [17:24] thanks for the review there. I've updated the PR and waiting on CI to land [17:25] If I try to edit the cloud.cfg file and add under section -- users -- the following [17:26] - name: myuser [17:26] system: false [17:28] if some of those field are empty do they still need to be present? [17:28] powersj: Yes, we should either doc the tagging requirement and/or adjust tooling. [17:28] blackboxsw: it does, now at your review has landed [17:29] chillysurfer: AIUI, the only effect that A declaring a Requires on B has is that if A is included in boot, then B will be included as well. [17:30] If systemd-networkd-wait-online.service is relevant to cloud-init, then something else should already "Requires" it, so it's included in boot. [17:30] If nothing else in boot Requires systemd-networkd-wait-online.service then that _probably_ means that networkd isn't in use on the system, so cloud-init "Requires"ing it would be incorrect. [17:31] The Wants=systemd-networkd-wait-online.service is effectively saying "if systemd-networkd-wait-online.service is included in boot, then cloud-init should run after it", which I believe is correct. [17:32] Odd_Bloke: right exactly, but there is no Wants directive, it's `After=systemd-networkd-wait-online.service` [17:32] i would think adding in Wants=systemd-networkd-wait-online.service would be a possible fix to ensuring network is up and running for cloud-init [17:33] chillysurfer: Are you seeing that systemd-networkd-wait-online.service isn't running during boot? [17:33] Odd_Bloke: correct [17:33] Odd_Bloke: and i think making it a requirement might alleviate this issue [17:34] like adding it as a Wants [17:34] OK, right. [17:34] I'm surprised that it isn't already included. [17:34] chillysurfer: What release are you looking at? [17:35] 19.2-36-g059d049c-0ubuntu2~18.04.1 [17:35] but smoser did bring up a good point. there are cases where a network requirement is a bad thing [17:35] Orion]: depending on the cloud platform, cloud-init does support configuring multiple ip addresses and/or multiple nics. It all depends on the metadata datasource used. Orion] if you are adding more nics on a platform that doesn't support configuring multiple IPs you could add additional nic configuration in /etc/network/interfaces.d/50-your-additional.cfg or you can either describe your own network config in [17:35] entirty to cloud-init (and cloud-init will write it out) or you can disable cloud-init network configuration in general by adding a network: {config:disabled} in a file somewhere in /etc/cloud/cloud.cfg.d/*.cfg per https://cloudinit.readthedocs.io/en/latest/topics/network-config.html [17:37] https://github.com/canonical/cloud-init/pull/120 landed thx meena [17:37] Odd_Bloke: i guess i could just add that Wants directive when i think it's necessary [17:37] maybe not a bad idea [17:39] chillysurfer: Can you pastebin `systemctl status systemd-networkd-wait-online`, please? I'm seeing it run in a lxd container locally, so I'm confused as to why it isn't running. [17:40] minor branch to validate CLA signing and fail CI with a message about reading hacking doc if the PR author hasn't already signed the CLA. [17:40] https://github.com/canonical/cloud-init/pull/124 [17:41] Odd_Bloke: i'll save the pastebin as it's easy to describe. it runs sometimes but takes a reboot for it to run [17:41] Odd_Bloke: here's what i'm seeing.... 1) first boot, cloud-init tries to provision with no networking as systemd-networkd is not running 2) something reboots the machine 3) networking comes up, but provisioning is already completed with errors [17:42] Odd_Bloke: so during the 1 phase above, networking just isn't up, including systemd-networkd-wait-online. and it's causing issues [17:42] systemd-networkd not running sounds like something is very broken, TBH. I'm not sure that's a cloud-init issue. [17:42] Odd_Bloke: my thoughts to remedy this include making cloud-init have a hard requirement [17:42] Odd_Bloke: no doubt, i completely agree with you. this was simply a workaround to tell cloud-init that it shouldn't run if systemd-networkd.service isn't running [17:42] it's a workaround for a much larger issue, i agree [17:44] chillysurfer: I guess what I'm confused about is how you've ended up with a system with this behaviour. Because I assume this isn't an issue in the default behaviour of bionic on Azure? [17:44] Odd_Bloke: correct. custom image [17:51] chillysurfer: at generator time, we can't know if systemd-networkd is present; and it won't be running; cloud-init local runs *before* networking on purpose; I don't understand the exact scenario; why is there no networking (missing nics?, not connected?) and how would cloud-init know that there isn't any networking available? [17:53] rharper: why is there no networking? great question. that's the root of the problem, i wanted to add a unit directive as a workaround to the problem in the interim [17:53] I was wondering if any of you seed cloud-init configs with packer. the goal is to customize existing AMIs and automate their build process. I can't seem to find much in terms of examples of such workflow [17:54] rharper: for some reason, networking isn't up for the 2-3 minutes on first boot so provisioning has issues. and then _something_ reboots the vm (don't know what). and then when it comes back up, networking works but provisioning is already complete (with issues) [17:58] chillysurfer: having a journal would be helpful [17:59] chillysurfer: do you want cloud-init to run [17:59] and without knowing why networking isn't available, it's not easy to drop-in some logic to check for such a state and have cloud-init skip running this boot [17:59] rharper: totally understand. i've combed through it pretty thoroughly. nothing that really jumps out as to why networking doesn't come up for some time [18:00] chillysurfer: I'm happy to look at the journal if you can share [18:00] rharper: yep totally understand [18:00] rharper: i'll have to check with a few people before i could share it, thanks for the offer! [18:00] sure, understood [18:00] rharper: thanks for your willingness to help! super appreciated [18:00] https://github.com/coreos/docs/issues/519 [18:01] chillysurfer: I suggest adding this to the image; it dumps way more debugging for networkd [18:01] rharper: oh wow that's super helpful actually [18:01] thank you [18:21] rharper: https://github.com/canonical/cloud-init/pull/123 for ubuntu/devel so I can upload to focal today if possible [18:21] the 19.4 release [18:22] also Travis validation of CLA is passing CI in https://github.com/canonical/cloud-init/pull/124/files [18:36] thanks for the review on ubuntu/devel smoser. True on adding RbxCloud to templates [18:47] done [19:17] blackboxsw: https://github.com/canonical/cloud-init/pull/124 reviewed [19:19] 28 contributors from 29 domains? [19:33] powersj: rharper: blackboxsw: https://github.com/canonical/cloud-init/pull/125 is the first workflow automation piece; it will auto-close PRs after 21 days of inactivity (it first marks a PR as stale after 14 days). [19:34] nice! [19:34] addressed 124 [20:03] powersj: blackboxsw: meena: I've responded to the comment that you either made or reacted on: https://github.com/canonical/cloud-init/pull/125#issuecomment-567190235 [20:11] Odd_Bloke: okay, fair. [20:11] Odd_Bloke: responded with an inline question/comment. I'm +1 regardless of your stance [20:12] blackboxsw: Yes, I think we do, let me update. [20:13] rharper: would you like to take a look at this again? https://github.com/canonical/cloud-init/pull/53 [20:17] blackboxsw: Updated. [20:17] blackboxsw / Odd_Bloke: would you like to take a look at this… [20:17] https://github.com/canonical/cloud-init/pull/69 [20:18] approved Odd_Bloke [20:18] looking meena [20:36] * blackboxsw will run through openstack once on that changeset meena, I'm +1 otherwise, but I don't think it'll actually trigger due to our current code structure in DatasourceOpenStack [20:36] it could possibly in Hetzner. [20:37] blackboxsw: it does; that's how i test Goneri's changeset :P [20:37] hrm, then I'll have to get through that live test to to re-educate myself about how that'd work across reboots (without cloud-init clean) [20:39] community notice: Upload to Ubuntu Focal (20.04) of cloud-init 19.4 complete [ubuntu/focal-proposed] cloud-init 19.4-1-g8c96cbc1-0ubuntu1 (Accepted) [21:09] powersj: This PR brought to you by having to read the cc_ssh docs repeatedly over the past few days: https://github.com/canonical/cloud-init/pull/126 :p [21:09] hahahaha! [21:10] Odd_Bloke, was this from a grep on 'ssh'? [21:21] powersj: `ait -s '[^-_/.a-zA-Z]ssh[^-/_a-zA-Z]' cloudinit doc/rtd` [21:21] (Where `ait` is `ag --ignore tests`.) [21:38] i keep forgetting that hetzner is overriding my system_info/distro [21:44] meena: Have you attempted to escalate that with them? [21:49] I've rebuilt my FreeBSD images, they go from 10 to 12: e.g: https://virt-lightning.org/images/freebsd-11/disk.qcow2 [21:54] https://www.youtube.com/watch?v=KOO5S4vxi0o [21:56] Odd_Bloke, you made me reconsider a lot of things in my life. [22:00] Odd_Bloke: nah, i have considered overriding vendor-data, however ;) [22:01] for one: FreeBSD doesn't need a rng seed; cuz it's got an implementation that doesn't suck [22:01] (sorry Linux folk) [22:01] However, i don't know how to do that! [22:01] * meena blames powersj [22:02] lol [22:04] https://bugs.launchpad.net/cloud-init/+bug/1837530 not sure if my reading comprehension is enough to get anything out of this bug report [22:04] Ubuntu bug 1837530 in cloud-init "cloud-config in vendordata overriden by user-data" [Undecided,Expired] [22:04] the merging doc needs to be updated, but I'll be honestly I don't understand it all enough to do the updates [22:05] generally user data always wins versus vendor data is about all I can somewhat intelligently say [22:05] so, given my vendor-data: https://bugs.launchpad.net/cloud-init/+bug/1855170 maybe it's enough that I say: system_info: distro: freebsd ; and be done with it [22:05] Ubuntu bug 1855170 in cloud-init "system_info can change the distro" [Undecided,Incomplete] [22:10] meena: That's certainly worth a try. [22:10] If it doesn't work, then disabling all vendor data should do it. [22:11] Odd_Bloke: that's not documented EITHER! [22:27] oh my gosh, it's like a dream [22:30] runcmd: section seems to have been ignored tho :( [22:33] meena: `alias man=grep -R` <-- fixed the docs problem [22:35] Odd_Bloke: lookit, my personal wisdom is, read the source, but STILL. [22:35] so, almost midnight, time to give up. [22:35] Goneri: found a "bug" [22:36] Goneri: https://gist.github.com/8fb0a17238b4264cfa8a37093017c7d1 can you find it too? ;) [22:37] so, bed time. [22:41] I've commented on https://github.com/canonical/cloud-init/pull/61 with my experience report ;) [22:41] next up, finding out why runcmd doesn't work… [22:41] or, sleep??? [22:44] anyway, i'm now paying hetzner double the money for the privilege of developing cloud-init [22:44] (almost 6€) [22:59] whoa, y'all have useful reviews comments! (it's much easier to give those useful reviews when you know the codebase better i guess;) [23:38] meena, I suspect the ifconfig_vtnet0 line to come from the ephemeral dhcp call [23:38] meena, could you send me the cloud metadata?