| otubo | I'm gonna leave this question here and leave for lunch because I know it might take some time for someone to pick it up and answer: On this bug https://bugzilla.redhat.com/show_bug.cgi?id=1593010 the systemd-generator and the ds-identify read correctly that the second boot doesn't have any DataSource and, from the logs, they disable cloud-init. But still, on the second boot, the network configuration gets | 09:55 |
|---|---|---|
| ubot5 | bugzilla.redhat.com bug 1593010 in cloud-init "cloud-init network configuration does not persist reboot [RHEL 7.8]" [High,New] | 09:55 |
| otubo | reset to dhcp. | 09:55 |
| otubo | If anyone has any hint on this, I'll be glad to thank personally on cloud-init summit :-) And now I'm off to lunch, be back soon. | 09:56 |
| otubo | perhaps rharper ^ | 10:34 |
| dunz0r | Is there some way I could get cloud-init to "ask" me for the hostname? Doing a template for a vmware-cluster. | 12:03 |
| nvz | dunz0r: :) | 12:33 |
| nvz | guess you beat me here | 12:33 |
| nvz | dunz0r: thanks for mentioning this, I think this could be quite useful at some point I think I may hang out here and get familiarized with the project | 12:34 |
| dunz0r | Yeah. This seems really useful to what I'm trying to do, especially since I don't have to reinvent the wheel, like I was planning on first. | 12:36 |
| nvz | would've been my natural reaction | 12:37 |
| nvz | this is a really nice looking project | 12:37 |
| nvz | I just can't believe it doesnt appear they've thought of making it have interactivity | 12:37 |
| nvz | Windows has for a long time had that thing where you do an OEM install and on the first run it asks you all those questions | 12:38 |
| nvz | so you can have the system installed but still allow the user to set their timezone, username, update preferences, etc.. | 12:38 |
| dunz0r | I can see why there's no interactivity, it isn't supposed to. This is for largescale deployments. Serious stuff | 12:39 |
| nvz | I'm fairly capable in python and this does interest me even though I dont have the kind of need for it right now like you seem to | 12:39 |
| nvz | I had always wandered about being able to configure linux systems like that to be able to have initial config done by the user and it looks like I could easily build that on top of this | 12:39 |
| nvz | like I just reinstalled debian on my cousin's laptop cause the m.2 ssd died and I had to install to his sata hdd he was using for /home.. and I'd have much rather sent it back to him so it booted up asking him for his user/password/fingerprints, etc.. | 12:41 |
| nvz | rather than having to set what I could and leave a README on his desktop explaining the rest | 12:41 |
| nvz | and if I ever start doing more serious stuff with VMs this could really be useful there too | 12:41 |
| dunz0r | nvz: My dream here at work is having ALL configuration in salt-stack and essentially never touch the machines, but that's waaay ahead in the future :) | 12:46 |
| nvz | Ah, I'd never heard of salt-stack I thought you were referring to the salt minion thing in cloud-init | 12:48 |
| nvz | I've heard of ansible, puppet, chef, but these things are news to me | 12:48 |
| nvz | what amazes me about this cloud-init is like python itself, its well documented and I immediately felt comfortable with it just glancing over the site | 12:49 |
| nvz | it all just makes sense | 12:49 |
| nvz | these sorts of things can often be so cumbersome they're not really useful | 12:49 |
| dunz0r | nvz: Salt-minion is part of saltstack :) | 12:50 |
| dunz0r | Saltstack is the whole shebang | 12:50 |
| nvz | hmm.. I should probably familiarize with these things they're bound to come of use at some point.. saltstack, ansible, puppet, chef, vagrant, cloud-init | 12:51 |
| nvz | terraform | 12:51 |
| nvz | hmm.. see now this damn saltstack has one of them fancy websites that tells you a whole lot of nothing | 13:00 |
| Odd_Bloke | dunz0r: Are you asking for a tool that will help you generate cloud-config? Or some sort of interactivity in the boot process? | 13:01 |
| nvz | I glanced about 90 seconds at the cloud-init site and I could probably use it like I been doing it for months.. I glance at the saltstack page and I'm still not even sure wth it is | 13:01 |
| nvz | Odd_Bloke: that was the general idea, they want to clone vms and be able to supply things on the first boot | 13:01 |
| dunz0r | Odd_Bloke: Interactivity in the boot-process. The machines aren't on AWS or something like that, so no way to input a config-file | 13:02 |
| dunz0r | nvz: Saltstacks frontpage is awful, yeah. https://docs.saltstack.com/en/latest/ is what you want :) | 13:03 |
| Odd_Bloke | dunz0r: Where are these machines? | 13:03 |
| dunz0r | Odd_Bloke: In a VMWare cluster. | 13:03 |
| nvz | dunz0r: they make it sound like something commercial.. is it something you have to buy? | 13:03 |
| Odd_Bloke | Interactivity during boot isn't something that would be generally useful (because most launches that involve cloud-init generally aren't accessible to the user at all until SSH is up). | 13:04 |
| dunz0r | nvz: No, but they have an "enterprise" version. But I've never felt limited by the foss-version. | 13:04 |
| dunz0r | Odd_Bloke: Well, I've got console access. | 13:04 |
| nvz | Odd_Bloke: that is a good point | 13:05 |
| Odd_Bloke | Sure, some places you will have it, but that's not generally true. | 13:05 |
| dunz0r | Odd_Bloke: Today we create VMs from a template, set hostname/ip/regenerate ssh-host-keys/etc via the console, and that's a hassle | 13:05 |
| dunz0r | Odd_Bloke: Well, not generally no. But in my particular use case it is | 13:05 |
| nvz | Odd_Bloke: however I don't know of anything that is like this that could offer that OEM installed OS kinda feel and this seems to be ready to build that kinda thing on top of | 13:05 |
| dunz0r | I think I'll write something that gets input from the user during boot and then uses cloud-init to do the actual setup. | 13:06 |
| dunz0r | "user" generally being me or my colleagues | 13:07 |
| nvz | Odd_Bloke: my idea, and mind you I just heard about this project, and glanced breifly at the site.. but is it possible to just ship a cloud-config, then have the bit that allows you to run commands at boot hook something to basically interactively make vendor data? | 13:07 |
| Odd_Bloke | You could also look at generating a config drive from the input, and attaching that to the instances. | 13:07 |
| nvz | on the first boot that is | 13:07 |
| Odd_Bloke | I think the tricky part would be getting input from the console; cloud-init just runs as a daemon and outputs to the console via syslog. | 13:09 |
| dunz0r | Odd_Bloke: My plan is to write a bash script that interactively asks for hostname and IP, and then generate a cloud.cfg from that info and a base template | 13:09 |
| dunz0r | and THEN use cloud-init via the script to "setup" the machine | 13:09 |
| nvz | they have things that you can use to have an auto-login getty and such | 13:10 |
| nvz | you could have the base config set that up then get input and supply that input as vendor data overriding any config as supplied by the user | 13:10 |
| Odd_Bloke | So vendor-data doesn't override user-data, intentionally; the user should always be in control of their instances, regardless of what their vendor wants the configuration to be. | 13:11 |
| Odd_Bloke | dunz0r: You may want to consider having your script generate a config drive: https://askubuntu.com/a/867958 | 13:12 |
| Odd_Bloke | Then you can attach that to the VM as you launch it, and cloud-init should find the configuration from it without intervention during boot. | 13:12 |
| nvz | I didn't even read about user data.. when I glanced it over I saw the main cloud-config then I seen the bit about vendor data that it says overrides cloud-config | 13:12 |
| Odd_Bloke | So cloud-config is really a format; it's YAML with a "#cloud-config" header so cloud-init knows that it's intended to be treated as cloud-config. | 13:14 |
| Odd_Bloke | Both user-data and vendor-data can specify cloud-config. | 13:14 |
| nvz | yeah I noticed that much.. and I'm a fan of yaml.. when I first heard of it I avoided even looking at it because I thought it was another damn markup language.. years later when I finally looked it over I felt stupid :P | 13:15 |
| nvz | turns out its not Yet Another Markup Language, its YAML Ain't Markup Language | 13:16 |
| dunz0r | Wait what:O | 13:16 |
| Odd_Bloke | dunz0r: Actually, looking at https://git.launchpad.net/cloud-init/tree/doc/sources/ovf might give you a better way of generating the ISO containing user-data. | 13:17 |
| Odd_Bloke | (I haven't played with VMWare much, so I don't know exactly how to do this, I'm afraid.) | 13:18 |
| dunz0r | Odd_Bloke: Generating an OVF would be a lot more work than what I have in mind, but I appreciate the input :) | 13:23 |
| Odd_Bloke | OK, I'll be interested to see what you come up with! :) | 13:25 |
| smoser | i dont' know how the answer at https://askubuntu.com/questions/867946/cloud-init-and-ova/867958#867958 would work | 13:28 |
| smoser | admittedly i dont understand lots of things about vmware. but that does not look like it'd work. | 13:29 |
| smoser | i would expect, though to be able to launch a vm on vmware that had an attached cdrom with a NoCloud datasource on it. | 13:31 |
| smoser | you can build such a datasource easily with cloud-localds | 13:31 |
| smoser | http://atom-lab-3.insieme.local:5880/pijaserami | 13:31 |
| smoser | err... sorry | 13:31 |
| smoser | http://paste.ubuntu.com/p/K9CcHyTd2V/ | 13:31 |
| smoser | strike two | 13:32 |
| smoser | http://paste.ubuntu.com/p/3GyHXxB2CW/ | 13:32 |
| smoser | there. | 13:32 |
| smoser | stupid finger memory for internal hastebin | 13:32 |
| otubo | if systemd-generator and/or ds-identify disabled cloud-init, it means totally disable or it still runs some things? I'm setting a network configuration with cloud-init, but when I reboot (both scripts identify that there's no datasource any more) and state as disabled | 13:45 |
| otubo | but the network configuration is reset to dhcp instead of the initial configuration which was static | 13:46 |
| Odd_Bloke | rharper: blackboxsw: I'm looking at validating a network configuration bug (https://bugs.launchpad.net/cloud-init/+bug/1812857) and I'd really appreciate some guidance as to how best to synthesise a networking environment for testing. | 13:46 |
| ubot5 | Launchpad bug 1812857 in cloud-init "RuntimeError: duplicate mac found! both 'ens4' and 'bond0' have mac '9c:XX:XX:46:5d:91'" [Medium,Fix released] | 13:46 |
| Odd_Bloke | otubo: I would expect that they wouldn't run at all, but /var/log/cloud-init.log would tell you the truth of that. | 13:46 |
| otubo | Odd_Bloke: I can see the pretty network configuration box with ascii art on /var/log/messages, setting everything back to dhcp | 13:48 |
| otubo | Odd_Bloke: and I can also see the logs in /run/cloud/ saying no datasource was found the cloud-init will be disabled | 13:49 |
| otubo | I'm a bit puzzled. | 13:49 |
| otubo | Odd_Bloke: I can also see this weird message on the logs, but I'm not sure if this is *causeing* the issue or if it is a *consequence* of the issue, or even not related: | 13:51 |
| otubo | https://pastebin.com/3NFJkwcK | 13:51 |
| smoser | otubo: if you have the generator enabled correctly | 13:59 |
| smoser | then cloud-init will not run *at all* if there is no datasource (or it is disabled) | 13:59 |
| smoser | otubo: cat /run/cloud-init/ds-identify.log | 14:00 |
| smoser | and /run/cloud-init/cloud-init-generator.log | 14:00 |
| otubo | smoser: both of them report as disabled, but I'm gonna paste here, sometimes I'm just missing something. | 14:00 |
| smoser | otubo: i suspect you have the services enabled. | 14:01 |
| smoser | the other services. | 14:01 |
| otubo | smoser: https://pastebin.com/LtdzQG1D | 14:01 |
| smoser | is suspect you have your units (cloud-init.service, cloud-init-local.service, cloud-final.service, cloud-config.service) enabled. | 14:03 |
| smoser | i suspect they are 'WantedBy=multi-user.target' | 14:03 |
| smoser | when they should be WantedBy=cloud-init.target | 14:03 |
| otubo | let me check that | 14:03 |
| smoser | the generator only handles enabling or disabling that target | 14:03 |
| otubo | smoser: bulls eye | 14:04 |
| otubo | https://pastebin.com/9PwbF6es | 14:04 |
| smoser | otubo: as upstream we'd love to have the generator enabled in rhel packaging | 14:24 |
| otubo | smoser: I'll send a patch soon, once I confirm this is the root cause :-) | 14:26 |
| otubo | smoser: thanks a lot for the help! | 14:26 |
| rharper | Odd_Bloke: yeah, I think launching an lxc with a network-config that has bond0 over eth0, should show that . | 14:44 |
| Odd_Bloke | rharper: Yeah, I got that working eventually but NoCloud doesn't seem to trigger the error. | 14:45 |
| Odd_Bloke | I can reproduce the error if I poke the appropriate function in a Python shell, but that seems like cheating. :p | 14:45 |
| rharper | hrm, it's somewhat related to the speed at which bond0 picksup the mac of the member | 14:46 |
| rharper | IIRC | 14:46 |
| * rharper re-reads the bug | 14:46 | |
| rharper | Odd_Bloke: alternatively, you could boot with the bond0 + eth0 setup; cloud-init clean; cloud-init init --local | 14:48 |
| rharper | that should trigger cloud-init seeing bond0 and eth0 with same mac | 14:48 |
| rharper | that way you don't have to rely on boot timing | 14:48 |
| Odd_Bloke | Oh, it's because the error is in openstack's convert_net_json. | 14:48 |
| Odd_Bloke | Well, it's further down, but that's where the traceback comes from. | 14:48 |
| rharper | yeah, they had an ironic deployment | 14:49 |
| Odd_Bloke | And NoCloud just passes my network config through verbatim, rather than having to convert it from OpenStack's format. | 14:49 |
| rharper | yes, I guess it's a matter of having get_interfaces() list called | 14:50 |
| Odd_Bloke | I (think I) reproduced in a KVM, but then couldn't get into it (presumably because cloud-init failed before setting my password or SSH keys?). | 14:50 |
| rharper | so either fallback networking, or some other network_config path which triggers that; and I still think best to clean and then run cloud-init init --local | 14:50 |
| rharper | you can backdoor the image | 14:50 |
| rharper | with root passwd first | 14:50 |
| Odd_Bloke | Right, yeah. Can you remind me how to do that? | 14:51 |
| rharper | yeah, I use sudo mount-image-callback --system-mounts bionic-server-cloudimg-amd64.img -- chroot _MOUNTPOINT_ /bin/bash | 14:51 |
| rharper | then passwd | 14:51 |
| Odd_Bloke | Thanks! | 14:53 |
| Odd_Bloke | rharper: Oh, right, but this is still going to boot with NoCloud. Do we have a good way of mimicking ConfigDrive locally? | 15:11 |
| smoser | Odd_Bloke: easier than that is | 15:39 |
| smoser | https://gist.github.com/smoser/8c65b8771d5ab1d99c44c285323dfff6/ | 15:39 |
| smoser | well, only slightly easier. but it makes sure that you can get in. | 15:39 |
| smoser | if you just ran 'passwd' and cloud-init got borked, you'd have no ssh host keys | 15:39 |
| smoser | but wrt config drive.. you can just craft one. easiest way to do it would probably be to just mount an existing and then change things you want. i think we possibly had a tool to make them at one point. but not sure where. | 15:41 |
| Odd_Bloke | Yeah, I do have console access so password is sufficient for this particular case, thankfully. | 15:53 |
| Odd_Bloke | Cool, I'll go find an example one for me to poke at. | 15:53 |
| Odd_Bloke | Thanks for the advice! | 15:53 |
| Odd_Bloke | OK, so I have a configdrive ISO now; how do I need to invoke kvm/QEMU to have it found properly? | 16:43 |
| Odd_Bloke | Oh, my ConfigDrive is invalid. | 16:58 |
| Odd_Bloke | That sounds like a problem for post-lunch Dan to deal with. | 16:58 |
| smoser | Odd_Bloke: use xkvm. and use '--disk=foo.img' | 17:17 |
| Odd_Bloke | smoser: Do you have a sample config drive I could take and modify? This tool I found is actually creating a NoCloud source despite having "config_drive" in its name. | 17:54 |
| smoser | that is odd in deed. | 17:55 |
| smoser | Odd_Bloke: what i'd do is launch a serverStack instance with '--config-drive=1' on the openstack cli | 17:56 |
| smoser | and then dd to get the disk. | 17:56 |
| smoser | there are probably some examples attached to bugs though | 17:56 |
| smoser | Odd_Bloke: https://bugs.launchpad.net/cloud-init/+bug/1671927 | 17:59 |
| ubot5 | Launchpad bug 1671927 in cloud-init "init local crash - unknown subnet type 'loopback'" [Medium,Fix released] | 17:59 |
| smoser | attachment 'sr0.gz' there is at least a start. | 17:59 |
| smoser | sorry, 'config-drive' is the name of the attachment | 17:59 |
| Odd_Bloke | Thank you! | 18:00 |
| Odd_Bloke | Oh right, except now I need to somehow write OpenStack network config language to reproduce this. | 18:16 |
| Odd_Bloke | rharper: You thought we might be able to reproduce this (https://bugs.launchpad.net/cloud-init/+bug/1812857) in a booted instance at stand-up. Did our conversation change that, when we realised it was ConfigDrive-specific? | 18:22 |
| ubot5 | Launchpad bug 1812857 in cloud-init "RuntimeError: duplicate mac found! both 'ens4' and 'bond0' have mac '9c:XX:XX:46:5d:91'" [Medium,Fix released] | 18:22 |
| Odd_Bloke | Because I really don't want to have to work out how to construct correct OpenStack network_data.json by hand. | 18:23 |
| rharper | it's not config drive specific; it can happen on any path in cloud-init which requires us to use net.get_interfaces() | 18:23 |
| rharper | I don't think we need to, really; I think just bringing up bond0/eth0, and then python3 -c 'from cloudinit import net; print(net.get_interfaces())' and ensure that bond0 isn't in the list | 18:24 |
| Odd_Bloke | Are we happy for me to reproduce it in a Python shell then? | 18:24 |
| rharper | I'm fine with that; | 18:24 |
| Odd_Bloke | OK, cool, I'm good with that. | 18:24 |
| rharper | I think it reasonable to show the net config, verify that eth0 and bond0 have the same mac, that cloudinit.net.is_bond() returns true on bond0 , false on eth0; and then that bond0 doesn't show up in get_interfaces(); | 18:26 |
| rharper | before/after upgrade should show that we do get bond0 in the list and it raises duplicate exception, post-upgrade we don't get the exception and bond0 isn't in the list; | 18:27 |
| Odd_Bloke | OK, cool, I'll continue with my lxd container setup I started on before thinking I might be able to reproduce the exact bug. | 18:28 |
| Odd_Bloke | Thanks! | 18:28 |
| smoser | Odd_Bloke: what you need to do is write a renderer for Openstack networking ocnfig in cloud-init | 18:41 |
| smoser | and then you could just use net-convert | 18:41 |
| Odd_Bloke | Trivial! | 18:44 |
| rharper | we already have that | 18:47 |
| rharper | smoser: Odd_Bloke: net-convert has a json input | 18:47 |
| rharper | which is effectively the openstack format | 18:48 |
| rharper | --kind json | 18:48 |
| blackboxsw | ok copr build for CentOS7 is updated for the cloud-init version which is currently undergoing SRU validation in ubuntu: 19.2.~385 https://copr.fedorainfracloud.org/coprs/g/cloud-init/el-testing/ | 18:50 |
| smoser | thats input | 18:50 |
| smoser | he needed output | 18:50 |
| rharper | ? | 18:51 |
| rharper | ah, right | 18:51 |
| rharper | I have input (aka output) that includes bonds and macs like Ironic | 18:51 |
| rharper | but even that's more complicated than just verifying that net.get_interfaces() skips bonds | 18:51 |
| Odd_Bloke | Yep, I'll proceed with this. | 19:00 |
| blackboxsw | rharper: are we waiting on bionic and disco for https://github.com/cloud-init/ubuntu-sru/pull/45/files? | 19:52 |
| rharper | ugh, right | 19:52 |
| rharper | lemme do that | 19:52 |
| rharper | blackboxsw: thanks | 19:53 |
| blackboxsw | no worries | 19:53 |
| blackboxsw | pending cloud-init SRU 19.2.24 publish isn't until Tuesday next week anyway. so still time :) | 19:53 |
| rharper | but I had forgotten, so thanks | 19:54 |
| Odd_Bloke | OK, I have a test case, but it doesn't work on xenial because the bond never gets created because: /etc/network/if-pre-up.d/ifenslave: 37: /etc/network/if-pre-up.d/ifenslave: cannot create /sys/class/net/bonding_masters: Permission denied | 20:10 |
| Odd_Bloke | Ugh, pretty sure this is a problem because I'm in a container; the owner is different outside of a container. | 20:23 |
| rharper | modprobe bonding IIRC | 20:35 |
| rharper | on the host | 20:35 |
| rharper | IIRC | 20:35 |
| rharper | Odd_Bloke: ^ | 20:35 |
| rharper | container bonding is PITA | 20:36 |
| Odd_Bloke | Nah, it doesn't work even in a privileged container with the module loaded; "read-only filesystem". | 20:38 |
| Odd_Bloke | I think I'm going to need to do the xenial test in a VM. | 20:38 |
| Odd_Bloke | Which is fine, because we've established that using NoCloud is fine. | 20:38 |
| rharper | https://github.com/cloud-init/ubuntu-sru/pull/45 | 21:49 |
| rharper | blackboxsw: updated | 21:49 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!