[09:55] <otubo> I'm gonna leave this question here and leave for lunch because I know it might take some time for someone to pick it up and answer: On this bug https://bugzilla.redhat.com/show_bug.cgi?id=1593010 the systemd-generator and the ds-identify read correctly that the second boot doesn't have any DataSource and, from the logs, they disable cloud-init. But still, on the second boot, the network configuration gets
[09:55] <ubot5> bugzilla.redhat.com bug 1593010 in cloud-init "cloud-init network configuration does not persist reboot [RHEL 7.8]" [High,New]
[09:55] <otubo> reset to dhcp.
[09:56] <otubo> If anyone has any hint on this, I'll be glad to thank personally on cloud-init summit :-) And now I'm off to lunch, be back soon.
[10:34] <otubo> perhaps rharper ^
[12:03] <dunz0r> Is there some way I could get cloud-init to "ask" me for the hostname? Doing a template for a vmware-cluster.
[12:33] <nvz> dunz0r: :)
[12:33] <nvz> guess you beat me here
[12:34] <nvz> dunz0r: thanks for mentioning this, I think this could be quite useful at some point I think I may hang out here and get familiarized with the project
[12:36] <dunz0r> Yeah. This seems really useful to what I'm trying to do, especially since I don't have to reinvent the wheel, like I was planning on first.
[12:37] <nvz> would've been my natural reaction
[12:37] <nvz> this is a really nice looking project
[12:37] <nvz> I just can't believe it doesnt appear they've thought of making it have interactivity
[12:38] <nvz> Windows has for a long time had that thing where you do an OEM install and on the first run it asks you all those questions
[12:38] <nvz> so you can have the system installed but still allow the user to set their timezone, username, update preferences, etc..
[12:39] <dunz0r> I can see why there's no interactivity, it isn't supposed to. This is for largescale deployments. Serious stuff
[12:39] <nvz> I'm fairly capable in python and this does interest me even though I dont have the kind of need for it right now like you seem to
[12:39] <nvz> I had always wandered about being able to configure linux systems like that to be able to have initial config done by the user and it looks like I could easily build that on top of this
[12:41] <nvz> like I just reinstalled debian on my cousin's laptop cause the m.2 ssd died and I had to install to his sata hdd he was using for /home.. and I'd have much rather sent it back to him so it booted up asking him for his user/password/fingerprints, etc..
[12:41] <nvz> rather than having to set what I could and leave a README on his desktop explaining the rest
[12:41] <nvz> and if I ever start doing more serious stuff with VMs this could really be useful there too
[12:46] <dunz0r> nvz: My dream here at work is having ALL configuration in salt-stack and essentially never touch the machines, but that's waaay ahead in the future :)
[12:48] <nvz> Ah, I'd never heard of salt-stack I thought you were referring to the salt minion thing in cloud-init
[12:48] <nvz> I've heard of ansible, puppet, chef, but these things are news to me
[12:49] <nvz> what amazes me about this cloud-init is like python itself, its well documented and I immediately felt comfortable with it just glancing over the site
[12:49] <nvz> it all just makes sense
[12:49] <nvz> these sorts of things can often be so cumbersome they're not really useful
[12:50] <dunz0r> nvz: Salt-minion is part of saltstack :)
[12:50] <dunz0r> Saltstack is the whole shebang
[12:51] <nvz> hmm.. I should probably familiarize with these things they're bound to come of use at some point.. saltstack, ansible, puppet, chef, vagrant, cloud-init
[12:51] <nvz> terraform
[13:00] <nvz> hmm.. see now this damn saltstack has one of them fancy websites that tells you a whole lot of nothing
[13:01] <Odd_Bloke> dunz0r: Are you asking for a tool that will help you generate cloud-config?  Or some sort of interactivity in the boot process?
[13:01] <nvz> I glanced about 90 seconds at the cloud-init site and I could probably use it like I been doing it for months.. I glance at the saltstack page and I'm still not even sure wth it is
[13:01] <nvz> Odd_Bloke: that was the general idea, they want to clone vms and be able to supply things on the first boot
[13:02] <dunz0r> Odd_Bloke: Interactivity in the boot-process. The machines aren't on AWS or something like that, so no way to input a config-file
[13:03] <dunz0r> nvz: Saltstacks frontpage is awful, yeah. https://docs.saltstack.com/en/latest/ is what you want :)
[13:03] <Odd_Bloke> dunz0r: Where are these machines?
[13:03] <dunz0r> Odd_Bloke: In a VMWare cluster.
[13:03] <nvz> dunz0r: they make it sound like something commercial.. is it something you have to buy?
[13:04] <Odd_Bloke> Interactivity during boot isn't something that would be generally useful (because most launches that involve cloud-init generally aren't accessible to the user at all until SSH is up).
[13:04] <dunz0r> nvz: No, but they have an "enterprise" version. But I've never felt limited by the foss-version.
[13:04] <dunz0r> Odd_Bloke: Well, I've got console access.
[13:05] <nvz> Odd_Bloke: that is a good point
[13:05] <Odd_Bloke> Sure, some places you will have it, but that's not generally true.
[13:05] <dunz0r> Odd_Bloke: Today we create VMs from a template, set hostname/ip/regenerate ssh-host-keys/etc via the console, and that's a hassle
[13:05] <dunz0r> Odd_Bloke: Well, not generally no. But in my particular use case it is
[13:05] <nvz> Odd_Bloke: however I don't know of anything that is like this that could offer that OEM installed OS kinda feel and this seems to be ready to build that kinda thing on top of
[13:06] <dunz0r> I think I'll write something that gets input from the user during boot and then uses cloud-init to do the actual setup.
[13:07] <dunz0r> "user" generally being me or my colleagues
[13:07] <nvz> Odd_Bloke: my idea, and mind you I just heard about this project, and glanced breifly at the site.. but is it possible to just ship a cloud-config, then have the bit that allows you to run commands at boot hook something to basically interactively make vendor data?
[13:07] <Odd_Bloke> You could also look at generating a config drive from the input, and attaching that to the instances.
[13:07] <nvz> on the first boot that is
[13:09] <Odd_Bloke> I think the tricky part would be getting input from the console; cloud-init just runs as a daemon and outputs to the console via syslog.
[13:09] <dunz0r> Odd_Bloke: My plan is to write a bash script that interactively asks for hostname and IP, and then generate a cloud.cfg from that info and a base template
[13:09] <dunz0r> and THEN use cloud-init via the script to "setup" the machine
[13:10] <nvz> they have things that you can use to have an auto-login getty and such
[13:10] <nvz> you could have the base config set that up then get input and supply that input as vendor data overriding any config as supplied by the user
[13:11] <Odd_Bloke> So vendor-data doesn't override user-data, intentionally; the user should always be in control of their instances, regardless of what their vendor wants the configuration to be.
[13:12] <Odd_Bloke> dunz0r: You may want to consider having your script generate a config drive: https://askubuntu.com/a/867958
[13:12] <Odd_Bloke> Then you can attach that to the VM as you launch it, and cloud-init should find the configuration from it without intervention during boot.
[13:12] <nvz> I didn't even read about user data.. when I glanced it over I saw the main cloud-config then I seen the bit about vendor data that it says overrides cloud-config
[13:14] <Odd_Bloke> So cloud-config is really a format; it's YAML with a "#cloud-config" header so cloud-init knows that it's intended to be treated as cloud-config.
[13:14] <Odd_Bloke> Both user-data and vendor-data can specify cloud-config.
[13:15] <nvz> yeah I noticed that much.. and I'm a fan of yaml.. when I first heard of it I avoided even looking at it because I thought it was another damn markup language.. years later when I finally looked it over I felt stupid :P
[13:16] <nvz> turns out its not Yet Another Markup Language, its YAML Ain't Markup Language
[13:16] <dunz0r> Wait what:O
[13:17] <Odd_Bloke> dunz0r: Actually, looking at https://git.launchpad.net/cloud-init/tree/doc/sources/ovf might give you a better way of generating the ISO containing user-data.
[13:18] <Odd_Bloke> (I haven't played with VMWare much, so I don't know exactly how to do this, I'm afraid.)
[13:23] <dunz0r> Odd_Bloke: Generating an OVF would be a lot more work than what I have in mind, but I appreciate the input :)
[13:25] <Odd_Bloke> OK, I'll be interested to see what you come up with! :)
[13:28] <smoser> i dont' know how the answer at https://askubuntu.com/questions/867946/cloud-init-and-ova/867958#867958 would work
[13:29] <smoser> admittedly i dont understand lots of things about vmware.  but that does not look like it'd work.
[13:31] <smoser> i would expect, though to be able to launch a vm on vmware that had an attached cdrom with a NoCloud datasource on it.
[13:31] <smoser> you can build such a datasource easily with cloud-localds
[13:31] <smoser> http://atom-lab-3.insieme.local:5880/pijaserami
[13:31] <smoser> err... sorry
[13:31] <smoser> http://paste.ubuntu.com/p/K9CcHyTd2V/
[13:32] <smoser> strike two
[13:32] <smoser> http://paste.ubuntu.com/p/3GyHXxB2CW/
[13:32] <smoser> there.
[13:32] <smoser> stupid finger memory for internal hastebin
[13:45] <otubo> if systemd-generator and/or ds-identify disabled cloud-init, it means totally disable or it still runs some things? I'm setting a network configuration with cloud-init, but when I reboot (both scripts identify that there's no datasource any more) and state as disabled
[13:46] <otubo> but the network configuration is reset to dhcp instead of the initial configuration which was static
[13:46] <Odd_Bloke> rharper: blackboxsw: I'm looking at validating a network configuration bug (https://bugs.launchpad.net/cloud-init/+bug/1812857) and I'd really appreciate some guidance as to how best to synthesise a networking environment for testing.
[13:46] <ubot5> Launchpad bug 1812857 in cloud-init "RuntimeError: duplicate mac found! both 'ens4' and 'bond0' have mac '9c:XX:XX:46:5d:91'" [Medium,Fix released]
[13:46] <Odd_Bloke> otubo: I would expect that they wouldn't run at all, but /var/log/cloud-init.log would tell you the truth of that.
[13:48] <otubo> Odd_Bloke: I can see the pretty network configuration box with ascii art on /var/log/messages, setting everything back to dhcp
[13:49] <otubo> Odd_Bloke: and I can also see the logs in /run/cloud/ saying no datasource was found the cloud-init will be disabled
[13:49] <otubo> I'm a bit puzzled.
[13:51] <otubo> Odd_Bloke: I can also see this weird message on the logs, but I'm not sure if this is *causeing* the issue or if it is a *consequence* of the issue, or even not related:
[13:51] <otubo> https://pastebin.com/3NFJkwcK
[13:59] <smoser> otubo: if you have the generator enabled correctly
[13:59] <smoser> then cloud-init will not run *at all* if there is no datasource (or it is disabled)
[14:00] <smoser> otubo: cat /run/cloud-init/ds-identify.log
[14:00] <smoser> and /run/cloud-init/cloud-init-generator.log
[14:00] <otubo> smoser: both of them report as disabled, but I'm gonna paste here, sometimes I'm just missing something.
[14:01] <smoser> otubo: i suspect you have the services enabled.
[14:01] <smoser> the other services.
[14:01] <otubo> smoser: https://pastebin.com/LtdzQG1D
[14:03] <smoser> is suspect you have your units (cloud-init.service, cloud-init-local.service, cloud-final.service, cloud-config.service) enabled.
[14:03] <smoser> i suspect they are 'WantedBy=multi-user.target'
[14:03] <smoser> when they should be WantedBy=cloud-init.target
[14:03] <otubo> let me check that
[14:03] <smoser> the generator only handles enabling or disabling that target
[14:04] <otubo> smoser: bulls eye
[14:04] <otubo> https://pastebin.com/9PwbF6es
[14:24] <smoser> otubo: as upstream we'd love to have the generator enabled in rhel packaging
[14:26] <otubo> smoser: I'll send a patch soon, once I confirm this is the root cause :-)
[14:26] <otubo> smoser: thanks a lot for the help!
[14:44] <rharper> Odd_Bloke: yeah, I think launching an lxc with a network-config that has bond0 over eth0, should show that .
[14:45] <Odd_Bloke> rharper: Yeah, I got that working eventually but NoCloud doesn't seem to trigger the error.
[14:45] <Odd_Bloke> I can reproduce the error if I poke the appropriate function in a Python shell, but that seems like cheating. :p
[14:46] <rharper> hrm,  it's somewhat related to the speed at which bond0 picksup the mac of the member
[14:46] <rharper> IIRC
[14:46]  * rharper re-reads the bug
[14:48] <rharper> Odd_Bloke: alternatively, you could boot with the bond0 + eth0 setup; cloud-init clean;  cloud-init init --local
[14:48] <rharper> that should trigger cloud-init seeing bond0 and eth0 with same mac
[14:48] <rharper> that way you don't have to rely on boot timing
[14:48] <Odd_Bloke> Oh, it's because the error is in openstack's convert_net_json.
[14:48] <Odd_Bloke> Well, it's further down, but that's where the traceback comes from.
[14:49] <rharper> yeah, they had an ironic deployment
[14:49] <Odd_Bloke> And NoCloud just passes my network config through verbatim, rather than having to convert it from OpenStack's format.
[14:50] <rharper> yes, I guess it's a matter of having get_interfaces() list called
[14:50] <Odd_Bloke> I (think I) reproduced in a KVM, but then couldn't get into it (presumably because cloud-init failed before setting my password or SSH keys?).
[14:50] <rharper> so either fallback networking, or some other network_config path which triggers that;  and I still think best to clean and then run cloud-init init --local
[14:50] <rharper> you can backdoor the image
[14:50] <rharper> with root passwd first
[14:51] <Odd_Bloke> Right, yeah.  Can you remind me how to do that?
[14:51] <rharper> yeah, I use  sudo mount-image-callback --system-mounts bionic-server-cloudimg-amd64.img -- chroot _MOUNTPOINT_ /bin/bash
[14:51] <rharper> then passwd
[14:53] <Odd_Bloke> Thanks!
[15:11] <Odd_Bloke> rharper: Oh, right, but this is still going to boot with NoCloud.  Do we have a good way of mimicking ConfigDrive locally?
[15:39] <smoser> Odd_Bloke: easier than that is
[15:39] <smoser>  https://gist.github.com/smoser/8c65b8771d5ab1d99c44c285323dfff6/
[15:39] <smoser> well, only slightly easier. but it makes sure that you can get in.
[15:39] <smoser> if you just ran 'passwd' and cloud-init got borked, you'd have no ssh host keys
[15:41] <smoser> but wrt config drive.. you can just craft one. easiest way to do it would probably be to just mount an existing and then change things you want.  i think we possibly had a tool to make them at one point. but not sure where.
[15:53] <Odd_Bloke> Yeah, I do have console access so password is sufficient for this particular case, thankfully.
[15:53] <Odd_Bloke> Cool, I'll go find an example one for me to poke at.
[15:53] <Odd_Bloke> Thanks for the advice!
[16:43] <Odd_Bloke> OK, so I have a configdrive ISO now; how do I need to invoke kvm/QEMU to have it found properly?
[16:58] <Odd_Bloke> Oh, my ConfigDrive is invalid.
[16:58] <Odd_Bloke> That sounds like a problem for post-lunch Dan to deal with.
[17:17] <smoser> Odd_Bloke: use xkvm. and use '--disk=foo.img'
[17:54] <Odd_Bloke> smoser: Do you have a sample config drive I could take and modify?  This tool I found is actually creating a NoCloud source despite having "config_drive" in its name.
[17:55] <smoser> that is odd in deed.
[17:56] <smoser> Odd_Bloke: what i'd do is launch a serverStack instance with '--config-drive=1' on the openstack cli
[17:56] <smoser> and then dd to get the disk.
[17:56] <smoser> there are probably some examples attached to bugs though
[17:59] <smoser> Odd_Bloke: https://bugs.launchpad.net/cloud-init/+bug/1671927
[17:59] <ubot5> Launchpad bug 1671927 in cloud-init "init local crash - unknown subnet type 'loopback'" [Medium,Fix released]
[17:59] <smoser> attachment 'sr0.gz' there is at least a start.
[17:59] <smoser> sorry, 'config-drive' is the name of the attachment
[18:00] <Odd_Bloke> Thank you!
[18:16] <Odd_Bloke> Oh right, except now I need to somehow write OpenStack network config language to reproduce this.
[18:22] <Odd_Bloke> rharper: You thought we might be able to reproduce this (https://bugs.launchpad.net/cloud-init/+bug/1812857) in a booted instance at stand-up.  Did our conversation change that, when we realised it was ConfigDrive-specific?
[18:22] <ubot5> Launchpad bug 1812857 in cloud-init "RuntimeError: duplicate mac found! both 'ens4' and 'bond0' have mac '9c:XX:XX:46:5d:91'" [Medium,Fix released]
[18:23] <Odd_Bloke> Because I really don't want to have to work out how to construct correct OpenStack network_data.json by hand.
[18:23] <rharper> it's not config drive specific; it can happen on any path in cloud-init which requires us to use net.get_interfaces()
[18:24] <rharper> I don't think we need to, really; I think just bringing up bond0/eth0, and then python3 -c 'from cloudinit import net;  print(net.get_interfaces())'  and ensure that bond0 isn't in the list
[18:24] <Odd_Bloke> Are we happy for me to reproduce it in a Python shell then?
[18:24] <rharper> I'm fine with that;
[18:24] <Odd_Bloke> OK, cool, I'm good with that.
[18:26] <rharper> I think it reasonable to show the net config, verify that eth0 and bond0 have the same mac, that cloudinit.net.is_bond() returns true on bond0 , false on eth0; and then that bond0 doesn't show up in get_interfaces();
[18:27] <rharper> before/after upgrade should show that we do get bond0 in the list and it raises duplicate exception, post-upgrade we don't get the exception and bond0 isn't in the list;
[18:28] <Odd_Bloke> OK, cool, I'll continue with my lxd container setup I started on before thinking I might be able to reproduce the exact bug.
[18:28] <Odd_Bloke> Thanks!
[18:41] <smoser> Odd_Bloke: what you need to do is write a renderer for Openstack networking ocnfig in cloud-init
[18:41] <smoser> and then you could just use net-convert
[18:44] <Odd_Bloke> Trivial!
[18:47] <rharper> we already have that
[18:47] <rharper> smoser: Odd_Bloke:  net-convert  has a json input
[18:48] <rharper> which is effectively the openstack format
[18:48] <rharper> --kind json
[18:50] <blackboxsw> ok copr build for CentOS7 is updated for the cloud-init version which is currently undergoing SRU validation in ubuntu: 19.2.~385 https://copr.fedorainfracloud.org/coprs/g/cloud-init/el-testing/
[18:50] <smoser> thats input
[18:50] <smoser> he needed output
[18:51] <rharper> ?
[18:51] <rharper> ah, right
[18:51] <rharper> I have input (aka output) that includes bonds and macs like Ironic
[18:51] <rharper> but even that's more complicated than just verifying that net.get_interfaces() skips bonds
[19:00] <Odd_Bloke> Yep, I'll proceed with this.
[19:52] <blackboxsw> rharper: are we waiting on bionic and disco for https://github.com/cloud-init/ubuntu-sru/pull/45/files?
[19:52] <rharper> ugh, right
[19:52] <rharper> lemme do that
[19:53] <rharper> blackboxsw: thanks
[19:53] <blackboxsw> no worries
[19:53] <blackboxsw> pending cloud-init SRU 19.2.24 publish isn't until Tuesday next week anyway. so still time :)
[19:54] <rharper> but I had forgotten, so thanks
[20:10] <Odd_Bloke> OK, I have a test case, but it doesn't work on xenial because the bond never gets created because: /etc/network/if-pre-up.d/ifenslave: 37: /etc/network/if-pre-up.d/ifenslave: cannot create /sys/class/net/bonding_masters: Permission denied
[20:23] <Odd_Bloke> Ugh, pretty sure this is a problem because I'm in a container; the owner is different outside of a container.
[20:35] <rharper> modprobe bonding IIRC
[20:35] <rharper> on the host
[20:35] <rharper> IIRC
[20:35] <rharper> Odd_Bloke: ^
[20:36] <rharper> container bonding is PITA
[20:38] <Odd_Bloke> Nah, it doesn't work even in a privileged container with the module loaded; "read-only filesystem".
[20:38] <Odd_Bloke> I think I'm going to need to do the xenial test in a VM.
[20:38] <Odd_Bloke> Which is fine, because we've established that using NoCloud is fine.
[21:49] <rharper> https://github.com/cloud-init/ubuntu-sru/pull/45
[21:49] <rharper> blackboxsw: updated