| prologic | Can anyone help me debug why cloud-init isn't working here correctly? It only seems to do half of it's job on the creation of a new instance. But if I rm -rf /var/lib/cloud and reboot it does everything | 04:52 |
|---|---|---|
| prologic | Problem I have is I already nuked the cloud-init state in my package template before pushing the image and initially creating an instance from it | 04:53 |
| prologic | this is really baffling me | 04:53 |
| prologic | Hmm 🤔 I don't get it | 05:12 |
| meena | prologic: maybe if you post the log from that first run, we could see what's up and help | 06:58 |
| prologic | umm sure okay, I've pretty much given up for the day with no luck | 06:59 |
| prologic | I've been scouring through it quite a bit myself | 06:59 |
| prologic | what would I have been looking for specifically? | 06:59 |
| prologic | Why does cloud-init clean && reboot work? But an initial boot not? | 06:59 |
| prologic | I feel like I'm missing something incredibly stupid here :/ | 07:00 |
| === waxfire7 is now known as waxfire | ||
| prologic | meena here it is https://transfer.mills.io/8hwr8/cloud-init.log | 07:11 |
| meena | prologic: did you run cloud-init clean before finishing the package? | 08:17 |
| prologic | yes | 08:23 |
| prologic | it's part of the last step in the packer build | 08:23 |
| Odd_Bloke | prologic: Thanks for the log! I don't see anything immediately wrong, though a couple of notes: 19.4 is fairly old and Photon support isn't upstream, so this may either be already resolved or an issue for whoever has produced your package. That said, when you say "It only seems to do half of it's job on the creation of a new instance", what do you mean? Which parts does it do and which parts does it | 14:19 |
| Odd_Bloke | not? | 14:19 |
| prologic | For example it runs the final_message | 14:20 |
| prologic | I see that in the journal logs | 14:20 |
| prologic | But none of the write_files get run | 14:20 |
| prologic | I _believe_ run_cmd(s) run and some other stuff | 14:20 |
| prologic | it's really weird | 14:21 |
| Odd_Bloke | prologic: Oh, actually, looking even further, this is also using a datasource that isn't upstream, so we may not be able to get very far in supporting you. Where is this cloud-init being installed from? | 14:21 |
| prologic | well cloud-init is a package Photon are installing obviously | 14:22 |
| prologic | it's part of their ISO from the installer | 14:22 |
| prologic | I'm just doing a packer build and adding a few bits/pieces we need | 14:22 |
| prologic | But you are right, the data source is coming from VMGuestInfo from vSphere | 14:22 |
| prologic | But | 14:23 |
| prologic | When cloud-init runs on boot I _do_ see the full user-data I expect to see in /var/lib/cloud/instances/.../user-data.txt | 14:23 |
| Odd_Bloke | Well, I don't really know what Photon is: they haven't engaged with us upstream to my knowledge. :) | 14:23 |
| prologic | i.e: there is nothing missing form the data source | 14:23 |
| prologic | https://github.com/vmware/photon | 14:23 |
| prologic | My understanding is, it's just a repackaged/rebranded CentOS distribution optimized for running containers (although that's quite questionable IHMO) | 14:24 |
| Odd_Bloke | prologic: I see the 'init' stage running twice in that log: do you know why that's happening? | 14:32 |
| prologic | unfortuantely no | 14:33 |
| Odd_Bloke | OK, I'd suggest looking into systemd units to see if you can figure that out: that's not how we ship cloud-init upstream so it could be leading to strange behaviour. If Photon are doing that, then I'm afraid you'll have to take it up with them. | 14:35 |
| prologic | How is cloud-init normally run btw? | 14:35 |
| prologic | With or without SystemD's help? | 14:35 |
| * prologic grumbles about how much he loathes SystemD | 14:36 | |
| prologic | And also; (if you can help explain this, or maybe it might give you a clue?) If run cloud-init clean && reboot | 14:37 |
| prologic | The vm comes back with all the things in the user-data correctly run | 14:37 |
| prologic | it's just on the first boot it doesn't (i.e: terraform apply -> vsphere create vm blah) | 14:38 |
| prologic | this is the behaviour I really don't get at all | 14:38 |
| Odd_Bloke | cloud-init runs in 5 stages (the first of which is optional and only supported by systemd), you can read more at https://cloudinit.readthedocs.io/en/latest/topics/boot.html | 14:38 |
| Odd_Bloke | So your init system needs to be invoking cloud-init in the right order and at the right points in boot: if Photon's packaging hasn't hooked that up as upstream expects, then it's not surprising that we would see strange behaviour. | 14:40 |
| Odd_Bloke | As for why a `clean` fixes it: I'm not sure, but I think the root cause of why the first boot doesn't work is likely related to the multiple runs of `init` so I think investigating that is the best place to start. | 14:41 |
| prologic | Ok. | 14:42 |
| prologic | Thanks very much for this insight. | 14:42 |
| Odd_Bloke | :) | 14:43 |
| Odd_Bloke | Let us know how you get on! | 14:43 |
| prologic | Sure will do | 14:45 |
| prologic | But at this point if I try Photon v4 that that exhibits the same behaviour, I'm a bit unwilling to go debug this for VMWare :/ | 14:46 |
| prologic | I might have to decide whether running Photon as our base os for docker is really a good cohice | 14:46 |
| Krikke | how is one supposed to use the network config? does it also require a magic comment at the top of the file? | 15:07 |
| Krikke | https://pastebin.com/rQXm7eMY | 15:08 |
| Krikke | lol just realized you can mount the iso :) | 15:13 |
| Krikke | seems at least network-config exists there | 15:13 |
| Krikke | netplan is also installed I think, it's an archlinux openstack image | 15:14 |
| Krikke | if it's not an dependecy for something I'm installing | 15:14 |
| Krikke | required by: cloud-init | 15:15 |
| Krikke | and I'm not installing that explicitly | 15:15 |
| Krikke | ah, removing the networking: from the start did the trick | 15:23 |
| jas02 | Hello guys, I am having problem with cloud-init and ssh public key. What am I doing wrong? https://gist.github.com/jas02/868ab330e05b126e909934238e587607 | 16:55 |
| jas02 | Figured it out, it woks now | 17:06 |
| dwigton | In ubuntu 20.04 server, if I add some settings in /etc/cloud/cloud.cfg.d chould they automatically run on reboot or is that a seperate setting? Asking because it didn't take effect until I did "cloud-init clean" "cloud-init init" That seemed a bit overkill. | 17:35 |
| dwigton | I don't particularly want new server keys everytime I make a change. | 17:36 |
| === ijohnson is now known as ijohnson|lunch | ||
| === ijohnson|lunch is now known as ijohnson | ||
| blackboxsw | falcojr: and Odd_Bloke I finally came out of a tunnel/hole on integration tests for Azure handling case-insensitive product UUID across kernel upgrades. I *think* I've got my ducks in a row on this. but wanted some feedback on better debugging what's going on here | 21:54 |
| blackboxsw | just pushed ba8e07159b748228eb2bbf5d807cbc2956e535f5 to https://github.com/canonical/cloud-init/pull/798 | 21:55 |
| blackboxsw | oops n/m the help debugging. I didn't realize that there is repeated information exposing and reporting the user_data that I provided to the test case. | 21:58 |
| blackboxsw | I just hadn't scrolled up far enough to see a real traceback | 21:58 |
| falcojr | glad I could help ;) | 22:00 |
| Odd_Bloke | Good work falcojr. | 22:12 |
| Krikke | wait cloud-init is canonical's? | 23:54 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!