| minimal | effendy[m]: what about the systemd journal? | 00:10 |
|---|---|---|
| falcojr | I know minimal could only speak for Alpine, but the story is the same on Ubuntu. Cloud-init doesn't reboot on its own, nor does a standard Ubuntu image (including OVA images). One possible crude way to see if there's something external watching cloud-init is grepping "/lib/systemd" for "cloud". There's about a dozen or so files that match, but it | 04:40 |
| falcojr | shouldn't be hard to wade those to see if there's another service waiting on one cloud-init's services to complete. | 04:40 |
| effendy[m] | Unfortunately I'm not finding anything relevant, just cloud-init related stuff. Unless the service somehow disappears afterwards. | 07:57 |
| effendy[m] | Rather daunting, I have to say. This blocks my whole automation process. | 07:57 |
| effendy[m] | falcojr: https://devicetests.com/stop-automatic-rebooting-ubuntu-22-04 This article seems to be contradicting what you're saying in regards to ubuntu cloud-images. | 12:19 |
| effendy[m] | But this does seem to be related to 'autoinstall' (so specific to Ubuntu). | 12:20 |
| falcojr | effendy[m]: Are you using an installer image? Whats the name of the image you're using? | 13:17 |
| effendy[m] | Just to be clear, I'm not saying that cloud-init itself is the issue here, I'm just challenging the assumption that cloudimages generally don't reboot or at least that this would be unusual with a relatively "default" configuration. | 13:19 |
| effendy[m] | This is the image I'm using: https://cloud-images.ubuntu.com/jammy/current/ (the ova one) | 13:19 |
| falcojr | that's unfortunately a fairly misleading article. It's conflating cloud-init with subiquity, the Ubuntu server installer. When running the server installer, it will automatically reboot at the end of the process, but that's only when using the installer image, which is completely separate from cloud-init or a cloud image | 13:19 |
| effendy[m] | Yeah, actually, I thought of that too after reading it again :D | 13:20 |
| effendy[m] | And I'm not sure 'autoinstaller' has anything to do anymore at this stage, given that the cloud-images is an already installed operating system | 13:20 |
| minimal | effendy[m]: the "systemd[1]: Reloading." message you mentioned earlier is systemd *reloading* its config, not systemd rebooting... | 13:21 |
| effendy[m] | I've tried a previous image from May, but Canonical has retroactively changed things. It's quite surprising. The image from 24 May doesn't exist anymore, there's one from 31 May only. And funnily enough the ssh server isn't starting anymore, because the host keys aren't set up! Cloud-init also doesn't seem be to running at all (this is the first stage with packer). | 13:22 |
| effendy[m] | minimal: I know. I simply pasted the log messages around that time where the system was shutting down. | 13:23 |
| minimal | effendy[m]: ok but when you showed that message you said "That's the start of the rebooting process" | 13:23 |
| effendy[m] | I still haven't found out a solution to understand what is rebooting it, I've installed auditd, but I'm only following the executables (as in halt, poweroff, reboot, etc.) | 13:23 |
| effendy[m] | minimal: ... | 13:23 |
| effendy[m] | It is. It is in the proximity. | 13:24 |
| effendy[m] | Never mind. Now it's clear, I hope. | 13:24 |
| minimal | effendy[m]: I.m being precise - a message showing systemd reloading its config is not a sign of systemd rebooting | 13:24 |
| effendy[m] | Ok. | 13:24 |
| minimal | have you tried asking for help in the ubuntu channel? | 13:24 |
| effendy[m] | I did. No one is answering :) So I'm bothering you. | 13:24 |
| minimal | well from what you've said it doesn't sound like a cloud-init issue but rather something else relating to the ubuntu cloud images | 13:25 |
| effendy[m] | Basically I've never got a question related to cloud-init answered on Ubuntu. Ever. And I did try asking several times. | 13:25 |
| effendy[m] | Yes, ok. Then I'll leave it at that I guess. | 13:25 |
| falcojr | Here? We've answered your questions. You just don't like the answers :D | 13:26 |
| minimal | effendy[m]: but it is not (directly) related to cloud-init from the information you've provided | 13:26 |
| effendy[m] | falcojr: I was talking about the Ubuntu channel. | 13:26 |
| minimal | There's no ubuntu cloud channel afaik | 13:27 |
| falcojr | #ubuntu-server | 13:27 |
| effendy[m] | It is related at least to the implementation of Canonical of cloud-init, given that, when cloud-init doesn't start, the VM isn't rebooting. | 13:27 |
| falcojr | 'I'm just challenging the assumption that cloudimages generally don't reboot or at least that this would be unusual with a relatively "default" configuration.' Is there anything I can do to convince you? I'm a Canonical employee and have been booting cloud images for years on every major cloud | 13:27 |
| effendy[m] | So it's pretty close anyway. But yes, I understand your point. | 13:27 |
| effendy[m] | Then maybe it's a vSphere-related issue... Yeah, ok, I get it. | 13:28 |
| minimal | effendy[m]: you downloaded the images from cloud-images.ubuntu.com? The images I see there are Ubuntu Server, and so that's with subiquity | 13:28 |
| effendy[m] | Yes, from there. I didn't know subiquity played a role here. | 13:29 |
| falcojr | minimal: our cloud images don't have subiquity | 13:29 |
| effendy[m] | Right... | 13:29 |
| minimal | falcojr: I thought all ubuntu server images had subiquity | 13:30 |
| effendy[m] | Subiquity runs only when you install the operating system from the start, right? With the installer and all that? | 13:30 |
| minimal | rather than any ubuntu (non-server) images | 13:30 |
| falcojr | effendy[m]: right, it's the name of the server installer | 13:30 |
| falcojr | minimal: no, it's the separate installer image | 13:31 |
| minimal | ah ok | 13:31 |
| effendy[m] | falcojr: so 'autoinstaller' would also make no difference in the context of cloud-images, I assume. | 13:31 |
| falcojr | effendy[m]: correct, they don't contain the installer | 13:32 |
| effendy[m] | When I built templates starting from the beginning, I didn't have this issue, the VM wouldn't reboot. And I used autoinstaller. But cloud-images, are of course, much more convenient. | 13:34 |
| effendy[m] | Well, ok, I'll try to figure it somehow, but I'm not sure how, I'll probably have to find a different setup altogether, 'cause there's nothing relevant out there. | 13:34 |
| minimal | like any problem it generally cannot be fixed until we know what the problem/what is causing the problem actually is | 13:35 |
| minimal | perhaps try asking on ubuntu-server channel as that is specific to the server release? | 13:36 |
| effendy[m] | ... | 13:37 |
| effendy[m] | I'm not sure if you just read between lines and not the lines themselves :D But I'll leave it it that :) | 13:37 |
| effendy[m] | Thanks. | 13:37 |
| minimal | I'm trying to help but it sounds like you don't want it so I'll leave it at that | 13:39 |
| effendy[m] | No, you're trying to superficially get the gist of the problem. And I understand that, you don't owe me anything, you're doing this freely, but it's not great to have to have to answer the exact same questions for which I've been giving answers already. But really, I don't want to argue, I get it, as I said, you don't owe me anything. It's ok. | 13:44 |
| minimal | if your "if you just read between the lines" is referring to me suggesting that you ask on the ubnutu-server channel, you previously mentioned asking on the ubnutu channel but I don't remember you ever mentioned asking already on the ubnutu-server channel | 13:52 |
| minimal | I managed to mistype ubuntu the same way 3 times, impressive :-) | 13:53 |
| effendy[m] | I'm also referring to our discussion yesterday :D But again, I don't want to have a go at anymore. The point is, yes, I've reached a sort of dead end and I understand you cannot help me further. If I do come up with a solution, I'll leave a message here, in case anyone was wondering what was going on (I bet everyone is waiting impatiently for one) | 13:55 |
| effendy[m] | at anyone* :) | 13:59 |
| effendy[m] | A different/unrelated question: is the syntax that is injected in netplan by the cloud-init network configuration the responsibility of the Ubuntu developers or that of the cloud-init ones? | 14:15 |
| effendy[m] | By default, cloud-init enters deprecated syntax related to routes, i.e. 'gateway4'. | 14:15 |
| falcojr | effendy[m]: that would be cloud-init. That has been updated in a recent release. Let me check where | 14:16 |
| effendy[m] | I think I might have found something. | 14:22 |
| effendy[m] | look into /var/log/vmware-imc/toolsDeployPkg.log, I've come across the following: | 14:23 |
| effendy[m] | sSkipReboot: 'false', forceSkipReboot 'false'. | 14:23 |
| effendy[m] | Do not trigger reboot if cloud-init is executing. | 14:23 |
| effendy[m] | Command to exec : '/usr/bin/cloud-init'. | 14:23 |
| effendy[m] | It then acknowledges a few times that cloud-init is running until cloud-init stops. Unbelievable :) | 14:24 |
| falcojr | re: netplan. https://github.com/canonical/cloud-init/pull/1794 , if you're on 23.1 or greater, you shouldn't be seeing it. If so, that might be a bug | 14:24 |
| -ubottu:#cloud-init- Pull 1794 in canonical/cloud-init "Netplan deprecated keys (SC-1050)" [Merged] | 14:24 | |
| effendy[m] | Timed out waiting for cloud-init execution done. | 14:24 |
| effendy[m] | Ran DeployPkg_DeployPackageFromFile successfully | 14:24 |
| effendy[m] | then it says 'Trigger reboot', 'rebooting' etc. | 14:25 |
| falcojr | not super knowledgeable about VMWare, but I'm assuming that log is from VMWare tooling? | 14:27 |
| effendy[m] | Yeah, it seems to be related to the vm tools/guest agent/whatever. | 14:28 |
| effendy[m] | Now I'm trying to figure out how to stop the madness :D | 14:28 |
| falcojr | nice...at least it gets you going in the right direction | 14:30 |
| effendy[m] | Yeah, well... the documentation related to this seems non-existent. But at least, yes, I'm looking in the right direction. | 14:32 |
| effendy[m] | Thanks for the discussion. It did help, despite my being (understandably) upset :D | 14:32 |
| minimal | https://kb.vmware.com/s/article/90331 | 14:42 |
| minimal | "Linux virtual machine is usually rebooted at the end of the Perl based customization. If cloud-init is still running by the time of rebooting happens, it will be terminated and fail to apply user data to virtual machine. To avoid such scenario, from VMware Tools version 12.1.5, the Perl based customization will not reboot virtual machine until it detects cloud-init execution has finished or fails to finish before timeout, this is to make sure | 14:42 |
| minimal | cloud-init can load and apply user data completely without interrupted by rebooting triggered by Perl based customization." | 14:42 |
| effendy[m] | Yes, I've just come across this too. I'm trying to see if disable_vmware_customization directive in cloud.cfg works :) | 14:55 |
| effendy[m] | What is the difference between cloud-init GOS and cloud-init GOSC? | 15:35 |
| effendy[m] | disable_vmware_customization set to false means GOS, true means GOSC it seems. | 15:35 |
| effendy[m] | I have a different issue now which doesn't make a lot of sense to me. This works sometimes, but other times it doesn't. I stop the ssh server during the packer template once (because I'm fixing the reboot problem for the second stage). I'm using bootcmd with the following command: - [ cloud-init-per, once, systemctl, stop, ssh ] | 17:22 |
| effendy[m] | However cloud-init complains about not being able to run bootcmd module returning 127. | 17:22 |
| effendy[m] | cloud-init-per doesn't seem to be available at that stage or something to that effect, I'm not sure. | 17:22 |
| effendy[m] | Ah, I think cloud-init-per is not the issue, but 'systemctl' itself might not be available at that stage. | 17:25 |
| effendy[m] | (although sometimes it is, which is just great :) ) | 17:26 |
| minimal | effendy[m]: wouldn't it be the other way around? disable_vmware_customization=false would mean GOSC (i.e. *do* customisation)? | 18:01 |
| minimal | as you are NOT disabling the customisation | 18:01 |
| effendy[m] | This is Photon OS, vmware's operating system. | 18:41 |
| effendy[m] | well, linux distribution. | 18:41 |
| effendy[m] | To be honest, I still don't really understand what the setting is doing :D | 18:42 |
| effendy[m] | Both provisioning with cloud-init and with vmtools perl script guest os customisation (or whatever is called) presuppose this to be set to 'false' :) | 18:43 |
| minimal | yes, disable=flase means enabled, which matches with the quote I posted earlier regarding Perl based customisation | 18:46 |
| effendy[m] | Means 'enabled' for what? | 18:52 |
| effendy[m] | I see that the default is 'true', according to cloud-init documentation related to OVF: https://cloudinit.readthedocs.io/en/22.1_a/topics/datasources/ovf.html | 18:53 |
| effendy[m] | I guess I don't really understand what 'vmware_customization' actually means. | 18:53 |
| minimal | means vmware customisation is enabled, whatever that may be | 18:54 |
| minimal | that's a VMware matter though | 18:54 |
| minimal | btw are you aware that there is also a VMware DataSource for cloud-init, separate from the OVF one? | 18:55 |
| minimal | and that, briefly, mentions this setting | 18:55 |
| effendy[m] | Yes. I tried that one. In the vmware documentation (https://kb.vmware.com/s/article/59557) it says that I should be using VMware instead of OVF for the cloud-init version higher than 23.1 (which is my case). When I tried it I there was nothing in user-data. On the other hand, the network settings (which I'm guessing belongs to meta-data) did work. The netplan settings were there (although they didn't override the previous settings for | 18:58 |
| effendy[m] | some reason, so I also had the template's IP somehow) | 18:58 |
| minimal | VMware themselves have done a lot of the work on the cloud-init VMware datasource | 19:01 |
| effendy[m] | I used the VMware datasource when I installed the operating system from scratch (starting with grub, autoinstall/subiquity, whatever). And that worked ok. It's just that it took way too long. The longest part was the compulsory updating of the security packages. | 19:02 |
| effendy[m] | Then I switched to OVF, because VMware didn't work with the OVA/OVF cloudimage/templates. | 19:03 |
| effendy[m] | I simply adapted the datasource to my needs, it wasn't a choice for me. | 19:03 |
| minimal | did you see this commit? https://github.com/canonical/cloud-init/commit/f5431e50a3b29db0ee044e7fa8a6a279d6cb14f7 | 19:03 |
| -ubottu:#cloud-init- Commit f5431e5 in canonical/cloud-init "VMware: Move Guest Customization transport from OVF to VMware (#1573)" | 19:03 | |
| effendy[m] | (I mean switching to a different datasource wasn't, switching the cloudimages, of course, was) | 19:04 |
| effendy[m] | No, I didn't. I wonder if this applies to a newer version of vSphere also. I in their documentation that if you wanted to switch to 'pure' cloud-init, you'd need vSphere version 7.0 Update 3 (or something like that), which we don't have yet unfortunately. Maybe an upgrading would be beneficial. | 19:06 |
| effendy[m] | does 'cloud-init clean' reset /etc/cloud/cloud.cfg settings? | 19:07 |
| effendy[m] | I've just realised that I'm running this after I add the disable_vmware_customization directive to cloud.cfg. | 19:07 |
| minimal | that commit was made by VMware in November last year which was 1 day *after* cloud-init 22.4 was released and so the next release was 23.1 (which matches with that VMware doc saying to use VMware DS with c-i >= 23.1) | 19:08 |
| minimal | cloud.cfg is created from cloud.cfg.tmpl but that only happens when cloud-init is packaged (by distro) | 19:09 |
| minimal | it is a "build time" thing, not a run-time thing | 19:09 |
| effendy[m] | I'm not challenging the fact VMware should be used as a datasource, but I'm not sure how I'm supposed to solve this issue, given that, in my case, only OVF seems to be population user-data. With VMware there's nothing in user-data. | 19:10 |
| minimal | sounds like something to take up with VMware as they author both VMware/VSphere and the cloud-init VMware datasource | 19:11 |
| minimal | pr raise a cloud-init Issue specific to the VMware datasource and "ping" one of the VMware guys in the Issue | 19:11 |
| minimal | s/pr/or/ | 19:11 |
| effendy[m] | Yeah, I've already written on their forum in relation to how to disable that reboot. Maybe someone answers there :) | 19:12 |
| minimal | try my 2nd suggestion? | 19:12 |
| effendy[m] | I've written them today. I'll wait a while. I'm not sure how I can convince anyone from cloudinit at this stage in regards to this issue, given that I'm not using the latest version of vSphere and... now I'd really understand how this would be particular to my setup. | 19:13 |
| effendy[m] | Ah, ok, I see what you mean. | 19:13 |
| effendy[m] | Yeah, maybe I'll do that too. I'll wait a little bit, I'll test things out and see what will happen. | 19:14 |
| minimal | if you look at the history of the c-i VMware datasource you'll see the Github names of several Vmware staff who work on the VMware datasource | 19:14 |
| minimal | many of the cloud-init datasources are written by the particular hypervisor/cloud vendors themselves rather than by Canonical or 3rd party individuals | 19:15 |
| effendy[m] | Yes, I might do that :) | 19:33 |
| minimal | effendy[m]: looking at the commit history for c-i's OVF datasource many of the changes there also come from VMware people | 19:34 |
| effendy[m] | When you say 'c-i', what you are refering to? | 19:34 |
| minimal | so as those are VMware staff who develop cloud-init's OVF and VMware datasources I'd expect those would be *exactly* the people to discuss your issues with | 19:35 |
| minimal | sorry, c-i is a abbreviation for cloud-init | 19:35 |
| effendy[m] | ah, ok :) | 19:35 |
| effendy[m] | Yes, it makes sense, of course. | 19:35 |
| effendy[m] | Thanks for pointing this out. | 19:36 |
| minimal | I never understand the whole VMware ESX/ESXi/VSphere/Fusion product line the OVF vs VMware c-i datasources, was confusing where trying to script OS image creation without actually having any VMware stuff hands-on to figure stuff out lol | 19:38 |
| minimal | s/where/when/ | 19:38 |
| minimal | effendy[m]: BTW the AltCloud also mentions its use for Vsphere.... | 19:59 |
| minimal | s/AltCloud/cloud-init AltCloud datasource/ | 19:59 |
| minimal | so that's a 3rd option :-) | 20:00 |
| effendy[m] | I don't like vSphere that much. I use what there is. I'd prefer proxmox to be honest, although their API is even worse when it comes to cloud-init, which you have to already have at the ready somehow as a snippet (so copying it from ssh etc. It doesn't work over the API). | 20:14 |
| effendy[m] | I feel I'm getting a little bit insane :) It seems that when creating the VM with terraform, cloud-init now is running correctly, it's not rebooting anymore, but the network interface is disconnected (doesn't start powered on). | 20:16 |
| effendy[m] | Probably related to the way the template is created because of that disable customization crap, who knows what stupid metadata is being misinterpreted?! crap :) | 20:16 |
| effendy[m] | I've now return to the initial setup (without disabling vm customization set to false) and I'm now brutally stopping and then starting again the vmware guest agent in the cloud-init script (which I was already running anyway). And I'll try to live for a while like that. It actually seems to be working. The solution horrifies me, but until I get more information, I'll make do. | 23:07 |
| effendy[m] | returned* | 23:07 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!