| === shokohsc510 is now known as shokohsc51 | ||
| bahamat | Maybe I'll have better luck here... | 17:08 |
|---|---|---|
| bahamat | I'm trying to debug unattended install with ubuntu-server. I've got `ds=nocloud-net;seedfrom=http://10.0.0.1:8080/ubuntu/22.04/` on the kernel cmdline, but the cloud-init log says there was no valid source there. | 17:09 |
| bahamat | I've confirmed from my web server log that it *is* getting the correct file. | 17:09 |
| minimal | bahamat: what else does the cloud-init.log say? | 17:13 |
| bahamat | minimal: Seed from http://10.0.0.1:8080/ubuntu/22.04/ not supported by DataSourceNoCloud [seed=None][dsmode=net] | 17:14 |
| bahamat | In /var/lib/cloud/seed/nocloud, both the meta-data and user-data file are empty | 17:14 |
| bahamat | But in /var/lib/cloud/instances/nocloud, both cloud-config.txt and user-data.txt have my correct config in it | 17:15 |
| bahamat | (cloud-config.txt looks like it's rearranged all the keys and changed the whitespace formatting, but otherwise it looks fine, I think) | 17:16 |
| dbungert | bahamat: a common problem with people doing autoinstall with cloud-init providing the data is that the autoinstall keys have to be under the top level key `autoinstall` - example here https://ubuntu.com/server/docs/install/autoinstall-quickstart | 17:16 |
| dbungert | s/providing/helping to carry/ | 17:16 |
| bahamat | If I do `sudo cloud-init schema --system`, it says `Config file /var/lib/cloud/instances/iid-datasource-none/cloud-config.txt does not exist` | 17:16 |
| bahamat | But if I symlink iid-datasource-none to nocloud, and run it again then it says `user-data at /var/lib/cloud/instances/iid-datasource-none/cloud-config.txt: Valid cloud-config: user-data` | 17:17 |
| bahamat | I do have everything under autoinstall | 17:18 |
| bahamat | user-data file is here: https://github.com/TritonDataCenter/triton-cloud-images/blob/main/http/ubuntu/22.04/user-data | 17:19 |
| bahamat | Full kernel cmdline is: autoinstall console=tty0 console=ttyS0,115200n8 ds=nocloud-net;seedfrom=http://10.0.0.1:8080/ubuntu/22.04/ tsc=reliable | 17:20 |
| dbungert | ok, I haven't looked too close, but the user-data file looks correct enough to start. I think you need to debug the cloud-init side of this. | 17:20 |
| minimal | bahamat: your meta-data file is empty | 17:21 |
| minimal | i.e. no instance-id specified | 17:22 |
| bahamat | Well, this worked literally two weeks ago with it empty. | 17:23 |
| dbungert | that's ok in this context, to have an empty meta-data file. I test like that routine | 17:23 |
| bahamat | So unless the installer was swapped out in the past two weeks and that's suddenly become a strict requirement, then I don't think that's the issue. | 17:23 |
| minimal | cloud-init generally makes use of instance-id so don't know what happens if you do not specify one | 17:23 |
| bahamat | My ubuntu-20.04 build still works, and that's here: https://github.com/TritonDataCenter/triton-cloud-images/blob/main/http/ubuntu/20.04/user-data | 17:24 |
| bahamat | It also has an empty meta-data file. | 17:24 |
| dbungert | bahamat: I suggest comparing the cloud-init logs of your working and failing cases | 17:25 |
| minimal | I'm not saying that is the problem, I'm merely pointing out that I've never seen cloud-init used without an instance-id being specified and that cloud-init makes use of the instance-id for various purposes | 17:25 |
| bahamat | minimal: ok, for the moment, let's assume it's not. dbungert indicates that it's not an issue, in my experience it hasn't been an issue. | 17:26 |
| minimal | bahamat: so is debugging enabled for cloud-init? | 17:29 |
| bahamat | I assume not, since I didn't explicitly enable it. | 17:29 |
| bahamat | How would I go about doing that? | 17:29 |
| bahamat | cloud-init.log has this at the top: | 17:30 |
| bahamat | 2023-07-25 17:12:34,370 - main.py[DEBUG]: No kernel command line url found. | 17:30 |
| bahamat | So...maybe something in my cmdline is fucked up? | 17:30 |
| bahamat | in /proc/cmdline it has: BOOT_IMAGE=/casper/vmlinuz --- autoinstall console=tty0 console=ttyS0,115200n8 ds=nocloud-net;seedfrom=http://10.0.0.1:8080/ubuntu/22.04/ tsc=reliable | 17:30 |
| bahamat | And I *can* curl `http://10.0.0.1:8080/ubuntu/22.04/user-data`, and it *is* the data I want. | 17:31 |
| minimal | bahamat: so it does appear to debug enabled. So what about the log lines around the "not supported by DataSourceNoCloud"? can you pastebin the logs somewhere? | 17:32 |
| bahamat | https://pastebin.com/dkqYyz2Q | 17:33 |
| minimal | yeah that line is misleading, it did load from the seed urls | 17:39 |
| bahamat | Yeah, reading much further down it does seem like it's fetching it, but then it's just not working | 17:41 |
| minimal | basically that was a failure for DataSourceNoCloud to handle seed url - as it does not handle seed urls, later DataSourceNoCloudNet succeeded, as it *does* handle see durls | 17:42 |
| dbungert | bahamat: can you also pastebin /var/log/installer/subiquity-server-debug.log? | 17:42 |
| bahamat | dbungert: https://pastebin.com/an2p5znz | 17:44 |
| minimal | I don't see any issues in the cloud-init.log | 17:45 |
| dbungert | bahamat: when you say you're testing 22.04, what iso specifically are you testing? I think you're hitting some bugs since Ubuntu 22.04.2 was released. | 17:46 |
| minimal | well aprt from this which may or may not be "normal" for Ubuntu: | 17:46 |
| minimal | 2023-07-25 17:12:36,168 - activators.py[WARNING]: Running ['netplan', 'apply'] resulted in stderr output: Failed to connect system bus: No such file or directory | 17:47 |
| minimal | Falling back to a hard restart of systemd-networkd.service | 17:47 |
| bahamat | So, background here, I'm trying to create cloud-origin images. | 17:47 |
| bahamat | I'm using a combination of packer+ansible, and I used this to publish images on July 11. | 17:47 |
| bahamat | But, I found something I need to change so now I'm running it again, and it's failing. | 17:48 |
| bahamat | bugs in the installer sound like it may very well be what's going on. | 17:48 |
| dbungert | bahamat: there are fixes in flight for the problem you're hitting on both cloud-init side and the subiquity side | 17:48 |
| dbungert | https://github.com/canonical/subiquity/pull/1735 needs to land + a compatible cloud-init version | 17:49 |
| -ubottu:#ubuntu-server- Pull 1735 in canonical/subiquity "Obtain autoinstall from combined-cloud-config" [Open] | 17:49 | |
| bahamat | OK, so maybe I can just force the 24.04.0 or 24.04.1 iso for now? | 17:49 |
| minimal | I see an IscDhclient error in the subquity log | 17:49 |
| dbungert | minimal: yea, things are hosed, pickle errors. cloud-init is kindly providing the data subiquity wants in a flat file to address that pickle error. | 17:50 |
| minimal | its c-i 23.2.1 and there were changes made to c-i around that time to support other DHCP clients | 17:51 |
| dbungert | bahamat: well, I think you mean 22.04, but yes forcing an older ISO is probably prudent. I expect this to all be resolved by the time 22.04.3 is released. | 17:52 |
| bahamat | Oh, yes. 22.04 | 17:52 |
| dbungert | no 24.04 yet ;) | 17:52 |
| dbungert | bahamat: another option, if you're building install media anyhow, is that you can put autoinstall data directly on the install media in a file named /autoinstall.yaml. That is not affected by the above bug. Remove the top level autoinstall key if doing so. | 17:55 |
| bahamat | I'm not building installer media | 17:55 |
| bahamat | Hard coding the ISO from jammy/daily-live/20230518 works | 18:02 |
| bahamat | Thanks for the help folks. It's good to know that I'm not completely insane and that it's not a hallucination that this *did* work two weeks ago. | 18:03 |
| dbungert | ah, you're using dailies, that's how you ran into this. also hard-coding the 22.04.2 iso should probably be ok. | 18:04 |
| === shokohsc516 is now known as shokohsc51 | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!