=== shokohsc510 is now known as shokohsc51
[17:08] <bahamat> Maybe I'll have better luck here...
[17:09] <bahamat> I'm trying to debug unattended install with ubuntu-server. I've got `ds=nocloud-net;seedfrom=http://10.0.0.1:8080/ubuntu/22.04/` on the kernel cmdline, but the cloud-init log says there was no valid source there.
[17:09] <bahamat> I've confirmed from my web server log that it *is* getting the correct file.
[17:13] <minimal> bahamat: what else does the cloud-init.log say?
[17:14] <bahamat> minimal: Seed from http://10.0.0.1:8080/ubuntu/22.04/ not supported by DataSourceNoCloud [seed=None][dsmode=net]
[17:14] <bahamat> In /var/lib/cloud/seed/nocloud, both the meta-data and user-data file are empty
[17:15] <bahamat> But in /var/lib/cloud/instances/nocloud, both cloud-config.txt and user-data.txt have my correct config in it
[17:16] <bahamat> (cloud-config.txt looks like it's rearranged all the keys and changed the whitespace formatting, but otherwise it looks fine, I think)
[17:16] <dbungert> bahamat: a common problem with people doing autoinstall with cloud-init providing the data is that the autoinstall keys have to be under the top level key `autoinstall` - example here https://ubuntu.com/server/docs/install/autoinstall-quickstart
[17:16] <dbungert> s/providing/helping to carry/
[17:16] <bahamat> If I do `sudo cloud-init schema --system`, it says `Config file /var/lib/cloud/instances/iid-datasource-none/cloud-config.txt does not exist`
[17:17] <bahamat> But if I symlink iid-datasource-none to nocloud, and run it again then it says `user-data at /var/lib/cloud/instances/iid-datasource-none/cloud-config.txt: Valid cloud-config: user-data`
[17:18] <bahamat> I do have everything under autoinstall
[17:19] <bahamat> user-data file is here: https://github.com/TritonDataCenter/triton-cloud-images/blob/main/http/ubuntu/22.04/user-data
[17:20] <bahamat> Full kernel cmdline is: autoinstall console=tty0 console=ttyS0,115200n8 ds=nocloud-net;seedfrom=http://10.0.0.1:8080/ubuntu/22.04/ tsc=reliable
[17:20] <dbungert> ok, I haven't looked too close, but the user-data file looks correct enough to start.  I think you need to debug the cloud-init side of this.
[17:21] <minimal> bahamat: your meta-data file is empty
[17:22] <minimal> i.e. no instance-id specified
[17:23] <bahamat> Well, this worked literally two weeks ago with it empty.
[17:23] <dbungert> that's ok in this context, to have an empty meta-data file.  I test like that routine 
[17:23] <bahamat> So unless the installer was swapped out in the past two weeks and that's suddenly become a strict requirement, then I don't think that's the issue.
[17:23] <minimal> cloud-init generally makes use of instance-id so don't know what happens if you do not specify one
[17:24] <bahamat> My ubuntu-20.04 build still works, and that's here: https://github.com/TritonDataCenter/triton-cloud-images/blob/main/http/ubuntu/20.04/user-data
[17:24] <bahamat> It also has an empty meta-data file.
[17:25] <dbungert> bahamat: I suggest comparing the cloud-init logs of your working and failing cases
[17:25] <minimal> I'm not saying that is the problem, I'm merely pointing out that I've never seen cloud-init used without an instance-id being specified and that cloud-init makes use of the instance-id for various purposes
[17:26] <bahamat> minimal: ok, for the moment, let's assume it's not. dbungert indicates that it's not an issue, in my experience it hasn't been an issue.
[17:29] <minimal> bahamat: so is debugging enabled for cloud-init?
[17:29] <bahamat> I assume not, since I didn't explicitly enable it.
[17:29] <bahamat> How would I go about doing that?
[17:30] <bahamat> cloud-init.log has this at the top:
[17:30] <bahamat> 2023-07-25 17:12:34,370 - main.py[DEBUG]: No kernel command line url found.
[17:30] <bahamat> So...maybe something in my cmdline is fucked up?
[17:30] <bahamat> in /proc/cmdline it has: BOOT_IMAGE=/casper/vmlinuz --- autoinstall console=tty0 console=ttyS0,115200n8 ds=nocloud-net;seedfrom=http://10.0.0.1:8080/ubuntu/22.04/ tsc=reliable
[17:31] <bahamat> And I *can* curl `http://10.0.0.1:8080/ubuntu/22.04/user-data`, and it *is* the data I want.
[17:32] <minimal> bahamat: so it does appear to debug enabled. So what about the log lines around the "not supported by DataSourceNoCloud"? can you pastebin the logs somewhere?
[17:33] <bahamat> https://pastebin.com/dkqYyz2Q
[17:39] <minimal> yeah that line is misleading, it did load from the seed urls
[17:41] <bahamat> Yeah, reading much further down it does seem like it's fetching it, but then it's just not working
[17:42] <minimal> basically that was a failure for DataSourceNoCloud to handle seed url - as it does not handle seed urls, later DataSourceNoCloudNet succeeded, as it *does* handle see durls
[17:42] <dbungert> bahamat: can you also pastebin /var/log/installer/subiquity-server-debug.log?
[17:44] <bahamat> dbungert: https://pastebin.com/an2p5znz
[17:45] <minimal> I don't see any issues in the cloud-init.log
[17:46] <dbungert> bahamat: when you say you're testing 22.04, what iso specifically are you testing?  I think you're hitting some bugs since Ubuntu 22.04.2 was released. 
[17:46] <minimal> well aprt from this which may or may not be "normal" for Ubuntu:
[17:47] <minimal> 2023-07-25 17:12:36,168 - activators.py[WARNING]: Running ['netplan', 'apply'] resulted in stderr output: Failed to connect system bus: No such file or directory
[17:47] <minimal> Falling back to a hard restart of systemd-networkd.service
[17:47] <bahamat> So, background here, I'm trying to create cloud-origin images.
[17:47] <bahamat> I'm using a combination of packer+ansible, and I used this to publish images on July 11.
[17:48] <bahamat> But, I found something I need to change so now I'm running it again, and it's failing.
[17:48] <bahamat> bugs in the installer sound like it may very well be what's going on.
[17:48] <dbungert> bahamat: there are fixes in flight for the problem you're hitting on both cloud-init side and the subiquity side
[17:49] <dbungert> https://github.com/canonical/subiquity/pull/1735 needs to land + a compatible cloud-init version
[17:49] -ubottu:#ubuntu-server- Pull 1735 in canonical/subiquity "Obtain autoinstall from combined-cloud-config" [Open]
[17:49] <bahamat> OK, so maybe I can just force the 24.04.0 or 24.04.1 iso for now?
[17:49] <minimal> I see an IscDhclient error in the subquity log
[17:50] <dbungert> minimal: yea, things are hosed, pickle errors.  cloud-init is kindly providing the data subiquity wants in a flat file to address that pickle error.
[17:51] <minimal> its c-i 23.2.1 and there were changes made to c-i around that time to support other DHCP clients
[17:52] <dbungert> bahamat: well, I think you mean 22.04, but yes forcing an older ISO is probably prudent.  I expect this to all be resolved by the time 22.04.3 is released.
[17:52] <bahamat> Oh, yes. 22.04
[17:52] <dbungert> no 24.04 yet ;)
[17:55] <dbungert> bahamat: another option, if you're building install media anyhow, is that you can put autoinstall data directly on the install media in a file named /autoinstall.yaml.  That is not affected by the above bug.  Remove the top level autoinstall key if doing so.
[17:55] <bahamat> I'm not building installer media
[18:02] <bahamat> Hard coding the ISO from jammy/daily-live/20230518 works
[18:03] <bahamat> Thanks for the help folks. It's good to know that I'm not completely insane and that it's not a hallucination that this *did* work two weeks ago.
[18:04] <dbungert> ah, you're using dailies, that's how you ran into this.  also hard-coding the 22.04.2 iso should probably be ok.
=== shokohsc516 is now known as shokohsc51