[08:07] Hi [08:07] https://bugs.launchpad.net/subiquity/+bug/1905296 [08:07] Launchpad bug 1905296 in subiquity "autoinstall crash with python3[1872]: segfault" [Undecided,New] [08:07] i found in my crash file that after installer ran, it mounnt-bind /etc/apt/source.list to /target/apt/source.list [08:08] problem is that i use an iso, uncompressed and served as nfsroot [08:08] so i get -> Failed to fetch file:/cdrom/dists/focal/main/binary-amd64/Packages [08:09] is there a way, maybe with a kernel paramater, to tell the iso to only use distant repositories ? [08:10] eoli3n_: usually /cdrom is a bind mount to the ISO9660; presumably you can adjust that for the unpacked scenario [08:11] to ways, first is to use an early_command in cloud-config to remove the cdrom source line [08:11] or early command to bind mount /cdrom to "i need to find where is mounted nfsroot" [08:12] but ubuntu-server iso should handle that case, and provide an "netboot=yes" option to cmdline [08:13] but but but its all 'cloud' nowadays ... which ironically doesn't play well for netboot! [08:15] TJ- or maybe i misuse something [08:15] beeing able to nfsroot insteaf of loading a 900MB iso should be handle [08:15] d [08:15] the first problem is that 900MB iso [08:15] no netboot UEFI iso anymore [08:15] Legacy one is 60MB [08:16] https://discourse.ubuntu.com/t/netbooting-the-live-server-installer/14510 [08:16] eoli3n_: don't need to tell me! I dispensed with the installer about 8 years ago [08:17] so how do you do ? [08:17] you bootstrap manually ? [08:17] with a personnal script ? [08:19] eoli3n_: we use debootstrap to create the base images and chroot to prepare them, then we use per-device bind-mounts for things like /etc/hostname /etc/hosts and /etc/machine-id [08:20] eoli3n_: those are served via PXE/NFS to the workstations [08:28] any resource to share ? [08:29] maybe debootstrap is the solution yes [08:29] in legacy, i deploy with netboot, then ansible installs everything as i want it too [08:29] in pull mode [08:33] TJ- https://x0.at/ioy.png [08:33] strange [08:33] so this should work [08:34] eoli3n_: there's nothing to share; deboostrap creates the base rootfs install (has options to add additional packages etc.) - you can use it to build foreign architecture (e.g. aarch64 on amd64) in conjunction with qemu-user-static too (q-u-s allows you to chroot into the foreign arch and work in it natively) [08:36] eoli3n_: your image suggests it should have worked assuming the command to access /cdrom/... was AFTER the mount was put in place [08:37] i don't get what you mean [08:37] here you can see that cdrom is actually working [08:37] https://x0.at/iaa.txt [08:38] file for exemple release.gpg worked [08:38] but not Packages [08:42] i reproduced, installer just crashed, then i checked: [08:43] the file /cdrom/dists/focal/main/binary-amd64/Packages doesnt exists [08:43] the file /cdrom/dists/focal/main/binary-amd64/Packages.gz does [08:45] eoli3n_: that does look weird ... apt should look for a .gz and fall back to plain uncompressed [08:46] eoli3n_: check what the 'Releases' file lists - does it contain the Packages.gz and Packages [08:47] eoli3n_: this is what the file in the archives looks like for reference: http://archive.ubuntu.com/ubuntu/dists/focal/Release [08:47] http://sprunge.us/Lshlss [08:47] oh ok [08:48] not that file [08:48] http://sprunge.us/6ry6y2 [08:48] so yes, it should fallback [08:49] no [08:49] it should find the first one which is .fgz [08:49] .gz [08:53] eoli3n_: trying to decompose your shell log, notice how early on after overlay-mounting /target/etc/apt/ it then does "mv /target/etc/apt/sources.list /target/etc/apt/sources.list.d/original.list" followed by "apt update" in separate namespace (using 'unshare') [08:53] eoli3n_: I wonder if this is a clue: "start: cmd-install/stage-curthooks/001-configure-apt/cmd-in-target: curtin command in-target " [08:53] yep but in any way, during the apt update process, some files are found, some other are not. [08:53] lets try to check that file [08:53] eoli3n_: what does " /target/etc/apt/sources.list.d/original.list" contain? [08:53] lets check [08:54] * TJ- loves debugging [08:55] http://sprunge.us/OpHYUC [08:56] eoli3n_: those messages about Packages 18.7KB must refer to the Release file "... 18748 main/binary-amd64/Packages.gz" ( Get:8 file:/cdrom focal/main amd64 Packages [18.7 kB] ) [08:56] but /target/etc/apt/source.list contains only the cdrom line [08:56] what do you mean ? [08:56] you want me to check Package.gz checksum ? [08:57] eoli3n_: those messages above tell us that apt was trying to read Packages.gz (since it is the 18.7KB file) [08:57] eoli3n_: no, you'd know if the hash didn't match - this is "file not found" [08:57] eoli3n_: show us the cdrom line from sources.list [08:58] /target/etc/apt/source.list : http://sprunge.us/ZSPK8j [08:59] du should output in Byte by default right ? [09:00] oh no ok -b [09:00] so yes Packages.gz is 18748 bytes [09:01] eoli3n_: on the NFS server, ensure that the entire path can be traversed and files read by a client [09:01] it does, because i'm using it right now [09:01] for all those tests [09:02] https://github.com/eoli3n/vagrant-pxe/blob/dev/roles/nfs/files/exports [09:02] eoli3n_: how about by the uid that runs the installer? notice it uses 'unshare' -- maybe that breaks something [09:02] unshare ? [09:02] what is that [09:03] just checked, subiquity runs as root [09:03] ohh [09:04] maybe a root squash or no root squash [09:04] i never remember which one maps "root" perms [09:04] it's a namespace tool - notice it is forking into a separate PID namespace -- and NFS client usually needs idmap and so on which'll remain in the parent PID namespace if my thinking is correct [09:04] see "man 7 pid_namespaces" [09:05] "no_root_squash" which i use, gives root perms on nfs server, so no problem with this [09:06] i'm reading this [09:06] but how to test if unshare messed up something [09:06] diff source and dest ? [09:06] eoli3n_: is /target/ still mounted? [09:06] it does [09:07] installer crashed but it is still running, so that behaviour let me check the exact state of the problem [09:07] eoli3n_: if so you can use "unshare --fork --pid -- chroot /target /usr/bin/bash" to enter and check if all of the /cdrom/ mount is readable [09:07] lets try [09:07] eoli3n_: if we're correct you'll not be able to find that Packages/Packages.gz file [09:08] eoli3n_: or you'll see the directory entry but not be able to read the file itself [09:08] i do see the file [09:08] but bash shows it in red color [09:08] i don't know what it means [09:09] eoli3n_: just thinking - on the NFS server, is the entire ISO permanently extracted, or is the iso just mounted [09:09] extracted -> https://github.com/eoli3n/vagrant-pxe/blob/dev/roles/ubuntu/tasks/main.yml#L14 [09:09] i can't tar xvf Packages.gz [09:09] eoli3n_: the colour comes from the 'ls' command - can't recall what red means but do "ls -l path/to/file" to get its details [09:10] ls -l show nothing strange [09:10] tar: This does not look like a tar archive [09:10] eoli3n_: try "zcat Packages.gz" [09:10] zcat works [09:10] eoli3n_: ok, and this is from the 'unshare' chroot ? [09:10] it does [09:10] eoli3n_: hmmm! [09:11] try runinng "apt-update" to replicate the original issue [09:11] yep, it broke with more informations [09:11] if we can reproduce the error, we can add debugging options to apt to get it to report what it is doing [09:11] let me paste [09:11] eoli3n_: OK, and I'll look up the appropriate Debug:: option [09:12] eoli3n_: this: "apt -o Debug::Acquire::cdrom=true update" [09:13] http://sprunge.us/VuoYWd [09:13] this should be useful [09:13] eoli3n_: other Debug:: options can be found towards the end of "man apt.conf" [09:13] oh [09:13] sorry the paste miss some lines [09:14] https://x0.at/7EZ.png [09:14] huhu spaced paste [09:14] do you still want with debug ? [09:16] https://askubuntu.com/a/908825 [09:18] i do not have any /cdrom/dists/focal/InRelease file [09:18] even outside the unshare chroot [09:19] if i chroot without unshare, i get that same error [09:19] so unshare, is not the problem [09:20] ohhh [09:21] i know [09:21] the mess up comes from https://github.com/eoli3n/vagrant-pxe/blob/dev/roles/ubuntu/tasks/main.yml#L33 [09:21] i chown -R /var/www/focal to www-data [09:23] it should be related, lets try to fix this [09:24] eoli3n_: that'd do it, because NFS relies on matching uid/gids [09:25] ok [09:26] thanks for your help [09:27] eoli3n_: note that often the InRelease file isn't used; that is just a combination of the Release + Release.gpg files ('In' means the GPG signature is 'inline' ) [09:27] ok [09:27] eoli3n_: so that may not be the problem [09:27] hm [09:28] lets fix this anyway [09:28] and retry [09:28] i wonder if i need to serve only initrd and vmlinuz as /var/www/focal with www-data [09:29] and export nfsroot from another dir [09:29] InRelease will start with "-----BEGIN PGP SIGNED MESSAGE---- and the Hash: used [09:29] eoli3n_: check on the NFS server for logs indicating errors/denied reads [09:29] i destroyed my stakc [09:30] in a couple of minute if i can reproduce [09:31] eoli3n_: the weird part here is this: If the chroot /target/ manages to read the /cdrom/dists/focal/Release file then there is no reason it shouldn't also read the /cdrom/main/binary-amd64/Packages.gz [09:32] eoli3n_: *unless* it is trying to locate Packages.gz by its hash and the hash symlink doesn't exist! [09:33] eoli3n_: this might be it! [09:33] eoli3n_: does the exploded /cdrom/ have a ./by-hash/ directory alongside the ./main/ ? [09:34] i answer all of that just when my server is up [09:34] :) [09:34] this would make sense; did you do a test earlier using that -o Debug::Acquire:cdrom=true ? [09:42] lets try with this -> https://github.com/eoli3n/vagrant-pxe/commit/3fcc67db45fed8a233dbee6f67dbca5cccb3eb5b [09:42] its provisionning [10:00] ok client is booting [10:00] lets wait for it to crash (or not hehe) [10:04] now installers is looping [10:04] ... [10:04] i already have that problem [10:04] had [10:05] third loop, lets try to see if it crash at a point [10:06] (it loops without rebooting) [10:06] looping? you mean the installer restarts itself? [10:07] yes, with my space eyes, i just see that i crash at the same point than before [10:07] just after "configuring apt" [10:07] but i can't be sure [10:07] lets try to catch a screenshot [10:07] what a debugger... thanks ubuntu [10:08] eoli3n_: if the installer runs multiple times (loops) and in a later run it fails like this, then the cause could be multiple mounts-on-mounts if it didn't clean up before starting the next loop [10:08] i screenshot just before it restart : https://x0.at/COS.png [10:09] hm [10:09] lets check mounts on tty2 [10:10] http://sprunge.us/o3E8MV [10:10] have you searched subiquity for similar bug reports? [10:10] yep [10:11] i found some "subitquity loop" issues [10:11] but nothing relevant [10:11] https://bugs.launchpad.net/snappy/+bug/1617232 [10:11] Launchpad bug 1617232 in subiquity (Ubuntu) "subiquity goes into endless configuration loop, no way to get out" [High,Invalid] === denningsrogue2 is now known as denningsrogue [10:36] that's crazy [10:36] at each run i get a different error [10:36] eoli3n_: have you checked for network errors? [10:36] https://x0.at/Civ.png [10:36] network errors ? [10:37] VMs are hosted on the same hypervisor [10:37] apt update works outside the chroot [10:39] maybe you're right, at some boot, i get moutn error [10:39] https://x0.at/pUY.png [10:42] mounting from the server itself failed until i exportfs -r [10:42] strange [10:49] ok i found, i have two interface on the vm [10:51] if i drop the second on, its working [10:51] one [10:51] problem is that, that interface give gateway to internet [10:54] by its working i mean, no nfs connectivity problem [10:55] but then i enter the endless loop anyway [11:00] Hi guys, what do you recommend for me to use to establish a _reliable_ remote desktop connection with my server? (GUI) [11:02] RDP or VNC? and which software specifically [11:05] mrkewl20 consider spice too, it is used by qemu by default [11:05] https://www.spice-space.org/index.html [11:06] I used xrdp but I want to switch since I have screen tearing a lot and the connection between and my server is completely fine [11:07] https://www.spice-space.org/spice-user-manual.html [11:07] Actually spice sounds cool I might give it a try thank you for mentioning it [11:08] you're welcome [11:15] TJ- with debug acquire -> https://x0.at/zUR.png [11:17] no "by-hash" dir [11:18] only "main Release Release.gpg restricted" [11:52] https://x0.at/3Pi.txt [11:52] how to find what is this "cmd-install/stage-curthooks/001-configure-apt/cmd-apt-config" [11:53] because it test something which is true [11:53] then it move source.list to source.list.d/...orig [11:53] at this point /target/etc/apt/source.list contains cdrom [11:53] between that mv, and the crash [11:54] something put cdrom as main source [11:54] complete crash file -> http://sprunge.us/ToqZRL [12:20] eoli3n_: despite extensive searching I cannot locate the original source-code for that 001-configure-apt - hecked both the curtin and subiquity source trees [12:20] hm [12:20] thanks for you help TJ- [12:21] asked on #ubuntu-devel [12:21] looks like another bloody snap! running '/snap/bin/subiquity.subiquity-configure-apt /snap/subiquity/1966/usr/bin/python3 true' [12:21] waiting for an answer [12:21] yep [12:21] didn't realise I wasn't in that channel! [12:21] maybe lets discuss it there ? [12:22] generally the -server team work in this channel, although most are likely in UTC+5+ timezones [12:27] eoli3n_: best clue so far "subiquity-configure-apt" at https://git.launchpad.net/subiquity/tree/examples/snaps/v2-snaps-subiquity.json [12:27] hm [12:27] it doesnt tell so much things [12:29] eoli3n_: found that at https://git.launchpad.net/subiquity/tree/bin/subiquity-configure-apt [12:30] TJ- the funny thing is that i'm trying to load installer with iso file instead of nfsroot [12:30] eoli3n_: notice items 4a and 4b in the header comment [12:30] everything going well, then it stuck at loading system [12:30] without any warning error info [12:30] nothing [12:30] eoli3n_: AND item 5! [12:30] "if network not expected to be working ..." mentioned several times [12:31] yep [12:31] i have a strange setup [12:31] line 47 sets the overlay [12:31] but what is the network test [12:31] ? [12:31] line 55/56 suggests your install HAS_NETWORK==true [12:32] line 56 is what we see happening in your crash log [12:32] yep but how HAS_NETWORK is set ? [12:32] HAS_NETWORK=$2 [12:32] so we need to find how that script is called [12:32] right, and reading to end of this script, you can see line 63 onwards is reflected in your crash log [12:33] I wouldn't worry about how it is called; that all looks sensible now we've found it [12:33] yep it does [12:33] good catch [12:33] my problem is that my VM has two interfaces [12:33] last line being apt-get update means if there is a failure it is in this script, or prior to this script being called [12:33] the first one has a gateway to internet [12:34] the second one is isolated network to pxe server [12:34] if a server has two interfaces with two gateways set by dhcp [12:35] which one will work ? [12:35] the latest set ? [12:35] 'depends' on which has the route to the target [12:35] which what ? [12:35] in your case I'd assume since it is managing to work from the NFS mount that routing is fine [12:36] TJ- not really because i'm facing that issue at a toper level [12:36] its random [12:36] when i boot pxe my VM, then ip=dhcp casper arguments set two interfaces [12:36] then if i'm lucky it has the right default if [12:36] if i'm not... [12:36] i think the same thing happens at differents level [12:37] i have 3 dhcp queries. first one is from pxe client, then from booloader, then from client [12:38] my guess is that bootloader and client network setup sets a random default route [12:38] default gateway* [12:38] you can have only one default gateway [12:38] so i need to find how to force one [12:39] maybe i need to ensure that my pxe dhcp doesn't give any default gateway [12:40] i don't push any gateway https://github.com/eoli3n/vagrant-pxe/blob/dev/roles/dhcp/files/dhcpd.conf [12:40] lets test routes on the client [12:42] https://x0.at/7fl.png [12:42] when i fails to boot you can see that the wrong IF is set first [12:43] then it goes [12:43] without waiting for the second one [12:43] 192.168.122.0 is the wrong network [12:43] eoli3n_: there can be multiple default routes, with differing metrics [12:43] i need ens6 first [12:43] never seen that [12:44] when i force ens6 interface in cmdline with ip=ens6 it seems to work [12:44] for that part [12:45] so you need to set which interface in the initrd [12:45] then i need to debug the client [12:45] yep but ens6 isnt predectible isn't it ? [12:45] i'm trying to force an interface name with vagrant [12:46] ah !! [12:46] eoli3n_: how about setting BOOTIF= or better still hwaddr= [12:46] https://x0.at/UyR.png [12:47] ah no, its like it should be [12:47] default on network which gateway has internet connectivity [12:48] eoli3n_: on a regular Ubuntu install do : "cd /usr/share/initramfs-tools/" then "grep -rn BOOTIF ." and you'll see that setting that on kernel-command line to the MAC address of the interface should do what you want [12:48] TJ- what differs from casper's ip=$if ? [12:49] ip=$if does the trick [12:49] or maybe i'm super lucky, then i need to make some more tests [12:49] eoli3n_: look at the comment against configure_networking() in scripts/functions [12:49] eoli3n_: it is designed for your multi-interface use-case [12:50] in scripts/functions of ? [12:50] sorry i loose it :/ [12:50] eoli3n_: /usr/share/initramfs-tools/scripts/functions [12:50] oh ok [12:51] eoli3n_: the scripts inside the initrd are rooted from /usr/share/initramfs-tools - in there, the 'init' script is what is being executed when the system is booted [12:51] that's the script the kernel executes, and is responsible for getting the root-fs ready [12:52] but what i don't get is, to solve what do you want me to set it ? [12:52] because my VM has network connectivity, if i ping google.fr , it goes [12:52] so problem seems to be about the test [12:53] HAS_NETWORK one [12:53] do we agree ? [12:53] eoli3n_: you said a few moments ago the boot-time i/f was wrong; this BOOTIF= is the way to solve that issue [12:53] yep but i said also that ip=$if solved this too [12:54] eoli3n_: no - because HAS_NETWORK is only a boolean, and as we've seen it is set to true, so that is the correct behaviour [12:54] where do you saw that its set to true ? [12:54] i saw that its set to $2 [12:54] or maybe i missed something [12:54] lets recheck [12:55] TJ-: line 55/56 suggests your install HAS_NETWORK==true [12:56] ok i get it now [12:56] no network issue [12:56] but it configure cdrom source in any cases [12:56] so, no way to disable it [12:57] eoli3n_: it's still not clear what causes the 'file not found' error for Packages.gz when Release/Release.gpg were read a moment before [12:57] you're right [12:58] but it looks on nfsroot, so maybe its related to this [12:58] that's why i wanted to test by loading http the iso [12:58] but i get stuck, and i don't want to debug this [13:01] yeah. I'd run inotifywait on the NFS server and see what file accesses are attempted [13:01] that way you'd know if reads were being attempted to those files on the server, or not [13:03] (tried BOOTIF=ens6, do not did the trick) [13:03] hm [13:04] my english is from the futur [13:04] didn't the trick* ? [13:05] it sounds bad [13:05] eoli3n_: you need to set BOOTIF= [13:06] my macadress isn't predictible [13:06] as vagrant creates VM [13:06] maybe i could force the mac adress [13:06] hm [13:06] oh sheesh! [13:06] sheesh ? [13:06] yes; it is possible to set a fixed MAC address [13:06] what does that strange word means ?µ [13:06] eoli3n_: that's an expression of exasperation :D [13:07] huhu [13:07] ip=ens6 [13:07] it works well so lets go with that [13:07] Yep [13:09] the inotify test seems crazy [13:09] eoli3n_: I use that frequently to prove a certain file access takes place, and its type [13:09] eoli3n_: my approach is to prove each piece and not assume [13:10] hm [13:10] lets try to [13:10] never did that [13:10] moved my irssi to Ubuntu instead of CentOS now [13:11] https://x0.at/559.txt [13:11] waiting [13:12] catch ! [13:12] /exports/focal/dists/focal/main/binary-amd64/Packages.gz ACCESS [13:12] so yep it does access [13:16] lets try to export rw [13:16] i don't know why, but i don't have any other idea [13:20] obviously, it didn't changed anything [13:31] so at least we know Packages.gz is read AND apt-update in-target doesn't get to see it! [13:31] Hey guys [13:31] I have ubuntu server on my nuc [13:32] I wanna know of i can use its wifi here [13:32] How do i check [13:33] eoli3n_: hang on... ACCESS isn't the same as READ if I recall correctly; that suggests there IS a permissions issue [13:35] eoli3n_: ignore me - ACCESS does include a read [13:36] Guys [13:36] How do i check if ubuntu server has support for the wifi [13:36] technoob: it does [13:37] technoob: assuming the wifi hardware has a driver and the interface is present, you can configure it via either netplan or directly with systemnd-networkd [13:38] Can you teach me how [13:38] How do i activate the wifi [13:39] I also wanna know if it can be used as a hotspot [13:39] TJ-: [13:40] Tj? [13:40] Anyone? [13:42] !patience | technoob [13:42] technoob: Don't feel ignored and repeat your question quickly; if nobody knows your answer, nobody will answer you. While you wait, try searching https://help.ubuntu.com or https://ubuntuforums.org or https://askubuntu.com/ [13:43] technoob: yes, as a hotspot you'd use 'hostapd' [13:43] Can i ask how to do this [13:43] TJ-: [13:44] technoob: this is a few years old but should get you started: https://help.ubuntu.com/community/WifiDocs/WirelessAccessPoint [13:48] How do i check iff i have hostapd installed already [14:04] Is there a way to connect to internet by plugging my phone which has data to my device running ubuntu server === halvors1 is now known as halvors [15:00] I have two servers that seem to be getting the same ip via dhcp. I checked the netplan directory and one of the servers has something that looks like this https://imgur.com/a/W0jmVbd and all those files have dhcp set to yes. The other server just has the standard 00-installer-config.yaml also with dhcp set to yes. [15:00] Shouldnt each be getting its own ip. two vms with different mac addr too. [17:15] well thats annoying, looks like i needed `dhcp-identifier: mac` in my netplan yaml file. dunno why they decided to change the DUID method. === ijohnson is now known as ijohnson|lunch [20:21] hi guys, any suggestions on how to configure 4 interfaces with netplan? [20:22] probably match blocks with macs [20:23] not sure I follow [20:39] devster31: well, now that I re-read the examples, matching by mac address is probably better reserved for cases where matching by name doesn't work [20:39] there's a bunch of examples on https://netplan.io/examples/ [20:47]