[18:03] any idea why an iso that would boot just fine locally (in virtualbox) would not boot over the network? I get an error about root device not being found and get dropped into initramfs [18:08] drab: sharing it via NFS ? [18:17] TJ-: no, I'm loading the whole iso via memdisk, boot over http, and I think I figured out the problem [18:17] I removed quiet splash and I can see the error /init: line 7: can't open /dev/sr0: no medium found [18:17] which makes sense, when I test with virtualbox it's mounted as the cdrom [18:17] when it's pulled in via network it's not... and I've no idea where it is [18:18] I wonder why other isos work, like UBCD [18:18] but not ubuntu liveCD [18:19] right. I recall a similar issue about 10 years ago whereby on certain Dell PowerEdge servers with SCSI CD-ROM, the installer would boot and then fail trying to mount /dev/sr0 - in that case the initrd didn't contain the correct scsi drivers [18:24] not sure it's a driver issue, I think it's a path issue, like I said it works just fine with UBCD or even systemrescueCD [18:24] so the approach to use memdisk + iso raw works if the iso is setup in a certain way, just not sure what way that is [18:24] yeah, I was just pointing out another failure mode that might be related [18:24] fair enough, will keep that in mind, thanks [18:24] could it be the path on the kernel command-line? [18:25] if you're dropped to shell in the initrd, you can examine the state of the system [18:25] yeah I'm trying to, cant figure out how, initramfs has no fdisk [18:25] not sure how to list the available drives and stuff [18:25] "cat /proc/partitions" [18:26] also /dev/block/ [18:26] oh, right, forgot about /proc, will try that, rebooting into it, was just testing UBCD again [18:27] I'm trying to figure out why your error report is /init line 7 ... that isn't from the standard initramfs-tools/init [18:27] which ISO image is it so I can check? [18:29] it's the ubuntu desktop 64bit, I then mounted the squash file and removed all bunch of porgram to slim it down, but didn't touch any of the init/kernel stuff so that should be intact [18:29] 16.04 [18:34] OK, 16.04 (the original) or 16.04.1, .2, .3 ? [18:34] .3 [18:36] I see running casper-premount script, then the flood begins [18:37] stdin: not a typewriter [18:37] /init: line 7: can't open /dev/sr0: No medium found [18:37] repeats that about a hundred times and then drops me in initramfs saying "unable to find a medium cotaining a live file system" [18:40] which looking at /proc/partitions I guess makes sense, I don't see any place where the iso has been mounted in memory, but maybe I'm misunderstnding how memdisk works [18:40] http://www.syslinux.org/wiki/index.php?title=MEMDISK [18:40] drab: OK, it's not about the /init! just extracted it and it is the /usr/share/initramfs-tools/init file. Line 7 is "[ -d /dev ] || mkdir -m 0755 /dev" ... so it must be referring to some file being sourced or called [18:40] the thing tho, ubcd and systemrescuecd work, so somehow it's possible to boot the iso just fine over http [18:44] right, but need to know which bit of code is failing so as to know what to give as parameters to fix it [18:51] what's the actual kernel command line? (cat /proc/cmdline) ... does it include boot=caper [18:52] TJ-: I think I'm gonna try a diff route, with squashfs booting, because this way even if I get it to work I'm still ending up with a much larger image than I need and a two stages boot [18:52] TJ-: yeah it does [18:52] boot=cascpe initrd=/casper/initrd.lz --- priority=low [18:52] casper* [18:52] that's the whole cmdline [18:52] drab: ok, so we know ./scripts/casper should be executed [18:53] the only thing 'cd' related was line 7 of that file: "scripts/casper:7:mountpoint=/cdrom " [18:53] but that wouldn't cause the line 7 error you reported [18:53] yeah that's probably it, since it's trying to mount /dev/sr0, the cdroom, to access the install medium [18:53] why not? [18:53] it seems to make sense to me [18:53] because it's only a variable assignment, not a command [18:53] initrd has booted and is trying to mount the installation medium on /cdrom [18:53] oh, I see [18:54] can't open /dev/sr0: no medium found suggests a "mount" command [18:54] right [19:13] one thing I just noticed in ./scripts/casper::mountroot() is it writes to "./casper.log" ... might be something useful in there [19:24] Got it! [19:26] TJ-: ? [19:26] in the meantime I failed badly at squashfs :( [19:26] I gues sI don't understand the boot process enough [19:26] and this seems mighty hard, much harder than I thought it'd be at least [19:27] in theory it seems pretty damn simple... fetch a file containing a root filesystem and mount it in ram, point init to that as the root device [19:27] but that seems asking for too much [19:27] at most I got an installer session going with a debootstrapped squashfs [19:28] but I can't get it to boot into it as if it was a normal root [19:28] ./scripts/casper::mountroot() calls ./scripts/casper::find_livefs() which iterates over /sys/block/ then calls ./scripts/casper-helpers::get_fstype() which calls ./bin/fstype (from the klibc-utils package (/usr/lib/klibc/bin/fstype on an installed system), which returns "No medium found" - try it locally with an empty CD/DVD drive and you get: "/usr/lib/klibc/bin/fstype /dev/sr0" >> "/dev/sr0: No [19:28] medium found" [19:28] wow that's cool that you figured it out [19:28] :) [19:29] I'm still stuck with an unbootable system tho :( [19:29] so, the upshot is, this is a warning as such not an error :) [19:29] heh [19:29] as in, it would be OK for CD-ROM drive to be empty (might be more than one in the system) [19:29] makes sense [19:30] however, if that fs_type binary returns non-zero as well as reporting it, the scripts may exit if they're running with "set -e" (means shell will exit on any error status returns) Not sure ./init is doing that but it's worth considering [19:31] so if you want to retry this method and it drops into the initrd shell again, the first thing I'd do would be to "cat ./casper.log" in case there's clues there [19:31] I tried that, didn't see anything interesting [19:32] were there messages in it? those can be valuable since tey indicate HOW FAR the execution got, and from which it is possible to figure out where the scripts died [19:32] there's a fingerprint like line numbers [19:32] s/there's/they're/ [19:32] nope, just the repetition of the stuff I typed above, but will look again [19:33] the thing is, looking at /proc/partitions, I don't know there the root OS is [19:33] I thought the syslinux memdisk would have loaded the iso in ram [19:33] and the root be available at something like /dev/ram0 [19:33] but I didn't see anything like that [19:33] it's in casper/filesystem.squashfs [20:40] I give up... don't get why it's so damn hard, all the tech is there, ramdisks, fetching squashfs, mounting, root pivot [20:40] but I've tried about 2 dozen combinations and scripts and always end up with some problem or another [20:40] what are you doing? local boot, or network? [20:41] network boot from iso or squashfs, not using nfsroot [20:41] PXE then? [20:41] so I can get a rescue system going to fix systems on the lan [20:41] yeah, PXE [20:41] I already pxe install stuff all the time, so the sever/dhcp and so on is in good shape [20:41] and like I said systemrescueCD or UBCD pxe boot from the iso work [20:42] but I want ubuntu/apt [20:42] for some reason there's no ubuntu based rescue CD, don't know why [20:43] that's because the LiveCd is used for such things [20:44] I found this in one of my last attempts, but it also fails: https://www.slax.org/en/ [20:44] in a very similar way... can't mount the root fs and drops me into a initramfs shell [20:45] it's even got a slack ipxe fetching an iso from http/the internet, so you'd think that'd work, but fails in the same way [20:46] have you considered it's your network? [20:47] I have not, if it was why would: 1) pxe installs normally work 2) pxe boot of UBCD and systemrescueCD work ? [20:47] isn't that enough to rule out it's my network? [20:47] I can even pxe boot a squash filesystem I custom made [20:47] the only problem is that it boots into the installer [20:47] instead of booting a usable system [20:47] we had a very similar issue in #maas where the network seemed to be dropping packets [20:48] how is the rootfs being provided? what protocol? [20:48] http [20:48] from local server, same pxe is running on [20:49] you're using PXE > memdisk > ISO ? [20:49] that was the hope, yes [20:50] what's the PXE/TFTP config for the Ubuntu menu entry? [20:50] and if I use nfsroot that also works, booting the liveCD "Try without installing" [20:50] sec [20:51] http://dpaste.com/305NG8W [20:51] the iso is a straightdownload from ubuntu.com [21:06] Question: for the working ISO boots, are they all contained within their initrd.img files? no additional rootfs to be fetched? [21:07] The thing is memdisk uses x86 real-mode interrupts, and loads the image into RAM. But once Linux starts it switches to protected mode which no longer has access to the real-mode access. There are workarounds involving finding the ISO image in RAM and loading a kernel module that maps/mounts it as a memory block device [21:08] TJ-: http://dpaste.com/0C7EP8J it's all the same for all ISOs [21:08] that's a straight iso download from http://www.ultimatebootcd.com/ [21:10] TJ-: I also found this, but could not get it to work either: https://github.com/medallia/ramroot [21:10] it sounded exactly like what I wanted (and motivated by the same reason - avoiding nfs) [21:33] the reason UBCD works is it has a self-contained squashfs inside the initrd.img [21:44] fair enough, I guess it makes not much of a diff to me, maybe I'll look how to back the squashfs into the initrd [21:45] rootram tho looks lik eit's doing the right thing, ie what would logically make sense to me [21:45] create a ramdisk and wget the root fs and untar it to it [21:45] however I'm stuck on "tar: applet not found" [21:45] https://github.com/medallia/ramroot/blob/master/initramfs-tools/scripts/ramdisk#L46 [21:45] which seems to be happenig there [21:45] no clue why busybox tar would fail like that... [21:51] not sure why we can't just add a simple http client fetch inside initrd for these boots. it'd be relatively small and simple [21:59] I wonder... there's tricks with bash that could do that :) [22:04] drab: you've given me a challenge now. the ubuntu intrd has ./bin/wget provided by busybox so it's possible to fetch the ISO into a tmpfs, loop mount it, and mount the squashfs [22:11] TJ-: dude if you got that working it'd be the best early xmas present anybody ever game me :P [22:11] just trying to think if it could be done without needing special arguments on the command line :D [22:12] in *theory* it could be the last-ditch fallback if the other methods fail, before the final drop to shell [22:13] another random question as I'm trying to make ramroot work at the same time... if possible [22:13] I found out that for whatever fun reason the busybox shipped in ubuntu does not come with tar... hence the fail [22:14] I apt-get'ed the source for busybox-initramfs [22:14] but it's not obvious to me wwhere the applets are enabled (normally it seems you'd run make menuconfig and pick) [22:16] drab: did you try telling memdisk it's a "hardisk" image not "iso" - after all, the ISOs are hybrid [22:16] TJ-: yes, I saw that and tried that too, made no diff [22:16] OK.. it got into the initrd though? [22:16] because indeed fdisk -l showed it was a disk like in the example they gave [22:16] ? [22:16] lost you there, what got into the initrd? [22:17] I mean it booted as far as the initrd then failed the same way it does with 'iso' method - dropping to shell [22:17] yeah [22:17] right, that really does confirm the problem there is the switch from real-mode to protected 'loses' access to the im-memory image [22:25] That ramroot project is OK, but misses the point. It could easily work with the existing ISO/squashfs images rather than enforcing having to repack it as tar.xz [22:26] TJ-: I can't get that to work either... recompiling busybox now, seems a rather wacky choice to ship busybox without tar... [22:26] I don't even know why/how it worked for him to begin with [22:26] maybe it used to ship it in 14.04 [22:27] we want it small as possible, and don't need tar in the initrd [22:28] yeah, I guessed as much, which makes sense [22:29] just been a long saturday... and thank you for helping, really appreciate it [22:31] I did all this stuff years ago; got scripts that automatically configure the PXE server for ISOs so it's simple to deploy, but I use NFS. [22:32] in case there's any nuggets in it, this is it from 10 years ago: http://tjworld.net/wiki/Linux/Ubuntu/NetbootPxeLiveCDMultipleReleases [22:33] TJ-: yeah, I have that working (with nfs), the whole problem started when trying to do this over http [22:33] nfs based stuff is no issue, works pretty well [22:33] yes, I understand [22:34] the end of the article deals with patching the ISO initrd [22:34] thanks, checking it out [22:35] back then the network modules weren't built-in to the kernel nor included in the initrd, so I had to modify it for getting netconsole [22:35] bah, got past the tar issue but now getting a whole new bunch of errors I can't even see because it scrolls too fast and then I can't page up, it's all locked up [22:35] failures during extraction maybe? [22:35] one per file in the image maybe? [22:36] you repacked the rootfs as a tar.xz I assume? [22:36] yeah [22:36] can't see where it's erroring first, scrolls real fast and dies, I see errors can't create /root/lib/firmware, can't mount /dev on /root/dev and eventually tar fs doesn't have /sbin/init [22:37] so it does sound like untarring an setting up failed somehow [22:37] I wonder why it all locks up, normally the initramfs prompt is usable and I can at least scroll up [22:58] did you increase the memtotal size to account for the size of your rootfs when uncompressed? [23:02] TJ-: for rootmem? no I didn't think of it, it wasn't in the instructions and I thoguht tmpfs was dynamic these days (and would have worst errored if no more space was not available) [23:03] but I have 4Gbs and the img is 100MB so I thought it wasn't a problem [23:07] look at the ramroot script, line 29. it defaults to 500MB which must take the uncompressed image. if not you have to set memtotal= on tha kernel command-line [23:22] TJ-: the unextracted/before tar ramroot is 400MB, but I will try [23:23] also I realized that earlier I tarred chroot, ie the tar would have chroot/{etc,var...} and maybe that's also no good [23:23] so retarring with the tar straight into the root [23:24] haha yes that'd be a problem [23:24] tar needs to start in the root of the rootfs :) [23:27] holy cow [23:27] there's good news and bad news [23:28] the good news, it booted \o/ [23:28] lol [23:28] the bad news, I guess it's missing some usb module because the usb keyword plugged doesn't work [23:28] check for usb_hid I think it is (human interface device) [23:29] check where? [23:29] I'd thought everything like that was included in the stock kernel I installed in the chroot (and loaded), but maybe not [23:29] it should be in the kernel image itself [23:30] i've had that previously where it needed some specific USB chipset driver to bring up the USB side [23:31] check "find /sys/bus/usb/ " see what's there [23:31] cant' type so can't check :) [23:31] but I'm looking in the chroot [23:32] to see if the module is there, I used linux-virtual-image which installs a minimal kernel [23:32] and doing /lib/modules/4.4.0-21-generic/kernel# sudo find . -name "*usb*" returns nothing [23:32] so they may not be there after all [23:33] yeah on my desktop that returns a lot of stuff, including usbhid.ko [23:33] trying again adding a diff kernel in the chroot [23:35] right; virtual-image doesn't need USB [23:35] it's gonna make the img fat tho... [23:35] * drab scratches his head [23:36] also seems I've recompiled busybox I'm thinking I could also make the root tar bz2 instead of xz [23:36] should make it much smaller [23:36] if you want to boot bare hardware it's better to have it builtin - in case the initrd fails :) [23:36] you could, but ramroot only supports xz as far as I noticed [23:36] yeah, well, in theory I guess I should just compile a static kernel [23:37] TJ-: I think it's just the code, to make it more portable as most busyboxes prolly aren't compiled with bzip2 [23:37] but maybe have xz, so I can prolly just change that part of the code to uncompress the tar [23:38] it's using pixz for decompressing; you'll need to add code to ramroot script if you're going to use bz2 [23:38] yeah [23:50] that's about 200MB more of modules... kinda whack, but let's see if it works [23:51] I guess if it does then I'll spend time compiling a kernel statically and adding a whole bunch of stuff, but avoiding modules I don't really need, like sound [23:52] comes to 180Mb vs 100MB it wa before [23:54] yep, bingo! got keyboard and even a nice framebuffer and it was up and running pretty quickly overall [23:58] :) well done