[18:03] <drab> any idea why an iso that would boot just fine locally (in virtualbox) would not boot over the network? I get an error about root device not being found and get dropped into initramfs
[18:08] <TJ-> drab: sharing it via NFS ?
[18:17] <drab> TJ-: no, I'm loading the whole iso via memdisk, boot over http, and I think I figured out the problem
[18:17] <drab> I removed quiet splash and I can see the error /init: line 7: can't open /dev/sr0: no medium found
[18:17] <drab> which makes sense, when I test with virtualbox it's mounted as the cdrom
[18:17] <drab> when it's pulled in via network it's not... and I've no idea where it is
[18:18] <drab> I wonder why other isos work, like UBCD
[18:18] <drab> but not ubuntu liveCD
[18:19] <TJ-> right. I recall a similar issue about 10 years ago whereby on certain Dell PowerEdge servers with SCSI CD-ROM, the installer would boot and then fail trying to mount /dev/sr0 - in that case the initrd didn't contain the correct scsi drivers
[18:24] <drab> not sure it's a driver issue, I think it's a path issue, like I said it works just fine with UBCD or even systemrescueCD
[18:24] <drab> so the approach to use memdisk + iso raw works if the iso is setup in a certain way, just not sure what way that is
[18:24] <TJ-> yeah, I was just pointing out another failure mode that might be related
[18:24] <drab> fair enough, will keep that in mind, thanks
[18:24] <TJ-> could it be the path on the kernel command-line?
[18:25] <TJ-> if you're dropped to shell in the initrd, you can examine the state of the system
[18:25] <drab> yeah I'm trying to, cant figure out how, initramfs has no fdisk
[18:25] <drab> not sure how to list the available drives and stuff
[18:25] <TJ-> "cat /proc/partitions"
[18:26] <TJ-> also /dev/block/
[18:26] <drab> oh, right, forgot about /proc, will try that, rebooting into it, was just testing UBCD again
[18:27] <TJ-> I'm trying to figure out why your error report is /init line 7 ... that isn't from the standard initramfs-tools/init
[18:27] <TJ-> which ISO image is it so I can check?
[18:29] <drab> it's the ubuntu desktop 64bit, I then mounted the squash file and removed all bunch of porgram to slim it down, but didn't touch any of the init/kernel stuff so that should be intact
[18:29] <drab> 16.04
[18:34] <TJ-> OK, 16.04 (the original) or 16.04.1, .2, .3 ?
[18:34] <drab> .3
[18:36] <drab> I see running casper-premount script, then the flood begins
[18:37] <drab> stdin: not a typewriter
[18:37] <drab>  /init: line 7: can't open /dev/sr0: No medium found
[18:37] <drab> repeats that about a hundred times and then drops me in initramfs saying "unable to find a medium cotaining a live file system"
[18:40] <drab> which looking at /proc/partitions I guess makes sense, I don't see any place where the iso has been mounted in memory, but maybe I'm misunderstnding how memdisk works
[18:40] <drab> http://www.syslinux.org/wiki/index.php?title=MEMDISK
[18:40] <TJ-> drab: OK, it's not about the /init! just extracted it and it is the /usr/share/initramfs-tools/init file. Line 7 is "[ -d /dev ] || mkdir -m 0755 /dev" ... so it must be referring to some file being sourced or called
[18:40] <drab> the thing tho, ubcd and systemrescuecd work, so somehow it's possible to boot the iso just fine over http
[18:44] <TJ-> right, but need to know which bit of code is failing so as to know what to give as parameters to fix it
[18:51] <TJ-> what's the actual kernel command line? (cat /proc/cmdline) ... does it include boot=caper
[18:52] <drab> TJ-: I think I'm gonna try a diff route, with squashfs booting, because this way even if I get it to work I'm still ending up with a much larger image than I need and a two stages boot
[18:52] <drab> TJ-: yeah it does
[18:52] <drab> boot=cascpe initrd=/casper/initrd.lz --- priority=low
[18:52] <drab> casper*
[18:52] <drab> that's the whole cmdline
[18:52] <TJ-> drab: ok, so we know ./scripts/casper should be executed
[18:53] <TJ-> the only thing 'cd' related was line 7 of that file: "scripts/casper:7:mountpoint=/cdrom "
[18:53] <TJ-> but that wouldn't cause the line 7 error you reported
[18:53] <drab> yeah that's probably it, since it's trying to mount /dev/sr0, the cdroom, to access the install medium
[18:53] <drab> why not?
[18:53] <drab> it seems to make sense to me
[18:53] <TJ-> because it's only a variable assignment, not a command
[18:53] <drab> initrd has booted and is trying to mount the installation  medium on /cdrom
[18:53] <drab> oh, I see
[18:54] <TJ-> can't open /dev/sr0: no medium found suggests a "mount" command
[18:54] <drab> right
[19:13] <TJ-> one thing I just noticed in ./scripts/casper::mountroot() is it writes to "./casper.log" ... might be something useful in there
[19:24] <TJ-> Got it!
[19:26] <drab> TJ-: ?
[19:26] <drab> in the meantime I failed badly at squashfs :(
[19:26] <drab> I gues sI don't understand the boot process enough
[19:26] <drab> and this seems mighty hard, much harder than I thought it'd be at least
[19:27] <drab> in theory it seems pretty damn simple... fetch a file containing a root filesystem and mount it in ram, point init to that as the root device
[19:27] <drab> but that seems asking for too much
[19:27] <drab> at most I got an installer session going with a debootstrapped squashfs
[19:28] <drab> but I can't get it to boot into it as if it was a normal root
[19:28] <TJ-> ./scripts/casper::mountroot() calls ./scripts/casper::find_livefs() which iterates over /sys/block/ then calls ./scripts/casper-helpers::get_fstype() which calls ./bin/fstype (from the klibc-utils package (/usr/lib/klibc/bin/fstype on an installed system), which returns "No medium found" - try it locally with an empty CD/DVD drive and you get: "/usr/lib/klibc/bin/fstype /dev/sr0" >> "/dev/sr0: No
[19:28] <TJ-> medium found"
[19:28] <drab> wow that's cool that you figured it out
[19:28] <TJ-> :)
[19:29] <drab> I'm still stuck with an unbootable system tho :(
[19:29] <TJ-> so, the upshot is, this is a warning as such not an error :)
[19:29] <drab> heh
[19:29] <TJ-> as in, it would be OK for CD-ROM drive to be empty (might be more than one in the system)
[19:29] <drab> makes sense
[19:30] <TJ-> however, if that fs_type binary returns non-zero as well as reporting it, the scripts may exit if they're running with "set -e" (means shell will exit on any error status returns) Not sure ./init is doing that but it's worth considering
[19:31] <TJ-> so if you want to retry this method and it drops into the initrd shell again, the first thing I'd do would be to "cat ./casper.log" in case there's clues there
[19:31] <drab> I tried that, didn't see anything interesting
[19:32] <TJ-> were there messages in it? those can be valuable since tey indicate HOW FAR the execution got, and from which it is possible to figure out where the scripts died
[19:32] <TJ-> there's a fingerprint like line numbers
[19:32] <TJ-> s/there's/they're/
[19:32] <drab> nope, just the repetition of the stuff I typed above, but will look again
[19:33] <drab> the thing is, looking at /proc/partitions, I don't know there the root OS is
[19:33] <drab> I thought the syslinux memdisk would have loaded the iso in ram
[19:33] <drab> and the root be available at something like /dev/ram0
[19:33] <drab> but I didn't see anything like that
[19:33] <TJ-> it's in casper/filesystem.squashfs
[20:40] <drab> I give up... don't get why it's so damn hard, all the tech is there, ramdisks, fetching squashfs, mounting, root pivot
[20:40] <drab> but I've tried about 2 dozen combinations and scripts and always end up with some problem or another
[20:40] <TJ-> what are you doing? local boot, or network?
[20:41] <drab> network boot from iso or squashfs, not using nfsroot
[20:41] <TJ-> PXE then?
[20:41] <drab> so I can get a rescue system going to fix systems on the lan
[20:41] <drab> yeah, PXE
[20:41] <drab> I already pxe install stuff all the time, so the sever/dhcp and so on is in good shape
[20:41] <drab> and like I said systemrescueCD or UBCD pxe boot from the iso work
[20:42] <drab> but I want ubuntu/apt
[20:42] <drab> for some reason there's no ubuntu based rescue CD, don't know why
[20:43] <TJ-> that's because the LiveCd is used for such things
[20:44] <drab> I found this in one of my last attempts, but it also fails: https://www.slax.org/en/
[20:44] <drab> in a very similar way... can't mount the root fs and drops me into a initramfs shell
[20:45] <drab> it's even got a slack ipxe fetching an iso from http/the internet, so you'd think that'd work, but fails in the same way
[20:46] <TJ-> have you considered it's your network?
[20:47] <drab> I have not, if it was why would: 1) pxe installs normally work 2) pxe boot of UBCD and systemrescueCD work ?
[20:47] <drab> isn't that enough to rule out it's my network?
[20:47] <drab> I can even pxe boot a squash filesystem I custom made
[20:47] <drab> the only problem is that it boots into the installer
[20:47] <drab> instead of booting a usable system
[20:47] <TJ-> we had a very similar issue in #maas where the network seemed to be dropping packets
[20:48] <TJ-> how is the rootfs being provided? what protocol?
[20:48] <drab> http
[20:48] <drab> from local server, same pxe is running on
[20:49] <TJ-> you're using PXE > memdisk > ISO ?
[20:49] <drab> that was the hope, yes
[20:50] <TJ-> what's the PXE/TFTP config for the Ubuntu menu entry?
[20:50] <drab> and if I use nfsroot that also works, booting the liveCD "Try without installing"
[20:50] <drab> sec
[20:51] <drab> http://dpaste.com/305NG8W
[20:51] <drab> the iso is a straightdownload from ubuntu.com
[21:06] <TJ-> Question: for the working ISO boots, are they all contained within their initrd.img files? no additional rootfs to be fetched?
[21:07] <TJ-> The thing is memdisk uses x86 real-mode interrupts, and loads the image into RAM. But once Linux starts it switches to protected mode which no longer has access to the real-mode access. There are workarounds involving finding the ISO image in RAM and loading a kernel module that maps/mounts it as a memory block device
[21:08] <drab> TJ-: http://dpaste.com/0C7EP8J it's all the same for all ISOs
[21:08] <drab> that's a straight iso download from http://www.ultimatebootcd.com/
[21:10] <drab> TJ-: I also found this, but could not get it to work either: https://github.com/medallia/ramroot
[21:10] <drab> it sounded exactly like what I wanted (and motivated by the same reason - avoiding nfs)
[21:33] <TJ-> the reason UBCD works is it has a self-contained squashfs inside the initrd.img
[21:44] <drab> fair enough, I guess it makes not much of a diff to me, maybe I'll look how to back the squashfs into the initrd
[21:45] <drab> rootram tho looks lik eit's doing the right thing, ie what would logically make sense to me
[21:45] <drab> create a ramdisk and wget the root fs and untar it to it
[21:45] <drab> however I'm stuck on "tar: applet not found"
[21:45] <drab> https://github.com/medallia/ramroot/blob/master/initramfs-tools/scripts/ramdisk#L46
[21:45] <drab> which seems to be happenig there
[21:45] <drab> no clue why busybox tar would fail like that...
[21:51] <TJ-> not sure why we can't just add a simple http client fetch inside initrd for these boots. it'd be relatively small and simple
[21:59] <TJ-> I wonder... there's tricks with bash that could do that :)
[22:04] <TJ-> drab: you've given me a challenge now. the ubuntu intrd has ./bin/wget provided by busybox so it's possible to fetch the ISO into a tmpfs, loop mount it, and mount the squashfs
[22:11] <drab> TJ-: dude if you got that working it'd be the best early xmas present anybody ever game me :P
[22:11] <TJ-> just trying to think if it could be done without needing special arguments on the command line :D
[22:12] <TJ-> in *theory* it could be the last-ditch fallback if the other methods fail, before the final drop to shell
[22:13] <drab> another random question as I'm trying to make ramroot work at the same time... if possible
[22:13] <drab> I found out that for whatever fun reason the busybox shipped in ubuntu does not come with tar... hence the fail
[22:14] <drab> I apt-get'ed the source for busybox-initramfs
[22:14] <drab> but it's not obvious to me wwhere the applets are enabled (normally it seems you'd run make menuconfig and pick)
[22:16] <TJ-> drab: did you try telling memdisk it's a "hardisk" image not "iso" - after all, the ISOs are hybrid
[22:16] <drab> TJ-: yes, I saw that and tried that too, made no diff
[22:16] <TJ-> OK.. it got into the initrd though?
[22:16] <drab> because indeed fdisk -l showed it was a disk like in the example they gave
[22:16] <drab> ?
[22:16] <drab> lost you there, what got into the initrd?
[22:17] <TJ-> I mean it booted as far as the initrd then failed the same way it does with 'iso' method - dropping to shell
[22:17] <drab> yeah
[22:17] <TJ-> right, that really does confirm the problem there is the switch from real-mode to protected 'loses' access to the im-memory image
[22:25] <TJ-> That ramroot project is OK, but misses the point. It could easily work with the existing ISO/squashfs images rather than enforcing having to repack it as tar.xz
[22:26] <drab> TJ-: I can't get that to work either... recompiling busybox now, seems a rather wacky choice to ship busybox without tar...
[22:26] <drab> I don't even know why/how it worked for him to begin with
[22:26] <drab> maybe it used to ship it in 14.04
[22:27] <TJ-> we want it small as possible, and don't need tar in the initrd
[22:28] <drab> yeah, I guessed as much, which makes sense
[22:29] <drab> just been a long saturday... and thank you for helping, really appreciate it
[22:31] <TJ-> I did all this stuff years ago; got scripts that automatically configure the PXE server for ISOs so it's simple to deploy, but I use NFS.
[22:32] <TJ-> in case there's any nuggets in it, this is it from 10 years ago: http://tjworld.net/wiki/Linux/Ubuntu/NetbootPxeLiveCDMultipleReleases
[22:33] <drab> TJ-: yeah, I have that working (with nfs), the whole problem started when trying to do this over http
[22:33] <drab> nfs based stuff is no issue, works pretty well
[22:33] <TJ-> yes, I understand
[22:34] <TJ-> the end of the article deals with patching the ISO initrd
[22:34] <drab> thanks, checking it out
[22:35] <TJ-> back then the network modules weren't built-in to the kernel nor included in the initrd, so I had to modify it for getting netconsole
[22:35] <drab> bah, got past the tar issue but now getting a whole new bunch of errors I can't even see because it scrolls too fast and then I can't page up, it's all locked up
[22:35] <TJ-> failures during extraction maybe?
[22:35] <TJ-> one per file in the image maybe?
[22:36] <TJ-> you repacked the rootfs as a tar.xz I assume?
[22:36] <drab> yeah
[22:36] <drab> can't see where it's erroring first, scrolls real fast and dies, I see errors can't create /root/lib/firmware, can't mount /dev on /root/dev and eventually tar fs doesn't have /sbin/init
[22:37] <drab> so it does sound like untarring an setting up failed somehow
[22:37] <drab> I wonder why it all locks up, normally the initramfs prompt is usable and I can at least scroll up
[22:58] <TJ-> did you increase the memtotal size to account for the size of your rootfs when uncompressed?
[23:02] <drab> TJ-: for rootmem? no I didn't think of it, it wasn't in the instructions and I thoguht tmpfs was dynamic these days (and would have worst errored if no more space was not available)
[23:03] <drab> but I have 4Gbs and the img is 100MB so I thought it wasn't a problem
[23:07] <TJ-> look at the ramroot script, line 29. it defaults to 500MB which must take the uncompressed image. if not you have to set memtotal= on tha kernel command-line
[23:22] <drab> TJ-: the unextracted/before tar ramroot is 400MB, but I will try
[23:23] <drab> also I realized that earlier I tarred chroot, ie the tar would have chroot/{etc,var...} and maybe that's also no good
[23:23] <drab> so retarring with the tar straight into the root
[23:24] <TJ-> haha yes that'd be a problem
[23:24] <TJ-> tar needs to start in the root of the rootfs :)
[23:27] <drab> holy cow
[23:27] <drab> there's good news and bad news
[23:28] <drab> the good news, it booted \o/
[23:28] <TJ-> lol
[23:28] <drab> the bad news, I guess it's missing some usb module because the usb keyword plugged doesn't work
[23:28] <TJ-> check for usb_hid I think it is (human interface device)
[23:29] <drab> check where?
[23:29] <drab> I'd thought everything like that was included in the stock kernel I installed in the chroot (and loaded), but maybe not
[23:29] <TJ-> it should be in the kernel image itself
[23:30] <TJ-> i've had that previously where it needed some specific USB chipset driver to bring up the USB side
[23:31] <TJ-> check "find /sys/bus/usb/ " see what's there
[23:31] <drab> cant' type so can't check :)
[23:31] <drab> but I'm looking in the chroot
[23:32] <drab> to see if the module is there, I used linux-virtual-image which installs a minimal kernel
[23:32] <drab> and doing /lib/modules/4.4.0-21-generic/kernel# sudo find . -name "*usb*" returns nothing
[23:32] <drab> so they may not be there after all
[23:33] <drab> yeah on my desktop that returns a lot of stuff, including usbhid.ko
[23:33] <drab> trying again adding a diff kernel in the chroot
[23:35] <TJ-> right; virtual-image doesn't need USB
[23:35] <drab> it's gonna make the img fat tho...
[23:35]  * drab scratches his head
[23:36] <drab> also seems I've recompiled busybox I'm thinking I could also make the root tar bz2 instead of xz
[23:36] <drab> should make it much smaller
[23:36] <TJ-> if you want to boot bare hardware it's better to have it builtin - in case the initrd fails :)
[23:36] <TJ-> you could, but ramroot only supports xz as far as I noticed
[23:36] <drab> yeah, well, in theory I guess I should just compile a static kernel
[23:37] <drab> TJ-: I think it's just the code, to make it more portable as most busyboxes prolly aren't compiled with bzip2
[23:37] <drab> but maybe have xz, so I can prolly just change that part of the code to uncompress the tar
[23:38] <TJ-> it's using pixz for decompressing; you'll need to add code to ramroot script if you're going to use bz2
[23:38] <drab> yeah
[23:50] <drab> that's about 200MB more of modules... kinda whack, but let's see if it works
[23:51] <drab> I guess if it does then I'll spend time compiling a kernel statically and adding a whole bunch of stuff, but avoiding modules I don't really need, like sound
[23:52] <drab> comes to 180Mb vs 100MB it wa before
[23:54] <drab> yep, bingo! got keyboard and even a nice framebuffer and it was up and running pretty quickly overall
[23:58] <TJ-> :) well done