| drab | any idea why an iso that would boot just fine locally (in virtualbox) would not boot over the network? I get an error about root device not being found and get dropped into initramfs | 18:03 |
|---|---|---|
| TJ- | drab: sharing it via NFS ? | 18:08 |
| drab | TJ-: no, I'm loading the whole iso via memdisk, boot over http, and I think I figured out the problem | 18:17 |
| drab | I removed quiet splash and I can see the error /init: line 7: can't open /dev/sr0: no medium found | 18:17 |
| drab | which makes sense, when I test with virtualbox it's mounted as the cdrom | 18:17 |
| drab | when it's pulled in via network it's not... and I've no idea where it is | 18:17 |
| drab | I wonder why other isos work, like UBCD | 18:18 |
| drab | but not ubuntu liveCD | 18:18 |
| TJ- | right. I recall a similar issue about 10 years ago whereby on certain Dell PowerEdge servers with SCSI CD-ROM, the installer would boot and then fail trying to mount /dev/sr0 - in that case the initrd didn't contain the correct scsi drivers | 18:19 |
| drab | not sure it's a driver issue, I think it's a path issue, like I said it works just fine with UBCD or even systemrescueCD | 18:24 |
| drab | so the approach to use memdisk + iso raw works if the iso is setup in a certain way, just not sure what way that is | 18:24 |
| TJ- | yeah, I was just pointing out another failure mode that might be related | 18:24 |
| drab | fair enough, will keep that in mind, thanks | 18:24 |
| TJ- | could it be the path on the kernel command-line? | 18:24 |
| TJ- | if you're dropped to shell in the initrd, you can examine the state of the system | 18:25 |
| drab | yeah I'm trying to, cant figure out how, initramfs has no fdisk | 18:25 |
| drab | not sure how to list the available drives and stuff | 18:25 |
| TJ- | "cat /proc/partitions" | 18:25 |
| TJ- | also /dev/block/ | 18:26 |
| drab | oh, right, forgot about /proc, will try that, rebooting into it, was just testing UBCD again | 18:26 |
| TJ- | I'm trying to figure out why your error report is /init line 7 ... that isn't from the standard initramfs-tools/init | 18:27 |
| TJ- | which ISO image is it so I can check? | 18:27 |
| drab | it's the ubuntu desktop 64bit, I then mounted the squash file and removed all bunch of porgram to slim it down, but didn't touch any of the init/kernel stuff so that should be intact | 18:29 |
| drab | 16.04 | 18:29 |
| TJ- | OK, 16.04 (the original) or 16.04.1, .2, .3 ? | 18:34 |
| drab | .3 | 18:34 |
| drab | I see running casper-premount script, then the flood begins | 18:36 |
| drab | stdin: not a typewriter | 18:37 |
| drab | /init: line 7: can't open /dev/sr0: No medium found | 18:37 |
| drab | repeats that about a hundred times and then drops me in initramfs saying "unable to find a medium cotaining a live file system" | 18:37 |
| drab | which looking at /proc/partitions I guess makes sense, I don't see any place where the iso has been mounted in memory, but maybe I'm misunderstnding how memdisk works | 18:40 |
| drab | http://www.syslinux.org/wiki/index.php?title=MEMDISK | 18:40 |
| TJ- | drab: OK, it's not about the /init! just extracted it and it is the /usr/share/initramfs-tools/init file. Line 7 is "[ -d /dev ] || mkdir -m 0755 /dev" ... so it must be referring to some file being sourced or called | 18:40 |
| drab | the thing tho, ubcd and systemrescuecd work, so somehow it's possible to boot the iso just fine over http | 18:40 |
| TJ- | right, but need to know which bit of code is failing so as to know what to give as parameters to fix it | 18:44 |
| TJ- | what's the actual kernel command line? (cat /proc/cmdline) ... does it include boot=caper | 18:51 |
| drab | TJ-: I think I'm gonna try a diff route, with squashfs booting, because this way even if I get it to work I'm still ending up with a much larger image than I need and a two stages boot | 18:52 |
| drab | TJ-: yeah it does | 18:52 |
| drab | boot=cascpe initrd=/casper/initrd.lz --- priority=low | 18:52 |
| drab | casper* | 18:52 |
| drab | that's the whole cmdline | 18:52 |
| TJ- | drab: ok, so we know ./scripts/casper should be executed | 18:52 |
| TJ- | the only thing 'cd' related was line 7 of that file: "scripts/casper:7:mountpoint=/cdrom " | 18:53 |
| TJ- | but that wouldn't cause the line 7 error you reported | 18:53 |
| drab | yeah that's probably it, since it's trying to mount /dev/sr0, the cdroom, to access the install medium | 18:53 |
| drab | why not? | 18:53 |
| drab | it seems to make sense to me | 18:53 |
| TJ- | because it's only a variable assignment, not a command | 18:53 |
| drab | initrd has booted and is trying to mount the installation medium on /cdrom | 18:53 |
| drab | oh, I see | 18:53 |
| TJ- | can't open /dev/sr0: no medium found suggests a "mount" command | 18:54 |
| drab | right | 18:54 |
| TJ- | one thing I just noticed in ./scripts/casper::mountroot() is it writes to "./casper.log" ... might be something useful in there | 19:13 |
| TJ- | Got it! | 19:24 |
| drab | TJ-: ? | 19:26 |
| drab | in the meantime I failed badly at squashfs :( | 19:26 |
| drab | I gues sI don't understand the boot process enough | 19:26 |
| drab | and this seems mighty hard, much harder than I thought it'd be at least | 19:26 |
| drab | in theory it seems pretty damn simple... fetch a file containing a root filesystem and mount it in ram, point init to that as the root device | 19:27 |
| drab | but that seems asking for too much | 19:27 |
| drab | at most I got an installer session going with a debootstrapped squashfs | 19:27 |
| drab | but I can't get it to boot into it as if it was a normal root | 19:28 |
| TJ- | ./scripts/casper::mountroot() calls ./scripts/casper::find_livefs() which iterates over /sys/block/ then calls ./scripts/casper-helpers::get_fstype() which calls ./bin/fstype (from the klibc-utils package (/usr/lib/klibc/bin/fstype on an installed system), which returns "No medium found" - try it locally with an empty CD/DVD drive and you get: "/usr/lib/klibc/bin/fstype /dev/sr0" >> "/dev/sr0: No | 19:28 |
| TJ- | medium found" | 19:28 |
| drab | wow that's cool that you figured it out | 19:28 |
| TJ- | :) | 19:28 |
| drab | I'm still stuck with an unbootable system tho :( | 19:29 |
| TJ- | so, the upshot is, this is a warning as such not an error :) | 19:29 |
| drab | heh | 19:29 |
| TJ- | as in, it would be OK for CD-ROM drive to be empty (might be more than one in the system) | 19:29 |
| drab | makes sense | 19:29 |
| TJ- | however, if that fs_type binary returns non-zero as well as reporting it, the scripts may exit if they're running with "set -e" (means shell will exit on any error status returns) Not sure ./init is doing that but it's worth considering | 19:30 |
| TJ- | so if you want to retry this method and it drops into the initrd shell again, the first thing I'd do would be to "cat ./casper.log" in case there's clues there | 19:31 |
| drab | I tried that, didn't see anything interesting | 19:31 |
| TJ- | were there messages in it? those can be valuable since tey indicate HOW FAR the execution got, and from which it is possible to figure out where the scripts died | 19:32 |
| TJ- | there's a fingerprint like line numbers | 19:32 |
| TJ- | s/there's/they're/ | 19:32 |
| drab | nope, just the repetition of the stuff I typed above, but will look again | 19:32 |
| drab | the thing is, looking at /proc/partitions, I don't know there the root OS is | 19:33 |
| drab | I thought the syslinux memdisk would have loaded the iso in ram | 19:33 |
| drab | and the root be available at something like /dev/ram0 | 19:33 |
| drab | but I didn't see anything like that | 19:33 |
| TJ- | it's in casper/filesystem.squashfs | 19:33 |
| drab | I give up... don't get why it's so damn hard, all the tech is there, ramdisks, fetching squashfs, mounting, root pivot | 20:40 |
| drab | but I've tried about 2 dozen combinations and scripts and always end up with some problem or another | 20:40 |
| TJ- | what are you doing? local boot, or network? | 20:40 |
| drab | network boot from iso or squashfs, not using nfsroot | 20:41 |
| TJ- | PXE then? | 20:41 |
| drab | so I can get a rescue system going to fix systems on the lan | 20:41 |
| drab | yeah, PXE | 20:41 |
| drab | I already pxe install stuff all the time, so the sever/dhcp and so on is in good shape | 20:41 |
| drab | and like I said systemrescueCD or UBCD pxe boot from the iso work | 20:41 |
| drab | but I want ubuntu/apt | 20:42 |
| drab | for some reason there's no ubuntu based rescue CD, don't know why | 20:42 |
| TJ- | that's because the LiveCd is used for such things | 20:43 |
| drab | I found this in one of my last attempts, but it also fails: https://www.slax.org/en/ | 20:44 |
| drab | in a very similar way... can't mount the root fs and drops me into a initramfs shell | 20:44 |
| drab | it's even got a slack ipxe fetching an iso from http/the internet, so you'd think that'd work, but fails in the same way | 20:45 |
| TJ- | have you considered it's your network? | 20:46 |
| drab | I have not, if it was why would: 1) pxe installs normally work 2) pxe boot of UBCD and systemrescueCD work ? | 20:47 |
| drab | isn't that enough to rule out it's my network? | 20:47 |
| drab | I can even pxe boot a squash filesystem I custom made | 20:47 |
| drab | the only problem is that it boots into the installer | 20:47 |
| drab | instead of booting a usable system | 20:47 |
| TJ- | we had a very similar issue in #maas where the network seemed to be dropping packets | 20:47 |
| TJ- | how is the rootfs being provided? what protocol? | 20:48 |
| drab | http | 20:48 |
| drab | from local server, same pxe is running on | 20:48 |
| TJ- | you're using PXE > memdisk > ISO ? | 20:49 |
| drab | that was the hope, yes | 20:49 |
| TJ- | what's the PXE/TFTP config for the Ubuntu menu entry? | 20:50 |
| drab | and if I use nfsroot that also works, booting the liveCD "Try without installing" | 20:50 |
| drab | sec | 20:50 |
| drab | http://dpaste.com/305NG8W | 20:51 |
| drab | the iso is a straightdownload from ubuntu.com | 20:51 |
| TJ- | Question: for the working ISO boots, are they all contained within their initrd.img files? no additional rootfs to be fetched? | 21:06 |
| TJ- | The thing is memdisk uses x86 real-mode interrupts, and loads the image into RAM. But once Linux starts it switches to protected mode which no longer has access to the real-mode access. There are workarounds involving finding the ISO image in RAM and loading a kernel module that maps/mounts it as a memory block device | 21:07 |
| drab | TJ-: http://dpaste.com/0C7EP8J it's all the same for all ISOs | 21:08 |
| drab | that's a straight iso download from http://www.ultimatebootcd.com/ | 21:08 |
| drab | TJ-: I also found this, but could not get it to work either: https://github.com/medallia/ramroot | 21:10 |
| drab | it sounded exactly like what I wanted (and motivated by the same reason - avoiding nfs) | 21:10 |
| TJ- | the reason UBCD works is it has a self-contained squashfs inside the initrd.img | 21:33 |
| drab | fair enough, I guess it makes not much of a diff to me, maybe I'll look how to back the squashfs into the initrd | 21:44 |
| drab | rootram tho looks lik eit's doing the right thing, ie what would logically make sense to me | 21:45 |
| drab | create a ramdisk and wget the root fs and untar it to it | 21:45 |
| drab | however I'm stuck on "tar: applet not found" | 21:45 |
| drab | https://github.com/medallia/ramroot/blob/master/initramfs-tools/scripts/ramdisk#L46 | 21:45 |
| drab | which seems to be happenig there | 21:45 |
| drab | no clue why busybox tar would fail like that... | 21:45 |
| TJ- | not sure why we can't just add a simple http client fetch inside initrd for these boots. it'd be relatively small and simple | 21:51 |
| TJ- | I wonder... there's tricks with bash that could do that :) | 21:59 |
| TJ- | drab: you've given me a challenge now. the ubuntu intrd has ./bin/wget provided by busybox so it's possible to fetch the ISO into a tmpfs, loop mount it, and mount the squashfs | 22:04 |
| drab | TJ-: dude if you got that working it'd be the best early xmas present anybody ever game me :P | 22:11 |
| TJ- | just trying to think if it could be done without needing special arguments on the command line :D | 22:11 |
| TJ- | in *theory* it could be the last-ditch fallback if the other methods fail, before the final drop to shell | 22:12 |
| drab | another random question as I'm trying to make ramroot work at the same time... if possible | 22:13 |
| drab | I found out that for whatever fun reason the busybox shipped in ubuntu does not come with tar... hence the fail | 22:13 |
| drab | I apt-get'ed the source for busybox-initramfs | 22:14 |
| drab | but it's not obvious to me wwhere the applets are enabled (normally it seems you'd run make menuconfig and pick) | 22:14 |
| TJ- | drab: did you try telling memdisk it's a "hardisk" image not "iso" - after all, the ISOs are hybrid | 22:16 |
| drab | TJ-: yes, I saw that and tried that too, made no diff | 22:16 |
| TJ- | OK.. it got into the initrd though? | 22:16 |
| drab | because indeed fdisk -l showed it was a disk like in the example they gave | 22:16 |
| drab | ? | 22:16 |
| drab | lost you there, what got into the initrd? | 22:16 |
| TJ- | I mean it booted as far as the initrd then failed the same way it does with 'iso' method - dropping to shell | 22:17 |
| drab | yeah | 22:17 |
| TJ- | right, that really does confirm the problem there is the switch from real-mode to protected 'loses' access to the im-memory image | 22:17 |
| TJ- | That ramroot project is OK, but misses the point. It could easily work with the existing ISO/squashfs images rather than enforcing having to repack it as tar.xz | 22:25 |
| drab | TJ-: I can't get that to work either... recompiling busybox now, seems a rather wacky choice to ship busybox without tar... | 22:26 |
| drab | I don't even know why/how it worked for him to begin with | 22:26 |
| drab | maybe it used to ship it in 14.04 | 22:26 |
| TJ- | we want it small as possible, and don't need tar in the initrd | 22:27 |
| drab | yeah, I guessed as much, which makes sense | 22:28 |
| drab | just been a long saturday... and thank you for helping, really appreciate it | 22:29 |
| TJ- | I did all this stuff years ago; got scripts that automatically configure the PXE server for ISOs so it's simple to deploy, but I use NFS. | 22:31 |
| TJ- | in case there's any nuggets in it, this is it from 10 years ago: http://tjworld.net/wiki/Linux/Ubuntu/NetbootPxeLiveCDMultipleReleases | 22:32 |
| drab | TJ-: yeah, I have that working (with nfs), the whole problem started when trying to do this over http | 22:33 |
| drab | nfs based stuff is no issue, works pretty well | 22:33 |
| TJ- | yes, I understand | 22:33 |
| TJ- | the end of the article deals with patching the ISO initrd | 22:34 |
| drab | thanks, checking it out | 22:34 |
| TJ- | back then the network modules weren't built-in to the kernel nor included in the initrd, so I had to modify it for getting netconsole | 22:35 |
| drab | bah, got past the tar issue but now getting a whole new bunch of errors I can't even see because it scrolls too fast and then I can't page up, it's all locked up | 22:35 |
| TJ- | failures during extraction maybe? | 22:35 |
| TJ- | one per file in the image maybe? | 22:35 |
| TJ- | you repacked the rootfs as a tar.xz I assume? | 22:36 |
| drab | yeah | 22:36 |
| drab | can't see where it's erroring first, scrolls real fast and dies, I see errors can't create /root/lib/firmware, can't mount /dev on /root/dev and eventually tar fs doesn't have /sbin/init | 22:36 |
| drab | so it does sound like untarring an setting up failed somehow | 22:37 |
| drab | I wonder why it all locks up, normally the initramfs prompt is usable and I can at least scroll up | 22:37 |
| TJ- | did you increase the memtotal size to account for the size of your rootfs when uncompressed? | 22:58 |
| drab | TJ-: for rootmem? no I didn't think of it, it wasn't in the instructions and I thoguht tmpfs was dynamic these days (and would have worst errored if no more space was not available) | 23:02 |
| drab | but I have 4Gbs and the img is 100MB so I thought it wasn't a problem | 23:03 |
| TJ- | look at the ramroot script, line 29. it defaults to 500MB which must take the uncompressed image. if not you have to set memtotal= on tha kernel command-line | 23:07 |
| drab | TJ-: the unextracted/before tar ramroot is 400MB, but I will try | 23:22 |
| drab | also I realized that earlier I tarred chroot, ie the tar would have chroot/{etc,var...} and maybe that's also no good | 23:23 |
| drab | so retarring with the tar straight into the root | 23:23 |
| TJ- | haha yes that'd be a problem | 23:24 |
| TJ- | tar needs to start in the root of the rootfs :) | 23:24 |
| drab | holy cow | 23:27 |
| drab | there's good news and bad news | 23:27 |
| drab | the good news, it booted \o/ | 23:28 |
| TJ- | lol | 23:28 |
| drab | the bad news, I guess it's missing some usb module because the usb keyword plugged doesn't work | 23:28 |
| TJ- | check for usb_hid I think it is (human interface device) | 23:28 |
| drab | check where? | 23:29 |
| drab | I'd thought everything like that was included in the stock kernel I installed in the chroot (and loaded), but maybe not | 23:29 |
| TJ- | it should be in the kernel image itself | 23:29 |
| TJ- | i've had that previously where it needed some specific USB chipset driver to bring up the USB side | 23:30 |
| TJ- | check "find /sys/bus/usb/ " see what's there | 23:31 |
| drab | cant' type so can't check :) | 23:31 |
| drab | but I'm looking in the chroot | 23:31 |
| drab | to see if the module is there, I used linux-virtual-image which installs a minimal kernel | 23:32 |
| drab | and doing /lib/modules/4.4.0-21-generic/kernel# sudo find . -name "*usb*" returns nothing | 23:32 |
| drab | so they may not be there after all | 23:32 |
| drab | yeah on my desktop that returns a lot of stuff, including usbhid.ko | 23:33 |
| drab | trying again adding a diff kernel in the chroot | 23:33 |
| TJ- | right; virtual-image doesn't need USB | 23:35 |
| drab | it's gonna make the img fat tho... | 23:35 |
| * drab scratches his head | 23:35 | |
| drab | also seems I've recompiled busybox I'm thinking I could also make the root tar bz2 instead of xz | 23:36 |
| drab | should make it much smaller | 23:36 |
| TJ- | if you want to boot bare hardware it's better to have it builtin - in case the initrd fails :) | 23:36 |
| TJ- | you could, but ramroot only supports xz as far as I noticed | 23:36 |
| drab | yeah, well, in theory I guess I should just compile a static kernel | 23:36 |
| drab | TJ-: I think it's just the code, to make it more portable as most busyboxes prolly aren't compiled with bzip2 | 23:37 |
| drab | but maybe have xz, so I can prolly just change that part of the code to uncompress the tar | 23:37 |
| TJ- | it's using pixz for decompressing; you'll need to add code to ramroot script if you're going to use bz2 | 23:38 |
| drab | yeah | 23:38 |
| drab | that's about 200MB more of modules... kinda whack, but let's see if it works | 23:50 |
| drab | I guess if it does then I'll spend time compiling a kernel statically and adding a whole bunch of stuff, but avoiding modules I don't really need, like sound | 23:51 |
| drab | comes to 180Mb vs 100MB it wa before | 23:52 |
| drab | yep, bingo! got keyboard and even a nice framebuffer and it was up and running pretty quickly overall | 23:54 |
| TJ- | :) well done | 23:58 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!