[10:55] Hiyas all [12:10] !info vlc [12:10] vlc (source: vlc): multimedia player and streamer. In component universe, is optional. Version 2.2.1-3 (wily), package size 1461 kB, installed size 4450 kB [12:10] so it IS a package [12:11] when I do sudo apt-get install vlc, it says E: Unable to locate package vlc [12:11] can anyone help? [12:12] tsimonq2: Is the universe repo enabled? [12:13] let me pastebinit my sources.list for ya [12:14] lordievader: I am assuming it is, right? http://paste.ubuntu.com/12734317/ [12:15] tsimonq2: Universe is enabled, what does 'apt-cache search vlc' output? [12:15] with sudo? [12:15] tsimonq2: No need for sudo. [12:16] the output: http://paste.ubuntu.com/12734335/ [12:16] tsimonq2: Interesting, what is the output of 'sudo apt-get update'? [12:17] lordievader: http://paste.ubuntu.com/12734346/ [12:18] lordievader: and if it helps, I am running in i386 [12:18] tsimonq2: Now run the apt-cache search again. [12:18] lordievader: http://paste.ubuntu.com/12734355/ [12:19] woah [12:19] There we go, your sources were outdated. [12:19] was it *really* the fact that I didn't update my repo list [12:19] wow fail XD [12:19] * tsimonq2 facepalms [12:19] well thank you [12:19] No problem ;) [12:20] see you over in #kubuntu-offtopic ;) [12:57] Has been libreoffice-style-crystal removed from wily ? [13:55] hi, any troubles with 15.10? [13:55] or can i upgrade? :D [15:40] Anyone succeeded a wily luks-root ? I got KP trying in virtualbox. [15:45] bcx: I use LUKS for /boot/ and for the root LVM VG [15:46] great, did you install with cd ot manual debootstrap ? [15:48] TJ:- i use deboostrap to build systems on virtualbox, when using cryptoroot initramfs hook I got KP related to innotek (virtualbox) input device (keyboard ?) ? [15:48] I do it myself usually, but I tested the installer in a VM and it worked [15:49] bcx: that sounds unrelated to LUKS; that sounds like a missing/bad module in the initrd [15:50] TJ-: building without cryptoroot succeeds [15:50] bcx: can you capture the output to a pastebin? [15:51] bcx: do you get the systemd-cryptsetup prompt for the LUKS pass-phrase? [15:51] TJ-: I can use serial console on virtualbox [15:51] bcx: I use a key-file so don't have to enter a pass-phrase [15:52] TJ-: i love passwords [15:52] only stored on my biological ram :) [15:52] I only have to enter the passphrase for GRUB to unlock its root file-system [15:53] ah ok [15:54] my installer script supports prompting either in grub or initramfs, i remember that both failed but will retry [15:55] will be back with logs in an hour or so, thanks TJ- [16:17] * penguin42 wonders what gpu-manager is and wth it took 16s according to systemd-analyse [16:45] It's the tool for trying to ensure the correct drivers are in place, especially when Optimus chipsets are present [16:49] TJ-: retried with serial console but I got interesting initramfs shell code on the real tty just before the KP, can it be redirected to serial ? [16:51] TJ-: https://paste.debian.net/315269/ [16:53] on real tty i can see 'echo "Somtehing went badly wrong in the initramfs"', 'panic "please file a bug on initramfs-tools"' [16:55] TJ-: Yes, but the interesting question is why it took 16s to figure out that my boring old intel doesnt [16:55] bcx: "Spurious NAK" - suggests the Vbox devices are causing an issue [16:55] penguin42: "/var/log/gpu-manager.log" maybe? [16:55] TJ-: yes but here it talking about the serial link, not sure I get the same msg w/o serial console [16:55] bcx: does the guest have some vbox 'guest extensions' installed? [16:56] TJ-: not sure [16:56] bcx: oh, you mean the serio0 is the Vbox serial console as seen from the guest? [16:56] i think so [16:58] TJ-: Nothing incorrect in there [16:58] penguin42: doesn't help there are no timestamps! [16:59] when i boot normally and use pause I can see a full shell script sources outputed on real tty [16:59] bcx: you see the *source code* of a script? [16:59] and finally KP not syncing: Attempted to kill init [16:59] TJ-: yes [17:00] bcx: is each line preceeded with "+" symbols? [17:00] TJ-: Right, but there are timestamps in the journalctl -u and that does show it taking 14 seconds [17:01] penguin42: can you use those to determine which gpu-manager operation took most of the time? [17:01] TJ-: No [17:02] TJ-: There are 4 moans of /etc/modprobe.d is not a file [17:02] penguin42: the biggest timestamp gap I see is after "Started Detect the available GPUs and deal with any system changes." [17:02] penguin42: 6 seconds here [17:02] TJ-: the other log mentioned grepping dmesg for 256=0 which sounds expensive [17:04] * penguin42 would love to paste you the contents from mine, but konsole just doesn't want to do a copy today - sigh [17:04] TJ-: no it it is really sources, not set -x output [17:05] penguin42: Personally I'd purge it: "gpu-manager -h" ==> "Segmentation fault (core dumped)" [17:05] bcx: so something is bad with the initrd.img then, rebuild it, test again [17:06] penguin42: apparently you can run it manually. maybe do that and see if it pauses between messages [17:07] TJ-: Yeh, it shouldn't take 6 seconds for you either - this machine is a crusty old core2 so I can see it taking longer, but 6 seconds during boot is still insane [17:08] penguin42: doesn't mean it is doing anything. It might be waiting from something to become available [17:09] penguin42: running it manually there is no pause [17:09] TJ-: That should be a dependency in the systemd config file for though shouldn't i [17:09] TJ-: Yeh similarly it's under a second here from a running system [17:10] penguin42: it starts when the DM is starting, so it is probably waiting for X or something [17:14] ls [17:14] oops [17:16] TJ-: It's a bit grim; C file spawning off a dmesg|grep to grep for pci ids [17:19] dmesg isn't large, so that shouldn't take a long time, unless dmesg is full of spam/errors [17:21] yeh only 880 lines === not_phunyguy is now known as phunyguy [21:10] TJ-: is there a simple way to pipe initramfs output to serial ? [21:11] TJ-: is there a simple way to debug initramfs other than adding 'set -x' in all scripts and 'read' for timing ? [21:12] TJ-: i'm not sure 'read' will receive stdin and interrupt execution [21:13] bcx: no, but you can use the maybe_break XXXX hooks with break=XXXX to bisect which 2 points the problem stems from [21:13] bcx: see "/usr/share/initramfs-tools/init" which is the /init script [21:24] has mencoder been removed from wily? [21:29] apparently [21:29] TJ-: know why? [21:31] It was removed for Utopic [21:32] for why? [21:34] No idea; from a browse on packages.ubuntu.com it looks like when mplayer was replaced by mplayer2 === trism_ is now known as trism === trism is now known as trism_ === trism_ is now known as trism === trism is now known as trism_ === qengho_ is now known as CardinalFang === trism_ is now known as trism === Termana_ is now known as Termana [21:48] TJ-: as i said previously i get irfs' init output to screen instead of being executed [21:50] bcx: that makes no sense at all. The only way I could imagine that happening would be for some strange kernel commandline or corrupted /init [21:51] i bet for the later [21:52] if it's getting as starting systemd I think there's a command line option to debug it [21:52] bcx: it sounds as if it is 'cat'ing itself [21:52] TJ-: i replaced all maybe_break calls with panic and cannot break in before [21:53] i can try to add an interrupt at the very top to check if init gets called [21:53] bcx: replaced? [21:54] in the init script [21:55] so i don't think it's self-cating or if it does it's before the first maybe_break call [21:56] bcx: I'm struggling to understand why you'd change those 'maybe_break XXX' lines [21:57] now i will try with 'read -p hello' as init first line [21:57] TJ-: for debugging that way if the init file was executed i could figure out the script part causing this [21:58] but as i said i don't think init gets executed [21:58] bcx: if you're messing with the script directly theres likelyhood you've broken it. [21:59] bcx: the kernel will execute "/init", see the she-bang in line 1, and load it via the shell [21:59] each time i restart from the virtualbox snapshot with initial configuration [21:59] the she-bang is a 'magic' that bintfmt recognises [22:00] s/bintfmt/binfmt/ [22:02] same thing, [22:03] now i will paste a very long recognizable pattern into the init file so i can catch it with pause [22:03] in case the irfs executed is not the one i debug [22:04] bcx: the initrd name will be in the boot-loader entry [22:06] TJ-: can i customize the name to be sure ? [22:06] bcx: it should be correctly set by update-grub if you use GRUB. the name will match the kernel version [22:08] as i have a unique kernel on this system I will use my simple technique: put 10k dash lines in the init file and catch it [22:09] caught [22:10] so i can confirm my issue: i got init script output to the console instead of executed [22:10] that's really weird [22:12] i use the same script to build ubuntu systems and it works flawlessly with trusty & vivid [22:13] it has nothing release specific except debootstrap --keyring=/etc/apt/trusted.gpg for wily [22:14] this issue may come from grub [22:14] which i use like you with cryptomount command [22:16] bcx: "dd if=/boot/initrd.img-$VERSION count=1 | hexdump -C | pastebinit" [22:20] https://paste.debian.net/315296/ [22:21] bcx: that's not a valid initrd image [22:21] TJ-: lsinitramfs can parse it [22:22] what's the problem ? [22:22] bcx: does it have a microcode header ? [22:22] i don't know what are you talking about [22:23] initial ramdisk images can contain binary prefixes containing microcode updates the kernel needs to apply before the init system starts [22:23] cool didn't know that [22:24] TJ-: Is that just to cope with the transactional hack on haswell? [22:25] TJ-: how can i check/disable this ? [22:26] penguin42: it's a generic mechanism, but is currently being used for such things [22:26] bcx: The thing is, that should not cause what you are experiencing but I suspect it is. I compared your hexdump with one on my 4.2.x system and they look different [22:30] bcx: I have a script I use to extract initrd images for inspection. Try it: https://iam.tj/projects/misc/initrd-extract.bash [22:31] bcx: usage for a single initrd.img (not all installed) is initrd-extract /boot/initrd.img-$VERSION [22:32] bcx: then hexdump the /init script again lets see if there are some bad magic bytes [22:34] previously i hexdumped the image not the script [22:35] TJ-: sorry i don't understand [22:38] btw you should add ' && [[ -n "$INITRD_LIST" ]]' to '[[ -z "$1" ]]' in case no image found in /boot [22:39] it would fail gracefully [22:39] bcx: Did you say you've got a special kernel build? [22:40] I've never needed it on a system that doesn't have initrd.img s :) [22:40] i use standard trusty livecd with zfs-native ppa modules [22:40] TJ-: i did ;) [22:41] bcx: once the script has extracted the initrd correctly (to a directory under /tmp/) you can hexdump the /init script correctly so we can check the magic at the start of the file [22:42] Processing 35551541 bytes of /boot/initrd.img-4.2.0-11-lowlatency [22:42] Skipping 21504 bytes pre-pended early initramfs image (possibly CPU microcode) [22:42] Creating /tmp/initrd.img.8844 for remaining 35530037 bytes [22:42] Processing 35530037 bytes of /tmp/initrd.img.8844 [22:42] Executing /bin/cat "/tmp/initrd.img.8844" | /bin/zcat | /bin/cpio --extract --make-directories [22:42] 184924 blocks [22:42] Extracted 1481 files to /tmp/initrd/4.2.0-11-lowlatency/ [22:42] * TJ- blows *raspberries* at the paste warning :) [22:42] but the magic i pasted previously is from the image, not the script [22:43] so maybe you want to see the script's magic, instead of the image's one [22:44] bcx: now I can do "hexdump -C /tmp/initrd/4.2.0-11-lowlatency/init | head [22:44] 00000000 23 21 2f 62 69 6e 2f 73 68 0a 0a 23 20 44 65 66 |#!/bin/sh..# Def| [22:44] bcx: correct, and the script magic is the important thing we want to see [22:45] understood [22:45] bcx: if that looks correct then it is possible the early-initramfs image is somehow breaking things [22:46] bcx: how many bytes does my script report it skipping for the early initramfs image? [22:48] it's clean [22:49] https://paste.debian.net/315297/ [22:50] TJ-: what/where is the "early" initramfs ? new to me [22:52] bcx: You said you were using zfs; on which partitions? [22:52] bcx: it is the first part of the initrd.img, an optional add-on used for loading microcode patches and other things the kernel needs before userspace starts [22:53] bcx: what does this report "file /tmp/initrd/*/bin/{busybox,sh}" [22:53] sorry, zfs is not incriminated, i get the same results w/o zfs, here i test with ext4 and no zfs modules [22:54] * TJ- suspects a 32-bit kernel and 64-bit userspace [22:54] oh that would be fun [22:55] all 64 [22:55] It would explain in some weird way the script not being executed, although I have difficulty imagining how that would come about :) [22:55] bcx: I have to suspect your build scripts at this point, something done/not done/done differently [22:56] bcx: If you pass init=/bin/sh on the command line do you get a shell? [22:56] good test [22:57] no [22:57] how does that fail? [22:58] same KP Attempted to kill init [22:58] what's before that line [22:58] cannot catch it [22:58] really? That's normally right near the end [22:58] can't see what is before the KP *block* [22:59] penguin42: because that is executed at the termination of the initrd /init script, not in place of it [22:59] oh yeh [22:59] penguin42: the initrd /init script passes that parameter in its final switch_root [23:01] bcx: Try adding boot_delay=50 it will cause all the text output during boot to slow down, maybe you can catch what happens easier? [23:01] increase 50 to taste [23:01] If the initrd /init script is being 'cat'-ed' not executed that suggests the /bin/sh (as in the she-bang magic) is faulty. /bin/sh is usually a multicall entry in a statically linked /bin/busybox [23:01] wow using init=/bin/sh i can see its binary code [23:02] that's a weird failure mode [23:02] penguin42: i have busybox-static package installed in my script [23:03] bcx: What exactly is your script doing? [23:04] debootstrap & miniman config with various advanced storage features zfs, crypto, fde [23:05] but in this case no feature [23:06] TJ-: will try w/o busybox-static [23:06] bcx: It shouldn't matter, but the usual busybox install is dynamically linked, for some reason [23:07] bcx: can you execute the busybox that was extracted under /tmp/... by my script? [23:11] TJ-: no ! [23:12] and the error is? [23:12] hangs [23:12] bcx: HAHA... so bin/sh will fail too. do "ldd bin/busybox" any missing libraries? [23:12] 64bit dynamically linked [23:13] you do have the dynamic linker - right? [23:14] https://paste.debian.net/315298/ [23:15] shouldn't be static as the package is installed ? [23:16] that pastebin looks to be cut-off early [23:17] here's what I see on 14.04: http://paste.ubuntu.com/12745262/ [23:17] sorry https://paste.debian.net/315299/ [23:18] it's a wily system debootstrapped on trusty [23:18] bcx: take a hash of the file in the initrd, compare to the hash of the file in the installed file-system [23:19] in the debootstrapped file system i got a static one [23:19] which i can run [23:20] don't know where this dynamically linked one comes from [23:20] sha1sum bin/busybox /usr/lib/initramfs-tools/bin/busybox [23:20] dc3e621c72cde19593c42a7703e143fd3dad5320 bin/busybox [23:20] dc3e621c72cde19593c42a7703e143fd3dad5320 /usr/lib/initramfs-tools/bin/busybox [23:22] https://paste.debian.net/315300/ [23:22] where does it come from ? [23:22] is the initrd one even valid? [23:23] i can run /mnt/target/usr/lib/initramfs-tools/bin/busybox [23:23] bcx: my hashes came from 14.04; I'll check on 15.10 now [23:23] bcx: so, that suggests the one in the initrd is bad/corrupt [23:24] of course [23:24] bravo TJ- [23:24] $ sha1sum /tmp/initrd/4.2.0-11-lowlatency/bin/busybox /usr/lib/initramfs-tools/bin/busybox [23:24] c90333979f56f38bbd41b81806015b0de502f3cc /tmp/initrd/4.2.0-11-lowlatency/bin/busybox [23:24] c90333979f56f38bbd41b81806015b0de502f3cc /usr/lib/initramfs-tools/bin/busybox [23:24] which matches you installed package, but not the initrd [23:28] still, that sha matches another one [23:28] https://wiki.strongswan.org/projects/strongswan/wiki/IMA/9 [23:29] so it's apparently an old /bin/sh [23:29] whether that's actually a busybox I don't know [23:29] you're mixing them up. That is the 14.04 busybox [23:29] the one that is corrupt is fce342dd3fce6e0e3ba88e4249cb7a210a42844a [23:29] as listed at https://paste.debian.net/315300/ [23:30] oops yes [23:31] the sha is preserved with update-initrd [23:31] update-initramfs i mean [23:31] I'd best start winding down for sleep; I'm out to catch the last flight of the XH558 Vulcan tomorrow [23:31] bcx: did you recreate the initrd, or update it? [23:31] ah, I think I missed that here today since I didn't know it was around [23:32] thank you TJ- [23:32] penguin42: I almost did but noticed it Friday night [23:32] penguin42: -c -k all [23:32] bcx, so is there some corruption being introduced? does adding '-v' to capture a verbose log show where that busybox is coming from? [23:33] some junk under /etc/initramfs-tools or the like? [23:33] https://paste.debian.net/3153001 [23:34] I still maintain the symptoms suggest that /init is being 'cat'-ed - what's that chance that sha1sum matches the hash for the /bin/cat ? [23:34] so it does! [23:34] then https://paste.debian.net/3153002 [23:34] fce342dd3fce6e0e3ba88e4249cb7a210a42844a /bin/cat [23:35] HAHA! catch! [23:35] * penguin42 blames it on caturday [23:35] bcx: there you go; something in your config is putting /bin/cat in there [23:35] LOL [23:36] zz-busybox-initramfs hook ? https://paste.debian.net/3153002 [23:37] bad link [23:38] no, he just types too many zeros in them :) [23:39] bcx: add "set -x" to zz-busybox-initramfs [23:39] oh [23:39] bcx: I don't think its that because the script is doing links, but best to be sure [23:41] https://paste.debian.net/315304 [23:43] bcx: sha1sum /usr/lib/initramfs-tools/bin/busybox [23:43] that looks fine, although I don't know why it uses symbolic links since cpio archives cannot preserve those, and end up adding 2 copies of the multicall binary [23:44] penguin42: from earlier: [23:44] root@ubuntu:~# sha1sum /mnt/target/usr/lib/initramfs-tools/bin/busybox [23:44] c90333979f56f38bbd41b81806015b0de502f3cc /mnt/target/usr/lib/initramfs-tools/bin/busybox [23:44] in https://paste.debian.net/315300/ [23:44] hmm [23:44] wat is copy_exec [23:45] bcx: a helper function [23:45] defined where ? [23:45] in the initramfs-tools hook script [23:45] bcx: before we get sidetracked, have you checked both bin/sh and bin/busybox from the initrd? [23:45] hoolk-functions [23:46] bcx: if bin/busybox --version works but bin/sh --version doesn't we know its not directly a zz-busybox-... issue [23:50] bcx: here's a nice way to figure out if things changed: "ls -latr bin/" ... all the busybox tools should be identical in size to bin/busybox itself [23:51] https://paste.debian.net/315307 copy_exec with set -x if $2==/bin/busybox [23:51] bcx: that may reveal other executables aren't in fact from BB [23:54] bcx: if bin/sh is actually /bin/cat then I reckon there's a script doing a complex shell replacement parameters operation that is failing and ending up doing "cp /bin/cat /target/.../bin/sh" instead of something like "cp $(...something...cat...) /target/.../bin/sh" [23:54] --version is ok [23:55] it's hanging when called with no arguments [23:55] the thing is if they're all linked together because they're all busybox, how does cat end up being different [23:55] oh it isn't, it's they're all cat [23:55] busybox misinterprets $0 [23:56] hooks/klibc:28: cp -pL /usr/lib/klibc/bin/sh ${DESTDIR}/bin/sh [23:56] https://paste.debian.net/315308 [23:57] penguin42: they're not linked together, that's the point. cpio archives cannot represent symbolic links, so it ends up as a copy operation, so we have loads of supposed identical binaries with different names, except right now bin/sh is not the same binary as busybox, its /bin/cat [23:58] bcx: it's looking like some other script is running AFTER zz-busybox [23:58] bcx: klibc being the candidate [23:58] yes ! [23:59] i have /etc/initramfs-tools/hooks/copy-cat.sh [23:59] bcx: can you show us the entire log of update-initramfs, rather than an extract?