| === Foxhoundz is now known as BenderRodriguez | ||
| cpaelzer | doko: cyphermox: It is always the new guy which struggles but due to that has a chance to make documentation better :-) I wanted to ask what you'd think of the following http://paste.ubuntu.com/p/JXMtDcnYBb/ ? | 06:04 |
|---|---|---|
| cpaelzer | Or to better map asciart with text http://paste.ubuntu.com/p/gX9ZYmntRX/ | 06:09 |
| alterjsive | sladen: it looks like it breaks after I install vagrant | 06:41 |
| alterjsive | I reinstalled my system, did a full update, did a reboot. Then I installed vagrant and now I get a busybox | 06:42 |
| alterjsive | after a reboot | 06:42 |
| sladen | alterjsive: how was 'vagrant' installed, and from where? | 06:45 |
| alterjsive | y boot breaks after I install vagrant(updates initfs?) | 06:49 |
| alterjsive | Steps: | 06:49 |
| alterjsive | install ubuntu & reboot | 06:49 |
| alterjsive | do a full update & reboot | 06:49 |
| alterjsive | install vagrant & reboot | 06:49 |
| alterjsive | => busybox | 06:49 |
| alterjsive | I'm a consultant, can I also pay for support? :) | 06:51 |
| sladen | alterjsive: how was 'vagrant' installed, and from where? | 06:52 |
| alterjsive | sladen: i just did a sudo apt-get install vagrant -y on a clean updated system. | 06:53 |
| sladen | alterjsive: ie. from an Ubuntu package (which exact version), or from a Debian package (which exact version), or from downloading from the internet | 06:53 |
| sladen | alterjsive: okay, that this single action (sudo apt-get install vagrant -y ; sudo reboot) results in getting a busybox loging? | 06:54 |
| sladen | login? | 06:54 |
| alterjsive | vagrant (2.0.2+dfsg-2ubuntu8) bionic; urgency=medium | 06:55 |
| cpaelzer | doko: cyphermox: I'll be unavailable next weeks MIR meeting (PTO) | 06:55 |
| cpaelzer | just FYI as I checked next weeks meetings | 06:56 |
| alterjsive | 1 sec i'll reboot to check the exact message | 06:56 |
| sladen | https://launchpad.net/ubuntu/+source/vagrant/2.0.2+dfsg-2ubuntu8 | 06:58 |
| * sladen pages Old_Bloke | 06:58 | |
| alterjsive | Busybox v 1.27.2 .... (initramfs) | 07:01 |
| alterjsive | sladen: any idea how to fix this initramfs issue? | 07:03 |
| sladen | alterjsive: what is the *exact* error message(s) on boot up | 07:04 |
| alterjsive | sladen: I get a busybox immediatly | 07:04 |
| sladen | alterjsive: what does the output say, immediately before "getting a busybox" ? | 07:05 |
| cpaelzer | alterjsive: FYI I'm trying to recreate that per steps you listed above to cheeck if it generally applies | 07:05 |
| alterjsive | sladen: can I check some logfile? | 07:05 |
| sladen | cpaelzer: https://github.com/chef/bento/issues/661 could be a possibility | 07:05 |
| sladen | cpaelzer: but don't want to distract alterjsive from trying to debug and get hard facts! | 07:07 |
| alterjsive | sladen: should I check the boot log from a bootable usb? | 07:10 |
| cpaelzer | steps of about 22 minutes ago work fine for me, vagrant is installed and reboot works as expected | 07:11 |
| alterjsive | after a failed boot /var/log/boot.log | 07:11 |
| cpaelzer | so yeah, lets try to get reasonable logs from alterjsive case | 07:11 |
| alterjsive | ok brb reboot | 07:12 |
| alterjsive | any other logs ? | 07:12 |
| alterjsive | I'll add /var/log/dmesg too | 07:13 |
| sladen | alterjsive: boot log, dmesg, etc. Something will say *why* this boot up has gone to busybox (and without logs the going to busybox is the only we have indication here of anything non-normal here...) | 07:13 |
| alterjsive | sladen: I don't see anything unusual in the boot.log | 07:27 |
| alterjsive | https://paste.ubuntu.com/p/vKFz4rc95y/ | 07:28 |
| alterjsive | dmesg is not a log I found out :) | 07:29 |
| alterjsive | I just copied all files in my log dir | 07:29 |
| alterjsive | sladen: https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1794318 i've added my logfiles | 07:34 |
| ubottu | Launchpad bug 1794318 in linux (Ubuntu) "unable to boot after installing vagrant?" [Undecided,Incomplete] | 07:34 |
| alterjsive | all of them | 07:35 |
| alterjsive | I'm sure you guys are busy , I should move this conversation to another channel right? | 07:43 |
| sladen | alterjsive: well here is good. keeps all the IRC logs in one place | 07:46 |
| alterjsive | did I say that right. an initfs update? | 07:47 |
| alterjsive | previous kernel version it works until another initfs update | 07:48 |
| alterjsive | my current kernel is working, can I make a backup ? in case I install something which does a update-initramfs ? | 07:49 |
| alterjsive | I need to continue my work now(deadline) | 07:50 |
| sladen | alterjsive: tihs syslog has four bootups in it. Which are the good ones, and which are the bad ones | 07:53 |
| sladen | this | 07:53 |
| sladen | alterjsive: "my current kernel is working" <-- *which* (exact version) kernel is working? | 07:55 |
| sladen | alterjsive: the syslog shows 3x boots with "(Ubuntu 4.15.0-29.31-generic 4.15.18)" and 1x boot with "(Ubuntu 4.15.0-34.37-generic 4.15.18)" | 07:56 |
| sladen | alterjsive: $ grep 'Linux version' log/syslog | 07:57 |
| alterjsive | grub says my latest kernel is 4.15.0-34-generic and the one i'm using now is 4.15.0-29-generic | 08:03 |
| sladen | alterjsive: so 4.15.0-34-generic is the *non*-working | 08:04 |
| alterjsive | yes | 08:04 |
| sladen | thank you | 08:04 |
| alterjsive | I was thinking about reinstalling it | 08:04 |
| sladen | and this is nothing to do with vagrant? | 08:04 |
| sladen | the *only* change is switching from kernel 4.15.0-29-generic (working) and rebooting with 4.15.0-34-generic (not working) | 08:05 |
| alterjsive | sladen: I'm under the impression that a update-initramfs breaks it, just a gut feeling | 08:05 |
| sladen | alterjsive: please confirm, that to replicate this, the *only* action needed is selecting one kernel, vs. the other kernel in Grub on bootup | 08:06 |
| sladen | s/replicate/reproduce this bug/ | 08:07 |
| alterjsive | sladen: yes | 08:07 |
| sladen | thank you | 08:07 |
| sladen | alterjsive: and that in neither case "safe mode" is being selected in Grub | 08:08 |
| alterjsive | sladen: yes | 08:08 |
| alterjsive | 1 sec I'll be more specific. After installing vagrant it did a update-initramfs which broke booting from kernel 4.15.0-34-generic when I switched back, using the grub menu, to a previous kernel 4.15.0-29-generic I could still boot without problems | 08:10 |
| alterjsive | any objections to sudo apt-get install --reinstall linux-image-generic linux-image | 08:11 |
| alterjsive | sladen: or do you need anyting more from me for debugging purposes? | 08:12 |
| sladen | alterjsive: please upload the working and non-working initramfs | 08:12 |
| alterjsive | sladen: how? | 08:13 |
| sladen | alterjsive: same way the lof files were uploaded | 08:13 |
| sladen | alterjsive: same way the log files were uploaded | 08:13 |
| alterjsive | where do they live? | 08:14 |
| sladen | alterjsive: ls -l /dev/*init*{4.15.0-29,4.15.0-34} | 08:15 |
| sladen | alterjsive: ls -l /boot/*init*{4.15.0-29,4.15.0-34} sorry | 08:15 |
| alterjsive | sladen: it's +-100MB, can I post it on launchpad? | 08:22 |
| alterjsive | together | 08:22 |
| sladen | alterjsive: or at least post the output of ls -l for the moment | 08:23 |
| alterjsive | sladen: https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1794318/+attachment/5193066/+files/initramfs.tar.gz | 08:24 |
| ubottu | Launchpad bug 1794318 in linux (Ubuntu) "unable to boot after installing vagrant" [Undecided,Incomplete] | 08:24 |
| alterjsive | ls -l /boot/*init* | 08:24 |
| alterjsive | -rw-r--r-- 1 root root 54060073 sep 26 08:20 /boot/initrd.img-4.15.0-29-generic | 08:24 |
| alterjsive | -rw-r--r-- 1 root root 59276945 sep 26 08:36 /boot/initrd.img-4.15.0-34-generic | 08:24 |
| alterjsive | sladen: my best guess is that when I install virtualbox, I will break my current kernel aswell | 08:27 |
| alterjsive | because it updates initramfs | 08:28 |
| alterjsive | sladen: can I try to reinstall the latest kernel now? | 08:29 |
| sladen | alterjsive: just before "Log ended: 2018-09-26 08:36:21" there is "Processing triggers for initramfs-tools (0.130ubuntu3.3)" "update-initramfs: Generating /boot/initrd.img-4.15.0-34-generic" | 08:29 |
| sladen | alterjsive: yes | 08:29 |
| alterjsive | sladen: I see, I don't know the implectations of that | 08:30 |
| sladen | alterjsive: this is a place were only "initrd.img-4.15.0-34-generic" is updated, and not the older one | 08:30 |
| sladen | alterjsive: did 4.15.0-34 work *before* the initramfs was updated? | 08:31 |
| alterjsive | sladen: yes | 08:34 |
| sladen | righht... | 08:34 |
| alterjsive | I only got the busybox after i installed vagrant | 08:34 |
| alterjsive | sudo apt-get install --reinstall linux-image-4.15.0-34-generic | 08:34 |
| alterjsive | brb reboot | 08:34 |
| alterjsive | sladen: no luck, still busybox | 08:37 |
| sladen | alterjsive: and if you remove apt-get remove --purge vagrant ? | 08:39 |
| alterjsive | sladen: won't it try to remove it from the current kernel? | 08:39 |
| alterjsive | ok maybe it's cheaper to buy a new laptop with hardware raid | 08:40 |
| alterjsive | sorry bit frustrated | 08:41 |
| alterjsive | apt-get remove --purge vagrant, brb reboot | 08:41 |
| alterjsive | sladen: https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1794318/+attachment/5193074/+files/cannot%20activate%20member%20md127.jpg | 08:51 |
| ubottu | Launchpad bug 1794318 in mdadm (Ubuntu) "unable to boot after installing vagrant" [Undecided,Incomplete] | 08:51 |
| alterjsive | sladen: looks like it's a fake raid issue | 08:52 |
| alterjsive | sladen: luckally I can still boot using the previous kernel | 08:53 |
| sladen | alterjsive: from the working kernel. Please upload cat /proc/mounts | 08:55 |
| sladen | alterjsive: thank you for the screenshot of the actual error message | 08:56 |
| alterjsive | https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1794318/comments/11 | 09:00 |
| ubottu | Launchpad bug 1794318 in mdadm (Ubuntu) "unable to boot after installing vagrant" [Undecided,Incomplete] | 09:00 |
| sladen | ... | 09:07 |
| === ghavil is now known as Guest58840 | ||
| sladen | alterjsive: looking at this tarball from earlier containing logs, the syslog contains these four boots, including one of -34 | 09:09 |
| alterjsive | sladen: I tried installing kernel 4.15.0-33-generic, but I get a busybox | 09:09 |
| sladen | alterjsive: if the mirror/raid away on mounted on root had not booted, then the syslog from the boot would not have saved. So this boot of -34 appears to have worked | 09:09 |
| sladen | s/away/array/ | 09:10 |
| alterjsive | sladen: when I installed 33, I tried to reboot, but I had to do a hard shutdown using the button because it hung on a stop job for WPA supplicants, i don't know if this is significant | 09:11 |
| alterjsive | this used to happen all the time | 09:12 |
| alterjsive | lately | 09:12 |
| alterjsive | sladen: yes it worked, before I installed vagrant | 09:12 |
| sladen | alterjsive: so 08:20 boot with -29, reboot to -34 at 08:30, install vagrant at 08:36, reboot, (reboot to -34 failed, no logs), reboot at 08:38 to -29, reboo at 08:59 to -29 | 09:14 |
| alterjsive | sladen: looks like it | 09:16 |
| sladen | alterjsive: please upload the output of find /etc/initramfs-tools/ -ls | 09:19 |
| alterjsive | sladen: I just installed virtualbox, I didn't see update-initramfs | 09:21 |
| alterjsive | sladen: https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1794318/comments/12 | 09:21 |
| ubottu | Launchpad bug 1794318 in mdadm (Ubuntu) "unable to boot after installing vagrant" [Undecided,Incomplete] | 09:21 |
| alterjsive | Setting up virtualbox-dkms (5.2.10-dfsg-6ubuntu18.04.1) ... | 09:24 |
| alterjsive | Loading new virtualbox-5.2.10 DKMS files... | 09:24 |
| alterjsive | Building for 4.15.0-29-generic 4.15.0-34-generic | 09:24 |
| alterjsive | Building initial module for 4.15.0-29-generic | 09:24 |
| alterjsive | just so you know | 09:25 |
| cpaelzer | doko: didrocks: I'd be happy if one of you here in my TZ could do a quick check if bug 1794219 is really ok (I found nothing) before I grant the MIR Team ack | 09:30 |
| ubottu | bug 1794219 in ledmon (Ubuntu) "[MIR] ledmon" [Undecided,Confirmed] https://launchpad.net/bugs/1794219 | 09:30 |
| cpaelzer | would you run this in valgrind to check for "Incautious use of malloc/sprintf" of the checks in https://wiki.ubuntu.com/MIRTeam ? | 09:34 |
| cpaelzer | II did, but just want to learn on best practise for these | 09:37 |
| alterjsive | sladen: could you change the status back to posted? https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1794318 | 10:05 |
| ubottu | Launchpad bug 1794318 in mdadm (Ubuntu) "unable to boot after installing vagrant" [Undecided,Incomplete] | 10:05 |
| alterjsive | sladen: it's complete now right? | 10:06 |
| sladen | alterjsive: did installing virtualbox cure things? | 10:09 |
| alterjsive | sladen: I don't dare to reboot | 10:10 |
| alterjsive | I will reboot the end of the day after my work | 10:10 |
| LocutusOfBorg | what is the virtualbox problem? | 10:10 |
| LocutusOfBorg | if none, nevermind | 10:10 |
| alterjsive | LocutusOfBorg: nothing yet, thx | 10:11 |
| alterjsive | sladen: if I want mirroring on linux and to use my self encrypting drive encryption. Should I buy a laptop with hardware raid? is this even possible? | 10:13 |
| alterjsive | sorry a bit off topic I know | 10:13 |
| alterjsive | ok I'll reboot, here goes nothing | 10:14 |
| mwhudson | cpaelzer, rbasak: hey can i ping you guys to get a new package imported? | 10:15 |
| mwhudson | cpaelzer, rbasak: in particular, rustc (it's a little large) | 10:15 |
| alterjsive | sladen: no lock, luckally I can still boot | 10:20 |
| alterjsive | on 27 | 10:20 |
| alterjsive | 29* | 10:20 |
| alterjsive | sladen: are you planning to pick this bug up soon, just for my own indication? | 10:27 |
| cpaelzer | mwhudson: I can give it a try | 10:39 |
| cpaelzer | mwhudson: do you need a one shot, or a regular re-import? | 10:39 |
| cpaelzer | mwhudson: I'll do a --no-push --active-series-only first to check if it is affected by any odd issues before going for all of the history | 10:41 |
| sladen | alterjsive: can you run mdadm --assemble --scan at some point | 10:41 |
| mwhudson | cpaelzer: regular would be awesomest | 10:41 |
| mwhudson | (can you not just import all of ubuntu already!! ;-p) | 10:42 |
| alterjsive | sladen: ok, will try it tonight, thx | 10:42 |
| sladen | xnox: this upload of mdadm (4.1~rc1-3~ubuntu18.04.1) ... https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1779685 | 11:15 |
| ubottu | Launchpad bug 1779685 in mdadm (Ubuntu Bionic) "[18.04.1] Backport support for Intel VROC arrays in mdadm" [High,Fix released] | 11:15 |
| doko | cpaelzer: looks good | 11:15 |
| sladen | xnox: (might) be the cause of https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1794318 slangasek in particular referred to some udev rules from upstream being pulled into the Ubuntu package | 11:16 |
| ubottu | Launchpad bug 1794318 in mdadm (Ubuntu) "unable to boot after installing vagrant" [Undecided,Incomplete] | 11:16 |
| sladen | xnox: "IMSM RAID geometry validation failed." and "platform does not support raid1 with 2 disks" + initramfs emergency shell is the result using an initramfs with this copy of mdadm | 11:19 |
| cpaelzer | doko: the bug update or the pastebin with the changes? | 11:23 |
| sladen | alterjsive: mdadm --detail-platform also please later | 11:32 |
| alterjsive | sladen: ok, will do | 11:33 |
| rbasak | mwhudson: I'll try an import and see how it goes | 11:36 |
| cpaelzer | rbasak: will you run the real import then, that would be nice | 11:45 |
| cpaelzer | rbasak: if my faster already started version stumble ssomewhere I'll let you know | 11:45 |
| rbasak | cpaelzer: I'm running it (in the screen) for real. If it succeeds it'll push. | 11:48 |
| rbasak | cpaelzer: though it's important to then add it to the whitelist so that it keeps up afterwards | 11:48 |
| xnox | sladen, i am sad =/ | 11:50 |
| xnox | sladen, disabling mdadm raid in the bios screens (typically Ctrl+I) and booting the drives directly might be an option. | 11:52 |
| sladen | xnox: currently the user is using their previous initramfs (with previous copy of mdadm included) | 11:52 |
| xnox | sladen, however, the install is done on top of intel raid.... as per partman | 11:53 |
| xnox | sladen, detail-platform would be useful, as well as the state/healthiness of the array from bios. quite likely there are unclean shutdowns and thus array is out of sync. | 11:53 |
| sladen | xnox: can you give a list of useful debugging commands for the user to run? | 11:54 |
| xnox | sladen, this user is interesting.... | 11:55 |
| * xnox is reading logs | 11:55 | |
| xnox | sladen, specifically they forced mdadm installation from kubuntu desktop | 11:58 |
| sladen | xnox: tarball::log/apt/history.log Start-Date: 2018-09-26 08:35:14 Commandline: apt-get install -y vagrant ... mdadm:amd64 (4.1~rc1-3~ubuntu18.04.1, automatic). What are you seeing? | 12:03 |
| xnox | sladen, i'm reading installer logs, as to how the system installation was done. | 12:27 |
| cpaelzer | rbasak: it ran a while without issues and is done (with the reduced set) | 12:27 |
| cpaelzer | rbasak: so your run should hoefully work as well | 12:27 |
| alterjsive | did you guys say anything about sed's and raid btw? | 12:29 |
| alterjsive | I know i'm a bit ambitious in my IT demands :))) | 12:29 |
| cpaelzer | rbasak: thanks for all the MP reviews! | 12:40 |
| rbasak | More to come :) | 12:47 |
| jbicha | cpaelzer: one comment on your paste: Ubuntu bugs easily get into Confirmed status so I consider New and Confirmed to be equivalent | 12:51 |
| tych0 | smoser: another aws noob question for you if you have a moment, I'm getting, "ClientError: Unsupported configuration: Multiple etc directories found" | 13:21 |
| tych0 | my root filesystem is an lvm thin pool, with several thin pools underneath it (all of them have /etc) | 13:22 |
| tych0 | there are some other lvm volume too (none of them have /etc), but in any case, the one that's specified as / in grub does have a reasonable /etc and is a thin pool. do you know if this is supported? or is something else going on? | 13:23 |
| cpaelzer | jbicha: yeah, since confirmed has no meaning yet we can define them as "the same" for this | 13:34 |
| cpaelzer | jbicha: updated in the draft thanks | 13:34 |
| cpaelzer | updated draft http://paste.ubuntu.com/p/HZXPnGk5sr/ | 13:36 |
| smoser | tych0: is this pvm ? | 13:42 |
| smoser | hvm should use grub2 and grub2 i would think might have issues, but such issues would not be specific to ec2 | 13:42 |
| smoser | pvm uses grub-legacy-ec2 which writes /boot/grub/menu.lst for a pv-grub loader to load. | 13:43 |
| smoser | that is hacky and generally deprecated, and i don tknow of the support for lvm volumes there. | 13:43 |
| tych0 | smoser: i don't know the difference :) | 13:44 |
| tych0 | smoser: i just used image-import by following the default docs | 13:44 |
| smoser | tych0: its probably hvm then. | 14:03 |
| smoser | but if that is the case hten you should most likelyi be able to reproduce outside ec2 | 14:04 |
| tych0 | smoser: sadly, the image boots fine other places | 14:04 |
| didrocks | cpaelzer: yeah, +1 for me, double checking by security is needed as you set | 14:04 |
| tych0 | and it's not that it doesn't boot here, it's that aws won't import it. presumably because it's looking through all th ethin pools and seeing multiple /etc dirs | 14:06 |
| smoser | can you do a ec2-describe-images call on it ? | 14:06 |
| smoser | i dont know their "import" process. i only have familiarity with the bare bones path | 14:06 |
| smoser | where you push bytes and then try to boot them. | 14:06 |
| tych0 | oh | 14:07 |
| tych0 | well that sounds interesting :) | 14:07 |
| tych0 | how do i do that? | 14:07 |
| tych0 | (smoser) | 14:11 |
| tych0 | smoser: i don't think i can do describe-images, because it never got successfully imported (the error i pasted above is from the import job) | 14:12 |
| smoser | oh. | 14:12 |
| smoser | its possible its just a buggy process flagging somethign that would otherwise work | 14:13 |
| tych0 | yeah, i'm wondering if that's what's happening | 14:13 |
| tych0 | not a lot of info on the interwebs about this, though | 14:13 |
| tych0 | admittedly, a nested thinpool for / is a strange situation, i suppose | 14:14 |
| hallyn | smoser: btw is it safe to assume that the code for generating and uploading the virtual images to aws is in lp? | 15:10 |
| infinity | hallyn: Ish. | 15:13 |
| infinity | hallyn: livecd-rootfs is responsible for all the heavy lifting there, but for public clouds, there's also a secret sauce branch that does some extra bits that aren't public. | 15:14 |
| infinity | hallyn: Oh, and the "and uploading" part is definitely not public. Though, I'm not sure there's any strategic reason why it couldn't be, just that it's not, AFAIR. | 15:15 |
| hallyn | ok. a matter of "it would cost time/money to figure out whether there's secret sauce or not" probably | 15:15 |
| hallyn | thanks infinity | 15:16 |
| === ledeni_ is now known as ledeni | ||
| === choopm is now known as Guest80724 | ||
| Sven_vB | hi! is this the right place for help with debugging ubiquity? | 23:29 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
