=== Foxhoundz is now known as BenderRodriguez
[06:04] <cpaelzer> doko: cyphermox: It is always the new guy which struggles but due to that has a chance to make documentation better :-) I wanted to ask what you'd think of the following http://paste.ubuntu.com/p/JXMtDcnYBb/ ?
[06:09] <cpaelzer> Or to better map asciart with text http://paste.ubuntu.com/p/gX9ZYmntRX/
[06:41] <alterjsive> sladen: it looks like it breaks after I install vagrant
[06:42] <alterjsive> I reinstalled my system, did a full update, did a reboot. Then I installed vagrant and now I get a busybox
[06:42] <alterjsive> after a reboot
[06:45] <sladen> alterjsive: how was 'vagrant' installed, and from where?
[06:49] <alterjsive> y boot breaks after I install vagrant(updates initfs?)
[06:49] <alterjsive> Steps:
[06:49] <alterjsive> install ubuntu & reboot
[06:49] <alterjsive> do a full update & reboot
[06:49] <alterjsive> install vagrant & reboot
[06:49] <alterjsive> => busybox
[06:51] <alterjsive> I'm a consultant, can I also pay for support? :)
[06:52] <sladen> alterjsive: how was 'vagrant' installed, and from where?
[06:53] <alterjsive> sladen: i just did a sudo apt-get install vagrant -y on a clean updated system.
[06:53] <sladen> alterjsive: ie. from an Ubuntu package (which exact version), or from a Debian package (which exact version), or from downloading from the internet
[06:54] <sladen> alterjsive: okay, that this single action (sudo apt-get install vagrant -y ; sudo reboot)  results in getting a busybox loging?
[06:54] <sladen> login?
[06:55] <alterjsive> vagrant (2.0.2+dfsg-2ubuntu8) bionic; urgency=medium
[06:55] <cpaelzer> doko: cyphermox: I'll be unavailable next weeks MIR meeting (PTO)
[06:56] <cpaelzer> just FYI as I checked next weeks meetings
[06:56] <alterjsive> 1 sec i'll reboot to check the exact message
[06:58] <sladen> https://launchpad.net/ubuntu/+source/vagrant/2.0.2+dfsg-2ubuntu8
[06:58]  * sladen pages Old_Bloke
[07:01] <alterjsive> Busybox  v 1.27.2 .... (initramfs)
[07:03] <alterjsive> sladen: any idea how to fix this initramfs issue?
[07:04] <sladen> alterjsive: what is the *exact* error message(s) on boot up
[07:04] <alterjsive> sladen: I get a busybox immediatly
[07:05] <sladen> alterjsive: what does the output say, immediately before "getting a busybox" ?
[07:05] <cpaelzer> alterjsive: FYI I'm trying to recreate that per steps you listed above to cheeck if it generally applies
[07:05] <alterjsive> sladen: can I check some logfile?
[07:05] <sladen> cpaelzer: https://github.com/chef/bento/issues/661  could be a possibility
[07:07] <sladen> cpaelzer: but don't want to distract alterjsive from trying to debug and get hard facts!
[07:10] <alterjsive> sladen: should I check the boot log from a bootable usb?
[07:11] <cpaelzer> steps of about 22 minutes ago work fine for me, vagrant is installed and reboot works as expected
[07:11] <alterjsive> after a failed boot /var/log/boot.log
[07:11] <cpaelzer> so yeah, lets try to get reasonable logs from alterjsive case
[07:12] <alterjsive> ok brb reboot
[07:12] <alterjsive> any other logs ?
[07:13] <alterjsive> I'll add /var/log/dmesg too
[07:13] <sladen> alterjsive: boot log, dmesg, etc.  Something will say *why* this boot up has gone to busybox (and without logs the going to busybox is the only we have indication here of anything non-normal here...)
[07:27] <alterjsive> sladen: I don't see anything unusual in the boot.log
[07:28] <alterjsive> https://paste.ubuntu.com/p/vKFz4rc95y/
[07:29] <alterjsive> dmesg is not a log I found out :)
[07:29] <alterjsive> I just copied all files in my log dir
[07:34] <alterjsive> sladen: https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1794318 i've added my logfiles
[07:34] <ubottu> Launchpad bug 1794318 in linux (Ubuntu) "unable to boot after installing vagrant?" [Undecided,Incomplete]
[07:35] <alterjsive> all of them
[07:43] <alterjsive> I'm sure you guys are busy , I should move this conversation to another channel right?
[07:46] <sladen> alterjsive: well here is good.  keeps all the IRC logs in one place
[07:47] <alterjsive> did I say that right. an initfs update?
[07:48] <alterjsive> previous kernel version it works until another initfs update
[07:49] <alterjsive> my current kernel is working, can I make a backup ? in case I install something which does a update-initramfs ?
[07:50] <alterjsive> I need to continue my work now(deadline)
[07:53] <sladen> alterjsive: tihs syslog has four bootups in it.  Which are the good ones, and which are the bad ones
[07:53] <sladen> this
[07:55] <sladen> alterjsive: "my current kernel is working"  <-- *which* (exact version) kernel is working?
[07:56] <sladen> alterjsive: the syslog shows 3x boots with "(Ubuntu 4.15.0-29.31-generic 4.15.18)" and 1x boot with "(Ubuntu 4.15.0-34.37-generic 4.15.18)"
[07:57] <sladen> alterjsive: $ grep 'Linux version' log/syslog
[08:03] <alterjsive> grub says my latest kernel is  4.15.0-34-generic and the one i'm using now is 4.15.0-29-generic
[08:04] <sladen> alterjsive: so 4.15.0-34-generic is the *non*-working
[08:04] <alterjsive> yes
[08:04] <sladen> thank you
[08:04] <alterjsive> I was thinking about reinstalling it
[08:04] <sladen> and this is nothing to do with vagrant?
[08:05] <sladen> the *only* change is switching from kernel 4.15.0-29-generic (working) and rebooting with 4.15.0-34-generic (not working)
[08:05] <alterjsive> sladen: I'm under the impression that a update-initramfs breaks it, just a gut feeling
[08:06] <sladen> alterjsive: please confirm, that to replicate this, the *only* action needed is selecting one kernel, vs. the other kernel in Grub on bootup
[08:07] <sladen> s/replicate/reproduce this bug/
[08:07] <alterjsive> sladen: yes
[08:07] <sladen> thank you
[08:08] <sladen> alterjsive: and that in neither case "safe mode" is being selected in Grub
[08:08] <alterjsive> sladen: yes
[08:10] <alterjsive> 1 sec I'll be more specific. After installing vagrant it did a update-initramfs which broke booting from kernel 4.15.0-34-generic when I switched back, using the grub menu, to a previous kernel  4.15.0-29-generic I could still boot without problems
[08:11] <alterjsive> any objections to sudo apt-get install --reinstall linux-image-generic linux-image
[08:12] <alterjsive> sladen: or do you need anyting more from me for debugging purposes?
[08:12] <sladen> alterjsive: please upload the working and non-working initramfs
[08:13] <alterjsive> sladen: how?
[08:13] <sladen> alterjsive: same way the lof files were uploaded
[08:13] <sladen> alterjsive: same way the log files were uploaded
[08:14] <alterjsive> where do they live?
[08:15] <sladen> alterjsive: ls -l /dev/*init*{4.15.0-29,4.15.0-34}
[08:15] <sladen> alterjsive: ls -l /boot/*init*{4.15.0-29,4.15.0-34}   sorry
[08:22] <alterjsive> sladen: it's +-100MB, can I post it on launchpad?
[08:22] <alterjsive> together
[08:23] <sladen> alterjsive: or at least post the output of ls -l for the moment
[08:24] <alterjsive> sladen: https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1794318/+attachment/5193066/+files/initramfs.tar.gz
[08:24] <ubottu> Launchpad bug 1794318 in linux (Ubuntu) "unable to boot after installing vagrant" [Undecided,Incomplete]
[08:24] <alterjsive> ls -l /boot/*init*
[08:24] <alterjsive> -rw-r--r-- 1 root root 54060073 sep 26 08:20 /boot/initrd.img-4.15.0-29-generic
[08:24] <alterjsive> -rw-r--r-- 1 root root 59276945 sep 26 08:36 /boot/initrd.img-4.15.0-34-generic
[08:27] <alterjsive> sladen: my best guess is that when I install virtualbox, I will break my current kernel aswell
[08:28] <alterjsive> because it updates initramfs
[08:29] <alterjsive> sladen: can I try to reinstall the latest kernel now?
[08:29] <sladen> alterjsive: just before "Log ended: 2018-09-26  08:36:21" there is "Processing triggers for initramfs-tools (0.130ubuntu3.3)" "update-initramfs: Generating /boot/initrd.img-4.15.0-34-generic"
[08:29] <sladen> alterjsive: yes
[08:30] <alterjsive> sladen: I see, I don't know the implectations of that
[08:30] <sladen> alterjsive: this is a place were only "initrd.img-4.15.0-34-generic" is updated, and not the older one
[08:31] <sladen> alterjsive: did 4.15.0-34 work *before* the initramfs was updated?
[08:34] <alterjsive> sladen: yes
[08:34] <sladen> righht...
[08:34] <alterjsive> I only got the busybox after i installed vagrant
[08:34] <alterjsive> sudo apt-get install --reinstall linux-image-4.15.0-34-generic
[08:34] <alterjsive> brb reboot
[08:37] <alterjsive> sladen: no luck, still busybox
[08:39] <sladen> alterjsive: and if you remove   apt-get remove --purge vagrant  ?
[08:39] <alterjsive> sladen: won't it try to remove it from the current kernel?
[08:40] <alterjsive> ok maybe it's cheaper to buy a new laptop with hardware raid
[08:41] <alterjsive> sorry bit frustrated
[08:41] <alterjsive> apt-get remove --purge vagrant, brb reboot
[08:51] <alterjsive> sladen: https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1794318/+attachment/5193074/+files/cannot%20activate%20member%20md127.jpg
[08:51] <ubottu> Launchpad bug 1794318 in mdadm (Ubuntu) "unable to boot after installing vagrant" [Undecided,Incomplete]
[08:52] <alterjsive> sladen: looks like it's a fake raid issue
[08:53] <alterjsive> sladen: luckally I can still boot using the previous kernel
[08:55] <sladen> alterjsive: from the working kernel.  Please upload   cat /proc/mounts
[08:56] <sladen> alterjsive: thank you for the screenshot of the actual error message
[09:00] <alterjsive> https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1794318/comments/11
[09:00] <ubottu> Launchpad bug 1794318 in mdadm (Ubuntu) "unable to boot after installing vagrant" [Undecided,Incomplete]
[09:07] <sladen> ...
=== ghavil is now known as Guest58840
[09:09] <sladen> alterjsive: looking at this tarball from earlier containing logs, the syslog contains these four boots, including one of -34
[09:09] <alterjsive> sladen: I tried installing kernel 4.15.0-33-generic, but I get a busybox
[09:09] <sladen> alterjsive: if the mirror/raid away on mounted on root had not booted, then the syslog from the boot would not have saved.  So this boot of -34 appears to have worked
[09:10] <sladen> s/away/array/
[09:11] <alterjsive> sladen: when I installed 33, I tried to reboot, but I had to do a hard shutdown using the button because it hung on a stop job for WPA supplicants, i don't know if this is significant
[09:12] <alterjsive> this used to happen all the time
[09:12] <alterjsive> lately
[09:12] <alterjsive> sladen: yes it worked, before I installed vagrant
[09:14] <sladen> alterjsive: so 08:20 boot with -29, reboot to -34 at 08:30, install vagrant at 08:36, reboot, (reboot to -34 failed, no logs), reboot at 08:38 to -29, reboo at 08:59 to -29
[09:16] <alterjsive> sladen: looks like it
[09:19] <sladen> alterjsive: please upload the output of  find /etc/initramfs-tools/ -ls
[09:21] <alterjsive> sladen: I just installed virtualbox, I didn't see update-initramfs
[09:21] <alterjsive> sladen: https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1794318/comments/12
[09:21] <ubottu> Launchpad bug 1794318 in mdadm (Ubuntu) "unable to boot after installing vagrant" [Undecided,Incomplete]
[09:24] <alterjsive> Setting up virtualbox-dkms (5.2.10-dfsg-6ubuntu18.04.1) ...
[09:24] <alterjsive> Loading new virtualbox-5.2.10 DKMS files...
[09:24] <alterjsive> Building for 4.15.0-29-generic 4.15.0-34-generic
[09:24] <alterjsive> Building initial module for 4.15.0-29-generic
[09:25] <alterjsive> just so you know
[09:30] <cpaelzer> doko: didrocks: I'd be happy if one of you here in my TZ could do a quick check if bug 1794219 is really ok (I found nothing) before I grant the MIR Team ack
[09:30] <ubottu> bug 1794219 in ledmon (Ubuntu) "[MIR] ledmon" [Undecided,Confirmed] https://launchpad.net/bugs/1794219
[09:34] <cpaelzer> would you run this in valgrind to check for "Incautious use of malloc/sprintf" of the checks in https://wiki.ubuntu.com/MIRTeam ?
[09:37] <cpaelzer> II did, but just want to learn on best practise for these
[10:05] <alterjsive> sladen: could you change the status back to posted? https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1794318
[10:05] <ubottu> Launchpad bug 1794318 in mdadm (Ubuntu) "unable to boot after installing vagrant" [Undecided,Incomplete]
[10:06] <alterjsive> sladen: it's complete now right?
[10:09] <sladen> alterjsive: did installing virtualbox cure things?
[10:10] <alterjsive> sladen: I don't dare to reboot
[10:10] <alterjsive> I will reboot the end of the day after my work
[10:10] <LocutusOfBorg> what is the virtualbox problem?
[10:10] <LocutusOfBorg> if none, nevermind
[10:11] <alterjsive> LocutusOfBorg: nothing yet, thx
[10:13] <alterjsive> sladen: if I want mirroring on linux and to use my self encrypting drive encryption. Should I buy a laptop with hardware raid? is this even possible?
[10:13] <alterjsive> sorry a bit off topic I know
[10:14] <alterjsive> ok I'll reboot, here goes nothing
[10:15] <mwhudson> cpaelzer, rbasak: hey can i ping you guys to get a new package imported?
[10:15] <mwhudson> cpaelzer, rbasak: in particular, rustc (it's a little large)
[10:20] <alterjsive> sladen: no lock, luckally I can still boot
[10:20] <alterjsive> on 27
[10:20] <alterjsive> 29*
[10:27] <alterjsive> sladen: are you planning to pick this bug up soon, just for my own indication?
[10:39] <cpaelzer> mwhudson: I can give it a try
[10:39] <cpaelzer> mwhudson: do you need a one shot, or a regular re-import?
[10:41] <cpaelzer> mwhudson: I'll do a --no-push --active-series-only first to check if it is affected by any odd issues before going for all of the history
[10:41] <sladen> alterjsive: can you run  mdadm --assemble --scan  at some point
[10:41] <mwhudson> cpaelzer: regular would be awesomest
[10:42] <mwhudson> (can you not just import all of ubuntu already!! ;-p)
[10:42] <alterjsive> sladen: ok, will try it tonight, thx
[11:15] <sladen> xnox: this upload of mdadm (4.1~rc1-3~ubuntu18.04.1) ... https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1779685
[11:15] <ubottu> Launchpad bug 1779685 in mdadm (Ubuntu Bionic) "[18.04.1] Backport support for Intel VROC arrays in mdadm" [High,Fix released]
[11:15] <doko> cpaelzer: looks good
[11:16] <sladen> xnox: (might) be the cause of  https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1794318  slangasek in particular referred to some udev rules from upstream being pulled into the Ubuntu package
[11:16] <ubottu> Launchpad bug 1794318 in mdadm (Ubuntu) "unable to boot after installing vagrant" [Undecided,Incomplete]
[11:19] <sladen> xnox: "IMSM RAID geometry validation failed." and "platform does not support raid1 with 2 disks" + initramfs emergency shell is the result using an initramfs with this copy of mdadm
[11:23] <cpaelzer> doko: the bug update or the pastebin with the changes?
[11:32] <sladen> alterjsive: mdadm --detail-platform  also please later
[11:33] <alterjsive> sladen: ok, will do
[11:36] <rbasak> mwhudson: I'll try an import and see how it goes
[11:45] <cpaelzer> rbasak: will you run the real import then, that would be nice
[11:45] <cpaelzer> rbasak: if my faster already started version stumble ssomewhere I'll let you know
[11:48] <rbasak> cpaelzer: I'm running it (in the screen) for real. If it succeeds it'll push.
[11:48] <rbasak> cpaelzer: though it's important to then add it to the whitelist so that it keeps up afterwards
[11:50] <xnox> sladen, i am sad =/
[11:52] <xnox> sladen, disabling mdadm raid in the bios screens (typically Ctrl+I) and booting the drives directly might be an option.
[11:52] <sladen> xnox: currently the user is using their previous initramfs (with previous copy of mdadm included)
[11:53] <xnox> sladen, however, the install is done on top of intel raid.... as per partman
[11:53] <xnox> sladen, detail-platform would be useful, as well as the state/healthiness of the array from bios. quite likely there are unclean shutdowns and thus array is out of sync.
[11:54] <sladen> xnox: can you give a list of useful debugging commands for the user to run?
[11:55] <xnox> sladen, this user is interesting....
[11:55]  * xnox is reading logs
[11:58] <xnox> sladen, specifically they forced mdadm installation from kubuntu desktop
[12:03] <sladen> xnox: tarball::log/apt/history.log  Start-Date: 2018-09-26  08:35:14  Commandline: apt-get install -y vagrant  ... mdadm:amd64 (4.1~rc1-3~ubuntu18.04.1, automatic).   What are you seeing?
[12:27] <xnox> sladen, i'm reading installer logs, as to how the system installation was done.
[12:27] <cpaelzer> rbasak: it ran a while without issues and is done (with the reduced set)
[12:27] <cpaelzer> rbasak: so your run should hoefully work as well
[12:29] <alterjsive> did you guys say anything about sed's and raid btw?
[12:29] <alterjsive> I know i'm a bit ambitious in my IT demands :)))
[12:40] <cpaelzer> rbasak: thanks for all the MP reviews!
[12:47] <rbasak> More to come :)
[12:51] <jbicha> cpaelzer: one comment on your paste: Ubuntu bugs easily get into Confirmed status so I consider New and Confirmed to be equivalent
[13:21] <tych0> smoser: another aws noob question for you if you have a moment, I'm getting, "ClientError: Unsupported configuration: Multiple etc directories found"
[13:22] <tych0> my root filesystem is an lvm thin pool, with several thin pools underneath it (all of them have /etc)
[13:23] <tych0> there are some other lvm volume too (none of them have /etc), but in any case, the one that's specified as / in grub does have a reasonable /etc and is a thin pool. do you know if this is supported? or is something else going on?
[13:34] <cpaelzer> jbicha: yeah, since confirmed has no meaning yet we can define them as "the same" for this
[13:34] <cpaelzer> jbicha: updated in the draft thanks
[13:36] <cpaelzer> updated draft http://paste.ubuntu.com/p/HZXPnGk5sr/
[13:42] <smoser> tych0: is this pvm ?
[13:42] <smoser> hvm should use grub2 and grub2 i would think might have issues, but such issues would not be specific to ec2
[13:43] <smoser> pvm uses grub-legacy-ec2 which writes /boot/grub/menu.lst for a pv-grub loader to load.
[13:43] <smoser> that is hacky and generally deprecated, and i don tknow of the support for lvm volumes there.
[13:44] <tych0> smoser: i don't know the difference :)
[13:44] <tych0> smoser: i just used image-import by following the default docs
[14:03] <smoser> tych0: its probably hvm then.
[14:04] <smoser> but if that is the case hten you should most likelyi be able to reproduce outside ec2
[14:04] <tych0> smoser: sadly, the image boots fine other places
[14:04] <didrocks> cpaelzer: yeah, +1 for me, double checking by security is needed as you set
[14:06] <tych0> and it's not that it doesn't boot here, it's that aws won't import it. presumably because it's looking through all th ethin pools and seeing multiple /etc dirs
[14:06] <smoser> can you do a ec2-describe-images call on it ?
[14:06] <smoser> i dont know their "import" process. i only have familiarity with the bare bones path
[14:06] <smoser> where you push bytes and then try to boot them.
[14:07] <tych0> oh
[14:07] <tych0> well that sounds interesting :)
[14:07] <tych0> how do i do that?
[14:11] <tych0> (smoser)
[14:12] <tych0> smoser: i don't think i can do describe-images, because it never got successfully imported (the error i pasted above is from the import job)
[14:12] <smoser> oh.
[14:13] <smoser> its possible its just a buggy process flagging somethign that would otherwise work
[14:13] <tych0> yeah, i'm wondering if that's what's happening
[14:13] <tych0> not a lot of info on the interwebs about this, though
[14:14] <tych0> admittedly, a nested thinpool for / is a strange situation, i suppose
[15:10] <hallyn> smoser: btw is it safe to assume that the code for generating and uploading the virtual images to aws is in lp?
[15:13] <infinity> hallyn: Ish.
[15:14] <infinity> hallyn: livecd-rootfs is responsible for all the heavy lifting there, but for public clouds, there's also a secret sauce branch that does some extra bits that aren't public.
[15:15] <infinity> hallyn: Oh, and the "and uploading" part is definitely not public.  Though, I'm not sure there's any strategic reason why it couldn't be, just that it's not, AFAIR.
[15:15] <hallyn> ok.  a matter of "it would cost time/money to figure out whether there's secret sauce or not" probably
[15:16] <hallyn> thanks infinity
=== ledeni_ is now known as ledeni
=== choopm is now known as Guest80724
[23:29] <Sven_vB> hi! is this the right place for help with debugging ubiquity?