[04:05] <babinlonston> I Want to allow one person to check my server for some issue, and he needs to check the server in root privileges, how can i give him login detail's using ssh key file
[08:21] <Tazmain> hi all, is there anyone here that has worked with squid before ?
[09:31] <hadifarnoud> is it a bad idea to have gzip compression turned on for low mem servers?
[09:44] <Oplex> anyone successfully linked deluge to sickbeard ?
[10:09] <maswan> So, how do I stop my trusty server from randomly renaming the p2p2 NIC to "rename5" on occasional reboots?
[10:10] <sarnold> maswan: first, please file a bug report about that, I know I've seen some complaints about it but I don't think they've been written down anywhere :/
[10:10] <maswan> sarnold: well, https://bugs.launchpad.net/ubuntu/+source/biosdevname/+bug/1284043 seems to cover it
[10:10] <sarnold> maswan: probably uninstalled the biosdevname package will do it, but that'll change all your names
[10:11] <maswan> but as usual there seems to be a fair bit of blahblah in the launchpad bug too
[10:15] <sarnold> maswan: heh, yeah, everyone has a story rather than just stating the bug. oh well. it's documented. :)
[10:15] <sarnold> maswan: so if you d'nt mind the eth0 eth1 eth2 etc names, you can uninstall the biosdevname package and rebuild your initramfs
[10:16] <maswan> sarnold: yeah. I kind of like the new names, but not to the point where drbd won't work because the internal networking went away because the nic suddenly was "rename5"
[10:17] <sarnold> maswan: yeah. it seems surprising it wasn't found and fixed earlier.
[11:11] <Pupeno> Should users used for running a web app and for deploying an app by system or regular users?
[16:30] <Pupeno> A custom upstart job I wrote is not starting and it's not writing anything to /var/log/upstart/jobname.log. Any other log files I could check for errors?
[16:37] <RoyK> Pupeno: perhaps /var/log/messages
[16:38] <Pupeno> RoyK: there's no such file in Ubuntu, at least 12.04.
[16:38] <RoyK> erm
[16:38] <RoyK> s/messages/syslog/
[16:38] <RoyK> sorry
[16:39]  * RoyK is using a lot of RHEL/CentOS at work and confuses things a bit sometimes
[16:40] <Pupeno> RoyK: no, nothing there :(
[16:41] <RoyK> Pupeno: then I don't know - sorry
[18:44] <dasjoe> maswan: have you checked /etc/udev/rules.d/70-persistent-net.rules?
[19:15] <soren> Suppose I have a number of outstanding security updates. Does anyone have any good ideas for how I can list the corresponding USN's?
[19:21] <soren> jdstrand: ^ Any good ideas?
[19:37] <XKM> Hello
[19:38] <XKM> I have an HP n54l with 8GB of RAM and an radeon HD540
[19:39] <XKM> the CPU is AMD Turion™ II Model Neo N54L
[19:39] <XKM> should i be running 64 bit or 32?
[19:39] <XKM> I think 64...am i wrong?
[19:39] <XKM> Its going to be a NAS btw if that makes any different
[19:43] <XKM> anyone?
[19:43] <RoyK> sec
[19:43] <XKM> ok
[19:43] <RoyK> it's a 64bit cpu AFAICS
[19:43] <RoyK> http://www.cpu-world.com/CPUs/K10/AMD-Turion%20II%20Neo%20N54L%20-%20TEN54LSDV23GME.html
[19:44] <RoyK> with 8GB RAM, it'd be best to run it in 64bit mode
[19:44] <XKM> Syeet
[19:44] <XKM> Sweet thanks!
[19:44] <XKM> How stable is the new 14.04?
[19:45] <XKM> Would i be better of witht he 12.04?
[19:48] <lorenz> Hey, I got a problem with my 14.04 Server, it won't boot anymore. It seems like that all upstart jobs are failing, because when I run it in debug mode, they start at a 5-second interval (which is the default for upstart when services fail). Does anybody know what's going on?
[19:49] <XKM> lol
[19:49] <lorenz> what lol?
[19:51] <TJ-> lorenz: disk corruption?
[19:52] <XKM> sorry i am thinking between 12.04 and 14.04 ubuntu server
[19:53] <lorenz> I checked my disk with btrfs scrub, btrfsck and rebalanced. Everything worked perfectly and no errors showed.
[19:53] <TJ-> lorenz: is the root file-system mounted read-only?
[19:54] <lorenz> no, I don't think so. At least in /etc/fstab it's mounted rw and the root subvolume is flagged rw.
[19:55] <TJ-> lorenz: Do you have a shell terminal you can type at?
[19:55] <lorenz> no, the shell doesn't work yet
[19:55] <lorenz> it fails when starting the plymouth-bridge
[19:55] <TJ-> lorenz: have you tried starting with "init=/bin/bash" so you can at least explore?
[19:56] <lorenz> oh, thank you
[19:56] <lorenz> I'll try that
[19:57] <XKM> so what about 12.04 vs 14.04?
[19:57] <XKM> which one should i go for knowing its going to be a NAS box
[19:58] <lorenz> I saw something: when i typed mount in, it showed me a warning that /etc/mtab is not writeable (read-only filesystem). But in the flags for / it shows rw. How can I find out if the disk is really getting mounted rw or ro?
[20:00] <TJ-> lorenz: "cat /proc/mounts" will show what is actually mounted as far as the kernel is concerned
[20:01] <lorenz> It is actually read-only.
[20:01] <lorenz> Can I just remount it rw or would that break something?
[20:02] <TJ-> lorenz: try to determine why
[20:02] <lorenz> I remounted it with mount -o remount,rw / and now it shows in /proc/mount as rw
[20:02] <TJ-> lorenz: has any file-system run out of space?
[20:03] <lorenz> no, all Fs have plenty of space
[20:03] <TJ-> lorenz: I wonder if the initrd.img could be corrupted
[20:04] <lorenz> Wouldn't update-initramfs -u fix that? Because I ran that command in a chroot and nothing changed.
[20:04] <TJ-> lorenz: it's the initrd.img's "/init" script that handles the root pivot from the kernel's rootfs to the real root
[20:04] <lorenz> ok
[20:04] <TJ-> lorenz: Yes, it should, assuming the /boot/ volume is good
[20:05] <TJ-> lorenz: Is /var/log/ in the root-filesystem?
[20:05] <lorenz> Yes, it is, but the ls command took ~1s to complete
[20:06] <TJ-> lorenz: "tail -f /var/log/kern.log" see what is being written
[20:06] <TJ-> lorenz: is all mass-storage local on the server ?
[20:06] <lorenz> a lot of retire_playback_urb: 832 callbacks suppressed
[20:07] <lorenz> yes, all storage is local (I currently run an image of the server on my computer in QEMU-KVM, so the hardware isn't the issue)
[20:08] <TJ-> lorenz: what USB devices connected?
[20:08] <lorenz> no USB devices connected
[20:08] <TJ-> Any sound devices?
[20:09] <lorenz> ac97
[20:09] <lorenz> default of qemu, I can remove it if that helps
[20:09] <TJ-> "retire_playback_urb" is a message from sound/usb/pcm.c
[20:09] <lorenz> ok
[20:09] <TJ-> hang on... this server is running in qemu-kvm ?
[20:11] <lorenz> The original server who failed to boot is physical. I took an image of the ssd on my local computer to help debugging after I failed to bring it back up. It behaves exactly the same in QEMU-KVM. But when it works again, I plan to write the modified image back to the boot ssd of the physical server.
[20:15] <TJ-> lorenz: That is *strange*
[20:16] <TJ-> lorenz: How large is the root file-system block device?
[20:16] <lorenz> The Image is 60 GB (It's taken from a Kingston KC300 60GB)
[20:19] <TJ-> lorenz: I'm wondering if it's a btrfs bug. If you have the space, I am thinking it might be worth creating a new blockdevice and formatting it ext4, then rsync the data from the current root-file system, and then modify things enough to try starting from that.
[20:20] <TJ-> The fact you had a read-only mount even on QEMU seems 'wrong' unless there is something wrong with the btrs volumes
[20:21] <lorenz> Well, I got a few terabytes on my computer, so I'm just gonna create  a new image with 60GB and copy stuff over.
[20:23] <lorenz> Why could I mount it rw if there's something wrong with the volume? The kernel didn't complain (and not even throw a warning in the logs).
[20:25] <TJ-> I have *no* idea, but I'm trying to reduce the problem space through eliminating potential issues... the more you can do that, the easier it is to focus on the real problem
[20:25] <lorenz> Something else: when I am in bash (from init=/bin/bash), can I then launch the normal init after I remounted / as rw?
[20:30] <lorenz> btrfs -> ext4 copy job is running
[20:30] <sarnold> lorenz: maybe. if you want to try, try exec /sbin/init
[20:30] <lorenz> ok, I'm gonna try that on another image
[20:34] <lorenz> ok, exec /sbin/init waits 60 seconds, afterwards it prints init: plymouth-upstart-bridge main process (196) terminated with status 1. So this fails too. I'm now waiting for the copy process.
[20:35] <sarnold> lorenz: oh, drat. :( thanks for trying, I realize now I'd never known if that could or should work, or why not if it doesn't..
[20:37] <Diplomat> hey guys, i installed 64bit ubuntu to my server.. and for some random reason it shows i686
[20:37] <Diplomat> any ideas ?
[20:37] <lorenz> sarnold: well, it was the same error as without /bin/bash, so it could work if it works without init=/bin/bash
[20:38] <lorenz> Diplomat: what is the output of uname -a
[20:38] <Diplomat> Linux os-comp2 3.11.0-22-generic #38~precise1-Ubuntu SMP Fri May 16 20:50:12 UTC 2014 i686 i686 i386 GNU/Linux
[20:38] <Diplomat> I'm more than 100% that I installed Ubuntu 12.04.4 64 bit version
[20:39] <ikonia> looks like a 32bit kernel
[20:39] <ikonia> seems %99.9 certain
[20:39] <lorenz> Diplomat: That's a 32bit ubuntu, I can't tell you more.
[20:39] <Diplomat> that's what im asking
[20:39] <Diplomat> how can it be 32bit when i used 64bit iso.. :/
[20:39] <ikonia> you didn't
[20:39] <ikonia> you used 32bit
[20:39] <lorenz> A 64bit ISO can't install a 32bit OS, it doesn't have the packages.
[20:40] <Diplomat> http://releases.ubuntu.com/12.04/ubuntu-12.04.4-server-amd64.iso is this 32 bit ?
[20:40] <lorenz> You have a 32bit OS
[20:40] <ikonia> Diplomat: that's not what you used to install
[20:40] <ikonia> Diplomat: please run "file /bin/bash" please
[20:40] <Diplomat> haha lol now how can you tell me that
[20:40] <Diplomat> I have installed ubuntu at least 50-60 times
[20:40] <ikonia> Diplomat: please run "file /bin/bash" please
[20:40] <RoyK> Diplomat: amd64 is 64bit - use the 32bit one
[20:41] <lorenz> RoyK: He wants 64 bit
[20:41] <Diplomat> ikonia, I checked it already it's 32
[20:41] <Diplomat> it shows
[20:41] <ikonia> Diplomat: ok, so you can't have used a 64bit install media
[20:41] <Diplomat> http://pastebin.com/rzkqrVQH
[20:41] <ikonia> the 64bit install media doesn't have the 32bit software on it
[20:41] <ikonia> so you have to have either a.) downloaded/installed the wrong media b.) the ubuntu site gave you the wrong media, but renamed it to 64bit media name (unlikley)
[20:41] <lorenz> ELF 32-bit LSB executable
[20:42] <ikonia> that's the only two options
[20:42] <Diplomat> ikonia, please, there is no way that I used 32 bit iso when I used that URL to download it to my desktop and then used that exact file for installing
[20:42] <lorenz> It could also be that the software to write the USB stick failed and left an old 32bit installer
[20:42] <Diplomat> I wouldnt be here making a joke of myself if i wouldnt be that sure
[20:42] <ikonia> Diplomat: repeating it won't change the facts you have a 32bit os
[20:42] <ikonia> Diplomat: so what do you want to do / happen now ?
[20:43] <RoyK> lorenz: 64bit doesn't work with 32bit hww
[20:43] <RoyK> lorenz: 64bit doesn't work with 32bit hw
[20:43] <Diplomat> nothing, i was just surprised and i wanted to ask about this issue
[20:43] <ikonia> well, as you are the only person who's ever reported this issue, it's most likley (but not fact) a simple user error mistake
[20:44] <ikonia> it does happen,
[20:44] <lorenz> RoyK: yes I know. But if it the script failed (didn't write anything on the stick), and there was another 32bit installer already on there.
[20:44] <ikonia> Diplomat: stick the install media in now, and config the volume label on it
[20:44] <lorenz> Diplomat: What installation media did you use?
[20:44] <guntbert> s/config/confirm/
[20:45] <Diplomat> I just booted it and installed like I have always done it
[20:45] <Diplomat> and if I failed some where.. then how lol
[20:45] <ikonia> check the volume label on the disk
[20:45] <lorenz> Was it an USB stick, CD, DVD or something else?
[20:45] <Diplomat> http://puu.sh/92ub6.jpg
[20:46] <ikonia> Diplomat: that's an iso image
[20:46] <Diplomat> just that file to empty VirtualBox
[20:46] <ikonia> Diplomat: not the boot media
[20:46] <ikonia> ahhh a virtual box install, sorry, I missed that earlier
[20:46] <lorenz> Boot order in VirtualBox?
[20:47] <Diplomat> What I did was I created a new vm.. and then it booted then it asked for installer I picked that ISO and then installed it and then it restarted and then i did some stuff there and then i realized that something is wrong and did "uname -a"
[20:47] <Diplomat> and voila
[20:47] <Diplomat> I dont have any USB or CD/DVDs connected
[20:48] <ikonia> Diplomat: boot the install media and check it with uname -a again
[20:48] <Diplomat> I was messing with this: http://docs.openstack.org/icehouse/install-guide/install/apt/content/nova-compute.html
[20:49] <lorenz> Then get an Ubuntu Cloud install image
[20:49] <Diplomat> http://docs.openstack.org/icehouse/install-guide/install/apt/content/basics-packages.html
[20:49] <Diplomat> this too
[20:49] <lorenz> no
[20:50] <lorenz> http://www.ubuntu.com/download/cloud
[20:50] <lorenz> this one
[20:50] <lorenz> Has OpenStack preinstalled
[20:50] <Diplomat> Yea, but that's why I didnt get it because I wanted to try myself
[20:51] <Diplomat> and it appears I was able to do the impossible lol
[20:51] <Diplomat> well
[20:51] <Diplomat> I'm gonna download that same iso again and try again
[20:51] <Diplomat> maybe i'm able to get it 16bit
[20:52] <sarnold> 8bit best bit
[20:53] <Diplomat> I'm trying man
[20:53] <lorenz> You will then have 256bits of ram available :D
[20:53] <Diplomat> lol
[20:54] <Diplomat> I might be able to open hello world text file then
[20:54] <Diplomat> that's pretty cool too
[21:00] <cloudman> Hi, just installed 14.04 for the first time on a server and activated root but sftp is saying no all the time...
[21:00] <Gallomimia> is your ssh server running?
[21:01] <cloudman> fresh install, got in as user and activated root etc so yes
[21:01] <Gallomimia> sftp runs over ssh. i'm sure your client is configured to get in that way too
[21:01] <lorenz> Is "Subsystem sftp /usr/lib/openssh/sftp-server" present in /etc/ssh/sshd_config
[21:02] <cloudman> tried about 4 times, checking pwd and sftp just keeps ignoting it??
[21:02] <Gallomimia> by the way sftp as root is pretty brutal. recommend using a different username than that
[21:02] <cloudman> Gallomimia: I need to upload a file to root dir
[21:02] <cloudman> I need root
[21:02] <lorenz> SFTP as root is maybe not very secure, but it works (I tried it once, I needed to move a whole lot of files to root)
[21:03] <cloudman> give me root everytime, let me take care of security, yet to be hacked since 1995
[21:04] <cloudman> is root sftp possible in 14.04??
[21:04] <Gallomimia> just as secure as anything. just can be easy to overwrite things you didn't want. anyway, viewing the file at /etc/sshd_config will let you look at a few options for the server
[21:04] <lorenz> Yes, it is possible
[21:04] <cloudman> how?  :)
[21:04] <lorenz> Should work out of the box
[21:04] <Gallomimia> having not run 14.04 yet im unsure. hm. do i have any ubuntu servers left?
[21:04] <cloudman> will do a fresh install but its having none of it here
[21:04] <lorenz> If you set your root password to something else than an empty hash
[21:05] <lorenz> Look at /etc/ssh/sshd_config
[21:05] <lorenz> Upload the contents somewhere on pastebin or similar
[21:06] <Gallomimia> there must be something turning off sftp for root if you having problems
[21:06] <Gallomimia> or your client is not configured properly
[21:06] <cloudman> beats me
[21:06] <cloudman> will check it out
[21:06] <TJ-> check "/var/log/auth.log"
[21:17] <cloudman> lorenz: http://paste.ubuntu.com/7524412/
[21:18] <lorenz> cloudman: that's ssh_config, not sshd_config
[21:18] <cloudman> ah, sry too much wine tonight
[21:19] <cloudman> http://paste.ubuntu.com/7524436/
[21:20] <cloudman> first time with 14.04
[21:20] <cloudman> all good fun anyway
[21:22] <cloudman> lorenz: http://paste.ubuntu.com/7524436/
[21:23] <lorenz> cloudman: Subsystem sftp /usr/lib/openssh/sftp-server is there, so the server should have no issues. Could you also post /var/log/auth.log?
[21:24] <cloudman> Yes as Tj mentioned checked that and it says pwd failure???  keyboard setup maube??
[21:24] <cloudman> maybe
[21:25] <cloudman> sshd[1357]: Failed password for root from
[21:26] <cloudman> how can that be though, I know my typing is a bit off tonight but...
[21:27] <cloudman> thing is I tried with a simple 123456 as well and got the same, sftp reject
[21:28] <lorenz> cloudman: Is there anything above the failed password line?
[21:30] <cloudman> lorenz:  pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=
[21:30] <lorenz> cloudman: anything above there?
[21:30] <genii> With "PermitRootLogin without-password" aren't you supposed to be using a key?
[21:30] <cloudman> fail2ban not built in to 14.04 is it :)
[21:31] <lorenz> cloudman: It's what genii said
[21:31] <cloudman> genii: not sure, 12.04 user here
[21:32] <lorenz> cloudman: Change line 28 of /etc/ssh/sshd_config to PermitRootLogin yes
[21:32] <cloudman> ok, hangon
[21:34] <cloudman> lol, need root
[21:35] <cloudman> won't let me do it
[21:35] <cloudman> hangon
[21:38] <cloudman> ok, should I restart ssh server
[21:38] <lorenz> cloudman: yes
[21:38] <cloudman> ty
[21:42] <cloudman> Guess what guys
[21:42] <cloudman> in as ./
[21:42] <cloudman> tahnks
[21:42] <cloudman> thanks
[21:42] <cloudman> tanked up as well
[21:42] <cloudman> why does it do that as default
[21:43] <cloudman> lorenz: ty
[21:43] <cloudman> genii: even more ty
[21:43] <cloudman> TJ ty
[21:43]  * genii hands out the coffee and cookies
[21:43] <cloudman> what a pain though
[21:44] <TJ-> it's done deliberately
[21:44] <cloudman> to annoy people?
[21:44] <cloudman> thanks guys, great help and fast response, superb
[21:45] <TJ-> cloudman: If I wanted a user to write to "/" I'd simply do "sudo adduser $USER root && sudo chmod g+w /"
[21:45] <lorenz> TJ-: Copy process is done, I will now try to setup grub
[21:46] <TJ-> lorenz: good luck, I hope it is that; at least you can then get on with fixing the problem
[21:46] <cloudman> 8 core 8 thread, 8Gb Ram DDR3 1Tb HD 35 euro a month anyone want one as a reward ;)
[21:47] <cloudman> not kidding just got a batch in
[21:48] <cloudman> now to see if VM installs fine on 14.04
[21:48] <cloudman> should be a gas
[21:48] <lorenz> cloudman: https://www.hetzner.de/en/hosting/produkte_rootserver/ex40 is 32GB RAM for 49 € per month
[21:48] <cloudman> yeah I have one
[21:49] <cloudman> mine is manged though
[21:49] <cloudman> managed
[21:49] <cloudman> not that anyone here should need a managed onr though I guess ;)
[21:51] <cloudman> talk about preaching to the converted lol
[21:51] <cloudman> thanks guys
[21:55] <cloudman> Virtualmin installer fails on 14.04 hope they sort it soon
[22:00] <TJ-> how did it fail?
[22:01] <genii> !webmin
[22:01] <cloudman> TJ http://paste.ubuntu.com/7524677/
[22:02] <cloudman> wow
[22:02] <TJ-> cloudman: "/root/virtualmin-install.log" ?
[22:05] <cloudman> TJ-: http://paste.ubuntu.com/7524686/
[22:06] <lorenz> TJ-: Unpacking grub-pc-bin in the rescue system is now running for like ~10min with no disk activity
[22:06] <TJ-> lorenz: that's not good
[22:06] <TJ-> lorenz: something very weird with your setups there
[22:07] <TJ-> cloudman: The line "Installing dependencies using command: ..." has the list of packages it was going to install. My guess is that one or more are no longer available or have changed SO version-names in 14.04
[22:07] <cloudman> They are sorting it soon
[22:08] <cloudman> webmin is ok but you just have to love virtualmin its so easy
[22:08] <lorenz> TJ-: It's a normal 14.04 installation, just standard packages and only a few PPAs (Percona, Elasticsearch, Ksplice, Owncloud, Passenger and XtreemFs)
[22:08] <cloudman> https://www.virtualmin.com/node/32970
[22:09] <TJ-> cloudman: I was already there :)
[22:17] <cloudman> :)
[22:17] <cloudman> be nice to get it all sorted
[22:22] <lorenz> TJ-: I'm trying with the boot-repair ISO, my grub-pc-bin package seems broken
[22:23] <TJ-> lorenz: ouch
[22:24] <cloudman> I have some servers that lose boot dir recently??
[22:39] <lorenz> TJ-: boot-repair also requires you to install this package in chroot. It doesn't work there either. But I have been able to attach a strace to the responsible dpkg process. It's writing with the speed of about 4KB/s :D
[22:40] <TJ-> lorenz: This is from the QEMU session?
[22:40] <lorenz> TJ-: It's inside QEMU, yes.
[22:41] <lorenz> TJ-: The Ext4 image is on a SSD, so that's not the issue
[22:41] <TJ-> lorenz: Are you *100%* sure that QEMU is using VT hardware support? It sounds as if its using software emulation
[22:42] <lorenz> TJ-: Everything else is very fast, I rechecked and KVM is enabled and the load on my host is something around 0.1
[22:42] <TJ-> lorenz: then something you've got there is playing silly buggers with you!
[22:42] <ikonia> how are you checking if kvm is being used
[22:43] <lorenz> ikonia: virt-manager shows Hypervisor: kvm
[22:43] <ikonia> lorenz: is the kvm module actually loaded ?
[22:44] <TJ-> lorenz: "ls /dev/kvm" or "kvm-ok"
[22:44] <lorenz> ikonia: lsmod | grep kvm shows kvm_intel and kvm
[22:44] <lorenz> TJ-: kvm-ok: KVM acceleration can be used
[22:45] <TJ-> lorenz: good... one more thing checked off :)
[22:45] <TJ-> lorenz: just to be clear, the host is fine, it's the guest image in the VM or when its run on the original bare-metal server, that is having problems?
[22:45] <lorenz> TJ-: Exactly
[22:46] <TJ-> lorenz: Then it has to be something in the images. Have you managed to start the ext4 clone of the root-file-system ?
[22:47] <lorenz> TJ-: I'm trying to install grub on it right now. The images are the simple .img files, so they shouldn't cause any problems. I need img to write them later back to the SSD.
[22:47] <TJ-> yeah, that's how I do it
[22:48] <lorenz> TJ-: It finished now! :D
[22:49] <Pupeno_> My custom upstart job stopped writing to /var/log/upstart/jobname.log. Any ideas what it could be or how to troubleshoot it?
[22:50] <TJ-> Pupeno_: logrotate not setting the permissions of the new file correctly?
[22:51] <lorenz> TJ-: The install dpkg for grub is now just sleeping (according to strace, it does a nanosleep)
[22:52] <TJ-> lorenz: one thing I can think of, is when hostname tries to be resolved outside the localhost... can affect many commands unexpectedly
[22:52] <lorenz> TJ-: Sorry, that was the top-level dpkg which was waiting for the children. But the child now also writes very slow.
[22:52] <Pupeno_> TJ-: there's no file right now, and it's not being created. I didn't touch the permissions of the parent directory since before it stopped working.
[22:53] <TJ-> Pupeno_: if the log file is missing then ownership and/or permissions will be the issue
[22:53] <lorenz> TJ-: localhost resolves to 127.0.0.1 according to /etc/hosts
[22:53] <Pupeno_> TJ-: that wasn't an issue when the computer was freshly installed, but I would be happy to explore that.
[22:53] <TJ-> lorenz: how about the hostname itself - the name of the machine
[22:54] <lorenz> TJ-: That also resolves to 127.0.1.1, which seems ok to me. Btw, is it correct that the kernel is responsible for executing the calls I see in strace?
[22:54] <Pupeno_> TJ-: created the file as 666 and still nothing got written to it.
[22:55] <TJ-> Pupeno_: I've seen instances of a daemon working fine until logrotate fires off, which changes ownerships and stops the daemon from writing... it can be caused by the daemon dropping privileges/to another unprivileged user-id and can therefore not have root:root access to access the log file
[22:55] <Pupeno_> The job is not starting, something is preventing it, but logs are empty.
[22:55] <TJ-> lorenz: You're seeing the syscalls mostly
[22:55] <TJ-> Pupeno_: Try starting the daemon manually in the foreground with full verbosity/debug enabled
[22:55] <Pupeno_> TJ-: I was working on the app this job is supposed to fire when it stopped working. I don't think it was logrotate, but still, without logs, I can't figure out what's stopping this app from starting.
[22:56] <Pupeno_> TJ-: that doesn't work, it fails, and that's all right. I'll get to that error after I fix upstart.
[22:56] <TJ-> Pupeno_: I'm lost; if the daemon won't start manually, how can upstart start it?
[22:57] <Pupeno_> TJ-: I wrote the app, there's a bug somewhere, which I'll resolve later. Right now, I want to see the error messages in the context of upstart. I'm not expecting upstart to work, I'm expecting upstart to log the error.
[22:59] <TJ-> Pupeno_: You'll need to do "initctl log-priority debug" to see more info in the syslog
[23:00] <Pupeno_> TJ-: still nothing in upstart/projectx.log or syslog.log. Should I look in another log file?
[23:01] <genii> "...in the syslog"
[23:01] <TJ-> Pupeno_: As far as I recall, /var/log/syslog is the place to look
[23:01] <Pupeno_> Should I add that line to my upstart conf file?
[23:05] <TJ-> what line?
[23:06] <Pupeno_> initctl log-priority debug
[23:06] <TJ-> Pupeno_: That's a command to issue directly, as root/sudo
[23:06] <Pupeno_> ok, that's what I did.
[23:07] <lorenz> TJ-: It has now worked, Boot-Repair is installing the new MBR. I'm trying to reboot shortly.
[23:07] <TJ-> lorenz: *fingers* crossed
[23:07] <TJ-> lorenz: but that slowness suggests something is dragging its feet terribly
[23:08] <lorenz> TJ-: Ok, it boots. BUT it is stuck at the same line as before.
[23:09] <lorenz> TJ-: Oh, I forgot to update /etc/fstab
[23:10] <TJ-> lorenz: *tuts*
[23:18] <lorenz> TJ-: Sorry, didn't help. The newly created ext4-machine (without EFI and BTRFS) does exactly the same. It shows init: plymouth-upstart-bridge main process (182) terminated with status 1 and mounts / as read-only
[23:19] <TJ-> lorenz: Then you've got some problem in the initrd.img I'd guess
[23:19] <lorenz> TJ-: Can I reinstall that somehow?
[23:20] <TJ-> lorenz: Before this first started, what was the last (few) sysconfig changes/system updates ?
[23:22] <lorenz> TJ-: I did nothing the day it started. The (physical) machine just stopped responding (I had a SSH session open, but it also didn't respond to pings and ARP-requests). Then I hard-rebooted and it didn't came up.
[23:23] <TJ-> lorenz: I have to start  thinking along the lines of "comprised" and "root-kit"
[23:23] <TJ-> lorenz: s/comprised/compromised/
[23:24] <lorenz> TJ-: It runs Snort, UFW, chkrootkit, OSSEC and is hardened with bastillion, so it's quite unlikely. But it could happen.
[23:26] <lorenz> TJ-: I had no alerts that day. And wouldn't somebody who attacks the machine try to not bring it down, because a disabled machine can't be used to do anything?
[23:26] <TJ-> lorenz: Well look at the facts... you've moved the same image to different hardware, inside a VM, using a different file-system, and it still does the same thing
[23:27] <TJ-> lorenz: extract the initrd and check it for anything 'weird' or 'missing' or 'badly configured'
[23:29] <lorenz> TJ-: Have you got a command ready to do that?
[23:29] <TJ-> lorenz: funny you should say that ;p
[23:30] <lorenz> TJ-: I found one.
[23:31] <TJ-> lorenz: "mkdir /tmp/initrd && pushd /tmp/initrd && zcat /boot/initrd.img-`uname -r` | cpio -i -d "
[23:31] <lorenz> TJ-: Thank you!
[23:32] <lorenz> TJ-: What should I look for?
[23:32] <TJ-> lorenz: $1,000,0000 question!
[23:33] <lorenz> TJ-: Ok :D
[23:33] <TJ-> first check in the conf/ directory and its sub-dirs, read the files, get a feel for what they're saying
[23:33] <TJ-> There should be some mdadm.conf stuff in there if you're using MD
[23:34] <lorenz> TJ-: I use btrfs RAID, so there is no mdadm.conf
[23:34] <lorenz> TJ-: /conf folder seems well, everything makes sense
[23:34] <TJ-> also, check the /libs/ and the lib/
[23:35] <TJ-> grrr, typo... "lib/"
[23:35] <lorenz> TJ-: Found one: klibc-P2s_k-gf23VtrGgO2_4pGkQgwMY.so
[23:35] <lorenz> TJ-: Does that look normal?
[23:36] <TJ-> lorenz: "md5sum:  4152a06877635bf3dc98a3cf6e48bd4a  lib/klibc-P2s_k-gf23VtrGgO2_4pGkQgwMY.so"
[23:37] <TJ-> lorenz: I notice there's "/etc/{lvm,mdadm}/"
[23:37] <lorenz> TJ-: 4152a06877635bf3dc98a3cf6e48bd4a  klibc-P2s_k-gf23VtrGgO2_4pGkQgwMY.so -- seems the same
[23:38] <lorenz> TJ-: I don't use LVM nor MDADM
[23:39] <TJ-> lorenz: OK... how about "cat $(find scripts -type f) | md5sum"          ==  2dc83e509c928077916b5951b8162cd9
[23:39] <TJ-> lorenz: that'll probably be different to mine; I have lvm, mdadm, and cryptsetup
[23:39] <lorenz> TJ-: That's different
[23:40] <TJ-> lorenz: not surprised, that was a lot of files to expect to be the same!
[23:42] <lorenz> TJ-: http://paste.ubuntu.com/7525216/ Are the sums of my scripts dir
[23:43] <TJ-> lorenz: OK, take a look at the "./init" shell script that controls the process. You'll see several function calls of the form "maybe_break XXXX" these are points where the script can be stopped by adding "break=XXXX" to the kernel command-line
[23:43] <lorenz> TJ-: I think I found something: my lib/modules are for 3.13.0-27-generic and my kernel is 3.13.0-24
[23:44] <TJ-> lorenz: In particular, look at "maybe_break mountroot" ... at that point it is about to call "mountroot". If you did "break=mountroot" and then manually executed that call, *maybe* you can find out why it is mounting it read-only
[23:44] <TJ-> lorenz: Really!?!
[23:44] <TJ-> lorenz: Let me extract the same version here
[23:45] <lorenz> TJ-: My newest image in /boot is initrd.img-3.13.0-24-generic, but the contents are 3.13.0-27-generic
[23:46] <TJ-> lorenz: "ls lib/modules/"  = "3.13.0-24-generic"
[23:46] <TJ-> lorenz: That'd sure mess things up!
[23:47] <TJ-> lorenz: "May 26 21:03:48 <TJ->   lorenz: I wonder if the initrd.img could be corrupted"
[23:47] <lorenz> TJ-: your script unpacked my host initrd, that was the problem.
[23:48] <TJ-> lorenz: lol ... I thought you'd adjusted the paths for the VM :) ... oh well, it's never *that* easy!
[23:49] <TJ-> lorenz: so, back to the idea of "break=mountroot"
[23:50] <lorenz> TJ-: ok, i'll see
[23:52] <TJ-> lorenz: If you do that, the first thing once you have the shell prompt is to source the function library with ". scripts/local" and then you can do "mountroot"
[23:52] <lorenz> TJ-: ok
[23:54] <TJ-> lorenz: also, *before* you call "mountroot" ensure there's verbose logging with export quiet="n"; set -x;
[23:56] <lorenz> TJ-: Kernel panic: Illegal number mountroot
[23:56] <TJ-> hmmm!
[23:57] <TJ-> lorenz: Did you see any messages/debug info before that?
[23:58] <lorenz> TJ-: I missed a newline before the mountroot, is now fixed. I'm compressing the next initrd.
[23:58] <TJ-> lorenz: OK
[23:59] <lorenz> TJ-: Nothing happens now (I put a newline between break and mountroot)