=== thumper is now known as thumper-otp
=== FreezingAlt is now known as FreezingCold
=== thumper-otp is now known as thumper
[04:05] <babinlonston> I Want to allow one person to check my server for some issue, and he needs to check the server in root privileges, how can i give him login detail's using ssh key file
=== Ursinha-afk is now known as Ursinha
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
[08:21] <Tazmain> hi all, is there anyone here that has worked with squid before ?
[09:31] <hadifarnoud> is it a bad idea to have gzip compression turned on for low mem servers?
[09:44] <Oplex> anyone successfully linked deluge to sickbeard ?
[10:09] <maswan> So, how do I stop my trusty server from randomly renaming the p2p2 NIC to "rename5" on occasional reboots?
[10:10] <sarnold> maswan: first, please file a bug report about that, I know I've seen some complaints about it but I don't think they've been written down anywhere :/
[10:10] <maswan> sarnold: well, https://bugs.launchpad.net/ubuntu/+source/biosdevname/+bug/1284043 seems to cover it
[10:10] <uvirtbot> Launchpad bug 1284043 in biosdevname "udev renaming the same hardware network i/f to different name, breaks networking and firewall" [High,Confirmed]
[10:10] <sarnold> maswan: probably uninstalled the biosdevname package will do it, but that'll change all your names
[10:11] <maswan> but as usual there seems to be a fair bit of blahblah in the launchpad bug too
[10:15] <sarnold> maswan: heh, yeah, everyone has a story rather than just stating the bug. oh well. it's documented. :)
[10:15] <sarnold> maswan: so if you d'nt mind the eth0 eth1 eth2 etc names, you can uninstall the biosdevname package and rebuild your initramfs
[10:16] <maswan> sarnold: yeah. I kind of like the new names, but not to the point where drbd won't work because the internal networking went away because the nic suddenly was "rename5"
[10:17] <sarnold> maswan: yeah. it seems surprising it wasn't found and fixed earlier.
=== Ursinha is now known as Ursinha-afk
[11:11] <Pupeno> Should users used for running a web app and for deploying an app by system or regular users?
=== Ursinha-afk is now known as Ursinha
=== Ursinha is now known as Ursinha-afk
[16:30] <Pupeno> A custom upstart job I wrote is not starting and it's not writing anything to /var/log/upstart/jobname.log. Any other log files I could check for errors?
[16:37] <RoyK> Pupeno: perhaps /var/log/messages
[16:38] <Pupeno> RoyK: there's no such file in Ubuntu, at least 12.04.
[16:38] <RoyK> erm
[16:38] <RoyK> s/messages/syslog/
[16:38] <RoyK> sorry
[16:39]  * RoyK is using a lot of RHEL/CentOS at work and confuses things a bit sometimes
[16:40] <Pupeno> RoyK: no, nothing there :(
[16:41] <RoyK> Pupeno: then I don't know - sorry
[18:44] <dasjoe> maswan: have you checked /etc/udev/rules.d/70-persistent-net.rules?
[19:15] <soren> Suppose I have a number of outstanding security updates. Does anyone have any good ideas for how I can list the corresponding USN's?
[19:21] <soren> jdstrand: ^ Any good ideas?
[19:37] <XKM> Hello
[19:38] <XKM> I have an HP n54l with 8GB of RAM and an radeon HD540
[19:39] <XKM> the CPU is AMD Turion™ II Model Neo N54L
[19:39] <XKM> should i be running 64 bit or 32?
[19:39] <XKM> I think 64...am i wrong?
[19:39] <XKM> Its going to be a NAS btw if that makes any different
[19:43] <XKM> anyone?
[19:43] <RoyK> sec
[19:43] <XKM> ok
[19:43] <RoyK> it's a 64bit cpu AFAICS
[19:43] <RoyK> http://www.cpu-world.com/CPUs/K10/AMD-Turion%20II%20Neo%20N54L%20-%20TEN54LSDV23GME.html
[19:44] <RoyK> with 8GB RAM, it'd be best to run it in 64bit mode
[19:44] <XKM> Syeet
[19:44] <XKM> Sweet thanks!
[19:44] <XKM> How stable is the new 14.04?
[19:45] <XKM> Would i be better of witht he 12.04?
[19:48] <lorenz> Hey, I got a problem with my 14.04 Server, it won't boot anymore. It seems like that all upstart jobs are failing, because when I run it in debug mode, they start at a 5-second interval (which is the default for upstart when services fail). Does anybody know what's going on?
[19:49] <XKM> lol
[19:49] <lorenz> what lol?
[19:51] <TJ-> lorenz: disk corruption?
[19:52] <XKM> sorry i am thinking between 12.04 and 14.04 ubuntu server
[19:53] <lorenz> I checked my disk with btrfs scrub, btrfsck and rebalanced. Everything worked perfectly and no errors showed.
[19:53] <TJ-> lorenz: is the root file-system mounted read-only?
[19:54] <lorenz> no, I don't think so. At least in /etc/fstab it's mounted rw and the root subvolume is flagged rw.
[19:55] <TJ-> lorenz: Do you have a shell terminal you can type at?
[19:55] <lorenz> no, the shell doesn't work yet
[19:55] <lorenz> it fails when starting the plymouth-bridge
[19:55] <TJ-> lorenz: have you tried starting with "init=/bin/bash" so you can at least explore?
[19:56] <lorenz> oh, thank you
[19:56] <lorenz> I'll try that
[19:57] <XKM> so what about 12.04 vs 14.04?
[19:57] <XKM> which one should i go for knowing its going to be a NAS box
[19:58] <lorenz> I saw something: when i typed mount in, it showed me a warning that /etc/mtab is not writeable (read-only filesystem). But in the flags for / it shows rw. How can I find out if the disk is really getting mounted rw or ro?
[20:00] <TJ-> lorenz: "cat /proc/mounts" will show what is actually mounted as far as the kernel is concerned
[20:01] <lorenz> It is actually read-only.
[20:01] <lorenz> Can I just remount it rw or would that break something?
[20:02] <TJ-> lorenz: try to determine why
[20:02] <lorenz> I remounted it with mount -o remount,rw / and now it shows in /proc/mount as rw
[20:02] <TJ-> lorenz: has any file-system run out of space?
[20:03] <lorenz> no, all Fs have plenty of space
[20:03] <TJ-> lorenz: I wonder if the initrd.img could be corrupted
[20:04] <lorenz> Wouldn't update-initramfs -u fix that? Because I ran that command in a chroot and nothing changed.
[20:04] <TJ-> lorenz: it's the initrd.img's "/init" script that handles the root pivot from the kernel's rootfs to the real root
[20:04] <lorenz> ok
[20:04] <TJ-> lorenz: Yes, it should, assuming the /boot/ volume is good
[20:05] <TJ-> lorenz: Is /var/log/ in the root-filesystem?
[20:05] <lorenz> Yes, it is, but the ls command took ~1s to complete
[20:06] <TJ-> lorenz: "tail -f /var/log/kern.log" see what is being written
[20:06] <TJ-> lorenz: is all mass-storage local on the server ?
[20:06] <lorenz> a lot of retire_playback_urb: 832 callbacks suppressed
[20:07] <lorenz> yes, all storage is local (I currently run an image of the server on my computer in QEMU-KVM, so the hardware isn't the issue)
[20:08] <TJ-> lorenz: what USB devices connected?
[20:08] <lorenz> no USB devices connected
[20:08] <TJ-> Any sound devices?
[20:09] <lorenz> ac97
[20:09] <lorenz> default of qemu, I can remove it if that helps
[20:09] <TJ-> "retire_playback_urb" is a message from sound/usb/pcm.c
[20:09] <lorenz> ok
[20:09] <TJ-> hang on... this server is running in qemu-kvm ?
[20:11] <lorenz> The original server who failed to boot is physical. I took an image of the ssd on my local computer to help debugging after I failed to bring it back up. It behaves exactly the same in QEMU-KVM. But when it works again, I plan to write the modified image back to the boot ssd of the physical server.
[20:15] <TJ-> lorenz: That is *strange*
[20:16] <TJ-> lorenz: How large is the root file-system block device?
[20:16] <lorenz> The Image is 60 GB (It's taken from a Kingston KC300 60GB)
[20:19] <TJ-> lorenz: I'm wondering if it's a btrfs bug. If you have the space, I am thinking it might be worth creating a new blockdevice and formatting it ext4, then rsync the data from the current root-file system, and then modify things enough to try starting from that.
[20:20] <TJ-> The fact you had a read-only mount even on QEMU seems 'wrong' unless there is something wrong with the btrs volumes
[20:21] <lorenz> Well, I got a few terabytes on my computer, so I'm just gonna create  a new image with 60GB and copy stuff over.
[20:23] <lorenz> Why could I mount it rw if there's something wrong with the volume? The kernel didn't complain (and not even throw a warning in the logs).
[20:25] <TJ-> I have *no* idea, but I'm trying to reduce the problem space through eliminating potential issues... the more you can do that, the easier it is to focus on the real problem
[20:25] <lorenz> Something else: when I am in bash (from init=/bin/bash), can I then launch the normal init after I remounted / as rw?
[20:30] <lorenz> btrfs -> ext4 copy job is running
[20:30] <sarnold> lorenz: maybe. if you want to try, try exec /sbin/init
[20:30] <lorenz> ok, I'm gonna try that on another image
[20:34] <lorenz> ok, exec /sbin/init waits 60 seconds, afterwards it prints init: plymouth-upstart-bridge main process (196) terminated with status 1. So this fails too. I'm now waiting for the copy process.
[20:35] <sarnold> lorenz: oh, drat. :( thanks for trying, I realize now I'd never known if that could or should work, or why not if it doesn't..
[20:37] <Diplomat> hey guys, i installed 64bit ubuntu to my server.. and for some random reason it shows i686
[20:37] <Diplomat> any ideas ?
[20:37] <lorenz> sarnold: well, it was the same error as without /bin/bash, so it could work if it works without init=/bin/bash
[20:38] <lorenz> Diplomat: what is the output of uname -a
[20:38] <Diplomat> Linux os-comp2 3.11.0-22-generic #38~precise1-Ubuntu SMP Fri May 16 20:50:12 UTC 2014 i686 i686 i386 GNU/Linux
[20:38] <Diplomat> I'm more than 100% that I installed Ubuntu 12.04.4 64 bit version
[20:39] <ikonia> looks like a 32bit kernel
[20:39] <ikonia> seems %99.9 certain
[20:39] <lorenz> Diplomat: That's a 32bit ubuntu, I can't tell you more.
[20:39] <Diplomat> that's what im asking
[20:39] <Diplomat> how can it be 32bit when i used 64bit iso.. :/
[20:39] <ikonia> you didn't
[20:39] <ikonia> you used 32bit
[20:39] <lorenz> A 64bit ISO can't install a 32bit OS, it doesn't have the packages.
[20:40] <Diplomat> http://releases.ubuntu.com/12.04/ubuntu-12.04.4-server-amd64.iso is this 32 bit ?
[20:40] <lorenz> You have a 32bit OS
[20:40] <ikonia> Diplomat: that's not what you used to install
[20:40] <ikonia> Diplomat: please run "file /bin/bash" please
[20:40] <Diplomat> haha lol now how can you tell me that
[20:40] <Diplomat> I have installed ubuntu at least 50-60 times
[20:40] <ikonia> Diplomat: please run "file /bin/bash" please
[20:40] <RoyK> Diplomat: amd64 is 64bit - use the 32bit one
[20:41] <lorenz> RoyK: He wants 64 bit
[20:41] <Diplomat> ikonia, I checked it already it's 32
[20:41] <Diplomat> it shows
[20:41] <ikonia> Diplomat: ok, so you can't have used a 64bit install media
[20:41] <Diplomat> http://pastebin.com/rzkqrVQH
[20:41] <ikonia> the 64bit install media doesn't have the 32bit software on it
=== _thumper_ is now known as thumper
[20:41] <ikonia> so you have to have either a.) downloaded/installed the wrong media b.) the ubuntu site gave you the wrong media, but renamed it to 64bit media name (unlikley)
[20:41] <lorenz> ELF 32-bit LSB executable
[20:42] <ikonia> that's the only two options
[20:42] <Diplomat> ikonia, please, there is no way that I used 32 bit iso when I used that URL to download it to my desktop and then used that exact file for installing
[20:42] <lorenz> It could also be that the software to write the USB stick failed and left an old 32bit installer
[20:42] <Diplomat> I wouldnt be here making a joke of myself if i wouldnt be that sure
[20:42] <ikonia> Diplomat: repeating it won't change the facts you have a 32bit os
[20:42] <ikonia> Diplomat: so what do you want to do / happen now ?
[20:43] <RoyK> lorenz: 64bit doesn't work with 32bit hww
[20:43] <RoyK> lorenz: 64bit doesn't work with 32bit hw
[20:43] <Diplomat> nothing, i was just surprised and i wanted to ask about this issue
[20:43] <ikonia> well, as you are the only person who's ever reported this issue, it's most likley (but not fact) a simple user error mistake
[20:44] <ikonia> it does happen,
[20:44] <lorenz> RoyK: yes I know. But if it the script failed (didn't write anything on the stick), and there was another 32bit installer already on there.
[20:44] <ikonia> Diplomat: stick the install media in now, and config the volume label on it
[20:44] <lorenz> Diplomat: What installation media did you use?
[20:44] <guntbert> s/config/confirm/
[20:45] <Diplomat> I just booted it and installed like I have always done it
[20:45] <Diplomat> and if I failed some where.. then how lol
[20:45] <ikonia> check the volume label on the disk
[20:45] <lorenz> Was it an USB stick, CD, DVD or something else?
[20:45] <Diplomat> http://puu.sh/92ub6.jpg
[20:46] <ikonia> Diplomat: that's an iso image
[20:46] <Diplomat> just that file to empty VirtualBox
[20:46] <ikonia> Diplomat: not the boot media
[20:46] <ikonia> ahhh a virtual box install, sorry, I missed that earlier
[20:46] <lorenz> Boot order in VirtualBox?
[20:47] <Diplomat> What I did was I created a new vm.. and then it booted then it asked for installer I picked that ISO and then installed it and then it restarted and then i did some stuff there and then i realized that something is wrong and did "uname -a"
[20:47] <Diplomat> and voila
[20:47] <Diplomat> I dont have any USB or CD/DVDs connected
[20:48] <ikonia> Diplomat: boot the install media and check it with uname -a again
[20:48] <Diplomat> I was messing with this: http://docs.openstack.org/icehouse/install-guide/install/apt/content/nova-compute.html
[20:49] <lorenz> Then get an Ubuntu Cloud install image
[20:49] <Diplomat> http://docs.openstack.org/icehouse/install-guide/install/apt/content/basics-packages.html
[20:49] <Diplomat> this too
[20:49] <lorenz> no
[20:50] <lorenz> http://www.ubuntu.com/download/cloud
[20:50] <lorenz> this one
[20:50] <lorenz> Has OpenStack preinstalled
[20:50] <Diplomat> Yea, but that's why I didnt get it because I wanted to try myself
[20:51] <Diplomat> and it appears I was able to do the impossible lol
[20:51] <Diplomat> well
[20:51] <Diplomat> I'm gonna download that same iso again and try again
[20:51] <Diplomat> maybe i'm able to get it 16bit
[20:52] <sarnold> 8bit best bit
[20:53] <Diplomat> I'm trying man
[20:53] <lorenz> You will then have 256bits of ram available :D
[20:53] <Diplomat> lol
[20:54] <Diplomat> I might be able to open hello world text file then
[20:54] <Diplomat> that's pretty cool too
[21:00] <cloudman> Hi, just installed 14.04 for the first time on a server and activated root but sftp is saying no all the time...
[21:00] <Gallomimia> is your ssh server running?
[21:01] <cloudman> fresh install, got in as user and activated root etc so yes
[21:01] <Gallomimia> sftp runs over ssh. i'm sure your client is configured to get in that way too
[21:01] <lorenz> Is "Subsystem sftp /usr/lib/openssh/sftp-server" present in /etc/ssh/sshd_config
[21:02] <cloudman> tried about 4 times, checking pwd and sftp just keeps ignoting it??
[21:02] <Gallomimia> by the way sftp as root is pretty brutal. recommend using a different username than that
[21:02] <cloudman> Gallomimia: I need to upload a file to root dir
[21:02] <cloudman> I need root
[21:02] <lorenz> SFTP as root is maybe not very secure, but it works (I tried it once, I needed to move a whole lot of files to root)
[21:03] <cloudman> give me root everytime, let me take care of security, yet to be hacked since 1995
[21:04] <cloudman> is root sftp possible in 14.04??
[21:04] <Gallomimia> just as secure as anything. just can be easy to overwrite things you didn't want. anyway, viewing the file at /etc/sshd_config will let you look at a few options for the server
[21:04] <lorenz> Yes, it is possible
[21:04] <cloudman> how?  :)
[21:04] <lorenz> Should work out of the box
[21:04] <Gallomimia> having not run 14.04 yet im unsure. hm. do i have any ubuntu servers left?
[21:04] <cloudman> will do a fresh install but its having none of it here
[21:04] <lorenz> If you set your root password to something else than an empty hash
[21:05] <lorenz> Look at /etc/ssh/sshd_config
[21:05] <lorenz> Upload the contents somewhere on pastebin or similar
[21:06] <Gallomimia> there must be something turning off sftp for root if you having problems
[21:06] <Gallomimia> or your client is not configured properly
[21:06] <cloudman> beats me
[21:06] <cloudman> will check it out
[21:06] <TJ-> check "/var/log/auth.log"
[21:17] <cloudman> lorenz: http://paste.ubuntu.com/7524412/
[21:18] <lorenz> cloudman: that's ssh_config, not sshd_config
[21:18] <cloudman> ah, sry too much wine tonight
[21:19] <cloudman> http://paste.ubuntu.com/7524436/
[21:20] <cloudman> first time with 14.04
[21:20] <cloudman> all good fun anyway
[21:22] <cloudman> lorenz: http://paste.ubuntu.com/7524436/
[21:23] <lorenz> cloudman: Subsystem sftp /usr/lib/openssh/sftp-server is there, so the server should have no issues. Could you also post /var/log/auth.log?
[21:24] <cloudman> Yes as Tj mentioned checked that and it says pwd failure???  keyboard setup maube??
[21:24] <cloudman> maybe
[21:25] <cloudman> sshd[1357]: Failed password for root from
[21:26] <cloudman> how can that be though, I know my typing is a bit off tonight but...
[21:27] <cloudman> thing is I tried with a simple 123456 as well and got the same, sftp reject
[21:28] <lorenz> cloudman: Is there anything above the failed password line?
[21:30] <cloudman> lorenz:  pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=
[21:30] <lorenz> cloudman: anything above there?
[21:30] <genii> With "PermitRootLogin without-password" aren't you supposed to be using a key?
[21:30] <cloudman> fail2ban not built in to 14.04 is it :)
[21:31] <lorenz> cloudman: It's what genii said
[21:31] <cloudman> genii: not sure, 12.04 user here
[21:32] <lorenz> cloudman: Change line 28 of /etc/ssh/sshd_config to PermitRootLogin yes
[21:32] <cloudman> ok, hangon
[21:34] <cloudman> lol, need root
[21:35] <cloudman> won't let me do it
[21:35] <cloudman> hangon
[21:38] <cloudman> ok, should I restart ssh server
[21:38] <lorenz> cloudman: yes
[21:38] <cloudman> ty
[21:42] <cloudman> Guess what guys
[21:42] <cloudman> in as ./
[21:42] <cloudman> tahnks
[21:42] <cloudman> thanks
[21:42] <cloudman> tanked up as well
[21:42] <cloudman> why does it do that as default
[21:43] <cloudman> lorenz: ty
[21:43] <cloudman> genii: even more ty
[21:43] <cloudman> TJ ty
[21:43]  * genii hands out the coffee and cookies
[21:43] <cloudman> what a pain though
[21:44] <TJ-> it's done deliberately
[21:44] <cloudman> to annoy people?
[21:44] <cloudman> thanks guys, great help and fast response, superb
[21:45] <TJ-> cloudman: If I wanted a user to write to "/" I'd simply do "sudo adduser $USER root && sudo chmod g+w /"
[21:45] <lorenz> TJ-: Copy process is done, I will now try to setup grub
[21:46] <TJ-> lorenz: good luck, I hope it is that; at least you can then get on with fixing the problem
[21:46] <cloudman> 8 core 8 thread, 8Gb Ram DDR3 1Tb HD 35 euro a month anyone want one as a reward ;)
[21:47] <cloudman> not kidding just got a batch in
[21:48] <cloudman> now to see if VM installs fine on 14.04
[21:48] <cloudman> should be a gas
[21:48] <lorenz> cloudman: https://www.hetzner.de/en/hosting/produkte_rootserver/ex40 is 32GB RAM for 49 € per month
[21:48] <cloudman> yeah I have one
[21:49] <cloudman> mine is manged though
[21:49] <cloudman> managed
[21:49] <cloudman> not that anyone here should need a managed onr though I guess ;)
[21:51] <cloudman> talk about preaching to the converted lol
[21:51] <cloudman> thanks guys
[21:55] <cloudman> Virtualmin installer fails on 14.04 hope they sort it soon
[22:00] <TJ-> how did it fail?
[22:01] <genii> !webmin
[22:01] <ubottu> webmin is no longer supported in Debian and Ubuntu. It is not compatible with the way that Ubuntu packages handle configuration files, and is likely to cause unexpected issues with your system.
[22:01] <cloudman> TJ http://paste.ubuntu.com/7524677/
[22:02] <cloudman> wow
[22:02] <TJ-> cloudman: "/root/virtualmin-install.log" ?
[22:05] <cloudman> TJ-: http://paste.ubuntu.com/7524686/
[22:06] <lorenz> TJ-: Unpacking grub-pc-bin in the rescue system is now running for like ~10min with no disk activity
[22:06] <TJ-> lorenz: that's not good
[22:06] <TJ-> lorenz: something very weird with your setups there
[22:07] <TJ-> cloudman: The line "Installing dependencies using command: ..." has the list of packages it was going to install. My guess is that one or more are no longer available or have changed SO version-names in 14.04
[22:07] <cloudman> They are sorting it soon
[22:08] <cloudman> webmin is ok but you just have to love virtualmin its so easy
[22:08] <lorenz> TJ-: It's a normal 14.04 installation, just standard packages and only a few PPAs (Percona, Elasticsearch, Ksplice, Owncloud, Passenger and XtreemFs)
[22:08] <cloudman> https://www.virtualmin.com/node/32970
[22:09] <TJ-> cloudman: I was already there :)
[22:17] <cloudman> :)
[22:17] <cloudman> be nice to get it all sorted
[22:22] <lorenz> TJ-: I'm trying with the boot-repair ISO, my grub-pc-bin package seems broken
[22:23] <TJ-> lorenz: ouch
[22:24] <cloudman> I have some servers that lose boot dir recently??
[22:39] <lorenz> TJ-: boot-repair also requires you to install this package in chroot. It doesn't work there either. But I have been able to attach a strace to the responsible dpkg process. It's writing with the speed of about 4KB/s :D
[22:40] <TJ-> lorenz: This is from the QEMU session?
[22:40] <lorenz> TJ-: It's inside QEMU, yes.
[22:41] <lorenz> TJ-: The Ext4 image is on a SSD, so that's not the issue
[22:41] <TJ-> lorenz: Are you *100%* sure that QEMU is using VT hardware support? It sounds as if its using software emulation
[22:42] <lorenz> TJ-: Everything else is very fast, I rechecked and KVM is enabled and the load on my host is something around 0.1
[22:42] <TJ-> lorenz: then something you've got there is playing silly buggers with you!
[22:42] <ikonia> how are you checking if kvm is being used
[22:43] <lorenz> ikonia: virt-manager shows Hypervisor: kvm
[22:43] <ikonia> lorenz: is the kvm module actually loaded ?
[22:44] <TJ-> lorenz: "ls /dev/kvm" or "kvm-ok"
[22:44] <lorenz> ikonia: lsmod | grep kvm shows kvm_intel and kvm
[22:44] <lorenz> TJ-: kvm-ok: KVM acceleration can be used
[22:45] <TJ-> lorenz: good... one more thing checked off :)
[22:45] <TJ-> lorenz: just to be clear, the host is fine, it's the guest image in the VM or when its run on the original bare-metal server, that is having problems?
[22:45] <lorenz> TJ-: Exactly
[22:46] <TJ-> lorenz: Then it has to be something in the images. Have you managed to start the ext4 clone of the root-file-system ?
[22:47] <lorenz> TJ-: I'm trying to install grub on it right now. The images are the simple .img files, so they shouldn't cause any problems. I need img to write them later back to the SSD.
[22:47] <TJ-> yeah, that's how I do it
[22:48] <lorenz> TJ-: It finished now! :D
[22:49] <Pupeno_> My custom upstart job stopped writing to /var/log/upstart/jobname.log. Any ideas what it could be or how to troubleshoot it?
[22:50] <TJ-> Pupeno_: logrotate not setting the permissions of the new file correctly?
[22:51] <lorenz> TJ-: The install dpkg for grub is now just sleeping (according to strace, it does a nanosleep)
[22:52] <TJ-> lorenz: one thing I can think of, is when hostname tries to be resolved outside the localhost... can affect many commands unexpectedly
[22:52] <lorenz> TJ-: Sorry, that was the top-level dpkg which was waiting for the children. But the child now also writes very slow.
[22:52] <Pupeno_> TJ-: there's no file right now, and it's not being created. I didn't touch the permissions of the parent directory since before it stopped working.
[22:53] <TJ-> Pupeno_: if the log file is missing then ownership and/or permissions will be the issue
[22:53] <lorenz> TJ-: localhost resolves to 127.0.0.1 according to /etc/hosts
[22:53] <Pupeno_> TJ-: that wasn't an issue when the computer was freshly installed, but I would be happy to explore that.
[22:53] <TJ-> lorenz: how about the hostname itself - the name of the machine
[22:54] <lorenz> TJ-: That also resolves to 127.0.1.1, which seems ok to me. Btw, is it correct that the kernel is responsible for executing the calls I see in strace?
[22:54] <Pupeno_> TJ-: created the file as 666 and still nothing got written to it.
[22:55] <TJ-> Pupeno_: I've seen instances of a daemon working fine until logrotate fires off, which changes ownerships and stops the daemon from writing... it can be caused by the daemon dropping privileges/to another unprivileged user-id and can therefore not have root:root access to access the log file
[22:55] <Pupeno_> The job is not starting, something is preventing it, but logs are empty.
[22:55] <TJ-> lorenz: You're seeing the syscalls mostly
[22:55] <TJ-> Pupeno_: Try starting the daemon manually in the foreground with full verbosity/debug enabled
[22:55] <Pupeno_> TJ-: I was working on the app this job is supposed to fire when it stopped working. I don't think it was logrotate, but still, without logs, I can't figure out what's stopping this app from starting.
[22:56] <Pupeno_> TJ-: that doesn't work, it fails, and that's all right. I'll get to that error after I fix upstart.
[22:56] <TJ-> Pupeno_: I'm lost; if the daemon won't start manually, how can upstart start it?
[22:57] <Pupeno_> TJ-: I wrote the app, there's a bug somewhere, which I'll resolve later. Right now, I want to see the error messages in the context of upstart. I'm not expecting upstart to work, I'm expecting upstart to log the error.
[22:59] <TJ-> Pupeno_: You'll need to do "initctl log-priority debug" to see more info in the syslog
[23:00] <Pupeno_> TJ-: still nothing in upstart/projectx.log or syslog.log. Should I look in another log file?
[23:01] <genii> "...in the syslog"
[23:01] <TJ-> Pupeno_: As far as I recall, /var/log/syslog is the place to look
[23:01] <Pupeno_> Should I add that line to my upstart conf file?
[23:05] <TJ-> what line?
[23:06] <Pupeno_> initctl log-priority debug
[23:06] <TJ-> Pupeno_: That's a command to issue directly, as root/sudo
[23:06] <Pupeno_> ok, that's what I did.
[23:07] <lorenz> TJ-: It has now worked, Boot-Repair is installing the new MBR. I'm trying to reboot shortly.
[23:07] <TJ-> lorenz: *fingers* crossed
[23:07] <TJ-> lorenz: but that slowness suggests something is dragging its feet terribly
[23:08] <lorenz> TJ-: Ok, it boots. BUT it is stuck at the same line as before.
[23:09] <lorenz> TJ-: Oh, I forgot to update /etc/fstab
[23:10] <TJ-> lorenz: *tuts*
[23:18] <lorenz> TJ-: Sorry, didn't help. The newly created ext4-machine (without EFI and BTRFS) does exactly the same. It shows init: plymouth-upstart-bridge main process (182) terminated with status 1 and mounts / as read-only
[23:19] <TJ-> lorenz: Then you've got some problem in the initrd.img I'd guess
[23:19] <lorenz> TJ-: Can I reinstall that somehow?
[23:20] <TJ-> lorenz: Before this first started, what was the last (few) sysconfig changes/system updates ?
[23:22] <lorenz> TJ-: I did nothing the day it started. The (physical) machine just stopped responding (I had a SSH session open, but it also didn't respond to pings and ARP-requests). Then I hard-rebooted and it didn't came up.
[23:23] <TJ-> lorenz: I have to start  thinking along the lines of "comprised" and "root-kit"
[23:23] <TJ-> lorenz: s/comprised/compromised/
[23:24] <lorenz> TJ-: It runs Snort, UFW, chkrootkit, OSSEC and is hardened with bastillion, so it's quite unlikely. But it could happen.
[23:26] <lorenz> TJ-: I had no alerts that day. And wouldn't somebody who attacks the machine try to not bring it down, because a disabled machine can't be used to do anything?
[23:26] <TJ-> lorenz: Well look at the facts... you've moved the same image to different hardware, inside a VM, using a different file-system, and it still does the same thing
[23:27] <TJ-> lorenz: extract the initrd and check it for anything 'weird' or 'missing' or 'badly configured'
[23:29] <lorenz> TJ-: Have you got a command ready to do that?
[23:29] <TJ-> lorenz: funny you should say that ;p
[23:30] <lorenz> TJ-: I found one.
[23:31] <TJ-> lorenz: "mkdir /tmp/initrd && pushd /tmp/initrd && zcat /boot/initrd.img-`uname -r` | cpio -i -d "
[23:31] <lorenz> TJ-: Thank you!
[23:32] <lorenz> TJ-: What should I look for?
[23:32] <TJ-> lorenz: $1,000,0000 question!
[23:33] <lorenz> TJ-: Ok :D
[23:33] <TJ-> first check in the conf/ directory and its sub-dirs, read the files, get a feel for what they're saying
[23:33] <TJ-> There should be some mdadm.conf stuff in there if you're using MD
[23:34] <lorenz> TJ-: I use btrfs RAID, so there is no mdadm.conf
[23:34] <lorenz> TJ-: /conf folder seems well, everything makes sense
[23:34] <TJ-> also, check the /libs/ and the lib/
[23:35] <TJ-> grrr, typo... "lib/"
[23:35] <lorenz> TJ-: Found one: klibc-P2s_k-gf23VtrGgO2_4pGkQgwMY.so
[23:35] <lorenz> TJ-: Does that look normal?
[23:36] <TJ-> lorenz: "md5sum:  4152a06877635bf3dc98a3cf6e48bd4a  lib/klibc-P2s_k-gf23VtrGgO2_4pGkQgwMY.so"
[23:37] <TJ-> lorenz: I notice there's "/etc/{lvm,mdadm}/"
[23:37] <lorenz> TJ-: 4152a06877635bf3dc98a3cf6e48bd4a  klibc-P2s_k-gf23VtrGgO2_4pGkQgwMY.so -- seems the same
[23:38] <lorenz> TJ-: I don't use LVM nor MDADM
[23:39] <TJ-> lorenz: OK... how about "cat $(find scripts -type f) | md5sum"          ==  2dc83e509c928077916b5951b8162cd9
[23:39] <TJ-> lorenz: that'll probably be different to mine; I have lvm, mdadm, and cryptsetup
[23:39] <lorenz> TJ-: That's different
[23:40] <TJ-> lorenz: not surprised, that was a lot of files to expect to be the same!
[23:42] <lorenz> TJ-: http://paste.ubuntu.com/7525216/ Are the sums of my scripts dir
[23:43] <TJ-> lorenz: OK, take a look at the "./init" shell script that controls the process. You'll see several function calls of the form "maybe_break XXXX" these are points where the script can be stopped by adding "break=XXXX" to the kernel command-line
[23:43] <lorenz> TJ-: I think I found something: my lib/modules are for 3.13.0-27-generic and my kernel is 3.13.0-24
[23:44] <TJ-> lorenz: In particular, look at "maybe_break mountroot" ... at that point it is about to call "mountroot". If you did "break=mountroot" and then manually executed that call, *maybe* you can find out why it is mounting it read-only
[23:44] <TJ-> lorenz: Really!?!
[23:44] <TJ-> lorenz: Let me extract the same version here
[23:45] <lorenz> TJ-: My newest image in /boot is initrd.img-3.13.0-24-generic, but the contents are 3.13.0-27-generic
[23:46] <TJ-> lorenz: "ls lib/modules/"  = "3.13.0-24-generic"
[23:46] <TJ-> lorenz: That'd sure mess things up!
[23:47] <TJ-> lorenz: "May 26 21:03:48 <TJ->   lorenz: I wonder if the initrd.img could be corrupted"
[23:47] <lorenz> TJ-: your script unpacked my host initrd, that was the problem.
[23:48] <TJ-> lorenz: lol ... I thought you'd adjusted the paths for the VM :) ... oh well, it's never *that* easy!
[23:49] <TJ-> lorenz: so, back to the idea of "break=mountroot"
[23:50] <lorenz> TJ-: ok, i'll see
[23:52] <TJ-> lorenz: If you do that, the first thing once you have the shell prompt is to source the function library with ". scripts/local" and then you can do "mountroot"
[23:52] <lorenz> TJ-: ok
[23:54] <TJ-> lorenz: also, *before* you call "mountroot" ensure there's verbose logging with export quiet="n"; set -x;
[23:56] <lorenz> TJ-: Kernel panic: Illegal number mountroot
[23:56] <TJ-> hmmm!
[23:57] <TJ-> lorenz: Did you see any messages/debug info before that?
[23:58] <lorenz> TJ-: I missed a newline before the mountroot, is now fixed. I'm compressing the next initrd.
[23:58] <TJ-> lorenz: OK
[23:59] <lorenz> TJ-: Nothing happens now (I put a newline between break and mountroot)