[10:08] <JonathanD> Morning.
[11:04] <InHisName> Morning JonathanD and all
[11:18] <JonathanD> hey InHisName
[12:10] <teddy-dbear> Morning peoples, dogs, turkeys and everything else
[13:17]  * ChinnoDog noms some bacon
[13:51] <MutantTurkey> I got a cat!
[13:52] <ChinnoDog> On purpose?
[13:52] <MutantTurkey> yeap
[13:53] <MutantTurkey> hopefully our other cat and him get along
[14:00] <ChinnoDog> I find that one cat is usually enough
[14:02] <MutantTurkey> two is better than one
[14:02] <waltman> Don't cats prey on turkeys?
[14:05] <MutantTurkey> still need advice? dump her
[14:05] <MutantTurkey> all i saw way 'need advice'
[14:05] <MutantTurkey> waltman: it's all about the powerplay
[14:08] <waltman> but what do you do after the 2 minutes are up?
[15:54] <MutantTurkey> uh oh... looks like my server is rejecting all ssh connections...
[15:57] <waltman> oops
[15:57] <MutantTurkey> yeah...
[15:57] <waltman> Is it a dynamic IP? Maybe it's on a nice address now?
[15:59] <MutantTurkey> nope.
[15:59] <MutantTurkey> i can ping it fine
[15:59] <MutantTurkey> mysql just went down as well...
[16:00] <bts3685> might have lost power and it may be fscking
[16:00] <bts3685> or sitting in busybox, heh
[16:00] <MutantTurkey> pings ok
[16:01] <bts3685> you can still have network with busybox depending on how your distro bundles it
[16:01] <bts3685> also, though, sidenote- 'swhy i use better-initramfs to boot
[16:01] <bts3685> automatic recovery is <3
[16:01] <bts3685> well. recovery MODE, i should say. accessible by ssh. yay!
[16:02] <MutantTurkey> gahh i am freaking out
[16:02] <waltman> Maybe something sucked up all your memory and the kernel's shutting stuff down?
[16:02] <MutantTurkey> NRPE still works!!!!
[16:03] <MutantTurkey> not that that helps
[16:03] <bts3685> heh
[16:03]  * bts3685 grabs bag of popcorn
[16:04] <jedijf> succeed2ban
[16:05]  * bts3685 chokes on popcorn
[16:05] <bts3685> jedijf: you're on the list
[16:06] <jedijf> \o/
[16:06] <MutantTurkey> alright... shit
[16:06] <MutantTurkey> sorry
[16:06] <MutantTurkey> our mysql daemon runs on that server for our very public database we just released...
[16:06] <bts3685> sucks to be you
[16:07] <bts3685> wait
[16:07] <bts3685> you run mysql publicly accessible?
[16:11] <MutantTurkey> no
[16:11] <MutantTurkey> i just have one machine that hosts all of our sql stuff for our other servers
[16:11] <MutantTurkey> one of which runs a website with a large mysql backend
[16:11] <MutantTurkey> since the machine htat's down has a large number cores and ram
[16:12] <MutantTurkey> well, now it's responding
[16:12] <MutantTurkey> so maybe it's just under very heavy load?
[16:13] <MutantTurkey> bts3685: Nagios is indivating that it's got a load average of 40.0 with only 24 cors
[16:17] <bts3685> check I/O
[16:18] <bts3685> if I/O's blocking, it'd not only cause tons of issues but also would kick that load up
[16:18] <bts3685> also amount of free memory, etc.
[16:21] <waltman> who is "our"?
[16:21] <waltman> Is this your lab's machine?
[16:48] <MutantTurkey> how could I check IO
[16:48] <MutantTurkey> waltman: one of them
[16:48] <MutantTurkey> i assume it's because the swap is kicking in
[16:48] <MutantTurkey> so paging is really slow
[16:48] <MutantTurkey> because they are running a crazu amound of jobs withoutconsideration to what the server load is...
[16:54] <bts3685> MutantTurkey: iotop, htop sort by I/O, sar, dstat... lots of options
[16:55] <MutantTurkey> bts3685: I can't access the server though
[16:55] <MutantTurkey> because bash is just hanging probably because it can't allocate resources
[16:56] <bts3685> is bash hanging or is ssh?
[16:56] <MutantTurkey> bash
[16:56] <bts3685> also, housed on-site or with a DC? because either way, you'll need to console that, the way it sounds.
[16:57] <MutantTurkey> unfortunately my boss just had a baby and they switched the keys... we don't have a kvm for that box...s
[16:57] <MutantTurkey> sooo!
[16:57] <MutantTurkey> how can I set the term with ssh?
[16:57] <MutantTurkey> ssh -C "TERM=xterm; htop" doesn't work
[16:58] <bts3685> i.... what? why would switching the keys have anything to do with a baby.
[16:58] <bts3685> shouldn't need to set TERM remotely
[16:58] <bts3685> unless you're using screen or something
[16:59] <bts3685> but, if you DID need to,
[16:59] <bts3685> env TERM=blabla ssh user@server
[16:59] <bts3685> err, actually
[16:59] <bts3685> env TERM=blabla ssh -o SendEnv=TERM user@server
[16:59] <bts3685> will probably be what you need
[17:01] <MutantTurkey> ah got it
[17:01] <MutantTurkey> ssh "export TERM=xterm; htop"
[17:01] <MutantTurkey> well, looks like everyones jobs ended and it's back to normal.
[17:06] <MutantTurkey> bts3685: any ideas how I can avoid that  in the future?
[17:18] <bts3685> better disk setup
[17:18] <bts3685> assuming it was I/O
[17:18] <bts3685> what's free -m look like? how much swap has been used?
[17:18] <bts3685> and what's the third column of load in w output?
[17:26] <MutantTurkey> no idea, by the time I got into the server, everything had dropped
[17:26] <MutantTurkey> since I had to run out to pick up some paperwork
[17:26]  * MutantTurkey sigh
[17:27] <bts3685> sar -q
[19:21] <waltman> MutantTurkey: You might try looking through syslog from around that time to see if you see anything unusual.
[19:27] <waltman> A friend is looking to ditch gmail (he's unhappy with their latest UI changes) and asked me for recommendations. The only linux mail clients I've run in a very long time are mutt (which I love) and thunderbird (which I don't). What mail clients are the cool kids running these days?
[19:29] <jedijf> gmail h
[19:31] <waltman> His issue was that he used some hack to compose messages in vim, but it broke in the redesign since the compose window is no longer a text area.
[19:31] <waltman> er, textarea
[19:31] <MutantTurkey> ah
[19:31] <MutantTurkey> well
[19:31] <MutantTurkey> if he hates the new social  split thing
[19:31] <MutantTurkey> you can get around that
[19:31] <waltman> social split thing?
[19:31] <MutantTurkey> waltman: I used that for a long time, for long wikipedia articles.
[19:32] <MutantTurkey> it's some sort of new tabbed gmail interface
[19:32] <MutantTurkey> I didn't see it until a friend showed me, because I have a different inbox style setup
[19:32] <waltman> I think he mainly hates that he can no longer compose his emails in vim
[19:32] <MutantTurkey> http://lifehacker.com/everything-you-need-to-know-about-gmails-new-super-co-511765933
[19:32] <pleia2> tabbed interface is not related to changes in composing box
[19:33] <MutantTurkey> pleia2: I understand
[19:33] <MutantTurkey> that's what I originally thought, but it seems not
[19:33] <waltman> pleia2: talk to mj and have him get someone to haX0r in vim supprto :)
[19:33] <waltman> support
[19:33] <bts3685> or vim plugin
[19:33] <bts3685> bet there's an emacs plugin already. :X
[19:34] <MutantTurkey> waltman: i know the problem. We ran a few to many services, I can see just one of users using almost 95% of our ram... bioinformatics is rather ram dependent
[19:34] <MutantTurkey> bts3685: at this point i'll be happy if my users use either.
[19:34] <waltman> Ah. Linux boxes aren't reall happy under heavy paging.
[19:34] <MutantTurkey> I cringe when they open gedit
[19:34] <bts3685> ...yeahhh sure enough. http://www.emacswiki.org/emacs/GnusGmail
[19:34] <MutantTurkey> waltman: yeah
[19:35] <pleia2> waltman: I just use vim :)
[19:35] <pleia2> in mutt!
[19:36] <pleia2> (actually my hosted domain hasn't gotten the awful new upgrade yet)
[19:36] <jedijf> waltman: can't they just stay in standard?
[19:36] <waltman> bts3685: I think that's different because gnus is just treating gmail like any other mail host. I believe with this now-broken plug-in you used the web interface but composed your messages in vim.
[19:36] <waltman> I thought they were disabling the old ui for everyone
[19:36] <waltman> (I don't use gmail)
[19:37] <jedijf> change the /u/   to /h/    all good
[19:37] <pleia2> they will eventually, hosted domains always lag behind gmail.com
[19:37] <pleia2> I hear all the screaming before I see it myself, fun times
[19:37] <MutantTurkey> waltman: i thought you switched after the whole cs thing crashed.
[19:37]  * jedijf checks plain ol gmail
[19:37]  * bts3685 imaps with thunderbird
[19:38] <pleia2> I use thunderbird for a bunch of accounts (shared ones that I help manage mostly)
[19:38] <pleia2> it sucks, but whatever
[19:38] <waltman> MutantTurkey: I still have my cs account, but I switched most of my day-to-day email to coe. And I use Mail.app to access it. This is for a friend who I believe is on a redhat box.
[19:39] <waltman> The UI on thunderbird is pretty dreadful.
[19:39] <waltman> The UI on outlook is even worse.
[19:39] <waltman> I use mutt and mail.app
[19:40] <jedijf> stamps
[19:41] <waltman> stamps?
[19:43] <CaptKyle> stamps. stationary. pens.
[19:44] <waltman> does anyone use evolution?
[19:45] <pleia2> no
[19:45]  * waltman looks at claws
[19:46] <pleia2> I've heard decent things about claws
[19:46] <pleia2> light on interface, but gets the job done
[19:47] <MutantTurkey> I used the mozilla one before
[19:47] <MutantTurkey> was pretty terrible
[19:47] <MutantTurkey> I found that mutt worked well for me
[19:47] <waltman> I'm a bit concerned that none of claws' screenshots show rich text and/or embedded images
[19:49] <jedijf> damn pleia2 answers for the world
[19:51] <pleia2> it's true
[19:51] <waltman> is thunderbird "the mozilla one"?
[19:51] <pleia2> yeah
[19:52] <waltman> It keeps getting picked as best linux email client year after year. Maybe the windows version I was running was just worse.
[19:53] <jedijf> could be the tallest midget award
[19:53] <pleia2> hehe
[19:54] <waltman> OSX's mail client is the first I've found since Euroda that doesn't make me tear my hair out.
[22:29] <pleia2> aw crap, new reply just hit my hosted domain
[22:29] <pleia2> er, sorry :)