=== dedi is now known as dedi_off === scottrigby is now known as scottrigby_away === scottrigby_away is now known as scottrigby === scottrigby is now known as scottrigby_away === dedi_off is now known as dedi === dedi is now known as dedi_off === scottrigby_away is now known as scottrigby === scottrigby is now known as scottrigby_away === scottrigby_away is now known as scottrigby === scottrigby is now known as scottrigby_away [10:08] Morning. [11:04] Morning JonathanD and all [11:18] hey InHisName [12:10] Morning peoples, dogs, turkeys and everything else [13:17] * ChinnoDog noms some bacon [13:51] I got a cat! [13:52] On purpose? [13:52] yeap [13:53] hopefully our other cat and him get along [14:00] I find that one cat is usually enough [14:02] two is better than one [14:02] Don't cats prey on turkeys? [14:05] still need advice? dump her [14:05] all i saw way 'need advice' [14:05] waltman: it's all about the powerplay [14:08] but what do you do after the 2 minutes are up? === scottrigby_away is now known as scottrigby [15:54] uh oh... looks like my server is rejecting all ssh connections... [15:57] oops [15:57] yeah... [15:57] Is it a dynamic IP? Maybe it's on a nice address now? [15:59] nope. [15:59] i can ping it fine [15:59] mysql just went down as well... [16:00] might have lost power and it may be fscking [16:00] or sitting in busybox, heh [16:00] pings ok [16:01] you can still have network with busybox depending on how your distro bundles it [16:01] also, though, sidenote- 'swhy i use better-initramfs to boot [16:01] automatic recovery is <3 [16:01] well. recovery MODE, i should say. accessible by ssh. yay! [16:02] gahh i am freaking out [16:02] Maybe something sucked up all your memory and the kernel's shutting stuff down? [16:02] NRPE still works!!!! [16:03] not that that helps [16:03] heh [16:03] * bts3685 grabs bag of popcorn [16:04] succeed2ban [16:05] * bts3685 chokes on popcorn [16:05] jedijf: you're on the list [16:06] \o/ [16:06] alright... shit [16:06] sorry [16:06] our mysql daemon runs on that server for our very public database we just released... [16:06] sucks to be you [16:07] wait [16:07] you run mysql publicly accessible? [16:11] no [16:11] i just have one machine that hosts all of our sql stuff for our other servers [16:11] one of which runs a website with a large mysql backend [16:11] since the machine htat's down has a large number cores and ram [16:12] well, now it's responding [16:12] so maybe it's just under very heavy load? [16:13] bts3685: Nagios is indivating that it's got a load average of 40.0 with only 24 cors [16:17] check I/O [16:18] if I/O's blocking, it'd not only cause tons of issues but also would kick that load up [16:18] also amount of free memory, etc. [16:21] who is "our"? [16:21] Is this your lab's machine? [16:48] how could I check IO [16:48] waltman: one of them [16:48] i assume it's because the swap is kicking in [16:48] so paging is really slow [16:48] because they are running a crazu amound of jobs withoutconsideration to what the server load is... [16:54] MutantTurkey: iotop, htop sort by I/O, sar, dstat... lots of options [16:55] bts3685: I can't access the server though [16:55] because bash is just hanging probably because it can't allocate resources [16:56] is bash hanging or is ssh? [16:56] bash [16:56] also, housed on-site or with a DC? because either way, you'll need to console that, the way it sounds. [16:57] unfortunately my boss just had a baby and they switched the keys... we don't have a kvm for that box...s [16:57] sooo! [16:57] how can I set the term with ssh? [16:57] ssh -C "TERM=xterm; htop" doesn't work [16:58] i.... what? why would switching the keys have anything to do with a baby. [16:58] shouldn't need to set TERM remotely [16:58] unless you're using screen or something [16:59] but, if you DID need to, [16:59] env TERM=blabla ssh user@server [16:59] err, actually [16:59] env TERM=blabla ssh -o SendEnv=TERM user@server [16:59] will probably be what you need [17:01] ah got it [17:01] ssh "export TERM=xterm; htop" [17:01] well, looks like everyones jobs ended and it's back to normal. [17:06] bts3685: any ideas how I can avoid that in the future? [17:18] better disk setup [17:18] assuming it was I/O [17:18] what's free -m look like? how much swap has been used? [17:18] and what's the third column of load in w output? [17:26] no idea, by the time I got into the server, everything had dropped [17:26] since I had to run out to pick up some paperwork [17:26] * MutantTurkey sigh [17:27] sar -q === scottrigby is now known as scottrigby_away === scottrigby_away is now known as scottrigby [19:21] MutantTurkey: You might try looking through syslog from around that time to see if you see anything unusual. [19:27] A friend is looking to ditch gmail (he's unhappy with their latest UI changes) and asked me for recommendations. The only linux mail clients I've run in a very long time are mutt (which I love) and thunderbird (which I don't). What mail clients are the cool kids running these days? [19:29] gmail h [19:31] His issue was that he used some hack to compose messages in vim, but it broke in the redesign since the compose window is no longer a text area. [19:31] er, textarea [19:31] ah [19:31] well [19:31] if he hates the new social split thing [19:31] you can get around that [19:31] social split thing? [19:31] waltman: I used that for a long time, for long wikipedia articles. [19:32] it's some sort of new tabbed gmail interface [19:32] I didn't see it until a friend showed me, because I have a different inbox style setup [19:32] I think he mainly hates that he can no longer compose his emails in vim [19:32] http://lifehacker.com/everything-you-need-to-know-about-gmails-new-super-co-511765933 [19:32] tabbed interface is not related to changes in composing box [19:33] pleia2: I understand [19:33] that's what I originally thought, but it seems not [19:33] pleia2: talk to mj and have him get someone to haX0r in vim supprto :) [19:33] support [19:33] or vim plugin [19:33] bet there's an emacs plugin already. :X [19:34] waltman: i know the problem. We ran a few to many services, I can see just one of users using almost 95% of our ram... bioinformatics is rather ram dependent [19:34] bts3685: at this point i'll be happy if my users use either. [19:34] Ah. Linux boxes aren't reall happy under heavy paging. [19:34] I cringe when they open gedit [19:34] ...yeahhh sure enough. http://www.emacswiki.org/emacs/GnusGmail [19:34] waltman: yeah [19:35] waltman: I just use vim :) [19:35] in mutt! [19:36] (actually my hosted domain hasn't gotten the awful new upgrade yet) [19:36] waltman: can't they just stay in standard? [19:36] bts3685: I think that's different because gnus is just treating gmail like any other mail host. I believe with this now-broken plug-in you used the web interface but composed your messages in vim. [19:36] I thought they were disabling the old ui for everyone [19:36] (I don't use gmail) [19:37] change the /u/ to /h/ all good [19:37] they will eventually, hosted domains always lag behind gmail.com [19:37] I hear all the screaming before I see it myself, fun times [19:37] waltman: i thought you switched after the whole cs thing crashed. [19:37] * jedijf checks plain ol gmail [19:37] * bts3685 imaps with thunderbird [19:38] I use thunderbird for a bunch of accounts (shared ones that I help manage mostly) [19:38] it sucks, but whatever [19:38] MutantTurkey: I still have my cs account, but I switched most of my day-to-day email to coe. And I use Mail.app to access it. This is for a friend who I believe is on a redhat box. [19:39] The UI on thunderbird is pretty dreadful. [19:39] The UI on outlook is even worse. [19:39] I use mutt and mail.app [19:40] stamps [19:41] stamps? [19:43] stamps. stationary. pens. [19:44] does anyone use evolution? [19:45] no [19:45] * waltman looks at claws [19:46] I've heard decent things about claws [19:46] light on interface, but gets the job done [19:47] I used the mozilla one before [19:47] was pretty terrible [19:47] I found that mutt worked well for me [19:47] I'm a bit concerned that none of claws' screenshots show rich text and/or embedded images [19:49] damn pleia2 answers for the world [19:51] it's true [19:51] is thunderbird "the mozilla one"? [19:51] yeah [19:52] It keeps getting picked as best linux email client year after year. Maybe the windows version I was running was just worse. [19:53] could be the tallest midget award [19:53] hehe [19:54] OSX's mail client is the first I've found since Euroda that doesn't make me tear my hair out. [22:29] aw crap, new reply just hit my hosted domain [22:29] er, sorry :) === dedi_off is now known as dedi === dedi is now known as dedi_off