=== Lcawte is now known as Lcawte|Away [01:00] Azelphur: the real question is why are you still using software centre? [01:00] ali1234: I'm not, but other new users are :P [01:00] oh, them. === Hornet- is now known as Hornet [09:12] morning boys and girls. === Lcawte|Away is now known as Lcawte === Monotoko is now known as Guest27897 [13:26] looks like popey is on pump.io [13:37] pope.io [13:38] :) [14:10] urgh [14:10] mugged it again [14:32] I appear to have got my server into a condition of stability by stopping the oom-killer from killing mysql [14:46] om nom nom oom [14:47] the linux oom killer is unfit for purpose [14:50] anyone got the link to that FAQ about how to write alsa applications properly so they work with pulseaudio? [14:54] it was like "alsa best-practices" or something [14:56] Dear work's laptop, I do recall I told you what would happen if your Outlook in box botred me to tears... I gave you fair warning... I warn you that I knew how to "delete all"... I did warn you that I knew where the off switch off was.... I did warn you that I Steam installed on my own pooter... ;) [14:57] *bored [14:58] * SuperEngineer kills work's laptop [14:58] [and laughs manically - movie style] [15:19] directhex: the oom-killer is just unintelligent and can't tell critical applications from non-critical ones [15:20] bashrc, it also can't complete a kill op before getting invoked again, on a machine with a lot of ram, since it doesn't scale [15:21] with a lot of ram you can set swapiness to zero, but I'm only on a Beaglebone with limited system resources [15:24] the next kernel release for the Beaglebone should also include zram, which may help [15:31] RamDoubler(tm) for Linux(tm)! === lee1 is now known as Cueball [18:24] hey guys [18:25] any idea how to identify a process pinging tonnes of udp connections? [18:25] 19:24:40.561770 IP 220.241.62.169.49838 > 85.119.82.243.10320: UDP, length 172 [18:31] that can be tricky [18:33] yeah, proving to be [18:33] have spent a few hours on it so far [18:34] have you tried looking at the contents of the packets to see if they suggest anything? [18:34] hmm I wonder if you can do it with perf's trace [18:35] the packets contents don't give me any clues [18:35] andylockran: try perf trace -a -e sendmsg [18:35] though I can tell it looks like it's being used as a rtp relay [18:36] says he guessing it's using sendmsg to send the packets [18:40] hmm, kinda buging me now [18:40] andylockran: Try that perf command - it should show every process using the sendmsg syscall [18:42] merlin:/# perf [18:42] /usr/bin/perf: line 24: exec: perf_2.6.32: not found [18:42] E: linux-tools-2.6.32 is not installed. [18:42] if you tab complete on perf_ do you have anything? [18:42] hmm bit old, might not have trace [18:46] andylockran: The other way that might work is creating an iptalbes rule to block them - not sure if that would show the process in the log? [18:50] Ok - blocking the process via ip seems to have blocked the data [18:50] but the process is still running [18:51] but are you generating logs showing it? [18:52] all I've got at the moment is the tcpdump [18:53] andylockran: I mean make the iptables log the rejects - I can't remember what info you get in the logs when you do that [18:53] ooh, ok [18:55] https://gist.github.com/anonymous/4dd0e8da0d82cfa8e63a [18:55] andylockran: ah, external syslogging? [18:56] what's pid 1881 then? [18:56] named [18:57] so is that saying that named is sending zillions of moans to rsyslog for some reason? [18:57] still going strong with those proceses stoped [18:58] did you check if you had perf_something installed ? [18:59] I've got perf_3.2 [18:59] perf: 'trace' is not a perf-command. See 'perf --help'. [19:00] yeh probably too new a feature [19:02] thing is if they were going to rsyslog I'd have expected some text in the packet - how are you capturing them with tcpdump? [19:02] https://gist.github.com/anonymous/793876c3889facf530c3 [19:03] yeh what tcpdump command are you using? [19:03] curious, do you have then coming inbound too? [19:03] I used wireshark [19:03] ShireWark [19:03] inbound, nthing. [19:04] andylockran: So with your wireshark, what's the contents of those packets - 200 bytes of something [19:04] https://gist.github.com/anonymous/a1c43dbb4ab4bea1f808 [19:05] hmm, 200 bytes of not much [19:05] 22:43:39.533789 IP (tos 0x80, ttl 56, id 30442, offset 0, flags [DF], proto TCP (6), length 100) [19:05] 80.229.11.208.57889 > 85.119.82.243.22: Flags [P.], cksum 0x67c6 (correct), seq 192:240, ack 196913, win 8192, options [nop,nop,TS val 512981144 ecr 183589], length 48 [19:06] there are loads - lots of them each second [19:07] andylockran: You could trace strace -p on every process one at a time until you find the victim [19:07] andylockran: is something showing up in top as using cpu - if it's shifting that much you'd think it would be [19:08] that last one I'd expect to see loads of [19:08] yeah - nothing is showing load [19:10] if you just run perf top for a few seconds what's it showing? [19:10] loads and loads. it's your ssh connection. so as you try to list the packets, it's creating more packets, so it lists more packets .. [19:11] shauno: Oh yeh, that's not the UDP packets he was previously complaining of though [19:11] right, just pointing out that the last one is a red herring :) [19:12] andylockran: So the packet you grabbed the hex of - how did you select that, have you got the actual udp packets you're worrying about [19:13] Just randomly selected [19:14] they all seem v. similar at the top level [19:14] the IP that they're comunicating with is weird though [19:14] 220.241.62.169 [19:15] which comes up as a phone site - so I guess I'm relaying their calls for free [19:15] hong kong. lovely. [19:16] even with inbound and outbound conns blocked to that ip, it continues. [19:16] andylockran: iptraffic rule? [19:16] iptables I mean [19:17] andylockran: However, if your box has been owned it could have a hidden process that won't show up in top or anything [19:17] set https://gist.github.com/anonymous/ec3cb8a7cc08801ff1b0 [19:17] penguin42: scared it's the latter :( [19:18] andylockran: so what does perf top show ? [19:19] penguin42: nowt [19:19] andylockran: How do you mean nowt [19:20] it doesn't show anything. just says 0 cycles (not sure how to use it though) [19:20] really? Oh never seen that [19:22] andylockran: What distro? [19:23] debian wheezy [19:23] the other possibility is it's just got something like an ipsec or other kernel level vpn enabled [19:24] andylockran: if you do mount -t debugfs nodev /sys/kernel/debug does it mount it? [19:24] https://gist.github.com/anonymous/d594efb9b8523dc679b8 [19:25] ok, what about ls /sys/kernel/debug/tracing ? [19:25] ls /sys/kernel/debug/tracing [19:25] available_events buffer_size_kb events per_cpu README set_event trace_clock trace_options tracing_cpumask tracing_on [19:25] available_tracers current_tracer options printk_formats saved_cmdlines trace trace_marker trace_pipe tracing_enabled [19:25] ooh that's promising - now can I figure out ftrace [19:25] cheers for your time buddy - much appreciated [19:26] andylockran: Can you install the trace-cmd package? [19:27] git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git? [19:27] yes, although it should be packaged - it is on ubuntu, not sure about wheezy [19:28] installed [19:29] andylockran: so hmm now the idea is to do trace-cmd record something to record something [19:29] andylockran: Maybe all the syscalls [19:31] https://gist.github.com/anonymous/ae9c17f2124c3a77612c [19:31] andylockran: how about trace-cmd record -e syscalls:sys_exit_sendmsg [19:32] https://gist.github.com/anonymous/7a07a716341c6ddc7368 [19:32] andylockran: trace-cmd list |grep send [19:34] merlin:~/trace-cmd# trace-cmd list |grep send [19:34] sched:sched_signal_send [19:34] hmm that's boring - it doesn't seem to have syscalls [19:35] how about grep for net instead of send? [19:36] if it has net_dev_xmit it would be good [19:36] nothing on net [19:36] ok, that's just too old [19:37] Well this seems to of worked. Server Upgrades FTW [19:37] perf_5.2 [19:37] andylockran: Other than strace'ing every pid I can't think of much [19:37] andylockran: Your kernel/perf is just too old for any of the funkier tools [19:38] aww . damnit :( [19:39] andylockran: Tried running rkhunter? [19:40] andylockran: Anything fun in dmesg? [19:40] nowt [19:42] andylockran: well I'm out of ideas - it's not looking good for the machine though [21:13] why do they always put blue LEDs in wifi dongles :/ if something's going on the side of my laptop, it shouldn't have a rescue strobe attached to it [21:44] shauno: Because we've run out of other LED colours that are 'cool' [21:45] I'm entirely in favour of no LEDs [22:58] hrrm. As far as I can tell, Ubuntu 14.04 LTS is released.... why doesn't do-release-upgrade see it? [23:00] * penguin42 has half a memory that in 10.04 the update didn't happen until 12.04.1 was out, but I might be imagining that [23:02] penguin42: you're right [23:02] Users of 12.04 LTS will be offered the automatic upgrade when 14.04.1 LTS is released, which is scheduled for July 24th [23:02] http://fridge.ubuntu.com/2014/04/17/ubuntu-14-04-trusty-tahr-released/ [23:03] thanks penguin42 [23:03] still, I should probably upgrade my dad's 12.10 [23:09] is there an ubuntu ppa related channel? [23:16] not that I'm aware of, there is #ubuntu-packaging if your problem is rying to package stuff [23:25] trying to find out if you can put a binary package you have no source for in a ppa [23:30] I suspect that's technically doable, not sure if it's allowed under the rules or not