[00:00] anyone else seeing weird tcp connection problem with 3.13+ kernel [00:08] define weird === Metacity is now known as edocb === edocb is now known as Guest71787 === Guest71787 is now known as Metacity === Lcawte is now known as Lcawte|Away [01:06] like.. [01:07] ssh to 127.0.0.1 and it hangs for anywhere from 10 seconds to minutes, sometimes connects and sometimes resets broken pipe etc [01:07] same thing with curls and such [01:07] starts happening about 10ish mins after reboot [01:07] works fine until then [01:07] almost like a memleak of some sort [01:07] happens in every kernel 3.13+ i've tried [01:07] seems fine in older ones [01:09] I'm on 3.13.0-44-generic and "time ssh localhost date" takes 1.9 seconds for the 'first' run and 0.2 seconds for each additional run [01:09] hangs with ssh often mean the server is trying to do a reverse DNS lookup on the client's IP address [01:10] yeah i know but it's not that [01:10] but that doesn't make sense re: ten minutes after reboot [01:10] localhost is in the /etc/hosts [01:10] plus it does it with curls (http request) [01:10] and on top of that i can't open any listening sockets either [01:10] like trying to start a service that listens on port 5000 [01:10] it won't listen on the port even though nothing else is there [01:11] it's extremely weird problem [01:11] ssh localhost is instant for some time [01:11] it asks for password instantly every time [01:11] and then hangs [01:12] is there anything strange in dmesg? how do netstat or ss output look? [01:12] nothing major, it's running several services [01:12] Transport Total IP IPv6 * 2108 - - RAW 0 0 0 UDP 11 8 3 TCP 107 104 3 INET 118 112 6 FRAG 0 0 0 [01:12] but not much connections [01:12] sometimes ssh connects in 10-60 seconds [01:12] and sometimes it broken pipe/reset by peer [01:13] after a few mins [01:13] but this is destroying the HTTP services running on the box [01:13] even curl to 127.0.0.1 does same thing so as uc an imagine http doesn't work :) [01:13] once a connection is established (ssh or whatever) it's fine [01:14] nothign in dmesg [01:14] was wondering if it had something to do with apparmor [01:15] X123: apparmor would show up with DENIED messages in dmesg or auditd logs [01:15] yeah, and it's not [01:15] but u never know :0 [01:15] does same on 3.16 [01:16] and multiple servers are doing it [01:16] not just one [01:16] but then again we have a few more servers that aren't AS BAD [01:16] like ssh takes 8-16s [01:16] and always seems to connect [01:17] and it's random that it does that, most of the time it's 1s or so [01:17] but no matter what it should always be instant [01:17] kind of driving a few of us nuts haha [01:19] if you have any clue or direction to figure it out would be appreciated [01:20] sniffing loopback and watching it shows interesting results sometimes [01:21] sometimes it gets stuck retramsitting over and over, and sometimes it gets no response [01:21] is very very odd [01:21] Can someone help me with an iptable to redirect all traffic from an interface to a port? [01:22] like what? [01:22] i mean you can't redirect ALL traffic from a layer2 interface to a layer3 port [01:22] all port 80 traffic to a Squid3 server [01:22] so have to be more specific :) [01:22] I know and iptables are very new to me lol [01:23] so when someone connects to port 80 of the ip address on your interface [01:23] u want it to go to some other ip on anothe rport [01:23] or do u want all traffic that gets forwareded through like a router to be redirected [01:23] Well the laptop or lan is connected to one port on my server which is vswitched to Eth1 of my Ubuntu serer [01:23] server [01:24] ok [01:24] I need all of that web traffic redirected to Squid3 and then on the outside redirected from Squid3 to Eth0 which is Vswitch to another physical port on the server. [01:24] All other traffic redirected straight from Eth1 to Eth0 [01:24] redirected.. as in? [01:24] like NAT [01:24] or routing ? [01:25] is there an service available that can monitor bandwidth usage? [01:25] That I am not sure of. Eth0 is on 192.168.1.0 net and Eth1 ison 192.168.2.0 net [01:25] so a "static" NAT for all the other traffic maybe? [01:25] cyclob|work: like a program you run on the command line, or what? [01:26] you can nat any traffic leaving eth0 [01:26] to the ip of eth0 [01:26] yeah so i can stick a box between a switch and router and graph the bandwidth usage over a week [01:26] X123, I tried this... http://pastebin.com/Lme2GAxU [01:26] why not just graph the bandwidth usage on the router or switch port [01:27] no snmp or router access sadly [01:27] whaa [01:27] yeah managed routers [01:27] so you want to insert a server in between it and bridge the interfaces [01:27] costs $$$$ to chagne anything on them. lol [01:27] cyclob|work: check out iptraf or munin or similar [01:27] just run SNMP on the server [01:27] and use cacti or something to graph the interfaces [01:27] Kind of... I want the Squid3 server to be directly behind my ISP modem and the Lan behind that [01:28] works just like it's a router or switch [01:28] what's the squid server for? [01:28] cool i'll check them out [01:28] its a proxy server... [01:28] to allow or disallow based off of rules. [01:28] yeah just apt-get install snmpd i think [01:28] Which I then build Dilidele WS on top of that for content filtering [01:29] so you want the squid server to transparent proxy [01:29] everything coming from your lan [01:29] Yes! [01:29] And I am familiar with firewall rules and such on ASA's but iptables is a foriegn language to me [01:29] that's easy enough [01:30] X123, this is for home networking btw [01:30] you really should specify interfaces in iptables [01:30] like postrouting -o eth1 [01:30] and I would prefer that because the ISP side may change [01:30] u don't want to nat everything on every interface [01:30] hey,,, if anyone can answer this i would be very appreciative https://askubuntu.com/questions/573904/setting-up-subdomains-for-ubuntu-server-14-04 [01:30] to give you a layout of what I have.. [01:31] Right now is it ISP>Router>Server>Laptop [01:31] for testing [01:31] server is acting as a switch [01:31] for the laptop yes. [01:31] And with NORMAL squid... IE... [01:31] so laptop has an ip from router [01:31] Directing the laptop to the squid it works [01:31] like eth0 on router goes to ISP [01:31] eth1 goes to server [01:31] umm ... [01:32] http://puu.sh/ew2LH/59f97f043e.png [01:32] top is what I have now [01:32] and I can't get it working [01:32] Once I figure it out ill put the server between the IPS and the internal LAN router [01:33] so the server is doing NAT also [01:33] to the router, which is doing nat to the isp lol [01:33] Server isnt yet... [01:33] I just have the interfaces on the server configured for those IP's [01:33] then how does the laptop get internet access [01:33] static [01:33] atm [01:34] Ohh umm [01:34] It doesn't [01:34] UNLESS [01:34] I manually put in the proxy info [01:34] which I don't want to do [01:34] oh so it has no internet at all [01:34] you just want port 80 to work [01:34] and ntohing else? [01:34] well eventually 80 and 443 to the proxy... [01:34] everything else straight out [01:34] if that makes sense lol [01:35] yeah [01:35] Right now, anything that has a destination port 80 [01:36] http://www.tldp.org/HOWTO/TransparentProxy-6.html [01:36] this 6.2 method is the best imo [01:36] checking it out now [01:37] hah, I read that HOWTO back in the ipfwadm days [01:37] nice to see it's been updated since then :) [01:37] haha [01:37] it still works :) [01:37] This sounds like the iptables is a seperate server.. [01:37] yeah the iptables is the router [01:37] squid is separate [01:38] can they be on one? [01:38] sure [01:38] squid-box = squid server ip right? [01:38] yeah [01:38] but you wouldn't need the extra routing [01:39] -s near the end of the first command is source? [01:40] http://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxRedirect [01:41] this should be easier to see for one device [01:41] yea I tried that page [01:41] of course i'd specify interfaces in iptables [01:41] i dunno why they didn't [01:42] i ALWAYS specify interface :) [01:42] -o and -i [01:42] Actually thats a different page [01:42] masquerade is outbound nat [01:42] 1 sec.. [01:43] it's ip unspecific is why they use it because if eth0 connects to the internet, the ip might change [01:43] so masquade just picks whatever IP is on that interface [01:43] and uses it [01:43] I don't see an eth0 on that page [01:44] there isn't :) [01:44] there should be.. :) [01:44] if eth0 is internet facing [01:44] then you want -o eth0 for the masq [01:44] this look good? http://pastebin.com/4PSmMVfw [01:44] just so there's no confusion [01:44] after masq [01:45] iptables -t nat -A POSTROUTING -j MASQUERADE -o eth0 [01:45] before, but im sure it will move it [01:45] basically means, anything exiting eth0, nat to the ip of eth0 [01:45] iptables -t nat -A POSTROUTING -j -o eth0 MASQUERADE ? [01:46] -A POSTROUTING -o eth0 -j MASQUERADE [01:46] k ill try [01:46] then the prerouting ones [01:46] u want -i [01:46] ... [01:46] input interface [01:46] i mean it's not necessary [01:46] i just don't like iptables doing things i don't want it to do [01:47] on all prerouting? [01:47] prerouting happens before any processing [01:48] http://pastebin.com/vF5wSbfc [01:48] like the first thing it does when a packet comes in an interface = prerouting [01:48] postrouting happens after it processes the packet in routing table and decides what interface is is going to send the packet out [01:48] i think the last prerouting is -i eth0 [01:48] ill try those commands [01:48] right sorry [01:48] because it's blokcing people on the internet from accessing the squid port [01:49] So if the dport isnt 3128 drop it [01:49] that drops everything going to 3128 [01:49] if it comes in eth0 [01:49] but it's not coming in eth0, it's coming in your lan eth1 [01:50] http://puu.sh/ex2wD/9242aef150.png [01:50] ill try it [01:52] nope... not even seeing access on the squid logs [01:53] is there a way to view traffic on the iptables? [01:53] u can see counters [01:53] how? lol [01:53] like iptables -t nat -L -vnx [01:53] or whatever table u are using -t mangle -t filter [01:54] postroute is only one with counters [01:54] well you are accessing a real internet ip from the laptop? [01:54] cnn.com [01:54] google.com [01:55] stuff like that [01:55] wait [01:55] Pings work [01:55] so.......... [01:55] DNS not working [01:55] you'd have to use DNAT for redirecting a real internet ip to the squid proxy [01:56] because redirect of a port will just change the port [01:56] but it will still end up going out [01:56] ok well getting closer lol [01:56] I havent been able to get that ping to work in 3 days trying this [01:56] Sec... gotta change the squid config to intercept [01:56] duh? [01:57] ping will work because of masq [01:57] gotcha.. and were not trying to intercept icmp [01:57] trying to intercept port 80 [01:58] actually it intercepts 3128 [01:58] hmm nslookup.. laptop is able to resolve [01:58] and you DNAT port 80 to 3128 [01:58] thats what udp 53? [01:58] aye [01:58] Ok so DNS is working [01:59] do a tcpdump -n -i eth0 [01:59] and try access web site [01:59] i bet you will see it trying to access 3128 [01:59] iptables -t nat -A PREROUTING -i eth1 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.2.1:3128 [01:59] try this? [01:59] yeah that's what u need [01:59] that redirects connection attemps to 80 to that ip [01:59] ok do I have to "restart" iptables? [01:59] not if you typed that from the shell [02:00] it puts it in instantly [02:00] but it appends it to the end [02:00] -A is append [02:00] so u have to look at where you are inserting it [02:00] ok it dropped it at the bottom.. top down is how it runs so how do I move it up? [02:00] and then u have to redirect the REPLY === markthomas is now known as markthomas|away [02:01] I don't see anything like sequence numbers in here [02:01] u can do --line-numbers [02:01] to see the sequence # [02:01] honestly i just edit a file [02:01] i do iptables-save >file [02:01] edit file [02:01] Where is it stored? [02:02] iptables-restore < file [02:02] that's how i do it :) [02:02] i don't even bother with it being saved anywhere [02:02] so make a new file with what I want then run that command [02:02] just type iptables-save [02:02] u will see [02:02] so u can > that to a file [02:02] edit that file [02:03] then iptables-restore < file [02:03] and it puts in in the order in the file [02:03] it's 100x easier [02:03] imo [02:03] ok... where did it save to? [02:04] nvm.. I did iptables-save > iptables.txt [02:05] you just need 2 rules [02:05] in iptables [02:05] plus the masq [02:05] http://puu.sh/ex4vX/17d898e47b.png [02:06] redirect rule should be lan [02:06] like [02:06] iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 3128 [02:06] if eth1 is your LAN int [02:07] no wait [02:07] should be eth0 [02:07] yea eth 1 [02:07] cuz it has to do it before it goes out eth0 [02:07] iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 3128 [02:07] is whats in there [02:07] so you will have two rules like this [02:07] iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j DNAT --to 192.168.1.1:3128 iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128 [02:07] or whatever your squid IP is [02:07] in that order [02:08] should work :) [02:08] i never tried it like this tho [02:08] so the DNAT is the outside [02:08] and then you'll have the masq [02:08] DNAT is LAN eth1 [02:08] it redirects anything on your LAN [02:08] that tries to connect to anything on port 80 [02:08] routed though this server [02:08] to that squid ip [02:09] ok so LAN is 192.168.2.0, Squid Eth1 is 192.168.2.1 so use 2.1 on the DNAT [02:11] squid server would be 192.168.1.1 in my example [02:11] http://puu.sh/ex5cd/95196f94b4.png [02:11] which is running on the local machine [02:11] that's what the -j REDIRECT does [02:11] redirects it to the local machine only [02:13] that last pic I sent still isnt working :/ lol [02:13] lemme check squid logs [02:13] nope. Squid log shows 0 traffic [02:14] http://www.cyberciti.biz/tips/linux-setup-transparent-proxy-squid-howto.html [02:14] check this [02:14] so at this point, I need to get port 80 traffic to 3128 just to start [02:14] that looks like it explains it better [02:14] you can verify that with tcpdump [02:14] but that 3rd rule under the NAT table should do it [02:14] that's how i do it [02:14] maybe rp_filter is on [02:14] and the machine is being a @()# [02:14] make sure rp_filter=0 [02:15] edit the /etc/sysctl.d/10-network-security.conf [02:15] and make those 0 [02:15] that breaks so much stuff that they default to 1 [02:15] soooo much [02:15] syncookie 0 too [02:16] this looks like my exact setup [02:16] yeah [02:16] but make sure rp filter! [02:16] is 0 [02:16] if you're going to turn off rp_filter, it's worth making sure your firewall rules enforce something sane [02:16] since u are doing nat tricks [02:16] i always turn it off, it causes more problems than anything [02:17] much easier to iptables stuff [02:17] X123: any luck with your problem? [02:17] sarnold no :/ [02:17] unfortunately [02:17] X123: dang ;( [02:17] it doesn't do it on centos running same thing [02:17] or previous kernels [02:17] I hoped I had just missed it in the middle of the rest.. [02:17] just 3.13+ so far [02:17] i'm dying to know what it is [02:18] X123: please file a bug against "linux" [02:18] well i've only tried the ubuntu kernels [02:18] i might download and compile one myself [02:18] and see [02:18] that's a lot of work though lol [02:19] X123: when you file one against linux, there's a robot that will ask you test some 'upstream kernels', all precompiled and ready to go [02:19] ill get one of the techs to do it.. i'm the network engineer :P [02:19] tested 3.16 [02:19] same stuff [02:19] rp_filter is the most annoying thing ever [02:19] it just silently discards packets [02:19] wish it was default 0 in every dist === zz_DenBeiren is now known as DenBeiren [02:31] k2gremlin u get it working [02:31] X123 nope not yet. Crusing through the iptables to figure out why its not forwading 80 to 3128 :/ [02:32] did u disable rp filter [02:32] That site is old... but I can sort of understand it lol [02:32] Yes [02:32] there were 2 commands wiht it [02:32] tcpdump :) [02:32] that is my friend [02:32] there's a script on that site that sets it up [02:32] check out the script, it has a bunch of iptables commands [02:33] Yea I ran it... [02:33] but some of the block a lot of stuff which u don't want [02:33] but the config portion for squid is really ol [02:33] old [02:33] the squid commands are not valid anymore. [02:33] ah [02:33] never used squid :) [02:33] i use nginx [02:33] only traffic was ssh with my putty and an arp request [02:33] use tcpdump to trace it [02:33] see what's coming in the lan int [02:33] see what's on lo [02:33] and see what's going out [02:34] can I grep an interface? [02:34] u can put options on tcpdump [02:34] like what do u want to do [02:34] see whats coming in eth1 [02:36] just tcpdump -n -i eth1 [02:37] 20:36:52.028698 IP 192.168.2.1 > 192.168.2.10: ICMP 192.168.2.1 udp port 53 unreachable, length 65 [02:37] yea see I can't even ping using their rules [02:37] lol [02:37] yeah like i said [02:38] it blocks pretty much everything [02:38] so u can just edit out all the blocking stuff [02:38] ping uses icmp, a different protocol than tcp... [02:38] you should use tcpdump -n -i lo [02:38] to see traffic on local machine [02:38] sarnold, tcpdump shows everything [02:39] even mpls, gre, layer2 info [02:39] everything :) [02:39] not sure why it's called tcpdump but it shows it all [02:39] X123: hehe, I meant that more along the lines of "you can't ping because you're only forwarding tcp traffic" ... [02:39] oh eheh [02:39] X123: tcpdump and I have a love/hate relationship :) [02:39] watching lo isnt generating anything [02:39] X123: ... I love the stupid thing but hate the language, I have to use the manpage every. single. time. [02:40] it's not tcpdump language [02:40] it's pcap [02:40] so, hate pcap :) [02:40] oh I do :) [02:40]