| mbff | Hello! Ubuntu seems to be preventing port forwarded traffic (I've tried Nginx and a python web server). Clients on the lan get a response. I've tried disabling UFW and I am seeing logs in tcp dump so the traffic seems to be getting to server. | 03:25 |
|---|---|---|
| mbff | One note would be that the port forwarded traffic isn't "internet" traffic but local traffic from the router "above" my router. | 03:26 |
| mason | mbff: iptables -n --list | 03:26 |
| mason | mbff: Also worth mentioning if this box is sitting on a LAN, is a gateway, is otherwise doing something interesting. | 03:27 |
| mbff | https://gist.github.com/marshallford/fb968423928a945dcb2a452f130ddd16 | 03:28 |
| mason | So, yeah, that's a bunch of rules with a default policy of "drop". | 03:29 |
| mbff | the ubuntu server running the web server is on a LAN inside a LAN. The inner network is 10.0.X, the outer 192.168.X. I am trying to hit the webserver from the 192.168.X ip, one NAT "up" | 03:29 |
| mbff | mason, I can disable ufw to help narrow down things | 03:30 |
| mason | mbff: I'm not immediately clear on what all the docker rules are accomplishing, but in general, clear your iptables rules out and slowly build up. | 03:30 |
| mason | If you're using random tools to build firewalls for you, make sure you understand how to tell them what you want. | 03:31 |
| mbff | I do know how to tell ufw what I want. I am very explict in my setup script/automation | 03:32 |
| mbff | the docker rules are added by, well docker | 03:32 |
| mbff | with ufw disabled: https://gist.github.com/marshallford/872911698f1eba780d47df57087dc635 | 03:32 |
| mason | That's not a cleared out list of iptables. :P | 03:33 |
| mbff | right... | 03:33 |
| mason | That said, maybe reassert all the rules, like what you'd get freshly booting, and look at hit counts. But since something is setting you up as default-drop, it might not be explicit rules knocking down packets. | 03:33 |
| mason | So, iptables -n -v --list | 03:34 |
| mason | or -L - not sure why I default to the long argument for list. =cough= | 03:34 |
| mason | Being a dinosaur, I build explicit rules and invoke them from /etc/network/interfaces, and it ends up never being mysterious, which I see as a win. | 03:35 |
| mbff | "Being a dinosaur" :) | 03:36 |
| mbff | iptables with verbose flag: https://gist.github.com/marshallford/e9edd9699f7869aaaaea6ea8f94135f4 | 03:37 |
| mbff | how would I check hit counts? | 03:38 |
| mason | mbff: Anyway, here's what I'd do. Maybe someone will have a better idea. Reboot, get all your rules up as normal, and see some traffic knocked down. Then pick one of iptables -v -n -L to see what rules are hitting. If none, change your default policy to ACCEPT with iptables -P and see if that lets traffic in. If it does, you'll have to craft rules to explicitly accept the stuff that's otherwise being | 03:38 |
| mason | dropped. | 03:38 |
| mason | Sorry, you get the hit counts from the -v | 03:41 |
| mbff | mkay | 03:42 |
| mbff | mason, this ufw-not-local chain seems suspicious | 03:46 |
| mason | mbff: Yeah, it has a couple hits. | 03:46 |
| mason | You want to generate the traffic that's being blocked and see if the hit count rises. | 03:47 |
| mbff | should I look up the pkts or bytes number? | 03:48 |
| mbff | ADDRTYPE match dst-type BROADCAST seems to be going up | 03:49 |
| mason | mbff: I'd think packets, because bytecount will vary based on how big the packets are. | 03:53 |
| mbff | My guess is one of the ufw auto generated rules is blocking requests from IPS that are class A,B,C that aren't the in the range of it's own ip class. | 03:53 |
| mason | mbff: Generate that traffic and watch the packet counts. No need to guess! :) | 03:53 |
| mbff | Can I get iptables to show me the change in traffic packets between rules? I have no idea which rule it is | 03:54 |
| mbff | Like can iptables narrow down output based on change | 03:54 |
| mason | mbff: save a series of outputs, and use diff to see what changed | 03:57 |
| mason | You can turn on logging to see hits logged, but that's too much work for this. | 03:58 |
| mbff | the diff is helping, but still a lot has changed | 04:01 |
| mbff | Would target RETURN cause the issue I am having? | 04:02 |
| mason | mbff: https://unix.stackexchange.com/questions/191607/iptables-and-return-target#191614 | 04:06 |
| mason | so it's not a target as such | 04:06 |
| mason | mbff: But if nothing explicitly accepts the traffic, your default DROP will get it. | 04:06 |
| mbff | After looking at the diff, I don't see anything that stands out. | 04:08 |
| mbff | How can I completly reset iptables, back to the system default before ufw was installed | 04:09 |
| mason | mbff: I'd reboot and get whatever comes up on boot, and try your blocked traffic after manually changing the policies to ACCEPT. | 04:11 |
| mason | That'll tell you if it's the default getting you. | 04:12 |
| qman__ | mbff: you can clear chains with iptables -F (must do nat and mangle separately, iptables -t nat -F, ...) | 04:13 |
| qman__ | and you can remove chains with iptables -X | 04:13 |
| mason | qman__: He needs to be careful with this, because flushing tables with a default DROP will lock him out if he's not on the console. | 04:13 |
| qman__ | yep, was about to mention setting policy | 04:13 |
| qman__ | iptables -P OUTPUT ACCEPT; iptables -P INPUT ACCEPT | 04:14 |
| mbff | here is the thing, this worked traffic worked earlier today on a base install of ubuntu server that was up to date. | 04:14 |
| mbff | So something in my ansible scripts (including ufw scripts) caused this, most likely | 04:15 |
| mason | mbff: A base install doesn't have a default DROP does it? Then again, I don't run ufw. | 04:15 |
| mbff | ufw isn't in the base install | 04:15 |
| qman__ | no, the base install has no rules and ACCEPT policies | 04:16 |
| mbff | so mostly likely ufw has some default rules it doesn't tell you about that are screwing me over | 04:17 |
| mbff | can I safetly run iptables -X ? I'm worried about docker rules coming back properly, since I didn | 04:18 |
| mbff | didn't write those | 04:18 |
| mason | mbff: Well. If you don't see a specific rule going up as that traffic is dropped, you'll want to add rules to explicitly allow the profile of whatever traffic you need to come in. | 04:18 |
| mason | mbff: That stuff will come back on boot if it's set up properly. | 04:18 |
| mbff | I do though, my script runs ufw allow 80 | 04:19 |
| mason | Reboot and see the world from that perspective, and see if changing policy temporarily fixes you. | 04:19 |
| mason | If it does, do whatever ufw requires to allow the specific traffic you want. | 04:19 |
| mbff | i have done that, `ufw allow 80` | 04:19 |
| qman__ | no, if you're running docker, it's not safe to flush iptables out or delete chains | 04:20 |
| qman__ | docker does its own mangling of iptables | 04:20 |
| mason | A blanket allow like that might be too wide. | 04:20 |
| mbff | I'm not really using it atm, I don't care about docker short term | 04:20 |
| mbff | using docker* | 04:20 |
| qman__ | if you don't care about breaking running apps, then yes, you can flush everything out; set policies to accept first, then flush, then delete chains | 04:21 |
| mbff | Still not working, my iptables look like https://gist.github.com/marshallford/a00d38d0ad02eabe0cc074974538d4d6 | 04:26 |
| mbff | I'm wondering if this isn't a firewall issue | 04:26 |
| qman__ | that is effectively no firewall, so I'd say so | 04:26 |
| mason | mbff: If it's not working with that, then it might not be a firewall issue. Look at your routing tables next... | 04:26 |
| mason | netstat -rn | 04:27 |
| mason | make sure you have a way to reply I guess? | 04:27 |
| mason | maybe use wireshark/tshark and make sure what's hitting your server is sane | 04:27 |
| qman__ | yeap | 04:27 |
| qman__ | also check your nat table | 04:27 |
| mason | maybe use wireshark on the other end of the connection and verify both ends at once | 04:27 |
| qman__ | iptables -nvL -t nat | 04:27 |
| qman__ | it shouldn't have anything in it, but if it does, it could break things | 04:28 |
| mbff | wep I am overwhelmed by advice, not a bad thing I guess haha | 04:28 |
| mason | mbff: Just save it all for later reference. :) | 04:29 |
| mbff | I figured it out! | 04:31 |
| mason | What was it? | 04:32 |
| mason | And congrats. :) | 04:32 |
| mbff | When openvpn is running, it doesn't work | 04:32 |
| mbff | when I run service openvpn stop && service nginx restart, It works | 04:32 |
| mason | mbff: Ah. I bet if you dig further you'll see routing tables telling you what was wrong. | 04:32 |
| mbff | the question is now how I get it to work when openvpn is running | 04:32 |
| mason | Like, you were taking your replies and trying to throw them back over the VPN. | 04:32 |
| mbff | yep | 04:33 |
| sarnold | try ip route get 8.8.8.8 or something similar | 04:33 |
| mason | mbff: Well. Check your routing tables, and if you can, add specific routes to do the right thing and tie the traffic that shouldn't go over OpenVPN to a real interface. | 04:33 |
| mbff | mason, I understand those words... | 04:33 |
| mbff | I have not done any route work before | 04:34 |
| sarnold | oh you are in for a treat^Wfrustrating evening! :) | 04:34 |
| mbff | lol | 04:34 |
| mbff | That's the reason I have this homelab | 04:34 |
| mason | mbff: netstat -rn or whatever the mystical new ip equivalent is to start | 04:34 |
| mbff | to learn | 04:34 |
| sarnold | seriously though, 'ip route get' is amazing stuff. | 04:34 |
| mason | mbff: In essence, what you're going to find is that your OpenVPN wants to grab traffic that matches the stuff coming in from your nat. | 04:35 |
| mason | Is there an 'ip route discard' to go with it? | 04:35 |
| sarnold | haha :D | 04:36 |
| mason | mbff: So, as an example here, I have a VPN that's greedy and wants to grab a big range, so I shove in a more-specific rule in front of that to save local traffic from being gobbled up. | 04:36 |
| mason | This way, my local traffic doesn't get sucked into a black hole, but I don't have to deal with a conflict with the VPN either. | 04:36 |
| mason | If there were systems on the other side of the VPN that used my local range I'd be stuck, but there aren't, so it works. You might end up wanting to do something similar. | 04:37 |
| mason | And if it's confusing, I apologize. Just be glad you're dealing with static routes. :) | 04:37 |
| mbff | I don't even know how to filter my problem | 04:37 |
| mbff | out | 04:37 |
| mbff | this guy has my same problem https://ubuntuforums.org/showthread.php?t=2296618 | 04:40 |
| mason | mbff: FWIW, use sarnold's ip route get foo syntax to be compliant with the modern world, but try netstat -rn for more human-parseable view of what's going on. | 04:40 |
| sarnold | mbff: be careful with forum posts .. it's too easy to find similar symptoms and get dragged off into the weeds chasing someone else's advice .. | 04:41 |
| qman__ | mbff: openvpn client? if so, make sure your openvpn config is not adding a default route over the VPN - this is most likely the issue | 04:42 |
| sarnold | mbff: in this case it might actually be perfect :) has your openvpn been configured to send *all* traffic on the machine through the pipe? that might be ideal if you're trying to get out from the great wall of china or something but rough if you just want a few systems linked together.. | 04:42 |
| mbff | I'm looking at it, I just don't understand how to parse it, I don't fully understand the terms and how they related to form the "story" X traffic goes to Y if Z" for example | 04:42 |
| mbff | sarnold, the vpn is not for connecting sites, strictly for ip masking, privacy, etc | 04:43 |
| sarnold | this might not be easy to fix | 04:44 |
| qman__ | it may not be fixable, they're essentially incompatible configurations | 04:44 |
| qman__ | it's possible in theory to convince the kernel to return the incoming traffic locally instead of following the default route, but it depends on the VPN's implementation | 04:45 |
| mbff | this guy solved the problem with a change the openvpn config https://serverfault.com/questions/614281/openvpn-client-breaks-remote-access-to-web-server | 04:46 |
| sarnold | he probably did not want to use openvpn for ip masking and privacy in the same way that you do. | 04:49 |
| * mbff shrugs | 04:51 | |
| qman__ | that might actually be the config you're looking for, it effectively sets routes for all outbound traffic over the VPN without disrupting the default route and without using the redirect-gateway feature | 04:51 |
| qman__ | you'll have to test it though | 04:52 |
| mbff | what does 128.0.0.0 mean? It seems arbitrary? | 04:53 |
| qman__ | it is arbitrary | 04:53 |
| qman__ | normally, a default route is set for 0.0.0.0/0 | 04:53 |
| qman__ | this configuration is setting routes for 0.0.0.0/1 and 128.0.0.0/1 | 04:53 |
| qman__ | they're narrower, scopes, splitting all address space in half | 04:53 |
| sarnold | hah I'd argue that is very much NOT arbitrary :) | 04:54 |
| qman__ | because they're narrower, they're preferred over the default route | 04:54 |
| sarnold | I mean you could achieve similar things with dozens of other configs .. but this is the simplest | 04:54 |
| mbff | so how would I figure out what routes/numbers I need? | 04:54 |
| qman__ | those are the exact routes you'd need | 04:55 |
| qman__ | it doesn't change | 04:55 |
| mbff | ie: why not 122.0.0.0/1 | 04:55 |
| sarnold | because that's not a well-formed CIDR input | 04:55 |
| qman__ | because 0.0.0.0/1 covers from 0.0.0.0 to 127.255.255.255, while 128.0.0.0/1 covers 128.0.0.0-255.255.255.255 | 04:55 |
| sarnold | (seriously, some tools have such a panic with that, security problems and all..) | 04:55 |
| mbff | I worried I'm exposing my real ip to sites that start with 128.X | 04:56 |
| sarnold | it'd be 128.x-255.255.255.254... | 04:56 |
| qman__ | those two routes cover the entirety of the ipv4 address space | 04:56 |
| mbff | like I said, I'm new to some of this networking stuff outside the typically lan world | 04:56 |
| qman__ | so, provided it works at all, it will work for all destination IPs | 04:56 |
| mbff | I'll trust yall... :) | 04:57 |
| sarnold | if it matters, it'd be worth *testing* this on non-hostile services first.. | 04:57 |
| qman__ | yes | 04:57 |
| qman__ | change the config, reconnect, and test with an IP checking service, like ipof.me | 04:58 |
| mbff | curl ifconfig.co gives me my real ip, but openvpn is running and the port forwarded traffic is getting though | 04:59 |
| lordievader | Good morning | 07:12 |
| jamespage | coreycb: ok where I got to was we should patch msgpack -> msgpack-python | 11:14 |
| jamespage | keystone and oslo.* done | 11:15 |
| jamespage | coreycb: keystone done; moving onto barbican for rc1 | 11:41 |
| jamespage | and congress and designate-* | 11:45 |
| jamespage | and then heat | 11:50 |
| jamespage | coreycb: I've moved branch builds over to stable/queens as well | 12:24 |
| jamespage | not sure all projects have cut those yet but its easier to mass move and then deal with excpetions | 12:24 |
| jamespage | coreycb: oging to horizon next | 12:30 |
| coreycb | jamespage: great, thanks | 12:57 |
| coreycb | jamespage: ok going to start on what's left. i'll work backward from z*. | 13:34 |
| jamespage | coreycb: okies | 14:51 |
| jamespage | coreycb: did you have a barbicanclient update in the works? | 15:11 |
| joop | hello | 15:11 |
| coreycb | jamespage: nope, that must be new | 15:12 |
| jamespage | coreycb: yeah heat needs it | 15:12 |
| joop | i would like to ask you some couple of question setting up a ubuntu server | 15:12 |
| jamespage | coreycb: lemme sort that out | 15:12 |
| joop | i need to run a php web app | 15:12 |
| joop | i installed apache2 | 15:12 |
| joop | but it doest not show only a blank page,afterall i installed xampp,which use apache | 15:13 |
| joop | so they have conflict with apache2 | 15:13 |
| joop | but main question is,i tried to intall webmin | 15:13 |
| joop | but the requested jcameron-key.asc file is not found anymore | 15:14 |
| coreycb | jamespage: oh i see, it's not in the upstream report for some reason. i'll figure out why. | 15:14 |
| joop | my question is how can i install webmin,and do i need actually or xampp do the same | 15:14 |
| coreycb | jamespage: done back to sahara. getting trove and vpnaas next. | 15:29 |
| rbasak | nacc: I don't see sssd in http://people.canonical.com/~ubuntu-archive/component-mismatches-proposed.svg ? | 16:46 |
| rbasak | I'd expect to see it there. | 16:46 |
| rbasak | Though it is in excuses | 16:46 |
| nacc | http://people.canonical.com/~ubuntu-archive/component-mismatches-proposed | 16:46 |
| nacc | rbasak: the svg doesn't show everything, aiui | 16:46 |
| coreycb | jamespage: working on the m* pkgs | 17:06 |
| administrador_ | hi | 17:07 |
| administrador_ | im trying to do wake on lan with nagios | 17:07 |
| jamespage | coreycb: horizon, designate and barbican done | 17:07 |
| dpb1 | rbasak: looking at that https everywhere bug, the one that carries the most weight is hiding the packages you are downloading. that is hard to refute. | 17:29 |
| rbasak | dpb1: HTTPS doesn't really hide that though. The package sizes, the dependency tree and recently published updates are all well known. That's enough to infer what you just downloaded. | 17:33 |
| rbasak | dpb1: if I had the time, I could write the tool to do it. | 17:33 |
| dpb1 | hm | 17:34 |
| dpb1 | rbasak: thanks, yes, interesting discussion | 17:37 |
| sdeziel | I suppose the above discussion is about HTTPS mirrors for apt, right? | 17:41 |
| dpb1 | sdeziel: yes | 17:41 |
| sdeziel | dpb1: there is one paper including a security analysis of various package manager: https://isis.poly.edu/~jcappos/papers/cappos_mirror_ccs_08.pdf | 17:42 |
| sdeziel | dpb1: rbasak: one of the benefit of using HTTPS was to defeat replay attacks and I believe this could happen when the connection to security.ubuntu.com is MITM | 17:43 |
| sdeziel | dpb1: rbasak: presumably having HTTPS on just security.ubuntu.com would prevent this without the nightmare associated with HTTPS mirrors | 17:43 |
| sdeziel | would be nice for the PPAs to also get HTTPS because they are only signed with a 1024 bit GPG key IIRC | 17:44 |
| dpb1 | sdeziel: ya, all the attacks in 3.1 seem relavent actually | 17:45 |
| sdeziel | dpb1: the "arbitrary package" and "extraneous dependencies" would fail GPG validation IMHO | 17:48 |
| sdeziel | but I haven't read the paper carefully | 17:48 |
| sdeziel | dpb1: could you share the bug number please? I couldn't find it quickly and would like to track it | 17:53 |
| sdeziel | https://www2.cs.arizona.edu/stork/packagemanagersecurity/otherattacks.html says that apt is subject to the endless data attack | 17:56 |
| dpb1 | sdeziel: ya, I had it, one sec | 17:57 |
| dpb1 | sdeziel: here's the main one: https://bugs.launchpad.net/ubuntu/+bug/1464064 | 17:57 |
| ubottu | Launchpad bug 1464064 in Ubuntu "Ubuntu apt repos are not available via HTTPS" [Undecided,Confirmed] | 17:57 |
| sdeziel | dpb1: thanks | 17:58 |
| dpb1 | sdeziel: thx for sharing your paper. I think freezing, endless data and replay are all very interesting and would certainly be mitigated by https+cert pinning | 17:58 |
| sdeziel | dpb1: endless data shouldn't even be exploitable because everything is there to close it. I don't know if recent apt versions fixed that or not but it doesn't require HTTPS | 17:59 |
| dpb1 | sdeziel: sure, the client should be proactive | 18:00 |
| dpb1 | sdeziel: at least on trusty, I had times where I was in the gigs of transfer from a broken repo | 18:00 |
| dpb1 | but maybe that is addressed now | 18:01 |
| sdeziel | yeah, that's what I'd like to know :) | 18:01 |
| sbeattie | sdeziel: new ppa gpg keys are 4096 bits. but they are per-owner, so those of us with an existing 1024 key can't ever seem to get out from under it, without creating a whole new launchpad identity. | 18:34 |
| sdeziel | sbeattie: hmm, that's interesting to say the least | 18:35 |
| jamespage | coreycb: https://launchpadlibrarian.net/357025783/buildlog_ubuntu-bionic-amd64.heat_1%3A10.0.0~rc1-0ubuntu1~ubuntu18.04.1~ppa201802141302_BUILDING.txt.gz | 18:35 |
| jamespage | I think we need new sdk | 18:36 |
| sbeattie | sdeziel: https://bugs.launchpad.net/launchpad/+bug/1240681 is where the default got bumped to 4096. | 18:36 |
| ubottu | Launchpad bug 1240681 in Launchpad itself "New GPG generated keys are 1024 bits" [High,Fix released] | 18:36 |
| coreycb | jamespage: i can get that | 18:37 |
| jamespage | coreycb: ta | 18:37 |
| jamespage | coreycb: the min barbicanclient version in global-requirements is to low as well | 18:37 |
| sdeziel | sbeattie: I sure would have hope for a way to transition to a newer key | 18:38 |
| sbeattie | +1 | 18:39 |
| sdeziel | https://bugs.launchpad.net/launchpad/+bug/1331914 | 18:40 |
| ubottu | Launchpad bug 1331914 in Launchpad itself "Allow users to re-generate a PPA signing key" [High,Triaged] | 18:40 |
| === transcend is now known as strive | ||
| rbasak | sdeziel: yeah, already noted. Valid-Until and Check-Valid-Until can mitigate that, but in the HTTPS case it'd be noticed quicker. | 19:43 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
