NAT and DNS/NTP servers

I'm observing lost packets when a gateway doing NAT also locally hosts a server for DNS or NTP. I believe this the result of the ambiguous conditions that can exist when routing server-to-server packets coming into the gateway.

I can avoid it by specifying --to-source ports in a SNAT rule for NTP, or with DNAT and --to-destination port to force usage of the gateway's server (which is likely the best solution). I've not tried this on DNS yet, but I expect it will solve the problem as well.

Is this a known situation for these protocols (and similar)? Or am I barking up the wrong tree?

Thanks.

Andrew E. Mileski

Show us your rules and explain what is ambiguous about them?

If you are hosting NTP on the machine running netfilter, why would you want a
SNAT rule? Show us an example?

Why? What address are the clients sending the packets to (such that they
need DNATting in order to be handled by the server you're talking about)?

This does not sound like a familiar problem (either which I have seen myself,
or heard other people discussing). I think if we saw a picture of your tree
we could tell you if it was the wrong one to bark up (ie: show us your ruleset and explain what goes wrong with it).

Regards,

Antony.

Success is a lousy teacher.  It seduces smart people into thinking they can't 
lose.

 - William H Gates III

                                                     Please reply to the list;
                                                           please don't CC me.

There is only one rule needed if all other defaults are ACCEPT:

This one is on my network:
iptables -t nat -A POSTROUTING -s ! $WAN_IP -o $WAN_IFC -j SNAT --to-source $WAN_IP

This one is on another network with the same problem:
iptables -t nat -A POSTROUTING -s $INTERNAL_NET -j MASQUERADE

Both DNS and NTP server-to-server UDP packets have the same source and destination ports. What I am seeing is a failure to remap the source port on outgoing packets when a local server is running on the gateway.

I speculate this is because an idle local server without an active connection doesn't have an entry in the connection table. Though I've also seen failures with an active local server, but I haven't got a good theory on the mechanics behind it yet.

I have seen port remapping in some situations, so I know that feature works. It just isn't being triggered in some situations, which I can't yet reason why.

As an aside, I think (S?)NTP pays attention to the source port, so a port remap may have side effects for this protocol. This isn't a concern of mine though.

Andrew E. Mileski

I just realized this theory is bogus too, as it should be easy to tell there is a listener on the local port. I assume this is probably checked.

Andrew E. Mileski

What is $WAN_IP, and what are the IP addresses on the machine on which this rule is running?

I still don't really understand why you need a SNAT rule on a machine which is running the service locally?

Please give a clear explanation of where the client is, where the server is,
and why you want a SNAT rule in the system. Also, please explain what is
"ambiguous" about all this, as you described it - I don't understand what you mean by that in this context.

Regards,

Antony.

Never write it in Perl if you can do it in Awk.
Never do it in Awk if sed can handle it.
Never use sed when tr can do the job.
Never invoke tr when cat is sufficient.
Avoid using cat whenever possible.

                                                     Please reply to the list;
                                                           please don't CC me.

LAN is a private network:
LAN_IFC=eth0
LAN_NET=192.168.0/24
LAN_IP=192.168.1.1

WAN is the internet:
WAN_IFC=eth1
WAN_NET=209.217.118.0/29
WAN_IP=209.217.118.226

The machine is a gateway to the internet as well. The gateway has to use SNAT to allow private hosts to reach the internet for other services, like say HTTP. Private hosts are not restricted to the services provided by the gateway.

A private host (192.168.1.2) on a private network (192.168.1.0/24) that is contacting a remote server on the internet.

I used the term "ambiguous" to attempt to describe the situation I'm seeing where there are two possible destinations for a reply packet coming from the internet. The packet can be routed to the server process on the gateway machine, or to a private host.

As I've stated I'm unclear on exactly what is happening, so don't attribute too much weight to it or my phraseology. I'm attempting to put words to a process I can not yet fully understand or explain. My appologies for this.

For example:

The NTP server process on the gateway sends a UDP packet to a NTP server on internet, let's say time.nist.gov. The packet sent out WAN_IFC has:
source address: WAN_IP = 209.217.118.226
source port: 123
destination address: time.nist.gov = 192.43.244.18 destination port: 123
A reply packet from time.nist.gov arrives on WAN_IFC with:
source address: time.nist.gov = 192.43.244.18
source port: 123
destination address: WAN_IP = 209.217.118.226
destination port: 123

When a private client (192.168.1.2) sends a packet to the same place, the packet arriving on the LAN_IFC has:
source address: private client = 192.168.1.2
source port: 123
destination address: time.nist.gov = 192.43.244.18 destination port: 123
This gets SNAT'ed by the gateway and sent out the WAN_IFC to the internet looking like this:
source address: WAN_IP = 209.217.118.226
source port: 123
destination address: time.nist.gov = 192.43.244.18 destination port: 123
A reply packet from time.nist.gov arrives on WAN_IFC with:
source address: time.nist.gov = 192.43.244.18
source port: 123
destination address: WAN_IP = 66.11.173.24
destination port: 123
Because the port wasn't remapped, there appears to be ambiguity in whether the reply packet should be routed to the private host (192.168.1.2) or the NTP server process on the router.

Andrew E. Mileski

Doh! made a mistake the WAN_IP in the reply above should be 209.217.118.226 not 66.11.173.24. Sorry. [That's why I tried to avoid typing numbers originally; to avoid mistakes.]

Andrew E. Mileski

Okay, I think you mean that the NTP *client* process on the gateway sends a
packet to an NTP server on the Internet. I was getting confused because you
kept talking about a server, and I couldn't see where SNAT would come into that, however for a client process I can understand it.

That all seems perfectly clear and reasonable.

Yup - I agree with that as well. The netfilter code will not change the
source port of a packet unless that source port is already bound to an existing connection.

If there is a connection tracking table entry (because the packet came from a
LAN client) then the reply packet will be routed back to the client. If
there isn't, then it won't be. I see no opportunity for ambiguity because
netfilter is stateful.

I do not think that netfilter will ever get "confused" about whether a reply packet is in response to a locally-generated request, or a NATted connection
from a LAN client machine. The gateway which you are running netfilter on
will always keep an accurate record of (a) locally-bound ports, and (b) connection tracking table entries for packets originating elsewhere.

What makes you think that netfilter is getting anything wrong? Are you
seeing packets in LOG entries which do not seem to correspond to current
connections? If you are simply seeing connections not succeeding from time
to time, do keep in mind that this is UDP we're talking about, which although is strictly known as User Datagram Protocol, can be fairly accurately interpreted as Unreliable Datagram Protocol (unlike TCP, which is a reliable service).

Regards,

Antony.

You can spend the whole of your life trying to be popular,
but at the end of the day the size of the crowd at your funeral
will be largely dictated by the weather.

 - Frank Skinner

                                                     Please reply to the list;
                                                           please don't CC me.

This is from a different machine (sorry), but my point is the same...

1) Make sure there are no existing entries

   # grep '123' /proc/net/ip_conntrack
   #

2) Start up a NTP server on the gateway with a poll period long enough to allow an entry in /proc/net/ip_conntrack to expire (I used minpoll 9 maxpoll 9)

   # grep '123' /proc/net/ip_conntrack
udp      17 166 src=66.11.173.24 dst=192.43.244.18 sport=123 dport=123 src=192.43.244.18 dst=66.11.173.24 sport=123 dport=123 [ASSURED] use=1

3) Start up a NTP server on a private client with a short poll period (I used minpoll 4 maxpoll 4)

   # grep '123' /proc/net/ip_conntrack
udp      17 177 src=66.11.173.24 dst=192.43.244.18 sport=123 dport=123 src=192.43.244.18 dst=66.11.173.24 sport=123 dport=123 [ASSURED] use=1
udp      17 162 src=192.168.2.244 dst=192.43.244.18 sport=123 dport=123 src=192.43.244.18 dst=66.11.173.24 sport=123 dport=12 [ASSURED] use=1

The port was remapped to 12. All is okay.

4) Stop the private host NTP server, keeping the gateway NTP server listening (long poll time remember), and make sure there are no existing entries (wait for any to expire)

   # grep '123' /proc/net/ip_conntrack
   #

5) Start a NTP server on a private client again with the short poll period

   # grep '123' /proc/net/ip_conntrack
udp      17 167 src=192.168.2.244 dst=192.43.244.18 sport=123 dport=123 src=192.43.244.18 dst=66.11.173.24 sport=123 dport=123 [ASSURED] use=1

The port wasn't remapped... incoming packets will ALWAYS be routed to this machine. The NTP server process on the gateway will never receive packets while this connecection exists.

It seems netfilter doesn't check for the existence of a local listener on a port before deciding whether or not to remap it. Only the connection table seems to be referenced.

Bug or feature, I don't know. I just wanted confirmation that this exists. I stated earlier that there are two ways around it.

Andrew E. Mileski

This is correct, because remapping is only relevant to packets leaving the machine (this machine acting as client), and a local listener is only relevant to packets arriving at the machine (this machine acting as server).

I remain confused from your description as to whether you are talking about the machine running the netfilter rules being an NTP server (servicing requests from other clients), or an NTP client (sending requests to other servers).

It may be that the problem you are experiencing is simply due to the default timeout on UDP "connections" (which is an artificial concept anyway, built into netfilter simply to try and make UDP conform to the stateful packet monitoring mechanism), and you need to adjust the timings by fiddling about with /proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout ?

Regards,

Antony.

If you want to be happy for an hour, get drunk.
If you want to be happy for a year, get married.
If you want to be happy for a lifetime, get a garden.

                                                     Please reply to the list;
                                                           please don't CC me.

The machine running the netfilter rules is an NTP and DNS server (servicing requests from other clients), _and_ a NTP and DNS client (sending requests to other servers). This is the nature of both of these protocols (peering).

The machine running the netfilter rules is also a gateway, passing requests from a private network to servers on the internet.

That will work on one condition: a guarantee can be made that the server on the gateway will get a packet out before forwarding one for a private host. The gateway server must own the first entry in the conntrack table for the port.

Otherwise the time workaround can easily be done with NTP, by setting ip_conntrack_udp_timeout to an amount longer than the poll period of the NTP server on the gateway.

However, with DNS one never knows when the server will go out to the internet. It can be quite a long time if all serviced requests reference cached entries.

Hence my solutions (both tested) of an explicit rule to either:
A) Force all private hosts to use the DNS/NTP servers on the gateway.
B) Force a port remap on all DNS/NTP packets from private hosts passing through the gateway.

Andrew E. Mileski

This sounds like a very good idea, for both protocols, since:

a) if you have a caching DNS server on your network, you may as well make maximum use of it, instead of sending requests out to the Internet direct from individual clients

b) if you have an NTP server on your network, it is more important that all your local machines are synchronised to it than that some of them are correct, but different from others.

Therefore I think pointing all local clients at the DNS / NTP servers on the gateway, and then pointing the single DNS / NTP clients running on the gateway to the outside world, is the correct solution.

Regards,

Antony.

I want to build a machine that will be proud of me.

 - Danny Hillis, creator of The Connection Machine

                                                     Please reply to the list;
                                                           please don't CC me.