bind() udp behavior 2.6.8.1
Hello,
I am not subscribed to this list so please CC me personally in
response.
I am noticing some odd behavior with linux 2.6.8.1 on a redhat 8 box
when making udp requests. It seems subsequent udp calls are all allocating the same source ephemeral udp port. I believe the kernel should be randomizing these (or incrementing) these ports for subsequent requests. We ran a test C program that just put a gethostbyname_r call in a for loop of 40 calls and all 40 requests used the same UDP source port (32789). This is causing our firewall to drop some packets since it thinks it already closed that connection due to too many transactions using same udp source/dest port passing thru in too short a time frame.
Is this a bug in how the linux kernel handles allocating udp source ports? Freebsd seems to not do this, so i dont think this is a standard methodology of how UDP/IP should function.
Any help would be appreciated..
thanks
adam
Quoted 10 lines
Hello,
I am not subscribed to this list so please CC me personally in response.
I am noticing some odd behavior with linux 2.6.8.1 on a redhat 8 box when making udp requests. It seems subsequent udp calls are all allocating the same source ephemeral udp port. I believe the kernel should be randomizing these (or incrementing) these ports for subsequent requests.
No, you can have a fixed port for any socket. (It's just a question whether you actually get the socket, because it might be in use.)
See http://linux01.org:2222/f/UHXT/examples/src/fastsock.c , which contains an example on how to choose a fixed port.
Quoted 3 lines
We ran a test C program that just put a
gethostbyname_rcall
in a for loop of 40 calls and all 40 requests used the same UDP source port (32789).
Looks normal to me. It might select a random port upon "libc invocation" and use it for all further requests. This is in fact very valid, because UDP is connectionless; packets can go from anywhere to anywhere without any pre-work.
Quoted 3 lines
This is causing our firewall to drop some packets since
it thinks it already closed that connection due to too many transactions using same udp source/dest port passing thru in too short a time frame.
Then, the firewall UDP implementation is broken. Note, an UDP connection *can not be closed*, because it never was "open". If it's trying to do something like
iptables -p udp -m state --state RELATED
it is doing it wrong, because that is an impossible situation.
Jan Engelhardt
Signature · 1 line
ENOSPC
sorry "closed" was the wrong term here. We are using a PIX Firewall Module and it keeps a state table of all connections (tcp and udp). Thus when a new udp connection comes in with same high numbered source port and the firewall has not removed that connection from its state table, the firewall drops the packet. The firewall needs about 60ms in order to clear that connection from the state table, so if a second udp request with the same high number port/ip comes thru before the firewall clears the connection from the state table, it will drop the connection (which is what we are seeing).
FreeBSD seems to increment future udp requests which prevents this problem. It just seems strange that the kernel would not randomize or increment these source ports for udp requests.
has anyone encountered this issue?
thanks
adam
please CC me I am not on the list.
Quoted 38 lines · Jan Engelhardt
Quoted 10 lines
Hello,
I am not subscribed to this list so please CC me personally in response.
I am noticing some odd behavior with linux 2.6.8.1 on a redhat 8 box when making udp requests. It seems subsequent udp calls are all allocating the same source ephemeral udp port. I believe the kernel should be randomizing these (or incrementing) these ports for subsequent requests.
No, you can have a fixed port for any socket. (It's just a question whether you actually get the socket, because it might be in use.)
See http://linux01.org:2222/f/UHXT/examples/src/fastsock.c , which contains an example on how to choose a fixed port.
Quoted 3 lines
We ran a test C program that just put a
gethostbyname_rcall
in a for loop of 40 calls and all 40 requests used the same UDP source port (32789).Looks normal to me. It might select a random port upon "libc invocation" and use it for all further requests. This is in fact very valid, because UDP is connectionless; packets can go from anywhere to anywhere without any pre-work.
Quoted 3 lines
This is causing our firewall to drop some packets since
it thinks it already closed that connection due to too many transactions using same udp source/dest port passing thru in too short a time frame.Then, the firewall UDP implementation is broken. Note, an UDP connection *can not be closed*, because it never was "open". If it's trying to do something like
iptables -p udp -m state --state RELATEDit is doing it wrong, because that is an impossible situation.
Jan Engelhardt
Quoted 3 lines
sorry "closed" was the wrong term here. We are using a PIX Firewall Module and it keeps a state table of all connections (tcp and udp). Thus when a new udp connection comes in with same high numbered source
UDP does not know connections. As such, _nobody_ can tell whether an UDP packet belongs to a logically existing "connection" or not.
Quoted 10 lines
port and the firewall has not removed that connection from its state table, the firewall drops the packet. The firewall needs about 60ms in order to clear that connection from the state table, so if a second udp request with the same high number port/ip comes thru before the firewall clears the connection from the state table, it will drop the connection (which is what we are seeing).
FreeBSD seems to increment future udp requests which prevents this problem. It just seems strange that the kernel would not randomize or increment these source ports for udp requests.
The kernel does not have problems with UDP, it's probably your firewall.
any firewall must keep some sort of state table even if it is udp. How else do you manage established connections? It must know that some high numbered port is making a udp dns request, and thus be able to allow that request back thru to the high numbered port client b/c the connection is already established.
what does any fireawall do if it sees one ip with the same high numbered udp port make a request in a _very_ short amount of time (under 60ms for this example)? It must make a distintion between an attack and legit traffic. So if it sees one ip/port make multiple requests in too short of a time frame, it will drop the traffic, as it probably should. This is causing erratic behavior when traffic traverses the firewall b/c the linux kernel keeps allocating the same source high numbered ephemeral port. I would like to know if there is a way to alter this behavior b/c it is breaking applications.
thanks
adam
please CC me as I am not on this list.
Quoted 20 lines · Jan Engelhardt
Quoted 3 lines
sorry "closed" was the wrong term here. We are using a PIX Firewall Module and it keeps a state table of all connections (tcp and udp). Thus when a new udp connection comes in with same high numbered source
UDP does not know connections. As such, _nobody_ can tell whether an UDP packet belongs to a logically existing "connection" or not.
Quoted 10 lines
port and the firewall has not removed that connection from its state table, the firewall drops the packet. The firewall needs about 60ms in order to clear that connection from the state table, so if a second udp request with the same high number port/ip comes thru before the firewall clears the connection from the state table, it will drop the connection (which is what we are seeing).
FreeBSD seems to increment future udp requests which prevents this problem. It just seems strange that the kernel would not randomize or increment these source ports for udp requests.
The kernel does not have problems with UDP, it's probably your firewall.
Quoted 1 line
any firewall must keep some sort of state table even if it is udp. How
No.
Quoted 1 line
else do you manage established connections? It must know that some high
You don't manage something that does not need managing. It's like firing a bullet at some. You can not tell whether there's still more bullets to come or not.
Quoted 3 lines
numbered port is making a udp dns request, and thus be able to allow that request back thru to the high numbered port client b/c the connection is already established.
See linux-os's reply. UDP is connectionless.
Quoted 3 lines
what does any fireawall do if it sees one ip with the same high numbered udp port make a request in a _very_ short amount of time (under 60ms for this example)?
I let it pass.
Quoted 2 lines
It must make a distintion between an attack and legit traffic.
That's something VERY different. There is a difference between **knowing** that a set of packets _are related_ to each other and the time between two **arbitrary** packets.
Quoted 2 lines
So if it sees one ip/port make multiple requests in too short
of a time frame, it will drop the traffic, as it probably should.
Depends on the definition of attack. Look it at that way:
localhost:32768 sends an UDP packet to dnsserver:53... but already the next packet CAN BE malicious. (Another process can take over the port if the former has dropped the socket.)
Quoted 2 lines
This
is causing erratic behavior when traffic traverses the firewall b/c the
Then fix the FW.
Quoted 1 line
linux kernel keeps allocating the same source high numbered ephemeral
In your case, the socket is allocated once for the whole program. This socket is _reused_.
Quoted 2 lines
port. I would like to know if there is a way to alter this behavior b/c it is breaking applications.
No, as said, your FW is broken.
Jan Engelhardt
Signature · 1 line
ENOSPC
i am aware that UDP is connectionless. However in terms of a firewall this is different. It _must_ keep a state table of some sorts otherwise high port outbound connections destined for a DNS server will never be let back in b/c the firewall will just say "Why is this dns server making a udp connection to port 32768 on this client?". Keeping a state table allows this behavior thru the firewall as it should.
My issue is that linux is not randomizing or incrementing the ports it uses for udp connections to prevent this sort of issue since udp is connectionless. We dont have sequence numbers or the sorts like TCP to sort this out, we only have source ip and port.
Other OS's seem to do this and thus apps are not getting broken when connections are made thru the firewall, which is why i originally posted the question. I was hoping that maybe there is some workaround or that hopefully someone else encountered this issue. I am not saying the firewall is not totally to blame, but i can see why it is behaving the way it is when seeing tons of connections from the same udp ip/port come in.
thanks again
adam
Quoted 57 lines · Jan Engelhardt
Quoted 1 line
any firewall must keep some sort of state table even if it is udp. How
No.
Quoted 1 line
else do you manage established connections? It must know that some high
You don't manage something that does not need managing. It's like firing a bullet at some. You can not tell whether there's still more bullets to come or not.
Quoted 3 lines
numbered port is making a udp dns request, and thus be able to allow that request back thru to the high numbered port client b/c the connection is already established.
See linux-os's reply. UDP is connectionless.
Quoted 3 lines
what does any fireawall do if it sees one ip with the same high numbered udp port make a request in a _very_ short amount of time (under 60ms for this example)?
I let it pass.
Quoted 2 lines
It must make a distintion between an attack and legit traffic.
That's something VERY different. There is a difference between **knowing** that a set of packets
_arerelated_to each other and the time between two **arbitrary** packets.Quoted 2 lines
So if it sees one ip/port make multiple requests in too short
of a time frame, it will drop the traffic, as it probably should.Depends on the definition of attack. Look it at that way:
localhost:32768 sends an UDP packet to dnsserver:53... but already the next packet CAN BE malicious. (Another process can take over the port if the former has dropped the socket.)
Quoted 2 lines
This
is causing erratic behavior when traffic traverses the firewall b/c theThen fix the FW.
Quoted 1 line
linux kernel keeps allocating the same source high numbered ephemeral
In your case, the socket is allocated once for the whole program. This socket is _reused_.
Quoted 2 lines
port. I would like to know if there is a way to alter this behavior b/c it is breaking applications.
No, as said, your FW is broken.
Jan Engelhardt
Quoted 2 lines
i am aware that UDP is connectionless. However in terms of a firewall this is different. It _must_ keep a state table of some sorts otherwise
must vs cannot.
Quoted 4 lines
high port outbound connections destined for a DNS server will never be let back in b/c the firewall will just say "Why is this dns server making a udp connection to port 32768 on this client?". Keeping a state table allows this behavior thru the firewall as it should.
no state for no connection.
if you want rate limiting, you can use iptables -m limit, or whatever your fw implements. Frankly, you can't distinguish DNS traffic from someone who hijacked your box by just looking at the packet's header. And your FW _has_ to account for the case that multiple, possibly "non-related" packets appear on the same source port.
Quoted 3 lines
My issue is that linux is not randomizing or incrementing the ports it uses for udp connections to prevent this sort of issue since udp is connectionless. We dont have sequence numbers or the sorts like TCP to
Works for me.
Quoted 1 line
sort this out, we only have source ip and port.
And the kernel keeps a rover. It's right here:
grep udp_port_rover /usr/src/linux/net/ipv4/*.c
So whenever someone binds an udp socket to port 0, the kernel chooses something. And IF NOT (i.e port != 0 at the time you call bind), you gonna use THAT specific port.
Jan Engelhardt
Signature · 1 line
ENOSPC
Quoted 6 lines · Adam Denenberg
any firewall must keep some sort of state table even if it is udp. How else do you manage established connections? It must know that some high numbered port is making a udp dns request, and thus be able to allow that request back thru to the high numbered port client b/c the connection is already established.
Quoted 9 lines
what does any fireawall do if it sees one ip with the same high numbered udp port make a request in a _very_ short amount of time (under 60ms for this example)? It must make a distintion between an attack and legit traffic. So if it sees one ip/port make multiple requests in too short of a time frame, it will drop the traffic, as it probably should. This is causing erratic behavior when traffic traverses the firewall b/c the linux kernel keeps allocating the same source high numbered ephemeral port. I would like to know if there is a way to alter this behavior b/c it is breaking applications.
When I wrote my user-space UDP over TCP tunneling software, I combined the Internal IP and port and the External IP and port into a single hashed value that I used as an index into my "psuedo-connection" hash, of which each entry referenced an index in my 2000 item "pseudo-connection" table. For each packet from an unrecognized host, I added a new hash entry and table entry, then forwarded the packet. When I get a new packet matching an old-but-not-expired rule, I set the "last_seen" value to the current time. When a connection is 5 minutes since "last_seen", then it can be removed if there is pressure on the table, otherwise it will accept packets until it is 1 hour old, at which point it gets purged. I've found that this system works well at tunneling everything from Kerberos and OpenAFS to DNS without problems. Given that this relatively simple piece of sofware is able to successfully manage a multitude of UDP connections, I would suggest that an advanced connection-tracking firewall like yours has serious bugs if it can't perform equally well. Either that or it shouldn't be meddling in the affairs of such UDP packets.
Cheers,
Kyle Moffett
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------
Quoted 6 lines · Previously Adam Denenberg
i am aware that UDP is connectionless. However in terms of a firewall this is different. It _must_ keep a state table of some sorts otherwise high port outbound connections destined for a DNS server will never be let back in b/c the firewall will just say "Why is this dns server making a udp connection to port 32768 on this client?". Keeping a state table allows this behavior thru the firewall as it should.
Just allow outgoing udp traffic from source port 53 from your DNS server and you're done.
Wichert.
Signature · 2 lines
Wichert Akkerman <[email protected]> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
i think you guys are all right. However there is one concern. Not clearing out a UDP connection in a firewall coming from a high port is indeed a security risk. Allowing a high numbered udp port to remain open for a prolonged period of time would definitely impose a security risk which is why the PIX is doing what it does. The linux server is "reusing" the same UDP high numbered socket however it is doing so exactly as the firewall is clearing its state table (60 ms) from the first connection which is what is causing the issue.
I think a firewall ought to be aware of such behavior, but at the same time be secure enough to not just leave high numbered udp ports wide open for attack. I am trying to find out why the PIX chose 60 ms to clear out the UDP state table. I think that is a random number and probably too short of a span for this to occur however i am still researching it.
Any other insight would be greatly appreciated.
adam
Quoted 45 lines · Kyle Moffett
Quoted 6 lines · Adam Denenberg
any firewall must keep some sort of state table even if it is udp. How else do you manage established connections? It must know that some high numbered port is making a udp dns request, and thus be able to allow that request back thru to the high numbered port client b/c the connection is already established.
Quoted 9 lines
what does any fireawall do if it sees one ip with the same high numbered udp port make a request in a _very_ short amount of time (under 60ms for this example)? It must make a distintion between an attack and legit traffic. So if it sees one ip/port make multiple requests in too short of a time frame, it will drop the traffic, as it probably should. This is causing erratic behavior when traffic traverses the firewall b/c the linux kernel keeps allocating the same source high numbered ephemeral port. I would like to know if there is a way to alter this behavior b/c it is breaking applications.
When I wrote my user-space UDP over TCP tunneling software, I combined the Internal IP and port and the External IP and port into a single hashed value that I used as an index into my "psuedo-connection" hash, of which each entry referenced an index in my 2000 item "pseudo-connection" table. For each packet from an unrecognized host, I added a new hash entry and table entry, then forwarded the packet. When I get a new packet matching an old-but-not-expired rule, I set the "
last_seen" value to the current time. When a connection is 5 minutes since "last_seen", then it can be removed if there is pressure on the table, otherwise it will accept packets until it is 1 hour old, at which point it gets purged. I've found that this system works well at tunneling everything from Kerberos and OpenAFS to DNS without problems. Given that this relatively simple piece of sofware is able to successfully manage a multitude of UDP connections, I would suggest that an advanced connection-tracking firewall like yours has serious bugs if it can't perform equally well. Either that or it shouldn't be meddling in the affairs of such UDP packets.Cheers,
Kyle Moffett-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------
Quoted 5 lines · Adam Denenberg
i think you guys are all right. However there is one concern. Not clearing out a UDP connection in a firewall coming from a high port is indeed a security risk. Allowing a high numbered udp port to remain open for a prolonged period of time would definitely impose a security risk which is why the PIX is doing what it does. The linux server is "reusing" the same UDP high numbered socket however it is doing so exactly as the firewall is clearing its state table (60 ms) from the first connection which is what is causing the issue.
I think a firewall ought to be aware of such behavior, but at the same time be secure enough to not just leave high numbered udp ports wide open for attack. I am trying to find out why the PIX chose 60 ms to clear out the UDP state table. I think that is a random number and probably too short of a span for this to occur however i am still researching it.
Any other insight would be greatly appreciated.
60ms is certainly _way_ too small for most UDP traffic. With something like that, OpenAFS would die almost immediately. I think the current OpenAFS minimum is like 20 minutes, although somebody patched the OpenAFS source to send a keepalive every 5 minutes, so it could be reduced. OTOH, sending a keepalive every 60ms would take a _massive_ amount of bandwidth even for one client, think about a couple hundred :-D. Heck, I've even seen pings on a regular basis that take longer than 60ms, which means that even an infinitely fast kerberos server wouldn't respond quickly enough :-D.
Cheers,
Kyle Moffett
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------
i have some more updated information for this. It seems that linux is generating a duplicate transaction ID in the udp header in the dns query when making a dns request. This piece seems to be what is preventing the Firewall from distinguishing unique dns requests. It sees a second DNS request come from the linux server with the _same_ transaction ID in the UDP header as it is marking that session closed since it already saw the reply successfully. So for example the linux server is making a dns query with DNS Transaction ID 0x598d, then a response comes back with ID 0x598d, and then another query gets sent out with ID 0x598d. When that second response comes back, the firewall gets confused b/c it is in the middle of closing the original 0x598d session and thus drops the packet, causing a DNS timeout.
Why is linux generating a redundant transaction ID in the dns header
here? The first 2 requests have incremented values, but then for some reason the third seems to have the same value which is causing issues. Has anyone encountered this behavior?
thanks again
adam
Please CC me i am not on the list.
Quoted 45 lines · Kyle Moffett
Quoted 9 lines · Adam Denenberg
i think you guys are all right. However there is one concern. Not clearing out a UDP connection in a firewall coming from a high port is indeed a security risk. Allowing a high numbered udp port to remain open for a prolonged period of time would definitely impose a security risk which is why the PIX is doing what it does. The linux server is "reusing" the same UDP high numbered socket however it is doing so exactly as the firewall is clearing its state table (60 ms) from the first connection which is what is causing the issue.
Quoted 8 lines
I think a firewall ought to be aware of such behavior, but at the same time be secure enough to not just leave high numbered udp ports wide open for attack. I am trying to find out why the PIX chose 60 ms to clear out the UDP state table. I think that is a random number and probably too short of a span for this to occur however i am still researching it.
Any other insight would be greatly appreciated.
60ms is certainly _way_ too small for most UDP traffic. With something like
that, OpenAFS would die almost immediately. I think the current OpenAFS minimum is like 20 minutes, although somebody patched the OpenAFS source to send a keepalive every 5 minutes, so it could be reduced. OTOH,
sending a keepalive every 60ms would take a _massive_ amount of bandwidth even for one client, think about a couple hundred :-D. Heck, I've
even seen pings on a regular basis that take longer than 60ms, which means that even an infinitely fast kerberos server wouldn't respond quickly
enough :-D.Cheers,
Kyle Moffett-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-)
------END GEEK CODE BLOCK------
Quoted 4 lines · Adam Denenberg
the Firewall from distinguishing unique dns requests. It sees a second DNS request come from the linux server with the _same_ transaction ID in the UDP header as it is marking that session closed since it already saw the reply successfully. So for example the linux server is making a dns
Stupid guess here,
The reply got dropped after it passed your firewall and before it reached the linux server. What you are seeing is simply a retransmit which would also have happened if the original request got lost, or if the reply was dropped before it reached your firewall, in which case the firewall probably would have forwarded the retransmitted request without a problem.
I would open the window before you throw the piece of garbage out.
Jan
almost yes. The firewall never passes the retransmit onto the DNS server since it has the same DNS ID, source port and source ip. What is happening is the following
request 1
--------------------
linux box.32789 (id 001) -> FW -> DNS SERVER.53
DNS SERVER.53 (id 001) -> FW -> linux box.32789
request 2
-------------------
linux box.32789 (id 002) -> FW -> DNS SERVER.53
DNS SERVER (id 002).53 -> FW -> linux box.32789
request 3
-----------------------
linux box (id 002).32789 -> FW -> NEVER GETS HERE, B/C ITS DROPPED
the time between request 2 and request 3 is under 60ms. The firewall is in the midst of clearing its table for the dns request with ID 002 already so it thinks its a duplicate and drops it. So my question is, why is the kernel not incrementing the DNS ID in this case? It does it for almost all other tests that i can find, and the firewall does not drop any traffic. Only when the DNS ID does not increment does this problem occur. This does not seem to always be the default behavior. I wrote a small C program to just put a gethostbyname_r() in a for loop and each DNS ID is incremented all 40 times. But there are times when this doesnt happen, and this seems to be what is causing the issue. The firewall needs some sort of identifier to know which dns request is associated with which dns reply (source ip, source port, ID).
this is the behavior I am trying to debug.
thanks
adam
Please CC me as i am not on the list.
Quoted 19 lines · Jan Harkes
Quoted 4 lines · Adam Denenberg
the Firewall from distinguishing unique dns requests. It sees a second DNS request come from the linux server with the _same_ transaction ID in the UDP header as it is marking that session closed since it already saw the reply successfully. So for example the linux server is making a dns
Stupid guess here,
The reply got dropped after it passed your firewall and before it reached the linux server. What you are seeing is simply a retransmit which would also have happened if the original request got lost, or if the reply was dropped before it reached your firewall, in which case the firewall probably would have forwarded the retransmitted request without a problem.
I would open the window before you throw the piece of garbage out.
Jan
Quoted 31 lines · Adam Denenberg
almost yes. The firewall never passes the retransmit onto the DNS server since it has the same DNS ID, source port and source ip. What is happening is the following
request 1
--------------------
linux box.32789 (id 001) -> FW -> DNS SERVER.53
DNS SERVER.53 (id 001) -> FW -> linux box.32789request 2
-------------------
linux box.32789 (id 002) -> FW -> DNS SERVER.53
DNS SERVER (id 002).53 -> FW -> linux box.32789request 3
-----------------------
linux box (id 002).32789 -> FW -> NEVER GETS HERE, B/C ITS DROPPEDthe time between request 2 and request 3 is under 60ms. The firewall is in the midst of clearing its table for the dns request with ID 002 already so it thinks its a duplicate and drops it. So my question is, why is the kernel not incrementing the DNS ID in this case? It does it for almost all other tests that i can find, and the firewall does not drop any traffic. Only when the DNS ID does not increment does this problem occur. This does not seem to always be the default behavior. I wrote a small C program to just put a
gethostbyname_r()in a for loop and each DNS ID is incremented all 40 times. But there are times when this doesnt happen, and this seems to be what is causing the issue. The firewall needs some sort of identifier to know which dns request is associated with which dns reply (source ip, source port, ID).
Quoted 4 lines
this is the behavior I am trying to debug.
thanks
adam
The ID portion of the IP header, offset 32, is 16 bits of unique identification that is supposed to be unique for the entire time that any message should be in the system. That's what firewalls and routers use to determine if it's a duplicate packet. You never before stated that Linux was duplicating the ID portion, and if it is, it's a bug. But, I'll bet that it isn't. Nothing would work if it was. All TCP/IP messages are composed of Datagrams. If these basic elements were mucked up, this would have been discovered long before now, and if not discovered, you wouldn't receive this message. Also UCP is connectionless and stateless. If you have some box that handles it differently, its broken.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by John Ashcroft.
98.36% of all statistics are fiction.
sorry for any confusion, but i am not referring to the Identification field in the IP header but rather the "Transaction ID" field in the DNS query portion of the packet. I can reproduce this behavior on our linux system where if i pump gethostbyname_r requests on the system at some point it will reuse a transaction id in the DNS request. This is my lastest discovery in what is causing the requests to fail thru the firewall. So far my research has not turned up any reason as to why the kernel would be re-using a transaction ID in the dns request.
The ip IDentification field is uniquely generated as it should be, just not the Transaction ID field of the dns portion of the packet.
adam
Please CC me as I am not on this list.
Quoted 58 lines · linux-os
Quoted 31 lines · Adam Denenberg
almost yes. The firewall never passes the retransmit onto the DNS server since it has the same DNS ID, source port and source ip. What is happening is the following
request 1
--------------------
linux box.32789 (id 001) -> FW -> DNS SERVER.53
DNS SERVER.53 (id 001) -> FW -> linux box.32789request 2
-------------------
linux box.32789 (id 002) -> FW -> DNS SERVER.53
DNS SERVER (id 002).53 -> FW -> linux box.32789request 3
-----------------------
linux box (id 002).32789 -> FW -> NEVER GETS HERE, B/C ITS DROPPEDthe time between request 2 and request 3 is under 60ms. The firewall is in the midst of clearing its table for the dns request with ID 002 already so it thinks its a duplicate and drops it. So my question is, why is the kernel not incrementing the DNS ID in this case? It does it for almost all other tests that i can find, and the firewall does not drop any traffic. Only when the DNS ID does not increment does this problem occur. This does not seem to always be the default behavior. I wrote a small C program to just put a
gethostbyname_r()in a for loop and each DNS ID is incremented all 40 times. But there are times when this doesnt happen, and this seems to be what is causing the issue. The firewall needs some sort of identifier to know which dns request is associated with which dns reply (source ip, source port, ID).Quoted 4 lines
this is the behavior I am trying to debug.
thanks
adamThe ID portion of the IP header, offset 32, is 16 bits of unique identification that is supposed to be unique for the entire time that any message should be in the system. That's what firewalls and routers use to determine if it's a duplicate packet. You never before stated that Linux was duplicating the ID portion, and if it is, it's a bug. But, I'll bet that it isn't. Nothing would work if it was. All TCP/IP messages are composed of Datagrams. If these basic elements were mucked up, this would have been discovered long before now, and if not discovered, you wouldn't receive this message. Also UCP is connectionless and stateless. If you have some box that handles it differently, its broken.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by John Ashcroft.
98.36% of all statistics are fiction.
Quoted 8 lines · Adam Denenberg
sorry for any confusion, but i am not referring to the Identification field in the IP header but rather the "Transaction ID" field in the DNS query portion of the packet. I can reproduce this behavior on our linux system where if i pump
gethostbyname_rrequests on the system at some point it will reuse a transaction id in the DNS request. This is my lastest discovery in what is causing the requests to fail thru the firewall. So far my research has not turned up any reason as to why the kernel would be re-using a transaction ID in the dns request.
That would be your DNS software reusing the transaction ID, since it's part of the packet data, not the UDP header. The kernel has nothing to do with it.
-Doug
but this would be an issue with glibc then correct?
Quoted 17 lines · Doug McNaught
Quoted 8 lines · Adam Denenberg
sorry for any confusion, but i am not referring to the Identification field in the IP header but rather the "Transaction ID" field in the DNS query portion of the packet. I can reproduce this behavior on our linux system where if i pump
gethostbyname_rrequests on the system at some point it will reuse a transaction id in the DNS request. This is my lastest discovery in what is causing the requests to fail thru the firewall. So far my research has not turned up any reason as to why the kernel would be re-using a transaction ID in the dns request.That would be your DNS software reusing the transaction ID, since it's part of the packet data, not the UDP header. The kernel has nothing to do with it.
-Doug
Quoted 1 line · Adam Denenberg
but this would be an issue with glibc then correct?
Whatever's sending the DNS request--the resolver library, BIND, or what have you...
-Doug
Hi,
Quoted 4 lines · Adam Denenberg
My issue is that linux is not randomizing or incrementing the ports it uses for udp connections to prevent this sort of issue since udp is connectionless. We dont have sequence numbers or the sorts like TCP to sort this out, we only have source ip and port.
Most UDP services use the server socket to access remote servers. For example, bind will only use its source port 53 to emit requests, ntpd will only use source port 123, this *is* how they have been working for ages. PIX is totally buggy and does not respect standards, it respects what seems to be "common practices" (remember TCP ECN ?). If the developpers have observed that freebsd wastes lots of source ports for DNS requests, they will decide that using the same one is probably an attack. I even remember that it does not support NTP correctly. When an NTP server on your LAN "connects" to another one outside, it translates the source port 123 to another source port < 1024, which most servers and/or firewalls drop (they only let 123 or >=1024 in).
Quoted 3 lines
Other OS's seem to do this and thus apps are not getting broken when connections are made thru the firewall, which is why i originally posted the question.
Perhaps the other OS you have seen simply close the socket as soon as they get their response, and have to create a new one for each new request... How does any DNS server forward to outside through your PIX ?
Quoted 5 lines
I was hoping that maybe there is some workaround or that hopefully someone else encountered this issue. I am not saying the firewall is not totally to blame, but i can see why it is behaving the way it is when seeing tons of connections from the same udp ip/port come in.
I doubt you can reliably use any UDP-based application through this firewall then... It seems more buggy than other pixes I have encountered, and I think you can configure it to be less silly, but I don't know how. You agree that a session is defined by these 5 numbers :
- proto (udp, tcp, icmp, ...)
- source ip, source port
- destination ip, destination port
If these 5 parameters are the same within a certain time frame, the packet belongs to a known session. If the combination is different, then it is a new session, and there is no reason for the firewall to drop a new session only based on the fact that 3 parameters out of 5 are the same as other ones. It is as stupid as saying that you refuse to establish a new TCP connection on some dest:80 because you just did it 30ms ago.
I you want to work around this buggy behaviour without reconfiguring the PIX, you can also play with iptables on the linux machine to use a random source port range :
iptables -t nat -A OUTPUT -p udp --dport 53 -j SNAT --to-source <your ip>:1024-65534
But this is a dirty hack...
Regards,
Willy
Quoted 5 lines · Adam Denenberg
i think you guys are all right. However there is one concern. Not clearing out a UDP connection in a firewall coming from a high port is indeed a security risk. Allowing a high numbered udp port to remain open for a prolonged period of time would definitely impose a security risk which is why the PIX is doing what it does.
Absolutely NO ! it is not "keeping the port open", it is "keeping a SESSION open". The firewall should allow traffic from the same ip:port to the other ip:port and from no other server on the net. You new session is totally unrelated to the old one.
Quoted 4 lines
The linux server is
"reusing" the same UDP high numbered socket however it is doing so exactly as the firewall is clearing its state table (60 ms) from the first connection which is what is causing the issue.
it is not the same session if you connect to a different remote server, and there is absolutely no reason to arbitrary prevent an internal server from connecting to two external ones from the same IP:port. Of course, if you reconnect to the same destIP:port, it should work because in this case it is a continuation of the same session.
Quoted 6 lines
I think a firewall ought to be aware of such behavior, but at the same time be secure enough to not just leave high numbered udp ports wide open for attack. I am trying to find out why the PIX chose 60 ms to clear out the UDP state table. I think that is a random number and probably too short of a span for this to occur however i am still researching it.
it is not the timing which causes trouble, it is the way it confuses new and already established sessions. Although 60 ms may seem short (you can probably never resolve anything on ADSL with a loaded link), it may be perfectly valid if the firewall agrees to open several sessions when you connect to several servers. And if you connect several times to the same server, of course it must re-open the session.
Quoted 1 line
Any other insight would be greatly appreciated.
unfortunately, googling for "pix udp problem" returns 25600 responses...
Regards,
willy
Quoted 1 line · Adam Denenberg
So my question is, why is the kernel not incrementing the DNS ID in this case?
Uhh ?
The *kernel* should change IDs in the *DNS* request ? It is the libc which does the DNS requests. The only ID the kernel could play with it the IP packet ID.
Willy
the issue with this currently is that the resolver is not using a new "Transaction ID" in the DNS porttion of the packet. This is the one unique piece of informatino that helps distinguish what query belongs to what response (since same source port and ip is used). So if the transaction id in the DNS header of the query is not updated, there is no real way to distinguish these dns requests. Also keep in mind the linux box is requesting the same A record, so even that is not unique among these requests.
I am trying to figure out why the resolver (redhat 8 glibc) is behaing this way, b/c it is breaking applications. If it simply updated its transaction ID in the packet i dont think this would be an issue.
thanks
adam
Please CC me I am not on this list.
Quoted 50 lines · Willy Tarreau
Quoted 5 lines · Adam Denenberg
i think you guys are all right. However there is one concern. Not clearing out a UDP connection in a firewall coming from a high port is indeed a security risk. Allowing a high numbered udp port to remain open for a prolonged period of time would definitely impose a security risk which is why the PIX is doing what it does.
Absolutely NO ! it is not "keeping the port open", it is "keeping a SESSION open". The firewall should allow traffic from the same ip:port to the other ip:port and from no other server on the net. You new session is totally unrelated to the old one.
Quoted 4 lines
The linux server is
"reusing" the same UDP high numbered socket however it is doing so exactly as the firewall is clearing its state table (60 ms) from the first connection which is what is causing the issue.it is not the same session if you connect to a different remote server, and there is absolutely no reason to arbitrary prevent an internal server from connecting to two external ones from the same IP:port. Of course, if you reconnect to the same destIP:port, it should work because in this case it is a continuation of the same session.
Quoted 6 lines
I think a firewall ought to be aware of such behavior, but at the same time be secure enough to not just leave high numbered udp ports wide open for attack. I am trying to find out why the PIX chose 60 ms to clear out the UDP state table. I think that is a random number and probably too short of a span for this to occur however i am still researching it.
it is not the timing which causes trouble, it is the way it confuses new and already established sessions. Although 60 ms may seem short (you can probably never resolve anything on ADSL with a loaded link), it may be perfectly valid if the firewall agrees to open several sessions when you connect to several servers. And if you connect several times to the same server, of course it must re-open the session.
Quoted 1 line
Any other insight would be greatly appreciated.
unfortunately, googling for "pix udp problem" returns 25600 responses...
Regards,
willy-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
I disagree. The linux server should be using unique Transaction ID's in the dns header for each unique dns request. Otherwise there is no way to distinguish them (same A record request). Of course the firewall is going to drop a reply that it thinks it already saw a reply for 30ms ago.
This appears to be a bug in the way glibc is handling things but i cannot be sure. That is the goal of my investigation.
adam
Please CC me i am not on the list.
Quoted 25 lines · Jan Harkes
Quoted 4 lines · Adam Denenberg
the Firewall from distinguishing unique dns requests. It sees a second DNS request come from the linux server with the _same_ transaction ID in the UDP header as it is marking that session closed since it already saw the reply successfully. So for example the linux server is making a dns
Stupid guess here,
The reply got dropped after it passed your firewall and before it reached the linux server. What you are seeing is simply a retransmit which would also have happened if the original request got lost, or if the reply was dropped before it reached your firewall, in which case the firewall probably would have forwarded the retransmitted request without a problem.
I would open the window before you throw the piece of garbage out.
Jan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 8 lines · Adam Denenberg
I disagree. The linux server should be using unique Transaction ID's in the dns header for each unique dns request. Otherwise there is no way to distinguish them (same A record request). Of course the firewall is going to drop a reply that it thinks it already saw a reply for 30ms ago.
This appears to be a bug in the way glibc is handling things but i cannot be sure. That is the goal of my investigation.
Ok, please stop it and _read_ the email, completely, a possible solution will be at the end.
This is not a problem with the linux kernel, the application (resolver library) is simply retransmitting the request and the kernel does exactly as it is told. So first of all this is completely off-topic for linux-kernel.
Second of all, it is -not- a glibc issue either. The reply packet was probably lost after it has passed through the firewall but before it reached the DNS client machine. So since it has not yet received the response it is doing the right thing, retransmitting the request.
The problem is... your firewall, it seems to assume that once it has seen a UDP reply packet that this packet will reach the destination and the session is closed. This is an incorrect assumption since UDP is per definition a lossy protocol (unreliable datagram protocol) and the packet can and will be dropped by any routers, switches, the network card in your machine, or even by the kernel if there is too much traffic or a single bit error which makes the checksum fail.
One solution that might work is to run a local caching DNS daemon on a machine behind the firewall (bind, dnsmasq or something) and set /etc/resolv.conf to only send requests to the local machine. The local caching daemon can then forward those requests from a known fixed local port to your upstream DNS servers. This allows you to open a hole in the firewall between local_dns_cache:53 and upstream_dns_server:53.
Jan
Ok thanks for all the help.
One final note before i end this (since there still seems to be some misunderstanding) is that the reply comes back successful for the first of the 2 redundant transaction IDs. Its not a retransmit. ID 101 gets sent from the linux box and receives a response successfully. It then sends another reqeust with ID 101 _after_ it received the response for 101. That is my issue and that is what is breaking the communication thru the firewall. It should be using a new ID after a successful dns transaction but it is not.
I understand this has deviated from the topic of the linux kernel so we can consider this thread dead.
thanks everybody for all their input and help.
adam
Quoted 40 lines · Jan Harkes
Quoted 8 lines · Adam Denenberg
I disagree. The linux server should be using unique Transaction ID's in the dns header for each unique dns request. Otherwise there is no way to distinguish them (same A record request). Of course the firewall is going to drop a reply that it thinks it already saw a reply for 30ms ago.
This appears to be a bug in the way glibc is handling things but i cannot be sure. That is the goal of my investigation.
Ok, please stop it and _read_ the email, completely, a possible solution will be at the end.
This is not a problem with the linux kernel, the application (resolver library) is simply retransmitting the request and the kernel does exactly as it is told. So first of all this is completely off-topic for linux-kernel.
Second of all, it is -not- a glibc issue either. The reply packet was probably lost after it has passed through the firewall but before it reached the DNS client machine. So since it has not yet received the response it is doing the right thing, retransmitting the request.
The problem is... your firewall, it seems to assume that once it has seen a UDP reply packet that this packet will reach the destination and the session is closed. This is an incorrect assumption since UDP is per definition a lossy protocol (unreliable datagram protocol) and the packet can and will be dropped by any routers, switches, the network card in your machine, or even by the kernel if there is too much traffic or a single bit error which makes the checksum fail.
One solution that might work is to run a local caching DNS daemon on a machine behind the firewall (bind, dnsmasq or something) and set
/etc/resolv.confto only send requests to the local machine. The local caching daemon can then forward those requests from a known fixed local port to your upstream DNS servers. This allows you to open a hole in the firewall between local_dns_cache:53 and upstream_dns_server:53.Jan