yet another icmp reply transition bug
Hi !
While debugging the network problem of a friend of mine, I have discovered yet another icmp reply transition bug, present in all kernel versions.
Imagine the following setup:
a NAT gateway is situated between two networks with different MTUs. Let's say 1500 on one side, and 1300 on the other side.
Poor user on the 1300 network initiates a TCP connection to some other end in the 1500 network. The other end will start sending 1500-sized datagrams with DONT_FRAG set.
The NAT gateway will generate an ICMP dest_unreach/frag_needed inside ip_forward.c, which will hit conntrack and NAT on the NF_IP_LOCAL_OUT hook.
ip_nat_core.c:icmp_reply_translation() will get called, and in line 916 [ if (info->manips[i].hooknum == opposite_hook[hooknum]) ] is evaluated.
If LOCAL_NAT is disabled, opposite_hook doesn't even have an entry for LOCAL_OUT... yes, we are reading over the end of the array :((
If LOCAL_NAT is enabled, opposite_hook would return NF_IP_LOCAL_IN, which is not what we want. The inverse translation needed for the inner icmp translation happened at NF_IP_PRE_ROUTING!
I'm now heading off to a party, leaving you to think about this *g*.
If there is no reasonable proposal on how to solve this problem, I'll come up with a solution tomorrow.
Signature · 5 lines
Live long and prosper - Harald Welte / [email protected] http://www.gnumonks.org/ ============================================================================ "If this were a dictatorship, it'd be a heck of a lot easier, just so long as I'm the dictator." -- George W. Bush Dec 18, 2000
Quoted 24 lines · Harald Welte
Hi !
While debugging the network problem of a friend of mine, I have discovered yet another icmp reply transition bug, present in all kernel versions.
Imagine the following setup:
a NAT gateway is situated between two networks with different MTUs. Let's say 1500 on one side, and 1300 on the other side.
Poor user on the 1300 network initiates a TCP connection to some other end in the 1500 network. The other end will start sending 1500-sized datagrams with
DONT_FRAGset.The NAT gateway will generate an ICMP dest_unreach/frag_needed inside
ip_forward.c, which will hit conntrack and NAT on theNF_IP_LOCAL_OUThook.ip_nat_core.c:icmp_reply_translation() will get called, and in line 916 [ if (info->manips[i].hooknum == opposite_hook[hooknum]) ] is evaluated.
If
LOCAL_NATis disabled,opposite_hookdoesn't even have an entry forLOCAL_OUT... yes, we are reading over the end of the array :((
Quoted 8 lines
If
LOCAL_NATis enabled,opposite_hookwould returnNF_IP_LOCAL_IN, which is not what we want. The inverse translation needed for the inner icmp translation happened atNF_IP_PRE_ROUTING!I'm now heading off to a party, leaving you to think about this *g*.
If there is no reasonable proposal on how to solve this problem, I'll come up with a solution tomorrow.
This is exactly the problem I have posted several times now. (I found it during my tproxy development). I have modified ICMP nat so that the inner packet is translated at the same time the outer packet is. I saw no real reasons to NAT ICMP packets in two phases (other than simmetricity).
The way it works now with two phases:
#1 translating outer header (this happens in the same hook the NAT manip was
registered in)
#2 translating the inner header (this happens in the opposite of #1)
If there's no opposite for #1, the inner packet is not modified.
Let's see my case I ran into during tproxy development:
I wanted to SNAT a locally generated session to a foreign IP address (my NAT translation happened implicitly without rules, but can be reproduced by using a simple iptables rule and netcat)
SNATs for locally generated sessions are set up in POSTROUTING (ip_nat_setup_info called with the hooknum argument set to POSTROUTING), and it registers a single manip to change the destination of reply packets at opposite_hook[hook] which is PREROUTING.
Looking above means that the outer header of ICMP replies is translatated at PREROUTING (which is correctly done), and the inner header would be translated in POSTROUTING (which happens never as this is a locally destined packet since it was generated locally)
Your case is similar IMHO, it is not guaranteed that the reply ICMP packet arrives to us in our opposite hook. It does in the simplest case when the session is simply forwarded _AND_ the icmp reply is not generated by us. If either of these conditions is FALSE, opposite_hook will be bogus.
When fixing the problem I came up with two solutions:
1) fix opposite_hook, and allow many possible points of inner header translation (e.g. a manip in PREROUTING may mean ICMP reply in POSTROUTING _or_ LOCAL_IN _or_ LOCAL_OUT)
2) as the first one doesn't seem too robust to me, I fixed the translation and did the whole thing in one step. The inner header is translated at the same time the outer header is translated. I have seen no real use in having the inconsistent (translated outer, and untranslated inner header) packet traverse the IP stack.
I've posted my fix several times, I'll post it again if you like.
Signature · 2 lines
Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
Quoted 4 lines · Balazs Scheidler
This is exactly the problem I have posted several times now. (I found it during my tproxy development). I have modified ICMP nat so that the inner packet is translated at the same time the outer packet is. I saw no real reasons to NAT ICMP packets in two phases (other than simmetricity).
Mh. Somehow I never really understdood the implications and the significance of your findings. I'm sorry for my ignorance.
Quoted 5 lines
When fixing the problem I came up with two solutions:
1) fix
opposite_hook, and allow many possible points of inner header translation (e.g. a manip in PREROUTING may mean ICMP reply in POSTROUTING _or_LOCAL_IN_or_LOCAL_OUT)
This was my initial idea as well. But it becomes horribly complex and looks more like a hack - the kind of code where you need more comments than code in order to make it half-way understandable :(
Quoted 5 lines
2) as the first one doesn't seem too robust to me, I fixed the translation and did the whole thing in one step. The inner header is translated at the same time the outer header is translated. I have seen no real use in having the inconsistent (translated outer, and untranslated inner header) packet traverse the IP stack.
This sounds reasonable. The question is: Do we ever have a case where we cannot translate both, but still want the packet to be passed on? But in case of uncertainty, passing less information is preferred than yet another information leak (about internal addresses, ...).
Quoted 1 line
I've posted my fix several times, I'll post it again if you like.
I'll pick it from the archives, thanks.
Expect more comments soon.
Quoted 1 line
Bazsi
Signature · 5 lines
Live long and prosper - Harald Welte / [email protected] http://www.gnumonks.org/ ============================================================================ "If this were a dictatorship, it'd be a heck of a lot easier, just so long as I'm the dictator." -- George W. Bush Dec 18, 2000
Quoted 9 lines · Harald Welte
Quoted 4 lines · Balazs Scheidler
This is exactly the problem I have posted several times now. (I found it during my tproxy development). I have modified ICMP nat so that the inner packet is translated at the same time the outer packet is. I saw no real reasons to NAT ICMP packets in two phases (other than simmetricity).
Mh. Somehow I never really understdood the implications and the significance of your findings. I'm sorry for my ignorance.
no problem.
Quoted 10 lines
Quoted 6 lines
When fixing the problem I came up with two solutions:
1) fix
opposite_hook, and allow many possible points of inner header translation (e.g. a manip in PREROUTING may mean ICMP reply in POSTROUTING _or_LOCAL_IN_or_LOCAL_OUT)This was my initial idea as well. But it becomes horribly complex and looks more like a hack - the kind of code where you need more comments than code in order to make it half-way understandable :(
that's what I meant on 'not robust' ;)
Quoted 11 lines
Quoted 6 lines
2) as the first one doesn't seem too robust to me, I fixed the translation and did the whole thing in one step. The inner header is translated at the same time the outer header is translated. I have seen no real use in having the inconsistent (translated outer, and untranslated inner header) packet traverse the IP stack.
This sounds reasonable. The question is: Do we ever have a case where we cannot translate both, but still want the packet to be passed on? But in case of uncertainty, passing less information is preferred than yet another information leak (about internal addresses, ...).
The same manip is applied to both the inner and outer headers. So I think there's no case when we don't know what to do. (once we have that single manip)
conntrack happens prior to NAT, so the icmp packet is already taken as related to a session, conntracking will not be affected.
The only possible problem I see is packet confirmation (a quick look shows that it doesn't care about ICMP though)
PS: I've been running with this fix applied for ages now, and I saw no problems.
Signature · 2 lines
Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1