xfrm: Don't lookup sk_policy for timewait sockets
Subsystems (4)
We are running a couple of thousand machines with 3.8 and 3.12. On very few systems (something below 10) we encounter panics in xfrm code. The main characteristic seams to be the usage of TPROXY.
Attached patch is only a workaround, as problems may also happen in other code portions (actually on even fewer systems this happens).
For timewait sockets the memory region of sk_policy does not belong to us anymore. So there may be someone else using it and we may panic because of corrupted pointers.
xfrm_sk_policy_lookup+0x38/0x66 xfrm_lookup+0x93/0x48f nf_nat_packet+0x92/0xa4 [nf_nat] _decode_session4+0xd9/0x294 nf_xfrm_me_harder+0x50/0xc5 [nf_nat] nf_nat_ipv4_out+0xad/0xc4 [iptable_nat] nf_iterate+0x42/0x7d ip_finish_output2+0x2b1/0x2b1 nf_hook_slow+0x22f/0x2c9 ip_finish_output2+0x2b1/0x2b1 ip_finish_output2+0x2b1/0x2b1 __xfrm_route_forward+0x7a/0x97 ip_finish_output2+0x2b1/0x2b1 NF_HOOK_COND+0x3f/0x54 ip_output+0x5a/0x5e __netif_receive_skb+0x4b2/0x514 process_backlog+0xee/0x1c5 net_rx_action+0xa7/0x1fe
--- net/xfrm/xfrm_policy.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
net/xfrm/xfrm_policy.c +2 -2
@@ -2072,7 +2072,7 @@ restart: xdst =NULL ; route =NULL ; -if (sk && sk->sk_policy[XFRM_POLICY_OUT ]) { +if (sk && sk->sk_state !=TCP_TIME_WAIT && sk->sk_policy[XFRM_POLICY_OUT ]) { num_pols =1 ; pols[0 ] =xfrm_sk_policy_lookup (sk,XFRM_POLICY_OUT , fl); err = xfrm_expand_policies(fl, family, pols,
@@ -2349,7 +2349,7 @@ int __xfrm_policy_check(struct sock *sk, int dir, struct sk_buff *skb, } pol =NULL ; -if (sk && sk->sk_policy[dir]) { +if (sk && sk->sk_state !=TCP_TIME_WAIT && sk->sk_policy[dir]) { pol =xfrm_sk_policy_lookup (sk, dir, &fl);if (IS_ERR (pol)) {XFRM_INC_STATS (net,LINUX_MIB_XFRMINPOLERROR );
Signature · 1 line
2.1.0
Quoted 49 lines · Sebastian Poehn
We are running a couple of thousand machines with 3.8 and 3.12. On very few systems (something below 10) we encounter panics in xfrm code. The main characteristic seams to be the usage of TPROXY.
Attached patch is only a workaround, as problems may also happen in other code portions (actually on even fewer systems this happens).
For timewait sockets the memory region of
sk_policydoes not belong to us anymore. So there may be someone else using it and we may panic because of corrupted pointers.xfrm_sk_policy_lookup+0x38/0x66 xfrm_lookup+0x93/0x48f nf_nat_packet+0x92/0xa4 [nf_nat] _decode_session4+0xd9/0x294 nf_xfrm_me_harder+0x50/0xc5 [nf_nat] nf_nat_ipv4_out+0xad/0xc4 [iptable_nat] nf_iterate+0x42/0x7d ip_finish_output2+0x2b1/0x2b1 nf_hook_slow+0x22f/0x2c9 ip_finish_output2+0x2b1/0x2b1 ip_finish_output2+0x2b1/0x2b1 __xfrm_route_forward+0x7a/0x97 ip_finish_output2+0x2b1/0x2b1 NF_HOOK_COND+0x3f/0x54 ip_output+0x5a/0x5e __netif_receive_skb+0x4b2/0x514 process_backlog+0xee/0x1c5 net_rx_action+0xa7/0x1feSigned-off-by Sebastian Poehn <[email protected]>--- net/xfrm/xfrm_policy.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)net/xfrm/xfrm_policy.c +1
-1@@ -2072,7 +2072,7 @@ restart: xdst =NULL ; route =NULL ; -if (sk && sk->sk_policy[XFRM_POLICY_OUT ]) { +if (sk && sk->sk_state !=TCP_TIME_WAIT && sk->sk_policy[XFRM_POLICY_OUT ]) { num_pols =1 ; pols[0 ] =xfrm_sk_policy_lookup (sk,XFRM_POLICY_OUT , fl); err = xfrm_expand_policies(fl, family, pols,
net/xfrm/xfrm_policy.c int __xfrm_policy_check(struct sock *sk, int dir, struct sk_buff *skb, · line 2355
Quoted 8 lines
net/xfrm/xfrm_policy.c +1
-1@@ -2349,7 +2349,7 @@ int __xfrm_policy_check(struct sock *sk, int dir, struct sk_buff *skb, } pol =NULL ; -if (sk && sk->sk_policy[dir]) { +if (sk && sk->sk_state !=TCP_TIME_WAIT && sk->sk_policy[dir]) { pol =xfrm_sk_policy_lookup (sk, dir, &fl);if (IS_ERR (pol)) {XFRM_INC_STATS (net,LINUX_MIB_XFRMINPOLERROR );
Hmm.. interesting.
TCP stack never sends packets attached to a socket in timewait state.
On IPv4, the ACK packets sent on behalf of TIME_WAIT follow this path :
tcp_v4_timewait_ack()
tcp_v4_send_ack()
ip_send_unicast_reply(*this_cpu_ptr(net->ipv4.tcp_sk))
The per cpu socket used to attach these skb are not in TCP_TIME_WAIT state.
TPROXY can handle a socket in TCP_TIME_WAIT state only in input path.
So I am a bit confused. Could you elaborate ?
In any case, you probably should use sk_fullsock() new helper, as I presume you'll have a similar issue with TCP_NEW_SYN_RECV pseudo sockets when I am done with tcp listener refactoring.
Thanks.
Quoted 69 lines · Eric Dumazet
Quoted 49 lines · Sebastian Poehn
We are running a couple of thousand machines with 3.8 and 3.12. On very few systems (something below 10) we encounter panics in xfrm code. The main characteristic seams to be the usage of TPROXY.
Attached patch is only a workaround, as problems may also happen in other code portions (actually on even fewer systems this happens).
For timewait sockets the memory region of
sk_policydoes not belong to us anymore. So there may be someone else using it and we may panic because of corrupted pointers.xfrm_sk_policy_lookup+0x38/0x66 xfrm_lookup+0x93/0x48f nf_nat_packet+0x92/0xa4 [nf_nat] _decode_session4+0xd9/0x294 nf_xfrm_me_harder+0x50/0xc5 [nf_nat] nf_nat_ipv4_out+0xad/0xc4 [iptable_nat] nf_iterate+0x42/0x7d ip_finish_output2+0x2b1/0x2b1 nf_hook_slow+0x22f/0x2c9 ip_finish_output2+0x2b1/0x2b1 ip_finish_output2+0x2b1/0x2b1 __xfrm_route_forward+0x7a/0x97 ip_finish_output2+0x2b1/0x2b1 NF_HOOK_COND+0x3f/0x54 ip_output+0x5a/0x5e __netif_receive_skb+0x4b2/0x514 process_backlog+0xee/0x1c5 net_rx_action+0xa7/0x1feSigned-off-by Sebastian Poehn <[email protected]>--- net/xfrm/xfrm_policy.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)net/xfrm/xfrm_policy.c +1
-1@@ -2072,7 +2072,7 @@ restart: xdst =NULL ; route =NULL ; -if (sk && sk->sk_policy[XFRM_POLICY_OUT ]) { +if (sk && sk->sk_state !=TCP_TIME_WAIT && sk->sk_policy[XFRM_POLICY_OUT ]) { num_pols =1 ; pols[0 ] =xfrm_sk_policy_lookup (sk,XFRM_POLICY_OUT , fl); err = xfrm_expand_policies(fl, family, pols,net/xfrm/xfrm_policy.c int __xfrm_policy_check(struct sock *sk, int dir, struct sk_buff *skb, · line 2355
Quoted 8 lines
net/xfrm/xfrm_policy.c +1
-1@@ -2349,7 +2349,7 @@ int __xfrm_policy_check(struct sock *sk, int dir, struct sk_buff *skb, } pol =NULL ; -if (sk && sk->sk_policy[dir]) { +if (sk && sk->sk_state !=TCP_TIME_WAIT && sk->sk_policy[dir]) { pol =xfrm_sk_policy_lookup (sk, dir, &fl);if (IS_ERR (pol)) {XFRM_INC_STATS (net,LINUX_MIB_XFRMINPOLERROR );Hmm.. interesting.
TCP stack never sends packets attached to a socket in timewait state.
On IPv4, the ACK packets sent on behalf of
TIME_WAITfollow this path :
tcp_v4_timewait_ack()
tcp_v4_send_ack()
ip_send_unicast_reply(*this_cpu_ptr(net->ipv4.tcp_sk))
Quoted 6 lines
The per cpu socket used to attach these skb are not in
TCP_TIME_WAITstate.TPROXY can handle a socket in
TCP_TIME_WAITstate only in input path.So I am a bit confused. Could you elaborate ?
The setup is a usual transparent proxy one. So TPROXY intercepts the first connection packet and later on we use socket match to direct to the local IP_TRANSPARENT socket. So sorry for the confusion - xt_socket was meant.
The strange thing I noticed is that tw_transparent was 0 (we don't use xt_socket --transparent). But then I wonder how the tuple can match.
I will collect some more information and post them.
Quoted 4 lines
In any case, you probably should use
sk_fullsock()new helper, as I presume you'll have a similar issue withTCP_NEW_SYN_RECVpseudo sockets when I am done with tcp listener refactoring.
I already saw and like sk_fullsock.
Quoted 4 lines
Thanks.
David Miller
maintainerFrom: Eric Dumazet <[email protected]>
Date: Thu, 09 Apr 2015 02:07:41 -0700
Quoted 1 line
TCP stack never sends packets attached to a socket in timewait state.
TPROXY assigns timewait sockets to skb->sk, that's the bug.
In net/netfilter/xt_TPROXY.c:
tproxy_tg4()
...
sk = nf_tproxy_get_sock_v4(dev_net(skb->dev), iph->protocol,
iph->saddr, iph->daddr,
hp->source, hp->dest,
skb->dev, NFT_LOOKUP_ESTABLISHED);
/* NOTE: assign_sock consumes our sk reference */
if (sk && tproxy_sk_is_transparent(sk)) {
/* This should be in a separate target, but we don't do multiple
targets on the same rule yet */
skb->mark = (skb->mark & ~mark_mask) ^ mark_value;
pr_debug("redirecting: proto %hhu %pI4:%hu -> %pI4:%hu, mark: %x\n",
iph->protocol, &iph->daddr, ntohs(hp->dest),
&laddr, ntohs(lport), skb->mark);
nf_tproxy_assign_sock(skb, sk); ...
/* assign a socket to the skb -- consumes sk */
static voidnf_tproxy_assign_sock(struct sk_buff *skb, struct sock *sk)
{
skb_orphan(skb);
skb->sk = sk;
skb->destructor = sock_edemux;
}Quoted 6 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Thu, 09 Apr 2015 02:07:41 -0700Quoted 1 line
TCP stack never sends packets attached to a socket in timewait state.
TPROXY assigns timewait sockets to
skb->sk, that's the bug.
I re-introduced this in fd158d79d33d3c under the assumption that the input path handles skb->sk timewait sockets correctly after all the early demux changes, afaics tcp edemux can also assign skb->sk timewait sockets.
Also, reporter mentions 3.8 as affected which should not assign tw sockets to skb->sk.
Even more strange, the reporters backtrace seems to indicate crash at end of forward path.
Sebastian, can you disable tw assignment via TPROXY in 3.12 just to see if it makes a difference?
[ not doing the assignment is safe provided you still set tproxy mark
on the skb; policy routing will ensure local delivery ].
David Miller
maintainerFrom: Florian Westphal <[email protected]>
Date: Thu, 9 Apr 2015 21:14:41 +0200
Quoted 8 lines
I re-introduced this in
fd158d79d33d3cunder the assumption that the input path handlesskb->sktimewait sockets correctly after all the early demux changes, afaics tcp edemux can also assignskb->sktimewait sockets.Also, reporter mentions 3.8 as affected which should not assign tw sockets to
skb->sk.
Quoted 8 lines
Even more strange, the reporters backtrace seems to indicate crash at end of forward path.
Sebastian, can you disable tw assignment via TPROXY in 3.12 just to see if it makes a difference?
[ not doing the assignment is safe provided you still set tproxy mark
on the skb; policy routing will ensure local delivery ].
My assumption in my analysis is that TPROXY writes the socket to skb->sk, and it is also being forwarded. And yes this is based upon his backtrace.
Quoted 17 lines · David Miller
From: Florian Westphal <[email protected]>
Date: Thu, 9 Apr 2015 21:14:41 +0200Quoted 14 lines
I re-introduced this in
fd158d79d33d3cunder the assumption that the input path handlesskb->sktimewait sockets correctly after all the early demux changes, afaics tcp edemux can also assignskb->sktimewait sockets.Also, reporter mentions 3.8 as affected which should not assign tw sockets to
skb->sk.Even more strange, the reporters backtrace seems to indicate crash at end of forward path.
Sebastian, can you disable tw assignment via TPROXY in 3.12 just to see if it makes a difference?
Quoted 6 lines
Quoted 2 lines
[ not doing the assignment is safe provided you still set tproxy mark
on the skb; policy routing will ensure local delivery ].My assumption in my analysis is that TPROXY writes the socket to
skb->sk, and it is also being forwarded. And yes this is based upon his backtrace.
Right. However, I think we might have to make more changes than just tproxy.
If we have sockets bound to non-local addresses then why would tcp edemux not cause same issue?
Quoted 21 lines · Florian Westphal
Quoted 20 lines · David Miller
From: Florian Westphal <[email protected]>
Date: Thu, 9 Apr 2015 21:14:41 +0200Quoted 16 lines
I re-introduced this in
fd158d79d33d3cunder the assumption that the input path handlesskb->sktimewait sockets correctly after all the early demux changes, afaics tcp edemux can also assignskb->sktimewait sockets.Also, reporter mentions 3.8 as affected which should not assign tw sockets to
skb->sk.Even more strange, the reporters backtrace seems to indicate crash at end of forward path.
Sebastian, can you disable tw assignment via TPROXY in 3.12 just to see if it makes a difference?
[ not doing the assignment is safe provided you still set tproxy mark
on the skb; policy routing will ensure local delivery ].
Quoted 8 lines
Quoted 3 lines
My assumption in my analysis is that TPROXY writes the socket to
skb->sk, and it is also being forwarded. And yes this is based upon his backtrace.Right. However, I think we might have to make more changes than just tproxy.
If we have sockets bound to non-local addresses then why would tcp edemux not cause same issue?
Thanks for all the helpful inputs. I will try to provide some more information and try a bit around with TPROXY not assigning tw sockets.
I will provide you with an update the next days.
Unfortunately this is a very rare event and (yet) impossible to reproduce in-house.
Am 10.04.2015 13:14 schrieb "Sebastian Poehn" <[email protected]>:
Quoted 32 lines · Florian Westphal
Quoted 21 lines
Quoted 20 lines · David Miller
From: Florian Westphal <[email protected]>
Date: Thu, 9 Apr 2015 21:14:41 +0200Quoted 16 lines
I re-introduced this in
fd158d79d33d3cunder the assumption that the input path handlesskb->sktimewait sockets correctly after all the early demux changes, afaics tcp edemux can also assignskb->sktimewait sockets.Also, reporter mentions 3.8 as affected which should not assign tw sockets to
skb->sk.Even more strange, the reporters backtrace seems to indicate crash at end of forward path.
Sebastian, can you disable tw assignment via TPROXY in 3.12 just to see if it makes a difference?
[ not doing the assignment is safe provided you still set tproxy mark
on the skb; policy routing will ensure local delivery ].Quoted 8 lines
Quoted 3 lines
My assumption in my analysis is that TPROXY writes the socket to
skb->sk, and it is also being forwarded. And yes this is based upon his backtrace.Right. However, I think we might have to make more changes than just tproxy.
If we have sockets bound to non-local addresses then why would tcp edemux not cause same issue?
Quoted 8 lines
Thanks for all the helpful inputs. I will try to provide some more information and try a bit around with TPROXY not assigning tw sockets.
I will provide you with an update the next days.
Unfortunately this is a very rare event and (yet) impossible to reproduce in-house.
Played around with sending crafted packets to a transparent tw socket.
For SYN tproxy does the re-lookup of the listening socket, which is fine. But for packets without SYN is assigns the tw socket. However this is not an issue as the fw mark is set, policy routing processes frame, so it becomes input and finally is dropped in TCP receive path. But if I remove the policy routing rule the frame enters the forwarding path.
Unfortunately this did not trigger the panic but this may be just by chance.
However I can't explain what is wrong with the ip rule maybe setup related.
Quoted 9 lines · Sebastian Poehn
Played around with sending crafted packets to a transparent tw socket.
For SYN tproxy does the re-lookup of the listening socket, which is fine. But for packets without SYN is assigns the tw socket. However this is not an issue as the fw mark is set, policy routing processes frame, so it becomes input and finally is dropped in TCP receive path. But if I remove the policy routing rule the frame enters the forwarding path.
Quoted 4 lines
Unfortunately this did not trigger the panic but this may be just by chance.
However I can't explain what is wrong with the ip rule maybe setup related.
First of all: This issue will only happen if there is something screwed up with policy routing. We don't use any 'exotic' policy to match the TPROXY traffic nor is there anything that could damage the mark.
ip rule add from all fwmark 0x1/0x1 lookup X
However it happens - maybe a race with configuration.
I found TPROXY behavior correct:
1) For SYN on tw socket it assigns listening socket
2) Otherwise tw socket is assigned with is required for protocol conformity
Principally the problem is that TPROXY cannot ensure that policy routing is working correctly. Florian suggested me to clean skb->sk in ip_forward. I even think dropping the frame is fine. Not sure if this is suited for mainline.
net/ipv4/ip_forward.c +4 -0
@@ -82,6 +82,10 @@ int ip_forward(struct sk_buff *skb)if (skb->pkt_type !=PACKET_HOST ) goto drop; +/* this should happen neither */ +if (unlikely (skb->sk)) + goto drop; +if (skb_warn_if_lro (skb)) goto drop;
Quoted 3 lines · Sebastian Poehn
Principally the problem is that TPROXY cannot ensure that policy routing is working correctly. Florian suggested me to clean
skb->skinip_forward. I even think dropping the frame is fine. Not sure if this is suited for mainline.
No objections from me, this looks very reasonable.
David Miller
maintainerFrom: Eric Dumazet <[email protected]>
Date: Mon, 13 Apr 2015 08:39:04 -0700
Quoted 7 lines · Sebastian Poehn
Quoted 4 lines
Principally the problem is that TPROXY cannot ensure that policy routing is working correctly. Florian suggested me to clean
skb->skinip_forward. I even think dropping the frame is fine. Not sure if this is suited for mainline.No objections from me, this looks very reasonable.
Someone please submit this to me formally.
Quoted 22 lines · Sebastian Poehn
Quoted 9 lines · Sebastian Poehn
Played around with sending crafted packets to a transparent tw socket.
For SYN tproxy does the re-lookup of the listening socket, which is fine. But for packets without SYN is assigns the tw socket. However this is not an issue as the fw mark is set, policy routing processes frame, so it becomes input and finally is dropped in TCP receive path. But if I remove the policy routing rule the frame enters the forwarding path.
Quoted 4 lines
Unfortunately this did not trigger the panic but this may be just by chance.
However I can't explain what is wrong with the ip rule maybe setup related.
First of all: This issue will only happen if there is something screwed up with policy routing. We don't use any 'exotic' policy to match the TPROXY traffic nor is there anything that could damage the mark.
ip rule add from all fwmark 0x1/0x1 lookup X
However it happens - maybe a race with configuration.
Quoted 7 lines
I found TPROXY behavior correct:
1) For SYN on tw socket it assigns listening socket
2) Otherwise tw socket is assigned with is required for protocol conformityPrincipally the problem is that TPROXY cannot ensure that policy routing is working correctly. Florian suggested me to clean
skb->skinip_forward. I even think dropping the frame is fine. Not sure if this is suited for mainline.
I agree, drop is preferable. I also think this should go in mainline, kernel shouldn't be prone to oopses just because someone flushed ip rules at wrong moment.
Thanks Sebastian.
Quoted 30 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Thu, 09 Apr 2015 02:07:41 -0700Quoted 1 line
TCP stack never sends packets attached to a socket in timewait state.
TPROXY assigns timewait sockets to
skb->sk, that's the bug.In
net/netfilter/xt_TPROXY.c:tproxy_tg4() ... sk = nf_tproxy_get_sock_v4(dev_net(skb->dev), iph->protocol, iph->saddr, iph->daddr, hp->source, hp->dest, skb->dev, NFT_LOOKUP_ESTABLISHED); /* NOTE: assign_sock consumes our sk reference */ if (sk && tproxy_sk_is_transparent(sk)) { /* This should be in a separate target, but we don't do multiple targets on the same rule yet */ skb->mark = (skb->mark & ~mark_mask) ^ mark_value; pr_debug("redirecting: proto %hhu %pI4:%hu -> %pI4:%hu, mark: %x\n", iph->protocol, &iph->daddr, ntohs(hp->dest), &laddr, ntohs(lport), skb->mark); nf_tproxy_assign_sock(skb, sk);...
/* assign a socket to the skb -- consumes sk */
static voidnf_tproxy_assign_sock(struct sk_buff *skb, struct sock *sk)
Quoted 5 lines
{
skb_orphan(skb);
skb->sk = sk;
skb->destructor = sock_edemux;
}Right, but stack trace shown by Sebastian seems to be an input frame, and we transmit a frame. This is the part I do not understand, yet.
Quoted 38 lines · Eric Dumazet
Quoted 30 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Thu, 09 Apr 2015 02:07:41 -0700Quoted 1 line
TCP stack never sends packets attached to a socket in timewait state.
TPROXY assigns timewait sockets to
skb->sk, that's the bug.In
net/netfilter/xt_TPROXY.c:tproxy_tg4() ... sk = nf_tproxy_get_sock_v4(dev_net(skb->dev), iph->protocol, iph->saddr, iph->daddr, hp->source, hp->dest, skb->dev, NFT_LOOKUP_ESTABLISHED); /* NOTE: assign_sock consumes our sk reference */ if (sk && tproxy_sk_is_transparent(sk)) { /* This should be in a separate target, but we don't do multiple targets on the same rule yet */ skb->mark = (skb->mark & ~mark_mask) ^ mark_value; pr_debug("redirecting: proto %hhu %pI4:%hu -> %pI4:%hu, mark: %x\n", iph->protocol, &iph->daddr, ntohs(hp->dest), &laddr, ntohs(lport), skb->mark); nf_tproxy_assign_sock(skb, sk);...
/* assign a socket to the skb -- consumes sk */
static voidnf_tproxy_assign_sock(struct sk_buff *skb, struct sock *sk)Quoted 5 lines
{ skb_orphan(skb); skb->sk = sk; skb->destructor = sock_edemux; }
Quoted 2 lines
Right, but stack trace shown by Sebastian seems to be an input frame, and we transmit a frame. This is the part I do not understand, yet.
Maybe policy routing is wedged so skb is erronously entering forward path?
After all TPROXY depends on correct marking of skb + ip rule magic to redirect skb to localhost.