Skip to contentSkip to conversation

net: fix quite a few dst_cache crashes reported by syzbot

PATCH In review Xin Long 2 participants 5 messages Jun 17, 2019 – Jun 19, 2019

Subsystems (4)
lore
Share & tools
lore

Xin Long

author
#0 lore raw

cover net: fix quite a few dst_cache crashes reported by syzbot

There are two kinds of crashes reported many times by syzbot with no reproducer. Call Traces are like:

     BUG: KASAN: slab-out-of-bounds in rt_cache_valid+0x158/0x190
     net/ipv4/route.c:1556
       rt_cache_valid+0x158/0x190 net/ipv4/route.c:1556
       __mkroute_output net/ipv4/route.c:2332 [inline]
       ip_route_output_key_hash_rcu+0x819/0x2d50 net/ipv4/route.c:2564
       ip_route_output_key_hash+0x1ef/0x360 net/ipv4/route.c:2393
       __ip_route_output_key include/net/route.h:125 [inline]
       ip_route_output_flow+0x28/0xc0 net/ipv4/route.c:2651
       ip_route_output_key include/net/route.h:135 [inline]
     ...

or:

     kasan: GPF could be caused by NULL-ptr deref or user memory access
     RIP: 0010:dst_dev_put+0x24/0x290 net/core/dst.c:168
       <IRQ>
       rt_fibinfo_free_cpus net/ipv4/fib_semantics.c:200 [inline]
       free_fib_info_rcu+0x2e1/0x490 net/ipv4/fib_semantics.c:217
       __rcu_reclaim kernel/rcu/rcu.h:240 [inline]
       rcu_do_batch kernel/rcu/tree.c:2437 [inline]
       invoke_rcu_callbacks kernel/rcu/tree.c:2716 [inline]
       rcu_process_callbacks+0x100a/0x1ac0 kernel/rcu/tree.c:2697
     ...

They were caused by the fib_nh_common percpu member 'nhc_pcpu_rth_output' overwritten by another percpu variable 'dev->tstats' access overflow in tipc udp media xmit path when counting packets on a non tunnel device.

The fix is to make udp tunnel work with no tunnel device by allowing not to count packets on the tstats when the tunnel dev is NULL in Patches 1/3 and 2/3, then pass a NULL tunnel dev in tipc_udp_tunnel() in Patch 3/3.

Xin Long (3):
ip_tunnel: allow not to count pkts on tstats by setting skb's dev to
NULL
ip6_tunnel: allow not to count pkts on tstats by passing dev as NULL
tipc: pass tunnel dev as NULL to udp_tunnel(6)_xmit_skb

 include/net/ip6_tunnel.h  | 9 ++++++---
 net/ipv4/ip_tunnel_core.c | 9 ++++++---
 net/tipc/udp_media.c      | 8 +++-----
 3 files changed, 15 insertions(+), 11 deletions(-)
Signature · 1 line
2.1.0
author
#1 lore raw

PATCH 1/3 ip_tunnel: allow not to count pkts on tstats by setting skb's dev to NULL

iptunnel_xmit() works as a common function, also used by a udp tunnel which doesn't have to have a tunnel device, like how TIPC works with udp media.

In these cases, we should allow not to count pkts on dev's tstats, so that udp tunnel can work with no tunnel device safely.

---
 net/ipv4/ip_tunnel_core.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
net/ipv4/ip_tunnel_core.c +6 -3
index 30c1c26..5073e3c 100644
@@ -89,9 +89,12 @@ void iptunnel_xmit(struct sock *sk, struct rtable *rt, struct sk_buff *skb,
 	__ip_select_ident(net, iph, skb_shinfo(skb)->gso_segs ?: 1);
 
 	err = ip_local_out(net, sk, skb);
-	if (unlikely(net_xmit_eval(err)))
-		pkt_len = 0;
-	iptunnel_xmit_stats(dev, pkt_len);
+
+	if (dev) {
+		if (unlikely(net_xmit_eval(err)))
+			pkt_len = 0;
+		iptunnel_xmit_stats(dev, pkt_len);
+	}
 }
 EXPORT_SYMBOL_GPL(iptunnel_xmit);
 
Signature · 1 line
2.1.0
author
#2 lore raw

PATCH 2/3 ip6_tunnel: allow not to count pkts on tstats by passing dev as NULL

A similar fix to Patch "ip_tunnel: allow not to count pkts on tstats by setting skb's dev to NULL" is also needed by ip6_tunnel.

---
 include/net/ip6_tunnel.h | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
include/net/ip6_tunnel.h +6 -3
index 69b4bcf..028eaea 100644
@@ -158,9 +158,12 @@ static inline void ip6tunnel_xmit(struct sock *sk, struct sk_buff *skb,
 	memset(skb->cb, 0, sizeof(struct inet6_skb_parm));
 	pkt_len = skb->len - skb_inner_network_offset(skb);
 	err = ip6_local_out(dev_net(skb_dst(skb)->dev), sk, skb);
-	if (unlikely(net_xmit_eval(err)))
-		pkt_len = -1;
-	iptunnel_xmit_stats(dev, pkt_len);
+
+	if (dev) {
+		if (unlikely(net_xmit_eval(err)))
+			pkt_len = -1;
+		iptunnel_xmit_stats(dev, pkt_len);
+	}
 }
 #endif
 #endif
Signature · 1 line
2.1.0
author
#3 lore raw

PATCH 3/3 tipc: pass tunnel dev as NULL to udp_tunnel(6)_xmit_skb

udp_tunnel(6)_xmit_skb() called by tipc_udp_xmit() expects a tunnel device to count packets on dev->tstats, a perpcu variable. However, TIPC is using udp tunnel with no tunnel device, and pass the lower dev, like veth device that only initializes dev->lstats(a perpcu variable) when creating it.

Later iptunnel_xmit_stats() called by ip(6)tunnel_xmit() thinks the dev as a tunnel device, and uses dev->tstats instead of dev->lstats. tstats' each pointer points to a bigger struct than lstats, so when tstats->tx_bytes is increased, other percpu variable's members could be overwritten.

syzbot has reported quite a few crashes due to fib_nh_common percpu member 'nhc_pcpu_rth_output' overwritten, call traces are like:

  BUG: KASAN: slab-out-of-bounds in rt_cache_valid+0x158/0x190
  net/ipv4/route.c:1556
    rt_cache_valid+0x158/0x190 net/ipv4/route.c:1556
    __mkroute_output net/ipv4/route.c:2332 [inline]
    ip_route_output_key_hash_rcu+0x819/0x2d50 net/ipv4/route.c:2564
    ip_route_output_key_hash+0x1ef/0x360 net/ipv4/route.c:2393
    __ip_route_output_key include/net/route.h:125 [inline]
    ip_route_output_flow+0x28/0xc0 net/ipv4/route.c:2651
    ip_route_output_key include/net/route.h:135 [inline]
  ...

or:

kasan: GPF could be caused by NULL-ptr deref or user memory access

  RIP: 0010:dst_dev_put+0x24/0x290 net/core/dst.c:168
    <IRQ>
    rt_fibinfo_free_cpus net/ipv4/fib_semantics.c:200 [inline]
    free_fib_info_rcu+0x2e1/0x490 net/ipv4/fib_semantics.c:217
    __rcu_reclaim kernel/rcu/rcu.h:240 [inline]
    rcu_do_batch kernel/rcu/tree.c:2437 [inline]
    invoke_rcu_callbacks kernel/rcu/tree.c:2716 [inline]
    rcu_process_callbacks+0x100a/0x1ac0 kernel/rcu/tree.c:2697
  ...

The issue exists since tunnel stats update is moved to iptunnel_xmit by Commit 039f50629b7f ("ip_tunnel: Move stats update to iptunnel_xmit()"), and here to fix it by passing a NULL tunnel dev to udp_tunnel(6)_xmit_skb so that the packets counting won't happen on dev->tstats.

Reported-by [email protected]
Reported-by [email protected]
Reported-by [email protected]
Reported-by [email protected]
Reported-by [email protected]
Reported-by [email protected]
Fixes 039f50629b7f ("ip_tunnel: Move stats update to iptunnel_xmit()")
---
 net/tipc/udp_media.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
net/tipc/udp_media.c +3 -5
index 7fc02d8..1405ccc 100644
@@ -176,7 +176,6 @@ static int tipc_udp_xmit(struct net *net, struct sk_buff *skb,
 			goto tx_error;
 		}
 
-		skb->dev = rt->dst.dev;
 		ttl = ip4_dst_hoplimit(&rt->dst);
 		udp_tunnel_xmit_skb(rt, ub->ubsock->sk, skb, src->ipv4.s_addr,
 				    dst->ipv4.s_addr, 0, ttl, 0, src->port,
@@ -195,10 +194,9 @@ static int tipc_udp_xmit(struct net *net, struct sk_buff *skb,
 		if (err)
 			goto tx_error;
 		ttl = ip6_dst_hoplimit(ndst);
-		err = udp_tunnel6_xmit_skb(ndst, ub->ubsock->sk, skb,
-					   ndst->dev, &src->ipv6,
-					   &dst->ipv6, 0, ttl, 0, src->port,
-					   dst->port, false);
+		err = udp_tunnel6_xmit_skb(ndst, ub->ubsock->sk, skb, NULL,
+					   &src->ipv6, &dst->ipv6, 0, ttl, 0,
+					   src->port, dst->port, false);
 #endif
 	}
 	return err;
Signature · 1 line
2.1.0
maintainer
#4 lore raw
In reply to Xin Long

From: Xin Long <[email protected]>
Date: Mon, 17 Jun 2019 21:34:12 +0800

Quoted 2 lines

There are two kinds of crashes reported many times by syzbot with no reproducer. Call Traces are like:

...

Quoted 7 lines

They were caused by the fib_nh_common percpu member 'nhc_pcpu_rth_output' overwritten by another percpu variable 'dev->tstats' access overflow in tipc udp media xmit path when counting packets on a non tunnel device.

The fix is to make udp tunnel work with no tunnel device by allowing not to count packets on the tstats when the tunnel dev is NULL in Patches 1/3 and 2/3, then pass a NULL tunnel dev in tipc_udp_tunnel() in Patch 3/3.

Series applied and queued up for -stable.

Thanks for putting all of those syzbot reported by tags in there.