Skip to contentSkip to conversation

ipv4: fix race condition between route lookup and invalidation

PATCH In review Wei Wang 3 participants 3 messages Oct 16, 2019 – Oct 18, 2019

Subsystems (3)
lore
Share & tools
lore

Wei Wang

author
#0 lore raw

PATCH 1/1 ipv4: fix race condition between route lookup and invalidation

Jesse and Ido reported the following race condition:
<CPU A, t0> - Received packet A is forwarded and cached dst entry is taken from the nexthop ('nhc->nhc_rth_input'). Calls skb_dst_set()

<t1> - Given Jesse has busy routers ("ingesting full BGP routing tables from multiple ISPs"), route is added / deleted and rt_cache_flush() is called

<CPU B, t2> - Received packet B tries to use the same cached dst entry from t0, but rt_cache_valid() is no longer true and it is replaced in rt_cache_route() by the newer one. This calls dst_dev_put() on the original dst entry which assigns the blackhole netdev to 'dst->dev'

<CPU A, t3> - dst_input(skb) is called on packet A and it is dropped due to 'dst->dev' being the blackhole netdev

There are 2 issues in the v4 routing code:
1. A per-netns counter is used to do the validation of the route. That means whenever a route is changed in the netns, users of all routes in the netns needs to redo lookup. v6 has an implementation of only updating fn_sernum for routes that are affected.
2. When rt_cache_valid() returns false, rt_cache_route() is called to throw away the current cache, and create a new one. This seems unnecessary because as long as this route does not change, the route cache does not need to be recreated.

To fully solve the above 2 issues, it probably needs quite some code changes and requires careful testing, and does not suite for net branch.

So this patch only tries to add the deleted cached rt into the uncached list, so user could still be able to use it to receive packets until it's done.

Fixes 95c47f9cf5e0 ("ipv4: call dst_dev_put() properly")
Cc David Ahern <[email protected]>
---
 net/ipv4/route.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/ipv4/route.c +1 -1
index 14654876127e..9e0c8dff2cd6 100644
@@ -1482,7 +1482,7 @@ static bool rt_cache_route(struct fib_nh_common *nhc, struct rtable *rt)
 	prev = cmpxchg(p, orig, rt);
 	if (prev == orig) {
 		if (orig) {
-			dst_dev_put(&orig->dst);
+			rt_add_uncached_list(orig);
 			dst_release(&orig->dst);
 		}
 	} else {
Signature · 1 line
2.23.0.700.g56cf767bdb-goog
Reviewed-by on v1 1/1
#1 lore raw
Quoted 33 lines · Wei Wang

Jesse and Ido reported the following race condition:
<CPU A, t0> - Received packet A is forwarded and cached dst entry is taken from the nexthop ('nhc->nhc_rth_input'). Calls skb_dst_set()

<t1> - Given Jesse has busy routers ("ingesting full BGP routing tables from multiple ISPs"), route is added / deleted and rt_cache_flush() is called

<CPU B, t2> - Received packet B tries to use the same cached dst entry from t0, but rt_cache_valid() is no longer true and it is replaced in rt_cache_route() by the newer one. This calls dst_dev_put() on the original dst entry which assigns the blackhole netdev to 'dst->dev'

<CPU A, t3> - dst_input(skb) is called on packet A and it is dropped due to 'dst->dev' being the blackhole netdev

There are 2 issues in the v4 routing code:
1. A per-netns counter is used to do the validation of the route. That means whenever a route is changed in the netns, users of all routes in the netns needs to redo lookup. v6 has an implementation of only updating fn_sernum for routes that are affected.
2. When rt_cache_valid() returns false, rt_cache_route() is called to throw away the current cache, and create a new one. This seems unnecessary because as long as this route does not change, the route cache does not need to be recreated.

To fully solve the above 2 issues, it probably needs quite some code changes and requires careful testing, and does not suite for net branch.

So this patch only tries to add the deleted cached rt into the uncached list, so user could still be able to use it to receive packets until it's done.

Quoted 7 lines
Fixes 95c47f9cf5e0 ("ipv4: call dst_dev_put() properly")
Cc David Ahern <[email protected]>
maintainer
#2 lore raw
In reply to Wei Wang

From: Wei Wang <[email protected]>
Date: Wed, 16 Oct 2019 12:03:15 -0700

Quoted 33 lines

Jesse and Ido reported the following race condition:
<CPU A, t0> - Received packet A is forwarded and cached dst entry is taken from the nexthop ('nhc->nhc_rth_input'). Calls skb_dst_set()

<t1> - Given Jesse has busy routers ("ingesting full BGP routing tables from multiple ISPs"), route is added / deleted and rt_cache_flush() is called

<CPU B, t2> - Received packet B tries to use the same cached dst entry from t0, but rt_cache_valid() is no longer true and it is replaced in rt_cache_route() by the newer one. This calls dst_dev_put() on the original dst entry which assigns the blackhole netdev to 'dst->dev'

<CPU A, t3> - dst_input(skb) is called on packet A and it is dropped due to 'dst->dev' being the blackhole netdev

There are 2 issues in the v4 routing code:
1. A per-netns counter is used to do the validation of the route. That means whenever a route is changed in the netns, users of all routes in the netns needs to redo lookup. v6 has an implementation of only updating fn_sernum for routes that are affected.
2. When rt_cache_valid() returns false, rt_cache_route() is called to throw away the current cache, and create a new one. This seems unnecessary because as long as this route does not change, the route cache does not need to be recreated.

To fully solve the above 2 issues, it probably needs quite some code changes and requires careful testing, and does not suite for net branch.

So this patch only tries to add the deleted cached rt into the uncached list, so user could still be able to use it to receive packets until it's done.

Quoted 6 lines
Fixes 95c47f9cf5e0 ("ipv4: call dst_dev_put() properly")

Applied and queued up for -stable.