Skip to contentSkip to conversation

netfilter: conntrack race between dump_table and destroy

RFC In review Stephen Hemminger 3 participants 8 messages Nov 25, 2010 – Jan 9, 2011

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC 1/1 netfilter: conntrack race between dump_table and destroy

A customer reported a crash and the backtrace showed that ctnetlink_dump_table was running while a conntrack entry was being destroyed. It looks like the code for walking the table with hlist_nulls_for_each_entry_rcu is not correctly handling the case where it finds a deleted entry.

According to RCU documentation, when using hlist_nulls the reader must handle the case of seeing a deleted entry and not proceed further down the linked list. For lookup the correct behavior would be to restart the scan, but that would generate duplicate entries.

This patch is the simplest one of three alternatives:
1) if dead entry detected, skip the rest of the hash chain (see below)
2) remember skb location at start of hash chain and rescan that chain
3) switch to using a full lock when scanning rather than RCU.
It all depends on the amount of effort versus consistency of results.

net/netfilter/nf_conntrack_netlink.c +5 -1
@@ -651,8 +651,12 @@ restart:
 			if (NF_CT_DIRECTION(h) != IP_CT_DIR_ORIGINAL)
 				continue;
 			ct = nf_ct_tuplehash_to_ctrack(h);
+
+			/* if entry is being deleted then can not proceed
+			 * past this point. */
 			if (!atomic_inc_not_zero(&ct->ct_general.use))
-				continue;
+				break;
+
 			/* Dump entries of a given L3 protocol number.
 			 * If it is not specified, ie. l3proto == 0,
 			 * then dump everything. */
#1 lore raw
Quoted 27 lines · Stephen Hemminger

A customer reported a crash and the backtrace showed that ctnetlink_dump_table was running while a conntrack entry was being destroyed. It looks like the code for walking the table with hlist_nulls_for_each_entry_rcu is not correctly handling the case where it finds a deleted entry.

According to RCU documentation, when using hlist_nulls the reader must handle the case of seeing a deleted entry and not proceed further down the linked list. For lookup the correct behavior would be to restart the scan, but that would generate duplicate entries.

This patch is the simplest one of three alternatives:
1) if dead entry detected, skip the rest of the hash chain (see below)
2) remember skb location at start of hash chain and rescan that chain
3) switch to using a full lock when scanning rather than RCU.
It all depends on the amount of effort versus consistency of results.

net/netfilter/nf_conntrack_netlink.c +1 -0
@@ -651,8 +651,12 @@ restart:
 			if (NF_CT_DIRECTION(h) != IP_CT_DIR_ORIGINAL)
 				continue;
 			ct = nf_ct_tuplehash_to_ctrack(h);
+

net/netfilter/nf_conntrack_netlink.c restart:

Quoted 10 lines
net/netfilter/nf_conntrack_netlink.c +4 -2
+			/* if entry is being deleted then can not proceed
+			 * past this point. */
 			if (!atomic_inc_not_zero(&ct->ct_general.use))
-				continue;
+				break;
+
 			/* Dump entries of a given L3 protocol number.
 			 * If it is not specified, ie. l3proto == 0,
 			 * then dump everything. */
--

Hmm...

How restarting the loop can be a problem ?

There must be a bug somewhere else that your patch try to avoid, not to really fix.

Normally, destroyed ct is removed eventually from the chain, so this lookup should stop.

--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 41 lines · On Thu, 25 Nov 2010 07:34:33 +0100 Eric Dumazet <[email protected]> wrote:
Quoted 27 lines · Stephen Hemminger

A customer reported a crash and the backtrace showed that ctnetlink_dump_table was running while a conntrack entry was being destroyed. It looks like the code for walking the table with hlist_nulls_for_each_entry_rcu is not correctly handling the case where it finds a deleted entry.

According to RCU documentation, when using hlist_nulls the reader must handle the case of seeing a deleted entry and not proceed further down the linked list. For lookup the correct behavior would be to restart the scan, but that would generate duplicate entries.

This patch is the simplest one of three alternatives:
1) if dead entry detected, skip the rest of the hash chain (see below)
2) remember skb location at start of hash chain and rescan that chain
3) switch to using a full lock when scanning rather than RCU.
It all depends on the amount of effort versus consistency of results.

net/netfilter/nf_conntrack_netlink.c +1 -0
@@ -651,8 +651,12 @@ restart:
 			if (NF_CT_DIRECTION(h) != IP_CT_DIR_ORIGINAL)
 				continue;
 			ct = nf_ct_tuplehash_to_ctrack(h);
+

net/netfilter/nf_conntrack_netlink.c restart:

Quoted 10 lines
net/netfilter/nf_conntrack_netlink.c +4 -2
+			/* if entry is being deleted then can not proceed
+			 * past this point. */
 			if (!atomic_inc_not_zero(&ct->ct_general.use))
-				continue;
+				break;
+
 			/* Dump entries of a given L3 protocol number.
 			 * If it is not specified, ie. l3proto == 0,
 			 * then dump everything. */
--

Hmm...

Quoted 1 line

How restarting the loop can be a problem ?

At this point in the loop, some entries have been placed in the netlink dump buffer. Restarting the loop will cause duplicate entries.

Quoted 5 lines

There must be a bug somewhere else that your patch try to avoid, not to really fix.

Normally, destroyed ct is removed eventually from the chain, so this lookup should stop.

Because hlist_nulls it is possible to walk into a dead entry, in that case the next pointer is no longer valid.

Signature · 4 lines
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#3 lore raw
Quoted 44 lines · Stephen Hemminger
Quoted 41 lines · On Thu, 25 Nov 2010 07:34:33 +0100 Eric Dumazet <[email protected]> wrote:
Quoted 27 lines · Stephen Hemminger

A customer reported a crash and the backtrace showed that ctnetlink_dump_table was running while a conntrack entry was being destroyed. It looks like the code for walking the table with hlist_nulls_for_each_entry_rcu is not correctly handling the case where it finds a deleted entry.

According to RCU documentation, when using hlist_nulls the reader must handle the case of seeing a deleted entry and not proceed further down the linked list. For lookup the correct behavior would be to restart the scan, but that would generate duplicate entries.

This patch is the simplest one of three alternatives:
1) if dead entry detected, skip the rest of the hash chain (see below)
2) remember skb location at start of hash chain and rescan that chain
3) switch to using a full lock when scanning rather than RCU.
It all depends on the amount of effort versus consistency of results.

net/netfilter/nf_conntrack_netlink.c +1 -0
@@ -651,8 +651,12 @@ restart:
 			if (NF_CT_DIRECTION(h) != IP_CT_DIR_ORIGINAL)
 				continue;
 			ct = nf_ct_tuplehash_to_ctrack(h);
+

net/netfilter/nf_conntrack_netlink.c restart:

Quoted 10 lines
net/netfilter/nf_conntrack_netlink.c +4 -2
+			/* if entry is being deleted then can not proceed
+			 * past this point. */
 			if (!atomic_inc_not_zero(&ct->ct_general.use))
-				continue;
+				break;
+
 			/* Dump entries of a given L3 protocol number.
 			 * If it is not specified, ie. l3proto == 0,
 			 * then dump everything. */
--

Hmm...

Quoted 5 lines
Quoted 1 line

How restarting the loop can be a problem ?

At this point in the loop, some entries have been placed in the netlink dump buffer. Restarting the loop will cause duplicate entries.

Then this is another problem.

We cannot use RCU at all here.

Or we miss valid entries in the chain.

Quoted 9 lines
Quoted 5 lines

There must be a bug somewhere else that your patch try to avoid, not to really fix.

Normally, destroyed ct is removed eventually from the chain, so this lookup should stop.

Because hlist_nulls it is possible to walk into a dead entry, in that case the next pointer is no longer valid.

RCU should be used where needed, in fast path only, to find one entry, not to find _all_ entries.

For example, we cannot use it for UDP multicast delivery for the same reasons : If we find a deleted or moved socket, we must restart the loop and forget all accumulated sockets.

If netlink dumps each entry in the final destination container, then we cannot restart loop, and cannot use RCU for chain lookup.

#4 lore raw

PATCH 1/1 netfilter: fix race in conntrack between dump_table and destroy

The netlink interface to dump the connection tracking table has a race when entries are deleted at the same time. A customer reported a crash and the backtrace showed thatctnetlink_dump_table was running while a conntrack entry wasbeing destroyed.
(see https://bugzilla.vyatta.com/show_bug.cgi?id=6402).

According to RCU documentation, when using hlist_nulls the reader must handle the case of seeing a deleted entry and not proceed further down the linked list. The old code would continue which caused the scan to walk into the free list.

This patch uses locking (rather than RCU) for this operation which is guaranteed safe, and no longer requires getting reference while doing dump operation.

net/netfilter/nf_conntrack_netlink.c +5 -9
@@ -642,25 +642,23 @@ ctnetlink_dump_table(struct sk_buff *skb
 	struct nfgenmsg *nfmsg = nlmsg_data(cb->nlh);
 	u_int8_t l3proto = nfmsg->nfgen_family;
 
-	rcu_read_lock();
+	spin_lock_bh(&nf_conntrack_lock);
 	last = (struct nf_conn *)cb->args[1];
 	for (; cb->args[0] < net->ct.htable_size; cb->args[0]++) {
 restart:
-		hlist_nulls_for_each_entry_rcu(h, n, &net->ct.hash[cb->args[0]],
+		hlist_nulls_for_each_entry(h, n, &net->ct.hash[cb->args[0]],
 					 hnnode) {
 			if (NF_CT_DIRECTION(h) != IP_CT_DIR_ORIGINAL)
 				continue;
 			ct = nf_ct_tuplehash_to_ctrack(h);
-			if (!atomic_inc_not_zero(&ct->ct_general.use))
-				continue;
 			/* Dump entries of a given L3 protocol number.
 			 * If it is not specified, ie. l3proto == 0,
 			 * then dump everything. */
 			if (l3proto && nf_ct_l3num(ct) != l3proto)
-				goto releasect;
+				continue;
 			if (cb->args[1]) {
 				if (ct != last)
-					goto releasect;
+					continue;
 				cb->args[1] = 0;
 			}
 			if (ctnetlink_fill_info(skb, NETLINK_CB(cb->skb).pid,
@@ -678,8 +676,6 @@ restart:
 				if (acct)
 					memset(acct, 0, sizeof(struct nf_conn_counter[IP_CT_DIR_MAX]));
 			}
-releasect:
-		nf_ct_put(ct);
 		}
 		if (cb->args[1]) {
 			cb->args[1] = 0;
@@ -687,7 +683,7 @@ releasect:
 		}
 	}
 out:
-	rcu_read_unlock();
+	spin_unlock_bh(&nf_conntrack_lock);
 	if (last)
 		nf_ct_put(last);
 
Acked-by on v1 1/1
#5 lore raw

Re: [PATCH] netfilter: fix race in conntrack between dump_table and destroy

Quoted 11 lines · Stephen Hemminger

The netlink interface to dump the connection tracking table has a race when entries are deleted at the same time. A customer reported a crash and the backtrace showed thatctnetlink_dump_table was running while a conntrack entry wasbeing destroyed.
(see https://bugzilla.vyatta.com/show_bug.cgi?id=6402).

According to RCU documentation, when using hlist_nulls the reader must handle the case of seeing a deleted entry and not proceed further down the linked list. The old code would continue which caused the scan to walk into the free list.

Quoted 5 lines

This patch uses locking (rather than RCU) for this operation which is guaranteed safe, and no longer requires getting reference while doing dump operation.

--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw

Re: [PATCH] netfilter: fix race in conntrack between dump_table and destroy

In reply to Stephen Hemminger
Quoted 11 lines · On Fri, 26 Nov 2010 13:51:01 -0800 Stephen Hemminger <[email protected]>…

The netlink interface to dump the connection tracking table has a race when entries are deleted at the same time. A customer reported a crash and the backtrace showed thatctnetlink_dump_table was running while a conntrack entry wasbeing destroyed.
(see https://bugzilla.vyatta.com/show_bug.cgi?id=6402).

According to RCU documentation, when using hlist_nulls the reader must handle the case of seeing a deleted entry and not proceed further down the linked list. The old code would continue which caused the scan to walk into the free list.

Quoted 5 lines

This patch uses locking (rather than RCU) for this operation which is guaranteed safe, and no longer requires getting reference while doing dump operation.

This should go in net-2.6 and stable for 2.6.32, 2.6.35, and 2.6.36

#7 lore raw

Re: [PATCH] netfilter: fix race in conntrack between dump_table and destroy

In reply to Stephen Hemminger
Quoted 6 lines · Stephen Hemminger

The netlink interface to dump the connection tracking table has a race when entries are deleted at the same time. A customer reported a crash and the backtrace showed thatctnetlink_dump_table was running while a conntrack entry wasbeing destroyed.
(see https://bugzilla.vyatta.com/show_bug.cgi?id=6402).

Quoted 8 lines

According to RCU documentation, when using hlist_nulls the reader must handle the case of seeing a deleted entry and not proceed further down the linked list. The old code would continue which caused the scan to walk into the free list.

This patch uses locking (rather than RCU) for this operation which is guaranteed safe, and no longer requires getting reference while doing dump operation.

I have put this in my tree:

http://1984.lsi.us.es/git/?p=net-2.6/.git;a=summary

I'll pass it to David for -stable inclusion.