Skip to contentSkip to conversation

Revert "netfilter: unlock xt_table earlier in __do_replace"

PATCH In review Sean Tranchetti 3 participants 4 messages Jan 23, 2020 – Feb 3, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Revert "netfilter: unlock xt_table earlier in __do_replace"

A recently reported crash in the x_tables framework seems to stem from a potential race condition between adding rules to a table and having a packet traversing the table at the same time.

In the crash, the jumpstack being used by the table traversal was freed by the table replace code. After performing some bisection, it seems that commit f31e5f1a891f ("netfilter: unlock xt_table earlier in __do_replace") exposed this race condition by unlocking the table before the get_old_counters() routine was called to perform the synchronization.

Call Stack:

	Unable to handle kernel paging request at virtual address
	006b6b6b6b6b6bc5

	pc : ipt_do_table+0x3b8/0x660
	lr : ipt_do_table+0x31c/0x660
	Call trace:
	ipt_do_table+0x3b8/0x660
	iptable_mangle_hook+0x58/0xf8
	nf_hook_slow+0x48/0xd8
	__ip_local_out+0xf4/0x138
	__ip_queue_xmit+0x348/0x3a0
	ip_queue_xmit+0x10/0x18
---
 net/ipv4/netfilter/arp_tables.c | 3 +--
 net/ipv4/netfilter/ip_tables.c  | 3 +--
 net/ipv6/netfilter/ip6_tables.c | 3 +--
 3 files changed, 3 insertions(+), 6 deletions(-)
net/ipv4/netfilter/arp_tables.c +1 -2
index f1f78a7..85cb189 100644
@@ -921,8 +921,6 @@ static int __do_replace(struct net *net, const char *name,
 	    (newinfo->number <= oldinfo->initial_entries))
 		module_put(t->me);
 
-	xt_table_unlock(t);
-
 	get_old_counters(oldinfo, counters);
 
 	/* Decrease module usage counts and free resource */
@@ -937,6 +935,7 @@ static int __do_replace(struct net *net, const char *name,
 		net_warn_ratelimited("arptables: counters copy to user failed while replacing table\n");
 	}
 	vfree(counters);
+	xt_table_unlock(t);
 	return ret;
 
  put_module:
net/ipv4/netfilter/ip_tables.c +1 -2
index 10b91eb..9f98bc5 100644
@@ -1076,8 +1076,6 @@ static int get_info(struct net *net, void __user *user,
 	    (newinfo->number <= oldinfo->initial_entries))
 		module_put(t->me);
 
-	xt_table_unlock(t);
-
 	get_old_counters(oldinfo, counters);
 
 	/* Decrease module usage counts and free resource */
@@ -1091,6 +1089,7 @@ static int get_info(struct net *net, void __user *user,
 		net_warn_ratelimited("iptables: counters copy to user failed while replacing table\n");
 	}
 	vfree(counters);
+	xt_table_unlock(t);
 	return ret;
 
  put_module:
net/ipv6/netfilter/ip6_tables.c +1 -2
index c973ace..f2637bfb 100644
@@ -1093,8 +1093,6 @@ static int get_info(struct net *net, void __user *user,
 	    (newinfo->number <= oldinfo->initial_entries))
 		module_put(t->me);
 
-	xt_table_unlock(t);
-
 	get_old_counters(oldinfo, counters);
 
 	/* Decrease module usage counts and free resource */
@@ -1108,6 +1106,7 @@ static int get_info(struct net *net, void __user *user,
 		net_warn_ratelimited("ip6tables: counters copy to user failed while replacing table\n");
 	}
 	vfree(counters);
+	xt_table_unlock(t);
 	return ret;
 
  put_module:
Signature · 1 line
1.9.1
maintainer
#1 lore raw

Sean Tranchetti <[email protected]> wrote:

[ CC Xin Long ]

Quoted 9 lines

A recently reported crash in the x_tables framework seems to stem from a potential race condition between adding rules to a table and having a packet traversing the table at the same time.

In the crash, the jumpstack being used by the table traversal was freed by the table replace code. After performing some bisection, it seems that commit f31e5f1a891f ("netfilter: unlock xt_table earlier in __do_replace") exposed this race condition by unlocking the table before the get_old_counters() routine was called to perform the synchronization.

But the packet path doesn't grab the table mutex.

Quoted 26 lines

Call Stack:

	Unable to handle kernel paging request at virtual address
	006b6b6b6b6b6bc5

	pc : ipt_do_table+0x3b8/0x660
	lr : ipt_do_table+0x31c/0x660
	Call trace:
	ipt_do_table+0x3b8/0x660
	iptable_mangle_hook+0x58/0xf8
	nf_hook_slow+0x48/0xd8
	__ip_local_out+0xf4/0x138
	__ip_queue_xmit+0x348/0x3a0
	ip_queue_xmit+0x10/0x18

---

net/ipv4/netfilter/arp_tables.c +0 -2
@@ -921,8 +921,6 @@ static int __do_replace(struct net *net, const char *name,
 	    (newinfo->number <= oldinfo->initial_entries))
 		module_put(t->me);
 
-	xt_table_unlock(t);
-
 	get_old_counters(oldinfo, counters);
 
 	/* Decrease module usage counts and free resource */

net/ipv4/netfilter/arp_tables.c static int __do_replace(struct net *net, const char *name, · line 938

Quoted 4 lines
net/ipv4/netfilter/arp_tables.c +1 -0
@@ -937,6 +935,7 @@ static int __do_replace(struct net *net, const char *name,
 		net_warn_ratelimited("arptables: counters copy to user failed while replacing table\n");
 	}
 	vfree(counters);
+	xt_table_unlock(t);

I don't see how this changes anything wrt. packet path.
This disallows another instance of iptables(-restore) to come in before the counters have been copied/freed and the destructors have run.

But as those have nothing to do with the jumpstack I don't see how this helps.

#2 lore raw
Quoted 7 lines · Florian Westphal

I don't see how this changes anything wrt. packet path.
This disallows another instance of iptables(-restore) to come in before the counters have been copied/freed and the destructors have run.

But as those have nothing to do with the jumpstack I don't see how this helps.

Based on on the stack of the iptables-restore task that freed the jumpstack being accessed in the ipt_do_table() routine, we end up in __do_replace()

      0xFFFFFF9239243AE0, ->kvfree
      0xFFFFFF923A1969EC, ->xt_free_table_info+0x50
      0xFFFFFF923A2100E0, ->__do_replace+0x200

Prior to the original patch, this xt_free_table_info was under lock, so it seems that having this call under lock guarantees that the new table->private entry that contains the jumpstack is seen across all CPUs.

Quoted 3 lines

But the packet path doesn't grab the table mutex.

Good point. Perhaps the reason that moving this lock helps is because it prevents multiple writers from stepping on one another in such a way that the private entry is left in a bad state. Or this whole thing is a red herring and the problem is actually that xt_replace_table() is able to return prematurely and not all CPUs are finished with the old jumpstack by the time the old table info is freed.

#3 lore raw
In reply to Florian Westphal
Quoted 9 lines · Florian Westphal

Sean Tranchetti <[email protected]> wrote:

[ CC Xin Long ]

Quoted 4 lines

A recently reported crash in the x_tables framework seems to stem from a potential race condition between adding rules to a table and having a packet traversing the table at the same time.

Quoted 21 lines
Quoted 5 lines

In the crash, the jumpstack being used by the table traversal was freed by the table replace code. After performing some bisection, it seems that commit f31e5f1a891f ("netfilter: unlock xt_table earlier in __do_replace") exposed this race condition by unlocking the table before the get_old_counters() routine was called to perform the synchronization.

But the packet path doesn't grab the table mutex.

Quoted 13 lines

Call Stack:

      Unable to handle kernel paging request at virtual address
      006b6b6b6b6b6bc5

      pc : ipt_do_table+0x3b8/0x660
      lr : ipt_do_table+0x31c/0x660
      Call trace:
      ipt_do_table+0x3b8/0x660
      iptable_mangle_hook+0x58/0xf8
      nf_hook_slow+0x48/0xd8
      __ip_local_out+0xf4/0x138
      __ip_queue_xmit+0x348/0x3a0
      ip_queue_xmit+0x10/0x18

I don't see how this happens either.

Hi Sean,
do you have a script to reproduce this issue?

Thanks.

Quoted 24 lines
Quoted 13 lines

---

net/ipv4/netfilter/arp_tables.c +0 -2
@@ -921,8 +921,6 @@ static int __do_replace(struct net *net, const char *name,
          (newinfo->number <= oldinfo->initial_entries))
              module_put(t->me);
 
-     xt_table_unlock(t);
-
      get_old_counters(oldinfo, counters);
 
      /* Decrease module usage counts and free resource */

net/ipv4/netfilter/arp_tables.c static int __do_replace(struct net *net, const char *name, · line 938

Quoted 4 lines
net/ipv4/netfilter/arp_tables.c +1 -0
@@ -937,6 +935,7 @@ static int __do_replace(struct net *net, const char *name,
              net_warn_ratelimited("arptables: counters copy to user failed while replacing table\n");
      }
      vfree(counters);
+     xt_table_unlock(t);

I don't see how this changes anything wrt. packet path.
This disallows another instance of iptables(-restore) to come in before the counters have been copied/freed and the destructors have run.

But as those have nothing to do with the jumpstack I don't see how this helps.