Revert "netfilter: unlock xt_table earlier in __do_replace"
Subsystems (2)
PATCH 1/1 Revert "netfilter: unlock xt_table earlier in __do_replace"
A recently reported crash in the x_tables framework seems to stem from a potential race condition between adding rules to a table and having a packet traversing the table at the same time.
In the crash, the jumpstack being used by the table traversal was freed by the table replace code. After performing some bisection, it seems that commit f31e5f1a891f ("netfilter: unlock xt_table earlier in __do_replace") exposed this race condition by unlocking the table before the get_old_counters() routine was called to perform the synchronization.
Call Stack:
Unable to handle kernel paging request at virtual address 006b6b6b6b6b6bc5 pc : ipt_do_table+0x3b8/0x660 lr : ipt_do_table+0x31c/0x660 Call trace: ipt_do_table+0x3b8/0x660 iptable_mangle_hook+0x58/0xf8 nf_hook_slow+0x48/0xd8 __ip_local_out+0xf4/0x138 __ip_queue_xmit+0x348/0x3a0 ip_queue_xmit+0x10/0x18
--- net/ipv4/netfilter/arp_tables.c | 3 +-- net/ipv4/netfilter/ip_tables.c | 3 +-- net/ipv6/netfilter/ip6_tables.c | 3 +-- 3 files changed, 3 insertions(+), 6 deletions(-)
net/ipv4/netfilter/arp_tables.c +1 -2
@@ -921,8 +921,6 @@ static int __do_replace(struct net *net, const char *name, (newinfo->number <= oldinfo->initial_entries))module_put (t->me); -xt_table_unlock (t); -get_old_counters (oldinfo, counters);/* Decrease module usage counts and free resource */
@@ -937,6 +935,7 @@ static int __do_replace(struct net *net, const char *name,net_warn_ratelimited ("arptables: counters copy to user failed while replacing table\n" ); }vfree (counters); +xt_table_unlock (t);return ret; put_module:
net/ipv4/netfilter/ip_tables.c +1 -2
@@ -1076,8 +1076,6 @@ static int get_info(struct net *net, void __user *user, (newinfo->number <= oldinfo->initial_entries))module_put (t->me); -xt_table_unlock (t); -get_old_counters (oldinfo, counters);/* Decrease module usage counts and free resource */
@@ -1091,6 +1089,7 @@ static int get_info(struct net *net, void __user *user,net_warn_ratelimited ("iptables: counters copy to user failed while replacing table\n" ); }vfree (counters); +xt_table_unlock (t);return ret; put_module:
net/ipv6/netfilter/ip6_tables.c +1 -2
@@ -1093,8 +1093,6 @@ static int get_info(struct net *net, void __user *user, (newinfo->number <= oldinfo->initial_entries))module_put (t->me); -xt_table_unlock (t); -get_old_counters (oldinfo, counters);/* Decrease module usage counts and free resource */
@@ -1108,6 +1106,7 @@ static int get_info(struct net *net, void __user *user,net_warn_ratelimited ("ip6tables: counters copy to user failed while replacing table\n" ); }vfree (counters); +xt_table_unlock (t);return ret; put_module:
Signature · 1 line
1.9.1
Florian Westphal
maintainerSean Tranchetti <[email protected]> wrote:
[ CC Xin Long ]
Quoted 9 lines
A recently reported crash in the
x_tablesframework seems to stem from a potential race condition between adding rules to a table and having a packet traversing the table at the same time.In the crash, the jumpstack being used by the table traversal was freed by the table replace code. After performing some bisection, it seems that commit
f31e5f1a891f("netfilter: unlockxt_tableearlier in__do_replace") exposed this race condition by unlocking the table before theget_old_counters()routine was called to perform the synchronization.
But the packet path doesn't grab the table mutex.
Quoted 26 lines
Call Stack:
Unable to handle kernel paging request at virtual address 006b6b6b6b6b6bc5 pc : ipt_do_table+0x3b8/0x660 lr : ipt_do_table+0x31c/0x660 Call trace: ipt_do_table+0x3b8/0x660 iptable_mangle_hook+0x58/0xf8 nf_hook_slow+0x48/0xd8 __ip_local_out+0xf4/0x138 __ip_queue_xmit+0x348/0x3a0 ip_queue_xmit+0x10/0x18Signed-off-by Sean Tranchetti <[email protected]>---
net/ipv4/netfilter/arp_tables.c +0
-2@@ -921,8 +921,6 @@ static int __do_replace(struct net *net, const char *name, (newinfo->number <= oldinfo->initial_entries))module_put (t->me); -xt_table_unlock (t); -get_old_counters (oldinfo, counters);/* Decrease module usage counts and free resource */
net/ipv4/netfilter/arp_tables.c static int __do_replace(struct net *net, const char *name, · line 938
Quoted 4 lines
net/ipv4/netfilter/arp_tables.c +1
-0@@ -937,6 +935,7 @@ static int __do_replace(struct net *net, const char *name,net_warn_ratelimited ("arptables: counters copy to user failed while replacing table\n" ); }vfree (counters); +xt_table_unlock (t);
I don't see how this changes anything wrt. packet path.
This disallows another instance of iptables(-restore) to come in before the counters have been copied/freed and the destructors have run.
But as those have nothing to do with the jumpstack I don't see how this helps.
Quoted 7 lines · Florian Westphal
I don't see how this changes anything wrt. packet path.
This disallows another instance of iptables(-restore) to come in before the counters have been copied/freed and the destructors have run.But as those have nothing to do with the jumpstack I don't see how this helps.
Based on on the stack of the iptables-restore task that freed the jumpstack being accessed in the ipt_do_table() routine, we end up in __do_replace()
0xFFFFFF9239243AE0, ->kvfree
0xFFFFFF923A1969EC, ->xt_free_table_info+0x50
0xFFFFFF923A2100E0, ->__do_replace+0x200Prior to the original patch, this xt_free_table_info was under lock, so it seems that having this call under lock guarantees that the new table->private entry that contains the jumpstack is seen across all CPUs.
Quoted 3 lines
But the packet path doesn't grab the table mutex.
Good point. Perhaps the reason that moving this lock helps is because it prevents multiple writers from stepping on one another in such a way that the private entry is left in a bad state. Or this whole thing is a red herring and the problem is actually that xt_replace_table() is able to return prematurely and not all CPUs are finished with the old jumpstack by the time the old table info is freed.
Quoted 9 lines · Florian Westphal
Sean Tranchetti <[email protected]> wrote:
[ CC Xin Long ]
Quoted 4 lines
A recently reported crash in the
x_tablesframework seems to stem from a potential race condition between adding rules to a table and having a packet traversing the table at the same time.
Quoted 21 lines
Quoted 5 lines
In the crash, the jumpstack being used by the table traversal was freed by the table replace code. After performing some bisection, it seems that commit
f31e5f1a891f("netfilter: unlockxt_tableearlier in__do_replace") exposed this race condition by unlocking the table before theget_old_counters()routine was called to perform the synchronization.But the packet path doesn't grab the table mutex.
Quoted 13 lines
Call Stack:
Unable to handle kernel paging request at virtual address 006b6b6b6b6b6bc5 pc : ipt_do_table+0x3b8/0x660 lr : ipt_do_table+0x31c/0x660 Call trace: ipt_do_table+0x3b8/0x660 iptable_mangle_hook+0x58/0xf8 nf_hook_slow+0x48/0xd8 __ip_local_out+0xf4/0x138 __ip_queue_xmit+0x348/0x3a0 ip_queue_xmit+0x10/0x18
I don't see how this happens either.
Hi Sean,
do you have a script to reproduce this issue?
Thanks.
Quoted 24 lines
Quoted 13 lines
Signed-off-by Sean Tranchetti <[email protected]>---
net/ipv4/netfilter/arp_tables.c +0
-2@@ -921,8 +921,6 @@ static int __do_replace(struct net *net, const char *name, (newinfo->number <= oldinfo->initial_entries))module_put (t->me); -xt_table_unlock (t); -get_old_counters (oldinfo, counters);/* Decrease module usage counts and free resource */ net/ipv4/netfilter/arp_tables.c static int __do_replace(struct net *net, const char *name, · line 938
Quoted 4 lines
net/ipv4/netfilter/arp_tables.c +1
-0@@ -937,6 +935,7 @@ static int __do_replace(struct net *net, const char *name,net_warn_ratelimited ("arptables: counters copy to user failed while replacing table\n" ); }vfree (counters); +xt_table_unlock (t);I don't see how this changes anything wrt. packet path.
This disallows another instance of iptables(-restore) to come in before the counters have been copied/freed and the destructors have run.But as those have nothing to do with the jumpstack I don't see how this helps.