WARNING: at include/net/sock.h:417 udp_lib_unhash
Subsystems (8)
I see the following trace during netperf stress mixed UDP/TCP IPv4/6 traffic. This is on recent pulls from net-2.6 and net-next.
[45197.989163] ------------[ cut here ]------------ [45197.994309] WARNING: at include/net/sock.h:417 udp_lib_unhash+0x81/0xab() [45197.994311] Hardware name: X7DA8 [45197.994314] Modules linked in: e1000 [last unloaded: e1000] [45197.994326] Pid: 7110, comm: netserver Tainted: G W 2.6.31-rc1-net-next-e1000-06250902 #8 [45197.994331] Call Trace: [45197.994336] [<ffffffff8135e0dc>] ? udp_lib_unhash+0x81/0xab [45197.994344] [<ffffffff8103cac9>] warn_slowpath_common+0x77/0x8f [45197.994349] [<ffffffff8103caf0>] warn_slowpath_null+0xf/0x11 [45197.994352] [<ffffffff8135e0dc>] udp_lib_unhash+0x81/0xab [45197.994357] [<ffffffff81301acb>] sk_common_release+0x2f/0xb4 [45197.994364] [<ffffffff813a0256>] udp_lib_close+0x9/0xb [45197.994369] [<ffffffff81364259>] inet_release+0x58/0x5f [45197.994374] [<ffffffff8138c8bd>] inet6_release+0x30/0x35 [45197.994383] [<ffffffff812ff273>] sock_release+0x1a/0x6c [45197.994386] [<ffffffff812ff763>] sock_close+0x22/0x26 [45197.994392] [<ffffffff810c69a0>] __fput+0xf0/0x18c [45197.994395] [<ffffffff810c6d00>] fput+0x15/0x19 [45197.994399] [<ffffffff810c3c3e>] filp_close+0x5c/0x67 [45197.994404] [<ffffffff810c3cc4>] sys_close+0x7b/0xb6 [45197.994412] [<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b [45197.994418] ---[ end trace 5acab6fc0afdaaa3 ]---
Emil
Quoted 25 lines · Emil S
I see the following trace during netperf stress mixed UDP/TCP IPv4/6 traffic. This is on recent pulls from net-2.6 and net-next.
[45197.989163] ------------[ cut here ]------------ [45197.994309] WARNING: at include/net/sock.h:417 udp_lib_unhash+0x81/0xab() [45197.994311] Hardware name: X7DA8 [45197.994314] Modules linked in: e1000 [last unloaded: e1000] [45197.994326] Pid: 7110, comm: netserver Tainted: G W 2.6.31-rc1-net-next-e1000-06250902 #8 [45197.994331] Call Trace: [45197.994336] [<ffffffff8135e0dc>] ? udp_lib_unhash+0x81/0xab [45197.994344] [<ffffffff8103cac9>] warn_slowpath_common+0x77/0x8f [45197.994349] [<ffffffff8103caf0>] warn_slowpath_null+0xf/0x11 [45197.994352] [<ffffffff8135e0dc>] udp_lib_unhash+0x81/0xab [45197.994357] [<ffffffff81301acb>] sk_common_release+0x2f/0xb4 [45197.994364] [<ffffffff813a0256>] udp_lib_close+0x9/0xb [45197.994369] [<ffffffff81364259>] inet_release+0x58/0x5f [45197.994374] [<ffffffff8138c8bd>] inet6_release+0x30/0x35 [45197.994383] [<ffffffff812ff273>] sock_release+0x1a/0x6c [45197.994386] [<ffffffff812ff763>] sock_close+0x22/0x26 [45197.994392] [<ffffffff810c69a0>] __fput+0xf0/0x18c [45197.994395] [<ffffffff810c6d00>] fput+0x15/0x19 [45197.994399] [<ffffffff810c3c3e>] filp_close+0x5c/0x67 [45197.994404] [<ffffffff810c3cc4>] sys_close+0x7b/0xb6 [45197.994412] [<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b [45197.994418] ---[ end trace 5acab6fc0afdaaa3 ]---
Quoted 1 line
Emil--
Thanks for this report Emil.
I could not find a recent change in this area in last kernels. If struct sk is hashed (sk_hashed() true), then sk_refcnt was incremented in sk_nulls_add_node_rcu(), thus its value should be >= 2.
Maybe we have a missing memory barrier somewhere or a list corruption.
1) Could you try CONFIG_DEBUG_LIST=y ?
2) Could you give model of cpu, since it reminds me the ongoing discussion raised by Jiri Olsa.
CPU1 does an atomic_inc(&sk->sk_refcnt) : refcnt changes from 1 to 2 then CPU2 does an atomic_read(&sk->sk_refcnt) and reads 1 instead of 2
David, maybe this test is not safe and if we really want to do a check we need to use a stronger atomic function.
If you can reproduce this problem easily could you try following patch ?
Thank you
include/net/sock.h +6 -5
@@ -393,8 +393,9 @@ static __inline__ int sk_del_node_init(struct sock *sk)if (rc) {/* paranoid for a while -acme */ -WARN_ON (atomic_read (&sk->sk_refcnt) ==1 ); -__sock_put (sk); +int res =atomic_dec_return (&sk->sk_refcnt); + +WARN_ON (res <=0 ); }return rc; }
@@ -413,9 +414,9 @@ static __inline__ int sk_nulls_del_node_init_rcu(struct sock *sk)int rc =__sk_nulls_del_node_init_rcu (sk);if (rc) { -/* paranoid for a while -acme */ -WARN_ON (atomic_read (&sk->sk_refcnt) ==1 ); -__sock_put (sk); +int res =atomic_dec_return (&sk->sk_refcnt); + +WARN_ON (res <=0 ); }return rc; }
Quoted 30 lines · Eric Dumazet
Quoted 25 lines · Emil S
I see the following trace during netperf stress mixed UDP/TCP IPv4/6 traffic. This is on recent pulls from net-2.6 and net-next.
[45197.989163] ------------[ cut here ]------------ [45197.994309] WARNING: at include/net/sock.h:417 udp_lib_unhash+0x81/0xab() [45197.994311] Hardware name: X7DA8 [45197.994314] Modules linked in: e1000 [last unloaded: e1000] [45197.994326] Pid: 7110, comm: netserver Tainted: G W 2.6.31-rc1-net-next-e1000-06250902 #8 [45197.994331] Call Trace: [45197.994336] [<ffffffff8135e0dc>] ? udp_lib_unhash+0x81/0xab [45197.994344] [<ffffffff8103cac9>] warn_slowpath_common+0x77/0x8f [45197.994349] [<ffffffff8103caf0>] warn_slowpath_null+0xf/0x11 [45197.994352] [<ffffffff8135e0dc>] udp_lib_unhash+0x81/0xab [45197.994357] [<ffffffff81301acb>] sk_common_release+0x2f/0xb4 [45197.994364] [<ffffffff813a0256>] udp_lib_close+0x9/0xb [45197.994369] [<ffffffff81364259>] inet_release+0x58/0x5f [45197.994374] [<ffffffff8138c8bd>] inet6_release+0x30/0x35 [45197.994383] [<ffffffff812ff273>] sock_release+0x1a/0x6c [45197.994386] [<ffffffff812ff763>] sock_close+0x22/0x26 [45197.994392] [<ffffffff810c69a0>] __fput+0xf0/0x18c [45197.994395] [<ffffffff810c6d00>] fput+0x15/0x19 [45197.994399] [<ffffffff810c3c3e>] filp_close+0x5c/0x67 [45197.994404] [<ffffffff810c3cc4>] sys_close+0x7b/0xb6 [45197.994412] [<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b [45197.994418] ---[ end trace 5acab6fc0afdaaa3 ]---Quoted 1 line
Emil--
Thanks for this report Emil.
Quoted 7 lines
I could not find a recent change in this area in last kernels. If struct sk is hashed (
sk_hashed()true), thensk_refcntwas incremented insk_nulls_add_node_rcu(), thus its value should be >= 2.Maybe we have a missing memory barrier somewhere or a list corruption.
1) Could you try
CONFIG_DEBUG_LIST=y?
I am running a test with this option now. Sorry for the delayed response, I was out last week.
Quoted 1 line
2) Could you give model of cpu, since it reminds me the ongoing discussion raised by Jiri Olsa.
processor : 0vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5450 @ 3.00GHz
stepping : 6
cpu MHz : 2999.790
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yesfpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority
bogomips : 5999.58
clflush size : 64
2 quad core Xeons, I only included the output from the first to reduce clutter.
Quoted 7 lines
CPU1 does an
atomic_inc(&sk->sk_refcnt): refcnt changes from 1 to 2 then CPU2 does anatomic_read(&sk->sk_refcnt)and reads 1 instead of 2David, maybe this test is not safe and if we really want to do a check we need to use a stronger atomic function.
If you can reproduce this problem easily could you try following patch ?
It varies from few minutes to hours, but it does reproduce consistently in my tests so far. I will try your patch once I manage to get a trace with CONFIG_DEBUG_LIST
Thanks,
Emil
Quoted 73 lines · Emil S Tantilov
Quoted 29 lines · Eric Dumazet
Quoted 25 lines · Emil S
I see the following trace during netperf stress mixed UDP/TCP IPv4/6 traffic. This is on recent pulls from net-2.6 and net-next.
[45197.989163] ------------[ cut here ]------------ [45197.994309] WARNING: at include/net/sock.h:417 udp_lib_unhash+0x81/0xab() [45197.994311] Hardware name: X7DA8 [45197.994314] Modules linked in: e1000 [last unloaded: e1000] [45197.994326] Pid: 7110, comm: netserver Tainted: G W 2.6.31-rc1-net-next-e1000-06250902 #8 [45197.994331] Call Trace: [45197.994336] [<ffffffff8135e0dc>] ? udp_lib_unhash+0x81/0xab [45197.994344] [<ffffffff8103cac9>] warn_slowpath_common+0x77/0x8f [45197.994349] [<ffffffff8103caf0>] warn_slowpath_null+0xf/0x11 [45197.994352] [<ffffffff8135e0dc>] udp_lib_unhash+0x81/0xab [45197.994357] [<ffffffff81301acb>] sk_common_release+0x2f/0xb4 [45197.994364] [<ffffffff813a0256>] udp_lib_close+0x9/0xb [45197.994369] [<ffffffff81364259>] inet_release+0x58/0x5f [45197.994374] [<ffffffff8138c8bd>] inet6_release+0x30/0x35 [45197.994383] [<ffffffff812ff273>] sock_release+0x1a/0x6c [45197.994386] [<ffffffff812ff763>] sock_close+0x22/0x26 [45197.994392] [<ffffffff810c69a0>] __fput+0xf0/0x18c [45197.994395] [<ffffffff810c6d00>] fput+0x15/0x19 [45197.994399] [<ffffffff810c3c3e>] filp_close+0x5c/0x67 [45197.994404] [<ffffffff810c3cc4>] sys_close+0x7b/0xb6 [45197.994412] [<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b [45197.994418] ---[ end trace 5acab6fc0afdaaa3 ]---Quoted 1 line
Emil--
Thanks for this report Emil.
Quoted 7 lines
I could not find a recent change in this area in last kernels. If struct sk is hashed (
sk_hashed()true), thensk_refcntwas incremented insk_nulls_add_node_rcu(), thus its value should be >= 2.Maybe we have a missing memory barrier somewhere or a list corruption.
1) Could you try
CONFIG_DEBUG_LIST=y?I am running a test with this option now. Sorry for the delayed response, I was out last week.
Quoted 1 line
2) Could you give model of cpu, since it reminds me the ongoing discussion raised by Jiri Olsa.
processor : 0
vendor_id: GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5450 @ 3.00GHz
stepping : 6
cpu MHz : 2999.790
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yesfpu_exception: yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lmconstant_tscarch_perfmonpebs btsrep_goodpni dtes64 monitords_cplvmx est tm2 ssse3 cx16 xtpr pdcm dcasse4_1lahf_lmtpr_shadowvnmi flexpriority
bogomips : 5999.58
clflush size : 642 quad core Xeons, I only included the output from the first to reduce clutter.
Quoted 3 lines
CPU1 does an
atomic_inc(&sk->sk_refcnt): refcnt changes from 1 to 2 then CPU2 does anatomic_read(&sk->sk_refcnt)and reads 1 instead of 2
Quoted 9 lines
Quoted 4 lines
David, maybe this test is not safe and if we really want to do a check we need to use a stronger atomic function.
If you can reproduce this problem easily could you try following patch ?
It varies from few minutes to hours, but it does reproduce consistently in my tests so far. I will try your patch once I manage to get a trace with
CONFIG_DEBUG_LIST
Eventually, could you also use following debug/quick&dirty patch ?
include/net/sock.h +2 -0
@@ -376,6 +376,7 @@ static __inline__ int __sk_del_node_init(struct sock *sk)static inline void sock_hold (struct sock *sk) { +WARN_ON (atomic_read (&sk->sk_refcnt) ==0 );atomic_inc (&sk->sk_refcnt); }
@@ -385,6 +386,7 @@ static inline void sock_hold(struct sock *sk)static inline void __sock_put (struct sock *sk) {atomic_dec (&sk->sk_refcnt); +WARN_ON (atomic_read (&sk->sk_refcnt) ==0 ); }static __inline__int sk_del_node_init (struct sock *sk)
Quoted 100 lines · Eric Dumazet
Quoted 72 lines · Emil S Tantilov
Quoted 29 lines · Eric Dumazet
Quoted 25 lines · Emil S
I see the following trace during netperf stress mixed UDP/TCP IPv4/6 traffic. This is on recent pulls from net-2.6 and net-next.
[45197.989163] ------------[ cut here ]------------ [45197.994309] WARNING: at include/net/sock.h:417 udp_lib_unhash+0x81/0xab() [45197.994311] Hardware name: X7DA8 [45197.994314] Modules linked in: e1000 [last unloaded: e1000] [45197.994326] Pid: 7110, comm: netserver Tainted: G W 2.6.31-rc1-net-next-e1000-06250902 #8 [45197.994331] Call Trace: [45197.994336] [<ffffffff8135e0dc>] ? udp_lib_unhash+0x81/0xab [45197.994344] [<ffffffff8103cac9>] warn_slowpath_common+0x77/0x8f [45197.994349] [<ffffffff8103caf0>] warn_slowpath_null+0xf/0x11 [45197.994352] [<ffffffff8135e0dc>] udp_lib_unhash+0x81/0xab [45197.994357] [<ffffffff81301acb>] sk_common_release+0x2f/0xb4 [45197.994364] [<ffffffff813a0256>] udp_lib_close+0x9/0xb [45197.994369] [<ffffffff81364259>] inet_release+0x58/0x5f [45197.994374] [<ffffffff8138c8bd>] inet6_release+0x30/0x35 [45197.994383] [<ffffffff812ff273>] sock_release+0x1a/0x6c [45197.994386] [<ffffffff812ff763>] sock_close+0x22/0x26 [45197.994392] [<ffffffff810c69a0>] __fput+0xf0/0x18c [45197.994395] [<ffffffff810c6d00>] fput+0x15/0x19 [45197.994399] [<ffffffff810c3c3e>] filp_close+0x5c/0x67 [45197.994404] [<ffffffff810c3cc4>] sys_close+0x7b/0xb6 [45197.994412] [<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b [45197.994418] ---[ end trace 5acab6fc0afdaaa3 ]---Quoted 1 line
Emil--
Thanks for this report Emil.
Quoted 7 lines
I could not find a recent change in this area in last kernels. If struct sk is hashed (
sk_hashed()true), thensk_refcntwas incremented insk_nulls_add_node_rcu(), thus its value should be >= 2.Maybe we have a missing memory barrier somewhere or a list corruption.
1) Could you try
CONFIG_DEBUG_LIST=y?I am running a test with this option now. Sorry for the delayed response, I was out last week.
Quoted 1 line
2) Could you give model of cpu, since it reminds me the ongoing discussion raised by Jiri Olsa.
processor : 0
vendor_id: GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5450 @ 3.00GHz
stepping : 6
cpu MHz : 2999.790
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yesfpu_exception: yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lmconstant_tscarch_perfmonpebs btsrep_goodpni dtes64 monitords_cplvmx est tm2 ssse3 cx16 xtpr pdcm dcasse4_1lahf_lmtpr_shadowvnmi flexpriority
bogomips : 5999.58
clflush size : 642 quad core Xeons, I only included the output from the first to reduce clutter.
Quoted 3 lines
CPU1 does an
atomic_inc(&sk->sk_refcnt): refcnt changes from 1 to 2 then CPU2 does anatomic_read(&sk->sk_refcnt)and reads 1 instead of 2Quoted 8 lines
Quoted 4 lines
David, maybe this test is not safe and if we really want to do a check we need to use a stronger atomic function.
If you can reproduce this problem easily could you try following patch ?
It varies from few minutes to hours, but it does reproduce consistently in my tests so far. I will try your patch once I manage to get a trace with
CONFIG_DEBUG_LISTEventually, could you also use following debug/quick&dirty patch ?
include/net/sock.h +1
-0@@ -376,6 +376,7 @@ static __inline__ int __sk_del_node_init(struct sock *sk)static inline void sock_hold (struct sock *sk) { +WARN_ON (atomic_read (&sk->sk_refcnt) ==0 );atomic_inc (&sk->sk_refcnt); }
include/net/sock.h static inline void sock_hold(struct sock *sk) · line 392
Quoted 7 lines
include/net/sock.h +1
-0@@ -385,6 +386,7 @@ static inline void sock_hold(struct sock *sk)static inline void __sock_put (struct sock *sk) {atomic_dec (&sk->sk_refcnt); +WARN_ON (atomic_read (&sk->sk_refcnt) ==0 ); }static __inline__int sk_del_node_init (struct sock *sk)
Hold on, I found the problem and will submit a patch (after testing it) in following hours
Thanks
PATCH 1/1 net: sk_alloc() should not blindly overwrite memory
Quoted 4 lines · Eric Dumazet
Hold on, I found the problem and will submit a patch (after testing it) in following hours
David, while looking at Emil case, I found following problem.
I am not sure this bug is responsible for the BUG hit by Emil, but it seems a candidate for stable as well...
Thanks
[PATCH] net: sk_alloc() should not blindly overwrite memory
Some sockets use SLAB_DESTROY_BY_RCU, and our RCU code rely that some fields should not be blindly overwritten, even with null.
These fields are sk->sk_refcnt and sk->sk_nulls_node.next
Current sk_alloc() implementation doesnt respect this hypothesis, calling kmem_alloc with __GFP_ZERO and setting sk_refcnt to 1 instead of atomically increment it.
--- include/net/sock.h | 24 +++++++++++------- net/core/sock.c | 41 ++++++++++++++++++++++++++++---- net/ipv4/inet_timewait_sock.c | 2 - 3 files changed, 52 insertions(+), 15 deletions(-)
include/net/sock.h +15 -9
@@ -103,15 +103,15 @@ struct net;/** * struct sock_common - minimal network layer representation of sockets +* @skc_node: main hash linkage for various protocol lookup tables +* @skc_nulls_node: main hash linkage for UDP/UDP-Lite protocol +* @skc_refcnt: reference count +* @skc_hash: hash value used with various protocol lookup tables * @skc_family: network address family * @skc_state: Connection state * @skc_reuse: %SO_REUSEADDR setting * @skc_bound_dev_if: bound device index if != 0 -* @skc_node: main hash linkage for various protocol lookup tables -* @skc_nulls_node: main hash linkage for UDP/UDP-Lite protocol * @skc_bind_node: bind hash linkage for various protocol lookup tables -* @skc_refcnt: reference count -* @skc_hash: hash value used with various protocol lookup tables * @skc_prot: protocol handlers inside a network family * @skc_net: reference to the network namespace of this socket *
@@ -119,17 +119,23 @@ struct net;* for struct sock and struct inet_timewait_sock. */ struct sock_common { -unsigned short skc_family; -volatile unsigned char skc_state; -unsigned char skc_reuse; -int skc_bound_dev_if; +/* +* WARNING : skc_node, skc_refcnt must be first elements of this struct +* (sk_prot_alloc() should not clear skc_node.next & skc_refcnt because +* of RCU lookups) +*/ union {struct hlist_node skc_node;struct hlist_nulls_node skc_nulls_node; }; -struct hlist_node skc_bind_node;atomic_t skc_refcnt; +unsigned int skc_hash; +unsigned short skc_family; +volatile unsigned char skc_state; +unsigned char skc_reuse; +int skc_bound_dev_if; +struct hlist_node skc_bind_node;struct proto *skc_prot;#ifdef CONFIG_NET_NS struct net *skc_net;
net/core/sock.c +36 -5
77 lines not on this page · raw message
net/ipv4/inet_timewait_sock.c +1 -1
@@ -139,7 +139,7 @@ struct inet_timewait_sock *inet_twsk_alloc(const struct sock *sk, const int stat tw->tw_transparent = inet->transparent; tw->tw_prot = sk->sk_prot_creator;twsk_net_set (tw,hold_net (sock_net (sk))); -atomic_set (&tw->tw_refcnt,1 ); +atomic_inc (&tw->tw_refcnt);inet_twsk_dead_node_init (tw);__module_get (tw->tw_prot->owner); }
RE: [PATCH] net: sk_alloc() should not blindly overwrite memory
Quoted 181 lines · Eric Dumazet
Quoted 5 lines · Eric Dumazet
Hold on, I found the problem and will submit a patch (after testing it) in following hours
David, while looking at Emil case, I found following problem.
I am not sure this bug is responsible for the BUG hit by Emil, but it seems a candidate for stable as well...
Thanks
[PATCH] net:
sk_alloc()should not blindly overwrite memorySome sockets use
SLAB_DESTROY_BY_RCU, and our RCU code rely that some fields should not be blindly overwritten, even with null.These fields are
sk->sk_refcntandsk->sk_nulls_node.nextCurrent
sk_alloc()implementation doesnt respect this hypothesis, callingkmem_allocwith__GFP_ZEROand settingsk_refcntto 1 instead of atomically increment it.Reported-by Emil S Tantilov <[email protected]>Signed-off-by Eric Dumazet <[email protected]>--- include/net/sock.h | 24 +++++++++++------- net/core/sock.c | 41 ++++++++++++++++++++++++++++---- net/ipv4/inet_timewait_sock.c | 2 - 3 files changed, 52 insertions(+), 15 deletions(-)include/net/sock.h +0
-0@@ -103,15 +103,15 @@ struct net;/** * struct sock_common - minimal network layer representation of sockets + * @skc_node: main hash linkage for various protocol lookup
tables + * @skc_nulls_node: main hash linkage for UDP/UDP-Lite
protocol + * @skc_refcnt: reference count
+ * @skc_hash: hash value used with various protocol lookup tables
* @skc_family: network address family
* @skc_state: Connection state
* @skc_reuse: %SO_REUSEADDR setting
* @skc_bound_dev_if: bound device index if != 0
- * @skc_node: main hash linkage for various protocol lookup tables
- * @skc_nulls_node: main hash linkage for UDP/UDP-Lite protocol
* @skc_bind_node: bind hash linkage for various protocol lookup
tables
- * @skc_refcnt: reference count
- * @skc_hash: hash value used with various protocol lookup tables
* @skc_prot: protocol handlers inside a network family
* @skc_net: reference to the network namespace of this socket
*include/net/sock.h +2
-4@@ -119,17 +119,23 @@ struct net;* for struct sock and struct inet_timewait_sock. */ struct sock_common { -unsigned short skc_family; -volatile unsigned char skc_state; -unsigned char skc_reuse; -int skc_bound_dev_if; +/* +* WARNING : skc_node, skc_refcnt must be first elements of this struct + * (
sk_prot_alloc()should not clear skc_node.next &skc_refcntbecause + * of RCU lookups)include/net/sock.h +7
-1+*/ union {struct hlist_node skc_node;struct hlist_nulls_node skc_nulls_node; }; -struct hlist_node skc_bind_node;atomic_t skc_refcnt; +unsigned int skc_hash; +unsigned short skc_family; +volatile unsigned char skc_state; +unsigned char skc_reuse; +int skc_bound_dev_if; +struct hlist_node skc_bind_node;struct proto *skc_prot;#ifdef CONFIG_NET_NS struct net *skc_net;net/core/sock.c +19
-2@@ -994,8 +994,25 @@ struct sock *sk_alloc(struct net *net, int family, gfp_t priority, {struct sock *sk; - sk =sk_prot_alloc (prot, priority | __GFP_ZERO, family); + sk =sk_prot_alloc (prot, priority, family);if (sk) { +/* +* caches using SLAB_DESTROY_BY_RCU should let sk_node +* and sk_refcnt untouched. +* +* Following makes sense only if sk first fields are +* in following order : sk_node, refcnt, sk_hash +*/ +BUILD_BUG_ON (offsetof(struct sock , sk_node) !=0 ); +BUILD_BUG_ON (offsetof(struct sock , sk_refcnt) != +sizeof (sk->sk_node)); +BUILD_BUG_ON (offsetof(struct sock , sk_hash) != +sizeof (sk->sk_node) +sizeof (sk->sk_refcnt)); +if (prot->slab_flags &SLAB_DESTROY_BY_RCU ) +memset (&sk->sk_hash,0 , + prot->obj_size - offsetof(struct sock , sk_hash)); +else +memset (sk,0 , prot->obj_size); sk->sk_family = family;/* * See comment in struct sock definition to understand @@ -1125,7 +1142,7 @@ struct sock *sk_clone(const struct sock *sk, const gfp_t priority) newsk->sk_err = 0; newsk->sk_priority =0 ; -atomic_set (&newsk->sk_refcnt,2 ); +atomic_add (2 , &newsk->sk_refcnt);/* net/core/sock.c +1
-1* Increment the counter in the same struct proto as the master @@ -1840,7 +1857,7 @@ void sock_init_data(struct socket *sock, struct sock *sk) sk->sk_stamp = ktime_set(-1L, 0); -atomic_set (&sk->sk_refcnt,1 ); +atomic_inc (&sk->sk_refcnt);atomic_set (&sk->sk_wmem_alloc,1 );atomic_set (&sk->sk_drops,0 );}
net/core/sock.c +2
-0@@ -2140,12 +2157,25 @@ static inline void release_proto_idx(struct proto *prot) }#endif +/* +* We need to initialize skc->skc_refcnt to 0, and
skc->skc_node.pprevto NULL + * but only on newly allocated objects (checksk_prot_alloc()) + */net/core/sock.c +12
-2+static void sk_init_once (void *arg) +{ +struct sock_common *skc = arg; + +atomic_set (&skc->skc_refcnt,0 ); + skc->skc_node.pprev =NULL ; +} +int proto_register (struct proto *prot,int alloc_slab) {if (alloc_slab) { prot->slab =kmem_cache_create (prot->name, prot->obj_size,0 ,SLAB_HWCACHE_ALIGN | prot->slab_flags, -NULL ); + prot->slab_flags &SLAB_DESTROY_BY_RCU ? + sk_init_once :NULL );if (prot->slab ==NULL ) { printk(KERN_CRIT "%s: Can't create sock SLAB cache!\n",@@ -2187,7 +2217,8 @@ int proto_register(struct proto *prot, int alloc_slab) 0,SLAB_HWCACHE_ALIGN | prot->slab_flags, - NULL); + prot->slab_flags &SLAB_DESTROY_BY_RCU ? + sk_init_once : NULL);if (prot->twsk_prot->twsk_slab ==NULL ) goto out_free_timewait_sock_slab_name; }net/ipv4/inet_timewait_sock.c +0
-0
net/ipv4/inet_timewait_sock.c struct inet_timewait_sock *inet_twsk_alloc(const · line 145
Quoted 8 lines
net/ipv4/inet_timewait_sock.c +1
-1@@ -139,7 +139,7 @@ struct inet_timewait_sock *inet_twsk_alloc(const struct sock *sk, const int stat tw->tw_transparent = inet->transparent; tw->tw_prot = sk->sk_prot_creator;twsk_net_set (tw,hold_net (sock_net (sk))); -atomic_set (&tw->tw_refcnt,1 ); +atomic_inc (&tw->tw_refcnt);inet_twsk_dead_node_init (tw);__module_get (tw->tw_prot->owner); }
Eric,
With this patch applied, I get panic on boot:
[ 5.334653] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 [ 5.344225] IP: [<ffffffff811ed364>] selinux_socket_post_create+0x78/0x95 [ 5.352263] PGD 0 [ 5.354952] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC [ 5.360599] last sysfs file: [ 5.364292] CPU 0 [ 5.366985] Modules linked in: [ 5.370836] Pid: 1, comm: swapper Not tainted 2.6.31-rc1-net-2.6-igb-ed-patch-07071123 #2 S5520HC [ 5.381465] RIP: 0010:[<ffffffff811ed364>] [<ffffffff811ed364>] selinux_socket_post_create+0x78/0x95 [ 5.392551] RSP: 0018:ffff8801ef0a5d60 EFLAGS: 00010286 [ 5.398871] RAX: 0000000000000001 RBX: ffff88036e5ace80 RCX: 0000000000000006 [ 5.407227] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000002 [ 5.415587] RBP: ffff8801ef0a5d80 R08: 0000000000000001 R09: 0000000000000000 [ 5.423945] R10: ffff88036e000000 R11: ffff8801ef0a5c20 R12: ffff88036ec09c80 [ 5.432300] R13: 0000000000000002 R14: 0000000000000002 R15: 0000000000000003 [ 5.440662] FS: 0000000000000000(0000) GS:ffffc90000000000(0000) knlGS:0000000000000000 [ 5.450386] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [ 5.457189] CR2: 0000000000000010 CR3: 0000000001001000 CR4: 00000000000006b0 [ 5.465545] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5.473908] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 5.482267] Process swapper (pid: 1, threadinfo ffff8801ef0a4000, task ffff88036f040000) [ 5.491994] Stack: [ 5.494606] 01ffffff816d0040 00000000ffffff9f ffffffff817244a0 ffff88036ec09c80 [ 5.503262] <0> ffff8801ef0a5d90 ffffffff811e5ef0 ffff8801ef0a5df0 ffffffff8138c041 [ 5.512853] <0> ffffffff8138bf93 00000001ef0a5de0 ffff8801ef0a5e18 0000000681478fb3 [ 5.523046] Call Trace: [ 5.526158] [<ffffffff811e5ef0>] security_socket_post_create+0x11/0x13 [ 5.533932] [<ffffffff8138c041>] __sock_create+0x1a4/0x1ff [ 5.540541] [<ffffffff8138bf93>] ? __sock_create+0xf6/0x1ff [ 5.547243] [<ffffffff8184028c>] ? inet_init+0x0/0x204 [ 5.553463] [<ffffffff8138c0bb>] sock_create_kern+0x1f/0x21 [ 5.560162] [<ffffffff813edd5a>] inet_ctl_sock_create+0x29/0x5d [ 5.567255] [<ffffffff8184028c>] ? inet_init+0x0/0x204 [ 5.573473] [<ffffffff8183fb60>] tcp_sk_init+0x25/0x27 [ 5.579687] [<ffffffff81396716>] register_pernet_operations+0x18/0x1a [ 5.587365] [<ffffffff81396824>] register_pernet_subsys+0x29/0x3d [ 5.594657] [<ffffffff8184028c>] ? inet_init+0x0/0x204 [ 5.600882] [<ffffffff8183fb27>] tcp_v4_init+0x1c/0x30 [ 5.607106] [<ffffffff818403cd>] inet_init+0x141/0x204 [ 5.613333] [<ffffffff8100905c>] do_one_initcall+0x56/0x130 [ 5.620038] [<ffffffff810a2a72>] ? register_irq_proc+0xae/0xca [ 5.627041] [<ffffffff81130000>] ? d_add+0x16/0x1d [ 5.632875] [<ffffffff81811915>] kernel_init+0x15e/0x1b4 [ 5.639294] [<ffffffff81028b9a>] child_rip+0xa/0x20 [ 5.645214] [<ffffffff8102857c>] ? restore_args+0x0/0x30 [ 5.651622] [<ffffffff818117b7>] ? kernel_init+0x0/0x1b4 [ 5.658033] [<ffffffff81028b90>] ? child_rip+0x0/0x20 [ 5.664158] Code: ca 44 89 ef e8 d8 b6 ff ff c6 43 22 01 66 89 43 20 31 c0 49 8b 54 24 60 48 85 d2 74 22 8b 43 1c 48 8b 92 a8 03 00 00 41 0f b7 f5 <89> 42 10 8b 43 20 66 89 42 18 49 8b 7c 24 60 e8 de 4a 00 00 41 [ 5.690779] RIP [<ffffffff811ed364>] selinux_socket_post_create+0x78/0x95 [ 5.698921] RSP <ffff8801ef0a5d60> [ 5.703202] CR2: 0000000000000010 [ 5.707298] ---[ end trace a7919e7f17c0a725 ]--- [ 5.712848] swapper used greatest stack depth: 4680 bytes left [ 5.727678] Kernel panic - not syncing: Attempted to kill init! [ 5.734672] Pid: 1, comm: swapper Tainted: G D 2.6.31-rc1-net-2.6-igb-ed-patch-07071123 #2 [ 5.745375] Call Trace: [ 5.748485] [<ffffffff81057cfd>] panic+0xdb/0x190 [ 5.754217] [<ffffffff8105ae3b>] ? do_exit+0x35f/0x6d8 [ 5.760439] [<ffffffff8105adeb>] ? do_exit+0x30f/0x6d8 [ 5.766659] [<ffffffff8105adeb>] ? do_exit+0x30f/0x6d8 [ 5.772875] [<ffffffff8107bdbb>] ? trace_hardirqs_on+0xd/0xf [ 5.779688] [<ffffffff8147a295>] ? _write_unlock_irq+0x2b/0x31 [ 5.786690] [<ffffffff8105ab55>] do_exit+0x79/0x6d8 [ 5.792622] [<ffffffff8147b07d>] oops_end+0xb2/0xba [ 5.798555] [<ffffffff810407b4>] no_context+0x1ef/0x1fe [ 5.804875] [<ffffffff8107ba3b>] ? mark_lock+0x22/0x1fb [ 5.811196] [<ffffffff8107af82>] ? save_trace+0x3f/0x96 [ 5.817516] [<ffffffff81040a03>] __bad_area_nosemaphore+0x186/0x1a9 [ 5.825009] [<ffffffff813e5bbf>] ? raw_hash_sk+0x2a/0x73 [ 5.831421] [<ffffffff8147c354>] ? do_page_fault+0xbd/0x22e [ 5.838128] [<ffffffff81040a9c>] bad_area_nosemaphore+0xe/0x10 [ 5.845132] [<ffffffff8147c3b6>] do_page_fault+0x11f/0x22e [ 5.851747] [<ffffffff8147a4ef>] page_fault+0x1f/0x30 [ 5.857871] [<ffffffff811ed364>] ? selinux_socket_post_create+0x78/0x95 [ 5.865754] [<ffffffff811e5ef0>] security_socket_post_create+0x11/0x13 [ 5.873536] [<ffffffff8138c041>] __sock_create+0x1a4/0x1ff [ 5.880149] [<ffffffff8138bf93>] ? __sock_create+0xf6/0x1ff [ 5.886859] [<ffffffff8184028c>] ? inet_init+0x0/0x204 [ 5.893072] [<ffffffff8138c0bb>] sock_create_kern+0x1f/0x21 [ 5.899774] [<ffffffff813edd5a>] inet_ctl_sock_create+0x29/0x5d [ 5.906863] [<ffffffff8184028c>] ? inet_init+0x0/0x204 [ 5.913080] [<ffffffff8183fb60>] tcp_sk_init+0x25/0x27 [ 5.919299] [<ffffffff81396716>] register_pernet_operations+0x18/0x1a [ 5.926986] [<ffffffff81396824>] register_pernet_subsys+0x29/0x3d [ 5.934274] [<ffffffff8184028c>] ? inet_init+0x0/0x204 [ 5.940504] [<ffffffff8183fb27>] tcp_v4_init+0x1c/0x30 [ 5.946730] [<ffffffff818403cd>] inet_init+0x141/0x204 [ 5.952957] [<ffffffff8100905c>] do_one_initcall+0x56/0x130 [ 5.959668] [<ffffffff810a2a72>] ? register_irq_proc+0xae/0xca [ 5.966666] [<ffffffff81130000>] ? d_add+0x16/0x1d [ 5.972495] [<ffffffff81811915>] kernel_init+0x15e/0x1b4 [ 5.978909] [<ffffffff81028b9a>] child_rip+0xa/0x20 [ 5.984845] [<ffffffff8102857c>] ? restore_args+0x0/0x30 [ 5.991266] [<ffffffff818117b7>] ? kernel_init+0x0/0x1b4 [ 5.997686] [<ffffffff81028b90>] ? child_rip+0x0/0x20
Emil
PATCH 1/1 net: sk_prot_alloc() should not blindly overwrite memory
Quoted 24 lines · Emil S
Quoted 23 lines · Eric Dumazet
Quoted 4 lines · Eric Dumazet
Hold on, I found the problem and will submit a patch (after testing it) in following hours
David, while looking at Emil case, I found following problem.
I am not sure this bug is responsible for the BUG hit by Emil, but it seems a candidate for stable as well...
Thanks
[PATCH] net:
sk_alloc()should not blindly overwrite memorySome sockets use
SLAB_DESTROY_BY_RCU, and our RCU code rely that some fields should not be blindly overwritten, even with null.These fields are
sk->sk_refcntandsk->sk_nulls_node.nextCurrent
sk_alloc()implementation doesnt respect this hypothesis, callingkmem_allocwith__GFP_ZEROand settingsk_refcntto 1 instead of atomically increment it.
Quoted 6 lines
Quoted 2 lines
Reported-by Emil S Tantilov <[email protected]>Signed-off-by Eric Dumazet <[email protected]>Eric,
With this patch applied, I get panic on boot:
Oops, this stuff has to be done inside sk_prot_alloc() or breaks selinux
Thanks for testing Emil, here is a second try !
[PATCH] net: sk_prot_alloc() should not blindly overwrite memory
Some sockets use SLAB_DESTROY_BY_RCU, and our RCU code rely that some fields should not be blindly overwritten, even with null.
These fields are sk->sk_refcnt and sk->sk_nulls_node.next
Current sk_prot_alloc() implementation doesnt respect this hypothesis, calling kmem_cache_alloc() with __GFP_ZERO and setting sk_refcnt to 1 instead of atomically increment it.
--- include/net/sock.h | 24 +++++++++------ net/core/sock.c | 49 ++++++++++++++++++++++++++++---- net/ipv4/inet_timewait_sock.c | 2 - 3 files changed, 59 insertions(+), 16 deletions(-)
include/net/sock.h +15 -9
@@ -103,15 +103,15 @@ struct net;/** * struct sock_common - minimal network layer representation of sockets +* @skc_node: main hash linkage for various protocol lookup tables +* @skc_nulls_node: main hash linkage for UDP/UDP-Lite protocol +* @skc_refcnt: reference count +* @skc_hash: hash value used with various protocol lookup tables * @skc_family: network address family * @skc_state: Connection state * @skc_reuse: %SO_REUSEADDR setting * @skc_bound_dev_if: bound device index if != 0 -* @skc_node: main hash linkage for various protocol lookup tables -* @skc_nulls_node: main hash linkage for UDP/UDP-Lite protocol * @skc_bind_node: bind hash linkage for various protocol lookup tables -* @skc_refcnt: reference count -* @skc_hash: hash value used with various protocol lookup tables * @skc_prot: protocol handlers inside a network family * @skc_net: reference to the network namespace of this socket *
@@ -119,17 +119,23 @@ struct net;* for struct sock and struct inet_timewait_sock. */ struct sock_common { -unsigned short skc_family; -volatile unsigned char skc_state; -unsigned char skc_reuse; -int skc_bound_dev_if; +/* +* WARNING : skc_node, skc_refcnt must be first elements of this struct +* (sk_prot_alloc() should not clear skc_node.next & skc_refcnt because +* of RCU lookups) +*/ union {struct hlist_node skc_node;struct hlist_nulls_node skc_nulls_node; }; -struct hlist_node skc_bind_node;atomic_t skc_refcnt; +unsigned int skc_hash; +unsigned short skc_family; +volatile unsigned char skc_state; +unsigned char skc_reuse; +int skc_bound_dev_if; +struct hlist_node skc_bind_node;struct proto *skc_prot;#ifdef CONFIG_NET_NS struct net *skc_net;
net/core/sock.c +43 -6
84 lines not on this page · raw message
net/ipv4/inet_timewait_sock.c +1 -1
@@ -139,7 +139,7 @@ struct inet_timewait_sock *inet_twsk_alloc(const struct sock *sk, const int stat tw->tw_transparent = inet->transparent; tw->tw_prot = sk->sk_prot_creator;twsk_net_set (tw,hold_net (sock_net (sk))); -atomic_set (&tw->tw_refcnt,1 ); +atomic_inc (&tw->tw_refcnt);inet_twsk_dead_node_init (tw);__module_get (tw->tw_prot->owner); }
David Miller
maintainerRe: [PATCH] net: sk_prot_alloc() should not blindly overwrite memory
From: Eric Dumazet <[email protected]>
Date: Wed, 08 Jul 2009 00:33:29 +0200
Quoted 5 lines
[PATCH] net:
sk_prot_alloc()should not blindly overwrite memorySome sockets use
SLAB_DESTROY_BY_RCU, and our RCU code rely that some fields should not be blindly overwritten, even with null.
Quoted 8 lines
These fields are
sk->sk_refcntandsk->sk_nulls_node.nextCurrent
sk_prot_alloc()implementation doesnt respect this hypothesis, callingkmem_cache_alloc()with__GFP_ZEROand settingsk_refcntto 1 instead of atomically increment it.Reported-by Emil S Tantilov <[email protected]>Signed-off-by Eric Dumazet <[email protected]>
I've applied this but will wait for some more testing before I push it out for real to kernel.org
Re: [PATCH] net: sk_prot_alloc() should not blindly overwrite memory
Quoted 14 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Wed, 08 Jul 2009 00:33:29 +0200Quoted 11 lines
[PATCH] net:
sk_prot_alloc()should not blindly overwrite memorySome sockets use
SLAB_DESTROY_BY_RCU, and our RCU code rely that some fields should not be blindly overwritten, even with null.These fields are
sk->sk_refcntandsk->sk_nulls_node.nextCurrent
sk_prot_alloc()implementation doesnt respect this hypothesis, callingkmem_cache_alloc()with__GFP_ZEROand settingsk_refcntto 1 instead of atomically increment it.
Quoted 5 lines
Quoted 2 lines
Reported-by Emil S Tantilov <[email protected]>Signed-off-by Eric Dumazet <[email protected]>I've applied this but will wait for some more testing before I push it out for real to kernel.org
Thanks David
I forgot to CC Paul and Patrick, so I'll ask them to look at this patch.
Patrick, a similar fix is needed in conntrack as well, we currently uses "ct = kmem_cache_zalloc(nf_conntrack_cachep, gfp);" and thus
overwrite struct hlist_nulls_node hnnode; contained
in "struct nf_conntrack_tuple_hash", while lockless readers still potentialy need them. Setting hnnode.next to NULL is dangerous since last bit is not set (not a nulls value), a reader could try to dereference this NULL pointer and trap.
Here is the patch again so that Paul & Patrick can comment on it.
I am not sure about the refcnt thing (blindly setting it to 0 again should be OK in fact, since no reader should/can to the atomic_inc_if_not_zero on it), but the nulls.next thing is problematic.
[PATCH] net: sk_prot_alloc() should not blindly overwrite memory
Some sockets use SLAB_DESTROY_BY_RCU, and our RCU code rely that some fields should not be blindly overwritten, even with null.
These fields are sk->sk_refcnt and sk->sk_nulls_node.next
Current sk_prot_alloc() implementation doesnt respect this hypothesis, calling kmem_cache_alloc() with __GFP_ZERO and setting sk_refcnt to 1 instead of atomically increment it.
--- include/net/sock.h | 24 +++++++++------ net/core/sock.c | 49 ++++++++++++++++++++++++++++---- net/ipv4/inet_timewait_sock.c | 2 - 3 files changed, 59 insertions(+), 16 deletions(-)
include/net/sock.h +15 -9
@@ -103,15 +103,15 @@ struct net;/** * struct sock_common - minimal network layer representation of sockets +* @skc_node: main hash linkage for various protocol lookup tables +* @skc_nulls_node: main hash linkage for UDP/UDP-Lite protocol +* @skc_refcnt: reference count +* @skc_hash: hash value used with various protocol lookup tables * @skc_family: network address family * @skc_state: Connection state * @skc_reuse: %SO_REUSEADDR setting * @skc_bound_dev_if: bound device index if != 0 -* @skc_node: main hash linkage for various protocol lookup tables -* @skc_nulls_node: main hash linkage for UDP/UDP-Lite protocol * @skc_bind_node: bind hash linkage for various protocol lookup tables -* @skc_refcnt: reference count -* @skc_hash: hash value used with various protocol lookup tables * @skc_prot: protocol handlers inside a network family * @skc_net: reference to the network namespace of this socket *
@@ -119,17 +119,23 @@ struct net;* for struct sock and struct inet_timewait_sock. */ struct sock_common { -unsigned short skc_family; -volatile unsigned char skc_state; -unsigned char skc_reuse; -int skc_bound_dev_if; +/* +* WARNING : skc_node, skc_refcnt must be first elements of this struct +* (sk_prot_alloc() should not clear skc_node.next & skc_refcnt because +* of RCU lookups) +*/ union {struct hlist_node skc_node;struct hlist_nulls_node skc_nulls_node; }; -struct hlist_node skc_bind_node;atomic_t skc_refcnt; +unsigned int skc_hash; +unsigned short skc_family; +volatile unsigned char skc_state; +unsigned char skc_reuse; +int skc_bound_dev_if; +struct hlist_node skc_bind_node;struct proto *skc_prot;#ifdef CONFIG_NET_NS struct net *skc_net;
net/core/sock.c +43 -6
84 lines not on this page · raw message
net/ipv4/inet_timewait_sock.c +1 -1
@@ -139,7 +139,7 @@ struct inet_timewait_sock *inet_twsk_alloc(const struct sock *sk, const int stat tw->tw_transparent = inet->transparent; tw->tw_prot = sk->sk_prot_creator;twsk_net_set (tw,hold_net (sock_net (sk))); -atomic_set (&tw->tw_refcnt,1 ); +atomic_inc (&tw->tw_refcnt);inet_twsk_dead_node_init (tw);__module_get (tw->tw_prot->owner); }
Re: [PATCH] net: sk_prot_alloc() should not blindly overwrite memory
Quoted 33 lines · Eric Dumazet
Quoted 10 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Wed, 08 Jul 2009 00:33:29 +0200Quoted 7 lines
[PATCH] net:
sk_prot_alloc()should not blindly overwrite memorySome sockets use
SLAB_DESTROY_BY_RCU, and our RCU code rely that some fields should not be blindly overwritten, even with null.These fields are
sk->sk_refcntandsk->sk_nulls_node.nextQuoted 8 lines
Quoted 6 lines
Current
sk_prot_alloc()implementation doesnt respect this hypothesis, callingkmem_cache_alloc()with__GFP_ZEROand settingsk_refcntto 1 instead of atomically increment it.Reported-by Emil S Tantilov <[email protected]>Signed-off-by Eric Dumazet <[email protected]>I've applied this but will wait for some more testing before I push it out for real to kernel.org
Thanks David
I forgot to CC Paul and Patrick, so I'll ask them to look at this patch.
Patrick, a similar fix is needed in conntrack as well, we currently uses "ct =
kmem_cache_zalloc(nf_conntrack_cachep, gfp);" and thus
overwrite structhlist_nulls_nodehnnode; contained
in "structnf_conntrack_tuple_hash", while lockless readers still potentialy need them. Setting hnnode.next to NULL is dangerous since last bit is not set (not a nulls value), a reader could try to dereference this NULL pointer and trap.
Quoted 5 lines
Here is the patch again so that Paul & Patrick can comment on it.
I am not sure about the refcnt thing (blindly setting it to 0 again should be OK in fact, since no reader should/can to the
atomic_inc_if_not_zeroon it), but the nulls.next thing is problematic.
Here is an updated and much simpler patch, taking care of sk_node.next being not set to 0
This patch applies to >= 2.6.29 kernels
[PATCH] net: sk_prot_alloc() should not blindly overwrite memory
Some sockets use SLAB_DESTROY_BY_RCU, and our RCU code correctness depends on sk->sk_nulls_node.next being always valid. A NULL value is not allowed as it might fault a lockless reader.
Current sk_prot_alloc() implementation doesnt respect this hypothesis, calling kmem_cache_alloc() with __GFP_ZERO. Just call memset() around the forbidden field.
---
net/core/sock.c +17 -2
@@ -939,8 +939,23 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority,struct kmem_cache *slab; slab = prot->slab; -if (slab !=NULL ) - sk =kmem_cache_alloc (slab, priority); +if (slab !=NULL ) { + sk =kmem_cache_alloc (slab, priority & ~__GFP_ZERO); +if (!sk) +return sk; +if (priority & __GFP_ZERO) { +/* +* caches using SLAB_DESTROY_BY_RCU should let +* sk_node.next un-modified. Special care is taken +* when initializing object to zero. +*/ + if (offsetof(struct sock, sk_node.next) != 0) + memset(sk, 0, offsetof(struct sock, sk_node.next)); +memset (&sk->sk_node.pprev,0 , + prot->obj_size - offsetof(struct sock , + sk_node.pprev)); + } + }else sk =kmalloc (prot->obj_size, priority);
Re: [PATCH] net: sk_prot_alloc() should not blindly overwrite memory
Quoted 36 lines · Eric Dumazet
Quoted 35 lines · Eric Dumazet
Quoted 10 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Wed, 08 Jul 2009 00:33:29 +0200Quoted 7 lines
[PATCH] net:
sk_prot_alloc()should not blindly overwrite memorySome sockets use
SLAB_DESTROY_BY_RCU, and our RCU code rely that some fields should not be blindly overwritten, even with null.These fields are
sk->sk_refcntandsk->sk_nulls_node.nextQuoted 8 lines
Quoted 6 lines
Current
sk_prot_alloc()implementation doesnt respect this hypothesis, callingkmem_cache_alloc()with__GFP_ZEROand settingsk_refcntto 1 instead of atomically increment it.Reported-by Emil S Tantilov <[email protected]>Signed-off-by Eric Dumazet <[email protected]>I've applied this but will wait for some more testing before I push it out for real to kernel.org
Thanks David
I forgot to CC Paul and Patrick, so I'll ask them to look at this patch.
Patrick, a similar fix is needed in conntrack as well, we currently uses "ct =
kmem_cache_zalloc(nf_conntrack_cachep, gfp);" and thus
overwrite structhlist_nulls_nodehnnode; contained
in "structnf_conntrack_tuple_hash", while lockless readers still potentialy need them. Setting hnnode.next to NULL is dangerous since last bit is not set (not a nulls value), a reader could try to dereference this NULL pointer and trap.Here is the patch again so that Paul & Patrick can comment on it.
Quoted 7 lines
Quoted 3 lines
I am not sure about the refcnt thing (blindly setting it to 0 again should be OK in fact, since no reader should/can to the
atomic_inc_if_not_zeroon it), but the nulls.next thing is problematic.Here is an updated and much simpler patch, taking care of sk_node.next being not set to 0
This patch applies to >= 2.6.29 kernels
Does this one also need the rearrangement of struct elements in the earlier patch? (And apologies about being slow to get to that one.)
Thanx, Paul
Quoted 42 lines
[PATCH] net:
sk_prot_alloc()should not blindly overwrite memorySome sockets use
SLAB_DESTROY_BY_RCU, and our RCU code correctness depends onsk->sk_nulls_node.nextbeing always valid. A NULL value is not allowed as it might fault a lockless reader.Current
sk_prot_alloc()implementation doesnt respect this hypothesis, callingkmem_cache_alloc()with__GFP_ZERO. Just callmemset()around the forbidden field.Signed-off-by Eric Dumazet <[email protected]>---
net/core/sock.c +17
-2@@ -939,8 +939,23 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority,struct kmem_cache *slab; slab = prot->slab; -if (slab !=NULL ) - sk =kmem_cache_alloc (slab, priority); +if (slab !=NULL ) { + sk =kmem_cache_alloc (slab, priority & ~__GFP_ZERO); +if (!sk) +return sk; +if (priority & __GFP_ZERO) { +/* +* caches using SLAB_DESTROY_BY_RCU should let +* sk_node.next un-modified. Special care is taken +* when initializing object to zero. +*/ + if (offsetof(struct sock, sk_node.next) != 0) + memset(sk, 0, offsetof(struct sock, sk_node.next)); +memset (&sk->sk_node.pprev,0 , + prot->obj_size - offsetof(struct sock , + sk_node.pprev)); + } + }else sk =kmalloc (prot->obj_size, priority);
Re: [PATCH] net: sk_prot_alloc() should not blindly overwrite memory
Quoted 39 lines · Paul E. McKenney
Quoted 38 lines · Eric Dumazet
Quoted 37 lines · Eric Dumazet
Quoted 10 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Wed, 08 Jul 2009 00:33:29 +0200Quoted 7 lines
[PATCH] net:
sk_prot_alloc()should not blindly overwrite memorySome sockets use
SLAB_DESTROY_BY_RCU, and our RCU code rely that some fields should not be blindly overwritten, even with null.These fields are
sk->sk_refcntandsk->sk_nulls_node.nextQuoted 8 lines
Quoted 6 lines
Current
sk_prot_alloc()implementation doesnt respect this hypothesis, callingkmem_cache_alloc()with__GFP_ZEROand settingsk_refcntto 1 instead of atomically increment it.Reported-by Emil S Tantilov <[email protected]>Signed-off-by Eric Dumazet <[email protected]>I've applied this but will wait for some more testing before I push it out for real to kernel.org
Thanks David
I forgot to CC Paul and Patrick, so I'll ask them to look at this patch.
Patrick, a similar fix is needed in conntrack as well, we currently uses "ct =
kmem_cache_zalloc(nf_conntrack_cachep, gfp);" and thus
overwrite structhlist_nulls_nodehnnode; contained
in "structnf_conntrack_tuple_hash", while lockless readers still potentialy need them. Setting hnnode.next to NULL is dangerous since last bit is not set (not a nulls value), a reader could try to dereference this NULL pointer and trap.Here is the patch again so that Paul & Patrick can comment on it.
I am not sure about the refcnt thing (blindly setting it to 0 again should be OK in fact, since no reader should/can to the
atomic_inc_if_not_zeroon it), but the nulls.next thing is problematic.
Quoted 7 lines
Quoted 3 lines
Here is an updated and much simpler patch, taking care of sk_node.next being not set to 0
This patch applies to >= 2.6.29 kernels
Does this one also need the rearrangement of struct elements in the earlier patch? (And apologies about being slow to get to that one.)
No, because only one field (sk_node.next) needs special attention, I felt it was not really necessary to reorder fields. First memset is inlined because of constant size, so small cost.
Thanks
Quoted 44 lines
Thanx, Paul
Quoted 42 lines
[PATCH] net:
sk_prot_alloc()should not blindly overwrite memorySome sockets use
SLAB_DESTROY_BY_RCU, and our RCU code correctness depends onsk->sk_nulls_node.nextbeing always valid. A NULL value is not allowed as it might fault a lockless reader.Current
sk_prot_alloc()implementation doesnt respect this hypothesis, callingkmem_cache_alloc()with__GFP_ZERO. Just callmemset()around the forbidden field.Signed-off-by Eric Dumazet <[email protected]>---
net/core/sock.c +17
-2@@ -939,8 +939,23 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority,struct kmem_cache *slab; slab = prot->slab; -if (slab !=NULL ) - sk =kmem_cache_alloc (slab, priority); +if (slab !=NULL ) { + sk =kmem_cache_alloc (slab, priority & ~__GFP_ZERO); +if (!sk) +return sk; +if (priority & __GFP_ZERO) { +/* +* caches using SLAB_DESTROY_BY_RCU should let +* sk_node.next un-modified. Special care is taken +* when initializing object to zero. +*/ + if (offsetof(struct sock, sk_node.next) != 0) + memset(sk, 0, offsetof(struct sock, sk_node.next)); +memset (&sk->sk_node.pprev,0 , + prot->obj_size - offsetof(struct sock , + sk_node.pprev)); + } + }else sk =kmalloc (prot->obj_size, priority);
David Miller
maintainerRe: [PATCH] net: sk_prot_alloc() should not blindly overwrite memory
In reply to Eric DumazetFrom: Eric Dumazet <[email protected]>
Date: Thu, 09 Jul 2009 07:36:05 +0200
Quoted 11 lines
[PATCH] net:
sk_prot_alloc()should not blindly overwrite memorySome sockets use
SLAB_DESTROY_BY_RCU, and our RCU code correctness depends onsk->sk_nulls_node.nextbeing always valid. A NULL value is not allowed as it might fault a lockless reader.Current
sk_prot_alloc()implementation doesnt respect this hypothesis, callingkmem_cache_alloc()with__GFP_ZERO. Just callmemset()around the forbidden field.Signed-off-by Eric Dumazet <[email protected]>
APplied and queued up for -stable.
Re: [PATCH] net: sk_prot_alloc() should not blindly overwrite memory
Quoted 9 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Thu, 09 Jul 2009 07:36:05 +0200Quoted 6 lines
[PATCH] net:
sk_prot_alloc()should not blindly overwrite memorySome sockets use
SLAB_DESTROY_BY_RCU, and our RCU code correctness depends onsk->sk_nulls_node.nextbeing always valid. A NULL value is not allowed as it might fault a lockless reader.
Quoted 7 lines
Quoted 5 lines
Current
sk_prot_alloc()implementation doesnt respect this hypothesis, callingkmem_cache_alloc()with__GFP_ZERO. Just callmemset()around the forbidden field.Signed-off-by Eric Dumazet <[email protected]>APplied and queued up for -stable.
I'll try to find some time to fix netfilter conntrack as well.
Thanks
PATCH 1/1 net: nf_conntrack_alloc() should not use kmem_cache_zalloc()
David, Patrick
Here is a fix for nf_conntrack, candidate for linux-2.6.31 and stable (linux-2.6.30)
Thank you
[PATCH] net: nf_conntrack_alloc() should not use kmem_cache_zalloc()
When a slab cache uses SLAB_DESTROY_BY_RCU, we must be careful when allocating objects, since slab allocator could give a freed object still used by lockless readers.
In particular, nf_conntrack RCU lookups rely on ct->tuplehash[xxx].hnnode.next being always valid (ie containing a valid 'nulls' value, or a valid pointer to next object in hash chain.)
kmem_cache_zalloc() setups object with NULL values, but a NULL value is not valid for ct->tuplehash[xxx].hnnode.next.
Fix is to call kmem_cache_alloc() and do the zeroing ourself.
---
net/netfilter/nf_conntrack_core.c +13 -2
@@ -561,17 +561,28 @@ struct nf_conn *nf_conntrack_alloc(struct net *net, } } - ct =kmem_cache_zalloc (nf_conntrack_cachep, gfp); +/* +* Do not use kmem_cache_zalloc(), as this cache uses +* SLAB_DESTROY_BY_RCU. +*/ + ct =kmem_cache_alloc (nf_conntrack_cachep, gfp);if (ct ==NULL ) {pr_debug ("nf_conntrack_alloc: Can't alloc conntrack.\n" );atomic_dec (&net->ct.count);return ERR_PTR (-ENOMEM ); } - +/* +* Let ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode.next +* and ct->tuplehash[IP_CT_DIR_REPLY].hnnode.next unchanged. +*/ +memset (&ct->tuplehash[IP_CT_DIR_MAX ],0 , + sizeof(*ct) - offsetof(struct nf_conn, tuplehash[IP_CT_DIR_MAX]));spin_lock_init (&ct->lock);atomic_set (&ct->ct_general.use,1 ); ct->tuplehash[IP_CT_DIR_ORIGINAL ].tuple = *orig; + ct->tuplehash[IP_CT_DIR_ORIGINAL ].hnnode.pprev =NULL ; ct->tuplehash[IP_CT_DIR_REPLY ].tuple = *repl; + ct->tuplehash[IP_CT_DIR_REPLY ].hnnode.pprev =NULL ;/* Don't set timer yet: wait for confirmation */ setup_timer (&ct->timeout, death_by_timeout, (unsigned long )ct);#ifdef CONFIG_NET_NS
Re: [PATCH] net: nf_conntrack_alloc() should not use kmem_cache_zalloc()
Quoted 6 lines · Eric Dumazet
[PATCH] net:
nf_conntrack_alloc()should not usekmem_cache_zalloc()When a slab cache uses
SLAB_DESTROY_BY_RCU, we must be careful when allocating objects, since slab allocator could give a freed object still used by lockless readers.
Quoted 8 lines
In particular,
nf_conntrackRCU lookups rely on ct->tuplehash[xxx].hnnode.next being always valid (ie containing a valid 'nulls' value, or a valid pointer to next object in hash chain.)
kmem_cache_zalloc()setups object with NULL values, but a NULL value is not valid for ct->tuplehash[xxx].hnnode.next.Fix is to call
kmem_cache_alloc()and do the zeroing ourself.
I think this is still racy, please see below:
Quoted 10 lines
net/netfilter/nf_conntrack_core.c +1
-1@@ -561,17 +561,28 @@ struct nf_conn *nf_conntrack_alloc(struct net *net, } } - ct =kmem_cache_zalloc (nf_conntrack_cachep, gfp); +/*
net/netfilter/nf_conntrack_core.c struct nf_conn *nf_conntrack_alloc(struct net *net, · line 570
Quoted 10 lines
net/netfilter/nf_conntrack_core.c +4
-1+* Do not use kmem_cache_zalloc(), as this cache uses +* SLAB_DESTROY_BY_RCU. +*/ + ct =kmem_cache_alloc (nf_conntrack_cachep, gfp);if (ct ==NULL ) {pr_debug ("nf_conntrack_alloc: Can't alloc conntrack.\n" );atomic_dec (&net->ct.count);return ERR_PTR (-ENOMEM ); } -
__nf_conntrack_find() on another CPU finds the entry at this point.
net/netfilter/nf_conntrack_core.c struct nf_conn *nf_conntrack_alloc(struct net *net, · line 581
Quoted 8 lines
net/netfilter/nf_conntrack_core.c +6
-0+/* +* Let ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode.next +* and ct->tuplehash[IP_CT_DIR_REPLY].hnnode.next unchanged. +*/ +memset (&ct->tuplehash[IP_CT_DIR_MAX ],0 , + sizeof(*ct) - offsetof(struct nf_conn, tuplehash[IP_CT_DIR_MAX]));spin_lock_init (&ct->lock);atomic_set (&ct->ct_general.use,1 );
nf_conntrack_find_get() successfully tries to atomic_inc_not_zero() at this point, following by another tuple comparison which is also successful.
Am I missing something? I think we need to make sure the reference count is not increased until the new tuples are visible.
Quoted 8 lines
net/netfilter/nf_conntrack_core.c +2
-0ct->tuplehash[IP_CT_DIR_ORIGINAL ].tuple = *orig; + ct->tuplehash[IP_CT_DIR_ORIGINAL ].hnnode.pprev =NULL ; ct->tuplehash[IP_CT_DIR_REPLY ].tuple = *repl; + ct->tuplehash[IP_CT_DIR_REPLY ].hnnode.pprev =NULL ;/* Don't set timer yet: wait for confirmation */ setup_timer (&ct->timeout, death_by_timeout, (unsigned long )ct);#ifdef CONFIG_NET_NS
PATCH 1/1 net: nf_conntrack_alloc() fixes
Quoted 11 lines · Patrick McHardy
Quoted 10 lines · Eric Dumazet
[PATCH] net:
nf_conntrack_alloc()should not usekmem_cache_zalloc()When a slab cache uses
SLAB_DESTROY_BY_RCU, we must be careful when allocating objects, since slab allocator could give a freed object still used by lockless readers.In particular,
nf_conntrackRCU lookups rely on ct->tuplehash[xxx].hnnode.next being always valid (ie containing a valid 'nulls' value, or a valid pointer to next object in hash chain.)
Quoted 6 lines
Quoted 4 lines
kmem_cache_zalloc()setups object with NULL values, but a NULL value is not valid for ct->tuplehash[xxx].hnnode.next.Fix is to call
kmem_cache_alloc()and do the zeroing ourself.I think this is still racy, please see below:
Nice catch indeed !
Quoted 37 lines
Quoted 11 lines
net/netfilter/nf_conntrack_core.c +1
-1@@ -561,17 +561,28 @@ struct nf_conn *nf_conntrack_alloc(struct net *net, } } - ct =kmem_cache_zalloc (nf_conntrack_cachep, gfp); +/* net/netfilter/nf_conntrack_core.c struct nf_conn *nf_conntrack_alloc(struct net *net, · line 570
Quoted 10 lines
net/netfilter/nf_conntrack_core.c +4
-1+* Do not use kmem_cache_zalloc(), as this cache uses +* SLAB_DESTROY_BY_RCU. +*/ + ct =kmem_cache_alloc (nf_conntrack_cachep, gfp);if (ct ==NULL ) {pr_debug ("nf_conntrack_alloc: Can't alloc conntrack.\n" );atomic_dec (&net->ct.count);return ERR_PTR (-ENOMEM ); } -
__nf_conntrack_find()on another CPU finds the entry at this point.net/netfilter/nf_conntrack_core.c struct nf_conn *nf_conntrack_alloc(struct net *net, · line 581
Quoted 8 lines
net/netfilter/nf_conntrack_core.c +6
-0+/* +* Let ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode.next +* and ct->tuplehash[IP_CT_DIR_REPLY].hnnode.next unchanged. +*/ +memset (&ct->tuplehash[IP_CT_DIR_MAX ],0 , + sizeof(*ct) - offsetof(struct nf_conn, tuplehash[IP_CT_DIR_MAX]));spin_lock_init (&ct->lock);atomic_set (&ct->ct_general.use,1 );
nf_conntrack_find_get()successfully tries toatomic_inc_not_zero()at this point, following by another tuple comparison which is also successful.
Quoted 2 lines
Am I missing something? I think we need to make sure the reference count is not increased until the new tuples are visible.
Yes you are right, and Documentation/RCU/rculist_nulls.txt should be updated to reflect this as well (in insert algo, must use smp_wmb() between obj->key assignment and refcnt assigment)
We'll have to change socket allocation too, this will be addressed by a followup patch
Thanks Patrick !
[PATCH] net: nf_conntrack_alloc() fixes
When a slab cache uses SLAB_DESTROY_BY_RCU, we must be careful when allocating objects, since slab allocator could give a freed object still used by lockless readers.
In particular, nf_conntrack RCU lookups rely on ct->tuplehash[xxx].hnnode.next being always valid (ie containing a valid 'nulls' value, or a valid pointer to next object in hash chain.)
kmem_cache_zalloc() setups object with NULL values, but a NULL value is not valid for ct->tuplehash[xxx].hnnode.next.
Fix is to call kmem_cache_alloc() and do the zeroing ourself.
As spotted by Patrick, we also need to make sure lookup keys are committed to memory before setting refcount to 1, or a lockless reader could get a reference on the old version of the object. Its key re-check could then pass the barrier.
--- Documentation/RCU/rculist_nulls.txt | 7 ++++++- net/netfilter/nf_conntrack_core.c | 21 ++++++++++++++++++--- 2 files changed, 24 insertions(+), 4 deletions(-)
Documentation/RCU/rculist_nulls.txt +6 -1
@@ -83,11 +83,12 @@ not detect it missed following items in original chain. obj = kmem_cache_alloc(...); lock_chain(); // typically a spin_lock() obj->key = key; -atomic_inc(&obj->refcnt); /* * we need to make sure obj->key is updated before obj->next + * or obj->refcnt */ smp_wmb(); +atomic_set(&obj->refcnt, 1); hlist_add_head_rcu(&obj->obj_node, list); unlock_chain(); // typically a spin_unlock()
@@ -159,6 +160,10 @@ out: obj = kmem_cache_alloc(cachep); lock_chain(); // typically a spin_lock() obj->key = key; +/* + * changes to obj->key must be visible before refcnt one + */ +smp_wmb(); atomic_set(&obj->refcnt, 1); /* * insert obj in RCU way (readers might be traversing chain)
net/netfilter/nf_conntrack_core.c +18 -3
@@ -561,23 +561,38 @@ struct nf_conn *nf_conntrack_alloc(struct net *net, } } - ct =kmem_cache_zalloc (nf_conntrack_cachep, gfp); +/* +* Do not use kmem_cache_zalloc(), as this cache uses +* SLAB_DESTROY_BY_RCU. +*/ + ct =kmem_cache_alloc (nf_conntrack_cachep, gfp);if (ct ==NULL ) {pr_debug ("nf_conntrack_alloc: Can't alloc conntrack.\n" );atomic_dec (&net->ct.count);return ERR_PTR (-ENOMEM ); } - +/* +* Let ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode.next +* and ct->tuplehash[IP_CT_DIR_REPLY].hnnode.next unchanged. +*/ +memset (&ct->tuplehash[IP_CT_DIR_MAX ],0 , + sizeof(*ct) - offsetof(struct nf_conn, tuplehash[IP_CT_DIR_MAX]));spin_lock_init (&ct->lock); -atomic_set (&ct->ct_general.use,1 ); ct->tuplehash[IP_CT_DIR_ORIGINAL ].tuple = *orig; + ct->tuplehash[IP_CT_DIR_ORIGINAL ].hnnode.pprev =NULL ; ct->tuplehash[IP_CT_DIR_REPLY ].tuple = *repl; + ct->tuplehash[IP_CT_DIR_REPLY ].hnnode.pprev =NULL ;/* Don't set timer yet: wait for confirmation */ setup_timer (&ct->timeout, death_by_timeout, (unsigned long )ct);#ifdef CONFIG_NET_NS ct->ct_net = net;#endif +/* +* changes to lookup keys must be done before setting refcnt to 1 +*/ +smp_wmb (); +atomic_set (&ct->ct_general.use,1 );return ct; }EXPORT_SYMBOL_GPL (nf_conntrack_alloc);
PATCH 1/1 net: sock_copy() fixes
David, I believe following patch could be 2.6.31 candidate (and stable (2.6.29/2.6.30) as well)
Note : Because we clear all sock fields in sk_prot_alloc(), setting sk_refcnt to 1 in sock_init_data() is safe, even if sock not yet added to hash table (with correct keys). Lockless readers have to re-check their key after getting a reference on the found object anyway, their re-check should not catch old key values.
Next problem to fix (but it can wait 2.6.32 I believe) is to delay sk_refcnt initialization *after* sock lookup keys are setup for new socket, not after being set to 0.
A lockless reader could :
- find an old sock, with old keys
(at this point, another cpu re-allocate this sock, clear fields,
and sets sk_refcnt to 1)
- succeed to increment sk_refcnt
(at this point, other cpu is filling lookup fields (remote address/port, local address...)
- re-check keys and find a match, while socket still being initialized by other cpu.
Only way to fix this is to set sk_refcnt to a non null value only at the point it is safe for a reader to find the socket and use it.
Thanks
[PATCH] net: sock_copy() fixes
Commit e912b1142be8f1e2c71c71001dc992c6e5eb2ec1 (net: sk_prot_alloc() should not blindly overwrite memory) took care of not zeroing whole new socket at allocation time.
sock_copy() is another spot where we should be very careful. We should not set refcnt to a non null value, until we are sure other fields are correctly setup, or
a lockless reader could catch this socket by mistake, while not fully (re)initialized.
This patch puts sk_node & sk_refcnt to the very beginning of struct sock to ease sock_copy() & sk_prot_alloc() job.
We add appropriate smp_wmb() before sk_refcnt initializations to match our RCU requirements (changes to sock keys should be committed to memory before sk_refcnt setting)
--- include/net/sock.h | 32 +++++++++++++++++++------------- net/core/sock.c | 20 ++++++++++++++++++-- 2 files changed, 37 insertions(+), 15 deletions(-)
include/net/sock.h +19 -13
@@ -104,15 +104,15 @@ struct net;/** * struct sock_common - minimal network layer representation of sockets +* @skc_node: main hash linkage for various protocol lookup tables +* @skc_nulls_node: main hash linkage for UDP/UDP-Lite protocol +* @skc_refcnt: reference count +* @skc_hash: hash value used with various protocol lookup tables * @skc_family: network address family * @skc_state: Connection state * @skc_reuse: %SO_REUSEADDR setting * @skc_bound_dev_if: bound device index if != 0 -* @skc_node: main hash linkage for various protocol lookup tables -* @skc_nulls_node: main hash linkage for UDP/UDP-Lite protocol * @skc_bind_node: bind hash linkage for various protocol lookup tables -* @skc_refcnt: reference count -* @skc_hash: hash value used with various protocol lookup tables * @skc_prot: protocol handlers inside a network family * @skc_net: reference to the network namespace of this socket *
@@ -120,17 +120,21 @@ struct net;* for struct sock and struct inet_timewait_sock. */ struct sock_common { -unsigned short skc_family; -volatile unsigned char skc_state; -unsigned char skc_reuse; -int skc_bound_dev_if; +/* +* first fields are not copied in sock_copy() +*/ union {struct hlist_node skc_node;struct hlist_nulls_node skc_nulls_node; }; -struct hlist_node skc_bind_node;atomic_t skc_refcnt; +unsigned int skc_hash; +unsigned short skc_family; +volatile unsigned char skc_state; +unsigned char skc_reuse; +int skc_bound_dev_if; +struct hlist_node skc_bind_node;struct proto *skc_prot;#ifdef CONFIG_NET_NS struct net *skc_net;
@@ -208,15 +212,17 @@ struct sock {* don't add nothing before this first member (__sk_common) --acme */ struct sock_common __sk_common; +#define sk_node __sk_common.skc_node +#define sk_nulls_node __sk_common.skc_nulls_node +#define sk_refcnt __sk_common.skc_refcnt + +#define sk_copy_start __sk_common.skc_hash +#define sk_hash __sk_common.skc_hash#define sk_family __sk_common.skc_family#define sk_state __sk_common.skc_state#define sk_reuse __sk_common.skc_reuse#define sk_bound_dev_if __sk_common.skc_bound_dev_if -#define sk_node __sk_common.skc_node -#define sk_nulls_node __sk_common.skc_nulls_node#define sk_bind_node __sk_common.skc_bind_node -#define sk_refcnt __sk_common.skc_refcnt -#define sk_hash __sk_common.skc_hash#define sk_prot __sk_common.skc_prot#define sk_net __sk_common.skc_netkmemcheck_bitfield_begin (flags);
net/core/sock.c +18 -2
@@ -919,13 +919,19 @@ static inline void sock_lock_init(struct sock *sk) af_family_keys + sk->sk_family); } +/* +* Copy all fields from osk to nsk but nsk->sk_refcnt must not change yet, +* even temporarly, because of RCU lookups. sk_node should also be left as is. +*/ static void sock_copy (struct sock *nsk,const struct sock *osk) {#ifdef CONFIG_SECURITY_NETWORK void *sptr = nsk->sk_security;#endif - -memcpy (nsk, osk, osk->sk_prot->obj_size); +BUILD_BUG_ON (offsetof(struct sock , sk_copy_start) != +sizeof (osk->sk_node) +sizeof (osk->sk_refcnt)); +memcpy (&nsk->sk_copy_start, &osk->sk_copy_start, + osk->sk_prot->obj_size - offsetof(struct sock , sk_copy_start));#ifdef CONFIG_SECURITY_NETWORK nsk->sk_security = sptr;security_sk_clone (osk, nsk);
@@ -1140,6 +1146,11 @@ struct sock *sk_clone(const struct sock *sk, const gfp_t priority) newsk->sk_err =0 ; newsk->sk_priority =0 ; +/* +* Before updating sk_refcnt, we must commit prior changes to memory +* (Documentation/RCU/rculist_nulls.txt for details) +*/ +smp_wmb ();atomic_set (&newsk->sk_refcnt,2 );/*
@@ -1855,6 +1866,11 @@ void sock_init_data(struct socket *sock, struct sock *sk) sk->sk_stamp =ktime_set (-1L ,0 ); +/* +* Before updating sk_refcnt, we must commit prior changes to memory +* (Documentation/RCU/rculist_nulls.txt for details) +*/ +smp_wmb ();atomic_set (&sk->sk_refcnt,1 );atomic_set (&sk->sk_wmem_alloc,1 );atomic_set (&sk->sk_drops,0 );
David Miller
maintainerRe: [PATCH] net: sock_copy() fixes
From: Eric Dumazet <[email protected]>
Date: Thu, 16 Jul 2009 11:13:10 +0200
Quoted 12 lines
[PATCH] net:
sock_copy()fixesCommit
e912b1142be8f1e2c71c71001dc992c6e5eb2ec1(net:sk_prot_alloc()should not blindly overwrite memory) took care of not zeroing whole new socket at allocation time.
sock_copy()is another spot where we should be very careful. We should not set refcnt to a non null value, until we are sure other fields are correctly setup, or
a lockless reader could catch this socket by mistake, while not fully (re)initialized.
Quoted 8 lines
This patch puts
sk_node&sk_refcntto the very beginning of struct sock to easesock_copy()&sk_prot_alloc()job.We add appropriate
smp_wmb()beforesk_refcntinitializations to match our RCU requirements (changes to sock keys should be committed to memory beforesk_refcntsetting)Signed-off-by Eric Dumazet <[email protected]>
Applied and queued for -stable, thanks!
Re: [PATCH] net: nf_conntrack_alloc() fixes
In reply to Eric DumazetQuoted 10 lines · Eric Dumazet
[PATCH] net:
nf_conntrack_alloc()fixesWhen a slab cache uses
SLAB_DESTROY_BY_RCU, we must be careful when allocating objects, since slab allocator could give a freed object still used by lockless readers.In particular,
nf_conntrackRCU lookups rely on ct->tuplehash[xxx].hnnode.next being always valid (ie containing a valid 'nulls' value, or a valid pointer to next object in hash chain.)
Quoted 8 lines
kmem_cache_zalloc()setups object with NULL values, but a NULL value is not valid for ct->tuplehash[xxx].hnnode.next.Fix is to call
kmem_cache_alloc()and do the zeroing ourself.As spotted by Patrick, we also need to make sure lookup keys are committed to memory before setting refcount to 1, or a lockless reader could get a reference on the old version of the object. Its key re-check could then pass the barrier.
Looks good to me. Applied, thanks Eric. I'll push it to -stable with the other fixes in a couple of days.
RE: [PATCH] net: sk_prot_alloc() should not blindly overwrite memory
In reply to David MillerQuoted 14 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Wed, 08 Jul 2009 00:33:29 +0200Quoted 11 lines
[PATCH] net:
sk_prot_alloc()should not blindly overwrite memorySome sockets use
SLAB_DESTROY_BY_RCU, and our RCU code rely that some fields should not be blindly overwritten, even with null.These fields are
sk->sk_refcntandsk->sk_nulls_node.nextCurrent
sk_prot_alloc()implementation doesnt respect this hypothesis, callingkmem_cache_alloc()with__GFP_ZEROand settingsk_refcntto 1 instead of atomically increment it.
Quoted 5 lines
Quoted 2 lines
Reported-by Emil S Tantilov <[email protected]>Signed-off-by Eric Dumazet <[email protected]>I've applied this but will wait for some more testing before I push it out for real to kernel.org
Still seeing traces during the test even with this patch applied:
[ 1089.430093] ------------[ cut here ]------------ [ 1089.435667] WARNING: at include/net/sock.h:423 udp_lib_unhash+0x73/0xa0() [ 1089.435670] Hardware name: S5520HC [ 1089.435671] Modules linked in: igb dca mdio [last unloaded: ixgbe] [ 1089.435678] Pid: 15545, comm: netserver Not tainted 2.6.31-rc1-net-2.6-igb-ed-07071641 #4 [ 1089.435681] Call Trace: [ 1089.435686] [<ffffffff813e8a2f>] ? udp_lib_unhash+0x73/0xa0 [ 1089.435691] [<ffffffff81057b49>] warn_slowpath_common+0x77/0x8f [ 1089.435696] [<ffffffff81057b70>] warn_slowpath_null+0xf/0x11 [ 1089.435700] [<ffffffff813e8a2f>] udp_lib_unhash+0x73/0xa0 [ 1089.435705] [<ffffffff8138e616>] sk_common_release+0x2f/0xb4 [ 1089.435710] [<ffffffff81429028>] udp_lib_close+0x9/0xb [ 1089.435715] [<ffffffff813ee62a>] inet_release+0x58/0x5f [ 1089.435720] [<ffffffff814158e5>] inet6_release+0x30/0x35 [ 1089.435725] [<ffffffff8138be4b>] sock_release+0x1a/0x6c [ 1089.435729] [<ffffffff8138c366>] sock_close+0x22/0x26 [ 1089.435735] [<ffffffff810ec923>] __fput+0xf0/0x18c [ 1089.435739] [<ffffffff810eccd1>] fput+0x15/0x18 [ 1089.435742] [<ffffffff810e9bfa>] filp_close+0x5c/0x67 [ 1089.435746] [<ffffffff810e9c80>] sys_close+0x7b/0xb6 [ 1089.435751] [<ffffffff81027aab>] system_call_fastpath+0x16/0x1b [ 1089.435755] ---[ end trace a79410bd00b8b1ac ]---
Emil
David Miller
maintainerRe: [PATCH] net: sk_prot_alloc() should not blindly overwrite memory
From: "Tantilov, Emil S" <[email protected]>
Date: Wed, 8 Jul 2009 11:02:22 -0600
Quoted 5 lines
Still seeing traces during the test even with this patch applied:
[ 1089.430093] ------------[ cut here ]------------ [ 1089.435667] WARNING: at include/net/sock.h:423 udp_lib_unhash+0x73/0xa0() [ 1089.435670] Hardware name: S5520HC
Ok I'll back this out for now, needs more investigation obviously.
Re: [PATCH] net: sk_prot_alloc() should not blindly overwrite memory
Quoted 11 lines · David Miller
From: "Tantilov, Emil S" <[email protected]>
Date: Wed, 8 Jul 2009 11:02:22 -0600Quoted 5 lines
Still seeing traces during the test even with this patch applied:
[ 1089.430093] ------------[ cut here ]------------ [ 1089.435667] WARNING: at include/net/sock.h:423 udp_lib_unhash+0x73/0xa0() [ 1089.435670] Hardware name: S5520HCOk I'll back this out for now, needs more investigation obviously.
Hmm... I never said it was supposed to fix Emil problem, just that I discovered one potential problem by code inspection.
I could not find yet sk_refcnt mismatch.
As we do less atomic ops per packet than before, some old bug could surface now...
Emil, is it easy to reproduce this problem, considering I have a similar platform than yours (dual quad core machine, E5450 cpus @ 3GHz) ?
RE: [PATCH] net: sk_prot_alloc() should not blindly overwrite memory
Quoted 16 lines · Eric Dumazet
Quoted 11 lines · David Miller
From: "Tantilov, Emil S" <[email protected]>
Date: Wed, 8 Jul 2009 11:02:22 -0600Quoted 5 lines
Still seeing traces during the test even with this patch applied:
[ 1089.430093] ------------[ cut here ]------------ [ 1089.435667] WARNING: at include/net/sock.h:423udp_lib_unhash+0x73/0xa0() [ 1089.435670] Hardware name: S5520HC
Ok I'll back this out for now, needs more investigation obviously.
Hmm... I never said it was supposed to fix Emil problem, just that I discovered one potential problem by code inspection.
Quoted 7 lines
I could not find yet
sk_refcntmismatch.
As we do less atomic ops per packet than before, some old bug could surface now...Emil, is it easy to reproduce this problem, considering I have a similar platform than yours (dual quad core machine, E5450 cpus @ 3GHz) ?
Eric,
It should be easy to reproduce. At least I have been able to consistently reproduce it on several different systems with different drivers (e1000, e1000e, igb).
The test I'm running is a mix of IPV4/6 TCP/UDP traffic with netperf (also mixing different types TCP/UDP_STREAM, TCP_MAERTS, TCP_UDP_RR etc). How much this matters I don't know - it's possible that just UDP traffic would do it. I also think it may have something to do with IPv6 because of the trace, but I am not sure.
If you need more information let me know.
Thanks,
Emil
PATCH 1/1 net: ip_push_pending_frames() fix
Quoted 25 lines · Emil S
Quoted 14 lines · Eric Dumazet
Quoted 10 lines · David Miller
From: "Tantilov, Emil S" <[email protected]>
Date: Wed, 8 Jul 2009 11:02:22 -0600Quoted 5 lines
Still seeing traces during the test even with this patch applied:
[ 1089.430093] ------------[ cut here ]------------ [ 1089.435667] WARNING: at include/net/sock.h:423udp_lib_unhash+0x73/0xa0() [ 1089.435670] Hardware name: S5520HC
Ok I'll back this out for now, needs more investigation obviously.
Hmm... I never said it was supposed to fix Emil problem, just that I discovered one potential problem by code inspection.
Quoted 7 lines
I could not find yet
sk_refcntmismatch.
As we do less atomic ops per packet than before, some old bug could surface now...Emil, is it easy to reproduce this problem, considering I have a similar platform than yours (dual quad core machine, E5450 cpus @ 3GHz) ?
Eric,
Quoted 8 lines
It should be easy to reproduce. At least I have been able to consistently reproduce it on several different systems with different drivers (e1000, e1000e, igb).
The test I'm running is a mix of IPV4/6 TCP/UDP traffic with netperf (also mixing different types TCP/UDP_STREAM,
TCP_MAERTS,TCP_UDP_RRetc). How much this matters I don't know - it's possible that just UDP traffic would do it. I also think it may have something to do with IPv6 because of the trace, but I am not sure.If you need more information let me know.
OK thanks, this was helpful, corking or not corking, that is the question :)
I think ip6_push_pending_frames() & ip_push_pending_frames
have a problem after recent commit 2b85a34e911bf483c27cfdd124aeb1605145dc80 (net: No more expensive sock_hold()/sock_put() on each tx)
[PATCH] net: ip_push_pending_frames() fix
After commit 2b85a34e911bf483c27cfdd124aeb1605145dc80
(net: No more expensive sock_hold()/sock_put() on each tx) we do not take any more references on sk->sk_refcnt on outgoing packets.
I forgot to delete two __sock_put() from ip_push_pending_frames() and ip6_push_pending_frames().
---
net/ipv4/ip_output.c +0 -1
@@ -1243,7 +1243,6 @@ int ip_push_pending_frames(struct sock *sk) skb->len += tmp_skb->len; skb->data_len += tmp_skb->len; skb->truesize += tmp_skb->truesize; -__sock_put (tmp_skb->sk); tmp_skb->destructor =NULL ; tmp_skb->sk =NULL ; }
net/ipv6/ip6_output.c +0 -1
@@ -1484,7 +1484,6 @@ int ip6_push_pending_frames(struct sock *sk) skb->len += tmp_skb->len; skb->data_len += tmp_skb->len; skb->truesize += tmp_skb->truesize; -__sock_put (tmp_skb->sk); tmp_skb->destructor =NULL ; tmp_skb->sk =NULL ; }
RE: [PATCH] net: ip_push_pending_frames() fix
Quoted 78 lines · Eric Dumazet
Quoted 30 lines · Emil S
Quoted 14 lines · Eric Dumazet
Quoted 10 lines · David Miller
From: "Tantilov, Emil S" <[email protected]>
Date: Wed, 8 Jul 2009 11:02:22 -0600Quoted 5 lines
Still seeing traces during the test even with this patch applied:
[ 1089.430093] ------------[ cut here ]------------ [ 1089.435667] WARNING: at include/net/sock.h:423udp_lib_unhash+0x73/0xa0() [ 1089.435670] Hardware name: S5520HC
Ok I'll back this out for now, needs more investigation obviously.
Hmm... I never said it was supposed to fix Emil problem, just that I discovered one potential problem by code inspection.
Quoted 7 lines
I could not find yet
sk_refcntmismatch.
As we do less atomic ops per packet than before, some old bug could surface now...Emil, is it easy to reproduce this problem, considering I have a similar platform than yours (dual quad core machine, E5450 cpus @ 3GHz) ?
Eric,
It should be easy to reproduce. At least I have been able to consistently
reproduce it on several different systems with different drivers (e1000, e1000e, igb).Quoted 8 lines
The test I'm running is a mix of IPV4/6 TCP/UDP traffic with netperf (also mixing different types TCP/UDP_STREAM,
TCP_MAERTS,TCP_UDP_RRetc). How much this matters I don't know - it's possible that just UDP traffic would do it. I also think it may have something to do with IPv6 because of the trace, but I am not sure.If you need more information let me know.
OK thanks, this was helpful, corking or not corking, that is the question :)
I think
ip6_push_pending_frames()&ip_push_pending_frameshave a problem after recent commit2b85a34e911bf483c27cfdd124aeb1605145dc80(net: No more expensive sock_hold()/sock_put() on each tx)[PATCH] net:
ip_push_pending_frames()fixAfter commit
2b85a34e911bf483c27cfdd124aeb1605145dc80(net: No more expensive sock_hold()/sock_put() on each tx) we do not take any more references onsk->sk_refcnton outgoing packets.I forgot to delete two
__sock_put()fromip_push_pending_frames()andip6_push_pending_frames().Reported-by Emil S Tantilov <[email protected]>Signed-off-by Eric Dumazet <[email protected]>---
net/ipv4/ip_output.c +0
-1@@ -1243,7 +1243,6 @@ int ip_push_pending_frames(struct sock *sk) skb->len += tmp_skb->len; skb->data_len += tmp_skb->len; skb->truesize += tmp_skb->truesize; -__sock_put (tmp_skb->sk); tmp_skb->destructor =NULL ; tmp_skb->sk =NULL ; }net/ipv6/ip6_output.c +0
-0
net/ipv6/ip6_output.c int ip6_push_pending_frames(struct sock *sk) · line 1489
Quoted 7 lines
net/ipv6/ip6_output.c +0
-1@@ -1484,7 +1484,6 @@ int ip6_push_pending_frames(struct sock *sk) skb->len += tmp_skb->len; skb->data_len += tmp_skb->len; skb->truesize += tmp_skb->truesize; -__sock_put (tmp_skb->sk); tmp_skb->destructor =NULL ; tmp_skb->sk =NULL ; }
Thanks Eric,
With this patch the test ran all night without issues.
Emil
Eric Dumazet
Tested-by Emil S TantilovRe: [PATCH] net: ip_push_pending_frames() fix
Quoted 87 lines · Emil S
Quoted 76 lines · Eric Dumazet
Quoted 29 lines · Emil S
Quoted 14 lines · Eric Dumazet
Quoted 10 lines · David Miller
From: "Tantilov, Emil S" <[email protected]>
Date: Wed, 8 Jul 2009 11:02:22 -0600Quoted 5 lines
Still seeing traces during the test even with this patch applied:
[ 1089.430093] ------------[ cut here ]------------ [ 1089.435667] WARNING: at include/net/sock.h:423udp_lib_unhash+0x73/0xa0() [ 1089.435670] Hardware name: S5520HC
Ok I'll back this out for now, needs more investigation obviously.
Hmm... I never said it was supposed to fix Emil problem, just that I discovered one potential problem by code inspection.
Quoted 7 lines
I could not find yet
sk_refcntmismatch.
As we do less atomic ops per packet than before, some old bug could surface now...Emil, is it easy to reproduce this problem, considering I have a similar platform than yours (dual quad core machine, E5450 cpus @ 3GHz) ?
Eric,
It should be easy to reproduce. At least I have been able to consistently
reproduce it on several different systems with different drivers (e1000, e1000e, igb).Quoted 8 lines
The test I'm running is a mix of IPV4/6 TCP/UDP traffic with netperf (also mixing different types TCP/UDP_STREAM,
TCP_MAERTS,TCP_UDP_RRetc). How much this matters I don't know - it's possible that just UDP traffic would do it. I also think it may have something to do with IPv6 because of the trace, but I am not sure.If you need more information let me know.
OK thanks, this was helpful, corking or not corking, that is the question :)
I think
ip6_push_pending_frames()&ip_push_pending_frameshave a problem after recent commit2b85a34e911bf483c27cfdd124aeb1605145dc80(net: No more expensive sock_hold()/sock_put() on each tx)[PATCH] net:
ip_push_pending_frames()fixAfter commit
2b85a34e911bf483c27cfdd124aeb1605145dc80(net: No more expensive sock_hold()/sock_put() on each tx) we do not take any more references onsk->sk_refcnton outgoing packets.I forgot to delete two
__sock_put()fromip_push_pending_frames()andip6_push_pending_frames().Reported-by Emil S Tantilov <[email protected]>Signed-off-by Eric Dumazet <[email protected]>---
net/ipv4/ip_output.c +0
-1@@ -1243,7 +1243,6 @@ int ip_push_pending_frames(struct sock *sk) skb->len += tmp_skb->len; skb->data_len += tmp_skb->len; skb->truesize += tmp_skb->truesize; -__sock_put (tmp_skb->sk); tmp_skb->destructor =NULL ; tmp_skb->sk =NULL ; }net/ipv6/ip6_output.c +0
-0net/ipv6/ip6_output.c int ip6_push_pending_frames(struct sock *sk) · line 1489
Quoted 7 lines
net/ipv6/ip6_output.c +0
-1@@ -1484,7 +1484,6 @@ int ip6_push_pending_frames(struct sock *sk) skb->len += tmp_skb->len; skb->data_len += tmp_skb->len; skb->truesize += tmp_skb->truesize; -__sock_put (tmp_skb->sk); tmp_skb->destructor =NULL ; tmp_skb->sk =NULL ; }Thanks Eric,
Quoted 2 lines
With this patch the test ran all night without issues.
Thanks a lot Emil for testing and your feedback.
David, could you please add another tag ?
Thanks
David Miller
maintainerRe: [PATCH] net: ip_push_pending_frames() fix
In reply to Eric DumazetFrom: Eric Dumazet <[email protected]>
Date: Thu, 09 Jul 2009 02:20:42 +0200
Quoted 11 lines
[PATCH] net:
ip_push_pending_frames()fixAfter commit
2b85a34e911bf483c27cfdd124aeb1605145dc80
(net: No more expensive sock_hold()/sock_put() on each tx) we do not take any more references onsk->sk_refcnton outgoing packets.I forgot to delete two
__sock_put()fromip_push_pending_frames()andip6_push_pending_frames().Reported-by Emil S Tantilov <[email protected]>Signed-off-by Eric Dumazet <[email protected]>
Applied, with the Tested-by marker added.