Skip to contentSkip to conversation

Inet-hashtable: Change the range of sk->hash lock to avoid the race condition.

PATCH In review Jun Chen 2 participants 6 messages Sep 12, 2013 – Sep 13, 2013

Subsystems (3)
lore
Share & tools
lore

Jun Chen

author
#0 lore raw

PATCH 1/1 Inet-hashtable: Change the range of sk->hash lock to avoid the race condition.

When try to add node to list in __inet_hash_nolisten function, first get the list and then to lock for using, but in extremeness case, others can del this node before locking it, then the node should be null.So this patch try to lock firstly and then get the list for using to avoid this race condition.

---
 net/ipv4/inet_hashtables.c |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)
net/ipv4/inet_hashtables.c +6 -3
index 7bd8983..76e846e 100644
@@ -407,12 +407,15 @@ int __inet_hash_nolisten(struct sock *sk, struct inet_timewait_sock *tw)
 	WARN_ON(!sk_unhashed(sk));
 
 	sk->sk_hash = inet_sk_ehashfn(sk);
-	head = inet_ehash_bucket(hashinfo, sk->sk_hash);
-	list = &head->chain;
 	lock = inet_ehash_lockp(hashinfo, sk->sk_hash);
 
 	spin_lock(lock);
-	__sk_nulls_add_node_rcu(sk, list);
+	if (sk_hashed(sk)) {
+		head = inet_ehash_bucket(hashinfo, sk->sk_hash);
+		list = &head->chain;
+		__sk_nulls_add_node_rcu(sk, list);
+	}
+
 	if (tw) {
 		WARN_ON(sk->sk_hash != tw->tw_hash);
 		twrefcnt = inet_twsk_unhash(tw);
Signature · 1 line
1.7.4.1
#1 lore raw
Quoted 4 lines · Jun Chen

When try to add node to list in __inet_hash_nolisten function, first get the list and then to lock for using, but in extremeness case, others can del this node before locking it, then the node should be null.So this patch try to lock firstly and then get the list for using to avoid this race condition.

I suspect another bug. This should not happen.

Care to describe the problem you got ?

Thanks

author
#2 lore raw
Quoted 6 lines · Eric Dumazet
Quoted 4 lines · Jun Chen

When try to add node to list in __inet_hash_nolisten function, first get the list and then to lock for using, but in extremeness case, others can del this node before locking it, then the node should be null.So this patch try to lock firstly and then get the list for using to avoid this race condition.

Quoted 7 lines

I suspect another bug. This should not happen.

Care to describe the problem you got ?

Thanks

Ok, I just got this call stack and no more info, pls help to look it. thanks!

<1>[ 88.548263] BUG: unable to handle kernel NULL pointer dereference at
00000004
<1>[ 88.548490] IP: [] __inet_hash_nolisten+0xc1/0x140
<4>[ 88.548617] *pde = 00000000
<4>[ 88.549927] EIP is at __inet_hash_nolisten+0xc1/0x140
<4>[ 88.550008] EAX: 00000000 EBX: e08c0000 ECX: edf846e0 EDX: e08c0020
<4>[ 88.550055] ESI: c20213c0 EDI: edc12dc0 EBP: ce4bfdfc ESP: ce4bfde8
<4>[ 88.550137] DS: 007b ES: 007b FS: 00d8 GS: 003b SS: 0068
<4>[ 88.550184] CR0: 80050033 CR2: 00000004 CR3: 2b4ff000 CR4: 001007d0
<4>[ 88.550266] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
<4>[ 88.550346] DR6: ffff0ff0 DR7: 00000400
<0>[ 88.550392] Process WebViewCoreThre (pid: 2137, ti=ce4be000
task=eb193c80 task.ti=ce4be000)
<0>[ 88.551746] Call Trace:
<4>[ 88.551797] [] __inet_hash_connect+0x295/0x2d0
<4>[ 88.551883] [] inet_hash_connect+0x40/0x50
<4>[ 88.551932] [] ? inet_unhash+0x90/0x90
<4>[ 88.551981] [] ? __inet_lookup_listener+0x1b0/0x1b0
<4>[ 88.552067] [] tcp_v4_connect+0x247/0x4a0
<4>[ 88.552117] [] ? lock_sock_nested+0x3e/0x50
<4>[ 88.552205] [] inet_stream_connect+0xe2/0x290
<4>[ 88.552254] [] ? _copy_from_user+0x35/0x50
<4>[ 88.552342] [] sys_connect+0xb2/0xd0
<4>[ 88.552393] [] ? alloc_file+0x20/0xa0
<4>[ 88.552441] [] ? tcp_setsockopt+0x50/0x60
<4>[ 88.552525] [] ? fget_light+0x44/0xe0
<4>[ 88.552574] [] ? sock_common_setsockopt+0x27/0x40
<4>[ 88.552659] [] ? _copy_from_user+0x35/0x50
<4>[ 88.552708] [] sys_socketcall+0xab/0x2b0
<4>[ 88.552790] [] ? trace_hardirqs_on_thunk+0xc/0x10
<4>[ 88.552840] [] syscall_call+0x7/0xb
<4>[ 88.552923] [] ? mutex_trylock+0x30/0x140
#3 lore raw
Quoted 42 lines · Jun Chen
Quoted 6 lines · Eric Dumazet
Quoted 4 lines · Jun Chen

When try to add node to list in __inet_hash_nolisten function, first get the list and then to lock for using, but in extremeness case, others can del this node before locking it, then the node should be null.So this patch try to lock firstly and then get the list for using to avoid this race condition.

Quoted 7 lines

I suspect another bug. This should not happen.

Care to describe the problem you got ?

Thanks

Ok, I just got this call stack and no more info, pls help to look it. thanks!

<1>[ 88.548263] BUG: unable to handle kernel NULL pointer dereference at
00000004
<1>[ 88.548490] IP: [] __inet_hash_nolisten+0xc1/0x140
<4>[ 88.548617] *pde = 00000000
<4>[ 88.549927] EIP is at __inet_hash_nolisten+0xc1/0x140
<4>[ 88.550008] EAX: 00000000 EBX: e08c0000 ECX: edf846e0 EDX: e08c0020
<4>[ 88.550055] ESI: c20213c0 EDI: edc12dc0 EBP: ce4bfdfc ESP: ce4bfde8
<4>[ 88.550137] DS: 007b ES: 007b FS: 00d8 GS: 003b SS: 0068
<4>[ 88.550184] CR0: 80050033 CR2: 00000004 CR3: 2b4ff000 CR4: 001007d0
<4>[ 88.550266] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
<4>[ 88.550346] DR6: ffff0ff0 DR7: 00000400
<0>[ 88.550392] Process WebViewCoreThre (pid: 2137, ti=ce4be000
task=eb193c80 task.ti=ce4be000)
<0>[ 88.551746] Call Trace:
<4>[ 88.551797] [] __inet_hash_connect+0x295/0x2d0
<4>[ 88.551883] [] inet_hash_connect+0x40/0x50
<4>[ 88.551932] [] ? inet_unhash+0x90/0x90
<4>[ 88.551981] [] ? __inet_lookup_listener+0x1b0/0x1b0
<4>[ 88.552067] [] tcp_v4_connect+0x247/0x4a0
<4>[ 88.552117] [] ? lock_sock_nested+0x3e/0x50
<4>[ 88.552205] [] inet_stream_connect+0xe2/0x290
<4>[ 88.552254] [] ? _copy_from_user+0x35/0x50
<4>[ 88.552342] [] sys_connect+0xb2/0xd0
<4>[ 88.552393] [] ? alloc_file+0x20/0xa0
Quoted 9 lines
<4>[ 88.552441] [] ? tcp_setsockopt+0x50/0x60
<4>[ 88.552525] [] ? fget_light+0x44/0xe0
<4>[ 88.552574] [] ? sock_common_setsockopt+0x27/0x40
<4>[ 88.552659] [] ? _copy_from_user+0x35/0x50
<4>[ 88.552708] [] sys_socketcall+0xab/0x2b0
<4>[ 88.552790] [] ? trace_hardirqs_on_thunk+0xc/0x10
<4>[ 88.552840] [] syscall_call+0x7/0xb
<4>[ 88.552923] [] ? mutex_trylock+0x30/0x140

This makes no sense to me. This could be a random memory corruption.

Do you have disassembly of __inet_hash_nolisten ?

author
#4 lore raw
Quoted 53 lines · Eric Dumazet
Quoted 42 lines · Jun Chen
Quoted 6 lines · Eric Dumazet
Quoted 4 lines · Jun Chen

When try to add node to list in __inet_hash_nolisten function, first get the list and then to lock for using, but in extremeness case, others can del this node before locking it, then the node should be null.So this patch try to lock firstly and then get the list for using to avoid this race condition.

Quoted 7 lines

I suspect another bug. This should not happen.

Care to describe the problem you got ?

Thanks

Ok, I just got this call stack and no more info, pls help to look it. thanks!

<1>[ 88.548263] BUG: unable to handle kernel NULL pointer dereference at
00000004
<1>[ 88.548490] IP: [] __inet_hash_nolisten+0xc1/0x140
<4>[ 88.548617] *pde = 00000000
<4>[ 88.549927] EIP is at __inet_hash_nolisten+0xc1/0x140
<4>[ 88.550008] EAX: 00000000 EBX: e08c0000 ECX: edf846e0 EDX: e08c0020
<4>[ 88.550055] ESI: c20213c0 EDI: edc12dc0 EBP: ce4bfdfc ESP: ce4bfde8
<4>[ 88.550137] DS: 007b ES: 007b FS: 00d8 GS: 003b SS: 0068
<4>[ 88.550184] CR0: 80050033 CR2: 00000004 CR3: 2b4ff000 CR4: 001007d0
<4>[ 88.550266] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
<4>[ 88.550346] DR6: ffff0ff0 DR7: 00000400
<0>[ 88.550392] Process WebViewCoreThre (pid: 2137, ti=ce4be000
task=eb193c80 task.ti=ce4be000)
<0>[ 88.551746] Call Trace:
<4>[ 88.551797] [] __inet_hash_connect+0x295/0x2d0
<4>[ 88.551883] [] inet_hash_connect+0x40/0x50
<4>[ 88.551932] [] ? inet_unhash+0x90/0x90
<4>[ 88.551981] [] ? __inet_lookup_listener+0x1b0/0x1b0
<4>[ 88.552067] [] tcp_v4_connect+0x247/0x4a0
<4>[ 88.552117] [] ? lock_sock_nested+0x3e/0x50
<4>[ 88.552205] [] inet_stream_connect+0xe2/0x290
<4>[ 88.552254] [] ? _copy_from_user+0x35/0x50
<4>[ 88.552342] [] sys_connect+0xb2/0xd0
<4>[ 88.552393] [] ? alloc_file+0x20/0xa0
Quoted 9 lines
<4>[ 88.552441] [] ? tcp_setsockopt+0x50/0x60
<4>[ 88.552525] [] ? fget_light+0x44/0xe0
<4>[ 88.552574] [] ? sock_common_setsockopt+0x27/0x40
<4>[ 88.552659] [] ? _copy_from_user+0x35/0x50
<4>[ 88.552708] [] sys_socketcall+0xab/0x2b0
<4>[ 88.552790] [] ? trace_hardirqs_on_thunk+0xc/0x10
<4>[ 88.552840] [] syscall_call+0x7/0xb
<4>[ 88.552923] [] ? mutex_trylock+0x30/0x140
Quoted 5 lines

This makes no sense to me. This could be a random memory corruption.

Do you have disassembly of __inet_hash_nolisten ?

I had disassembled the __inet_hash_nolisten+0xc1,
the corruption is located on the:

__inet_hash_nolisten -->
__sk_nulls_add_node_rcu(sk, list); --> __sk_nulls_add_node_rcu -->
static inline void hlist_nulls_add_head_rcu(struct hlist_nulls_node *n,
struct hlist_nulls_head *h)
{
...

    if (!is_a_nulls(first))
        first->pprev = &n->next;  (this line trigger corruption)

...
}

Thanks!

#5 lore raw
Quoted 69 lines · Jun Chen
Quoted 53 lines · Eric Dumazet
Quoted 42 lines · Jun Chen
Quoted 6 lines · Eric Dumazet
Quoted 4 lines · Jun Chen

When try to add node to list in __inet_hash_nolisten function, first get the list and then to lock for using, but in extremeness case, others can del this node before locking it, then the node should be null.So this patch try to lock firstly and then get the list for using to avoid this race condition.

Quoted 7 lines

I suspect another bug. This should not happen.

Care to describe the problem you got ?

Thanks

Ok, I just got this call stack and no more info, pls help to look it. thanks!

<1>[ 88.548263] BUG: unable to handle kernel NULL pointer dereference at
00000004
<1>[ 88.548490] IP: [] __inet_hash_nolisten+0xc1/0x140
<4>[ 88.548617] *pde = 00000000
<4>[ 88.549927] EIP is at __inet_hash_nolisten+0xc1/0x140
<4>[ 88.550008] EAX: 00000000 EBX: e08c0000 ECX: edf846e0 EDX: e08c0020
<4>[ 88.550055] ESI: c20213c0 EDI: edc12dc0 EBP: ce4bfdfc ESP: ce4bfde8
<4>[ 88.550137] DS: 007b ES: 007b FS: 00d8 GS: 003b SS: 0068
<4>[ 88.550184] CR0: 80050033 CR2: 00000004 CR3: 2b4ff000 CR4: 001007d0
<4>[ 88.550266] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
<4>[ 88.550346] DR6: ffff0ff0 DR7: 00000400
<0>[ 88.550392] Process WebViewCoreThre (pid: 2137, ti=ce4be000
task=eb193c80 task.ti=ce4be000)
<0>[ 88.551746] Call Trace:
<4>[ 88.551797] [] __inet_hash_connect+0x295/0x2d0
<4>[ 88.551883] [] inet_hash_connect+0x40/0x50
<4>[ 88.551932] [] ? inet_unhash+0x90/0x90
<4>[ 88.551981] [] ? __inet_lookup_listener+0x1b0/0x1b0
<4>[ 88.552067] [] tcp_v4_connect+0x247/0x4a0
<4>[ 88.552117] [] ? lock_sock_nested+0x3e/0x50
<4>[ 88.552205] [] inet_stream_connect+0xe2/0x290
<4>[ 88.552254] [] ? _copy_from_user+0x35/0x50
<4>[ 88.552342] [] sys_connect+0xb2/0xd0
<4>[ 88.552393] [] ? alloc_file+0x20/0xa0
Quoted 9 lines
<4>[ 88.552441] [] ? tcp_setsockopt+0x50/0x60
<4>[ 88.552525] [] ? fget_light+0x44/0xe0
<4>[ 88.552574] [] ? sock_common_setsockopt+0x27/0x40
<4>[ 88.552659] [] ? _copy_from_user+0x35/0x50
<4>[ 88.552708] [] sys_socketcall+0xab/0x2b0
<4>[ 88.552790] [] ? trace_hardirqs_on_thunk+0xc/0x10
<4>[ 88.552840] [] syscall_call+0x7/0xb
<4>[ 88.552923] [] ? mutex_trylock+0x30/0x140
Quoted 5 lines

This makes no sense to me. This could be a random memory corruption.

Do you have disassembly of __inet_hash_nolisten ?

I had disassembled the __inet_hash_nolisten+0xc1,
the corruption is located on the:

__inet_hash_nolisten -->
__sk_nulls_add_node_rcu(sk, list); --> __sk_nulls_add_node_rcu -->
static inline void hlist_nulls_add_head_rcu(struct hlist_nulls_node *n,
struct hlist_nulls_head *h)
{
...

Quoted 4 lines
    if (!is_a_nulls(first))
        first->pprev = &n->next;  (this line trigger corruption)

...
}

first is NULL, which is absolutely not possible.

You had a memory corruption on some sort.