Skip to contentSkip to conversation

lockdep: fix sk->sk_callback_lock locking

PATCH In review Peter Zijlstra 4 participants 6 messages Nov 28, 2006 – Nov 29, 2006 3 messages not in the archive

Subsystems (3)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 1164635760.6588.27.camel@twins

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw

PATCH 1/1 lockdep: fix sk->sk_callback_lock locking

=========================================================
[ INFO: possible irq lock inversion dependency detected ]
2.6.19-rc6 #4
---------------------------------------------------------
nc/1854 just changed the state of lock:
(af_callback_keys + sk->sk_family#2){-.-?}, at: [<c0268a7f>] sock_def_error_report+0x1f/0x90
but this lock was taken by another, soft-irq-safe lock in the past:
(slock-AF_INET){-+..}

and interrupts could create inverse lock ordering between them.

stack backtrace:
 [<c0103f36>] show_trace_log_lvl+0x26/0x40
 [<c010406b>] show_trace+0x1b/0x20
 [<c01049e4>] dump_stack+0x24/0x30
 [<c013738c>] print_irq_inversion_bug+0x10c/0x130
 [<c01374f2>] check_usage_backwards+0x42/0x50
 [<c0137912>] mark_lock+0x312/0x620
 [<c0138925>] __lock_acquire+0x4c5/0xcb0
 [<c0139499>] lock_acquire+0x69/0x90
 [<c02ddc49>] _read_lock+0x39/0x50
 [<c026778c>] sock_def_wakeup+0x1c/0x60
 [<c02b7d73>] inet_shutdown+0x93/0xf0
 [<c02648d2>] sys_shutdown+0x32/0x60
 [<c0266284>] sys_socketcall+0x1d4/0x260
 [<c01031f3>] syscall_call+0x7/0xb
 =======================

stack backtrace:
 [<c0103f36>] show_trace_log_lvl+0x26/0x40
 [<c010406b>] show_trace+0x1b/0x20
 [<c01049e4>] dump_stack+0x24/0x30
 [<c013738c>] print_irq_inversion_bug+0x10c/0x130
 [<c01374f2>] check_usage_backwards+0x42/0x50
 [<c0137912>] mark_lock+0x312/0x620
 [<c0138925>] __lock_acquire+0x4c5/0xcb0
 [<c0139499>] lock_acquire+0x69/0x90
 [<c02ddc59>] _read_lock+0x39/0x50
 [<c0268a7f>] sock_def_error_report+0x1f/0x90
 [<c029b968>] tcp_disconnect+0x318/0x490
 [<c02b9110>] inet_stream_connect+0x220/0x260
 [<c0264adb>] sys_connect+0x6b/0x90
 [<c026614f>] sys_socketcall+0x9f/0x260
 [<c01031f3>] syscall_call+0x7/0xb
 =======================

sk->sk_callback_lock is usually only read locked from softirq context however it seems lockdep found two spots that are reachable from process context, thus creating the possibility of a deadlock.

For now fix these two call sites with manual disabling of softirqs; if more of these sites are found we might consider changing the read_lock() to read_lock_bh().

CC Martin Josefsson <[email protected]>
---
 net/ipv4/af_inet.c |    2 ++
 net/ipv4/tcp.c     |    2 ++
 2 files changed, 4 insertions(+)
net/ipv4/af_inet.c +2 -0
Index: linux-2.6-netdev/net/ipv4/af_inet.c
===================================================================
@@ -731,7 +731,9 @@ int inet_shutdown(struct socket *sock, i
 	}
 
 	/* Wake up anyone sleeping in poll. */
+	local_bh_disable();
 	sk->sk_state_change(sk);
+	local_bh_enable();
 	release_sock(sk);
 	return err;
 }
net/ipv4/tcp.c +2 -0
Index: linux-2.6-netdev/net/ipv4/tcp.c
===================================================================
@@ -1765,7 +1765,9 @@ int tcp_disconnect(struct sock *sk, int 
 
 	BUG_TRAP(!inet->num || icsk->icsk_bind_hash);
 
+	local_bh_disable();
 	sk->sk_error_report(sk);
+	local_bh_enable();
 	return err;
 }
 
#4 lore raw
Quoted 11 lines · Peter Zijlstra

=========================================================
[ INFO: possible irq lock inversion dependency detected ]
2.6.19-rc6 #4
---------------------------------------------------------
nc/1854 just changed the state of lock:
(af_callback_keys + sk->sk_family#2){-.-?}, at: [<c0268a7f>] sock_def_error_report+0x1f/0x90 but this lock was taken by another, soft-irq-safe lock in the past:
(slock-AF_INET){-+..}

and interrupts could create inverse lock ordering between them.

I think this is bogus. The slock is not a standard lock. When we hold it in process context we don't actually hold the spin lock part of it. However, it does prevent the softirq path from running in critical sections which also prevents any attempt to grab the callback lock from softirq context.

If you still think there is a problem, please show an actual scenario where it dead locks.

Thanks,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#5 lore raw
Quoted 12 lines · Herbert Xu
Quoted 10 lines · Peter Zijlstra

=========================================================
[ INFO: possible irq lock inversion dependency detected ]
2.6.19-rc6 #4
---------------------------------------------------------
nc/1854 just changed the state of lock:
(af_callback_keys + sk->sk_family#2){-.-?}, at: [<c0268a7f>] sock_def_error_report+0x1f/0x90 but this lock was taken by another, soft-irq-safe lock in the past:
(slock-AF_INET){-+..}

and interrupts could create inverse lock ordering between them.

Quoted 8 lines

I think this is bogus. The slock is not a standard lock. When we hold it in process context we don't actually hold the spin lock part of it. However, it does prevent the softirq path from running in critical sections which also prevents any attempt to grab the callback lock from softirq context.

If you still think there is a problem, please show an actual scenario where it dead locks.

It would be nice to have a look at other parts of stack backtraces probably with softirq part, which took that
lock: sk->sk_family#2){-.-?}

Jarek P.

#6 lore raw
In reply to Herbert Xu
Quoted 13 lines · Herbert Xu
Quoted 11 lines · Peter Zijlstra

=========================================================
[ INFO: possible irq lock inversion dependency detected ]
2.6.19-rc6 #4
---------------------------------------------------------
nc/1854 just changed the state of lock:
(af_callback_keys + sk->sk_family#2){-.-?}, at: [<c0268a7f>] sock_def_error_report+0x1f/0x90 but this lock was taken by another, soft-irq-safe lock in the past:
(slock-AF_INET){-+..}

and interrupts could create inverse lock ordering between them.

Quoted 8 lines

I think this is bogus. The slock is not a standard lock. When we hold it in process context we don't actually hold the spin lock part of it. However, it does prevent the softirq path from running in critical sections which also prevents any attempt to grab the callback lock from softirq context.

If you still think there is a problem, please show an actual scenario where it dead locks.

process context does lock_sock(sk) which is basically a sleeping lock and sets an owner field when acquired.

BH context does bh_lock_sock(sk); which spins on the spinlock protecting the owner field; and checks for an owner under this lock. When an owner is found it will stick the skb on a queue for later processing.

This scheme does indeed seem to avoid the reported deadlock scenario - although I didn't audit all code paths.

However I'm not quite sure yet how to teach lockdep about this. The proposed patch will shut it up though.

#7 lore raw
Quoted 3 lines · Peter Zijlstra

However I'm not quite sure yet how to teach lockdep about this. The proposed patch will shut it up though.

As a rule I think we should never make semantic changes to shut up lockdep.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
maintainer
#8 lore raw

From: Herbert Xu <[email protected]>
Date: Wed, 29 Nov 2006 23:07:09 +1100

Quoted 7 lines · Peter Zijlstra
Quoted 3 lines

However I'm not quite sure yet how to teach lockdep about this. The proposed patch will shut it up though.

As a rule I think we should never make semantic changes to shut up lockdep.

Especially ones which are costly, as this proposed change is in that it disables software interrupts in a place where that is completely unnecessary.

Let's not even consider this patch :)