Possible Circular Locking in TIPC
Subsystems (3)
hi,
while running my usual stuff on 2.6.20-rc1-git5, sfuzz (http://www.digitaldwarf.be/products/sfuzz.c)
did the following, to produce the lockdep warning below:
socket(1, 3, 1)
accept(3, "\x54\x1f\x6d\x30\x0b\x0b\x44\xb2\x6c\x57\x8f\xcd\x12\x8b\x67\xa0", 32)
accept(3, "\x7d\xa0\x2c\x64\x45\x57\xdf\x36\xe5\x7f\xd0\x1c\x5e\x78\x6b\x0e", 16)
setsockopt(3, 48, 206, "\x6e\xec\x11\x2a\xd1\xb0\x5d\xb9\xd9\xf4\x4e\x64\xe6\x78\xce\x46", 4)
fstatfs(3, 0xbfff1cb8)
listen(3, 5)
fstatfs(3, 0xbfff1cb8)socket(16, 3, 31)
fstat(3, 0xbfff1cb8)
getpeername(3, "\x7a\xcc\xda\xb9\x08\x52\x58\xfe\x30\x6e\xa7\x1e\x26\x6a\x3a\xbf", *-1652368178)
accept(3, "\x9b\xf9\xf8\x16\x0f\x6c\x8a\x66\x0b\xf3\x5e\xeb\xcb\xa8\xac\x09", -106)
getsockname(3, "\x68\x78\xb8\x46\x19\xe8\x61\xe1\xa7\x31\x5e\x6b\xd6\xe6\xbe\x7f", *256)
listen(3, 9)
shutdown(3, 0)
shutdown(3, 0)
fstatfs(3, 0xbfff1cb8)
getsockopt(3, 28, 127, "\x41\x0e\x19\x52\x1e\x3f\xe9\x3c\xa3\xf0\x36\x9e\xd2\x02\x84\x03", *128)
getsockopt(3, -170, 5, "\xfb\x48\x97\xdf\x7d\x4a\x79\x7a\x7e\x89\xe4\x3c\x36\x9f\x12\x30", *64)
connect(3, "\xca\xbd\x43\xff\x7e\xa6\x47\x9a\x5a\x89\x1d\x3a\xfe\xcc\x6b\x21", -140)
shutdown(3, 0)
fstatfs(3, 0xbfff1cb8)
bind(3, "\x25\x74\xed\xaf\x20\x5a\x1a\x75\xbc\x06\x37\x1d\x0a\x01\x41\x47", 32)
fstat(3, 0xbfff1cb8)
bind(3, "\xfd\xb5\x0c\x63\xd9\x32\xe6\x54\x0a\x93\xb7\xa6\x23\x68\x07\x00", 32)
bind(3, "\x00\x7c\x05\xc6\xc7\xc2\x07\x68\xbd\x1d\x23\xe3\xea\x00\xec\xe7", 128)
listen(3, 19)
setsockopt(3, 197, -140, "\x84\x7b\xa2\xf8\x29\xf9\x2a\xd1\xd4\xe1\xa1\xcf\x5e\x7f\x6b\x4c", 32)Here is the stacktrace:
[ 313.239556] =======================================================
[ 313.239718] [ INFO: possible circular locking dependency detected ]
[ 313.239795] 2.6.20-rc1-git5 #26
[ 313.239858] -------------------------------------------------------
[ 313.239929] sfuzz/4133 is trying to acquire lock:
[ 313.239996] (ref_table_lock){-+..}, at: [<c04cd0a9>] tipc_ref_discard+0x29/0xe0
[ 313.241101]
[ 313.241105] but task is already holding lock:
[ 313.241225] (&table[i].lock){-+..}, at: [<c04cb7c0>] tipc_deleteport+0x40/0x1a0
[ 313.241524]
[ 313.241528] which lock already depends on the new lock.
[ 313.241535]
[ 313.241709]
[ 313.241713] the existing dependency chain (in reverse order) is:
[ 313.241837]
[ 313.241841] -> #1 (&table[i].lock){-+..}:
[ 313.242096] [<c01366c5>] __lock_acquire+0xd05/0xde0
[ 313.242562] [<c0136809>] lock_acquire+0x69/0xa0
[ 313.243013] [<c04d4040>] _spin_lock_bh+0x40/0x60
[ 313.243476] [<c04cd1cb>] tipc_ref_acquire+0x6b/0xe0
[ 313.244115] [<c04cac73>] tipc_createport_raw+0x33/0x260
[ 313.244562] [<c04cbe21>] tipc_createport+0x41/0x120
[ 313.245007] [<c04c57ec>] tipc_subscr_start+0xcc/0x120
[ 313.245458] [<c04bdb56>] process_signal_queue+0x56/0xa0
[ 313.245906] [<c011ea18>] tasklet_action+0x38/0x80
[ 313.246361] [<c011ecbb>] __do_softirq+0x5b/0xc0
[ 313.246817] [<c01060e8>] do_softirq+0x88/0xe0
[ 313.247450] [<ffffffff>] 0xffffffff
[ 313.247894]
[ 313.247898] -> #0 (ref_table_lock){-+..}:
[ 313.248155] [<c0136415>] __lock_acquire+0xa55/0xde0
[ 313.248601] [<c0136809>] lock_acquire+0x69/0xa0
[ 313.249037] [<c04d4100>] _write_lock_bh+0x40/0x60
[ 313.249486] [<c04cd0a9>] tipc_ref_discard+0x29/0xe0
[ 313.249922] [<c04cb7da>] tipc_deleteport+0x5a/0x1a0
[ 313.250543] [<c04cd4f8>] tipc_create+0x58/0x160
[ 313.250980] [<c0431cb2>] __sock_create+0x112/0x280
[ 313.251422] [<c0431e5a>] sock_create+0x1a/0x20
[ 313.251863] [<c04320fb>] sys_socket+0x1b/0x40
[ 313.252301] [<c0432a72>] sys_socketcall+0x92/0x260
[ 313.252738] [<c0102fd0>] syscall_call+0x7/0xb
[ 313.253175] [<ffffffff>] 0xffffffff
[ 313.253778]
[ 313.253782] other info that might help us debug this:
[ 313.253790]
[ 313.253956] 1 lock held by sfuzz/4133:
[ 313.254019] #0: (&table[i].lock){-+..}, at: [<c04cb7c0>] tipc_deleteport+0x40/0x1a0
[ 313.254346]
[ 313.254351] stack backtrace:
[ 313.254470] [<c01045da>] show_trace_log_lvl+0x1a/0x40
[ 313.254594] [<c0104d72>] show_trace+0x12/0x20
[ 313.254711] [<c0104e79>] dump_stack+0x19/0x20
[ 313.254829] [<c013480f>] print_circular_bug_tail+0x6f/0x80
[ 313.254952] [<c0136415>] __lock_acquire+0xa55/0xde0
[ 313.255070] [<c0136809>] lock_acquire+0x69/0xa0
[ 313.255188] [<c04d4100>] _write_lock_bh+0x40/0x60
[ 313.255315] [<c04cd0a9>] tipc_ref_discard+0x29/0xe0
[ 313.255435] [<c04cb7da>] tipc_deleteport+0x5a/0x1a0
[ 313.255565] [<c04cd4f8>] tipc_create+0x58/0x160
[ 313.255687] [<c0431cb2>] __sock_create+0x112/0x280
[ 313.255811] [<c0431e5a>] sock_create+0x1a/0x20
[ 313.255942] [<c04320fb>] sys_socket+0x1b/0x40
[ 313.256059] [<c0432a72>] sys_socketcall+0x92/0x260
[ 313.256179] [<c0102fd0>] syscall_call+0x7/0xb
[ 313.256300] =======================Greetings, Eric
PATCH 1/1 tipc: checking returns and Re: Possible Circular Locking in TIPC
Quoted 4 lines · Eric Sesterhenn
hi,
while running my usual stuff on 2.6.20-rc1-git5, sfuzz (http://www.digitaldwarf.be/products/sfuzz.c)
did the following, to produce the lockdep warning below:
...
Quoted 68 lines
Here is the stacktrace:
[ 313.239556] ======================================================= [ 313.239718] [ INFO: possible circular locking dependency detected ] [ 313.239795] 2.6.20-rc1-git5 #26 [ 313.239858] ------------------------------------------------------- [ 313.239929] sfuzz/4133 is trying to acquire lock: [ 313.239996] (ref_table_lock){-+..}, at: [<c04cd0a9>] tipc_ref_discard+0x29/0xe0 [ 313.241101] [ 313.241105] but task is already holding lock: [ 313.241225] (&table[i].lock){-+..}, at: [<c04cb7c0>] tipc_deleteport+0x40/0x1a0 [ 313.241524] [ 313.241528] which lock already depends on the new lock. [ 313.241535] [ 313.241709] [ 313.241713] the existing dependency chain (in reverse order) is: [ 313.241837] [ 313.241841] -> #1 (&table[i].lock){-+..}: [ 313.242096] [<c01366c5>] __lock_acquire+0xd05/0xde0 [ 313.242562] [<c0136809>] lock_acquire+0x69/0xa0 [ 313.243013] [<c04d4040>] _spin_lock_bh+0x40/0x60 [ 313.243476] [<c04cd1cb>] tipc_ref_acquire+0x6b/0xe0 [ 313.244115] [<c04cac73>] tipc_createport_raw+0x33/0x260 [ 313.244562] [<c04cbe21>] tipc_createport+0x41/0x120 [ 313.245007] [<c04c57ec>] tipc_subscr_start+0xcc/0x120 [ 313.245458] [<c04bdb56>] process_signal_queue+0x56/0xa0 [ 313.245906] [<c011ea18>] tasklet_action+0x38/0x80 [ 313.246361] [<c011ecbb>] __do_softirq+0x5b/0xc0 [ 313.246817] [<c01060e8>] do_softirq+0x88/0xe0 [ 313.247450] [<ffffffff>] 0xffffffff [ 313.247894] [ 313.247898] -> #0 (ref_table_lock){-+..}: [ 313.248155] [<c0136415>] __lock_acquire+0xa55/0xde0 [ 313.248601] [<c0136809>] lock_acquire+0x69/0xa0 [ 313.249037] [<c04d4100>] _write_lock_bh+0x40/0x60 [ 313.249486] [<c04cd0a9>] tipc_ref_discard+0x29/0xe0 [ 313.249922] [<c04cb7da>] tipc_deleteport+0x5a/0x1a0 [ 313.250543] [<c04cd4f8>] tipc_create+0x58/0x160 [ 313.250980] [<c0431cb2>] __sock_create+0x112/0x280 [ 313.251422] [<c0431e5a>] sock_create+0x1a/0x20 [ 313.251863] [<c04320fb>] sys_socket+0x1b/0x40 [ 313.252301] [<c0432a72>] sys_socketcall+0x92/0x260 [ 313.252738] [<c0102fd0>] syscall_call+0x7/0xb [ 313.253175] [<ffffffff>] 0xffffffff [ 313.253778] [ 313.253782] other info that might help us debug this: [ 313.253790] [ 313.253956] 1 lock held by sfuzz/4133: [ 313.254019] #0: (&table[i].lock){-+..}, at: [<c04cb7c0>] tipc_deleteport+0x40/0x1a0 [ 313.254346] [ 313.254351] stack backtrace: [ 313.254470] [<c01045da>] show_trace_log_lvl+0x1a/0x40 [ 313.254594] [<c0104d72>] show_trace+0x12/0x20 [ 313.254711] [<c0104e79>] dump_stack+0x19/0x20 [ 313.254829] [<c013480f>] print_circular_bug_tail+0x6f/0x80 [ 313.254952] [<c0136415>] __lock_acquire+0xa55/0xde0 [ 313.255070] [<c0136809>] lock_acquire+0x69/0xa0 [ 313.255188] [<c04d4100>] _write_lock_bh+0x40/0x60 [ 313.255315] [<c04cd0a9>] tipc_ref_discard+0x29/0xe0 [ 313.255435] [<c04cb7da>] tipc_deleteport+0x5a/0x1a0 [ 313.255565] [<c04cd4f8>] tipc_create+0x58/0x160 [ 313.255687] [<c0431cb2>] __sock_create+0x112/0x280 [ 313.255811] [<c0431e5a>] sock_create+0x1a/0x20 [ 313.255942] [<c04320fb>] sys_socket+0x1b/0x40 [ 313.256059] [<c0432a72>] sys_socketcall+0x92/0x260 [ 313.256179] [<c0102fd0>] syscall_call+0x7/0xb [ 313.256300] =======================
Quoted 1 line
Greetings, Eric
Hello,
Maybe I misinterpret this but, IMHO lockdep complains about locks acquired in different
order: tipc_ref_acquire() gets ref_table_lock and then tipc_ret_table.entries[index]->lock, but tipc_deleteport() inversely (with:tipc_port_lock() and tipc_ref_discard()). I hope maintainers will decide the correct order.
Btw. there is a problem with tipc_ref_discard():
it should be called with tipc_port_lock, but how to discard a ref if this lock can't be acquired? Is it OK to call it without the lock like in subscr_named_msg_event()?
Btw. #2: during this checking I've found two places where return values from tipc_ref_lock() and tipc_port_lock() are not checked, so I attach a patch proposal for this (compiled but not tested):
Regards,
Jarek P.
---
[PATCH] tipc: checking returns from locking functions
Checking of return values from tipc_ref_lock() and tipc_port_lock() added in 2 places.
---
net/tipc/port.c +6 -1
@@ -238,7 +238,12 @@ u32 tipc_createport_raw(void *usr_handlereturn 0 ; } -tipc_port_lock (ref); +if (!tipc_port_lock (ref)) { +tipc_ref_discard (ref); +warn ("Port creation failed, reference table invalid\n" ); +kfree (p_ptr); +return 0 ; + } p_ptr->publ.ref = ref; msg = &p_ptr->publ.phdr;msg_init (msg,DATA_LOW ,TIPC_NAMED_MSG ,TIPC_OK ,LONG_H_SIZE ,0 );
net/tipc/subscr.c +6 -1
@@ -499,7 +499,12 @@ static void subscr_named_msg_event(void/* Add subscriber to topology server's subscriber list */ -tipc_ref_lock (subscriber->ref); +if (!tipc_ref_lock (subscriber->ref)) { +warn ("Subscriber rejected, unable to find port\n" ); +tipc_ref_discard (subscriber->ref); +kfree (subscriber); +return ; + }spin_lock_bh (&topsrv.lock);list_add (&subscriber->subscriber_list, &topsrv.subscriber_list);spin_unlock_bh (&topsrv.lock);
Re: [PATCH] tipc: checking returns and Re: Possible Circular Locking in TIPC
See my comments below.
Regards
///jon
Quoted 11 lines · Jarek Poplawski
..........
Maybe I misinterpret this but, IMHO lockdep
complains about locks acquired in different
order:tipc_ref_acquire()getsref_table_lockand then tipc_ret_table.entries[index]->lock, buttipc_deleteport()inversely (with:tipc_port_lock()andtipc_ref_discard()).
I hope maintainers will decide the correct
order.
This order is correct. There can never be parallel access to the same _instance_ of tipc_ret_table.entries[index]->lock from the two functions you mention.
Note that tipc_deleteport() takes as argument the reference (=index) returned from tipc_ref_acquire(), so it can not be (and is not) called until and unless the latter function has returned a valid reference. As a parallel, you can't do free() on a memory chunk until malloc() has given you a pointer to it.
Quoted 7 lines
Btw. there is a problem with
tipc_ref_discard():
it should be called withtipc_port_lock, but
how to discard a ref if this lock can't be
acquired? Is it OK to call it without the lock
like insubscr_named_msg_event()?
I suspect you are mixing up things here. We are handling two different reference entries and two different locks in this function.
One reference entry points to a subscription instance, and its
reference (index) is obtainable from subscriber->ref. So, we
could easily lock the entry if needed. However, in this
particular case it is unnecessary, since there is no chance that
anybody else could have obtained the new reference, and
hence no risk for race conditions.
The other reference entry was intended to point to a new port,
but, since we didn't obtain any reference in the first place,
there is no port to delete and no reference to discard.
Quoted 6 lines
Btw. #2: during this checking I've found
two places where return values fromtipc_ref_lock()andtipc_port_lock()are not checked, so I attach a patch proposal for this (compiled but not tested):
Thanks.
Quoted 53 lines
Regards,
Jarek P.
---[PATCH] tipc: checking returns from locking functions
Checking of return values from
tipc_ref_lock()
andtipc_port_lock()added in 2 places.Signed-off-by Jarek Poplawski <[email protected]>---
diff -Nurp
linux-2.6.20-rc2-/net/tipc/port.clinux-2.6.20-rc2/net/tipc/port.cnet/tipc/port.c +6
-1@@ -238,7 +238,12 @@ u32 tipc_createport_raw(void *usr_handlereturn 0 ; } -tipc_port_lock (ref); +if (!tipc_port_lock (ref)) { +tipc_ref_discard (ref); +warn ("Port creation failed, reference table invalid\n" ); +kfree (p_ptr); +return 0 ; + } p_ptr->publ.ref = ref; msg = &p_ptr->publ.phdr;msg_init (msg,DATA_LOW ,TIPC_NAMED_MSG ,TIPC_OK ,LONG_H_SIZE ,0 );diff -Nurp
linux-2.6.20-rc2-/net/tipc/subscr.clinux-2.6.20-rc2/net/tipc/subscr.cnet/tipc/subscr.c +6
-2@@ -499,7 +499,12 @@ static void subscr_named_msg_event(void/* Add subscriber to topology server's subscriber list */ -tipc_ref_lock (subscriber->ref); +if (!tipc_ref_lock (subscriber->ref)) { +warn ("Subscriber rejected, unable to find port\n" ); +tipc_ref_discard (subscriber->ref); +kfree (subscriber); +return ; + }spin_lock_bh (&topsrv.lock);list_add (&subscriber->subscriber_list, &topsrv.subscriber_list);spin_unlock_bh (&topsrv.lock); -To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] tipc: checking returns and Re: Possible Circular Locking in TIPC
Quoted 18 lines · Jon Maloy
See my comments below.
Regards
///jonQuoted 12 lines · Jarek Poplawski
..........
Maybe I misinterpret this but, IMHO lockdep complains about locks acquired in different
order:tipc_ref_acquire()getsref_table_lockand then tipc_ret_table.entries[index]->lock, buttipc_deleteport()inversely (with:tipc_port_lock()andtipc_ref_discard()). I hope maintainers will decide the correct order.
Quoted 8 lines
This order is correct. There can never be parallel access to the same _instance_ of tipc_ret_table.entries[index]->lock from the two functions you mention.
Note thattipc_deleteport()takes as argument the reference (=index) returned fromtipc_ref_acquire(), so it can not be (and is not) called until and unless the latter function has returned a valid reference. As a parallel, you can't dofree()on a memory chunk untilmalloc()has given you a pointer to it.
I'm happy the order is correct! But the warning probably will be back. I know lockdep is sometimes too careful but nevertheless some change is needed to fix a real bug or give additional information to lockdep.
Quoted 7 lines
Quoted 7 lines
Btw. there is a problem with
tipc_ref_discard():
it should be called withtipc_port_lock, but how to discard a ref if this lock can't be acquired? Is it OK to call it without the lock like insubscr_named_msg_event()?
Quoted 12 lines
I suspect you are mixing up things here.
We are handling two different reference entries and two different locks in this function.
One reference entry points to a subscription instance, and its reference (index) is obtainable fromsubscriber->ref. So, we could easily lock the entry if needed. However, in this particular case it is unnecessary, since there is no chance that anybody else could have obtained the new reference, and hence no risk for race conditions.
The other reference entry was intended to point to a new port, but, since we didn't obtain any reference in the first place, there is no port to delete and no reference to discard.
I admit I don't know this program and I hope I didn't mislead anybody with my message. I only tried to point at some doubts and maybe this function could be better commented about when the lock is needed.
Thanks for explanations & best regards,
Jarek P.
Re: [PATCH] tipc: checking returns and Re: Possible Circular Locking in TIPC
Regards
///jon
Quoted 7 lines · Jarek Poplawski
I know lockdep is sometimes
too careful but nevertheless some change is needed to fix a real bug or give additional information to lockdep.
I don't know lockdep well enough yet, but I will try to find out if that is possible.
Quoted 25 lines
Quoted 11 lines
Quoted 11 lines
Btw. there is a problem with
tipc_ref_discard():
it should be called withtipc_port_lock, but
how to discard a ref if this lock can't be
acquired? Is it OK to call it without the lock
like insubscr_named_msg_event()?Quoted 13 lines
I suspect you are mixing up things here. We are handling two different reference entries and two different locks in this function.
One reference entry points to a subscription instance, and its
reference (index) is obtainable fromsubscriber->ref. So, we
could easily lock the entry if needed. However, in this
particular case it is unnecessary, since there is no chance that
anybody else could have obtained the new reference, and
hence no risk for race conditions.
The other reference entry was intended to point to a new port,
but, since we didn't obtain any reference in the first place,
there is no port to delete and no reference to discard.
Quoted 7 lines
I admit I don't know this program and I hope I didn't mislead anybody with my message. I only tried to point at some doubts and maybe this function could be better commented about when the lock is needed.
Agreed.
Quoted 6 lines
Thanks for explanations & best regards,
Jarek P.
Re: [PATCH] tipc: checking returns and Re: Possible Circular Locking in TIPC
Quoted 6 lines · Jon Maloy
Regards
///jonQuoted 1 line · Jarek Poplawski
Quoted 8 lines
Quoted 6 lines
I know lockdep is sometimes
too careful but nevertheless some change is needed to fix a real bug or give additional information to lockdep.I don't know lockdep well enough yet, but I will try to find out if that is possible.
If you are sure there is no circular locking possible between these two functions and this entry->lock here isn't endangered by other functions, you could try to make lockdep "silent" like this:
write_lock_bh(&ref_table_lock);
if (tipc_ref_table.first_free) {
index = tipc_ref_table.first_free;
entry = &(tipc_ref_table.entries[index]);
index_mask = tipc_ref_table.index_mask;
/* take lock in case a previous user of entry still holds it */- spin_lock_bh(&entry->lock, );
+ local_bh_disable();
+ spin_lock_nested(&entry->lock, SINGLE_DEPTH_NESTING);
next_plus_upper = entry->data.next_plus_upper;
tipc_ref_table.first_free = next_plus_upper & index_mask;
reference = (next_plus_upper & ~index_mask) + index;
entry->data.reference = reference;
entry->object = object;
if (lock != 0)
*lock = &entry->lock;/* may stay as is or: */
- spin_unlock_bh(&entry->lock);
+ spin_unlock(&entry->lock);
+ local_bh_enable();
}
write_unlock_bh(&ref_table_lock);Cheers,
Jarek P.
Re: [PATCH] tipc: checking returns and Re: Possible Circular Locking in TIPC
Quoted 31 lines · Jarek Poplawski
If you are sure there is no circular locking possible between these two functions and this
entry->lockhere isn't endangered by other functions, you could try to make lockdep "silent" like this:write_lock_bh(&ref_table_lock); if (tipc_ref_table.first_free) { index = tipc_ref_table.first_free; entry = &(tipc_ref_table.entries[index]); index_mask = tipc_ref_table.index_mask; /* take lock in case a previous user of entry still holds it */(quoted diff) +2
-1- spin_lock_bh(&entry->lock, ); + local_bh_disable(); + spin_lock_nested(&entry->lock, SINGLE_DEPTH_NESTING); next_plus_upper = entry->data.next_plus_upper; tipc_ref_table.first_free = next_plus_upper & index_mask; reference = (next_plus_upper & ~index_mask) + index; entry->data.reference = reference; entry->object = object; if (lock != 0) *lock = &entry->lock;/* may stay as is or: */
Review
Quoted 6 lines
(quoted diff) +2
-1- spin_unlock_bh(&entry->lock); + spin_unlock(&entry->lock); + local_bh_enable(); } write_unlock_bh(&ref_table_lock);
Looks like an acceptable solution. I will try this. Thanks
///Jon