Skip to contentSkip to conversation

starting mc triggers lockdep

DISCUSSION Dave Jones 4 participants 6 messages Jul 7, 2006 – Jul 8, 2006

Subsystems (2)
lore
Share & tools
lore

Dave Jones

author
#0 lore raw

With 2.6.18rc1 + a selection of the lockdep tweaks found so far, midnight commander makes the kernel unhappy.

Dave

=======================================================
[ INFO: possible circular locking dependency detected ]
-------------------------------------------------------
mc/4913 is trying to acquire lock:
(sk_lock-AF_INET){--..}, at: [<ffffffff8022800c>] tcp_sendmsg+0x1f/0xb1a

but task is already holding lock:
(&inode->i_mutex){--..}, at: [<ffffffff802692e0>] mutex_lock+0x2a/0x2e

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #2 (&inode->i_mutex){--..}:

       [<ffffffff802ab6d5>] lock_acquire+0x4a/0x69
       [<ffffffff80269102>] __mutex_lock_slowpath+0xeb/0x29f
       [<ffffffff802692df>] mutex_lock+0x29/0x2e
       [<ffffffff8030f4a0>] create_dir+0x2c/0x1e2
       [<ffffffff8030fa5b>] sysfs_create_dir+0x59/0x78
       [<ffffffff8034d2e2>] kobject_add+0x114/0x1d8
       [<ffffffff803bb1e7>] class_device_add+0xb5/0x49d
       [<ffffffff804300b1>] netdev_register_sysfs+0x98/0xa2
       [<ffffffff80426f58>] register_netdevice+0x28c/0x376
       [<ffffffff8042709c>] register_netdev+0x5a/0x69
       [<ffffffff8098aa12>] loopback_init+0x4e/0x53
       [<ffffffff8098a918>] net_olddevs_init+0xb/0xb7
       [<ffffffff80270918>] init+0x177/0x348
       [<ffffffff80263cdd>] child_rip+0x7/0x12

-> #1 (rtnl_mutex){--..}:

       [<ffffffff802ab6d5>] lock_acquire+0x4a/0x69
       [<ffffffff80269102>] __mutex_lock_slowpath+0xeb/0x29f
       [<ffffffff802692df>] mutex_lock+0x29/0x2e
       [<ffffffff8042e0a2>] rtnl_lock+0xf/0x12
       [<ffffffff8045c7b8>] ip_mc_leave_group+0x1e/0xae
       [<ffffffff804467f7>] do_ip_setsockopt+0x6ad/0x9b2
       [<ffffffff80446baa>] ip_setsockopt+0x2a/0x84
       [<ffffffff80454a55>] udp_setsockopt+0xd/0x1c
       [<ffffffff8041f7ea>] sock_common_setsockopt+0xe/0x11
       [<ffffffff8041e965>] sys_setsockopt+0x8e/0xb4
       [<ffffffff80262f19>] tracesys+0xd0/0xdb

-> #0 (sk_lock-AF_INET){--..}:

       [<ffffffff802ab6d5>] lock_acquire+0x4a/0x69
       [<ffffffff802371ea>] lock_sock+0xd4/0xe7
       [<ffffffff8022800b>] tcp_sendmsg+0x1e/0xb1a
       [<ffffffff80248f4b>] inet_sendmsg+0x45/0x53
       [<ffffffff80259d25>] sock_sendmsg+0x110/0x130
       [<ffffffff8041f462>] kernel_sendmsg+0x3c/0x52
       [<ffffffff885399e9>] xs_tcp_send_request+0x117/0x320 [sunrpc]
       [<ffffffff885388d5>] xprt_transmit+0x105/0x21e [sunrpc]
       [<ffffffff8853771e>] call_transmit+0x1f4/0x239 [sunrpc]
       [<ffffffff8853c06e>] __rpc_execute+0x9b/0x1e6 [sunrpc]
       [<ffffffff8853c1de>] rpc_execute+0x1a/0x1d [sunrpc]
       [<ffffffff885364ad>] rpc_call_sync+0x87/0xb9 [sunrpc]
       [<ffffffff885a2587>] nfs3_rpc_wrapper+0x2e/0x74 [nfs]
       [<ffffffff885a2a14>] nfs3_proc_lookup+0xe0/0x163 [nfs]
       [<ffffffff88594b10>] nfs_lookup+0xef/0x1d6 [nfs]
       [<ffffffff8020d300>] do_lookup+0xd0/0x18c
       [<ffffffff80209f27>] __link_path_walk+0xa29/0xf7d
       [<ffffffff8020f076>] link_path_walk+0x69/0x101
       [<ffffffff8020a22b>] __link_path_walk+0xd2d/0xf7d
       [<ffffffff8020f076>] link_path_walk+0x69/0x101
       [<ffffffff8020d096>] do_path_lookup+0x27b/0x2e7
       [<ffffffff802258da>] __user_walk_fd+0x40/0x5c
       [<ffffffff8022ae4a>] vfs_stat_fd+0x26/0x5d
       [<ffffffff80225592>] sys_newstat+0x21/0x3c
       [<ffffffff80262f19>] tracesys+0xd0/0xdb

other info that might help us debug this:

1 lock held by mc/4913:
#0: (&inode->i_mutex){--..}, at: [<ffffffff802692e0>] mutex_lock+0x2a/0x2e

stack backtrace:

Call Trace:
 [<ffffffff80271910>] show_trace+0xaa/0x23d
 [<ffffffff80271ab8>] dump_stack+0x15/0x17
 [<ffffffff802a992f>] print_circular_bug_tail+0x6c/0x77
 [<ffffffff802aaf34>] __lock_acquire+0x853/0xa54
 [<ffffffff802ab6d6>] lock_acquire+0x4b/0x69
 [<ffffffff802371eb>] lock_sock+0xd5/0xe7
 [<ffffffff8022800c>] tcp_sendmsg+0x1f/0xb1a
 [<ffffffff80248f4c>] inet_sendmsg+0x46/0x53
 [<ffffffff80259d26>] sock_sendmsg+0x111/0x130
 [<ffffffff8041f463>] kernel_sendmsg+0x3d/0x52
 [<ffffffff885399ea>] :sunrpc:xs_tcp_send_request+0x118/0x320
 [<ffffffff885388d6>] :sunrpc:xprt_transmit+0x106/0x21e
 [<ffffffff8853771f>] :sunrpc:call_transmit+0x1f5/0x239
 [<ffffffff8853c06f>] :sunrpc:__rpc_execute+0x9c/0x1e6
 [<ffffffff8853c1df>] :sunrpc:rpc_execute+0x1b/0x1d
 [<ffffffff885364ae>] :sunrpc:rpc_call_sync+0x88/0xb9
 [<ffffffff885a2588>] :nfs:nfs3_rpc_wrapper+0x2f/0x74
 [<ffffffff885a2a15>] :nfs:nfs3_proc_lookup+0xe1/0x163
 [<ffffffff88594b11>] :nfs:nfs_lookup+0xf0/0x1d6
 [<ffffffff8020d301>] do_lookup+0xd1/0x18c
 [<ffffffff80209f28>] __link_path_walk+0xa2a/0xf7d
 [<ffffffff8020f077>] link_path_walk+0x6a/0x101
 [<ffffffff8020a22c>] __link_path_walk+0xd2e/0xf7d
 [<ffffffff8020f077>] link_path_walk+0x6a/0x101
 [<ffffffff8020d097>] do_path_lookup+0x27c/0x2e7
 [<ffffffff802258db>] __user_walk_fd+0x41/0x5c
 [<ffffffff8022ae4b>] vfs_stat_fd+0x27/0x5d
 [<ffffffff80225593>] sys_newstat+0x22/0x3c
 [<ffffffff80262f1a>] tracesys+0xd1/0xdb
Signature · 1 line
http://www.codemonkey.org.uk
#1 lore raw
Quoted 12 lines · Dave Jones

With 2.6.18rc1 + a selection of the lockdep tweaks found so far, midnight commander makes the kernel unhappy.

Dave

=======================================================
[ INFO: possible circular locking dependency detected ]
-------------------------------------------------------
mc/4913 is trying to acquire lock:
(sk_lock-AF_INET){--..}, at: [<ffffffff8022800c>] tcp_sendmsg+0x1f/0xb1a

Quoted 4 lines

but task is already holding lock:
(&inode->i_mutex){--..}, at: [<ffffffff802692e0>] mutex_lock+0x2a/0x2e

which lock already depends on the new lock.

ok this looks like a fun three-way variant of the traditional AB-BA deadlock... an ABCA deadlock :)

Lock A: rtnl_mutex
Lock B: i_mutex
Lock C: sk_lock for AF_INET

i_mutex is taken within rtln_mutex like this:

       [<ffffffff8030f4a0>] create_dir+0x2c/0x1e2
       [<ffffffff8030fa5b>] sysfs_create_dir+0x59/0x78
       [<ffffffff8034d2e2>] kobject_add+0x114/0x1d8
       [<ffffffff803bb1e7>] class_device_add+0xb5/0x49d
       [<ffffffff804300b1>] netdev_register_sysfs+0x98/0xa2
       [<ffffffff80426f58>] register_netdevice+0x28c/0x376
       [<ffffffff8042709c>] register_netdev+0x5a/0x69

creating the AB dependency

now for the second part:
if you use the IP_DROP_MEMBERSHIP socket option, you end up going into the function do_ip_setsockopt(), which first does a lock_sock(), and then a call to ip_mc_leave_group(), which calls rtnl_lock() So we have the CA dependency right here.

now for the third part, which involves the nfs client:
stat on an nfs file, which ends up taken the i_mutex of a directory in the path (obvious), and then does

       [<ffffffff8022800b>] tcp_sendmsg+0x1e/0xb1a
       [<ffffffff80248f4b>] inet_sendmsg+0x45/0x53
       [<ffffffff80259d25>] sock_sendmsg+0x110/0x130
       [<ffffffff8041f462>] kernel_sendmsg+0x3c/0x52
       [<ffffffff885399e9>] xs_tcp_send_request+0x117/0x320 [sunrpc]
       [<ffffffff885388d5>] xprt_transmit+0x105/0x21e [sunrpc]
       [<ffffffff8853771e>] call_transmit+0x1f4/0x239 [sunrpc]
       [<ffffffff8853c06e>] __rpc_execute+0x9b/0x1e6 [sunrpc]
       [<ffffffff8853c1de>] rpc_execute+0x1a/0x1d [sunrpc]
       [<ffffffff885364ad>] rpc_call_sync+0x87/0xb9 [sunrpc]
       [<ffffffff885a2587>] nfs3_rpc_wrapper+0x2e/0x74 [nfs]
       [<ffffffff885a2a14>] nfs3_proc_lookup+0xe0/0x163 [nfs]

where tcp_sendmsg calls lock_sock. So this is the BC dependency.

In summary, we have an AB dependency, a BC dependency and a CA dependency. This is reason for lockdep to conclude that there is a circular dependency happening, which, looking at my analysis, seems to be at least reasonably right...

#2 lore raw
Quoted 10 lines · Arjan van de Ven

i_mutex is taken within rtln_mutex like this:

      [<ffffffff8030f4a0>] create_dir+0x2c/0x1e2
      [<ffffffff8030fa5b>] sysfs_create_dir+0x59/0x78
      [<ffffffff8034d2e2>] kobject_add+0x114/0x1d8
      [<ffffffff803bb1e7>] class_device_add+0xb5/0x49d
      [<ffffffff804300b1>] netdev_register_sysfs+0x98/0xa2
      [<ffffffff80426f58>] register_netdevice+0x28c/0x376
      [<ffffffff8042709c>] register_netdev+0x5a/0x69

creating the AB dependency

This is a sysfs inode.

Quoted 15 lines

now for the third part, which involves the nfs client:
stat on an nfs file, which ends up taken the i_mutex of a directory in the path (obvious), and then does

      [<ffffffff8022800b>] tcp_sendmsg+0x1e/0xb1a
      [<ffffffff80248f4b>] inet_sendmsg+0x45/0x53
      [<ffffffff80259d25>] sock_sendmsg+0x110/0x130
      [<ffffffff8041f462>] kernel_sendmsg+0x3c/0x52
      [<ffffffff885399e9>] xs_tcp_send_request+0x117/0x320 [sunrpc]
      [<ffffffff885388d5>] xprt_transmit+0x105/0x21e [sunrpc]
      [<ffffffff8853771e>] call_transmit+0x1f4/0x239 [sunrpc]
      [<ffffffff8853c06e>] __rpc_execute+0x9b/0x1e6 [sunrpc]
      [<ffffffff8853c1de>] rpc_execute+0x1a/0x1d [sunrpc]
      [<ffffffff885364ad>] rpc_call_sync+0x87/0xb9 [sunrpc]
      [<ffffffff885a2587>] nfs3_rpc_wrapper+0x2e/0x74 [nfs]
      [<ffffffff885a2a14>] nfs3_proc_lookup+0xe0/0x163 [nfs]
Quoted 1 line

where tcp_sendmsg calls lock_sock. So this is the BC dependency.

This is an nfs inode.

Did I miss something?

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#3 lore raw
Quoted 31 lines · Herbert Xu
Quoted 10 lines · Arjan van de Ven

i_mutex is taken within rtln_mutex like this:

      [<ffffffff8030f4a0>] create_dir+0x2c/0x1e2
      [<ffffffff8030fa5b>] sysfs_create_dir+0x59/0x78
      [<ffffffff8034d2e2>] kobject_add+0x114/0x1d8
      [<ffffffff803bb1e7>] class_device_add+0xb5/0x49d
      [<ffffffff804300b1>] netdev_register_sysfs+0x98/0xa2
      [<ffffffff80426f58>] register_netdevice+0x28c/0x376
      [<ffffffff8042709c>] register_netdev+0x5a/0x69

creating the AB dependency

This is a sysfs inode.

Quoted 15 lines

now for the third part, which involves the nfs client:
stat on an nfs file, which ends up taken the i_mutex of a directory in the path (obvious), and then does

      [<ffffffff8022800b>] tcp_sendmsg+0x1e/0xb1a
      [<ffffffff80248f4b>] inet_sendmsg+0x45/0x53
      [<ffffffff80259d25>] sock_sendmsg+0x110/0x130
      [<ffffffff8041f462>] kernel_sendmsg+0x3c/0x52
      [<ffffffff885399e9>] xs_tcp_send_request+0x117/0x320 [sunrpc]
      [<ffffffff885388d5>] xprt_transmit+0x105/0x21e [sunrpc]
      [<ffffffff8853771e>] call_transmit+0x1f4/0x239 [sunrpc]
      [<ffffffff8853c06e>] __rpc_execute+0x9b/0x1e6 [sunrpc]
      [<ffffffff8853c1de>] rpc_execute+0x1a/0x1d [sunrpc]
      [<ffffffff885364ad>] rpc_call_sync+0x87/0xb9 [sunrpc]
      [<ffffffff885a2587>] nfs3_rpc_wrapper+0x2e/0x74 [nfs]
      [<ffffffff885a2a14>] nfs3_proc_lookup+0xe0/0x163 [nfs]
Quoted 1 line

where tcp_sendmsg calls lock_sock. So this is the BC dependency.

Quoted 3 lines

This is an nfs inode.

Did I miss something?

is it not possible to nfs export /sys, and then mount it over loopback?

#4 lore raw
Quoted 18 lines · Arjan van de Ven
Quoted 18 lines
Quoted 17 lines

now for the third part, which involves the nfs client:
stat on an nfs file, which ends up taken the i_mutex of a directory in the path (obvious), and then does

      [<ffffffff8022800b>] tcp_sendmsg+0x1e/0xb1a
      [<ffffffff80248f4b>] inet_sendmsg+0x45/0x53
      [<ffffffff80259d25>] sock_sendmsg+0x110/0x130
      [<ffffffff8041f462>] kernel_sendmsg+0x3c/0x52
      [<ffffffff885399e9>] xs_tcp_send_request+0x117/0x320 [sunrpc]
      [<ffffffff885388d5>] xprt_transmit+0x105/0x21e [sunrpc]
      [<ffffffff8853771e>] call_transmit+0x1f4/0x239 [sunrpc]
      [<ffffffff8853c06e>] __rpc_execute+0x9b/0x1e6 [sunrpc]
      [<ffffffff8853c1de>] rpc_execute+0x1a/0x1d [sunrpc]
      [<ffffffff885364ad>] rpc_call_sync+0x87/0xb9 [sunrpc]
      [<ffffffff885a2587>] nfs3_rpc_wrapper+0x2e/0x74 [nfs]
      [<ffffffff885a2a14>] nfs3_proc_lookup+0xe0/0x163 [nfs]

where tcp_sendmsg calls lock_sock. So this is the BC dependency.

Quoted 5 lines
Quoted 3 lines

This is an nfs inode.

Did I miss something?

is it not possible to nfs export /sys, and then mount it over loopback?

Possibly, but not with the backtrace above. You'd need an nfs server backtrace to get the real sysfs inode.

In any case, the sock lock from the other backtrace that you had (udp setsockopt) cannot be held by the kernel nfs client or server since the kernel nfs sockets are not visible to user space.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#5 lore raw
In reply to Arjan van de Ven
Quoted 3 lines · Arjan van de Ven
Quoted 1 line

Did I miss something?

is it not possible to nfs export /sys, and then mount it over loopback?

No, it's not. A filesystem needs dedicated routines to support nfs exporting and sysfs doesn't have those.