possible circular locking dependency

The lock dependency checker found this circular lock dependency warning on the 2.6.31 kernel plus some s390 patches. But the problem occurs in common SCSI code in 5 steps:

#4 first acquires scan_mutex in scsi_remove_device,
then sd_ref_mutex in scsi_disk_get_from_dev

#3 first acquires rport_delete_work in run_workqueue (inlined in worker_thread),
then scan_mutex in scsi_remove_device

#2 first acquires fc_host->work_q in run_workqueue,
then rport_delete_work also in run_workqueue

#1 first acquires cpu_add_remove_lock in destroy_workqueue,
then fc_host->work_q in cleanup_workqueue_thread

#0 first acquires sd_ref_mutex in scsi_disk_put,
then cpu_add_remove_lock in destroy_workqueue

I think this is only a theoretical warning which will be very hard or impossible to trigger in reality. But at least the warning should be fixed to keep the lock dependency checker useful.

Does anybody have an idea how to break this dependency chain?

The complete output of the lock dependency checker:

=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.31 #12
-------------------------------------------------------
multipathd/2285 is trying to acquire lock:
(cpu_add_remove_lock){+.+.+.}, at: [<000000000006a38e>] destroy_workqueue+0x3a/0x274

but task is already holding lock:
(sd_ref_mutex){+.+.+.}, at: [<0000000000284202>] scsi_disk_put+0x36/0x5c

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #4 (sd_ref_mutex){+.+.+.}:

       [<0000000000086782>] __lock_acquire+0xe76/0x1940
       [<00000000000872dc>] lock_acquire+0x90/0xb8
       [<000000000046fccc>] mutex_lock_nested+0x80/0x41c
       [<0000000000284190>] scsi_disk_get_from_dev+0x30/0x6c
       [<0000000000284830>] sd_shutdown+0x28/0x160
       [<0000000000284ca4>] sd_remove+0x68/0xac
       [<0000000000257450>] __device_release_driver+0x98/0x108
       [<00000000002575e8>] device_release_driver+0x38/0x48
       [<000000000025674a>] bus_remove_device+0xd6/0x11c
       [<000000000025458c>] device_del+0x160/0x218
       [<0000000000272650>] __scsi_remove_device+0x6c/0xb4
       [<00000000002726da>] scsi_remove_device+0x42/0x54
       [<00000000002727c6>] __scsi_remove_target+0xce/0x108
       [<00000000002728ae>] __remove_child+0x3a/0x4c
       [<0000000000253b0e>] device_for_each_child+0x72/0xbc
       [<000000000027284e>] scsi_remove_target+0x4e/0x74
       [<000000000027929a>] fc_rport_final_delete+0xb2/0x20c
       [<0000000000069ed0>] worker_thread+0x25c/0x318
       [<000000000006ff62>] kthread+0x9a/0xa4
       [<000000000001c952>] kernel_thread_starter+0x6/0xc
       [<000000000001c94c>] kernel_thread_starter+0x0/0xc

-> #3 (&shost->scan_mutex){+.+.+.}:

       [<0000000000086782>] __lock_acquire+0xe76/0x1940
       [<00000000000872dc>] lock_acquire+0x90/0xb8
       [<000000000046fccc>] mutex_lock_nested+0x80/0x41c
       [<00000000002726d0>] scsi_remove_device+0x38/0x54
       [<00000000002727c6>] __scsi_remove_target+0xce/0x108
       [<00000000002728ae>] __remove_child+0x3a/0x4c
       [<0000000000253b0e>] device_for_each_child+0x72/0xbc
       [<000000000027284e>] scsi_remove_target+0x4e/0x74
       [<000000000027929a>] fc_rport_final_delete+0xb2/0x20c
       [<0000000000069ed0>] worker_thread+0x25c/0x318
       [<000000000006ff62>] kthread+0x9a/0xa4
       [<000000000001c952>] kernel_thread_starter+0x6/0xc
       [<000000000001c94c>] kernel_thread_starter+0x0/0xc

-> #2 (&rport->rport_delete_work){+.+.+.}:

       [<0000000000086782>] __lock_acquire+0xe76/0x1940
       [<00000000000872dc>] lock_acquire+0x90/0xb8
       [<0000000000069eca>] worker_thread+0x256/0x318
       [<000000000006ff62>] kthread+0x9a/0xa4
       [<000000000001c952>] kernel_thread_starter+0x6/0xc
       [<000000000001c94c>] kernel_thread_starter+0x0/0xc

-> #1 ((fc_host->work_q_name)){+.+.+.}:

       [<0000000000086782>] __lock_acquire+0xe76/0x1940
       [<00000000000872dc>] lock_acquire+0x90/0xb8
       [<000000000006a2ae>] cleanup_workqueue_thread+0x62/0xac
       [<000000000006a420>] destroy_workqueue+0xcc/0x274
       [<0000000000279c4a>] fc_remove_host+0x1de/0x210
       [<000000000034556e>] zfcp_adapter_scsi_unregister+0x96/0xc4
       [<0000000000343df0>] zfcp_ccw_remove+0x9c/0x370
       [<00000000002c2a6a>] ccw_device_remove+0x3e/0x1a8
       [<0000000000257450>] __device_release_driver+0x98/0x108
       [<00000000002575e8>] device_release_driver+0x38/0x48
       [<000000000025674a>] bus_remove_device+0xd6/0x11c
       [<000000000025458c>] device_del+0x160/0x218
       [<00000000002c3404>] ccw_device_unregister+0x5c/0x7c
       [<00000000002c3490>] io_subchannel_remove+0x6c/0x9c
       [<00000000002be32e>] css_remove+0x3e/0x7c
       [<0000000000257450>] __device_release_driver+0x98/0x108
       [<00000000002575e8>] device_release_driver+0x38/0x48
       [<000000000025674a>] bus_remove_device+0xd6/0x11c
       [<000000000025458c>] device_del+0x160/0x218
       [<000000000025466a>] device_unregister+0x26/0x38
       [<00000000002be4bc>] css_sch_device_unregister+0x44/0x54
       [<00000000002c435e>] ccw_device_call_sch_unregister+0x4e/0x78
       [<0000000000069ed0>] worker_thread+0x25c/0x318
       [<000000000006ff62>] kthread+0x9a/0xa4
       [<000000000001c952>] kernel_thread_starter+0x6/0xc
       [<000000000001c94c>] kernel_thread_starter+0x0/0xc

-> #0 (cpu_add_remove_lock){+.+.+.}:

       [<0000000000086e5a>] __lock_acquire+0x154e/0x1940
       [<00000000000872dc>] lock_acquire+0x90/0xb8
       [<000000000046fccc>] mutex_lock_nested+0x80/0x41c
       [<000000000006a38e>] destroy_workqueue+0x3a/0x274
       [<0000000000265bb0>] scsi_host_dev_release+0x88/0x104
       [<000000000025396a>] device_release+0x36/0xa0
       [<000000000022ae92>] kobject_release+0x62/0xa8
       [<000000000022c11c>] kref_put+0x74/0x94
       [<00000000002771cc>] fc_rport_dev_release+0x2c/0x40
       [<000000000025396a>] device_release+0x36/0xa0
       [<000000000022ae92>] kobject_release+0x62/0xa8
       [<000000000022c11c>] kref_put+0x74/0x94
       [<000000000025396a>] device_release+0x36/0xa0
       [<000000000022ae92>] kobject_release+0x62/0xa8
       [<000000000022c11c>] kref_put+0x74/0x94
       [<000000000006ba9c>] execute_in_process_context+0xa4/0xbc
       [<000000000025396a>] device_release+0x36/0xa0
       [<000000000022ae92>] kobject_release+0x62/0xa8
       [<000000000022c11c>] kref_put+0x74/0x94
       [<0000000000284216>] scsi_disk_put+0x4a/0x5c
       [<0000000000285560>] sd_release+0x6c/0x108
       [<0000000000126364>] __blkdev_put+0x1b8/0x1cc
       [<00000000000f224e>] __fput+0x12a/0x240
       [<00000000000ee4c0>] filp_close+0x78/0xa8
       [<00000000000ee5d0>] SyS_close+0xe0/0x148
       [<000000000002a042>] sysc_noemu+0x10/0x16
       [<0000020000041160>] 0x20000041160

other info that might help us debug this:

2 locks held by multipathd/2285:
#0: (&bdev->bd_mutex){+.+.+.}, at: [<00000000001261f2>] __blkdev_put+0x46/0x1cc #1: (sd_ref_mutex){+.+.+.}, at: [<0000000000284202>] scsi_disk_put+0x36/0x5c

stack backtrace:
CPU: 1 Not tainted 2.6.31 #12
Process multipathd (pid: 2285, task: 000000002d87b900, ksp: 000000002eca7800)
0000000000000000 000000002eca7770 0000000000000002 0000000000000000 
       000000002eca7810 000000002eca7788 000000002eca7788 000000000046db82 
       0000000000000000 0000000000000001 000000002d87bfd0 0000000000000000 
       000000000000000d 0000000000000000 000000002eca77d8 000000000000000e 
       000000000047fc30 0000000000017d80 000000002eca7770 000000002eca77b8 
Call Trace:
([<0000000000017c82>] show_trace+0xee/0x144)
 [<000000000008532e>] print_circular_bug_tail+0x10a/0x110
 [<0000000000086e5a>] __lock_acquire+0x154e/0x1940
 [<00000000000872dc>] lock_acquire+0x90/0xb8
 [<000000000046fccc>] mutex_lock_nested+0x80/0x41c
 [<000000000006a38e>] destroy_workqueue+0x3a/0x274
 [<0000000000265bb0>] scsi_host_dev_release+0x88/0x104
 [<000000000025396a>] device_release+0x36/0xa0
 [<000000000022ae92>] kobject_release+0x62/0xa8
 [<000000000022c11c>] kref_put+0x74/0x94
 [<00000000002771cc>] fc_rport_dev_release+0x2c/0x40
 [<000000000025396a>] device_release+0x36/0xa0
 [<000000000022ae92>] kobject_release+0x62/0xa8
 [<000000000022c11c>] kref_put+0x74/0x94
 [<000000000025396a>] device_release+0x36/0xa0
 [<000000000022ae92>] kobject_release+0x62/0xa8
 [<000000000022c11c>] kref_put+0x74/0x94
 [<000000000006ba9c>] execute_in_process_context+0xa4/0xbc
 [<000000000025396a>] device_release+0x36/0xa0
 [<000000000022ae92>] kobject_release+0x62/0xa8
 [<000000000022c11c>] kref_put+0x74/0x94
 [<0000000000284216>] scsi_disk_put+0x4a/0x5c
 [<0000000000285560>] sd_release+0x6c/0x108
 [<0000000000126364>] __blkdev_put+0x1b8/0x1cc
 [<00000000000f224e>] __fput+0x12a/0x240
 [<00000000000ee4c0>] filp_close+0x78/0xa8
 [<00000000000ee5d0>] SyS_close+0xe0/0x148
 [<000000000002a042>] sysc_noemu+0x10/0x16
 [<0000020000041160>] 0x20000041160

INFO: lockdep is turned off.

--
Christof Schmitt

This still happens with 2.6.32. I think it boils down to:

#4: The work function acquiring the sd_ref_mutex gives:

    cpu_add_remove_lock -> sd_ref_mutex

#0: Calling destroy_workqueue from scsi_host_dev_release introduces

    the dependency
    sd_ref_mutex -> cpu_add_remove_lock

But the sd_ref_mutex is required for the scsi_disk references. So far, i don't see a good way to approach this.