Skip to contentSkip to conversation

fc_remove_host / scsi_host_dev_release deadlock

DISCUSSION Joe Eykholt 1 participant 3 messages Aug 7, 2009 SCSI

lore
Share & tools
lore

Joe Eykholt

author
#0 lore raw

This is different from the 3-way deadlock I posted on 7/25
http://marc.info/?l=linux-scsi&m=124856078500527&w=2

A user process is closing an sd block dev file.
It holds the sd_ref_mutex and is in scsi_host_dev_release() doing a destroy_workqueue() which needs the cpu_add_remove_lock.

The process that holds cpu_add_remove_lock is doing fcoe_destroy (deleting an FCoE instance) which calls fc_remove_host() which does destroy_workqueue() and waiting in flush_cpu_workqueue() for a work item to finish.

The stuck work item is doing fc_rport_final_delete(), and that's in device_release_driver() in sd_remove() which is waiting for the sd_ref_mutex.

I'd like to hear some discussion about the right approach to fix this. There should be some clear rules about how work items can destroy or flush other workqueues. I'd like to be able to state the clear rule that's being violated here.

Perhaps the way cpu_add_remove_lock is used to in destroy_workqueue() should be changed, or maybe fc_rport_final_delete and fc_remove_host need to be better about using workqueues somehow.

I do see a lock dependency warning about this, so that system works. I saw this once on a system based on 2.6.31-rc4 under the current fcoe-next.git tree.

The following are the stacks I collected via /proc/*/stacks and the lockdep warnings.

Thanks,
Joe

Name:   hald-probe-volu
State:  D (disk sleep)
cmd:    /usr/libexec/hald-probe-volume
wchan:  cpu_maps_update_begin

--- waiting cpu_add_remove_lock
--- has sd_ref_mutex in scsi_disk_put

[<ffffffff81042bd2>] cpu_maps_update_begin+0x12/0x14
[<ffffffff81052ee0>] destroy_workqueue+0x2b/0x9e
[<ffffffff8129ef4f>] scsi_host_dev_release+0x5a/0xbd
[<ffffffff81291941>] device_release+0x49/0x75
[<ffffffff811d39d8>] kobject_release+0x51/0x67
[<ffffffff811d4789>] kref_put+0x43/0x4f
[<ffffffff811d38e1>] kobject_put+0x47/0x4b
[<ffffffff812912f2>] put_device+0x12/0x14
[<ffffffffa004826b>] fc_rport_dev_release+0x18/0x24 [scsi_transport_fc]
[<ffffffff81291941>] device_release+0x49/0x75
[<ffffffff811d39d8>] kobject_release+0x51/0x67
[<ffffffff811d4789>] kref_put+0x43/0x4f
[<ffffffff811d38e1>] kobject_put+0x47/0x4b
[<ffffffff812912f2>] put_device+0x12/0x14
[<ffffffff812a4d44>] scsi_target_dev_release+0x1d/0x21
[<ffffffff81291941>] device_release+0x49/0x75
[<ffffffff811d39d8>] kobject_release+0x51/0x67
[<ffffffff811d4789>] kref_put+0x43/0x4f
[<ffffffff811d38e1>] kobject_put+0x47/0x4b
[<ffffffff812912f2>] put_device+0x12/0x14
[<ffffffff812a7709>] scsi_device_dev_release_usercontext+0x118/0x124
[<ffffffff81053894>] execute_in_process_context+0x2a/0x70
[<ffffffff812a75ef>] scsi_device_dev_release+0x17/0x19
[<ffffffff81291941>] device_release+0x49/0x75
[<ffffffff811d39d8>] kobject_release+0x51/0x67
[<ffffffff811d4789>] kref_put+0x43/0x4f
[<ffffffff811d38e1>] kobject_put+0x47/0x4b
[<ffffffff812912f2>] put_device+0x12/0x14
[<ffffffff8129dfd5>] scsi_device_put+0x3d/0x42
[<ffffffff812acb57>] scsi_disk_put+0x30/0x41
[<ffffffff812ad9c3>] sd_release+0x4d/0x54
[<ffffffff810f9ca0>] __blkdev_put+0xa7/0x16e
[<ffffffff810f9d57>] __blkdev_put+0x15e/0x16e
[<ffffffff810f9d72>] blkdev_put+0xb/0xd
[<ffffffff810f9dab>] blkdev_close+0x37/0x3c
[<ffffffff810d6450>] __fput+0x10d/0x1bb
[<ffffffff810d6516>] fput+0x18/0x1a
[<ffffffff810d35b0>] filp_close+0x67/0x72
[<ffffffff810d3660>] sys_close+0xa5/0xe4
[<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

Name:   fcoeadm
State:  D (disk sleep)
cmd:    fcoeadm -d eth4
wchan:  flush_cpu_workqueue

--- waits on cpu work to finish. -- fc_workqueue 28596 --- holds cpu_add_remove_lock

[<ffffffff81052dbd>] flush_cpu_workqueue+0x7b/0x87
[<ffffffff81052e33>] cleanup_workqueue_thread+0x6a/0xb8
[<ffffffff81052f18>] destroy_workqueue+0x63/0x9e
[<ffffffffa004a5a7>] fc_remove_host+0x148/0x171 [scsi_transport_fc]
[<ffffffffa03e48b2>] fcoe_if_destroy+0x1a6/0x1db [fcoe]
[<ffffffffa03e6001>] fcoe_destroy+0x86/0xad [fcoe]
[<ffffffff81055034>] param_attr_store+0x25/0x35
[<ffffffff81055089>] module_attr_store+0x21/0x25
[<ffffffff811279fe>] sysfs_write_file+0xe4/0x119
[<ffffffff810d5a12>] vfs_write+0xab/0x105
[<ffffffff810d5b30>] sys_write+0x47/0x6f
[<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

Name:   fc_wq_137
State:  D (disk sleep)

cmd:
wchan: scsi_disk_get_from_dev

-- waits on sd_ref_mutex

[<ffffffff812acaf8>] scsi_disk_get_from_dev+0x1a/0x49
[<ffffffff812acfdd>] sd_shutdown+0x12/0x117
[<ffffffff812ad20f>] sd_remove+0x51/0x8a
[<ffffffff8129443e>] __device_release_driver+0x80/0xc9
[<ffffffff81294552>] device_release_driver+0x1e/0x2b
[<ffffffff81293ae6>] bus_remove_device+0xa8/0xc9
[<ffffffff812921d6>] device_del+0x13f/0x1ac
[<ffffffff812a7a58>] __scsi_remove_device+0x44/0x81
[<ffffffff812a7abb>] scsi_remove_device+0x26/0x33
[<ffffffff812a7b6d>] __scsi_remove_target+0x93/0xd7
[<ffffffff812a7c17>] __remove_child+0x1e/0x25
[<ffffffff81291a22>] device_for_each_child+0x38/0x6f
[<ffffffff812a7bec>] scsi_remove_target+0x3b/0x48
[<ffffffffa0049db7>] fc_starget_delete+0x21/0x25 [scsi_transport_fc]
[<ffffffffa0049eb1>] fc_rport_final_delete+0xf6/0x188 [scsi_transport_fc]
[<ffffffff810527ec>] worker_thread+0x1fa/0x30a
[<ffffffff81056c2d>] kthread+0x88/0x90
[<ffffffff8100cbfa>] child_rip+0xa/0x20

[<ffffffffffffffff>] 0xffffffffffffffff

[ INFO: possible circular locking dependency detected ]
2.6.31-rc4-rp9 #5
-------------------------------------------------------
hald-probe-volu/23829 is trying to acquire lock:
(cpu_add_remove_lock){+.+.+.}, at: [<ffffffff81042bd2>] cpu_maps_update_begin+0x12/0x14

but task is already holding lock:
(sd_ref_mutex){+.+.+.}, at: [<ffffffff812acb47>] scsi_disk_put+0x20/0x41

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #4 (sd_ref_mutex){+.+.+.}:

        [<ffffffff8106700c>] __lock_acquire+0xa48/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff8150d536>] __mutex_lock_common+0x48/0x328
        [<ffffffff8150d88c>] mutex_lock_nested+0x24/0x29
        [<ffffffff812acaf8>] scsi_disk_get_from_dev+0x1a/0x49
        [<ffffffff812acfdd>] sd_shutdown+0x12/0x117
        [<ffffffff812ad20f>] sd_remove+0x51/0x8a
        [<ffffffff8129443e>] __device_release_driver+0x80/0xc9
        [<ffffffff81294552>] device_release_driver+0x1e/0x2b
        [<ffffffff81293ae6>] bus_remove_device+0xa8/0xc9
        [<ffffffff812921d6>] device_del+0x13f/0x1ac
        [<ffffffff812a7a58>] __scsi_remove_device+0x44/0x81
        [<ffffffff812a7abb>] scsi_remove_device+0x26/0x33
        [<ffffffff812a7b6d>] __scsi_remove_target+0x93/0xd7
        [<ffffffff812a7c17>] __remove_child+0x1e/0x25
        [<ffffffff81291a22>] device_for_each_child+0x38/0x6f
        [<ffffffff812a7bec>] scsi_remove_target+0x3b/0x48
        [<ffffffffa0049db7>] fc_starget_delete+0x21/0x25 [scsi_transport_fc]
        [<ffffffffa0049eb1>] fc_rport_final_delete+0xf6/0x188 [scsi_transport_fc]
        [<ffffffff810527ec>] worker_thread+0x1fa/0x30a
        [<ffffffff81056c2d>] kthread+0x88/0x90
        [<ffffffff8100cbfa>] child_rip+0xa/0x20
        [<ffffffffffffffff>] 0xffffffffffffffff

-> #3 (&shost->scan_mutex){+.+.+.}:

        [<ffffffff8106700c>] __lock_acquire+0xa48/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff8150d536>] __mutex_lock_common+0x48/0x328
        [<ffffffff8150d88c>] mutex_lock_nested+0x24/0x29
        [<ffffffff812a7ab3>] scsi_remove_device+0x1e/0x33
        [<ffffffff812a7b6d>] __scsi_remove_target+0x93/0xd7
        [<ffffffff812a7c17>] __remove_child+0x1e/0x25
        [<ffffffff81291a22>] device_for_each_child+0x38/0x6f
        [<ffffffff812a7bec>] scsi_remove_target+0x3b/0x48
        [<ffffffffa0049db7>] fc_starget_delete+0x21/0x25 [scsi_transport_fc]
        [<ffffffffa0049eb1>] fc_rport_final_delete+0xf6/0x188 [scsi_transport_fc]
        [<ffffffff810527ec>] worker_thread+0x1fa/0x30a
        [<ffffffff81056c2d>] kthread+0x88/0x90
        [<ffffffff8100cbfa>] child_rip+0xa/0x20
        [<ffffffffffffffff>] 0xffffffffffffffff

-> #2 (&rport->rport_delete_work){+.+.+.}:

        [<ffffffff8106700c>] __lock_acquire+0xa48/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff810527e3>] worker_thread+0x1f1/0x30a
        [<ffffffff81056c2d>] kthread+0x88/0x90
        [<ffffffff8100cbfa>] child_rip+0xa/0x20
        [<ffffffffffffffff>] 0xffffffffffffffff

-> #1 ((fc_host->work_q_name)){+.+.+.}:

        [<ffffffff8106700c>] __lock_acquire+0xa48/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff81052e0e>] cleanup_workqueue_thread+0x45/0xb8
        [<ffffffff81052f18>] destroy_workqueue+0x63/0x9e
        [<ffffffffa004a5a7>] fc_remove_host+0x148/0x171 [scsi_transport_fc]
        [<ffffffffa00738b2>] 0xffffffffa00738b2
        [<ffffffffa0075001>] 0xffffffffa0075001
        [<ffffffff81055034>] param_attr_store+0x25/0x35
        [<ffffffff81055089>] module_attr_store+0x21/0x25
        [<ffffffff811279fe>] sysfs_write_file+0xe4/0x119
        [<ffffffff810d5a12>] vfs_write+0xab/0x105
        [<ffffffff810d5b30>] sys_write+0x47/0x6f
        [<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b
        [<ffffffffffffffff>] 0xffffffffffffffff

-> #0 (cpu_add_remove_lock){+.+.+.}:

        [<ffffffff81066f00>] __lock_acquire+0x93c/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff8150d536>] __mutex_lock_common+0x48/0x328
        [<ffffffff8150d88c>] mutex_lock_nested+0x24/0x29
        [<ffffffff81042bd2>] cpu_maps_update_begin+0x12/0x14
        [<ffffffff81052ee0>] destroy_workqueue+0x2b/0x9e
        [<ffffffff8129ef4f>] scsi_host_dev_release+0x5a/0xbd
        [<ffffffff81291941>] device_release+0x49/0x75
        [<ffffffff811d39d8>] kobject_release+0x51/0x67
        [<ffffffff811d4789>] kref_put+0x43/0x4f
        [<ffffffff811d38e1>] kobject_put+0x47/0x4b
        [<ffffffff812912f2>] put_device+0x12/0x14
        [<ffffffffa004826b>] fc_rport_dev_release+0x18/0x24 [scsi_transport_fc]
        [<ffffffff81291941>] device_release+0x49/0x75
        [<ffffffff811d39d8>] kobject_release+0x51/0x67
        [<ffffffff811d4789>] kref_put+0x43/0x4f
        [<ffffffff811d38e1>] kobject_put+0x47/0x4b
        [<ffffffff812912f2>] put_device+0x12/0x14
        [<ffffffff812a4d44>] scsi_target_dev_release+0x1d/0x21
        [<ffffffff81291941>] device_release+0x49/0x75
        [<ffffffff811d39d8>] kobject_release+0x51/0x67
        [<ffffffff811d4789>] kref_put+0x43/0x4f
        [<ffffffff811d38e1>] kobject_put+0x47/0x4b
        [<ffffffff812912f2>] put_device+0x12/0x14
        [<ffffffff812a7709>] scsi_device_dev_release_usercontext+0x118/0x124
        [<ffffffff81053894>] execute_in_process_context+0x2a/0x70
        [<ffffffff812a75ef>] scsi_device_dev_release+0x17/0x19
        [<ffffffff81291941>] device_release+0x49/0x75
        [<ffffffff811d39d8>] kobject_release+0x51/0x67
        [<ffffffff811d4789>] kref_put+0x43/0x4f
        [<ffffffff811d38e1>] kobject_put+0x47/0x4b
        [<ffffffff812912f2>] put_device+0x12/0x14
        [<ffffffff8129dfd5>] scsi_device_put+0x3d/0x42
        [<ffffffff812acb57>] scsi_disk_put+0x30/0x41
        [<ffffffff812ad9c3>] sd_release+0x4d/0x54
        [<ffffffff810f9ca0>] __blkdev_put+0xa7/0x16e
        [<ffffffff810f9d57>] __blkdev_put+0x15e/0x16e
        [<ffffffff810f9d72>] blkdev_put+0xb/0xd
        [<ffffffff810f9dab>] blkdev_close+0x37/0x3c
        [<ffffffff810d6450>] __fput+0x10d/0x1bb
        [<ffffffff810d6516>] fput+0x18/0x1a
        [<ffffffff810d35b0>] filp_close+0x67/0x72
        [<ffffffff810d3660>] sys_close+0xa5/0xe4
        [<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b
        [<ffffffffffffffff>] 0xffffffffffffffff

other info that might help us debug this:

2 locks held by hald-probe-volu/23829:
#0: (&bdev->bd_mutex/1){+.+.+.}, at: [<ffffffff810f9c27>] __blkdev_put+0x2e/0x16e #1: (sd_ref_mutex){+.+.+.}, at: [<ffffffff812acb47>] scsi_disk_put+0x20/0x41

 stack backtrace:
 Pid: 23829, comm: hald-probe-volu Not tainted 2.6.31-rc4-rp9 #5
 Call Trace:
  [<ffffffff8106624e>] print_circular_bug_tail+0x71/0x7c
  [<ffffffff81066f00>] __lock_acquire+0x93c/0xbd0
  [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
  [<ffffffff81042bd2>] ? cpu_maps_update_begin+0x12/0x14
  [<ffffffff8150d536>] __mutex_lock_common+0x48/0x328
  [<ffffffff81042bd2>] ? cpu_maps_update_begin+0x12/0x14
  [<ffffffff81065c1d>] ? trace_hardirqs_on+0xd/0xf
  [<ffffffff8150e64c>] ? _spin_unlock_irq+0x2b/0x30
  [<ffffffff81042bd2>] ? cpu_maps_update_begin+0x12/0x14
  [<ffffffff8150ce9e>] ? wait_for_common+0xf7/0x112
  [<ffffffff8103c93f>] ? default_wake_function+0x0/0xf
  [<ffffffff8150d88c>] mutex_lock_nested+0x24/0x29
  [<ffffffff81042bd2>] cpu_maps_update_begin+0x12/0x14
  [<ffffffff81052ee0>] destroy_workqueue+0x2b/0x9e
  [<ffffffff8129ef4f>] scsi_host_dev_release+0x5a/0xbd
  [<ffffffff81291941>] device_release+0x49/0x75
  [<ffffffff811d39d8>] kobject_release+0x51/0x67
  [<ffffffff811d3987>] ? kobject_release+0x0/0x67
  [<ffffffff811d4789>] kref_put+0x43/0x4f
  [<ffffffff811d38e1>] kobject_put+0x47/0x4b
  [<ffffffff812912f2>] put_device+0x12/0x14
  [<ffffffffa004826b>] fc_rport_dev_release+0x18/0x24 [scsi_transport_fc]
  [<ffffffff81291941>] device_release+0x49/0x75
  [<ffffffff811d39d8>] kobject_release+0x51/0x67
  [<ffffffff811d3987>] ? kobject_release+0x0/0x67
  [<ffffffff811d4789>] kref_put+0x43/0x4f
  [<ffffffff811d38e1>] kobject_put+0x47/0x4b
  [<ffffffff812912f2>] put_device+0x12/0x14
  [<ffffffff812a4d44>] scsi_target_dev_release+0x1d/0x21
  [<ffffffff81291941>] device_release+0x49/0x75
  [<ffffffff811d39d8>] kobject_release+0x51/0x67
  [<ffffffff811d3987>] ? kobject_release+0x0/0x67
  [<ffffffff811d4789>] kref_put+0x43/0x4f
  [<ffffffff811d38e1>] kobject_put+0x47/0x4b
  [<ffffffff812912f2>] put_device+0x12/0x14
  [<ffffffff812a7709>] scsi_device_dev_release_usercontext+0x118/0x124
  [<ffffffff812a75f1>] ? scsi_device_dev_release_usercontext+0x0/0x124
  [<ffffffff81053894>] execute_in_process_context+0x2a/0x70
  [<ffffffff812a75ef>] scsi_device_dev_release+0x17/0x19
  [<ffffffff81291941>] device_release+0x49/0x75
  [<ffffffff811d39d8>] kobject_release+0x51/0x67
  [<ffffffff811d3987>] ? kobject_release+0x0/0x67
  [<ffffffff811d4789>] kref_put+0x43/0x4f
  [<ffffffff811d38e1>] kobject_put+0x47/0x4b
  [<ffffffff812912f2>] put_device+0x12/0x14
  [<ffffffff8129dfd5>] scsi_device_put+0x3d/0x42
  [<ffffffff812acb57>] scsi_disk_put+0x30/0x41
  [<ffffffff812ad9c3>] sd_release+0x4d/0x54
  [<ffffffff810f9ca0>] __blkdev_put+0xa7/0x16e
  [<ffffffff810f9d57>] __blkdev_put+0x15e/0x16e
  [<ffffffff810f9d72>] blkdev_put+0xb/0xd
  [<ffffffff810f9dab>] blkdev_close+0x37/0x3c
  [<ffffffff810d6450>] __fput+0x10d/0x1bb
  [<ffffffff810d6516>] fput+0x18/0x1a
  [<ffffffff810d35b0>] filp_close+0x67/0x72
  [<ffffffff810d3660>] sys_close+0xa5/0xe4
  [<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b

-- end

author
#1 lore raw
Quoted 16 lines · Joe Eykholt

This is different from the 3-way deadlock I posted on 7/25
http://marc.info/?l=linux-scsi&m=124856078500527&w=2

A user process is closing an sd block dev file.
It holds the sd_ref_mutex and is in scsi_host_dev_release() doing a destroy_workqueue() which needs the cpu_add_remove_lock.

The process that holds cpu_add_remove_lock is doing fcoe_destroy (deleting an FCoE instance) which calls fc_remove_host() which does destroy_workqueue() and waiting in flush_cpu_workqueue() for a work item to finish.

The stuck work item is doing fc_rport_final_delete(), and that's in device_release_driver() in sd_remove() which is waiting for the sd_ref_mutex.

Quoted 5 lines

I'd like to hear some discussion about the right approach to fix this. There should be some clear rules about how work items can destroy or flush other workqueues. I'd like to be able to state the clear rule that's being violated here.

Now it seems to me that that the rule is:

A workqueue item may not destroy a workqueue if it is running on a workqueue that itself may be destroyed.

That's because destroy_workqueue takes the cpu_add_remove lock and, while holding it, does flush_cpu_workqueue().

If a work item on workqueue A destroy workqueue B, and thread C destroys workqueue A, then thread C may get the cpu_add_remove lock first, work item on A can wait forever for the lock, and thread C can hang waiting for workqueue A to flush.

So we're violating that rule if we let fc_rport_final_delete()
do the destroy of the device's workqueue. I still don't
have a good solution. Maybe fc_host_remove() could flush the workqueue before destroying it?

Maybe destroy_workqueue() should flush before grabbing the cpu_add_remove_lock, but maybe that doesn't completely fix it.

Joe

Quoted 325 lines

Perhaps the way cpu_add_remove_lock is used to in
destroy_workqueue() should be changed, or maybe
fc_rport_final_delete and fc_remove_host need to
be better about using workqueues somehow.

I do see a lock dependency warning about this, so that system works. I saw this once on a system based on 2.6.31-rc4 under the current fcoe-next.git tree.

The following are the stacks I collected via /proc/*/stacks and the lockdep warnings.

Thanks,
Joe

Name:   hald-probe-volu
State:  D (disk sleep)
cmd:    /usr/libexec/hald-probe-volume
wchan:  cpu_maps_update_begin

--- waiting cpu_add_remove_lock
--- has sd_ref_mutex in scsi_disk_put

[<ffffffff81042bd2>] cpu_maps_update_begin+0x12/0x14
[<ffffffff81052ee0>] destroy_workqueue+0x2b/0x9e
[<ffffffff8129ef4f>] scsi_host_dev_release+0x5a/0xbd
[<ffffffff81291941>] device_release+0x49/0x75
[<ffffffff811d39d8>] kobject_release+0x51/0x67
[<ffffffff811d4789>] kref_put+0x43/0x4f
[<ffffffff811d38e1>] kobject_put+0x47/0x4b
[<ffffffff812912f2>] put_device+0x12/0x14
[<ffffffffa004826b>] fc_rport_dev_release+0x18/0x24 [scsi_transport_fc]
[<ffffffff81291941>] device_release+0x49/0x75
[<ffffffff811d39d8>] kobject_release+0x51/0x67
[<ffffffff811d4789>] kref_put+0x43/0x4f
[<ffffffff811d38e1>] kobject_put+0x47/0x4b
[<ffffffff812912f2>] put_device+0x12/0x14
[<ffffffff812a4d44>] scsi_target_dev_release+0x1d/0x21
[<ffffffff81291941>] device_release+0x49/0x75
[<ffffffff811d39d8>] kobject_release+0x51/0x67
[<ffffffff811d4789>] kref_put+0x43/0x4f
[<ffffffff811d38e1>] kobject_put+0x47/0x4b
[<ffffffff812912f2>] put_device+0x12/0x14
[<ffffffff812a7709>] scsi_device_dev_release_usercontext+0x118/0x124
[<ffffffff81053894>] execute_in_process_context+0x2a/0x70
[<ffffffff812a75ef>] scsi_device_dev_release+0x17/0x19
[<ffffffff81291941>] device_release+0x49/0x75
[<ffffffff811d39d8>] kobject_release+0x51/0x67
[<ffffffff811d4789>] kref_put+0x43/0x4f
[<ffffffff811d38e1>] kobject_put+0x47/0x4b
[<ffffffff812912f2>] put_device+0x12/0x14
[<ffffffff8129dfd5>] scsi_device_put+0x3d/0x42
[<ffffffff812acb57>] scsi_disk_put+0x30/0x41
[<ffffffff812ad9c3>] sd_release+0x4d/0x54
[<ffffffff810f9ca0>] __blkdev_put+0xa7/0x16e
[<ffffffff810f9d57>] __blkdev_put+0x15e/0x16e
[<ffffffff810f9d72>] blkdev_put+0xb/0xd
[<ffffffff810f9dab>] blkdev_close+0x37/0x3c
[<ffffffff810d6450>] __fput+0x10d/0x1bb
[<ffffffff810d6516>] fput+0x18/0x1a
[<ffffffff810d35b0>] filp_close+0x67/0x72
[<ffffffff810d3660>] sys_close+0xa5/0xe4
[<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

Name:   fcoeadm
State:  D (disk sleep)
cmd:    fcoeadm -d eth4
wchan:  flush_cpu_workqueue

--- waits on cpu work to finish. -- fc_workqueue 28596 --- holds cpu_add_remove_lock

[<ffffffff81052dbd>] flush_cpu_workqueue+0x7b/0x87
[<ffffffff81052e33>] cleanup_workqueue_thread+0x6a/0xb8
[<ffffffff81052f18>] destroy_workqueue+0x63/0x9e
[<ffffffffa004a5a7>] fc_remove_host+0x148/0x171 [scsi_transport_fc]
[<ffffffffa03e48b2>] fcoe_if_destroy+0x1a6/0x1db [fcoe]
[<ffffffffa03e6001>] fcoe_destroy+0x86/0xad [fcoe]
[<ffffffff81055034>] param_attr_store+0x25/0x35
[<ffffffff81055089>] module_attr_store+0x21/0x25
[<ffffffff811279fe>] sysfs_write_file+0xe4/0x119
[<ffffffff810d5a12>] vfs_write+0xab/0x105
[<ffffffff810d5b30>] sys_write+0x47/0x6f
[<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

Name:   fc_wq_137
State:  D (disk sleep)

cmd:
wchan: scsi_disk_get_from_dev

-- waits on sd_ref_mutex

[<ffffffff812acaf8>] scsi_disk_get_from_dev+0x1a/0x49
[<ffffffff812acfdd>] sd_shutdown+0x12/0x117
[<ffffffff812ad20f>] sd_remove+0x51/0x8a
[<ffffffff8129443e>] __device_release_driver+0x80/0xc9
[<ffffffff81294552>] device_release_driver+0x1e/0x2b
[<ffffffff81293ae6>] bus_remove_device+0xa8/0xc9
[<ffffffff812921d6>] device_del+0x13f/0x1ac
[<ffffffff812a7a58>] __scsi_remove_device+0x44/0x81
[<ffffffff812a7abb>] scsi_remove_device+0x26/0x33
[<ffffffff812a7b6d>] __scsi_remove_target+0x93/0xd7
[<ffffffff812a7c17>] __remove_child+0x1e/0x25
[<ffffffff81291a22>] device_for_each_child+0x38/0x6f
[<ffffffff812a7bec>] scsi_remove_target+0x3b/0x48
[<ffffffffa0049db7>] fc_starget_delete+0x21/0x25 [scsi_transport_fc]
[<ffffffffa0049eb1>] fc_rport_final_delete+0xf6/0x188 [scsi_transport_fc]
[<ffffffff810527ec>] worker_thread+0x1fa/0x30a
[<ffffffff81056c2d>] kthread+0x88/0x90
[<ffffffff8100cbfa>] child_rip+0xa/0x20

[<ffffffffffffffff>] 0xffffffffffffffff

[ INFO: possible circular locking dependency detected ] 2.6.31-rc4-rp9 #5
-------------------------------------------------------
hald-probe-volu/23829 is trying to acquire lock:
(cpu_add_remove_lock){+.+.+.}, at: [<ffffffff81042bd2>]
cpu_maps_update_begin+0x12/0x14

but task is already holding lock:
(sd_ref_mutex){+.+.+.}, at: [<ffffffff812acb47>] scsi_disk_put+0x20/0x41

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #4 (sd_ref_mutex){+.+.+.}:

        [<ffffffff8106700c>] __lock_acquire+0xa48/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff8150d536>] __mutex_lock_common+0x48/0x328
        [<ffffffff8150d88c>] mutex_lock_nested+0x24/0x29
        [<ffffffff812acaf8>] scsi_disk_get_from_dev+0x1a/0x49
        [<ffffffff812acfdd>] sd_shutdown+0x12/0x117
        [<ffffffff812ad20f>] sd_remove+0x51/0x8a
        [<ffffffff8129443e>] __device_release_driver+0x80/0xc9
        [<ffffffff81294552>] device_release_driver+0x1e/0x2b
        [<ffffffff81293ae6>] bus_remove_device+0xa8/0xc9
        [<ffffffff812921d6>] device_del+0x13f/0x1ac
        [<ffffffff812a7a58>] __scsi_remove_device+0x44/0x81
        [<ffffffff812a7abb>] scsi_remove_device+0x26/0x33
        [<ffffffff812a7b6d>] __scsi_remove_target+0x93/0xd7
        [<ffffffff812a7c17>] __remove_child+0x1e/0x25
        [<ffffffff81291a22>] device_for_each_child+0x38/0x6f
        [<ffffffff812a7bec>] scsi_remove_target+0x3b/0x48
        [<ffffffffa0049db7>] fc_starget_delete+0x21/0x25 
[scsi_transport_fc]
        [<ffffffffa0049eb1>] fc_rport_final_delete+0xf6/0x188 
[scsi_transport_fc]
        [<ffffffff810527ec>] worker_thread+0x1fa/0x30a
        [<ffffffff81056c2d>] kthread+0x88/0x90
        [<ffffffff8100cbfa>] child_rip+0xa/0x20
        [<ffffffffffffffff>] 0xffffffffffffffff

-> #3 (&shost->scan_mutex){+.+.+.}:

        [<ffffffff8106700c>] __lock_acquire+0xa48/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff8150d536>] __mutex_lock_common+0x48/0x328
        [<ffffffff8150d88c>] mutex_lock_nested+0x24/0x29
        [<ffffffff812a7ab3>] scsi_remove_device+0x1e/0x33
        [<ffffffff812a7b6d>] __scsi_remove_target+0x93/0xd7
        [<ffffffff812a7c17>] __remove_child+0x1e/0x25
        [<ffffffff81291a22>] device_for_each_child+0x38/0x6f
        [<ffffffff812a7bec>] scsi_remove_target+0x3b/0x48
        [<ffffffffa0049db7>] fc_starget_delete+0x21/0x25 
[scsi_transport_fc]
        [<ffffffffa0049eb1>] fc_rport_final_delete+0xf6/0x188 
[scsi_transport_fc]
        [<ffffffff810527ec>] worker_thread+0x1fa/0x30a
        [<ffffffff81056c2d>] kthread+0x88/0x90
        [<ffffffff8100cbfa>] child_rip+0xa/0x20
        [<ffffffffffffffff>] 0xffffffffffffffff

-> #2 (&rport->rport_delete_work){+.+.+.}:

        [<ffffffff8106700c>] __lock_acquire+0xa48/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff810527e3>] worker_thread+0x1f1/0x30a
        [<ffffffff81056c2d>] kthread+0x88/0x90
        [<ffffffff8100cbfa>] child_rip+0xa/0x20
        [<ffffffffffffffff>] 0xffffffffffffffff

-> #1 ((fc_host->work_q_name)){+.+.+.}:

        [<ffffffff8106700c>] __lock_acquire+0xa48/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff81052e0e>] cleanup_workqueue_thread+0x45/0xb8
        [<ffffffff81052f18>] destroy_workqueue+0x63/0x9e
        [<ffffffffa004a5a7>] fc_remove_host+0x148/0x171 [scsi_transport_fc]
        [<ffffffffa00738b2>] 0xffffffffa00738b2
        [<ffffffffa0075001>] 0xffffffffa0075001
        [<ffffffff81055034>] param_attr_store+0x25/0x35
        [<ffffffff81055089>] module_attr_store+0x21/0x25
        [<ffffffff811279fe>] sysfs_write_file+0xe4/0x119
        [<ffffffff810d5a12>] vfs_write+0xab/0x105
        [<ffffffff810d5b30>] sys_write+0x47/0x6f
        [<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b
        [<ffffffffffffffff>] 0xffffffffffffffff

-> #0 (cpu_add_remove_lock){+.+.+.}:

        [<ffffffff81066f00>] __lock_acquire+0x93c/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff8150d536>] __mutex_lock_common+0x48/0x328
        [<ffffffff8150d88c>] mutex_lock_nested+0x24/0x29
        [<ffffffff81042bd2>] cpu_maps_update_begin+0x12/0x14
        [<ffffffff81052ee0>] destroy_workqueue+0x2b/0x9e
        [<ffffffff8129ef4f>] scsi_host_dev_release+0x5a/0xbd
        [<ffffffff81291941>] device_release+0x49/0x75
        [<ffffffff811d39d8>] kobject_release+0x51/0x67
        [<ffffffff811d4789>] kref_put+0x43/0x4f
        [<ffffffff811d38e1>] kobject_put+0x47/0x4b
        [<ffffffff812912f2>] put_device+0x12/0x14
        [<ffffffffa004826b>] fc_rport_dev_release+0x18/0x24 
[scsi_transport_fc]
        [<ffffffff81291941>] device_release+0x49/0x75
        [<ffffffff811d39d8>] kobject_release+0x51/0x67
        [<ffffffff811d4789>] kref_put+0x43/0x4f
        [<ffffffff811d38e1>] kobject_put+0x47/0x4b
        [<ffffffff812912f2>] put_device+0x12/0x14
        [<ffffffff812a4d44>] scsi_target_dev_release+0x1d/0x21
        [<ffffffff81291941>] device_release+0x49/0x75
        [<ffffffff811d39d8>] kobject_release+0x51/0x67
        [<ffffffff811d4789>] kref_put+0x43/0x4f
        [<ffffffff811d38e1>] kobject_put+0x47/0x4b
        [<ffffffff812912f2>] put_device+0x12/0x14
        [<ffffffff812a7709>] 
scsi_device_dev_release_usercontext+0x118/0x124
        [<ffffffff81053894>] execute_in_process_context+0x2a/0x70
        [<ffffffff812a75ef>] scsi_device_dev_release+0x17/0x19
        [<ffffffff81291941>] device_release+0x49/0x75
        [<ffffffff811d39d8>] kobject_release+0x51/0x67
        [<ffffffff811d4789>] kref_put+0x43/0x4f
        [<ffffffff811d38e1>] kobject_put+0x47/0x4b
        [<ffffffff812912f2>] put_device+0x12/0x14
        [<ffffffff8129dfd5>] scsi_device_put+0x3d/0x42
        [<ffffffff812acb57>] scsi_disk_put+0x30/0x41
        [<ffffffff812ad9c3>] sd_release+0x4d/0x54
        [<ffffffff810f9ca0>] __blkdev_put+0xa7/0x16e
        [<ffffffff810f9d57>] __blkdev_put+0x15e/0x16e
        [<ffffffff810f9d72>] blkdev_put+0xb/0xd
        [<ffffffff810f9dab>] blkdev_close+0x37/0x3c
        [<ffffffff810d6450>] __fput+0x10d/0x1bb
        [<ffffffff810d6516>] fput+0x18/0x1a
        [<ffffffff810d35b0>] filp_close+0x67/0x72
        [<ffffffff810d3660>] sys_close+0xa5/0xe4
        [<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b
        [<ffffffffffffffff>] 0xffffffffffffffff

other info that might help us debug this:

2 locks held by hald-probe-volu/23829:
#0: (&bdev->bd_mutex/1){+.+.+.}, at: [<ffffffff810f9c27>]

__blkdev_put+0x2e/0x16e
  #1:  (sd_ref_mutex){+.+.+.}, at: [<ffffffff812acb47>] 
scsi_disk_put+0x20/0x41

 stack backtrace:
 Pid: 23829, comm: hald-probe-volu Not tainted 2.6.31-rc4-rp9 #5
 Call Trace:
  [<ffffffff8106624e>] print_circular_bug_tail+0x71/0x7c
  [<ffffffff81066f00>] __lock_acquire+0x93c/0xbd0
  [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
  [<ffffffff81042bd2>] ? cpu_maps_update_begin+0x12/0x14
  [<ffffffff8150d536>] __mutex_lock_common+0x48/0x328
  [<ffffffff81042bd2>] ? cpu_maps_update_begin+0x12/0x14
  [<ffffffff81065c1d>] ? trace_hardirqs_on+0xd/0xf
  [<ffffffff8150e64c>] ? _spin_unlock_irq+0x2b/0x30
  [<ffffffff81042bd2>] ? cpu_maps_update_begin+0x12/0x14
  [<ffffffff8150ce9e>] ? wait_for_common+0xf7/0x112
  [<ffffffff8103c93f>] ? default_wake_function+0x0/0xf
  [<ffffffff8150d88c>] mutex_lock_nested+0x24/0x29
  [<ffffffff81042bd2>] cpu_maps_update_begin+0x12/0x14
  [<ffffffff81052ee0>] destroy_workqueue+0x2b/0x9e
  [<ffffffff8129ef4f>] scsi_host_dev_release+0x5a/0xbd
  [<ffffffff81291941>] device_release+0x49/0x75
  [<ffffffff811d39d8>] kobject_release+0x51/0x67
  [<ffffffff811d3987>] ? kobject_release+0x0/0x67
  [<ffffffff811d4789>] kref_put+0x43/0x4f
  [<ffffffff811d38e1>] kobject_put+0x47/0x4b
  [<ffffffff812912f2>] put_device+0x12/0x14
  [<ffffffffa004826b>] fc_rport_dev_release+0x18/0x24 [scsi_transport_fc]
  [<ffffffff81291941>] device_release+0x49/0x75
  [<ffffffff811d39d8>] kobject_release+0x51/0x67
  [<ffffffff811d3987>] ? kobject_release+0x0/0x67
  [<ffffffff811d4789>] kref_put+0x43/0x4f
  [<ffffffff811d38e1>] kobject_put+0x47/0x4b
  [<ffffffff812912f2>] put_device+0x12/0x14
  [<ffffffff812a4d44>] scsi_target_dev_release+0x1d/0x21
  [<ffffffff81291941>] device_release+0x49/0x75
  [<ffffffff811d39d8>] kobject_release+0x51/0x67
  [<ffffffff811d3987>] ? kobject_release+0x0/0x67
  [<ffffffff811d4789>] kref_put+0x43/0x4f
  [<ffffffff811d38e1>] kobject_put+0x47/0x4b
  [<ffffffff812912f2>] put_device+0x12/0x14
  [<ffffffff812a7709>] scsi_device_dev_release_usercontext+0x118/0x124
  [<ffffffff812a75f1>] ? scsi_device_dev_release_usercontext+0x0/0x124
  [<ffffffff81053894>] execute_in_process_context+0x2a/0x70
  [<ffffffff812a75ef>] scsi_device_dev_release+0x17/0x19
  [<ffffffff81291941>] device_release+0x49/0x75
  [<ffffffff811d39d8>] kobject_release+0x51/0x67
  [<ffffffff811d3987>] ? kobject_release+0x0/0x67
  [<ffffffff811d4789>] kref_put+0x43/0x4f
  [<ffffffff811d38e1>] kobject_put+0x47/0x4b
  [<ffffffff812912f2>] put_device+0x12/0x14
  [<ffffffff8129dfd5>] scsi_device_put+0x3d/0x42
  [<ffffffff812acb57>] scsi_disk_put+0x30/0x41
  [<ffffffff812ad9c3>] sd_release+0x4d/0x54
  [<ffffffff810f9ca0>] __blkdev_put+0xa7/0x16e
  [<ffffffff810f9d57>] __blkdev_put+0x15e/0x16e
  [<ffffffff810f9d72>] blkdev_put+0xb/0xd
  [<ffffffff810f9dab>] blkdev_close+0x37/0x3c
  [<ffffffff810d6450>] __fput+0x10d/0x1bb
  [<ffffffff810d6516>] fput+0x18/0x1a
  [<ffffffff810d35b0>] filp_close+0x67/0x72
  [<ffffffff810d3660>] sys_close+0xa5/0xe4
  [<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b

-- end

Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
author
#2 lore raw
Quoted 23 lines · Joe Eykholt
Quoted 16 lines · Joe Eykholt

This is different from the 3-way deadlock I posted on 7/25
http://marc.info/?l=linux-scsi&m=124856078500527&w=2

A user process is closing an sd block dev file.
It holds the sd_ref_mutex and is in scsi_host_dev_release() doing a destroy_workqueue() which needs the cpu_add_remove_lock.

The process that holds cpu_add_remove_lock is doing fcoe_destroy (deleting an FCoE instance) which calls fc_remove_host() which does destroy_workqueue() and waiting in flush_cpu_workqueue() for a work item to finish.

The stuck work item is doing fc_rport_final_delete(), and that's in device_release_driver() in sd_remove() which is waiting for the sd_ref_mutex.

Quoted 5 lines

I'd like to hear some discussion about the right approach to fix this. There should be some clear rules about how work items can destroy or flush other workqueues. I'd like to be able to state the clear rule that's being violated here.

Quoted 5 lines

Now it seems to me that that the rule is:

A workqueue item may not destroy a workqueue if it is running on a workqueue that itself may be destroyed.

Rephrasing:

If a workqueue may be destroyed, it's handlers may not destroy any workqueue.

Quoted 14 lines

That's because destroy_workqueue takes the cpu_add_remove lock and, while holding it, does flush_cpu_workqueue().

If a work item on workqueue A destroy workqueue B, and thread C destroys workqueue A, then thread C may get the cpu_add_remove lock first, work item on A can wait forever for the lock, and thread C can hang waiting for workqueue A to flush.

So we're violating that rule if we let fc_rport_final_delete()
do the destroy of the device's workqueue. I still don't
have a good solution. Maybe fc_host_remove() could flush the workqueue before destroying it?

Quoted 4 lines

Maybe destroy_workqueue() should flush before grabbing the cpu_add_remove_lock, but maybe that doesn't completely fix it.

Joe

Sorry for the noise.

The rule above is true, but it's
not really what's causing the problem here, which is:

sd_ref_mutex is held during a destroy_workqueue() for the shost->work_q. It means sd_release must have been the the last one to release the scsi_host shost_gendev.

Since fc_remove_host() is running, but it must be for the other fcoe instance, not the same host.

sd_ref_mutex is also grabbed from a work item handler on the fc_host fc_wq.

Since both these workqueues can be destroyed, its wrong to hold sd_ref mutex while destroying shost->work_q. Since it may also be grabbed in a work item that needs to be finished before destroying its work queue.

So my second rule is:

If a workqueue may be destroyed, it's handlers may not take a lock if that lock is ever held while destroying any workqueue.

These two rules are caused by the use of cpu_add_remove_lock. Maybe that's the issue to work on?

The first rule is violated in fc_rport_final_delete() and the second rule is violated for sd_ref_mutex also in fc_rport_final_delete().

Would it be OK to run fc_rport_final_delete() on the general event workqueue (non-destroyable)? I'm not sure that fixes it.

Is there a way to fix the violations, or should we manage to change the rules somehow? Does anyone find a problem with the rules I've stated?

Thanks,
Joe

Quoted 332 lines
Quoted 327 lines

Perhaps the way cpu_add_remove_lock is used to in
destroy_workqueue() should be changed, or maybe
fc_rport_final_delete and fc_remove_host need to
be better about using workqueues somehow.

I do see a lock dependency warning about this, so that system works. I saw this once on a system based on 2.6.31-rc4 under the current fcoe-next.git tree.

The following are the stacks I collected via /proc/*/stacks and the lockdep warnings.

Thanks,
Joe

Name:   hald-probe-volu
State:  D (disk sleep)
cmd:    /usr/libexec/hald-probe-volume
wchan:  cpu_maps_update_begin

--- waiting cpu_add_remove_lock
--- has sd_ref_mutex in scsi_disk_put

[<ffffffff81042bd2>] cpu_maps_update_begin+0x12/0x14
[<ffffffff81052ee0>] destroy_workqueue+0x2b/0x9e
[<ffffffff8129ef4f>] scsi_host_dev_release+0x5a/0xbd
[<ffffffff81291941>] device_release+0x49/0x75
[<ffffffff811d39d8>] kobject_release+0x51/0x67
[<ffffffff811d4789>] kref_put+0x43/0x4f
[<ffffffff811d38e1>] kobject_put+0x47/0x4b
[<ffffffff812912f2>] put_device+0x12/0x14
[<ffffffffa004826b>] fc_rport_dev_release+0x18/0x24 [scsi_transport_fc]
[<ffffffff81291941>] device_release+0x49/0x75
[<ffffffff811d39d8>] kobject_release+0x51/0x67
[<ffffffff811d4789>] kref_put+0x43/0x4f
[<ffffffff811d38e1>] kobject_put+0x47/0x4b
[<ffffffff812912f2>] put_device+0x12/0x14
[<ffffffff812a4d44>] scsi_target_dev_release+0x1d/0x21
[<ffffffff81291941>] device_release+0x49/0x75
[<ffffffff811d39d8>] kobject_release+0x51/0x67
[<ffffffff811d4789>] kref_put+0x43/0x4f
[<ffffffff811d38e1>] kobject_put+0x47/0x4b
[<ffffffff812912f2>] put_device+0x12/0x14
[<ffffffff812a7709>] scsi_device_dev_release_usercontext+0x118/0x124
[<ffffffff81053894>] execute_in_process_context+0x2a/0x70
[<ffffffff812a75ef>] scsi_device_dev_release+0x17/0x19
[<ffffffff81291941>] device_release+0x49/0x75
[<ffffffff811d39d8>] kobject_release+0x51/0x67
[<ffffffff811d4789>] kref_put+0x43/0x4f
[<ffffffff811d38e1>] kobject_put+0x47/0x4b
[<ffffffff812912f2>] put_device+0x12/0x14
[<ffffffff8129dfd5>] scsi_device_put+0x3d/0x42
[<ffffffff812acb57>] scsi_disk_put+0x30/0x41
[<ffffffff812ad9c3>] sd_release+0x4d/0x54
[<ffffffff810f9ca0>] __blkdev_put+0xa7/0x16e
[<ffffffff810f9d57>] __blkdev_put+0x15e/0x16e
[<ffffffff810f9d72>] blkdev_put+0xb/0xd
[<ffffffff810f9dab>] blkdev_close+0x37/0x3c
[<ffffffff810d6450>] __fput+0x10d/0x1bb
[<ffffffff810d6516>] fput+0x18/0x1a
[<ffffffff810d35b0>] filp_close+0x67/0x72
[<ffffffff810d3660>] sys_close+0xa5/0xe4
[<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

Name:   fcoeadm
State:  D (disk sleep)
cmd:    fcoeadm -d eth4
wchan:  flush_cpu_workqueue

--- waits on cpu work to finish. -- fc_workqueue 28596 --- holds cpu_add_remove_lock

[<ffffffff81052dbd>] flush_cpu_workqueue+0x7b/0x87
[<ffffffff81052e33>] cleanup_workqueue_thread+0x6a/0xb8
[<ffffffff81052f18>] destroy_workqueue+0x63/0x9e
[<ffffffffa004a5a7>] fc_remove_host+0x148/0x171 [scsi_transport_fc]
[<ffffffffa03e48b2>] fcoe_if_destroy+0x1a6/0x1db [fcoe]
[<ffffffffa03e6001>] fcoe_destroy+0x86/0xad [fcoe]
[<ffffffff81055034>] param_attr_store+0x25/0x35
[<ffffffff81055089>] module_attr_store+0x21/0x25
[<ffffffff811279fe>] sysfs_write_file+0xe4/0x119
[<ffffffff810d5a12>] vfs_write+0xab/0x105
[<ffffffff810d5b30>] sys_write+0x47/0x6f
[<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

Name:   fc_wq_137
State:  D (disk sleep)

cmd:
wchan: scsi_disk_get_from_dev

-- waits on sd_ref_mutex

[<ffffffff812acaf8>] scsi_disk_get_from_dev+0x1a/0x49
[<ffffffff812acfdd>] sd_shutdown+0x12/0x117
[<ffffffff812ad20f>] sd_remove+0x51/0x8a
[<ffffffff8129443e>] __device_release_driver+0x80/0xc9
[<ffffffff81294552>] device_release_driver+0x1e/0x2b
[<ffffffff81293ae6>] bus_remove_device+0xa8/0xc9
[<ffffffff812921d6>] device_del+0x13f/0x1ac
[<ffffffff812a7a58>] __scsi_remove_device+0x44/0x81
[<ffffffff812a7abb>] scsi_remove_device+0x26/0x33
[<ffffffff812a7b6d>] __scsi_remove_target+0x93/0xd7
[<ffffffff812a7c17>] __remove_child+0x1e/0x25
[<ffffffff81291a22>] device_for_each_child+0x38/0x6f
[<ffffffff812a7bec>] scsi_remove_target+0x3b/0x48
[<ffffffffa0049db7>] fc_starget_delete+0x21/0x25 [scsi_transport_fc]
[<ffffffffa0049eb1>] fc_rport_final_delete+0xf6/0x188 [scsi_transport_fc]
[<ffffffff810527ec>] worker_thread+0x1fa/0x30a
[<ffffffff81056c2d>] kthread+0x88/0x90
[<ffffffff8100cbfa>] child_rip+0xa/0x20

[<ffffffffffffffff>] 0xffffffffffffffff

[ INFO: possible circular locking dependency detected ] 2.6.31-rc4-rp9 #5
-------------------------------------------------------
hald-probe-volu/23829 is trying to acquire lock:
(cpu_add_remove_lock){+.+.+.}, at: [<ffffffff81042bd2>]
cpu_maps_update_begin+0x12/0x14

but task is already holding lock:
(sd_ref_mutex){+.+.+.}, at: [<ffffffff812acb47>]
scsi_disk_put+0x20/0x41

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #4 (sd_ref_mutex){+.+.+.}:

        [<ffffffff8106700c>] __lock_acquire+0xa48/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff8150d536>] __mutex_lock_common+0x48/0x328
        [<ffffffff8150d88c>] mutex_lock_nested+0x24/0x29
        [<ffffffff812acaf8>] scsi_disk_get_from_dev+0x1a/0x49
        [<ffffffff812acfdd>] sd_shutdown+0x12/0x117
        [<ffffffff812ad20f>] sd_remove+0x51/0x8a
        [<ffffffff8129443e>] __device_release_driver+0x80/0xc9
        [<ffffffff81294552>] device_release_driver+0x1e/0x2b
        [<ffffffff81293ae6>] bus_remove_device+0xa8/0xc9
        [<ffffffff812921d6>] device_del+0x13f/0x1ac
        [<ffffffff812a7a58>] __scsi_remove_device+0x44/0x81
        [<ffffffff812a7abb>] scsi_remove_device+0x26/0x33
        [<ffffffff812a7b6d>] __scsi_remove_target+0x93/0xd7
        [<ffffffff812a7c17>] __remove_child+0x1e/0x25
        [<ffffffff81291a22>] device_for_each_child+0x38/0x6f
        [<ffffffff812a7bec>] scsi_remove_target+0x3b/0x48
        [<ffffffffa0049db7>] fc_starget_delete+0x21/0x25 
[scsi_transport_fc]
        [<ffffffffa0049eb1>] fc_rport_final_delete+0xf6/0x188 
[scsi_transport_fc]
        [<ffffffff810527ec>] worker_thread+0x1fa/0x30a
        [<ffffffff81056c2d>] kthread+0x88/0x90
        [<ffffffff8100cbfa>] child_rip+0xa/0x20
        [<ffffffffffffffff>] 0xffffffffffffffff

-> #3 (&shost->scan_mutex){+.+.+.}:

        [<ffffffff8106700c>] __lock_acquire+0xa48/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff8150d536>] __mutex_lock_common+0x48/0x328
        [<ffffffff8150d88c>] mutex_lock_nested+0x24/0x29
        [<ffffffff812a7ab3>] scsi_remove_device+0x1e/0x33
        [<ffffffff812a7b6d>] __scsi_remove_target+0x93/0xd7
        [<ffffffff812a7c17>] __remove_child+0x1e/0x25
        [<ffffffff81291a22>] device_for_each_child+0x38/0x6f
        [<ffffffff812a7bec>] scsi_remove_target+0x3b/0x48
        [<ffffffffa0049db7>] fc_starget_delete+0x21/0x25 
[scsi_transport_fc]
        [<ffffffffa0049eb1>] fc_rport_final_delete+0xf6/0x188 
[scsi_transport_fc]
        [<ffffffff810527ec>] worker_thread+0x1fa/0x30a
        [<ffffffff81056c2d>] kthread+0x88/0x90
        [<ffffffff8100cbfa>] child_rip+0xa/0x20
        [<ffffffffffffffff>] 0xffffffffffffffff

-> #2 (&rport->rport_delete_work){+.+.+.}:

        [<ffffffff8106700c>] __lock_acquire+0xa48/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff810527e3>] worker_thread+0x1f1/0x30a
        [<ffffffff81056c2d>] kthread+0x88/0x90
        [<ffffffff8100cbfa>] child_rip+0xa/0x20
        [<ffffffffffffffff>] 0xffffffffffffffff

-> #1 ((fc_host->work_q_name)){+.+.+.}:

        [<ffffffff8106700c>] __lock_acquire+0xa48/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff81052e0e>] cleanup_workqueue_thread+0x45/0xb8
        [<ffffffff81052f18>] destroy_workqueue+0x63/0x9e
        [<ffffffffa004a5a7>] fc_remove_host+0x148/0x171 
[scsi_transport_fc]
        [<ffffffffa00738b2>] 0xffffffffa00738b2
        [<ffffffffa0075001>] 0xffffffffa0075001
        [<ffffffff81055034>] param_attr_store+0x25/0x35
        [<ffffffff81055089>] module_attr_store+0x21/0x25
        [<ffffffff811279fe>] sysfs_write_file+0xe4/0x119
        [<ffffffff810d5a12>] vfs_write+0xab/0x105
        [<ffffffff810d5b30>] sys_write+0x47/0x6f
        [<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b
        [<ffffffffffffffff>] 0xffffffffffffffff

-> #0 (cpu_add_remove_lock){+.+.+.}:

        [<ffffffff81066f00>] __lock_acquire+0x93c/0xbd0
        [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
        [<ffffffff8150d536>] __mutex_lock_common+0x48/0x328
        [<ffffffff8150d88c>] mutex_lock_nested+0x24/0x29
        [<ffffffff81042bd2>] cpu_maps_update_begin+0x12/0x14
        [<ffffffff81052ee0>] destroy_workqueue+0x2b/0x9e
        [<ffffffff8129ef4f>] scsi_host_dev_release+0x5a/0xbd
        [<ffffffff81291941>] device_release+0x49/0x75
        [<ffffffff811d39d8>] kobject_release+0x51/0x67
        [<ffffffff811d4789>] kref_put+0x43/0x4f
        [<ffffffff811d38e1>] kobject_put+0x47/0x4b
        [<ffffffff812912f2>] put_device+0x12/0x14
        [<ffffffffa004826b>] fc_rport_dev_release+0x18/0x24 
[scsi_transport_fc]
        [<ffffffff81291941>] device_release+0x49/0x75
        [<ffffffff811d39d8>] kobject_release+0x51/0x67
        [<ffffffff811d4789>] kref_put+0x43/0x4f
        [<ffffffff811d38e1>] kobject_put+0x47/0x4b
        [<ffffffff812912f2>] put_device+0x12/0x14
        [<ffffffff812a4d44>] scsi_target_dev_release+0x1d/0x21
        [<ffffffff81291941>] device_release+0x49/0x75
        [<ffffffff811d39d8>] kobject_release+0x51/0x67
        [<ffffffff811d4789>] kref_put+0x43/0x4f
        [<ffffffff811d38e1>] kobject_put+0x47/0x4b
        [<ffffffff812912f2>] put_device+0x12/0x14
        [<ffffffff812a7709>] 
scsi_device_dev_release_usercontext+0x118/0x124
        [<ffffffff81053894>] execute_in_process_context+0x2a/0x70
        [<ffffffff812a75ef>] scsi_device_dev_release+0x17/0x19
        [<ffffffff81291941>] device_release+0x49/0x75
        [<ffffffff811d39d8>] kobject_release+0x51/0x67
        [<ffffffff811d4789>] kref_put+0x43/0x4f
        [<ffffffff811d38e1>] kobject_put+0x47/0x4b
        [<ffffffff812912f2>] put_device+0x12/0x14
        [<ffffffff8129dfd5>] scsi_device_put+0x3d/0x42
        [<ffffffff812acb57>] scsi_disk_put+0x30/0x41
        [<ffffffff812ad9c3>] sd_release+0x4d/0x54
        [<ffffffff810f9ca0>] __blkdev_put+0xa7/0x16e
        [<ffffffff810f9d57>] __blkdev_put+0x15e/0x16e
        [<ffffffff810f9d72>] blkdev_put+0xb/0xd
        [<ffffffff810f9dab>] blkdev_close+0x37/0x3c
        [<ffffffff810d6450>] __fput+0x10d/0x1bb
        [<ffffffff810d6516>] fput+0x18/0x1a
        [<ffffffff810d35b0>] filp_close+0x67/0x72
        [<ffffffff810d3660>] sys_close+0xa5/0xe4
        [<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b
        [<ffffffffffffffff>] 0xffffffffffffffff

other info that might help us debug this:

2 locks held by hald-probe-volu/23829:
#0: (&bdev->bd_mutex/1){+.+.+.}, at: [<ffffffff810f9c27>]

__blkdev_put+0x2e/0x16e
  #1:  (sd_ref_mutex){+.+.+.}, at: [<ffffffff812acb47>] 
scsi_disk_put+0x20/0x41

 stack backtrace:
 Pid: 23829, comm: hald-probe-volu Not tainted 2.6.31-rc4-rp9 #5
 Call Trace:
  [<ffffffff8106624e>] print_circular_bug_tail+0x71/0x7c
  [<ffffffff81066f00>] __lock_acquire+0x93c/0xbd0
  [<ffffffff8106724d>] lock_acquire+0xb9/0xdd
  [<ffffffff81042bd2>] ? cpu_maps_update_begin+0x12/0x14
  [<ffffffff8150d536>] __mutex_lock_common+0x48/0x328
  [<ffffffff81042bd2>] ? cpu_maps_update_begin+0x12/0x14
  [<ffffffff81065c1d>] ? trace_hardirqs_on+0xd/0xf
  [<ffffffff8150e64c>] ? _spin_unlock_irq+0x2b/0x30
  [<ffffffff81042bd2>] ? cpu_maps_update_begin+0x12/0x14
  [<ffffffff8150ce9e>] ? wait_for_common+0xf7/0x112
  [<ffffffff8103c93f>] ? default_wake_function+0x0/0xf
  [<ffffffff8150d88c>] mutex_lock_nested+0x24/0x29
  [<ffffffff81042bd2>] cpu_maps_update_begin+0x12/0x14
  [<ffffffff81052ee0>] destroy_workqueue+0x2b/0x9e
  [<ffffffff8129ef4f>] scsi_host_dev_release+0x5a/0xbd
  [<ffffffff81291941>] device_release+0x49/0x75
  [<ffffffff811d39d8>] kobject_release+0x51/0x67
  [<ffffffff811d3987>] ? kobject_release+0x0/0x67
  [<ffffffff811d4789>] kref_put+0x43/0x4f
  [<ffffffff811d38e1>] kobject_put+0x47/0x4b
  [<ffffffff812912f2>] put_device+0x12/0x14
  [<ffffffffa004826b>] fc_rport_dev_release+0x18/0x24 [scsi_transport_fc]
  [<ffffffff81291941>] device_release+0x49/0x75
  [<ffffffff811d39d8>] kobject_release+0x51/0x67
  [<ffffffff811d3987>] ? kobject_release+0x0/0x67
  [<ffffffff811d4789>] kref_put+0x43/0x4f
  [<ffffffff811d38e1>] kobject_put+0x47/0x4b
  [<ffffffff812912f2>] put_device+0x12/0x14
  [<ffffffff812a4d44>] scsi_target_dev_release+0x1d/0x21
  [<ffffffff81291941>] device_release+0x49/0x75
  [<ffffffff811d39d8>] kobject_release+0x51/0x67
  [<ffffffff811d3987>] ? kobject_release+0x0/0x67
  [<ffffffff811d4789>] kref_put+0x43/0x4f
  [<ffffffff811d38e1>] kobject_put+0x47/0x4b
  [<ffffffff812912f2>] put_device+0x12/0x14
  [<ffffffff812a7709>] scsi_device_dev_release_usercontext+0x118/0x124
  [<ffffffff812a75f1>] ? scsi_device_dev_release_usercontext+0x0/0x124
  [<ffffffff81053894>] execute_in_process_context+0x2a/0x70
  [<ffffffff812a75ef>] scsi_device_dev_release+0x17/0x19
  [<ffffffff81291941>] device_release+0x49/0x75
  [<ffffffff811d39d8>] kobject_release+0x51/0x67
  [<ffffffff811d3987>] ? kobject_release+0x0/0x67
  [<ffffffff811d4789>] kref_put+0x43/0x4f
  [<ffffffff811d38e1>] kobject_put+0x47/0x4b
  [<ffffffff812912f2>] put_device+0x12/0x14
  [<ffffffff8129dfd5>] scsi_device_put+0x3d/0x42
  [<ffffffff812acb57>] scsi_disk_put+0x30/0x41
  [<ffffffff812ad9c3>] sd_release+0x4d/0x54
  [<ffffffff810f9ca0>] __blkdev_put+0xa7/0x16e
  [<ffffffff810f9d57>] __blkdev_put+0x15e/0x16e
  [<ffffffff810f9d72>] blkdev_put+0xb/0xd
  [<ffffffff810f9dab>] blkdev_close+0x37/0x3c
  [<ffffffff810d6450>] __fput+0x10d/0x1bb
  [<ffffffff810d6516>] fput+0x18/0x1a
  [<ffffffff810d35b0>] filp_close+0x67/0x72
  [<ffffffff810d3660>] sys_close+0xa5/0xe4
  [<ffffffff8100baeb>] system_call_fastpath+0x16/0x1b

-- end

Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html