Fix a hard lockup in scsi_remove_target()
cover Fix a hard lockup in scsi_remove_target()
On October 2, 2015 Johannes Thumshirn reported the following hard lockup (see also http://thread.gmane.org/gmane.linux.kernel/2052359):
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0 [...] Call Trace: [<ffffffff8100471d>] dump_trace+0x7d/0x2d0 [<ffffffff81004a04>] show_stack_log_lvl+0x94/0x170 [<ffffffff81005cc1>] show_stack+0x21/0x50 [<ffffffff8151aa75>] dump_stack+0x41/0x51 [<ffffffff8151545a>] panic+0xc8/0x1d7 [<ffffffff810fbdda>] watchdog_overflow_callback+0xba/0xc0 [<ffffffff811336c8>] __perf_event_overflow+0x88/0x240 [<ffffffff8101e3aa>] intel_pmu_handle_irq+0x1fa/0x3e0 [<ffffffff81522836>] perf_event_nmi_handler+0x26/0x40 [<ffffffff81521fcd>] nmi_handle.isra.2+0x8d/0x180 [<ffffffff815221e6>] do_nmi+0x126/0x3c0 [<ffffffff8152159b>] end_repeat_nmi+0x1a/0x1e [<ffffffffa00212e8>] scsi_remove_target+0x68/0x240 [scsi_mod] [<ffffffff81072742>] process_one_work+0x172/0x420 [<ffffffff810733ba>] worker_thread+0x11a/0x3c0 [<ffffffff81079d34>] kthread+0xb4/0xc0 [<ffffffff81528cd8>] ret_from_fork+0x58/0x90
The following two patches fix this lockup:
0001-Separate-target-visibility-from-reaped-state-informa.patch 0002-Restart-list-search-after-unlock-in-scsi_remove_targ.patch
These patches have been tested with the SRP initiator (ib_srp).
PATCH 1/2 Separate target visibility from reaped state information
Instead of representing the states "visible in sysfs" and "has been removed from the target list" by a single state variable, use two variables to represent this information.
--- drivers/scsi/scsi_scan.c | 31 +++---------------------------- drivers/scsi/scsi_sysfs.c | 7 ++++--- include/scsi/scsi_device.h | 3 ++- 3 files changed, 9 insertions(+), 32 deletions(-)
drivers/scsi/scsi_scan.c +3 -28
@@ -314,7 +314,6 @@ static void scsi_target_destroy(struct scsi_target *starget)struct Scsi_Host *shost =dev_to_shost (dev->parent);unsigned long flags; - starget->state =STARGET_DEL ;transport_destroy_device (dev);spin_lock_irqsave (shost->host_lock, flags);if (shost->hostt->target_destroy)
@@ -379,19 +378,15 @@ static void scsi_target_reap_ref_release(struct kref *kref)struct scsi_target *starget = container_of(kref, struct scsi_target, reap_ref); -/* -* if we get here and the target is still in the CREATED state that -* means it was allocated but never made visible (because a scan -* turned up no LUNs), so don't call device_del() on it. -*/ -if (starget->state !=STARGET_CREATED ) { +if (starget->visible) { + starget->visible = false;transport_remove_device (&starget->dev);device_del (&starget->dev); }scsi_target_destroy (starget); } -static void scsi_target_reap_ref_put (struct scsi_target *starget) +void scsi_target_reap (struct scsi_target *starget) {kref_put (&starget->reap_ref, scsi_target_reap_ref_release); }
@@ -437,7 +432,6 @@ static struct scsi_target *scsi_alloc_target(struct device *parent, starget->can_queue =0 ;INIT_LIST_HEAD (&starget->siblings);INIT_LIST_HEAD (&starget->devices); - starget->state =STARGET_CREATED ; starget->scsi_level =SCSI_2 ; starget->max_target_blocked =SCSI_DEFAULT_TARGET_BLOCKED ; retry:
@@ -498,25 +492,6 @@ static struct scsi_target *scsi_alloc_target(struct device *parent, }/** -* scsi_target_reap - check to see if target is in use and destroy if not -* @starget: target to be checked -* -* This is used after removing a LUN or doing a last put of the target -* it checks atomically that nothing is using the target and removes -* it if so. -*/ -void scsi_target_reap (struct scsi_target *starget) -{ -/* -* serious problem if this triggers: STARGET_DEL is only set in the if -* the reap_ref drops to zero, so we're trying to do another final put -* on an already released kref -*/ -BUG_ON (starget->state ==STARGET_DEL ); -scsi_target_reap_ref_put (starget); -} - -/** * sanitize_inquiry_string - remove non-graphical chars from an INQUIRY result string * @s: INQUIRY result string to sanitize * @len: length of the string
drivers/scsi/scsi_sysfs.c +4 -3
@@ -974,7 +974,7 @@ static int scsi_target_add(struct scsi_target *starget) {int error; -if (starget->state !=STARGET_CREATED ) +if (starget->visible)return 0 ; error =device_add (&starget->dev);
@@ -983,7 +983,7 @@ static int scsi_target_add(struct scsi_target *starget)return error; }transport_add_device (&starget->dev); - starget->state =STARGET_RUNNING ; + starget->visible = true;pm_runtime_set_active (&starget->dev);pm_runtime_enable (&starget->dev);
@@ -1166,11 +1166,12 @@ void scsi_remove_target(struct device *dev)*/ spin_lock_irqsave (shost->host_lock, flags);list_for_each_entry (starget, &shost->__targets, siblings) { -if (starget->state ==STARGET_DEL ) +if (starget->reaped)continue ;if (starget->dev.parent == dev || &starget->dev == dev) {/* assuming new targets arrive at the end */ kref_get (&starget->reap_ref); + starget->reaped = true;spin_unlock_irqrestore (shost->host_lock, flags);if (last)scsi_target_reap (last);
include/scsi/scsi_device.h +2 -1
@@ -267,6 +267,8 @@ struct scsi_target { unsigned int expecting_lun_change:1; /* A device has reported* a 3F/0E UA, other devices on * the same target will also. */ + unsigned int visible:1; /* visible in sysfs */ + unsigned int reaped:1; /* removed from target list *//* commands actually active on LLD. */ atomic_t target_busy;atomic_t target_blocked;
@@ -280,7 +282,6 @@ struct scsi_target {#define SCSI_DEFAULT_TARGET_BLOCKED 3char scsi_level; -enum scsi_target_state state;void *hostdata;/* available to low-level driver */ unsigned long starget_data[0 ];/* for the transport */ /* starget_data must be the last element!!!! */
Signature · 1 line
2.6.2.307.g37023ba
Re: [PATCH 1/2] Separate target visibility from reaped state information
Looks good:
A few things I noticed when looking over this and the previous iterations, which are more notes for future work than an actual comment on this patch:
- what lock protects ->visible and ->reaped (or previously ->state)? It seems like the synchronization rules in this code are a little fragile.
- why don't we make the target visible as soon as we allocated it? We can still tear it down without a whole more effort even if it is visible.
PATCH 2/2 Restart list search after unlock in scsi_remove_target
In reply to Bart Van AsscheWhen dropping a lock while iterating a list we must restart the search as other threads could have manipulated the list under us. Without this we can get stuck in an endless loop.
This is a slightly modified version of a patch from Christoph Hellwig (see also https://www.spinics.net/lists/linux-scsi/msg89416.html).
--- drivers/scsi/scsi_sysfs.c | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-)
drivers/scsi/scsi_sysfs.c +4 -12
@@ -1158,32 +1158,24 @@ static void __scsi_remove_target(struct scsi_target *starget)void scsi_remove_target (struct device *dev) {struct Scsi_Host *shost =dev_to_shost (dev->parent); -struct scsi_target *starget, *last =NULL ; +struct scsi_target *starget;unsigned long flags; -/* remove targets being careful to lookup next entry before -* deleting the last -*/ +restart:spin_lock_irqsave (shost->host_lock, flags);list_for_each_entry (starget, &shost->__targets, siblings) {if (starget->reaped)continue ;if (starget->dev.parent == dev || &starget->dev == dev) { -/* assuming new targets arrive at the end */ kref_get (&starget->reap_ref); starget->reaped = true;spin_unlock_irqrestore (shost->host_lock, flags); -if (last) -scsi_target_reap (last); - last = starget;__scsi_remove_target (starget); -spin_lock_irqsave (shost->host_lock, flags); +scsi_target_reap (starget); + goto restart; } }spin_unlock_irqrestore (shost->host_lock, flags); - -if (last) -scsi_target_reap (last); }EXPORT_SYMBOL (scsi_remove_target);
Signature · 1 line
2.6.2.307.g37023ba
Re: [PATCH 2/2] Restart list search after unlock in scsi_remove_target
Quoted 47 lines · Bart Van Assche
When dropping a lock while iterating a list we must restart the search as other threads could have manipulated the list under us. Without this we can get stuck in an endless loop.
This is a slightly modified version of a patch from Christoph Hellwig (see also https://www.spinics.net/lists/linux-scsi/msg89416.html).
Reported-by Johannes Thumshirn <[email protected]>Signed-off-by Bart Van Assche <[email protected]>Cc Johannes Thumshirn <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Dan Williams <[email protected]>Cc stable <[email protected]>--- drivers/scsi/scsi_sysfs.c | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-)drivers/scsi/scsi_sysfs.c +2
-9@@ -1158,32 +1158,24 @@ static void __scsi_remove_target(struct scsi_target *starget)void scsi_remove_target (struct device *dev) {struct Scsi_Host *shost =dev_to_shost (dev->parent); -struct scsi_target *starget, *last =NULL ; +struct scsi_target *starget;unsigned long flags; -/* remove targets being careful to lookup next entry before -* deleting the last -*/ +restart:spin_lock_irqsave (shost->host_lock, flags);list_for_each_entry (starget, &shost->__targets, siblings) {if (starget->reaped)continue ;if (starget->dev.parent == dev || &starget->dev == dev) { -/* assuming new targets arrive at the end */ kref_get (&starget->reap_ref); starget->reaped = true;spin_unlock_irqrestore (shost->host_lock, flags); -if (last) -scsi_target_reap (last); - last = starget;__scsi_remove_target (starget); -spin_lock_irqsave (shost->host_lock, flags);
drivers/scsi/scsi_sysfs.c static void __scsi_remove_target(struct scsi_target *starget) · line 1180
Quoted 10 lines
drivers/scsi/scsi_sysfs.c +2
-3+scsi_target_reap (starget); + goto restart; } }spin_unlock_irqrestore (shost->host_lock, flags); - -if (last) -scsi_target_reap (last); }EXPORT_SYMBOL (scsi_remove_target);
(replying to my own e-mail)
Hello Christoph,
Is it OK for you if I mention you as author of this e-mail ?
Thanks,
Bart.
James Bottomley
maintainerRe: [PATCH 2/2] Restart list search after unlock in scsi_remove_target
Quoted 61 lines · Bart Van Assche
Quoted 47 lines · Bart Van Assche
When dropping a lock while iterating a list we must restart the search as other threads could have manipulated the list under us. Without this we can get stuck in an endless loop.
This is a slightly modified version of a patch from Christoph Hellwig (see also https://www.spinics.net/lists/linux-scsi/msg89416.html).
Reported-by Johannes Thumshirn <[email protected]>Signed-off-by Bart Van Assche <[email protected]>Cc Johannes Thumshirn <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Dan Williams <[email protected]>Cc stable <[email protected]>--- drivers/scsi/scsi_sysfs.c | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-)drivers/scsi/scsi_sysfs.c +2
-9@@ -1158,32 +1158,24 @@ static void __scsi_remove_target(struct scsi_target *starget)void scsi_remove_target (struct device *dev) {struct Scsi_Host *shost =dev_to_shost (dev->parent); -struct scsi_target *starget, *last =NULL ; +struct scsi_target *starget;unsigned long flags; -/* remove targets being careful to lookup next entry before -* deleting the last -*/ +restart:spin_lock_irqsave (shost->host_lock, flags);list_for_each_entry (starget, &shost->__targets, siblings) {if (starget->reaped)continue ;if (starget->dev.parent == dev || &starget->dev == dev) { -/* assuming new targets arrive at the end */ kref_get (&starget->reap_ref); starget->reaped = true;spin_unlock_irqrestore (shost->host_lock, flags); -if (last) -scsi_target_reap (last); - last = starget;__scsi_remove_target (starget); -spin_lock_irqsave (shost->host_lock, flags);drivers/scsi/scsi_sysfs.c static void __scsi_remove_target(struct scsi_target *starget) · line 1180
Quoted 10 lines
drivers/scsi/scsi_sysfs.c +2
-3+scsi_target_reap (starget); + goto restart; } }spin_unlock_irqrestore (shost->host_lock, flags); - -if (last) -scsi_target_reap (last); }EXPORT_SYMBOL (scsi_remove_target);(replying to my own e-mail)
Quoted 3 lines
Hello Christoph,
Is it OK for you if I mention you as author of this e-mail ?
Could you just both co-operate, especially since there's not much difference between the patches.
The fundamental problem with this is how have the conditions that caused us to move away from list restart:
commit bc3f02a795d3b4faa99d37390174be2a75d091bd
Author: Dan Williams <[email protected]> Date: Tue Aug 28 22:12:10 2012 -0700
[SCSI] scsi_remove_target: fix softlockup regression on hot remove
Which was triggered by this bug report
http://thread.gmane.org/gmane.linux.kernel/1348679
been mitigated?
James
Re: [PATCH 2/2] Restart list search after unlock in scsi_remove_target
Quoted 9 lines · James Bottomley
Quoted 5 lines · Bart Van Assche
(replying to my own e-mail)
Hello Christoph,
Is it OK for you if I mention you as author of this e-mail ?
Could you just both co-operate, especially since there's not much difference between the patches.
Hello James and Christoph,
Patch [2/2] is identical to the patch Christoph had posted except that it has been rebased on top of patch [1/2] of this series. I will be happy to step back if Christoph or someone else wants to take the lead here. The only reason I started looking into this is that there was no further progress on the original e-mail thread about this subject (http://thread.gmane.org/gmane.linux.kernel/2052359).
Bart.
Re: [PATCH 2/2] Restart list search after unlock in scsi_remove_target
Bart, can you resend your patch on top of 4.4-rc1? I think we really need it so we should get it into 4.4 and backport it to -stable.
Re: [PATCH 2/2] Restart list search after unlock in scsi_remove_target
In reply to James BottomleyOn Wed, Nov 4, 2015 at 2:44 PM, James Bottomley
<[email protected]> wrote:
[..]
Quoted 7 lines
The fundamental problem with this is how have the conditions that caused us to move away from list restart:
commit
bc3f02a795d3b4faa99d37390174be2a75d091bdAuthor: Dan Williams <[email protected]> Date: Tue Aug 28 22:12:10 2012 -0700
Quoted 8 lines
[SCSI]
scsi_remove_target: fix softlockup regression on hot removeWhich was triggered by this bug report
http://thread.gmane.org/gmane.linux.kernel/1348679
been mitigated?
I think it has because the problem that led to that report was the fact that scsi_target_destroy() did not advance the target state, but we changed that in f2495e228fce.
James Bottomley
maintainerRe: [PATCH 2/2] Restart list search after unlock in scsi_remove_target
Quoted 16 lines · Dan Williams
On Wed, Nov 4, 2015 at 2:44 PM, James Bottomley
<[email protected]> wrote:
[..]Quoted 13 lines
The fundamental problem with this is how have the conditions that caused us to move away from list restart:
commit
bc3f02a795d3b4faa99d37390174be2a75d091bdAuthor: Dan Williams <[email protected]> Date: Tue Aug 28 22:12:10 2012 -0700[SCSI]
scsi_remove_target: fix softlockup regression on hot removeWhich was triggered by this bug report
Quoted 8 lines
Quoted 2 lines
been mitigated?
I think it has because the problem that led to that report was the fact that
scsi_target_destroy()did not advance the target state, but we changed that inf2495e228fce.
Great, thanks, we'll put Christoph's version in then, because it can be cc'd to stable without problems. Bart, you can redo your state updates on top of this because that's inessential to the current problem.
James
Christoph Hellwig
Reviewed-byRe: [PATCH 2/2] Restart list search after unlock in scsi_remove_target
In reply to Bart Van AsscheQuoted 5 lines · Bart Van Assche
(replying to my own e-mail)
Hello Christoph,
Is it OK for you if I mention you as author of this e-mail ?
I don't care whom this fix is attributed to, but let's get it in:
or
at your preference :)