scsi: core: move scsi_host_busy() out of host lock for waking up EH handler
PATCH 1/1 scsi: core: move scsi_host_busy() out of host lock for waking up EH handler
Inside scsi_eh_wakeup(), scsi_host_busy() is called & checked with host lock every time for deciding if error handler kthread needs to be waken up.
This way can be too heavy in case of recovery, such as:
- N hardware queues
- queue depth is M for each hardware queue
- each scsi_host_busy() iterates over (N * M) tag/requests
If recovery is triggered in case that all requests are in-flight, each scsi_eh_wakeup() is strictly serialized, when scsi_eh_wakeup() is called for the last in-flight request, scsi_host_busy() has been run for (N * M - 1) times, and request has been iterated for (N*M - 1) * (N * M) times.
If both N and M are big enough, hard lockup can be triggered on acquiring host lock, and it is observed on mpi3mr(128 hw queues, queue depth 8169).
Fix the issue by calling scsi_host_busy() outside host lock, and we don't need host lock for getting busy count because host lock never covers that.
---
BTW, another way is to wake up EH handler unconditionally, and it should work too.
drivers/scsi/scsi_error.c | 11 +++++++---- drivers/scsi/scsi_lib.c | 4 +++- drivers/scsi/scsi_priv.h | 2 +- 3 files changed, 11 insertions(+), 6 deletions(-)
drivers/scsi/scsi_error.c +7 -4
@@ -61,11 +61,11 @@ static int scsi_eh_try_stu(struct scsi_cmnd *scmd);static enum scsi_disposition scsi_try_to_abort_cmd (const struct scsi_host_template *,struct scsi_cmnd *); -void scsi_eh_wakeup (struct Scsi_Host *shost) +void scsi_eh_wakeup (struct Scsi_Host *shost,unsigned int busy) {lockdep_assert_held (shost->host_lock); -if (scsi_host_busy (shost) == shost->host_failed) { +if (busy == shost->host_failed) {trace_scsi_eh_wakeup (shost);wake_up_process (shost->ehandler); SCSI_LOG_ERROR_RECOVERY(5, shost_printk(KERN_INFO, shost,
@@ -87,8 +87,10 @@ void scsi_schedule_eh(struct Scsi_Host *shost)if (scsi_host_set_state (shost,SHOST_RECOVERY ) ==0 ||scsi_host_set_state (shost,SHOST_CANCEL_RECOVERY ) ==0 ) { +unsigned int busy =scsi_host_busy (shost); + shost->host_eh_scheduled++; -scsi_eh_wakeup (shost); +scsi_eh_wakeup (shost, busy); }spin_unlock_irqrestore (shost->host_lock, flags);
@@ -283,10 +285,11 @@ static void scsi_eh_inc_host_failed(struct rcu_head *head)struct scsi_cmnd *scmd =container_of (head,typeof (*scmd), rcu);struct Scsi_Host *shost = scmd->device->host;unsigned long flags; +unsigned int busy =scsi_host_busy (shost);spin_lock_irqsave (shost->host_lock, flags); shost->host_failed++; -scsi_eh_wakeup (shost); +scsi_eh_wakeup (shost, busy);spin_unlock_irqrestore (shost->host_lock, flags); }
drivers/scsi/scsi_lib.c +3 -1
@@ -278,9 +278,11 @@ static void scsi_dec_host_busy(struct Scsi_Host *shost, struct scsi_cmnd *cmd)rcu_read_lock ();__clear_bit (SCMD_STATE_INFLIGHT , &cmd->state);if (unlikely (scsi_host_in_recovery (shost))) { +unsigned int busy =scsi_host_busy (shost); +spin_lock_irqsave (shost->host_lock, flags);if (shost->host_failed || shost->host_eh_scheduled) -scsi_eh_wakeup (shost); +scsi_eh_wakeup (shost, busy);spin_unlock_irqrestore (shost->host_lock, flags); }rcu_read_unlock ();
drivers/scsi/scsi_priv.h +1 -1
@@ -92,7 +92,7 @@ extern void scmd_eh_abort_handler(struct work_struct *work);extern enum blk_eh_timer_return scsi_timeout (struct request *req);extern int scsi_error_handler (void *host);extern enum scsi_disposition scsi_decide_disposition (struct scsi_cmnd *cmd); -extern void scsi_eh_wakeup (struct Scsi_Host *shost); +extern void scsi_eh_wakeup (struct Scsi_Host *shost,unsigned int busy);extern void scsi_eh_scmd_add (struct scsi_cmnd *); void scsi_eh_ready_devs(struct Scsi_Host *shost, struct list_head *work_q,
Signature · 1 line
2.42.0
Quoted 14 lines · Ming Lei
Inside
scsi_eh_wakeup(),scsi_host_busy()is called & checked with host lock every time for deciding if error handler kthread needs to be waken up.This way can be too heavy in case of recovery, such as:
- N hardware queues
- queue depth is M for each hardware queue
- eachscsi_host_busy()iterates over (N * M) tag/requestsIf recovery is triggered in case that all requests are in-flight, each
scsi_eh_wakeup()is strictly serialized, whenscsi_eh_wakeup()is called for the last in-flight request,scsi_host_busy()has been run for (N * M - 1) times, and request has been iterated for (N*M - 1) * (N * M) times.
Quoted 7 lines
If both N and M are big enough, hard lockup can be triggered on acquiring host lock, and it is observed on
mpi3mr(128 hw queues, queue depth 8169).Fix the issue by calling
scsi_host_busy()outside host lock, and we don't need host lock for getting busy count because host lock never covers that.
Can you share details for the hard lockup?
I do agree that scsi_host_busy() is an expensive operation, so it might not be ideal to call it under a spin lock.
But I wonder where the lockup comes in here.
Care to explain?
And if it leads to a lockup, aren't other instances calling scsi_host_busy() under a spinlock affected, as well?
Cheers,
Hannes
Quoted 22 lines · Hannes Reinecke
Quoted 14 lines · Ming Lei
Inside
scsi_eh_wakeup(),scsi_host_busy()is called & checked with host lock every time for deciding if error handler kthread needs to be waken up.This way can be too heavy in case of recovery, such as:
- N hardware queues
- queue depth is M for each hardware queue
- eachscsi_host_busy()iterates over (N * M) tag/requestsIf recovery is triggered in case that all requests are in-flight, each
scsi_eh_wakeup()is strictly serialized, whenscsi_eh_wakeup()is called for the last in-flight request,scsi_host_busy()has been run for (N * M - 1) times, and request has been iterated for (N*M - 1) * (N * M) times.Quoted 7 lines
If both N and M are big enough, hard lockup can be triggered on acquiring host lock, and it is observed on
mpi3mr(128 hw queues, queue depth 8169).Fix the issue by calling
scsi_host_busy()outside host lock, and we don't need host lock for getting busy count because host lock never covers that.
Quoted 5 lines
Can you share details for the hard lockup?
I do agree thatscsi_host_busy()is an expensive operation, so it might not be ideal to call it under a spin lock.
But I wonder where the lockup comes in here.
Care to explain?
Recovery happens when there is N * M inflight requests, then scsi_dec_host_busy() can be called for each inflight request/scmnd from irq context.
host lock serializes every scsi_eh_wakeup().
Given each hardware queue has its own irq handler, so there could be one request, scsi_dec_host_busy() is called and the host lock is spinned until it is released from scsi_dec_host_busy() for all requests from all other hardware queues.
The spin time can be long enough to trigger the hard lockup if N and M is big enough, and the total wait time can be:
(N - 1) * M * time_taken_in_scsi_host_busy().
Meantime the same story happens on scsi_eh_inc_host_failed() which is called from softirq context, so host lock spin can be much more worse.
It is observed on mpi3mr with 128(N) hw queues and 8169(M) queue depth.
Quoted 3 lines
And if it leads to a lockup, aren't other instances calling
scsi_host_busy()under a spinlock affected, as well?
It is only possible when it is called in per-command situation.
Thanks,
Ming
Ewan Milne
Reviewed-byQuoted 51 lines · Ming Lei
Quoted 22 lines · Hannes Reinecke
Quoted 14 lines · Ming Lei
Inside
scsi_eh_wakeup(),scsi_host_busy()is called & checked with host lock every time for deciding if error handler kthread needs to be waken up.This way can be too heavy in case of recovery, such as:
- N hardware queues
- queue depth is M for each hardware queue
- eachscsi_host_busy()iterates over (N * M) tag/requestsIf recovery is triggered in case that all requests are in-flight, each
scsi_eh_wakeup()is strictly serialized, whenscsi_eh_wakeup()is called for the last in-flight request,scsi_host_busy()has been run for (N * M - 1) times, and request has been iterated for (N*M - 1) * (N * M) times.Quoted 7 lines
If both N and M are big enough, hard lockup can be triggered on acquiring host lock, and it is observed on
mpi3mr(128 hw queues, queue depth 8169).Fix the issue by calling
scsi_host_busy()outside host lock, and we don't need host lock for getting busy count because host lock never covers that.Quoted 5 lines
Can you share details for the hard lockup?
I do agree thatscsi_host_busy()is an expensive operation, so it might not be ideal to call it under a spin lock.
But I wonder where the lockup comes in here.
Care to explain?Recovery happens when there is N * M inflight requests, then
scsi_dec_host_busy()can be called for each inflight request/scmnd from irq context.host lock serializes every
scsi_eh_wakeup().Given each hardware queue has its own irq handler, so there could be one request,
scsi_dec_host_busy()is called and the host lock is spinned until it is released fromscsi_dec_host_busy()for all requests from all other hardware queues.The spin time can be long enough to trigger the hard lockup if N and M is big enough, and the total wait time can be:
(N - 1) * M * time_taken_in_scsi_host_busy().Meantime the same story happens on
scsi_eh_inc_host_failed()which is called from softirq context, so host lock spin can be much more worse.It is observed on mpi3mr with 128(N) hw queues and 8169(M) queue depth.
Quoted 1 line
Quoted 9 lines
Quoted 2 lines
And if it leads to a lockup, aren't other instances calling
scsi_host_busy()under a spinlock affected, as well?It is only possible when it is called in per-command situation.
Thanks,
Ming
I can't see why this wouldn't work, or cause a problem with a lost wakeup, but the cost of iterating to obtain the host_busy value is still being paid, just outside the host_lock. If this has triggered a hard lockup, should we revisit the algorithm, e.g. are we still delaying EH wakeup for a noticeable amount of time? O(n^2) algorithms in the kernel don't seem like the best idea.
In any case...
-Ewan
Quoted 62 lines · Ewan Milne
Quoted 51 lines · Ming Lei
Quoted 22 lines · Hannes Reinecke
Quoted 14 lines · Ming Lei
Inside
scsi_eh_wakeup(),scsi_host_busy()is called & checked with host lock every time for deciding if error handler kthread needs to be waken up.This way can be too heavy in case of recovery, such as:
- N hardware queues
- queue depth is M for each hardware queue
- eachscsi_host_busy()iterates over (N * M) tag/requestsIf recovery is triggered in case that all requests are in-flight, each
scsi_eh_wakeup()is strictly serialized, whenscsi_eh_wakeup()is called for the last in-flight request,scsi_host_busy()has been run for (N * M - 1) times, and request has been iterated for (N*M - 1) * (N * M) times.Quoted 7 lines
If both N and M are big enough, hard lockup can be triggered on acquiring host lock, and it is observed on
mpi3mr(128 hw queues, queue depth 8169).Fix the issue by calling
scsi_host_busy()outside host lock, and we don't need host lock for getting busy count because host lock never covers that.Quoted 5 lines
Can you share details for the hard lockup?
I do agree thatscsi_host_busy()is an expensive operation, so it might not be ideal to call it under a spin lock.
But I wonder where the lockup comes in here.
Care to explain?Recovery happens when there is N * M inflight requests, then
scsi_dec_host_busy()can be called for each inflight request/scmnd from irq context.host lock serializes every
scsi_eh_wakeup().Given each hardware queue has its own irq handler, so there could be one request,
scsi_dec_host_busy()is called and the host lock is spinned until it is released fromscsi_dec_host_busy()for all requests from all other hardware queues.The spin time can be long enough to trigger the hard lockup if N and M is big enough, and the total wait time can be:
(N - 1) * M * time_taken_in_scsi_host_busy().Meantime the same story happens on
scsi_eh_inc_host_failed()which is called from softirq context, so host lock spin can be much more worse.It is observed on mpi3mr with 128(N) hw queues and 8169(M) queue depth.
Quoted 1 line
Quoted 9 lines
Quoted 2 lines
And if it leads to a lockup, aren't other instances calling
scsi_host_busy()under a spinlock affected, as well?It is only possible when it is called in per-command situation.
Thanks,
Ming
Quoted 5 lines
I can't see why this wouldn't work, or cause a problem with a lost wakeup, but the cost of iterating to obtain the
host_busyvalue is still being paid, just outside thehost_lock. If this has triggered a hard lockup, should we revisit the algorithm, e.g. are we still delaying EH wakeup for a noticeable amount of time?
SCSI EH is designed to start handling until all in-flight commands are failed, so it waits until all requests are failed first.
Quoted 1 line
O(n^2) algorithms in the kernel don't seem like the best idea.
It is actually O(n) because each hardware queue handles request in parallel.
It is degraded to O(n^2) or O(n * m) just because of shared host lock.
Single or N scsi_host_busy() won't take too long without host lock, what matters is actually the per-host lock spin time which can be accumulated as too big.
Quoted 3 lines
In any case...
Reviewed-by Ewan D. Milne <[email protected]>
Thanks for the review!
Signature · 1 line
Ming
Quoted 85 lines · Ming Lei
Quoted 62 lines · Ewan Milne
Quoted 51 lines · Ming Lei
Quoted 22 lines · Hannes Reinecke
Quoted 14 lines · Ming Lei
Inside
scsi_eh_wakeup(),scsi_host_busy()is called & checked with host lock every time for deciding if error handler kthread needs to be waken up.This way can be too heavy in case of recovery, such as:
- N hardware queues
- queue depth is M for each hardware queue
- eachscsi_host_busy()iterates over (N * M) tag/requestsIf recovery is triggered in case that all requests are in-flight, each
scsi_eh_wakeup()is strictly serialized, whenscsi_eh_wakeup()is called for the last in-flight request,scsi_host_busy()has been run for (N * M - 1) times, and request has been iterated for (N*M - 1) * (N * M) times.Quoted 7 lines
If both N and M are big enough, hard lockup can be triggered on acquiring host lock, and it is observed on
mpi3mr(128 hw queues, queue depth 8169).Fix the issue by calling
scsi_host_busy()outside host lock, and we don't need host lock for getting busy count because host lock never covers that.Quoted 5 lines
Can you share details for the hard lockup?
I do agree thatscsi_host_busy()is an expensive operation, so it might not be ideal to call it under a spin lock.
But I wonder where the lockup comes in here.
Care to explain?Recovery happens when there is N * M inflight requests, then
scsi_dec_host_busy()can be called for each inflight request/scmnd from irq context.host lock serializes every
scsi_eh_wakeup().Given each hardware queue has its own irq handler, so there could be one request,
scsi_dec_host_busy()is called and the host lock is spinned until it is released fromscsi_dec_host_busy()for all requests from all other hardware queues.The spin time can be long enough to trigger the hard lockup if N and M is big enough, and the total wait time can be:
(N - 1) * M * time_taken_in_scsi_host_busy().Meantime the same story happens on
scsi_eh_inc_host_failed()which is called from softirq context, so host lock spin can be much more worse.It is observed on mpi3mr with 128(N) hw queues and 8169(M) queue depth.
Quoted 1 line
Quoted 9 lines
Quoted 2 lines
And if it leads to a lockup, aren't other instances calling
scsi_host_busy()under a spinlock affected, as well?It is only possible when it is called in per-command situation.
Thanks,
MingQuoted 5 lines
I can't see why this wouldn't work, or cause a problem with a lost wakeup, but the cost of iterating to obtain the
host_busyvalue is still being paid, just outside thehost_lock. If this has triggered a hard lockup, should we revisit the algorithm, e.g. are we still delaying EH wakeup for a noticeable amount of time?SCSI EH is designed to start handling until all in-flight commands are failed, so it waits until all requests are failed first.
Quoted 1 line
O(n^2) algorithms in the kernel don't seem like the best idea.
It is actually O(n) because each hardware queue handles request in parallel.
It is degraded to O(n^2) or
O(n * m)just because of shared host lock.Single or N
scsi_host_busy()won't take too long without host lock, what matters is actually the per-host lock spin time which can be accumulated as too big.Quoted 1 line
Quoted 10 lines
Quoted 2 lines
In any case...
Reviewed-by Ewan D. Milne <[email protected]>Thanks for the review!
--
Ming
drivers/scsi/scsi_error.c void scsi_schedule_eh(struct Scsi_Host *shost) · line 94
Quoted 10 lines · Ming Lei
drivers/scsi/scsi_error.c +3
-1@@ -87,8 +87,10 @@ void scsi_schedule_eh(struct Scsi_Host *shost)if (scsi_host_set_state (shost,SHOST_RECOVERY ) ==0 ||scsi_host_set_state (shost,SHOST_CANCEL_RECOVERY ) ==0 ) { +unsigned int busy =scsi_host_busy (shost); + shost->host_eh_scheduled++; -scsi_eh_wakeup (shost); +scsi_eh_wakeup (shost, busy); }
No new variable is needed here. If this patch is reposted, please change the above into the following:
+ scsi_eh_wakeup(shost, scsi_host_busy(shost));
With or without that additional change:
Martin K. Petersen
AppliedQuoted 10 lines · Ming Lei
Inside
scsi_eh_wakeup(),scsi_host_busy()is called & checked with host lock every time for deciding if error handler kthread needs to be waken up.This way can be too heavy in case of recovery, such as:
- N hardware queues
- queue depth is M for each hardware queue
- eachscsi_host_busy()iterates over (N * M) tag/requests[...]
Applied to 6.8/scsi-fixes, thanks!
[1/1] scsi: core: move scsi_host_busy() out of host lock for waking up EH handler
https://git.kernel.org/mkp/scsi/c/4373534a9850
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Quoted 12 lines · Martin K. Petersen
Quoted 9 lines · Ming Lei
Inside
scsi_eh_wakeup(),scsi_host_busy()is called & checked with host lock every time for deciding if error handler kthread needs to be waken up.This way can be too heavy in case of recovery, such as:
- N hardware queues
- queue depth is M for each hardware queue
- eachscsi_host_busy()iterates over (N * M) tag/requests
Quoted 6 lines
Quoted 1 line
[...]
Applied to 6.8/scsi-fixes, thanks!
[1/1] scsi: core: move
scsi_host_busy()out of host lock for waking up EH handlerhttps://git.kernel.org/mkp/scsi/c/4373534a9850
Hi Martin,
When I started to backport this commit, I found it was merged as wrong, the point is that scsi_host_busy() needs to be moved out of host lock.
I will send one new patch to fix it.
Thanks,
Ming