scsi: Fix dm-multipath starvation when scsi host is busy
Subsystems (2)
PATCH 1/1 scsi: Fix dm-multipath starvation when scsi host is busy
When a scsi host is kept busy by a dm-mpath device, other dm-mpath device on the same host could be starved. For example:
http://www.redhat.com/archives/dm-devel/2012-May/msg00123.html
It happens because dm-mpath delays request submission when the underlying scsi host is busy even if sdev is not busy.
For case like this, it is better to send the request down and let scsi do appropriate starvation control over the shared resource.
Though it might seem odd to change scsi's definition of a LLD being "busy", it is reasonable because scsi_lld_busy (and blk_lld_busy) was introduced to provide a hint for request-based stacking driver (i.e. dm-multipath) and dm-multipath is the only user of this function.
--- drivers/scsi/scsi_lib.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/scsi/scsi_lib.c +1 -4
@@ -1378,16 +1378,13 @@ static int scsi_lld_busy(struct request_queue *q) {struct scsi_device *sdev = q->queuedata;struct Scsi_Host *shost; -struct scsi_target *starget;if (!sdev)return 0 ; shost = sdev->host; - starget =scsi_target (sdev); -if (scsi_host_in_recovery (shost) ||scsi_host_is_busy (shost) || -scsi_target_is_busy (starget) ||scsi_device_is_busy (sdev)) +if (scsi_host_in_recovery (shost) ||scsi_device_is_busy (sdev))return 1 ;return 0 ;
James Bottomley
maintainerQuoted 7 lines · Jun'ichi Nomura
When a scsi host is kept busy by a dm-mpath device, other dm-mpath device on the same host could be starved. For example:
http://www.redhat.com/archives/dm-devel/2012-May/msg00123.htmlIt happens because dm-mpath delays request submission when the underlying scsi host is busy even if sdev is not busy.
Quoted 7 lines
For case like this, it is better to send the request down and let scsi do appropriate starvation control over the shared resource.
Though it might seem odd to change scsi's definition of a LLD being "busy", it is reasonable because
scsi_lld_busy(andblk_lld_busy) was introduced to provide a hint for request-based stacking driver (i.e. dm-multipath) and dm-multipath is the only user of this function.
This explanation is rather dense.
Isn't a more understandable explanation:
block congestion control doesn't have any concept of fairness across multiple queues. This means that if SCSI reports the host as busy in the queue congestion control it can result in an unfair starvation situation in dm-mp if there are multiple multipath devices on the same host. The fix for this is to report only the sdev busy state (and ignore the host busy state) in the block congestion control call back. The host is still congested, but the SCSI subsystem will sort out the congestion in a fair way because it knows the relation between the queues and the host.
And please put a comment in the code as well otherwise someone will eventually send a "fix" for this because we're not paying attention to host busy (and I'll have forgotten about the issue by then and might apply it).
A final note is that this is more a band aid than a fix because this is still a congestion situation dm-mp should be aware of.
James
PATCH v2 1/1 scsi: Fix dm-multipath starvation when scsi host is busy
Quoted 1 line · James Bottomley
Isn't a more understandable explanation:
Thank you. That's quite elegant.
I replaced the description with it.
Quoted 4 lines
And please put a comment in the code as well otherwise someone will eventually send a "fix" for this because we're not paying attention to host busy (and I'll have forgotten about the issue by then and might apply it).
Added the comment in code.
Quoted 2 lines
A final note is that this is more a band aid than a fix because this is still a congestion situation dm-mp should be aware of.
Yes. To do that, we have to generalize the concept of "host" and share it with block layer.
Attached below is the revised patch.
----------------------------------------------------------
[PATCH v2] scsi: Fix dm-multipath starvation when scsi host is busy
block congestion control doesn't have any concept of fairness across multiple queues. This means that if SCSI reports the host as busy in the queue congestion control it can result in an unfair starvation situation in dm-mp if there are multiple multipath devices on the same host. For example:
http://www.redhat.com/archives/dm-devel/2012-May/msg00123.html
The fix for this is to report only the sdev busy state (and ignore the host busy state) in the block congestion control call back. The host is still congested, but the SCSI subsystem will sort out the congestion in a fair way because it knows the relation between the queues and the host.
--- drivers/scsi/scsi_lib.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
drivers/scsi/scsi_lib.c +7 -4
@@ -1378,16 +1378,19 @@ static int scsi_lld_busy(struct request_queue *q) {struct scsi_device *sdev = q->queuedata;struct Scsi_Host *shost; -struct scsi_target *starget;if (!sdev)return 0 ; shost = sdev->host; - starget =scsi_target (sdev); -if (scsi_host_in_recovery (shost) ||scsi_host_is_busy (shost) || -scsi_target_is_busy (starget) ||scsi_device_is_busy (sdev)) +/* +* Ignore host/starget busy state. +* Since block layer does not have a concept of fairness across +* multiple queues, congestion of host/starget needs to be handled +* in SCSI layer. +*/ +if (scsi_host_in_recovery (shost) ||scsi_device_is_busy (sdev))return 1 ;return 0 ;
Quoted 37 lines · Jun'ichi Nomura
Quoted 1 line · James Bottomley
Isn't a more understandable explanation:
Thank you. That's quite elegant.
I replaced the description with it.Quoted 4 lines
And please put a comment in the code as well otherwise someone will eventually send a "fix" for this because we're not paying attention to host busy (and I'll have forgotten about the issue by then and might apply it).
Added the comment in code.
Quoted 2 lines
A final note is that this is more a band aid than a fix because this is still a congestion situation dm-mp should be aware of.
Yes. To do that, we have to generalize the concept of "host" and share it with block layer.
Attached below is the revised patch.
----------------------------------------------------------
[PATCH v2] scsi: Fix dm-multipath starvation when scsi host is busyblock congestion control doesn't have any concept of fairness across multiple queues. This means that if SCSI reports the host as busy in the queue congestion control it can result in an unfair starvation situation in dm-mp if there are multiple multipath devices on the same host. For example:
http://www.redhat.com/archives/dm-devel/2012-May/msg00123.htmlThe fix for this is to report only the sdev busy state (and ignore the host busy state) in the block congestion control call back. The host is still congested, but the SCSI subsystem will sort out the congestion in a fair way because it knows the relation between the queues and the host.
Quoted 5 lines
Reported-by Bernd Schubert <[email protected]>Tested-by Bernd Schubert <[email protected]>Signed-off-by Jun'ichi Nomura <[email protected]>Cc James Bottomley <[email protected]>Cc Mike Snitzer <[email protected]>
Hannes Reinecke
Acked-byQuoted 40 lines · Mike Snitzer
Quoted 37 lines · Jun'ichi Nomura
Quoted 1 line · James Bottomley
Isn't a more understandable explanation:
Thank you. That's quite elegant.
I replaced the description with it.Quoted 4 lines
And please put a comment in the code as well otherwise someone will eventually send a "fix" for this because we're not paying attention to host busy (and I'll have forgotten about the issue by then and might apply it).
Added the comment in code.
Quoted 2 lines
A final note is that this is more a band aid than a fix because this is still a congestion situation dm-mp should be aware of.
Yes. To do that, we have to generalize the concept of "host" and share it with block layer.
Attached below is the revised patch.
----------------------------------------------------------
[PATCH v2] scsi: Fix dm-multipath starvation when scsi host is busyblock congestion control doesn't have any concept of fairness across multiple queues. This means that if SCSI reports the host as busy in the queue congestion control it can result in an unfair starvation situation in dm-mp if there are multiple multipath devices on the same host. For example:
http://www.redhat.com/archives/dm-devel/2012-May/msg00123.htmlThe fix for this is to report only the sdev busy state (and ignore the host busy state) in the block congestion control call back. The host is still congested, but the SCSI subsystem will sort out the congestion in a fair way because it knows the relation between the queues and the host.
Quoted 8 lines
Quoted 5 lines
Reported-by Bernd Schubert <[email protected]>Tested-by Bernd Schubert <[email protected]>Signed-off-by Jun'ichi Nomura <[email protected]>Cc James Bottomley <[email protected]>Cc Mike Snitzer <[email protected]>Acked-by Mike Snitzer <[email protected]>
Cheers,
Hannes
Signature · 8 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html