queue_depth tracking from LLD
I just came across the SCSI midlayer function scsi_track_queue_full.
If a SCSI command is returned with a status of QUEUE_FULL, then this is mapped to ADD_TO_MLQUEUE and "device blocked". So, there is already a mechanism in place. Is a LLD driver expected to additionally call something like this to decrease the queue depth?
if (status_byte(scmd->result) == QUEUE_FULL)
scsi_track_queue_full(sdev, sdev->queue_depth - 1))
If a LLD does this, should it also increase the queue depth again when no more QUEUE_FULL status are seen? To me this looks like a duplication of the midlayer device blocking, but i assume there is a reason in having both, scsi_track_queue_full and the device blocking.
--
Christof Schmitt
The mid-layer queue depth handling is really designed/optimized around behavior for a JBOD. This, if it's a single-lun device, the LLDD could largely ignore doing anything with adjusting the queue depth.
However, for arrays, with multiple luns, the queue depth is usually a target-level resource, so the midlayer/block-layer's implementation falls on its face fairly quickly. I brought this up 2 yrs ago at storage summit. What needs to happen is the creation of queue ramp-down and ramp-up policies that can be selected on a per-lun basis, and have these implemented in the midlayer (why should the LLDD ever look at scsi command results). What will make this difficult is the ramp-up policies, as it can be very target device-specific or configuration/load centric.
In the meantime, if you look at any LLDD that is worth its salt, and it will be implementing it's own queue ramp-down and ramp-up algorithms internally. They will look for QUEUE_FULLs to ramp-down, and selecting a rate and methodology for the ramp-up. They will use this routine to do the queue depth changing.
-- james s
Quoted 23 lines · Christof Schmitt
I just came across the SCSI midlayer function
scsi_track_queue_full.If a SCSI command is returned with a status of
QUEUE_FULL, then this is mapped toADD_TO_MLQUEUEand "device blocked". So, there is already a mechanism in place. Is a LLD driver expected to additionally call something like this to decrease the queue depth?if (
status_byte(scmd->result)==QUEUE_FULL)scsi_track_queue_full(sdev, sdev->queue_depth - 1))If a LLD does this, should it also increase the queue depth again when no more
QUEUE_FULLstatus are seen? To me this looks like a duplication of the midlayer device blocking, but i assume there is a reason in having both,scsi_track_queue_fulland the device blocking.--
Christof Schmitt
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · James Smart
The mid-layer queue depth handling is really designed/optimized around behavior for a JBOD. This, if it's a single-lun device, the LLDD could largely ignore doing anything with adjusting the queue depth.
However, for arrays, with multiple luns, the queue depth is usually a target-level resource, so the midlayer/block-layer's implementation falls on its face fairly quickly. I brought this up 2 yrs ago at storage summit. What needs to happen is the creation of queue ramp-down and ramp-up policies that can be selected on a per-lun basis, and have these implemented in the midlayer (why should the LLDD ever look at scsi command results). What will make this difficult is the ramp-up policies, as it can be very target device-specific or configuration/load centric.
For the rampup are you referring to code like lpfc_rampup_queue_depth? Were were just talking about this on the fcoe list. Why did lpfc and qla2xxx end up implememting their own code? We started to look into moving this into the scsi layer. It does not seem like there was a major reason why it should not have been more common. Was it just one of those things where it got added in one driver then added in another?
If we moved code like that to the scsi layer, then is all the is needed is a interface to config this?
Quoted 15 lines · Mike Christie
Quoted 4 lines · James Smart
The mid-layer queue depth handling is really designed/optimized around behavior for a JBOD. This, if it's a single-lun device, the LLDD could largely ignore doing anything with adjusting the queue depth.
Quoted 8 lines
However, for arrays, with multiple luns, the queue depth is usually a target-level resource, so the midlayer/block-layer's implementation falls on its face fairly quickly. I brought this up 2 yrs ago at storage summit. What needs to happen is the creation of queue ramp-down and ramp-up policies that can be selected on a per-lun basis, and have these implemented in the midlayer (why should the LLDD ever look at scsi command results). What will make this difficult is the ramp-up policies, as it can be very target device-specific or configuration/load centric.
Quoted 2 lines
For the rampup are you referring to code like
lpfc_rampup_queue_depth? Were were just talking about this on the fcoe list. Why did lpfc and qla2xxx end up implememting their own code? We started to look into moving this into the scsi layer. It does not seem like there was a major reason why it should not have been more common. Was it just one of those things where it got added in one driver then added in another?
No good reason. It should be in the midlayer, and that was the recommendation I made at storage summit a couple of years ago. It hasn't as, for the drivers that care, they had already implemented it. It also isn't a relished task, as there will be lots of discussion on how the ramp-up should be implemented - which may mean, the need for more algorithms.
Quoted 2 lines
If we moved code like that to the scsi layer, then is all the is needed is a interface to config this?
Yep. As mentioned, figuring out what algorithm, for what device and configuration, will be the more interesting thing.
-- james
Quoted 1 line · James Smart
Mike Christie wrote:
[...]
Quoted 6 lines
No good reason. It should be in the midlayer, and that was the recommendation I made at storage summit a couple of years ago. It hasn't as, for the drivers that care, they had already implemented it. It also isn't a relished task, as there will be lots of discussion on how the ramp-up should be implemented - which may mean, the need for more algorithms.
zfcp is one of the drivers that don't have ramp-up/down mechanism in place. And i am trying to understand what is required here. Is there currently work being done to get queue_depth ramp-up/down in the midlayer?
Quoted 5 lines
Quoted 2 lines
If we moved code like that to the scsi layer, then is all the is needed is a interface to config this?
Yep. As mentioned, figuring out what algorithm, for what device and configuration, will be the more interesting thing.
If the LLDs that currently have a private ramp-up/down mechanism in place use a similar strategy, would it make sense to first move them to common code that can be activated from a LLD? And later refine it with device-specific behaviour?
--
Christof Schmitt
Quoted 4 lines · Christof Schmitt
zfcp is one of the drivers that don't have ramp-up/down mechanism in place. And i am trying to understand what is required here. Is there currently work being done to get
queue_depthramp-up/down in the midlayer?
No work started - but desired.
I'd recommend, rather than implementing it in zfcp, implement it in the midlayer for everyone.
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines
If we moved code like that to the scsi layer, then is all the is needed is a interface to config this?
Yep. As mentioned, figuring out what algorithm, for what device and configuration, will be the more interesting thing.
If the LLDs that currently have a private ramp-up/down mechanism in place use a similar strategy, would it make sense to first move them to common code that can be activated from a LLD? And later refine it with device-specific behaviour?
Yes.
-- james s
Quoted 4 lines · James Smart
However, for arrays, with multiple luns, the queue depth is usually a target-level resource,
so the midlayer/block-layer's implementation falls on its face fairly quickly. I brought this
If the problem were as simple as the resource being target-level instead of LUN-level, it would be fairly easy to fix (we could do accounting per-target instead of per-LUN). The problem, AIUI, is multi-initiator where you can't know whether resources are in use or not.
Quoted 9 lines
up 2 yrs ago at storage summit. What needs to happen is the creation of queue ramp-down
and ramp-up policies that can be selected on a per-lun basis, and have these implemented
in the midlayer (why should the LLDD ever look at scsi command results). What will make
this difficult is the ramp-up policies, as it can be very target device-specific or configuration/load
centric.
While not disagreeing that it's complex, I don't think putting it in the driver makes it less complex. I completely agree that LLDDs should not be snooping scsi commands or scsi command results. It should all be in the midlayer so we all share the same bugs ;-)
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Completely Agree. The multi-initiator point is the one I try to hammer home. It's what the current algorithm completely misses.
Even though I said its complex - it's really not that difficult. The pain is just figuring out what to group and what the rates should be.
-- james s
Quoted 25 lines · Matthew Wilcox
Quoted 3 lines · James Smart
However, for arrays, with multiple luns, the queue depth is usually a target-level resource, so the midlayer/block-layer's implementation falls on its face fairly quickly. I brought this
If the problem were as simple as the resource being target-level instead of LUN-level, it would be fairly easy to fix (we could do accounting per-target instead of per-LUN). The problem, AIUI, is multi-initiator where you can't know whether resources are in use or not.
Quoted 6 lines
up 2 yrs ago at storage summit. What needs to happen is the creation of queue ramp-down and ramp-up policies that can be selected on a per-lun basis, and have these implemented in the midlayer (why should the LLDD ever look at scsi command results). What will make this difficult is the ramp-up policies, as it can be very target device-specific or configuration/load centric.
While not disagreeing that it's complex, I don't think putting it in the driver makes it less complex. I completely agree that LLDDs should not be snooping scsi commands or scsi command results. It should all be in the midlayer so we all share the same bugs ;-)