scsi-mq performance check

Hi,

I have started to enable scsi-mq on the HiSilicon SAS driver.

Are there hints/checks I should use to make sure it is configured correctly/optimally? In my initial testing I have seen some performance improvements, but none like what I have seen in presentations.

Quoted 9 lines · John Garry

Hi,

I have started to enable scsi-mq on the HiSilicon SAS driver.

Are there hints/checks I should use to make sure it is configured correctly/optimally? In my initial testing I have seen some performance improvements, but none like what I have seen in presentations.

The whole thing is build around having symmetric submit and receive queues, so that we can tack a send/receive queue pair to the same CPU. With that we can ensure that we don't have any cache invalidation, as the request is already in the cache for that CPU when the completion is recieved. _And_ we can get rid of most spinlocks as other CPUs cannot access our request.

So make sure to have the submit and receive queues properly done, and ensure you don't have any global resources within your driver which needs to be locked. Or move access to those resources out of the fast path.

Quoted 17 lines · Hannes Reinecke

Quoted 9 lines · John Garry

Hi,

I have started to enable scsi-mq on the HiSilicon SAS driver.

Are there hints/checks I should use to make sure it is configured correctly/optimally? In my initial testing I have seen some performance improvements, but none like what I have seen in presentations.

The whole thing is build around having symmetric submit and receive queues, so that we can tack a send/receive queue pair to the same CPU. With that we can ensure that we don't have any cache invalidation, as the request is already in the cache for that CPU when the completion is recieved. _And_ we can get rid of most spinlocks as other CPUs cannot access our request.

Quoted 4 lines

So make sure to have the submit and receive queues properly done, and ensure you don't have any global resources within your driver which needs to be locked. Or move access to those resources out of the fast path.

Hello John,

It's great news that you started looking into scsi-mq support :-) As Hannes wrote, if the performance improvement is not as big as you expected this could be caused e.g. by lock contention. Are you familiar with the perf tool ? The perf tool can be a great help to verify whether lock contention occurs and also which lock(s) cause it.

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 39 lines · Bart Van Assche

Quoted 21 lines · Hannes Reinecke

Quoted 9 lines · John Garry

Hi,

I have started to enable scsi-mq on the HiSilicon SAS driver.

Are there hints/checks I should use to make sure it is configured correctly/optimally? In my initial testing I have seen some performance improvements, but none like what I have seen in presentations.

The whole thing is build around having symmetric submit and receive queues, so that we can tack a send/receive queue pair to the same CPU.
With that we can ensure that we don't have any cache invalidation, as
the request is already in the cache for that CPU when the completion is
recieved. _And_ we can get rid of most spinlocks as other CPUs cannot
access our request.

Quoted 6 lines

So make sure to have the submit and receive queues properly done, and
ensure you don't have any global resources within your driver which needs to be locked. Or move access to those resources out of the fast
path.

Hello John,

It's great news that you started looking into scsi-mq support :-) As Hannes wrote, if the performance improvement is not as big as you expected this could be caused e.g. by lock contention. Are you familiar
with the perf tool ? The perf tool can be a great help to verify whether
lock contention occurs and also which lock(s) cause it.

Quoted 7 lines

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Thanks for the replies.

One of my main concerns is how we use a spinlock in our task exec function to prepare and deliver a frame to the hardware:
hisi_sas_task_exec()

{
     ...

     /* protect task_prep and start_delivery sequence */
     spin_lock_irqsave(&hisi_hba->lock, flags);
     rc = hisi_sas_task_prep(task, hisi_hba, is_tmf, tmf, &pass);
     ...
     hisi_hba->hw->start_delivery(hisi_hba);
     spin_unlock_irqrestore(&hisi_hba->lock, flags);

     ...
}

We have to lock due to how we reserve a slot in the delivery queue. We are looking to optimise this, but it's not straightforward.

Perf is a good strategy, but, to be honest, I have not spent a lot of time looking at this so I'm looking for low hanging fruit initially.

FYI, our hardware does have the same number of delivery and completion queues (32), and 16 cores. One thing to note is that a command which was sent on queue x is not quaranteed to complete on queue y.

Quoted 4 lines

We have to lock due to how we reserve a slot in the delivery queue. We are looking to optimise this, but it's not straightforward.

Quoted 8 lines

Perf is a good strategy, but, to be honest, I have not spent a lot of time looking at this so I'm looking for low hanging fruit initially.

FYI, our hardware does have the same number of delivery and completion queues (32), and 16 cores. One thing to note is that a command which was sent on queue x is not quaranteed to complete on queue y.

... then don't bother looking at scsi-mq. That is the very thing it relies on ...

Time to change the firmware?

Cheers,

Hannes