SMP affinity and the blk-mq I/O submission path
Hi Jens,
I've taken a pick at your mpt3sas branch, trying to make the mpt3sas multiqueue aware. Sadly I've encountered some issues on the way, which make me think we _might_ run into a locking issue.
Which also might be attributed to my imperfect knowledge of multiqueue submission and SMP affinity in general, hence my question:
Quoted 1 line
From my understanding blk-mq works due to the fact that I/O submission
is local to the CPU, and hence we won't need (most) locks. However, looking at the code I find that indeed the software queues are CPU-local, but the hardware context is not.
Plus I haven't really seen any provisions for ensure the hardware context (and hence the actual I/O submission) is running on any given CPU. So if I have a device with just one MSI-X vector (bloody mpt2sas firmware restriction) I would assign this vector to a specific CPU. But then I have the problem that I/O submission can in principle happen on _any_ CPU, whereas I/O completion will only happen on a specific CPU. Which consequently means that we might run into locking issue if the I/O submission happens on another CPU than the I/O completion.
Is this intended?
Plus, wouldn't we have a cacheline bouncing between those CPUs? Shouldn't rather run the I/O submission in the same CPU than the I/O completion?
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg)
Quoted 8 lines · Hannes Reinecke
Hi Jens,
I've taken a pick at your mpt3sas branch, trying to make the mpt3sas multiqueue aware. Sadly I've encountered some issues on the way, which make me think we _might_ run into a locking issue.
Which also might be attributed to my imperfect knowledge of multiqueue submission and SMP affinity in general, hence my question:
Quoted 6 lines
From my understanding blk-mq works due to the fact that I/O submission is local to the CPU, and hence we won't need (most) locks. However, looking at the code I find that indeed the software queues are CPU-local, but the hardware context is not.
Plus I haven't really seen any provisions for ensure the hardware context (and hence the actual I/O submission) is running on any given CPU=
.
Quoted 6 lines
So if I have a device with just one MSI-X vector (bloody mpt2sas firmware restriction) I would assign this vector to a specific CPU. But then I have the problem that I/O submission can in principle happen on _any_ CPU, whereas I/O completion will only happen on a specific CPU. Which consequently means that we might run into locking issue if the I/O submission happens on another CPU than the I/O completion.
The hardware queue lock needn't in I/O completion path.
Quoted 4 lines
Is this intended?
Plus, wouldn't we have a cacheline bouncing between those CPUs? Shouldn't rather run the I/O submission in the same CPU than the I/O completion?
The default mq flag includes QUEUE_FLAG_SAME_COMP, which should make the completion handler run in same LLC domain with the submission CPU, please see blk_mq_ipi_complete_request(). But the driver need to define .complete callback of blk_mq_ops.
Thanks,
Ming
Quoted 14 lines
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
[email protected] +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N=C3=BCrnberg
GF: F. Imend=C3=B6rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG N=C3=BCrnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--=20
Ming Lei