nvme: NUMA locality for fabrics
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v3 Latest posting 4/40 of 3 patches 0 repliesNew
- v3 posting 3/40 of 3 patches 0 repliesSuperseded
- v3 posting 2/40 of 3 patches 0 repliesSuperseded
Earlier versions · 2
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v3 posting 1/4 | 3 patches | 0 replies | Superseded | |
| v1 | 2 patches | 0 replies | Superseded |
Patches in v3 posting 4/4 No applies recorded yet
None of the 3 patches the cover letter announces are in this archive.
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.
Cover letter [PATCHv3 0/3] nvme: NUMA locality for fabrics
Quoted 3 lines · James Smart
- What are the latencies that are meaningful ?? is it solely single digit us ? 10-30us ?? 1ms ?? 50ms ?? 100ms ???? How do things change if the latency change ?
Well, we did spend a lot of effort on making sure sub-10us latencies work. For real life setups with multiple cpus in use the 100us order of magniture is probably more interesting.
Quoted 8 lines
- At what point does it become more important to get commands to the subsystem (via a different queue or queues on different controllers) so they can be being worked on in parallel vs the latency of a single io ??? How is this need communicated to the nvme layer so we can make the right choices ??? Queue counts, queue size, and MAXCMD limits (thanks Keith), may cause throttles that increase this need.?? To that end - what are your expectations for queue size or MAXCMD limits vs the latencies vs load from a single cpu?? or a set of cpus ?
If you fully load the queue you are of course going to see massive additional latency. That's why you'll see typical NVMe PCIe device (or RDMA setups) massively overprovisioned in number of queues and/or queue depth.
Quoted 8 lines
- Must every cpu get a queue ??? what if the controller won't support a queue per cpu ?? how do things change if only 1 in 16 or less of the cpu's get queues ?? Today, if less than cpu count queues aren't supported - aren't queues for the different controllers likely mapping to the same cpus ? I don't think there's awareness of what queues on other controllers are present so that there could be redundant paths mapped to cpus that aren't bound already. And if such logic were added, how does that affect the multipathing choices ?
Less than cpu count queues are perfectly supported, we'll just start sharing queues. In general as long as you share queues between cores on the same socket things still work reasonably fine. Once you start sharing a queue between sockets you are going to be in a world of pain.
Quoted 5 lines
- What if application load is driven only by specific cpu's - if a "cpu was given" to the specific task (constrained app, VMs, or containers. how would we know that?) how does that map if multiple cpus are sharing a queue ? will multipathing and load choices be made system-wide, specific to a cpu set, or a single cpu ?
Right now we do multipathing devisions per node (aka per socket for todays typical systems).
Quoted 4 lines
- What if a cpu is finally out of cycles due to load, how can we find out if the io must be limited to a cpu or whether affinity can be subverted so that use of other idle cpus can share in the processing load for the queue(s) for that cpu ?
For the traditional interrupt driven model we really need to process the interrupts on the submitting cpu to throttle. For an interesting model to move the I/O completion load to another thread and thus potential cpu look at the aio poll patches Jens just posted.
Quoted 6 lines
If we're completely focused on cpu affinity with a queue, what happens when not all queues are equal. There is lots of talk of specific queues providing specific functionality that other queues wouldn't support. What if queues 1-M do writes with dedup, queues N-P do writes with compression, and Q-Z do accelerated writes to a local cache.? How do you see the current linux implementation migrating to something like that ?
Very unlikely. We have support for a few queu types now in the for-4.21 block tree (default, reads, polling), but the above just sounds way too magic.