block: avoid hctx spinlock for plug with multiple queues
Status from this message: Applied, thanks!
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v2 Latest3 patches 3 repliesApplied
- v13 patches 4 repliesSuperseded
Changes in v2 from the cover letter
- Leave unmatched requests in plug list instead of building a new list
- Add Reviewed-by tags
Patches in v2 Applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/3 | block: take rq_list instead of plug in dispatch functionsblock/ block/ | +13 | 0 replies | |
| 2/3 | block: factor out blk_mq_dispatch_queue_requests() helperblock/ | +22 | 0 replies | |
| 3/3 | block: avoid hctx spinlock for plug with multiple queuesblock/ | +48 | 1 reply |
Reviews trailers per patch
All 3 patches have a Reviewed-by or Acked-by.
| Patch | Christoph Hellwig |
|---|---|
| 1/3 | R-b |
| 2/3 | R-b |
| 3/3 | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH v2 0/3] block: avoid hctx spinlock for plug with multiple queues
blk_mq_flush_plug_list() has a fast path if all requests in the plug are destined for the same request_queue. It calls ->queue_rqs() with the whole batch of requests, falling back on ->queue_rq() for any requests not handled by ->queue_rqs(). However, if the requests are destined for multiple queues, blk_mq_flush_plug_list() has a slow path that calls blk_mq_dispatch_list() repeatedly to filter the requests by ctx/hctx. Each queue's requests are inserted into the hctx's dispatch list under a spinlock, then __blk_mq_sched_dispatch_requests() takes them out of the dispatch list (taking the spinlock again), and finally blk_mq_dispatch_rq_list() calls ->queue_rq() on each request.
Acquiring the hctx spinlock twice and calling ->queue_rq() instead of ->queue_rqs() makes the slow path significantly more expensive. Thus, batching more requests into a single plug (e.g. io_uring_enter syscall) can counterintuitively hurt performance by causing the plug to span multiple queues. We have observed 2-3% of CPU time spent acquiring the hctx spinlock alone on workloads issuing requests to multiple NVMe devices in the same io_uring SQE batches.
Add a medium path in blk_mq_flush_plug_list() for plugs that don't have elevators or come from a schedule, but do span multiple queues. Filter the requests by queue and call ->queue_rqs()/->queue_rq() on the list of requests destined to each request_queue.
With this change, we no longer see any CPU time spent in _raw_spin_lock from blk_mq_flush_plug_list and throughput increases accordingly.
Caleb Sander Mateos (3):
block: take rq_list instead of plug in dispatch functions
block: factor out blk_mq_dispatch_queue_requests() helper
block: avoid hctx spinlock for plug with multiple queues
block/blk-mq.c | 110 +++++++++++++++++++++++++++++++------------- block/mq-deadline.c | 2 +- 2 files changed, 79 insertions(+), 33 deletions(-)
v2:
- Leave unmatched requests in plug list instead of building a new list
- Add Reviewed-by tags
Signature · 1 line
2.45.2