Skip to content

scsi/block: NUMA-local allocations and false-sharing fixes

Sumit Saxena linux-block linux-scsi 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest3 patches 4 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/3scsi: use NUMA-local allocation for sdev and stargetdrivers/scsi/scsi_scan.c+6 -30 replies
2/3block: align nr_active_requests_shared_tags to avoid cache line contentioninclude/linux/blkdev.h+3 -12 replies
3/3scsi: align scsi_device iodone_cnt to avoid cache line contentioninclude/scsi/scsi_device.h+3 -12 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/3] scsi/block: NUMA-local allocations and false-sharing fixes

This series contains three performance improvements targeting the SCSI and block layers on multi-socket NUMA systems.

On multi-socket NUMA systems we observed extreme I/O throughput variance of 50-60% between runs. This series identifies and fixes two root causes:
cross-node memory accesses due to NUMA-unaware allocations in the scan path, and false sharing between hot atomic counters in
struct request_queue and struct scsi_device.

The first patch makes the SCSI scan path allocate scsi_device and scsi_target on the NUMA node of the host adapter.

The second patch addresses false sharing in struct request_queue. This patch touches include/linux/blkdev.h, so needs review from linux-block, an Acked-by from the block maintainer is requested before merging via the SCSI tree.

The third patch addresses a false-sharing problem in struct scsi_device.

Performance notes:

Tested on a dual-socket NUMA system with an mpi3mr HBA, running fio (random read, 4K, QD 64, 16 jobs, 60s, direct I/O).
IOPS figures are in KIOPS (thousands of IOPS):

  Configuration                    Avg KIOPS   Range (KIOPS)   Spread
  Baseline                         6,255       4,200 - 6,700   ~37%
  Baseline + patches 2-3 (align)   6,653       6,000 - 7,000   ~15%
  Baseline + all patches (1-3)     6,649       6,400 - 7,000    ~9%

Key findings:
- Cacheline alignment patches (2-3) raise average IOPS by ~6% and cut throughput spread from ~37% to ~15%.
- Adding the NUMA allocation patch (1) further tightens the spread to ~9% with negligible impact on average throughput.
- The combined effect reduces the observed 50-60% run-to-run variance to under 10%, significantly improving workload predictability.

No functional regressions observed.

This patch series is based on Martin's for-next tree.

James Rizzo (3):
scsi: use NUMA-local allocation for sdev and starget
block: align nr_active_requests_shared_tags to avoid cache line
contention
scsi: align scsi_device iodone_cnt to avoid cache line contention

 drivers/scsi/scsi_scan.c   | 9 ++++++---
 include/linux/blkdev.h     | 4 +++-
 include/scsi/scsi_device.h | 4 +++-
 3 files changed, 12 insertions(+), 5 deletions(-)
Signature · 1 line
2.43.7