Skip to content

md/raid5: size stripe-cache and worker tuning from the hardware

Hiroshi Nishida linux-raid lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest6 patches 6 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/6md/raid5: size the worker group array by nr_node_idsdrivers/md/raid5.c+7 -11 reply
2/6md/raid5: size stripe-cache hash locks from the CPU countdrivers/md/raid5-cache.c drivers/md/raid5.c drivers/md/raid5.h+84 -331 reply
3/6md/raid5: scale the stripe_cache_size limit with system memorydrivers/md/raid5.c+36 -11 reply
4/6md/raid5: make the stripe batch size a module parameterdrivers/md/raid5.c drivers/md/raid5.h+23 -41 reply
5/6md/raid5: scale the default stripe cache size with system memorydrivers/md/raid5.c+41 -31 reply
6/6md/raid5: derive the default group_thread_cnt from the hardwaredrivers/md/raid5.c+42 -31 reply

Reviews trailers per patch

1 of 6 patches has a Reviewed-by or Acked-by.

PatchYu Kuai
1/6R-b
2/6
3/6
4/6
5/6
6/6

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 0/6] md/raid5: size stripe-cache and worker tuning from the hardware

md/raid5 has several tuning values that are compile-time constants chosen for the modest systems of years ago:

- NR_STRIPE_HASH_LOCKS (8) stripe-cache hash lock striping
- the 32768 stripe_cache_size ceiling

  - NR_STRIPES             (256)  initial stripe cache size
  - MAX_STRIPE_BATCH       (8)    stripes handled per device_lock
  - worker_cnt_per_group   (0)    raid5 worker threads (group_thread_cnt)

On a large-memory, many-core host backing a wide array these are all too small, and the only remedy today is to retune each one by hand after every assembly, or to recompile. On a small NAS a fixed larger value would just waste memory.

This series makes each value derive a default from the hardware -- system memory for the cache sizes, CPU count for the lock and worker counts -- while keeping every one overridable:

1/6 (prerequisite) size the worker_groups[] array by nr_node_ids 2/6 size the stripe-cache hash locks from the CPU count (8..32) 3/6 scale the stripe_cache_size limit with memory (never below 32768) 4/6 make the stripe batch size a module parameter
5/6 scale the default stripe cache size with memory (256..4096) 6/6 derive the default group_thread_cnt from the CPU count

Patch 1 is a prerequisite correctness fix that patch 6 depends on:
alloc_thread_groups() sizes the worker_groups[] array by num_possible_nodes() (a node count) but indexes it by cpu_to_node() (a node id), so a sparse NUMA node map can index out of bounds. That is only reachable once worker groups are enabled, which patch 6 does by default -- so the fix goes first. It stands alone (Fixes: 851c30c9badf) and can be taken independently.

Each tunable default only rises on hardware that can back it: the lock and worker counts (2, 6) scale with the CPU count, the cache sizes (3, 5) with RAM. A genuinely small system -- a few cores and a few GB -- therefore keeps today's values and today's memory footprint. Patches 2-6 each add a module parameter to override or pin the value (patch 6 also keeps the existing per-array group_thread_cnt sysfs attribute), so an administrator can force any of them, including back to the historical behaviour.

Patches 2-5 are sizing/capability changes with no throughput claim -- the intent is simply that the out-of-the-box configuration tracks the machine instead of a fixed constant. Patch 6 does move throughput: enabling md's worker groups by default measures 2.1-3.2x on a 16-disk NVMe array (a 32-vCPU / 16-core host, steady state), with no regression on small machines -- a 4-CPU box gets 2 workers and is never slower, a box with two or fewer CPUs gets 0 and is unchanged.

Tested on a KASAN + lockdep + DEBUG_LIST kernel (RAID5 on loop devices):
create, 180MB write + sha256 verify, disk fail + spare rebuild (data verified), scrub (mismatch_cnt=0), and bitmap add/remove -- which drives raid5_quiesce()'s lock_all_device_hash_locks_irq() path -- with the parameters both left at their hardware-derived defaults and pinned to explicit values, including nr_stripe_hash_locks=32 (the 33-lock quiesce path). No KASAN, lockdep, or list-debug reports.

Hiroshi Nishida (6):
md/raid5: size the worker group array by nr_node_ids md/raid5: size stripe-cache hash locks from the CPU count md/raid5: scale the stripe_cache_size limit with system memory md/raid5: make the stripe batch size a module parameter md/raid5: scale the default stripe cache size with system memory md/raid5: derive the default group_thread_cnt from the hardware

 drivers/md/raid5-cache.c |   2 +-
 drivers/md/raid5.c       | 233 +++++++++++++++++++++++++++++++++------
 drivers/md/raid5.h       |  43 ++++++--
 3 files changed, 233 insertions(+), 45 deletions(-)

base-commit: 55b77337bdd088c77461588e5ec094421b89911b --
2.43.0