md/raid5: size stripe-cache and worker tuning from the hardware
Get series with b4
b4 am [email protected]Versions newest first
- v1 Latest6 patches 6 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/6 | md/raid5: size the worker group array by nr_node_idsdrivers/ | +7 | 1 reply | |
| 2/6 | md/raid5: size stripe-cache hash locks from the CPU countdrivers/ drivers/ drivers/ | +84 | 1 reply | |
| 3/6 | md/raid5: scale the stripe_cache_size limit with system memorydrivers/ | +36 | 1 reply | |
| 4/6 | md/raid5: make the stripe batch size a module parameterdrivers/ drivers/ | +23 | 1 reply | |
| 5/6 | md/raid5: scale the default stripe cache size with system memorydrivers/ | +41 | 1 reply | |
| 6/6 | md/raid5: derive the default group_thread_cnt from the hardwaredrivers/ | +42 | 1 reply |
Reviews trailers per patch
1 of 6 patches has a Reviewed-by or Acked-by.
| Patch | Yu Kuai |
|---|---|
| 1/6 | R-b |
| 2/6 | |
| 3/6 | |
| 4/6 | |
| 5/6 | |
| 6/6 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH 0/6] md/raid5: size stripe-cache and worker tuning from the hardware
md/raid5 has several tuning values that are compile-time constants chosen for the modest systems of years ago:
- NR_STRIPE_HASH_LOCKS (8) stripe-cache hash lock striping
- the 32768 stripe_cache_size ceiling
- NR_STRIPES (256) initial stripe cache size - MAX_STRIPE_BATCH (8) stripes handled per device_lock - worker_cnt_per_group (0) raid5 worker threads (group_thread_cnt)
On a large-memory, many-core host backing a wide array these are all too small, and the only remedy today is to retune each one by hand after every assembly, or to recompile. On a small NAS a fixed larger value would just waste memory.
This series makes each value derive a default from the hardware -- system memory for the cache sizes, CPU count for the lock and worker counts -- while keeping every one overridable:
1/6 (prerequisite) size the worker_groups[] array by nr_node_ids 2/6 size the stripe-cache hash locks from the CPU count (8..32) 3/6 scale the stripe_cache_size limit with memory (never below 32768) 4/6 make the stripe batch size a module parameter
5/6 scale the default stripe cache size with memory (256..4096) 6/6 derive the default group_thread_cnt from the CPU count
Patch 1 is a prerequisite correctness fix that patch 6 depends on:alloc_thread_groups() sizes the worker_groups[] array by num_possible_nodes() (a node count) but indexes it by cpu_to_node() (a node id), so a sparse NUMA node map can index out of bounds. That is only reachable once worker groups are enabled, which patch 6 does by default -- so the fix goes first. It stands alone (Fixes: 851c30c9badf) and can be taken independently.
Each tunable default only rises on hardware that can back it: the lock and worker counts (2, 6) scale with the CPU count, the cache sizes (3, 5) with RAM. A genuinely small system -- a few cores and a few GB -- therefore keeps today's values and today's memory footprint. Patches 2-6 each add a module parameter to override or pin the value (patch 6 also keeps the existing per-array group_thread_cnt sysfs attribute), so an administrator can force any of them, including back to the historical behaviour.
Patches 2-5 are sizing/capability changes with no throughput claim -- the intent is simply that the out-of-the-box configuration tracks the machine instead of a fixed constant. Patch 6 does move throughput: enabling md's worker groups by default measures 2.1-3.2x on a 16-disk NVMe array (a 32-vCPU / 16-core host, steady state), with no regression on small machines -- a 4-CPU box gets 2 workers and is never slower, a box with two or fewer CPUs gets 0 and is unchanged.
Tested on a KASAN + lockdep + DEBUG_LIST kernel (RAID5 on loop devices):
create, 180MB write + sha256 verify, disk fail + spare rebuild (data verified), scrub (mismatch_cnt=0), and bitmap add/remove -- which drives raid5_quiesce()'s lock_all_device_hash_locks_irq() path -- with the parameters both left at their hardware-derived defaults and pinned to explicit values, including nr_stripe_hash_locks=32 (the 33-lock quiesce path). No KASAN, lockdep, or list-debug reports.
Hiroshi Nishida (6):
md/raid5: size the worker group array by nr_node_ids md/raid5: size stripe-cache hash locks from the CPU count md/raid5: scale the stripe_cache_size limit with system memory md/raid5: make the stripe batch size a module parameter md/raid5: scale the default stripe cache size with system memory md/raid5: derive the default group_thread_cnt from the hardware
drivers/md/raid5-cache.c | 2 +- drivers/md/raid5.c | 233 +++++++++++++++++++++++++++++++++------ drivers/md/raid5.h | 43 ++++++-- 3 files changed, 233 insertions(+), 45 deletions(-)
base-commit: 55b77337bdd088c77461588e5ec094421b89911b --
2.43.0