test/btt-stress.sh: add stress test for BTT lane race
Alison Schofield
maintainerPATCH 1/1 test/btt-stress.sh: add stress test for BTT lane race
The btt-check unit test exposed data mismatches during BTT I/O in a CI environment, indicating a race in lane acquisition that can lead to silent data corruption. The failure was not reliably reproduced under typical test conditions.
Add a targeted stress test that repeatedly writes, reads, and verifies data on a BTT namespace while background readers contend for BTT lanes and CPU loops increase preemption pressure.
The test reproduces the race on an unfixed kernel and passes with the lane ownership fix applied.
--- test/btt-stress.sh | 111 +++++++++++++++++++++++++++++++++++++++++++++ test/meson.build | 2 + 2 files changed, 113 insertions(+) create mode 100755 test/btt-stress.sh
test/btt-stress.sh new file +111 -0
@@ -0,0 +1,111 @@ +#!/bin/bash -E +# SPDX-License-Identifier: GPL-2.0 + +dev="" +mode="" +size="" +sector_size="" +blockdev="" +rc=77 + +. $(dirname $0)/common + +trap 'err $LINENO' ERR + +check_min_kver "7.1" || do_skip "known BTT lane race before fix" + +# Stress BTT I/O under contention to exercise lane acquisition races. +# Background readers contend for lanes while CPU loops increase +# preemption pressure. + +create() { + json=$($NDCTL create-namespace -b "$NFIT_TEST_BUS0" -t pmem -m sector) + rc=2 + eval "$(echo "$json" | json2var)" + [ -n "$dev" ] || err "$LINENO" + [ "$mode" = "sector" ] || err "$LINENO" + [ -n "$size" ] || err "$LINENO" + [ -n "$sector_size" ] || err "$LINENO" + [ -n "$blockdev" ] || err "$LINENO" + [ "$size" -gt 0 ] || err "$LINENO" +} + +# Start background workers: +# - readers contend for lanes +# - CPU loops increase preemption +start_bg_workers() { + local ncpus + ncpus=$(nproc) + local nworkers=$((ncpus / 2)) + + # Ensure at least one worker, cap to limit runtime noise + [ $nworkers -lt 1 ] && nworkers=1 + [ $nworkers -gt 8 ] && nworkers=8 + + BG_PIDS=() + local i + for i in $(seq 1 $nworkers); do + # Reader: contends for lanes (use O_DIRECT to avoid page cache) + (while :; do + dd if=/dev/"$blockdev" of=/dev/null \ + bs="$sector_size" count=256 \ + iflag=direct >/dev/null 2>&1 || true + done) & + BG_PIDS+=($!) + + # CPU hog: increase preemption + (while :; do :; done) & + BG_PIDS+=($!) + done + echo "started $nworkers readers + $nworkers CPU hogs" +} + +stop_bg_workers() { + local pid + for pid in "${BG_PIDS[@]}"; do + kill "$pid" 2>/dev/null || true + done + wait "${BG_PIDS[@]}" 2>/dev/null || true + BG_PIDS=() +} + +# Write, read, and verify data +do_io_verify() { + dd if=/dev/urandom of=test-bin \ + bs="$sector_size" count=$((size / sector_size)) >/dev/null 2>&1 + dd if=test-bin of=/dev/"$blockdev" \ + bs="$sector_size" count=$((size / sector_size)) >/dev/null 2>&1 + dd if=/dev/"$blockdev" of=test-bin-read \ + bs="$sector_size" count=$((size / sector_size)) >/dev/null 2>&1 + diff test-bin test-bin-read + rm -f test-bin* +} + +# Run verification under contention +test_io_stress() { + local iterations=${1:-20} + echo "=== ${FUNCNAME[0]} ($iterations iterations) ===" + + start_bg_workers + trap 'stop_bg_workers; err $LINENO' ERR + + local i + for i in $(seq 1 "$iterations"); do + echo "--- iteration $i/$iterations ---" + do_io_verify + done + + stop_bg_workers + trap 'err $LINENO' ERR +} + +modprobe nfit_test +rc=1 +reset && create + +# 30 iterations balances runtime and reproduction probability +test_io_stress 30 + +reset +_cleanup +exit 0
test/meson.build +2 -0
@@ -150,6 +150,7 @@ sector_mode = find_program('sector-mode.sh') inject_error = find_program('inject-error.sh') btt_errors = find_program('btt-errors.sh') btt_pad_compat = find_program('btt-pad-compat.sh') +btt_stress = find_program('btt-stress.sh') firmware_update = find_program('firmware-update.sh') rescan_partitions = find_program('rescan-partitions.sh') inject_smart = find_program('inject-smart.sh')
@@ -185,6 +186,7 @@ tests = [ [ 'sector-mode.sh', sector_mode, 'ndctl' ], [ 'inject-error.sh', inject_error, 'ndctl' ], [ 'btt-errors.sh', btt_errors, 'ndctl' ], + [ 'btt-stress.sh', btt_stress, 'ndctl' ], [ 'hugetlb', hugetlb, 'ndctl' ], [ 'btt-pad-compat.sh', btt_pad_compat, 'ndctl' ], [ 'ack-shutdown-count-set', ack_shutdown_count, 'ndctl' ],
Signature · 1 line
2.37.3
Quoted 9 lines · Alison Schofield
The btt-check unit test exposed data mismatches during BTT I/O in a CI environment, indicating a race in lane acquisition that can lead to silent data corruption. The failure was not reliably reproduced under typical test conditions.
Add a targeted stress test that repeatedly writes, reads, and verifies data on a BTT namespace while background readers contend for BTT lanes and CPU loops increase preemption pressure.
Quoted 5 lines
The test reproduces the race on an unfixed kernel and passes with the lane ownership fix applied.
Assisted-By Claude Sonnet 4.5Signed-off-by Alison Schofield <[email protected]>
Just one nit below, otherwise looks good.
Also, side note, looks like this ran for ~45 seconds on my laptop making it the second longest running test in ndctl - maybe that's fine. Also all the 'while :; do :;' processes pegged half my CPUs to 100% for the entire time, but maybe that's also fine :) - I see that's intentional of course.
Quoted 23 lines
---
test/btt-stress.sh | 111 +++++++++++++++++++++++++++++++++++++++++++++ test/meson.build | 2 +2 files changed, 113 insertions(+)create mode 100755
test/btt-stress.shtest/btt-stress.sh new file +11
-0@@ -0,0 +1,111 @@ +#!/bin/bash -E +# SPDX-License-Identifier: GPL-2.0 + +dev="" +mode="" +size="" +sector_size="" +blockdev="" +rc=77 + +. $(dirname $0)/common
test/btt-stress.sh line 21
Quoted 10 lines
test/btt-stress.sh new file +10
-0+ +trap 'err $LINENO' ERR + +check_min_kver "7.1" || do_skip "known BTT lane race before fix" + +# Stress BTT I/O under contention to exercise lane acquisition races. +# Background readers contend for lanes while CPU loops increase +# preemption pressure. + +create() {
Usually for other scripts in ndctl, the convention has been to be more C-like and put the opening { on the next line.
test/btt-stress.sh line 24
Quoted 3 lines
test/btt-stress.sh new file +3
-0+ json=$($NDCTL create-namespace -b "$NFIT_TEST_BUS0" -t pmem -m sector) + rc=2 + eval "$(echo "$json" | json2var)"
I was going to complain about the eval and the json2var 'sed' parsing to get stuff out of json, but I see this has been around since forever and used in a bunch of other places.
Ideally this should just be multiple jq -r <filter> invocations, to extract each variable - more wordy, but avoids the eval.. But this can be a future treewide cleanup.
Quoted 107 lines
test/btt-stress.sh new file +87
-0+ [ -n "$dev" ] || err "$LINENO" + [ "$mode" = "sector" ] || err "$LINENO" + [ -n "$size" ] || err "$LINENO" + [ -n "$sector_size" ] || err "$LINENO" + [ -n "$blockdev" ] || err "$LINENO" + [ "$size" -gt 0 ] || err "$LINENO" +} + +# Start background workers: +# - readers contend for lanes +# - CPU loops increase preemption +start_bg_workers() { + local ncpus + ncpus=$(nproc) + local nworkers=$((ncpus / 2)) + + # Ensure at least one worker, cap to limit runtime noise + [ $nworkers -lt 1 ] && nworkers=1 + [ $nworkers -gt 8 ] && nworkers=8 + + BG_PIDS=() + local i + for i in $(seq 1 $nworkers); do + # Reader: contends for lanes (use O_DIRECT to avoid page cache) + (while :; do + dd if=/dev/"$blockdev" of=/dev/null \ + bs="$sector_size" count=256 \ + iflag=direct >/dev/null 2>&1 || true + done) & + BG_PIDS+=($!) + + # CPU hog: increase preemption + (while :; do :; done) & + BG_PIDS+=($!) + done + echo "started $nworkers readers + $nworkers CPU hogs" +} + +stop_bg_workers() { + local pid + for pid in "${BG_PIDS[@]}"; do + kill "$pid" 2>/dev/null || true + done + wait "${BG_PIDS[@]}" 2>/dev/null || true + BG_PIDS=() +} + +# Write, read, and verify data +do_io_verify() { + dd if=/dev/urandom of=test-bin \ + bs="$sector_size" count=$((size / sector_size)) >/dev/null 2>&1 + dd if=test-bin of=/dev/"$blockdev" \ + bs="$sector_size" count=$((size / sector_size)) >/dev/null 2>&1 + dd if=/dev/"$blockdev" of=test-bin-read \ + bs="$sector_size" count=$((size / sector_size)) >/dev/null 2>&1 + diff test-bin test-bin-read + rm -f test-bin* +} + +# Run verification under contention +test_io_stress() { + local iterations=${1:-20} + echo "=== ${FUNCNAME[0]} ($iterations iterations) ===" + + start_bg_workers + trap 'stop_bg_workers; err $LINENO' ERR + + local i + for i in $(seq 1 "$iterations"); do + echo "--- iteration $i/$iterations ---" + do_io_verify + done + + stop_bg_workers + trap 'err $LINENO' ERR +} + +modprobe nfit_test +rc=1 +reset && create + +# 30 iterations balances runtime and reproduction probability +test_io_stress 30 + +reset +_cleanup +exit 0test/meson.build +2
-0@@ -150,6 +150,7 @@ sector_mode = find_program('sector-mode.sh') inject_error = find_program('inject-error.sh') btt_errors = find_program('btt-errors.sh') btt_pad_compat = find_program('btt-pad-compat.sh') +btt_stress = find_program('btt-stress.sh') firmware_update = find_program('firmware-update.sh') rescan_partitions = find_program('rescan-partitions.sh') inject_smart = find_program('inject-smart.sh')@@ -185,6 +186,7 @@ tests = [ [ 'sector-mode.sh', sector_mode, 'ndctl' ], [ 'inject-error.sh', inject_error, 'ndctl' ], [ 'btt-errors.sh', btt_errors, 'ndctl' ], + [ 'btt-stress.sh', btt_stress, 'ndctl' ], [ 'hugetlb', hugetlb, 'ndctl' ], [ 'btt-pad-compat.sh', btt_pad_compat, 'ndctl' ], [ 'ack-shutdown-count-set', ack_shutdown_count, 'ndctl' ],
Alison Schofield
maintainerQuoted 18 lines · Vishal Verma
Quoted 9 lines · Alison Schofield
The btt-check unit test exposed data mismatches during BTT I/O in a CI environment, indicating a race in lane acquisition that can lead to silent data corruption. The failure was not reliably reproduced under typical test conditions.
Add a targeted stress test that repeatedly writes, reads, and verifies data on a BTT namespace while background readers contend for BTT lanes and CPU loops increase preemption pressure.
Quoted 5 lines
The test reproduces the race on an unfixed kernel and passes with the lane ownership fix applied.
Assisted-By Claude Sonnet 4.5Signed-off-by Alison Schofield <[email protected]>Just one nit below, otherwise looks good.
Quoted 7 lines
Reviewed-by Vishal Verma <[email protected]>Also, side note, looks like this ran for ~45 seconds on my laptop making it the second longest running test in ndctl - maybe that's fine. Also all the 'while :; do :;' processes pegged half my CPUs to 100% for the entire time, but maybe that's also fine :) - I see that's intentional of course.
Thanks for the review Vishal!
ICYMI I take unit test time hogs very seriously, so aside from the obvious finding here: you deserve a laptop upgrade, let's look at it:
Alone:
1/1 ndctl:ndctl / btt-stress.sh OK 10.71s
En Suite:
1/24 ndctl:ndctl / libndctl OK 1.76s
2/24 ndctl:ndctl / dsm-fail OK 0.30s 3/24 ndctl:ndctl / create.sh OK 0.41s 4/24 ndctl:ndctl / clear.sh OK 14.70s 5/24 ndctl:ndctl / pmem-errors.sh OK 13.61s 6/24 ndctl:ndctl / btt-check.sh OK 2.71s 7/24 ndctl:ndctl / label-compat.sh OK 0.58s 8/24 ndctl:ndctl / sector-mode.sh OK 0.67s 9/24 ndctl:ndctl / inject-error.sh OK 13.99s 10/24 ndctl:ndctl / btt-errors.sh OK 24.14s 11/24 ndctl:ndctl / btt-stress.sh OK 11.15s 12/24 ndctl:ndctl / hugetlb OK 0.32s 13/24 ndctl:ndctl / btt-pad-compat.sh OK 0.95s 14/24 ndctl:ndctl / ack-shutdown-count-set OK 0.21s 15/24 ndctl:ndctl / rescan-partitions.sh OK 6.76s 16/24 ndctl:ndctl / inject-smart.sh OK 1.41s 17/24 ndctl:ndctl / monitor.sh OK 49.71s 18/24 ndctl:ndctl / max_extent_ns OK 0.79s 19/24 ndctl:ndctl / pfn-meta-errors.sh OK 14.79s 20/24 ndctl:ndctl / track-uuid.sh OK 0.55s 21/24 ndctl:ndctl / firmware-update.sh OK 10.32s 22/24 ndctl:ndctl / pmem-ns OK 1.04s 23/24 ndctl:ndctl / align.sh OK 1.13s 24/24 ndctl:ndctl / nfit-security.sh OK 1.27s
To be complete - that btt race condition took a lot for me to reproduce. I was stripping out all things that slowed the kernel until I got it to hit, and then came up with that level of stress in the unit test.
I do wonder if you would even hit the problem with the monster config you likely are using based on your runtimes. Did you test this with and without the fix applied?
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ +create() {Usually for other scripts in ndctl, the convention has been to be more C-like and put the opening { on the next line.
Yeah, this is a thing. I've started using shfmt and that brace placement for function bodies is not a style choice. And when I think about enforcing it across the unit test scripts I do see the C like convention you note.
Let me take a look at how the kernel scripts in tools/testing deal with formatting. I expect it is some flavor of checkpatch.
I'm after uniformity and no exceptions! Same thing for shellcheck.
Quoted 11 lines
test/btt-stress.sh line 24
Quoted 3 lines
test/btt-stress.sh new file +3
-0+ json=$($NDCTL create-namespace -b "$NFIT_TEST_BUS0" -t pmem -m sector) + rc=2 + eval "$(echo "$json" | json2var)"I was going to complain about the eval and the json2var 'sed' parsing to get stuff out of json, but I see this has been around since forever and used in a bunch of other places.
Ideally this should just be multiple jq -r <filter> invocations, to extract each variable - more wordy, but avoids the eval.. But this can be a future treewide cleanup.
Agreed the eval/json2var pattern isn't ideal, but since it's the convention across the whole test/ suite, I'd rather not have this one test diverge. Happy to do a treewide jq conversion as a separate series.
snip
Quoted 11 lines · Alison Schofield
The btt-check unit test exposed data mismatches during BTT I/O in a CI environment, indicating a race in lane acquisition that can lead to silent data corruption. The failure was not reliably reproduced under typical test conditions.
Add a targeted stress test that repeatedly writes, reads, and verifies data on a BTT namespace while background readers contend for BTT lanes and CPU loops increase preemption pressure.
The test reproduces the race on an unfixed kernel and passes with the lane ownership fix applied.
Applied to pending for NDCTL v86
https://github.com/pmem/ndctl/commits/pending/
Quoted 7 lines
Assisted-By Claude Sonnet 4.5Signed-off-by Alison Schofield <[email protected]>--- test/btt-stress.sh | 111 +++++++++++++++++++++++++++++++++++++++++++++ test/meson.build | 2 + 2 files changed, 113 insertions(+)
snip