Skip to contentSkip to conversation

test/btt-stress.sh: add stress test for BTT lane race

PATCH Applied Alison Schofield 2 participants 4 messages Apr 24 – Jun 30

Subsystems (2)
lore
Share & tools
lore

Alison Schofield

author maintainer
#0 lore raw

PATCH 1/1 test/btt-stress.sh: add stress test for BTT lane race

The btt-check unit test exposed data mismatches during BTT I/O in a CI environment, indicating a race in lane acquisition that can lead to silent data corruption. The failure was not reliably reproduced under typical test conditions.

Add a targeted stress test that repeatedly writes, reads, and verifies data on a BTT namespace while background readers contend for BTT lanes and CPU loops increase preemption pressure.

The test reproduces the race on an unfixed kernel and passes with the lane ownership fix applied.

Assisted-By Claude Sonnet 4.5
---
 test/btt-stress.sh | 111 +++++++++++++++++++++++++++++++++++++++++++++
 test/meson.build   |   2 +
 2 files changed, 113 insertions(+)
 create mode 100755 test/btt-stress.sh
test/btt-stress.sh new file +111 -0
new file mode 100755
index 000000000000..c1892f536d75
@@ -0,0 +1,111 @@
+#!/bin/bash -E
+# SPDX-License-Identifier: GPL-2.0
+
+dev=""
+mode=""
+size=""
+sector_size=""
+blockdev=""
+rc=77
+
+. $(dirname $0)/common
+
+trap 'err $LINENO' ERR
+
+check_min_kver "7.1" || do_skip "known BTT lane race before fix"
+
+# Stress BTT I/O under contention to exercise lane acquisition races.
+# Background readers contend for lanes while CPU loops increase
+# preemption pressure.
+
+create() {
+	json=$($NDCTL create-namespace -b "$NFIT_TEST_BUS0" -t pmem -m sector)
+	rc=2
+	eval "$(echo "$json" | json2var)"
+	[ -n "$dev" ] || err "$LINENO"
+	[ "$mode" = "sector" ] || err "$LINENO"
+	[ -n "$size" ] || err "$LINENO"
+	[ -n "$sector_size" ] || err "$LINENO"
+	[ -n "$blockdev" ] || err "$LINENO"
+	[ "$size" -gt 0 ] || err "$LINENO"
+}
+
+# Start background workers:
+#   - readers contend for lanes
+#   - CPU loops increase preemption
+start_bg_workers() {
+	local ncpus
+	ncpus=$(nproc)
+	local nworkers=$((ncpus / 2))
+
+	# Ensure at least one worker, cap to limit runtime noise
+	[ $nworkers -lt 1 ] && nworkers=1
+	[ $nworkers -gt 8 ] && nworkers=8
+
+	BG_PIDS=()
+	local i
+	for i in $(seq 1 $nworkers); do
+		# Reader: contends for lanes (use O_DIRECT to avoid page cache)
+		(while :; do
+			dd if=/dev/"$blockdev" of=/dev/null \
+				bs="$sector_size" count=256 \
+				iflag=direct >/dev/null 2>&1 || true
+		done) &
+		BG_PIDS+=($!)
+
+		# CPU hog: increase preemption
+		(while :; do :; done) &
+		BG_PIDS+=($!)
+	done
+	echo "started $nworkers readers + $nworkers CPU hogs"
+}
+
+stop_bg_workers() {
+	local pid
+	for pid in "${BG_PIDS[@]}"; do
+		kill "$pid" 2>/dev/null || true
+	done
+	wait "${BG_PIDS[@]}" 2>/dev/null || true
+	BG_PIDS=()
+}
+
+# Write, read, and verify data
+do_io_verify() {
+	dd if=/dev/urandom of=test-bin \
+		bs="$sector_size" count=$((size / sector_size)) >/dev/null 2>&1
+	dd if=test-bin of=/dev/"$blockdev" \
+		bs="$sector_size" count=$((size / sector_size)) >/dev/null 2>&1
+	dd if=/dev/"$blockdev" of=test-bin-read \
+		bs="$sector_size" count=$((size / sector_size)) >/dev/null 2>&1
+	diff test-bin test-bin-read
+	rm -f test-bin*
+}
+
+# Run verification under contention
+test_io_stress() {
+	local iterations=${1:-20}
+	echo "=== ${FUNCNAME[0]} ($iterations iterations) ==="
+
+	start_bg_workers
+	trap 'stop_bg_workers; err $LINENO' ERR
+
+	local i
+	for i in $(seq 1 "$iterations"); do
+		echo "--- iteration $i/$iterations ---"
+		do_io_verify
+	done
+
+	stop_bg_workers
+	trap 'err $LINENO' ERR
+}
+
+modprobe nfit_test
+rc=1
+reset && create
+
+# 30 iterations balances runtime and reproduction probability
+test_io_stress 30
+
+reset
+_cleanup
+exit 0
test/meson.build +2 -0
index e0e2193bfd51..ee6a18762a17 100644
@@ -150,6 +150,7 @@ sector_mode = find_program('sector-mode.sh')
 inject_error = find_program('inject-error.sh')
 btt_errors = find_program('btt-errors.sh')
 btt_pad_compat = find_program('btt-pad-compat.sh')
+btt_stress = find_program('btt-stress.sh')
 firmware_update = find_program('firmware-update.sh')
 rescan_partitions = find_program('rescan-partitions.sh')
 inject_smart = find_program('inject-smart.sh')
@@ -185,6 +186,7 @@ tests = [
   [ 'sector-mode.sh',         sector_mode,        'ndctl' ],
   [ 'inject-error.sh',        inject_error,	  'ndctl' ],
   [ 'btt-errors.sh',          btt_errors,	  'ndctl' ],
+  [ 'btt-stress.sh',          btt_stress,	  'ndctl' ],
   [ 'hugetlb',                hugetlb,		  'ndctl' ],
   [ 'btt-pad-compat.sh',      btt_pad_compat,	  'ndctl' ],
   [ 'ack-shutdown-count-set', ack_shutdown_count, 'ndctl' ],
Signature · 1 line
2.37.3
maintainer Reviewed-by on v1 1/1
#1 lore raw
Quoted 9 lines · Alison Schofield

The btt-check unit test exposed data mismatches during BTT I/O in a CI environment, indicating a race in lane acquisition that can lead to silent data corruption. The failure was not reliably reproduced under typical test conditions.

Add a targeted stress test that repeatedly writes, reads, and verifies data on a BTT namespace while background readers contend for BTT lanes and CPU loops increase preemption pressure.

Quoted 5 lines

The test reproduces the race on an unfixed kernel and passes with the lane ownership fix applied.

Assisted-By Claude Sonnet 4.5

Just one nit below, otherwise looks good.

Also, side note, looks like this ran for ~45 seconds on my laptop making it the second longest running test in ndctl - maybe that's fine. Also all the 'while :; do :;' processes pegged half my CPUs to 100% for the entire time, but maybe that's also fine :) - I see that's intentional of course.

Quoted 23 lines

---
 test/btt-stress.sh | 111 +++++++++++++++++++++++++++++++++++++++++++++ test/meson.build   |   2 +

 2 files changed, 113 insertions(+)

 create mode 100755 test/btt-stress.sh

test/btt-stress.sh new file +11 -0
new file mode 100755
index 000000000000..c1892f536d75
@@ -0,0 +1,111 @@
+#!/bin/bash -E
+# SPDX-License-Identifier: GPL-2.0
+
+dev=""
+mode=""
+size=""
+sector_size=""
+blockdev=""
+rc=77
+
+. $(dirname $0)/common

test/btt-stress.sh line 21

Quoted 10 lines
test/btt-stress.sh new file +10 -0
+
+trap 'err $LINENO' ERR
+
+check_min_kver "7.1" || do_skip "known BTT lane race before fix"
+
+# Stress BTT I/O under contention to exercise lane acquisition races.
+# Background readers contend for lanes while CPU loops increase
+# preemption pressure.
+
+create() {

Usually for other scripts in ndctl, the convention has been to be more C-like and put the opening { on the next line.

test/btt-stress.sh line 24

Quoted 3 lines
test/btt-stress.sh new file +3 -0
+	json=$($NDCTL create-namespace -b "$NFIT_TEST_BUS0" -t pmem -m sector)
+	rc=2
+	eval "$(echo "$json" | json2var)"

I was going to complain about the eval and the json2var 'sed' parsing to get stuff out of json, but I see this has been around since forever and used in a bunch of other places.

Ideally this should just be multiple jq -r <filter> invocations, to extract each variable - more wordy, but avoids the eval.. But this can be a future treewide cleanup.

Quoted 107 lines
test/btt-stress.sh new file +87 -0
+	[ -n "$dev" ] || err "$LINENO"
+	[ "$mode" = "sector" ] || err "$LINENO"
+	[ -n "$size" ] || err "$LINENO"
+	[ -n "$sector_size" ] || err "$LINENO"
+	[ -n "$blockdev" ] || err "$LINENO"
+	[ "$size" -gt 0 ] || err "$LINENO"
+}
+
+# Start background workers:
+#   - readers contend for lanes
+#   - CPU loops increase preemption
+start_bg_workers() {
+	local ncpus
+	ncpus=$(nproc)
+	local nworkers=$((ncpus / 2))
+
+	# Ensure at least one worker, cap to limit runtime noise
+	[ $nworkers -lt 1 ] && nworkers=1
+	[ $nworkers -gt 8 ] && nworkers=8
+
+	BG_PIDS=()
+	local i
+	for i in $(seq 1 $nworkers); do
+		# Reader: contends for lanes (use O_DIRECT to avoid page cache)
+		(while :; do
+			dd if=/dev/"$blockdev" of=/dev/null \
+				bs="$sector_size" count=256 \
+				iflag=direct >/dev/null 2>&1 || true
+		done) &
+		BG_PIDS+=($!)
+
+		# CPU hog: increase preemption
+		(while :; do :; done) &
+		BG_PIDS+=($!)
+	done
+	echo "started $nworkers readers + $nworkers CPU hogs"
+}
+
+stop_bg_workers() {
+	local pid
+	for pid in "${BG_PIDS[@]}"; do
+		kill "$pid" 2>/dev/null || true
+	done
+	wait "${BG_PIDS[@]}" 2>/dev/null || true
+	BG_PIDS=()
+}
+
+# Write, read, and verify data
+do_io_verify() {
+	dd if=/dev/urandom of=test-bin \
+		bs="$sector_size" count=$((size / sector_size)) >/dev/null 2>&1
+	dd if=test-bin of=/dev/"$blockdev" \
+		bs="$sector_size" count=$((size / sector_size)) >/dev/null 2>&1
+	dd if=/dev/"$blockdev" of=test-bin-read \
+		bs="$sector_size" count=$((size / sector_size)) >/dev/null 2>&1
+	diff test-bin test-bin-read
+	rm -f test-bin*
+}
+
+# Run verification under contention
+test_io_stress() {
+	local iterations=${1:-20}
+	echo "=== ${FUNCNAME[0]} ($iterations iterations) ==="
+
+	start_bg_workers
+	trap 'stop_bg_workers; err $LINENO' ERR
+
+	local i
+	for i in $(seq 1 "$iterations"); do
+		echo "--- iteration $i/$iterations ---"
+		do_io_verify
+	done
+
+	stop_bg_workers
+	trap 'err $LINENO' ERR
+}
+
+modprobe nfit_test
+rc=1
+reset && create
+
+# 30 iterations balances runtime and reproduction probability
+test_io_stress 30
+
+reset
+_cleanup
+exit 0
test/meson.build +2 -0
index e0e2193bfd51..ee6a18762a17 100644
@@ -150,6 +150,7 @@ sector_mode = find_program('sector-mode.sh')
  inject_error = find_program('inject-error.sh')
  btt_errors = find_program('btt-errors.sh')
  btt_pad_compat = find_program('btt-pad-compat.sh')
+btt_stress = find_program('btt-stress.sh')
  firmware_update = find_program('firmware-update.sh')
  rescan_partitions = find_program('rescan-partitions.sh')
  inject_smart = find_program('inject-smart.sh')
@@ -185,6 +186,7 @@ tests = [
    [ 'sector-mode.sh',         sector_mode,        'ndctl' ],
    [ 'inject-error.sh',        inject_error,	  'ndctl' ],
    [ 'btt-errors.sh',          btt_errors,	  'ndctl' ],
+  [ 'btt-stress.sh',          btt_stress,	  'ndctl' ],
    [ 'hugetlb',                hugetlb,		  'ndctl' ],
    [ 'btt-pad-compat.sh',      btt_pad_compat,	  'ndctl' ],
    [ 'ack-shutdown-count-set', ack_shutdown_count, 'ndctl' ],
author maintainer
#2 lore raw
Quoted 18 lines · Vishal Verma
Quoted 9 lines · Alison Schofield

The btt-check unit test exposed data mismatches during BTT I/O in a CI environment, indicating a race in lane acquisition that can lead to silent data corruption. The failure was not reliably reproduced under typical test conditions.

Add a targeted stress test that repeatedly writes, reads, and verifies data on a BTT namespace while background readers contend for BTT lanes and CPU loops increase preemption pressure.

Quoted 5 lines

The test reproduces the race on an unfixed kernel and passes with the lane ownership fix applied.

Assisted-By Claude Sonnet 4.5

Just one nit below, otherwise looks good.

Quoted 7 lines

Also, side note, looks like this ran for ~45 seconds on my laptop making it the second longest running test in ndctl - maybe that's fine. Also all the 'while :; do :;' processes pegged half my CPUs to 100% for the entire time, but maybe that's also fine :) - I see that's intentional of course.

Thanks for the review Vishal!

ICYMI I take unit test time hogs very seriously, so aside from the obvious finding here: you deserve a laptop upgrade, let's look at it:

Alone:
1/1 ndctl:ndctl / btt-stress.sh OK 10.71s

En Suite:
1/24 ndctl:ndctl / libndctl OK 1.76s

 2/24 ndctl:ndctl / dsm-fail                       OK               0.30s
 3/24 ndctl:ndctl / create.sh                      OK               0.41s
 4/24 ndctl:ndctl / clear.sh                       OK              14.70s
 5/24 ndctl:ndctl / pmem-errors.sh                 OK              13.61s
 6/24 ndctl:ndctl / btt-check.sh                   OK               2.71s
 7/24 ndctl:ndctl / label-compat.sh                OK               0.58s
 8/24 ndctl:ndctl / sector-mode.sh                 OK               0.67s
 9/24 ndctl:ndctl / inject-error.sh                OK              13.99s
10/24 ndctl:ndctl / btt-errors.sh                  OK              24.14s
11/24 ndctl:ndctl / btt-stress.sh                  OK              11.15s
12/24 ndctl:ndctl / hugetlb                        OK               0.32s
13/24 ndctl:ndctl / btt-pad-compat.sh              OK               0.95s
14/24 ndctl:ndctl / ack-shutdown-count-set         OK               0.21s
15/24 ndctl:ndctl / rescan-partitions.sh           OK               6.76s
16/24 ndctl:ndctl / inject-smart.sh                OK               1.41s
17/24 ndctl:ndctl / monitor.sh                     OK              49.71s
18/24 ndctl:ndctl / max_extent_ns                  OK               0.79s
19/24 ndctl:ndctl / pfn-meta-errors.sh             OK              14.79s
20/24 ndctl:ndctl / track-uuid.sh                  OK               0.55s
21/24 ndctl:ndctl / firmware-update.sh             OK              10.32s
22/24 ndctl:ndctl / pmem-ns                        OK               1.04s
23/24 ndctl:ndctl / align.sh                       OK               1.13s
24/24 ndctl:ndctl / nfit-security.sh               OK               1.27s

To be complete - that btt race condition took a lot for me to reproduce. I was stripping out all things that slowed the kernel until I got it to hit, and then came up with that level of stress in the unit test.

I do wonder if you would even hit the problem with the monster config you likely are using based on your runtimes. Did you test this with and without the fix applied?

Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+create() {

Usually for other scripts in ndctl, the convention has been to be more C-like and put the opening { on the next line.

Yeah, this is a thing. I've started using shfmt and that brace placement for function bodies is not a style choice. And when I think about enforcing it across the unit test scripts I do see the C like convention you note.

Let me take a look at how the kernel scripts in tools/testing deal with formatting. I expect it is some flavor of checkpatch.

I'm after uniformity and no exceptions! Same thing for shellcheck.

Quoted 11 lines

test/btt-stress.sh line 24

Quoted 3 lines
test/btt-stress.sh new file +3 -0
+	json=$($NDCTL create-namespace -b "$NFIT_TEST_BUS0" -t pmem -m sector)
+	rc=2
+	eval "$(echo "$json" | json2var)"

I was going to complain about the eval and the json2var 'sed' parsing to get stuff out of json, but I see this has been around since forever and used in a bunch of other places.

Ideally this should just be multiple jq -r <filter> invocations, to extract each variable - more wordy, but avoids the eval.. But this can be a future treewide cleanup.

Agreed the eval/json2var pattern isn't ideal, but since it's the convention across the whole test/ suite, I'd rather not have this one test diverge. Happy to do a treewide jq conversion as a separate series.

snip

author maintainer Applied
#3 lore raw
In reply to Alison Schofield
Quoted 11 lines · Alison Schofield

The btt-check unit test exposed data mismatches during BTT I/O in a CI environment, indicating a race in lane acquisition that can lead to silent data corruption. The failure was not reliably reproduced under typical test conditions.

Add a targeted stress test that repeatedly writes, reads, and verifies data on a BTT namespace while background readers contend for BTT lanes and CPU loops increase preemption pressure.

The test reproduces the race on an unfixed kernel and passes with the lane ownership fix applied.

Applied to pending for NDCTL v86
https://github.com/pmem/ndctl/commits/pending/

Quoted 7 lines
Assisted-By Claude Sonnet 4.5
---
 test/btt-stress.sh | 111 +++++++++++++++++++++++++++++++++++++++++++++
 test/meson.build   |   2 +
 2 files changed, 113 insertions(+)

snip