Skip to contentSkip to conversation

selftests/bpf: Add stress test for rqspinlock in NMI

PATCH Superseded Kumar Kartikeya Dwivedi 2 participants 4 messages Sep 26, 2025 – Sep 27, 2025

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Kumar Kartikeya Dwivedi

author maintainer
#0 lore raw

PATCH 1/1 selftests/bpf: Add stress test for rqspinlock in NMI

Introduce a kernel module that will exercise lock acquisition in the NMI path, and bias toward creating contention such that NMI waiters end up being non-head waiters. Prior to the rqspinlock fix made in the commit 0d80e7f951be ("rqspinlock: Choose trylock fallback for NMI waiters"), it was possible for the queueing path of non-head waiters to get stuck in NMI, which this stress test reproduces fairly easily with just 3 CPUs.

Both AA and ABBA flavors are supported, and it will serve as a test case for future fixes that address this corner case. More information about the problem in question is available in the commit cited above. When the fix is reverted, this stress test will lock up the system.

To enable this test automatically through the test_progs infrastructure, add a load_module_params API to exercise both AA and ABBA cases when running the test.

Note that the test runs for at most 5 seconds, and becomes a noop after that, in order to allow the system to make forward progress. In addition, CPU 0 is always kept untouched by the created threads and NMIs. The test will automatically scale to the number of available online CPUs.

Note that at least 3 CPUs are necessary to run this test, hence skip the selftest in case the environment has less than 3 CPUs available.

---
 tools/testing/selftests/bpf/Makefile          |   2 +-
 .../selftests/bpf/prog_tests/res_spin_lock.c  |  16 ++
 .../testing/selftests/bpf/test_kmods/Makefile |   2 +-
 .../bpf/test_kmods/bpf_test_rqspinlock.c      | 208 ++++++++++++++++++
 tools/testing/selftests/bpf/testing_helpers.c |  14 +-
 tools/testing/selftests/bpf/testing_helpers.h |   1 +
 6 files changed, 239 insertions(+), 4 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/test_kmods/bpf_test_rqspinlock.c
tools/testing/selftests/bpf/Makefile +1 -1
index 0b6ee902bce5..f00587d4ede6 100644
@@ -120,7 +120,7 @@ TEST_PROGS_EXTENDED := \
 	test_bpftool.py
 
 TEST_KMODS := bpf_testmod.ko bpf_test_no_cfi.ko bpf_test_modorder_x.ko \
-	bpf_test_modorder_y.ko
+	bpf_test_modorder_y.ko bpf_test_rqspinlock.ko
 TEST_KMOD_TARGETS = $(addprefix $(OUTPUT)/,$(TEST_KMODS))
 
 # Compile but not part of 'make run_tests'
tools/testing/selftests/bpf/prog_tests/res_spin_lock.c +16 -0
index 0703e987df89..8c6c2043a432 100644
@@ -99,3 +99,19 @@ void test_res_spin_lock_success(void)
 	res_spin_lock__destroy(skel);
 	return;
 }
+
+void serial_test_res_spin_lock_stress(void)
+{
+	if (libbpf_num_possible_cpus() < 3) {
+		test__skip();
+		return;
+	}
+
+	ASSERT_OK(load_module("bpf_test_rqspinlock.ko", false), "load module AA");
+	sleep(5);
+	unload_module("bpf_test_rqspinlock", false);
+
+	ASSERT_OK(load_module_params("bpf_test_rqspinlock.ko", "test_ab=1", false), "load module ABBA");
+	sleep(5);
+	unload_module("bpf_test_rqspinlock", false);
+}
tools/testing/selftests/bpf/test_kmods/Makefile +1 -1
index d4e50c4509c9..63c4d3f6a12f 100644
@@ -8,7 +8,7 @@ Q = @
 endif
 
 MODULES = bpf_testmod.ko bpf_test_no_cfi.ko bpf_test_modorder_x.ko \
-	bpf_test_modorder_y.ko
+	bpf_test_modorder_y.ko bpf_test_rqspinlock.ko
 
 $(foreach m,$(MODULES),$(eval obj-m += $(m:.ko=.o)))
 
tools/testing/selftests/bpf/test_kmods/bpf_test_rqspinlock.c new file +208 -0

208 lines not on this page · raw message

tools/testing/selftests/bpf/testing_helpers.c +12 -2
index 5e9f16683be5..16eb37e5bad6 100644
@@ -399,7 +399,7 @@ int unload_module(const char *name, bool verbose)
 	return 0;
 }
 
-int load_module(const char *path, bool verbose)
+static int __load_module(const char *path, const char *param_values, bool verbose)
 {
 	int fd;
 
@@ -411,7 +411,7 @@ int load_module(const char *path, bool verbose)
 		fprintf(stdout, "Can't find %s kernel module: %d\n", path, -errno);
 		return -ENOENT;
 	}
-	if (finit_module(fd, "", 0)) {
+	if (finit_module(fd, param_values, 0)) {
 		fprintf(stdout, "Failed to load %s into the kernel: %d\n", path, -errno);
 		close(fd);
 		return -EINVAL;
@@ -423,6 +423,16 @@ int load_module(const char *path, bool verbose)
 	return 0;
 }
 
+int load_module_params(const char *path, const char *param_values, bool verbose)
+{
+	return __load_module(path, param_values, verbose);
+}
+
+int load_module(const char *path, bool verbose)
+{
+	return __load_module(path, "", verbose);
+}
+
 int unload_bpf_testmod(bool verbose)
 {
 	return unload_module("bpf_testmod", verbose);
tools/testing/selftests/bpf/testing_helpers.h +1 -0
index 46d7f7089f63..eb20d3772218 100644
@@ -39,6 +39,7 @@ int kern_sync_rcu(void);
 int finit_module(int fd, const char *param_values, int flags);
 int delete_module(const char *name, int flags);
 int load_module(const char *path, bool verbose);
+int load_module_params(const char *path, const char *param_values, bool verbose);
 int unload_module(const char *name, bool verbose);
 
 static inline __u64 get_time_ns(void)

--
2.51.0

author maintainer
#1 lore raw
Quoted 23 lines · Kumar Kartikeya Dwivedi

Introduce a kernel module that will exercise lock acquisition in the NMI path, and bias toward creating contention such that NMI waiters end up being non-head waiters. Prior to the rqspinlock fix made in the commit 0d80e7f951be ("rqspinlock: Choose trylock fallback for NMI waiters"), it was possible for the queueing path of non-head waiters to get stuck in NMI, which this stress test reproduces fairly easily with just 3 CPUs.

Both AA and ABBA flavors are supported, and it will serve as a test case for future fixes that address this corner case. More information about the problem in question is available in the commit cited above. When the fix is reverted, this stress test will lock up the system.

To enable this test automatically through the test_progs infrastructure, add a load_module_params API to exercise both AA and ABBA cases when running the test.

Note that the test runs for at most 5 seconds, and becomes a noop after that, in order to allow the system to make forward progress. In addition, CPU 0 is always kept untouched by the created threads and NMIs. The test will automatically scale to the number of available online CPUs.

Quoted 5 lines

Note that at least 3 CPUs are necessary to run this test, hence skip the selftest in case the environment has less than 3 CPUs available.

---

This currently won't trigger in BPF CI, as the VMs have two CPUs allocated to them.
I was wondering if this is a simple change, otherwise I can rework the module to work with less than 3 CPUs.

#2 lore raw
Quoted 28 lines · Kumar Kartikeya Dwivedi
Quoted 26 lines · Kumar Kartikeya Dwivedi

Introduce a kernel module that will exercise lock acquisition in the NMI path, and bias toward creating contention such that NMI waiters end up being non-head waiters. Prior to the rqspinlock fix made in the commit 0d80e7f951be ("rqspinlock: Choose trylock fallback for NMI waiters"), it was possible for the queueing path of non-head waiters to get stuck in NMI, which this stress test reproduces fairly easily with just 3 CPUs.

Both AA and ABBA flavors are supported, and it will serve as a test case for future fixes that address this corner case. More information about the problem in question is available in the commit cited above. When the fix is reverted, this stress test will lock up the system.

To enable this test automatically through the test_progs infrastructure, add a load_module_params API to exercise both AA and ABBA cases when running the test.

Note that the test runs for at most 5 seconds, and becomes a noop after that, in order to allow the system to make forward progress. In addition, CPU 0 is always kept untouched by the created threads and NMIs. The test will automatically scale to the number of available online CPUs.

Note that at least 3 CPUs are necessary to run this test, hence skip the selftest in case the environment has less than 3 CPUs available.

Quoted 7 lines
Quoted 2 lines

---

This currently won't trigger in BPF CI, as the VMs have two CPUs allocated to them.
I was wondering if this is a simple change, otherwise I can rework the module to work with less than 3 CPUs.

It's fine. We can bump CI's cpu count.

But please see the bug spotted by our brand new AI review. Despite false positives it's amazing what it catches.

Don't click on "Logs for ai-review" pw entry.
It's hard to navigate from there.
Click on "PR summary" and scroll down.

author maintainer
#3 lore raw
Quoted 39 lines · Alexei Starovoitov
Quoted 28 lines · Kumar Kartikeya Dwivedi
Quoted 26 lines · Kumar Kartikeya Dwivedi

Introduce a kernel module that will exercise lock acquisition in the NMI path, and bias toward creating contention such that NMI waiters end up being non-head waiters. Prior to the rqspinlock fix made in the commit 0d80e7f951be ("rqspinlock: Choose trylock fallback for NMI waiters"), it was possible for the queueing path of non-head waiters to get stuck in NMI, which this stress test reproduces fairly easily with just 3 CPUs.

Both AA and ABBA flavors are supported, and it will serve as a test case for future fixes that address this corner case. More information about the problem in question is available in the commit cited above. When the fix is reverted, this stress test will lock up the system.

To enable this test automatically through the test_progs infrastructure, add a load_module_params API to exercise both AA and ABBA cases when running the test.

Note that the test runs for at most 5 seconds, and becomes a noop after that, in order to allow the system to make forward progress. In addition, CPU 0 is always kept untouched by the created threads and NMIs. The test will automatically scale to the number of available online CPUs.

Note that at least 3 CPUs are necessary to run this test, hence skip the selftest in case the environment has less than 3 CPUs available.

Quoted 7 lines
Quoted 2 lines

---

This currently won't trigger in BPF CI, as the VMs have two CPUs allocated to them.
I was wondering if this is a simple change, otherwise I can rework the module to work with less than 3 CPUs.

Quoted 8 lines

It's fine. We can bump CI's cpu count.

But please see the bug spotted by our brand new AI review. Despite false positives it's amazing what it catches.

Don't click on "Logs for ai-review" pw entry.
It's hard to navigate from there.
Click on "PR summary" and scroll down.

Honored to be the first recipient (thanks for working on this Ihor and Chris). Should we use a Reported-by: to credit such bugs?
I can also respin as is for now and we can decide later, but we probably should keep track just for statistics.