Skip to contentSkip to conversation

Fixes for test-ww_mutex stress test

RFC PATCH In review John Stultz 3 participants 8 messages Aug 8, 2023 Locking primitives

lore
Share & tools
lore
#0 lore raw

cover Fixes for test-ww_mutex stress test

As part of recent stabilizing of the proxy-execution series, I've seen a number of issues from the test-ww_mutex module.

This is great for shaking out problems in the patches, but in some cases it seems the patch series has made it easier to uncover problems in the test-ww_mutex stress tests.

So this patch series provides a few fixes that I've come up with in testing with qemu using large cpu counts.

Feedback would be greatly appreciated!

thanks
-john

Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Will Deacon <[email protected]>
Cc Waiman Long <[email protected]>
Cc Boqun Feng <[email protected]>
Cc "Paul E . McKenney" <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Li Zhijian <[email protected]>
Cc Dietmar Eggemann <[email protected]>

John Stultz (3):
test-ww_mutex: Use prng instead of rng to avoid hangs at bootup
test-ww_mutex: Fix potential workqueue corruption
test-ww_mutex: Make sure we bail out instead of livelock

 kernel/locking/test-ww_mutex.c | 48 ++++++++++++++++++++++++----------
 1 file changed, 34 insertions(+), 14 deletions(-)
Signature · 1 line
2.41.0.640.ga95def55d0-goog
#1 lore raw

RFC PATCH 1/3 test-ww_mutex: Use prng instead of rng to avoid hangs at bootup

Booting w/ qemu without kvm, I noticed we'd sometimes seem to get stuck in get_random_u32_below(). This seems potentially to be entropy exhaustion (with the test module linked statically, it runs pretty early in the bootup).

I'm not 100% sure on this, but this patch switches to use the prng instead since we don't need true randomness, just mixed up orders for testing ww_mutex lock acquisitions.

With this patch, I no longer see hangs in get_random_u32_below()

Feedback would be appreciated!

Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Will Deacon <[email protected]>
Cc Waiman Long <[email protected]>
Cc Boqun Feng <[email protected]>
Cc "Paul E . McKenney" <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Li Zhijian <[email protected]>
Cc Dietmar Eggemann <[email protected]>
---
 kernel/locking/test-ww_mutex.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
kernel/locking/test-ww_mutex.c +17 -2
index 93cca6e69860..9bceba65858a 100644
@@ -9,7 +9,7 @@
 #include <linux/delay.h>
 #include <linux/kthread.h>
 #include <linux/module.h>
-#include <linux/random.h>
+#include <linux/prandom.h>
 #include <linux/slab.h>
 #include <linux/ww_mutex.h>
 
@@ -386,6 +386,19 @@ struct stress {
 	int nlocks;
 };
 
+struct rnd_state rng;
+DEFINE_SPINLOCK(rng_lock);
+
+static inline u32 prandom_u32_below(u32 ceil)
+{
+	u32 ret;
+
+	spin_lock(&rng_lock);
+	ret = prandom_u32_state(&rng) % ceil;
+	spin_unlock(&rng_lock);
+	return ret;
+}
+
 static int *get_random_order(int count)
 {
 	int *order;
@@ -399,7 +412,7 @@ static int *get_random_order(int count)
 		order[n] = n;
 
 	for (n = count - 1; n > 1; n--) {
-		r = get_random_u32_below(n + 1);
+		r = prandom_u32_below(n + 1);
 		if (r != n) {
 			tmp = order[n];
 			order[n] = order[r];
@@ -625,6 +638,8 @@ static int __init test_ww_mutex_init(void)
 
 	printk(KERN_INFO "Beginning ww mutex selftests\n");
 
+	prandom_seed_state(&rng, get_random_u64());
+
 	wq = alloc_workqueue("test-ww_mutex", WQ_UNBOUND, 0);
 	if (!wq)
 		return -ENOMEM;
Signature · 1 line
2.41.0.640.ga95def55d0-goog
maintainer
#2 lore raw

Re: [RFC][PATCH 1/3] test-ww_mutex: Use prng instead of rng to avoid hangs at bootup

Quoted 5 lines · John Stultz

Booting w/ qemu without kvm, I noticed we'd sometimes seem to get stuck in get_random_u32_below(). This seems potentially to be entropy exhaustion (with the test module linked statically, it runs pretty early in the bootup).

Quoted 7 lines

I'm not 100% sure on this, but this patch switches to use the prng instead since we don't need true randomness, just mixed up orders for testing ww_mutex lock acquisitions.

With this patch, I no longer see hangs in get_random_u32_below()

Feedback would be appreciated!

Jason, I thought part of the 'recent' random rework was avoiding the exhaustion problem, could you please give an opinion on the below?

Quoted 59 lines
---
 kernel/locking/test-ww_mutex.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
kernel/locking/test-ww_mutex.c +17 -2
index 93cca6e69860..9bceba65858a 100644
@@ -9,7 +9,7 @@
 #include <linux/delay.h>
 #include <linux/kthread.h>
 #include <linux/module.h>
-#include <linux/random.h>
+#include <linux/prandom.h>
 #include <linux/slab.h>
 #include <linux/ww_mutex.h>
 
@@ -386,6 +386,19 @@ struct stress {
 	int nlocks;
 };
 
+struct rnd_state rng;
+DEFINE_SPINLOCK(rng_lock);
+
+static inline u32 prandom_u32_below(u32 ceil)
+{
+	u32 ret;
+
+	spin_lock(&rng_lock);
+	ret = prandom_u32_state(&rng) % ceil;
+	spin_unlock(&rng_lock);
+	return ret;
+}
+
 static int *get_random_order(int count)
 {
 	int *order;
@@ -399,7 +412,7 @@ static int *get_random_order(int count)
 		order[n] = n;
 
 	for (n = count - 1; n > 1; n--) {
-		r = get_random_u32_below(n + 1);
+		r = prandom_u32_below(n + 1);
 		if (r != n) {
 			tmp = order[n];
 			order[n] = order[r];
@@ -625,6 +638,8 @@ static int __init test_ww_mutex_init(void)
 
 	printk(KERN_INFO "Beginning ww mutex selftests\n");
 
+	prandom_seed_state(&rng, get_random_u64());
+
 	wq = alloc_workqueue("test-ww_mutex", WQ_UNBOUND, 0);
 	if (!wq)
 		return -ENOMEM;
Signature · 1 line
2.41.0.640.ga95def55d0-goog
#3 lore raw

Re: [RFC][PATCH 1/3] test-ww_mutex: Use prng instead of rng to avoid hangs at bootup

Hi Peter, John,

Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · John Stultz

Booting w/ qemu without kvm, I noticed we'd sometimes seem to get stuck in get_random_u32_below(). This seems potentially to be entropy exhaustion (with the test module linked statically, it runs pretty early in the bootup).

I'm not 100% sure on this, but this patch switches to use the prng instead since we don't need true randomness, just mixed up orders for testing ww_mutex lock acquisitions.

Quoted 6 lines
Quoted 3 lines

With this patch, I no longer see hangs in get_random_u32_below()

Feedback would be appreciated!

Jason, I thought part of the 'recent' random rework was avoiding the exhaustion problem, could you please give an opinion on the below?

Thanks for looping me in. I actually can't reproduce this. I'm using a minimal config and using QEMU without KVM. The RNG doesn't initialize until much later on in the boot process, expectedly, yet get_random_u32_below() does _not_ hang in my trials. And indeed it's designed to never hang, since that would create boot deadlocks. So I'm not sure why you're seeing a hang.

It is worth noting that in those early boot test-case scenarios, before the RNG initializes, get_random_u32_below() will be somewhat slower than it normally is, and also slower than prandom_u32_state(). (But only in this early boot scenario edge case; this isn't a general statement about speed.) It's possible that in your QEMU machine, things are slow enough that you're simply noticing the difference. On my system, however, I replaced get_random_u32_below() with static u32 x; return ++x % ceil; and I didn't see any difference running it under TCG -- it took about 7 seconds either way.

So, from my perspective, you shouldn't see any hang. That function never blocks. I'm happy to look more into what's happening on your end though. Maybe share your .config and qemu command line and I'll see if I can repro?

Jason

#4 lore raw

Re: [RFC][PATCH 1/3] test-ww_mutex: Use prng instead of rng to avoid hangs at bootup

Quoted 22 lines · Jason A. Donenfeld

Hi Peter, John,

Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · John Stultz

Booting w/ qemu without kvm, I noticed we'd sometimes seem to get stuck in get_random_u32_below(). This seems potentially to be entropy exhaustion (with the test module linked statically, it runs pretty early in the bootup).

I'm not 100% sure on this, but this patch switches to use the prng instead since we don't need true randomness, just mixed up orders for testing ww_mutex lock acquisitions.

Quoted 6 lines
Quoted 3 lines

With this patch, I no longer see hangs in get_random_u32_below()

Feedback would be appreciated!

Jason, I thought part of the 'recent' random rework was avoiding the exhaustion problem, could you please give an opinion on the below?

Quoted 6 lines

Thanks for looping me in. I actually can't reproduce this. I'm using a minimal config and using QEMU without KVM. The RNG doesn't initialize until much later on in the boot process, expectedly, yet get_random_u32_below() does _not_ hang in my trials. And indeed it's designed to never hang, since that would create boot deadlocks. So I'm not sure why you're seeing a hang.

Ok. My hesitancy here was in part due to my understanding the entropy exhaustion hangs weren't supposed to be an issue anymore (thanks, btw).

Quoted 10 lines

It is worth noting that in those early boot test-case scenarios, before the RNG initializes, get_random_u32_below() will be somewhat slower than it normally is, and also slower than prandom_u32_state(). (But only in this early boot scenario edge case; this isn't a general statement about speed.) It's possible that in your QEMU machine, things are slow enough that you're simply noticing the difference. On my system, however, I replaced get_random_u32_below() with static u32 x; return ++x % ceil; and I didn't see any difference running it under TCG -- it took about 7 seconds either way.

Quoted 4 lines

So, from my perspective, you shouldn't see any hang. That function never blocks. I'm happy to look more into what's happening on your end though. Maybe share your .config and qemu command line and I'll see if I can repro?

Yeah, it may just be that the real RNG is slow enough that I'm hitting the hung task watchdog?
(I'm running with 64 cpus, so the test is trying to use 128 threads all hitting get_random_u32_below over and over to create their own random order of 16 locks)

When the softlockup watchdog hits, the trace looks like:

[   64.268881]  _get_random_bytes+0x3b/0x140
[   64.268881]  ? __kmem_cache_alloc_node+0x17a/0x310
[   64.268881]  ? get_random_order+0x28/0xa0
[   64.268881]  get_random_u32+0x1c2/0x1e0
[   64.268881]  __get_random_u32_below+0xd/0x60
[   64.268881]  get_random_order+0x60/0xa0
[   64.268881]  stress_inorder_work+0x26/0x390
[   64.268881]  ? lock_acquire+0xd4/0x290
[   64.268881]  process_one_work+0x267/0x520
[   64.268881]  worker_thread+0x4a/0x390
[   64.268881]  ? __pfx_worker_thread+0x10/0x10
[   64.268881]  kthread+0xf5/0x130
[   64.268881]  ? __pfx_kthread+0x10/0x10
[   64.268881]  ret_from_fork+0x2b/0x40
[   64.268881]  ? __pfx_kthread+0x10/0x10
[   64.268881]  ret_from_fork_asm+0x1b/0x30

Attached is the defconfig I just used.

My script to run is below.

thanks
-john

#base boot args
kcmd="root=/dev/vda2 console=ttyS0 earlycon "
#debug helper
kcmd+="ftrace_dump_on_oops sysrq_always_enabled debug_boot_weak_hash " #locktorture bits
kcmd+="torture.random_shuffle=1 locktorture.writer_fifo=1 locktorture.torture_type=mutex_lock locktorture.nested_locks=8 locktorture.rt_boost=1 locktorture.rt_boost_factor=50 locktorture.stutter=0 "

cpu_count=64
mem_size="6G"
#use_kvm="-enable-kvm"

device_config="-smp ${cpu_count} -m ${mem_size} ${use_kvm}      \
                -drive if=virtio,format=qcow2,file=ubuntu.img   \
                -device virtio-scsi-pci,id=scsi0                \
                -object rng-random,filename=/dev/urandom,id=rng0\
                -device virtio-rng-pci,rng=rng0                 \
                -device virtio-net-pci,netdev=net0              \
                -netdev user,id=net0,hostfwd=tcp::8022-:22      \
                -chardev stdio,id=char0,mux=on,logfile=serial.log,signal=off \
                -serial chardev:char0 -mon chardev=char0        \
                -nographic                                      \
                "

kbin="bzImage"

qemu-system-x86_64                                      \
                -kernel "${kbin}" -initrd initrd.img -append "${kcmd}"  \
                ${device_config}
#5 lore raw

Re: [RFC][PATCH 1/3] test-ww_mutex: Use prng instead of rng to avoid hangs at bootup

Quoted 11 lines · John Stultz
Quoted 4 lines · Jason A. Donenfeld

So, from my perspective, you shouldn't see any hang. That function never blocks. I'm happy to look more into what's happening on your end though. Maybe share your .config and qemu command line and I'll see if I can repro?

Yeah, it may just be that the real RNG is slow enough that I'm hitting the hung task watchdog?
(I'm running with 64 cpus, so the test is trying to use 128 threads all hitting get_random_u32_below over and over to create their own random order of 16 locks)

Just following up on this point, I went through and disabled all the hung task and delay detection (and pushed the rcu stall boundary up to two minutes), and indeed the test did complete without actually hanging. However, the test took something like 90 seconds to finish using the get_random_u32_below() calls, whereas with this patch it finishes in ~18s.

So indeed it's not blocking, just not fast enough to avoid the hung task watchdogs in this admittedly contrived case (though one that has been helpful in uncovering issues with proposed scheduler changes) .

I'll try to rework the commit message so the above is clear and resubmit.

thanks
-john

#6 lore raw

RFC PATCH 2/3 test-ww_mutex: Fix potential workqueue corruption

In reply to John Stultz

In some cases running with the test-ww_mutex code, I was seeing odd behavior where sometimes it seemed flush_workqueue was returning before all the work threads were finished.

Often this would cause strange crashes as the mutexes would be freed while they were being used.

Looking at the code, there is a lifetime problem as the controlling thread that spawns the work allocates the "struct stress" structures that are passed to the workqueue threads. Then when the workqueue threads are finished, they free the stress struct that was passed to them.

Unfortunately the workqueue work_struct node is in the stress struct. Which means the work_struct is freed before the work thread returns and while flush_workqueue is waiting.

It seems like a better idea to have the controlling thread both allocate and free the stress structures, so that we can be sure we don't corrupt the workqueue by freeing the structure prematurely.

So this patch reworks the test to do so, and with this change I no longer see the early flush_workqueue returns.

Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Will Deacon <[email protected]>
Cc Waiman Long <[email protected]>
Cc Boqun Feng <[email protected]>
Cc "Paul E . McKenney" <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Li Zhijian <[email protected]>
Cc Dietmar Eggemann <[email protected]>
---
 kernel/locking/test-ww_mutex.c | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)
kernel/locking/test-ww_mutex.c +12 -8
index 9bceba65858a..358d66150426 100644
@@ -479,7 +479,6 @@ static void stress_inorder_work(struct work_struct *work)
 	} while (!time_after(jiffies, stress->timeout));
 
 	kfree(order);
-	kfree(stress);
 }
 
 struct reorder_lock {
@@ -544,7 +543,6 @@ static void stress_reorder_work(struct work_struct *work)
 	list_for_each_entry_safe(ll, ln, &locks, link)
 		kfree(ll);
 	kfree(order);
-	kfree(stress);
 }
 
 static void stress_one_work(struct work_struct *work)
@@ -565,8 +563,6 @@ static void stress_one_work(struct work_struct *work)
 			break;
 		}
 	} while (!time_after(jiffies, stress->timeout));
-
-	kfree(stress);
 }
 
 #define STRESS_INORDER BIT(0)
@@ -577,15 +573,24 @@ static void stress_one_work(struct work_struct *work)
 static int stress(int nlocks, int nthreads, unsigned int flags)
 {
 	struct ww_mutex *locks;
-	int n;
+	struct stress *stress_array;
+	int n, count;
 
 	locks = kmalloc_array(nlocks, sizeof(*locks), GFP_KERNEL);
 	if (!locks)
 		return -ENOMEM;
 
+	stress_array = kmalloc_array(nthreads, sizeof(*stress_array),
+				     GFP_KERNEL);
+	if (!stress_array) {
+		kfree(locks);
+		return -ENOMEM;
+	}
+
 	for (n = 0; n < nlocks; n++)
 		ww_mutex_init(&locks[n], &ww_class);
 
+	count = 0;
 	for (n = 0; nthreads; n++) {
 		struct stress *stress;
 		void (*fn)(struct work_struct *work);
@@ -609,9 +614,7 @@ static int stress(int nlocks, int nthreads, unsigned int flags)
 		if (!fn)
 			continue;
 
-		stress = kmalloc(sizeof(*stress), GFP_KERNEL);
-		if (!stress)
-			break;
+		stress = &stress_array[count++];
 
 		INIT_WORK(&stress->work, fn);
 		stress->locks = locks;
@@ -626,6 +629,7 @@ static int stress(int nlocks, int nthreads, unsigned int flags)
 
 	for (n = 0; n < nlocks; n++)
 		ww_mutex_destroy(&locks[n]);
+	kfree(stress_array);
 	kfree(locks);
 
 	return 0;
Signature · 1 line
2.41.0.640.ga95def55d0-goog
#7 lore raw

RFC PATCH 3/3 test-ww_mutex: Make sure we bail out instead of livelock

In reply to John Stultz

I've seen what appears to be livelocks in the stress_inorder_work() function, and looking at the code it is clear we can have a case where we continually retry acquiring the locks and never check to see if we have passed the specified timeout.

This patch reworks that function so we always check the timeout before iterating through the loop again.

I believe others may have hit this previously here:
https://lore.kernel.org/lkml/[email protected]/

Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Will Deacon <[email protected]>
Cc Waiman Long <[email protected]>
Cc Boqun Feng <[email protected]>
Cc "Paul E . McKenney" <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Li Zhijian <[email protected]>
Cc Dietmar Eggemann <[email protected]>
---
 kernel/locking/test-ww_mutex.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
kernel/locking/test-ww_mutex.c +5 -4
index 358d66150426..78719e1ef1b1 100644
@@ -465,17 +465,18 @@ static void stress_inorder_work(struct work_struct *work)
 			ww_mutex_unlock(&locks[order[n]]);
 
 		if (err == -EDEADLK) {
-			ww_mutex_lock_slow(&locks[order[contended]], &ctx);
-			goto retry;
+			if (!time_after(jiffies, stress->timeout)) {
+				ww_mutex_lock_slow(&locks[order[contended]], &ctx);
+				goto retry;
+			}
 		}
 
+		ww_acquire_fini(&ctx);
 		if (err) {
 			pr_err_once("stress (%s) failed with %d\n",
 				    __func__, err);
 			break;
 		}
-
-		ww_acquire_fini(&ctx);
 	} while (!time_after(jiffies, stress->timeout));
 
 	kfree(order);
Signature · 1 line
2.41.0.640.ga95def55d0-goog