Fixes for test-ww_mutex stress test
cover Fixes for test-ww_mutex stress test
As part of recent stabilizing of the proxy-execution series, I've seen a number of issues from the test-ww_mutex module.
This is great for shaking out problems in the patches, but in some cases it seems the patch series has made it easier to uncover problems in the test-ww_mutex stress tests.
So this patch series provides a few fixes that I've come up with in testing with qemu using large cpu counts.
Feedback would be greatly appreciated!
thanks
-john
John Stultz (3):
test-ww_mutex: Use prng instead of rng to avoid hangs at bootup
test-ww_mutex: Fix potential workqueue corruption
test-ww_mutex: Make sure we bail out instead of livelock
kernel/locking/test-ww_mutex.c | 48 ++++++++++++++++++++++++---------- 1 file changed, 34 insertions(+), 14 deletions(-)
Signature · 1 line
2.41.0.640.ga95def55d0-goog
RFC PATCH 1/3 test-ww_mutex: Use prng instead of rng to avoid hangs at bootup
Booting w/ qemu without kvm, I noticed we'd sometimes seem to get stuck in get_random_u32_below(). This seems potentially to be entropy exhaustion (with the test module linked statically, it runs pretty early in the bootup).
I'm not 100% sure on this, but this patch switches to use the prng instead since we don't need true randomness, just mixed up orders for testing ww_mutex lock acquisitions.
With this patch, I no longer see hangs in get_random_u32_below()
Feedback would be appreciated!
--- kernel/locking/test-ww_mutex.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
kernel/locking/test-ww_mutex.c +17 -2
@@ -9,7 +9,7 @@#include <linux/delay.h> #include <linux/kthread.h> #include <linux/module.h> -#include <linux/random.h> +#include <linux/prandom.h> #include <linux/slab.h> #include <linux/ww_mutex.h>
@@ -386,6 +386,19 @@ struct stress {int nlocks; }; +struct rnd_state rng; +DEFINE_SPINLOCK (rng_lock); + +static inline u32 prandom_u32_below (u32 ceil) +{ +u32 ret; + +spin_lock (&rng_lock); + ret =prandom_u32_state (&rng) % ceil; +spin_unlock (&rng_lock); +return ret; +} +static int *get_random_order (int count) {int *order;
@@ -399,7 +412,7 @@ static int *get_random_order(int count) order[n] = n;for (n = count -1 ; n >1 ; n--) { - r =get_random_u32_below (n +1 ); + r =prandom_u32_below (n +1 );if (r != n) { tmp = order[n]; order[n] = order[r];
@@ -625,6 +638,8 @@ static int __init test_ww_mutex_init(void)printk (KERN_INFO "Beginning ww mutex selftests\n" ); +prandom_seed_state (&rng,get_random_u64 ()); + wq =alloc_workqueue ("test-ww_mutex" ,WQ_UNBOUND ,0 );if (!wq)return -ENOMEM ;
Signature · 1 line
2.41.0.640.ga95def55d0-goog
Peter Zijlstra
maintainerRe: [RFC][PATCH 1/3] test-ww_mutex: Use prng instead of rng to avoid hangs at bootup
Quoted 5 lines · John Stultz
Booting w/ qemu without kvm, I noticed we'd sometimes seem to get stuck in
get_random_u32_below(). This seems potentially to be entropy exhaustion (with the test module linked statically, it runs pretty early in the bootup).
Quoted 7 lines
I'm not 100% sure on this, but this patch switches to use the prng instead since we don't need true randomness, just mixed up orders for testing
ww_mutexlock acquisitions.With this patch, I no longer see hangs in
get_random_u32_below()Feedback would be appreciated!
Jason, I thought part of the 'recent' random rework was avoiding the exhaustion problem, could you please give an opinion on the below?
Quoted 59 lines
Signed-off-by John Stultz <[email protected]>--- kernel/locking/test-ww_mutex.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)kernel/locking/test-ww_mutex.c +17
-2@@ -9,7 +9,7 @@#include <linux/delay.h> #include <linux/kthread.h> #include <linux/module.h> -#include <linux/random.h> +#include <linux/prandom.h> #include <linux/slab.h> #include <linux/ww_mutex.h> @@ -386,6 +386,19 @@ struct stress {int nlocks; }; +struct rnd_state rng; +DEFINE_SPINLOCK (rng_lock); + +static inline u32 prandom_u32_below (u32 ceil) +{ +u32 ret; + +spin_lock (&rng_lock); + ret =prandom_u32_state (&rng) % ceil; +spin_unlock (&rng_lock); +return ret; +} +static int *get_random_order (int count) {int *order;@@ -399,7 +412,7 @@ static int *get_random_order(int count) order[n] = n;for (n = count -1 ; n >1 ; n--) { - r =get_random_u32_below (n +1 ); + r =prandom_u32_below (n +1 );if (r != n) { tmp = order[n]; order[n] = order[r];@@ -625,6 +638,8 @@ static int __init test_ww_mutex_init(void)printk (KERN_INFO "Beginning ww mutex selftests\n" ); +prandom_seed_state (&rng,get_random_u64 ()); + wq =alloc_workqueue ("test-ww_mutex" ,WQ_UNBOUND ,0 );if (!wq)return -ENOMEM ;Signature · 1 line
2.41.0.640.ga95def55d0-goog
Re: [RFC][PATCH 1/3] test-ww_mutex: Use prng instead of rng to avoid hangs at bootup
Hi Peter, John,
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · John Stultz
Booting w/ qemu without kvm, I noticed we'd sometimes seem to get stuck in
get_random_u32_below(). This seems potentially to be entropy exhaustion (with the test module linked statically, it runs pretty early in the bootup).I'm not 100% sure on this, but this patch switches to use the prng instead since we don't need true randomness, just mixed up orders for testing
ww_mutexlock acquisitions.
Quoted 6 lines
Quoted 3 lines
With this patch, I no longer see hangs in
get_random_u32_below()Feedback would be appreciated!
Jason, I thought part of the 'recent' random rework was avoiding the exhaustion problem, could you please give an opinion on the below?
Thanks for looping me in. I actually can't reproduce this. I'm using a minimal config and using QEMU without KVM. The RNG doesn't initialize until much later on in the boot process, expectedly, yet get_random_u32_below() does _not_ hang in my trials. And indeed it's designed to never hang, since that would create boot deadlocks. So I'm not sure why you're seeing a hang.
It is worth noting that in those early boot test-case scenarios, before the RNG initializes, get_random_u32_below() will be somewhat slower than it normally is, and also slower than prandom_u32_state(). (But only in this early boot scenario edge case; this isn't a general statement about speed.) It's possible that in your QEMU machine, things are slow enough that you're simply noticing the difference. On my system, however, I replaced get_random_u32_below() with static u32 x; return ++x % ceil; and I didn't see any difference running it under TCG -- it took about 7 seconds either way.
So, from my perspective, you shouldn't see any hang. That function never blocks. I'm happy to look more into what's happening on your end though. Maybe share your .config and qemu command line and I'll see if I can repro?
Jason
Re: [RFC][PATCH 1/3] test-ww_mutex: Use prng instead of rng to avoid hangs at bootup
Quoted 22 lines · Jason A. Donenfeld
Hi Peter, John,
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · John Stultz
Booting w/ qemu without kvm, I noticed we'd sometimes seem to get stuck in
get_random_u32_below(). This seems potentially to be entropy exhaustion (with the test module linked statically, it runs pretty early in the bootup).I'm not 100% sure on this, but this patch switches to use the prng instead since we don't need true randomness, just mixed up orders for testing
ww_mutexlock acquisitions.Quoted 6 lines
Quoted 3 lines
With this patch, I no longer see hangs in
get_random_u32_below()Feedback would be appreciated!
Jason, I thought part of the 'recent' random rework was avoiding the exhaustion problem, could you please give an opinion on the below?
Quoted 6 lines
Thanks for looping me in. I actually can't reproduce this. I'm using a minimal config and using QEMU without KVM. The RNG doesn't initialize until much later on in the boot process, expectedly, yet
get_random_u32_below()does _not_ hang in my trials. And indeed it's designed to never hang, since that would create boot deadlocks. So I'm not sure why you're seeing a hang.
Ok. My hesitancy here was in part due to my understanding the entropy exhaustion hangs weren't supposed to be an issue anymore (thanks, btw).
Quoted 10 lines
It is worth noting that in those early boot test-case scenarios, before the RNG initializes,
get_random_u32_below()will be somewhat slower than it normally is, and also slower thanprandom_u32_state(). (But only in this early boot scenario edge case; this isn't a general statement about speed.) It's possible that in your QEMU machine, things are slow enough that you're simply noticing the difference. On my system, however, I replacedget_random_u32_below()withstatic u32 x; return ++x % ceil;and I didn't see any difference running it under TCG -- it took about 7 seconds either way.
Quoted 4 lines
So, from my perspective, you shouldn't see any hang. That function never blocks. I'm happy to look more into what's happening on your end though. Maybe share your .config and qemu command line and I'll see if I can repro?
Yeah, it may just be that the real RNG is slow enough that I'm hitting the hung task watchdog?
(I'm running with 64 cpus, so the test is trying to use 128 threads all hitting get_random_u32_below over and over to create their own random order of 16 locks)
When the softlockup watchdog hits, the trace looks like:
[ 64.268881] _get_random_bytes+0x3b/0x140 [ 64.268881] ? __kmem_cache_alloc_node+0x17a/0x310 [ 64.268881] ? get_random_order+0x28/0xa0 [ 64.268881] get_random_u32+0x1c2/0x1e0 [ 64.268881] __get_random_u32_below+0xd/0x60 [ 64.268881] get_random_order+0x60/0xa0 [ 64.268881] stress_inorder_work+0x26/0x390 [ 64.268881] ? lock_acquire+0xd4/0x290 [ 64.268881] process_one_work+0x267/0x520 [ 64.268881] worker_thread+0x4a/0x390 [ 64.268881] ? __pfx_worker_thread+0x10/0x10 [ 64.268881] kthread+0xf5/0x130 [ 64.268881] ? __pfx_kthread+0x10/0x10 [ 64.268881] ret_from_fork+0x2b/0x40 [ 64.268881] ? __pfx_kthread+0x10/0x10 [ 64.268881] ret_from_fork_asm+0x1b/0x30
Attached is the defconfig I just used.
My script to run is below.
thanks
-john
#base boot args
kcmd="root=/dev/vda2 console=ttyS0 earlycon "
#debug helper
kcmd+="ftrace_dump_on_oops sysrq_always_enabled debug_boot_weak_hash " #locktorture bits
kcmd+="torture.random_shuffle=1 locktorture.writer_fifo=1 locktorture.torture_type=mutex_lock locktorture.nested_locks=8 locktorture.rt_boost=1 locktorture.rt_boost_factor=50 locktorture.stutter=0 "
cpu_count=64
mem_size="6G"
#use_kvm="-enable-kvm"
device_config="-smp ${cpu_count} -m ${mem_size} ${use_kvm} \
-drive if=virtio,format=qcow2,file=ubuntu.img \
-device virtio-scsi-pci,id=scsi0 \
-object rng-random,filename=/dev/urandom,id=rng0\
-device virtio-rng-pci,rng=rng0 \
-device virtio-net-pci,netdev=net0 \
-netdev user,id=net0,hostfwd=tcp::8022-:22 \
-chardev stdio,id=char0,mux=on,logfile=serial.log,signal=off \
-serial chardev:char0 -mon chardev=char0 \
-nographic \
"kbin="bzImage"
qemu-system-x86_64 \
-kernel "${kbin}" -initrd initrd.img -append "${kcmd}" \
${device_config}Re: [RFC][PATCH 1/3] test-ww_mutex: Use prng instead of rng to avoid hangs at bootup
Quoted 11 lines · John Stultz
Quoted 4 lines · Jason A. Donenfeld
So, from my perspective, you shouldn't see any hang. That function never blocks. I'm happy to look more into what's happening on your end though. Maybe share your .config and qemu command line and I'll see if I can repro?
Yeah, it may just be that the real RNG is slow enough that I'm hitting the hung task watchdog?
(I'm running with 64 cpus, so the test is trying to use 128 threads all hittingget_random_u32_belowover and over to create their own random order of 16 locks)
Just following up on this point, I went through and disabled all the hung task and delay detection (and pushed the rcu stall boundary up to two minutes), and indeed the test did complete without actually hanging. However, the test took something like 90 seconds to finish using the get_random_u32_below() calls, whereas with this patch it finishes in ~18s.
So indeed it's not blocking, just not fast enough to avoid the hung task watchdogs in this admittedly contrived case (though one that has been helpful in uncovering issues with proposed scheduler changes) .
I'll try to rework the commit message so the above is clear and resubmit.
thanks
-john
RFC PATCH 2/3 test-ww_mutex: Fix potential workqueue corruption
In reply to John StultzIn some cases running with the test-ww_mutex code, I was seeing odd behavior where sometimes it seemed flush_workqueue was returning before all the work threads were finished.
Often this would cause strange crashes as the mutexes would be freed while they were being used.
Looking at the code, there is a lifetime problem as the controlling thread that spawns the work allocates the "struct stress" structures that are passed to the workqueue threads. Then when the workqueue threads are finished, they free the stress struct that was passed to them.
Unfortunately the workqueue work_struct node is in the stress struct. Which means the work_struct is freed before the work thread returns and while flush_workqueue is waiting.
It seems like a better idea to have the controlling thread both allocate and free the stress structures, so that we can be sure we don't corrupt the workqueue by freeing the structure prematurely.
So this patch reworks the test to do so, and with this change I no longer see the early flush_workqueue returns.
--- kernel/locking/test-ww_mutex.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-)
kernel/locking/test-ww_mutex.c +12 -8
@@ -479,7 +479,6 @@ static void stress_inorder_work(struct work_struct *work) }while (!time_after (jiffies, stress->timeout));kfree (order); -kfree (stress); }struct reorder_lock {
@@ -544,7 +543,6 @@ static void stress_reorder_work(struct work_struct *work)list_for_each_entry_safe (ll, ln, &locks, link)kfree (ll);kfree (order); -kfree (stress); }static void stress_one_work (struct work_struct *work)
@@ -565,8 +563,6 @@ static void stress_one_work(struct work_struct *work)break ; } }while (!time_after (jiffies, stress->timeout)); - -kfree (stress); }#define STRESS_INORDER BIT(0)
@@ -577,15 +573,24 @@ static void stress_one_work(struct work_struct *work)static int stress (int nlocks,int nthreads,unsigned int flags) {struct ww_mutex *locks; -int n; +struct stress *stress_array; +int n, count; locks =kmalloc_array (nlocks,sizeof (*locks),GFP_KERNEL );if (!locks)return -ENOMEM ; + stress_array =kmalloc_array (nthreads,sizeof (*stress_array), +GFP_KERNEL ); +if (!stress_array) { +kfree (locks); +return -ENOMEM ; + } +for (n =0 ; n < nlocks; n++)ww_mutex_init (&locks[n], &ww_class); + count =0 ;for (n =0 ; nthreads; n++) {struct stress *stress;void (*fn)(struct work_struct *work);
@@ -609,9 +614,7 @@ static int stress(int nlocks, int nthreads, unsigned int flags)if (!fn)continue ; - stress =kmalloc (sizeof (*stress),GFP_KERNEL ); -if (!stress) -break ; + stress = &stress_array[count++];INIT_WORK (&stress->work, fn); stress->locks = locks;
@@ -626,6 +629,7 @@ static int stress(int nlocks, int nthreads, unsigned int flags)for (n =0 ; n < nlocks; n++)ww_mutex_destroy (&locks[n]); +kfree (stress_array);kfree (locks);return 0 ;
Signature · 1 line
2.41.0.640.ga95def55d0-goog
RFC PATCH 3/3 test-ww_mutex: Make sure we bail out instead of livelock
In reply to John StultzI've seen what appears to be livelocks in the stress_inorder_work() function, and looking at the code it is clear we can have a case where we continually retry acquiring the locks and never check to see if we have passed the specified timeout.
This patch reworks that function so we always check the timeout before iterating through the loop again.
I believe others may have hit this previously here:
https://lore.kernel.org/lkml/[email protected]/
--- kernel/locking/test-ww_mutex.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
kernel/locking/test-ww_mutex.c +5 -4
@@ -465,17 +465,18 @@ static void stress_inorder_work(struct work_struct *work)ww_mutex_unlock (&locks[order[n]]);if (err == -EDEADLK ) { -ww_mutex_lock_slow (&locks[order[contended]], &ctx); - goto retry; +if (!time_after (jiffies, stress->timeout)) { +ww_mutex_lock_slow (&locks[order[contended]], &ctx); + goto retry; + } } +ww_acquire_fini (&ctx);if (err) {pr_err_once ("stress (%s) failed with %d\n" , __func__, err);break ; } - -ww_acquire_fini (&ctx); }while (!time_after (jiffies, stress->timeout));kfree (order);
Signature · 1 line
2.41.0.640.ga95def55d0-goog