KVM: selftests: Fix the dirty_log_test semaphore imbalance
PATCH v2 1/1 KVM: selftests: Fix the dirty_log_test semaphore imbalance
When execute the dirty_log_test on some aarch64 machine, it sometimes trigger the ASSERT:
==== Test Assertion Failure ====
dirty_log_test.c:384: dirty_ring_vcpu_ring_full
pid=14854 tid=14854 errno=22 - Invalid argument
1 0x00000000004033eb: dirty_ring_collect_dirty_pages at dirty_log_test.c:384
2 0x0000000000402d27: log_mode_collect_dirty_pages at dirty_log_test.c:505
3 (inlined by) run_test at dirty_log_test.c:802
4 0x0000000000403dc7: for_each_guest_mode at guest_modes.c:100
5 0x0000000000401dff: main at dirty_log_test.c:941 (discriminator 3)
6 0x0000ffff9be173c7: ?? ??:0
7 0x0000ffff9be1749f: ?? ??:0
8 0x000000000040206f: _start at ??:?Didn't continue vcpu even without ring full
The dirty_log_test fails when execute the dirty-ring test, this is because the sem_vcpu_cont and the sem_vcpu_stop is non-zero value when execute the dirty_ring_collect_dirty_pages() function. When those two sem_t variables are non-zero, the dirty_ring_wait_vcpu() at the beginning of the dirty_ring_collect_dirty_pages() will not wait for the vcpu to stop, but continue to execute the following code. In this case, before vcpu stop, if the dirty_ring_vcpu_ring_full is true, and the dirty_ring_collect_dirty_pages() has passed the check for the dirty_ring_vcpu_ring_full but hasn't execute the check for the continued_vcpu, the vcpu stop, and set the dirty_ring_vcpu_ring_full to false. Then dirty_ring_collect_dirty_pages() will trigger the ASSERT.
Why sem_vcpu_cont and sem_vcpu_stop can be non-zero value? It's because the dirty_ring_before_vcpu_join() execute the sem_post(&sem_vcpu_cont) at the end of each dirty-ring test. It can cause two cases:
1. sem_vcpu_cont be non-zero. When we set the host_quit to be true, the vcpu_worker directly see the host_quit to be true, it quit. So the log_mode_before_vcpu_join() function will set the sem_vcpu_cont to 1, since the vcpu_worker has quit, it won't consume it.
2. sem_vcpu_stop be non-zero. When we set the host_quit to be true, the vcpu_worker has entered the guest state, the next time it exit from guest state, it will set the sem_vcpu_stop to 1, and then see the host_quit, no one will consume the sem_vcpu_stop.
When execute more and more dirty-ring tests, the sem_vcpu_cont and sem_vcpu_stop can be larger and larger, which makes many code paths don't wait for the sem_t. Thus finally cause the problem.
To fix this problem, we can wait a while before set the host_quit to true, which gives the vcpu time to enter the guest state, so it will exit again. Then we can wait the vcpu to exit, and let it continue again, then the vcpu will see the host_quit. Thus the sem_vcpu_cont and sem_vcpu_stop will be both zero when test finished.
---v1->v2:
- Fix the real logic bug, not just fresh the context.
v1: https://lore.kernel.org/all/[email protected]/
--- tools/testing/selftests/kvm/dirty_log_test.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/dirty_log_test.c +15 -1
@@ -417,7 +417,8 @@ static void dirty_ring_after_vcpu_run(struct kvm_vcpu *vcpu, int ret, int err)static void dirty_ring_before_vcpu_join (void ) { -/* Kick another round of vcpu just to make sure it will quit */ +/* Wait vcpu exit, and let it continue to see the host_quit. */ +dirty_ring_wait_vcpu ();sem_post (&sem_vcpu_cont); }
@@ -719,6 +720,7 @@ static void run_test(enum vm_guest_mode mode, void *arg)struct kvm_vm *vm;unsigned long *bmap;uint32_t ring_buf_idx =0 ; +int sem_val;if (!log_mode_supported ()) { print_skip("Log mode '%s' not supported",
@@ -726,6 +728,11 @@ static void run_test(enum vm_guest_mode mode, void *arg)return ; } +sem_getvalue (&sem_vcpu_stop, &sem_val); +assert (sem_val ==0 ); +sem_getvalue (&sem_vcpu_cont, &sem_val); +assert (sem_val ==0 ); +/* * We reserve page table for 2 times of extra dirty mem which * will definitely cover the original (1G+) test range. Here
@@ -825,6 +832,13 @@ static void run_test(enum vm_guest_mode mode, void *arg)sync_global_to_guest (vm, iteration); } +/* +* +* Before we set the host_quit, let the vcpu has time to run, to make +* sure we consume the sem_vcpu_stop and the vcpu consume the +* sem_vcpu_cont, to keep the semaphore balance. +*/ +usleep (p->interval *1000 );/* Tell the vcpu thread to quit */ host_quit = true;log_mode_before_vcpu_join ();
Signature · 1 line
2.40.1
Kindly ping..
Quoted 109 lines · Shaoqin Huang
When execute the
dirty_log_teston some aarch64 machine, it sometimes trigger the ASSERT:==== Test Assertion Failure ====
dirty_log_test.c:384:dirty_ring_vcpu_ring_fullpid=14854 tid=14854 errno=22 - Invalid argument1 0x00000000004033eb: dirty_ring_collect_dirty_pages at dirty_log_test.c:384 2 0x0000000000402d27: log_mode_collect_dirty_pages at dirty_log_test.c:505 3 (inlined by) run_test at dirty_log_test.c:802 4 0x0000000000403dc7: for_each_guest_mode at guest_modes.c:100 5 0x0000000000401dff: main at dirty_log_test.c:941 (discriminator 3) 6 0x0000ffff9be173c7: ?? ??:0 7 0x0000ffff9be1749f: ?? ??:0 8 0x000000000040206f: _start at ??:?Didn't continue vcpu even without ring full
The
dirty_log_testfails when execute the dirty-ring test, this is because thesem_vcpu_contand thesem_vcpu_stopis non-zero value when execute thedirty_ring_collect_dirty_pages()function. When those twosem_tvariables are non-zero, thedirty_ring_wait_vcpu()at the beginning of thedirty_ring_collect_dirty_pages()will not wait for the vcpu to stop, but continue to execute the following code. In this case, before vcpu stop, if thedirty_ring_vcpu_ring_fullis true, and thedirty_ring_collect_dirty_pages()has passed the check for thedirty_ring_vcpu_ring_fullbut hasn't execute the check for thecontinued_vcpu, the vcpu stop, and set thedirty_ring_vcpu_ring_fullto false. Thendirty_ring_collect_dirty_pages()will trigger the ASSERT.Why
sem_vcpu_contandsem_vcpu_stopcan be non-zero value? It's because thedirty_ring_before_vcpu_join()execute thesem_post(&sem_vcpu_cont)at the end of each dirty-ring test. It can cause two cases:1.
sem_vcpu_contbe non-zero. When we set thehost_quitto be true, thevcpu_workerdirectly see thehost_quitto be true, it quit. So thelog_mode_before_vcpu_join()function will set thesem_vcpu_contto 1, since thevcpu_workerhas quit, it won't consume it.
2.sem_vcpu_stopbe non-zero. When we set thehost_quitto be true, thevcpu_workerhas entered the guest state, the next time it exit from guest state, it will set thesem_vcpu_stopto 1, and then see thehost_quit, no one will consume thesem_vcpu_stop.When execute more and more dirty-ring tests, the
sem_vcpu_contandsem_vcpu_stopcan be larger and larger, which makes many code paths don't wait for thesem_t. Thus finally cause the problem.To fix this problem, we can wait a while before set the
host_quitto true, which gives the vcpu time to enter the guest state, so it will exit again. Then we can wait the vcpu to exit, and let it continue again, then the vcpu will see thehost_quit. Thus thesem_vcpu_contandsem_vcpu_stopwill be both zero when test finished.Signed-off-by Shaoqin Huang <[email protected]>---
v1->v2:
- Fix the real logic bug, not just fresh the context.v1: https://lore.kernel.org/all/[email protected]/
--- tools/testing/selftests/kvm/dirty_log_test.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)tools/testing/selftests/kvm/dirty_log_test.c +15
-1@@ -417,7 +417,8 @@ static void dirty_ring_after_vcpu_run(struct kvm_vcpu *vcpu, int ret, int err)static void dirty_ring_before_vcpu_join (void ) { -/* Kick another round of vcpu just to make sure it will quit */ +/* Wait vcpu exit, and let it continue to see the host_quit. */ +dirty_ring_wait_vcpu ();sem_post (&sem_vcpu_cont); }@@ -719,6 +720,7 @@ static void run_test(enum vm_guest_mode mode, void *arg)struct kvm_vm *vm;unsigned long *bmap;uint32_t ring_buf_idx =0 ; +int sem_val;if (!log_mode_supported ()) { print_skip("Log mode '%s' not supported",@@ -726,6 +728,11 @@ static void run_test(enum vm_guest_mode mode, void *arg)return ; } +sem_getvalue (&sem_vcpu_stop, &sem_val); +assert (sem_val ==0 ); +sem_getvalue (&sem_vcpu_cont, &sem_val); +assert (sem_val ==0 ); +/* * We reserve page table for 2 times of extra dirty mem which * will definitely cover the original (1G+) test range. Here @@ -825,6 +832,13 @@ static void run_test(enum vm_guest_mode mode, void *arg)sync_global_to_guest (vm, iteration); } +/* +* +* Before we set the host_quit, let the vcpu has time to run, to make +* sure we consume the sem_vcpu_stop and the vcpu consume the +* sem_vcpu_cont, to keep the semaphore balance. +*/ +usleep (p->interval *1000 );/* Tell the vcpu thread to quit */ host_quit = true;log_mode_before_vcpu_join ();
Signature · 1 line
Shaoqin
Sean Christopherson
reviewer+Peter
Quoted 84 lines · Shaoqin Huang
When execute the
dirty_log_teston some aarch64 machine, it sometimes trigger the ASSERT:==== Test Assertion Failure ====
dirty_log_test.c:384:dirty_ring_vcpu_ring_full
pid=14854 tid=14854 errno=22 - Invalid argument1 0x00000000004033eb: dirty_ring_collect_dirty_pages at dirty_log_test.c:384 2 0x0000000000402d27: log_mode_collect_dirty_pages at dirty_log_test.c:505 3 (inlined by) run_test at dirty_log_test.c:802 4 0x0000000000403dc7: for_each_guest_mode at guest_modes.c:100 5 0x0000000000401dff: main at dirty_log_test.c:941 (discriminator 3) 6 0x0000ffff9be173c7: ?? ??:0 7 0x0000ffff9be1749f: ?? ??:0 8 0x000000000040206f: _start at ??:?Didn't continue vcpu even without ring full
The
dirty_log_testfails when execute the dirty-ring test, this is because thesem_vcpu_contand thesem_vcpu_stopis non-zero value when execute thedirty_ring_collect_dirty_pages()function. When those twosem_tvariables are non-zero, thedirty_ring_wait_vcpu()at the beginning of thedirty_ring_collect_dirty_pages()will not wait for the vcpu to stop, but continue to execute the following code. In this case, before vcpu stop, if thedirty_ring_vcpu_ring_fullis true, and thedirty_ring_collect_dirty_pages()has passed the check for thedirty_ring_vcpu_ring_fullbut hasn't execute the check for thecontinued_vcpu, the vcpu stop, and set thedirty_ring_vcpu_ring_fullto false. Thendirty_ring_collect_dirty_pages()will trigger the ASSERT.Why
sem_vcpu_contandsem_vcpu_stopcan be non-zero value? It's because thedirty_ring_before_vcpu_join()execute thesem_post(&sem_vcpu_cont)at the end of each dirty-ring test. It can cause two cases:1.
sem_vcpu_contbe non-zero. When we set thehost_quitto be true, thevcpu_workerdirectly see thehost_quitto be true, it quit. So thelog_mode_before_vcpu_join()function will set thesem_vcpu_contto 1, since thevcpu_workerhas quit, it won't consume it.
2.sem_vcpu_stopbe non-zero. When we set thehost_quitto be true, thevcpu_workerhas entered the guest state, the next time it exit from guest state, it will set thesem_vcpu_stopto 1, and then see thehost_quit, no one will consume thesem_vcpu_stop.When execute more and more dirty-ring tests, the
sem_vcpu_contandsem_vcpu_stopcan be larger and larger, which makes many code paths don't wait for thesem_t. Thus finally cause the problem.To fix this problem, we can wait a while before set the
host_quitto true, which gives the vcpu time to enter the guest state, so it will exit again. Then we can wait the vcpu to exit, and let it continue again, then the vcpu will see thehost_quit. Thus thesem_vcpu_contandsem_vcpu_stopwill be both zero when test finished.Signed-off-by Shaoqin Huang <[email protected]>---
v1->v2:
- Fix the real logic bug, not just fresh the context.v1: https://lore.kernel.org/all/[email protected]/
--- tools/testing/selftests/kvm/dirty_log_test.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)tools/testing/selftests/kvm/dirty_log_test.c +3
-1@@ -417,7 +417,8 @@ static void dirty_ring_after_vcpu_run(struct kvm_vcpu *vcpu, int ret, int err)static void dirty_ring_before_vcpu_join (void ) { -/* Kick another round of vcpu just to make sure it will quit */ +/* Wait vcpu exit, and let it continue to see the host_quit. */ +dirty_ring_wait_vcpu ();sem_post (&sem_vcpu_cont); }@@ -719,6 +720,7 @@ static void run_test(enum vm_guest_mode mode, void *arg)struct kvm_vm *vm;unsigned long *bmap;uint32_t ring_buf_idx =0 ; +int sem_val;if (!log_mode_supported ()) { print_skip("Log mode '%s' not supported",
tools/testing/selftests/kvm/dirty_log_test.c static void run_test(enum vm_guest_mode mode, void *arg) · line 734
Quoted 7 lines
tools/testing/selftests/kvm/dirty_log_test.c +4
-0@@ -726,6 +728,11 @@ static void run_test(enum vm_guest_mode mode, void *arg)return ; } +sem_getvalue (&sem_vcpu_stop, &sem_val); +assert (sem_val ==0 ); +sem_getvalue (&sem_vcpu_cont, &sem_val); +assert (sem_val ==0 );
Never use bare assert() in selftests, they'll either get compiled out or IIRC, will cause the test to silently exit if they fail. Either do nothing, or use TEST_ASSERT().
Quoted 5 lines
tools/testing/selftests/kvm/dirty_log_test.c +1
-0+/* * We reserve page table for 2 times of extra dirty mem which * will definitely cover the original (1G+) test range. Here
tools/testing/selftests/kvm/dirty_log_test.c static void run_test(enum vm_guest_mode mode, void *arg) · line 841
Quoted 10 lines
tools/testing/selftests/kvm/dirty_log_test.c +7
-0@@ -825,6 +832,13 @@ static void run_test(enum vm_guest_mode mode, void *arg)sync_global_to_guest (vm, iteration); } +/* +* +* Before we set the host_quit, let the vcpu has time to run, to make +* sure we consume the sem_vcpu_stop and the vcpu consume the +* sem_vcpu_cont, to keep the semaphore balance. +*/ +usleep (p->interval *1000 );
Please no. "Wait for a while" is never a complete solution for fixing races. In rare cases, adding a delay might be the only sane workaround, but I doubt that's the case here.
Hi Sean,
Thanks for your reviewing.
Quoted 90 lines · Sean Christopherson
+Peter
Quoted 87 lines · Shaoqin Huang
When execute the
dirty_log_teston some aarch64 machine, it sometimes trigger the ASSERT:==== Test Assertion Failure ====
dirty_log_test.c:384:dirty_ring_vcpu_ring_fullpid=14854 tid=14854 errno=22 - Invalid argument1 0x00000000004033eb: dirty_ring_collect_dirty_pages at dirty_log_test.c:384 2 0x0000000000402d27: log_mode_collect_dirty_pages at dirty_log_test.c:505 3 (inlined by) run_test at dirty_log_test.c:802 4 0x0000000000403dc7: for_each_guest_mode at guest_modes.c:100 5 0x0000000000401dff: main at dirty_log_test.c:941 (discriminator 3) 6 0x0000ffff9be173c7: ?? ??:0 7 0x0000ffff9be1749f: ?? ??:0 8 0x000000000040206f: _start at ??:?Didn't continue vcpu even without ring full
The
dirty_log_testfails when execute the dirty-ring test, this is because thesem_vcpu_contand thesem_vcpu_stopis non-zero value when execute thedirty_ring_collect_dirty_pages()function. When those twosem_tvariables are non-zero, thedirty_ring_wait_vcpu()at the beginning of thedirty_ring_collect_dirty_pages()will not wait for the vcpu to stop, but continue to execute the following code. In this case, before vcpu stop, if thedirty_ring_vcpu_ring_fullis true, and thedirty_ring_collect_dirty_pages()has passed the check for thedirty_ring_vcpu_ring_fullbut hasn't execute the check for thecontinued_vcpu, the vcpu stop, and set thedirty_ring_vcpu_ring_fullto false. Thendirty_ring_collect_dirty_pages()will trigger the ASSERT.Why
sem_vcpu_contandsem_vcpu_stopcan be non-zero value? It's because thedirty_ring_before_vcpu_join()execute thesem_post(&sem_vcpu_cont)at the end of each dirty-ring test. It can cause two cases:1.
sem_vcpu_contbe non-zero. When we set thehost_quitto be true, thevcpu_workerdirectly see thehost_quitto be true, it quit. So thelog_mode_before_vcpu_join()function will set thesem_vcpu_contto 1, since thevcpu_workerhas quit, it won't consume it.
2.sem_vcpu_stopbe non-zero. When we set thehost_quitto be true, thevcpu_workerhas entered the guest state, the next time it exit from guest state, it will set thesem_vcpu_stopto 1, and then see thehost_quit, no one will consume thesem_vcpu_stop.When execute more and more dirty-ring tests, the
sem_vcpu_contandsem_vcpu_stopcan be larger and larger, which makes many code paths don't wait for thesem_t. Thus finally cause the problem.To fix this problem, we can wait a while before set the
host_quitto true, which gives the vcpu time to enter the guest state, so it will exit again. Then we can wait the vcpu to exit, and let it continue again, then the vcpu will see thehost_quit. Thus thesem_vcpu_contandsem_vcpu_stopwill be both zero when test finished.Signed-off-by Shaoqin Huang <[email protected]>---
v1->v2:
- Fix the real logic bug, not just fresh the context.v1: https://lore.kernel.org/all/[email protected]/
--- tools/testing/selftests/kvm/dirty_log_test.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)tools/testing/selftests/kvm/dirty_log_test.c +3
-1@@ -417,7 +417,8 @@ static void dirty_ring_after_vcpu_run(struct kvm_vcpu *vcpu, int ret, int err)static void dirty_ring_before_vcpu_join (void ) { -/* Kick another round of vcpu just to make sure it will quit */ +/* Wait vcpu exit, and let it continue to see the host_quit. */ +dirty_ring_wait_vcpu ();sem_post (&sem_vcpu_cont); }@@ -719,6 +720,7 @@ static void run_test(enum vm_guest_mode mode, void *arg)struct kvm_vm *vm;unsigned long *bmap;uint32_t ring_buf_idx =0 ; +int sem_val;if (!log_mode_supported ()) { print_skip("Log mode '%s' not supported",@@ -726,6 +728,11 @@ static void run_test(enum vm_guest_mode mode, void *arg)return ; }
Quoted 9 lines
tools/testing/selftests/kvm/dirty_log_test.c static void run_test(enum vm_guest_mode mode, void *arg) · line 734
Quoted 4 lines
tools/testing/selftests/kvm/dirty_log_test.c +4
-0+sem_getvalue (&sem_vcpu_stop, &sem_val); +assert (sem_val ==0 ); +sem_getvalue (&sem_vcpu_cont, &sem_val); +assert (sem_val ==0 );Never use bare
assert()in selftests, they'll either get compiled out or IIRC, will cause the test to silently exit if they fail. Either do nothing, or useTEST_ASSERT().
Sure. I will replace them with TEST_ASSERT().
Quoted 8 lines
Quoted 8 lines
tools/testing/selftests/kvm/dirty_log_test.c +1
-0+/* * We reserve page table for 2 times of extra dirty mem which * will definitely cover the original (1G+) test range. Here @@ -825,6 +832,13 @@ static void run_test(enum vm_guest_mode mode, void *arg)sync_global_to_guest (vm, iteration); }
Quoted 11 lines
tools/testing/selftests/kvm/dirty_log_test.c static void run_test(enum vm_guest_mode mode, void *arg) · line 841
Quoted 7 lines
tools/testing/selftests/kvm/dirty_log_test.c +7
-0+/* +* +* Before we set the host_quit, let the vcpu has time to run, to make +* sure we consume the sem_vcpu_stop and the vcpu consume the +* sem_vcpu_cont, to keep the semaphore balance. +*/ +usleep (p->interval *1000 );Please no. "Wait for a while" is never a complete solution for fixing races. In rare cases, adding a delay might be the only sane workaround, but I doubt that's the case here.
If that's the case. I guess I should keep the current solution. Except you have any better solution, please let me know.
Thanks,
Shaoqin
Quoted 1 line
Signature · 1 line
Shaoqin
Sean Christopherson
reviewerQuoted 18 lines · Shaoqin Huang
Quoted 9 lines
Quoted 9 lines
/*
* We reserve page table for 2 times of extra dirty mem which
* will definitely cover the original (1G+) test range. Heretools/testing/selftests/kvm/dirty_log_test.c +3
-0@@ -825,6 +832,13 @@ static void run_test(enum vm_guest_mode mode, void *arg)sync_global_to_guest (vm, iteration); } +/* +* +* Before we set the host_quit, let the vcpu has time to run, to make Quoted 8 lines
tools/testing/selftests/kvm/dirty_log_test.c static void run_test(enum vm_guest_mode mode, void *arg) · line 841
Quoted 4 lines
tools/testing/selftests/kvm/dirty_log_test.c +4
-0+* sure we consume the sem_vcpu_stop and the vcpu consume the +* sem_vcpu_cont, to keep the semaphore balance. +*/ +usleep (p->interval *1000 );Please no. "Wait for a while" is never a complete solution for fixing races. In rare cases, adding a delay might be the only sane workaround, but I doubt that's the case here.
Quoted 2 lines
If that's the case. I guess I should keep the current solution. Except you have any better solution, please let me know.
Unfortunately I don't have a better solution, and I don't have cycles to stare at this deeply to figure out what how to make the synchronization rock solid. Sorry :-/