s390x: SIGP fixes
cover s390x: SIGP fixes
Hi all,
Some of you may remember the recent discussions to straighten out some serialization issues with the SIGNAL PROCESSOR instruction, and its interaction with userspace. This resulted in a few kernel patches [1] that reliably solve the issues I was seeing.
The attached kvm-unit-tests series adapts the existing smp tests such that it can reproduce the problems I saw when those patches are reverted (typically by simultaneously doing a kernel compile), and also demonstrate that those patches indeed fix the issue.
There's some cleanup in here too, based on my understanding of the smp tests as I was walking through here. Thoughts?
[1] 812de04661c4 KVM: s390: Clarify SIGP orders versus STOP/RESTART 67cf68b6a5cc KVM: s390: Add a routine for setting userspace CPU state 8eeba194a32e KVM: s390: Simplify SIGP Set Arch handling
Eric Farman (6):
lib: s390x: smp: Retry SIGP SENSE on CC2
s390x: smp: Test SIGP RESTART against stopped CPU
s390x: smp: Fix checks for SIGP STOP STORE STATUS
s390x: smp: Create and use a non-waiting CPU stop
s390x: smp: Create and use a non-waiting CPU restart
lib: s390x: smp: Convert remaining smp_sigp to _retry
lib/s390x/smp.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++--- lib/s390x/smp.h | 2 ++ s390x/smp.c | 39 +++++++++++++++++++----------------- 3 files changed, 73 insertions(+), 21 deletions(-)
Signature · 1 line
2.32.0
PATCH 1/6 lib: s390x: smp: Retry SIGP SENSE on CC2
The routine smp_cpu_stopped() issues a SIGP SENSE, and returns true if it received a CC1 (STATUS STORED) with the STOPPED or CHECK STOP bits enabled. Otherwise, it returns false.
This is misleading, because a CC2 (BUSY) merely indicates that the order code could not be processed, not that the CPU is operating. It could be operating but in the process of being stopped.
Convert the invocation of the SIGP SENSE to retry when a CC2 is received, so we get a more definitive answer.
--- lib/s390x/smp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/s390x/smp.c +1 -1
@@ -78,7 +78,7 @@ bool smp_cpu_stopped(uint16_t idx) {uint32_t status; -if (smp_sigp (idx,SIGP_SENSE ,0 , &status) !=SIGP_CC_STATUS_STORED ) +if (smp_sigp_retry (idx,SIGP_SENSE ,0 , &status) !=SIGP_CC_STATUS_STORED )return false;return !!(status & (SIGP_STATUS_CHECK_STOP |SIGP_STATUS_STOPPED )); }
Signature · 1 line
2.32.0
Re: [PATCH kvm-unit-tests v1 1/6] lib: s390x: smp: Retry SIGP SENSE on CC2
Quoted 4 lines · Eric Farman
The routine
smp_cpu_stopped()issues a SIGP SENSE, and returns true if it received a CC1 (STATUS STORED) with the STOPPED or CHECK STOP bits enabled. Otherwise, it returns false.
Quoted 8 lines
This is misleading, because a CC2 (BUSY) merely indicates that the order code could not be processed, not that the CPU is operating. It could be operating but in the process of being stopped.
Convert the invocation of the SIGP SENSE to retry when a CC2 is received, so we get a more definitive answer.
Signed-off-by Eric Farman <[email protected]>
Re: [PATCH kvm-unit-tests v1 1/6] lib: s390x: smp: Retry SIGP SENSE on CC2
In reply to Eric FarmanQuoted 8 lines · On Thu, 3 Mar 2022 22:04:20 +0100 Eric Farman <[email protected]> wrote:
The routine
smp_cpu_stopped()issues a SIGP SENSE, and returns true if it received a CC1 (STATUS STORED) with the STOPPED or CHECK STOP bits enabled. Otherwise, it returns false.This is misleading, because a CC2 (BUSY) merely indicates that the order code could not be processed, not that the CPU is operating. It could be operating but in the process of being stopped.
Quoted 5 lines
Convert the invocation of the SIGP SENSE to retry when a CC2 is received, so we get a more definitive answer.
Signed-off-by Eric Farman <[email protected]>Reviewed-by Janosch Frank <[email protected]>
Quoted 18 lines
--- lib/s390x/smp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)lib/s390x/smp.c +1
-1@@ -78,7 +78,7 @@ bool smp_cpu_stopped(uint16_t idx) {uint32_t status; -if (smp_sigp (idx,SIGP_SENSE ,0 , &status) != SIGP_CC_STATUS_STORED) +if (smp_sigp_retry (idx,SIGP_SENSE ,0 , &status) !=SIGP_CC_STATUS_STORED )return false; return !!(status &(SIGP_STATUS_CHECK_STOP|SIGP_STATUS_STOPPED)); }
PATCH 2/6 s390x: smp: Test SIGP RESTART against stopped CPU
In reply to Eric Farmantest_restart() makes two smp_cpu_restart() calls against CPU 1. It claims to perform both of them against running (operating) CPUs, but the first invocation tries to achieve this by calling smp_cpu_stop() to CPU 0. This will be rejected by the library.
Let's fix this by making the first restart operate on a stopped CPU, to ensure it gets test coverage instead of relying on other callers.
--- s390x/smp.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
s390x/smp.c +2 -6
@@ -50,10 +50,6 @@ static void test_start(void)report_pass ("start" ); } -/* -* Does only test restart when the target is running. -* The other tests do restarts when stopped multiple times already. -*/ static void test_restart (void ) {struct cpu *cpu =smp_cpu_from_idx (1 );
@@ -62,8 +58,8 @@ static void test_restart(void) lc->restart_new_psw.mask =extract_psw_mask (); lc->restart_new_psw.addr = (unsigned long )test_func; -/* Make sure cpu is running */ -smp_cpu_stop (0 ); +/* Make sure cpu is stopped */ +smp_cpu_stop (1 );set_flag (0 );smp_cpu_restart (1 );wait_for_flag ();
Signature · 1 line
2.32.0
Re: [PATCH kvm-unit-tests v1 2/6] s390x: smp: Test SIGP RESTART against stopped CPU
Quoted 4 lines · Eric Farman
test_restart()makes twosmp_cpu_restart()calls against CPU 1. It claims to perform both of them against running (operating) CPUs, but the first invocation tries to achieve this by callingsmp_cpu_stop()to CPU 0. This will be rejected by the library.
I played myself there :)
Quoted 6 lines
Let's fix this by making the first restart operate on a stopped CPU, to ensure it gets test coverage instead of relying on other callers.
Fixes 166da884d ("s390x: smp: Add restart when running test")Signed-off-by Eric Farman <[email protected]>
If you want to you can add a report_pass() after the first wait flag.
Quoted 30 lines
--- s390x/smp.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)s390x/smp.c +2
-6@@ -50,10 +50,6 @@ static void test_start(void)report_pass ("start" ); } -/* -* Does only test restart when the target is running. -* The other tests do restarts when stopped multiple times already. -*/ static void test_restart (void ) {struct cpu *cpu =smp_cpu_from_idx (1 );@@ -62,8 +58,8 @@ static void test_restart(void) lc->restart_new_psw.mask =extract_psw_mask (); lc->restart_new_psw.addr = (unsigned long )test_func; -/* Make sure cpu is running */ -smp_cpu_stop (0 ); +/* Make sure cpu is stopped */ +smp_cpu_stop (1 );set_flag (0 );smp_cpu_restart (1 );wait_for_flag ();
Re: [PATCH kvm-unit-tests v1 2/6] s390x: smp: Test SIGP RESTART against stopped CPU
Quoted 7 lines · Janosch Frank
Quoted 4 lines · Eric Farman
test_restart()makes twosmp_cpu_restart()calls against CPU 1. It claims to perform both of them against running (operating) CPUs, but the first invocation tries to achieve this by callingsmp_cpu_stop()to CPU 0. This will be rejected by the library.I played myself there :)
:)
Quoted 10 lines
Quoted 8 lines
Let's fix this by making the first restart operate on a stopped CPU,
to ensure it gets test coverage instead of relying on other callers.Fixes 166da884d ("s390x: smp: Add restart when running test")Signed-off-by Eric Farman <[email protected]>If you want to you can add a
report_pass()after the first wait flag.
I did in patch 5. I can move that here, but that patch has some additional prefix changes for those reports, so didn't want to move TOO much of that to this fix.
Quoted 2 lines
Reviewed-by Janosch Frank <[email protected]>
Thank you!
Quoted 32 lines
Quoted 32 lines
--- s390x/smp.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)s390x/smp.c +0
-3@@ -50,10 +50,6 @@ static void test_start(void)report_pass ("start" ); } -/* -* Does only test restart when the target is running. -* The other tests do restarts when stopped multiple times already.
- */
static voidtest_restart(void)
{
struct cpu *cpu =smp_cpu_from_idx(1);s390x/smp.c +2
-2@@ -62,8 +58,8 @@ static void test_restart(void) lc->restart_new_psw.mask =extract_psw_mask (); lc->restart_new_psw.addr = (unsigned long )test_func; -/* Make sure cpu is running */ -smp_cpu_stop (0 ); +/* Make sure cpu is stopped */ +smp_cpu_stop (1 );set_flag (0 );smp_cpu_restart (1 );wait_for_flag ();
Re: [PATCH kvm-unit-tests v1 2/6] s390x: smp: Test SIGP RESTART against stopped CPU
In reply to Eric FarmanQuoted 10 lines · Eric Farman
test_restart()makes twosmp_cpu_restart()calls against CPU 1. It claims to perform both of them against running (operating) CPUs, but the first invocation tries to achieve this by callingsmp_cpu_stop()to CPU 0. This will be rejected by the library.Let's fix this by making the first restart operate on a stopped CPU, to ensure it gets test coverage instead of relying on other callers.
Fixes 166da884d ("s390x: smp: Add restart when running test")Signed-off-by Eric Farman <[email protected]>
Re: [PATCH kvm-unit-tests v1 2/6] s390x: smp: Test SIGP RESTART against stopped CPU
In reply to Eric FarmanQuoted 10 lines · On Thu, 3 Mar 2022 22:04:21 +0100 Eric Farman <[email protected]> wrote:
test_restart()makes twosmp_cpu_restart()calls against CPU 1. It claims to perform both of them against running (operating) CPUs, but the first invocation tries to achieve this by callingsmp_cpu_stop()to CPU 0. This will be rejected by the library.Let's fix this by making the first restart operate on a stopped CPU, to ensure it gets test coverage instead of relying on other callers.
Fixes 166da884d ("s390x: smp: Add restart when running test")Signed-off-by Eric Farman <[email protected]>
Quoted 30 lines
--- s390x/smp.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)s390x/smp.c +2
-6@@ -50,10 +50,6 @@ static void test_start(void)report_pass ("start" ); } -/* -* Does only test restart when the target is running. -* The other tests do restarts when stopped multiple times already. -*/ static void test_restart (void ) {struct cpu *cpu =smp_cpu_from_idx (1 );@@ -62,8 +58,8 @@ static void test_restart(void) lc->restart_new_psw.mask =extract_psw_mask (); lc->restart_new_psw.addr = (unsigned long )test_func; -/* Make sure cpu is running */ -smp_cpu_stop (0 ); +/* Make sure cpu is stopped */ +smp_cpu_stop (1 );set_flag (0 );smp_cpu_restart (1 );wait_for_flag ();
PATCH 3/6 s390x: smp: Fix checks for SIGP STOP STORE STATUS
In reply to Eric FarmanIn the routine test_stop_store_status(), the "running" part of the test checks a few of the fields in lowcore (to verify the "STORE STATUS" part of the SIGP order), and then ensures that the CPU has stopped. But this is backwards, and leads to false errors.
According to the Principles of Operation:
The addressed CPU performs the stop function, fol- lowed by the store-status operation (see “Store Sta- tus” on page 4-82).
By checking the results how they are today, the contents of the lowcore fields are unreliable until the CPU is stopped. Thus, check that the CPU is stopped first, before ensuring that the STORE STATUS was performed correctly.
While here, add the same check to the second part of the test, even though the CPU is explicitly stopped prior to the SIGP.
--- s390x/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
s390x/smp.c +2 -1
@@ -98,9 +98,9 @@ static void test_stop_store_status(void) lc->grs_sa[15 ] =0 ;smp_cpu_stop_store_status (1 );mb (); +report (smp_cpu_stopped (1 ),"cpu stopped" );report (lc->prefix_sa == (uint32_t )(uintptr_t )cpu->lowcore,"prefix" );report (lc->grs_sa[15 ],"stack" ); -report (smp_cpu_stopped (1 ),"cpu stopped" );report_prefix_pop ();report_prefix_push ("stopped" );
@@ -108,6 +108,7 @@ static void test_stop_store_status(void) lc->grs_sa[15 ] =0 ;smp_cpu_stop_store_status (1 );mb (); +report (smp_cpu_stopped (1 ),"cpu stopped" );report (lc->prefix_sa == (uint32_t )(uintptr_t )cpu->lowcore,"prefix" );report (lc->grs_sa[15 ],"stack" );report_prefix_pop ();
Signature · 1 line
2.32.0
Re: [PATCH kvm-unit-tests v1 3/6] s390x: smp: Fix checks for SIGP STOP STORE STATUS
Quoted 11 lines · Eric Farman
In the routine
test_stop_store_status(), the "running" part of the test checks a few of the fields in lowcore (to verify the "STORE STATUS" part of the SIGP order), and then ensures that the CPU has stopped. But this is backwards, and leads to false errors.According to the Principles of Operation:
The addressed CPU performs the stop function, fol- lowed by the store-status operation (see “Store Sta- tus” on page 4-82).
Quoted 4 lines
By checking the results how they are today, the contents of the lowcore fields are unreliable until the CPU is stopped. Thus, check that the CPU is stopped first, before ensuring that the STORE STATUS was performed correctly.
The results are undefined until the cpu is not busy via SIGP sense, no? You cover that via doing the smp_cpu_stopped() check since that does a sigp sense.
Where the stop check is located doesn't really matter since the library waits until the cpu is stopped and it does that via smp_cpu_stopped()
So:
Are we really fixing something here?
Please improve the commit description.
For me this looks more like making checks more explicit and symmetrical which I'm generally ok with. We just need to specify correctly why we're doing that.
Quoted 33 lines
While here, add the same check to the second part of the test, even though the CPU is explicitly stopped prior to the SIGP.
Fixes fc67b07a4 ("s390x: smp: Test stop and store status on a running and stopped cpu")Signed-off-by Eric Farman <[email protected]>--- s390x/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)s390x/smp.c +2
-1@@ -98,9 +98,9 @@ static void test_stop_store_status(void) lc->grs_sa[15 ] =0 ;smp_cpu_stop_store_status (1 );mb (); +report (smp_cpu_stopped (1 ),"cpu stopped" );report (lc->prefix_sa == (uint32_t )(uintptr_t )cpu->lowcore,"prefix" );report (lc->grs_sa[15 ],"stack" ); -report (smp_cpu_stopped (1 ),"cpu stopped" );report_prefix_pop ();report_prefix_push ("stopped" );@@ -108,6 +108,7 @@ static void test_stop_store_status(void) lc->grs_sa[15 ] =0 ;smp_cpu_stop_store_status (1 );mb (); +report (smp_cpu_stopped (1 ),"cpu stopped" );report (lc->prefix_sa == (uint32_t )(uintptr_t )cpu->lowcore,"prefix" );report (lc->grs_sa[15 ],"stack" );report_prefix_pop ();
Re: [PATCH kvm-unit-tests v1 3/6] s390x: smp: Fix checks for SIGP STOP STORE STATUS
Quoted 23 lines · Janosch Frank
Quoted 11 lines · Eric Farman
In the routine
test_stop_store_status(), the "running" part of the test checks a few of the fields in lowcore (to verify the "STORE STATUS" part of the SIGP order), and then ensures that the CPU has stopped. But this is backwards, and leads to false errors.According to the Principles of Operation:
The addressed CPU performs the stop function, fol- lowed by the store-status operation (see “Store Sta- tus” on page 4-82).Quoted 4 lines
By checking the results how they are today, the contents of the lowcore fields are unreliable until the CPU is stopped. Thus, check that the CPU is stopped first, before ensuring that the STORE STATUS was performed correctly.
The results are undefined until the cpu is not busy via SIGP sense, no?
You cover that via doing thesmp_cpu_stopped()check since that does a
sigp sense.
Quoted 7 lines
Where the stop check is located doesn't really matter since the library
waits until the cpu is stopped and it does that viasmp_cpu_stopped()So:
Are we really fixing something here?
Hrm, I thought so, but I got focused on the order of these checks and overlooked the point that the library already does this looping. I do trip up on these checks; let me revisit them.
Quoted 43 lines
Please improve the commit description.
For me this looks more like making checks more explicit and symmetrical
which I'm generally ok with. We just need to specify correctly why we're
doing that.Quoted 35 lines
While here, add the same check to the second part of the test, even though the CPU is explicitly stopped prior to the SIGP.
Fixes fc67b07a4 ("s390x: smp: Test stop and store status on arunning and stopped cpu")
Signed-off-by Eric Farman <[email protected]>--- s390x/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)s390x/smp.c +1
-1@@ -98,9 +98,9 @@ static void test_stop_store_status(void) lc->grs_sa[15 ] =0 ;smp_cpu_stop_store_status (1 );mb (); +report (smp_cpu_stopped (1 ),"cpu stopped" );report (lc->prefix_sa == (uint32_t )(uintptr_t )cpu->lowcore,"prefix" );report (lc->grs_sa[15 ],"stack" ); -report (smp_cpu_stopped (1 ),"cpu stopped" );report_prefix_pop ();report_prefix_push("stopped");s390x/smp.c +1
-0@@ -108,6 +108,7 @@ static void test_stop_store_status(void) lc->grs_sa[15 ] =0 ;smp_cpu_stop_store_status (1 );mb (); +report (smp_cpu_stopped (1 ),"cpu stopped" );report (lc->prefix_sa == (uint32_t )(uintptr_t )cpu->lowcore,"prefix" );report (lc->grs_sa[15 ],"stack" );report_prefix_pop();
Re: [PATCH kvm-unit-tests v1 3/6] s390x: smp: Fix checks for SIGP STOP STORE STATUS
Quoted 31 lines · Eric Farman
Quoted 30 lines · Janosch Frank
Quoted 11 lines · Eric Farman
In the routine
test_stop_store_status(), the "running" part of the test checks a few of the fields in lowcore (to verify the "STORE STATUS" part of the SIGP order), and then ensures that the CPU has stopped. But this is backwards, and leads to false errors.According to the Principles of Operation:
The addressed CPU performs the stop function, fol- lowed by the store-status operation (see “Store Sta- tus” on page 4-82).Quoted 4 lines
By checking the results how they are today, the contents of the lowcore fields are unreliable until the CPU is stopped. Thus, check that the CPU is stopped first, before ensuring that the STORE STATUS was performed correctly.
The results are undefined until the cpu is not busy via SIGP sense, no?
You cover that via doing thesmp_cpu_stopped()check since that does
a
sigp sense.Where the stop check is located doesn't really matter since the library
waits until the cpu is stopped and it does that viasmp_cpu_stopped()
Quoted 6 lines
Quoted 2 lines
So:
Are we really fixing something here?Hrm, I thought so, but I got focused on the order of these checks and overlooked the point that the library already does this looping. I do trip up on these checks; let me revisit them.
Ah, my turn to fool myself. To test all the different combinations, I had both old/new SIGP behavior in otherwise identical kernels and QEMU binaries. But I appear to have mislabeled QEMU, so the failures I was seeing was due to running the old QEMU, and not anything in kvm-unit- tests itself. My apologies.
So, per your next paragraph, I'll keep this patch but tidy up the commit message accordingly.
Quoted 45 lines
Quoted 45 lines
Please improve the commit description.
For me this looks more like making checks more explicit and symmetrical
which I'm generally ok with. We just need to specify correctly why we're
doing that.Quoted 37 lines
While here, add the same check to the second part of the test, even though the CPU is explicitly stopped prior to the SIGP.
Fixes fc67b07a4 ("s390x: smp: Test stop and store status on arunning and stopped cpu")
Signed-off-by Eric Farman <[email protected]>--- s390x/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)s390x/smp.c +1
-1@@ -98,9 +98,9 @@ static void test_stop_store_status(void) lc->grs_sa[15 ] =0 ;smp_cpu_stop_store_status (1 );mb (); +report (smp_cpu_stopped (1 ),"cpu stopped" );report (lc->prefix_sa == (uint32_t )(uintptr_t )cpu- >lowcore,"prefix" );report (lc->grs_sa[15 ],"stack" ); -report (smp_cpu_stopped (1 ),"cpu stopped" );report_prefix_pop ();report_prefix_push("stopped");s390x/smp.c +1
-0@@ -108,6 +108,7 @@ static void test_stop_store_status(void) lc->grs_sa[15 ] =0 ;smp_cpu_stop_store_status (1 );mb (); +report (smp_cpu_stopped (1 ),"cpu stopped" );report (lc->prefix_sa == (uint32_t )(uintptr_t )cpu- >lowcore,"prefix" );report(lc->grs_sa[15], "stack"); report_prefix_pop();
PATCH 4/6 s390x: smp: Create and use a non-waiting CPU stop
In reply to Eric FarmanWhen stopping a CPU, kvm-unit-tests serializes/waits for everything to finish, in order to get a consistent result whenever those functions are used.
But to test the SIGP STOP itself, these additional measures could mask other problems. For example, did the STOP work, or is the CPU still operating?
Let's create a non-waiting SIGP STOP and use it here, to ensure that the CPU is correctly stopped. A smp_cpu_stopped() call will still be used to see that the SIGP STOP has been processed, and the state of the CPU can be used to determine whether the test passes/fails.
--- lib/s390x/smp.c | 25 +++++++++++++++++++++++++ lib/s390x/smp.h | 1 + s390x/smp.c | 10 ++-------- 3 files changed, 28 insertions(+), 8 deletions(-)
lib/s390x/smp.c +25 -0
@@ -119,6 +119,31 @@ int smp_cpu_stop(uint16_t idx)return rc; } +/* +* Functionally equivalent to smp_cpu_stop(), but without the +* elements that wait/serialize matters itself. +* Used to see if KVM itself is serialized correctly. +*/ +int smp_cpu_stop_nowait (uint16_t idx) +{ +/* refuse to work on the boot CPU */ +if (idx ==0 ) +return -1 ; + +spin_lock (&lock); + +/* Don't suppress a CC2 with sigp_retry() */ +if (smp_sigp (idx,SIGP_STOP ,0 ,NULL )) { +spin_unlock (&lock); +return -1 ; + } + + cpus[idx].active = false; +spin_unlock (&lock); + +return 0 ; +} +int smp_cpu_stop_store_status (uint16_t idx) {int rc;
lib/s390x/smp.h +1 -0
@@ -44,6 +44,7 @@ bool smp_sense_running_status(uint16_t idx);int smp_cpu_restart (uint16_t idx);int smp_cpu_start (uint16_t idx,struct psw psw);int smp_cpu_stop (uint16_t idx); +int smp_cpu_stop_nowait (uint16_t idx);int smp_cpu_stop_store_status (uint16_t idx);int smp_cpu_destroy (uint16_t idx);int smp_cpu_setup (uint16_t idx,struct psw psw);
s390x/smp.c +2 -8
@@ -76,14 +76,8 @@ static void test_restart(void)static void test_stop (void ) { -smp_cpu_stop (1 ); -/* -* The smp library waits for the CPU to shut down, but let's -* also do it here, so we don't rely on the library -* implementation -*/ -while (!smp_cpu_stopped (1 )) {} -report_pass ("stop" ); +smp_cpu_stop_nowait (1 ); +report (smp_cpu_stopped (1 ),"stop" ); }static void test_stop_store_status (void )
Signature · 1 line
2.32.0
Re: [PATCH kvm-unit-tests v1 4/6] s390x: smp: Create and use a non-waiting CPU stop
On Thu, 2022-03-03 at 22:04 +0100, Eric Farman wrote:
[...]
Quoted 7 lines
s390x/smp.c +0
-0@@ -76,14 +76,8 @@ static void test_restart(void)
static voidtest_stop(void)
{
- smp_cpu_stop(1);
Quoted 8 lines
- /*
- * The smp library waits for the CPU to shut down, but let's - * also do it here, so we don't rely on the library - * implementation
- */
- while (!smp_cpu_stopped(1)) {}
- report_pass("stop");
+ smp_cpu_stop_nowait(1);
Now that this can fail because of CC=2, should we check the return value here?
Re: [PATCH kvm-unit-tests v1 4/6] s390x: smp: Create and use a non-waiting CPU stop
Quoted 7 lines · Nico Boehr
On Thu, 2022-03-03 at 22:04 +0100, Eric Farman wrote:
[...]Quoted 4 lines
s390x/smp.c +0
-0@@ -76,14 +76,8 @@ static void test_restart(void)
Quoted 15 lines
s390x/smp.c static void test_restart(void) · line 79
Quoted 12 lines
s390x/smp.c +0
-3static void test_stop (void ) { -smp_cpu_stop (1 ); -/* -* The smp library waits for the CPU to shut down, but let's
s390x/smp.c +1
-5-* also do it here, so we don't rely on the library -* implementation -*/ -while (!smp_cpu_stopped (1 )) {} -report_pass ("stop" ); +smp_cpu_stop_nowait (1 );Now that this can fail because of CC=2, should we check the return value here?
Ah, yes it should. Thanks,
Eric
Re: [PATCH kvm-unit-tests v1 4/6] s390x: smp: Create and use a non-waiting CPU stop
In reply to Eric FarmanQuoted 75 lines · On Thu, 3 Mar 2022 22:04:23 +0100 Eric Farman <[email protected]> wrote:
When stopping a CPU, kvm-unit-tests serializes/waits for everything to finish, in order to get a consistent result whenever those functions are used.
But to test the SIGP STOP itself, these additional measures could mask other problems. For example, did the STOP work, or is the CPU still operating?
Let's create a non-waiting SIGP STOP and use it here, to ensure that the CPU is correctly stopped. A
smp_cpu_stopped()call will still be used to see that the SIGP STOP has been processed, and the state of the CPU can be used to determine whether the test passes/fails.Signed-off-by Eric Farman <[email protected]>--- lib/s390x/smp.c | 25 +++++++++++++++++++++++++ lib/s390x/smp.h | 1 + s390x/smp.c | 10 ++-------- 3 files changed, 28 insertions(+), 8 deletions(-)lib/s390x/smp.c +25
-0@@ -119,6 +119,31 @@ int smp_cpu_stop(uint16_t idx)return rc; } +/* +* Functionally equivalent to smp_cpu_stop(), but without the +* elements that wait/serialize matters itself. +* Used to see if KVM itself is serialized correctly. +*/ +int smp_cpu_stop_nowait (uint16_t idx) +{ +/* refuse to work on the boot CPU */ +if (idx ==0 ) +return -1 ; + +spin_lock (&lock); + +/* Don't suppress a CC2 with sigp_retry() */ +if (smp_sigp (idx,SIGP_STOP ,0 ,NULL )) { +spin_unlock (&lock); +return -1 ; + } + + cpus[idx].active = false; +spin_unlock (&lock); + +return 0 ; +} +int smp_cpu_stop_store_status (uint16_t idx) {int rc;lib/s390x/smp.h +1
-0@@ -44,6 +44,7 @@ bool smp_sense_running_status(uint16_t idx);int smp_cpu_restart (uint16_t idx);int smp_cpu_start (uint16_t idx,struct psw psw);int smp_cpu_stop (uint16_t idx); +int smp_cpu_stop_nowait (uint16_t idx);int smp_cpu_stop_store_status (uint16_t idx);int smp_cpu_destroy (uint16_t idx);int smp_cpu_setup (uint16_t idx,struct psw psw);s390x/smp.c +0
-0@@ -76,14 +76,8 @@ static void test_restart(void)static void test_stop (void )
s390x/smp.c static void test_restart(void) · line 79
Quoted 10 lines
s390x/smp.c +1
-8{ -smp_cpu_stop (1 ); -/* -* The smp library waits for the CPU to shut down, but let's -* also do it here, so we don't rely on the library -* implementation -*/ -while (!smp_cpu_stopped (1 )) {} -report_pass ("stop" ); +smp_cpu_stop_nowait (1 );
can it happen that the SIGP STOP order is accepted, but the target CPU is still running (and not even busy)?
s390x/smp.c static void test_restart(void) · line 80
Quoted 1 line
s390x/smp.c +1
-0+report (smp_cpu_stopped (1 ),"stop" );
e.g. can this ^ check race with the actual stopping of the CPU?
Quoted 3 lines
s390x/smp.c +0
-0}static void test_stop_store_status (void )
Re: [PATCH kvm-unit-tests v1 4/6] s390x: smp: Create and use a non-waiting CPU stop
Quoted 78 lines · Claudio Imbrenda
Quoted 75 lines · On Thu, 3 Mar 2022 22:04:23 +0100 Eric Farman <[email protected]> wrote:
When stopping a CPU, kvm-unit-tests serializes/waits for everything to finish, in order to get a consistent result whenever those functions are used.
But to test the SIGP STOP itself, these additional measures could mask other problems. For example, did the STOP work, or is the CPU still operating?
Let's create a non-waiting SIGP STOP and use it here, to ensure that
the CPU is correctly stopped. Asmp_cpu_stopped()call will still be used to see that the SIGP STOP has been processed, and the state of the CPU can be used to determine whether the test passes/fails.Signed-off-by Eric Farman <[email protected]>--- lib/s390x/smp.c | 25 +++++++++++++++++++++++++ lib/s390x/smp.h | 1 + s390x/smp.c | 10 ++-------- 3 files changed, 28 insertions(+), 8 deletions(-)lib/s390x/smp.c +25
-0@@ -119,6 +119,31 @@ int smp_cpu_stop(uint16_t idx)return rc; } +/* +* Functionally equivalent to smp_cpu_stop(), but without the +* elements that wait/serialize matters itself. +* Used to see if KVM itself is serialized correctly. +*/ +int smp_cpu_stop_nowait (uint16_t idx) +{ +/* refuse to work on the boot CPU */ +if (idx ==0 ) +return -1 ; + +spin_lock (&lock); + +/* Don't suppress a CC2 with sigp_retry() */ +if (smp_sigp (idx,SIGP_STOP ,0 ,NULL )) { +spin_unlock (&lock); +return -1 ; + } + + cpus[idx].active = false; +spin_unlock (&lock); + +return 0 ; +} +int smp_cpu_stop_store_status (uint16_t idx) {int rc;lib/s390x/smp.h +1
-0@@ -44,6 +44,7 @@ bool smp_sense_running_status(uint16_t idx);int smp_cpu_restart (uint16_t idx);int smp_cpu_start (uint16_t idx,struct psw psw);int smp_cpu_stop (uint16_t idx); +int smp_cpu_stop_nowait (uint16_t idx);int smp_cpu_stop_store_status (uint16_t idx);int smp_cpu_destroy (uint16_t idx);int smp_cpu_setup (uint16_t idx,struct psw psw);s390x/smp.c +0
-0@@ -76,14 +76,8 @@ static void test_restart(void)
Quoted 15 lines
s390x/smp.c static void test_restart(void) · line 79
Quoted 11 lines
s390x/smp.c +1
-8static void test_stop (void ) { -smp_cpu_stop (1 ); -/* -* The smp library waits for the CPU to shut down, but let's -* also do it here, so we don't rely on the library -* implementation -*/ -while (!smp_cpu_stopped (1 )) {} -report_pass ("stop" ); +smp_cpu_stop_nowait (1 );can it happen that the SIGP STOP order is accepted, but the target CPU
is still running (and not even busy)?
Of course. A SIGP that's processed by userspace (which is many of them) injects a STOP IRQ back to the kernel, which means the CPU might not be stopped for some time. But...
Quoted 4 lines
s390x/smp.c static void test_restart(void) · line 80
Quoted 2 lines
s390x/smp.c +1
-0+report (smp_cpu_stopped (1 ),"stop" );e.g. can this ^ check race with the actual stopping of the CPU?
...the smp_cpu_stopped() routine now loops on the CC2 that SIGP SENSE returns because of that pending IRQ. If SIGP SENSE returns CC0/1, then the CPU can correctly be identified stopped/operating, and the test can correctly pass/fail.
Quoted 4 lines
Quoted 4 lines
s390x/smp.c +0
-0}static void test_stop_store_status (void )
Re: [PATCH kvm-unit-tests v1 4/6] s390x: smp: Create and use a non-waiting CPU stop
Quoted 100 lines · On Mon, 07 Mar 2022 14:03:45 -0500 Eric Farman <[email protected]> wrote:
Quoted 78 lines · Claudio Imbrenda
Quoted 75 lines · On Thu, 3 Mar 2022 22:04:23 +0100 Eric Farman <[email protected]> wrote:
When stopping a CPU, kvm-unit-tests serializes/waits for everything to finish, in order to get a consistent result whenever those functions are used.
But to test the SIGP STOP itself, these additional measures could mask other problems. For example, did the STOP work, or is the CPU still operating?
Let's create a non-waiting SIGP STOP and use it here, to ensure that
the CPU is correctly stopped. Asmp_cpu_stopped()call will still be used to see that the SIGP STOP has been processed, and the state of the CPU can be used to determine whether the test passes/fails.Signed-off-by Eric Farman <[email protected]>--- lib/s390x/smp.c | 25 +++++++++++++++++++++++++ lib/s390x/smp.h | 1 + s390x/smp.c | 10 ++-------- 3 files changed, 28 insertions(+), 8 deletions(-)lib/s390x/smp.c +25
-0@@ -119,6 +119,31 @@ int smp_cpu_stop(uint16_t idx)return rc; } +/* +* Functionally equivalent to smp_cpu_stop(), but without the +* elements that wait/serialize matters itself. +* Used to see if KVM itself is serialized correctly. +*/ +int smp_cpu_stop_nowait (uint16_t idx) +{ +/* refuse to work on the boot CPU */ +if (idx ==0 ) +return -1 ; + +spin_lock (&lock); + +/* Don't suppress a CC2 with sigp_retry() */ +if (smp_sigp (idx,SIGP_STOP ,0 ,NULL )) { +spin_unlock (&lock); +return -1 ; + } + + cpus[idx].active = false; +spin_unlock (&lock); + +return 0 ; +} +int smp_cpu_stop_store_status (uint16_t idx) {int rc;lib/s390x/smp.h +1
-0@@ -44,6 +44,7 @@ bool smp_sense_running_status(uint16_t idx);int smp_cpu_restart (uint16_t idx);int smp_cpu_start (uint16_t idx,struct psw psw);int smp_cpu_stop (uint16_t idx); +int smp_cpu_stop_nowait (uint16_t idx);int smp_cpu_stop_store_status (uint16_t idx);int smp_cpu_destroy (uint16_t idx);int smp_cpu_setup (uint16_t idx,struct psw psw);s390x/smp.c +0
-0@@ -76,14 +76,8 @@ static void test_restart(void)Quoted 15 lines
s390x/smp.c static void test_restart(void) · line 79
Quoted 11 lines
s390x/smp.c +1
-8static void test_stop (void ) { -smp_cpu_stop (1 ); -/* -* The smp library waits for the CPU to shut down, but let's -* also do it here, so we don't rely on the library -* implementation -*/ -while (!smp_cpu_stopped (1 )) {} -report_pass ("stop" ); +smp_cpu_stop_nowait (1 );can it happen that the SIGP STOP order is accepted, but the target CPU
is still running (and not even busy)?Of course. A SIGP that's processed by userspace (which is many of them) injects a STOP IRQ back to the kernel, which means the CPU might not be stopped for some time. But...
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
s390x/smp.c static void test_restart(void) · line 80
Quoted 1 line
s390x/smp.c +1
-0+report (smp_cpu_stopped (1 ),"stop" );e.g. can this ^ check race with the actual stopping of the CPU?
...the
smp_cpu_stopped()routine now loops on the CC2 that SIGP SENSE returns because of that pending IRQ. If SIGP SENSE returns CC0/1, then the CPU can correctly be identified stopped/operating, and the test can correctly pass/fail.
my question was: is it possible architecturally that there is a window where the STOP order is accepted, but a SENSE on the target CPU still successfully returns that the CPU is running?
in other words: is it specified architecturally that, once an order is accepted for a target CPU, that CPU can't accept any other order (and will return CC2), including SENSE, until the order has been completed successfully?
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
s390x/smp.c +0
-0}static void test_stop_store_status (void )
Re: [PATCH kvm-unit-tests v1 4/6] s390x: smp: Create and use a non-waiting CPU stop
Quoted 125 lines · Claudio Imbrenda
Quoted 114 lines · On Mon, 07 Mar 2022 14:03:45 -0500 Eric Farman <[email protected]> wrote:
Quoted 93 lines · Claudio Imbrenda
Quoted 90 lines · On Thu, 3 Mar 2022 22:04:23 +0100 Eric Farman <[email protected]> wrote:
When stopping a CPU, kvm-unit-tests serializes/waits for everything
to finish, in order to get a consistent result whenever those functions are used.But to test the SIGP STOP itself, these additional measures could
mask other problems. For example, did the STOP work, or is the CPU
still operating?Let's create a non-waiting SIGP STOP and use it here, to ensure that
the CPU is correctly stopped. Asmp_cpu_stopped()call will still
be used to see that the SIGP STOP has been processed, and the state
of the CPU can be used to determine whether the test passes/fails.Signed-off-by Eric Farman <[email protected]>--- lib/s390x/smp.c | 25 +++++++++++++++++++++++++ lib/s390x/smp.h | 1 + s390x/smp.c | 10 ++-------- 3 files changed, 28 insertions(+), 8 deletions(-)lib/s390x/smp.c +25
-0@@ -119,6 +119,31 @@ int smp_cpu_stop(uint16_t idx)return rc; } +/* +* Functionally equivalent to smp_cpu_stop(), but without the +* elements that wait/serialize matters itself. +* Used to see if KVM itself is serialized correctly. +*/ +int smp_cpu_stop_nowait (uint16_t idx) +{ +/* refuse to work on the boot CPU */ +if (idx ==0 ) +return -1 ; + +spin_lock (&lock); + +/* Don't suppress a CC2 with sigp_retry() */ +if (smp_sigp (idx,SIGP_STOP ,0 ,NULL )) { +spin_unlock (&lock); +return -1 ; + } + + cpus[idx].active = false; +spin_unlock (&lock); + +return 0 ; +} +int smp_cpu_stop_store_status (uint16_t idx) {int rc;lib/s390x/smp.h +1
-0@@ -44,6 +44,7 @@ bool smp_sense_running_status(uint16_t idx);int smp_cpu_restart (uint16_t idx);int smp_cpu_start (uint16_t idx,struct psw psw);int smp_cpu_stop (uint16_t idx); +int smp_cpu_stop_nowait (uint16_t idx);int smp_cpu_stop_store_status (uint16_t idx);int smp_cpu_destroy (uint16_t idx);int smp_cpu_setup (uint16_t idx,struct psw psw);s390x/smp.c +0
-3@@ -76,14 +76,8 @@ static void test_restart(void)static void test_stop (void ) { -smp_cpu_stop (1 ); -/* -* The smp library waits for the CPU to shut down, but let's
s390x/smp.c +0
-3-* also do it here, so we don't rely on the library -* implementation -*/ Quoted 8 lines
s390x/smp.c static void test_restart(void) · line 79
Quoted 3 lines
s390x/smp.c +1
-2-while (!smp_cpu_stopped (1 )) {} -report_pass ("stop" ); +smp_cpu_stop_nowait (1 );can it happen that the SIGP STOP order is accepted, but the target
CPU
is still running (and not even busy)?Of course. A SIGP that's processed by userspace (which is many of them)
injects a STOP IRQ back to the kernel, which means the CPU might not be
stopped for some time. But...Quoted 4 lines
s390x/smp.c static void test_restart(void) · line 80
Quoted 2 lines
s390x/smp.c +1
-0+report (smp_cpu_stopped (1 ),"stop" );e.g. can this ^ check race with the actual stopping of the CPU?
Quoted 7 lines
...the
smp_cpu_stopped()routine now loops on the CC2 that SIGP SENSE
returns because of that pending IRQ. If SIGP SENSE returns CC0/1, then
the CPU can correctly be identified stopped/operating, and the test can
correctly pass/fail.
Quoted 4 lines
my question was: is it possible architecturally that there is a window
where the STOP order is accepted, but a SENSE on the target CPU still successfully returns that the CPU is running?
Not to my knowledge. The "Conditions Determining Response" section of POPS (v12, page 4-95; also below) specifically states that the SIGP SENSE will return CC2 when a SIGP STOP order is outstanding.
In our implementation, the stop IRQ will be injected before the STOP order gets accepted, such that a SIGP SENSE would see it immediately.
"""
A previously issued start, stop, restart, stop- and-store-status, set-prefix, store-status-at- address order, or store-additional-status-at- address has been accepted by the
addressed CPU, and execution of the func- tion requested by the order has not yet been completed.
...
If the currently specified order is sense, external call, emergency signal, start, stop, restart, stop and store status, set prefix, store status at address, set architecture, set multithreading, or store additional status at address, then the order is rejected, and condition code 2 is set. """
Quoted 6 lines
in other words: is it specified architecturally that, once an order is
accepted for a target CPU, that CPU can't accept any other order (and will return CC2), including SENSE, until the order has been completed successfully?
The POPS quote I placed above excludes store-extended-status-address, conditional-emergency-signal, and sense-running-status orders as returning a CC2. That's something Janosch and I have chatted about offline, but don't have an answer to at this time. Besides that, any other order would get a CC2 until the STOP has been completed.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
}
static void
test_stop_store_status(void)
Claudio Imbrenda
Reviewed-byRe: [PATCH kvm-unit-tests v1 4/6] s390x: smp: Create and use a non-waiting CPU stop
Quoted 162 lines · On Tue, 08 Mar 2022 16:18:16 -0500 Eric Farman <[email protected]> wrote:
Quoted 125 lines · Claudio Imbrenda
Quoted 114 lines · On Mon, 07 Mar 2022 14:03:45 -0500 Eric Farman <[email protected]> wrote:
Quoted 93 lines · Claudio Imbrenda
Quoted 90 lines · On Thu, 3 Mar 2022 22:04:23 +0100 Eric Farman <[email protected]> wrote:
When stopping a CPU, kvm-unit-tests serializes/waits for everything
to finish, in order to get a consistent result whenever those functions are used.But to test the SIGP STOP itself, these additional measures could
mask other problems. For example, did the STOP work, or is the CPU
still operating?Let's create a non-waiting SIGP STOP and use it here, to ensure that
the CPU is correctly stopped. Asmp_cpu_stopped()call will still
be used to see that the SIGP STOP has been processed, and the state
of the CPU can be used to determine whether the test passes/fails.Signed-off-by Eric Farman <[email protected]>--- lib/s390x/smp.c | 25 +++++++++++++++++++++++++ lib/s390x/smp.h | 1 + s390x/smp.c | 10 ++-------- 3 files changed, 28 insertions(+), 8 deletions(-)lib/s390x/smp.c +25
-0@@ -119,6 +119,31 @@ int smp_cpu_stop(uint16_t idx)return rc; } +/* +* Functionally equivalent to smp_cpu_stop(), but without the +* elements that wait/serialize matters itself. +* Used to see if KVM itself is serialized correctly. +*/ +int smp_cpu_stop_nowait (uint16_t idx) +{ +/* refuse to work on the boot CPU */ +if (idx ==0 ) +return -1 ; + +spin_lock (&lock); + +/* Don't suppress a CC2 with sigp_retry() */ +if (smp_sigp (idx,SIGP_STOP ,0 ,NULL )) { +spin_unlock (&lock); +return -1 ; + } + + cpus[idx].active = false; +spin_unlock (&lock); + +return 0 ; +} +int smp_cpu_stop_store_status (uint16_t idx) {int rc;lib/s390x/smp.h +1
-0@@ -44,6 +44,7 @@ bool smp_sense_running_status(uint16_t idx);int smp_cpu_restart (uint16_t idx);int smp_cpu_start (uint16_t idx,struct psw psw);int smp_cpu_stop (uint16_t idx); +int smp_cpu_stop_nowait (uint16_t idx);int smp_cpu_stop_store_status (uint16_t idx);int smp_cpu_destroy (uint16_t idx);int smp_cpu_setup (uint16_t idx,struct psw psw);s390x/smp.c +0
-3@@ -76,14 +76,8 @@ static void test_restart(void)static void test_stop (void ) { -smp_cpu_stop (1 ); -/* -* The smp library waits for the CPU to shut down, but let's
s390x/smp.c +0
-3-* also do it here, so we don't rely on the library -* implementation -*/ Quoted 8 lines
s390x/smp.c static void test_restart(void) · line 79
Quoted 3 lines
s390x/smp.c +1
-2-while (!smp_cpu_stopped (1 )) {} -report_pass ("stop" ); +smp_cpu_stop_nowait (1 );can it happen that the SIGP STOP order is accepted, but the target
CPU
is still running (and not even busy)?Of course. A SIGP that's processed by userspace (which is many of them)
injects a STOP IRQ back to the kernel, which means the CPU might not be
stopped for some time. But...Quoted 4 lines
s390x/smp.c static void test_restart(void) · line 80
Quoted 2 lines
s390x/smp.c +1
-0+report (smp_cpu_stopped (1 ),"stop" );e.g. can this ^ check race with the actual stopping of the CPU?
Quoted 7 lines
...the
smp_cpu_stopped()routine now loops on the CC2 that SIGP SENSE
returns because of that pending IRQ. If SIGP SENSE returns CC0/1, then
the CPU can correctly be identified stopped/operating, and the test can
correctly pass/fail.Quoted 4 lines
my question was: is it possible architecturally that there is a window
where the STOP order is accepted, but a SENSE on the target CPU still successfully returns that the CPU is running?Not to my knowledge. The "Conditions Determining Response" section of POPS (v12, page 4-95; also below) specifically states that the SIGP SENSE will return CC2 when a SIGP STOP order is outstanding.
In our implementation, the stop IRQ will be injected before the STOP order gets accepted, such that a SIGP SENSE would see it immediately.
"""
A previously issued start, stop, restart, stop- and-store-status, set-prefix, store-status-at- address order, or store-additional-status-at- address has been accepted by the
addressed CPU, and execution of the func- tion requested by the order has not yet been completed.
...
If the currently specified order is sense, external call, emergency signal, start, stop, restart, stop and store status, set prefix, store status at address, set architecture, set multithreading, or store additional status at address, then the order is rejected, and condition code 2 is set. """Quoted 6 lines
in other words: is it specified architecturally that, once an order is
accepted for a target CPU, that CPU can't accept any other order (and will return CC2), including SENSE, until the order has been completed successfully?
Quoted 5 lines
The POPS quote I placed above excludes store-extended-status-address, conditional-emergency-signal, and sense-running-status orders as returning a CC2. That's something Janosch and I have chatted about offline, but don't have an answer to at this time. Besides that, any other order would get a CC2 until the STOP has been completed.
this is what I wanted to know, thanks
you can add
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
}
static void
test_stop_store_status(void)
PATCH 5/6 s390x: smp: Create and use a non-waiting CPU restart
In reply to Eric FarmanThe kvm-unit-tests infrastructure for a CPU restart waits for the SIGP RESTART to complete. In order to test the restart itself, create a variation that does not wait, and test the state of the CPU directly.
While here, add some better report prefixes/messages, to clarify which condition is being examined (similar to test_stop_store_status()).
--- lib/s390x/smp.c | 22 ++++++++++++++++++++++ lib/s390x/smp.h | 1 + s390x/smp.c | 18 +++++++++++++++--- 3 files changed, 38 insertions(+), 3 deletions(-)
lib/s390x/smp.c +22 -0
@@ -192,6 +192,28 @@ int smp_cpu_restart(uint16_t idx)return rc; } +/* +* Functionally equivalent to smp_cpu_restart(), but without the +* elements that wait/serialize matters here in the test. +* Used to see if KVM itself is serialized correctly. +*/ +int smp_cpu_restart_nowait (uint16_t idx) +{ +spin_lock (&lock); + +/* Don't suppress a CC2 with sigp_retry() */ +if (smp_sigp (idx,SIGP_RESTART ,0 ,NULL )) { +spin_unlock (&lock); +return -1 ; + } + + cpus[idx].active = true; + +spin_unlock (&lock); + +return 0 ; +} +int smp_cpu_start (uint16_t idx,struct psw psw) {int rc;
lib/s390x/smp.h +1 -0
@@ -42,6 +42,7 @@ uint16_t smp_cpu_addr(uint16_t idx);bool smp_cpu_stopped (uint16_t idx);bool smp_sense_running_status (uint16_t idx);int smp_cpu_restart (uint16_t idx); +int smp_cpu_restart_nowait (uint16_t idx);int smp_cpu_start (uint16_t idx,struct psw psw);int smp_cpu_stop (uint16_t idx);int smp_cpu_stop_nowait (uint16_t idx);
s390x/smp.c +15 -3
@@ -55,23 +55,35 @@ static void test_restart(void)struct cpu *cpu =smp_cpu_from_idx (1 );struct lowcore *lc = cpu->lowcore; +report_prefix_push ("restart" ); +report_prefix_push ("stopped" ); + lc->restart_new_psw.mask =extract_psw_mask (); lc->restart_new_psw.addr = (unsigned long )test_func;/* Make sure cpu is stopped */ smp_cpu_stop (1 );set_flag (0 ); -smp_cpu_restart (1 ); +smp_cpu_restart_nowait (1 ); +report (!smp_cpu_stopped (1 ),"cpu started" );wait_for_flag (); +report_pass ("test flag" ); + +report_prefix_pop (); +report_prefix_push ("running" );/* * Wait until cpu 1 has set the flag because it executed the * restart function. */ set_flag (0 ); -smp_cpu_restart (1 ); +smp_cpu_restart_nowait (1 ); +report (!smp_cpu_stopped (1 ),"cpu started" );wait_for_flag (); -report_pass ("restart while running" ); +report_pass ("test flag" ); + +report_prefix_pop (); +report_prefix_pop (); }static void test_stop (void )
Signature · 1 line
2.32.0
Re: [PATCH kvm-unit-tests v1 5/6] s390x: smp: Create and use a non-waiting CPU restart
Quoted 70 lines · On Thu, 3 Mar 2022 22:04:24 +0100 Eric Farman <[email protected]> wrote:
The kvm-unit-tests infrastructure for a CPU restart waits for the SIGP RESTART to complete. In order to test the restart itself, create a variation that does not wait, and test the state of the CPU directly.
While here, add some better report prefixes/messages, to clarify which condition is being examined (similar to
test_stop_store_status()).Signed-off-by Eric Farman <[email protected]>--- lib/s390x/smp.c | 22 ++++++++++++++++++++++ lib/s390x/smp.h | 1 + s390x/smp.c | 18 +++++++++++++++--- 3 files changed, 38 insertions(+), 3 deletions(-)lib/s390x/smp.c +22
-0@@ -192,6 +192,28 @@ int smp_cpu_restart(uint16_t idx)return rc; } +/* +* Functionally equivalent to smp_cpu_restart(), but without the +* elements that wait/serialize matters here in the test. +* Used to see if KVM itself is serialized correctly. +*/ +int smp_cpu_restart_nowait (uint16_t idx) +{ +spin_lock (&lock); + +/* Don't suppress a CC2 with sigp_retry() */ +if (smp_sigp (idx,SIGP_RESTART ,0 ,NULL )) { +spin_unlock (&lock); +return -1 ; + } + + cpus[idx].active = true; + +spin_unlock (&lock); + +return 0 ; +} +int smp_cpu_start (uint16_t idx,struct psw psw) {int rc;lib/s390x/smp.h +1
-0@@ -42,6 +42,7 @@ uint16_t smp_cpu_addr(uint16_t idx);bool smp_cpu_stopped (uint16_t idx);bool smp_sense_running_status (uint16_t idx);int smp_cpu_restart (uint16_t idx); +int smp_cpu_restart_nowait (uint16_t idx);int smp_cpu_start (uint16_t idx,struct psw psw);int smp_cpu_stop (uint16_t idx);int smp_cpu_stop_nowait (uint16_t idx);s390x/smp.c +2
-0@@ -55,23 +55,35 @@ static void test_restart(void)struct cpu *cpu =smp_cpu_from_idx (1 );struct lowcore *lc = cpu->lowcore; +report_prefix_push ("restart" ); +report_prefix_push ("stopped" );
s390x/smp.c static void test_restart(void) · line 68
Quoted 10 lines
s390x/smp.c +3
-1+ lc->restart_new_psw.mask =extract_psw_mask (); lc->restart_new_psw.addr = (unsigned long )test_func;/* Make sure cpu is stopped */ smp_cpu_stop (1 );set_flag (0 ); -smp_cpu_restart (1 ); +smp_cpu_restart_nowait (1 ); +report (!smp_cpu_stopped (1 ),"cpu started" );
can this check ^ race?
we are using the flag to check if the CPU actually restarts, right?
Quoted 4 lines
s390x/smp.c +3
-0wait_for_flag (); +report_pass ("test flag" ); + +report_prefix_pop ();
s390x/smp.c static void test_restart(void) · line 81
Quoted 10 lines
s390x/smp.c +3
-1+report_prefix_push ("running" );/* * Wait until cpu 1 has set the flag because it executed the * restart function. */ set_flag (0 ); -smp_cpu_restart (1 ); +smp_cpu_restart_nowait (1 ); +report (!smp_cpu_stopped (1 ),"cpu started" );
same here
Quoted 9 lines
s390x/smp.c +4
-1wait_for_flag (); -report_pass ("restart while running" ); +report_pass ("test flag" ); + +report_prefix_pop (); +report_prefix_pop (); }static void test_stop (void )
PATCH 6/6 lib: s390x: smp: Convert remaining smp_sigp to _retry
In reply to Eric FarmanA SIGP SENSE is used to determine if a CPU is stopped or operating, and thus has a vested interest in ensuring it received a CC0 or CC1, instead of a CC2 (BUSY). But, any order could receive a CC2 response, and is probably ill-equipped to respond to it.
In practice, the order is likely to only encounter this when racing with a SIGP STOP (AND STORE STATUS) or SIGP RESTART order, which are unlikely. But, since it's not impossible, let's convert the library calls that issue a SIGP to loop on CC2 so the callers do not need to react to that possible outcome.
--- lib/s390x/smp.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
lib/s390x/smp.c +2 -2
@@ -85,7 +85,7 @@ bool smp_cpu_stopped(uint16_t idx)bool smp_sense_running_status (uint16_t idx) { -if (smp_sigp (idx,SIGP_SENSE_RUNNING ,0 ,NULL ) !=SIGP_CC_STATUS_STORED ) +if (smp_sigp_retry (idx,SIGP_SENSE_RUNNING ,0 ,NULL ) !=SIGP_CC_STATUS_STORED )return true;/* Status stored condition code is equivalent to cpu not running. */ return false;
@@ -169,7 +169,7 @@ static int smp_cpu_restart_nolock(uint16_t idx, struct psw *psw)* running after the restart. */ smp_cpu_stop_nolock (idx, false); - rc =smp_sigp (idx,SIGP_RESTART ,0 ,NULL ); + rc =smp_sigp_retry (idx,SIGP_RESTART ,0 ,NULL );if (rc)return rc;/*
Signature · 1 line
2.32.0
Re: [PATCH kvm-unit-tests v1 6/6] lib: s390x: smp: Convert remaining smp_sigp to _retry
Quoted 4 lines · Eric Farman
A SIGP SENSE is used to determine if a CPU is stopped or operating, and thus has a vested interest in ensuring it received a CC0 or CC1, instead of a CC2 (BUSY). But, any order could receive a CC2 response, and is probably ill-equipped to respond to it.
sigp sense running status doesn't return a cc2, only sigp sense does afaik.
Looking at the KVM implementation tells me that it's not doing more than looking at the R bit in the sblk.
Quoted 34 lines
In practice, the order is likely to only encounter this when racing with a SIGP STOP (AND STORE STATUS) or SIGP RESTART order, which are unlikely. But, since it's not impossible, let's convert the library calls that issue a SIGP to loop on CC2 so the callers do not need to react to that possible outcome.
Signed-off-by Eric Farman <[email protected]>--- lib/s390x/smp.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)lib/s390x/smp.c +2
-2@@ -85,7 +85,7 @@ bool smp_cpu_stopped(uint16_t idx)bool smp_sense_running_status (uint16_t idx) { -if (smp_sigp (idx,SIGP_SENSE_RUNNING ,0 ,NULL ) !=SIGP_CC_STATUS_STORED ) +if (smp_sigp_retry (idx,SIGP_SENSE_RUNNING ,0 ,NULL ) !=SIGP_CC_STATUS_STORED )return true;/* Status stored condition code is equivalent to cpu not running. */ return false;@@ -169,7 +169,7 @@ static int smp_cpu_restart_nolock(uint16_t idx, struct psw *psw)* running after the restart. */ smp_cpu_stop_nolock (idx, false); - rc =smp_sigp (idx,SIGP_RESTART ,0 ,NULL ); + rc =smp_sigp_retry (idx,SIGP_RESTART ,0 ,NULL );if (rc)return rc;/*
Re: [PATCH kvm-unit-tests v1 6/6] lib: s390x: smp: Convert remaining smp_sigp to _retry
Quoted 10 lines · Janosch Frank
Quoted 6 lines · Eric Farman
A SIGP SENSE is used to determine if a CPU is stopped or operating, and thus has a vested interest in ensuring it received a CC0 or CC1,
instead of a CC2 (BUSY). But, any order could receive a CC2 response,
and is probably ill-equipped to respond to it.sigp sense running status doesn't return a cc2, only sigp sense does afaik.
The KVM routine handle_sigp_dst() returns the CC2 if a STOP/RESTART IRQ is pending for any non-reset order, before it gets to the switch statement that would route to the SIGP SENSE RUNNING handler.
Quoted 42 lines
Looking at the KVM implementation tells me that it's not doing more than
looking at the R bit in the sblk.Quoted 38 lines
In practice, the order is likely to only encounter this when racing with a SIGP STOP (AND STORE STATUS) or SIGP RESTART order, which are
unlikely. But, since it's not impossible, let's convert the library calls that issue a SIGP to loop on CC2 so the callers do not need to react to that possible outcome.Signed-off-by Eric Farman <[email protected]>--- lib/s390x/smp.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)lib/s390x/smp.c +1
-1@@ -85,7 +85,7 @@ bool smp_cpu_stopped(uint16_t idx)bool smp_sense_running_status (uint16_t idx) { -if (smp_sigp (idx,SIGP_SENSE_RUNNING ,0 ,NULL ) != SIGP_CC_STATUS_STORED) +if (smp_sigp_retry (idx,SIGP_SENSE_RUNNING ,0 ,NULL ) !=SIGP_CC_STATUS_STORED )return true;/* Status stored condition code is equivalent to cpu not
running. */
return false;lib/s390x/smp.c +1
-1@@ -169,7 +169,7 @@ static int smp_cpu_restart_nolock(uint16_t idx,struct psw *psw) * running after the restart. */ smp_cpu_stop_nolock (idx, false); - rc =smp_sigp (idx,SIGP_RESTART ,0 ,NULL ); + rc =smp_sigp_retry (idx,SIGP_RESTART ,0 ,NULL );if (rc)return rc;/*
Re: [PATCH kvm-unit-tests v1 6/6] lib: s390x: smp: Convert remaining smp_sigp to _retry
In reply to Janosch FrankQuoted 8 lines · Janosch Frank
Quoted 6 lines · Eric Farman
A SIGP SENSE is used to determine if a CPU is stopped or operating, and thus has a vested interest in ensuring it received a CC0 or CC1,
instead of a CC2 (BUSY). But, any order could receive a CC2 response,
and is probably ill-equipped to respond to it.
Quoted 5 lines
sigp sense running status doesn't return a cc2, only sigp sense does afaik.
Looking at the KVM implementation tells me that it's not doing more than
looking at the R bit in the sblk.
From the POP I read _all_ orders may indeed return CC=2: case 1 under "Conditions precluding Interpretation of the Order Code".
That being said, there are a few more users of smp_sigp (no retry) in smp.c (the test, not the lib).
Does it make sense to fix them aswell?
Re: [PATCH kvm-unit-tests v1 6/6] lib: s390x: smp: Convert remaining smp_sigp to _retry
Quoted 17 lines · Nico Boehr
Quoted 9 lines · Janosch Frank
Quoted 7 lines · Eric Farman
A SIGP SENSE is used to determine if a CPU is stopped or operating,
and thus has a vested interest in ensuring it received a CC0 or CC1,
instead of a CC2 (BUSY). But, any order could receive a CC2 response,
and is probably ill-equipped to respond to it.Quoted 6 lines
sigp sense running status doesn't return a cc2, only sigp sense does
afaik.
Looking at the KVM implementation tells me that it's not doing more than
looking at the R bit in the sblk.
Quoted 7 lines
From the POP I read _all_ orders may indeed return CC=2: case 1 under "Conditions precluding Interpretation of the Order Code".
That being said, there are a few more users of
smp_sigp(no retry) insmp.c(the test, not the lib).Does it make sense to fix them aswell?
I thought it made sense to do the lib, since other places expect those things to "just work."
But for the tests themselves, I struggle to convince myself with one path over another. The only way KVM returns a CC2 is because of a concurrent STOP/RESTART, which isn't a possibility because of the waiting the lib itself does when invoking the STOP/RESTART. So should the tests be looking for an unexpected CC2? Or just loop when they occur? If the latter, shouldn't the lib itself do that?
I'm happy to make changes, I just can't decide which it should be. Any opinions?
Eric
Re: [PATCH kvm-unit-tests v1 6/6] lib: s390x: smp: Convert remaining smp_sigp to _retry
Quoted 36 lines · Eric Farman
Quoted 17 lines · Nico Boehr
Quoted 9 lines · Janosch Frank
Quoted 7 lines · Eric Farman
A SIGP SENSE is used to determine if a CPU is stopped or operating,
and thus has a vested interest in ensuring it received a CC0 or CC1,
instead of a CC2 (BUSY). But, any order could receive a CC2 response,
and is probably ill-equipped to respond to it.Quoted 6 lines
sigp sense running status doesn't return a cc2, only sigp sense does
afaik.
Looking at the KVM implementation tells me that it's not doing more than
looking at the R bit in the sblk.Quoted 7 lines
From the POP I read _all_ orders may indeed return CC=2: case 1 under
"Conditions precluding Interpretation of the Order Code".That being said, there are a few more users of
smp_sigp(no retry) insmp.c(the test, not the lib).Does it make sense to fix them aswell?
I thought it made sense to do the lib, since other places expect those things to "just work."
But for the tests themselves, I struggle to convince myself with one path over another. The only way KVM returns a CC2 is because of a concurrent STOP/RESTART, which isn't a possibility because of the waiting the lib itself does when invoking the STOP/RESTART. So should the tests be looking for an unexpected CC2? Or just loop when they occur? If the latter, shouldn't the lib itself do that?
Quoted 2 lines
I'm happy to make changes, I just can't decide which it should be. Any opinions?
Before we continue bikeshedding, let's add the cc2 retry. If it never returns cc2 we'll never loop on it but the dead code won't kill us either.