Skip to contentSkip to conversation

s390x: Workaround smp stop and store status race

PATCH In review Janosch Frank 4 participants 7 messages Feb 9, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 s390x: Workaround smp stop and store status race

KVM and QEMU handle a SIGP stop and store status in two steps:
1) Stop the CPU by injecting a stop request
2) Store when the CPU has left SIE because of the stop request

The problem is that the SIGP order is already considered completed by KVM/QEMU when step 1 has been performed and not once both have completed. In addition we currently don't implement the busy CC so a kernel has no way of knowing that the store has finished other than checking the location for the store.

This workaround is based on the fact that for a new SIE entry (via the added smp restart) a stop with the store status has to be finished first.

Correct handling of this in KVM/QEMU will need some thought and time.

---
 s390x/smp.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
s390x/smp.c +4 -1
index b0ece491..32f284a2 100644
@@ -102,12 +102,15 @@ static void test_stop_store_status(void)
 	lc->grs_sa[15] = 0;
 	smp_cpu_stop_store_status(1);
 	mb();
+	report(smp_cpu_stopped(1), "cpu stopped");
+	/* For the cpu to be started it should have finished storing */
+	smp_cpu_restart(1);
 	report(lc->prefix_sa == (uint32_t)(uintptr_t)cpu->lowcore, "prefix");
 	report(lc->grs_sa[15], "stack");
-	report(smp_cpu_stopped(1), "cpu stopped");
 	report_prefix_pop();
 
 	report_prefix_push("stopped");
+	smp_cpu_stop(1);
 	lc->prefix_sa = 0;
 	lc->grs_sa[15] = 0;
 	smp_cpu_stop_store_status(1);
Signature · 1 line
2.25.1
Tested-by Reviewed-by on v1 1/1
#1 lore raw
Quoted 32 lines · Janosch Frank

KVM and QEMU handle a SIGP stop and store status in two steps:
1) Stop the CPU by injecting a stop request
2) Store when the CPU has left SIE because of the stop request

The problem is that the SIGP order is already considered completed by KVM/QEMU when step 1 has been performed and not once both have completed. In addition we currently don't implement the busy CC so a kernel has no way of knowing that the store has finished other than checking the location for the store.

This workaround is based on the fact that for a new SIE entry (via the added smp restart) a stop with the store status has to be finished first.

Correct handling of this in KVM/QEMU will need some thought and time.

---
  s390x/smp.c | 5 ++++-
  1 file changed, 4 insertions(+), 1 deletion(-)
s390x/smp.c +3 -0
index b0ece491..32f284a2 100644
@@ -102,12 +102,15 @@ static void test_stop_store_status(void)
  	lc->grs_sa[15] = 0;
  	smp_cpu_stop_store_status(1);
  	mb();
+	report(smp_cpu_stopped(1), "cpu stopped");
+	/* For the cpu to be started it should have finished storing */
+	smp_cpu_restart(1);

s390x/smp.c static void test_stop_store_status(void) · line 116

Quoted 10 lines
s390x/smp.c +1 -1
  	report(lc->prefix_sa == (uint32_t)(uintptr_t)cpu->lowcore, "prefix");
  	report(lc->grs_sa[15], "stack");
-	report(smp_cpu_stopped(1), "cpu stopped");
  	report_prefix_pop();
 
  	report_prefix_push("stopped");
+	smp_cpu_stop(1);
  	lc->prefix_sa = 0;
  	lc->grs_sa[15] = 0;
  	smp_cpu_stop_store_status(1);

Thanks, this fixes the flaky test for me!

#2 lore raw
In reply to Janosch Frank
Quoted 10 lines · On Tue, 9 Feb 2021 09:15:54 -0500 Janosch Frank <[email protected]> wrote:

KVM and QEMU handle a SIGP stop and store status in two steps:
1) Stop the CPU by injecting a stop request
2) Store when the CPU has left SIE because of the stop request

The problem is that the SIGP order is already considered completed by KVM/QEMU when step 1 has been performed and not once both have completed. In addition we currently don't implement the busy CC so a kernel has no way of knowing that the store has finished other than checking the location for the store.

Quoted 5 lines

This workaround is based on the fact that for a new SIE entry (via the added smp restart) a stop with the store status has to be finished first.

Correct handling of this in KVM/QEMU will need some thought and time.

do I understand correctly that you are here "fixing" the test by not triggering the KVM bug? Shouldn't we try to trigger as many bugs as possible instead?

Quoted 27 lines
---
 s390x/smp.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
s390x/smp.c +4 -1
index b0ece491..32f284a2 100644
@@ -102,12 +102,15 @@ static void test_stop_store_status(void)
 	lc->grs_sa[15] = 0;
 	smp_cpu_stop_store_status(1);
 	mb();
+	report(smp_cpu_stopped(1), "cpu stopped");
+	/* For the cpu to be started it should have finished storing
 */
+	smp_cpu_restart(1);
 	report(lc->prefix_sa == (uint32_t)(uintptr_t)cpu->lowcore,
 "prefix"); report(lc->grs_sa[15], "stack");
-	report(smp_cpu_stopped(1), "cpu stopped");
 	report_prefix_pop();
 
 	report_prefix_push("stopped");
+	smp_cpu_stop(1);
 	lc->prefix_sa = 0;
 	lc->grs_sa[15] = 0;

smp_cpu_stop_store_status(1);

#3 lore raw
Quoted 17 lines · Claudio Imbrenda
Quoted 14 lines · On Tue, 9 Feb 2021 09:15:54 -0500 Janosch Frank <[email protected]> wrote:

KVM and QEMU handle a SIGP stop and store status in two steps:
1) Stop the CPU by injecting a stop request
2) Store when the CPU has left SIE because of the stop request

The problem is that the SIGP order is already considered completed by KVM/QEMU when step 1 has been performed and not once both have completed. In addition we currently don't implement the busy CC so a kernel has no way of knowing that the store has finished other than checking the location for the store.

This workaround is based on the fact that for a new SIE entry (via the added smp restart) a stop with the store status has to be finished first.

Quoted 5 lines
Quoted 1 line

Correct handling of this in KVM/QEMU will need some thought and time.

do I understand correctly that you are here "fixing" the test by not triggering the KVM bug? Shouldn't we try to trigger as many bugs as possible instead?

This is not a bug, it's missing code :-)

We trigger a higher number of bugs by running tests and this workaround does exactly that by letting Thomas use the smp test in the CI again.

Quoted 29 lines
Quoted 28 lines
---
 s390x/smp.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
s390x/smp.c +4 -1
index b0ece491..32f284a2 100644
@@ -102,12 +102,15 @@ static void test_stop_store_status(void)
 	lc->grs_sa[15] = 0;
 	smp_cpu_stop_store_status(1);
 	mb();
+	report(smp_cpu_stopped(1), "cpu stopped");
+	/* For the cpu to be started it should have finished storing
 */
+	smp_cpu_restart(1);
 	report(lc->prefix_sa == (uint32_t)(uintptr_t)cpu->lowcore,
 "prefix"); report(lc->grs_sa[15], "stack");
-	report(smp_cpu_stopped(1), "cpu stopped");
 	report_prefix_pop();
 
 	report_prefix_push("stopped");
+	smp_cpu_stop(1);
 	lc->prefix_sa = 0;
 	lc->grs_sa[15] = 0;

smp_cpu_stop_store_status(1);

#4 lore raw
Quoted 20 lines · Janosch Frank
Quoted 19 lines · Claudio Imbrenda
Quoted 15 lines · On Tue, 9 Feb 2021 09:15:54 -0500 Janosch Frank <[email protected]> wrote:

KVM and QEMU handle a SIGP stop and store status in two steps:
1) Stop the CPU by injecting a stop request
2) Store when the CPU has left SIE because of the stop request

The problem is that the SIGP order is already considered completed by KVM/QEMU when step 1 has been performed and not once both have completed. In addition we currently don't implement the busy CC so a kernel has no way of knowing that the store has finished other than checking the location for the store.

This workaround is based on the fact that for a new SIE entry (via the added smp restart) a stop with the store status has to be finished first.

Correct handling of this in KVM/QEMU will need some thought and time.

Quoted 8 lines
Quoted 3 lines

do I understand correctly that you are here "fixing" the test by not triggering the KVM bug? Shouldn't we try to trigger as many bugs as possible instead?

This is not a bug, it's missing code :-)

We trigger a higher number of bugs by running tests and this workaround does exactly that by letting Thomas use the smp test in the CI again.

Alternatively, we could use report_xfail here to make the test pass, but still have the problem reported so that we do not forget to fix it later.

Thomas

#5 lore raw
Quoted 27 lines · Thomas Huth
Quoted 26 lines · Janosch Frank
Quoted 17 lines · Claudio Imbrenda
Quoted 14 lines · On Tue, 9 Feb 2021 09:15:54 -0500 Janosch Frank <[email protected]> wrote:

KVM and QEMU handle a SIGP stop and store status in two steps:
1) Stop the CPU by injecting a stop request
2) Store when the CPU has left SIE because of the stop request

The problem is that the SIGP order is already considered completed by KVM/QEMU when step 1 has been performed and not once both have completed. In addition we currently don't implement the busy CC so a kernel has no way of knowing that the store has finished other than checking the location for the store.

This workaround is based on the fact that for a new SIE entry (via the added smp restart) a stop with the store status has to be finished first.

Quoted 5 lines
Quoted 1 line

Correct handling of this in KVM/QEMU will need some thought and time.

do I understand correctly that you are here "fixing" the test by not triggering the KVM bug? Shouldn't we try to trigger as many bugs as possible instead?

This is not a bug, it's missing code :-)

Quoted 8 lines
Quoted 2 lines

We trigger a higher number of bugs by running tests and this workaround does exactly that by letting Thomas use the smp test in the CI again.

Alternatively, we could use report_xfail here to make the test pass, but still have the problem reported so that we do not forget to fix it later.

Thomas

I have no strong opinion on that although I'll need to have a look if our CI flags XPASS/XFAIL as fails.

#6 lore raw
In reply to Janosch Frank
Quoted 7 lines · Janosch Frank

KVM and QEMU handle a SIGP stop and store status in two steps:
1) Stop the CPU by injecting a stop request
2) Store when the CPU has left SIE because of the stop request

The problem is that the SIGP order is already considered completed by KVM/QEMU when step 1 has been performed and not once both have completed. In addition we currently don't implement the busy CC so a

QEMU remembers that stop-and-store-status is active in "cpu->env.sigp_order" and rejects other orders until completed with SIGP_CC_BUSY. See handle_sigp_single_dst().

The "issue" is that the kernel does not know about that. The kernel will continue handling
- SIGP_SENSE
- SIGP_EXTERNAL_CALL
- SIGP_EMERGENCY_SIGNAL
- SIGP_COND_EMERGENCY_SIGNAL
- SIGP_SENSE_RUNNING
itself and not go to user space where that information is present. And for some of these orders we really don't want to go to user space.

I remember that at least SIGP_SENSE_RUNNING doesn't give any guarantees, so that one might be correct.

I remember that it was a little unclear if all of these orders actually have to wait for other orders to finish (IOW: return SIGP_CC_BUSY). Especially with SIGPIF even the hardware has no idea if we are emulating a SIGP STOP .* right now, how should it know? We would have to disable the facility, and that's most probably not what the HW/facility was designed for.

Yeah, it's complicated. We would have to let QEMU "set" and "clear" a SIGP busy status in the kernel. Then we could at least let SIGP SENSE be correct and fast (I think that's the only critical one to get right).

Signature · 3 lines
Thanks,

David / dhildenb