Skip to contentSkip to conversation

s390x: firq: floating interrupt test

PATCH In review David Hildenbrand 3 participants 12 messages Dec 2, 2021 – Dec 6, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover s390x: firq: floating interrupt test

From patch #2:

"
We had a KVM BUG fixed by kernel commit a3e03bc1368c ("KVM: s390: index kvm->arch.idle_mask by vcpu_idx"), whereby a floating interrupt might get stuck forever because a CPU in the wait state would not get woken up.

The issue can be triggered when CPUs are created in a nonlinear fashion, such that the CPU address ("core-id") and the KVM cpu id don't match.

So let's start with a floating interrupt test that will trigger a floating interrupt (via SCLP) to be delivered to a CPU in the wait state. "

v1 -> v2:
- Remove flag logic
- Extend comments
- Minor cleanups
- sclp_clear_busy() before printing to the SCLP console

David Hildenbrand (2):
s390x: make smp_cpu_setup() return 0 on success
s390x: firq: floating interrupt test

 lib/s390x/sclp.c    |  11 ++--
 lib/s390x/sclp.h    |   1 +
 lib/s390x/smp.c     |   1 +
 s390x/Makefile      |   1 +
 s390x/firq.c        | 122 ++++++++++++++++++++++++++++++++++++++++++++
 s390x/unittests.cfg |  10 ++++
 6 files changed, 143 insertions(+), 3 deletions(-)
 create mode 100644 s390x/firq.c
Signature · 1 line
2.31.1
#1 lore raw

PATCH v2 1/2 s390x: make smp_cpu_setup() return 0 on success

Properly return "0" on success so callers can check if the setup was successful.

The return value is yet unused, which is why this wasn't noticed so far.

---
 lib/s390x/smp.c | 1 +
 1 file changed, 1 insertion(+)
lib/s390x/smp.c +1 -0
index da6d32f..b753eab 100644
@@ -212,6 +212,7 @@ int smp_cpu_setup(uint16_t addr, struct psw psw)
 	/* Wait until the cpu has finished setup and started the provided psw */
 	while (lc->restart_new_psw.addr != psw.addr)
 		mb();
+	rc = 0;
 out:
 	spin_unlock(&lock);
 	return rc;
Signature · 1 line
2.31.1
#2 lore raw

PATCH v2 2/2 s390x: firq: floating interrupt test

In reply to David Hildenbrand

We had a KVM BUG fixed by kernel commit a3e03bc1368c ("KVM: s390: index kvm->arch.idle_mask by vcpu_idx"), whereby a floating interrupt might get stuck forever because a CPU in the wait state would not get woken up.

The issue can be triggered when CPUs are created in a nonlinear fashion, such that the CPU address ("core-id") and the KVM cpu id don't match.

So let's start with a floating interrupt test that will trigger a floating interrupt (via SCLP) to be delivered to a CPU in the wait state.

---
 lib/s390x/sclp.c    |  11 ++--
 lib/s390x/sclp.h    |   1 +
 s390x/Makefile      |   1 +
 s390x/firq.c        | 122 ++++++++++++++++++++++++++++++++++++++++++++
 s390x/unittests.cfg |  10 ++++
 5 files changed, 142 insertions(+), 3 deletions(-)
 create mode 100644 s390x/firq.c
lib/s390x/sclp.c +8 -3
index 0272249..33985eb 100644
@@ -60,9 +60,7 @@ void sclp_setup_int(void)
 void sclp_handle_ext(void)
 {
 	ctl_clear_bit(0, CTL0_SERVICE_SIGNAL);
-	spin_lock(&sclp_lock);
-	sclp_busy = false;
-	spin_unlock(&sclp_lock);
+	sclp_clear_busy();
 }
 
 void sclp_wait_busy(void)
@@ -89,6 +87,13 @@ void sclp_mark_busy(void)
 	}
 }
 
+void sclp_clear_busy(void)
+{
+	spin_lock(&sclp_lock);
+	sclp_busy = false;
+	spin_unlock(&sclp_lock);
+}
+
 static void sclp_read_scp_info(ReadInfo *ri, int length)
 {
 	unsigned int commands[] = { SCLP_CMDW_READ_SCP_INFO_FORCED,
lib/s390x/sclp.h +1 -0
index 61e9cf5..fead007 100644
@@ -318,6 +318,7 @@ void sclp_setup_int(void);
 void sclp_handle_ext(void);
 void sclp_wait_busy(void);
 void sclp_mark_busy(void);
+void sclp_clear_busy(void);
 void sclp_console_setup(void);
 void sclp_print(const char *str);
 void sclp_read_info(void);
s390x/Makefile +1 -0
index f95f2e6..1e567c1 100644
@@ -25,6 +25,7 @@ tests += $(TEST_DIR)/uv-host.elf
 tests += $(TEST_DIR)/edat.elf
 tests += $(TEST_DIR)/mvpg-sie.elf
 tests += $(TEST_DIR)/spec_ex-sie.elf
+tests += $(TEST_DIR)/firq.elf
 
 tests_binary = $(patsubst %.elf,%.bin,$(tests))
 ifneq ($(HOST_KEY_DOCUMENT),)
s390x/firq.c new file +122 -0

122 lines not on this page · raw message

s390x/unittests.cfg +10 -0
index 3b454b7..054560c 100644
@@ -112,3 +112,13 @@ file = mvpg-sie.elf
 
 [spec_ex-sie]
 file = spec_ex-sie.elf
+
+[firq-linear-cpu-ids]
+file = firq.elf
+timeout = 20
+extra_params = -smp 1,maxcpus=3 -cpu qemu -device qemu-s390x-cpu,core-id=1 -device qemu-s390x-cpu,core-id=2
+
+[firq-nonlinear-cpu-ids]
+file = firq.elf
+timeout = 20
+extra_params = -smp 1,maxcpus=3 -cpu qemu -device qemu-s390x-cpu,core-id=2 -device qemu-s390x-cpu,core-id=1
Signature · 1 line
2.31.1
Reviewed-by on v2 2/2
#3 lore raw
Quoted 11 lines · On Thu, 2 Dec 2021 13:35:53 +0100 David Hildenbrand <[email protected]> wrote:

We had a KVM BUG fixed by kernel commit a3e03bc1368c ("KVM: s390: index kvm->arch.idle_mask by vcpu_idx"), whereby a floating interrupt might get stuck forever because a CPU in the wait state would not get woken up.

The issue can be triggered when CPUs are created in a nonlinear fashion, such that the CPU address ("core-id") and the KVM cpu id don't match.

So let's start with a floating interrupt test that will trigger a floating interrupt (via SCLP) to be delivered to a CPU in the wait state.

Quoted 208 lines
---
 lib/s390x/sclp.c    |  11 ++--
 lib/s390x/sclp.h    |   1 +
 s390x/Makefile      |   1 +
 s390x/firq.c        | 122 ++++++++++++++++++++++++++++++++++++++++++++
 s390x/unittests.cfg |  10 ++++
 5 files changed, 142 insertions(+), 3 deletions(-)
 create mode 100644 s390x/firq.c
lib/s390x/sclp.c +8 -3
index 0272249..33985eb 100644
@@ -60,9 +60,7 @@ void sclp_setup_int(void)
 void sclp_handle_ext(void)
 {
 	ctl_clear_bit(0, CTL0_SERVICE_SIGNAL);
-	spin_lock(&sclp_lock);
-	sclp_busy = false;
-	spin_unlock(&sclp_lock);
+	sclp_clear_busy();
 }
 
 void sclp_wait_busy(void)
@@ -89,6 +87,13 @@ void sclp_mark_busy(void)
 	}
 }
 
+void sclp_clear_busy(void)
+{
+	spin_lock(&sclp_lock);
+	sclp_busy = false;
+	spin_unlock(&sclp_lock);
+}
+
 static void sclp_read_scp_info(ReadInfo *ri, int length)
 {
 	unsigned int commands[] = { SCLP_CMDW_READ_SCP_INFO_FORCED,
lib/s390x/sclp.h +1 -0
index 61e9cf5..fead007 100644
@@ -318,6 +318,7 @@ void sclp_setup_int(void);
 void sclp_handle_ext(void);
 void sclp_wait_busy(void);
 void sclp_mark_busy(void);
+void sclp_clear_busy(void);
 void sclp_console_setup(void);
 void sclp_print(const char *str);
 void sclp_read_info(void);
s390x/Makefile +1 -0
index f95f2e6..1e567c1 100644
@@ -25,6 +25,7 @@ tests += $(TEST_DIR)/uv-host.elf
 tests += $(TEST_DIR)/edat.elf
 tests += $(TEST_DIR)/mvpg-sie.elf
 tests += $(TEST_DIR)/spec_ex-sie.elf
+tests += $(TEST_DIR)/firq.elf
 
 tests_binary = $(patsubst %.elf,%.bin,$(tests))
 ifneq ($(HOST_KEY_DOCUMENT),)
s390x/firq.c new file +122 -0
new file mode 100644
index 0000000..1f87718
@@ -0,0 +1,122 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Floating interrupt tests.
+ *
+ * Copyright 2021 Red Hat Inc
+ *
+ * Authors:
+ *    David Hildenbrand <[email protected]>
+ */
+#include <libcflat.h>
+#include <asm/asm-offsets.h>
+#include <asm/interrupt.h>
+#include <asm/page.h>
+#include <asm-generic/barrier.h>
+
+#include <sclp.h>
+#include <smp.h>
+#include <alloc_page.h>
+
+static void wait_for_sclp_int(void)
+{
+	/* Enable SCLP interrupts on this CPU only. */
+	ctl_set_bit(0, CTL0_SERVICE_SIGNAL);
+
+	/* Enable external interrupts and go to the wait state. */
+	wait_for_interrupt(PSW_MASK_EXT);
+}
+
+/*
+ * Some KVM versions might mix CPUs when looking for a floating IRQ target,
+ * accidentially detecting a stopped CPU as waiting and resulting in the actually
+ * waiting CPU not getting woken up for the interrupt.
+ */
+static void test_wait_state_delivery(void)
+{
+	struct psw psw;
+	SCCBHeader *h;
+	int ret;
+
+	report_prefix_push("wait state delivery");
+
+	if (smp_query_num_cpus() < 3) {
+		report_skip("need at least 3 CPUs for this test");
+		goto out;
+	}
+
+	if (stap()) {
+		report_skip("need to start on CPU #0");
+		goto out;
+	}
+
+	/*
+	 * We want CPU #2 to be stopped. This should be the case at this
+	 * point, however, we want to sense if it even exists as well.
+	 */
+	ret = smp_cpu_stop(2);
+	if (ret) {
+		report_skip("CPU #2 not found");
+		goto out;
+	}
+
+	/*
+	 * We're going to perform an SCLP service call but expect
+	 * the interrupt on CPU #1 while it is in the wait state.
+	 */
+	sclp_mark_busy();
+
+	/* Start CPU #1 and let it wait for the interrupt. */
+	psw.mask = extract_psw_mask();
+	psw.addr = (unsigned long)wait_for_sclp_int;
+	ret = smp_cpu_setup(1, psw);
+	if (ret) {
+		sclp_clear_busy();
+		report_skip("cpu #1 not found");
+		goto out;
+	}
+
+	/*
+	 * We'd have to jump trough some hoops to sense e.g., via SIGP
+	 * CONDITIONAL EMERGENCY SIGNAL if CPU #1 is already in the
+	 * wait state.
+	 *
+	 * Although not completely reliable, use SIGP SENSE RUNNING STATUS
+	 * until not reported as running -- after all, our SCLP processing
+	 * will take some time as well and smp_cpu_setup() returns when we're
+	 * either already in wait_for_sclp_int() or just about to execute it.
+	 */
+	while(smp_sense_running_status(1));
+
+	h = alloc_page();
+	h->length = 4096;
+	ret = servc(SCLP_CMDW_READ_CPU_INFO, __pa(h));
+	if (ret) {
+		sclp_clear_busy();
+		report_fail("SCLP_CMDW_READ_CPU_INFO failed");
+		goto out_destroy;
+	}
+
+	/*
+	 * Wait until the interrupt gets delivered on CPU #1, marking the
+	 * SCLP requests as done.
+	 */
+	sclp_wait_busy();
+
+	report(true, "sclp interrupt delivered");
+
+out_destroy:
+	free_page(h);
+	smp_cpu_destroy(1);
+out:
+	report_prefix_pop();
+}
+
+int main(void)
+{
+	report_prefix_push("firq");
+
+	test_wait_state_delivery();
+
+	report_prefix_pop();
+	return report_summary();
+}
s390x/unittests.cfg +10 -0
index 3b454b7..054560c 100644
@@ -112,3 +112,13 @@ file = mvpg-sie.elf
 
 [spec_ex-sie]
 file = spec_ex-sie.elf
+
+[firq-linear-cpu-ids]
+file = firq.elf
+timeout = 20
+extra_params = -smp 1,maxcpus=3 -cpu qemu -device qemu-s390x-cpu,core-id=1 -device qemu-s390x-cpu,core-id=2
+
+[firq-nonlinear-cpu-ids]
+file = firq.elf
+timeout = 20
+extra_params = -smp 1,maxcpus=3 -cpu qemu -device qemu-s390x-cpu,core-id=2 -device qemu-s390x-cpu,core-id=1
Reviewed-by on v2 2/2
#4 lore raw
In reply to David Hildenbrand
Quoted 9 lines · David Hildenbrand

We had a KVM BUG fixed by kernel commit a3e03bc1368c ("KVM: s390: index kvm->arch.idle_mask by vcpu_idx"), whereby a floating interrupt might get stuck forever because a CPU in the wait state would not get woken up.

The issue can be triggered when CPUs are created in a nonlinear fashion, such that the CPU address ("core-id") and the KVM cpu id don't match.

So let's start with a floating interrupt test that will trigger a floating interrupt (via SCLP) to be delivered to a CPU in the wait state.

Thank you very much for tackling this! Some remarks below...

Quoted 86 lines
---
  lib/s390x/sclp.c    |  11 ++--
  lib/s390x/sclp.h    |   1 +
  s390x/Makefile      |   1 +
  s390x/firq.c        | 122 ++++++++++++++++++++++++++++++++++++++++++++
  s390x/unittests.cfg |  10 ++++
  5 files changed, 142 insertions(+), 3 deletions(-)

create mode 100644 s390x/firq.c

lib/s390x/sclp.c +8 -3
index 0272249..33985eb 100644
@@ -60,9 +60,7 @@ void sclp_setup_int(void)
  void sclp_handle_ext(void)
  {
  	ctl_clear_bit(0, CTL0_SERVICE_SIGNAL);
-	spin_lock(&sclp_lock);
-	sclp_busy = false;
-	spin_unlock(&sclp_lock);
+	sclp_clear_busy();
  }
 
  void sclp_wait_busy(void)
@@ -89,6 +87,13 @@ void sclp_mark_busy(void)
  	}
  }
 
+void sclp_clear_busy(void)
+{
+	spin_lock(&sclp_lock);
+	sclp_busy = false;
+	spin_unlock(&sclp_lock);
+}
+
  static void sclp_read_scp_info(ReadInfo *ri, int length)
  {
  	unsigned int commands[] = { SCLP_CMDW_READ_SCP_INFO_FORCED,
lib/s390x/sclp.h +1 -0
index 61e9cf5..fead007 100644
@@ -318,6 +318,7 @@ void sclp_setup_int(void);
  void sclp_handle_ext(void);
  void sclp_wait_busy(void);
  void sclp_mark_busy(void);
+void sclp_clear_busy(void);
  void sclp_console_setup(void);
  void sclp_print(const char *str);
  void sclp_read_info(void);
s390x/Makefile +1 -0
index f95f2e6..1e567c1 100644
@@ -25,6 +25,7 @@ tests += $(TEST_DIR)/uv-host.elf
  tests += $(TEST_DIR)/edat.elf
  tests += $(TEST_DIR)/mvpg-sie.elf
  tests += $(TEST_DIR)/spec_ex-sie.elf
+tests += $(TEST_DIR)/firq.elf
 
  tests_binary = $(patsubst %.elf,%.bin,$(tests))
  ifneq ($(HOST_KEY_DOCUMENT),)
s390x/firq.c new file +17 -0
new file mode 100644
index 0000000..1f87718
@@ -0,0 +1,122 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Floating interrupt tests.
+ *
+ * Copyright 2021 Red Hat Inc
+ *
+ * Authors:
+ *    David Hildenbrand <[email protected]>
+ */
+#include <libcflat.h>
+#include <asm/asm-offsets.h>
+#include <asm/interrupt.h>
+#include <asm/page.h>
+#include <asm-generic/barrier.h>
+
+#include <sclp.h>
+#include <smp.h>

s390x/firq.c line 27

Quoted 10 lines
s390x/firq.c new file +10 -0
+#include <alloc_page.h>
+
+static void wait_for_sclp_int(void)
+{
+	/* Enable SCLP interrupts on this CPU only. */
+	ctl_set_bit(0, CTL0_SERVICE_SIGNAL);
+
+	/* Enable external interrupts and go to the wait state. */
+	wait_for_interrupt(PSW_MASK_EXT);
+}

What happens if the CPU got an interrupt? Should there be a "while (true)" at the end of the function to avoid that the CPU ends up crashing at the end of the function?

Quoted 12 lines
s390x/firq.c new file +12 -0
+/*
+ * Some KVM versions might mix CPUs when looking for a floating IRQ target,
+ * accidentially detecting a stopped CPU as waiting and resulting in the actually
+ * waiting CPU not getting woken up for the interrupt.
+ */
+static void test_wait_state_delivery(void)
+{
+	struct psw psw;
+	SCCBHeader *h;
+	int ret;
+
+	report_prefix_push("wait state delivery");

s390x/firq.c line 50

Quoted 10 lines
s390x/firq.c new file +10 -0
+
+	if (smp_query_num_cpus() < 3) {
+		report_skip("need at least 3 CPUs for this test");
+		goto out;
+	}
+
+	if (stap()) {
+		report_skip("need to start on CPU #0");
+		goto out;
+	}

I think I'd rather turn this into an assert() instead ... no strong opinion about it, though.

s390x/firq.c line 58

Quoted 8 lines
s390x/firq.c new file +8 -0
+
+	/*
+	 * We want CPU #2 to be stopped. This should be the case at this
+	 * point, however, we want to sense if it even exists as well.
+	 */
+	ret = smp_cpu_stop(2);
+	if (ret) {
+		report_skip("CPU #2 not found");

Since you already queried for the availablity of at least 3 CPUs above, I think you could turn this into a report_fail() instead?

Quoted 44 lines
s390x/firq.c new file +44 -0
+		goto out;
+	}
+
+	/*
+	 * We're going to perform an SCLP service call but expect
+	 * the interrupt on CPU #1 while it is in the wait state.
+	 */
+	sclp_mark_busy();
+
+	/* Start CPU #1 and let it wait for the interrupt. */
+	psw.mask = extract_psw_mask();
+	psw.addr = (unsigned long)wait_for_sclp_int;
+	ret = smp_cpu_setup(1, psw);
+	if (ret) {
+		sclp_clear_busy();
+		report_skip("cpu #1 not found");
+		goto out;
+	}
+
+	/*
+	 * We'd have to jump trough some hoops to sense e.g., via SIGP
+	 * CONDITIONAL EMERGENCY SIGNAL if CPU #1 is already in the
+	 * wait state.
+	 *
+	 * Although not completely reliable, use SIGP SENSE RUNNING STATUS
+	 * until not reported as running -- after all, our SCLP processing
+	 * will take some time as well and smp_cpu_setup() returns when we're
+	 * either already in wait_for_sclp_int() or just about to execute it.
+	 */
+	while(smp_sense_running_status(1));
+
+	h = alloc_page();
+	h->length = 4096;
+	ret = servc(SCLP_CMDW_READ_CPU_INFO, __pa(h));
+	if (ret) {
+		sclp_clear_busy();
+		report_fail("SCLP_CMDW_READ_CPU_INFO failed");
+		goto out_destroy;
+	}
+
+	/*
+	 * Wait until the interrupt gets delivered on CPU #1, marking the
+	 * SCLP requests as done.
+	 */

s390x/firq.c line 112

Quoted 10 lines
s390x/firq.c new file +10 -0
+	sclp_wait_busy();
+
+	report(true, "sclp interrupt delivered");
+
+out_destroy:
+	free_page(h);
+	smp_cpu_destroy(1);
+out:
+	report_prefix_pop();
+}

Anyway, code looks fine for me, either with my comments addressed or not:

#5 lore raw
Quoted 103 lines · On Fri, 3 Dec 2021 11:55:31 +0100 Thomas Huth <[email protected]> wrote:
Quoted 9 lines · David Hildenbrand

We had a KVM BUG fixed by kernel commit a3e03bc1368c ("KVM: s390: index kvm->arch.idle_mask by vcpu_idx"), whereby a floating interrupt might get stuck forever because a CPU in the wait state would not get woken up.

The issue can be triggered when CPUs are created in a nonlinear fashion, such that the CPU address ("core-id") and the KVM cpu id don't match.

So let's start with a floating interrupt test that will trigger a floating interrupt (via SCLP) to be delivered to a CPU in the wait state.

Thank you very much for tackling this! Some remarks below...

Quoted 90 lines
---
  lib/s390x/sclp.c    |  11 ++--
  lib/s390x/sclp.h    |   1 +
  s390x/Makefile      |   1 +
  s390x/firq.c        | 122 ++++++++++++++++++++++++++++++++++++++++++++
  s390x/unittests.cfg |  10 ++++
  5 files changed, 142 insertions(+), 3 deletions(-)

create mode 100644 s390x/firq.c

lib/s390x/sclp.c +8 -3
index 0272249..33985eb 100644
@@ -60,9 +60,7 @@ void sclp_setup_int(void)
  void sclp_handle_ext(void)
  {
  	ctl_clear_bit(0, CTL0_SERVICE_SIGNAL);
-	spin_lock(&sclp_lock);
-	sclp_busy = false;
-	spin_unlock(&sclp_lock);
+	sclp_clear_busy();
  }
 
  void sclp_wait_busy(void)
@@ -89,6 +87,13 @@ void sclp_mark_busy(void)
  	}
  }
 
+void sclp_clear_busy(void)
+{
+	spin_lock(&sclp_lock);
+	sclp_busy = false;
+	spin_unlock(&sclp_lock);
+}
+
  static void sclp_read_scp_info(ReadInfo *ri, int length)
  {
  	unsigned int commands[] = { SCLP_CMDW_READ_SCP_INFO_FORCED,
lib/s390x/sclp.h +1 -0
index 61e9cf5..fead007 100644
@@ -318,6 +318,7 @@ void sclp_setup_int(void);
  void sclp_handle_ext(void);
  void sclp_wait_busy(void);
  void sclp_mark_busy(void);
+void sclp_clear_busy(void);
  void sclp_console_setup(void);
  void sclp_print(const char *str);
  void sclp_read_info(void);
s390x/Makefile +1 -0
index f95f2e6..1e567c1 100644
@@ -25,6 +25,7 @@ tests += $(TEST_DIR)/uv-host.elf
  tests += $(TEST_DIR)/edat.elf
  tests += $(TEST_DIR)/mvpg-sie.elf
  tests += $(TEST_DIR)/spec_ex-sie.elf
+tests += $(TEST_DIR)/firq.elf
 
  tests_binary = $(patsubst %.elf,%.bin,$(tests))
  ifneq ($(HOST_KEY_DOCUMENT),)
s390x/firq.c new file +21 -0
new file mode 100644
index 0000000..1f87718
@@ -0,0 +1,122 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Floating interrupt tests.
+ *
+ * Copyright 2021 Red Hat Inc
+ *
+ * Authors:
+ *    David Hildenbrand <[email protected]>
+ */
+#include <libcflat.h>
+#include <asm/asm-offsets.h>
+#include <asm/interrupt.h>
+#include <asm/page.h>
+#include <asm-generic/barrier.h>
+
+#include <sclp.h>
+#include <smp.h>
+#include <alloc_page.h>
+
+static void wait_for_sclp_int(void)
+{
Quoted 8 lines

s390x/firq.c line 27

Quoted 6 lines
s390x/firq.c new file +6 -0
+	/* Enable SCLP interrupts on this CPU only. */
+	ctl_set_bit(0, CTL0_SERVICE_SIGNAL);
+
+	/* Enable external interrupts and go to the wait state. */
+	wait_for_interrupt(PSW_MASK_EXT);
+}  

What happens if the CPU got an interrupt? Should there be a "while (true)"

it should not get any interrupts, but if it does anyway...

Quoted 2 lines

at the end of the function to avoid that the CPU ends up crashing at the end of the function?

... we have this in smp_cpu_setup_state, after the call to the actual function body:

/* If the function returns, just loop here */
0: j 0

so if the function returns, it will hang in there anyway

Quoted 18 lines
Quoted 18 lines
s390x/firq.c new file +17 -0
+/*
+ * Some KVM versions might mix CPUs when looking for a floating IRQ target,
+ * accidentially detecting a stopped CPU as waiting and resulting in the actually
+ * waiting CPU not getting woken up for the interrupt.
+ */
+static void test_wait_state_delivery(void)
+{
+	struct psw psw;
+	SCCBHeader *h;
+	int ret;
+
+	report_prefix_push("wait state delivery");
+
+	if (smp_query_num_cpus() < 3) {
+		report_skip("need at least 3 CPUs for this test");
+		goto out;
+	}
Quoted 8 lines

s390x/firq.c line 50

Quoted 5 lines
s390x/firq.c new file +5 -0
+
+	if (stap()) {
+		report_skip("need to start on CPU #0");
+		goto out;
+	}  

I think I'd rather turn this into an assert() instead ... no strong opinion about it, though.

I agree, including the part about no strong opinions (which is why I did not comment on it before)

Quoted 12 lines

s390x/firq.c line 58

Quoted 9 lines
s390x/firq.c new file +8 -0
+
+	/*
+	 * We want CPU #2 to be stopped. This should be the case at this
+	 * point, however, we want to sense if it even exists as well.
+	 */
+	ret = smp_cpu_stop(2);
+	if (ret) {
+		report_skip("CPU #2 not found");  

Since you already queried for the availablity of at least 3 CPUs above, I think you could turn this into a report_fail() instead?

either that or an assert, but again, no strong opinions

Quoted 60 lines
Quoted 45 lines
s390x/firq.c new file +44 -0
+		goto out;
+	}
+
+	/*
+	 * We're going to perform an SCLP service call but expect
+	 * the interrupt on CPU #1 while it is in the wait state.
+	 */
+	sclp_mark_busy();
+
+	/* Start CPU #1 and let it wait for the interrupt. */
+	psw.mask = extract_psw_mask();
+	psw.addr = (unsigned long)wait_for_sclp_int;
+	ret = smp_cpu_setup(1, psw);
+	if (ret) {
+		sclp_clear_busy();
+		report_skip("cpu #1 not found");
+		goto out;
+	}
+
+	/*
+	 * We'd have to jump trough some hoops to sense e.g., via SIGP
+	 * CONDITIONAL EMERGENCY SIGNAL if CPU #1 is already in the
+	 * wait state.
+	 *
+	 * Although not completely reliable, use SIGP SENSE RUNNING STATUS
+	 * until not reported as running -- after all, our SCLP processing
+	 * will take some time as well and smp_cpu_setup() returns when we're
+	 * either already in wait_for_sclp_int() or just about to execute it.
+	 */
+	while(smp_sense_running_status(1));
+
+	h = alloc_page();
+	h->length = 4096;
+	ret = servc(SCLP_CMDW_READ_CPU_INFO, __pa(h));
+	if (ret) {
+		sclp_clear_busy();
+		report_fail("SCLP_CMDW_READ_CPU_INFO failed");
+		goto out_destroy;
+	}
+
+	/*
+	 * Wait until the interrupt gets delivered on CPU #1, marking the
+	 * SCLP requests as done.
+	 */

s390x/firq.c line 112

Quoted 10 lines
s390x/firq.c new file +10 -0
+	sclp_wait_busy();
+
+	report(true, "sclp interrupt delivered");
+
+out_destroy:
+	free_page(h);
+	smp_cpu_destroy(1);
+out:
+	report_prefix_pop();
+}  

Anyway, code looks fine for me, either with my comments addressed or not:

#6 lore raw
Quoted 120 lines · Claudio Imbrenda
Quoted 103 lines · On Fri, 3 Dec 2021 11:55:31 +0100 Thomas Huth <[email protected]> wrote:
Quoted 9 lines · David Hildenbrand

We had a KVM BUG fixed by kernel commit a3e03bc1368c ("KVM: s390: index kvm->arch.idle_mask by vcpu_idx"), whereby a floating interrupt might get stuck forever because a CPU in the wait state would not get woken up.

The issue can be triggered when CPUs are created in a nonlinear fashion, such that the CPU address ("core-id") and the KVM cpu id don't match.

So let's start with a floating interrupt test that will trigger a floating interrupt (via SCLP) to be delivered to a CPU in the wait state.

Thank you very much for tackling this! Some remarks below...

Quoted 90 lines
---
   lib/s390x/sclp.c    |  11 ++--
   lib/s390x/sclp.h    |   1 +
   s390x/Makefile      |   1 +
   s390x/firq.c        | 122 ++++++++++++++++++++++++++++++++++++++++++++
   s390x/unittests.cfg |  10 ++++
   5 files changed, 142 insertions(+), 3 deletions(-)

create mode 100644 s390x/firq.c

lib/s390x/sclp.c +8 -3
index 0272249..33985eb 100644
@@ -60,9 +60,7 @@ void sclp_setup_int(void)
   void sclp_handle_ext(void)
   {
   	ctl_clear_bit(0, CTL0_SERVICE_SIGNAL);
-	spin_lock(&sclp_lock);
-	sclp_busy = false;
-	spin_unlock(&sclp_lock);
+	sclp_clear_busy();
   }
 
   void sclp_wait_busy(void)
@@ -89,6 +87,13 @@ void sclp_mark_busy(void)
   	}
   }
 
+void sclp_clear_busy(void)
+{
+	spin_lock(&sclp_lock);
+	sclp_busy = false;
+	spin_unlock(&sclp_lock);
+}
+
   static void sclp_read_scp_info(ReadInfo *ri, int length)
   {
   	unsigned int commands[] = { SCLP_CMDW_READ_SCP_INFO_FORCED,
lib/s390x/sclp.h +1 -0
index 61e9cf5..fead007 100644
@@ -318,6 +318,7 @@ void sclp_setup_int(void);
   void sclp_handle_ext(void);
   void sclp_wait_busy(void);
   void sclp_mark_busy(void);
+void sclp_clear_busy(void);
   void sclp_console_setup(void);
   void sclp_print(const char *str);
   void sclp_read_info(void);
s390x/Makefile +1 -0
index f95f2e6..1e567c1 100644
@@ -25,6 +25,7 @@ tests += $(TEST_DIR)/uv-host.elf
   tests += $(TEST_DIR)/edat.elf
   tests += $(TEST_DIR)/mvpg-sie.elf
   tests += $(TEST_DIR)/spec_ex-sie.elf
+tests += $(TEST_DIR)/firq.elf
 
   tests_binary = $(patsubst %.elf,%.bin,$(tests))
   ifneq ($(HOST_KEY_DOCUMENT),)
s390x/firq.c new file +21 -0
new file mode 100644
index 0000000..1f87718
@@ -0,0 +1,122 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Floating interrupt tests.
+ *
+ * Copyright 2021 Red Hat Inc
+ *
+ * Authors:
+ *    David Hildenbrand <[email protected]>
+ */
+#include <libcflat.h>
+#include <asm/asm-offsets.h>
+#include <asm/interrupt.h>
+#include <asm/page.h>
+#include <asm-generic/barrier.h>
+
+#include <sclp.h>
+#include <smp.h>
+#include <alloc_page.h>
+
+static void wait_for_sclp_int(void)
+{
Quoted 8 lines

s390x/firq.c line 27

Quoted 6 lines
s390x/firq.c new file +6 -0
+	/* Enable SCLP interrupts on this CPU only. */
+	ctl_set_bit(0, CTL0_SERVICE_SIGNAL);
+
+	/* Enable external interrupts and go to the wait state. */
+	wait_for_interrupt(PSW_MASK_EXT);
+}

What happens if the CPU got an interrupt? Should there be a "while (true)"

it should not get any interrupts, but if it does anyway...

Quoted 2 lines

at the end of the function to avoid that the CPU ends up crashing at the end of the function?

Quoted 7 lines

... we have this in smp_cpu_setup_state, after the call to the actual function body:

/* If the function returns, just loop here */
0: j 0

so if the function returns, it will hang in there anyway

Ah, great, so we're fine indeed!

Thomas

#7 lore raw
In reply to Claudio Imbrenda
Quoted 14 lines
Quoted 13 lines

s390x/firq.c line 50

Quoted 10 lines
s390x/firq.c new file +9 -0
+	if (smp_query_num_cpus() < 3) {
+		report_skip("need at least 3 CPUs for this test");
+		goto out;
+	}
+
+	if (stap()) {
+		report_skip("need to start on CPU #0");
+		goto out;
+	}  

I think I'd rather turn this into an assert() instead ... no strong opinion about it, though.

Quoted 2 lines

I agree, including the part about no strong opinions (which is why I did not comment on it before)

Would it be the case on any system we might end up running, even under LPAR ... and whoever could run these tests ?

Quoted 14 lines
Quoted 13 lines

s390x/firq.c line 58

Quoted 10 lines
s390x/firq.c new file +8 -0
+
+	/*
+	 * We want CPU #2 to be stopped. This should be the case at this
+	 * point, however, we want to sense if it even exists as well.
+	 */
+	ret = smp_cpu_stop(2);
+	if (ret) {
+		report_skip("CPU #2 not found");  

Since you already queried for the availablity of at least 3 CPUs above, I think you could turn this into a report_fail() instead?

Quoted 2 lines

either that or an assert, but again, no strong opinions

Just because there are >= 3 CPUs doesn't imply that CPU #2 is around.

What we could remove is the "if (smp_query_num_cpus() < 3) {" check, though!

Thanks!

Signature · 3 lines
Thanks,

David / dhildenb
#8 lore raw
Quoted 14 lines · David Hildenbrand
Quoted 14 lines
Quoted 13 lines

s390x/firq.c line 50

Quoted 10 lines
s390x/firq.c new file +9 -0
+	if (smp_query_num_cpus() < 3) {
+		report_skip("need at least 3 CPUs for this test");
+		goto out;
+	}
+
+	if (stap()) {
+		report_skip("need to start on CPU #0");
+		goto out;
+	}

I think I'd rather turn this into an assert() instead ... no strong opinion about it, though.

Quoted 5 lines
Quoted 2 lines

I agree, including the part about no strong opinions (which is why I did not comment on it before)

Would it be the case on any system we might end up running, even under LPAR ... and whoever could run these tests ?

Well, ok, since it likely doesn't happen in real life anyway, simply keep the report_skip().

Quoted 14 lines
Quoted 14 lines
Quoted 13 lines

s390x/firq.c line 58

Quoted 10 lines
s390x/firq.c new file +8 -0
+
+	/*
+	 * We want CPU #2 to be stopped. This should be the case at this
+	 * point, however, we want to sense if it even exists as well.
+	 */
+	ret = smp_cpu_stop(2);
+	if (ret) {
+		report_skip("CPU #2 not found");

Since you already queried for the availablity of at least 3 CPUs above, I think you could turn this into a report_fail() instead?

Quoted 4 lines
Quoted 2 lines

either that or an assert, but again, no strong opinions

Just because there are >= 3 CPUs doesn't imply that CPU #2 is around.

Ok, fair point. But if #2 is not around, it means that the test has been run in the wrong way by the user... I wonder what's better in that case - to skip this test or to go out with a bang. Skipping the test has the advantage of looking a little bit more "polite", but it has the disadvantage that it might get lost in automation, e.g. if somebody enabled the test in their CI, but did something wrong in the settings, they might not notice that the test is not run at all...

Quoted 1 line

What we could remove is the "if (smp_query_num_cpus() < 3) {" check, though!

Yes, that seems to be redundant, indeed.

Thomas

#9 lore raw
Quoted 19 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines

s390x/firq.c line 58

Quoted 10 lines
s390x/firq.c new file +8 -0
+
+	/*
+	 * We want CPU #2 to be stopped. This should be the case at this
+	 * point, however, we want to sense if it even exists as well.
+	 */
+	ret = smp_cpu_stop(2);
+	if (ret) {
+		report_skip("CPU #2 not found");

Since you already queried for the availablity of at least 3 CPUs above, I think you could turn this into a report_fail() instead?

Quoted 4 lines
Quoted 2 lines

either that or an assert, but again, no strong opinions

Just because there are >= 3 CPUs doesn't imply that CPU #2 is around.

Quoted 7 lines

Ok, fair point. But if #2 is not around, it means that the test has been run in the wrong way by the user... I wonder what's better in that case - to skip this test or to go out with a bang. Skipping the test has the advantage of looking a little bit more "polite", but it has the disadvantage that it might get lost in automation, e.g. if somebody enabled the test in their CI, but did something wrong in the settings, they might not notice that the test is not run at all...

I sticked to what we have in s390x/smp.c, where we fail if we only have a single CPU.

But I don't particularly care (and have to move on doing other stuff), so I'll do whatever maintainers want and resend :)

Signature · 3 lines
Thanks,

David / dhildenb
#10 lore raw
Quoted 27 lines · On Mon, 6 Dec 2021 09:15:00 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 19 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines

s390x/firq.c line 58

Quoted 10 lines
s390x/firq.c new file +8 -0
+
+	/*
+	 * We want CPU #2 to be stopped. This should be the case at this
+	 * point, however, we want to sense if it even exists as well.
+	 */
+	ret = smp_cpu_stop(2);
+	if (ret) {
+		report_skip("CPU #2 not found");  

Since you already queried for the availablity of at least 3 CPUs above, I think you could turn this into a report_fail() instead?

Quoted 4 lines
Quoted 2 lines

either that or an assert, but again, no strong opinions

Just because there are >= 3 CPUs doesn't imply that CPU #2 is around.

Quoted 7 lines

Ok, fair point. But if #2 is not around, it means that the test has been run in the wrong way by the user... I wonder what's better in that case - to skip this test or to go out with a bang. Skipping the test has the advantage of looking a little bit more "polite", but it has the disadvantage that it might get lost in automation, e.g. if somebody enabled the test in their CI, but did something wrong in the settings, they might not notice that the test is not run at all...

Quoted 6 lines

I sticked to what we have in s390x/smp.c, where we fail if we only have a single CPU.

But I don't particularly care (and have to move on doing other stuff), so I'll do whatever maintainers want and resend :)

a better solution for number != ID is needed (aka: I'll try to fix it when I have the time), for now it works, so leave it as it is.

#11 lore raw
In reply to David Hildenbrand
Quoted 7 lines · On Thu, 2 Dec 2021 13:35:51 +0100 David Hildenbrand <[email protected]> wrote:

From patch #2:

"
We had a KVM BUG fixed by kernel commit a3e03bc1368c ("KVM: s390: index kvm->arch.idle_mask by vcpu_idx"), whereby a floating interrupt might get stuck forever because a CPU in the wait state would not get woken up.

Quoted 6 lines

The issue can be triggered when CPUs are created in a nonlinear fashion, such that the CPU address ("core-id") and the KVM cpu id don't match.

So let's start with a floating interrupt test that will trigger a floating interrupt (via SCLP) to be delivered to a CPU in the wait state. "

ok I'll try picking this series

Quoted 20 lines

v1 -> v2:
- Remove flag logic
- Extend comments
- Minor cleanups
- sclp_clear_busy() before printing to the SCLP console

David Hildenbrand (2):
s390x: make smp_cpu_setup() return 0 on success
s390x: firq: floating interrupt test

 lib/s390x/sclp.c    |  11 ++--
 lib/s390x/sclp.h    |   1 +
 lib/s390x/smp.c     |   1 +
 s390x/Makefile      |   1 +
 s390x/firq.c        | 122 ++++++++++++++++++++++++++++++++++++++++++++
 s390x/unittests.cfg |  10 ++++
 6 files changed, 143 insertions(+), 3 deletions(-)
 create mode 100644 s390x/firq.c