Skip to contentSkip to conversation

S390x: CPU Topology Information

PATCH Superseded Pierre Morel 3 participants 11 messages Feb 2, 2023 – Feb 15, 2023

Subsystems (2)

v6 of 21 Superseded Latest is v21 →

lore
Share & tools
lore
#0 lore raw

v6 cover S390x: CPU Topology Information

Hi,

new version of the kvm-unit-test s390x CPU topology series.

1. what is done
---------------

- First part is checking PTF errors, for KVM and LPAR

- Second part is checking PTF polarization change and STSI with the cpu topology including drawers and books. This tests are run for KVM only.

To run these tests under KVM successfully you need Linux 6.0 and the QEMU patches you find at:

https://lists.gnu.org/archive/html/qemu-devel/2023-02/msg00081.html

To start the test in KVM just do:

# ./run_tests.sh topology

or

# ./s390x-run s390x/topology.elf \
	-smp 5,drawers=3,books=3,sockets=4,cores=4,maxcpus=144 \
	-append '-drawers 3 -books 3 -sockets 4 -cores 4'

Of course the declaration of the number of socket and core must be coherent in -smp and -append arguments.

To start the tests in LPAR you do not need any argument.

Regards,
Pierre

Pierre Morel (2):
s390x: topology: Check the Perform Topology Function
s390x: topology: Checking Configuration Topology Information

 lib/s390x/sclp.h    |   3 +-
 lib/s390x/stsi.h    |  44 +++++
 s390x/Makefile      |   1 +
 s390x/topology.c    | 399 ++++++++++++++++++++++++++++++++++++++++++++
 s390x/unittests.cfg |   4 +
 5 files changed, 450 insertions(+), 1 deletion(-)
 create mode 100644 s390x/topology.c
Signature · 17 lines
2.31.1

Changelog:

From v5:

- added drawers and books

- mnest parameter is not needed anymore, replaced
  by the use of sclp.

- Added several tests for polarity change and reset
  for PTF

From v4:

- Simplify the tests for socket and cores only.
#1 lore raw

PATCH v6 1/2 s390x: topology: Check the Perform Topology Function

We check that the PTF instruction is working correctly when the cpu topology facility is available.

For KVM only, we test changing of the polarity between horizontal and vertical and that a reset set the horizontal polarity.

---
 s390x/Makefile      |   1 +
 s390x/topology.c    | 155 ++++++++++++++++++++++++++++++++++++++++++++
 s390x/unittests.cfg |   3 +
 3 files changed, 159 insertions(+)
 create mode 100644 s390x/topology.c
s390x/Makefile +1 -0
index 52a9d82..b5fe8a3 100644
@@ -39,6 +39,7 @@ tests += $(TEST_DIR)/panic-loop-extint.elf
 tests += $(TEST_DIR)/panic-loop-pgm.elf
 tests += $(TEST_DIR)/migration-sck.elf
 tests += $(TEST_DIR)/exittime.elf
+tests += $(TEST_DIR)/topology.elf
 
 pv-tests += $(TEST_DIR)/pv-diags.elf
 
s390x/topology.c new file +155 -0

155 lines not on this page · raw message

s390x/unittests.cfg +3 -0
index 3caf81e..3530cc4 100644
@@ -208,3 +208,6 @@ groups = migration
 [exittime]
 file = exittime.elf
 smp = 2
+
+[topology]
+file = topology.elf
Signature · 1 line
2.31.1
#2 lore raw

Re: [kvm-unit-tests PATCH v6 1/2] s390x: topology: Check the Perform Topology Function

Quoted 57 lines · Pierre Morel

We check that the PTF instruction is working correctly when the cpu topology facility is available.

For KVM only, we test changing of the polarity between horizontal and vertical and that a reset set the horizontal polarity.

---
  s390x/Makefile      |   1 +
  s390x/topology.c    | 155 ++++++++++++++++++++++++++++++++++++++++++++
  s390x/unittests.cfg |   3 +
  3 files changed, 159 insertions(+)

create mode 100644 s390x/topology.c

s390x/Makefile +1 -0
index 52a9d82..b5fe8a3 100644
@@ -39,6 +39,7 @@ tests += $(TEST_DIR)/panic-loop-extint.elf
  tests += $(TEST_DIR)/panic-loop-pgm.elf
  tests += $(TEST_DIR)/migration-sck.elf
  tests += $(TEST_DIR)/exittime.elf
+tests += $(TEST_DIR)/topology.elf
 
  pv-tests += $(TEST_DIR)/pv-diags.elf
 
s390x/topology.c new file +25 -0
new file mode 100644
index 0000000..20f7ba2
@@ -0,0 +1,155 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * CPU Topology
+ *
+ * Copyright IBM Corp. 2022
+ *
+ * Authors:
+ *  Pierre Morel <[email protected]>
+ */
+
+#include <libcflat.h>
+#include <asm/page.h>
+#include <asm/asm-offsets.h>
+#include <asm/interrupt.h>
+#include <asm/facility.h>
+#include <smp.h>
+#include <sclp.h>
+#include <s390x/hardware.h>
+
+#define PTF_REQ_HORIZONTAL	0
+#define PTF_REQ_VERTICAL	1
+#define PTF_REQ_CHECK		2
+
+#define PTF_ERR_NO_REASON	0
+#define PTF_ERR_ALRDY_POLARIZED	1

s390x/topology.c line 35

Quoted 10 lines
s390x/topology.c new file +10 -0
+#define PTF_ERR_IN_PROGRESS	2
+
+extern int diag308_load_reset(u64);
+
+static int ptf(unsigned long fc, unsigned long *rc)
+{
+	int cc;
+
+	asm volatile(
+		"       .insn   rre,0xb9a20000,%1,0\n"

Why are you specifying the instruction manually? I think both, GCC and Clang should know the "ptf" mnemonic, shouldn't they?

Quoted 6 lines
s390x/topology.c new file +6 -0
+		"       ipm     %0\n"
+		"       srl     %0,28\n"
+		: "=d" (cc), "+d" (fc)
+		:
+		: "cc");
+

s390x/topology.c line 51

Quoted 10 lines
s390x/topology.c new file +10 -0
+	*rc = fc >> 8;
+	return cc;
+}
+
+static void test_ptf(void)
+{
+	unsigned long rc;
+	int cc;
+
+	/* PTF is a privilege instruction */

s/privilege/privileged/ ?

Quoted 8 lines
s390x/topology.c new file +8 -0
+	report_prefix_push("Privilege");
+	enter_pstate();
+	expect_pgm_int();
+	ptf(PTF_REQ_CHECK, &rc);
+	check_pgm_int_code(PGM_INT_CODE_PRIVILEGED_OPERATION);
+	report_prefix_pop();
+
+	report_prefix_push("Wrong fc");

s390x/topology.c line 69

Quoted 10 lines
s390x/topology.c new file +10 -0
+	expect_pgm_int();
+	ptf(0xff, &rc);
+	check_pgm_int_code(PGM_INT_CODE_SPECIFICATION);
+	report_prefix_pop();
+
+	report_prefix_push("Reserved bits");
+	expect_pgm_int();
+	ptf(0xffffffffffffff00UL, &rc);
+	check_pgm_int_code(PGM_INT_CODE_SPECIFICATION);
+	report_prefix_pop();

This function is quite big ... I'd maybe group the above checks for error conditions into a separate function instead.

Quoted 6 lines
s390x/topology.c new file +6 -0
+	report_prefix_push("Topology Report pending");
+	/*
+	 * At this moment the topology may already have changed
+	 * since the VM has been started.
+	 * However, we can test if a second PTF instruction
+	 * reports that the topology did not change since the

s390x/topology.c line 86

Quoted 10 lines
s390x/topology.c new file +10 -0
+	 * preceding PFT instruction.
+	 */
+	ptf(PTF_REQ_CHECK, &rc);
+	cc = ptf(PTF_REQ_CHECK, &rc);
+	report(cc == 0, "PTF check should clear topology report");
+	report_prefix_pop();
+
+	report_prefix_push("Topology polarisation check");
+	/*
+	 * We can not assume the state of the polarization for

s/can not/cannot/ ?

Also, you sometimes write polarization with "z" and sometimes with "s". I'd suggest to standardize on "z" (as in "IBM Z" ;-))

Quoted 27 lines
s390x/topology.c new file +27 -0
+	 * any Virtual Machine but KVM.
+	 * Let's skip the polarisation tests for other VMs.
+	 */
+	if (!host_is_kvm()) {
+		report_skip("Topology polarisation check is done for KVM only");
+		goto end;
+	}
+
+	/*
+	 * Set vertical polarization to verify that RESET sets
+	 * horizontal polarization back.
+	 */
+	cc = ptf(PTF_REQ_VERTICAL, &rc);
+	report(cc == 0, "Set vertical polarization.");
+
+	report(diag308_load_reset(1), "load normal reset done");
+
+	cc = ptf(PTF_REQ_CHECK, &rc);
+	report(cc == 0, "Reset should clear topology report");
+
+	cc = ptf(PTF_REQ_HORIZONTAL, &rc);
+	report(cc == 2 && rc == PTF_ERR_ALRDY_POLARIZED,
+	       "After RESET polarization is horizontal");
+
+	/* Flip between vertical and horizontal polarization */
+	cc = ptf(PTF_REQ_VERTICAL, &rc);
+	report(cc == 0, "Change to vertical polarization.");

s390x/topology.c line 123

Quoted 10 lines
s390x/topology.c new file +10 -0
+
+	cc = ptf(PTF_REQ_CHECK, &rc);
+	report(cc == 1, "Polarization change should set topology report");
+
+	cc = ptf(PTF_REQ_HORIZONTAL, &rc);
+	report(cc == 0, "Change to horizontal polarization.");
+
+end:
+	report_prefix_pop();
+}

Apart from the nits, the patch looks fine to me.

Thomas

#3 lore raw

Re: [kvm-unit-tests PATCH v6 1/2] s390x: topology: Check the Perform Topology Function

Quoted 60 lines · Thomas Huth
Quoted 59 lines · Pierre Morel

We check that the PTF instruction is working correctly when the cpu topology facility is available.

For KVM only, we test changing of the polarity between horizontal and vertical and that a reset set the horizontal polarity.

---
  s390x/Makefile      |   1 +
  s390x/topology.c    | 155 ++++++++++++++++++++++++++++++++++++++++++++ s390x/unittests.cfg |   3 +

  3 files changed, 159 insertions(+)

  create mode 100644 s390x/topology.c

s390x/Makefile +0 -0
index 52a9d82..b5fe8a3 100644
@@ -39,6 +39,7 @@ tests += $(TEST_DIR)/panic-loop-extint.elf

  tests += $(TEST_DIR)/panic-loop-pgm.elf
  tests += $(TEST_DIR)/migration-sck.elf
  tests += $(TEST_DIR)/exittime.elf
+tests += $(TEST_DIR)/topology.elf
  pv-tests += $(TEST_DIR)/pv-diags.elf

s390x/topology.c new file +29 -0
new file mode 100644
index 0000000..20f7ba2
@@ -0,0 +1,155 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * CPU Topology
+ *
+ * Copyright IBM Corp. 2022
+ *
+ * Authors:
+ *  Pierre Morel <[email protected]>
+ */
+
+#include <libcflat.h>
+#include <asm/page.h>
+#include <asm/asm-offsets.h>
+#include <asm/interrupt.h>
+#include <asm/facility.h>
+#include <smp.h>
+#include <sclp.h>
+#include <s390x/hardware.h>
+
+#define PTF_REQ_HORIZONTAL    0
+#define PTF_REQ_VERTICAL    1
+#define PTF_REQ_CHECK        2
+
+#define PTF_ERR_NO_REASON    0
+#define PTF_ERR_ALRDY_POLARIZED    1
+#define PTF_ERR_IN_PROGRESS    2
+
+extern int diag308_load_reset(u64);
+
Quoted 8 lines

s390x/topology.c line 35

Quoted 6 lines
s390x/topology.c new file +6 -0
+static int ptf(unsigned long fc, unsigned long *rc)
+{
+    int cc;
+
+    asm volatile(
+        "       .insn   rre,0xb9a20000,%1,0\n"

Why are you specifying the instruction manually? I think both, GCC and Clang should know the "ptf" mnemonic, shouldn't they?

:D right !

Quoted 11 lines
Quoted 11 lines
s390x/topology.c new file +10 -0
+        "       ipm     %0\n"
+        "       srl     %0,28\n"
+        : "=d" (cc), "+d" (fc)
+        :
+        : "cc");
+
+    *rc = fc >> 8;
+    return cc;
+}
+
Quoted 8 lines

s390x/topology.c line 51

Quoted 6 lines
s390x/topology.c new file +6 -0
+static void test_ptf(void)
+{
+    unsigned long rc;
+    int cc;
+
+    /* PTF is a privilege instruction */

s/privilege/privileged/ ?

Yes, thanks

Quoted 13 lines
Quoted 13 lines
s390x/topology.c new file +12 -0
+    report_prefix_push("Privilege");
+    enter_pstate();
+    expect_pgm_int();
+    ptf(PTF_REQ_CHECK, &rc);
+    check_pgm_int_code(PGM_INT_CODE_PRIVILEGED_OPERATION);
+    report_prefix_pop();
+
+    report_prefix_push("Wrong fc");
+    expect_pgm_int();
+    ptf(0xff, &rc);
+    check_pgm_int_code(PGM_INT_CODE_SPECIFICATION);
+    report_prefix_pop();
Quoted 8 lines

s390x/topology.c line 69

Quoted 6 lines
s390x/topology.c new file +6 -0
+
+    report_prefix_push("Reserved bits");
+    expect_pgm_int();
+    ptf(0xffffffffffffff00UL, &rc);
+    check_pgm_int_code(PGM_INT_CODE_SPECIFICATION);
+    report_prefix_pop();

This function is quite big ... I'd maybe group the above checks for error conditions into a separate function instead.

OK

Quoted 18 lines
Quoted 7 lines
s390x/topology.c new file +6 -0
+    report_prefix_push("Topology Report pending");
+    /*
+     * At this moment the topology may already have changed
+     * since the VM has been started.
+     * However, we can test if a second PTF instruction
+     * reports that the topology did not change since the

s390x/topology.c line 86

Quoted 10 lines
s390x/topology.c new file +10 -0
+     * preceding PFT instruction.
+     */
+    ptf(PTF_REQ_CHECK, &rc);
+    cc = ptf(PTF_REQ_CHECK, &rc);
+    report(cc == 0, "PTF check should clear topology report");
+    report_prefix_pop();
+
+    report_prefix_push("Topology polarisation check");
+    /*
+     * We can not assume the state of the polarization for
Quoted 1 line

s/can not/cannot/ ?

OK

Quoted 2 lines

Also, you sometimes write polarization with "z" and sometimes with "s". I'd suggest to standardize on "z" (as in "IBM Z" ;-))

OK

Quoted 41 lines
Quoted 28 lines
s390x/topology.c new file +27 -0
+     * any Virtual Machine but KVM.
+     * Let's skip the polarisation tests for other VMs.
+     */
+    if (!host_is_kvm()) {
+        report_skip("Topology polarisation check is done for KVM only");
+        goto end;
+    }
+
+    /*
+     * Set vertical polarization to verify that RESET sets
+     * horizontal polarization back.
+     */
+    cc = ptf(PTF_REQ_VERTICAL, &rc);
+    report(cc == 0, "Set vertical polarization.");
+
+    report(diag308_load_reset(1), "load normal reset done");
+
+    cc = ptf(PTF_REQ_CHECK, &rc);
+    report(cc == 0, "Reset should clear topology report");
+
+    cc = ptf(PTF_REQ_HORIZONTAL, &rc);
+    report(cc == 2 && rc == PTF_ERR_ALRDY_POLARIZED,
+           "After RESET polarization is horizontal");
+
+    /* Flip between vertical and horizontal polarization */
+    cc = ptf(PTF_REQ_VERTICAL, &rc);
+    report(cc == 0, "Change to vertical polarization.");

s390x/topology.c line 123

Quoted 10 lines
s390x/topology.c new file +10 -0
+
+    cc = ptf(PTF_REQ_CHECK, &rc);
+    report(cc == 1, "Polarization change should set topology report");
+
+    cc = ptf(PTF_REQ_HORIZONTAL, &rc);
+    report(cc == 0, "Change to horizontal polarization.");
+
+end:
+    report_prefix_pop();
+}

Apart from the nits, the patch looks fine to me.

Quoted 2 lines

 Thomas

Thanks,

Regards.
Pierre

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
#4 lore raw

Re: [kvm-unit-tests PATCH v6 1/2] s390x: topology: Check the Perform Topology Function

In reply to Pierre Morel
Quoted 48 lines · Pierre Morel

We check that the PTF instruction is working correctly when the cpu topology facility is available.

For KVM only, we test changing of the polarity between horizontal and vertical and that a reset set the horizontal polarity.

---
 s390x/Makefile      |   1 +
 s390x/topology.c    | 155 ++++++++++++++++++++++++++++++++++++++++++++
 s390x/unittests.cfg |   3 +
 3 files changed, 159 insertions(+)
 create mode 100644 s390x/topology.c
s390x/Makefile +1 -0
index 52a9d82..b5fe8a3 100644
@@ -39,6 +39,7 @@ tests += $(TEST_DIR)/panic-loop-extint.elf
 tests += $(TEST_DIR)/panic-loop-pgm.elf
 tests += $(TEST_DIR)/migration-sck.elf
 tests += $(TEST_DIR)/exittime.elf
+tests += $(TEST_DIR)/topology.elf
 
 pv-tests += $(TEST_DIR)/pv-diags.elf
 
s390x/topology.c new file +16 -0
new file mode 100644
index 0000000..20f7ba2
@@ -0,0 +1,155 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * CPU Topology
+ *
+ * Copyright IBM Corp. 2022
+ *
+ * Authors:
+ *  Pierre Morel <[email protected]>
+ */
+
+#include <libcflat.h>
+#include <asm/page.h>
+#include <asm/asm-offsets.h>
+#include <asm/interrupt.h>
+#include <asm/facility.h>
+#include <smp.h>

s390x/topology.c line 26

Quoted 10 lines
s390x/topology.c new file +10 -0
+#include <sclp.h>
+#include <s390x/hardware.h>
+
+#define PTF_REQ_HORIZONTAL	0
+#define PTF_REQ_VERTICAL	1
+#define PTF_REQ_CHECK		2
+
+#define PTF_ERR_NO_REASON	0
+#define PTF_ERR_ALRDY_POLARIZED	1
+#define PTF_ERR_IN_PROGRESS	2

Maybe also give the CC codes names for improved readability.

Quoted 4 lines
s390x/topology.c new file +4 -0
+
+extern int diag308_load_reset(u64);
+
+static int ptf(unsigned long fc, unsigned long *rc)

s390x/topology.c line 40

Quoted 10 lines
s390x/topology.c new file +10 -0
+{
+	int cc;
+
+	asm volatile(
+		"       .insn   rre,0xb9a20000,%1,0\n"
+		"       ipm     %0\n"
+		"       srl     %0,28\n"
+		: "=d" (cc), "+d" (fc)
+		:
+		: "cc");

Personally I always name asm arguments, but it is a very short snippet, so still very readable. Could also pull the shift into C code, but again, small difference.

Quoted 7 lines
s390x/topology.c new file +7 -0
+
+	*rc = fc >> 8;
+	return cc;
+}
+
+static void test_ptf(void)
+{

s390x/topology.c line 57

Quoted 10 lines
s390x/topology.c new file +10 -0
+	unsigned long rc;
+	int cc;
+
+	/* PTF is a privilege instruction */
+	report_prefix_push("Privilege");
+	enter_pstate();
+	expect_pgm_int();
+	ptf(PTF_REQ_CHECK, &rc);
+	check_pgm_int_code(PGM_INT_CODE_PRIVILEGED_OPERATION);
+	report_prefix_pop();

IMO, you should repeat this test for all FCs, since some are emulated, others interpreted by SIE.

s390x/topology.c line 59

Quoted 2 lines
s390x/topology.c new file +2 -0
+
+	report_prefix_push("Wrong fc");

"Undefined fc" is more informative IMO.

Quoted 18 lines
s390x/topology.c new file +18 -0
+	expect_pgm_int();
+	ptf(0xff, &rc);
+	check_pgm_int_code(PGM_INT_CODE_SPECIFICATION);
+	report_prefix_pop();
+
+	report_prefix_push("Reserved bits");
+	expect_pgm_int();
+	ptf(0xffffffffffffff00UL, &rc);
+	check_pgm_int_code(PGM_INT_CODE_SPECIFICATION);
+	report_prefix_pop();
+
+	report_prefix_push("Topology Report pending");
+	/*
+	 * At this moment the topology may already have changed
+	 * since the VM has been started.
+	 * However, we can test if a second PTF instruction
+	 * reports that the topology did not change since the
+	 * preceding PFT instruction.

s390x/topology.c line 87

Quoted 10 lines
s390x/topology.c new file +10 -0
+	 */
+	ptf(PTF_REQ_CHECK, &rc);
+	cc = ptf(PTF_REQ_CHECK, &rc);
+	report(cc == 0, "PTF check should clear topology report");
+	report_prefix_pop();
+
+	report_prefix_push("Topology polarisation check");
+	/*
+	 * We can not assume the state of the polarization for
+	 * any Virtual Machine but KVM.

Random Capitalization :)
Why can you not test the same thing for other hypervisors/LPAR?

s390x/topology.c line 98

Quoted 11 lines
s390x/topology.c new file +11 -0
+	 * Let's skip the polarisation tests for other VMs.
+	 */
+	if (!host_is_kvm()) {
+		report_skip("Topology polarisation check is done for KVM only");
+		goto end;
+	}
+
+	/*
+	 * Set vertical polarization to verify that RESET sets
+	 * horizontal polarization back.
+	 */

You might want to do a reset here also, since there could be some other test case that could have run before and modified the polarization. There isn't right now of course, but doing a reset improves separation of tests.

Quoted 68 lines
s390x/topology.c new file +57 -0
+	cc = ptf(PTF_REQ_VERTICAL, &rc);
+	report(cc == 0, "Set vertical polarization.");
+
+	report(diag308_load_reset(1), "load normal reset done");
+
+	cc = ptf(PTF_REQ_CHECK, &rc);
+	report(cc == 0, "Reset should clear topology report");
+
+	cc = ptf(PTF_REQ_HORIZONTAL, &rc);
+	report(cc == 2 && rc == PTF_ERR_ALRDY_POLARIZED,
+	       "After RESET polarization is horizontal");
+
+	/* Flip between vertical and horizontal polarization */
+	cc = ptf(PTF_REQ_VERTICAL, &rc);
+	report(cc == 0, "Change to vertical polarization.");
+
+	cc = ptf(PTF_REQ_CHECK, &rc);
+	report(cc == 1, "Polarization change should set topology report");
+
+	cc = ptf(PTF_REQ_HORIZONTAL, &rc);
+	report(cc == 0, "Change to horizontal polarization.");
+
+end:
+	report_prefix_pop();
+}
+
+static struct {
+	const char *name;
+	void (*func)(void);
+} tests[] = {
+	{ "PTF", test_ptf},
+	{ NULL, NULL }
+};
+
+int main(int argc, char *argv[])
+{
+	int i;
+
+	report_prefix_push("CPU Topology");
+
+	if (!test_facility(11)) {
+		report_skip("Topology facility not present");
+		goto end;
+	}
+
+	report_info("Virtual machine level %ld", stsi_get_fc());
+
+	for (i = 0; tests[i].name; i++) {
+		report_prefix_push(tests[i].name);
+		tests[i].func();
+		report_prefix_pop();
+	}
+
+end:
+	report_prefix_pop();
+	return report_summary();
+}
s390x/unittests.cfg +3 -0
index 3caf81e..3530cc4 100644
@@ -208,3 +208,6 @@ groups = migration
 [exittime]
 file = exittime.elf
 smp = 2
+
+[topology]
+file = topology.elf
#5 lore raw

Re: [kvm-unit-tests PATCH v6 1/2] s390x: topology: Check the Perform Topology Function

Quoted 53 lines · Nina Schoetterl-Glausch
Quoted 52 lines · Pierre Morel

We check that the PTF instruction is working correctly when the cpu topology facility is available.

For KVM only, we test changing of the polarity between horizontal and vertical and that a reset set the horizontal polarity.

---
  s390x/Makefile      |   1 +
  s390x/topology.c    | 155 ++++++++++++++++++++++++++++++++++++++++++++
  s390x/unittests.cfg |   3 +
  3 files changed, 159 insertions(+)

create mode 100644 s390x/topology.c

s390x/Makefile +1 -0
index 52a9d82..b5fe8a3 100644
@@ -39,6 +39,7 @@ tests += $(TEST_DIR)/panic-loop-extint.elf
  tests += $(TEST_DIR)/panic-loop-pgm.elf
  tests += $(TEST_DIR)/migration-sck.elf
  tests += $(TEST_DIR)/exittime.elf
+tests += $(TEST_DIR)/topology.elf
 
  pv-tests += $(TEST_DIR)/pv-diags.elf
 
s390x/topology.c new file +20 -0
new file mode 100644
index 0000000..20f7ba2
@@ -0,0 +1,155 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * CPU Topology
+ *
+ * Copyright IBM Corp. 2022
+ *
+ * Authors:
+ *  Pierre Morel <[email protected]>
+ */
+
+#include <libcflat.h>
+#include <asm/page.h>
+#include <asm/asm-offsets.h>
+#include <asm/interrupt.h>
+#include <asm/facility.h>
+#include <smp.h>
+#include <sclp.h>
+#include <s390x/hardware.h>
+
+#define PTF_REQ_HORIZONTAL	0
Quoted 8 lines

s390x/topology.c line 26

Quoted 6 lines
s390x/topology.c new file +6 -0
+#define PTF_REQ_VERTICAL	1
+#define PTF_REQ_CHECK		2
+
+#define PTF_ERR_NO_REASON	0
+#define PTF_ERR_ALRDY_POLARIZED	1
+#define PTF_ERR_IN_PROGRESS	2

Maybe also give the CC codes names for improved readability.

OK

Quoted 32 lines
Quoted 5 lines
s390x/topology.c new file +4 -0
+
+extern int diag308_load_reset(u64);
+
+static int ptf(unsigned long fc, unsigned long *rc)

s390x/topology.c line 40

Quoted 10 lines
s390x/topology.c new file +10 -0
+{
+	int cc;
+
+	asm volatile(
+		"       .insn   rre,0xb9a20000,%1,0\n"
+		"       ipm     %0\n"
+		"       srl     %0,28\n"
+		: "=d" (cc), "+d" (fc)
+		:
+		: "cc");

Personally I always name asm arguments, but it is a very short snippet, so still very readable. Could also pull the shift into C code, but again, small difference.

Quoted 12 lines
s390x/topology.c new file +12 -0
+
+	*rc = fc >> 8;
+	return cc;
+}
+
+static void test_ptf(void)
+{
+	unsigned long rc;
+	int cc;
+
+	/* PTF is a privilege instruction */
+	report_prefix_push("Privilege");
Quoted 8 lines

s390x/topology.c line 57

Quoted 5 lines
s390x/topology.c new file +5 -0
+	enter_pstate();
+	expect_pgm_int();
+	ptf(PTF_REQ_CHECK, &rc);
+	check_pgm_int_code(PGM_INT_CODE_PRIVILEGED_OPERATION);
+	report_prefix_pop();

IMO, you should repeat this test for all FCs, since some are emulated, others interpreted by SIE.

right

Quoted 5 lines

s390x/topology.c line 59

Quoted 3 lines
s390x/topology.c new file +2 -0
+
+	report_prefix_push("Wrong fc");

"Undefined fc" is more informative IMO.

OK

Quoted 23 lines
Quoted 23 lines
s390x/topology.c new file +22 -0
+	expect_pgm_int();
+	ptf(0xff, &rc);
+	check_pgm_int_code(PGM_INT_CODE_SPECIFICATION);
+	report_prefix_pop();
+
+	report_prefix_push("Reserved bits");
+	expect_pgm_int();
+	ptf(0xffffffffffffff00UL, &rc);
+	check_pgm_int_code(PGM_INT_CODE_SPECIFICATION);
+	report_prefix_pop();
+
+	report_prefix_push("Topology Report pending");
+	/*
+	 * At this moment the topology may already have changed
+	 * since the VM has been started.
+	 * However, we can test if a second PTF instruction
+	 * reports that the topology did not change since the
+	 * preceding PFT instruction.
+	 */
+	ptf(PTF_REQ_CHECK, &rc);
+	cc = ptf(PTF_REQ_CHECK, &rc);
+	report(cc == 0, "PTF check should clear topology report");
Quoted 8 lines

s390x/topology.c line 87

Quoted 6 lines
s390x/topology.c new file +6 -0
+	report_prefix_pop();
+
+	report_prefix_push("Topology polarisation check");
+	/*
+	 * We can not assume the state of the polarization for
+	 * any Virtual Machine but KVM.

Random Capitalization :)

OK

Quoted 1 line

Why can you not test the same thing for other hypervisors/LPAR?

At first QEMU did not support vertical polarization so my tests would have get a false negative on LPAR. I could have done different tests but did not.

I think that now it is alright to do the checks on LPAR too.

Quoted 16 lines

s390x/topology.c line 98

Quoted 12 lines
s390x/topology.c new file +11 -0
+	 * Let's skip the polarisation tests for other VMs.
+	 */
+	if (!host_is_kvm()) {
+		report_skip("Topology polarisation check is done for KVM only");
+		goto end;
+	}
+
+	/*
+	 * Set vertical polarization to verify that RESET sets
+	 * horizontal polarization back.
+	 */

You might want to do a reset here also, since there could be some other test case that could have run before and modified the polarization. There isn't right now of course, but doing a reset improves separation of tests.

Not sure about this but it does not arm so why not.

Thanks.

regards,
Pierre

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
#6 lore raw

PATCH v6 2/2 s390x: topology: Checking Configuration Topology Information

In reply to Pierre Morel

STSI with function code 15 is used to store the CPU configuration topology.

We retrieve the maximum nested level with SCLP and use the topology tree provided by the drawers, books, sockets, cores arguments.

We check :
- if the topology stored is coherent between the QEMU -smp parameters and kernel parameters.
- the number of CPUs
- the maximum number of CPUs
- the number of containers of each levels for every STSI(15.1.x) instruction allowed by the machine.

---
 lib/s390x/sclp.h    |   3 +-
 lib/s390x/stsi.h    |  44 ++++++++
 s390x/topology.c    | 244 ++++++++++++++++++++++++++++++++++++++++++++
 s390x/unittests.cfg |   1 +
 4 files changed, 291 insertions(+), 1 deletion(-)
lib/s390x/sclp.h +2 -1
index 853529b..6ecfb0a 100644
@@ -150,7 +150,8 @@ typedef struct ReadInfo {
 	SCCBHeader h;
 	uint16_t rnmax;
 	uint8_t rnsize;
-	uint8_t  _reserved1[16 - 11];       /* 11-15 */
+	uint8_t  _reserved1[15 - 11];       /* 11-14 */
+	uint8_t stsi_parm;                  /* 15-15 */
 	uint16_t entries_cpu;               /* 16-17 */
 	uint16_t offset_cpu;                /* 18-19 */
 	uint8_t  _reserved2[24 - 20];       /* 20-23 */
lib/s390x/stsi.h +44 -0

48 lines not on this page · raw message

s390x/topology.c +244 -0

267 lines not on this page · raw message

s390x/unittests.cfg +1 -0
index 3530cc4..b697aca 100644
@@ -211,3 +211,4 @@ smp = 2
 
 [topology]
 file = topology.elf
+extra_params=-smp 5,drawers=3,books=3,sockets=4,cores=4,maxcpus=144 -append '-drawers 3 -books 3 -sockets 4 -cores 4'
Signature · 1 line
2.31.1
#7 lore raw

Re: [kvm-unit-tests PATCH v6 2/2] s390x: topology: Checking Configuration Topology Information

Quoted 17 lines · Pierre Morel

STSI with function code 15 is used to store the CPU configuration topology.

We retrieve the maximum nested level with SCLP and use the topology tree provided by the drawers, books, sockets, cores arguments.

We check :
- if the topology stored is coherent between the QEMU -smp parameters and kernel parameters.
- the number of CPUs
- the maximum number of CPUs
- the number of containers of each levels for every STSI(15.1.x) instruction allowed by the machine.

---

...

lib/s390x/stsi.h struct sysinfo_3_2_2 { · line 73

Quoted 8 lines
lib/s390x/stsi.h +8 -0
+static inline int cpus_in_tle_mask(uint64_t val)
+{
+	int i, n;
+
+	for (i = 0, n = 0; i < 64; i++, val >>= 1)
+		if (val & 0x01)
+			n++;
+	return n;

I'd suggest to use __builtin_popcountl here instead of looping.

Quoted 8 lines
lib/s390x/stsi.h +2 -0
+}
+
  #endif  /* _S390X_STSI_H_ */
s390x/topology.c +0 -0
index 20f7ba2..f21c653 100644

s390x/topology.c line 21

Quoted 6 lines
s390x/topology.c +3 -0
@@ -16,6 +16,18 @@
  #include <smp.h>
  #include <sclp.h>
  #include <s390x/hardware.h>
+#include <s390x/stsi.h>
+
+static uint8_t pagebuf[PAGE_SIZE * 2] __attribute__((aligned(PAGE_SIZE * 2)));

Isn't the SYSIB just one page only? Why reserve two pages here?

Quoted 21 lines
s390x/topology.c +14 -0
+static int max_nested_lvl;
+static int number_of_cpus;
+static int max_cpus = 1;
+
+/* Topology level as defined by architecture */
+static int arch_topo_lvl[CPU_TOPOLOGY_MAX_LEVEL];
+/* Topology nested level as reported in STSI */
+static int stsi_nested_lvl[CPU_TOPOLOGY_MAX_LEVEL];
 
  #define PTF_REQ_HORIZONTAL	0
  #define PTF_REQ_VERTICAL	1
@@ -122,11 +134,241 @@ end:
  	report_prefix_pop();
  }
 
+/*
+ * stsi_check_maxcpus
+ * @info: Pointer to the stsi information
+ *
+ * The product of the numbers of containers per level
+ * is the maximum number of CPU allowed by the machine.

s390x/topology.c end: · line 152

Quoted 10 lines
s390x/topology.c +10 -0
+ */
+static void stsi_check_maxcpus(struct sysinfo_15_1_x *info)
+{
+	int n, i;
+
+	report_prefix_push("maximum cpus");
+
+	for (i = 0, n = 1; i < CPU_TOPOLOGY_MAX_LEVEL; i++) {
+		report_info("Mag%d: %d", CPU_TOPOLOGY_MAX_LEVEL - i, info->mag[i]);
+		n *= info->mag[i] ? info->mag[i] : 1;

You could use the Elvis operator here instead.

Quoted 15 lines
s390x/topology.c +15 -0
+	}
+	report(n == max_cpus, "Maximum CPUs %d expected %d", n, max_cpus);
+
+	report_prefix_pop();
+}
+
+/*
+ * stsi_check_tle_coherency
+ * @info: Pointer to the stsi information
+ * @sel2: Topology level to check.
+ *
+ * We verify that we get the expected number of Topology List Entry
+ * containers for a specific level.
+ */
+static void stsi_check_tle_coherency(struct sysinfo_15_1_x *info, int sel2)

s390x/topology.c end: · line 177

Quoted 10 lines
s390x/topology.c +10 -0
+{
+	struct topology_container *tc, *end;
+	struct topology_core *cpus;
+	int n = 0;
+	int i;
+
+	report_prefix_push("TLE coherency");
+
+	tc = &info->tle[0].container;
+	end = (struct topology_container *)((unsigned long)info + info->length);

s/unsigned long/uintptr_t/ please!

s390x/topology.c end: · line 180

Quoted 3 lines
s390x/topology.c +3 -0
+
+	for (i = 0; i < CPU_TOPOLOGY_MAX_LEVEL; i++)
+		stsi_nested_lvl[i] = 0;

memset(stsi_nested_lvl, 0, sizeof(stsi_nested_lvl)) ?

s390x/topology.c end: · line 183

Quoted 2 lines
s390x/topology.c +2 -0
+	while (tc < end) {
+		if (tc->nl > 5) {

Use ">= CPU_TOPOLOGY_MAX_LEVEL" instead of "> 5" ?

Quoted 7 lines
s390x/topology.c +7 -0
+			report_abort("Unexpected TL Entry: tle->nl: %d", tc->nl);
+			return;
+		}
+		if (tc->nl == 0) {
+			cpus = (struct topology_core *)tc;
+			n += cpus_in_tle_mask(cpus->mask);
+			report_info("cpu type %02x  d: %d pp: %d", cpus->type, cpus->d, cpus->pp);

s390x/topology.c end: · line 200

Quoted 10 lines
s390x/topology.c +10 -0
+			report_info("origin : %04x mask %016lx", cpus->origin, cpus->mask);
+		}
+
+		stsi_nested_lvl[tc->nl]++;
+		report_info("level %d: lvl: %d id: %d cnt: %d",
+			    tc->nl, tc->nl, tc->id, stsi_nested_lvl[tc->nl]);
+
+		/* trick: CPU TLEs are twice the size of containers TLE */
+		if (tc->nl == 0)
+			tc++;

IMHO it might be cleaner to have a "uint8_t *" or "void *" to the current position in the sysinfo block, and do the pointer arithmetic on that pointer instead... well, it's likely just a matter of taste.

Quoted 55 lines
s390x/topology.c +55 -0
+		tc++;
+	}
+	report(n == number_of_cpus, "Number of CPUs  : %d expect %d", n, number_of_cpus);
+	/*
+	 * For KVM we accept
+	 * - only 1 type of CPU
+	 * - only horizontal topology
+	 * - only dedicated CPUs
+	 * This leads to expect the number of entries of level 0 CPU
+	 * Topology Level Entry (TLE) to be:
+	 * 1 + (number_of_cpus - 1)  / arch_topo_lvl[0]
+	 *
+	 * For z/VM or LPAR this number can only be greater if different
+	 * polarity, CPU types because there may be a nested level 0 CPU TLE
+	 * for each of the CPU/polarity/sharing types in a level 1 container TLE.
+	 */
+	n =  (number_of_cpus - 1)  / arch_topo_lvl[0];
+	report(stsi_nested_lvl[0] >=  n + 1,
+	       "CPU Type TLE    : %d expect %d", stsi_nested_lvl[0], n + 1);
+
+	/* For each level found in STSI */
+	for (i = 1; i < CPU_TOPOLOGY_MAX_LEVEL; i++) {
+		/*
+		 * For non QEMU/KVM hypervisor the concatenation of the levels
+		 * above level 1 are architecture dependent.
+		 * Skip these checks.
+		 */
+		if (!host_is_kvm() && sel2 != 2)
+			continue;
+
+		/* For QEMU/KVM we expect a simple calculation */
+		if (sel2 > i) {
+			report(stsi_nested_lvl[i] ==  n + 1,
+			       "Container TLE  %d: %d expect %d", i, stsi_nested_lvl[i], n + 1);
+			n /= arch_topo_lvl[i];
+		}
+	}
+
+	report_prefix_pop();
+}
+
+/*
+ * check_sysinfo_15_1_x
+ * @info: pointer to the STSI info structure
+ * @sel2: the selector giving the topology level to check
+ *
+ * Check if the validity of the STSI instruction and then
+ * calls specific checks on the information buffer.
+ */
+static void check_sysinfo_15_1_x(struct sysinfo_15_1_x *info, int sel2)
+{
+	int ret;
+
+	report_prefix_pushf("mnested %d 15_1_%d", max_nested_lvl, sel2);
+

s390x/topology.c end: · line 265

Quoted 10 lines
s390x/topology.c +10 -0
+	ret = stsi(pagebuf, 15, 1, sel2);
+	if (max_nested_lvl >= sel2) {
+		report(!ret, "Valid stsi instruction");
+	} else {
+		report(ret, "Invalid stsi instruction");
+		goto end;
+	}
+
+	stsi_check_maxcpus(info);
+	stsi_check_tle_coherency(info, sel2);

You could also move the two stsi_check_* calls into the first part of the if-statement, then you could get rid of the goto in the second part.

Quoted 80 lines
s390x/topology.c +80 -0
+end:
+	report_prefix_pop();
+}
+
+static int sclp_get_mnest(void)
+{
+	ReadInfo *sccb = (void *)_sccb;
+
+	sclp_mark_busy();
+	memset(_sccb, 0, PAGE_SIZE);
+	sccb->h.length = PAGE_SIZE;
+
+	sclp_service_call(SCLP_CMDW_READ_SCP_INFO, sccb);
+	assert(sccb->h.response_code == SCLP_RC_NORMAL_READ_COMPLETION);
+
+	return sccb->stsi_parm;
+}
+
+/*
+ * test_stsi
+ *
+ * Retrieves the maximum nested topology level supported by the architecture
+ * and the number of CPUs.
+ * Calls the checking for the STSI instruction in sel2 reverse level order
+ * from 6 (CPU_TOPOLOGY_MAX_LEVEL) to 2 to have the most interesting level,
+ * the one triggering a topology-change-report-pending condition, level 2,
+ * at the end of the report.
+ *
+ */
+static void test_stsi(void)
+{
+	int sel2;
+
+	max_nested_lvl = sclp_get_mnest();
+	report_info("SCLP maximum nested level : %d", max_nested_lvl);
+
+	number_of_cpus = sclp_get_cpu_num();
+	report_info("SCLP number of CPU: %d", number_of_cpus);
+
+	/* STSI selector 2 can takes values between 2 and 6 */
+	for (sel2 = 6; sel2 >= 2; sel2--)
+		check_sysinfo_15_1_x((struct sysinfo_15_1_x *)pagebuf, sel2);
+}
+
+/*
+ * parse_topology_args
+ * @argc: number of arguments
+ * @argv: argument array
+ *
+ * This function initialize the architecture topology levels
+ * which should be the same as the one provided by the hypervisor.
+ *
+ * We use the current names found in IBM/Z literature, Linux and QEMU:
+ * cores, sockets/packages, books, drawers and nodes to facilitate the
+ * human machine interface but store the result in a machine abstract
+ * array of architecture topology levels.
+ * Note that when QEMU uses socket as a name for the topology level 1
+ * Linux uses package or physical_package.
+ */
+static void parse_topology_args(int argc, char **argv)
+{
+	int i;
+
+	report_info("%d arguments", argc);
+	for (i = 1; i < argc; i++) {
+		if (!strcmp("-cores", argv[i])) {
+			i++;
+			if (i >= argc)
+				report_abort("-cores needs a parameter");
+			arch_topo_lvl[0] = atol(argv[i]);
+			report_info("cores: %d", arch_topo_lvl[0]);
+		} else if (!strcmp("-sockets", argv[i])) {
+			i++;
+			if (i >= argc)
+				report_abort("-sockets needs a parameter");
+			arch_topo_lvl[1] = atol(argv[i]);
+			report_info("sockets: %d", arch_topo_lvl[1]);
+		} else if (!strcmp("-books", argv[i])) {
+			i++;
+			if (i >= argc)

s390x/topology.c end: · line 356

Quoted 10 lines
s390x/topology.c +10 -0
+				report_abort("-books needs a parameter");
+			arch_topo_lvl[2] = atol(argv[i]);
+			report_info("books: %d", arch_topo_lvl[2]);
+		} else if (!strcmp("-drawers", argv[i])) {
+			i++;
+			if (i >= argc)
+				report_abort("-drawers needs a parameter");
+			arch_topo_lvl[3] = atol(argv[i]);
+			report_info("drawers: %d", arch_topo_lvl[3]);
+		}

Maybe abort on unkown parameters, to avoid that typos go unnoticed?

s390x/topology.c end: · line 364

Quoted 8 lines
s390x/topology.c +8 -0
+	}
+
+	for (i = 0; i < CPU_TOPOLOGY_MAX_LEVEL; i++) {
+		if (!arch_topo_lvl[i])
+			arch_topo_lvl[i] = 1;
+		max_cpus *= arch_topo_lvl[i];
+	}
+}

Thomas

#8 lore raw

Re: [kvm-unit-tests PATCH v6 2/2] s390x: topology: Checking Configuration Topology Information

Quoted 19 lines · Thomas Huth
Quoted 17 lines · Pierre Morel

STSI with function code 15 is used to store the CPU configuration topology.

We retrieve the maximum nested level with SCLP and use the topology tree provided by the drawers, books, sockets, cores arguments.

We check :
- if the topology stored is coherent between the QEMU -smp parameters and kernel parameters.
- the number of CPUs
- the maximum number of CPUs
- the number of containers of each levels for every STSI(15.1.x) instruction allowed by the machine.

---

...

Quoted 10 lines

lib/s390x/stsi.h struct sysinfo_3_2_2 { · line 73

Quoted 8 lines
lib/s390x/stsi.h +8 -0
+static inline int cpus_in_tle_mask(uint64_t val)
+{
+    int i, n;
+
+    for (i = 0, n = 0; i < 64; i++, val >>= 1)
+        if (val & 0x01)
+            n++;
+    return n;

I'd suggest to use __builtin_popcountl here instead of looping.

OK

Quoted 10 lines
Quoted 10 lines
(quoted diff) +2 -0
+}
+

  #endif  /* _S390X_STSI_H_ */

s390x/topology.c +0 -0
index 20f7ba2..f21c653 100644
@@ -16,6 +16,18 @@
   #include <smp.h>
Quoted 8 lines

s390x/topology.c

Quoted 6 lines
s390x/topology.c +3 -0
   #include <sclp.h>
   #include <s390x/hardware.h>
+#include <s390x/stsi.h>
+
+static uint8_t pagebuf[PAGE_SIZE * 2] 
 __attribute__((aligned(PAGE_SIZE * 2)));

Isn't the SYSIB just one page only? Why reserve two pages here?

Yes it is, I change it to a single page.

Quoted 25 lines
Quoted 25 lines
s390x/topology.c +19 -0
+static int max_nested_lvl;
+static int number_of_cpus;
+static int max_cpus = 1;
+
+/* Topology level as defined by architecture */
+static int arch_topo_lvl[CPU_TOPOLOGY_MAX_LEVEL];
+/* Topology nested level as reported in STSI */
+static int stsi_nested_lvl[CPU_TOPOLOGY_MAX_LEVEL];
   #define PTF_REQ_HORIZONTAL    0
   #define PTF_REQ_VERTICAL    1
@@ -122,11 +134,241 @@ end:
       report_prefix_pop();
   }
+/*
+ * stsi_check_maxcpus
+ * @info: Pointer to the stsi information
+ *
+ * The product of the numbers of containers per level
+ * is the maximum number of CPU allowed by the machine.
+ */
+static void stsi_check_maxcpus(struct sysinfo_15_1_x *info)
+{
+    int n, i;
+
Quoted 8 lines

s390x/topology.c end: · line 152

Quoted 6 lines
s390x/topology.c +5 -0
+    report_prefix_push("maximum cpus");
+
+    for (i = 0, n = 1; i < CPU_TOPOLOGY_MAX_LEVEL; i++) {
+        report_info("Mag%d: %d", CPU_TOPOLOGY_MAX_LEVEL - i, 
 info->mag[i]);
+        n *= info->mag[i] ? info->mag[i] : 1;

You could use the Elvis operator here instead.

Right thanks.

Quoted 22 lines
Quoted 22 lines
s390x/topology.c +20 -0
+    }
+    report(n == max_cpus, "Maximum CPUs %d expected %d", n, max_cpus);
+
+    report_prefix_pop();
+}
+
+/*
+ * stsi_check_tle_coherency
+ * @info: Pointer to the stsi information
+ * @sel2: Topology level to check.
+ *
+ * We verify that we get the expected number of Topology List Entry
+ * containers for a specific level.
+ */
+static void stsi_check_tle_coherency(struct sysinfo_15_1_x *info, int 
 sel2)
+{
+    struct topology_container *tc, *end;
+    struct topology_core *cpus;
+    int n = 0;
+    int i;
Quoted 8 lines

s390x/topology.c end:

Quoted 6 lines
s390x/topology.c +5 -0
+
+    report_prefix_push("TLE coherency");
+
+    tc = &info->tle[0].container;
+    end = (struct topology_container *)((unsigned long)info + 
 info->length);

s/unsigned long/uintptr_t/ please!

OK, thanks

Quoted 7 lines

s390x/topology.c end: · line 180

Quoted 5 lines
s390x/topology.c +3 -0
+
+    for (i = 0; i < CPU_TOPOLOGY_MAX_LEVEL; i++)
+        stsi_nested_lvl[i] = 0;

memset(stsi_nested_lvl, 0, sizeof(stsi_nested_lvl)) ?

better, thanks

Quoted 5 lines

s390x/topology.c end: · line 183

Quoted 3 lines
s390x/topology.c +2 -0
+    while (tc < end) {
+        if (tc->nl > 5) {

Use ">= CPU_TOPOLOGY_MAX_LEVEL" instead of "> 5" ?

OK

Quoted 14 lines
Quoted 14 lines
s390x/topology.c +11 -0
+            report_abort("Unexpected TL Entry: tle->nl: %d", tc->nl);
+            return;
+        }
+        if (tc->nl == 0) {
+            cpus = (struct topology_core *)tc;
+            n += cpus_in_tle_mask(cpus->mask);
+            report_info("cpu type %02x  d: %d pp: %d", cpus->type, 
 cpus->d, cpus->pp);
+            report_info("origin : %04x mask %016lx", cpus->origin, 
 cpus->mask);
+        }
+
+        stsi_nested_lvl[tc->nl]++;
Quoted 8 lines

s390x/topology.c end: · line 200

Quoted 6 lines
s390x/topology.c +6 -0
+        report_info("level %d: lvl: %d id: %d cnt: %d",
+                tc->nl, tc->nl, tc->id, stsi_nested_lvl[tc->nl]);
+
+        /* trick: CPU TLEs are twice the size of containers TLE */
+        if (tc->nl == 0)
+            tc++;

IMHO it might be cleaner to have a "uint8_t *" or "void *" to the current position in the sysinfo block, and do the pointer arithmetic on that pointer instead... well, it's likely just a matter of taste.

OK

Quoted 63 lines
Quoted 63 lines
s390x/topology.c +15 -0
+        tc++;
+    }
+    report(n == number_of_cpus, "Number of CPUs  : %d expect %d", n, 
 number_of_cpus);
+    /*
+     * For KVM we accept
+     * - only 1 type of CPU
+     * - only horizontal topology
+     * - only dedicated CPUs
+     * This leads to expect the number of entries of level 0 CPU
+     * Topology Level Entry (TLE) to be:
+     * 1 + (number_of_cpus - 1)  / arch_topo_lvl[0]
+     *
+     * For z/VM or LPAR this number can only be greater if different
+     * polarity, CPU types because there may be a nested level 0 CPU TLE
+     * for each of the CPU/polarity/sharing types in a level 1 

container TLE.
+     */
+    n =  (number_of_cpus - 1)  / arch_topo_lvl[0];
+    report(stsi_nested_lvl[0] >=  n + 1,
+           "CPU Type TLE    : %d expect %d", stsi_nested_lvl[0], n + 1);

s390x/topology.c +15 -0
+
+    /* For each level found in STSI */
+    for (i = 1; i < CPU_TOPOLOGY_MAX_LEVEL; i++) {
+        /*
+         * For non QEMU/KVM hypervisor the concatenation of the levels
+         * above level 1 are architecture dependent.
+         * Skip these checks.
+         */
+        if (!host_is_kvm() && sel2 != 2)
+            continue;
+
+        /* For QEMU/KVM we expect a simple calculation */
+        if (sel2 > i) {
+            report(stsi_nested_lvl[i] ==  n + 1,
+                   "Container TLE  %d: %d expect %d", i, 

stsi_nested_lvl[i], n + 1);
+            n /= arch_topo_lvl[i];
+        }
+    }

s390x/topology.c +22 -0
+
+    report_prefix_pop();
+}
+
+/*
+ * check_sysinfo_15_1_x
+ * @info: pointer to the STSI info structure
+ * @sel2: the selector giving the topology level to check
+ *
+ * Check if the validity of the STSI instruction and then
+ * calls specific checks on the information buffer.
+ */
+static void check_sysinfo_15_1_x(struct sysinfo_15_1_x *info, int sel2)
+{
+    int ret;
+
+    report_prefix_pushf("mnested %d 15_1_%d", max_nested_lvl, sel2);
+
+    ret = stsi(pagebuf, 15, 1, sel2);
+    if (max_nested_lvl >= sel2) {
+        report(!ret, "Valid stsi instruction");
+    } else {
Quoted 8 lines
Quoted 6 lines

+        report(ret, "Invalid stsi instruction");
+        goto end;
+    }
+
+    stsi_check_maxcpus(info);
+    stsi_check_tle_coherency(info, sel2);

You could also move the two stsi_check_* calls into the first part of the if-statement, then you could get rid of the goto in the second part.

Thanks, yes.

Quoted 89 lines
Quoted 89 lines
s390x/topology.c +22 -0
+end:
+    report_prefix_pop();
+}
+
+static int sclp_get_mnest(void)
+{
+    ReadInfo *sccb = (void *)_sccb;
+
+    sclp_mark_busy();
+    memset(_sccb, 0, PAGE_SIZE);
+    sccb->h.length = PAGE_SIZE;
+
+    sclp_service_call(SCLP_CMDW_READ_SCP_INFO, sccb);
+    assert(sccb->h.response_code == SCLP_RC_NORMAL_READ_COMPLETION);
+
+    return sccb->stsi_parm;
+}
+
+/*
+ * test_stsi
+ *
+ * Retrieves the maximum nested topology level supported by the 

architecture
+ * and the number of CPUs.
+ * Calls the checking for the STSI instruction in sel2 reverse level order
+ * from 6 (CPU_TOPOLOGY_MAX_LEVEL) to 2 to have the most interesting level,
+ * the one triggering a topology-change-report-pending condition, level 2,
+ * at the end of the report.
+ *
+ */

s390x/topology.c +55 -0
+static void test_stsi(void)
+{
+    int sel2;
+
+    max_nested_lvl = sclp_get_mnest();
+    report_info("SCLP maximum nested level : %d", max_nested_lvl);
+
+    number_of_cpus = sclp_get_cpu_num();
+    report_info("SCLP number of CPU: %d", number_of_cpus);
+
+    /* STSI selector 2 can takes values between 2 and 6 */
+    for (sel2 = 6; sel2 >= 2; sel2--)
+        check_sysinfo_15_1_x((struct sysinfo_15_1_x *)pagebuf, sel2);
+}
+
+/*
+ * parse_topology_args
+ * @argc: number of arguments
+ * @argv: argument array
+ *
+ * This function initialize the architecture topology levels
+ * which should be the same as the one provided by the hypervisor.
+ *
+ * We use the current names found in IBM/Z literature, Linux and QEMU:
+ * cores, sockets/packages, books, drawers and nodes to facilitate the
+ * human machine interface but store the result in a machine abstract
+ * array of architecture topology levels.
+ * Note that when QEMU uses socket as a name for the topology level 1
+ * Linux uses package or physical_package.
+ */
+static void parse_topology_args(int argc, char **argv)
+{
+    int i;
+
+    report_info("%d arguments", argc);
+    for (i = 1; i < argc; i++) {
+        if (!strcmp("-cores", argv[i])) {
+            i++;
+            if (i >= argc)
+                report_abort("-cores needs a parameter");
+            arch_topo_lvl[0] = atol(argv[i]);
+            report_info("cores: %d", arch_topo_lvl[0]);
+        } else if (!strcmp("-sockets", argv[i])) {
+            i++;
+            if (i >= argc)
+                report_abort("-sockets needs a parameter");
+            arch_topo_lvl[1] = atol(argv[i]);
+            report_info("sockets: %d", arch_topo_lvl[1]);
+        } else if (!strcmp("-books", argv[i])) {
+            i++;
+            if (i >= argc)
+                report_abort("-books needs a parameter");
+            arch_topo_lvl[2] = atol(argv[i]);
+            report_info("books: %d", arch_topo_lvl[2]);
+        } else if (!strcmp("-drawers", argv[i])) {
Quoted 8 lines
Quoted 6 lines

+            i++;
+            if (i >= argc)
+                report_abort("-drawers needs a parameter"); +            arch_topo_lvl[3] = atol(argv[i]);
+            report_info("drawers: %d", arch_topo_lvl[3]); +        }

Maybe abort on unkown parameters, to avoid that typos go unnoticed?

Yes, better.
Thanks,

Regards.
Pierre

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
#9 lore raw

Re: [kvm-unit-tests PATCH v6 2/2] s390x: topology: Checking Configuration Topology Information

In reply to Pierre Morel
Quoted 67 lines · Pierre Morel

STSI with function code 15 is used to store the CPU configuration topology.

We retrieve the maximum nested level with SCLP and use the topology tree provided by the drawers, books, sockets, cores arguments.

We check :
- if the topology stored is coherent between the QEMU -smp parameters and kernel parameters.
- the number of CPUs
- the maximum number of CPUs
- the number of containers of each levels for every STSI(15.1.x) instruction allowed by the machine.

---
 lib/s390x/sclp.h    |   3 +-
 lib/s390x/stsi.h    |  44 ++++++++
 s390x/topology.c    | 244 ++++++++++++++++++++++++++++++++++++++++++++
 s390x/unittests.cfg |   1 +
 4 files changed, 291 insertions(+), 1 deletion(-)
lib/s390x/sclp.h +2 -1
index 853529b..6ecfb0a 100644
@@ -150,7 +150,8 @@ typedef struct ReadInfo {
 	SCCBHeader h;
 	uint16_t rnmax;
 	uint8_t rnsize;
-	uint8_t  _reserved1[16 - 11];       /* 11-15 */
+	uint8_t  _reserved1[15 - 11];       /* 11-14 */
+	uint8_t stsi_parm;                  /* 15-15 */
 	uint16_t entries_cpu;               /* 16-17 */
 	uint16_t offset_cpu;                /* 18-19 */
 	uint8_t  _reserved2[24 - 20];       /* 20-23 */
lib/s390x/stsi.h +22 -0
index bebc492..8dbbfc2 100644
@@ -29,4 +29,48 @@ struct sysinfo_3_2_2 {
 	uint8_t ext_names[8][256];
 };
 
+struct topology_core {
+	uint8_t nl;
+	uint8_t reserved1[3];
+	uint8_t reserved4:5;
+	uint8_t d:1;
+	uint8_t pp:2;
+	uint8_t type;
+	uint16_t origin;
+	uint64_t mask;
+};
+
+struct topology_container {
+	uint8_t nl;
+	uint8_t reserved[6];
+	uint8_t id;
+};
+
+union topology_entry {
+	uint8_t nl;
+	struct topology_core cpu;
+	struct topology_container container;
+};

lib/s390x/stsi.h struct sysinfo_3_2_2 { · line 63

Quoted 10 lines
lib/s390x/stsi.h +10 -0
+
+#define CPU_TOPOLOGY_MAX_LEVEL 6
+struct sysinfo_15_1_x {
+	uint8_t reserved0[2];
+	uint16_t length;
+	uint8_t mag[CPU_TOPOLOGY_MAX_LEVEL];
+	uint8_t reserved0a;
+	uint8_t mnest;
+	uint8_t reserved0c[4];
+	union topology_entry tle[0];

I'd use [] since it's C99.

Quoted 18 lines
lib/s390x/stsi.h +12 -0
+};
+
+static inline int cpus_in_tle_mask(uint64_t val)
+{
+	int i, n;
+
+	for (i = 0, n = 0; i < 64; i++, val >>= 1)
+		if (val & 0x01)
+			n++;
+	return n;
+}
+
 #endif  /* _S390X_STSI_H_ */
s390x/topology.c +0 -0
index 20f7ba2..f21c653 100644

s390x/topology.c line 25

Quoted 10 lines
s390x/topology.c +7 -0
@@ -16,6 +16,18 @@
 #include <smp.h>
 #include <sclp.h>
 #include <s390x/hardware.h>
+#include <s390x/stsi.h>
+
+static uint8_t pagebuf[PAGE_SIZE * 2] __attribute__((aligned(PAGE_SIZE * 2)));
+
+static int max_nested_lvl;
+static int number_of_cpus;
+static int max_cpus = 1;

IMO you shouldn't initialize this, it's a bit confusing because it gets modified later. I think it's good to initialize globals where one would expect it, so in main, or at the very top of the individual test case.

Quoted 82 lines
s390x/topology.c +75 -0
+
+/* Topology level as defined by architecture */
+static int arch_topo_lvl[CPU_TOPOLOGY_MAX_LEVEL];
+/* Topology nested level as reported in STSI */
+static int stsi_nested_lvl[CPU_TOPOLOGY_MAX_LEVEL];
 
 #define PTF_REQ_HORIZONTAL	0
 #define PTF_REQ_VERTICAL	1
@@ -122,11 +134,241 @@ end:
 	report_prefix_pop();
 }
 
+/*
+ * stsi_check_maxcpus
+ * @info: Pointer to the stsi information
+ *
+ * The product of the numbers of containers per level
+ * is the maximum number of CPU allowed by the machine.
+ */
+static void stsi_check_maxcpus(struct sysinfo_15_1_x *info)
+{
+	int n, i;
+
+	report_prefix_push("maximum cpus");
+
+	for (i = 0, n = 1; i < CPU_TOPOLOGY_MAX_LEVEL; i++) {
+		report_info("Mag%d: %d", CPU_TOPOLOGY_MAX_LEVEL - i, info->mag[i]);
+		n *= info->mag[i] ? info->mag[i] : 1;
+	}
+	report(n == max_cpus, "Maximum CPUs %d expected %d", n, max_cpus);
+
+	report_prefix_pop();
+}
+
+/*
+ * stsi_check_tle_coherency
+ * @info: Pointer to the stsi information
+ * @sel2: Topology level to check.
+ *
+ * We verify that we get the expected number of Topology List Entry
+ * containers for a specific level.
+ */
+static void stsi_check_tle_coherency(struct sysinfo_15_1_x *info, int sel2)
+{
+	struct topology_container *tc, *end;
+	struct topology_core *cpus;
+	int n = 0;
+	int i;
+
+	report_prefix_push("TLE coherency");
+
+	tc = &info->tle[0].container;
+	end = (struct topology_container *)((unsigned long)info + info->length);
+
+	for (i = 0; i < CPU_TOPOLOGY_MAX_LEVEL; i++)
+		stsi_nested_lvl[i] = 0;
+
+	while (tc < end) {
+		if (tc->nl > 5) {
+			report_abort("Unexpected TL Entry: tle->nl: %d", tc->nl);
+			return;
+		}
+		if (tc->nl == 0) {
+			cpus = (struct topology_core *)tc;
+			n += cpus_in_tle_mask(cpus->mask);
+			report_info("cpu type %02x  d: %d pp: %d", cpus->type, cpus->d, cpus->pp);
+			report_info("origin : %04x mask %016lx", cpus->origin, cpus->mask);
+		}
+
+		stsi_nested_lvl[tc->nl]++;
+		report_info("level %d: lvl: %d id: %d cnt: %d",
+			    tc->nl, tc->nl, tc->id, stsi_nested_lvl[tc->nl]);
+
+		/* trick: CPU TLEs are twice the size of containers TLE */
+		if (tc->nl == 0)
+			tc++;
+		tc++;
+	}
+	report(n == number_of_cpus, "Number of CPUs  : %d expect %d", n, number_of_cpus);
+	/*
+	 * For KVM we accept
+	 * - only 1 type of CPU

s390x/topology.c end: · line 216

Quoted 10 lines
s390x/topology.c +10 -0
+	 * - only horizontal topology
+	 * - only dedicated CPUs
+	 * This leads to expect the number of entries of level 0 CPU
+	 * Topology Level Entry (TLE) to be:
+	 * 1 + (number_of_cpus - 1)  / arch_topo_lvl[0]
+	 *
+	 * For z/VM or LPAR this number can only be greater if different
+	 * polarity, CPU types because there may be a nested level 0 CPU TLE
+	 * for each of the CPU/polarity/sharing types in a level 1 container TLE.
+	 */

IMO you should only test what is defined by the architecture. This behavior isn't even dependent on KVM, but on user space, ok effectively this is QEMU but you never know. And that behavior could change in the future.

Quoted 14 lines
s390x/topology.c +14 -0
+	n =  (number_of_cpus - 1)  / arch_topo_lvl[0];
+	report(stsi_nested_lvl[0] >=  n + 1,
+	       "CPU Type TLE    : %d expect %d", stsi_nested_lvl[0], n + 1);
+
+	/* For each level found in STSI */
+	for (i = 1; i < CPU_TOPOLOGY_MAX_LEVEL; i++) {
+		/*
+		 * For non QEMU/KVM hypervisor the concatenation of the levels
+		 * above level 1 are architecture dependent.
+		 * Skip these checks.
+		 */
+		if (!host_is_kvm() && sel2 != 2)
+			continue;
+

s390x/topology.c end: · line 240

Quoted 10 lines
s390x/topology.c +10 -0
+		/* For QEMU/KVM we expect a simple calculation */
+		if (sel2 > i) {
+			report(stsi_nested_lvl[i] ==  n + 1,
+			       "Container TLE  %d: %d expect %d", i, stsi_nested_lvl[i], n + 1);
+			n /= arch_topo_lvl[i];
+		}
+	}
+
+	report_prefix_pop();
+}

I think you could make this test stricter and more readable by using recursion. You have a function check_container which:
* checks that each child has nesting level one level lower
* calls check_container on the child
* gets back the number of cpus in the child
* gets back the next child
* if the current "child" isn't actually a child but has one higher nesting level we're done
* check if sum of child cpus makes sense
* error out on any anomaly (e.g required 0 bits)
* return "child" and sum of cpus

For CPU entries you have a special checking function:
* check as much as you can
* calculate number of cpus in mask
* you can also test if the actual cpus exist by:
* building a bitmap of all cpus first before the test
* checking if the bits in the mask are a subset of existing cpus

Quoted 33 lines
s390x/topology.c +33 -0
+
+/*
+ * check_sysinfo_15_1_x
+ * @info: pointer to the STSI info structure
+ * @sel2: the selector giving the topology level to check
+ *
+ * Check if the validity of the STSI instruction and then
+ * calls specific checks on the information buffer.
+ */
+static void check_sysinfo_15_1_x(struct sysinfo_15_1_x *info, int sel2)
+{
+	int ret;
+
+	report_prefix_pushf("mnested %d 15_1_%d", max_nested_lvl, sel2);
+
+	ret = stsi(pagebuf, 15, 1, sel2);
+	if (max_nested_lvl >= sel2) {
+		report(!ret, "Valid stsi instruction");
+	} else {
+		report(ret, "Invalid stsi instruction");
+		goto end;
+	}
+
+	stsi_check_maxcpus(info);
+	stsi_check_tle_coherency(info, sel2);
+
+end:
+	report_prefix_pop();
+}
+
+static int sclp_get_mnest(void)
+{
+	ReadInfo *sccb = (void *)_sccb;

s390x/topology.c end: · line 283

Quoted 10 lines
s390x/topology.c +10 -0
+
+	sclp_mark_busy();
+	memset(_sccb, 0, PAGE_SIZE);
+	sccb->h.length = PAGE_SIZE;
+
+	sclp_service_call(SCLP_CMDW_READ_SCP_INFO, sccb);
+	assert(sccb->h.response_code == SCLP_RC_NORMAL_READ_COMPLETION);
+
+	return sccb->stsi_parm;
+}

I think it would be better if you just add a function sclp_get_stsi_max_sel to lib/s390x/sclp.[ch] that reads the value from read_info and calculates the maximum selector value.

Quoted 43 lines
s390x/topology.c +43 -0
+
+/*
+ * test_stsi
+ *
+ * Retrieves the maximum nested topology level supported by the architecture
+ * and the number of CPUs.
+ * Calls the checking for the STSI instruction in sel2 reverse level order
+ * from 6 (CPU_TOPOLOGY_MAX_LEVEL) to 2 to have the most interesting level,
+ * the one triggering a topology-change-report-pending condition, level 2,
+ * at the end of the report.
+ *
+ */
+static void test_stsi(void)
+{
+	int sel2;
+
+	max_nested_lvl = sclp_get_mnest();
+	report_info("SCLP maximum nested level : %d", max_nested_lvl);
+
+	number_of_cpus = sclp_get_cpu_num();
+	report_info("SCLP number of CPU: %d", number_of_cpus);
+
+	/* STSI selector 2 can takes values between 2 and 6 */
+	for (sel2 = 6; sel2 >= 2; sel2--)
+		check_sysinfo_15_1_x((struct sysinfo_15_1_x *)pagebuf, sel2);
+}
+
+/*
+ * parse_topology_args
+ * @argc: number of arguments
+ * @argv: argument array
+ *
+ * This function initialize the architecture topology levels
+ * which should be the same as the one provided by the hypervisor.
+ *
+ * We use the current names found in IBM/Z literature, Linux and QEMU:
+ * cores, sockets/packages, books, drawers and nodes to facilitate the
+ * human machine interface but store the result in a machine abstract
+ * array of architecture topology levels.
+ * Note that when QEMU uses socket as a name for the topology level 1
+ * Linux uses package or physical_package.
+ */
+static void parse_topology_args(int argc, char **argv)

s390x/topology.c end: · line 336

Quoted 10 lines
s390x/topology.c +10 -0
+{
+	int i;
+
+	report_info("%d arguments", argc);
+	for (i = 1; i < argc; i++) {
+		if (!strcmp("-cores", argv[i])) {
+			i++;
+			if (i >= argc)
+				report_abort("-cores needs a parameter");
+			arch_topo_lvl[0] = atol(argv[i]);

You don't do any error checking of the number parsing, and don't check the sign. You could use parse_unsigned in spec_ex.c, should maybe be in a lib somewhere instead.

Quoted 10 lines
s390x/topology.c +10 -0
+			report_info("cores: %d", arch_topo_lvl[0]);
+		} else if (!strcmp("-sockets", argv[i])) {
+			i++;
+			if (i >= argc)
+				report_abort("-sockets needs a parameter");
+			arch_topo_lvl[1] = atol(argv[i]);
+			report_info("sockets: %d", arch_topo_lvl[1]);
+		} else if (!strcmp("-books", argv[i])) {
+			i++;
+			if (i >= argc)

s390x/topology.c end: · line 356

Quoted 10 lines
s390x/topology.c +10 -0
+				report_abort("-books needs a parameter");
+			arch_topo_lvl[2] = atol(argv[i]);
+			report_info("books: %d", arch_topo_lvl[2]);
+		} else if (!strcmp("-drawers", argv[i])) {
+			i++;
+			if (i >= argc)
+				report_abort("-drawers needs a parameter");
+			arch_topo_lvl[3] = atol(argv[i]);
+			report_info("drawers: %d", arch_topo_lvl[3]);
+		}

Might be nice to error out if no flags match.
I'm guessing, currently, -sockets -cores 1 would "parse", but the result is nonsense

s390x/topology.c end: · line 357

Quoted 1 line
s390x/topology.c +1 -0
+	}

You can also get rid of some code redundancy:

        char *levels[] = { "cores", "sockets", "books", "drawers" };
        for (...) {
                char *flag = argv[i];
                int level;

                if (flag[0] != '-')
                        report_abort("Expected ...");
                flag++;
                for (level = 0; ARRAY_SIZE(levels); level++) {
                        if (!strcmp(levels[level]), flag)
                                break;
                }
                if (level == ARRAY_SIZE(levels))
                        report_abort("Unknown ...");

                arch_topo_lvl[level] = atol(argv[++i]);
                report_info("%s: %d", levels[level], arch_topo_lvl[level]);
        }

s390x/topology.c end: · line 363

Quoted 6 lines
s390x/topology.c +6 -0
+
+	for (i = 0; i < CPU_TOPOLOGY_MAX_LEVEL; i++) {
+		if (!arch_topo_lvl[i])
+			arch_topo_lvl[i] = 1;
+		max_cpus *= arch_topo_lvl[i];
+	}

Might be nice to split this function up into two, one that parses the values into an array and one that calculates max_cpus.
Then you can do max_cpus = calculate_max_cpus or whatever and when someone is looking for where max_cpus is defined searching for "max_cpus =" works.

Quoted 29 lines
s390x/topology.c +5 -0
+}
+
 static struct {
 	const char *name;
 	void (*func)(void);
 } tests[] = {
 	{ "PTF", test_ptf},
+	{ "STSI", test_stsi},
 	{ NULL, NULL }
 };
 
@@ -136,6 +378,8 @@ int main(int argc, char *argv[])
 
 	report_prefix_push("CPU Topology");
 
+	parse_topology_args(argc, argv);
+
 	if (!test_facility(11)) {
 		report_skip("Topology facility not present");
 		goto end;
s390x/unittests.cfg +1 -0
index 3530cc4..b697aca 100644
@@ -211,3 +211,4 @@ smp = 2
 
 [topology]
 file = topology.elf
+extra_params=-smp 5,drawers=3,books=3,sockets=4,cores=4,maxcpus=144 -append '-drawers 3 -books 3 -sockets 4 -cores 4'
#10 lore raw

Re: [kvm-unit-tests PATCH v6 2/2] s390x: topology: Checking Configuration Topology Information

Quoted 72 lines · Nina Schoetterl-Glausch
Quoted 71 lines · Pierre Morel

STSI with function code 15 is used to store the CPU configuration topology.

We retrieve the maximum nested level with SCLP and use the topology tree provided by the drawers, books, sockets, cores arguments.

We check :
- if the topology stored is coherent between the QEMU -smp parameters and kernel parameters.
- the number of CPUs
- the maximum number of CPUs
- the number of containers of each levels for every STSI(15.1.x) instruction allowed by the machine.

---
  lib/s390x/sclp.h    |   3 +-
  lib/s390x/stsi.h    |  44 ++++++++
  s390x/topology.c    | 244 ++++++++++++++++++++++++++++++++++++++++++++
  s390x/unittests.cfg |   1 +
  4 files changed, 291 insertions(+), 1 deletion(-)
lib/s390x/sclp.h +2 -1
index 853529b..6ecfb0a 100644
@@ -150,7 +150,8 @@ typedef struct ReadInfo {
  	SCCBHeader h;
  	uint16_t rnmax;
  	uint8_t rnsize;
-	uint8_t  _reserved1[16 - 11];       /* 11-15 */
+	uint8_t  _reserved1[15 - 11];       /* 11-14 */
+	uint8_t stsi_parm;                  /* 15-15 */
  	uint16_t entries_cpu;               /* 16-17 */
  	uint16_t offset_cpu;                /* 18-19 */
  	uint8_t  _reserved2[24 - 20];       /* 20-23 */
lib/s390x/stsi.h +26 -0
index bebc492..8dbbfc2 100644
@@ -29,4 +29,48 @@ struct sysinfo_3_2_2 {
  	uint8_t ext_names[8][256];
  };
 
+struct topology_core {
+	uint8_t nl;
+	uint8_t reserved1[3];
+	uint8_t reserved4:5;
+	uint8_t d:1;
+	uint8_t pp:2;
+	uint8_t type;
+	uint16_t origin;
+	uint64_t mask;
+};
+
+struct topology_container {
+	uint8_t nl;
+	uint8_t reserved[6];
+	uint8_t id;
+};
+
+union topology_entry {
+	uint8_t nl;
+	struct topology_core cpu;
+	struct topology_container container;
+};
+
+#define CPU_TOPOLOGY_MAX_LEVEL 6
+struct sysinfo_15_1_x {
+	uint8_t reserved0[2];
Quoted 8 lines

lib/s390x/stsi.h struct sysinfo_3_2_2 { · line 63

Quoted 6 lines
lib/s390x/stsi.h +6 -0
+	uint16_t length;
+	uint8_t mag[CPU_TOPOLOGY_MAX_LEVEL];
+	uint8_t reserved0a;
+	uint8_t mnest;
+	uint8_t reserved0c[4];
+	union topology_entry tle[0];

I'd use [] since it's C99.

OK

Quoted 25 lines
Quoted 25 lines
lib/s390x/stsi.h +12 -0
+};
+
+static inline int cpus_in_tle_mask(uint64_t val)
+{
+	int i, n;
+
+	for (i = 0, n = 0; i < 64; i++, val >>= 1)
+		if (val & 0x01)
+			n++;
+	return n;
+}
+
  #endif  /* _S390X_STSI_H_ */
s390x/topology.c +3 -0
index 20f7ba2..f21c653 100644
@@ -16,6 +16,18 @@
  #include <smp.h>
  #include <sclp.h>
  #include <s390x/hardware.h>
+#include <s390x/stsi.h>
+
+static uint8_t pagebuf[PAGE_SIZE * 2] __attribute__((aligned(PAGE_SIZE * 2)));
Quoted 8 lines

s390x/topology.c line 25

Quoted 4 lines
s390x/topology.c +4 -0
+
+static int max_nested_lvl;
+static int number_of_cpus;
+static int max_cpus = 1;

IMO you shouldn't initialize this, it's a bit confusing because it gets modified later. I think it's good to initialize globals where one would expect it, so in main, or at the very top of the individual test case.

OK

Quoted 89 lines
Quoted 89 lines
s390x/topology.c +81 -0
+
+/* Topology level as defined by architecture */
+static int arch_topo_lvl[CPU_TOPOLOGY_MAX_LEVEL];
+/* Topology nested level as reported in STSI */
+static int stsi_nested_lvl[CPU_TOPOLOGY_MAX_LEVEL];
 
  #define PTF_REQ_HORIZONTAL	0
  #define PTF_REQ_VERTICAL	1
@@ -122,11 +134,241 @@ end:
  	report_prefix_pop();
  }
 
+/*
+ * stsi_check_maxcpus
+ * @info: Pointer to the stsi information
+ *
+ * The product of the numbers of containers per level
+ * is the maximum number of CPU allowed by the machine.
+ */
+static void stsi_check_maxcpus(struct sysinfo_15_1_x *info)
+{
+	int n, i;
+
+	report_prefix_push("maximum cpus");
+
+	for (i = 0, n = 1; i < CPU_TOPOLOGY_MAX_LEVEL; i++) {
+		report_info("Mag%d: %d", CPU_TOPOLOGY_MAX_LEVEL - i, info->mag[i]);
+		n *= info->mag[i] ? info->mag[i] : 1;
+	}
+	report(n == max_cpus, "Maximum CPUs %d expected %d", n, max_cpus);
+
+	report_prefix_pop();
+}
+
+/*
+ * stsi_check_tle_coherency
+ * @info: Pointer to the stsi information
+ * @sel2: Topology level to check.
+ *
+ * We verify that we get the expected number of Topology List Entry
+ * containers for a specific level.
+ */
+static void stsi_check_tle_coherency(struct sysinfo_15_1_x *info, int sel2)
+{
+	struct topology_container *tc, *end;
+	struct topology_core *cpus;
+	int n = 0;
+	int i;
+
+	report_prefix_push("TLE coherency");
+
+	tc = &info->tle[0].container;
+	end = (struct topology_container *)((unsigned long)info + info->length);
+
+	for (i = 0; i < CPU_TOPOLOGY_MAX_LEVEL; i++)
+		stsi_nested_lvl[i] = 0;
+
+	while (tc < end) {
+		if (tc->nl > 5) {
+			report_abort("Unexpected TL Entry: tle->nl: %d", tc->nl);
+			return;
+		}
+		if (tc->nl == 0) {
+			cpus = (struct topology_core *)tc;
+			n += cpus_in_tle_mask(cpus->mask);
+			report_info("cpu type %02x  d: %d pp: %d", cpus->type, cpus->d, cpus->pp);
+			report_info("origin : %04x mask %016lx", cpus->origin, cpus->mask);
+		}
+
+		stsi_nested_lvl[tc->nl]++;
+		report_info("level %d: lvl: %d id: %d cnt: %d",
+			    tc->nl, tc->nl, tc->id, stsi_nested_lvl[tc->nl]);
+
+		/* trick: CPU TLEs are twice the size of containers TLE */
+		if (tc->nl == 0)
+			tc++;
+		tc++;
+	}
+	report(n == number_of_cpus, "Number of CPUs  : %d expect %d", n, number_of_cpus);
+	/*
+	 * For KVM we accept
+	 * - only 1 type of CPU
+	 * - only horizontal topology
+	 * - only dedicated CPUs
+	 * This leads to expect the number of entries of level 0 CPU
+	 * Topology Level Entry (TLE) to be:
+	 * 1 + (number_of_cpus - 1)  / arch_topo_lvl[0]
+	 *
Quoted 8 lines

s390x/topology.c end: · line 216

Quoted 4 lines
s390x/topology.c +4 -0
+	 * For z/VM or LPAR this number can only be greater if different
+	 * polarity, CPU types because there may be a nested level 0 CPU TLE
+	 * for each of the CPU/polarity/sharing types in a level 1 container TLE.
+	 */

IMO you should only test what is defined by the architecture. This behavior isn't even dependent on KVM, but on user space, ok effectively this is QEMU but you never know. And that behavior could change in the future.

Right, I will let fall this check as I did not find any specification about how the levels collapse in the documentation. And the measures on LPAR gives unexpected results.

Quoted 37 lines
Quoted 15 lines
s390x/topology.c +14 -0
+	n =  (number_of_cpus - 1)  / arch_topo_lvl[0];
+	report(stsi_nested_lvl[0] >=  n + 1,
+	       "CPU Type TLE    : %d expect %d", stsi_nested_lvl[0], n + 1);
+
+	/* For each level found in STSI */
+	for (i = 1; i < CPU_TOPOLOGY_MAX_LEVEL; i++) {
+		/*
+		 * For non QEMU/KVM hypervisor the concatenation of the levels
+		 * above level 1 are architecture dependent.
+		 * Skip these checks.
+		 */
+		if (!host_is_kvm() && sel2 != 2)
+			continue;
+

s390x/topology.c end: · line 240

Quoted 10 lines
s390x/topology.c +10 -0
+		/* For QEMU/KVM we expect a simple calculation */
+		if (sel2 > i) {
+			report(stsi_nested_lvl[i] ==  n + 1,
+			       "Container TLE  %d: %d expect %d", i, stsi_nested_lvl[i], n + 1);
+			n /= arch_topo_lvl[i];
+		}
+	}
+
+	report_prefix_pop();
+}

I think you could make this test stricter and more readable by using recursion. You have a function check_container which:
* checks that each child has nesting level one level lower
* calls check_container on the child
* gets back the number of cpus in the child
* gets back the next child
* if the current "child" isn't actually a child but has one higher nesting level we're done
* check if sum of child cpus makes sense
* error out on any anomaly (e.g required 0 bits)
* return "child" and sum of cpus

Quoted 6 lines

For CPU entries you have a special checking function:
* check as much as you can
* calculate number of cpus in mask
* you can also test if the actual cpus exist by:
* building a bitmap of all cpus first before the test
* checking if the bits in the mask are a subset of existing cpus

That is very interesting however I wonder if we should not propose the test without these checks first, because it is quite finish then, and expand the test in a second series.

IMO having a test now even not complete and get more intense checking later is better than to have nothing until we finally have all tests we can think about in the framework.

Quoted 39 lines
Quoted 39 lines
s390x/topology.c +39 -0
+
+/*
+ * check_sysinfo_15_1_x
+ * @info: pointer to the STSI info structure
+ * @sel2: the selector giving the topology level to check
+ *
+ * Check if the validity of the STSI instruction and then
+ * calls specific checks on the information buffer.
+ */
+static void check_sysinfo_15_1_x(struct sysinfo_15_1_x *info, int sel2)
+{
+	int ret;
+
+	report_prefix_pushf("mnested %d 15_1_%d", max_nested_lvl, sel2);
+
+	ret = stsi(pagebuf, 15, 1, sel2);
+	if (max_nested_lvl >= sel2) {
+		report(!ret, "Valid stsi instruction");
+	} else {
+		report(ret, "Invalid stsi instruction");
+		goto end;
+	}
+
+	stsi_check_maxcpus(info);
+	stsi_check_tle_coherency(info, sel2);
+
+end:
+	report_prefix_pop();
+}
+
+static int sclp_get_mnest(void)
+{
+	ReadInfo *sccb = (void *)_sccb;
+
+	sclp_mark_busy();
+	memset(_sccb, 0, PAGE_SIZE);
+	sccb->h.length = PAGE_SIZE;
+
+	sclp_service_call(SCLP_CMDW_READ_SCP_INFO, sccb);
Quoted 8 lines

s390x/topology.c end: · line 283

Quoted 4 lines
s390x/topology.c +4 -0
+	assert(sccb->h.response_code == SCLP_RC_NORMAL_READ_COMPLETION);
+
+	return sccb->stsi_parm;
+}

I think it would be better if you just add a function sclp_get_stsi_max_sel to lib/s390x/sclp.[ch] that reads the value from read_info and calculates the maximum selector value.

OK

Quoted 49 lines
Quoted 49 lines
s390x/topology.c +48 -0
+
+/*
+ * test_stsi
+ *
+ * Retrieves the maximum nested topology level supported by the architecture
+ * and the number of CPUs.
+ * Calls the checking for the STSI instruction in sel2 reverse level order
+ * from 6 (CPU_TOPOLOGY_MAX_LEVEL) to 2 to have the most interesting level,
+ * the one triggering a topology-change-report-pending condition, level 2,
+ * at the end of the report.
+ *
+ */
+static void test_stsi(void)
+{
+	int sel2;
+
+	max_nested_lvl = sclp_get_mnest();
+	report_info("SCLP maximum nested level : %d", max_nested_lvl);
+
+	number_of_cpus = sclp_get_cpu_num();
+	report_info("SCLP number of CPU: %d", number_of_cpus);
+
+	/* STSI selector 2 can takes values between 2 and 6 */
+	for (sel2 = 6; sel2 >= 2; sel2--)
+		check_sysinfo_15_1_x((struct sysinfo_15_1_x *)pagebuf, sel2);
+}
+
+/*
+ * parse_topology_args
+ * @argc: number of arguments
+ * @argv: argument array
+ *
+ * This function initialize the architecture topology levels
+ * which should be the same as the one provided by the hypervisor.
+ *
+ * We use the current names found in IBM/Z literature, Linux and QEMU:
+ * cores, sockets/packages, books, drawers and nodes to facilitate the
+ * human machine interface but store the result in a machine abstract
+ * array of architecture topology levels.
+ * Note that when QEMU uses socket as a name for the topology level 1
+ * Linux uses package or physical_package.
+ */
+static void parse_topology_args(int argc, char **argv)
+{
+	int i;
+
+	report_info("%d arguments", argc);
+	for (i = 1; i < argc; i++) {
Quoted 8 lines

s390x/topology.c end: · line 336

Quoted 5 lines
s390x/topology.c +5 -0
+		if (!strcmp("-cores", argv[i])) {
+			i++;
+			if (i >= argc)
+				report_abort("-cores needs a parameter");
+			arch_topo_lvl[0] = atol(argv[i]);

You don't do any error checking of the number parsing, and don't check the sign. You could use parse_unsigned in spec_ex.c, should maybe be in a lib somewhere instead.

OK

Quoted 16 lines
Quoted 16 lines
s390x/topology.c +15 -0
+			report_info("cores: %d", arch_topo_lvl[0]);
+		} else if (!strcmp("-sockets", argv[i])) {
+			i++;
+			if (i >= argc)
+				report_abort("-sockets needs a parameter");
+			arch_topo_lvl[1] = atol(argv[i]);
+			report_info("sockets: %d", arch_topo_lvl[1]);
+		} else if (!strcmp("-books", argv[i])) {
+			i++;
+			if (i >= argc)
+				report_abort("-books needs a parameter");
+			arch_topo_lvl[2] = atol(argv[i]);
+			report_info("books: %d", arch_topo_lvl[2]);
+		} else if (!strcmp("-drawers", argv[i])) {
+			i++;
Quoted 8 lines

s390x/topology.c end: · line 356

Quoted 5 lines
s390x/topology.c +5 -0
+			if (i >= argc)
+				report_abort("-drawers needs a parameter");
+			arch_topo_lvl[3] = atol(argv[i]);
+			report_info("drawers: %d", arch_topo_lvl[3]);
+		}

Might be nice to error out if no flags match.
I'm guessing, currently, -sockets -cores 1 would "parse", but the result is nonsense

OK

Quoted 14 lines

s390x/topology.c end: · line 357

Quoted 2 lines
s390x/topology.c +1 -0
+	}

You can also get rid of some code redundancy:

         char *levels[] = { "cores", "sockets", "books", "drawers" };
         for (...) {
                 char *flag = argv[i];
                 int level;

                 if (flag[0] != '-')
                         report_abort("Expected ...");
                 flag++;
                 for (level = 0; ARRAY_SIZE(levels); level++) {
                         if (!strcmp(levels[level]), flag)
Quoted 8 lines
                                 break;
                 }
                 if (level == ARRAY_SIZE(levels))
                         report_abort("Unknown ...");

                 arch_topo_lvl[level] = atol(argv[++i]);
                 report_info("%s: %d", levels[level], arch_topo_lvl[level]);
         }

OK, I take it.

Quoted 11 lines

s390x/topology.c end: · line 363

Quoted 6 lines
s390x/topology.c +6 -0
+
+	for (i = 0; i < CPU_TOPOLOGY_MAX_LEVEL; i++) {
+		if (!arch_topo_lvl[i])
+			arch_topo_lvl[i] = 1;
+		max_cpus *= arch_topo_lvl[i];
+	}

Might be nice to split this function up into two, one that parses the values into an array and one that calculates max_cpus.
Then you can do max_cpus = calculate_max_cpus or whatever and when someone is looking for where max_cpus is defined searching for "max_cpus =" works.

OK

Regards,
Pierre

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen