Skip to contentSkip to conversation

s390x: add migration test for CMM

PATCH Superseded Nico Boehr 3 participants 11 messages May 9, 2022 – May 11, 2022

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Nico Boehr

author
#0 lore raw

cover s390x: add migration test for CMM

Upon migration, we expect the CMM page states to be preserved. Add a test which checks for that.

The new test gets a new file so the existing cmm test can still run when the prerequisites for running migration tests aren't given (netcat). Therefore, move some definitions to a common header to be able to re-use them.

Nico Boehr (2):
lib: s390x: add header for CMM related defines
s390x: add cmm migration test

 lib/s390x/asm/cmm.h   | 50 +++++++++++++++++++++++++++
 s390x/Makefile        |  1 +
 s390x/cmm-migration.c | 78 +++++++++++++++++++++++++++++++++++++++++++
 s390x/cmm.c           | 25 ++------------
 s390x/unittests.cfg   |  4 +++
 5 files changed, 136 insertions(+), 22 deletions(-)
 create mode 100644 lib/s390x/asm/cmm.h
 create mode 100644 s390x/cmm-migration.c
Signature · 1 line
2.31.1
author
#1 lore raw

PATCH 1/2 lib: s390x: add header for CMM related defines

Since we're going to need the definitions in an upcoming migration test for CMM, add a header for CMM related defines. It is based on arch/s390/include/asm/page-states.h from linux.

While at it, use the constants in existing calls to CMM related functions.

Also move essa() and test_availability() there to be able to use it outside cmm.c.

---
 lib/s390x/asm/cmm.h | 50 +++++++++++++++++++++++++++++++++++++++++++++
 s390x/cmm.c         | 25 +++--------------------
 2 files changed, 53 insertions(+), 22 deletions(-)
 create mode 100644 lib/s390x/asm/cmm.h
lib/s390x/asm/cmm.h new file +50 -0
new file mode 100644
index 000000000000..554a60031fbf
@@ -0,0 +1,50 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ *    Copyright IBM Corp. 2017, 2022
+ *    Author(s): Claudio Imbrenda <[email protected]>
+ *               Nico Boehr <[email protected]>
+ */
+#include <asm/interrupt.h>
+
+#ifndef PAGE_STATES_H
+#define PAGE_STATES_H
+
+#define ESSA_GET_STATE			0
+#define ESSA_SET_STABLE			1
+#define ESSA_SET_UNUSED			2
+#define ESSA_SET_VOLATILE		3
+#define ESSA_SET_POT_VOLATILE		4
+#define ESSA_SET_STABLE_RESIDENT	5
+#define ESSA_SET_STABLE_IF_RESIDENT	6
+#define ESSA_SET_STABLE_NODAT		7
+
+#define ESSA_MAX	ESSA_SET_STABLE_NODAT
+
+#define ESSA_USAGE_STABLE		0
+#define ESSA_USAGE_UNUSED		1
+#define ESSA_USAGE_POT_VOLATILE		2
+#define ESSA_USAGE_VOLATILE		3
+
+static unsigned long essa(uint8_t state, unsigned long paddr)
+{
+	uint64_t extr_state;
+
+	asm volatile(".insn rrf,0xb9ab0000,%[extr_state],%[addr],%[new_state],0" \
+			: [extr_state] "=d" (extr_state) \
+			: [addr] "a" (paddr), [new_state] "i" (state));
+
+	return (unsigned long)extr_state;
+}
+
+/*
+ * Unfortunately the availability is not indicated by stfl bits, but
+ * we have to try to execute it and test for an operation exception.
+ */
+static inline bool check_essa_available(void)
+{
+	expect_pgm_int();
+	essa(ESSA_GET_STATE, 0);
+	return clear_pgm_int() == 0;
+}
+
+#endif
s390x/cmm.c +3 -22
index c3f0c931ae36..af852838851e 100644
@@ -12,19 +12,10 @@
 #include <asm/asm-offsets.h>
 #include <asm/interrupt.h>
 #include <asm/page.h>
+#include <asm/cmm.h>
 
 static uint8_t pagebuf[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
 
-static unsigned long essa(uint8_t state, unsigned long paddr)
-{
-	uint64_t extr_state;
-
-	asm volatile(".insn rrf,0xb9ab0000,%[extr_state],%[addr],%[new_state],0"
-			: [extr_state] "=d" (extr_state)
-			: [addr] "a" (paddr), [new_state] "i" (state));
-	return (unsigned long)extr_state;
-}
-
 static void test_params(void)
 {
 	report_prefix_push("invalid ORC 8");
@@ -39,24 +30,14 @@ static void test_priv(void)
 	report_prefix_push("privileged");
 	expect_pgm_int();
 	enter_pstate();
-	essa(0, (unsigned long)pagebuf);
+	essa(ESSA_GET_STATE, (unsigned long)pagebuf);
 	check_pgm_int_code(PGM_INT_CODE_PRIVILEGED_OPERATION);
 	report_prefix_pop();
 }
 
-/* Unfortunately the availability is not indicated by stfl bits, but
- * we have to try to execute it and test for an operation exception.
- */
-static bool test_availability(void)
-{
-	expect_pgm_int();
-	essa(0, (unsigned long)pagebuf);
-	return clear_pgm_int() == 0;
-}
-
 int main(void)
 {
-	bool has_essa = test_availability();
+	bool has_essa = check_essa_available();
 
 	report_prefix_push("cmm");
 	if (!has_essa) {
Signature · 1 line
2.31.1
author
#2 lore raw

PATCH 2/2 s390x: add cmm migration test

In reply to Nico Boehr

When a VM is migrated, we expect the page states to be preserved. Add a test which checks for that.

---
 s390x/Makefile        |  1 +
 s390x/cmm-migration.c | 78 +++++++++++++++++++++++++++++++++++++++++++
 s390x/unittests.cfg   |  4 +++
 3 files changed, 83 insertions(+)
 create mode 100644 s390x/cmm-migration.c
s390x/Makefile +1 -0
index a8e04aa6fe4d..8ac0afdfd994 100644
@@ -32,6 +32,7 @@ tests += $(TEST_DIR)/epsw.elf
 tests += $(TEST_DIR)/adtl-status.elf
 tests += $(TEST_DIR)/migration.elf
 tests += $(TEST_DIR)/pv-attest.elf
+tests += $(TEST_DIR)/cmm-migration.elf
 
 pv-tests += $(TEST_DIR)/pv-diags.elf
 
s390x/cmm-migration.c new file +78 -0

78 lines not on this page · raw message

s390x/unittests.cfg +4 -0
index b456b2881448..625026d90e52 100644
@@ -176,3 +176,7 @@ extra_params = -cpu qemu,gs=off,vx=off
 file = migration.elf
 groups = migration
 smp = 2
+
+[cmm-migration]
+file = cmm-migration.elf
+groups = migration
Signature · 1 line
2.31.1
#3 lore raw

Re: [kvm-unit-tests PATCH v1 2/2] s390x: add cmm migration test

Quoted 67 lines · On Mon, 9 May 2022 14:08:05 +0200 Nico Boehr <[email protected]> wrote:

When a VM is migrated, we expect the page states to be preserved. Add a test which checks for that.

---
 s390x/Makefile        |  1 +
 s390x/cmm-migration.c | 78 +++++++++++++++++++++++++++++++++++++++++++
 s390x/unittests.cfg   |  4 +++
 3 files changed, 83 insertions(+)
 create mode 100644 s390x/cmm-migration.c
s390x/Makefile +1 -0
index a8e04aa6fe4d..8ac0afdfd994 100644
@@ -32,6 +32,7 @@ tests += $(TEST_DIR)/epsw.elf
 tests += $(TEST_DIR)/adtl-status.elf
 tests += $(TEST_DIR)/migration.elf
 tests += $(TEST_DIR)/pv-attest.elf
+tests += $(TEST_DIR)/cmm-migration.elf
 
 pv-tests += $(TEST_DIR)/pv-diags.elf
 
s390x/cmm-migration.c new file +38 -0
new file mode 100644
index 000000000000..4a7b50e40fc6
@@ -0,0 +1,78 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * CMM migration tests (ESSA)
+ *
+ * Copyright IBM Corp. 2022
+ *
+ * Authors:
+ *  Nico Boehr <[email protected]>
+ */
+
+#include <libcflat.h>
+#include <asm/asm-offsets.h>
+#include <asm/interrupt.h>
+#include <asm/page.h>
+#include <asm/cmm.h>
+#include <bitops.h>
+
+#define NUM_PAGES 128
+static uint8_t pagebuf[NUM_PAGES][PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
+
+static void test_migration(void)
+{
+	int i, state_mask, actual_state;
+	/*
+	 * Maps ESSA actions to states the page is allowed to be in after the
+	 * respective action was executed.
+	 */
+	int allowed_essa_state_masks[4] = {
+		BIT(ESSA_USAGE_STABLE),					/* ESSA_SET_STABLE */
+		BIT(ESSA_USAGE_UNUSED),					/* ESSA_SET_UNUSED */
+		BIT(ESSA_USAGE_VOLATILE),				/* ESSA_SET_VOLATILE */
+		BIT(ESSA_USAGE_VOLATILE) | BIT(ESSA_USAGE_POT_VOLATILE) /* ESSA_SET_POT_VOLATILE */
+	};
+
+	for (i = 0; i < NUM_PAGES; i++) {
+		switch(i % 4) {
+			case 0:
+				essa(ESSA_SET_STABLE, (unsigned long)pagebuf[i]);

s390x/cmm-migration.c line 48

Quoted 10 lines
s390x/cmm-migration.c new file +10 -0
+			break;
+			case 1:
+				essa(ESSA_SET_UNUSED, (unsigned long)pagebuf[i]);
+			break;
+			case 2:
+				essa(ESSA_SET_VOLATILE, (unsigned long)pagebuf[i]);
+			break;
+			case 3:
+				essa(ESSA_SET_POT_VOLATILE, (unsigned long)pagebuf[i]);
+			break;

const int essa_commands[4] = {ESSA_SET_STABLE, ESSA_SET_UNUSED, ...

for (i = 0; i < NUM_PAGES; i++)

	essa(essa_commands[i % 4], ...

I think it would look more compact and more readable

Quoted 42 lines
s390x/cmm-migration.c new file +30 -0
+		}
+	}
+
+	puts("Please migrate me, then press return\n");
+	(void)getchar();
+
+	for (i = 0; i < NUM_PAGES; i++) {
+		actual_state = essa(ESSA_GET_STATE, (unsigned long)pagebuf[i]);
+		/* extract the usage state in bits 60 and 61 */
+		actual_state = (actual_state >> 2) & 0x3;
+		state_mask = allowed_essa_state_masks[i % ARRAY_SIZE(allowed_essa_state_masks)];
+		report(BIT(actual_state) & state_mask, "page %d state: expected_mask=0x%x actual_mask=0x%lx", i, state_mask, BIT(actual_state));
+	}
+}
+
+int main(void)
+{
+	bool has_essa = check_essa_available();
+
+	report_prefix_push("cmm-migration");
+	if (!has_essa) {
+		report_skip("ESSA is not available");
+		goto done;
+	}
+
+	test_migration();
+done:
+	report_prefix_pop();
+	return report_summary();
+}
s390x/unittests.cfg +4 -0
index b456b2881448..625026d90e52 100644
@@ -176,3 +176,7 @@ extra_params = -cpu qemu,gs=off,vx=off
 file = migration.elf
 groups = migration
 smp = 2
+
+[cmm-migration]
+file = cmm-migration.elf
+groups = migration
author
#4 lore raw

Re: [kvm-unit-tests PATCH v1 2/2] s390x: add cmm migration test

Quoted 19 lines · Claudio Imbrenda
Quoted 10 lines

+       for (i = 0; i < NUM_PAGES; i++) {
+               switch(i % 4) {
+                       case 0:
+                               essa(ESSA_SET_STABLE, (unsigned long)pagebuf[i]);
+                       break;
+                       case 1:
+                               essa(ESSA_SET_UNUSED, (unsigned long)pagebuf[i]);
+                       break;

Quoted 8 lines

+                       case 2:
+                               essa(ESSA_SET_VOLATILE, (unsigned long)pagebuf[i]);
+                       break;
+                       case 3:
+                               essa(ESSA_SET_POT_VOLATILE, (unsigned long)pagebuf[i]);
+                       break;

Quoted 6 lines

const int essa_commands[4] = {ESSA_SET_STABLE, ESSA_SET_UNUSED, ...

for (i = 0; i < NUM_PAGES; i++)
        essa(essa_commands[i % 4], ...

I think it would look more compact and more readable

I like your idea a lot, but the compiler doesn't :-(:

/home/nrb/kvm-unit-tests/lib/asm/cmm.h: In function ‘main’:
/home/nrb/kvm-unit-tests/lib/asm/cmm.h:32:9: error: ‘asm’ operand 2 probably does not match constraints [-Werror]
32 | asm volatile(".insn

rrf,0xb9ab0000,%[extr_state],%[addr],%[new_state],0" \
      |         ^~~

/home/nrb/kvm-unit-tests/lib/asm/cmm.h:32:9: error: impossible constraint in ‘asm’

To satify the "i" constraint, new_state needs to be a compile time constant, which it won't be any more with your suggestion unfortunately.

We can do crazy stuff like this in cmm.h:

#define __essa(state) \
asm volatile(".insn

rrf,0xb9ab0000,%[extr_state],%[addr],%[new_state],0" \
			: [extr_state] "=d" (extr_state) \
			: [addr] "a" (paddr), [new_state] "i"
(state));
static unsigned long essa(uint8_t state, unsigned long paddr)
{
	uint64_t extr_state = 0;

	switch(state) {
		case ESSA_SET_STABLE:
			__essa(ESSA_SET_STABLE);
		break;
		case ESSA_SET_UNUSED:
			__essa(ESSA_SET_UNUSED);
		break;
		case ESSA_SET_VOLATILE:
			__essa(ESSA_SET_VOLATILE);
		break;
		case ESSA_SET_POT_VOLATILE:
			__essa(ESSA_SET_POT_VOLATILE);
		break;
	}

	return (unsigned long)extr_state;
}

But that essentially just shifts the readability problem to a different file. What do you think?

Or we make essa a marco, which doesn't sound like a particularily attractive alternative either.

#5 lore raw

Re: [kvm-unit-tests PATCH v1 2/2] s390x: add cmm migration test

Quoted 20 lines · On Tue, 10 May 2022 15:25:09 +0200 Nico Boehr <[email protected]> wrote:
Quoted 19 lines · Claudio Imbrenda
Quoted 18 lines

+       for (i = 0; i < NUM_PAGES; i++) {
+               switch(i % 4) {
+                       case 0:
+                               essa(ESSA_SET_STABLE, (unsigned long)pagebuf[i]);
+                       break;
+                       case 1:
+                               essa(ESSA_SET_UNUSED, (unsigned long)pagebuf[i]);
+                       break;
+                       case 2:
+                               essa(ESSA_SET_VOLATILE, (unsigned long)pagebuf[i]);
+                       break;
+                       case 3:
+                               essa(ESSA_SET_POT_VOLATILE, (unsigned long)pagebuf[i]);
+                       break;

Quoted 8 lines
Quoted 6 lines

const int essa_commands[4] = {ESSA_SET_STABLE, ESSA_SET_UNUSED, ...

for (i = 0; i < NUM_PAGES; i++)
        essa(essa_commands[i % 4], ...

I think it would look more compact and more readable

I like your idea a lot, but the compiler doesn't :-(:

ouch, I had completely forgotten about that.

Quoted 44 lines

/home/nrb/kvm-unit-tests/lib/asm/cmm.h: In function ‘main’:
/home/nrb/kvm-unit-tests/lib/asm/cmm.h:32:9: error: ‘asm’ operand 2 probably does not match constraints [-Werror]
32 | asm volatile(".insn

rrf,0xb9ab0000,%[extr_state],%[addr],%[new_state],0" \
      |         ^~~

/home/nrb/kvm-unit-tests/lib/asm/cmm.h:32:9: error: impossible constraint in ‘asm’

To satify the "i" constraint, new_state needs to be a compile time constant, which it won't be any more with your suggestion unfortunately.

We can do crazy stuff like this in cmm.h:

#define __essa(state) \
asm volatile(".insn

rrf,0xb9ab0000,%[extr_state],%[addr],%[new_state],0" \
			: [extr_state] "=d" (extr_state) \
			: [addr] "a" (paddr), [new_state] "i"
(state));
static unsigned long essa(uint8_t state, unsigned long paddr)
{
	uint64_t extr_state = 0;

	switch(state) {
		case ESSA_SET_STABLE:
			__essa(ESSA_SET_STABLE);
		break;
		case ESSA_SET_UNUSED:
			__essa(ESSA_SET_UNUSED);
		break;
		case ESSA_SET_VOLATILE:
			__essa(ESSA_SET_VOLATILE);
		break;
		case ESSA_SET_POT_VOLATILE:
			__essa(ESSA_SET_POT_VOLATILE);
		break;
	}

	return (unsigned long)extr_state;
}
Quoted 5 lines

But that essentially just shifts the readability problem to a different file. What do you think?

Or we make essa a marco, which doesn't sound like a particularily attractive alternative either.

ok, next less ugly thing: unroll the loop

for (i = 0; i < NUM_PAGES; i += 4) {
	essa(ESSA_SET_STABLE, (unsigned long)pagebuf[i]);
	essa(ESSA_SET_UNUSED, (unsigned long)pagebuf[i + 1]);
	... etc
}

maybe assert(NUM_PAGES % 4 == 0) before that, just for good measure

author
#6 lore raw

Re: [kvm-unit-tests PATCH v1 2/2] s390x: add cmm migration test

Quoted 9 lines · Claudio Imbrenda

ok, next less ugly thing: unroll the loop

for (i = 0; i < NUM_PAGES; i += 4) { essa(ESSA_SET_STABLE, (unsigned long)pagebuf[i]); essa(ESSA_SET_UNUSED, (unsigned long)pagebuf[i + 1]); ... etc
}

maybe assert(NUM_PAGES % 4 == 0) before that, just for good measure

That's nicer, thanks, fixed.

#7 lore raw
In reply to Nico Boehr
Quoted 5 lines · On Mon, 9 May 2022 14:08:03 +0200 Nico Boehr <[email protected]> wrote:

Upon migration, we expect the CMM page states to be preserved. Add a test which checks for that.

The new test gets a new file so the existing cmm test can still run when the prerequisites for running migration tests aren't given (netcat). Therefore, move

I wonder if we are going to have more of these "split" tests.

is there a way to make sure migration prerequisites are always present? or rewrite things so that we don't need them?

Quoted 15 lines

some definitions to a common header to be able to re-use them.

Nico Boehr (2):
lib: s390x: add header for CMM related defines
s390x: add cmm migration test

 lib/s390x/asm/cmm.h   | 50 +++++++++++++++++++++++++++
 s390x/Makefile        |  1 +
 s390x/cmm-migration.c | 78 +++++++++++++++++++++++++++++++++++++++++++
 s390x/cmm.c           | 25 ++------------
 s390x/unittests.cfg   |  4 +++
 5 files changed, 136 insertions(+), 22 deletions(-)
 create mode 100644 lib/s390x/asm/cmm.h
 create mode 100644 s390x/cmm-migration.c
author
#8 lore raw
Quoted 4 lines · Claudio Imbrenda

I wonder if we are going to have more of these "split" tests.

is there a way to make sure migration prerequisites are always present?

We could not run _any_ tests if netcat is not installed, which seems like a bad idea.

Quoted 1 line

or rewrite things so that we don't need them?

We need ncat to communicate with the QEMU QMP over unix socket. I am not aware of a way to use unix sockets in Bash, but no expert either.

We could ship our own version of netcat and build it for the host, which adds additional complexity and maintenance burden.

I honestly can't think of a good way.

Or we just put all cmm tests in a single file and accept the fact that if you don't have all the migration related requirements installed, you don't get all the tests - even some which are not at all related to migration. I did not like that so I went with the extra file.

#9 lore raw
Quoted 17 lines · Nico Boehr
Quoted 4 lines · Claudio Imbrenda

I wonder if we are going to have more of these "split" tests.

is there a way to make sure migration prerequisites are always present?

We could not run _any_ tests if netcat is not installed, which seems like a bad idea.

Quoted 1 line

or rewrite things so that we don't need them?

We need ncat to communicate with the QEMU QMP over unix socket. I am not aware of a way to use unix sockets in Bash, but no expert either.

We could ship our own version of netcat and build it for the host, which adds additional complexity and maintenance burden.

Quoted 6 lines

I honestly can't think of a good way.

Or we just put all cmm tests in a single file and accept the fact that if you don't have all the migration related requirements installed, you don't get all the tests - even some which are not at all related to migration. I did not like that so I went with the extra file.

Having them separate is fine. I'd change the file name to migration-cmm.elf though. We might also want to change the name of the first migration test in the future to make the name more specific.

author
#10 lore raw
Quoted 4 lines · Janosch Frank

Having them separate is fine. I'd change the file name to migration-cmm.elf though. We might also want to change the name of the
first migration test in the future to make the name more specific.

Yes, good thought. I have it on my TODO.