Skip to contentSkip to conversation

s390x: Add snippet support

PATCH In review Janosch Frank 2 participants 9 messages Jun 29, 2021 – Jul 5, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover s390x: Add snippet support

The SIE support allows us to run guests and test the hypervisor's (V)SIE implementation. However it requires that the guest instructions are binary which limits the complexity of the guest code.

The snippet support provides a way to write guest code as ASM or C and simply memcpy it into guest memory. Some of the KVM-unit-test library can be re-used which further speeds up guest code development.

The included mvpg-sie test helped us to deliver the KVM mvpg fixes which Claudio posted a short while ago. In the future I'll post Secure Execution snippet support patches which was my initial goal with this series anyway.

v2:
* Minor flat.lds fixes
* mvpg-sie.c include cleanup
* mvpg-sie.c added some comments
* mvpg-sie.c replace pei report calculations with variables src/dst

RFC -> V1:
* Replaced handle_validity() with assert()
* Added clearing of registers before sigp in cstart.S
* Added a nice way to compile the snippets

Janosch Frank (2):
s390x: snippets: Add gitignore as well as linker script and start
assembly
s390x: mvpg: Add SIE mvpg test

Steffen Eiden (1):
s390x: snippets: Add snippet compilation

 .gitignore                      |   1 +
 s390x/Makefile                  |  29 ++++--
 s390x/mvpg-sie.c                | 151 ++++++++++++++++++++++++++++++++
 s390x/snippets/c/cstart.S       |  16 ++++
 s390x/snippets/c/flat.lds       |  51 +++++++++++
 s390x/snippets/c/mvpg-snippet.c |  33 +++++++
 s390x/unittests.cfg             |   3 +
 7 files changed, 278 insertions(+), 6 deletions(-)
 create mode 100644 s390x/mvpg-sie.c
 create mode 100644 s390x/snippets/c/cstart.S
 create mode 100644 s390x/snippets/c/flat.lds
 create mode 100644 s390x/snippets/c/mvpg-snippet.c
Signature · 1 line
2.30.2
#1 lore raw

PATCH v2 1/3 s390x: snippets: Add gitignore as well as linker script and start assembly

Snippets are small guests That can be run under a unit test as the hypervisor. They can be written in C or assembly. The C code needs a linker script and a start assembly file that jumps to main to work properly. So let's add that as well as a gitignore entry for the new files.

---
 .gitignore                |  1 +
 s390x/snippets/c/cstart.S | 16 ++++++++++++
 s390x/snippets/c/flat.lds | 51 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 68 insertions(+)
 create mode 100644 s390x/snippets/c/cstart.S
 create mode 100644 s390x/snippets/c/flat.lds
.gitignore +1 -0
index 8534fb7..b3cf2cb 100644
@@ -23,3 +23,4 @@ cscope.*
 /api/dirty-log
 /api/dirty-log-perf
 /s390x/*.bin
+/s390x/snippets/*/*.gbin
s390x/snippets/c/cstart.S new file +16 -0
new file mode 100644
index 0000000..242568d
@@ -0,0 +1,16 @@
+#include <asm/sigp.h>
+
+.section .init
+	.globl start
+start:
+	/* XOR all registers with themselves to clear them fully. */
+	.irp i, 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
+	xgr \i,\i
+	.endr
+	/* 0x3000 is the stack page for now */
+	lghi	%r15, 0x4000 - 160
+	sam64
+	brasl	%r14, main
+	/* For now let's only use cpu 0 in snippets so this will always work. */
+	xgr	%r0, %r0
+	sigp    %r2, %r0, SIGP_STOP
s390x/snippets/c/flat.lds new file +51 -0

51 lines not on this page · raw message

Signature · 1 line
2.30.2
Reviewed-by on v2 1/3
#2 lore raw

Re: [kvm-unit-tests PATCH v2 1/3] s390x: snippets: Add gitignore as well as linker script and start assembly

Quoted 2 lines · Janosch Frank

Snippets are small guests That can be run under a unit test as the hypervisor.

s/That/that/. Maybe also rather something like:

Snippets are small guests which can be run under a unit test that implements a simple hypervisor.

?

Quoted 4 lines

They can be written in C or assembly. The C code needs a linker script and a start assembly file that jumps to main to work properly. So let's add that as well as a gitignore entry for the new files.

Rest of the file looks fine to me.

#3 lore raw

PATCH v2 2/3 s390x: snippets: Add snippet compilation

In reply to Janosch Frank

From: Steffen Eiden <[email protected]>

This patch provides the necessary make targets to properly compile snippets and link them into their associated host.

To define the guest-host definition, we use the make-feature SECONDEXPANSION in combination with Target specific Variable Values. The variable snippets has different values depending on the current target. This enables us to define which snippets (=guests) belong to which hosts. Furthermore, using the second-expansion, we can use snippets in the perquisites of the host's %.elf rule, which otherwise would be not set.

---
 s390x/Makefile | 27 +++++++++++++++++++++------
 1 file changed, 21 insertions(+), 6 deletions(-)
s390x/Makefile +21 -6
index 8820e99..ba32f4c 100644
@@ -76,11 +76,26 @@ OBJDIRS += lib/s390x
 asmlib = $(TEST_DIR)/cstart64.o $(TEST_DIR)/cpu.o
 
 FLATLIBS = $(libcflat)
-%.elf: %.o $(FLATLIBS) $(SRCDIR)/s390x/flat.lds $(asmlib)
-	$(CC) $(CFLAGS) -c -o $(@:.elf=.aux.o) \
-		$(SRCDIR)/lib/auxinfo.c -DPROGNAME=\"$@\"
-	$(CC) $(LDFLAGS) -o $@ -T $(SRCDIR)/s390x/flat.lds \
-		$(filter %.o, $^) $(FLATLIBS) $(@:.elf=.aux.o)
+
+SNIPPET_DIR = $(TEST_DIR)/snippets
+snippet_asmlib = $(SNIPPET_DIR)/c/cstart.o
+
+# perquisites (=guests) for the snippet hosts.
+# $(TEST_DIR)/<snippet-host>.elf: snippets = $(SNIPPET_DIR)/<c/asm>/<snippet>.gbin
+
+$(SNIPPET_DIR)/asm/%.gbin: $(SNIPPET_DIR)/asm/%.o $(FLATLIBS)
+	$(OBJCOPY) -O binary $(patsubst %.gbin,%.o,$@) $@
+	$(OBJCOPY) -I binary -O elf64-s390 -B "s390:64-bit" $@ $@
+
+$(SNIPPET_DIR)/c/%.gbin: $(SNIPPET_DIR)/c/%.o $(snippet_asmlib) $(FLATLIBS)
+	$(CC) $(LDFLAGS) -o $@ -T $(SNIPPET_DIR)/c/flat.lds $(patsubst %.gbin,%.o,$@) $(snippet_asmlib) $(FLATLIBS)
+	$(OBJCOPY) -O binary $@ $@
+	$(OBJCOPY) -I binary -O elf64-s390 -B "s390:64-bit" $@ $@
+
+.SECONDEXPANSION:
+%.elf: $$(snippets) %.o $(FLATLIBS) $(SRCDIR)/s390x/flat.lds $(asmlib)
+	$(CC) $(CFLAGS) -c -o $(@:.elf=.aux.o) $(SRCDIR)/lib/auxinfo.c -DPROGNAME=\"$@\"
+	$(CC) $(LDFLAGS) -o $@ -T $(SRCDIR)/s390x/flat.lds $(filter %.o, $^) $(FLATLIBS) $(snippets) $(@:.elf=.aux.o)
 	$(RM) $(@:.elf=.aux.o)
 	@chmod a-x $@
 
@@ -94,7 +109,7 @@ FLATLIBS = $(libcflat)
 	$(GENPROTIMG) --host-key-document $(HOST_KEY_DOCUMENT) --no-verify --image $< -o $@
 
 arch_clean: asm_offsets_clean
-	$(RM) $(TEST_DIR)/*.{o,elf,bin} $(TEST_DIR)/.*.d lib/s390x/.*.d
+	$(RM) $(TEST_DIR)/*.{o,elf,bin} $(TEST_DIR)/.*.d $(SNIPPET_DIR)/c/*.{o,gbin} $(SNIPPET_DIR)/c/.*.d lib/s390x/.*.d
 
 generated-files = $(asm-offsets)
 $(tests:.elf=.o) $(asmlib) $(cflatobjs): $(generated-files)
Signature · 1 line
2.30.2
#4 lore raw

PATCH v2 3/3 s390x: mvpg: Add SIE mvpg test

In reply to Janosch Frank

Let's also check the PEI values to make sure our VSIE implementation is correct.

---
 s390x/Makefile                  |   2 +
 s390x/mvpg-sie.c                | 151 ++++++++++++++++++++++++++++++++
 s390x/snippets/c/mvpg-snippet.c |  33 +++++++
 s390x/unittests.cfg             |   3 +
 4 files changed, 189 insertions(+)
 create mode 100644 s390x/mvpg-sie.c
 create mode 100644 s390x/snippets/c/mvpg-snippet.c
s390x/Makefile +2 -0
index ba32f4c..07af26d 100644
@@ -23,6 +23,7 @@ tests += $(TEST_DIR)/sie.elf
 tests += $(TEST_DIR)/mvpg.elf
 tests += $(TEST_DIR)/uv-host.elf
 tests += $(TEST_DIR)/edat.elf
+tests += $(TEST_DIR)/mvpg-sie.elf
 
 tests_binary = $(patsubst %.elf,%.bin,$(tests))
 ifneq ($(HOST_KEY_DOCUMENT),)
@@ -82,6 +83,7 @@ snippet_asmlib = $(SNIPPET_DIR)/c/cstart.o
 
 # perquisites (=guests) for the snippet hosts.
 # $(TEST_DIR)/<snippet-host>.elf: snippets = $(SNIPPET_DIR)/<c/asm>/<snippet>.gbin
+$(TEST_DIR)/mvpg-sie.elf: snippets = $(SNIPPET_DIR)/c/mvpg-snippet.gbin
 
 $(SNIPPET_DIR)/asm/%.gbin: $(SNIPPET_DIR)/asm/%.o $(FLATLIBS)
 	$(OBJCOPY) -O binary $(patsubst %.gbin,%.o,$@) $@
s390x/mvpg-sie.c new file +151 -0

151 lines not on this page · raw message

s390x/snippets/c/mvpg-snippet.c new file +33 -0
new file mode 100644
index 0000000..c1eb5d7
@@ -0,0 +1,33 @@
+#include <libcflat.h>
+
+static inline void force_exit(void)
+{
+	asm volatile("	diag	0,0,0x44\n");
+}
+
+static inline int mvpg(unsigned long r0, void *dest, void *src)
+{
+	register unsigned long reg0 asm ("0") = r0;
+	int cc;
+
+	asm volatile("	mvpg    %1,%2\n"
+		     "	ipm     %0\n"
+		     "	srl     %0,28"
+		     : "=d" (cc) : "a" (dest), "a" (src), "d" (reg0)
+		     : "memory", "cc");
+	return cc;
+}
+
+static void test_mvpg_real(void)
+{
+	mvpg(0, (void *)0x5000, (void *)0x6000);
+	force_exit();
+}
+
+__attribute__((section(".text"))) int main(void)
+{
+	test_mvpg_real();
+	test_mvpg_real();
+	test_mvpg_real();
+	return 0;
+}
s390x/unittests.cfg +3 -0
index a0ec886..9e1802f 100644
@@ -106,3 +106,6 @@ timeout = 10
 
 [edat]
 file = edat.elf
+
+[mvpg-sie]
+file = mvpg-sie.elf
Signature · 1 line
2.30.2
#5 lore raw

Re: [kvm-unit-tests PATCH v2 3/3] s390x: mvpg: Add SIE mvpg test

Quoted 63 lines · Janosch Frank

Let's also check the PEI values to make sure our VSIE implementation is correct.

---
  s390x/Makefile                  |   2 +
  s390x/mvpg-sie.c                | 151 ++++++++++++++++++++++++++++++++
  s390x/snippets/c/mvpg-snippet.c |  33 +++++++
  s390x/unittests.cfg             |   3 +
  4 files changed, 189 insertions(+)

create mode 100644 s390x/mvpg-sie.c
create mode 100644 s390x/snippets/c/mvpg-snippet.c

s390x/Makefile +2 -0
index ba32f4c..07af26d 100644
@@ -23,6 +23,7 @@ tests += $(TEST_DIR)/sie.elf
  tests += $(TEST_DIR)/mvpg.elf
  tests += $(TEST_DIR)/uv-host.elf
  tests += $(TEST_DIR)/edat.elf
+tests += $(TEST_DIR)/mvpg-sie.elf
 
  tests_binary = $(patsubst %.elf,%.bin,$(tests))
  ifneq ($(HOST_KEY_DOCUMENT),)
@@ -82,6 +83,7 @@ snippet_asmlib = $(SNIPPET_DIR)/c/cstart.o
 
  # perquisites (=guests) for the snippet hosts.
  # $(TEST_DIR)/<snippet-host>.elf: snippets = $(SNIPPET_DIR)/<c/asm>/<snippet>.gbin
+$(TEST_DIR)/mvpg-sie.elf: snippets = $(SNIPPET_DIR)/c/mvpg-snippet.gbin
 
  $(SNIPPET_DIR)/asm/%.gbin: $(SNIPPET_DIR)/asm/%.o $(FLATLIBS)
  	$(OBJCOPY) -O binary $(patsubst %.gbin,%.o,$@) $@
s390x/mvpg-sie.c new file +24 -0
new file mode 100644
index 0000000..3536c6a
@@ -0,0 +1,151 @@
+#include <libcflat.h>
+#include <asm/asm-offsets.h>
+#include <asm-generic/barrier.h>
+#include <asm/pgtable.h>
+#include <mmu.h>
+#include <asm/page.h>
+#include <asm/facility.h>
+#include <asm/mem.h>
+#include <alloc_page.h>
+#include <vm.h>
+#include <sclp.h>
+#include <sie.h>
+
+static u8 *guest;
+static u8 *guest_instr;
+static struct vm vm;
+
+static uint8_t *src;
+static uint8_t *dst;
+static uint8_t *cmp;
+
+extern const char _binary_s390x_snippets_c_mvpg_snippet_gbin_start[];
+extern const char _binary_s390x_snippets_c_mvpg_snippet_gbin_end[];
+int binary_size;

s390x/mvpg-sie.c line 34

Quoted 10 lines
s390x/mvpg-sie.c new file +10 -0
+
+static void sie(struct vm *vm)
+{
+	/* Reset icptcode so we don't trip over it below */
+	vm->sblk->icptcode = 0;
+
+	while (vm->sblk->icptcode == 0) {
+		sie64a(vm->sblk, &vm->save_area);
+		if (vm->sblk->icptcode == ICPT_VALIDITY)
+			assert(0);

Please replace the above two lines with:

		assert(vm->sblk->icptcode != ICPT_VALIDITY);
Quoted 38 lines
s390x/mvpg-sie.c new file +38 -0
+	}
+	vm->save_area.guest.grs[14] = vm->sblk->gg14;
+	vm->save_area.guest.grs[15] = vm->sblk->gg15;
+}
+
+static void test_mvpg_pei(void)
+{
+	uint64_t **pei_dst = (uint64_t **)((uintptr_t) vm.sblk + 0xc0);
+	uint64_t **pei_src = (uint64_t **)((uintptr_t) vm.sblk + 0xc8);
+
+	report_prefix_push("pei");
+
+	report_prefix_push("src");
+	memset(dst, 0, PAGE_SIZE);
+	protect_page(src, PAGE_ENTRY_I);
+	sie(&vm);
+	report(vm.sblk->icptcode == ICPT_PARTEXEC, "Partial execution");
+	report((uintptr_t)**pei_src == (uintptr_t)src + PAGE_ENTRY_I, "PEI_SRC correct");
+	report((uintptr_t)**pei_dst == (uintptr_t)dst, "PEI_DST correct");
+	unprotect_page(src, PAGE_ENTRY_I);
+	report(!memcmp(cmp, dst, PAGE_SIZE), "Destination intact");
+	/*
+	 * We need to execute the diag44 which is used as a blocker
+	 * behind the mvpg. It makes sure we fail the tests above if
+	 * the mvpg wouldn't have intercepted.
+	 */
+	sie(&vm);
+	/* Make sure we intercepted for the diag44 and nothing else */
+	assert(vm.sblk->icptcode == ICPT_INST &&
+	       vm.sblk->ipa == 0x8300 && vm.sblk->ipb == 0x440000);
+	report_prefix_pop();
+
+	/* Clear PEI data for next check */
+	report_prefix_push("dst");
+	memset((uint64_t *)((uintptr_t) vm.sblk + 0xc0), 0, 16);
+	memset(dst, 0, PAGE_SIZE);
+	protect_page(dst, PAGE_ENTRY_I);
+	sie(&vm);

s390x/mvpg-sie.c line 82

Quoted 10 lines
s390x/mvpg-sie.c new file +10 -0
+	report(vm.sblk->icptcode == ICPT_PARTEXEC, "Partial execution");
+	report((uintptr_t)**pei_src == (uintptr_t)src, "PEI_SRC correct");
+	report((uintptr_t)**pei_dst == (uintptr_t)dst + PAGE_ENTRY_I, "PEI_DST correct");
+	/* Needed for the memcmp and general cleanup */
+	unprotect_page(dst, PAGE_ENTRY_I);
+	report(!memcmp(cmp, dst, PAGE_SIZE), "Destination intact");
+	report_prefix_pop();
+
+	report_prefix_pop();
+}

Still quite a lot of magic values in above code ... any chance to introduce some #defines finally?

s390x/mvpg-sie.c line 93

Quoted 10 lines
s390x/mvpg-sie.c new file +10 -0
+static void test_mvpg(void)
+{
+	int binary_size = ((uintptr_t)_binary_s390x_snippets_c_mvpg_snippet_gbin_end -
+			   (uintptr_t)_binary_s390x_snippets_c_mvpg_snippet_gbin_start);
+
+	memcpy(guest, _binary_s390x_snippets_c_mvpg_snippet_gbin_start, binary_size);
+	memset(src, 0x42, PAGE_SIZE);
+	memset(dst, 0x43, PAGE_SIZE);
+	sie(&vm);
+	mb();

I think you don't need the mb() here.

s390x/mvpg-sie.c line 94

Quoted 2 lines
s390x/mvpg-sie.c new file +2 -0
+	report(!memcmp(src, dst, PAGE_SIZE) && *dst == 0x42, "Page moved");
+}

Thomas

#6 lore raw

Re: [kvm-unit-tests PATCH v2 3/3] s390x: mvpg: Add SIE mvpg test

Quoted 77 lines · Thomas Huth
Quoted 63 lines · Janosch Frank

Let's also check the PEI values to make sure our VSIE implementation is correct.

---
  s390x/Makefile                  |   2 +
  s390x/mvpg-sie.c                | 151 ++++++++++++++++++++++++++++++++
  s390x/snippets/c/mvpg-snippet.c |  33 +++++++
  s390x/unittests.cfg             |   3 +
  4 files changed, 189 insertions(+)

create mode 100644 s390x/mvpg-sie.c
create mode 100644 s390x/snippets/c/mvpg-snippet.c

s390x/Makefile +2 -0
index ba32f4c..07af26d 100644
@@ -23,6 +23,7 @@ tests += $(TEST_DIR)/sie.elf
  tests += $(TEST_DIR)/mvpg.elf
  tests += $(TEST_DIR)/uv-host.elf
  tests += $(TEST_DIR)/edat.elf
+tests += $(TEST_DIR)/mvpg-sie.elf
 
  tests_binary = $(patsubst %.elf,%.bin,$(tests))
  ifneq ($(HOST_KEY_DOCUMENT),)
@@ -82,6 +83,7 @@ snippet_asmlib = $(SNIPPET_DIR)/c/cstart.o
 
  # perquisites (=guests) for the snippet hosts.
  # $(TEST_DIR)/<snippet-host>.elf: snippets = $(SNIPPET_DIR)/<c/asm>/<snippet>.gbin
+$(TEST_DIR)/mvpg-sie.elf: snippets = $(SNIPPET_DIR)/c/mvpg-snippet.gbin
 
  $(SNIPPET_DIR)/asm/%.gbin: $(SNIPPET_DIR)/asm/%.o $(FLATLIBS)
  	$(OBJCOPY) -O binary $(patsubst %.gbin,%.o,$@) $@
s390x/mvpg-sie.c new file +24 -0
new file mode 100644
index 0000000..3536c6a
@@ -0,0 +1,151 @@
+#include <libcflat.h>
+#include <asm/asm-offsets.h>
+#include <asm-generic/barrier.h>
+#include <asm/pgtable.h>
+#include <mmu.h>
+#include <asm/page.h>
+#include <asm/facility.h>
+#include <asm/mem.h>
+#include <alloc_page.h>
+#include <vm.h>
+#include <sclp.h>
+#include <sie.h>
+
+static u8 *guest;
+static u8 *guest_instr;
+static struct vm vm;
+
+static uint8_t *src;
+static uint8_t *dst;
+static uint8_t *cmp;
+
+extern const char _binary_s390x_snippets_c_mvpg_snippet_gbin_start[];
+extern const char _binary_s390x_snippets_c_mvpg_snippet_gbin_end[];
+int binary_size;

s390x/mvpg-sie.c line 34

Quoted 10 lines
s390x/mvpg-sie.c new file +10 -0
+
+static void sie(struct vm *vm)
+{
+	/* Reset icptcode so we don't trip over it below */
+	vm->sblk->icptcode = 0;
+
+	while (vm->sblk->icptcode == 0) {
+		sie64a(vm->sblk, &vm->save_area);
+		if (vm->sblk->icptcode == ICPT_VALIDITY)
+			assert(0);

Please replace the above two lines with:

Quoted 1 line
		assert(vm->sblk->icptcode != ICPT_VALIDITY);

Sure

Quoted 44 lines
Quoted 44 lines
s390x/mvpg-sie.c new file +43 -0
+	}
+	vm->save_area.guest.grs[14] = vm->sblk->gg14;
+	vm->save_area.guest.grs[15] = vm->sblk->gg15;
+}
+
+static void test_mvpg_pei(void)
+{
+	uint64_t **pei_dst = (uint64_t **)((uintptr_t) vm.sblk + 0xc0);
+	uint64_t **pei_src = (uint64_t **)((uintptr_t) vm.sblk + 0xc8);
+
+	report_prefix_push("pei");
+
+	report_prefix_push("src");
+	memset(dst, 0, PAGE_SIZE);
+	protect_page(src, PAGE_ENTRY_I);
+	sie(&vm);
+	report(vm.sblk->icptcode == ICPT_PARTEXEC, "Partial execution");
+	report((uintptr_t)**pei_src == (uintptr_t)src + PAGE_ENTRY_I, "PEI_SRC correct");
+	report((uintptr_t)**pei_dst == (uintptr_t)dst, "PEI_DST correct");
+	unprotect_page(src, PAGE_ENTRY_I);
+	report(!memcmp(cmp, dst, PAGE_SIZE), "Destination intact");
+	/*
+	 * We need to execute the diag44 which is used as a blocker
+	 * behind the mvpg. It makes sure we fail the tests above if
+	 * the mvpg wouldn't have intercepted.
+	 */
+	sie(&vm);
+	/* Make sure we intercepted for the diag44 and nothing else */
+	assert(vm.sblk->icptcode == ICPT_INST &&
+	       vm.sblk->ipa == 0x8300 && vm.sblk->ipb == 0x440000);
+	report_prefix_pop();
+
+	/* Clear PEI data for next check */
+	report_prefix_push("dst");
+	memset((uint64_t *)((uintptr_t) vm.sblk + 0xc0), 0, 16);
+	memset(dst, 0, PAGE_SIZE);
+	protect_page(dst, PAGE_ENTRY_I);
+	sie(&vm);
+	report(vm.sblk->icptcode == ICPT_PARTEXEC, "Partial execution");
+	report((uintptr_t)**pei_src == (uintptr_t)src, "PEI_SRC correct");
+	report((uintptr_t)**pei_dst == (uintptr_t)dst + PAGE_ENTRY_I, "PEI_DST correct");
+	/* Needed for the memcmp and general cleanup */
+	unprotect_page(dst, PAGE_ENTRY_I);
Quoted 8 lines

s390x/mvpg-sie.c line 82

Quoted 5 lines
s390x/mvpg-sie.c new file +5 -0
+	report(!memcmp(cmp, dst, PAGE_SIZE), "Destination intact");
+	report_prefix_pop();
+
+	report_prefix_pop();
+}

Still quite a lot of magic values in above code ... any chance to introduce some #defines finally?

Currently not really.
I added a comment for the diag 44 which should be enough right now. If needed I can add a comment to the pei variables as well.

Quoted 13 lines

s390x/mvpg-sie.c line 93

Quoted 11 lines
s390x/mvpg-sie.c new file +10 -0
+static void test_mvpg(void)
+{
+	int binary_size = ((uintptr_t)_binary_s390x_snippets_c_mvpg_snippet_gbin_end -
+			   (uintptr_t)_binary_s390x_snippets_c_mvpg_snippet_gbin_start);
+
+	memcpy(guest, _binary_s390x_snippets_c_mvpg_snippet_gbin_start, binary_size);
+	memset(src, 0x42, PAGE_SIZE);
+	memset(dst, 0x43, PAGE_SIZE);
+	sie(&vm);
+	mb();

I think you don't need the mb() here.

Right

Quoted 6 lines

s390x/mvpg-sie.c line 94

Quoted 3 lines
s390x/mvpg-sie.c new file +2 -0
+	report(!memcmp(src, dst, PAGE_SIZE) && *dst == 0x42, "Page moved");
+}

Thomas

Acked-by
#7 lore raw

Re: [kvm-unit-tests PATCH v2 3/3] s390x: mvpg: Add SIE mvpg test

Quoted 135 lines · Janosch Frank
Quoted 77 lines · Thomas Huth
Quoted 63 lines · Janosch Frank

Let's also check the PEI values to make sure our VSIE implementation is correct.

---
   s390x/Makefile                  |   2 +
   s390x/mvpg-sie.c                | 151 ++++++++++++++++++++++++++++++++
   s390x/snippets/c/mvpg-snippet.c |  33 +++++++
   s390x/unittests.cfg             |   3 +
   4 files changed, 189 insertions(+)

create mode 100644 s390x/mvpg-sie.c
create mode 100644 s390x/snippets/c/mvpg-snippet.c

s390x/Makefile +2 -0
index ba32f4c..07af26d 100644
@@ -23,6 +23,7 @@ tests += $(TEST_DIR)/sie.elf
   tests += $(TEST_DIR)/mvpg.elf
   tests += $(TEST_DIR)/uv-host.elf
   tests += $(TEST_DIR)/edat.elf
+tests += $(TEST_DIR)/mvpg-sie.elf
 
   tests_binary = $(patsubst %.elf,%.bin,$(tests))
   ifneq ($(HOST_KEY_DOCUMENT),)
@@ -82,6 +83,7 @@ snippet_asmlib = $(SNIPPET_DIR)/c/cstart.o
 
   # perquisites (=guests) for the snippet hosts.
   # $(TEST_DIR)/<snippet-host>.elf: snippets = $(SNIPPET_DIR)/<c/asm>/<snippet>.gbin
+$(TEST_DIR)/mvpg-sie.elf: snippets = $(SNIPPET_DIR)/c/mvpg-snippet.gbin
 
   $(SNIPPET_DIR)/asm/%.gbin: $(SNIPPET_DIR)/asm/%.o $(FLATLIBS)
   	$(OBJCOPY) -O binary $(patsubst %.gbin,%.o,$@) $@
s390x/mvpg-sie.c new file +24 -0
new file mode 100644
index 0000000..3536c6a
@@ -0,0 +1,151 @@
+#include <libcflat.h>
+#include <asm/asm-offsets.h>
+#include <asm-generic/barrier.h>
+#include <asm/pgtable.h>
+#include <mmu.h>
+#include <asm/page.h>
+#include <asm/facility.h>
+#include <asm/mem.h>
+#include <alloc_page.h>
+#include <vm.h>
+#include <sclp.h>
+#include <sie.h>
+
+static u8 *guest;
+static u8 *guest_instr;
+static struct vm vm;
+
+static uint8_t *src;
+static uint8_t *dst;
+static uint8_t *cmp;
+
+extern const char _binary_s390x_snippets_c_mvpg_snippet_gbin_start[];
+extern const char _binary_s390x_snippets_c_mvpg_snippet_gbin_end[];
+int binary_size;

s390x/mvpg-sie.c line 34

Quoted 10 lines
s390x/mvpg-sie.c new file +10 -0
+
+static void sie(struct vm *vm)
+{
+	/* Reset icptcode so we don't trip over it below */
+	vm->sblk->icptcode = 0;
+
+	while (vm->sblk->icptcode == 0) {
+		sie64a(vm->sblk, &vm->save_area);
+		if (vm->sblk->icptcode == ICPT_VALIDITY)
+			assert(0);

Please replace the above two lines with:

Quoted 1 line
		assert(vm->sblk->icptcode != ICPT_VALIDITY);

Sure

Quoted 44 lines
Quoted 44 lines
s390x/mvpg-sie.c new file +43 -0
+	}
+	vm->save_area.guest.grs[14] = vm->sblk->gg14;
+	vm->save_area.guest.grs[15] = vm->sblk->gg15;
+}
+
+static void test_mvpg_pei(void)
+{
+	uint64_t **pei_dst = (uint64_t **)((uintptr_t) vm.sblk + 0xc0);
+	uint64_t **pei_src = (uint64_t **)((uintptr_t) vm.sblk + 0xc8);
+
+	report_prefix_push("pei");
+
+	report_prefix_push("src");
+	memset(dst, 0, PAGE_SIZE);
+	protect_page(src, PAGE_ENTRY_I);
+	sie(&vm);
+	report(vm.sblk->icptcode == ICPT_PARTEXEC, "Partial execution");
+	report((uintptr_t)**pei_src == (uintptr_t)src + PAGE_ENTRY_I, "PEI_SRC correct");
+	report((uintptr_t)**pei_dst == (uintptr_t)dst, "PEI_DST correct");
+	unprotect_page(src, PAGE_ENTRY_I);
+	report(!memcmp(cmp, dst, PAGE_SIZE), "Destination intact");
+	/*
+	 * We need to execute the diag44 which is used as a blocker
+	 * behind the mvpg. It makes sure we fail the tests above if
+	 * the mvpg wouldn't have intercepted.
+	 */
+	sie(&vm);
+	/* Make sure we intercepted for the diag44 and nothing else */
+	assert(vm.sblk->icptcode == ICPT_INST &&
+	       vm.sblk->ipa == 0x8300 && vm.sblk->ipb == 0x440000);
+	report_prefix_pop();
+
+	/* Clear PEI data for next check */
+	report_prefix_push("dst");
+	memset((uint64_t *)((uintptr_t) vm.sblk + 0xc0), 0, 16);
+	memset(dst, 0, PAGE_SIZE);
+	protect_page(dst, PAGE_ENTRY_I);
+	sie(&vm);
+	report(vm.sblk->icptcode == ICPT_PARTEXEC, "Partial execution");
+	report((uintptr_t)**pei_src == (uintptr_t)src, "PEI_SRC correct");
+	report((uintptr_t)**pei_dst == (uintptr_t)dst + PAGE_ENTRY_I, "PEI_DST correct");
+	/* Needed for the memcmp and general cleanup */
+	unprotect_page(dst, PAGE_ENTRY_I);
Quoted 8 lines

s390x/mvpg-sie.c line 82

Quoted 5 lines
s390x/mvpg-sie.c new file +5 -0
+	report(!memcmp(cmp, dst, PAGE_SIZE), "Destination intact");
+	report_prefix_pop();
+
+	report_prefix_pop();
+}

Still quite a lot of magic values in above code ... any chance to introduce some #defines finally?

Quoted 3 lines

Currently not really.
I added a comment for the diag 44 which should be enough right now. If needed I can add a comment to the pei variables as well.

Ok, fine for me, we can still clean up later if necessary. Thus with the assert() fixed and the mb() removed:

#8 lore raw

Re: [kvm-unit-tests PATCH v2 3/3] s390x: mvpg: Add SIE mvpg test

Quoted 136 lines · Thomas Huth
Quoted 135 lines · Janosch Frank
Quoted 77 lines · Thomas Huth
Quoted 63 lines · Janosch Frank

Let's also check the PEI values to make sure our VSIE implementation is correct.

---
   s390x/Makefile                  |   2 +
   s390x/mvpg-sie.c                | 151 ++++++++++++++++++++++++++++++++
   s390x/snippets/c/mvpg-snippet.c |  33 +++++++
   s390x/unittests.cfg             |   3 +
   4 files changed, 189 insertions(+)

create mode 100644 s390x/mvpg-sie.c
create mode 100644 s390x/snippets/c/mvpg-snippet.c

s390x/Makefile +2 -0
index ba32f4c..07af26d 100644
@@ -23,6 +23,7 @@ tests += $(TEST_DIR)/sie.elf
   tests += $(TEST_DIR)/mvpg.elf
   tests += $(TEST_DIR)/uv-host.elf
   tests += $(TEST_DIR)/edat.elf
+tests += $(TEST_DIR)/mvpg-sie.elf
 
   tests_binary = $(patsubst %.elf,%.bin,$(tests))
   ifneq ($(HOST_KEY_DOCUMENT),)
@@ -82,6 +83,7 @@ snippet_asmlib = $(SNIPPET_DIR)/c/cstart.o
 
   # perquisites (=guests) for the snippet hosts.
   # $(TEST_DIR)/<snippet-host>.elf: snippets = $(SNIPPET_DIR)/<c/asm>/<snippet>.gbin
+$(TEST_DIR)/mvpg-sie.elf: snippets = $(SNIPPET_DIR)/c/mvpg-snippet.gbin
 
   $(SNIPPET_DIR)/asm/%.gbin: $(SNIPPET_DIR)/asm/%.o $(FLATLIBS)
   	$(OBJCOPY) -O binary $(patsubst %.gbin,%.o,$@) $@
s390x/mvpg-sie.c new file +24 -0
new file mode 100644
index 0000000..3536c6a
@@ -0,0 +1,151 @@
+#include <libcflat.h>
+#include <asm/asm-offsets.h>
+#include <asm-generic/barrier.h>
+#include <asm/pgtable.h>
+#include <mmu.h>
+#include <asm/page.h>
+#include <asm/facility.h>
+#include <asm/mem.h>
+#include <alloc_page.h>
+#include <vm.h>
+#include <sclp.h>
+#include <sie.h>
+
+static u8 *guest;
+static u8 *guest_instr;
+static struct vm vm;
+
+static uint8_t *src;
+static uint8_t *dst;
+static uint8_t *cmp;
+
+extern const char _binary_s390x_snippets_c_mvpg_snippet_gbin_start[];
+extern const char _binary_s390x_snippets_c_mvpg_snippet_gbin_end[];
+int binary_size;

s390x/mvpg-sie.c line 34

Quoted 10 lines
s390x/mvpg-sie.c new file +10 -0
+
+static void sie(struct vm *vm)
+{
+	/* Reset icptcode so we don't trip over it below */
+	vm->sblk->icptcode = 0;
+
+	while (vm->sblk->icptcode == 0) {
+		sie64a(vm->sblk, &vm->save_area);
+		if (vm->sblk->icptcode == ICPT_VALIDITY)
+			assert(0);

Please replace the above two lines with:

Quoted 1 line
		assert(vm->sblk->icptcode != ICPT_VALIDITY);

Sure

Quoted 52 lines
Quoted 39 lines
s390x/mvpg-sie.c new file +38 -0
+	}
+	vm->save_area.guest.grs[14] = vm->sblk->gg14;
+	vm->save_area.guest.grs[15] = vm->sblk->gg15;
+}
+
+static void test_mvpg_pei(void)
+{
+	uint64_t **pei_dst = (uint64_t **)((uintptr_t) vm.sblk + 0xc0);
+	uint64_t **pei_src = (uint64_t **)((uintptr_t) vm.sblk + 0xc8);
+
+	report_prefix_push("pei");
+
+	report_prefix_push("src");
+	memset(dst, 0, PAGE_SIZE);
+	protect_page(src, PAGE_ENTRY_I);
+	sie(&vm);
+	report(vm.sblk->icptcode == ICPT_PARTEXEC, "Partial execution");
+	report((uintptr_t)**pei_src == (uintptr_t)src + PAGE_ENTRY_I, "PEI_SRC correct");
+	report((uintptr_t)**pei_dst == (uintptr_t)dst, "PEI_DST correct");
+	unprotect_page(src, PAGE_ENTRY_I);
+	report(!memcmp(cmp, dst, PAGE_SIZE), "Destination intact");
+	/*
+	 * We need to execute the diag44 which is used as a blocker
+	 * behind the mvpg. It makes sure we fail the tests above if
+	 * the mvpg wouldn't have intercepted.
+	 */
+	sie(&vm);
+	/* Make sure we intercepted for the diag44 and nothing else */
+	assert(vm.sblk->icptcode == ICPT_INST &&
+	       vm.sblk->ipa == 0x8300 && vm.sblk->ipb == 0x440000);
+	report_prefix_pop();
+
+	/* Clear PEI data for next check */
+	report_prefix_push("dst");
+	memset((uint64_t *)((uintptr_t) vm.sblk + 0xc0), 0, 16);
+	memset(dst, 0, PAGE_SIZE);
+	protect_page(dst, PAGE_ENTRY_I);
+	sie(&vm);

s390x/mvpg-sie.c line 82

Quoted 10 lines
s390x/mvpg-sie.c new file +10 -0
+	report(vm.sblk->icptcode == ICPT_PARTEXEC, "Partial execution");
+	report((uintptr_t)**pei_src == (uintptr_t)src, "PEI_SRC correct");
+	report((uintptr_t)**pei_dst == (uintptr_t)dst + PAGE_ENTRY_I, "PEI_DST correct");
+	/* Needed for the memcmp and general cleanup */
+	unprotect_page(dst, PAGE_ENTRY_I);
+	report(!memcmp(cmp, dst, PAGE_SIZE), "Destination intact");
+	report_prefix_pop();
+
+	report_prefix_pop();
+}

Still quite a lot of magic values in above code ... any chance to introduce some #defines finally?

Quoted 9 lines
Quoted 3 lines

Currently not really.
I added a comment for the diag 44 which should be enough right now. If needed I can add a comment to the pei variables as well.

Ok, fine for me, we can still clean up later if necessary. Thus with the assert() fixed and the mb() removed:

Thanks, I'll fix this up and queue the series