s390x: Add support for running guests without MSO/MSL
v4 cover s390x: Add support for running guests without MSO/MSL
v4:
---
- add static assert for PSW bitfield (Janosch, Claudio)
- remove unneeded includes (Janosch)
- move variable decls to function start (Janosch)
- remove unneeded imports (Janosch)
- lowerocase hex (Janosch)
- remove unneeded attr (Janosch)
- tyop :-) fixes (Janosch)
v3:
---
* introduce bitfield for the PSW to make handling less clumsy
* some variable renames (Claudio)
* remove unneeded barriers (Claudio)
* remove rebase leftover sie_had_pgm_int (Claudio)
* move read_pgm_int_code to header (Claudio)
* squash include fix commit into the one causing the issue (Claudio)
v2:
---
* add function to change DAT/AS mode for all irq handlers (Janosch, Claudio)
* instead of a new flag in PROG0C, check the pgm int code in lowcore (Janosch)
* fix indents, comments (Nina)
Right now, all SIE tests in kvm-unit-tests (i.e. where kvm-unit-test is the hypervisor) run using MSO/MSL.
This is convenient, because it's simple. But it also comes with disadvantages, for example some features are unavailabe with MSO/MSL.
This series adds support for running guests without MSO/MSL with dedicated guest page tables for the GPA->HPA translation.
Since SIE implicitly uses the primary space mode for the guest, the host can't run in the primary space mode, too. To avoid moving all tests to the home space mode, only switch to home space mode when it is actually needed.
This series also comes with various bugfixes that were caught while develoing this.
Nico Boehr (6):
lib: s390x: introduce bitfield for PSW mask
s390x: add function to set DAT mode for all interrupts
s390x: sie: switch to home space mode before entering SIE
s390x: lib: don't forward PSW when handling exception in SIE
s390x: lib: sie: don't reenter SIE on pgm int
s390x: add a test for SIE without MSO/MSL
lib/s390x/asm/arch_def.h | 27 ++++++++- lib/s390x/asm/interrupt.h | 18 ++++++ lib/s390x/asm/mem.h | 1 + lib/s390x/interrupt.c | 36 ++++++++++++ lib/s390x/mmu.c | 5 +- lib/s390x/sie.c | 22 ++++++- s390x/Makefile | 2 + s390x/selftest.c | 40 +++++++++++++ s390x/sie-dat.c | 115 +++++++++++++++++++++++++++++++++++++ s390x/snippets/c/sie-dat.c | 58 +++++++++++++++++++ s390x/unittests.cfg | 3 + 11 files changed, 323 insertions(+), 4 deletions(-) create mode 100644 s390x/sie-dat.c create mode 100644 s390x/snippets/c/sie-dat.c
Signature · 1 line
2.39.1
PATCH v4 1/6 lib: s390x: introduce bitfield for PSW mask
Changing the PSW mask is currently little clumsy, since there is only the PSW_MASK_* defines. This makes it hard to change e.g. only the address space in the current PSW without a lot of bit fiddling.
Introduce a bitfield for the PSW mask. This makes this kind of modifications much simpler and easier to read.
--- lib/s390x/asm/arch_def.h | 26 +++++++++++++++++++++++++- s390x/selftest.c | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+), 1 deletion(-)
lib/s390x/asm/arch_def.h +25 -1
@@ -37,12 +37,36 @@ struct stack_frame_int { };struct psw { -uint64_t mask; +union { +uint64_t mask; +struct { +uint8_t reserved00:1 ; +uint8_t per:1 ; +uint8_t reserved02:3 ; +uint8_t dat:1 ; +uint8_t io:1 ; +uint8_t ext:1 ; +uint8_t key:4 ; +uint8_t reserved12:1 ; +uint8_t mchk:1 ; +uint8_t wait:1 ; +uint8_t pstate:1 ; +uint8_t as:2 ; +uint8_t cc:2 ; +uint8_t prg_mask:4 ; +uint8_t reserved24:7 ; +uint8_t ea:1 ; +uint8_t ba:1 ; +uint32_t reserved33:31 ; + }; + };uint64_t addr; }; +_Static_assert (sizeof (struct psw ) ==16 ,"PSW size" );#define PSW (m, a) ((struct psw){ .mask = (m), .addr = (uint64_t)(a) }) +struct short_psw {uint32_t mask;uint32_t addr;
s390x/selftest.c +40 -0
@@ -74,6 +74,45 @@ static void test_malloc(void)report_prefix_pop (); } +static void test_psw_mask (void ) +{ +uint64_t expected_key =0xF ; +struct psw test_psw =PSW (0 ,0 ); + +report_prefix_push ("PSW mask" ); + test_psw.dat =1 ; +report (test_psw.mask ==PSW_MASK_DAT ,"DAT matches expected=0x%016lx actual=0x%016lx" ,PSW_MASK_DAT , test_psw.mask); + + test_psw.mask =0 ; + test_psw.io =1 ; +report (test_psw.mask ==PSW_MASK_IO ,"IO matches expected=0x%016lx actual=0x%016lx" ,PSW_MASK_IO , test_psw.mask); + + test_psw.mask =0 ; + test_psw.ext =1 ; +report (test_psw.mask ==PSW_MASK_EXT ,"EXT matches expected=0x%016lx actual=0x%016lx" ,PSW_MASK_EXT , test_psw.mask); + + test_psw.mask = expected_key << (63 -11 ); +report (test_psw.key == expected_key,"PSW Key matches expected=0x%lx actual=0x%x" , expected_key, test_psw.key); + + test_psw.mask =1UL << (63 -13 ); +report (test_psw.mchk,"MCHK matches" ); + + test_psw.mask =0 ; + test_psw.wait =1 ; +report (test_psw.mask ==PSW_MASK_WAIT ,"Wait matches expected=0x%016lx actual=0x%016lx" ,PSW_MASK_WAIT , test_psw.mask); + + test_psw.mask =0 ; + test_psw.pstate =1 ; +report (test_psw.mask ==PSW_MASK_PSTATE ,"Pstate matches expected=0x%016lx actual=0x%016lx" ,PSW_MASK_PSTATE , test_psw.mask); + + test_psw.mask =0 ; + test_psw.ea =1 ; + test_psw.ba =1 ; +report (test_psw.mask ==PSW_MASK_64 ,"BA/EA matches expected=0x%016lx actual=0x%016lx" ,PSW_MASK_64 , test_psw.mask); + +report_prefix_pop (); +} +int main (int argc,char **argv) {report_prefix_push ("selftest" );
@@ -89,6 +128,7 @@ int main(int argc, char**argv)test_fp ();test_pgm_int ();test_malloc (); +test_psw_mask ();return report_summary (); }
Signature · 1 line
2.39.1
PATCH v4 2/6 s390x: add function to set DAT mode for all interrupts
In reply to Nico BoehrWhen toggling DAT or switch address space modes, it is likely that interrupts should be handled in the same DAT or address space mode.
Add a function which toggles DAT and address space mode for all interruptions, except restart interrupts.
--- lib/s390x/asm/interrupt.h | 4 ++++ lib/s390x/interrupt.c | 35 +++++++++++++++++++++++++++++++++++ lib/s390x/mmu.c | 5 +++-- 3 files changed, 42 insertions(+), 2 deletions(-)
lib/s390x/asm/interrupt.h +4 -0
@@ -83,6 +83,10 @@ void expect_ext_int(void);uint16_t clear_pgm_int (void );void check_pgm_int_code (uint16_t code); +#define IRQ_DAT_ON true +#define IRQ_DAT_OFF false +void irq_set_dat_mode (bool dat,uint64_t as); +/* Activate low-address protection */ static inline void low_prot_enable (void ) {
lib/s390x/interrupt.c +35 -0
@@ -9,6 +9,7 @@*/ #include <libcflat.h> #include <asm/barrier.h> +#include <asm/mem.h> #include <asm/asm-offsets.h> #include <sclp.h> #include <interrupt.h>
@@ -104,6 +105,40 @@ void register_ext_cleanup_func(void (*f)(struct stack_frame_int *))THIS_CPU ->ext_cleanup_func = f; } +/** +* irq_set_dat_mode - Set the DAT mode of all interrupt handlers, except for +* restart. +* This will update the DAT mode and address space mode of all interrupt new +* PSWs. +* +* Since enabling DAT needs initalized CRs and the restart new PSW is often used +* to initalize CRs, the restart new PSW is never touched to avoid the chicken +* and egg situation. +* +* @dat specifies whether to use DAT or not +* @as specifies the address space mode to use - one of AS_PRIM, AS_ACCR, +* AS_SECN or AS_HOME. +*/ +void irq_set_dat_mode (bool dat,uint64_t as) +{ +struct psw * irq_psws[] = { +OPAQUE_PTR (GEN_LC_EXT_NEW_PSW ), +OPAQUE_PTR (GEN_LC_SVC_NEW_PSW ), +OPAQUE_PTR (GEN_LC_PGM_NEW_PSW ), +OPAQUE_PTR (GEN_LC_MCCK_NEW_PSW ), +OPAQUE_PTR (GEN_LC_IO_NEW_PSW ), +NULL /* sentinel */ + }; + +assert (as ==AS_PRIM || as ==AS_ACCR || as ==AS_SECN || as ==AS_HOME ); + +for (struct psw *psw = irq_psws[0 ]; psw !=NULL ; psw++) { + psw->dat = dat; +if (dat) + psw->as = as; + } +} +static void fixup_pgm_int (struct stack_frame_int *stack) {/* If we have an error on SIE we directly move to sie_exit */
lib/s390x/mmu.c +3 -2
@@ -12,6 +12,7 @@#include <asm/pgtable.h> #include <asm/arch_def.h> #include <asm/barrier.h> +#include <asm/interrupt.h> #include <vmalloc.h> #include "mmu.h"
@@ -41,8 +42,8 @@ static void mmu_enable(pgd_t *pgtable)/* enable dat (primary == 0 set as default) */ enable_dat (); -/* we can now also use DAT unconditionally in our PGM handler */ - lowcore.pgm_new_psw.mask |=PSW_MASK_DAT ; +/* we can now also use DAT in all interrupt handlers */ +irq_set_dat_mode (IRQ_DAT_ON ,AS_PRIM ); }/*
Signature · 1 line
2.39.1
Re: [kvm-unit-tests PATCH v4 2/6] s390x: add function to set DAT mode for all interrupts
Quoting Nico Boehr (2023-07-11 09:35:10)
[...]
Quoted 4 lines
lib/s390x/interrupt.c +0
-0
[...]
Quoted 4 lines
lib/s390x/interrupt.c +4
-0+void irq_set_dat_mode (bool dat,uint64_t as) +{ +struct psw * irq_psws[] = { +OPAQUE_PTR (GEN_LC_EXT_NEW_PSW ),
lib/s390x/interrupt.c void register_ext_cleanup_func(void (*f)(struct stack_frame_int *)) · line 135
Quoted 10 lines
lib/s390x/interrupt.c +10
-0+OPAQUE_PTR (GEN_LC_SVC_NEW_PSW ), +OPAQUE_PTR (GEN_LC_PGM_NEW_PSW ), +OPAQUE_PTR (GEN_LC_MCCK_NEW_PSW ), +OPAQUE_PTR (GEN_LC_IO_NEW_PSW ), + NULL /* sentinel */ + }; + +assert (as ==AS_PRIM || as ==AS_ACCR || as ==AS_SECN || as ==AS_HOME ); + +for (struct psw *psw = irq_psws[0 ]; psw !=NULL ; psw++) {
This is obviously completely wrong and leads to a bunch of random memory in lowcore being overwritten. For weird optimization reasons, this actually terminates. I will sent a fixed version, where I simply don't do fancy pointer stuff but just iterate over an array index...
Thanks Claudio.
PATCH v4 3/6 s390x: sie: switch to home space mode before entering SIE
In reply to Nico BoehrThis is to prepare for running guests without MSO/MSL, which is currently not possible.
We already have code in sie64a to setup a guest primary ASCE before entering SIE, so we can in theory switch to the page tables which translate gpa to hpa.
But the host is running in primary space mode already, so changing the primary ASCE before entering SIE will also affect the host's code and data.
To make this switch useful, the host should run in a different address space mode. Hence, set up and change to home address space mode before installing the guest ASCE.
The home space ASCE is just copied over from the primary space ASCE, so no functional change is intended, also for tests that want to use MSO/MSL. If a test intends to use a different primary space ASCE, it can now just set the guest.asce in the save_area.
--- lib/s390x/asm/arch_def.h | 1 + lib/s390x/sie.c | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+)
lib/s390x/asm/arch_def.h +1 -0
@@ -91,6 +91,7 @@ struct cpu {#define AS_HOME 3#define PSW_MASK_DAT 0x0400000000000000UL +#define PSW_MASK_HOME 0x0000C00000000000UL#define PSW_MASK_IO 0x0200000000000000UL#define PSW_MASK_EXT 0x0100000000000000UL#define PSW_MASK_KEY 0x00F0000000000000UL
lib/s390x/sie.c +18 -0
@@ -46,6 +46,8 @@ void sie_handle_validity(struct vm *vm)void sie (struct vm *vm) { +uint64_t old_cr13; +if (vm->sblk->sdf ==2 )memcpy (vm->sblk->pv_grregs, vm->save_area.guest.grs,sizeof (vm->save_area.guest.grs));
@@ -53,6 +55,16 @@ void sie(struct vm *vm)/* Reset icptcode so we don't trip over it below */ vm->sblk->icptcode =0 ; +/* set up home address space to match primary space */ + old_cr13 =stctg (13 ); +lctlg (13 ,stctg (1 )); + +/* switch to home space so guest tables can be different from host */ +psw_mask_set_bits (PSW_MASK_HOME ); + +/* also handle all interruptions in home space while in SIE */ +irq_set_dat_mode (IRQ_DAT_ON ,AS_HOME ); +while (vm->sblk->icptcode ==0 ) {sie64a (vm->sblk, &vm->save_area);sie_handle_validity (vm);
@@ -60,6 +72,12 @@ void sie(struct vm *vm) vm->save_area.guest.grs[14 ] = vm->sblk->gg14; vm->save_area.guest.grs[15 ] = vm->sblk->gg15; +irq_set_dat_mode (IRQ_DAT_ON ,AS_PRIM ); +psw_mask_clear_bits (PSW_MASK_HOME ); + +/* restore the old CR 13 */ +lctlg (13 , old_cr13); +if (vm->sblk->sdf ==2 )memcpy (vm->save_area.guest.grs, vm->sblk->pv_grregs,sizeof (vm->save_area.guest.grs));
Signature · 1 line
2.39.1
PATCH v4 4/6 s390x: lib: don't forward PSW when handling exception in SIE
In reply to Nico BoehrWhen we're handling a pgm int in SIE, we want to return to the SIE cleanup after handling the exception. That's why we set pgm_old_psw to the sie_exit label in fixup_pgm_int.
On nullifing pgm ints, fixup_pgm_int will also forward the old PSW such that we don't cause an pgm int again.
However, when we want to return to the sie_exit label, this is not needed (since we've manually set pgm_old_psw). Instead, forwarding the PSW might cause us to skip an instruction or end up in the middle of an instruction.
So, let's just skip the rest of the fixup in case we're inside SIE.
Note that we're intentionally not fixing up the PSW in the guest; that's best left to the test at hand by registering their own psw fixup.
--- lib/s390x/interrupt.c | 1 + 1 file changed, 1 insertion(+)
lib/s390x/interrupt.c +1 -0
@@ -145,6 +145,7 @@ static void fixup_pgm_int(struct stack_frame_int *stack)if (lowcore.pgm_old_psw.addr >= (uint64_t )&sie_entry && lowcore.pgm_old_psw.addr <= (uint64_t )&sie_exit) { lowcore.pgm_old_psw.addr = (uint64_t )&sie_exit; +return ; }switch (lowcore.pgm_int_code) {
Signature · 1 line
2.39.1
PATCH v4 5/6 s390x: lib: sie: don't reenter SIE on pgm int
In reply to Nico BoehrAt the moment, when a PGM int occurs while in SIE, we will just reenter SIE after the interrupt handler was called.
This is because sie() has a loop which checks icptcode and re-enters SIE if it is zero.
However, this behaviour is quite undesirable for SIE tests, since it doesn't give the host the chance to assert on the PGM int. Instead, we will just re-enter SIE, on nullifing conditions even causing the exception again.
In sie(), check whether a pgm int code is set in lowcore. If it has, exit the loop so the test can react to the interrupt. Add a new function read_pgm_int_code() to obtain the interrupt code.
Note that this introduces a slight oddity with sie and pgm int in certain cases: If a PGM int occurs between a expect_pgm_int() and sie(), we will now never enter SIE until the pgm_int_code is cleared by e.g. clear_pgm_int().
Also add missing include of facility.h to mem.h.
--- lib/s390x/asm/interrupt.h | 14 ++++++++++++++ lib/s390x/asm/mem.h | 1 + lib/s390x/sie.c | 4 +++- 3 files changed, 18 insertions(+), 1 deletion(-)
lib/s390x/asm/interrupt.h +14 -0
@@ -99,4 +99,18 @@ static inline void low_prot_disable(void) ctl_clear_bit(0, CTL0_LOW_ADDR_PROT); } +/** +* read_pgm_int_code - Get the program interruption code of the last pgm int +* on the current CPU. +* +* This is similar to clear_pgm_int(), except that it doesn't clear the +* interruption information from lowcore. +* +* Returns 0 when none occurred. +*/ +static inline uint16_t read_pgm_int_code (void ) +{ +return lowcore.pgm_int_code; +} +#endif
lib/s390x/asm/mem.h +1 -0
@@ -8,6 +8,7 @@#ifndef _ASMS390X_MEM_H_#define _ASMS390X_MEM_H_#include <asm/arch_def.h> +#include <asm/facility.h> /* create pointer while avoiding compiler warnings */ #define OPAQUE_PTR (x) ((void *)(((uint64_t)&lowcore) + (x)))
lib/s390x/sie.c +3 -1
@@ -13,6 +13,7 @@#include <libcflat.h> #include <sie.h> #include <asm/page.h> +#include <asm/interrupt.h> #include <libcflat.h> #include <alloc_page.h>
@@ -65,7 +66,8 @@ void sie(struct vm *vm)/* also handle all interruptions in home space while in SIE */ irq_set_dat_mode (IRQ_DAT_ON ,AS_HOME ); -while (vm->sblk->icptcode ==0 ) { +/* leave SIE when we have an intercept or an interrupt so the test can react to it */ +while (vm->sblk->icptcode ==0 && !read_pgm_int_code ()) {sie64a (vm->sblk, &vm->save_area);sie_handle_validity (vm); }
Signature · 1 line
2.39.1
PATCH v4 6/6 s390x: add a test for SIE without MSO/MSL
In reply to Nico BoehrSince we now have the ability to run guests without MSO/MSL, add a test to make sure this doesn't break.
--- s390x/Makefile | 2 + s390x/sie-dat.c | 115 +++++++++++++++++++++++++++++++++++++ s390x/snippets/c/sie-dat.c | 58 +++++++++++++++++++ s390x/unittests.cfg | 3 + 4 files changed, 178 insertions(+) create mode 100644 s390x/sie-dat.c create mode 100644 s390x/snippets/c/sie-dat.c
s390x/Makefile +2 -0
@@ -40,6 +40,7 @@ tests += $(TEST_DIR)/panic-loop-pgm.elf tests += $(TEST_DIR)/migration-sck.elf tests += $(TEST_DIR)/exittime.elf tests += $(TEST_DIR)/ex.elf +tests += $(TEST_DIR)/sie-dat.elf pv-tests += $(TEST_DIR)/pv-diags.elf
@@ -120,6 +121,7 @@ snippet_lib = $(snippet_asmlib) lib/auxinfo.o # perquisites (=guests) for the snippet hosts. # $(TEST_DIR)/<snippet-host>.elf: snippets = $(SNIPPET_DIR)/<c/asm>/<snippet>.gbin $(TEST_DIR)/mvpg-sie.elf: snippets = $(SNIPPET_DIR)/c/mvpg-snippet.gbin +$(TEST_DIR)/sie-dat.elf: snippets = $(SNIPPET_DIR)/c/sie-dat.gbin $(TEST_DIR)/spec_ex-sie.elf: snippets = $(SNIPPET_DIR)/c/spec_ex.gbin $(TEST_DIR)/pv-diags.elf: pv-snippets += $(SNIPPET_DIR)/asm/snippet-pv-diag-yield.gbin
s390x/sie-dat.c new file +115 -0
115 lines not on this page · raw message
s390x/snippets/c/sie-dat.c new file +58 -0
58 lines not on this page · raw message
s390x/unittests.cfg +3 -0
@@ -218,3 +218,6 @@ extra_params = -append '--parallel' [execute] file = ex.elf + +[sie-dat] +file = sie-dat.elf
Signature · 1 line
2.39.1