s390x: Add support for running guests without MSO/MSL
cover s390x: Add support for running guests without MSO/MSL
Right now, all SIE tests in kvm-unit-tests (i.e. where kvm-unit-test is the hypervisor) run using MSO/MSL.
This is convenient, because it's simple. But it also comes with disadvantages, for example some features are unavailabe with MSO/MSL.
This series adds support for running guests without MSO/MSL with dedicated guest page tables for the GPA->HPA translation.
Since SIE implicitly uses the primary space mode for the guest, the host can't run in the primary space mode, too. To avoid moving all tests to the home space mode, only switch to home space mode when it is actually needed.
This series also comes with various bugfixes that were caught while develoing this.
Nico Boehr (4):
s390x: sie: switch to home space mode before entering SIE
s390x: lib: don't forward PSW when handling exception in SIE
s390x: lib: sie: don't reenter SIE on pgm int
s390x: add a test for SIE without MSO/MSL
lib/s390x/asm/arch_def.h | 2 + lib/s390x/interrupt.c | 7 +++ lib/s390x/sie.c | 36 ++++++++++- lib/s390x/sie.h | 2 + s390x/Makefile | 2 + s390x/sie-dat.c | 121 +++++++++++++++++++++++++++++++++++++ s390x/snippets/c/sie-dat.c | 58 ++++++++++++++++++ s390x/unittests.cfg | 3 + 8 files changed, 230 insertions(+), 1 deletion(-) create mode 100644 s390x/sie-dat.c create mode 100644 s390x/snippets/c/sie-dat.c
Signature · 1 line
2.39.1
PATCH 1/4 s390x: sie: switch to home space mode before entering SIE
This is to prepare for running guests without MSO/MSL, which is currently not possible.
We already have code in sie64a to setup a guest primary ASCE before entering SIE, so we can in theory switch to the page tables which translate gpa to hpa.
But the host is running in primary space mode already, so changing the primary ASCE before entering SIE will also affect the host's code and data.
To make this switch useful, the host should run in a different address space mode. Hence, set up and change to home address space mode before installing the guest ASCE.
The home space ASCE is just copied over from the primary space ASCE, so no functional change is intended, also for tests that want to use MSO/MSL. If a test intends to use a different primary space ASCE, it can now just set the guest.asce in the save_area.
--- lib/s390x/asm/arch_def.h | 2 ++ lib/s390x/sie.c | 26 ++++++++++++++++++++++++++ lib/s390x/sie.h | 1 + 3 files changed, 29 insertions(+)
lib/s390x/asm/arch_def.h +2 -0
@@ -67,6 +67,8 @@ struct cpu {#define AS_HOME 3#define PSW_MASK_DAT 0x0400000000000000UL +#define PSW_MASK_DAT_HOME 0x0400C00000000000UL +#define PSW_MASK_HOME 0x0000C00000000000UL#define PSW_MASK_IO 0x0200000000000000UL#define PSW_MASK_EXT 0x0100000000000000UL#define PSW_MASK_KEY 0x00F0000000000000UL
lib/s390x/sie.c +26 -0
@@ -46,6 +46,8 @@ void sie_handle_validity(struct vm *vm)void sie (struct vm *vm) { +uint64_t old_cr13; +if (vm->sblk->sdf ==2 )memcpy (vm->sblk->pv_grregs, vm->save_area.guest.grs,sizeof (vm->save_area.guest.grs));
@@ -53,6 +55,19 @@ void sie(struct vm *vm)/* Reset icptcode so we don't trip over it below */ vm->sblk->icptcode =0 ; +/* set up home address space to match primary space */ + old_cr13 =stctg (13 ); +lctlg (13 ,stctg (1 )); + +/* switch to home space so guest tables can be different from host */ +psw_mask_set_bits (PSW_MASK_HOME ); + +/* also handle all interruptions in home space while in SIE */ + lowcore.pgm_new_psw.mask |=PSW_MASK_DAT_HOME ; + lowcore.ext_new_psw.mask |=PSW_MASK_DAT_HOME ; + lowcore.io_new_psw.mask |=PSW_MASK_DAT_HOME ; +mb (); +while (vm->sblk->icptcode ==0 ) {sie64a (vm->sblk, &vm->save_area);sie_handle_validity (vm);
@@ -60,6 +75,17 @@ void sie(struct vm *vm) vm->save_area.guest.grs[14 ] = vm->sblk->gg14; vm->save_area.guest.grs[15 ] = vm->sblk->gg15; + lowcore.pgm_new_psw.mask &= ~PSW_MASK_HOME ; + lowcore.ext_new_psw.mask &= ~PSW_MASK_HOME ; + lowcore.io_new_psw.mask &= ~PSW_MASK_HOME ; +mb (); + +psw_mask_clear_bits (PSW_MASK_HOME ); + +/* restore the old CR 13 */ +lctlg (13 , old_cr13); + +if (vm->sblk->sdf ==2 )memcpy (vm->save_area.guest.grs, vm->sblk->pv_grregs,sizeof (vm->save_area.guest.grs));
lib/s390x/sie.h +1 -0
@@ -284,5 +284,6 @@ void sie_handle_validity(struct vm *vm);void sie_guest_sca_create (struct vm *vm);void sie_guest_create (struct vm *vm,uint64_t guest_mem,uint64_t guest_mem_len);void sie_guest_destroy (struct vm *vm); +bool sie_had_pgm_int (struct vm *vm);#endif /* _S390X_SIE_H_ */
Signature · 1 line
2.39.1
Re: [kvm-unit-tests PATCH v1 1/4] s390x: sie: switch to home space mode before entering SIE
Quoted 20 lines · Nico Boehr
This is to prepare for running guests without MSO/MSL, which is currently not possible.
We already have code in sie64a to setup a guest primary ASCE before entering SIE, so we can in theory switch to the page tables which translate gpa to hpa.
But the host is running in primary space mode already, so changing the primary ASCE before entering SIE will also affect the host's code and data.
To make this switch useful, the host should run in a different address space mode. Hence, set up and change to home address space mode before installing the guest ASCE.
The home space ASCE is just copied over from the primary space ASCE, so no functional change is intended, also for tests that want to use MSO/MSL. If a test intends to use a different primary space ASCE, it can now just set the guest.asce in the
save_area.
[...]
lib/s390x/sie.c void sie(struct vm *vm) · line 68
Quoted 12 lines
lib/s390x/sie.c +11
-0+/* set up home address space to match primary space */ + old_cr13 =stctg (13 ); +lctlg (13 ,stctg (1 )); + +/* switch to home space so guest tables can be different from host */ +psw_mask_set_bits (PSW_MASK_HOME ); + +/* also handle all interruptions in home space while in SIE */ + lowcore.pgm_new_psw.mask |=PSW_MASK_DAT_HOME ; + lowcore.ext_new_psw.mask |=PSW_MASK_DAT_HOME ; + lowcore.io_new_psw.mask |=PSW_MASK_DAT_HOME ;
We didn't enable DAT in these two cases as far as I can see so this is superfluous or we should change the mmu code. Also it's missing the svc and machine check.
The whole bit manipulation thing looks a bit crude. It might make more sense to drop into real mode for a few instructions and have a dedicated storage location for an extended PSW mask and an interrupt ASCE as part of the interrupt call code instead.
Opinions?
Quoted 34 lines
lib/s390x/sie.c +13
-0+mb (); +while (vm->sblk->icptcode ==0 ) {sie64a (vm->sblk, &vm->save_area);sie_handle_validity (vm);@@ -60,6 +75,17 @@ void sie(struct vm *vm) vm->save_area.guest.grs[14 ] = vm->sblk->gg14; vm->save_area.guest.grs[15 ] = vm->sblk->gg15; + lowcore.pgm_new_psw.mask &= ~PSW_MASK_HOME ; + lowcore.ext_new_psw.mask &= ~PSW_MASK_HOME ; + lowcore.io_new_psw.mask &= ~PSW_MASK_HOME ; +mb (); + +psw_mask_clear_bits (PSW_MASK_HOME ); + +/* restore the old CR 13 */ +lctlg (13 , old_cr13); + +if (vm->sblk->sdf ==2 )memcpy (vm->save_area.guest.grs, vm->sblk->pv_grregs,sizeof (vm->save_area.guest.grs));lib/s390x/sie.h +1
-0@@ -284,5 +284,6 @@ void sie_handle_validity(struct vm *vm);void sie_guest_sca_create (struct vm *vm);void sie_guest_create (struct vm *vm,uint64_t guest_mem,uint64_t guest_mem_len);void sie_guest_destroy (struct vm *vm); +bool sie_had_pgm_int (struct vm *vm);#endif /* _S390X_SIE_H_ */
Re: [kvm-unit-tests PATCH v1 1/4] s390x: sie: switch to home space mode before entering SIE
Quoting Janosch Frank (2023-03-28 16:13:04)
Quoted 32 lines · Nico Boehr
Quoted 20 lines
This is to prepare for running guests without MSO/MSL, which is currently not possible.
We already have code in sie64a to setup a guest primary ASCE before entering SIE, so we can in theory switch to the page tables which translate gpa to hpa.
But the host is running in primary space mode already, so changing the primary ASCE before entering SIE will also affect the host's code and data.
To make this switch useful, the host should run in a different address space mode. Hence, set up and change to home address space mode before installing the guest ASCE.
The home space ASCE is just copied over from the primary space ASCE, so no functional change is intended, also for tests that want to use MSO/MSL. If a test intends to use a different primary space ASCE, it can now just set the guest.asce in the
save_area.[...]
Quoted 9 lines
lib/s390x/sie.c +9
-0+/* set up home address space to match primary space */ + old_cr13 =stctg (13 ); +lctlg (13 ,stctg (1 )); + +/* switch to home space so guest tables can be different from host */ +psw_mask_set_bits (PSW_MASK_HOME ); + +/* also handle all interruptions in home space while in SIE */ + lowcore.pgm_new_psw.mask |=PSW_MASK_DAT_HOME ;
Quoted 5 lines
lib/s390x/sie.c void sie(struct vm *vm) · line 68
Quoted 2 lines
lib/s390x/sie.c +2
-0+ lowcore.ext_new_psw.mask |=PSW_MASK_DAT_HOME ; + lowcore.io_new_psw.mask |=PSW_MASK_DAT_HOME ;We didn't enable DAT in these two cases as far as I can see so this is superfluous or we should change the mmu code. Also it's missing the svc and machine check.
Right. Is there a particular reason why we only run DAT on for PGM ints?
Quoted 6 lines
The whole bit manipulation thing looks a bit crude. It might make more sense to drop into real mode for a few instructions and have a dedicated storage location for an extended PSW mask and an interrupt ASCE as part of the interrupt call code instead.
Opinions?
Maybe I don't get it, but I personally don't quite see the advantage. It seems to me this would make things much more complicated just to avoid a few simple bitops.
It maybe also depends on how many new_psws we have to touch. If it's really just the PGM, the current solution seems simple enough.
But if others also prefer Janosch's suggestion, I am happy to implement it.
Re: [kvm-unit-tests PATCH v1 1/4] s390x: sie: switch to home space mode before entering SIE
Quoted 33 lines · On Wed, 29 Mar 2023 14:50:50 +0200 Nico Boehr <[email protected]> wrote:
Quoting Janosch Frank (2023-03-28 16:13:04)
Quoted 32 lines · Nico Boehr
Quoted 20 lines
This is to prepare for running guests without MSO/MSL, which is currently not possible.
We already have code in sie64a to setup a guest primary ASCE before entering SIE, so we can in theory switch to the page tables which translate gpa to hpa.
But the host is running in primary space mode already, so changing the primary ASCE before entering SIE will also affect the host's code and data.
To make this switch useful, the host should run in a different address space mode. Hence, set up and change to home address space mode before installing the guest ASCE.
The home space ASCE is just copied over from the primary space ASCE, so no functional change is intended, also for tests that want to use MSO/MSL. If a test intends to use a different primary space ASCE, it can now just set the guest.asce in the
save_area.[...]
Quoted 9 lines
lib/s390x/sie.c +9
-0+/* set up home address space to match primary space */ + old_cr13 =stctg (13 ); +lctlg (13 ,stctg (1 )); + +/* switch to home space so guest tables can be different from host */ +psw_mask_set_bits (PSW_MASK_HOME ); + +/* also handle all interruptions in home space while in SIE */ + lowcore.pgm_new_psw.mask |=PSW_MASK_DAT_HOME ;
Quoted 7 lines
Quoted 5 lines
lib/s390x/sie.c void sie(struct vm *vm) · line 68
Quoted 2 lines
lib/s390x/sie.c +2
-0+ lowcore.ext_new_psw.mask |=PSW_MASK_DAT_HOME ; + lowcore.io_new_psw.mask |=PSW_MASK_DAT_HOME ;We didn't enable DAT in these two cases as far as I can see so this is superfluous or we should change the mmu code. Also it's missing the svc and machine check.
Right. Is there a particular reason why we only run DAT on for PGM ints?
a fixup handler for PGM it might need to run with DAT on (e.g. to access data that is not identity mapped), whereas for other interrupts it's not needed (at least not yet ;) )
Quoted 16 lines
Quoted 7 lines
The whole bit manipulation thing looks a bit crude. It might make more sense to drop into real mode for a few instructions and have a dedicated storage location for an extended PSW mask and an interrupt ASCE as part of the interrupt call code instead.
Opinions?
Maybe I don't get it, but I personally don't quite see the advantage. It seems to me this would make things much more complicated just to avoid a few simple bitops.
It maybe also depends on how many
new_pswswe have to touch. If it's really just the PGM, the current solution seems simple enough.But if others also prefer Janosch's suggestion, I am happy to implement it.
Re: [kvm-unit-tests PATCH v1 1/4] s390x: sie: switch to home space mode before entering SIE
Quoted 42 lines · Claudio Imbrenda
Quoted 39 lines · On Wed, 29 Mar 2023 14:50:50 +0200 Nico Boehr <[email protected]> wrote:
Quoting Janosch Frank (2023-03-28 16:13:04)
Quoted 37 lines · Nico Boehr
Quoted 20 lines
This is to prepare for running guests without MSO/MSL, which is currently not possible.
We already have code in sie64a to setup a guest primary ASCE before entering SIE, so we can in theory switch to the page tables which translate gpa to hpa.
But the host is running in primary space mode already, so changing the primary ASCE before entering SIE will also affect the host's code and data.
To make this switch useful, the host should run in a different address space mode. Hence, set up and change to home address space mode before installing the guest ASCE.
The home space ASCE is just copied over from the primary space ASCE, so no functional change is intended, also for tests that want to use MSO/MSL. If a test intends to use a different primary space ASCE, it can now just set the guest.asce in the
save_area.[...]
lib/s390x/sie.c void sie(struct vm *vm) · line 68
Quoted 12 lines
lib/s390x/sie.c +11
-0+/* set up home address space to match primary space */ + old_cr13 =stctg (13 ); +lctlg (13 ,stctg (1 )); + +/* switch to home space so guest tables can be different from host */ +psw_mask_set_bits (PSW_MASK_HOME ); + +/* also handle all interruptions in home space while in SIE */ + lowcore.pgm_new_psw.mask |=PSW_MASK_DAT_HOME ; + lowcore.ext_new_psw.mask |=PSW_MASK_DAT_HOME ; + lowcore.io_new_psw.mask |=PSW_MASK_DAT_HOME ;We didn't enable DAT in these two cases as far as I can see so this is superfluous or we should change the mmu code. Also it's missing the svc and machine check.
Quoted 5 lines
Quoted 1 line
Right. Is there a particular reason why we only run DAT on for PGM ints?
a fixup handler for PGM it might need to run with DAT on (e.g. to access data that is not identity mapped), whereas for other interrupts it's not needed (at least not yet ;) )
At the time where the mmu code was written, the other handlers were very basic or a direct abort (like the IO IRQ that was introduced by Pierre). But I'd rather have them all behave the same so they are at least consistent.
If we don't introduce a location where we load the PSW from, then add a function that sets the masks for all IRQs and also convert the mmu enablement to use it.
Something to the likes of:
void irq_new_set_mask(bool dat, uint8_t as)
{
loop over psws {
- Remove dat and as bits from new PSW
- Or in the new dat + as bits
}
}Re: [kvm-unit-tests PATCH v1 1/4] s390x: sie: switch to home space mode before entering SIE
In reply to Claudio ImbrendaQuoting Claudio Imbrenda (2023-03-29 15:00:32)
Quoted 42 lines
Quoted 39 lines · On Wed, 29 Mar 2023 14:50:50 +0200 Nico Boehr <[email protected]> wrote:
Quoting Janosch Frank (2023-03-28 16:13:04)
Quoted 37 lines · Nico Boehr
Quoted 20 lines
This is to prepare for running guests without MSO/MSL, which is currently not possible.
We already have code in sie64a to setup a guest primary ASCE before entering SIE, so we can in theory switch to the page tables which translate gpa to hpa.
But the host is running in primary space mode already, so changing the primary ASCE before entering SIE will also affect the host's code and data.
To make this switch useful, the host should run in a different address space mode. Hence, set up and change to home address space mode before installing the guest ASCE.
The home space ASCE is just copied over from the primary space ASCE, so no functional change is intended, also for tests that want to use MSO/MSL. If a test intends to use a different primary space ASCE, it can now just set the guest.asce in the
save_area.[...]
lib/s390x/sie.c void sie(struct vm *vm) · line 68
Quoted 12 lines
lib/s390x/sie.c +11
-0+/* set up home address space to match primary space */ + old_cr13 =stctg (13 ); +lctlg (13 ,stctg (1 )); + +/* switch to home space so guest tables can be different from host */ +psw_mask_set_bits (PSW_MASK_HOME ); + +/* also handle all interruptions in home space while in SIE */ + lowcore.pgm_new_psw.mask |=PSW_MASK_DAT_HOME ; + lowcore.ext_new_psw.mask |=PSW_MASK_DAT_HOME ; + lowcore.io_new_psw.mask |=PSW_MASK_DAT_HOME ;We didn't enable DAT in these two cases as far as I can see so this is superfluous or we should change the mmu code. Also it's missing the svc and machine check.
Quoted 5 lines
Quoted 1 line
Right. Is there a particular reason why we only run DAT on for PGM ints?
a fixup handler for PGM it might need to run with DAT on (e.g. to access data that is not identity mapped), whereas for other interrupts it's not needed (at least not yet ;) )
Makes sense.
Since one can register a cleanup for io and ext, too, I think we need to fix the mmu init for these cases while at it.
Will do that in v2.
PATCH 2/4 s390x: lib: don't forward PSW when handling exception in SIE
In reply to Nico BoehrWhen we're handling a pgm int in SIE, we want to return to the SIE cleanup after handling the exception. That's why we set pgm_old_psw to the sie_exit label in fixup_pgm_int.
On nullifing pgm ints, fixup_pgm_int will also forward the old PSW such that we don't cause an pgm int again.
However, when we want to return to the sie_exit label, this is not needed (since we've manually set pgm_old_psw). Instead, forwarding the PSW might cause us to skip an instruction or end up in the middle of an instruction.
So, let's just skip the rest of the fixup in case we're inside SIE.
Note that we're intentionally not fixing up the PSW in the guest; that's best left to the test at hand by registering their own psw fixup.
--- lib/s390x/interrupt.c | 1 + 1 file changed, 1 insertion(+)
lib/s390x/interrupt.c +1 -0
@@ -110,6 +110,7 @@ static void fixup_pgm_int(struct stack_frame_int *stack)if (lowcore.pgm_old_psw.addr >= (uint64_t )&sie_entry && lowcore.pgm_old_psw.addr <= (uint64_t )&sie_exit) { lowcore.pgm_old_psw.addr = (uint64_t )&sie_exit; +return ; }switch (lowcore.pgm_int_code) {
Signature · 1 line
2.39.1
PATCH 3/4 s390x: lib: sie: don't reenter SIE on pgm int
In reply to Nico BoehrAt the moment, when a PGM int occurs while in SIE, we will just reenter SIE after the interrupt handler was called.
This is because sie() has a loop which checks icptcode and re-enters SIE if it is zero.
However, this behaviour is quite undesirable for SIE tests, since it doesn't give the host the chance to assert on the PGM int. Instead, we will just re-enter SIE, on nullifing conditions even causing the exception again.
Add a flag PROG_PGM_IN_SIE set by the pgm int fixup which indicates a program interrupt has occured in SIE. Check for the flag in sie() and if it's set return from sie() to give the host the ability to react on the exception. The host may check if a PGM int has occured in the guest using the new function sie_had_pgm_int().
--- lib/s390x/interrupt.c | 6 ++++++ lib/s390x/sie.c | 10 +++++++++- lib/s390x/sie.h | 1 + 3 files changed, 16 insertions(+), 1 deletion(-)
lib/s390x/interrupt.c +6 -0
@@ -106,10 +106,16 @@ void register_ext_cleanup_func(void (*f)(struct stack_frame_int *))static void fixup_pgm_int (struct stack_frame_int *stack) { +struct kvm_s390_sie_block *sblk; +/* If we have an error on SIE we directly move to sie_exit */ if (lowcore.pgm_old_psw.addr >= (uint64_t )&sie_entry && lowcore.pgm_old_psw.addr <= (uint64_t )&sie_exit) { lowcore.pgm_old_psw.addr = (uint64_t )&sie_exit; + +/* set a marker in sie_block that a PGM int occured */ + sblk = *((struct kvm_s390_sie_block **)(stack->grs0[13 ] + __SF_SIE_CONTROL)); + sblk->prog0c |=PROG_PGM_IN_SIE ;return ; }
lib/s390x/sie.c +9 -1
@@ -44,6 +44,11 @@ void sie_handle_validity(struct vm *vm) vm->validity_expected = false; } +bool sie_had_pgm_int (struct vm *vm) +{ +return vm->sblk->prog0c &PROG_PGM_IN_SIE ; +} +void sie (struct vm *vm) {uint64_t old_cr13;
@@ -68,7 +73,10 @@ void sie(struct vm *vm) lowcore.io_new_psw.mask |=PSW_MASK_DAT_HOME ;mb (); -while (vm->sblk->icptcode ==0 ) { +/* clear PGM int marker, which might still be set */ + vm->sblk->prog0c &= ~PROG_PGM_IN_SIE ; + +while (vm->sblk->icptcode ==0 && !sie_had_pgm_int (vm)) {sie64a (vm->sblk, &vm->save_area);sie_handle_validity (vm); }
lib/s390x/sie.h +1 -0
@@ -37,6 +37,7 @@ struct kvm_s390_sie_block { uint32_t ibc : 12;uint8_t reserved08[4 ];/* 0x0008 */ #define PROG_IN_SIE (1<<0) +#define PROG_PGM_IN_SIE (1<<1)uint32_t prog0c;/* 0x000c */ union {uint8_t reserved10[16 ];/* 0x0010 */
Signature · 1 line
2.39.1
Re: [kvm-unit-tests PATCH v1 3/4] s390x: lib: sie: don't reenter SIE on pgm int
Quoted 11 lines · Nico Boehr
At the moment, when a PGM int occurs while in SIE, we will just reenter SIE after the interrupt handler was called.
This is because
sie()has a loop which checks icptcode and re-enters SIE if it is zero.However, this behaviour is quite undesirable for SIE tests, since it doesn't give the host the chance to assert on the PGM int. Instead, we will just re-enter SIE, on nullifing conditions even causing the exception again.
Quoted 5 lines
Add a flag
PROG_PGM_IN_SIEset by the pgm int fixup which indicates a program interrupt has occured in SIE. Check for the flag insie()and if it's set return fromsie()to give the host the ability to react on the exception. The host may check if a PGM int has occured in the guest using the new functionsie_had_pgm_int().
We could simply check "!lowcore.pgm_int_code" by introducing:uint16_t read_pgm_int(void)
{
mb();
return lowcore.pgm_int_code;
}into interrupt.c.
Or to be a bit more verbose:
I don't see a reason why we'd want to store a per sblk PGM in SIE bit when all we want to know is either: was there a PGM at all (to stop the SIE loop) or was there a PGM between the expect and the check_pgm_int_code().
Re: [kvm-unit-tests PATCH v1 3/4] s390x: lib: sie: don't reenter SIE on pgm int
Quoting Janosch Frank (2023-03-28 15:42:26)
Quoted 25 lines · Nico Boehr
Quoted 11 lines
At the moment, when a PGM int occurs while in SIE, we will just reenter SIE after the interrupt handler was called.
This is because
sie()has a loop which checks icptcode and re-enters SIE if it is zero.However, this behaviour is quite undesirable for SIE tests, since it doesn't give the host the chance to assert on the PGM int. Instead, we will just re-enter SIE, on nullifing conditions even causing the exception again.
Quoted 5 lines
Add a flag
PROG_PGM_IN_SIEset by the pgm int fixup which indicates a program interrupt has occured in SIE. Check for the flag insie()and if it's set return fromsie()to give the host the ability to react on the exception. The host may check if a PGM int has occured in the guest using the new functionsie_had_pgm_int().We could simply check "!lowcore.pgm_int_code" by introducing:
uint16_tread_pgm_int(void){ mb(); return lowcore.pgm_int_code; }
Quoted 8 lines
into
interrupt.c.Or to be a bit more verbose:
I don't see a reason why we'd want to store a per sblk PGM in SIE bit when all we want to know is either: was there a PGM at all (to stop the SIE loop) or was there a PGM between the expect and thecheck_pgm_int_code().
Yes, makes perfect sense, I just didn't see this possiblity. Thanks, will change.
Re: [kvm-unit-tests PATCH v1 3/4] s390x: lib: sie: don't reenter SIE on pgm int
In reply to Nico BoehrQuoted 78 lines · On Mon, 27 Mar 2023 10:21:17 +0200 Nico Boehr <[email protected]> wrote:
At the moment, when a PGM int occurs while in SIE, we will just reenter SIE after the interrupt handler was called.
This is because
sie()has a loop which checks icptcode and re-enters SIE if it is zero.However, this behaviour is quite undesirable for SIE tests, since it doesn't give the host the chance to assert on the PGM int. Instead, we will just re-enter SIE, on nullifing conditions even causing the exception again.
Add a flag
PROG_PGM_IN_SIEset by the pgm int fixup which indicates a program interrupt has occured in SIE. Check for the flag insie()and if it's set return fromsie()to give the host the ability to react on the exception. The host may check if a PGM int has occured in the guest using the new functionsie_had_pgm_int().Signed-off-by Nico Boehr <[email protected]>--- lib/s390x/interrupt.c | 6 ++++++ lib/s390x/sie.c | 10 +++++++++- lib/s390x/sie.h | 1 + 3 files changed, 16 insertions(+), 1 deletion(-)lib/s390x/interrupt.c +6
-0@@ -106,10 +106,16 @@ void register_ext_cleanup_func(void (*f)(struct stack_frame_int *))static void fixup_pgm_int (struct stack_frame_int *stack) { +struct kvm_s390_sie_block *sblk; +/* If we have an error on SIE we directly move to sie_exit */ if (lowcore.pgm_old_psw.addr >= (uint64_t )&sie_entry && lowcore.pgm_old_psw.addr <= (uint64_t )&sie_exit) { lowcore.pgm_old_psw.addr = (uint64_t )&sie_exit; + +/* set a marker in sie_block that a PGM int occured */ + sblk = *((struct kvm_s390_sie_block **)(stack->grs0[13 ] + __SF_SIE_CONTROL)); + sblk->prog0c |=PROG_PGM_IN_SIE ;return ; }lib/s390x/sie.c +9
-1@@ -44,6 +44,11 @@ void sie_handle_validity(struct vm *vm) vm->validity_expected = false; } +bool sie_had_pgm_int (struct vm *vm) +{ +return vm->sblk->prog0c &PROG_PGM_IN_SIE ; +} +void sie (struct vm *vm) {uint64_t old_cr13;@@ -68,7 +73,10 @@ void sie(struct vm *vm) lowcore.io_new_psw.mask |=PSW_MASK_DAT_HOME ;mb (); -while (vm->sblk->icptcode ==0 ) { +/* clear PGM int marker, which might still be set */ + vm->sblk->prog0c &= ~PROG_PGM_IN_SIE ; + +while (vm->sblk->icptcode ==0 && !sie_had_pgm_int (vm)) {sie64a (vm->sblk, &vm->save_area);sie_handle_validity (vm); }lib/s390x/sie.h +0
-0
lib/s390x/sie.h struct kvm_s390_sie_block { · line 40
Quoted 4 lines
lib/s390x/sie.h +1
-0@@ -37,6 +37,7 @@ struct kvm_s390_sie_block { uint32_t ibc : 12;uint8_t reserved08[4 ];/* 0x0008 */ #define PROG_IN_SIE (1<<0) +#define PROG_PGM_IN_SIE (1<<1)
please align the body of the macros with tabs, so they are more readable
Quoted 3 lines
lib/s390x/sie.h +0
-0uint32_t prog0c;/* 0x000c */ union {uint8_t reserved10[16 ];/* 0x0010 */
Re: [kvm-unit-tests PATCH v1 3/4] s390x: lib: sie: don't reenter SIE on pgm int
Quoting Claudio Imbrenda (2023-03-28 19:01:06)
[...]
Quoted 11 lines
lib/s390x/sie.h struct kvm_s390_sie_block { · line 40
Quoted 9 lines
lib/s390x/sie.h +1
-0@@ -37,6 +37,7 @@ struct kvm_s390_sie_block { uint32_t ibc : 12;uint8_t reserved08[4 ];/* 0x0008 */ #define PROG_IN_SIE (1<<0) +#define PROG_PGM_IN_SIE (1<<1)please align the body of the macros with tabs, so they are more readable
Thanks, would do, but this is gonna go away anyways in favor of Janoschs suggestion.
PATCH 4/4 s390x: add a test for SIE without MSO/MSL
In reply to Nico BoehrSince we now have the ability to run guests without MSO/MSL, add a test to make sure this doesn't break.
--- s390x/Makefile | 2 + s390x/sie-dat.c | 121 +++++++++++++++++++++++++++++++++++++ s390x/snippets/c/sie-dat.c | 58 ++++++++++++++++++ s390x/unittests.cfg | 3 + 4 files changed, 184 insertions(+) create mode 100644 s390x/sie-dat.c create mode 100644 s390x/snippets/c/sie-dat.c
s390x/Makefile +2 -0
@@ -39,6 +39,7 @@ tests += $(TEST_DIR)/panic-loop-extint.elf tests += $(TEST_DIR)/panic-loop-pgm.elf tests += $(TEST_DIR)/migration-sck.elf tests += $(TEST_DIR)/exittime.elf +tests += $(TEST_DIR)/sie-dat.elf pv-tests += $(TEST_DIR)/pv-diags.elf
@@ -114,6 +115,7 @@ snippet_lib = $(snippet_asmlib) lib/auxinfo.o # perquisites (=guests) for the snippet hosts. # $(TEST_DIR)/<snippet-host>.elf: snippets = $(SNIPPET_DIR)/<c/asm>/<snippet>.gbin $(TEST_DIR)/mvpg-sie.elf: snippets = $(SNIPPET_DIR)/c/mvpg-snippet.gbin +$(TEST_DIR)/sie-dat.elf: snippets = $(SNIPPET_DIR)/c/sie-dat.gbin $(TEST_DIR)/spec_ex-sie.elf: snippets = $(SNIPPET_DIR)/c/spec_ex.gbin $(TEST_DIR)/pv-diags.elf: pv-snippets += $(SNIPPET_DIR)/asm/snippet-pv-diag-yield.gbin
s390x/sie-dat.c new file +121 -0
121 lines not on this page · raw message
s390x/snippets/c/sie-dat.c new file +58 -0
58 lines not on this page · raw message
s390x/unittests.cfg +3 -0
@@ -215,3 +215,6 @@ file = migration-skey.elf smp = 2 groups = migration extra_params = -append '--parallel' + +[sie-dat] +file = sie-dat.elf
Signature · 1 line
2.39.1
Re: [kvm-unit-tests PATCH v1 4/4] s390x: add a test for SIE without MSO/MSL
Quoted 4 lines · Nico Boehr
Since we now have the ability to run guests without MSO/MSL, add a test to make sure this doesn't break.
Signed-off-by Nico Boehr <[email protected]>
Quoted 9 lines
--- s390x/Makefile | 2 + s390x/sie-dat.c | 121 +++++++++++++++++++++++++++++++++++++ s390x/snippets/c/sie-dat.c | 58 ++++++++++++++++++ s390x/unittests.cfg | 3 + 4 files changed, 184 insertions(+) create mode 100644 s390x/sie-dat.c create mode 100644 s390x/snippets/c/sie-dat.c
Test looks good to me. Some comments below.
[...]
Quoted 7 lines
s390x/sie-dat.c new file +0
-0@@ -0,0 +1,121 @@
[...]
s390x/sie-dat.c line 32
Quoted 3 lines
s390x/sie-dat.c new file +3
-0+ +/* keep in sync with TOTAL_PAGE_COUNT in s390x/snippets/c/sie-dat.c */ +#define GUEST_TOTAL_PAGE_COUNT 256
This (1M) is the maximum snippet size (see snippet_setup_guest), is this intentional? In that case the comment is inaccurate, since you'd want to sync it with the maximum snippet size. You also know the actual snippet size SNIPPET_LEN(c, sie_dat) so I don't see why you'd need a define at all.
Quoted 4 lines
s390x/sie-dat.c new file +3
-0+ +static void test_sie_dat (void ) +{
[...]
s390x/sie-dat.c
Quoted 2 lines
s390x/sie-dat.c new file +2
-0+ +/* the guest will now write to an unmapped address and we check that this causes a segment translation */
I'd prefer "causes a segment translation exception"
[...]
Quoted 20 lines
s390x/snippets/c/sie-dat.c new file +14
-0@@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* Snippet used by the sie-dat.c test to verify paging without MSO/MSL +* +* Copyright (c) 2023 IBM Corp +* +* Authors: +* Nico Boehr <[email protected]> +*/ +#include <stddef.h> +#include <inttypes.h> +#include <string.h> +#include <asm-generic/page.h> +
s390x/snippets/c/sie-dat.c line 24
Quoted 10 lines
s390x/snippets/c/sie-dat.c new file +10
-0+/* keep in sync with GUEST_TEST_PAGE_COUNT in s390x/sie-dat.c */ +#define TEST_PAGE_COUNT 10 +static uint8_t test_page[TEST_PAGE_COUNT * PAGE_SIZE] __attribute__((__aligned__(PAGE_SIZE))); + +/* keep in sync with GUEST_TOTAL_PAGE_COUNT in s390x/sie-dat.c */ +#define TOTAL_PAGE_COUNT 256 + +static inline void force_exit (void ) +{ + asmvolatile (" diag 0,0,0x44\n" );
Pretty sure the compiler will generate a leading tab, so this will be doubly indented.
s390x/snippets/c/sie-dat.c line 35
Quoted 11 lines
s390x/snippets/c/sie-dat.c new file +11
-0+} + +static inline void force_exit_value (uint64_t val) +{ + asmvolatile ( +" diag %[val],0,0x9c\n" + : : [val]"d" (val) + ); +} + +__attribute__((section (".text" )))int main (void )
Why is the attribute necessary? I know all the snippets have it, but I don't see why it's necessary.
@Janosch ?
Quoted 6 lines
s390x/snippets/c/sie-dat.c new file +6
-0+{ +uint8_t *invalid_ptr; + +memset (test_page,0 ,sizeof (test_page)); +/* tell the host the page's physical address (we're running DAT off) */ +force_exit_value ((uint64_t )test_page);
s390x/snippets/c/sie-dat.c line 51
Quoted 10 lines
s390x/snippets/c/sie-dat.c new file +10
-0+ +/* write some value to the page so the host can verify it */ +for (size_t i =0 ; i <TEST_PAGE_COUNT ; i++) + test_page[i *PAGE_SIZE ] =42 + i; + +/* indicate we've written all pages */ +force_exit (); + +/* the first unmapped address */ + invalid_ptr = (uint8_t *)(TOTAL_PAGE_COUNT *PAGE_SIZE );
Why not just use an address high enough you know it will not be mapped? -1 should do just fine.
Quoted 18 lines
s390x/snippets/c/sie-dat.c new file +7
-0+ *invalid_ptr =42 ; + +/* indicate we've written the non-allowed page (should never get here) */ +force_exit (); + +return 0 ; +}s390x/unittests.cfg +3
-0@@ -215,3 +215,6 @@ file = migration-skey.elf smp = 2 groups = migration extra_params = -append '--parallel' + +[sie-dat] +file = sie-dat.elf
Re: [kvm-unit-tests PATCH v1 4/4] s390x: add a test for SIE without MSO/MSL
Quoted 6 lines · Nina Schoetterl-Glausch
Quoted 5 lines · Nico Boehr
Since we now have the ability to run guests without MSO/MSL, add a test to make sure this doesn't break.
Signed-off-by Nico Boehr <[email protected]>---
[...]
Quoted 5 lines
Quoted 5 lines
s390x/snippets/c/sie-dat.c new file +5
-0+static inline void force_exit_value (uint64_t val) +{ + asmvolatile ( +" diag %[val],0,0x9c\n" + : : [val] "d"(val)
Quoted 8 lines
s390x/snippets/c/sie-dat.c line 35
Quoted 4 lines
s390x/snippets/c/sie-dat.c new file +4
-0+ ); +} + +__attribute__((section (".text" )))int main (void )Why is the attribute necessary? I know all the snippets have it, but I don't see why it's necessary.
@Janosch ?
"Historical growth" :)
If it doesn't work without it then we need to find a way to fix it. If it does work without it then we should remove the attribute from all snippets.
But this issue is a low priority for me.
Quoted 40 lines
Quoted 7 lines
s390x/snippets/c/sie-dat.c new file +6
-0+{ +uint8_t *invalid_ptr; + +memset (test_page,0 ,sizeof (test_page)); +/* tell the host the page's physical address (we're running DAT off) */ +force_exit_value ((uint64_t )test_page);s390x/snippets/c/sie-dat.c line 51
Quoted 10 lines
s390x/snippets/c/sie-dat.c new file +10
-0+ +/* write some value to the page so the host can verify it */ +for (size_t i =0 ; i <TEST_PAGE_COUNT ; i++) + test_page[i *PAGE_SIZE ] =42 + i; + +/* indicate we've written all pages */ +force_exit (); + +/* the first unmapped address */ + invalid_ptr = (uint8_t *)(TOTAL_PAGE_COUNT *PAGE_SIZE );Why not just use an address high enough you know it will not be mapped? -1 should do just fine.
Quoted 18 lines
s390x/snippets/c/sie-dat.c new file +7
-0+ *invalid_ptr =42 ; + +/* indicate we've written the non-allowed page (should never get here) */ +force_exit (); + +return 0 ; +}s390x/unittests.cfg +3
-0@@ -215,3 +215,6 @@ file = migration-skey.elf smp = 2 groups = migration extra_params = -append '--parallel' + +[sie-dat] +file = sie-dat.elf
Re: [kvm-unit-tests PATCH v1 4/4] s390x: add a test for SIE without MSO/MSL
In reply to Nina Schoetterl-GlauschQuoting Nina Schoetterl-Glausch (2023-04-05 21:55:01)
Quoted 26 lines · Nico Boehr
Quoted 4 lines
Since we now have the ability to run guests without MSO/MSL, add a test to make sure this doesn't break.
Signed-off-by Nico Boehr <[email protected]>Quoted 9 lines
--- s390x/Makefile | 2 + s390x/sie-dat.c | 121 +++++++++++++++++++++++++++++++++++++ s390x/snippets/c/sie-dat.c | 58 ++++++++++++++++++ s390x/unittests.cfg | 3 + 4 files changed, 184 insertions(+) create mode 100644 s390x/sie-dat.c create mode 100644 s390x/snippets/c/sie-dat.cTest looks good to me. Some comments below.
[...]Quoted 7 lines
s390x/sie-dat.c new file +0
-0@@ -0,0 +1,121 @@[...]
Quoted 5 lines
s390x/sie-dat.c line 32
Quoted 3 lines
s390x/sie-dat.c new file +3
-0+ +/* keep in sync with TOTAL_PAGE_COUNT in s390x/snippets/c/sie-dat.c */ +#define GUEST_TOTAL_PAGE_COUNT 256This (1M) is the maximum snippet size (see
snippet_setup_guest), is this intentional?
It is by accident the maximum snippet size. It is completely fine to stay at 256 pages when we increase the maximum snippet size.
Quoted 3 lines
In that case the comment is inaccurate, since you'd want to sync it with the maximum snippet size. You also know the actual snippet size
SNIPPET_LEN(c, sie_dat)so I don't see why you'd need a define at all.
The snippet size is not the same as the number of mapped pages in the guest, no?
[...]
Quoted 9 lines
Quoted 4 lines
s390x/sie-dat.c new file +3
-0+ +static void test_sie_dat (void ) +{[...]
s390x/sie-dat.c
Quoted 2 lines
s390x/sie-dat.c new file +2
-0+ +/* the guest will now write to an unmapped address and we check that this causes a segment translation */ I'd prefer "causes a segment translation exception"
OK
[...]
Quoted 5 lines
Quoted 5 lines
s390x/snippets/c/sie-dat.c new file +0
-0
[...]
Quoted 5 lines
s390x/snippets/c/sie-dat.c line 24
Quoted 3 lines
s390x/snippets/c/sie-dat.c new file +3
-0+static inline void force_exit (void ) +{ + asmvolatile (" diag 0,0,0x44\n" );Pretty sure the compiler will generate a leading tab, so this will be doubly indented.
Copy-paste artifact, I can remove it.
[...]
Quoted 11 lines
Quoted 11 lines
s390x/snippets/c/sie-dat.c new file +11
-0+{ +uint8_t *invalid_ptr; + +memset (test_page,0 ,sizeof (test_page)); +/* tell the host the page's physical address (we're running DAT off) */ +force_exit_value ((uint64_t )test_page); + +/* write some value to the page so the host can verify it */ +for (size_t i =0 ; i <TEST_PAGE_COUNT ; i++) + test_page[i *PAGE_SIZE ] =42 + i; +
Quoted 8 lines
s390x/snippets/c/sie-dat.c line 51
Quoted 5 lines
s390x/snippets/c/sie-dat.c new file +5
-0+/* indicate we've written all pages */ +force_exit (); + +/* the first unmapped address */ + invalid_ptr = (uint8_t *)(TOTAL_PAGE_COUNT *PAGE_SIZE );Why not just use an address high enough you know it will not be mapped? -1 should do just fine.
I wanted to make sure exactly the right amount of pages is mapped and no more.
-1 would defeat that purpose.
Re: [kvm-unit-tests PATCH v1 4/4] s390x: add a test for SIE without MSO/MSL
Quoted 33 lines · Nico Boehr
Quoting Nina Schoetterl-Glausch (2023-04-05 21:55:01)
Quoted 26 lines · Nico Boehr
Quoted 4 lines
Since we now have the ability to run guests without MSO/MSL, add a test to make sure this doesn't break.
Signed-off-by Nico Boehr <[email protected]>Quoted 9 lines
--- s390x/Makefile | 2 + s390x/sie-dat.c | 121 +++++++++++++++++++++++++++++++++++++ s390x/snippets/c/sie-dat.c | 58 ++++++++++++++++++ s390x/unittests.cfg | 3 + 4 files changed, 184 insertions(+) create mode 100644 s390x/sie-dat.c create mode 100644 s390x/snippets/c/sie-dat.cTest looks good to me. Some comments below.
[...]Quoted 7 lines
s390x/sie-dat.c new file +0
-0@@ -0,0 +1,121 @@[...]
Quoted 5 lines
s390x/sie-dat.c line 32
Quoted 3 lines
s390x/sie-dat.c new file +3
-0+ +/* keep in sync with TOTAL_PAGE_COUNT in s390x/snippets/c/sie-dat.c */ +#define GUEST_TOTAL_PAGE_COUNT 256This (1M) is the maximum snippet size (see
snippet_setup_guest), is this intentional?
Quoted 7 lines
It is by accident the maximum snippet size. It is completely fine to stay at 256 pages when we increase the maximum snippet size.
Quoted 3 lines
In that case the comment is inaccurate, since you'd want to sync it with the maximum snippet size. You also know the actual snippet size
SNIPPET_LEN(c, sie_dat)so I don't see why you'd need a define at all.The snippet size is not the same as the number of mapped pages in the guest, no?
No, but one probably wouldn't want to map less that the snippet size. And there probably isn't a reason to map more either.
[...]
Quoted 10 lines
Quoted 6 lines
s390x/sie-dat.c
Quoted 3 lines
s390x/sie-dat.c new file +2
-0+/* the first unmapped address */ + invalid_ptr = (uint8_t *)(TOTAL_PAGE_COUNT *PAGE_SIZE );Why not just use an address high enough you know it will not be mapped? -1 should do just fine.
I wanted to make sure exactly the right amount of pages is mapped and no more.
-1 would defeat that purpose.
You want to touch the first page that isn't mapped?
Do you also want to map a certain number of pages?
I.e fill an entire page table and have the invalid pointer be mapped by another segment table entry?
With a small linker script change the snippet could know it's own length.
Then you could map just the required number of pages and don't need to keep those numbers in sync.
Re: [kvm-unit-tests PATCH v1 4/4] s390x: add a test for SIE without MSO/MSL
Quoting Nina Schoetterl-Glausch (2023-04-13 18:33:50) [...]
Quoted 2 lines
With a small linker script change the snippet could know it's own length.
Then you could map just the required number of pages and don't need to keep those numbers in sync.
Maybe it's because my knowledge about linker scripts is really limited or I don't get it, but I fail to see the advantage of the additional complexity.
My assumption would be that the number of pages mapped for the guest memory will never really change. Keeping a define in sync seems more pragmatic than going through linker script magic.
Re: [kvm-unit-tests PATCH v1 4/4] s390x: add a test for SIE without MSO/MSL
Quoted 11 lines · Nico Boehr
Quoting Nina Schoetterl-Glausch (2023-04-13 18:33:50) [...]
Quoted 2 lines
With a small linker script change the snippet could know it's own length.
Then you could map just the required number of pages and don't need to keep those numbers in sync.Maybe it's because my knowledge about linker scripts is really limited or I don't get it, but I fail to see the advantage of the additional complexity.
My assumption would be that the number of pages mapped for the guest memory will never really change. Keeping a define in sync seems more pragmatic than going through linker script magic.
I think just
+ . = ALIGN(4K);
+ esnippet = .;
/* End data */at the bottom of the snippet linker script would suffice.
Then you could use esnippet as an extern symbol in the C code. But yeah, might not be worth it unless it would help with other snippets too.