s390x: Align SIE tests to 2GB
cover s390x: Align SIE tests to 2GB
Some environments on s390x require guests to be aligned to 2GB. This is a problem when kvm-unit-tests act as a hypervisor, since guests run with MSO/MSL there and guest memory doesn't satisfy this requirement.
This series ensures kvm-unit-tests fulfills the 2G alignment requirement.
Note that this currently breaks the mvpg-sie test case under KVM due to prefix issues, a fix is due for upstream submission.
Nico Boehr (2):
s390x: spec_ex-sie: refactor to use snippet API
s390x: sie: ensure guests are aligned to 2GB
lib/s390x/sie.c | 42 ++++++++++++++++++++++++++++++++++++++++++ lib/s390x/sie.h | 2 ++ lib/s390x/snippet.h | 9 +++------ s390x/sie.c | 4 ++-- s390x/spec_ex-sie.c | 13 ++++++------- 5 files changed, 55 insertions(+), 15 deletions(-)
Signature · 1 line
2.41.0
PATCH 1/2 s390x: spec_ex-sie: refactor to use snippet API
spec_ex-sie uses a snippet, but allocated the memory on its own without obvious reason to do so.
Refactor the test to use snippet_init().
--- s390x/spec_ex-sie.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
s390x/spec_ex-sie.c +6 -7
@@ -17,19 +17,18 @@#include <hardware.h> static struct vm vm; -extern const char SNIPPET_NAME_START (c ,spec_ex )[]; -extern const char SNIPPET_NAME_END (c ,spec_ex )[];static bool strict;static void setup_guest (void ) { -char *guest; -int binary_size =SNIPPET_LEN (c, spec_ex); +extern const char SNIPPET_NAME_START (c ,spec_ex )[]; +extern const char SNIPPET_NAME_END (c ,spec_ex )[];setup_vm (); - guest =alloc_pages (8 ); -memcpy (guest,SNIPPET_NAME_START (c, spec_ex), binary_size); -sie_guest_create (&vm, (uint64_t ) guest,HPAGE_SIZE ); + +snippet_setup_guest (&vm, false); +snippet_init (&vm,SNIPPET_NAME_START (c, spec_ex), +SNIPPET_LEN (c, spec_ex),SNIPPET_UNPACK_OFF ); }static void reset_guest (void )
Signature · 1 line
2.41.0
Re: [kvm-unit-tests PATCH v1 1/2] s390x: spec_ex-sie: refactor to use snippet API
Quoted 4 lines · Nico Boehr
spec_ex-sie uses a snippet, but allocated the memory on its own without obvious reason to do so.
Refactor the test to use
snippet_init().
Hmm, is the test pre-snippet?
Anyway:
PATCH 2/2 s390x: sie: ensure guests are aligned to 2GB
In reply to Nico BoehrUntil now, kvm-unit-tests has aligned guests to 1 MB in the host virtual address space. Unfortunately, some s390x environments require guests to be 2GB aligned in the host virtual address space, preventing kvm-unit-tests which act as a hypervisor from running there.
We can't easily put guests at address 0, since we want to be able to run with MSO/MSL without having to maintain separate page tables for the guest physical memory. 2GB is also not a good choice, since the alloc_pages allocator will place its metadata there when the host has more than 2GB of memory. In addition, we also want a bit of space after the end of the host physical memory to be able to catch accesses beyond the end of physical memory.
The vmalloc allocator unfortunately allocates memory starting at the highest virtual address which is not suitable for guest memory either due to additional constraints of some environments.
The physical page allocator in memalign_pages() is also not a optimal choice, since every test running SIE would then require at least 4GB+1MB of physical memory.
This results in a few quite complex allocation requirements, hence add a new function sie_guest_alloc() which allocates memory for a guest and then establishes a properly aligned virtual space mapping.
Rework snippet test and sie tests to use the new sie_guest_alloc() function.
--- lib/s390x/sie.c | 42 ++++++++++++++++++++++++++++++++++++++++++ lib/s390x/sie.h | 2 ++ lib/s390x/snippet.h | 9 +++------ s390x/sie.c | 4 ++-- 4 files changed, 49 insertions(+), 8 deletions(-)
lib/s390x/sie.c +42 -0
@@ -15,6 +15,8 @@#include <asm/page.h> #include <libcflat.h> #include <alloc_page.h> +#include <vmalloc.h> +#include <sclp.h> void sie_expect_validity (struct vm *vm) {
@@ -111,6 +113,46 @@ void sie_guest_create(struct vm *vm, uint64_t guest_mem, uint64_t guest_mem_len) vm->sblk->crycbd = (uint32_t )(uintptr_t )vm->crycb; } +/* +* Allocate memory for a guest and map it in virtual address space such that +* it is properly aligned. +* @guest_size the desired size of the guest in bytes. +*/ +uint8_t *sie_guest_alloc (uint64_t guest_size) +{ +pgd_t *root; +u8 *guest_phys, *guest_virt; +static unsigned long guest_counter =1 ; +unsigned long i; + +setup_vm (); + root = (pgd_t *)(stctg (1 ) &PAGE_MASK ); + +/* +* Start of guest memory in host virtual space needs to be aligned to +* 2GB for some environments. It also can't be at 2GB since the memory +* allocator stores its page_states metadata there. +* Thus we use the next multiple of 4GB after the end of physical +* mapping. This also leaves space after end of physical memory so the +* page immediately after physical memory is guaranteed not to be +* present. +*/ + guest_virt = (uint8_t *)ALIGN (get_ram_size () + guest_counter *4UL *SZ_1G ,SZ_2G ); + guest_counter++; + + guest_phys =alloc_pages (get_order (guest_size) -12 ); +/* +* Establish a new mapping of the guest memory so it can be 2GB aligned +* without actually requiring 2GB physical memory. +*/ +for (i =0 ; i <= guest_size; i +=PAGE_SIZE ) { +install_page (root,__pa (guest_phys + i), guest_virt + i); + } +memset (guest_virt,0 , guest_size); + +return guest_virt; +} +/* Frees the memory that was gathered on initialization */ void sie_guest_destroy (struct vm *vm) {
lib/s390x/sie.h +2 -0
@@ -285,4 +285,6 @@ void sie_guest_sca_create(struct vm *vm);void sie_guest_create (struct vm *vm,uint64_t guest_mem,uint64_t guest_mem_len);void sie_guest_destroy (struct vm *vm); +uint8_t *sie_guest_alloc (uint64_t guest_size); +#endif /* _S390X_SIE_H_ */
lib/s390x/snippet.h +3 -6
@@ -125,14 +125,11 @@ static inline void snippet_pv_init(struct vm *vm, const char *gbin,/* Allocates and sets up a snippet based guest */ static inline void snippet_setup_guest (struct vm *vm,bool is_pv) { -u8 *guest; - -/* Allocate 1MB as guest memory */ - guest =alloc_pages (8 ); -memset (guest,0 ,HPAGE_SIZE ); +const unsigned long guest_size =SZ_1M ; +uint8_t *guest_start =sie_guest_alloc (guest_size);/* Initialize the vm struct and allocate control blocks */ -sie_guest_create (vm, (uint64_t )guest,HPAGE_SIZE ); +sie_guest_create (vm, (uint64_t )guest_start, guest_size);if (is_pv) {/* FMT4 needs a ESCA */
s390x/sie.c +2 -2
@@ -89,8 +89,8 @@ static void setup_guest(void) {setup_vm (); -/* Allocate 1MB as guest memory */ - guest =alloc_pages (8 ); + guest =sie_guest_alloc (SZ_1M ); +/* The first two pages are the lowcore */ guest_instr = guest +PAGE_SIZE *2 ;
Signature · 1 line
2.41.0
Re: [kvm-unit-tests PATCH v1 2/2] s390x: sie: ensure guests are aligned to 2GB
On Fri, 3 Nov 2023 07:35:47 +0100
Nico Boehr <[email protected]> wrote:
[...]
Quoted 14 lines
lib/s390x/sie.c +2
-0@@ -15,6 +15,8 @@#include <asm/page.h> #include <libcflat.h> #include <alloc_page.h> +#include <vmalloc.h> +#include <sclp.h> void sie_expect_validity (struct vm *vm) {
lib/s390x/sie.c void sie_guest_create(struct vm *vm, uint64_t guest_mem, uint64_t guest_mem_len) · line 116
Quoted 4 lines
lib/s390x/sie.c +1
-0@@ -111,6 +113,46 @@ void sie_guest_create(struct vm *vm, uint64_t guest_mem, uint64_t guest_mem_len) vm->sblk->crycbd = (uint32_t )(uintptr_t )vm->crycb; } +/*
/**
* sie_guest_alloc - Allocate memory [...]
lib/s390x/sie.c void sie_guest_create(struct vm *vm, uint64_t guest_mem, uint64_t guest_mem_len) · line 119
Quoted 3 lines
lib/s390x/sie.c +3
-0+* Allocate memory for a guest and map it in virtual address space such that +* it is properly aligned. +* @guest_size the desired size of the guest in bytes.
* @guest_size: the desired [...]
lib/s390x/sie.c void sie_guest_create(struct vm *vm, uint64_t guest_mem, uint64_t guest_mem_len) · line 120
Quoted 1 line
lib/s390x/sie.c +1
-0+*/
let's try to use a consistent style for comments. in many places we use the kerneldoc style -- let's stick to that
in the future we should pay more attention to the style, there are several places where we have __almost__ kerneldoc style.
lib/s390x/sie.c void sie_guest_create(struct vm *vm, uint64_t guest_mem, uint64_t guest_mem_len) · line 126
Quoted 6 lines
lib/s390x/sie.c +6
-0+uint8_t *sie_guest_alloc (uint64_t guest_size) +{ +pgd_t *root; +u8 *guest_phys, *guest_virt; +static unsigned long guest_counter =1 ; +unsigned long i;
reverse Christmas tree, please
Quoted 12 lines
lib/s390x/sie.c +12
-0+ +setup_vm (); + root = (pgd_t *)(stctg (1 ) &PAGE_MASK ); + +/* +* Start of guest memory in host virtual space needs to be aligned to +* 2GB for some environments. It also can't be at 2GB since the memory +* allocator stores its page_states metadata there. +* Thus we use the next multiple of 4GB after the end of physical +* mapping. This also leaves space after end of physical memory so the +* page immediately after physical memory is guaranteed not to be +* present.
lib/s390x/sie.c void sie_guest_create(struct vm *vm, uint64_t guest_mem, uint64_t guest_mem_len) · line 148
Quoted 10 lines
lib/s390x/sie.c +10
-0+*/ + guest_virt = (uint8_t *)ALIGN (get_ram_size () + guest_counter *4UL *SZ_1G ,SZ_2G ); + guest_counter++; + + guest_phys =alloc_pages (get_order (guest_size) -12 ); +/* +* Establish a new mapping of the guest memory so it can be 2GB aligned +* without actually requiring 2GB physical memory. +*/ +for (i =0 ; i <= guest_size; i +=PAGE_SIZE ) {
i < guest_size ?
Quoted 10 lines
lib/s390x/sie.c +7
-0+install_page (root,__pa (guest_phys + i), guest_virt + i); + } +memset (guest_virt,0 , guest_size); + +return guest_virt; +} +/* Frees the memory that was gathered on initialization */ void sie_guest_destroy (struct vm *vm) {
[...]
Quoted 8 lines · On Fri, 3 Nov 2023 07:35:45 +0100 Nico Boehr <[email protected]> wrote:
Some environments on s390x require guests to be aligned to 2GB. This is a problem when kvm-unit-tests act as a hypervisor, since guests run with MSO/MSL there and guest memory doesn't satisfy this requirement.
This series ensures kvm-unit-tests fulfills the 2G alignment requirement.
Note that this currently breaks the mvpg-sie test case under KVM due to prefix issues, a fix is due for upstream submission.
I disagree with the wording.
The mvpg-sie test does not break, it will fail because there is an actual KVM bug, so the test works as intended :)
Quoted 12 lines
Nico Boehr (2):
s390x: spec_ex-sie: refactor to use snippet API
s390x: sie: ensure guests are aligned to 2GBlib/s390x/sie.c | 42 ++++++++++++++++++++++++++++++++++++++++++ lib/s390x/sie.h | 2 ++ lib/s390x/snippet.h | 9 +++------ s390x/sie.c | 4 ++-- s390x/spec_ex-sie.c | 13 ++++++------- 5 files changed, 55 insertions(+), 15 deletions(-)