KVM: arm64: Selftest IPA fixes and 16kB support

PATCH Applied Marc Zyngier 2 participants 15 messages Dec 27, 2021 – Dec 28, 2021

Subsystems (3)

KVM Kselftest KVM arm64

lore

Share & tools

lore

Marc Zyngier

author maintainer

2021-12-27 12:49 UTC #0 lore raw

v2 cover KVM: arm64: Selftest IPA fixes and 16kB support

Now that the common variety of fruity arm64 systems makes a pretty remarkable KVM host, I have decided to run the collection of selftests on it.

Nothing works. Oh no!

As it turns out, the tests have a notion of "default mode" (40bit PA, 4kB), which cannot work in general on arm64 because there no such thing as an IPA size that would be valid everywhere (apart from the minimal 32bit), nor a guaranteed to be supported page size.

This small series is a first attempt at making these things computed at runtime by making the default something else on systems that do not support it. It also makes the supported page sizes dynamically discovered, which allows us to implement support for 16kB pages.

With that, the selftests do run on the M1, with the exception of the memslot tests that are freaked out by the use of 16kB pages on the host and 4kB in the guest (these tests are hardcoded to use 4kB pages even if the VM uses something else -- oh well...).

* From v1:
- Rebased on 5.16-rc7 and dropped the initial patch, now that non-x86 systems are up and running again
- Used Andrew's trick to initialise the default mode as a constructor instead of hacking every single test (I decided to make it an arm64-special instead of an all-arch thing though)
- Reworked the way TCR_EL1 gets configured for easier integration of new modes
- Added support for various 16kB modes
- Various cleanups as requested by Andrew

Marc Zyngier (6):
KVM: selftests: arm64: Initialise default guest mode at test startup
time
KVM: selftests: arm64: Introduce a variable default IPA size
KVM: selftests: arm64: Check for supported page sizes
KVM: selftests: arm64: Rework TCR_EL1 configuration
KVM: selftests: arm64: Add support for VM_MODE_P36V48_{4K,64K}
KVM: selftests: arm64: Add support for various modes with 16kB page
size

 .../selftests/kvm/include/aarch64/processor.h |  3 +
 .../testing/selftests/kvm/include/kvm_util.h  | 10 ++-
 .../selftests/kvm/lib/aarch64/processor.c     | 82 +++++++++++++++++--
 tools/testing/selftests/kvm/lib/guest_modes.c | 49 +++++++++--
 tools/testing/selftests/kvm/lib/kvm_util.c    | 22 +++++
 5 files changed, 152 insertions(+), 14 deletions(-)

Signature · 6 lines

2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Marc Zyngier

author maintainer

2021-12-27 12:49 UTC #1 lore raw

PATCH v2 1/6 KVM: selftests: arm64: Initialise default guest mode at test startup time

As we are going to add support for a variable default mode on arm64, let's make sure it is setup first by using a constructor that gets called before the actual test runs.

Signed-off-by Marc Zyngier <[email protected]>

---
 tools/testing/selftests/kvm/lib/aarch64/processor.c | 9 +++++++++
 1 file changed, 9 insertions(+)

tools/testing/selftests/kvm/lib/aarch64/processor.c +9 -0

index b4eeeafd2a70..b509341b8411 100644
@@ -432,3 +432,12 @@ uint32_t guest_get_vcpuid(void)
 {
 	return read_sysreg(tpidr_el1);
 }
+
+/*
+ * arm64 doesn't have a true default mode, so start by computing the
+ * available IPA space and page sizes early.
+ */
+void __attribute__((constructor)) init_guest_modes(void)
+{
+       guest_modes_append_default();
+}

Signature · 6 lines

2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Andrew Jones

Reviewed-by on v2 1/6

2021-12-28 09:37 UTC #2 lore raw

Re: [PATCH v2 1/6] KVM: selftests: arm64: Initialise default guest mode at test startup time

Quoted 30 lines · Marc Zyngier

As we are going to add support for a variable default mode on arm64, let's make sure it is setup first by using a constructor that gets called before the actual test runs.
Suggested-by Andrew Jones <[email protected]>
Signed-off-by Marc Zyngier <[email protected]>
---
 tools/testing/selftests/kvm/lib/aarch64/processor.c | 9 +++++++++
 1 file changed, 9 insertions(+)
tools/testing/selftests/kvm/lib/aarch64/processor.c +9 -0
index b4eeeafd2a70..b509341b8411 100644
@@ -432,3 +432,12 @@ uint32_t guest_get_vcpuid(void)
 {
 	return read_sysreg(tpidr_el1);
 }
+
+/*
+ * arm64 doesn't have a true default mode, so start by computing the
+ * available IPA space and page sizes early.
+ */
+void __attribute__((constructor)) init_guest_modes(void)
+{
+       guest_modes_append_default();
+}
Signature · 1 line
2.30.2

Reviewed-by Andrew Jones <[email protected]>

Marc Zyngier

author maintainer

2021-12-27 12:49 UTC #3 lore raw

PATCH v2 2/6 KVM: selftests: arm64: Introduce a variable default IPA size

In reply to Marc Zyngier

Contrary to popular belief, there is no such thing as a default IPA size on arm64. Anything goes, and implementations are the usual Wild West.

The selftest infrastructure default to 40bit IPA, which obviously doesn't work for some systems out there.

Turn VM_MODE_DEFAULT from a constant into a variable, and let guest_modes_append_default() populate it, depending on what the HW can do. In order to preserve the current behaviour, we still pick 40bits IPA as the default if it is available, and the largest supported IPA space otherwise.

Signed-off-by Marc Zyngier <[email protected]>

---
 .../testing/selftests/kvm/include/kvm_util.h  |  4 ++-
 tools/testing/selftests/kvm/lib/guest_modes.c | 30 +++++++++++++++++--
 2 files changed, 30 insertions(+), 4 deletions(-)

tools/testing/selftests/kvm/include/kvm_util.h +3 -1

index 2d62edc49d67..7fa0a93d7526 100644
@@ -53,7 +53,9 @@ enum vm_guest_mode {
 
 #if defined(__aarch64__)
 
-#define VM_MODE_DEFAULT			VM_MODE_P40V48_4K
+extern enum vm_guest_mode vm_mode_default;
+
+#define VM_MODE_DEFAULT			vm_mode_default
 #define MIN_PAGE_SHIFT			12U
 #define ptes_per_page(page_size)	((page_size) / 8)

tools/testing/selftests/kvm/lib/guest_modes.c +27 -3

index c330f414ef96..5e3fdbd992fd 100644
@@ -4,22 +4,46 @@
  */
 #include "guest_modes.h"
 
+#ifdef __aarch64__
+enum vm_guest_mode vm_mode_default;
+#endif
+
 struct guest_mode guest_modes[NUM_VM_MODES];
 
 void guest_modes_append_default(void)
 {
+#ifndef __aarch64__
 	guest_mode_append(VM_MODE_DEFAULT, true, true);
-
-#ifdef __aarch64__
-	guest_mode_append(VM_MODE_P40V48_64K, true, true);
+#else
 	{
 		unsigned int limit = kvm_check_cap(KVM_CAP_ARM_VM_IPA_SIZE);
+		int i;
+
+		vm_mode_default = NUM_VM_MODES;
+
 		if (limit >= 52)
 			guest_mode_append(VM_MODE_P52V48_64K, true, true);
 		if (limit >= 48) {
 			guest_mode_append(VM_MODE_P48V48_4K, true, true);
 			guest_mode_append(VM_MODE_P48V48_64K, true, true);
 		}
+		if (limit >= 40) {
+			guest_mode_append(VM_MODE_P40V48_4K, true, true);
+			guest_mode_append(VM_MODE_P40V48_64K, true, true);
+			vm_mode_default = VM_MODE_P40V48_4K;
+		}
+
+		/*
+		 * Pick the first supported IPA size if the default
+		 * isn't available.
+		 */
+		for (i = 0; vm_mode_default == NUM_VM_MODES && i < NUM_VM_MODES; i++) {
+			if (guest_modes[i].supported && guest_modes[i].enabled)
+				vm_mode_default = i;
+		}
+
+		TEST_ASSERT(vm_mode_default != NUM_VM_MODES,
+			    "No supported mode!");
 	}
 #endif
 #ifdef __s390x__

Signature · 1 line

2.30.2

Andrew Jones

2021-12-28 09:26 UTC #4 lore raw

Re: [PATCH v2 2/6] KVM: selftests: arm64: Introduce a variable default IPA size

Quoted 71 lines · Marc Zyngier

Contrary to popular belief, there is no such thing as a default IPA size on arm64. Anything goes, and implementations are the usual Wild West.

The selftest infrastructure default to 40bit IPA, which obviously doesn't work for some systems out there.

Signed-off-by Marc Zyngier <[email protected]>

---
 .../testing/selftests/kvm/include/kvm_util.h  |  4 ++-
 tools/testing/selftests/kvm/lib/guest_modes.c | 30 +++++++++++++++++--
 2 files changed, 30 insertions(+), 4 deletions(-)

tools/testing/selftests/kvm/include/kvm_util.h +3 -1

index 2d62edc49d67..7fa0a93d7526 100644
@@ -53,7 +53,9 @@ enum vm_guest_mode {
 
 #if defined(__aarch64__)
 
-#define VM_MODE_DEFAULT			VM_MODE_P40V48_4K
+extern enum vm_guest_mode vm_mode_default;
+
+#define VM_MODE_DEFAULT			vm_mode_default
 #define MIN_PAGE_SHIFT			12U
 #define ptes_per_page(page_size)	((page_size) / 8)

tools/testing/selftests/kvm/lib/guest_modes.c +13 -3

index c330f414ef96..5e3fdbd992fd 100644
@@ -4,22 +4,46 @@
  */
 #include "guest_modes.h"
 
+#ifdef __aarch64__
+enum vm_guest_mode vm_mode_default;
+#endif
+
 struct guest_mode guest_modes[NUM_VM_MODES];
 
 void guest_modes_append_default(void)
 {
+#ifndef __aarch64__
 	guest_mode_append(VM_MODE_DEFAULT, true, true);
-
-#ifdef __aarch64__
-	guest_mode_append(VM_MODE_P40V48_64K, true, true);
+#else
 	{
 		unsigned int limit = kvm_check_cap(KVM_CAP_ARM_VM_IPA_SIZE);
+		int i;
+
+		vm_mode_default = NUM_VM_MODES;
+
 		if (limit >= 52)
 			guest_mode_append(VM_MODE_P52V48_64K, true, true);
 		if (limit >= 48) {
 			guest_mode_append(VM_MODE_P48V48_4K, true, true);
 			guest_mode_append(VM_MODE_P48V48_64K, true, true);
 		}
+		if (limit >= 40) {
+			guest_mode_append(VM_MODE_P40V48_4K, true, true);
+			guest_mode_append(VM_MODE_P40V48_64K, true, true);

tools/testing/selftests/kvm/lib/guest_modes.c line 42

Quoted 10 lines

tools/testing/selftests/kvm/lib/guest_modes.c +10 -0

+			vm_mode_default = VM_MODE_P40V48_4K;
+		}
+
+		/*
+		 * Pick the first supported IPA size if the default
+		 * isn't available.
+		 */
+		for (i = 0; vm_mode_default == NUM_VM_MODES && i < NUM_VM_MODES; i++) {
+			if (guest_modes[i].supported && guest_modes[i].enabled)
+				vm_mode_default = i;

Since we don't have a 'break' here, this picks the last supported size (of the guest_modes list), not the first, as the comment implies it should do.

Thanks,
drew

Quoted 10 lines

tools/testing/selftests/kvm/lib/guest_modes.c +4 -0

+		}
+
+		TEST_ASSERT(vm_mode_default != NUM_VM_MODES,
+			    "No supported mode!");
 	}
 #endif
 #ifdef __s390x__

Signature · 1 line

2.30.2

Marc Zyngier

author maintainer

2021-12-28 10:29 UTC #5 lore raw

Re: [PATCH v2 2/6] KVM: selftests: arm64: Introduce a variable default IPA size

Quoted 79 lines · Andrew Jones

Quoted 77 lines · Marc Zyngier

Contrary to popular belief, there is no such thing as a default IPA size on arm64. Anything goes, and implementations are the usual Wild West.

The selftest infrastructure default to 40bit IPA, which obviously doesn't work for some systems out there.

Signed-off-by Marc Zyngier <[email protected]>

---
 .../testing/selftests/kvm/include/kvm_util.h  |  4 ++-
 tools/testing/selftests/kvm/lib/guest_modes.c | 30 +++++++++++++++++--
 2 files changed, 30 insertions(+), 4 deletions(-)

tools/testing/selftests/kvm/include/kvm_util.h +3 -1

index 2d62edc49d67..7fa0a93d7526 100644
@@ -53,7 +53,9 @@ enum vm_guest_mode {
 
 #if defined(__aarch64__)
 
-#define VM_MODE_DEFAULT			VM_MODE_P40V48_4K
+extern enum vm_guest_mode vm_mode_default;
+
+#define VM_MODE_DEFAULT			vm_mode_default
 #define MIN_PAGE_SHIFT			12U
 #define ptes_per_page(page_size)	((page_size) / 8)

tools/testing/selftests/kvm/lib/guest_modes.c +19 -3

index c330f414ef96..5e3fdbd992fd 100644
@@ -4,22 +4,46 @@
  */
 #include "guest_modes.h"
 
+#ifdef __aarch64__
+enum vm_guest_mode vm_mode_default;
+#endif
+
 struct guest_mode guest_modes[NUM_VM_MODES];
 
 void guest_modes_append_default(void)
 {
+#ifndef __aarch64__
 	guest_mode_append(VM_MODE_DEFAULT, true, true);
-
-#ifdef __aarch64__
-	guest_mode_append(VM_MODE_P40V48_64K, true, true);
+#else
 	{
 		unsigned int limit = kvm_check_cap(KVM_CAP_ARM_VM_IPA_SIZE);
+		int i;
+
+		vm_mode_default = NUM_VM_MODES;
+
 		if (limit >= 52)
 			guest_mode_append(VM_MODE_P52V48_64K, true, true);
 		if (limit >= 48) {
 			guest_mode_append(VM_MODE_P48V48_4K, true, true);
 			guest_mode_append(VM_MODE_P48V48_64K, true, true);
 		}
+		if (limit >= 40) {
+			guest_mode_append(VM_MODE_P40V48_4K, true, true);
+			guest_mode_append(VM_MODE_P40V48_64K, true, true);
+			vm_mode_default = VM_MODE_P40V48_4K;
+		}
+
+		/*
+		 * Pick the first supported IPA size if the default
+		 * isn't available.

Quoted 8 lines

tools/testing/selftests/kvm/lib/guest_modes.c line 42
Quoted 4 lines
tools/testing/selftests/kvm/lib/guest_modes.c +4 -0
+		 */
+		for (i = 0; vm_mode_default == NUM_VM_MODES && i < NUM_VM_MODES; i++) {
+			if (guest_modes[i].supported && guest_modes[i].enabled)
+				vm_mode_default = i;
Since we don't have a 'break' here, this picks the last supported size (of the guest_modes list), not the first, as the comment implies it should do.

This is checked in the for() loop condition, and the first matching mode will cause the loop to terminate. This is the same check that avoids scanning for a mode when VM_MODE_P40V48_4K is selected.

M.

Signature · 1 line

Without deviation from the norm, progress is not possible.

Andrew Jones

Reviewed-by

2021-12-28 10:33 UTC #6 lore raw

Re: [PATCH v2 2/6] KVM: selftests: arm64: Introduce a variable default IPA size

Quoted 8 lines · Marc Zyngier

Quoted 7 lines
tools/testing/selftests/kvm/lib/guest_modes.c line 42
Quoted 3 lines
tools/testing/selftests/kvm/lib/guest_modes.c +3 -0
+		for (i = 0; vm_mode_default == NUM_VM_MODES && i < NUM_VM_MODES; i++) {
+			if (guest_modes[i].supported && guest_modes[i].enabled)
+				vm_mode_default = i;
Since we don't have a 'break' here, this picks the last supported size (of the guest_modes list), not the first, as the comment implies it should do.

Quoted 4 lines

This is checked in the for() loop condition, and the first matching mode will cause the loop to terminate. This is the same check that avoids scanning for a mode when VM_MODE_P40V48_4K is selected.

Ah, of course, sorry for the noise.

Reviewed-by Andrew Jones <[email protected]>

Marc Zyngier

author maintainer

2021-12-27 12:49 UTC #7 lore raw

PATCH v2 3/6 KVM: selftests: arm64: Check for supported page sizes

In reply to Marc Zyngier

Just as arm64 implemenations don't necessary support all IPA ranges, they don't all support the same page sizes either. Fun.

Create a dummy VM to snapshot the page sizes supported by the host, and filter the supported modes.

Signed-off-by Marc Zyngier <[email protected]>

---
 .../selftests/kvm/include/aarch64/processor.h |  3 ++
 .../selftests/kvm/lib/aarch64/processor.c     | 36 +++++++++++++++++++
 tools/testing/selftests/kvm/lib/guest_modes.c | 17 +++++----
 3 files changed, 50 insertions(+), 6 deletions(-)

tools/testing/selftests/kvm/include/aarch64/processor.h +3 -0

index 27d8e1bb5b36..8f9f46979a00 100644
@@ -113,6 +113,9 @@ enum {
 #define ESR_EC_WP_CURRENT	0x35
 #define ESR_EC_BRK_INS		0x3c
 
+void aarch64_get_supported_page_sizes(uint32_t ipa,
+				      bool *ps4k, bool *ps16k, bool *ps64k);
+
 void vm_init_descriptor_tables(struct kvm_vm *vm);
 void vcpu_init_descriptor_tables(struct kvm_vm *vm, uint32_t vcpuid);

tools/testing/selftests/kvm/lib/aarch64/processor.c +36 -0

index b509341b8411..49fcfe9768e0 100644
@@ -8,6 +8,7 @@
 #include <linux/compiler.h>
 #include <assert.h>
 
+#include "guest_modes.h"
 #include "kvm_util.h"
 #include "../kvm_util_internal.h"
 #include "processor.h"

@@ -433,6 +434,41 @@ uint32_t guest_get_vcpuid(void)
 	return read_sysreg(tpidr_el1);
 }
 
+void aarch64_get_supported_page_sizes(uint32_t ipa,
+				      bool *ps4k, bool *ps16k, bool *ps64k)
+{
+	struct kvm_vcpu_init preferred_init;
+	int kvm_fd, vm_fd, vcpu_fd, err;
+	uint64_t val;
+	struct kvm_one_reg reg = {
+		.id	= KVM_ARM64_SYS_REG(SYS_ID_AA64MMFR0_EL1),
+		.addr	= (uint64_t)&val,
+	};
+
+	kvm_fd = open_kvm_dev_path_or_exit();
+	vm_fd = ioctl(kvm_fd, KVM_CREATE_VM, ipa);
+	TEST_ASSERT(vm_fd >= 0, "Can't create VM");
+
+	vcpu_fd = ioctl(vm_fd, KVM_CREATE_VCPU, 0);
+	TEST_ASSERT(vcpu_fd >= 0, "Can't create vcpu");
+
+	err = ioctl(vm_fd, KVM_ARM_PREFERRED_TARGET, &preferred_init);
+	TEST_ASSERT(err == 0, "Can't get target");
+	err = ioctl(vcpu_fd, KVM_ARM_VCPU_INIT, &preferred_init);
+	TEST_ASSERT(err == 0, "Can't get init vcpu");
+
+	err = ioctl(vcpu_fd, KVM_GET_ONE_REG, &reg);
+	TEST_ASSERT(err == 0, "Can't get MMFR0");
+
+	*ps4k = ((val >> 28) & 0xf) != 0xf;
+	*ps64k = ((val >> 24) & 0xf) == 0;
+	*ps16k = ((val >> 20) & 0xf) != 0;
+
+	close(vcpu_fd);
+	close(vm_fd);
+	close(kvm_fd);
+}
+
 /*
  * arm64 doesn't have a true default mode, so start by computing the
  * available IPA space and page sizes early.

tools/testing/selftests/kvm/lib/guest_modes.c +11 -6

index 5e3fdbd992fd..67144fdac433 100644
@@ -5,6 +5,7 @@
 #include "guest_modes.h"
 
 #ifdef __aarch64__
+#include "processor.h"
 enum vm_guest_mode vm_mode_default;
 #endif

@@ -17,20 +18,24 @@ void guest_modes_append_default(void)
 #else
 	{
 		unsigned int limit = kvm_check_cap(KVM_CAP_ARM_VM_IPA_SIZE);
+		bool ps4k, ps16k, ps64k;
 		int i;
 
+		aarch64_get_supported_page_sizes(limit, &ps4k, &ps16k, &ps64k);
+
 		vm_mode_default = NUM_VM_MODES;
 
 		if (limit >= 52)
-			guest_mode_append(VM_MODE_P52V48_64K, true, true);
+			guest_mode_append(VM_MODE_P52V48_64K, ps64k, ps64k);
 		if (limit >= 48) {
-			guest_mode_append(VM_MODE_P48V48_4K, true, true);
-			guest_mode_append(VM_MODE_P48V48_64K, true, true);
+			guest_mode_append(VM_MODE_P48V48_4K, ps4k, ps4k);
+			guest_mode_append(VM_MODE_P48V48_64K, ps64k, ps64k);
 		}
 		if (limit >= 40) {
-			guest_mode_append(VM_MODE_P40V48_4K, true, true);
-			guest_mode_append(VM_MODE_P40V48_64K, true, true);
-			vm_mode_default = VM_MODE_P40V48_4K;
+			guest_mode_append(VM_MODE_P40V48_4K, ps4k, ps4k);
+			guest_mode_append(VM_MODE_P40V48_64K, ps64k, ps64k);
+			if (ps4k)
+				vm_mode_default = VM_MODE_P40V48_4K;
 		}
 
 		/*

Signature · 1 line

2.30.2

Andrew Jones

Reviewed-by on v2 3/6

2021-12-28 09:37 UTC #8 lore raw

Re: [PATCH v2 3/6] KVM: selftests: arm64: Check for supported page sizes

Quoted 6 lines · Marc Zyngier

Just as arm64 implemenations don't necessary support all IPA ranges, they don't all support the same page sizes either. Fun.
Create a dummy VM to snapshot the page sizes supported by the host, and filter the supported modes.

Quoted 7 lines

Signed-off-by Marc Zyngier <[email protected]>

---
 .../selftests/kvm/include/aarch64/processor.h |  3 ++
 .../selftests/kvm/lib/aarch64/processor.c     | 36 +++++++++++++++++++
 tools/testing/selftests/kvm/lib/guest_modes.c | 17 +++++----
 3 files changed, 50 insertions(+), 6 deletions(-)

Reviewed-by Andrew Jones <[email protected]>

Marc Zyngier

author maintainer

2021-12-27 12:49 UTC #9 lore raw

PATCH v2 4/6 KVM: selftests: arm64: Rework TCR_EL1 configuration

In reply to Marc Zyngier

The current way we initialise TCR_EL1 is a bit cumbersome, as we mix setting TG0 and IPS in the same swtch statement.

Split it into two statements (one for the base granule size, and another for the IPA size), allowing new modes to be added in a more elegant way.

No functional change intended.

Signed-off-by Marc Zyngier <[email protected]>

---
 .../selftests/kvm/lib/aarch64/processor.c     | 21 ++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)

tools/testing/selftests/kvm/lib/aarch64/processor.c +14 -7

index 49fcfe9768e0..d005543aa3e2 100644
@@ -238,6 +238,7 @@ void aarch64_vcpu_setup(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_vcpu_init
 	get_reg(vm, vcpuid, KVM_ARM64_SYS_REG(SYS_SCTLR_EL1), &sctlr_el1);
 	get_reg(vm, vcpuid, KVM_ARM64_SYS_REG(SYS_TCR_EL1), &tcr_el1);
 
+	/* Configure base granule size */
 	switch (vm->mode) {
 	case VM_MODE_P52V48_4K:
 		TEST_FAIL("AArch64 does not support 4K sized pages "

@@ -246,23 +247,29 @@ void aarch64_vcpu_setup(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_vcpu_init
 		TEST_FAIL("AArch64 does not support 4K sized pages "
 			  "with ANY-bit physical address ranges");
 	case VM_MODE_P52V48_64K:
+	case VM_MODE_P48V48_64K:
+	case VM_MODE_P40V48_64K:
 		tcr_el1 |= 1ul << 14; /* TG0 = 64KB */
-		tcr_el1 |= 6ul << 32; /* IPS = 52 bits */
 		break;
 	case VM_MODE_P48V48_4K:
+	case VM_MODE_P40V48_4K:
 		tcr_el1 |= 0ul << 14; /* TG0 = 4KB */
-		tcr_el1 |= 5ul << 32; /* IPS = 48 bits */
 		break;
+	default:
+		TEST_FAIL("Unknown guest mode, mode: 0x%x", vm->mode);
+	}
+
+	/* Configure output size */
+	switch (vm->mode) {
+	case VM_MODE_P52V48_64K:
+		tcr_el1 |= 6ul << 32; /* IPS = 52 bits */
+		break;
+	case VM_MODE_P48V48_4K:
 	case VM_MODE_P48V48_64K:
-		tcr_el1 |= 1ul << 14; /* TG0 = 64KB */
 		tcr_el1 |= 5ul << 32; /* IPS = 48 bits */
 		break;
 	case VM_MODE_P40V48_4K:
-		tcr_el1 |= 0ul << 14; /* TG0 = 4KB */
-		tcr_el1 |= 2ul << 32; /* IPS = 40 bits */
-		break;
 	case VM_MODE_P40V48_64K:
-		tcr_el1 |= 1ul << 14; /* TG0 = 64KB */
 		tcr_el1 |= 2ul << 32; /* IPS = 40 bits */
 		break;
 	default:

Signature · 1 line

2.30.2

Andrew Jones

Reviewed-by on v2 4/6

2021-12-28 09:37 UTC #10 lore raw

Re: [PATCH v2 4/6] KVM: selftests: arm64: Rework TCR_EL1 configuration

Quoted 7 lines · Marc Zyngier

The current way we initialise TCR_EL1 is a bit cumbersome, as we mix setting TG0 and IPS in the same swtch statement.
Split it into two statements (one for the base granule size, and another for the IPA size), allowing new modes to be added in a more elegant way.

Quoted 7 lines

No functional change intended.
Signed-off-by Marc Zyngier <[email protected]>
---
 .../selftests/kvm/lib/aarch64/processor.c     | 21 ++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)

Reviewed-by Andrew Jones <[email protected]>

Marc Zyngier

author maintainer

2021-12-27 12:49 UTC #11 lore raw

PATCH v2 5/6 KVM: selftests: arm64: Add support for VM_MODE_P36V48_{4K,64K}

In reply to Marc Zyngier

Some of the arm64 systems out there have an IPA space that is positively tiny. Nonetheless, they make great KVM hosts.

Add support for 36bit IPA support with 4kB pages, which makes some of the fruity machines happy. Whilst we're at it, add support for 64kB pages as well, though these boxes have no support for it.

Reviewed-by Andrew Jones <[email protected]>

Signed-off-by Marc Zyngier <[email protected]>

---
 tools/testing/selftests/kvm/include/kvm_util.h      | 2 ++
 tools/testing/selftests/kvm/lib/aarch64/processor.c | 6 ++++++
 tools/testing/selftests/kvm/lib/guest_modes.c       | 4 ++++
 tools/testing/selftests/kvm/lib/kvm_util.c          | 6 ++++++
 4 files changed, 18 insertions(+)

tools/testing/selftests/kvm/include/kvm_util.h +2 -0

index 7fa0a93d7526..fd1397fc7ad5 100644
@@ -48,6 +48,8 @@ enum vm_guest_mode {
 	VM_MODE_PXXV48_4K,	/* For 48bits VA but ANY bits PA */
 	VM_MODE_P47V64_4K,
 	VM_MODE_P44V64_4K,
+	VM_MODE_P36V48_4K,
+	VM_MODE_P36V48_64K,
 	NUM_VM_MODES,
 };

tools/testing/selftests/kvm/lib/aarch64/processor.c +6 -0

index d005543aa3e2..70395c777ea4 100644
@@ -249,10 +249,12 @@ void aarch64_vcpu_setup(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_vcpu_init
 	case VM_MODE_P52V48_64K:
 	case VM_MODE_P48V48_64K:
 	case VM_MODE_P40V48_64K:
+	case VM_MODE_P36V48_64K:
 		tcr_el1 |= 1ul << 14; /* TG0 = 64KB */
 		break;
 	case VM_MODE_P48V48_4K:
 	case VM_MODE_P40V48_4K:
+	case VM_MODE_P36V48_4K:
 		tcr_el1 |= 0ul << 14; /* TG0 = 4KB */
 		break;
 	default:

@@ -272,6 +274,10 @@ void aarch64_vcpu_setup(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_vcpu_init
 	case VM_MODE_P40V48_64K:
 		tcr_el1 |= 2ul << 32; /* IPS = 40 bits */
 		break;
+	case VM_MODE_P36V48_4K:
+	case VM_MODE_P36V48_64K:
+		tcr_el1 |= 1ul << 32; /* IPS = 36 bits */
+		break;
 	default:
 		TEST_FAIL("Unknown guest mode, mode: 0x%x", vm->mode);
 	}

tools/testing/selftests/kvm/lib/guest_modes.c +4 -0

index 67144fdac433..240f2d2e2d23 100644
@@ -37,6 +37,10 @@ void guest_modes_append_default(void)
 			if (ps4k)
 				vm_mode_default = VM_MODE_P40V48_4K;
 		}
+		if (limit >= 36) {
+			guest_mode_append(VM_MODE_P36V48_4K, ps4k, ps4k);
+			guest_mode_append(VM_MODE_P36V48_64K, ps64k, ps64k);
+		}
 
 		/*
 		 * Pick the first supported IPA size if the default

tools/testing/selftests/kvm/lib/kvm_util.c +6 -0

index 53d2b5d04b82..9da71e27cd84 100644
@@ -172,6 +172,8 @@ const char *vm_guest_mode_string(uint32_t i)
 		[VM_MODE_PXXV48_4K]	= "PA-bits:ANY, VA-bits:48,  4K pages",
 		[VM_MODE_P47V64_4K]	= "PA-bits:47,  VA-bits:64,  4K pages",
 		[VM_MODE_P44V64_4K]	= "PA-bits:44,  VA-bits:64,  4K pages",
+		[VM_MODE_P36V48_4K]	= "PA-bits:36,  VA-bits:48,  4K pages",
+		[VM_MODE_P36V48_64K]	= "PA-bits:36,  VA-bits:48, 64K pages",
 	};
 	_Static_assert(sizeof(strings)/sizeof(char *) == NUM_VM_MODES,
 		       "Missing new mode strings?");

@@ -191,6 +193,8 @@ const struct vm_guest_mode_params vm_guest_mode_params[] = {
 	[VM_MODE_PXXV48_4K]	= {  0,  0,  0x1000, 12 },
 	[VM_MODE_P47V64_4K]	= { 47, 64,  0x1000, 12 },
 	[VM_MODE_P44V64_4K]	= { 44, 64,  0x1000, 12 },
+	[VM_MODE_P36V48_4K]	= { 36, 48,  0x1000, 12 },
+	[VM_MODE_P36V48_64K]	= { 36, 48, 0x10000, 16 },
 };
 _Static_assert(sizeof(vm_guest_mode_params)/sizeof(struct vm_guest_mode_params) == NUM_VM_MODES,
 	       "Missing new mode params?");

@@ -252,9 +256,11 @@ struct kvm_vm *vm_create(enum vm_guest_mode mode, uint64_t phy_pages, int perm)
 		vm->pgtable_levels = 3;
 		break;
 	case VM_MODE_P40V48_4K:
+	case VM_MODE_P36V48_4K:
 		vm->pgtable_levels = 4;
 		break;
 	case VM_MODE_P40V48_64K:
+	case VM_MODE_P36V48_64K:
 		vm->pgtable_levels = 3;
 		break;
 	case VM_MODE_PXXV48_4K:

Signature · 6 lines

2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Marc Zyngier

author maintainer

2021-12-27 12:49 UTC #12 lore raw

PATCH v2 6/6 KVM: selftests: arm64: Add support for various modes with 16kB page size

In reply to Marc Zyngier

The 16kB page size is not a popular choice, due to only a few CPUs actually implementing support for it. However, it can lead to some interesting performance improvements given the right uarch choices.

Add support for this page size for various PA/VA combinations.

Signed-off-by Marc Zyngier <[email protected]>

---
 tools/testing/selftests/kvm/include/kvm_util.h   |  4 ++++
 .../selftests/kvm/lib/aarch64/processor.c        | 10 ++++++++++
 tools/testing/selftests/kvm/lib/guest_modes.c    |  4 ++++
 tools/testing/selftests/kvm/lib/kvm_util.c       | 16 ++++++++++++++++
 4 files changed, 34 insertions(+)

tools/testing/selftests/kvm/include/kvm_util.h +4 -0

index fd1397fc7ad5..e1f27626bbdc 100644
@@ -42,14 +42,18 @@ enum vm_guest_mode {
 	VM_MODE_P52V48_4K,
 	VM_MODE_P52V48_64K,
 	VM_MODE_P48V48_4K,
+	VM_MODE_P48V48_16K,
 	VM_MODE_P48V48_64K,
 	VM_MODE_P40V48_4K,
+	VM_MODE_P40V48_16K,
 	VM_MODE_P40V48_64K,
 	VM_MODE_PXXV48_4K,	/* For 48bits VA but ANY bits PA */
 	VM_MODE_P47V64_4K,
 	VM_MODE_P44V64_4K,
 	VM_MODE_P36V48_4K,
+	VM_MODE_P36V48_16K,
 	VM_MODE_P36V48_64K,
+	VM_MODE_P36V47_16K,
 	NUM_VM_MODES,
 };

tools/testing/selftests/kvm/lib/aarch64/processor.c +10 -0

index 70395c777ea4..9343d82519b4 100644
@@ -252,6 +252,12 @@ void aarch64_vcpu_setup(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_vcpu_init
 	case VM_MODE_P36V48_64K:
 		tcr_el1 |= 1ul << 14; /* TG0 = 64KB */
 		break;
+	case VM_MODE_P48V48_16K:
+	case VM_MODE_P40V48_16K:
+	case VM_MODE_P36V48_16K:
+	case VM_MODE_P36V47_16K:
+		tcr_el1 |= 2ul << 14; /* TG0 = 16KB */
+		break;
 	case VM_MODE_P48V48_4K:
 	case VM_MODE_P40V48_4K:
 	case VM_MODE_P36V48_4K:

@@ -267,15 +273,19 @@ void aarch64_vcpu_setup(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_vcpu_init
 		tcr_el1 |= 6ul << 32; /* IPS = 52 bits */
 		break;
 	case VM_MODE_P48V48_4K:
+	case VM_MODE_P48V48_16K:
 	case VM_MODE_P48V48_64K:
 		tcr_el1 |= 5ul << 32; /* IPS = 48 bits */
 		break;
 	case VM_MODE_P40V48_4K:
+	case VM_MODE_P40V48_16K:
 	case VM_MODE_P40V48_64K:
 		tcr_el1 |= 2ul << 32; /* IPS = 40 bits */
 		break;
 	case VM_MODE_P36V48_4K:
+	case VM_MODE_P36V48_16K:
 	case VM_MODE_P36V48_64K:
+	case VM_MODE_P36V47_16K:
 		tcr_el1 |= 1ul << 32; /* IPS = 36 bits */
 		break;
 	default:

tools/testing/selftests/kvm/lib/guest_modes.c +4 -0

index 240f2d2e2d23..da315d3373a3 100644
@@ -29,17 +29,21 @@ void guest_modes_append_default(void)
 			guest_mode_append(VM_MODE_P52V48_64K, ps64k, ps64k);
 		if (limit >= 48) {
 			guest_mode_append(VM_MODE_P48V48_4K, ps4k, ps4k);
+			guest_mode_append(VM_MODE_P48V48_16K, ps16k, ps16k);
 			guest_mode_append(VM_MODE_P48V48_64K, ps64k, ps64k);
 		}
 		if (limit >= 40) {
 			guest_mode_append(VM_MODE_P40V48_4K, ps4k, ps4k);
+			guest_mode_append(VM_MODE_P40V48_16K, ps16k, ps16k);
 			guest_mode_append(VM_MODE_P40V48_64K, ps64k, ps64k);
 			if (ps4k)
 				vm_mode_default = VM_MODE_P40V48_4K;
 		}
 		if (limit >= 36) {
 			guest_mode_append(VM_MODE_P36V48_4K, ps4k, ps4k);
+			guest_mode_append(VM_MODE_P36V48_16K, ps16k, ps16k);
 			guest_mode_append(VM_MODE_P36V48_64K, ps64k, ps64k);
+			guest_mode_append(VM_MODE_P36V47_16K, ps16k, ps16k);
 		}
 
 		/*

tools/testing/selftests/kvm/lib/kvm_util.c +16 -0

index 9da71e27cd84..658a964ad641 100644
@@ -166,14 +166,18 @@ const char *vm_guest_mode_string(uint32_t i)
 		[VM_MODE_P52V48_4K]	= "PA-bits:52,  VA-bits:48,  4K pages",
 		[VM_MODE_P52V48_64K]	= "PA-bits:52,  VA-bits:48, 64K pages",
 		[VM_MODE_P48V48_4K]	= "PA-bits:48,  VA-bits:48,  4K pages",
+		[VM_MODE_P48V48_16K]	= "PA-bits:48,  VA-bits:48, 16K pages",
 		[VM_MODE_P48V48_64K]	= "PA-bits:48,  VA-bits:48, 64K pages",
 		[VM_MODE_P40V48_4K]	= "PA-bits:40,  VA-bits:48,  4K pages",
+		[VM_MODE_P40V48_16K]	= "PA-bits:40,  VA-bits:48, 16K pages",
 		[VM_MODE_P40V48_64K]	= "PA-bits:40,  VA-bits:48, 64K pages",
 		[VM_MODE_PXXV48_4K]	= "PA-bits:ANY, VA-bits:48,  4K pages",
 		[VM_MODE_P47V64_4K]	= "PA-bits:47,  VA-bits:64,  4K pages",
 		[VM_MODE_P44V64_4K]	= "PA-bits:44,  VA-bits:64,  4K pages",
 		[VM_MODE_P36V48_4K]	= "PA-bits:36,  VA-bits:48,  4K pages",
+		[VM_MODE_P36V48_16K]	= "PA-bits:36,  VA-bits:48, 16K pages",
 		[VM_MODE_P36V48_64K]	= "PA-bits:36,  VA-bits:48, 64K pages",
+		[VM_MODE_P36V47_16K]	= "PA-bits:36,  VA-bits:47, 16K pages",
 	};
 	_Static_assert(sizeof(strings)/sizeof(char *) == NUM_VM_MODES,
 		       "Missing new mode strings?");

@@ -187,14 +191,18 @@ const struct vm_guest_mode_params vm_guest_mode_params[] = {
 	[VM_MODE_P52V48_4K]	= { 52, 48,  0x1000, 12 },
 	[VM_MODE_P52V48_64K]	= { 52, 48, 0x10000, 16 },
 	[VM_MODE_P48V48_4K]	= { 48, 48,  0x1000, 12 },
+	[VM_MODE_P48V48_16K]	= { 48, 48,  0x4000, 14 },
 	[VM_MODE_P48V48_64K]	= { 48, 48, 0x10000, 16 },
 	[VM_MODE_P40V48_4K]	= { 40, 48,  0x1000, 12 },
+	[VM_MODE_P40V48_16K]	= { 40, 48,  0x4000, 14 },
 	[VM_MODE_P40V48_64K]	= { 40, 48, 0x10000, 16 },
 	[VM_MODE_PXXV48_4K]	= {  0,  0,  0x1000, 12 },
 	[VM_MODE_P47V64_4K]	= { 47, 64,  0x1000, 12 },
 	[VM_MODE_P44V64_4K]	= { 44, 64,  0x1000, 12 },
 	[VM_MODE_P36V48_4K]	= { 36, 48,  0x1000, 12 },
+	[VM_MODE_P36V48_16K]	= { 36, 48,  0x4000, 14 },
 	[VM_MODE_P36V48_64K]	= { 36, 48, 0x10000, 16 },
+	[VM_MODE_P36V47_16K]	= { 36, 47,  0x4000, 14 },
 };
 _Static_assert(sizeof(vm_guest_mode_params)/sizeof(struct vm_guest_mode_params) == NUM_VM_MODES,
 	       "Missing new mode params?");

@@ -263,6 +271,14 @@ struct kvm_vm *vm_create(enum vm_guest_mode mode, uint64_t phy_pages, int perm)
 	case VM_MODE_P36V48_64K:
 		vm->pgtable_levels = 3;
 		break;
+	case VM_MODE_P48V48_16K:
+	case VM_MODE_P40V48_16K:
+	case VM_MODE_P36V48_16K:
+		vm->pgtable_levels = 4;
+		break;
+	case VM_MODE_P36V47_16K:
+		vm->pgtable_levels = 3;
+		break;
 	case VM_MODE_PXXV48_4K:
 #ifdef __x86_64__
 		kvm_get_cpu_address_width(&vm->pa_bits, &vm->va_bits);

Signature · 1 line

2.30.2

Andrew Jones

Reviewed-by on v2 6/6

2021-12-28 09:38 UTC #13 lore raw

Re: [PATCH v2 6/6] KVM: selftests: arm64: Add support for various modes with 16kB page size

Quoted 6 lines · Marc Zyngier

The 16kB page size is not a popular choice, due to only a few CPUs actually implementing support for it. However, it can lead to some interesting performance improvements given the right uarch choices.
Add support for this page size for various PA/VA combinations.

Quoted 8 lines

Signed-off-by Marc Zyngier <[email protected]>

---
 tools/testing/selftests/kvm/include/kvm_util.h   |  4 ++++
 .../selftests/kvm/lib/aarch64/processor.c        | 10 ++++++++++
 tools/testing/selftests/kvm/lib/guest_modes.c    |  4 ++++
 tools/testing/selftests/kvm/lib/kvm_util.c       | 16 ++++++++++++++++
 4 files changed, 34 insertions(+)

Reviewed-by Andrew Jones <[email protected]>

Marc Zyngier

author maintainer Applied

2021-12-28 11:09 UTC #14 lore raw

In reply to Marc Zyngier

Quoted 4 lines · Marc Zyngier

Now that the common variety of fruity arm64 systems makes a pretty remarkable KVM host, I have decided to run the collection of selftests on it.

Quoted 8 lines

Nothing works. Oh no!
As it turns out, the tests have a notion of "default mode" (40bit PA, 4kB), which cannot work in general on arm64 because there no such thing as an IPA size that would be valid everywhere (apart from the minimal 32bit), nor a guaranteed to be supported page size.
[...]

Applied to next, thanks!

[1/6] KVM: selftests: arm64: Initialise default guest mode at test startup time

      commit: cb7c4f364abd09abd1865fa049ef492fb43e6bf3
[2/6] KVM: selftests: arm64: Introduce a variable default IPA size
      commit: 357c628e1248dd53f5c43a768246a83478a7f489
[3/6] KVM: selftests: arm64: Check for supported page sizes
      commit: 0303ffdb9ecffac4654b16bbf69ba84d131eb8b7
[4/6] KVM: selftests: arm64: Rework TCR_EL1 configuration
      commit: 2f41a61c54fb6410202b2cc08be80ae9554d599d
[5/6] KVM: selftests: arm64: Add support for VM_MODE_P36V48_{4K,64K}
      commit: e7f58a6bd28bfd2e4f60312abf48f07de2c4121c
[6/6] KVM: selftests: arm64: Add support for various modes with 16kB page size
      commit: aa674de1dc3d2bdf2c67ad195dc81977972323c6

Cheers,

M.

Signature · 1 line

Without deviation from the norm, progress is not possible.