Skip to contentSkip to conversation

KVM: arm64: nv: Implement FEAT_XNX and FEAT_HAF

PATCH Applied Oliver Upton 4 participants 21 messages Nov 24, 2025 – Mar 17

Subsystems (4)
KVM Kselftest arm64 KVM arm64
lore
Share & tools
lore

Oliver Upton

author maintainer
#0 lore raw

v3 cover KVM: arm64: nv: Implement FEAT_XNX and FEAT_HAF

Fixes all the issues Marc found in v2:

v2: https://lore.kernel.org/kvmarm/[email protected]/

v2 -> v3:
- Don't use uninitialized XN value in kvm_pgtable_stage2_relax_perms()
- Pass vcpu pointer directly in the stage-2 PTW
- Ensure LL/SC swap implementation correctly handles compare failures
- Fix SRCU imbalance

Oliver Upton (15):
arm64: Detect FEAT_XNX
KVM: arm64: Add support for FEAT_XNX stage-2 permissions
KVM: arm64: nv: Forward FEAT_XNX permissions to the shadow stage-2
KVM: arm64: Teach ptdump about FEAT_XNX permissions
KVM: arm64: nv: Advertise support for FEAT_XNX
KVM: arm64: Call helper for reading descriptors directly
KVM: arm64: nv: Stop passing vCPU through void ptr in S2 PTW
KVM: arm64: Handle endianness in read helper for emulated PTW
KVM: arm64: nv: Use pgtable definitions in stage-2 walk
KVM: arm64: Add helper for swapping guest descriptor
KVM: arm64: Propagate PTW errors up to AT emulation
KVM: arm64: Implement HW access flag management in stage-1 SW PTW
KVM: arm64: nv: Implement HW access flag management in stage-2 SW PTW
KVM: arm64: nv: Expose hardware access flag management to NV guests
KVM: arm64: selftests: Add test for AT emulation

 arch/arm64/include/asm/kvm_asm.h              |   6 +-
 arch/arm64/include/asm/kvm_nested.h           |  40 +++-
 arch/arm64/include/asm/kvm_pgtable.h          |  17 +-
 arch/arm64/kernel/cpufeature.c                |   7 +
 arch/arm64/kvm/at.c                           | 186 ++++++++++++++++--
 arch/arm64/kvm/hyp/pgtable.c                  |  58 +++++-
 arch/arm64/kvm/mmu.c                          |  28 ++-
 arch/arm64/kvm/nested.c                       | 126 ++++++++----
 arch/arm64/kvm/ptdump.c                       |  35 +++-
 arch/arm64/kvm/sys_regs.c                     |   9 +-
 arch/arm64/tools/cpucaps                      |   1 +
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 tools/testing/selftests/kvm/arm64/at.c        | 172 ++++++++++++++++
 .../testing/selftests/kvm/include/kvm_util.h  |   1 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |  10 +
 15 files changed, 602 insertions(+), 95 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/arm64/at.c

base-commit: dcb6fa37fd7bc9c3d2b066329b0d27dedf8becaa

Signature · 1 line
2.47.3
author maintainer
#1 lore raw

PATCH v3 1/15 arm64: Detect FEAT_XNX

Detect the feature in anticipation of using it in KVM.

Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/kernel/cpufeature.c | 7 +++++++
 arch/arm64/tools/cpucaps       | 1 +
 2 files changed, 8 insertions(+)
arch/arm64/kernel/cpufeature.c +7 -0
index 5ed401ff79e3..aa3ecae252d3 100644
@@ -3088,6 +3088,13 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.capability = ARM64_HAS_GICV5_LEGACY,
 		.matches = test_has_gicv5_legacy,
 	},
+	{
+		.desc = "XNX",
+		.capability = ARM64_HAS_XNX,
+		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+		.matches = has_cpuid_feature,
+		ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, XNX, IMP)
+	},
 	{},
 };
arch/arm64/tools/cpucaps +1 -0
index 1b32c1232d28..ee74199107d3 100644
@@ -64,6 +64,7 @@ HAS_TLB_RANGE
 HAS_VA52
 HAS_VIRT_HOST_EXTN
 HAS_WFXT
+HAS_XNX
 HAFT
 HW_DBM
 KVM_HVHE
Signature · 1 line
2.47.3
author maintainer
#2 lore raw

PATCH v3 2/15 KVM: arm64: Add support for FEAT_XNX stage-2 permissions

In reply to Oliver Upton

FEAT_XNX adds support for encoding separate execute permissions for EL0 and EL1 at stage-2. Add support for this to the page table library, hiding the unintuitive encoding scheme behind generic pX and uX permission flags.

Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/include/asm/kvm_pgtable.h | 17 ++++----
 arch/arm64/kvm/hyp/pgtable.c         | 58 ++++++++++++++++++++++++----
 2 files changed, 60 insertions(+), 15 deletions(-)
arch/arm64/include/asm/kvm_pgtable.h +10 -7
index 2888b5d03757..c72149a607d6 100644
@@ -89,7 +89,7 @@ typedef u64 kvm_pte_t;
 
 #define KVM_PTE_LEAF_ATTR_HI_S1_XN	BIT(54)
 
-#define KVM_PTE_LEAF_ATTR_HI_S2_XN	BIT(54)
+#define KVM_PTE_LEAF_ATTR_HI_S2_XN	GENMASK(54, 53)
 
 #define KVM_PTE_LEAF_ATTR_HI_S1_GP	BIT(50)
 

@@ -251,12 +251,15 @@ enum kvm_pgtable_stage2_flags {
  * @KVM_PGTABLE_PROT_SW3:	Software bit 3.
  */
 enum kvm_pgtable_prot {
-	KVM_PGTABLE_PROT_X			= BIT(0),
-	KVM_PGTABLE_PROT_W			= BIT(1),
-	KVM_PGTABLE_PROT_R			= BIT(2),
-
-	KVM_PGTABLE_PROT_DEVICE			= BIT(3),
-	KVM_PGTABLE_PROT_NORMAL_NC		= BIT(4),
+	KVM_PGTABLE_PROT_PX			= BIT(0),
+	KVM_PGTABLE_PROT_UX			= BIT(1),
+	KVM_PGTABLE_PROT_X			= KVM_PGTABLE_PROT_PX	|
+						  KVM_PGTABLE_PROT_UX,
+	KVM_PGTABLE_PROT_W			= BIT(2),
+	KVM_PGTABLE_PROT_R			= BIT(3),
+
+	KVM_PGTABLE_PROT_DEVICE			= BIT(4),
+	KVM_PGTABLE_PROT_NORMAL_NC		= BIT(5),
 
 	KVM_PGTABLE_PROT_SW0			= BIT(55),
 	KVM_PGTABLE_PROT_SW1			= BIT(56),
arch/arm64/kvm/hyp/pgtable.c +50 -8

94 lines not on this page · raw message

Signature · 1 line
2.47.3
author maintainer
#3 lore raw

PATCH v3 3/15 KVM: arm64: nv: Forward FEAT_XNX permissions to the shadow stage-2

In reply to Oliver Upton

Add support for FEAT_XNX to shadow stage-2 MMUs, being careful to only evaluate XN[0] when the feature is actually exposed to the VM. Restructure the layering of permissions in the fault handler to assume pX and uX then restricting based on the guest's stage-2 afterwards.

Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/include/asm/kvm_nested.h | 37 +++++++++++++++++++++++++++--
 arch/arm64/kvm/mmu.c                | 23 ++++++++++++++----
 arch/arm64/kvm/nested.c             |  5 +++-
 3 files changed, 57 insertions(+), 8 deletions(-)
arch/arm64/include/asm/kvm_nested.h +35 -2
index f7c06a840963..5d967b60414c 100644
@@ -120,9 +120,42 @@ static inline bool kvm_s2_trans_writable(struct kvm_s2_trans *trans)
 	return trans->writable;
 }
 
-static inline bool kvm_s2_trans_executable(struct kvm_s2_trans *trans)
+static inline bool kvm_has_xnx(struct kvm *kvm)
 {
-	return !(trans->desc & BIT(54));
+	return cpus_have_final_cap(ARM64_HAS_XNX) &&
+		kvm_has_feat(kvm, ID_AA64MMFR1_EL1, XNX, IMP);
+}
+
+static inline bool kvm_s2_trans_exec_el0(struct kvm *kvm, struct kvm_s2_trans *trans)
+{
+	u8 xn = FIELD_GET(KVM_PTE_LEAF_ATTR_HI_S2_XN, trans->desc);
+
+	if (!kvm_has_xnx(kvm))
+		xn &= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b10);
+
+	switch (xn) {
+	case 0b00:
+	case 0b01:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static inline bool kvm_s2_trans_exec_el1(struct kvm *kvm, struct kvm_s2_trans *trans)
+{
+	u8 xn = FIELD_GET(KVM_PTE_LEAF_ATTR_HI_S2_XN, trans->desc);
+
+	if (!kvm_has_xnx(kvm))
+		xn &= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b10);
+
+	switch (xn) {
+	case 0b00:
+	case 0b11:
+		return true;
+	default:
+		return false;
+	}
 }
 
 extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa,
arch/arm64/kvm/mmu.c +18 -5
index 7cc964af8d30..96f1786c72fe 100644
@@ -1521,6 +1521,16 @@ static void adjust_nested_fault_perms(struct kvm_s2_trans *nested,
 	*prot |= kvm_encode_nested_level(nested);
 }
 
+static void adjust_nested_exec_perms(struct kvm *kvm,
+				     struct kvm_s2_trans *nested,
+				     enum kvm_pgtable_prot *prot)
+{
+	if (!kvm_s2_trans_exec_el0(kvm, nested))
+		*prot &= ~KVM_PGTABLE_PROT_UX;
+	if (!kvm_s2_trans_exec_el1(kvm, nested))
+		*prot &= ~KVM_PGTABLE_PROT_PX;
+}
+
 #define KVM_PGTABLE_WALK_MEMABORT_FLAGS (KVM_PGTABLE_WALK_HANDLE_FAULT | KVM_PGTABLE_WALK_SHARED)
 
 static int gmem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,

@@ -1572,11 +1582,12 @@ static int gmem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	if (writable)
 		prot |= KVM_PGTABLE_PROT_W;
 
-	if (exec_fault ||
-	    (cpus_have_final_cap(ARM64_HAS_CACHE_DIC) &&
-	     (!nested || kvm_s2_trans_executable(nested))))
+	if (exec_fault || cpus_have_final_cap(ARM64_HAS_CACHE_DIC))
 		prot |= KVM_PGTABLE_PROT_X;
 
+	if (nested)
+		adjust_nested_exec_perms(kvm, nested, &prot);
+
 	kvm_fault_lock(kvm);
 	if (mmu_invalidate_retry(kvm, mmu_seq)) {
 		ret = -EAGAIN;

@@ -1851,11 +1862,13 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 			prot |= KVM_PGTABLE_PROT_NORMAL_NC;
 		else
 			prot |= KVM_PGTABLE_PROT_DEVICE;
-	} else if (cpus_have_final_cap(ARM64_HAS_CACHE_DIC) &&
-		   (!nested || kvm_s2_trans_executable(nested))) {
+	} else if (cpus_have_final_cap(ARM64_HAS_CACHE_DIC)) {
 		prot |= KVM_PGTABLE_PROT_X;
 	}
 
+	if (nested)
+		adjust_nested_exec_perms(kvm, nested, &prot);
+
 	/*
 	 * Under the premise of getting a FSC_PERM fault, we just need to relax
 	 * permissions only if vma_pagesize equals fault_granule. Otherwise,
arch/arm64/kvm/nested.c +4 -1
index f04cda40545b..92b2a69f0b89 100644
@@ -788,7 +788,10 @@ int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, struct kvm_s2_trans *trans)
 		return 0;
 
 	if (kvm_vcpu_trap_is_iabt(vcpu)) {
-		forward_fault = !kvm_s2_trans_executable(trans);
+		if (vcpu_mode_priv(vcpu))
+			forward_fault = !kvm_s2_trans_exec_el1(vcpu->kvm, trans);
+		else
+			forward_fault = !kvm_s2_trans_exec_el0(vcpu->kvm, trans);
 	} else {
 		bool write_fault = kvm_is_write_fault(vcpu);
Signature · 1 line
2.47.3
author maintainer
#4 lore raw

PATCH v3 4/15 KVM: arm64: Teach ptdump about FEAT_XNX permissions

In reply to Oliver Upton

Although KVM doesn't make direct use of the feature, guest hypervisors can use FEAT_XNX which influences the permissions of the shadow stage-2. Update ptdump to separately print the privileged and unprivileged execute permissions.

Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/kvm/ptdump.c | 35 +++++++++++++++++++++++++++--------
 1 file changed, 27 insertions(+), 8 deletions(-)
arch/arm64/kvm/ptdump.c +27 -8
index dc5acfb00af9..bd722383d0e3 100644
@@ -31,27 +31,46 @@ static const struct ptdump_prot_bits stage2_pte_bits[] = {
 		.val	= PTE_VALID,
 		.set	= " ",
 		.clear	= "F",
-	}, {
+	},
+	{
 		.mask	= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R,
 		.val	= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R,
 		.set	= "R",
 		.clear	= " ",
-	}, {
+	},
+	{
 		.mask	= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W,
 		.val	= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W,
 		.set	= "W",
 		.clear	= " ",
-	}, {
+	},
+	{
 		.mask	= KVM_PTE_LEAF_ATTR_HI_S2_XN,
-		.val	= KVM_PTE_LEAF_ATTR_HI_S2_XN,
-		.set	= "NX",
-		.clear	= "x ",
-	}, {
+		.val	= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b00),
+		.set	= "px ux ",
+	},
+	{
+		.mask	= KVM_PTE_LEAF_ATTR_HI_S2_XN,
+		.val	= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b01),
+		.set	= "PXNux ",
+	},
+	{
+		.mask	= KVM_PTE_LEAF_ATTR_HI_S2_XN,
+		.val	= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b10),
+		.set	= "PXNUXN",
+	},
+	{
+		.mask	= KVM_PTE_LEAF_ATTR_HI_S2_XN,
+		.val	= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b11),
+		.set	= "px UXN",
+	},
+	{
 		.mask	= KVM_PTE_LEAF_ATTR_LO_S2_AF,
 		.val	= KVM_PTE_LEAF_ATTR_LO_S2_AF,
 		.set	= "AF",
 		.clear	= "  ",
-	}, {
+	},
+	{
 		.mask	= PMD_TYPE_MASK,
 		.val	= PMD_TYPE_SECT,
 		.set	= "BLK",
Signature · 1 line
2.47.3
#5 lore raw

Re: [PATCH v3 04/15] KVM: arm64: Teach ptdump about FEAT_XNX permissions

Hi Oliver,

Quoted 72 lines · Oliver Upton

Although KVM doesn't make direct use of the feature, guest hypervisors can use FEAT_XNX which influences the permissions of the shadow stage-2. Update ptdump to separately print the privileged and unprivileged execute permissions.

Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/kvm/ptdump.c | 35 +++++++++++++++++++++++++++--------
 1 file changed, 27 insertions(+), 8 deletions(-)
arch/arm64/kvm/ptdump.c +27 -8
index dc5acfb00af9..bd722383d0e3 100644
@@ -31,27 +31,46 @@ static const struct ptdump_prot_bits stage2_pte_bits[] = {
 		.val	= PTE_VALID,
 		.set	= " ",
 		.clear	= "F",
-	}, {
+	},
+	{
 		.mask	= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R,
 		.val	= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R,
 		.set	= "R",
 		.clear	= " ",
-	}, {
+	},
+	{
 		.mask	= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W,
 		.val	= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W,
 		.set	= "W",
 		.clear	= " ",
-	}, {
+	},
+	{
 		.mask	= KVM_PTE_LEAF_ATTR_HI_S2_XN,
-		.val	= KVM_PTE_LEAF_ATTR_HI_S2_XN,
-		.set	= "NX",
-		.clear	= "x ",
-	}, {
+		.val	= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b00),
+		.set	= "px ux ",
+	},
+	{
+		.mask	= KVM_PTE_LEAF_ATTR_HI_S2_XN,
+		.val	= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b01),
+		.set	= "PXNux ",
+	},
+	{
+		.mask	= KVM_PTE_LEAF_ATTR_HI_S2_XN,
+		.val	= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b10),
+		.set	= "PXNUXN",
+	},
+	{
+		.mask	= KVM_PTE_LEAF_ATTR_HI_S2_XN,
+		.val	= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b11),
+		.set	= "px UXN",
+	},
+	{
 		.mask	= KVM_PTE_LEAF_ATTR_LO_S2_AF,
 		.val	= KVM_PTE_LEAF_ATTR_LO_S2_AF,
 		.set	= "AF",
 		.clear	= "  ",
-	}, {
+	},
+	{
 		.mask	= PMD_TYPE_MASK,
 		.val	= PMD_TYPE_SECT,
 		.set	= "BLK",
Signature · 1 line
2.47.3

This change as commit 81e9fe5c6f25 ("KVM: arm64: Teach ptdump about FEAT_XNX permissions") in -next does not build for me.

GCC 15.2.0

In file included from arch/arm64/include/asm/sysreg.h:1155,

                   from arch/arm64/include/asm/memory.h:219,
                   from arch/arm64/include/asm/thread_info.h:17,
                   from include/linux/thread_info.h:60,
                   from arch/arm64/include/asm/preempt.h:5,
                   from include/linux/preempt.h:79,
                   from include/linux/spinlock.h:56,
                   from include/linux/wait.h:9,
                   from include/linux/wait_bit.h:8,
                   from include/linux/fs.h:7,
                   from include/linux/debugfs.h:15,
                   from arch/arm64/kvm/ptdump.c:9:
  include/linux/bitfield.h:114:9: error: braced-group within expression allowed only inside a function
    114 |         ({                                                              \
        |         ^
  arch/arm64/kvm/ptdump.c:49:27: note: in expansion of macro 'FIELD_PREP'
     49 |                 .val    = FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b00),
        |                           ^~~~~~~~~~
  include/linux/bitfield.h:114:9: error: braced-group within expression allowed only inside a function
    114 |         ({                                                              \
        |         ^
  arch/arm64/kvm/ptdump.c:54:27: note: in expansion of macro 'FIELD_PREP'
     54 |                 .val    = FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b01),
        |                           ^~~~~~~~~~
  include/linux/bitfield.h:114:9: error: braced-group within expression allowed only inside a function
    114 |         ({                                                              \
        |         ^
  arch/arm64/kvm/ptdump.c:59:27: note: in expansion of macro 'FIELD_PREP'
     59 |                 .val    = FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b10),
        |                           ^~~~~~~~~~
  include/linux/bitfield.h:114:9: error: braced-group within expression allowed only inside a function
    114 |         ({                                                              \
        |         ^
  arch/arm64/kvm/ptdump.c:64:27: note: in expansion of macro 'FIELD_PREP'
     64 |                 .val    = FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b11),
        |                           ^~~~~~~~~~

clang 21.1.6:

arch/arm64/kvm/ptdump.c:49:10: error: statement expression not allowed at file scope

     49 |                 .val    = FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b00),
        |                           ^
  include/linux/bitfield.h:114:2: note: expanded from macro 'FIELD_PREP'
    114 |         ({                                                              \
        |         ^
  arch/arm64/kvm/ptdump.c:54:10: error: statement expression not allowed at file scope
     54 |                 .val    = FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b01),
        |                           ^
  include/linux/bitfield.h:114:2: note: expanded from macro 'FIELD_PREP'
    114 |         ({                                                              \
        |         ^
  arch/arm64/kvm/ptdump.c:59:10: error: statement expression not allowed at file scope
     59 |                 .val    = FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b10),
        |                           ^
  include/linux/bitfield.h:114:2: note: expanded from macro 'FIELD_PREP'
    114 |         ({                                                              \
        |         ^
  arch/arm64/kvm/ptdump.c:64:10: error: statement expression not allowed at file scope
     64 |                 .val    = FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b11),
        |                           ^
  include/linux/bitfield.h:114:2: note: expanded from macro 'FIELD_PREP'
    114 |         ({                                                              \
        |         ^

Cheers,
Nathan

author maintainer
#6 lore raw

PATCH v3 5/15 KVM: arm64: nv: Advertise support for FEAT_XNX

In reply to Oliver Upton

Everything is in place to support FEAT_XNX, advertise support.

Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/kvm/nested.c | 1 -
 1 file changed, 1 deletion(-)
arch/arm64/kvm/nested.c +0 -1
index 92b2a69f0b89..08839a320a45 100644
@@ -1559,7 +1559,6 @@ u64 limit_nv_id_reg(struct kvm *kvm, u32 reg, u64 val)
 		val &= ~(ID_AA64MMFR1_EL1_CMOW		|
 			 ID_AA64MMFR1_EL1_nTLBPA	|
 			 ID_AA64MMFR1_EL1_ETS		|
-			 ID_AA64MMFR1_EL1_XNX		|
 			 ID_AA64MMFR1_EL1_HAFDBS);
 		/* FEAT_E2H0 implies no VHE */
 		if (test_bit(KVM_ARM_VCPU_HAS_EL2_E2H0, kvm->arch.vcpu_features))
Signature · 1 line
2.47.3
author maintainer
#7 lore raw

PATCH v3 6/15 KVM: arm64: Call helper for reading descriptors directly

In reply to Oliver Upton

Going through a function pointer doesn't serve much purpose when there's only one implementation.

Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/kvm/nested.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)
arch/arm64/kvm/nested.c +8 -10
index 08839a320a45..e4928a6a3672 100644
@@ -124,7 +124,6 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu)
 }
 
 struct s2_walk_info {
-	int	     (*read_desc)(phys_addr_t pa, u64 *desc, void *data);
 	void	     *data;
 	u64	     baddr;
 	unsigned int max_oa_bits;

@@ -199,6 +198,13 @@ static int check_output_size(struct s2_walk_info *wi, phys_addr_t output)
 	return 0;
 }
 
+static int read_guest_s2_desc(phys_addr_t pa, u64 *desc, void *data)
+{
+	struct kvm_vcpu *vcpu = data;
+
+	return kvm_read_guest(vcpu->kvm, pa, desc, sizeof(*desc));
+}
+
 /*
  * This is essentially a C-version of the pseudo code from the ARM ARM
  * AArch64.TranslationTableWalk  function.  I strongly recommend looking at

@@ -257,7 +263,7 @@ static int walk_nested_s2_pgd(phys_addr_t ipa,
 			>> (addr_bottom - 3);
 
 		paddr = base_addr | index;
-		ret = wi->read_desc(paddr, &desc, wi->data);
+		ret = read_guest_s2_desc(paddr, &desc, wi->data);
 		if (ret < 0)
 			return ret;
 

@@ -325,13 +331,6 @@ static int walk_nested_s2_pgd(phys_addr_t ipa,
 	return 0;
 }
 
-static int read_guest_s2_desc(phys_addr_t pa, u64 *desc, void *data)
-{
-	struct kvm_vcpu *vcpu = data;
-
-	return kvm_read_guest(vcpu->kvm, pa, desc, sizeof(*desc));
-}
-
 static void vtcr_to_walk_info(u64 vtcr, struct s2_walk_info *wi)
 {
 	wi->t0sz = vtcr & TCR_EL2_T0SZ_MASK;

@@ -364,7 +363,6 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa,
 	if (!vcpu_has_nv(vcpu))
 		return 0;
 
-	wi.read_desc = read_guest_s2_desc;
 	wi.data = vcpu;
 	wi.baddr = vcpu_read_sys_reg(vcpu, VTTBR_EL2);
Signature · 1 line
2.47.3
author maintainer
#8 lore raw

PATCH v3 7/15 KVM: arm64: nv: Stop passing vCPU through void ptr in S2 PTW

In reply to Oliver Upton

The stage-2 table walker passes down the vCPU as a void pointer. That might've made sense if the walker was generic although at this point it is clear this will only ever be used in the context of a vCPU.

Suggested-by Marc Zyngier <[email protected]>
Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/kvm/nested.c | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)
arch/arm64/kvm/nested.c +4 -8
index e4928a6a3672..94eff8307aad 100644
@@ -124,7 +124,6 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu)
 }
 
 struct s2_walk_info {
-	void	     *data;
 	u64	     baddr;
 	unsigned int max_oa_bits;
 	unsigned int pgshift;

@@ -198,10 +197,8 @@ static int check_output_size(struct s2_walk_info *wi, phys_addr_t output)
 	return 0;
 }
 
-static int read_guest_s2_desc(phys_addr_t pa, u64 *desc, void *data)
+static int read_guest_s2_desc(struct kvm_vcpu *vcpu, phys_addr_t pa, u64 *desc)
 {
-	struct kvm_vcpu *vcpu = data;
-
 	return kvm_read_guest(vcpu->kvm, pa, desc, sizeof(*desc));
 }
 

@@ -212,7 +209,7 @@ static int read_guest_s2_desc(phys_addr_t pa, u64 *desc, void *data)
  *
  * Must be called with the kvm->srcu read lock held
  */
-static int walk_nested_s2_pgd(phys_addr_t ipa,
+static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa,
 			      struct s2_walk_info *wi, struct kvm_s2_trans *out)
 {
 	int first_block_level, level, stride, input_size, base_lower_bound;

@@ -263,7 +260,7 @@ static int walk_nested_s2_pgd(phys_addr_t ipa,
 			>> (addr_bottom - 3);
 
 		paddr = base_addr | index;
-		ret = read_guest_s2_desc(paddr, &desc, wi->data);
+		ret = read_guest_s2_desc(vcpu, paddr, &desc);
 		if (ret < 0)
 			return ret;
 

@@ -363,14 +360,13 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa,
 	if (!vcpu_has_nv(vcpu))
 		return 0;
 
-	wi.data = vcpu;
 	wi.baddr = vcpu_read_sys_reg(vcpu, VTTBR_EL2);
 
 	vtcr_to_walk_info(vtcr, &wi);
 
 	wi.be = vcpu_read_sys_reg(vcpu, SCTLR_EL2) & SCTLR_ELx_EE;
 
-	ret = walk_nested_s2_pgd(gipa, &wi, result);
+	ret = walk_nested_s2_pgd(vcpu, gipa, &wi, result);
 	if (ret)
 		result->esr |= (kvm_vcpu_get_esr(vcpu) & ~ESR_ELx_FSC);
Signature · 1 line
2.47.3
author maintainer
#9 lore raw

PATCH v3 8/15 KVM: arm64: Handle endianness in read helper for emulated PTW

In reply to Oliver Upton

Implementing FEAT_HAFDBS means adding another descriptor accessor that needs to deal with the guest-configured endianness. Prepare by moving the endianness handling into the read accessor and out of the main body of the S1/S2 PTWs.

Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/kvm/at.c     | 25 +++++++++++++++++++------
 arch/arm64/kvm/nested.c | 32 ++++++++++++++++++++------------
 2 files changed, 39 insertions(+), 18 deletions(-)
arch/arm64/kvm/at.c +19 -6
index be26d5aa668c..a295a37dd3b1 100644
@@ -362,6 +362,24 @@ static int setup_s1_walk(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
 	return -EFAULT;
 }
 
+static int kvm_read_s1_desc(struct kvm_vcpu *vcpu, u64 pa, u64 *desc,
+			    struct s1_walk_info *wi)
+{
+	u64 val;
+	int r;
+
+	r = kvm_read_guest(vcpu->kvm, pa, &val, sizeof(val));
+	if (r)
+		return r;
+
+	if (wi->be)
+		*desc = be64_to_cpu((__force __be64)val);
+	else
+		*desc = le64_to_cpu((__force __le64)val);
+
+	return 0;
+}
+
 static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
 		   struct s1_walk_result *wr, u64 va)
 {

@@ -414,17 +432,12 @@ static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
 				return ret;
 		}
 
-		ret = kvm_read_guest(vcpu->kvm, ipa, &desc, sizeof(desc));
+		ret = kvm_read_s1_desc(vcpu, ipa, &desc, wi);
 		if (ret) {
 			fail_s1_walk(wr, ESR_ELx_FSC_SEA_TTW(level), false);
 			return ret;
 		}
 
-		if (wi->be)
-			desc = be64_to_cpu((__force __be64)desc);
-		else
-			desc = le64_to_cpu((__force __le64)desc);
-
 		/* Invalid descriptor */
 		if (!(desc & BIT(0)))
 			goto transfault;
arch/arm64/kvm/nested.c +20 -12
index 94eff8307aad..75d26c0ba3e0 100644
@@ -197,9 +197,26 @@ static int check_output_size(struct s2_walk_info *wi, phys_addr_t output)
 	return 0;
 }
 
-static int read_guest_s2_desc(struct kvm_vcpu *vcpu, phys_addr_t pa, u64 *desc)
+static int read_guest_s2_desc(struct kvm_vcpu *vcpu, phys_addr_t pa, u64 *desc,
+			      struct s2_walk_info *wi)
 {
-	return kvm_read_guest(vcpu->kvm, pa, desc, sizeof(*desc));
+	u64 val;
+	int r;
+
+	r = kvm_read_guest(vcpu->kvm, pa, &val, sizeof(val));
+	if (r)
+		return r;
+
+	/*
+	 * Handle reversedescriptors if endianness differs between the
+	 * host and the guest hypervisor.
+	 */
+	if (wi->be)
+		*desc = be64_to_cpu((__force __be64)val);
+	else
+		*desc = le64_to_cpu((__force __le64)val);
+
+	return 0;
 }
 
 /*

@@ -260,19 +277,10 @@ static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa,
 			>> (addr_bottom - 3);
 
 		paddr = base_addr | index;
-		ret = read_guest_s2_desc(vcpu, paddr, &desc);
+		ret = read_guest_s2_desc(vcpu, paddr, &desc, wi);
 		if (ret < 0)
 			return ret;
 
-		/*
-		 * Handle reversedescriptors if endianness differs between the
-		 * host and the guest hypervisor.
-		 */
-		if (wi->be)
-			desc = be64_to_cpu((__force __be64)desc);
-		else
-			desc = le64_to_cpu((__force __le64)desc);
-
 		/* Check for valid descriptor at this point */
 		if (!(desc & 1) || ((desc & 3) == 1 && level == 3)) {
 			out->esr = compute_fsc(level, ESR_ELx_FSC_FAULT);
Signature · 1 line
2.47.3
author maintainer
#10 lore raw

PATCH v3 9/15 KVM: arm64: nv: Use pgtable definitions in stage-2 walk

In reply to Oliver Upton

Use the existing page table definitions instead of magic numbers for the stage-2 table walk.

Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/kvm/nested.c | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)
arch/arm64/kvm/nested.c +15 -6
index 75d26c0ba3e0..a096766c6ec3 100644
@@ -282,14 +282,23 @@ static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa,
 			return ret;
 
 		/* Check for valid descriptor at this point */
-		if (!(desc & 1) || ((desc & 3) == 1 && level == 3)) {
+		if (!(desc & KVM_PTE_VALID)) {
 			out->esr = compute_fsc(level, ESR_ELx_FSC_FAULT);
 			out->desc = desc;
 			return 1;
 		}
 
-		/* We're at the final level or block translation level */
-		if ((desc & 3) == 1 || level == 3)
+		if (FIELD_GET(KVM_PTE_TYPE, desc) == KVM_PTE_TYPE_BLOCK) {
+			if (level < 3)
+				break;
+
+			out->esr = compute_fsc(level, ESR_ELx_FSC_FAULT);
+			out->desc = desc;
+			return 1;
+		}
+
+		/* We're at the final level */
+		if (level == 3)
 			break;
 
 		if (check_output_size(wi, desc)) {

@@ -316,7 +325,7 @@ static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa,
 		return 1;
 	}
 
-	if (!(desc & BIT(10))) {
+	if (!(desc & KVM_PTE_LEAF_ATTR_LO_S2_AF)) {
 		out->esr = compute_fsc(level, ESR_ELx_FSC_ACCESS);
 		out->desc = desc;
 		return 1;

@@ -329,8 +338,8 @@ static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa,
 		(ipa & GENMASK_ULL(addr_bottom - 1, 0));
 	out->output = paddr;
 	out->block_size = 1UL << ((3 - level) * stride + wi->pgshift);
-	out->readable = desc & (0b01 << 6);
-	out->writable = desc & (0b10 << 6);
+	out->readable = desc & KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R;
+	out->writable = desc & KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W;
 	out->level = level;
 	out->desc = desc;
 	return 0;
Signature · 1 line
2.47.3
author maintainer
#11 lore raw

PATCH v3 10/15 KVM: arm64: Add helper for swapping guest descriptor

In reply to Oliver Upton

Implementing FEAT_HAFDBS in KVM's software PTWs requires the ability to CAS a descriptor to update the in-memory value. Add an accessor to do exactly that, coping with the fact that guest descriptors are in user memory (duh).

While FEAT_LSE required on any system that implements NV, KVM now uses the stage-1 PTW for non-nested use cases meaning an LL/SC implementation is necessary as well.

Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/include/asm/kvm_nested.h |  2 +
 arch/arm64/kvm/at.c                 | 87 +++++++++++++++++++++++++++++
 2 files changed, 89 insertions(+)
arch/arm64/include/asm/kvm_nested.h +2 -0
index 5d967b60414c..6dbc2908aed9 100644
@@ -403,4 +403,6 @@ void kvm_handle_s1e2_tlbi(struct kvm_vcpu *vcpu, u32 inst, u64 val);
 		(FIX_VNCR - __c);				\
 	})
 
+int __kvm_at_swap_desc(struct kvm *kvm, gpa_t ipa, u64 old, u64 new);
+
 #endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/at.c +87 -0

90 lines not on this page · raw message

Signature · 1 line
2.47.3
author maintainer
#12 lore raw

PATCH v3 11/15 KVM: arm64: Propagate PTW errors up to AT emulation

In reply to Oliver Upton

KVM's software PTW will soon support 'hardware' updates to the access flag. Similar to fault handling, races to update the descriptor will be handled by restarting the instruction. Prepare for this by propagating errors up to the AT emulation, only retiring the instruction if the walk succeeds.

Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/include/asm/kvm_asm.h |  6 ++---
 arch/arm64/kvm/at.c              | 43 ++++++++++++++++++++++----------
 arch/arm64/kvm/sys_regs.c        |  9 ++++---
 3 files changed, 39 insertions(+), 19 deletions(-)
arch/arm64/include/asm/kvm_asm.h +3 -3
index 9da54d4ee49e..090f7b740bdc 100644
@@ -246,9 +246,9 @@ extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu);
 extern int __kvm_tlbi_s1e2(struct kvm_s2_mmu *mmu, u64 va, u64 sys_encoding);
 
 extern void __kvm_timer_set_cntvoff(u64 cntvoff);
-extern void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
-extern void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
-extern void __kvm_at_s12(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
+extern int __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
+extern int __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
+extern int __kvm_at_s12(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
 
 extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
arch/arm64/kvm/at.c +30 -13

120 lines not on this page · raw message

arch/arm64/kvm/sys_regs.c +6 -3
index e67eb39ddc11..61830eb3607c 100644
@@ -3767,7 +3767,8 @@ static bool handle_at_s1e01(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
 {
 	u32 op = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2);
 
-	__kvm_at_s1e01(vcpu, op, p->regval);
+	if (__kvm_at_s1e01(vcpu, op, p->regval))
+		return false;
 
 	return true;
 }

@@ -3784,7 +3785,8 @@ static bool handle_at_s1e2(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
 		return false;
 	}
 
-	__kvm_at_s1e2(vcpu, op, p->regval);
+	if (__kvm_at_s1e2(vcpu, op, p->regval))
+		return false;
 
 	return true;
 }

@@ -3794,7 +3796,8 @@ static bool handle_at_s12(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
 {
 	u32 op = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2);
 
-	__kvm_at_s12(vcpu, op, p->regval);
+	if (__kvm_at_s12(vcpu, op, p->regval))
+		return false;
 
 	return true;
 }
Signature · 1 line
2.47.3
author maintainer
#13 lore raw

PATCH v3 12/15 KVM: arm64: Implement HW access flag management in stage-1 SW PTW

In reply to Oliver Upton

Atomically update the Access flag at stage-1 when the guest has configured the MMU to do so. Make the implementation choice (and liberal interpretation of speculation) that any access type updates the Access flag, including AT and CMO instructions.

Restart the entire walk by returning to the exception-generating instruction in the case of a failed Access flag update.

Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/include/asm/kvm_nested.h |  1 +
 arch/arm64/kvm/at.c                 | 33 +++++++++++++++++++++++++++--
 2 files changed, 32 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_nested.h +1 -0
index 6dbc2908aed9..905c658057a4 100644
@@ -353,6 +353,7 @@ struct s1_walk_info {
 	bool	     		be;
 	bool	     		s2;
 	bool			pa52bit;
+	bool			ha;
 };
 
 struct s1_walk_result {
arch/arm64/kvm/at.c +31 -2
index 2a99380ada6f..e39f814d247f 100644
@@ -346,6 +346,8 @@ static int setup_s1_walk(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
 
 	wi->baddr &= GENMASK_ULL(wi->max_oa_bits - 1, x);
 
+	wi->ha = tcr & TCR_HA;
+
 	return 0;
 
 addrsz:

@@ -380,10 +382,24 @@ static int kvm_read_s1_desc(struct kvm_vcpu *vcpu, u64 pa, u64 *desc,
 	return 0;
 }
 
+static int kvm_swap_s1_desc(struct kvm_vcpu *vcpu, u64 pa, u64 old, u64 new,
+			    struct s1_walk_info *wi)
+{
+	if (wi->be) {
+		old = cpu_to_be64(old);
+		new = cpu_to_be64(new);
+	} else {
+		old = cpu_to_le64(old);
+		new = cpu_to_le64(new);
+	}
+
+	return __kvm_at_swap_desc(vcpu->kvm, pa, old, new);
+}
+
 static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
 		   struct s1_walk_result *wr, u64 va)
 {
-	u64 va_top, va_bottom, baddr, desc;
+	u64 va_top, va_bottom, baddr, desc, new_desc, ipa;
 	int level, stride, ret;
 
 	level = wi->sl;

@@ -393,7 +409,7 @@ static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
 	va_top = get_ia_size(wi) - 1;
 
 	while (1) {
-		u64 index, ipa;
+		u64 index;
 
 		va_bottom = (3 - level) * stride + wi->pgshift;
 		index = (va & GENMASK_ULL(va_top, va_bottom)) >> (va_bottom - 3);

@@ -438,6 +454,8 @@ static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
 			return ret;
 		}
 
+		new_desc = desc;
+
 		/* Invalid descriptor */
 		if (!(desc & BIT(0)))
 			goto transfault;

@@ -490,6 +508,17 @@ static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
 	if (check_output_size(baddr & GENMASK(52, va_bottom), wi))
 		goto addrsz;
 
+	if (wi->ha)
+		new_desc |= PTE_AF;
+
+	if (new_desc != desc) {
+		ret = kvm_swap_s1_desc(vcpu, ipa, desc, new_desc, wi);
+		if (ret)
+			return ret;
+
+		desc = new_desc;
+	}
+
 	if (!(desc & PTE_AF)) {
 		fail_s1_walk(wr, ESR_ELx_FSC_ACCESS_L(level), false);
 		return -EACCES;
Signature · 1 line
2.47.3
author maintainer
#14 lore raw

PATCH v3 13/15 KVM: arm64: nv: Implement HW access flag management in stage-2 SW PTW

In reply to Oliver Upton

Give the stage-2 walk similar treatment to stage-1: update the access flag during the table walk and do so for any walk context.

Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/kvm/mmu.c    |  5 +++++
 arch/arm64/kvm/nested.c | 44 ++++++++++++++++++++++++++++++++++-------
 2 files changed, 42 insertions(+), 7 deletions(-)
arch/arm64/kvm/mmu.c +5 -0
index 96f1786c72fe..b9aebca90f59 100644
@@ -2012,6 +2012,11 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
 		u32 esr;
 
 		ret = kvm_walk_nested_s2(vcpu, fault_ipa, &nested_trans);
+		if (ret == -EAGAIN) {
+			ret = 1;
+			goto out_unlock;
+		}
+
 		if (ret) {
 			esr = kvm_s2_trans_esr(&nested_trans);
 			kvm_inject_s2_fault(vcpu, esr);
arch/arm64/kvm/nested.c +37 -7

80 lines not on this page · raw message

Signature · 1 line
2.47.3
author maintainer
#15 lore raw

PATCH v3 14/15 KVM: arm64: nv: Expose hardware access flag management to NV guests

In reply to Oliver Upton

Everything is in place to update the access flag at S1 and S2. Expose support for the access flag flavor of FEAT_HAFDBS to NV guests.

Signed-off-by Oliver Upton <[email protected]>
---
 arch/arm64/kvm/nested.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
arch/arm64/kvm/nested.c +4 -2
index 6495442f400a..88d7dfb44410 100644
@@ -1599,11 +1599,13 @@ u64 limit_nv_id_reg(struct kvm *kvm, u32 reg, u64 val)
 	case SYS_ID_AA64MMFR1_EL1:
 		val &= ~(ID_AA64MMFR1_EL1_CMOW		|
 			 ID_AA64MMFR1_EL1_nTLBPA	|
-			 ID_AA64MMFR1_EL1_ETS		|
-			 ID_AA64MMFR1_EL1_HAFDBS);
+			 ID_AA64MMFR1_EL1_ETS);
+
 		/* FEAT_E2H0 implies no VHE */
 		if (test_bit(KVM_ARM_VCPU_HAS_EL2_E2H0, kvm->arch.vcpu_features))
 			val &= ~ID_AA64MMFR1_EL1_VH;
+
+		val = ID_REG_LIMIT_FIELD_ENUM(val, ID_AA64MMFR1_EL1, HAFDBS, AF);
 		break;
 
 	case SYS_ID_AA64MMFR2_EL1:
Signature · 1 line
2.47.3
author maintainer
#16 lore raw

PATCH v3 15/15 KVM: arm64: selftests: Add test for AT emulation

In reply to Oliver Upton

Add a basic test for AT emulation in the EL2&0 and EL1&0 translation regimes.

Signed-off-by Oliver Upton <[email protected]>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 tools/testing/selftests/kvm/arm64/at.c        | 166 ++++++++++++++++++
 .../testing/selftests/kvm/include/kvm_util.h  |   1 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |  10 ++
 4 files changed, 178 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/arm64/at.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 148d427ff24b..81b3aa54678a 100644
@@ -156,6 +156,7 @@ TEST_GEN_PROGS_EXTENDED_x86 += x86/nx_huge_pages_test
 TEST_GEN_PROGS_arm64 = $(TEST_GEN_PROGS_COMMON)
 TEST_GEN_PROGS_arm64 += arm64/aarch32_id_regs
 TEST_GEN_PROGS_arm64 += arm64/arch_timer_edge_cases
+TEST_GEN_PROGS_arm64 += arm64/at
 TEST_GEN_PROGS_arm64 += arm64/debug-exceptions
 TEST_GEN_PROGS_arm64 += arm64/hello_el2
 TEST_GEN_PROGS_arm64 += arm64/host_sve
tools/testing/selftests/kvm/arm64/at.c new file +166 -0

166 lines not on this page · raw message

tools/testing/selftests/kvm/include/kvm_util.h +1 -0
index d3f3e455c031..41467dad9178 100644
@@ -715,6 +715,7 @@ static inline bool vm_arch_has_protected_memory(struct kvm_vm *vm)
 #endif
 
 void vm_mem_region_set_flags(struct kvm_vm *vm, uint32_t slot, uint32_t flags);
+void vm_mem_region_reload(struct kvm_vm *vm, uint32_t slot);
 void vm_mem_region_move(struct kvm_vm *vm, uint32_t slot, uint64_t new_gpa);
 void vm_mem_region_delete(struct kvm_vm *vm, uint32_t slot);
 struct kvm_vcpu *__vm_vcpu_add(struct kvm_vm *vm, uint32_t vcpu_id);
tools/testing/selftests/kvm/lib/kvm_util.c +10 -0
index 1a93d6361671..d6538bb17740 100644
@@ -1201,6 +1201,16 @@ void vm_mem_region_set_flags(struct kvm_vm *vm, uint32_t slot, uint32_t flags)
 		ret, errno, slot, flags);
 }
 
+void vm_mem_region_reload(struct kvm_vm *vm, uint32_t slot)
+{
+	struct userspace_mem_region *region = memslot2region(vm, slot);
+	struct kvm_userspace_memory_region2 tmp = region->region;
+
+	tmp.memory_size = 0;
+	vm_ioctl(vm, KVM_SET_USER_MEMORY_REGION2, &tmp);
+	vm_ioctl(vm, KVM_SET_USER_MEMORY_REGION2, &region->region);
+}
+
 /*
  * VM Memory Region Move
  *
Signature · 1 line
2.47.3
#17 lore raw

Re: [PATCH v3 15/15] KVM: arm64: selftests: Add test for AT emulation

Hi Oliver,

Quoted 126 lines · Oliver Upton

Add a basic test for AT emulation in the EL2&0 and EL1&0 translation regimes.

Signed-off-by Oliver Upton <[email protected]>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 tools/testing/selftests/kvm/arm64/at.c        | 166 ++++++++++++++++++
 .../testing/selftests/kvm/include/kvm_util.h  |   1 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |  10 ++
 4 files changed, 178 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/arm64/at.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 148d427ff24b..81b3aa54678a 100644
@@ -156,6 +156,7 @@ TEST_GEN_PROGS_EXTENDED_x86 += x86/nx_huge_pages_test
 TEST_GEN_PROGS_arm64 = $(TEST_GEN_PROGS_COMMON)
 TEST_GEN_PROGS_arm64 += arm64/aarch32_id_regs
 TEST_GEN_PROGS_arm64 += arm64/arch_timer_edge_cases
+TEST_GEN_PROGS_arm64 += arm64/at
 TEST_GEN_PROGS_arm64 += arm64/debug-exceptions
 TEST_GEN_PROGS_arm64 += arm64/hello_el2
 TEST_GEN_PROGS_arm64 += arm64/host_sve
tools/testing/selftests/kvm/arm64/at.c new file +96 -0
new file mode 100644
index 000000000000..acecb6ab5071
@@ -0,0 +1,166 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * at - Test for KVM's AT emulation in the EL2&0 and EL1&0 translation regimes.
+ */
+#include "kvm_util.h"
+#include "processor.h"
+#include "test_util.h"
+#include "ucall.h"
+
+#include <asm/sysreg.h>
+
+#define TEST_ADDR	0x80000000
+
+enum {
+	CLEAR_ACCESS_FLAG,
+	TEST_ACCESS_FLAG,
+};
+
+static u64 *ptep_hva;
+
+#define copy_el2_to_el1(reg)						\
+	write_sysreg_s(read_sysreg_s(SYS_##reg##_EL1), SYS_##reg##_EL12)
+
+/* Yes, this is an ugly hack */
+#define __at(op, addr)	write_sysreg_s(addr, op)
+
+#define test_at_insn(op, expect_fault)							\
+do {											\
+	u64 par, fsc;									\
+	bool fault;									\
+											\
+	GUEST_SYNC(CLEAR_ACCESS_FLAG);							\
+											\
+	__at(OP_AT_##op, TEST_ADDR);							\
+	isb();										\
+	par = read_sysreg(par_el1);							\
+											\
+	fault = par & SYS_PAR_EL1_F;							\
+	fsc = FIELD_GET(SYS_PAR_EL1_FST, par);						\
+											\
+	__GUEST_ASSERT((expect_fault) == fault,						\
+		       "AT "#op": %sexpected fault (par: %lx)1",			\
+		       (expect_fault) ? "" : "un", par);				\
+	if ((expect_fault)) {								\
+		__GUEST_ASSERT(fsc == ESR_ELx_FSC_ACCESS_L(3),				\
+			       "AT "#op": expected access flag fault (par: %lx)",	\
+			       par);							\
+	} else {									\
+		GUEST_ASSERT_EQ(FIELD_GET(SYS_PAR_EL1_ATTR, par), MAIR_ATTR_NORMAL);	\
+		GUEST_ASSERT_EQ(FIELD_GET(SYS_PAR_EL1_SH, par), PTE_SHARED >> 8);	\
+		GUEST_ASSERT_EQ(par & SYS_PAR_EL1_PA, TEST_ADDR);			\
+		GUEST_SYNC(TEST_ACCESS_FLAG);						\
+	}										\
+} while (0)
+
+static void test_at(bool expect_fault)
+{
+	test_at_insn(S1E2R, expect_fault);
+	test_at_insn(S1E2W, expect_fault);
+
+	/* Reuse the stage-1 MMU context from EL2 at EL1 */
+	copy_el2_to_el1(SCTLR);
+	copy_el2_to_el1(MAIR);
+	copy_el2_to_el1(TCR);
+	copy_el2_to_el1(TTBR0);
+	copy_el2_to_el1(TTBR1);
+
+	/* Disable stage-2 translation and enter a non-host context */
+	write_sysreg(0, vtcr_el2);
+	write_sysreg(0, vttbr_el2);
+	sysreg_clear_set(hcr_el2, HCR_EL2_TGE | HCR_EL2_VM, 0);
+	isb();
+
+	test_at_insn(S1E1R, expect_fault);
+	test_at_insn(S1E1W, expect_fault);
+}
+
+static void guest_code(void)
+{
+	sysreg_clear_set(tcr_el1, TCR_HA, 0);
+	isb();
+
+	test_at(true);
+
+	if (!SYS_FIELD_GET(ID_AA64MMFR1_EL1, HAFDBS, read_sysreg(id_aa64mmfr1_el1)))
+		GUEST_DONE();
+
+	/*
+	 * KVM's software PTW makes the implementation choice that the AT
+	 * instruction sets the access flag.
+	 */
+	sysreg_clear_set(tcr_el1, 0, TCR_HA);
+	isb();
+	test_at(false);
+
+	GUEST_DONE();

tools/testing/selftests/kvm/arm64/at.c line 106

Quoted 10 lines
tools/testing/selftests/kvm/arm64/at.c new file +10 -0
+}
+
+static void handle_sync(struct kvm_vcpu *vcpu, struct ucall *uc)
+{
+	switch (uc->args[1]) {
+	case CLEAR_ACCESS_FLAG:
+		/*
+		 * Delete + reinstall the memslot to invalidate stage-2
+		 * mappings of the stage-1 page tables, forcing KVM to
+		 * use the 'slow' AT emulation path.

Once the guest continues executing instructions, we would get a stage 2 translation fault (on a S1 PTW) immediately and re-build the stage 2 mappings for the S1 page tables. Depends on how much we have re-built for S1 page tables, the "forcing KVM to use the 'slow' AT emulation path" may not always be true.

I tested it in a QEMU guest (with "-machine virt,virtualization=on \
-cpu max -accel tcg", so it's essentially a NV2 capable HW!) and it failed at the first test_at_insn(S1E2R, expect_fault):

[root@localhost arm64]# getconf PAGESIZE
65536
[root@localhost arm64]# ./at
Random seed: 0x6b8b4567
__vm_create: mode='PA-bits:40, VA-bits:48, 4K pages' type='0', pages='672' ==== Test Assertion Failure ====
arm64/at.c:58: (expect_fault) == fault
pid=824 tid=824 errno=4 - Interrupted system call

     1	0x0000000000402d8f: run_test at at.c:137
     2	0x00000000004020e3: main at at.c:162
     3	0x0000ffffb07daf3b: ?? ??:0
     4	0x0000ffffb07db007: ?? ??:0
     5	0x000000000040222f: _start at ??:?

AT S1E2R: expected fault (par: ff00000080000b80)1

because in my test, the S1 page tables used to translate TEST_ADDR are located at

	level 0: 0x180000
	level 1: 0x181000
	level 2: 0x187000
	level 3: 0x188000

All of them had already been re-mapped in stage 2 [*] (in the same 64KB physical page) before KVM started to emulate AT S1E2R. The AT_S1E1R (fast path) will successfully get the translation result and it's IMP DEF that whether an access flag fault will be triggered (QEMU chooses to not fault). Whatever, [*] is out of expect and result in the failure.

I have no idea that if we can actually force the slow emulation path.

Thanks,
Zenghui

Quoted 93 lines
tools/testing/selftests/kvm/arm64/at.c new file +60 -0
+		 *
+		 * This and clearing the access flag from host userspace
+		 * ensures that the access flag cannot be set speculatively
+		 * and is reliably cleared at the time of the AT instruction.
+		 */
+		clear_bit(__ffs(PTE_AF), ptep_hva);
+		vm_mem_region_reload(vcpu->vm, vcpu->vm->memslots[MEM_REGION_PT]);
+		break;
+	case TEST_ACCESS_FLAG:
+		TEST_ASSERT(test_bit(__ffs(PTE_AF), ptep_hva),
+			    "Expected access flag to be set (desc: %lu)", *ptep_hva);
+		break;
+	default:
+		TEST_FAIL("Unexpected SYNC arg: %lu", uc->args[1]);
+	}
+}
+
+static void run_test(struct kvm_vcpu *vcpu)
+{
+	struct ucall uc;
+
+	while (true) {
+		vcpu_run(vcpu);
+		switch (get_ucall(vcpu, &uc)) {
+		case UCALL_DONE:
+			return;
+		case UCALL_SYNC:
+			handle_sync(vcpu, &uc);
+			continue;
+		case UCALL_ABORT:
+			REPORT_GUEST_ASSERT(uc);
+			return;
+		default:
+			TEST_FAIL("Unexpeced ucall: %lu", uc.cmd);
+		}
+	}
+}
+
+int main(void)
+{
+	struct kvm_vcpu_init init;
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+
+	TEST_REQUIRE(kvm_check_cap(KVM_CAP_ARM_EL2));
+
+	vm = vm_create(1);
+
+	kvm_get_default_vcpu_target(vm, &init);
+	init.features[0] |= BIT(KVM_ARM_VCPU_HAS_EL2);
+	vcpu = aarch64_vcpu_add(vm, 0, &init, guest_code);
+	kvm_arch_vm_finalize_vcpus(vm);
+
+	virt_map(vm, TEST_ADDR, TEST_ADDR, 1);
+	ptep_hva = virt_get_pte_hva_at_level(vm, TEST_ADDR, 3);
+	run_test(vcpu);
+
+	kvm_vm_free(vm);
+	return 0;
+}
tools/testing/selftests/kvm/include/kvm_util.h +1 -0
index d3f3e455c031..41467dad9178 100644
@@ -715,6 +715,7 @@ static inline bool vm_arch_has_protected_memory(struct kvm_vm *vm)
 #endif
 
 void vm_mem_region_set_flags(struct kvm_vm *vm, uint32_t slot, uint32_t flags);
+void vm_mem_region_reload(struct kvm_vm *vm, uint32_t slot);
 void vm_mem_region_move(struct kvm_vm *vm, uint32_t slot, uint64_t new_gpa);
 void vm_mem_region_delete(struct kvm_vm *vm, uint32_t slot);
 struct kvm_vcpu *__vm_vcpu_add(struct kvm_vm *vm, uint32_t vcpu_id);
tools/testing/selftests/kvm/lib/kvm_util.c +10 -0
index 1a93d6361671..d6538bb17740 100644
@@ -1201,6 +1201,16 @@ void vm_mem_region_set_flags(struct kvm_vm *vm, uint32_t slot, uint32_t flags)
 		ret, errno, slot, flags);
 }
 
+void vm_mem_region_reload(struct kvm_vm *vm, uint32_t slot)
+{
+	struct userspace_mem_region *region = memslot2region(vm, slot);
+	struct kvm_userspace_memory_region2 tmp = region->region;
+
+	tmp.memory_size = 0;
+	vm_ioctl(vm, KVM_SET_USER_MEMORY_REGION2, &tmp);
+	vm_ioctl(vm, KVM_SET_USER_MEMORY_REGION2, &region->region);
+}
+
 /*
  * VM Memory Region Move
  *
#18 lore raw

Re: [PATCH v3 15/15] KVM: arm64: selftests: Add test for AT emulation

Quoted 45 lines · Zenghui Yu

Hi Oliver,

tools/testing/selftests/kvm/arm64/at.c line 106

Quoted 8 lines · Oliver Upton
tools/testing/selftests/kvm/arm64/at.c new file +8 -0
+static void handle_sync(struct kvm_vcpu *vcpu, struct ucall *uc)
+{
+	switch (uc->args[1]) {
+	case CLEAR_ACCESS_FLAG:
+		/*
+		 * Delete + reinstall the memslot to invalidate stage-2
+		 * mappings of the stage-1 page tables, forcing KVM to
+		 * use the 'slow' AT emulation path.

Once the guest continues executing instructions, we would get a stage 2 translation fault (on a S1 PTW) immediately and re-build the stage 2 mappings for the S1 page tables. Depends on how much we have re-built for S1 page tables, the "forcing KVM to use the 'slow' AT emulation path" may not always be true.

I tested it in a QEMU guest (with "-machine virt,virtualization=on \
-cpu max -accel tcg", so it's essentially a NV2 capable HW!) and it failed at the first test_at_insn(S1E2R, expect_fault):

[root@localhost arm64]# getconf PAGESIZE
65536
[root@localhost arm64]# ./at
Random seed: 0x6b8b4567
__vm_create: mode='PA-bits:40, VA-bits:48, 4K pages' type='0', pages='672' ==== Test Assertion Failure ====
arm64/at.c:58: (expect_fault) == fault
pid=824 tid=824 errno=4 - Interrupted system call

     1	0x0000000000402d8f: run_test at at.c:137
     2	0x00000000004020e3: main at at.c:162
     3	0x0000ffffb07daf3b: ?? ??:0
     4	0x0000ffffb07db007: ?? ??:0
     5	0x000000000040222f: _start at ??:?

AT S1E2R: expected fault (par: ff00000080000b80)1

because in my test, the S1 page tables used to translate TEST_ADDR are located at

	level 0: 0x180000
	level 1: 0x181000
	level 2: 0x187000
	level 3: 0x188000
Quoted 5 lines

All of them had already been re-mapped in stage 2 [*] (in the same 64KB physical page) before KVM started to emulate AT S1E2R. The AT_S1E1R (fast path) will successfully get the translation result and it's IMP DEF that whether an access flag fault will be triggered (QEMU chooses to not fault).

Fix myself up: an access flag fault should be taken in this case (with AF bit is 0 and HA is not enabled). I've just sent a fix to QEMU for that [*].

Given that we can't force the slow emulation path, we should still avoid testing the IMP DEF behavior (i.e., TEST_ACCESS_FLAG). I will send a patch to remove it.

[*] https://lore.kernel.org/r/[email protected]

Thanks,
Zenghui

maintainer Reviewed-by Tested-by on the v3 series
#19 lore raw
In reply to Oliver Upton
Quoted 10 lines · Oliver Upton

Fixes all the issues Marc found in v2:

v2: https://lore.kernel.org/kvmarm/[email protected]/

v2 -> v3:
- Don't use uninitialized XN value in kvm_pgtable_stage2_relax_perms()
- Pass vcpu pointer directly in the stage-2 PTW
- Ensure LL/SC swap implementation correctly handles compare failures
- Fix SRCU imbalance

Thanks for the quick respin, this looks pretty good now. I've taken it for a ride on the usual setup, and this time nothing caught fire.

FWIW:

Reviewed-by Marc Zyngier <[email protected]>
Tested-by Marc Zyngier <[email protected]>
	M.
Signature · 1 line
Jazz isn't dead. It just smells funny.
author maintainer Applied
#20 lore raw
In reply to Oliver Upton
Quoted 11 lines · Oliver Upton

Fixes all the issues Marc found in v2:

v2: https://lore.kernel.org/kvmarm/[email protected]/

v2 -> v3:
- Don't use uninitialized XN value in kvm_pgtable_stage2_relax_perms()
- Pass vcpu pointer directly in the stage-2 PTW
- Ensure LL/SC swap implementation correctly handles compare failures
- Fix SRCU imbalance

[...]

Applied to next, thanks!

[01/15] arm64: Detect FEAT_XNX

        https://git.kernel.org/kvmarm/kvmarm/c/dc31124379b6
[02/15] KVM: arm64: Add support for FEAT_XNX stage-2 permissions
        https://git.kernel.org/kvmarm/kvmarm/c/2608563b466b
[03/15] KVM: arm64: nv: Forward FEAT_XNX permissions to the shadow stage-2
        https://git.kernel.org/kvmarm/kvmarm/c/d93febe2ed2e
[04/15] KVM: arm64: Teach ptdump about FEAT_XNX permissions
        https://git.kernel.org/kvmarm/kvmarm/c/81e9fe5c6f25
[05/15] KVM: arm64: nv: Advertise support for FEAT_XNX
        https://git.kernel.org/kvmarm/kvmarm/c/5aef6974ec47
[06/15] KVM: arm64: Call helper for reading descriptors directly
        https://git.kernel.org/kvmarm/kvmarm/c/e218981560f6
[07/15] KVM: arm64: nv: Stop passing vCPU through void ptr in S2 PTW
        https://git.kernel.org/kvmarm/kvmarm/c/c2c5d3b248e3
[08/15] KVM: arm64: Handle endianness in read helper for emulated PTW
        https://git.kernel.org/kvmarm/kvmarm/c/a1b8922186f8
[09/15] KVM: arm64: nv: Use pgtable definitions in stage-2 walk
        https://git.kernel.org/kvmarm/kvmarm/c/0a610d2056c2
[10/15] KVM: arm64: Add helper for swapping guest descriptor
        https://git.kernel.org/kvmarm/kvmarm/c/9020b38e23db
[11/15] KVM: arm64: Propagate PTW errors up to AT emulation
        https://git.kernel.org/kvmarm/kvmarm/c/15e996ef9dc5
[12/15] KVM: arm64: Implement HW access flag management in stage-1 SW PTW
        https://git.kernel.org/kvmarm/kvmarm/c/c59ca4b5b0c3
[13/15] KVM: arm64: nv: Implement HW access flag management in stage-2 SW PTW
        https://git.kernel.org/kvmarm/kvmarm/c/39db933ba67f
[14/15] KVM: arm64: nv: Expose hardware access flag management to NV guests
        https://git.kernel.org/kvmarm/kvmarm/c/6774884fabf0
[15/15] KVM: arm64: selftests: Add test for AT emulation
        https://git.kernel.org/kvmarm/kvmarm/c/662d0d6cccba

--
Best,
Oliver