KVM: selftests: add powerpc support

Patch v5 -> v7:
================
(Suggestions from Sean):
1. Added kvm_is_ppc_pseries_lpar() helper in kvm_util.h to avoid sprinkling multiple #ifdef __powerpc64__ checks.
2. Enable all tests via TEST_GEN_PROGS_COMMON for powerpc
3. Enable hardware_disable_test and skip when running on powerpc pseries LPAR (nested).
4. Re-arranged the fixes in appropriate order.

This is based off v2 of Sean PPC pre-enablement series [1].
[1]: https://lore.kernel.org/kvm/[email protected]/

Patch v4 -> v5:
==============
- Fixed the issues reported by Anushree while running on kvm on powervm (nested).
- Fixed memslot_perf_test on powerpc.

PATCH v3 -> PATCH v4:
==================
- Rebased on top of Sean's new page allocator patch series v2 [1] This allowed to drop and patch-2 since it will now come from Sean's v2.
- Addressed review comments from Sean - squashed patches 6-10 in patch-1 of this.
- Dropped patch-5 to print vcpu_id when KVM_CREATE_VCPU ioctl fails.
- Added additional patch-3 to this series to allow dirty_log_test to run on all big-endian machines.

[1]: https://lore.kernel.org/kvm/[email protected]/
[v3]: https://lore.kernel.org/linuxppc-dev/2d2c46be7f051c7f1c645afaec67f57067d37a81.1781093720.git.ritesh.list@gmail.com/

Testing updates:
================
I have done some fair amount of testing of the selftests both on real HW and using various combinations on qemu (powernv+LE (default), pseries+cap-nested-hv, pseries+cap-nested-papr, powernv+BE). There were few powerpc kvm kernel fixes which were identified as part of this work. I will send those patches separately as those anyway needs to go in separately.

<pasted snippet of cover letter from previous versions>
=======================================================

Hi All,

This series primarly adds KVM selftests support for powerpc (64-bit, BookS, radix MMU).

This patch series is originally Nick's work. I have mainly only rebased it on the latest upstream tree. Since the rebase required few changes to all the four patches (Patch 1-4), I have dropped the earlier Acked-by from Michael Ellerman.

Since the last series was posted three years ago [1], I am resetting the version to RFC. This rebase was done as part of a larger effort to improve the selftests infrastructure for Linux on PowerPC tree. Thanks to Harsh and Maddy for their help on this.

Nicholas Piggin (1):
KVM: PPC: selftests: add support for powerpc

Ritesh Harjani (IBM) (7):
KVM: selftests: Don't limit LE dirty-bitmap bitops to s390x
KVM: selftests: Move memslot_perf_test off the 256M ELF load address
KVM: selftests: Skip the max vCPU ID test when KVM_CAP_PPC_SMT is set
KVM: selftests: Add helper for checking if running on powerpc pseries LPAR
KVM: selftests: Skip kvm_create_max_vcpus on powerpc pseries LPAR
KVM: selftests: Only warn for idle-page checks on powerpc pseries LPAR
KVM: selftests: Skip hardware_disable_test on powerpc pseries LPAR

 MAINTAINERS                                   |   2 +
 tools/testing/selftests/kvm/Makefile          |   2 +-
 tools/testing/selftests/kvm/Makefile.kvm      |   9 +
 .../selftests/kvm/access_tracking_perf_test.c |  10 +
 tools/testing/selftests/kvm/dirty_log_test.c  |  12 +-
 .../selftests/kvm/hardware_disable_test.c     |   3 +
 .../testing/selftests/kvm/include/kvm_util.h  |  26 +
 .../selftests/kvm/include/powerpc/hcall.h     |  17 +
 .../kvm/include/powerpc/kvm_util_arch.h       |  22 +
 .../selftests/kvm/include/powerpc/ppc_asm.h   |  32 ++
 .../selftests/kvm/include/powerpc/processor.h |  39 ++
 .../selftests/kvm/include/powerpc/ucall.h     |  21 +
 .../selftests/kvm/kvm_create_max_vcpus.c      |  14 +-
 tools/testing/selftests/kvm/lib/guest_modes.c |  20 +-
 tools/testing/selftests/kvm/lib/kvm_util.c    |   8 +
 .../selftests/kvm/lib/powerpc/handlers.S      |  93 ++++
 .../testing/selftests/kvm/lib/powerpc/hcall.c |  45 ++
 .../selftests/kvm/lib/powerpc/processor.c     | 512 ++++++++++++++++++
 .../testing/selftests/kvm/lib/powerpc/ucall.c |  22 +
 .../testing/selftests/kvm/memslot_perf_test.c |   2 +-
 20 files changed, 898 insertions(+), 13 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/include/powerpc/hcall.h
 create mode 100644 tools/testing/selftests/kvm/include/powerpc/kvm_util_arch.h
 create mode 100644 tools/testing/selftests/kvm/include/powerpc/ppc_asm.h
 create mode 100644 tools/testing/selftests/kvm/include/powerpc/processor.h
 create mode 100644 tools/testing/selftests/kvm/include/powerpc/ucall.h
 create mode 100644 tools/testing/selftests/kvm/lib/powerpc/handlers.S
 create mode 100644 tools/testing/selftests/kvm/lib/powerpc/hcall.c
 create mode 100644 tools/testing/selftests/kvm/lib/powerpc/processor.c
 create mode 100644 tools/testing/selftests/kvm/lib/powerpc/ucall.c

--
2.39.5

dirty_log_test already has BITOP_LE_SWIZZLE helpers, but they were compiled in only for s390x, so every other big-endian host used native test_bit().

The test fails on big-endian powerpc because it treated dirty host pages as clean (because of the endianness issue)

w/o the fix the test fails with:
Clear page 1 value (1) >= iteration (1)

---
 tools/testing/selftests/kvm/dirty_log_test.c | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

index af5eb0334a74..8189eaac713c 100644
@@ -44,8 +44,11 @@
 #define TEST_MIN_WRITES_PER_ITERATION	0x100
 
 /* Dirty bitmaps are always little endian, so we need to swap on big endian */
-#if defined(__s390x__)
+#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
 # define BITOP_LE_SWIZZLE	((BITS_PER_LONG-1) & ~0x7)
+#else
+# define BITOP_LE_SWIZZLE	0
+#endif
 # define test_bit_le(nr, addr) \
 	test_bit((nr) ^ BITOP_LE_SWIZZLE, addr)
 # define __set_bit_le(nr, addr) \

@@ -56,13 +59,6 @@
 	__test_and_set_bit((nr) ^ BITOP_LE_SWIZZLE, addr)
 # define __test_and_clear_bit_le(nr, addr) \
 	__test_and_clear_bit((nr) ^ BITOP_LE_SWIZZLE, addr)
-#else
-# define test_bit_le			test_bit
-# define __set_bit_le			__set_bit
-# define __clear_bit_le			__clear_bit
-# define __test_and_set_bit_le		__test_and_set_bit
-# define __test_and_clear_bit_le	__test_and_clear_bit
-#endif
 
 #define TEST_DIRTY_RING_COUNT		65536
 

2.39.5

KVM selftests are built with -fno-PIE/-no-pie so the binary uses fixed load addresses.
On powerpc64, GNU ld default links at 0x10000000 (256M) which is also the MEM_GPA, so the identity map trips with the error:
PTE already present at level: 4 gva: 0x10000000 pte:0xc00000000002018f

Put the test slots at 512M instead which is past the ELF. e.g. failure w/o this fix:

./memslot_perf_test -s 514
Random seed: 0x423558ef
Testing map performance with 1 runs, 5 seconds each

==== Test Assertion Failure ====
lib/powerpc/processor.c:235: !pte
pid=371 tid=371 errno=2 - No such file or directory

     1  0x0000000010018293: virt_arch_pg_map at processor.c:235
     2  0x0000000010008b3f: virt_pg_map at kvm_util.h:1325
     3   (inlined by) virt_map at kvm_util.c:1571
     4  0x0000000010001cab: prepare_vm at memslot_perf_test.c:353
     5   (inlined by) test_execute at memslot_perf_test.c:816
     6   (inlined by) test_loop at memslot_perf_test.c:1062
     7   (inlined by) main at memslot_perf_test.c:1136
     8  0x000000001001d713: __libc_start_call_main at libc-start.o:?
     9  0x000000001001db47: __libc_start_main_impl at ??:?

PTE already present at level: 4 gva: 0x10000000 pte:0xc00000000002018f

---
 tools/testing/selftests/kvm/memslot_perf_test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 844ea9b7bb41..2913a4de4cc9 100644
@@ -30,7 +30,7 @@
 #define MEM_EXTRA_SIZE		SZ_64K
 
 #define MEM_SIZE		(SZ_512M + MEM_EXTRA_SIZE)
-#define MEM_GPA			SZ_256M
+#define MEM_GPA			SZ_512M
 #define MEM_AUX_GPA		MEM_GPA
 #define MEM_SYNC_GPA		MEM_AUX_GPA
 #define MEM_TEST_GPA		(MEM_AUX_GPA + MEM_EXTRA_SIZE)

2.39.5

kvm_create_max_vcpus creates vCPUs from 0..MAX_VCPUS-1, then again with ids at the top of KVM_CAP_MAX_VCPU_ID.

On PowerPC the vCPU id is not a flat index: it encodes SMT thread plus core. So on powerpc maximum permitted vCPU ID depends on the VM's SMT mode and the maximum reported by KVM_CAP_MAX_VCPU_ID exceeds a simple non-SMT VM's limit.

So, skip that second loop when KVM_CAP_PPC_SMT is non-zero. The cap is 0 on other arches, so they still run it. The 0..MAX_VCPUS pass is unchanged.

---
 tools/testing/selftests/kvm/kvm_create_max_vcpus.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

index c5310736ed06..1b95027d41ba 100644
@@ -58,7 +58,13 @@ int main(int argc, char *argv[])
 
 	test_vcpu_creation(0, kvm_max_vcpus);
 
-	if (kvm_max_vcpu_id > kvm_max_vcpus)
+	/*
+	 * Skip the vCPU ID test when running on PowerPC with SMT support, in
+	 * which case system topology is encoded into the vCPU ID and so a VM
+	 * can't use the full advertised vCPU ID range without crafting a valid
+	 * platform specific topology.
+	 */
+	if (kvm_max_vcpu_id > kvm_max_vcpus && !kvm_check_cap(KVM_CAP_PPC_SMT))
 		test_vcpu_creation(
 			kvm_max_vcpu_id - kvm_max_vcpus, kvm_max_vcpus);
 

--
2.39.5

From: Nicholas Piggin <[email protected]>

Implement KVM selftests support for powerpc (Book3S-64).

ucalls are implemented with an unsupported PAPR hcall number which will always cause KVM to exit to userspace.

Virtual memory is implemented for the radix MMU, and only a base page size is supported (both 4K and 64K).

Guest interrupts are taken in real-mode, so require a page allocated at gRA 0x0. Interrupt entry is complicated because gVA:gRA is not 1:1 mapped (like the kernel is), so the MMU can not just be switched on and off.

This patch series also changes the following types as done in other selftests. These changes are squashed in this main patch as discussed here [1]:
- Use u64 instead of uint64_t
- Use s64 instead of int64_t
- Use u32 instead of uint32_t
- Use u8 instead of uint8_t
- Replace u64 gpa, u64 gva|vaddr with gpa_t and gva_t
- Adopt to the new page table allocator APIs
- Start the guest at the entry point and not the descriptor in case of elfv1

[1]: https://lore.kernel.org/linuxppc-dev/[email protected]/
[Rebased to latest mainline tree, "cc" to clobber list, fix the
size calculation in kvm_arch_vm_post_create()]

---
 MAINTAINERS                                   |   2 +
 tools/testing/selftests/kvm/Makefile          |   2 +-
 tools/testing/selftests/kvm/Makefile.kvm      |   7 +
 .../testing/selftests/kvm/include/kvm_util.h  |   9 +
 .../selftests/kvm/include/powerpc/hcall.h     |  17 +
 .../kvm/include/powerpc/kvm_util_arch.h       |  22 +
 .../selftests/kvm/include/powerpc/ppc_asm.h   |  32 ++
 .../selftests/kvm/include/powerpc/processor.h |  39 ++
 .../selftests/kvm/include/powerpc/ucall.h     |  21 +
 tools/testing/selftests/kvm/lib/guest_modes.c |  20 +-
 tools/testing/selftests/kvm/lib/kvm_util.c    |   8 +
 .../selftests/kvm/lib/powerpc/handlers.S      |  93 ++++
 .../testing/selftests/kvm/lib/powerpc/hcall.c |  45 ++
 .../selftests/kvm/lib/powerpc/processor.c     | 512 ++++++++++++++++++
 .../testing/selftests/kvm/lib/powerpc/ucall.c |  22 +
 15 files changed, 848 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/include/powerpc/hcall.h
 create mode 100644 tools/testing/selftests/kvm/include/powerpc/kvm_util_arch.h
 create mode 100644 tools/testing/selftests/kvm/include/powerpc/ppc_asm.h
 create mode 100644 tools/testing/selftests/kvm/include/powerpc/processor.h
 create mode 100644 tools/testing/selftests/kvm/include/powerpc/ucall.h
 create mode 100644 tools/testing/selftests/kvm/lib/powerpc/handlers.S
 create mode 100644 tools/testing/selftests/kvm/lib/powerpc/hcall.c
 create mode 100644 tools/testing/selftests/kvm/lib/powerpc/processor.c
 create mode 100644 tools/testing/selftests/kvm/lib/powerpc/ucall.c

index fc320b373656..35df0c39e991 100644
@@ -14349,6 +14349,8 @@ F:	arch/powerpc/include/asm/kvm*
 F:	arch/powerpc/include/uapi/asm/kvm*
 F:	arch/powerpc/kernel/kvm*
 F:	arch/powerpc/kvm/
+F:	tools/testing/selftests/kvm/*/powerpc/
+F:	tools/testing/selftests/kvm/powerpc/
 
 KERNEL VIRTUAL MACHINE FOR RISC-V (KVM/riscv)
 M:	Anup Patel <[email protected]>

index f2b223072b62..03d91f00092f 100644
@@ -3,7 +3,7 @@ top_srcdir = ../../../..
 include $(top_srcdir)/scripts/subarch.include
 ARCH            ?= $(SUBARCH)
 
-ifeq ($(ARCH),$(filter $(ARCH),arm64 s390 riscv x86 x86_64 loongarch))
+ifeq ($(ARCH),$(filter $(ARCH),arm64 s390 riscv x86 x86_64 loongarch powerpc))
 # Top-level selftests allows ARCH=x86_64 :-(
 ifeq ($(ARCH),x86_64)
 	override ARCH := x86

index 96bab7002d39..4b7f6ab0ee54 100644
@@ -53,6 +53,11 @@ LIBKVM_loongarch += lib/loongarch/processor.c
 LIBKVM_loongarch += lib/loongarch/ucall.c
 LIBKVM_loongarch += lib/loongarch/exception.S
 
+LIBKVM_powerpc += lib/powerpc/handlers.S
+LIBKVM_powerpc += lib/powerpc/processor.c
+LIBKVM_powerpc += lib/powerpc/ucall.c
+LIBKVM_powerpc += lib/powerpc/hcall.c
+
 # Non-compiled test targets
 TEST_PROGS_x86 += x86/nx_huge_pages_test.sh
 

@@ -251,6 +256,8 @@ TEST_GEN_PROGS_loongarch += memslot_perf_test
 TEST_GEN_PROGS_loongarch += set_memory_region_test
 TEST_GEN_PROGS_loongarch += steal_time
 
+TEST_GEN_PROGS_powerpc = $(TEST_GEN_PROGS_COMMON)
+
 SPLIT_TESTS += arch_timer
 SPLIT_TESTS += get-reg-list
 

index 89b4fc2af751..9df16c8dc82a 100644
@@ -241,6 +241,9 @@ enum vm_guest_mode {
 	VM_MODE_P41V48_4K,
 	VM_MODE_P41V39_4K,
 
+	VM_MODE_P52V52_4K,	/* For powerpc64 */
+	VM_MODE_P52V52_64K,
+
 	NUM_VM_MODES,
 };
 

@@ -300,6 +303,12 @@ extern enum vm_guest_mode vm_mode_default;
 #define MIN_PAGE_SHIFT			12U
 #define ptes_per_page(page_size)	((page_size) / 8)
 
+#elif defined(__powerpc64__)
+
+#define VM_MODE_DEFAULT			vm_mode_default
+#define MIN_PAGE_SHIFT			12U
+#define ptes_per_page(page_size)	((page_size) / 8)
+
 #endif
 
 #define VM_SHAPE_DEFAULT	VM_SHAPE(VM_MODE_DEFAULT)

new file mode 100644
index 000000000000..21b2bf9180f1
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * powerpc hcall defines
+ */
+#ifndef SELFTEST_KVM_HCALL_H
+#define SELFTEST_KVM_HCALL_H
+
+#include <linux/compiler.h>
+
+/* Ucalls use unimplemented PAPR hcall 0 which exits KVM */
+#define H_UCALL	0
+
+s64 hcall0(u64 token);
+s64 hcall1(u64 token, u64 arg1);
+s64 hcall2(u64 token, u64 arg1, u64 arg2);
+
+#endif

new file mode 100644
index 000000000000..5d45c25cd299
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef SELFTEST_KVM_UTIL_ARCH_H
+#define SELFTEST_KVM_UTIL_ARCH_H
+
+#include <stdint.h>
+
+#include "kvm_util_types.h"
+
+struct kvm_mmu_arch {};
+
+/* Page table fragment cache for guest page tables < page size */
+struct vm_pt_frag_cache {
+	gpa_t page;
+	size_t page_nr_used;
+};
+
+struct kvm_vm_arch {
+	gpa_t prtb; /* process table */
+	struct vm_pt_frag_cache pt_frag_cache[2]; /* 256B and 4KB PT caches */
+};
+
+#endif  /* SELFTEST_KVM_UTIL_ARCH_H */

new file mode 100644
index 000000000000..b9df64659792
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * powerpc asm specific defines
+ */
+#ifndef SELFTEST_KVM_PPC_ASM_H
+#define SELFTEST_KVM_PPC_ASM_H
+
+#define STACK_FRAME_MIN_SIZE	112 /* Could be 32 on ELFv2 */
+#define STACK_REDZONE_SIZE	512
+
+#define INT_FRAME_SIZE		(STACK_FRAME_MIN_SIZE + STACK_REDZONE_SIZE)
+
+#define SPR_SRR0	0x01a
+#define SPR_SRR1	0x01b
+#define SPR_CFAR	0x01c
+
+#define MSR_SF		0x8000000000000000ULL
+#define MSR_HV		0x1000000000000000ULL
+#define MSR_VEC		0x0000000002000000ULL
+#define MSR_VSX		0x0000000000800000ULL
+#define MSR_EE		0x0000000000008000ULL
+#define MSR_PR		0x0000000000004000ULL
+#define MSR_FP		0x0000000000002000ULL
+#define MSR_ME		0x0000000000001000ULL
+#define MSR_IR		0x0000000000000020ULL
+#define MSR_DR		0x0000000000000010ULL
+#define MSR_RI		0x0000000000000002ULL
+#define MSR_LE		0x0000000000000001ULL
+
+#define LPCR_ILE	0x0000000002000000ULL
+
+#endif

new file mode 100644
index 000000000000..f0ee21a8e2d4
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * powerpc processor specific defines
+ */
+#ifndef SELFTEST_KVM_PROCESSOR_H
+#define SELFTEST_KVM_PROCESSOR_H
+
+#include <linux/compiler.h>
+#include "ppc_asm.h"
+#include "kvm_util_types.h"
+
+extern unsigned char __interrupts_start[];
+extern unsigned char __interrupts_end[];
+
+struct kvm_vm;
+struct kvm_vcpu;
+
+struct ex_regs {
+	u64	gprs[32];
+	u64	nia;
+	u64	msr;
+	u64	cfar;
+	u64	lr;
+	u64	ctr;
+	u64	xer;
+	u32	cr;
+	u32	trap;
+	gva_t	gva; /* gva of this struct */
+};
+
+void vm_install_exception_handler(struct kvm_vm *vm, int vector,
+			void (*handler)(struct ex_regs *));
+
+static inline void cpu_relax(void)
+{
+	asm volatile("" ::: "memory");
+}
+
+#endif

new file mode 100644
index 000000000000..e0dbe91e8848
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef SELFTEST_KVM_UCALL_H
+#define SELFTEST_KVM_UCALL_H
+
+#include "hcall.h"
+
+#define UCALL_EXIT_REASON	KVM_EXIT_PAPR_HCALL
+
+#define UCALL_R4_UCALL	0x5715 /* regular ucall, r5 contains ucall pointer */
+#define UCALL_R4_SIMPLE	0x0000 /* simple exit usable by asm with no ucall data */
+
+static inline void ucall_arch_init(struct kvm_vm *vm, gpa_t mmio_gpa)
+{
+}
+
+static inline void ucall_arch_do_ucall(gva_t uc)
+{
+	hcall2(H_UCALL, UCALL_R4_UCALL, (uintptr_t)(uc));
+}
+
+#endif

index 7a96c43b5704..439766fad693 100644
@@ -4,16 +4,20 @@
  */
 #include "guest_modes.h"
 
-#if defined(__aarch64__) || defined(__riscv)
+#if defined(__aarch64__) || defined(__riscv) || defined(__powerpc64__)
 #include "processor.h"
 enum vm_guest_mode vm_mode_default;
 #endif
 
+#if defined(__powerpc64__)
+#include <unistd.h>
+#endif
+
 struct guest_mode guest_modes[NUM_VM_MODES];
 
 void guest_modes_append_default(void)
 {
-#if !defined(__aarch64__) && !defined(__riscv)
+#if !defined(__aarch64__) && !defined(__riscv) && !defined(__powerpc64__)
 	guest_mode_append(VM_MODE_DEFAULT, true);
 #endif
 

@@ -108,6 +112,18 @@ void guest_modes_append_default(void)
 		TEST_ASSERT(vm_mode_default != NUM_VM_MODES, "No supported mode!");
 	}
 #endif
+#ifdef __powerpc64__
+	{
+		TEST_REQUIRE(kvm_has_cap(KVM_CAP_PPC_MMU_RADIX));
+		/* Radix guest EA and RA are 52-bit on POWER9 and POWER10 */
+		if (sysconf(_SC_PAGESIZE) == 4096)
+			vm_mode_default = VM_MODE_P52V52_4K;
+		else
+			vm_mode_default = VM_MODE_P52V52_64K;
+		guest_mode_append(VM_MODE_P52V52_4K, true);
+		guest_mode_append(VM_MODE_P52V52_64K, true);
+	}
+#endif
 }
 
 void for_each_guest_mode(void (*func)(enum vm_guest_mode, void *), void *arg)

index 57b653fbd0b1..2be79c240ebf 100644
@@ -227,6 +227,8 @@ const char *vm_guest_mode_string(u32 i)
 		[VM_MODE_P41V57_4K]	= "PA-bits:41,  VA-bits:57,  4K pages",
 		[VM_MODE_P41V48_4K]	= "PA-bits:41,  VA-bits:48,  4K pages",
 		[VM_MODE_P41V39_4K]	= "PA-bits:41,  VA-bits:39,  4K pages",
+		[VM_MODE_P52V52_4K]	= "PA-bits:52,  VA-bits:52,  4K pages",
+		[VM_MODE_P52V52_64K]	= "PA-bits:52,  VA-bits:52, 64K pages",
 	};
 	_Static_assert(sizeof(strings)/sizeof(char *) == NUM_VM_MODES,
 		       "Missing new mode strings?");

@@ -263,6 +265,8 @@ const struct vm_guest_mode_params vm_guest_mode_params[] = {
 	[VM_MODE_P41V57_4K]	= { 41, 57,  0x1000, 12 },
 	[VM_MODE_P41V48_4K]	= { 41, 48,  0x1000, 12 },
 	[VM_MODE_P41V39_4K]	= { 41, 39,  0x1000, 12 },
+	[VM_MODE_P52V52_4K]	= { 52, 52,  0x1000, 12 },
+	[VM_MODE_P52V52_64K]	= { 52, 52, 0x10000, 16 },
 };
 _Static_assert(sizeof(vm_guest_mode_params)/sizeof(struct vm_guest_mode_params) == NUM_VM_MODES,
 	       "Missing new mode params?");

@@ -383,6 +387,10 @@ struct kvm_vm *____vm_create(struct vm_shape shape)
 	case VM_MODE_P41V39_4K:
 		vm->mmu.pgtable_levels = 3;
 		break;
+	case VM_MODE_P52V52_4K:
+	case VM_MODE_P52V52_64K:
+		vm->mmu.pgtable_levels = 4;
+		break;
 	default:
 		TEST_FAIL("Unknown guest mode: 0x%x", vm->mode);
 	}

new file mode 100644
index 000000000000..3481a7a0b850
@@ -0,0 +1,22 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ucall support. A ucall is a "hypercall to host userspace".
+ */
+#include "kvm_util.h"
+#include "ucall_common.h"
+#include "hcall.h"
+
+void *ucall_arch_get_ucall(struct kvm_vcpu *vcpu)
+{
+	struct kvm_run *run = vcpu->run;
+
+	if (run->exit_reason == UCALL_EXIT_REASON &&
+	    run->papr_hcall.nr == H_UCALL) {
+		struct kvm_regs regs;
+
+		vcpu_regs_get(vcpu, &regs);
+		if (regs.gpr[4] == UCALL_R4_UCALL)
+			return (void *)regs.gpr[5];
+	}
+	return NULL;
+}

2.39.5

PowerPC's PAPR L0 does not guarantee H_GUEST_CREATE/H_GUEST_CREATE_VCPU hcalls to create upto MAX_VCPUS or hundreds of VMs, this is due to limited resouces at L0 for guest management space. This helper allows us to skip such tests which creates a lot of VMs/VCPUs, when we are running nested.

Note that there is no easy way to differentiate between running nested on PowerNV v/s kvm-on-powervm case. So we use a common helper to detect any nested case.

The helper is added to the common kvm_util header file to avoid sprinkling multiple #ifdef __powerpc64__ everywhere.

---
 tools/testing/selftests/kvm/include/kvm_util.h | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

index 9df16c8dc82a..258ae5879a5e 100644
@@ -14,6 +14,7 @@
 #include <linux/kvm.h>
 #include "linux/rbtree.h"
 #include <linux/types.h>
+#include <unistd.h>
 
 #include <asm/atomic.h>
 #include <asm/kvm.h>

@@ -1063,6 +1064,22 @@ const char *exit_reason_str(unsigned int exit_reason);
 
 bool kvm_arch_needs_naturally_aligned_page_tables(void);
 
+/*
+ * True on powerpc when this host is a pseries LPAR (nested case).
+ * PowerPC's PAPR L0 does not guarantee H_GUEST_CREATE / H_GUEST_CREATE_VCPU up
+ * to MAX_VCPUS or creating hundreds of VMs - due to limited resouces at L0 for
+ * guest management space. This helper allows us to skip such tests when we are
+ * running nested.
+ */
+static inline bool kvm_is_ppc_pseries_lpar(void)
+{
+#ifdef __powerpc64__
+	return !access("/proc/device-tree/rtas/ibm,hypertas-functions", F_OK);
+#else
+	return false;
+#endif
+}
+
 gpa_t ____vm_phy_pages_alloc(struct kvm_vm *vm, size_t nr_pages, gpa_t min_gpa,
 			     u32 memslot, bool protected, bool naturally_aligned);
 gpa_t __vm_phy_pages_alloc(struct kvm_vm *vm, size_t nr_pages,

2.39.5

On a PAPR LPAR, KVM_CREATE_VCPU is H_GUEST_CREATE_VCPU and KVM_CAP_MAX_VCPUS is only the vcpuId range but is not a guarantee from L0 i.e. because L0 Guest Management Space can be exhausted before that many vCPUs exist, so creating MAX_VCPUS is not a valid assert.

So, skip the test when powerpc is pseries LPAR (running nested i.e. ibm,hypertas-functions is present).

---
 tools/testing/selftests/kvm/kvm_create_max_vcpus.c | 6 ++++++
 1 file changed, 6 insertions(+)

index 1b95027d41ba..22279c638d06 100644
@@ -39,6 +39,12 @@ int main(int argc, char *argv[])
 	int kvm_max_vcpu_id = kvm_check_cap(KVM_CAP_MAX_VCPU_ID);
 	int kvm_max_vcpus = kvm_check_cap(KVM_CAP_MAX_VCPUS);
 
+	/*
+	 * PowerPC's PAPR L0 does not guarantee KVM_CAP_MAX_VCPUS when running
+	 * on pseries LPAR (nested kvm-on-powervm).
+	 */
+	TEST_REQUIRE(!kvm_is_ppc_pseries_lpar());
+
 	pr_info("KVM_CAP_MAX_VCPU_ID: %d\n", kvm_max_vcpu_id);
 	pr_info("KVM_CAP_MAX_VCPUS: %d\n", kvm_max_vcpus);
 

2.39.5

Same nested TLB/access bit issue like x86. PowerPC nested KVM (pSeries LPAR) can hide accessed/idle, since KVM (L1) doesn't explicitly flush the TLB for access bit update, so L0 keeps translations cached. That also means that the L2 accesses do not refault and L1 thinks that the pages for L2 memory are still idle.

Use kvm_is_ppc_pseries_lpar() check to detect when running nested on powerpc pseries LPAR and only warn on the idle-page count.

---
 tools/testing/selftests/kvm/Makefile.kvm               |  1 +
 .../testing/selftests/kvm/access_tracking_perf_test.c  | 10 ++++++++++
 2 files changed, 11 insertions(+)

index 4b7f6ab0ee54..692bbae44a65 100644
@@ -257,6 +257,7 @@ TEST_GEN_PROGS_loongarch += set_memory_region_test
 TEST_GEN_PROGS_loongarch += steal_time
 
 TEST_GEN_PROGS_powerpc = $(TEST_GEN_PROGS_COMMON)
+TEST_GEN_PROGS_powerpc += access_tracking_perf_test
 
 SPLIT_TESTS += arch_timer
 SPLIT_TESTS += get-reg-list

index 4415c94b2866..eb6939065e79 100644
@@ -471,6 +471,16 @@ static int access_tracking_unreliable(void)
 		return 1;
 	}
 #endif
+	/*
+	 * PowerPC: Nested KVM (pSeries LPAR) can hide accessed/idle, since KVM
+	 * doesn't explicitely flush the TLB for access bit update. L0 keeps
+	 * translations cached so the guest won't see the idle bit cleared.
+	 */
+	if (kvm_is_ppc_pseries_lpar()) {
+		puts("Skipping idle page count sanity check, because the test is run nested");
+		return 1;
+	}
+
 	/*
 	 * When NUMA balancing is enabled, guest memory will be unmapped to get
 	 * NUMA faults, dropping the Accessed bits.

2.39.5

The test forks 512 children, each of which does KVM_CREATE_VM and then SIGKILLs without waiting. On a PAPR LPAR that ioctl is H_GUEST_CREATE and unreaped guests may still occupy L0 Guest Management Space, which means create can fail with -ENOMEM error.
Hence, skip the test when we are running on powerpc pseries LPAR.

---
 tools/testing/selftests/kvm/Makefile.kvm            | 1 +
 tools/testing/selftests/kvm/hardware_disable_test.c | 3 +++
 2 files changed, 4 insertions(+)

index 692bbae44a65..38c3900cac68 100644
@@ -258,6 +258,7 @@ TEST_GEN_PROGS_loongarch += steal_time
 
 TEST_GEN_PROGS_powerpc = $(TEST_GEN_PROGS_COMMON)
 TEST_GEN_PROGS_powerpc += access_tracking_perf_test
+TEST_GEN_PROGS_powerpc += hardware_disable_test
 
 SPLIT_TESTS += arch_timer
 SPLIT_TESTS += get-reg-list

index 43a36ef3ead8..0104b7e840b5 100644
@@ -130,6 +130,9 @@ int main(int argc, char **argv)
 	int s, r, cpu, i;
 	pid_t pid;
 
+	/* PowerPC PAPR L0 cannot guarantee 512 overlapping H_GUEST_CREATE from fork+SIGKILL. */
+	TEST_REQUIRE(!kvm_is_ppc_pseries_lpar());
+
 	kvm_sched_getaffinity(0, sizeof(cpu_set_t), &allowed_cpu_set);
 
 	for (i = 0; i < NR_VCPUS && CPU_COUNT(&allowed_cpu_set); i++) {

2.39.5

Hi Ritesh,

The series LGTM. I also tested it on KVM on LPAR and KVM on PowerNV systems (both L1 and L2).

Thanks,
Gautam

Sean/Paolo, hope you take this via your tree.

I've pushed this to kvm-x86/ppc_selftests, and to kvm-x86/next. It should show up in tomorrow's build of linux-next. If all goes well, I'll send "official" thanks in a few days (I'll be offline for the next ~48 hours), and include ppc_selftests in my pull requests for 7.4.

Applied to kvm-x86 ppc_selftests. I swapped the order of 5 and 6 so that there wouldn't ever be a window where the kvm_create_max_vcpus test could fail on LPAR setups. I also massaged the shortlog+changelog for 7 and 8, to explicitly call state the tests are being enabled (saying only "skip on LPAR" suggests the tests were already enabled, which they were not).

[1/8] KVM: selftests: Don't limit LE dirty-bitmap bitops to s390x

      https://github.com/kvm-x86/linux/commit/ab1d40006720
[2/8] KVM: selftests: Move memslot_perf_test off the 256M ELF load address
      https://github.com/kvm-x86/linux/commit/f51639c0ee7a
[3/8] KVM: selftests: Skip the max vCPU ID test when KVM_CAP_PPC_SMT is set
      https://github.com/kvm-x86/linux/commit/2bc2c4a8105f
[4/8] KVM: selftests: Add helper for checking if running on powerpc pseries LPAR
      https://github.com/kvm-x86/linux/commit/5527310b4d19
[5/8] KVM: selftests: Skip kvm_create_max_vcpus on powerpc pseries LPAR
      https://github.com/kvm-x86/linux/commit/9de211597e81
[6/8] KVM: PPC: selftests: add support for powerpc
      https://github.com/kvm-x86/linux/commit/1b53b581d4e8
[7/8] KVM: selftests: Enable access tracking perf test for PPC (with LPAR clause)
      https://github.com/kvm-x86/linux/commit/e5983ef19a6a
[8/8] KVM: selftests: Enable hardware disable test PPC (without pSeries LPAR)
      https://github.com/kvm-x86/linux/commit/23140d6e54ac

--
https://github.com/kvm-x86/linux/tree/next