Skip to contentSkip to conversation

kvmppc: Paravirtualize KVM to support ultravisor

PATCH Superseded Claudio Carvalho 7 participants 28 messages Jun 6, 2019 – Jul 1, 2019

Subsystems (2)

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v3 cover kvmppc: Paravirtualize KVM to support ultravisor

POWER platforms that supports the Protected Execution Facility (PEF) introduce features that combine hardware facilities and firmware to enable secure virtual machines. That includes a new processor mode (ultravisor mode) and the ultravisor firmware.

In PEF enabled systems, the ultravisor firmware runs at a privilege level above the hypervisor and also takes control over some system resources. The hypervisor, though, can make system calls to access these resources. Such system calls, a.k.a. ucalls, are handled by the ultravisor firmware.

The processor allows part of the system memory to be configured as secure memory, and introduces a a new mode, called secure mode, where any software entity in that mode can access secure memory. The hypervisor doesn't (and can't) run in secure mode, but a secure guest and the ultravisor firmware do.

This patch set adds support for ultravisor calls and do some preparation for running secure guests.

---
Changelog:
---
v2->v3:
- Squashed patches:
"[PATCH v2 08/10] KVM: PPC: Ultravisor: Return to UV for hcalls from SVM" "[PATCH v2 09/10] KVM: PPC: Book3S HV: Fixed for running secure guests"
- Renamed patch from/to:
"[PATCH v2 08/10] KVM: PPC: Ultravisor: Return to UV for hcalls from SVM" "[PATCH v3 08/09] KVM: PPC: Ultravisor: Enter a secure guest
- Rebased
- Addressed comments from Paul Mackerras
- Dropped ultravisor checks made in power8 code
- Updated the commit message for:
"[PATCH v3 08/09] KVM: PPC: Ultravisor: Enter a secure guest"
- Addressed comments from Maddy
- Dropped imc-pmu.c changes
- Changed opal-imc.c to fail the probe when the ultravisor is enabled
- Fixed "ucall defined but not used" issue when CONFIG_PPC_UV not set

v1->v2:
- Addressed comments from Paul Mackerras:
- Write the pate in HV's table before doing that in UV's
- Renamed and better documented the ultravisor header files. Also added all possible return codes for each ucall
- Updated the commit message that introduces the MSR_S bit
- Moved ultravisor.c and ucall.S to arch/powerpc/kernel
- Changed ucall.S to not save CR
- Rebased
- Changed the patches order
- Updated several commit messages
- Added FW_FEATURE_ULTRAVISOR to enable use of firmware_has_feature()
- Renamed CONFIG_PPC_KVM_UV to CONFIG_PPC_UV and used it to ifdef the ucall handler and the code that populates the powerpc_firmware_features for ultravisor
- Exported the ucall symbol. KVM may be built as module.
- Restricted LDBAR access if the ultravisor firmware is available
- Dropped patches:
"[PATCH 06/13] KVM: PPC: Ultravisor: UV_RESTRICTED_SPR_WRITE ucall" "[PATCH 07/13] KVM: PPC: Ultravisor: UV_RESTRICTED_SPR_READ ucall" "[PATCH 08/13] KVM: PPC: Ultravisor: fix mtspr and mfspr"
- Squashed patches:
"[PATCH 09/13] KVM: PPC: Ultravisor: Return to UV for hcalls from SVM" "[PATCH 13/13] KVM: PPC: UV: Have fast_guest_return check secure_guest"

Anshuman Khandual (1):
KVM: PPC: Ultravisor: Add PPC_UV config option

Claudio Carvalho (2):
powerpc: Introduce FW_FEATURE_ULTRAVISOR
KVM: PPC: Ultravisor: Restrict LDBAR access

Michael Anderson (2):
KVM: PPC: Ultravisor: Use UV_WRITE_PATE ucall to register a PATE
KVM: PPC: Ultravisor: Check for MSR_S during hv_reset_msr

Ram Pai (2):
KVM: PPC: Ultravisor: Add generic ultravisor call handler
KVM: PPC: Ultravisor: Restrict flush of the partition tlb cache

Sukadev Bhattiprolu (2):
KVM: PPC: Ultravisor: Introduce the MSR_S bit
KVM: PPC: Ultravisor: Enter a secure guest

 arch/powerpc/Kconfig                      | 20 +++++++
 arch/powerpc/include/asm/firmware.h       |  5 +-
 arch/powerpc/include/asm/kvm_host.h       |  1 +
 arch/powerpc/include/asm/reg.h            |  3 ++
 arch/powerpc/include/asm/ultravisor-api.h | 24 +++++++++
 arch/powerpc/include/asm/ultravisor.h     | 49 +++++++++++++++++
 arch/powerpc/kernel/Makefile              |  1 +
 arch/powerpc/kernel/asm-offsets.c         |  1 +
 arch/powerpc/kernel/prom.c                |  6 +++
 arch/powerpc/kernel/ucall.S               | 31 +++++++++++
 arch/powerpc/kernel/ultravisor.c          | 28 ++++++++++
 arch/powerpc/kvm/book3s_64_mmu_hv.c       |  1 +
 arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 39 +++++++++++---
 arch/powerpc/mm/book3s64/hash_utils.c     |  3 +-
 arch/powerpc/mm/book3s64/pgtable.c        | 65 +++++++++++++++++------
 arch/powerpc/mm/book3s64/radix_pgtable.c  |  9 ++--
 arch/powerpc/platforms/powernv/idle.c     |  6 ++-
 arch/powerpc/platforms/powernv/opal-imc.c |  7 +++
 18 files changed, 269 insertions(+), 30 deletions(-)
 create mode 100644 arch/powerpc/include/asm/ultravisor-api.h
 create mode 100644 arch/powerpc/include/asm/ultravisor.h
 create mode 100644 arch/powerpc/kernel/ucall.S
 create mode 100644 arch/powerpc/kernel/ultravisor.c
Signature · 1 line
2.20.1
#1 lore raw

PATCH v3 1/9 KVM: PPC: Ultravisor: Add PPC_UV config option

From: Anshuman Khandual <[email protected]>

CONFIG_PPC_UV adds support for ultravisor.

[Update config help and commit message]
---
 arch/powerpc/Kconfig | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
arch/powerpc/Kconfig +20 -0
index 8c1c636308c8..276c1857c335 100644
@@ -439,6 +439,26 @@ config PPC_TRANSACTIONAL_MEM
        ---help---
          Support user-mode Transactional Memory on POWERPC.
 
+config PPC_UV
+	bool "Ultravisor support"
+	depends on KVM_BOOK3S_HV_POSSIBLE
+	select HMM_MIRROR
+	select HMM
+	select ZONE_DEVICE
+	select MIGRATE_VMA_HELPER
+	select DEV_PAGEMAP_OPS
+	select DEVICE_PRIVATE
+	select MEMORY_HOTPLUG
+	select MEMORY_HOTREMOVE
+	default n
+	help
+	  This option paravirtualizes the kernel to run in POWER platforms that
+	  supports the Protected Execution Facility (PEF). In such platforms,
+	  the ultravisor firmware runs at a privilege level above the
+	  hypervisor.
+
+	  If unsure, say "N".
+
 config LD_HEAD_STUB_CATCH
 	bool "Reserve 256 bytes to cope with linker stubs in HEAD text" if EXPERT
 	depends on PPC64
Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH v3 1/9] KVM: PPC: Ultravisor: Add PPC_UV config option

Quoted 34 lines · Claudio Carvalho

From: Anshuman Khandual <[email protected]>

CONFIG_PPC_UV adds support for ultravisor.

[Update config help and commit message]
---
 arch/powerpc/Kconfig | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
arch/powerpc/Kconfig +13 -0
index 8c1c636308c8..276c1857c335 100644
@@ -439,6 +439,26 @@ config PPC_TRANSACTIONAL_MEM
        ---help---
          Support user-mode Transactional Memory on POWERPC.
 
+config PPC_UV
+	bool "Ultravisor support"
+	depends on KVM_BOOK3S_HV_POSSIBLE
+	select HMM_MIRROR
+	select HMM
+	select ZONE_DEVICE
+	select MIGRATE_VMA_HELPER
+	select DEV_PAGEMAP_OPS
+	select DEVICE_PRIVATE
+	select MEMORY_HOTPLUG
+	select MEMORY_HOTREMOVE
+	default n
+	help

arch/powerpc/Kconfig config PPC_TRANSACTIONAL_MEM · line 464

Quoted 10 lines
arch/powerpc/Kconfig +7 -0
+	  This option paravirtualizes the kernel to run in POWER platforms that
+	  supports the Protected Execution Facility (PEF). In such platforms,
+	  the ultravisor firmware runs at a privilege level above the
+	  hypervisor.
+
+	  If unsure, say "N".
+
 config LD_HEAD_STUB_CATCH
 	bool "Reserve 256 bytes to cope with linker stubs in HEAD text" if EXPERT
 	depends on PPC64

Maybe this patch should be the last of the series, as it may cause some bisect trouble to have this option enabled while missing some of the patches.

#3 lore raw

Re: [PATCH v3 1/9] KVM: PPC: Ultravisor: Add PPC_UV config option

Quoted 41 lines · Leonardo Bras
Quoted 39 lines · Claudio Carvalho

From: Anshuman Khandual <[email protected]>

CONFIG_PPC_UV adds support for ultravisor.

[Update config help and commit message]
---
 arch/powerpc/Kconfig | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
arch/powerpc/Kconfig +18 -0
index 8c1c636308c8..276c1857c335 100644
@@ -439,6 +439,26 @@ config PPC_TRANSACTIONAL_MEM
        ---help---
          Support user-mode Transactional Memory on POWERPC.
 
+config PPC_UV
+	bool "Ultravisor support"
+	depends on KVM_BOOK3S_HV_POSSIBLE
+	select HMM_MIRROR
+	select HMM
+	select ZONE_DEVICE
+	select MIGRATE_VMA_HELPER
+	select DEV_PAGEMAP_OPS
+	select DEVICE_PRIVATE
+	select MEMORY_HOTPLUG
+	select MEMORY_HOTREMOVE
+	default n
+	help
+	  This option paravirtualizes the kernel to run in POWER platforms that
+	  supports the Protected Execution Facility (PEF). In such platforms,
+	  the ultravisor firmware runs at a privilege level above the
+	  hypervisor.
+
Quoted 8 lines

arch/powerpc/Kconfig config PPC_TRANSACTIONAL_MEM · line 464

Quoted 5 lines
arch/powerpc/Kconfig +2 -0
+	  If unsure, say "N".
+
 config LD_HEAD_STUB_CATCH
 	bool "Reserve 256 bytes to cope with linker stubs in HEAD text" if EXPERT
 	depends on PPC64

Maybe this patch should be the last of the series, as it may cause some bisect trouble to have this option enabled while missing some of the patches.

Thanks Leonardo. I changed that for the next version.

Claudio

#4 lore raw

PATCH v3 2/9 KVM: PPC: Ultravisor: Introduce the MSR_S bit

In reply to Claudio Carvalho

From: Sukadev Bhattiprolu <[email protected]>

The ultravisor processor mode is introduced in POWER platforms that supports the Protected Execution Facility (PEF). Ultravisor is higher privileged than hypervisor mode.

In PEF enabled platforms, the MSR_S bit is used to indicate if the thread is in secure state. With the MSR_S bit, the privilege state of the thread is now determined by MSR_S, MSR_HV and MSR_PR, as follows:

S HV PR
-----------------------

0   x   1   problem
1   0   1   problem
x   x   0   privileged
x   1   0   hypervisor
1   1   0   ultravisor
1   1   1   reserved

The hypervisor doesn't (and can't) run with the MSR_S bit set, but a secure guest and the ultravisor firmware do.

---
 arch/powerpc/include/asm/reg.h | 3 +++
 1 file changed, 3 insertions(+)
arch/powerpc/include/asm/reg.h +3 -0
index 10caa145f98b..39b4c0a519f5 100644
@@ -38,6 +38,7 @@
 #define MSR_TM_LG	32		/* Trans Mem Available */
 #define MSR_VEC_LG	25	        /* Enable AltiVec */
 #define MSR_VSX_LG	23		/* Enable VSX */
+#define MSR_S_LG	22		/* Secure VM bit */
 #define MSR_POW_LG	18		/* Enable Power Management */
 #define MSR_WE_LG	18		/* Wait State Enable */
 #define MSR_TGPR_LG	17		/* TLB Update registers in use */
@@ -71,11 +72,13 @@
 #define MSR_SF		__MASK(MSR_SF_LG)	/* Enable 64 bit mode */
 #define MSR_ISF		__MASK(MSR_ISF_LG)	/* Interrupt 64b mode valid on 630 */
 #define MSR_HV 		__MASK(MSR_HV_LG)	/* Hypervisor state */
+#define MSR_S		__MASK(MSR_S_LG)	/* Secure state */
 #else
 /* so tests for these bits fail on 32-bit */
 #define MSR_SF		0
 #define MSR_ISF		0
 #define MSR_HV		0
+#define MSR_S		0
 #endif
 
 /*
Signature · 1 line
2.20.1
#5 lore raw

PATCH v3 3/9 powerpc: Introduce FW_FEATURE_ULTRAVISOR

In reply to Claudio Carvalho

This feature tells if the ultravisor firmware is available to handle ucalls.

[Device node name to "ibm,ultravisor"]
---
 arch/powerpc/include/asm/firmware.h   |  5 +++--
 arch/powerpc/include/asm/ultravisor.h | 15 +++++++++++++++
 arch/powerpc/kernel/Makefile          |  1 +
 arch/powerpc/kernel/prom.c            |  6 ++++++
 arch/powerpc/kernel/ultravisor.c      | 24 ++++++++++++++++++++++++
 5 files changed, 49 insertions(+), 2 deletions(-)
 create mode 100644 arch/powerpc/include/asm/ultravisor.h
 create mode 100644 arch/powerpc/kernel/ultravisor.c
arch/powerpc/include/asm/firmware.h +3 -2
index 00bc42d95679..43b48c4d3ca9 100644
@@ -54,6 +54,7 @@
 #define FW_FEATURE_DRC_INFO	ASM_CONST(0x0000000800000000)
 #define FW_FEATURE_BLOCK_REMOVE ASM_CONST(0x0000001000000000)
 #define FW_FEATURE_PAPR_SCM 	ASM_CONST(0x0000002000000000)
+#define FW_FEATURE_ULTRAVISOR	ASM_CONST(0x0000004000000000)
 
 #ifndef __ASSEMBLY__
 
@@ -72,9 +73,9 @@ enum {
 		FW_FEATURE_TYPE1_AFFINITY | FW_FEATURE_PRRN |
 		FW_FEATURE_HPT_RESIZE | FW_FEATURE_DRMEM_V2 |
 		FW_FEATURE_DRC_INFO | FW_FEATURE_BLOCK_REMOVE |
-		FW_FEATURE_PAPR_SCM,
+		FW_FEATURE_PAPR_SCM | FW_FEATURE_ULTRAVISOR,
 	FW_FEATURE_PSERIES_ALWAYS = 0,
-	FW_FEATURE_POWERNV_POSSIBLE = FW_FEATURE_OPAL,
+	FW_FEATURE_POWERNV_POSSIBLE = FW_FEATURE_OPAL | FW_FEATURE_ULTRAVISOR,
 	FW_FEATURE_POWERNV_ALWAYS = 0,
 	FW_FEATURE_PS3_POSSIBLE = FW_FEATURE_LPAR | FW_FEATURE_PS3_LV1,
 	FW_FEATURE_PS3_ALWAYS = FW_FEATURE_LPAR | FW_FEATURE_PS3_LV1,
arch/powerpc/include/asm/ultravisor.h new file +15 -0
new file mode 100644
index 000000000000..e5009b0d84ea
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Ultravisor definitions
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#ifndef _ASM_POWERPC_ULTRAVISOR_H
+#define _ASM_POWERPC_ULTRAVISOR_H
+
+/* Internal functions */
+extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
+					 int depth, void *data);
+
+#endif	/* _ASM_POWERPC_ULTRAVISOR_H */
arch/powerpc/kernel/Makefile +1 -0
index 0ea6c4aa3a20..c8ca219e54bf 100644
@@ -154,6 +154,7 @@ endif
 
 obj-$(CONFIG_EPAPR_PARAVIRT)	+= epapr_paravirt.o epapr_hcalls.o
 obj-$(CONFIG_KVM_GUEST)		+= kvm.o kvm_emul.o
+obj-$(CONFIG_PPC_UV)		+= ultravisor.o
 
 # Disable GCOV, KCOV & sanitizers in odd or sensitive code
 GCOV_PROFILE_prom_init.o := n
arch/powerpc/kernel/prom.c +6 -0
index 4221527b082f..8a9a8a319959 100644
@@ -59,6 +59,7 @@
 #include <asm/firmware.h>
 #include <asm/dt_cpu_ftrs.h>
 #include <asm/drmem.h>
+#include <asm/ultravisor.h>
 
 #include <mm/mmu_decl.h>
 
@@ -713,6 +714,11 @@ void __init early_init_devtree(void *params)
 	of_scan_flat_dt(early_init_dt_scan_fw_dump, NULL);
 #endif
 
+#if defined(CONFIG_PPC_UV)
+	/* Scan tree for ultravisor feature */
+	of_scan_flat_dt(early_init_dt_scan_ultravisor, NULL);
+#endif
+
 	/* Retrieve various informations from the /chosen node of the
 	 * device-tree, including the platform type, initrd location and
 	 * size, TCE reserve, and more ...
arch/powerpc/kernel/ultravisor.c new file +24 -0
new file mode 100644
index 000000000000..dc6021f63c97
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Ultravisor high level interfaces
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#include <linux/init.h>
+#include <linux/printk.h>
+#include <linux/string.h>
+
+#include <asm/ultravisor.h>
+#include <asm/firmware.h>
+
+int __init early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
+					 int depth, void *data)
+{
+	if (depth != 1 || strcmp(uname, "ibm,ultravisor") != 0)
+		return 0;
+
+	powerpc_firmware_features |= FW_FEATURE_ULTRAVISOR;
+	pr_debug("Ultravisor detected!\n");
+	return 1;
+}
Signature · 1 line
2.20.1
#6 lore raw

Re: [PATCH v3 3/9] powerpc: Introduce FW_FEATURE_ULTRAVISOR

Quoted 2 lines · Claudio Carvalho

This feature tells if the ultravisor firmware is available to handle ucalls.

Everything in this patch that depends on CONFIG_PPC_UV should just depend on CONFIG_PPC_POWERNV instead. The reason is that every host kernel needs to be able to do the ultracall to set partition table entry 0, in case it ends up being run on a machine with an ultravisor. Otherwise we will have the situation where a host kernel may crash early in boot just because the machine it's booted on happens to have an ultravisor running. The crash will be a particularly nasty one because it will happen before we have probed the machine type and initialized the console; therefore it will just look like the machine hangs for no discernable reason.

We also need to think about how to provide a way for petitboot to know whether the kernel it is booting knows how to do a ucall to set its partition table entry. One suggestion would be to modify vmlinux.lds.S to add a new PT_NOTE entry in the program header of the binary with (say) a 64-bit doubleword which is a bitmap indicating capabilities of the binary. We would define the first bit as indicating that the kernel knows how to run under an ultravisor. When running under an ultravisor, petitboot could then look for the PT_NOTE and the ultravisor-capable bit in it, and if the PT_NOTE is not there or the bit is zero, put up a dialog warning the user that the kernel will probably crash early in boot, and asking for explicit confirmation that the user wants to proceed.

Paul.

#7 lore raw

Re: [PATCH v3 3/9] powerpc: Introduce FW_FEATURE_ULTRAVISOR

Quoted 14 lines · Paul Mackerras
Quoted 2 lines · Claudio Carvalho

This feature tells if the ultravisor firmware is available to handle ucalls.

Everything in this patch that depends on CONFIG_PPC_UV should just depend on CONFIG_PPC_POWERNV instead. The reason is that every host kernel needs to be able to do the ultracall to set partition table entry 0, in case it ends up being run on a machine with an ultravisor. Otherwise we will have the situation where a host kernel may crash early in boot just because the machine it's booted on happens to have an ultravisor running. The crash will be a particularly nasty one because it will happen before we have probed the machine type and initialized the console; therefore it will just look like the machine hangs for no discernable reason.

Quoted 12 lines

We also need to think about how to provide a way for petitboot to know whether the kernel it is booting knows how to do a ucall to set its partition table entry. One suggestion would be to modify vmlinux.lds.S to add a new PT_NOTE entry in the program header of the binary with (say) a 64-bit doubleword which is a bitmap indicating capabilities of the binary. We would define the first bit as indicating that the kernel knows how to run under an ultravisor. When running under an ultravisor, petitboot could then look for the PT_NOTE and the ultravisor-capable bit in it, and if the PT_NOTE is not there or the bit is zero, put up a dialog warning the user that the kernel will probably crash early in boot, and asking for explicit confirmation that the user wants to proceed.

I just posted a separated RFC patch for the ELF note.

Thanks, Claudio.

Quoted 3 lines

Paul.

#8 lore raw

PATCH v3 4/9 KVM: PPC: Ultravisor: Add generic ultravisor call handler

In reply to Claudio Carvalho

From: Ram Pai <[email protected]>

Add the ucall() function, which can be used to make ultravisor calls with varied number of in and out arguments. Ultravisor calls can be made from the host or guests.

This copies the implementation of plpar_hcall().

[Change ucall.S to not save CR, rename and move the headers, build
ucall.S if CONFIG_PPC_UV set, and add some comments in the code]

---
 arch/powerpc/include/asm/ultravisor-api.h | 20 +++++++++++++++
 arch/powerpc/include/asm/ultravisor.h     | 20 +++++++++++++++
 arch/powerpc/kernel/Makefile              |  2 +-
 arch/powerpc/kernel/ucall.S               | 31 +++++++++++++++++++++++
 arch/powerpc/kernel/ultravisor.c          |  4 +++
 5 files changed, 76 insertions(+), 1 deletion(-)
 create mode 100644 arch/powerpc/include/asm/ultravisor-api.h
 create mode 100644 arch/powerpc/kernel/ucall.S
arch/powerpc/include/asm/ultravisor-api.h new file +20 -0
new file mode 100644
index 000000000000..5f538f33c704
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Ultravisor calls.
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#ifndef _ASM_POWERPC_ULTRAVISOR_API_H
+#define _ASM_POWERPC_ULTRAVISOR_API_H
+
+#include <asm/hvcall.h>
+
+/* Return codes */
+#define U_NOT_AVAILABLE		H_NOT_AVAILABLE
+#define U_SUCCESS		H_SUCCESS
+#define U_FUNCTION		H_FUNCTION
+#define U_PARAMETER		H_PARAMETER
+
+#endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
+
arch/powerpc/include/asm/ultravisor.h +20 -0
index e5009b0d84ea..7500771a8ebd 100644
@@ -8,8 +8,28 @@
 #ifndef _ASM_POWERPC_ULTRAVISOR_H
 #define _ASM_POWERPC_ULTRAVISOR_H
 
+#include <asm/ultravisor-api.h>
+
+#if !defined(__ASSEMBLY__)
+
 /* Internal functions */
 extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
 					 int depth, void *data);
 
+/* API functions */
+#define UCALL_BUFSIZE 4
+/**
+ * ucall: Make a powerpc ultravisor call.
+ * @opcode: The ultravisor call to make.
+ * @retbuf: Buffer to store up to 4 return arguments in.
+ *
+ * This call supports up to 6 arguments and 4 return arguments. Use
+ * UCALL_BUFSIZE to size the return argument buffer.
+ */
+#if defined(CONFIG_PPC_UV)
+long ucall(unsigned long opcode, unsigned long *retbuf, ...);
+#endif
+
+#endif /* !__ASSEMBLY__ */
+
 #endif	/* _ASM_POWERPC_ULTRAVISOR_H */
arch/powerpc/kernel/Makefile +1 -1
index c8ca219e54bf..43ff4546e469 100644
@@ -154,7 +154,7 @@ endif
 
 obj-$(CONFIG_EPAPR_PARAVIRT)	+= epapr_paravirt.o epapr_hcalls.o
 obj-$(CONFIG_KVM_GUEST)		+= kvm.o kvm_emul.o
-obj-$(CONFIG_PPC_UV)		+= ultravisor.o
+obj-$(CONFIG_PPC_UV)		+= ultravisor.o ucall.o
 
 # Disable GCOV, KCOV & sanitizers in odd or sensitive code
 GCOV_PROFILE_prom_init.o := n
arch/powerpc/kernel/ucall.S new file +31 -0
new file mode 100644
index 000000000000..ecc88998a13b
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Generic code to perform an ultravisor call.
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#include <asm/ppc_asm.h>
+
+/*
+ * This function is based on the plpar_hcall()
+ */
+_GLOBAL_TOC(ucall)
+	mr	r0,r3
+	std     r4,STK_PARAM(R4)(r1)     /* Save ret buffer */
+	mr	r3,r5
+	mr	r4,r6
+	mr	r5,r7
+	mr	r6,r8
+	mr	r7,r9
+	mr	r8,r10
+
+	sc 2				/* invoke the ultravisor */
+
+	ld	r12,STK_PARAM(R4)(r1)
+	std	r4,  0(r12)
+	std	r5,  8(r12)
+	std	r6, 16(r12)
+	std	r7, 24(r12)
+
+	blr				/* return r3 = status */
arch/powerpc/kernel/ultravisor.c +4 -0
index dc6021f63c97..02ddf79a9522 100644
@@ -8,10 +8,14 @@
 #include <linux/init.h>
 #include <linux/printk.h>
 #include <linux/string.h>
+#include <linux/export.h>
 
 #include <asm/ultravisor.h>
 #include <asm/firmware.h>
 
+/* in ucall.S */
+EXPORT_SYMBOL_GPL(ucall);
+
 int __init early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
 					 int depth, void *data)
 {
Signature · 1 line
2.20.1
#9 lore raw

Re: [PATCH v3 4/9] KVM: PPC: Ultravisor: Add generic ultravisor call handler

Quoted 7 lines · Claudio Carvalho

From: Ram Pai <[email protected]>

Add the ucall() function, which can be used to make ultravisor calls with varied number of in and out arguments. Ultravisor calls can be made from the host or guests.

This copies the implementation of plpar_hcall().

Again, we will want all of this on every powernv-capable kernel, since they will all need to do UV_WRITE_PATE, even if they have no other support for the ultravisor.

Paul.

#10 lore raw

Re: [PATCH v3 4/9] KVM: PPC: Ultravisor: Add generic ultravisor call handler

In reply to Claudio Carvalho
Quoted 7 lines · Claudio Carvalho

From: Ram Pai <[email protected]>

Add the ucall() function, which can be used to make ultravisor calls with varied number of in and out arguments. Ultravisor calls can be made from the host or guests.

This copies the implementation of plpar_hcall().

One point which I missed when I looked at this patch previously is that the ABI that we're defining here is different from the hcall ABI in that we are putting the ucall number in r0, whereas hcalls have the hcall number in r3. That makes ucalls more like syscalls, which have the syscall number in r0. So that last sentence quoted above is somewhat misleading.

The thing we need to consider is that when SMFCTRL[E] = 0, a ucall instruction becomes a hcall (that is, sc 2 is executed as if it was sc 1). In that case, the first argument to the ucall will be interpreted as the hcall number. Mostly that will happen not to be a valid hcall number, but sometimes it might unavoidably be a valid but unintended hcall number.

I think that will make it difficult to get ucalls to fail gracefully in the case where SMF/PEF is disabled. It seems like the assignment of ucall numbers was made so that they wouldn't overlap with valid hcall numbers; presumably that was so that we could tell when an hcall was actually intended to be a ucall. However, using a different GPR to pass the ucall number defeats that.

I realize that there is ultravisor code in development that takes the ucall number in r0, and also that having the ucall number in r3 would possibly make life more difficult for the place where we call UV_RETURN in assembler code. Nevertheless, perhaps we should consider changing the ABI to be like the hcall ABI before everything gets set in concrete.

Paul.

#11 lore raw

Re: Re: [PATCH v3 4/9] KVM: PPC: Ultravisor: Add generic ultravisor call handler

Quoted 23 lines · Paul Mackerras
Quoted 7 lines · Claudio Carvalho

From: Ram Pai <[email protected]>

Add the ucall() function, which can be used to make ultravisor calls with varied number of in and out arguments. Ultravisor calls can be made from the host or guests.

This copies the implementation of plpar_hcall().

One point which I missed when I looked at this patch previously is that the ABI that we're defining here is different from the hcall ABI in that we are putting the ucall number in r0, whereas hcalls have the hcall number in r3. That makes ucalls more like syscalls, which have the syscall number in r0. So that last sentence quoted above is somewhat misleading.

The thing we need to consider is that when SMFCTRL[E] = 0, a ucall instruction becomes a hcall (that is, sc 2 is executed as if it was sc 1). In that case, the first argument to the ucall will be interpreted as the hcall number. Mostly that will happen not to be a valid hcall number, but sometimes it might unavoidably be a valid but unintended hcall number.

Quoted 6 lines

I think that will make it difficult to get ucalls to fail gracefully in the case where SMF/PEF is disabled. It seems like the assignment of ucall numbers was made so that they wouldn't overlap with valid hcall numbers; presumably that was so that we could tell when an hcall was actually intended to be a ucall. However, using a different GPR to pass the ucall number defeats that.

Right this is a valid point. Glad that you caught it, otherwise it would have become a difficult to fix it in the future.

Quoted 5 lines

I realize that there is ultravisor code in development that takes the ucall number in r0, and also that having the ucall number in r3 would possibly make life more difficult for the place where we call UV_RETURN in assembler code.

Its called from one place in the hypervisor, and the changes look simple.

- LOAD_REG_IMMEDIATE(r0, UV_RETURN)
+ LOAD_REG_IMMEDIATE(r3, UV_RETURN)

        ld      r7, VCPU_GPR(R7)(r4)
        ld      r6, VCPU_GPR(R6)(r4)
        ld      r4, VCPU_GPR(R4)(r4)

What am i missing?

Quoted 3 lines

Nevertheless, perhaps we should consider
changing the ABI to be like the hcall ABI before everything gets set in concrete.

yes.

Thanks Paul!
RP

#12 lore raw

Re: [PATCH v3 4/9] KVM: PPC: Ultravisor: Add generic ultravisor call handler

In reply to Paul Mackerras

Hi Paul,

Quoted 6 lines · Paul Mackerras

The thing we need to consider is that when SMFCTRL[E] = 0, a ucall instruction becomes a hcall (that is, sc 2 is executed as if it was sc 1). In that case, the first argument to the ucall will be interpreted as the hcall number. Mostly that will happen not to be a valid hcall number, but sometimes it might unavoidably be a valid but unintended hcall number.

Shouldn't a caller of the ultravisor *know* that it is talking to the ultravisor in the first place? And not to the hypervisor.

Segher

#13 lore raw

Re: Re: [PATCH v3 4/9] KVM: PPC: Ultravisor: Add generic ultravisor call handler

Quoted 10 lines · Segher Boessenkool

Hi Paul,

Quoted 6 lines · Paul Mackerras

The thing we need to consider is that when SMFCTRL[E] = 0, a ucall instruction becomes a hcall (that is, sc 2 is executed as if it was sc 1). In that case, the first argument to the ucall will be interpreted as the hcall number. Mostly that will happen not to be a valid hcall number, but sometimes it might unavoidably be a valid but unintended hcall number.

Quoted 2 lines

Shouldn't a caller of the ultravisor *know* that it is talking to the ultravisor in the first place? And not to the hypervisor.

It may or may not. But if it knows and still decides to make the ucall,
the hypervisor must gracefully handle it.

We can't control who makes a ucall. A normal process within the VM could make a ucall too. Or a normal process running on top of the hypervisor could make a ucall.

RP

#14 lore raw

PATCH v3 5/9 KVM: PPC: Ultravisor: Use UV_WRITE_PATE ucall to register a PATE

In reply to Claudio Carvalho

From: Michael Anderson <[email protected]>

When running under an ultravisor, the ultravisor controls the real partition table and has it in secure memory where the hypervisor can't access it, and therefore we (the HV) have to do a ucall whenever we want to update an entry.

The HV still keeps a copy of its view of the partition table in normal memory so that the nest MMU can access it.

Both partition tables will have PATE entries for HV and normal virtual machines.

[Write the pate in HV's table before doing that in UV's]
---
 arch/powerpc/include/asm/ultravisor-api.h |  5 +++-
 arch/powerpc/include/asm/ultravisor.h     | 14 ++++++++++
 arch/powerpc/mm/book3s64/hash_utils.c     |  3 +-
 arch/powerpc/mm/book3s64/pgtable.c        | 34 +++++++++++++++++++++--
 arch/powerpc/mm/book3s64/radix_pgtable.c  |  9 ++++--
 5 files changed, 57 insertions(+), 8 deletions(-)
arch/powerpc/include/asm/ultravisor-api.h +4 -1
index 5f538f33c704..24bfb4c1737e 100644
@@ -15,6 +15,9 @@
 #define U_SUCCESS		H_SUCCESS
 #define U_FUNCTION		H_FUNCTION
 #define U_PARAMETER		H_PARAMETER
+#define U_PERMISSION		H_PERMISSION
 
-#endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
+/* opcodes */
+#define UV_WRITE_PATE			0xF104
 
+#endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
arch/powerpc/include/asm/ultravisor.h +14 -0
index 7500771a8ebd..4ffec7a36acd 100644
@@ -12,6 +12,8 @@
 
 #if !defined(__ASSEMBLY__)
 
+#include <linux/types.h>
+
 /* Internal functions */
 extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
 					 int depth, void *data);
@@ -28,8 +30,20 @@ extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
  */
 #if defined(CONFIG_PPC_UV)
 long ucall(unsigned long opcode, unsigned long *retbuf, ...);
+#else
+static long ucall(unsigned long opcode, unsigned long *retbuf, ...)
+{
+	return U_NOT_AVAILABLE;
+}
 #endif
 
+static inline int uv_register_pate(u64 lpid, u64 dw0, u64 dw1)
+{
+	unsigned long retbuf[UCALL_BUFSIZE];
+
+	return ucall(UV_WRITE_PATE, retbuf, lpid, dw0, dw1);
+}
+
 #endif /* !__ASSEMBLY__ */
 
 #endif	/* _ASM_POWERPC_ULTRAVISOR_H */
arch/powerpc/mm/book3s64/hash_utils.c +2 -1
index 1ff451892d7f..220a4e133240 100644
@@ -1080,9 +1080,10 @@ void hash__early_init_mmu_secondary(void)
 
 		if (!cpu_has_feature(CPU_FTR_ARCH_300))
 			mtspr(SPRN_SDR1, _SDR1);
-		else
+		else if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
 			mtspr(SPRN_PTCR,
 			      __pa(partition_tb) | (PATB_SIZE_SHIFT - 12));
+
 	}
 	/* Initialize SLB */
 	slb_initialize();
arch/powerpc/mm/book3s64/pgtable.c +31 -3
index 16bda049187a..40a9fc8b139f 100644
@@ -16,6 +16,8 @@
 #include <asm/tlb.h>
 #include <asm/trace.h>
 #include <asm/powernv.h>
+#include <asm/firmware.h>
+#include <asm/ultravisor.h>
 
 #include <mm/mmu_decl.h>
 #include <trace/events/thp.h>
@@ -206,12 +208,25 @@ void __init mmu_partition_table_init(void)
 	 * 64 K size.
 	 */
 	ptcr = __pa(partition_tb) | (PATB_SIZE_SHIFT - 12);
-	mtspr(SPRN_PTCR, ptcr);
+	/*
+	 * If ultravisor is available, it is responsible for creating and
+	 * managing partition table
+	 */
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		mtspr(SPRN_PTCR, ptcr);
+
+	/*
+	 * Since nestMMU cannot access secure memory. Create
+	 * and manage our own partition table. This table
+	 * contains entries for nonsecure and hypervisor
+	 * partition.
+	 */
 	powernv_set_nmmu_ptcr(ptcr);
 }
 
-void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
-				   unsigned long dw1)
+static void __mmu_partition_table_set_entry(unsigned int lpid,
+					    unsigned long dw0,
+					    unsigned long dw1)
 {
 	unsigned long old = be64_to_cpu(partition_tb[lpid].patb0);
 
@@ -238,6 +253,19 @@ void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
 	/* do we need fixup here ?*/
 	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
 }
+
+void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
+				  unsigned long dw1)
+{
+	__mmu_partition_table_set_entry(lpid, dw0, dw1);
+
+	if (firmware_has_feature(FW_FEATURE_ULTRAVISOR)) {
+		uv_register_pate(lpid, dw0, dw1);
+		pr_info("PATE registered by ultravisor: dw0 = 0x%lx, dw1 = 0x%lx\n",
+			dw0, dw1);
+	}
+}
+
 EXPORT_SYMBOL_GPL(mmu_partition_table_set_entry);
 
 static pmd_t *get_pmd_from_cache(struct mm_struct *mm)
arch/powerpc/mm/book3s64/radix_pgtable.c +6 -3
index c929d31f1043..670cb562d213 100644
@@ -655,8 +655,10 @@ void radix__early_init_mmu_secondary(void)
 		lpcr = mfspr(SPRN_LPCR);
 		mtspr(SPRN_LPCR, lpcr | LPCR_UPRT | LPCR_HR);
 
-		mtspr(SPRN_PTCR,
-		      __pa(partition_tb) | (PATB_SIZE_SHIFT - 12));
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			mtspr(SPRN_PTCR, __pa(partition_tb) |
+			      (PATB_SIZE_SHIFT - 12));
+
 		radix_init_amor();
 	}
 
@@ -672,7 +674,8 @@ void radix__mmu_cleanup_all(void)
 	if (!firmware_has_feature(FW_FEATURE_LPAR)) {
 		lpcr = mfspr(SPRN_LPCR);
 		mtspr(SPRN_LPCR, lpcr & ~LPCR_UPRT);
-		mtspr(SPRN_PTCR, 0);
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			mtspr(SPRN_PTCR, 0);
 		powernv_set_nmmu_ptcr(0);
 		radix__flush_tlb_all();
 	}
Signature · 1 line
2.20.1
#15 lore raw

Re: [PATCH v3 5/9] KVM: PPC: Ultravisor: Use UV_WRITE_PATE ucall to register a PATE

Quoted 7 lines · Claudio Carvalho

From: Michael Anderson <[email protected]>

When running under an ultravisor, the ultravisor controls the real partition table and has it in secure memory where the hypervisor can't access it, and therefore we (the HV) have to do a ucall whenever we want to update an entry.

Quoted 5 lines

The HV still keeps a copy of its view of the partition table in normal memory so that the nest MMU can access it.

Both partition tables will have PATE entries for HV and normal virtual machines.

As discussed before, all of this should depend only on CONFIG_PPC_POWERNV.

Paul.

#16 lore raw

PATCH v3 6/9 KVM: PPC: Ultravisor: Restrict flush of the partition tlb cache

In reply to Claudio Carvalho

From: Ram Pai <[email protected]>

Ultravisor is responsible for flushing the tlb cache, since it manages the PATE entries. Hence skip tlb flush, if the ultravisor firmware is available.

---
 arch/powerpc/mm/book3s64/pgtable.c | 33 +++++++++++++++++-------------
 1 file changed, 19 insertions(+), 14 deletions(-)
arch/powerpc/mm/book3s64/pgtable.c +19 -14
index 40a9fc8b139f..1eeb5fe87023 100644
@@ -224,6 +224,23 @@ void __init mmu_partition_table_init(void)
 	powernv_set_nmmu_ptcr(ptcr);
 }
 
+static void flush_partition(unsigned int lpid, unsigned long dw0)
+{
+	if (dw0 & PATB_HR) {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 1, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
+	} else {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 0) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
+	}
+	/* do we need fixup here ?*/
+	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
+}
+
 static void __mmu_partition_table_set_entry(unsigned int lpid,
 					    unsigned long dw0,
 					    unsigned long dw1)
@@ -238,20 +255,8 @@ static void __mmu_partition_table_set_entry(unsigned int lpid,
 	 * The type of flush (hash or radix) depends on what the previous
 	 * use of this partition ID was, not the new use.
 	 */
-	asm volatile("ptesync" : : : "memory");
-	if (old & PATB_HR) {
-		asm volatile(PPC_TLBIE_5(%0,%1,2,0,1) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
-	} else {
-		asm volatile(PPC_TLBIE_5(%0,%1,2,0,0) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
-	}
-	/* do we need fixup here ?*/
-	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		flush_partition(lpid, old);
 }
 
 void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
Signature · 1 line
2.20.1
#17 lore raw

Re: [PATCH v3 6/9] KVM: PPC: Ultravisor: Restrict flush of the partition tlb cache

Quoted 27 lines · Claudio Carvalho

From: Ram Pai <[email protected]>

Ultravisor is responsible for flushing the tlb cache, since it manages the PATE entries. Hence skip tlb flush, if the ultravisor firmware is available.

---
 arch/powerpc/mm/book3s64/pgtable.c | 33 +++++++++++++++++-------------
 1 file changed, 19 insertions(+), 14 deletions(-)
arch/powerpc/mm/book3s64/pgtable.c +7 -0
index 40a9fc8b139f..1eeb5fe87023 100644
@@ -224,6 +224,23 @@ void __init mmu_partition_table_init(void)
 	powernv_set_nmmu_ptcr(ptcr);
 }
 
+static void flush_partition(unsigned int lpid, unsigned long dw0)
+{
+	if (dw0 & PATB_HR) {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 1, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));

arch/powerpc/mm/book3s64/pgtable.c void __init mmu_partition_table_init(void) · line 243

Quoted 10 lines
arch/powerpc/mm/book3s64/pgtable.c +10 -0
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
+	} else {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 0) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
+	}
+	/* do we need fixup here ?*/
+	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
+}
+

checkpatch.pl seems to complain:

ERROR: need consistent spacing around '%' (ctx:WxV) #125: FILE: arch/powerpc/mm/book3s64/pgtable.c:230:
+ asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 1) : :

                                             ^

ERROR: need consistent spacing around '%' (ctx:WxV) #127: FILE: arch/powerpc/mm/book3s64/pgtable.c:232:
+ asm volatile(PPC_TLBIE_5(%0, %1, 2, 1, 1) : :

                                             ^

ERROR: need consistent spacing around '%' (ctx:WxV) #131: FILE: arch/powerpc/mm/book3s64/pgtable.c:236:
+ asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 0) : :

                                             ^
Quoted 28 lines
arch/powerpc/mm/book3s64/pgtable.c +2 -15
 static void __mmu_partition_table_set_entry(unsigned int lpid,
 					    unsigned long dw0,
 					    unsigned long dw1)
@@ -238,20 +255,8 @@ static void __mmu_partition_table_set_entry(unsigned int lpid,
 	 * The type of flush (hash or radix) depends on what the previous
 	 * use of this partition ID was, not the new use.
 	 */
-	asm volatile("ptesync" : : : "memory");
-	if (old & PATB_HR) {
-		asm volatile(PPC_TLBIE_5(%0,%1,2,0,1) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
-	} else {
-		asm volatile(PPC_TLBIE_5(%0,%1,2,0,0) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
-	}
-	/* do we need fixup here ?*/
-	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		flush_partition(lpid, old);
 }
 
 void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
--

2.20.1

#18 lore raw

Re: [PATCH v3 6/9] KVM: PPC: Ultravisor: Restrict flush of the partition tlb cache

Quoted 50 lines · Murilo Opsfelder Araújo
Quoted 27 lines · Claudio Carvalho

From: Ram Pai <[email protected]>

Ultravisor is responsible for flushing the tlb cache, since it manages the PATE entries. Hence skip tlb flush, if the ultravisor firmware is available.

---
 arch/powerpc/mm/book3s64/pgtable.c | 33 +++++++++++++++++-------------
 1 file changed, 19 insertions(+), 14 deletions(-)
arch/powerpc/mm/book3s64/pgtable.c +7 -0
index 40a9fc8b139f..1eeb5fe87023 100644
@@ -224,6 +224,23 @@ void __init mmu_partition_table_init(void)
 	powernv_set_nmmu_ptcr(ptcr);
 }
 
+static void flush_partition(unsigned int lpid, unsigned long dw0)
+{
+	if (dw0 & PATB_HR) {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 1, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));

arch/powerpc/mm/book3s64/pgtable.c void __init mmu_partition_table_init(void) · line 243

Quoted 10 lines
arch/powerpc/mm/book3s64/pgtable.c +10 -0
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
+	} else {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 0) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
+	}
+	/* do we need fixup here ?*/
+	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
+}
+

checkpatch.pl seems to complain:

ERROR: need consistent spacing around '%' (ctx:WxV) #125: FILE: arch/powerpc/mm/book3s64/pgtable.c:230:
+ asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 1) : :

                                             ^

ERROR: need consistent spacing around '%' (ctx:WxV) #127: FILE: arch/powerpc/mm/book3s64/pgtable.c:232:
+ asm volatile(PPC_TLBIE_5(%0, %1, 2, 1, 1) : :

Quoted 6 lines
                                             ^

ERROR: need consistent spacing around '%' (ctx:WxV) #131: FILE: arch/powerpc/mm/book3s64/pgtable.c:236:
+ asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 0) : :

                                             ^

Then clearly checkpatch.pl has a bug.

Paul.

#19 lore raw

PATCH v3 7/9 KVM: PPC: Ultravisor: Restrict LDBAR access

In reply to Claudio Carvalho

When the ultravisor firmware is available, it takes control over the LDBAR register. In this case, thread-imc updates and save/restore operations on the LDBAR register are handled by ultravisor.

---
 arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 2 ++
 arch/powerpc/platforms/powernv/idle.c     | 6 ++++--
 arch/powerpc/platforms/powernv/opal-imc.c | 7 +++++++
 3 files changed, 13 insertions(+), 2 deletions(-)
arch/powerpc/kvm/book3s_hv_rmhandlers.S +2 -0
index f9b2620fbecd..cffb365d9d02 100644
@@ -375,8 +375,10 @@ BEGIN_FTR_SECTION
 	mtspr	SPRN_RPR, r0
 	ld	r0, KVM_SPLIT_PMMAR(r6)
 	mtspr	SPRN_PMMAR, r0
+BEGIN_FW_FTR_SECTION_NESTED(70)
 	ld	r0, KVM_SPLIT_LDBAR(r6)
 	mtspr	SPRN_LDBAR, r0
+END_FW_FTR_SECTION_NESTED(FW_FEATURE_ULTRAVISOR, 0, 70)
 	isync
 FTR_SECTION_ELSE
 	/* On P9 we use the split_info for coordinating LPCR changes */
arch/powerpc/platforms/powernv/idle.c +4 -2
index c9133f7908ca..fd62435e3267 100644
@@ -679,7 +679,8 @@ static unsigned long power9_idle_stop(unsigned long psscr, bool mmu_on)
 		sprs.ptcr	= mfspr(SPRN_PTCR);
 		sprs.rpr	= mfspr(SPRN_RPR);
 		sprs.tscr	= mfspr(SPRN_TSCR);
-		sprs.ldbar	= mfspr(SPRN_LDBAR);
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			sprs.ldbar	= mfspr(SPRN_LDBAR);
 
 		sprs_saved = true;
 
@@ -762,7 +763,8 @@ static unsigned long power9_idle_stop(unsigned long psscr, bool mmu_on)
 	mtspr(SPRN_PTCR,	sprs.ptcr);
 	mtspr(SPRN_RPR,		sprs.rpr);
 	mtspr(SPRN_TSCR,	sprs.tscr);
-	mtspr(SPRN_LDBAR,	sprs.ldbar);
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		mtspr(SPRN_LDBAR,	sprs.ldbar);
 
 	if (pls >= pnv_first_tb_loss_level) {
 		/* TB loss */
arch/powerpc/platforms/powernv/opal-imc.c +7 -0
index 1b6932890a73..e9b641d313fb 100644
@@ -254,6 +254,13 @@ static int opal_imc_counters_probe(struct platform_device *pdev)
 	bool core_imc_reg = false, thread_imc_reg = false;
 	u32 type;
 
+	/*
+	 * When the Ultravisor is enabled, it is responsible for thread-imc
+	 * updates
+	 */
+	if (firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		return -EACCES;
+
 	/*
 	 * Check whether this is kdump kernel. If yes, force the engines to
 	 * stop and return.
Signature · 1 line
2.20.1
Acked-by on v3 7/9
#20 lore raw

Re: [PATCH v3 7/9] KVM: PPC: Ultravisor: Restrict LDBAR access

Quoted 56 lines · Claudio Carvalho

When the ultravisor firmware is available, it takes control over the LDBAR register. In this case, thread-imc updates and save/restore operations on the LDBAR register are handled by ultravisor.

---
  arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 2 ++
  arch/powerpc/platforms/powernv/idle.c     | 6 ++++--
  arch/powerpc/platforms/powernv/opal-imc.c | 7 +++++++
  3 files changed, 13 insertions(+), 2 deletions(-)
arch/powerpc/kvm/book3s_hv_rmhandlers.S +2 -0
index f9b2620fbecd..cffb365d9d02 100644
@@ -375,8 +375,10 @@ BEGIN_FTR_SECTION
  	mtspr	SPRN_RPR, r0
  	ld	r0, KVM_SPLIT_PMMAR(r6)
  	mtspr	SPRN_PMMAR, r0
+BEGIN_FW_FTR_SECTION_NESTED(70)
  	ld	r0, KVM_SPLIT_LDBAR(r6)
  	mtspr	SPRN_LDBAR, r0
+END_FW_FTR_SECTION_NESTED(FW_FEATURE_ULTRAVISOR, 0, 70)
  	isync
  FTR_SECTION_ELSE
  	/* On P9 we use the split_info for coordinating LPCR changes */
arch/powerpc/platforms/powernv/idle.c +4 -2
index c9133f7908ca..fd62435e3267 100644
@@ -679,7 +679,8 @@ static unsigned long power9_idle_stop(unsigned long psscr, bool mmu_on)
  		sprs.ptcr	= mfspr(SPRN_PTCR);
  		sprs.rpr	= mfspr(SPRN_RPR);
  		sprs.tscr	= mfspr(SPRN_TSCR);
-		sprs.ldbar	= mfspr(SPRN_LDBAR);
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			sprs.ldbar	= mfspr(SPRN_LDBAR);
 
  		sprs_saved = true;
 
@@ -762,7 +763,8 @@ static unsigned long power9_idle_stop(unsigned long psscr, bool mmu_on)
  	mtspr(SPRN_PTCR,	sprs.ptcr);
  	mtspr(SPRN_RPR,		sprs.rpr);
  	mtspr(SPRN_TSCR,	sprs.tscr);
-	mtspr(SPRN_LDBAR,	sprs.ldbar);
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		mtspr(SPRN_LDBAR,	sprs.ldbar);
 
  	if (pls >= pnv_first_tb_loss_level) {
  		/* TB loss */
arch/powerpc/platforms/powernv/opal-imc.c +0 -0
index 1b6932890a73..e9b641d313fb 100644

arch/powerpc/platforms/powernv/opal-imc.c static int opal_imc_counters_probe(struct platform_device *pdev) · line 260

Quoted 7 lines
arch/powerpc/platforms/powernv/opal-imc.c +4 -0
@@ -254,6 +254,13 @@ static int opal_imc_counters_probe(struct platform_device *pdev)
  	bool core_imc_reg = false, thread_imc_reg = false;
  	u32 type;
 
+	/*
+	 * When the Ultravisor is enabled, it is responsible for thread-imc
+	 * updates
+	 */

Would prefer the comment to be "Disable IMC devices, when Ultravisor is enabled"
Rest looks good.

Quoted 6 lines
arch/powerpc/platforms/powernv/opal-imc.c +3 -0
+	if (firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		return -EACCES;
+
  	/*
  	 * Check whether this is kdump kernel. If yes, force the engines to
  	 * stop and return.
#21 lore raw

Re: [PATCH v3 7/9] KVM: PPC: Ultravisor: Restrict LDBAR access

Quoted 75 lines · Madhavan Srinivasan
Quoted 61 lines · Claudio Carvalho

When the ultravisor firmware is available, it takes control over the LDBAR register. In this case, thread-imc updates and save/restore operations on the LDBAR register are handled by ultravisor.

---
  arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 2 ++ arch/powerpc/platforms/powernv/idle.c     | 6 ++++-- arch/powerpc/platforms/powernv/opal-imc.c | 7 +++++++

  3 files changed, 13 insertions(+), 2 deletions(-)
arch/powerpc/kvm/book3s_hv_rmhandlers.S +0 -0
index f9b2620fbecd..cffb365d9d02 100644
@@ -375,8 +375,10 @@ BEGIN_FTR_SECTION

      mtspr    SPRN_RPR, r0
      ld    r0, KVM_SPLIT_PMMAR(r6)
      mtspr    SPRN_PMMAR, r0 +BEGIN_FW_FTR_SECTION_NESTED(70)
      ld    r0, KVM_SPLIT_LDBAR(r6)
      mtspr    SPRN_LDBAR, r0 +END_FW_FTR_SECTION_NESTED(FW_FEATURE_ULTRAVISOR, 0, 70) isync
  FTR_SECTION_ELSE
      /* On P9 we use the split_info for coordinating LPCR changes */

arch/powerpc/platforms/powernv/idle.c +4 -2
index c9133f7908ca..fd62435e3267 100644
@@ -679,7 +679,8 @@ static unsigned long power9_idle_stop(unsigned long
 psscr, bool mmu_on)
           sprs.ptcr    = mfspr(SPRN_PTCR);
           sprs.rpr    = mfspr(SPRN_RPR);
           sprs.tscr    = mfspr(SPRN_TSCR);
-        sprs.ldbar    = mfspr(SPRN_LDBAR);
+        if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+            sprs.ldbar    = mfspr(SPRN_LDBAR);
 
           sprs_saved = true;
@@ -762,7 +763,8 @@ static unsigned long power9_idle_stop(unsigned long
 psscr, bool mmu_on)
       mtspr(SPRN_PTCR,    sprs.ptcr);
       mtspr(SPRN_RPR,        sprs.rpr);
       mtspr(SPRN_TSCR,    sprs.tscr);
-    mtspr(SPRN_LDBAR,    sprs.ldbar);
+    if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+        mtspr(SPRN_LDBAR,    sprs.ldbar);
 
       if (pls >= pnv_first_tb_loss_level) {

          /* TB loss */

arch/powerpc/platforms/powernv/opal-imc.c +0 -0
index 1b6932890a73..e9b641d313fb 100644

arch/powerpc/platforms/powernv/opal-imc.c static int opal_imc_counters_probe(struct · line 261

Quoted 8 lines
arch/powerpc/platforms/powernv/opal-imc.c +4 -0
@@ -254,6 +254,13 @@ static int opal_imc_counters_probe(struct
 platform_device *pdev)
       bool core_imc_reg = false, thread_imc_reg = false;
       u32 type;
 
+    /*
+     * When the Ultravisor is enabled, it is responsible for thread-imc
+     * updates
+     */

Would prefer the comment to be "Disable IMC devices, when Ultravisor is enabled"
Rest looks good.

Quoted 1 line

Thanks Maddy. I applied that change to the next version.

Claudio

Quoted 7 lines
Quoted 7 lines
arch/powerpc/platforms/powernv/opal-imc.c +3 -0
+    if (firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+        return -EACCES;
+
       /*
        * Check whether this is kdump kernel. If yes, force the engines to

       * stop and return.

Acked-by on v3 7/9
#22 lore raw

Re: [PATCH v3 7/9] KVM: PPC: Ultravisor: Restrict LDBAR access

In reply to Claudio Carvalho
Quoted 6 lines · Claudio Carvalho

When the ultravisor firmware is available, it takes control over the LDBAR register. In this case, thread-imc updates and save/restore operations on the LDBAR register are handled by ultravisor.

Just a note on the signed-off-by: it's a bit weird to have Ram's signoff when he is neither the author nor the sender of the patch. The author is assumed to be Claudio; if that is not correct then the patch should have a From: line at the start telling us who the author is, and ideally that person should have a signoff line before Claudio's signoff as the sender of the patch.

Paul.

#23 lore raw

Re: Re: [PATCH v3 7/9] KVM: PPC: Ultravisor: Restrict LDBAR access

Quoted 8 lines · Paul Mackerras
Quoted 6 lines · Claudio Carvalho

When the ultravisor firmware is available, it takes control over the LDBAR register. In this case, thread-imc updates and save/restore operations on the LDBAR register are handled by ultravisor.

Quoted 8 lines

Just a note on the signed-off-by: it's a bit weird to have Ram's signoff when he is neither the author nor the sender of the patch. The author is assumed to be Claudio; if that is not correct then the patch should have a From: line at the start telling us who the author is, and ideally that person should have a signoff line before Claudio's signoff as the sender of the patch.

Ryan originally wrote this patch, which I than modified, before Claudio further modified it to its current form.

So I think the author should be Ryan.

RP

#24 lore raw

PATCH v3 8/9 KVM: PPC: Ultravisor: Enter a secure guest

In reply to Claudio Carvalho

From: Sukadev Bhattiprolu <[email protected]>

To enter a secure guest, we have to go through the ultravisor, therefore we do a ucall when we are entering a secure guest.

This change is needed for any sort of entry to the secure guest from the hypervisor, whether it is a return from an hcall, a return from a hypervisor interrupt, or the first time that a secure guest vCPU is run.

If we are returning from an hcall, the results are already in the appropriate registers (R3:12), except for R6,7, which need to be restored before doing the ucall (UV_RETURN).

Have fast_guest_return check the kvm_arch.secure_guest field so that a new CPU enters UV when started (in response to a RTAS start-cpu call).

Thanks to input from Paul Mackerras, Ram Pai and Mike Anderson.

[Pass SRR1 in r11 for UV_RETURN, fix kvmppc_msr_interrupt to preserve
the MSR_S bit]

[Fix UV_RETURN token number and arch.secure_guest check]
[Update commit message and ret_to_ultra comment]
---
 arch/powerpc/include/asm/kvm_host.h       |  1 +
 arch/powerpc/include/asm/ultravisor-api.h |  1 +
 arch/powerpc/kernel/asm-offsets.c         |  1 +
 arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 37 +++++++++++++++++++----
 4 files changed, 34 insertions(+), 6 deletions(-)
arch/powerpc/include/asm/kvm_host.h +1 -0
index 013c76a0a03e..184becb62ea4 100644
@@ -294,6 +294,7 @@ struct kvm_arch {
 	cpumask_t cpu_in_guest;
 	u8 radix;
 	u8 fwnmi_enabled;
+	u8 secure_guest;
 	bool threads_indep;
 	bool nested_enable;
 	pgd_t *pgtable;
arch/powerpc/include/asm/ultravisor-api.h +1 -0
index 24bfb4c1737e..15e6ce77a131 100644
@@ -19,5 +19,6 @@
 
 /* opcodes */
 #define UV_WRITE_PATE			0xF104
+#define UV_RETURN			0xF11C
 
 #endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
arch/powerpc/kernel/asm-offsets.c +1 -0
index 8e02444e9d3d..44742724513e 100644
@@ -508,6 +508,7 @@ int main(void)
 	OFFSET(KVM_VRMA_SLB_V, kvm, arch.vrma_slb_v);
 	OFFSET(KVM_RADIX, kvm, arch.radix);
 	OFFSET(KVM_FWNMI, kvm, arch.fwnmi_enabled);
+	OFFSET(KVM_SECURE_GUEST, kvm, arch.secure_guest);
 	OFFSET(VCPU_DSISR, kvm_vcpu, arch.shregs.dsisr);
 	OFFSET(VCPU_DAR, kvm_vcpu, arch.shregs.dar);
 	OFFSET(VCPU_VPA, kvm_vcpu, arch.vpa.pinned_addr);
arch/powerpc/kvm/book3s_hv_rmhandlers.S +31 -6
index cffb365d9d02..d719d730d31e 100644
@@ -36,6 +36,7 @@
 #include <asm/asm-compat.h>
 #include <asm/feature-fixups.h>
 #include <asm/cpuidle.h>
+#include <asm/ultravisor-api.h>
 
 /* Sign-extend HDEC if not on POWER9 */
 #define EXTEND_HDEC(reg)			\
@@ -1092,16 +1093,12 @@ BEGIN_FTR_SECTION
 END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
 
 	ld	r5, VCPU_LR(r4)
-	ld	r6, VCPU_CR(r4)
 	mtlr	r5
-	mtcr	r6
 
 	ld	r1, VCPU_GPR(R1)(r4)
 	ld	r2, VCPU_GPR(R2)(r4)
 	ld	r3, VCPU_GPR(R3)(r4)
 	ld	r5, VCPU_GPR(R5)(r4)
-	ld	r6, VCPU_GPR(R6)(r4)
-	ld	r7, VCPU_GPR(R7)(r4)
 	ld	r8, VCPU_GPR(R8)(r4)
 	ld	r9, VCPU_GPR(R9)(r4)
 	ld	r10, VCPU_GPR(R10)(r4)
@@ -1119,10 +1116,35 @@ BEGIN_FTR_SECTION
 	mtspr	SPRN_HDSISR, r0
 END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
 
+	ld	r6, VCPU_KVM(r4)
+	lbz	r7, KVM_SECURE_GUEST(r6)
+	cmpdi	r7, 0
+	bne	ret_to_ultra
+
+	lwz	r6, VCPU_CR(r4)
+	mtcr	r6
+
+	ld	r7, VCPU_GPR(R7)(r4)
+	ld	r6, VCPU_GPR(R6)(r4)
 	ld	r0, VCPU_GPR(R0)(r4)
 	ld	r4, VCPU_GPR(R4)(r4)
 	HRFI_TO_GUEST
 	b	.
+/*
+ * We are entering a secure guest, so we have to invoke the ultravisor to do
+ * that. If we are returning from a hcall, the results are already in the
+ * appropriate registers (R3:12), except for R6,7 which we used as temporary
+ * registers above. Restore them, and set R0 to the ucall number (UV_RETURN).
+ */
+ret_to_ultra:
+	lwz	r6, VCPU_CR(r4)
+	mtcr	r6
+	mfspr	r11, SPRN_SRR1
+	LOAD_REG_IMMEDIATE(r0, UV_RETURN)
+	ld	r7, VCPU_GPR(R7)(r4)
+	ld	r6, VCPU_GPR(R6)(r4)
+	ld	r4, VCPU_GPR(R4)(r4)
+	sc	2
 
 /*
  * Enter the guest on a P9 or later system where we have exactly
@@ -3318,13 +3340,16 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_TYPE_RADIX)
  *   r0 is used as a scratch register
  */
 kvmppc_msr_interrupt:
+	andis.	r0, r11, MSR_S@h
 	rldicl	r0, r11, 64 - MSR_TS_S_LG, 62
-	cmpwi	r0, 2 /* Check if we are in transactional state..  */
+	cmpwi	cr1, r0, 2 /* Check if we are in transactional state..  */
 	ld	r11, VCPU_INTR_MSR(r9)
-	bne	1f
+	bne	cr1, 1f
 	/* ... if transactional, change to suspended */
 	li	r0, 1
 1:	rldimi	r11, r0, MSR_TS_S_LG, 63 - MSR_TS_T_LG
+	beqlr
+	oris	r11, r11, MSR_S@h		/* preserve MSR_S bit setting */
 	blr
 
 /*
Signature · 1 line
2.20.1
Acked-by on v3 8/9
#25 lore raw

Re: [PATCH v3 8/9] KVM: PPC: Ultravisor: Enter a secure guest

Quoted 18 lines · Claudio Carvalho

From: Sukadev Bhattiprolu <[email protected]>

To enter a secure guest, we have to go through the ultravisor, therefore we do a ucall when we are entering a secure guest.

This change is needed for any sort of entry to the secure guest from the hypervisor, whether it is a return from an hcall, a return from a hypervisor interrupt, or the first time that a secure guest vCPU is run.

If we are returning from an hcall, the results are already in the appropriate registers (R3:12), except for R6,7, which need to be restored before doing the ucall (UV_RETURN).

Have fast_guest_return check the kvm_arch.secure_guest field so that a new CPU enters UV when started (in response to a RTAS start-cpu call).

Thanks to input from Paul Mackerras, Ram Pai and Mike Anderson.

Quoted 8 lines

[Pass SRR1 in r11 for UV_RETURN, fix kvmppc_msr_interrupt to preserve
the MSR_S bit]

[Fix UV_RETURN token number and arch.secure_guest check]
[Update commit message and ret_to_ultra comment]

Paul.

#26 lore raw

PATCH v3 9/9 KVM: PPC: Ultravisor: Check for MSR_S during hv_reset_msr

In reply to Claudio Carvalho

From: Michael Anderson <[email protected]>

- Check for MSR_S so that kvmppc_set_msr will include. Prior to this change return to guest would not have the S bit set.

- Patch based on comment from Paul Mackerras <[email protected]>

---
 arch/powerpc/kvm/book3s_64_mmu_hv.c | 1 +
 1 file changed, 1 insertion(+)
arch/powerpc/kvm/book3s_64_mmu_hv.c +1 -0
index ab3d484c5e2e..ab62a66f9b4e 100644
@@ -295,6 +295,7 @@ static void kvmppc_mmu_book3s_64_hv_reset_msr(struct kvm_vcpu *vcpu)
 		msr |= MSR_TS_S;
 	else
 		msr |= vcpu->arch.shregs.msr & MSR_TS_MASK;
+	msr |= vcpu->arch.shregs.msr & MSR_S;
 	kvmppc_set_msr(vcpu, msr);
 }
 
Signature · 1 line
2.20.1
Acked-by on v3 9/9
#27 lore raw

Re: [PATCH v3 9/9] KVM: PPC: Ultravisor: Check for MSR_S during hv_reset_msr

Quoted 3 lines · Claudio Carvalho

From: Michael Anderson <[email protected]>

- Check for MSR_S so that kvmppc_set_msr will include. Prior to this

Will include what? "it" maybe?

Quoted 6 lines

change return to guest would not have the S bit set.

- Patch based on comment from Paul Mackerras <[email protected]>

but you should reword the commit message fix that first sentence.

Paul.