Skip to contentSkip to conversation

kvmppc: Paravirtualize KVM to support ultravisor

PATCH Superseded Claudio Carvalho 9 participants 49 messages Jun 28, 2019 – Jul 22, 2019

Subsystems (2)

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v4 cover kvmppc: Paravirtualize KVM to support ultravisor

POWER platforms that supports the Protected Execution Facility (PEF) introduce features that combine hardware facilities and firmware to enable secure virtual machines. That includes a new processor mode (ultravisor mode) and the ultravisor firmware.

In PEF enabled systems, the ultravisor firmware runs at a privilege level above the hypervisor and also takes control over some system resources. The hypervisor, though, can make system calls to access these resources. Such system calls, a.k.a. ucalls, are handled by the ultravisor firmware.

The processor allows part of the system memory to be configured as secure memory, and introduces a a new mode, called secure mode, where any software entity in that mode can access secure memory. The hypervisor doesn't (and can't) run in secure mode, but a secure guest and the ultravisor firmware do.

This patch set adds support for ultravisor calls and do some preparation for running secure guests.

---
Changelog:
---

v3->v4:

- Patch "KVM: PPC: Ultravisor: Add PPC_UV config option":
- Moved to the patchset "kvmppc: HMM driver to manage pages of secure guest" v5 that will be posted by Bharata Rao.

- Patch "powerpc: Introduce FW_FEATURE_ULTRAVISOR":
- Changed to depend only on CONFIG_PPC_POWERNV.

- Patch "KVM: PPC: Ultravisor: Add generic ultravisor call handler":
- Fixed whitespaces in ucall.S and in ultravisor-api.h.
- Changed to depend only on CONFIG_PPC_POWERNV.
- Changed the ucall wrapper to pass the ucall number in R3.

- Patch "KVM: PPC: Ultravisor: Use UV_WRITE_PATE ucall to register a PATE:
- Changed to depend only on CONFIG_PPC_POWERNV.

- Patch "KVM: PPC: Ultravisor: Restrict LDBAR access":
- Fixed comment in opal-imc.c to be "Disable IMC devices, when Ultravisor is enabled.
- Fixed signed-off-by.

- Patch "KVM: PPC: Ultravisor: Enter a secure guest":
- Changed the UV_RETURN assembly call to save the actual R3 in R0 for the ultravisor and pass the UV_RETURN call number in R3.

- Patch "KVM: PPC: Ultravisor: Check for MSR_S during hv_reset_msr":
- Fixed commit message.

- Rebased to powerpc/next.

v2->v3:

- Squashed patches:
- "KVM: PPC: Ultravisor: Return to UV for hcalls from SVM"
- "KVM: PPC: Book3S HV: Fixed for running secure guests"
- Renamed patch from/to:
- "KVM: PPC: Ultravisor: Return to UV for hcalls from SVM"
- "KVM: PPC: Ultravisor: Enter a secure guest
- Rebased
- Addressed comments from Paul Mackerras
- Dropped ultravisor checks made in power8 code
- Updated the commit message for:
"KVM: PPC: Ultravisor: Enter a secure guest"
- Addressed comments from Maddy
- Dropped imc-pmu.c changes
- Changed opal-imc.c to fail the probe when the ultravisor is enabled
- Fixed "ucall defined but not used" issue when CONFIG_PPC_UV not set

v1->v2:

- Addressed comments from Paul Mackerras:
- Write the pate in HV's table before doing that in UV's
- Renamed and better documented the ultravisor header files. Also added all possible return codes for each ucall
- Updated the commit message that introduces the MSR_S bit
- Moved ultravisor.c and ucall.S to arch/powerpc/kernel
- Changed ucall.S to not save CR
- Rebased
- Changed the patches order
- Updated several commit messages
- Added FW_FEATURE_ULTRAVISOR to enable use of firmware_has_feature()
- Renamed CONFIG_PPC_KVM_UV to CONFIG_PPC_UV and used it to ifdef the ucall handler and the code that populates the powerpc_firmware_features for ultravisor
- Exported the ucall symbol. KVM may be built as module.
- Restricted LDBAR access if the ultravisor firmware is available
- Dropped patches:
- "[PATCH 06/13] KVM: PPC: Ultravisor: UV_RESTRICTED_SPR_WRITE ucall"
- "[PATCH 07/13] KVM: PPC: Ultravisor: UV_RESTRICTED_SPR_READ ucall"
- "[PATCH 08/13] KVM: PPC: Ultravisor: fix mtspr and mfspr"
- Squashed patches:
- "[PATCH 09/13] KVM: PPC: Ultravisor: Return to UV for hcalls from SVM"
- "[PATCH 13/13] KVM: PPC: UV: Have fast_guest_return check secure_guest"

Claudio Carvalho (2):
powerpc: Introduce FW_FEATURE_ULTRAVISOR
KVM: PPC: Ultravisor: Restrict LDBAR access

Michael Anderson (2):
KVM: PPC: Ultravisor: Use UV_WRITE_PATE ucall to register a PATE
KVM: PPC: Ultravisor: Check for MSR_S during hv_reset_msr

Ram Pai (2):
KVM: PPC: Ultravisor: Add generic ultravisor call handler
KVM: PPC: Ultravisor: Restrict flush of the partition tlb cache

Sukadev Bhattiprolu (2):
KVM: PPC: Ultravisor: Introduce the MSR_S bit
KVM: PPC: Ultravisor: Enter a secure guest

 arch/powerpc/include/asm/firmware.h       |  5 +-
 arch/powerpc/include/asm/kvm_host.h       |  1 +
 arch/powerpc/include/asm/reg.h            |  3 ++
 arch/powerpc/include/asm/ultravisor-api.h | 24 +++++++++
 arch/powerpc/include/asm/ultravisor.h     | 49 +++++++++++++++++
 arch/powerpc/kernel/Makefile              |  1 +
 arch/powerpc/kernel/asm-offsets.c         |  1 +
 arch/powerpc/kernel/prom.c                |  4 ++
 arch/powerpc/kernel/ucall.S               | 30 +++++++++++
 arch/powerpc/kernel/ultravisor.c          | 28 ++++++++++
 arch/powerpc/kvm/book3s_64_mmu_hv.c       |  1 +
 arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 42 ++++++++++++---
 arch/powerpc/mm/book3s64/hash_utils.c     |  3 +-
 arch/powerpc/mm/book3s64/pgtable.c        | 65 +++++++++++++++++------
 arch/powerpc/mm/book3s64/radix_pgtable.c  |  9 ++--
 arch/powerpc/platforms/powernv/idle.c     |  6 ++-
 arch/powerpc/platforms/powernv/opal-imc.c |  4 ++
 17 files changed, 246 insertions(+), 30 deletions(-)
 create mode 100644 arch/powerpc/include/asm/ultravisor-api.h
 create mode 100644 arch/powerpc/include/asm/ultravisor.h
 create mode 100644 arch/powerpc/kernel/ucall.S
 create mode 100644 arch/powerpc/kernel/ultravisor.c
Signature · 1 line
2.20.1
#1 lore raw

PATCH v4 1/8 KVM: PPC: Ultravisor: Introduce the MSR_S bit

From: Sukadev Bhattiprolu <[email protected]>

The ultravisor processor mode is introduced in POWER platforms that supports the Protected Execution Facility (PEF). Ultravisor is higher privileged than hypervisor mode.

In PEF enabled platforms, the MSR_S bit is used to indicate if the thread is in secure state. With the MSR_S bit, the privilege state of the thread is now determined by MSR_S, MSR_HV and MSR_PR, as follows:

S HV PR
-----------------------

0   x   1   problem
1   0   1   problem
x   x   0   privileged
x   1   0   hypervisor
1   1   0   ultravisor
1   1   1   reserved

The hypervisor doesn't (and can't) run with the MSR_S bit set, but a secure guest and the ultravisor firmware do.

[ Update the commit message ]
---
 arch/powerpc/include/asm/reg.h | 3 +++
 1 file changed, 3 insertions(+)
arch/powerpc/include/asm/reg.h +3 -0
index 10caa145f98b..39b4c0a519f5 100644
@@ -38,6 +38,7 @@
 #define MSR_TM_LG	32		/* Trans Mem Available */
 #define MSR_VEC_LG	25	        /* Enable AltiVec */
 #define MSR_VSX_LG	23		/* Enable VSX */
+#define MSR_S_LG	22		/* Secure VM bit */
 #define MSR_POW_LG	18		/* Enable Power Management */
 #define MSR_WE_LG	18		/* Wait State Enable */
 #define MSR_TGPR_LG	17		/* TLB Update registers in use */
@@ -71,11 +72,13 @@
 #define MSR_SF		__MASK(MSR_SF_LG)	/* Enable 64 bit mode */
 #define MSR_ISF		__MASK(MSR_ISF_LG)	/* Interrupt 64b mode valid on 630 */
 #define MSR_HV 		__MASK(MSR_HV_LG)	/* Hypervisor state */
+#define MSR_S		__MASK(MSR_S_LG)	/* Secure state */
 #else
 /* so tests for these bits fail on 32-bit */
 #define MSR_SF		0
 #define MSR_ISF		0
 #define MSR_HV		0
+#define MSR_S		0
 #endif
 
 /*
Signature · 1 line
2.20.1
Reviewed-by on v4 1/8
#2 lore raw

Re: [PATCH v4 1/8] KVM: PPC: Ultravisor: Introduce the MSR_S bit

Quoted 19 lines · Claudio Carvalho

From: Sukadev Bhattiprolu <[email protected]>

The ultravisor processor mode is introduced in POWER platforms that supports the Protected Execution Facility (PEF). Ultravisor is higher privileged than hypervisor mode.

In PEF enabled platforms, the MSR_S bit is used to indicate if the thread is in secure state. With the MSR_S bit, the privilege state of the thread is now determined by MSR_S, MSR_HV and MSR_PR, as follows:

S HV PR
-----------------------

0   x   1   problem
1   0   1   problem
x   x   0   privileged
x   1   0   hypervisor
1   1   0   ultravisor
1   1   1   reserved
Quoted 7 lines

The hypervisor doesn't (and can't) run with the MSR_S bit set, but a secure guest and the ultravisor firmware do.

[ Update the commit message ]
Quoted 31 lines
---
 arch/powerpc/include/asm/reg.h | 3 +++
 1 file changed, 3 insertions(+)
arch/powerpc/include/asm/reg.h +3 -0
index 10caa145f98b..39b4c0a519f5 100644
@@ -38,6 +38,7 @@
 #define MSR_TM_LG	32		/* Trans Mem Available */
 #define MSR_VEC_LG	25	        /* Enable AltiVec */
 #define MSR_VSX_LG	23		/* Enable VSX */
+#define MSR_S_LG	22		/* Secure VM bit */
 #define MSR_POW_LG	18		/* Enable Power Management */
 #define MSR_WE_LG	18		/* Wait State Enable */
 #define MSR_TGPR_LG	17		/* TLB Update registers in use */
@@ -71,11 +72,13 @@
 #define MSR_SF		__MASK(MSR_SF_LG)	/* Enable 64 bit mode */
 #define MSR_ISF		__MASK(MSR_ISF_LG)	/* Interrupt 64b mode valid on 630 
 */
 #define MSR_HV 		__MASK(MSR_HV_LG)	/* Hypervisor state */
+#define MSR_S		__MASK(MSR_S_LG)	/* Secure state */
 #else
 /* so tests for these bits fail on 32-bit */
 #define MSR_SF		0
 #define MSR_ISF		0
 #define MSR_HV		0
+#define MSR_S		0
 #endif
 

/*

#3 lore raw

Re: [PATCH v4 1/8] KVM: PPC: Ultravisor: Introduce the MSR_S bit

In reply to Claudio Carvalho
Quoted 10 lines · Claudio Carvalho

From: Sukadev Bhattiprolu <[email protected]>

The ultravisor processor mode is introduced in POWER platforms that supports the Protected Execution Facility (PEF). Ultravisor is higher privileged than hypervisor mode.

In PEF enabled platforms, the MSR_S bit is used to indicate if the thread is in secure state. With the MSR_S bit, the privilege state of the thread is now determined by MSR_S, MSR_HV and MSR_PR, as follows:

Quoted 8 lines

S HV PR
-----------------------

0   x   1   problem
1   0   1   problem
x   x   0   privileged
x   1   0   hypervisor
1   1   0   ultravisor
1   1   1   reserved

What are you trying to express with the 'x' value?

I guess you mean it as "either" or "don't care" - but then you have cases where it could only have one value, such as hypervisor. I think it would be clearer if you spelled it out more explicitly.

Quoted 2 lines

The hypervisor doesn't (and can't) run with the MSR_S bit set, but a secure guest and the ultravisor firmware do.

I know you're trying to be helpful, but this comment is really just confusing to someone who doesn't have all the documentation.

I'd really like to see something in Documentation/powerpc describing at least the outline of how the system works. I'm pretty sure most of that is public, so even if it's mostly a list of references to other documentations or presentations that would be fine. But I'm not really happy with a whole new processor mode appearing with zero documentation in the tree.

Quoted 3 lines
[ Update the commit message ]

It's normal to prefix these comments with your handle to make it clear who is saying it.

Quoted 10 lines
---
 arch/powerpc/include/asm/reg.h | 3 +++
 1 file changed, 3 insertions(+)
arch/powerpc/include/asm/reg.h +0 -0
index 10caa145f98b..39b4c0a519f5 100644

arch/powerpc/include/asm/reg.h line 41

Quoted 4 lines
arch/powerpc/include/asm/reg.h +1 -0
@@ -38,6 +38,7 @@
 #define MSR_TM_LG	32		/* Trans Mem Available */
 #define MSR_VEC_LG	25	        /* Enable AltiVec */
 #define MSR_VSX_LG	23		/* Enable VSX */
+#define MSR_S_LG	22		/* Secure VM bit */

I don't think that's the best description, because it's also the Ultravisor bit when MSR[HV] = 1.

So "Secure state" as you have below would be better IMO.

cheers

Quoted 19 lines
arch/powerpc/include/asm/reg.h +2 -0
 #define MSR_POW_LG	18		/* Enable Power Management */
 #define MSR_WE_LG	18		/* Wait State Enable */
 #define MSR_TGPR_LG	17		/* TLB Update registers in use */
@@ -71,11 +72,13 @@
 #define MSR_SF		__MASK(MSR_SF_LG)	/* Enable 64 bit mode */
 #define MSR_ISF		__MASK(MSR_ISF_LG)	/* Interrupt 64b mode valid on 630 */
 #define MSR_HV 		__MASK(MSR_HV_LG)	/* Hypervisor state */
+#define MSR_S		__MASK(MSR_S_LG)	/* Secure state */
 #else
 /* so tests for these bits fail on 32-bit */
 #define MSR_SF		0
 #define MSR_ISF		0
 #define MSR_HV		0
+#define MSR_S		0
 #endif
 
 /*
Signature · 1 line
2.20.1
#4 lore raw

Re: [PATCH v4 1/8] KVM: PPC: Ultravisor: Introduce the MSR_S bit

Michael Ellerman's on July 11, 2019 10:57 pm:

Quoted 64 lines
Quoted 10 lines · Claudio Carvalho

From: Sukadev Bhattiprolu <[email protected]>

The ultravisor processor mode is introduced in POWER platforms that supports the Protected Execution Facility (PEF). Ultravisor is higher privileged than hypervisor mode.

In PEF enabled platforms, the MSR_S bit is used to indicate if the thread is in secure state. With the MSR_S bit, the privilege state of the thread is now determined by MSR_S, MSR_HV and MSR_PR, as follows:

Quoted 8 lines

S HV PR
-----------------------

0   x   1   problem
1   0   1   problem
x   x   0   privileged
x   1   0   hypervisor
1   1   0   ultravisor
1   1   1   reserved

What are you trying to express with the 'x' value?

I guess you mean it as "either" or "don't care" - but then you have cases where it could only have one value, such as hypervisor. I think it would be clearer if you spelled it out more explicitly.

Quoted 2 lines

The hypervisor doesn't (and can't) run with the MSR_S bit set, but a secure guest and the ultravisor firmware do.

I know you're trying to be helpful, but this comment is really just confusing to someone who doesn't have all the documentation.

I'd really like to see something in Documentation/powerpc describing at least the outline of how the system works. I'm pretty sure most of that is public, so even if it's mostly a list of references to other documentations or presentations that would be fine. But I'm not really happy with a whole new processor mode appearing with zero documentation in the tree.

Quoted 3 lines
[ Update the commit message ]

It's normal to prefix these comments with your handle to make it clear who is saying it.

Quoted 10 lines
---
 arch/powerpc/include/asm/reg.h | 3 +++
 1 file changed, 3 insertions(+)
arch/powerpc/include/asm/reg.h +0 -0
index 10caa145f98b..39b4c0a519f5 100644

arch/powerpc/include/asm/reg.h line 41

Quoted 4 lines
arch/powerpc/include/asm/reg.h +1 -0
@@ -38,6 +38,7 @@
 #define MSR_TM_LG	32		/* Trans Mem Available */
 #define MSR_VEC_LG	25	        /* Enable AltiVec */
 #define MSR_VSX_LG	23		/* Enable VSX */
+#define MSR_S_LG	22		/* Secure VM bit */

I don't think that's the best description, because it's also the Ultravisor bit when MSR[HV] = 1.

Quoted 1 line

So "Secure state" as you have below would be better IMO.

Ooops I see Michael covered everything I wrote, sorry for the noise I missed the thread.

Thanks,
Nick

#5 lore raw

Re: [PATCH v4 1/8] KVM: PPC: Ultravisor: Introduce the MSR_S bit

In reply to Claudio Carvalho

Claudio Carvalho's on June 29, 2019 6:08 am:

Quoted 10 lines

From: Sukadev Bhattiprolu <[email protected]>

The ultravisor processor mode is introduced in POWER platforms that supports the Protected Execution Facility (PEF). Ultravisor is higher privileged than hypervisor mode.

In PEF enabled platforms, the MSR_S bit is used to indicate if the thread is in secure state. With the MSR_S bit, the privilege state of the thread is now determined by MSR_S, MSR_HV and MSR_PR, as follows:

Quoted 8 lines

S HV PR
-----------------------

0   x   1   problem
1   0   1   problem
x   x   0   privileged
x   1   0   hypervisor
1   1   0   ultravisor
1   1   1   reserved

What does this table mean? I thought 'x' meant either, but in that case there are several states that can apply to the same combination of bits.

Would it be clearer to rearrange the table so the columns are the HV and PR bits we know and love, plus the effect of S=1 on each of them?

HV PR S=0 S=1
---------------------------------------------
0 0 privileged privileged (secure guest kernel)
0 1 problem problem (secure guest userspace)
1 0 hypervisor ultravisor
1 1 problem reserved

Is that accurate?

Quoted 25 lines

The hypervisor doesn't (and can't) run with the MSR_S bit set, but a secure guest and the ultravisor firmware do.

[ Update the commit message ]
---
 arch/powerpc/include/asm/reg.h | 3 +++
 1 file changed, 3 insertions(+)
arch/powerpc/include/asm/reg.h +1 -0
index 10caa145f98b..39b4c0a519f5 100644
@@ -38,6 +38,7 @@
 #define MSR_TM_LG	32		/* Trans Mem Available */
 #define MSR_VEC_LG	25	        /* Enable AltiVec */
 #define MSR_VSX_LG	23		/* Enable VSX */
+#define MSR_S_LG	22		/* Secure VM bit */
 #define MSR_POW_LG	18		/* Enable Power Management */
 #define MSR_WE_LG	18		/* Wait State Enable */
 #define MSR_TGPR_LG	17		/* TLB Update registers in use */

arch/powerpc/include/asm/reg.h line 75

Quoted 4 lines
arch/powerpc/include/asm/reg.h +1 -0
@@ -71,11 +72,13 @@
 #define MSR_SF		__MASK(MSR_SF_LG)	/* Enable 64 bit mode */
 #define MSR_ISF		__MASK(MSR_ISF_LG)	/* Interrupt 64b mode valid on 630 */
 #define MSR_HV 		__MASK(MSR_HV_LG)	/* Hypervisor state */
+#define MSR_S		__MASK(MSR_S_LG)	/* Secure state */

This is a real nitpick, but why two different comments for the bit number and the mask?

#6 lore raw

Re: [PATCH v4 1/8] KVM: PPC: Ultravisor: Introduce the MSR_S bit

Quoted 27 lines · Nicholas Piggin

Claudio Carvalho's on June 29, 2019 6:08 am:

Quoted 10 lines

From: Sukadev Bhattiprolu <[email protected]>

The ultravisor processor mode is introduced in POWER platforms that supports the Protected Execution Facility (PEF). Ultravisor is higher privileged than hypervisor mode.

In PEF enabled platforms, the MSR_S bit is used to indicate if the thread is in secure state. With the MSR_S bit, the privilege state of the thread is now determined by MSR_S, MSR_HV and MSR_PR, as follows:

Quoted 8 lines

S HV PR
-----------------------

0   x   1   problem
1   0   1   problem
x   x   0   privileged
x   1   0   hypervisor
1   1   0   ultravisor
1   1   1   reserved

What does this table mean? I thought 'x' meant either, but in that case there are several states that can apply to the same combination of bits.

Would it be clearer to rearrange the table so the columns are the HV and PR bits we know and love, plus the effect of S=1 on each of them?

Quoted 8 lines

HV PR S=0 S=1
---------------------------------------------
0 0 privileged privileged (secure guest kernel)
0 1 problem problem (secure guest userspace)
1 0 hypervisor ultravisor
1 1 problem reserved

Is that accurate?

I like that format.

cheers

#7 lore raw

Re: [PATCH v4 1/8] KVM: PPC: Ultravisor: Introduce the MSR_S bit

In reply to Nicholas Piggin
Quoted 19 lines · Nicholas Piggin

Claudio Carvalho's on June 29, 2019 6:08 am:

Quoted 10 lines

From: Sukadev Bhattiprolu <[email protected]>

The ultravisor processor mode is introduced in POWER platforms that supports the Protected Execution Facility (PEF). Ultravisor is higher privileged than hypervisor mode.

In PEF enabled platforms, the MSR_S bit is used to indicate if the thread is in secure state. With the MSR_S bit, the privilege state of the thread is now determined by MSR_S, MSR_HV and MSR_PR, as follows:

Quoted 8 lines

S HV PR
-----------------------

0   x   1   problem
1   0   1   problem
x   x   0   privileged
x   1   0   hypervisor
1   1   0   ultravisor
1   1   1   reserved
Quoted 1 line

What does this table mean? I thought 'x' meant either

Yes, it means either. The table was arranged that way to say that:
- hypervisor state is also a privileged state,
- ultravisor state is also a hypervisor state.

Quoted 7 lines

, but in that
case there are several states that can apply to the same combination of bits.

Would it be clearer to rearrange the table so the columns are the HV and PR bits we know and love, plus the effect of S=1 on each of them?

Quoted 8 lines

HV PR S=0 S=1
---------------------------------------------
0 0 privileged privileged (secure guest kernel)
0 1 problem problem (secure guest userspace)
1 0 hypervisor ultravisor
1 1 problem reserved

Is that accurate?

Yes, it is. I also like this format. I will consider it.

Quoted 24 lines
Quoted 24 lines

The hypervisor doesn't (and can't) run with the MSR_S bit set, but a secure guest and the ultravisor firmware do.

[ Update the commit message ]
---
 arch/powerpc/include/asm/reg.h | 3 +++
 1 file changed, 3 insertions(+)
arch/powerpc/include/asm/reg.h +1 -0
index 10caa145f98b..39b4c0a519f5 100644
@@ -38,6 +38,7 @@
 #define MSR_TM_LG	32		/* Trans Mem Available */
 #define MSR_VEC_LG	25	        /* Enable AltiVec */
 #define MSR_VSX_LG	23		/* Enable VSX */
+#define MSR_S_LG	22		/* Secure VM bit */
 #define MSR_POW_LG	18		/* Enable Power Management */
 #define MSR_WE_LG	18		/* Wait State Enable */
Quoted 8 lines

arch/powerpc/include/asm/reg.h line 75

Quoted 6 lines
arch/powerpc/include/asm/reg.h +1 -0
 #define MSR_TGPR_LG	17		/* TLB Update registers in use */
@@ -71,11 +72,13 @@
 #define MSR_SF		__MASK(MSR_SF_LG)	/* Enable 64 bit mode */
 #define MSR_ISF		__MASK(MSR_ISF_LG)	/* Interrupt 64b mode valid on 630 */
 #define MSR_HV 		__MASK(MSR_HV_LG)	/* Hypervisor state */
+#define MSR_S		__MASK(MSR_S_LG)	/* Secure state */

This is a real nitpick, but why two different comments for the bit number and the mask?

Fixed for the next version. Both comments will be /* Secure state */

Thanks
Claudio

#8 lore raw

PATCH v4 2/8 powerpc: Introduce FW_FEATURE_ULTRAVISOR

In reply to Claudio Carvalho

This feature tells if the ultravisor firmware is available to handle ucalls.

[ Device node name to "ibm,ultravisor" ]
---
 arch/powerpc/include/asm/firmware.h   |  5 +++--
 arch/powerpc/include/asm/ultravisor.h | 15 +++++++++++++++
 arch/powerpc/kernel/Makefile          |  1 +
 arch/powerpc/kernel/prom.c            |  4 ++++
 arch/powerpc/kernel/ultravisor.c      | 24 ++++++++++++++++++++++++
 5 files changed, 47 insertions(+), 2 deletions(-)
 create mode 100644 arch/powerpc/include/asm/ultravisor.h
 create mode 100644 arch/powerpc/kernel/ultravisor.c
arch/powerpc/include/asm/firmware.h +3 -2
index 00bc42d95679..43b48c4d3ca9 100644
@@ -54,6 +54,7 @@
 #define FW_FEATURE_DRC_INFO	ASM_CONST(0x0000000800000000)
 #define FW_FEATURE_BLOCK_REMOVE ASM_CONST(0x0000001000000000)
 #define FW_FEATURE_PAPR_SCM 	ASM_CONST(0x0000002000000000)
+#define FW_FEATURE_ULTRAVISOR	ASM_CONST(0x0000004000000000)
 
 #ifndef __ASSEMBLY__
 
@@ -72,9 +73,9 @@ enum {
 		FW_FEATURE_TYPE1_AFFINITY | FW_FEATURE_PRRN |
 		FW_FEATURE_HPT_RESIZE | FW_FEATURE_DRMEM_V2 |
 		FW_FEATURE_DRC_INFO | FW_FEATURE_BLOCK_REMOVE |
-		FW_FEATURE_PAPR_SCM,
+		FW_FEATURE_PAPR_SCM | FW_FEATURE_ULTRAVISOR,
 	FW_FEATURE_PSERIES_ALWAYS = 0,
-	FW_FEATURE_POWERNV_POSSIBLE = FW_FEATURE_OPAL,
+	FW_FEATURE_POWERNV_POSSIBLE = FW_FEATURE_OPAL | FW_FEATURE_ULTRAVISOR,
 	FW_FEATURE_POWERNV_ALWAYS = 0,
 	FW_FEATURE_PS3_POSSIBLE = FW_FEATURE_LPAR | FW_FEATURE_PS3_LV1,
 	FW_FEATURE_PS3_ALWAYS = FW_FEATURE_LPAR | FW_FEATURE_PS3_LV1,
arch/powerpc/include/asm/ultravisor.h new file +15 -0
new file mode 100644
index 000000000000..e5009b0d84ea
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Ultravisor definitions
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#ifndef _ASM_POWERPC_ULTRAVISOR_H
+#define _ASM_POWERPC_ULTRAVISOR_H
+
+/* Internal functions */
+extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
+					 int depth, void *data);
+
+#endif	/* _ASM_POWERPC_ULTRAVISOR_H */
arch/powerpc/kernel/Makefile +1 -0
index 0ea6c4aa3a20..f0caa302c8c0 100644
@@ -154,6 +154,7 @@ endif
 
 obj-$(CONFIG_EPAPR_PARAVIRT)	+= epapr_paravirt.o epapr_hcalls.o
 obj-$(CONFIG_KVM_GUEST)		+= kvm.o kvm_emul.o
+obj-$(CONFIG_PPC_POWERNV)	+= ultravisor.o
 
 # Disable GCOV, KCOV & sanitizers in odd or sensitive code
 GCOV_PROFILE_prom_init.o := n
arch/powerpc/kernel/prom.c +4 -0
index 4221527b082f..67a2c1b39252 100644
@@ -59,6 +59,7 @@
 #include <asm/firmware.h>
 #include <asm/dt_cpu_ftrs.h>
 #include <asm/drmem.h>
+#include <asm/ultravisor.h>
 
 #include <mm/mmu_decl.h>
 
@@ -706,6 +707,9 @@ void __init early_init_devtree(void *params)
 #ifdef CONFIG_PPC_POWERNV
 	/* Some machines might need OPAL info for debugging, grab it now. */
 	of_scan_flat_dt(early_init_dt_scan_opal, NULL);
+
+	/* Scan tree for ultravisor feature */
+	of_scan_flat_dt(early_init_dt_scan_ultravisor, NULL);
 #endif
 
 #ifdef CONFIG_FA_DUMP
arch/powerpc/kernel/ultravisor.c new file +24 -0
new file mode 100644
index 000000000000..dc6021f63c97
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Ultravisor high level interfaces
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#include <linux/init.h>
+#include <linux/printk.h>
+#include <linux/string.h>
+
+#include <asm/ultravisor.h>
+#include <asm/firmware.h>
+
+int __init early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
+					 int depth, void *data)
+{
+	if (depth != 1 || strcmp(uname, "ibm,ultravisor") != 0)
+		return 0;
+
+	powerpc_firmware_features |= FW_FEATURE_ULTRAVISOR;
+	pr_debug("Ultravisor detected!\n");
+	return 1;
+}
Signature · 1 line
2.20.1
Reviewed-by on v4 2/8
#9 lore raw

Re: [PATCH v4 2/8] powerpc: Introduce FW_FEATURE_ULTRAVISOR

Quoted 6 lines · Claudio Carvalho

This feature tells if the ultravisor firmware is available to handle ucalls.

[ Device node name to "ibm,ultravisor" ]
Quoted 124 lines
---
 arch/powerpc/include/asm/firmware.h   |  5 +++--
 arch/powerpc/include/asm/ultravisor.h | 15 +++++++++++++++
 arch/powerpc/kernel/Makefile          |  1 +
 arch/powerpc/kernel/prom.c            |  4 ++++
 arch/powerpc/kernel/ultravisor.c      | 24 ++++++++++++++++++++++++
 5 files changed, 47 insertions(+), 2 deletions(-)
 create mode 100644 arch/powerpc/include/asm/ultravisor.h
 create mode 100644 arch/powerpc/kernel/ultravisor.c
arch/powerpc/include/asm/firmware.h +3 -2
index 00bc42d95679..43b48c4d3ca9 100644
@@ -54,6 +54,7 @@
 #define FW_FEATURE_DRC_INFO	ASM_CONST(0x0000000800000000)
 #define FW_FEATURE_BLOCK_REMOVE ASM_CONST(0x0000001000000000)
 #define FW_FEATURE_PAPR_SCM 	ASM_CONST(0x0000002000000000)
+#define FW_FEATURE_ULTRAVISOR	ASM_CONST(0x0000004000000000)
 
 #ifndef __ASSEMBLY__
 
@@ -72,9 +73,9 @@ enum {
 		FW_FEATURE_TYPE1_AFFINITY | FW_FEATURE_PRRN |
 		FW_FEATURE_HPT_RESIZE | FW_FEATURE_DRMEM_V2 |
 		FW_FEATURE_DRC_INFO | FW_FEATURE_BLOCK_REMOVE |
-		FW_FEATURE_PAPR_SCM,
+		FW_FEATURE_PAPR_SCM | FW_FEATURE_ULTRAVISOR,
 	FW_FEATURE_PSERIES_ALWAYS = 0,
-	FW_FEATURE_POWERNV_POSSIBLE = FW_FEATURE_OPAL,
+	FW_FEATURE_POWERNV_POSSIBLE = FW_FEATURE_OPAL | 
 FW_FEATURE_ULTRAVISOR,
 	FW_FEATURE_POWERNV_ALWAYS = 0,
 	FW_FEATURE_PS3_POSSIBLE = FW_FEATURE_LPAR | FW_FEATURE_PS3_LV1,
arch/powerpc/include/asm/firmware.h +0 -0
 	FW_FEATURE_PS3_ALWAYS = FW_FEATURE_LPAR | FW_FEATURE_PS3_LV1,
arch/powerpc/include/asm/ultravisor.h new file +12 -0
new file mode 100644
index 000000000000..e5009b0d84ea
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Ultravisor definitions
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#ifndef _ASM_POWERPC_ULTRAVISOR_H
+#define _ASM_POWERPC_ULTRAVISOR_H
+
+/* Internal functions */
+extern int early_init_dt_scan_ultravisor(unsigned long node, const 

char *uname,

arch/powerpc/include/asm/ultravisor.h new file +3 -0
+					 int depth, void *data);
+
+#endif	/* _ASM_POWERPC_ULTRAVISOR_H */
arch/powerpc/kernel/Makefile +1 -0
index 0ea6c4aa3a20..f0caa302c8c0 100644
@@ -154,6 +154,7 @@ endif
 
 obj-$(CONFIG_EPAPR_PARAVIRT)	+= epapr_paravirt.o epapr_hcalls.o
 obj-$(CONFIG_KVM_GUEST)		+= kvm.o kvm_emul.o
+obj-$(CONFIG_PPC_POWERNV)	+= ultravisor.o
 
 # Disable GCOV, KCOV & sanitizers in odd or sensitive code
 GCOV_PROFILE_prom_init.o := n
arch/powerpc/kernel/prom.c +4 -0
index 4221527b082f..67a2c1b39252 100644
@@ -59,6 +59,7 @@
 #include <asm/firmware.h>
 #include <asm/dt_cpu_ftrs.h>
 #include <asm/drmem.h>
+#include <asm/ultravisor.h>
 
 #include <mm/mmu_decl.h>
 
@@ -706,6 +707,9 @@ void __init early_init_devtree(void *params)
 #ifdef CONFIG_PPC_POWERNV
 	/* Some machines might need OPAL info for debugging, grab it now. */
 	of_scan_flat_dt(early_init_dt_scan_opal, NULL);
+
+	/* Scan tree for ultravisor feature */
+	of_scan_flat_dt(early_init_dt_scan_ultravisor, NULL);
 #endif
 
 #ifdef CONFIG_FA_DUMP
arch/powerpc/kernel/ultravisor.c new file +15 -0
new file mode 100644
index 000000000000..dc6021f63c97
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Ultravisor high level interfaces
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#include <linux/init.h>
+#include <linux/printk.h>
+#include <linux/string.h>
+
+#include <asm/ultravisor.h>
+#include <asm/firmware.h>
+
+int __init early_init_dt_scan_ultravisor(unsigned long node, const 

char *uname,

arch/powerpc/kernel/ultravisor.c new file +9 -0
+					 int depth, void *data)
+{
+	if (depth != 1 || strcmp(uname, "ibm,ultravisor") != 0)
+		return 0;
+
+	powerpc_firmware_features |= FW_FEATURE_ULTRAVISOR;
+	pr_debug("Ultravisor detected!\n");
+	return 1;
+}
#10 lore raw

Re: [PATCH v4 2/8] powerpc: Introduce FW_FEATURE_ULTRAVISOR

In reply to Claudio Carvalho
Quoted 9 lines · Claudio Carvalho
arch/powerpc/include/asm/ultravisor.h new file +3 -0
new file mode 100644
index 000000000000..e5009b0d84ea
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Ultravisor definitions

arch/powerpc/include/asm/ultravisor.h line 13

Quoted 10 lines
arch/powerpc/include/asm/ultravisor.h new file +10 -0
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#ifndef _ASM_POWERPC_ULTRAVISOR_H
+#define _ASM_POWERPC_ULTRAVISOR_H
+
+/* Internal functions */
+extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
+					 int depth, void *data);

Please don't use extern in new headers.

arch/powerpc/kernel/ultravisor.c

Quoted 5 lines
arch/powerpc/kernel/ultravisor.c new file +0 -0
new file mode 100644
index 000000000000..dc6021f63c97

Is there a reason this (and other later files) aren't in platforms/powernv ?

Quoted 10 lines
arch/powerpc/kernel/ultravisor.c new file +9 -0
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Ultravisor high level interfaces
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#include <linux/init.h>
+#include <linux/printk.h>

arch/powerpc/kernel/ultravisor.c line 19

Quoted 10 lines
arch/powerpc/kernel/ultravisor.c new file +10 -0
+#include <linux/string.h>
+
+#include <asm/ultravisor.h>
+#include <asm/firmware.h>
+
+int __init early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
+					 int depth, void *data)
+{
+	if (depth != 1 || strcmp(uname, "ibm,ultravisor") != 0)
+		return 0;

I know you're following the example of OPAL, but this is not the best way to search for the ultravisor node.

It makes the location and name of the node part of the ABI, when there's no need for it to be.

If instead you just scan the tree for a node that is *compatible* with "ibm,ultravisor" (or whatever compatible string) then the node can be placed any where in the tree and have any name, which gives us the most flexibility in future to change the location of the device tree node.

cheers

#11 lore raw

Re: [PATCH v4 2/8] powerpc: Introduce FW_FEATURE_ULTRAVISOR

Quoted 22 lines · Michael Ellerman
Quoted 9 lines · Claudio Carvalho
arch/powerpc/include/asm/ultravisor.h new file +3 -0
new file mode 100644
index 000000000000..e5009b0d84ea
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Ultravisor definitions

arch/powerpc/include/asm/ultravisor.h line 13

Quoted 10 lines
arch/powerpc/include/asm/ultravisor.h new file +10 -0
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#ifndef _ASM_POWERPC_ULTRAVISOR_H
+#define _ASM_POWERPC_ULTRAVISOR_H
+
+/* Internal functions */
+extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
+					 int depth, void *data);

Please don't use extern in new headers.

Quoted 6 lines

arch/powerpc/kernel/ultravisor.c

Quoted 5 lines
arch/powerpc/kernel/ultravisor.c new file +0 -0
new file mode 100644
index 000000000000..dc6021f63c97

Is there a reason this (and other later files) aren't in platforms/powernv ?

Yes, there is.
https://www.spinics.net/lists/kvm-ppc/msg14998.html

We also need to do ucalls from a secure guest and its kernel may not have CONFIG_PPC_POWERNV=y. I can make it clear in the commit message.

Quoted 24 lines
Quoted 11 lines
arch/powerpc/kernel/ultravisor.c new file +9 -0
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Ultravisor high level interfaces
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#include <linux/init.h>
+#include <linux/printk.h>

arch/powerpc/kernel/ultravisor.c line 19

Quoted 10 lines
arch/powerpc/kernel/ultravisor.c new file +10 -0
+#include <linux/string.h>
+
+#include <asm/ultravisor.h>
+#include <asm/firmware.h>
+
+int __init early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
+					 int depth, void *data)
+{
+	if (depth != 1 || strcmp(uname, "ibm,ultravisor") != 0)
+		return 0;

I know you're following the example of OPAL, but this is not the best way to search for the ultravisor node.

Quoted 7 lines

It makes the location and name of the node part of the ABI, when there's no need for it to be.

If instead you just scan the tree for a node that is *compatible* with "ibm,ultravisor" (or whatever compatible string) then the node can be placed any where in the tree and have any name, which gives us the most flexibility in future to change the location of the device tree node.

I will do that in the next version.

Thanks,
Claudio

Quoted 3 lines

cheers

#12 lore raw

Re: [PATCH v4 2/8] powerpc: Introduce FW_FEATURE_ULTRAVISOR

Quoted 30 lines · Claudio Carvalho
Quoted 22 lines · Michael Ellerman
Quoted 9 lines · Claudio Carvalho
arch/powerpc/include/asm/ultravisor.h new file +3 -0
new file mode 100644
index 000000000000..e5009b0d84ea
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Ultravisor definitions

arch/powerpc/include/asm/ultravisor.h line 13

Quoted 10 lines
arch/powerpc/include/asm/ultravisor.h new file +10 -0
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#ifndef _ASM_POWERPC_ULTRAVISOR_H
+#define _ASM_POWERPC_ULTRAVISOR_H
+
+/* Internal functions */
+extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
+					 int depth, void *data);

Please don't use extern in new headers.

Quoted 6 lines

arch/powerpc/kernel/ultravisor.c

Quoted 5 lines
arch/powerpc/kernel/ultravisor.c new file +0 -0
new file mode 100644
index 000000000000..dc6021f63c97

Is there a reason this (and other later files) aren't in platforms/powernv ?

Quoted 5 lines

Yes, there is.
https://www.spinics.net/lists/kvm-ppc/msg14998.html

We also need to do ucalls from a secure guest and its kernel may not have CONFIG_PPC_POWERNV=y. I can make it clear in the commit message.

OK, sorry I missed Paul's comment. Yeah that is obviously a valid reason :)

Quoted 26 lines
Quoted 26 lines
Quoted 10 lines
arch/powerpc/kernel/ultravisor.c new file +9 -0
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Ultravisor high level interfaces
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#include <linux/init.h>
+#include <linux/printk.h>

arch/powerpc/kernel/ultravisor.c line 19

Quoted 10 lines
arch/powerpc/kernel/ultravisor.c new file +10 -0
+#include <linux/string.h>
+
+#include <asm/ultravisor.h>
+#include <asm/firmware.h>
+
+int __init early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
+					 int depth, void *data)
+{
+	if (depth != 1 || strcmp(uname, "ibm,ultravisor") != 0)
+		return 0;

I know you're following the example of OPAL, but this is not the best way to search for the ultravisor node.

It makes the location and name of the node part of the ABI, when there's no need for it to be.

Quoted 6 lines
Quoted 4 lines

If instead you just scan the tree for a node that is *compatible* with "ibm,ultravisor" (or whatever compatible string) then the node can be placed any where in the tree and have any name, which gives us the most flexibility in future to change the location of the device tree node.

I will do that in the next version.

Excellent, thanks.

cheers

#13 lore raw

PATCH v4 3/8 KVM: PPC: Ultravisor: Add generic ultravisor call handler

In reply to Claudio Carvalho

From: Ram Pai <[email protected]>

Add the ucall() function, which can be used to make ultravisor calls with varied number of in and out arguments. Ultravisor calls can be made from the host or guests.

This copies the implementation of plpar_hcall().

[ Change ucall.S to not save CR, rename and move headers, build ucall.S
if CONFIG_PPC_POWERNV set, use R3 for the ucall number and add some comments in the code ]

---
 arch/powerpc/include/asm/ultravisor-api.h | 20 +++++++++++++++
 arch/powerpc/include/asm/ultravisor.h     | 20 +++++++++++++++
 arch/powerpc/kernel/Makefile              |  2 +-
 arch/powerpc/kernel/ucall.S               | 30 +++++++++++++++++++++++
 arch/powerpc/kernel/ultravisor.c          |  4 +++
 5 files changed, 75 insertions(+), 1 deletion(-)
 create mode 100644 arch/powerpc/include/asm/ultravisor-api.h
 create mode 100644 arch/powerpc/kernel/ucall.S
arch/powerpc/include/asm/ultravisor-api.h new file +20 -0
new file mode 100644
index 000000000000..49e766adabc7
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Ultravisor API.
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#ifndef _ASM_POWERPC_ULTRAVISOR_API_H
+#define _ASM_POWERPC_ULTRAVISOR_API_H
+
+#include <asm/hvcall.h>
+
+/* Return codes */
+#define U_NOT_AVAILABLE		H_NOT_AVAILABLE
+#define U_SUCCESS		H_SUCCESS
+#define U_FUNCTION		H_FUNCTION
+#define U_PARAMETER		H_PARAMETER
+
+#endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
+
arch/powerpc/include/asm/ultravisor.h +20 -0
index e5009b0d84ea..a78a2dacfd0b 100644
@@ -8,8 +8,28 @@
 #ifndef _ASM_POWERPC_ULTRAVISOR_H
 #define _ASM_POWERPC_ULTRAVISOR_H
 
+#include <asm/ultravisor-api.h>
+
+#if !defined(__ASSEMBLY__)
+
 /* Internal functions */
 extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
 					 int depth, void *data);
 
+/* API functions */
+#define UCALL_BUFSIZE 4
+/**
+ * ucall: Make a powerpc ultravisor call.
+ * @opcode: The ultravisor call to make.
+ * @retbuf: Buffer to store up to 4 return arguments in.
+ *
+ * This call supports up to 6 arguments and 4 return arguments. Use
+ * UCALL_BUFSIZE to size the return argument buffer.
+ */
+#if defined(CONFIG_PPC_POWERNV)
+long ucall(unsigned long opcode, unsigned long *retbuf, ...);
+#endif
+
+#endif /* !__ASSEMBLY__ */
+
 #endif	/* _ASM_POWERPC_ULTRAVISOR_H */
arch/powerpc/kernel/Makefile +1 -1
index f0caa302c8c0..f28baccc0a79 100644
@@ -154,7 +154,7 @@ endif
 
 obj-$(CONFIG_EPAPR_PARAVIRT)	+= epapr_paravirt.o epapr_hcalls.o
 obj-$(CONFIG_KVM_GUEST)		+= kvm.o kvm_emul.o
-obj-$(CONFIG_PPC_POWERNV)	+= ultravisor.o
+obj-$(CONFIG_PPC_POWERNV)	+= ultravisor.o ucall.o
 
 # Disable GCOV, KCOV & sanitizers in odd or sensitive code
 GCOV_PROFILE_prom_init.o := n
arch/powerpc/kernel/ucall.S new file +30 -0
new file mode 100644
index 000000000000..1678f6eb7230
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Generic code to perform an ultravisor call.
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#include <asm/ppc_asm.h>
+
+/*
+ * This function is based on the plpar_hcall()
+ */
+_GLOBAL_TOC(ucall)
+	std	r4,STK_PARAM(R4)(r1)	/* Save ret buffer */
+	mr	r4,r5
+	mr	r5,r6
+	mr	r6,r7
+	mr	r7,r8
+	mr	r8,r9
+	mr	r9,r10
+
+	sc 2				/* Invoke the ultravisor */
+
+	ld	r12,STK_PARAM(R4)(r1)
+	std	r4,  0(r12)
+	std	r5,  8(r12)
+	std	r6, 16(r12)
+	std	r7, 24(r12)
+
+	blr				/* Return r3 = status */
arch/powerpc/kernel/ultravisor.c +4 -0
index dc6021f63c97..02ddf79a9522 100644
@@ -8,10 +8,14 @@
 #include <linux/init.h>
 #include <linux/printk.h>
 #include <linux/string.h>
+#include <linux/export.h>
 
 #include <asm/ultravisor.h>
 #include <asm/firmware.h>
 
+/* in ucall.S */
+EXPORT_SYMBOL_GPL(ucall);
+
 int __init early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
 					 int depth, void *data)
 {
Signature · 1 line
2.20.1
Reviewed-by on v4 3/8
#14 lore raw

Re: [PATCH v4 3/8] KVM: PPC: Ultravisor: Add generic ultravisor call handler

Quoted 7 lines · Claudio Carvalho

From: Ram Pai <[email protected]>

Add the ucall() function, which can be used to make ultravisor calls with varied number of in and out arguments. Ultravisor calls can be made
from the host or guests.

Quoted 7 lines

This copies the implementation of plpar_hcall().

[ Change ucall.S to not save CR, rename and move headers, build ucall.S
if CONFIG_PPC_POWERNV set, use R3 for the ucall number and add some comments in the code ]

Quoted 143 lines
---
 arch/powerpc/include/asm/ultravisor-api.h | 20 +++++++++++++++
 arch/powerpc/include/asm/ultravisor.h     | 20 +++++++++++++++
 arch/powerpc/kernel/Makefile              |  2 +-
 arch/powerpc/kernel/ucall.S               | 30 +++++++++++++++++++++++
 arch/powerpc/kernel/ultravisor.c          |  4 +++
 5 files changed, 75 insertions(+), 1 deletion(-)
 create mode 100644 arch/powerpc/include/asm/ultravisor-api.h
 create mode 100644 arch/powerpc/kernel/ucall.S
arch/powerpc/include/asm/ultravisor-api.h new file +20 -0
new file mode 100644
index 000000000000..49e766adabc7
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Ultravisor API.
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#ifndef _ASM_POWERPC_ULTRAVISOR_API_H
+#define _ASM_POWERPC_ULTRAVISOR_API_H
+
+#include <asm/hvcall.h>
+
+/* Return codes */
+#define U_NOT_AVAILABLE		H_NOT_AVAILABLE
+#define U_SUCCESS		H_SUCCESS
+#define U_FUNCTION		H_FUNCTION
+#define U_PARAMETER		H_PARAMETER
+
+#endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
+
arch/powerpc/include/asm/ultravisor.h +20 -0
index e5009b0d84ea..a78a2dacfd0b 100644
@@ -8,8 +8,28 @@
 #ifndef _ASM_POWERPC_ULTRAVISOR_H
 #define _ASM_POWERPC_ULTRAVISOR_H
 
+#include <asm/ultravisor-api.h>
+
+#if !defined(__ASSEMBLY__)
+
 /* Internal functions */
 extern int early_init_dt_scan_ultravisor(unsigned long node, const 
 char *uname,
 					 int depth, void *data);
 
+/* API functions */
+#define UCALL_BUFSIZE 4
+/**
+ * ucall: Make a powerpc ultravisor call.
+ * @opcode: The ultravisor call to make.
+ * @retbuf: Buffer to store up to 4 return arguments in.
+ *
+ * This call supports up to 6 arguments and 4 return arguments. Use
+ * UCALL_BUFSIZE to size the return argument buffer.
+ */
+#if defined(CONFIG_PPC_POWERNV)
+long ucall(unsigned long opcode, unsigned long *retbuf, ...);
+#endif
+
+#endif /* !__ASSEMBLY__ */
+

#endif /* _ASM_POWERPC_ULTRAVISOR_H */

arch/powerpc/kernel/Makefile +1 -1
index f0caa302c8c0..f28baccc0a79 100644
@@ -154,7 +154,7 @@ endif
 
 obj-$(CONFIG_EPAPR_PARAVIRT)	+= epapr_paravirt.o epapr_hcalls.o
 obj-$(CONFIG_KVM_GUEST)		+= kvm.o kvm_emul.o
-obj-$(CONFIG_PPC_POWERNV)	+= ultravisor.o
+obj-$(CONFIG_PPC_POWERNV)	+= ultravisor.o ucall.o
 
 # Disable GCOV, KCOV & sanitizers in odd or sensitive code
 GCOV_PROFILE_prom_init.o := n
arch/powerpc/kernel/ucall.S new file +30 -0
new file mode 100644
index 000000000000..1678f6eb7230
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Generic code to perform an ultravisor call.
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#include <asm/ppc_asm.h>
+
+/*
+ * This function is based on the plpar_hcall()
+ */
+_GLOBAL_TOC(ucall)
+	std	r4,STK_PARAM(R4)(r1)	/* Save ret buffer */
+	mr	r4,r5
+	mr	r5,r6
+	mr	r6,r7
+	mr	r7,r8
+	mr	r8,r9
+	mr	r9,r10
+
+	sc 2				/* Invoke the ultravisor */
+
+	ld	r12,STK_PARAM(R4)(r1)
+	std	r4,  0(r12)
+	std	r5,  8(r12)
+	std	r6, 16(r12)
+	std	r7, 24(r12)
+
+	blr				/* Return r3 = status */
arch/powerpc/kernel/ultravisor.c +4 -0
index dc6021f63c97..02ddf79a9522 100644
@@ -8,10 +8,14 @@
 #include <linux/init.h>
 #include <linux/printk.h>
 #include <linux/string.h>
+#include <linux/export.h>
 
 #include <asm/ultravisor.h>
 #include <asm/firmware.h>
 
+/* in ucall.S */
+EXPORT_SYMBOL_GPL(ucall);
+
 int __init early_init_dt_scan_ultravisor(unsigned long node, const 
 char *uname,
 					 int depth, void *data)

{

#15 lore raw

Re: [PATCH v4 3/8] KVM: PPC: Ultravisor: Add generic ultravisor call handler

In reply to Claudio Carvalho
Quoted 7 lines · Claudio Carvalho

From: Ram Pai <[email protected]>

Add the ucall() function, which can be used to make ultravisor calls with varied number of in and out arguments. Ultravisor calls can be made from the host or guests.

This copies the implementation of plpar_hcall().

.. with quite a few changes?

This is one of the things I'd like to see in a Documentation file, so that people can review the implementation vs the specification.

Quoted 4 lines

[ Change ucall.S to not save CR, rename and move headers, build ucall.S
if CONFIG_PPC_POWERNV set, use R3 for the ucall number and add some comments in the code ]

Why are we not saving CR? See previous comment about Documentation :)

Quoted 24 lines
---
 arch/powerpc/include/asm/ultravisor-api.h | 20 +++++++++++++++
 arch/powerpc/include/asm/ultravisor.h     | 20 +++++++++++++++
 arch/powerpc/kernel/Makefile              |  2 +-
 arch/powerpc/kernel/ucall.S               | 30 +++++++++++++++++++++++
 arch/powerpc/kernel/ultravisor.c          |  4 +++
 5 files changed, 75 insertions(+), 1 deletion(-)
 create mode 100644 arch/powerpc/include/asm/ultravisor-api.h
 create mode 100644 arch/powerpc/kernel/ucall.S
arch/powerpc/include/asm/ultravisor-api.h new file +7 -0
new file mode 100644
index 000000000000..49e766adabc7
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Ultravisor API.
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */

arch/powerpc/include/asm/ultravisor-api.h line 17

Quoted 10 lines
arch/powerpc/include/asm/ultravisor-api.h new file +10 -0
+#ifndef _ASM_POWERPC_ULTRAVISOR_API_H
+#define _ASM_POWERPC_ULTRAVISOR_API_H
+
+#include <asm/hvcall.h>
+
+/* Return codes */
+#define U_NOT_AVAILABLE		H_NOT_AVAILABLE
+#define U_SUCCESS		H_SUCCESS
+#define U_FUNCTION		H_FUNCTION
+#define U_PARAMETER		H_PARAMETER

Is there any benefit in redefining these?

arch/powerpc/include/asm/ultravisor.h line 13

Quoted 11 lines
arch/powerpc/include/asm/ultravisor.h +3 -0
index e5009b0d84ea..a78a2dacfd0b 100644
@@ -8,8 +8,28 @@
 #ifndef _ASM_POWERPC_ULTRAVISOR_H
 #define _ASM_POWERPC_ULTRAVISOR_H
 
+#include <asm/ultravisor-api.h>
+
+#if !defined(__ASSEMBLY__)

Just #ifndef is fine.

arch/powerpc/include/asm/ultravisor.h line 15

Quoted 1 line
arch/powerpc/include/asm/ultravisor.h +0 -0
 /* Internal functions */

How is it internal?

arch/powerpc/include/asm/ultravisor.h line 20

Quoted 5 lines
arch/powerpc/include/asm/ultravisor.h +2 -0
 extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
 					 int depth, void *data);
 
+/* API functions */
+#define UCALL_BUFSIZE 4

Please don't copy this design from the hcall code, it has led to bugs in the past.

See my (still unmerged) attempt to fix this for the hcall case:
https://patchwork.ozlabs.org/patch/683577/

Basically instead of asking callers nicely to define a certain sized buffer, and them forgetting, define a proper type that has the right size.

arch/powerpc/include/asm/ultravisor.h line 29

Quoted 9 lines
arch/powerpc/include/asm/ultravisor.h +9 -0
+/**
+ * ucall: Make a powerpc ultravisor call.
+ * @opcode: The ultravisor call to make.
+ * @retbuf: Buffer to store up to 4 return arguments in.
+ *
+ * This call supports up to 6 arguments and 4 return arguments. Use
+ * UCALL_BUFSIZE to size the return argument buffer.
+ */
+#if defined(CONFIG_PPC_POWERNV)

#ifdef

Quoted 11 lines
arch/powerpc/include/asm/ultravisor.h +5 -0
+long ucall(unsigned long opcode, unsigned long *retbuf, ...);
+#endif
+
+#endif /* !__ASSEMBLY__ */
+
 #endif	/* _ASM_POWERPC_ULTRAVISOR_H */
arch/powerpc/kernel/Makefile +0 -0
index f0caa302c8c0..f28baccc0a79 100644

arch/powerpc/kernel/Makefile endif · line 157

Quoted 5 lines
arch/powerpc/kernel/Makefile +1 -1
@@ -154,7 +154,7 @@ endif
 
 obj-$(CONFIG_EPAPR_PARAVIRT)	+= epapr_paravirt.o epapr_hcalls.o
 obj-$(CONFIG_KVM_GUEST)		+= kvm.o kvm_emul.o
-obj-$(CONFIG_PPC_POWERNV)	+= ultravisor.o
+obj-$(CONFIG_PPC_POWERNV)	+= ultravisor.o ucall.o

Same comment about being platforms/powernv ?

Quoted 8 lines
arch/powerpc/kernel/ucall.S new file +1 -0
new file mode 100644
index 000000000000..1678f6eb7230
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */

arch/powerpc/kernel/ucall.S line 11

Quoted 10 lines
arch/powerpc/kernel/ucall.S new file +10 -0
+/*
+ * Generic code to perform an ultravisor call.
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#include <asm/ppc_asm.h>
+
+/*
+ * This function is based on the plpar_hcall()

I don't think it meaningfully is any more.

arch/powerpc/kernel/ucall.S line 13

Quoted 2 lines
arch/powerpc/kernel/ucall.S new file +2 -0
+ */
+_GLOBAL_TOC(ucall)

You don't need the TOC setup here (unless a later patch does?).

arch/powerpc/kernel/ucall.S line 20

Quoted 7 lines
arch/powerpc/kernel/ucall.S new file +7 -0
+	std	r4,STK_PARAM(R4)(r1)	/* Save ret buffer */
+	mr	r4,r5
+	mr	r5,r6
+	mr	r6,r7
+	mr	r7,r8
+	mr	r8,r9
+	mr	r9,r10

Below you space the arguments, here you don't. Pick one or the other please.

Quoted 15 lines
arch/powerpc/kernel/ucall.S new file +10 -0
+
+	sc 2				/* Invoke the ultravisor */
+
+	ld	r12,STK_PARAM(R4)(r1)
+	std	r4,  0(r12)
+	std	r5,  8(r12)
+	std	r6, 16(r12)
+	std	r7, 24(r12)
+
+	blr				/* Return r3 = status */
arch/powerpc/kernel/ultravisor.c +0 -0
index dc6021f63c97..02ddf79a9522 100644

arch/powerpc/kernel/ultravisor.c line 17

Quoted 10 lines
arch/powerpc/kernel/ultravisor.c +3 -0
@@ -8,10 +8,14 @@
 #include <linux/init.h>
 #include <linux/printk.h>
 #include <linux/string.h>
+#include <linux/export.h>
 
 #include <asm/ultravisor.h>
 #include <asm/firmware.h>
 
+/* in ucall.S */
+EXPORT_SYMBOL_GPL(ucall);

This should be in ucall.S

cheers

#16 lore raw

Re: [PATCH v4 3/8] KVM: PPC: Ultravisor: Add generic ultravisor call handler

Quoted 7 lines · Michael Ellerman
Quoted 6 lines · Claudio Carvalho

From: Ram Pai <[email protected]>

Add the ucall() function, which can be used to make ultravisor calls with varied number of in and out arguments. Ultravisor calls can be made from the host or guests.

Quoted 5 lines
Quoted 1 line

This copies the implementation of plpar_hcall().

.. with quite a few changes?

This is one of the things I'd like to see in a Documentation file, so that people can review the implementation vs the specification.

I will document this (and other things) in a file under Documentation/powerpc.

Quoted 67 lines
Quoted 5 lines

[ Change ucall.S to not save CR, rename and move headers, build ucall.S
if CONFIG_PPC_POWERNV set, use R3 for the ucall number and add some comments in the code ]

Why are we not saving CR? See previous comment about Documentation :)

Quoted 24 lines
---
 arch/powerpc/include/asm/ultravisor-api.h | 20 +++++++++++++++
 arch/powerpc/include/asm/ultravisor.h     | 20 +++++++++++++++
 arch/powerpc/kernel/Makefile              |  2 +-
 arch/powerpc/kernel/ucall.S               | 30 +++++++++++++++++++++++
 arch/powerpc/kernel/ultravisor.c          |  4 +++
 5 files changed, 75 insertions(+), 1 deletion(-)
 create mode 100644 arch/powerpc/include/asm/ultravisor-api.h
 create mode 100644 arch/powerpc/kernel/ucall.S
arch/powerpc/include/asm/ultravisor-api.h new file +7 -0
new file mode 100644
index 000000000000..49e766adabc7
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Ultravisor API.
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */

arch/powerpc/include/asm/ultravisor-api.h line 17

Quoted 10 lines
arch/powerpc/include/asm/ultravisor-api.h new file +10 -0
+#ifndef _ASM_POWERPC_ULTRAVISOR_API_H
+#define _ASM_POWERPC_ULTRAVISOR_API_H
+
+#include <asm/hvcall.h>
+
+/* Return codes */
+#define U_NOT_AVAILABLE		H_NOT_AVAILABLE
+#define U_SUCCESS		H_SUCCESS
+#define U_FUNCTION		H_FUNCTION
+#define U_PARAMETER		H_PARAMETER

Is there any benefit in redefining these?

arch/powerpc/include/asm/ultravisor.h line 13

Quoted 11 lines
arch/powerpc/include/asm/ultravisor.h +3 -0
index e5009b0d84ea..a78a2dacfd0b 100644
@@ -8,8 +8,28 @@
 #ifndef _ASM_POWERPC_ULTRAVISOR_H
 #define _ASM_POWERPC_ULTRAVISOR_H
 
+#include <asm/ultravisor-api.h>
+
+#if !defined(__ASSEMBLY__)

Just #ifndef is fine.

arch/powerpc/include/asm/ultravisor.h line 15

Quoted 1 line
arch/powerpc/include/asm/ultravisor.h +0 -0
 /* Internal functions */

How is it internal?

arch/powerpc/include/asm/ultravisor.h line 20

Quoted 5 lines
arch/powerpc/include/asm/ultravisor.h +2 -0
 extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
 					 int depth, void *data);
 
+/* API functions */
+#define UCALL_BUFSIZE 4

Please don't copy this design from the hcall code, it has led to bugs in the past.

Quoted 5 lines

See my (still unmerged) attempt to fix this for the hcall case:
https://patchwork.ozlabs.org/patch/683577/

Basically instead of asking callers nicely to define a certain sized buffer, and them forgetting, define a proper type that has the right size.

I will keep that in mind. For now I think we don't need that since the v5 will have ucall_norets() instead.

Quoted 81 lines

arch/powerpc/include/asm/ultravisor.h line 29

Quoted 10 lines
arch/powerpc/include/asm/ultravisor.h +9 -0
+/**
+ * ucall: Make a powerpc ultravisor call.
+ * @opcode: The ultravisor call to make.
+ * @retbuf: Buffer to store up to 4 return arguments in.
+ *
+ * This call supports up to 6 arguments and 4 return arguments. Use
+ * UCALL_BUFSIZE to size the return argument buffer.
+ */
+#if defined(CONFIG_PPC_POWERNV)

#ifdef

Quoted 11 lines
arch/powerpc/include/asm/ultravisor.h +5 -0
+long ucall(unsigned long opcode, unsigned long *retbuf, ...);
+#endif
+
+#endif /* !__ASSEMBLY__ */
+
 #endif	/* _ASM_POWERPC_ULTRAVISOR_H */
arch/powerpc/kernel/Makefile +0 -0
index f0caa302c8c0..f28baccc0a79 100644

arch/powerpc/kernel/Makefile endif · line 157

Quoted 5 lines
arch/powerpc/kernel/Makefile +1 -1
@@ -154,7 +154,7 @@ endif
 
 obj-$(CONFIG_EPAPR_PARAVIRT)	+= epapr_paravirt.o epapr_hcalls.o
 obj-$(CONFIG_KVM_GUEST)		+= kvm.o kvm_emul.o
-obj-$(CONFIG_PPC_POWERNV)	+= ultravisor.o
+obj-$(CONFIG_PPC_POWERNV)	+= ultravisor.o ucall.o

Same comment about being platforms/powernv ?

Quoted 7 lines
arch/powerpc/kernel/ucall.S new file +1 -0
new file mode 100644
index 000000000000..1678f6eb7230
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */

arch/powerpc/kernel/ucall.S line 11

Quoted 10 lines
arch/powerpc/kernel/ucall.S new file +10 -0
+/*
+ * Generic code to perform an ultravisor call.
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#include <asm/ppc_asm.h>
+
+/*
+ * This function is based on the plpar_hcall()

I don't think it meaningfully is any more.

arch/powerpc/kernel/ucall.S line 13

Quoted 2 lines
arch/powerpc/kernel/ucall.S new file +2 -0
+ */
+_GLOBAL_TOC(ucall)

You don't need the TOC setup here (unless a later patch does?).

arch/powerpc/kernel/ucall.S line 20

Quoted 7 lines
arch/powerpc/kernel/ucall.S new file +7 -0
+	std	r4,STK_PARAM(R4)(r1)	/* Save ret buffer */
+	mr	r4,r5
+	mr	r5,r6
+	mr	r6,r7
+	mr	r7,r8
+	mr	r8,r9
+	mr	r9,r10

Below you space the arguments, here you don't. Pick one or the other please.

Quoted 20 lines
arch/powerpc/kernel/ucall.S new file +10 -0
+
+	sc 2				/* Invoke the ultravisor */
+
+	ld	r12,STK_PARAM(R4)(r1)
+	std	r4,  0(r12)
+	std	r5,  8(r12)
+	std	r6, 16(r12)
+	std	r7, 24(r12)
+
+	blr				/* Return r3 = status */
arch/powerpc/kernel/ultravisor.c +1 -0
index dc6021f63c97..02ddf79a9522 100644
@@ -8,10 +8,14 @@
 #include <linux/init.h>
 #include <linux/printk.h>
 #include <linux/string.h>
+#include <linux/export.h>
 
Quoted 6 lines

arch/powerpc/kernel/ultravisor.c line 17

Quoted 5 lines
arch/powerpc/kernel/ultravisor.c +2 -0
 #include <asm/ultravisor.h>
 #include <asm/firmware.h>
 
+/* in ucall.S */
+EXPORT_SYMBOL_GPL(ucall);

This should be in ucall.S

Here I'm following the same way that hypercall wrapper symbols are exported.

Last time I tried to export that in ucall.S the linker complained that the ucall
symbol will not be versioned. Something like this:
https://patchwork.kernel.org/patch/9452759/

Thanks,
Claudio

Quoted 2 lines

cheers

#17 lore raw

Re: [PATCH v4 3/8] KVM: PPC: Ultravisor: Add generic ultravisor call handler

Quoted 8 lines · Claudio Carvalho
Quoted 7 lines · Michael Ellerman
Quoted 6 lines · Claudio Carvalho

From: Ram Pai <[email protected]>

Add the ucall() function, which can be used to make ultravisor calls with varied number of in and out arguments. Ultravisor calls can be made from the host or guests.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

This copies the implementation of plpar_hcall().

.. with quite a few changes?

This is one of the things I'd like to see in a Documentation file, so that people can review the implementation vs the specification.

I will document this (and other things) in a file under Documentation/powerpc.

Thanks.

Quoted 5 lines
Quoted 5 lines

arch/powerpc/include/asm/ultravisor.h line 20

Quoted 2 lines
arch/powerpc/include/asm/ultravisor.h +2 -0
+/* API functions */
+#define UCALL_BUFSIZE 4

Please don't copy this design from the hcall code, it has led to bugs in the past.

Quoted 8 lines
Quoted 5 lines

See my (still unmerged) attempt to fix this for the hcall case:
https://patchwork.ozlabs.org/patch/683577/

Basically instead of asking callers nicely to define a certain sized buffer, and them forgetting, define a proper type that has the right size.

I will keep that in mind. For now I think we don't need that since the v5 will have ucall_norets() instead.

OK.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
arch/powerpc/kernel/ultravisor.c +1 -0
index dc6021f63c97..02ddf79a9522 100644
@@ -8,10 +8,14 @@
 #include <linux/init.h>
 #include <linux/printk.h>
 #include <linux/string.h>
+#include <linux/export.h>
 
Quoted 8 lines
Quoted 6 lines

arch/powerpc/kernel/ultravisor.c line 17

Quoted 5 lines
arch/powerpc/kernel/ultravisor.c +2 -0
 #include <asm/ultravisor.h>
 #include <asm/firmware.h>
 
+/* in ucall.S */
+EXPORT_SYMBOL_GPL(ucall);

This should be in ucall.S

Here I'm following the same way that hypercall wrapper symbols are exported.

Yeah we used to not be able to export from .S but that was fixed a few years ago.

Quoted 4 lines

Last time I tried to export that in ucall.S the linker complained that the ucall
symbol will not be versioned. Something like this:
https://patchwork.kernel.org/patch/9452759/

I think you just need to include <asm/export.h> in the .S ?

cheers

#18 lore raw

Re: [PATCH v4 3/8] KVM: PPC: Ultravisor: Add generic ultravisor call handler

In reply to Claudio Carvalho

Claudio Carvalho's on June 29, 2019 6:08 am:

Quoted 102 lines

From: Ram Pai <[email protected]>

Add the ucall() function, which can be used to make ultravisor calls with varied number of in and out arguments. Ultravisor calls can be made from the host or guests.

This copies the implementation of plpar_hcall().

[ Change ucall.S to not save CR, rename and move headers, build ucall.S
if CONFIG_PPC_POWERNV set, use R3 for the ucall number and add some comments in the code ]

---
 arch/powerpc/include/asm/ultravisor-api.h | 20 +++++++++++++++
 arch/powerpc/include/asm/ultravisor.h     | 20 +++++++++++++++
 arch/powerpc/kernel/Makefile              |  2 +-
 arch/powerpc/kernel/ucall.S               | 30 +++++++++++++++++++++++
 arch/powerpc/kernel/ultravisor.c          |  4 +++
 5 files changed, 75 insertions(+), 1 deletion(-)
 create mode 100644 arch/powerpc/include/asm/ultravisor-api.h
 create mode 100644 arch/powerpc/kernel/ucall.S
arch/powerpc/include/asm/ultravisor-api.h new file +20 -0
new file mode 100644
index 000000000000..49e766adabc7
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Ultravisor API.
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#ifndef _ASM_POWERPC_ULTRAVISOR_API_H
+#define _ASM_POWERPC_ULTRAVISOR_API_H
+
+#include <asm/hvcall.h>
+
+/* Return codes */
+#define U_NOT_AVAILABLE		H_NOT_AVAILABLE
+#define U_SUCCESS		H_SUCCESS
+#define U_FUNCTION		H_FUNCTION
+#define U_PARAMETER		H_PARAMETER
+
+#endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
+
arch/powerpc/include/asm/ultravisor.h +20 -0
index e5009b0d84ea..a78a2dacfd0b 100644
@@ -8,8 +8,28 @@
 #ifndef _ASM_POWERPC_ULTRAVISOR_H
 #define _ASM_POWERPC_ULTRAVISOR_H
 
+#include <asm/ultravisor-api.h>
+
+#if !defined(__ASSEMBLY__)
+
 /* Internal functions */
 extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
 					 int depth, void *data);
 
+/* API functions */
+#define UCALL_BUFSIZE 4
+/**
+ * ucall: Make a powerpc ultravisor call.
+ * @opcode: The ultravisor call to make.
+ * @retbuf: Buffer to store up to 4 return arguments in.
+ *
+ * This call supports up to 6 arguments and 4 return arguments. Use
+ * UCALL_BUFSIZE to size the return argument buffer.
+ */
+#if defined(CONFIG_PPC_POWERNV)
+long ucall(unsigned long opcode, unsigned long *retbuf, ...);
+#endif
+
+#endif /* !__ASSEMBLY__ */
+
 #endif	/* _ASM_POWERPC_ULTRAVISOR_H */
arch/powerpc/kernel/Makefile +1 -1
index f0caa302c8c0..f28baccc0a79 100644
@@ -154,7 +154,7 @@ endif
 
 obj-$(CONFIG_EPAPR_PARAVIRT)	+= epapr_paravirt.o epapr_hcalls.o
 obj-$(CONFIG_KVM_GUEST)		+= kvm.o kvm_emul.o
-obj-$(CONFIG_PPC_POWERNV)	+= ultravisor.o
+obj-$(CONFIG_PPC_POWERNV)	+= ultravisor.o ucall.o
 
 # Disable GCOV, KCOV & sanitizers in odd or sensitive code
 GCOV_PROFILE_prom_init.o := n
arch/powerpc/kernel/ucall.S new file +1 -0
new file mode 100644
index 000000000000..1678f6eb7230
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */

arch/powerpc/kernel/ucall.S line 11

Quoted 10 lines
arch/powerpc/kernel/ucall.S new file +10 -0
+/*
+ * Generic code to perform an ultravisor call.
+ *
+ * Copyright 2019, IBM Corporation.
+ *
+ */
+#include <asm/ppc_asm.h>
+
+/*
+ * This function is based on the plpar_hcall()

A better comment might be to specify calling convention.

arch/powerpc/kernel/ucall.S line 20

Quoted 9 lines
arch/powerpc/kernel/ucall.S new file +9 -0
+ */
+_GLOBAL_TOC(ucall)
+	std	r4,STK_PARAM(R4)(r1)	/* Save ret buffer */
+	mr	r4,r5
+	mr	r5,r6
+	mr	r6,r7
+	mr	r7,r8
+	mr	r8,r9
+	mr	r9,r10

I guess this is fine, is there any particular reason that you made the call convention this way rather than leaving regs in place and r4 is a scratch?

Quoted 9 lines
arch/powerpc/kernel/ucall.S new file +9 -0
+
+	sc 2				/* Invoke the ultravisor */
+
+	ld	r12,STK_PARAM(R4)(r1)
+	std	r4,  0(r12)
+	std	r5,  8(r12)
+	std	r6, 16(r12)
+	std	r7, 24(r12)
+
#19 lore raw

PATCH v4 4/8 KVM: PPC: Ultravisor: Use UV_WRITE_PATE ucall to register a PATE

In reply to Claudio Carvalho

From: Michael Anderson <[email protected]>

When running under an ultravisor, the ultravisor controls the real partition table and has it in secure memory where the hypervisor can't access it, and therefore we (the HV) have to do a ucall whenever we want to update an entry.

The HV still keeps a copy of its view of the partition table in normal memory so that the nest MMU can access it.

Both partition tables will have PATE entries for HV and normal virtual machines.

[ Write the pate in HV's table before doing that in UV's ]
---
 arch/powerpc/include/asm/ultravisor-api.h |  5 +++-
 arch/powerpc/include/asm/ultravisor.h     | 14 ++++++++++
 arch/powerpc/mm/book3s64/hash_utils.c     |  3 +-
 arch/powerpc/mm/book3s64/pgtable.c        | 34 +++++++++++++++++++++--
 arch/powerpc/mm/book3s64/radix_pgtable.c  |  9 ++++--
 5 files changed, 57 insertions(+), 8 deletions(-)
arch/powerpc/include/asm/ultravisor-api.h +4 -1
index 49e766adabc7..141940771add 100644
@@ -15,6 +15,9 @@
 #define U_SUCCESS		H_SUCCESS
 #define U_FUNCTION		H_FUNCTION
 #define U_PARAMETER		H_PARAMETER
+#define U_PERMISSION		H_PERMISSION
 
-#endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
+/* opcodes */
+#define UV_WRITE_PATE			0xF104
 
+#endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
arch/powerpc/include/asm/ultravisor.h +14 -0
index a78a2dacfd0b..996c1efd6c6d 100644
@@ -12,6 +12,8 @@
 
 #if !defined(__ASSEMBLY__)
 
+#include <linux/types.h>
+
 /* Internal functions */
 extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
 					 int depth, void *data);
@@ -28,8 +30,20 @@ extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
  */
 #if defined(CONFIG_PPC_POWERNV)
 long ucall(unsigned long opcode, unsigned long *retbuf, ...);
+#else
+static long ucall(unsigned long opcode, unsigned long *retbuf, ...)
+{
+	return U_NOT_AVAILABLE;
+}
 #endif
 
+static inline int uv_register_pate(u64 lpid, u64 dw0, u64 dw1)
+{
+	unsigned long retbuf[UCALL_BUFSIZE];
+
+	return ucall(UV_WRITE_PATE, retbuf, lpid, dw0, dw1);
+}
+
 #endif /* !__ASSEMBLY__ */
 
 #endif	/* _ASM_POWERPC_ULTRAVISOR_H */
arch/powerpc/mm/book3s64/hash_utils.c +2 -1
index 1ff451892d7f..220a4e133240 100644
@@ -1080,9 +1080,10 @@ void hash__early_init_mmu_secondary(void)
 
 		if (!cpu_has_feature(CPU_FTR_ARCH_300))
 			mtspr(SPRN_SDR1, _SDR1);
-		else
+		else if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
 			mtspr(SPRN_PTCR,
 			      __pa(partition_tb) | (PATB_SIZE_SHIFT - 12));
+
 	}
 	/* Initialize SLB */
 	slb_initialize();
arch/powerpc/mm/book3s64/pgtable.c +31 -3
index ad3dd977c22d..224c5c7c2e3d 100644
@@ -16,6 +16,8 @@
 #include <asm/tlb.h>
 #include <asm/trace.h>
 #include <asm/powernv.h>
+#include <asm/firmware.h>
+#include <asm/ultravisor.h>
 
 #include <mm/mmu_decl.h>
 #include <trace/events/thp.h>
@@ -206,12 +208,25 @@ void __init mmu_partition_table_init(void)
 	 * 64 K size.
 	 */
 	ptcr = __pa(partition_tb) | (PATB_SIZE_SHIFT - 12);
-	mtspr(SPRN_PTCR, ptcr);
+	/*
+	 * If ultravisor is available, it is responsible for creating and
+	 * managing partition table
+	 */
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		mtspr(SPRN_PTCR, ptcr);
+
+	/*
+	 * Since nestMMU cannot access secure memory. Create
+	 * and manage our own partition table. This table
+	 * contains entries for nonsecure and hypervisor
+	 * partition.
+	 */
 	powernv_set_nmmu_ptcr(ptcr);
 }
 
-void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
-				   unsigned long dw1)
+static void __mmu_partition_table_set_entry(unsigned int lpid,
+					    unsigned long dw0,
+					    unsigned long dw1)
 {
 	unsigned long old = be64_to_cpu(partition_tb[lpid].patb0);
 
@@ -238,6 +253,19 @@ void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
 	/* do we need fixup here ?*/
 	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
 }
+
+void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
+				  unsigned long dw1)
+{
+	__mmu_partition_table_set_entry(lpid, dw0, dw1);
+
+	if (firmware_has_feature(FW_FEATURE_ULTRAVISOR)) {
+		uv_register_pate(lpid, dw0, dw1);
+		pr_info("PATE registered by ultravisor: dw0 = 0x%lx, dw1 = 0x%lx\n",
+			dw0, dw1);
+	}
+}
+
 EXPORT_SYMBOL_GPL(mmu_partition_table_set_entry);
 
 static pmd_t *get_pmd_from_cache(struct mm_struct *mm)
arch/powerpc/mm/book3s64/radix_pgtable.c +6 -3
index 8904aa1243d8..da6a6b76a040 100644
@@ -656,8 +656,10 @@ void radix__early_init_mmu_secondary(void)
 		lpcr = mfspr(SPRN_LPCR);
 		mtspr(SPRN_LPCR, lpcr | LPCR_UPRT | LPCR_HR);
 
-		mtspr(SPRN_PTCR,
-		      __pa(partition_tb) | (PATB_SIZE_SHIFT - 12));
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			mtspr(SPRN_PTCR, __pa(partition_tb) |
+			      (PATB_SIZE_SHIFT - 12));
+
 		radix_init_amor();
 	}
 
@@ -673,7 +675,8 @@ void radix__mmu_cleanup_all(void)
 	if (!firmware_has_feature(FW_FEATURE_LPAR)) {
 		lpcr = mfspr(SPRN_LPCR);
 		mtspr(SPRN_LPCR, lpcr & ~LPCR_UPRT);
-		mtspr(SPRN_PTCR, 0);
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			mtspr(SPRN_PTCR, 0);
 		powernv_set_nmmu_ptcr(0);
 		radix__flush_tlb_all();
 	}
Signature · 1 line
2.20.1
Reviewed-by on v4 4/8
#20 lore raw

Re: [PATCH v4 4/8] KVM: PPC: Ultravisor: Use UV_WRITE_PATE ucall to register a PATE

Quoted 14 lines · Claudio Carvalho

From: Michael Anderson <[email protected]>

When running under an ultravisor, the ultravisor controls the real partition table and has it in secure memory where the hypervisor can't access it, and therefore we (the HV) have to do a ucall whenever we want
to update an entry.

The HV still keeps a copy of its view of the partition table in normal memory so that the nest MMU can access it.

Both partition tables will have PATE entries for HV and normal virtual machines.

Quoted 6 lines
[ Write the pate in HV's table before doing that in UV's ]
Quoted 172 lines
---
 arch/powerpc/include/asm/ultravisor-api.h |  5 +++-
 arch/powerpc/include/asm/ultravisor.h     | 14 ++++++++++
 arch/powerpc/mm/book3s64/hash_utils.c     |  3 +-
 arch/powerpc/mm/book3s64/pgtable.c        | 34 +++++++++++++++++++++--
 arch/powerpc/mm/book3s64/radix_pgtable.c  |  9 ++++--
 5 files changed, 57 insertions(+), 8 deletions(-)
arch/powerpc/include/asm/ultravisor-api.h +4 -1
index 49e766adabc7..141940771add 100644
@@ -15,6 +15,9 @@
 #define U_SUCCESS		H_SUCCESS
 #define U_FUNCTION		H_FUNCTION
 #define U_PARAMETER		H_PARAMETER
+#define U_PERMISSION		H_PERMISSION
 
-#endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
+/* opcodes */
+#define UV_WRITE_PATE			0xF104
 
+#endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
arch/powerpc/include/asm/ultravisor.h +2 -0
index a78a2dacfd0b..996c1efd6c6d 100644
@@ -12,6 +12,8 @@
 
 #if !defined(__ASSEMBLY__)
 
+#include <linux/types.h>
+
 /* Internal functions */
 extern int early_init_dt_scan_ultravisor(unsigned long node, const 
 char *uname,
arch/powerpc/include/asm/ultravisor.h +12 -0
 					 int depth, void *data);
@@ -28,8 +30,20 @@ extern int early_init_dt_scan_ultravisor(unsigned
 long node, const char *uname,
  */
 #if defined(CONFIG_PPC_POWERNV)
 long ucall(unsigned long opcode, unsigned long *retbuf, ...);
+#else
+static long ucall(unsigned long opcode, unsigned long *retbuf, ...)
+{
+	return U_NOT_AVAILABLE;
+}
 #endif
 
+static inline int uv_register_pate(u64 lpid, u64 dw0, u64 dw1)
+{
+	unsigned long retbuf[UCALL_BUFSIZE];
+
+	return ucall(UV_WRITE_PATE, retbuf, lpid, dw0, dw1);
+}
+
 #endif /* !__ASSEMBLY__ */
 

#endif /* _ASM_POWERPC_ULTRAVISOR_H */

arch/powerpc/mm/book3s64/hash_utils.c +2 -1
index 1ff451892d7f..220a4e133240 100644
@@ -1080,9 +1080,10 @@ void hash__early_init_mmu_secondary(void)
 
 		if (!cpu_has_feature(CPU_FTR_ARCH_300))
 			mtspr(SPRN_SDR1, _SDR1);
-		else
+		else if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
 			mtspr(SPRN_PTCR,
 			      __pa(partition_tb) | (PATB_SIZE_SHIFT - 12));
+
 	}
 	/* Initialize SLB */
 	slb_initialize();
arch/powerpc/mm/book3s64/pgtable.c +31 -3
index ad3dd977c22d..224c5c7c2e3d 100644
@@ -16,6 +16,8 @@
 #include <asm/tlb.h>
 #include <asm/trace.h>
 #include <asm/powernv.h>
+#include <asm/firmware.h>
+#include <asm/ultravisor.h>
 
 #include <mm/mmu_decl.h>
 #include <trace/events/thp.h>
@@ -206,12 +208,25 @@ void __init mmu_partition_table_init(void)
 	 * 64 K size.
 	 */
 	ptcr = __pa(partition_tb) | (PATB_SIZE_SHIFT - 12);
-	mtspr(SPRN_PTCR, ptcr);
+	/*
+	 * If ultravisor is available, it is responsible for creating and
+	 * managing partition table
+	 */
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		mtspr(SPRN_PTCR, ptcr);
+
+	/*
+	 * Since nestMMU cannot access secure memory. Create
+	 * and manage our own partition table. This table
+	 * contains entries for nonsecure and hypervisor
+	 * partition.
+	 */
 	powernv_set_nmmu_ptcr(ptcr);
 }
 
-void mmu_partition_table_set_entry(unsigned int lpid, unsigned long 
 dw0,
-				   unsigned long dw1)
+static void __mmu_partition_table_set_entry(unsigned int lpid,
+					    unsigned long dw0,
+					    unsigned long dw1)
 {
 	unsigned long old = be64_to_cpu(partition_tb[lpid].patb0);
@@ -238,6 +253,19 @@ void mmu_partition_table_set_entry(unsigned int
 lpid, unsigned long dw0,
 	/* do we need fixup here ?*/
 	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
 }
+
+void mmu_partition_table_set_entry(unsigned int lpid, unsigned long 
 dw0,
+				  unsigned long dw1)
+{
+	__mmu_partition_table_set_entry(lpid, dw0, dw1);
+
+	if (firmware_has_feature(FW_FEATURE_ULTRAVISOR)) {
+		uv_register_pate(lpid, dw0, dw1);
+		pr_info("PATE registered by ultravisor: dw0 = 0x%lx, dw1 = 0x%lx\n",
+			dw0, dw1);
+	}
+}
+
 EXPORT_SYMBOL_GPL(mmu_partition_table_set_entry);

static pmd_t *get_pmd_from_cache(struct mm_struct *mm)

arch/powerpc/mm/book3s64/radix_pgtable.c +6 -3
index 8904aa1243d8..da6a6b76a040 100644
@@ -656,8 +656,10 @@ void radix__early_init_mmu_secondary(void)
 		lpcr = mfspr(SPRN_LPCR);
 		mtspr(SPRN_LPCR, lpcr | LPCR_UPRT | LPCR_HR);
 
-		mtspr(SPRN_PTCR,
-		      __pa(partition_tb) | (PATB_SIZE_SHIFT - 12));
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			mtspr(SPRN_PTCR, __pa(partition_tb) |
+			      (PATB_SIZE_SHIFT - 12));
+
 		radix_init_amor();
 	}
 
@@ -673,7 +675,8 @@ void radix__mmu_cleanup_all(void)
 	if (!firmware_has_feature(FW_FEATURE_LPAR)) {
 		lpcr = mfspr(SPRN_LPCR);
 		mtspr(SPRN_LPCR, lpcr & ~LPCR_UPRT);
-		mtspr(SPRN_PTCR, 0);
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			mtspr(SPRN_PTCR, 0);
 		powernv_set_nmmu_ptcr(0);
 		radix__flush_tlb_all();
 	}
#21 lore raw

Re: [PATCH v4 4/8] KVM: PPC: Ultravisor: Use UV_WRITE_PATE ucall to register a PATE

In reply to Claudio Carvalho
Quoted 11 lines · Claudio Carvalho

From: Michael Anderson <[email protected]>

When running under an ultravisor, the ultravisor controls the real partition table and has it in secure memory where the hypervisor can't access it, and therefore we (the HV) have to do a ucall whenever we want to update an entry.

The HV still keeps a copy of its view of the partition table in normal memory so that the nest MMU can access it.

Both partition tables will have PATE entries for HV and normal virtual

Can you expand novel acronyms on their first usage please, ie. PATE.

Quoted 3 lines

machines.

"Suggested" implies this is optional, but it's not :)

I'm not sure exactly what Ryan's input was, but feel free to make up a better tag :)

Quoted 18 lines
[ Write the pate in HV's table before doing that in UV's ]
---
 arch/powerpc/include/asm/ultravisor-api.h |  5 +++-
 arch/powerpc/include/asm/ultravisor.h     | 14 ++++++++++
 arch/powerpc/mm/book3s64/hash_utils.c     |  3 +-
 arch/powerpc/mm/book3s64/pgtable.c        | 34 +++++++++++++++++++++--
 arch/powerpc/mm/book3s64/radix_pgtable.c  |  9 ++++--
 5 files changed, 57 insertions(+), 8 deletions(-)
arch/powerpc/include/asm/ultravisor-api.h +0 -0
index 49e766adabc7..141940771add 100644

arch/powerpc/include/asm/ultravisor-api.h line 19

Quoted 6 lines
arch/powerpc/include/asm/ultravisor-api.h +1 -1
@@ -15,6 +15,9 @@
 #define U_SUCCESS		H_SUCCESS
 #define U_FUNCTION		H_FUNCTION
 #define U_PARAMETER		H_PARAMETER
+#define U_PERMISSION		H_PERMISSION
 
-#endif /* _ASM_POWERPC_ULTRAVISOR_API_H */

What happened there? Just a diff artifact?

Quoted 18 lines
arch/powerpc/include/asm/ultravisor-api.h +3 -0
+/* opcodes */
+#define UV_WRITE_PATE			0xF104
 
+#endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
arch/powerpc/include/asm/ultravisor.h +2 -0
index a78a2dacfd0b..996c1efd6c6d 100644
@@ -12,6 +12,8 @@
 
 #if !defined(__ASSEMBLY__)
 
+#include <linux/types.h>
+
 /* Internal functions */
 extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
 					 int depth, void *data);

arch/powerpc/include/asm/ultravisor.h extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname, · line 34

Quoted 5 lines
arch/powerpc/include/asm/ultravisor.h +2 -0
@@ -28,8 +30,20 @@ extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
  */
 #if defined(CONFIG_PPC_POWERNV)
 long ucall(unsigned long opcode, unsigned long *retbuf, ...);
+#else
+static long ucall(unsigned long opcode, unsigned long *retbuf, ...)
          ^
          inline

arch/powerpc/include/asm/ultravisor.h extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname, · line 38

Quoted 4 lines
arch/powerpc/include/asm/ultravisor.h +3 -0
+{
+	return U_NOT_AVAILABLE;
+}
 #endif

That should have been in the previous patch.

arch/powerpc/include/asm/ultravisor.h extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname, · line 44

Quoted 5 lines
arch/powerpc/include/asm/ultravisor.h +5 -0
+static inline int uv_register_pate(u64 lpid, u64 dw0, u64 dw1)
+{
+	unsigned long retbuf[UCALL_BUFSIZE];
+
+	return ucall(UV_WRITE_PATE, retbuf, lpid, dw0, dw1);

You probably want a ucall_norets().

Quoted 10 lines
arch/powerpc/include/asm/ultravisor.h +2 -0
+}
+
 #endif /* !__ASSEMBLY__ */
 
 #endif	/* _ASM_POWERPC_ULTRAVISOR_H */
arch/powerpc/mm/book3s64/hash_utils.c +0 -0
index 1ff451892d7f..220a4e133240 100644

arch/powerpc/mm/book3s64/hash_utils.c void hash__early_init_mmu_secondary(void) · line 1085

Quoted 7 lines
arch/powerpc/mm/book3s64/hash_utils.c +1 -1
@@ -1080,9 +1080,10 @@ void hash__early_init_mmu_secondary(void)
 
 		if (!cpu_has_feature(CPU_FTR_ARCH_300))
 			mtspr(SPRN_SDR1, _SDR1);
-		else
+		else if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
 			mtspr(SPRN_PTCR,
 			      __pa(partition_tb) | (PATB_SIZE_SHIFT - 12));

Needs a comment for the else case.

Quoted 17 lines
arch/powerpc/mm/book3s64/hash_utils.c +0 -0
 	}
 	/* Initialize SLB */
 	slb_initialize();
arch/powerpc/mm/book3s64/pgtable.c +2 -0
index ad3dd977c22d..224c5c7c2e3d 100644
@@ -16,6 +16,8 @@
 #include <asm/tlb.h>
 #include <asm/trace.h>
 #include <asm/powernv.h>
+#include <asm/firmware.h>
+#include <asm/ultravisor.h>
 
 #include <mm/mmu_decl.h>
 #include <trace/events/thp.h>

arch/powerpc/mm/book3s64/pgtable.c void __init mmu_partition_table_init(void) · line 214

Quoted 8 lines
arch/powerpc/mm/book3s64/pgtable.c +4 -1
@@ -206,12 +208,25 @@ void __init mmu_partition_table_init(void)
 	 * 64 K size.
 	 */
 	ptcr = __pa(partition_tb) | (PATB_SIZE_SHIFT - 12);
-	mtspr(SPRN_PTCR, ptcr);
+	/*
+	 * If ultravisor is available, it is responsible for creating and
+	 * managing partition table
+	 */

But we just created the partition table?!

This comment and the one below would probably make more sense if they were merged and appeared further up, where we allocate the partition table.

arch/powerpc/mm/book3s64/pgtable.c void __init mmu_partition_table_init(void) · line 220

Quoted 6 lines
arch/powerpc/mm/book3s64/pgtable.c +6 -0
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		mtspr(SPRN_PTCR, ptcr);
+
+	/*
+	 * Since nestMMU cannot access secure memory. Create
+	 * and manage our own partition table. This table

But we just said the UV was responsible for it :)

Quoted 20 lines
arch/powerpc/mm/book3s64/pgtable.c +8 -2
+	 * contains entries for nonsecure and hypervisor
+	 * partition.
+	 */
 	powernv_set_nmmu_ptcr(ptcr);
 }
 
-void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
-				   unsigned long dw1)
+static void __mmu_partition_table_set_entry(unsigned int lpid,
+					    unsigned long dw0,
+					    unsigned long dw1)
 {
 	unsigned long old = be64_to_cpu(partition_tb[lpid].patb0);
 
@@ -238,6 +253,19 @@ void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
 	/* do we need fixup here ?*/
 	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
 }
+
+void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,

arch/powerpc/mm/book3s64/pgtable.c void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0, · line 267

Quoted 10 lines
arch/powerpc/mm/book3s64/pgtable.c +10 -0
+				  unsigned long dw1)
+{
+	__mmu_partition_table_set_entry(lpid, dw0, dw1);
+
+	if (firmware_has_feature(FW_FEATURE_ULTRAVISOR)) {
+		uv_register_pate(lpid, dw0, dw1);
+		pr_info("PATE registered by ultravisor: dw0 = 0x%lx, dw1 = 0x%lx\n",
+			dw0, dw1);
+	}
+}

I agree with Alexey that this patch and the next should be merged and the result cleaned up a bit.

arch/powerpc/mm/book3s64/pgtable.c void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0, · line 268

Quoted 1 line
arch/powerpc/mm/book3s64/pgtable.c +1 -0
+

No extra blank please.

Quoted 22 lines
arch/powerpc/mm/book3s64/pgtable.c +0 -0
 EXPORT_SYMBOL_GPL(mmu_partition_table_set_entry);
 
 

static pmd_t *get_pmd_from_cache(struct mm_struct *mm)

arch/powerpc/mm/book3s64/radix_pgtable.c +4 -2
index 8904aa1243d8..da6a6b76a040 100644
@@ -656,8 +656,10 @@ void radix__early_init_mmu_secondary(void)
 		lpcr = mfspr(SPRN_LPCR);
 		mtspr(SPRN_LPCR, lpcr | LPCR_UPRT | LPCR_HR);
 
-		mtspr(SPRN_PTCR,
-		      __pa(partition_tb) | (PATB_SIZE_SHIFT - 12));
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			mtspr(SPRN_PTCR, __pa(partition_tb) |
+			      (PATB_SIZE_SHIFT - 12));
+
 		radix_init_amor();
 	}
 

arch/powerpc/mm/book3s64/radix_pgtable.c void radix__mmu_cleanup_all(void) · line 682

Quoted 9 lines
arch/powerpc/mm/book3s64/radix_pgtable.c +2 -1
@@ -673,7 +675,8 @@ void radix__mmu_cleanup_all(void)
 	if (!firmware_has_feature(FW_FEATURE_LPAR)) {
 		lpcr = mfspr(SPRN_LPCR);
 		mtspr(SPRN_LPCR, lpcr & ~LPCR_UPRT);
-		mtspr(SPRN_PTCR, 0);
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			mtspr(SPRN_PTCR, 0);
 		powernv_set_nmmu_ptcr(0);
 		radix__flush_tlb_all();
 	}

There's four of these case where we skip touching the PTCR, which is right on the borderline of warranting an accessor. I guess we can do it as a cleanup later.

cheers

Reviewed-by
#22 lore raw

Re: [PATCH v4 4/8] KVM: PPC: Ultravisor: Use UV_WRITE_PATE ucall to register a PATE

Quoted 10 lines · Michael Ellerman
Quoted 9 lines · Claudio Carvalho

From: Michael Anderson <[email protected]>

When running under an ultravisor, the ultravisor controls the real partition table and has it in secure memory where the hypervisor can't
access it, and therefore we (the HV) have to do a ucall whenever we want
to update an entry.

Quoted 9 lines
Quoted 6 lines

The HV still keeps a copy of its view of the partition table in normal
memory so that the nest MMU can access it.

Both partition tables will have PATE entries for HV and normal virtual

Can you expand novel acronyms on their first usage please, ie. PATE.

Agreed. This confused me a while ago. It is "Partition Table Entry", correct?

Quoted 4 lines
Quoted 3 lines

machines.

Please remove this and add

Quoted 228 lines

"Suggested" implies this is optional, but it's not :)

I'm not sure exactly what Ryan's input was, but feel free to make up a
better tag :)

Quoted 20 lines
[ Write the pate in HV's table before doing that in UV's ]
---
 arch/powerpc/include/asm/ultravisor-api.h |  5 +++-
 arch/powerpc/include/asm/ultravisor.h     | 14 ++++++++++
 arch/powerpc/mm/book3s64/hash_utils.c     |  3 +-
 arch/powerpc/mm/book3s64/pgtable.c        | 34
+++++++++++++++++++++--
 arch/powerpc/mm/book3s64/radix_pgtable.c  |  9 ++++--
 5 files changed, 57 insertions(+), 8 deletions(-)
arch/powerpc/include/asm/ultravisor-api.h +0 -0
index 49e766adabc7..141940771add 100644

arch/powerpc/include/asm/ultravisor-api.h line 19

Quoted 6 lines
arch/powerpc/include/asm/ultravisor-api.h +1 -1
@@ -15,6 +15,9 @@
 #define U_SUCCESS		H_SUCCESS
 #define U_FUNCTION		H_FUNCTION
 #define U_PARAMETER		H_PARAMETER
+#define U_PERMISSION		H_PERMISSION
 
-#endif /* _ASM_POWERPC_ULTRAVISOR_API_H */

What happened there? Just a diff artifact?

Quoted 27 lines
arch/powerpc/include/asm/ultravisor-api.h +3 -0
+/* opcodes */
+#define UV_WRITE_PATE			0xF104
 
+#endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
arch/powerpc/include/asm/ultravisor.h +2 -0
index a78a2dacfd0b..996c1efd6c6d 100644
@@ -12,6 +12,8 @@
 
 #if !defined(__ASSEMBLY__)
 
+#include <linux/types.h>
+
 /* Internal functions */
 extern int early_init_dt_scan_ultravisor(unsigned long node, const
 char *uname,
arch/powerpc/include/asm/ultravisor.h +0 -0
 					 int depth, void *data);
@@ -28,8 +30,20 @@ extern int

early_init_dt_scan_ultravisor(unsigned long node, const char *uname,
*/
#if defined(CONFIG_PPC_POWERNV)
long ucall(unsigned long opcode, unsigned long *retbuf, ...);

arch/powerpc/include/asm/ultravisor.h +2 -0
+#else
+static long ucall(unsigned long opcode, unsigned long *retbuf,
Quoted 1 line

...)

          ^
          inline

arch/powerpc/include/asm/ultravisor.h extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname, · line 38

Quoted 4 lines
arch/powerpc/include/asm/ultravisor.h +3 -0
+{
+	return U_NOT_AVAILABLE;
+}
 #endif

That should have been in the previous patch.

arch/powerpc/include/asm/ultravisor.h extern int early_init_dt_scan_ultravisor(unsigned long node, const char *uname, · line 44

Quoted 5 lines
arch/powerpc/include/asm/ultravisor.h +5 -0
+static inline int uv_register_pate(u64 lpid, u64 dw0, u64 dw1)
+{
+	unsigned long retbuf[UCALL_BUFSIZE];
+
+	return ucall(UV_WRITE_PATE, retbuf, lpid, dw0, dw1);

You probably want a ucall_norets().

Quoted 11 lines
arch/powerpc/include/asm/ultravisor.h +2 -0
+}
+
 #endif /* !__ASSEMBLY__ */
 
 #endif	/* _ASM_POWERPC_ULTRAVISOR_H */
arch/powerpc/mm/book3s64/hash_utils.c +0 -0
index 1ff451892d7f..220a4e133240 100644

arch/powerpc/mm/book3s64/hash_utils.c void hash__early_init_mmu_secondary(void) · line 1086

Quoted 8 lines
arch/powerpc/mm/book3s64/hash_utils.c +1 -1
@@ -1080,9 +1080,10 @@ void hash__early_init_mmu_secondary(void)
 
 		if (!cpu_has_feature(CPU_FTR_ARCH_300))
 			mtspr(SPRN_SDR1, _SDR1);
-		else
+		else if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
 			mtspr(SPRN_PTCR,
 			      __pa(partition_tb) | (PATB_SIZE_SHIFT -
 12));

Needs a comment for the else case.

Quoted 25 lines
arch/powerpc/mm/book3s64/hash_utils.c +0 -0
 	}
 	/* Initialize SLB */
 	slb_initialize();
arch/powerpc/mm/book3s64/pgtable.c +4 -1
index ad3dd977c22d..224c5c7c2e3d 100644
@@ -16,6 +16,8 @@
 #include <asm/tlb.h>
 #include <asm/trace.h>
 #include <asm/powernv.h>
+#include <asm/firmware.h>
+#include <asm/ultravisor.h>
 
 #include <mm/mmu_decl.h>
 #include <trace/events/thp.h>
@@ -206,12 +208,25 @@ void __init mmu_partition_table_init(void)
 	 * 64 K size.
 	 */
 	ptcr = __pa(partition_tb) | (PATB_SIZE_SHIFT - 12);
-	mtspr(SPRN_PTCR, ptcr);
+	/*
+	 * If ultravisor is available, it is responsible for creating

and

arch/powerpc/mm/book3s64/pgtable.c

Quoted 2 lines
arch/powerpc/mm/book3s64/pgtable.c +2 -0
+	 * managing partition table
+	 */

But we just created the partition table?!

This comment and the one below would probably make more sense if they were merged and appeared further up, where we allocate the partition table.

arch/powerpc/mm/book3s64/pgtable.c void __init mmu_partition_table_init(void) · line 220

Quoted 6 lines
arch/powerpc/mm/book3s64/pgtable.c +6 -0
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		mtspr(SPRN_PTCR, ptcr);
+
+	/*
+	 * Since nestMMU cannot access secure memory. Create
+	 * and manage our own partition table. This table

But we just said the UV was responsible for it :)

Quoted 24 lines
arch/powerpc/mm/book3s64/pgtable.c +3 -1
+	 * contains entries for nonsecure and hypervisor
+	 * partition.
+	 */
 	powernv_set_nmmu_ptcr(ptcr);
 }
 
-void mmu_partition_table_set_entry(unsigned int lpid, unsigned

long dw0,

arch/powerpc/mm/book3s64/pgtable.c +6 -1
-				   unsigned long dw1)
+static void __mmu_partition_table_set_entry(unsigned int lpid,
+					    unsigned long dw0,
+					    unsigned long dw1)
 {
 	unsigned long old = be64_to_cpu(partition_tb[lpid].patb0);
@@ -238,6 +253,19 @@ void mmu_partition_table_set_entry(unsigned
 int lpid, unsigned long dw0,
 	/* do we need fixup here ?*/
 	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
 }
+
+void mmu_partition_table_set_entry(unsigned int lpid, unsigned
 long dw0,
+				  unsigned long dw1)

arch/powerpc/mm/book3s64/pgtable.c void mmu_partition_table_set_entry(unsigned · line 270

Quoted 10 lines
arch/powerpc/mm/book3s64/pgtable.c +9 -0
+{
+	__mmu_partition_table_set_entry(lpid, dw0, dw1);
+
+	if (firmware_has_feature(FW_FEATURE_ULTRAVISOR)) {
+		uv_register_pate(lpid, dw0, dw1);
+		pr_info("PATE registered by ultravisor: dw0 = 0x%lx,
 dw1 = 0x%lx\n",
+			dw0, dw1);
+	}
+}

I agree with Alexey that this patch and the next should be merged and the result cleaned up a bit.

arch/powerpc/mm/book3s64/pgtable.c void mmu_partition_table_set_entry(unsigned · line 271

Quoted 1 line
arch/powerpc/mm/book3s64/pgtable.c +1 -0
+

No extra blank please.

Quoted 22 lines

EXPORT_SYMBOL_GPL(mmu_partition_table_set_entry);

static pmd_t *get_pmd_from_cache(struct mm_struct *mm)

arch/powerpc/mm/book3s64/radix_pgtable.c +4 -2
index 8904aa1243d8..da6a6b76a040 100644
@@ -656,8 +656,10 @@ void radix__early_init_mmu_secondary(void)
 		lpcr = mfspr(SPRN_LPCR);
 		mtspr(SPRN_LPCR, lpcr | LPCR_UPRT | LPCR_HR);
 
-		mtspr(SPRN_PTCR,
-		      __pa(partition_tb) | (PATB_SIZE_SHIFT - 12));
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			mtspr(SPRN_PTCR, __pa(partition_tb) |
+			      (PATB_SIZE_SHIFT - 12));
+
 		radix_init_amor();
 	}
 

arch/powerpc/mm/book3s64/radix_pgtable.c void radix__mmu_cleanup_all(void) · line 682

Quoted 9 lines
arch/powerpc/mm/book3s64/radix_pgtable.c +2 -1
@@ -673,7 +675,8 @@ void radix__mmu_cleanup_all(void)
 	if (!firmware_has_feature(FW_FEATURE_LPAR)) {
 		lpcr = mfspr(SPRN_LPCR);
 		mtspr(SPRN_LPCR, lpcr & ~LPCR_UPRT);
-		mtspr(SPRN_PTCR, 0);
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			mtspr(SPRN_PTCR, 0);
 		powernv_set_nmmu_ptcr(0);
 		radix__flush_tlb_all();
 	}

There's four of these case where we skip touching the PTCR, which is right on the borderline of warranting an accessor. I guess we can do it
as a cleanup later.

cheers

#23 lore raw

Re: [PATCH v4 4/8] KVM: PPC: Ultravisor: Use UV_WRITE_PATE ucall to register a PATE

In reply to Michael Ellerman
Quoted 20 lines · Michael Ellerman
Quoted 20 lines

static pmd_t *get_pmd_from_cache(struct mm_struct *mm)

arch/powerpc/mm/book3s64/radix_pgtable.c +4 -2
index 8904aa1243d8..da6a6b76a040 100644
@@ -656,8 +656,10 @@ void radix__early_init_mmu_secondary(void)
 		lpcr = mfspr(SPRN_LPCR);
 		mtspr(SPRN_LPCR, lpcr | LPCR_UPRT | LPCR_HR);
 
-		mtspr(SPRN_PTCR,
-		      __pa(partition_tb) | (PATB_SIZE_SHIFT - 12));
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			mtspr(SPRN_PTCR, __pa(partition_tb) |
+			      (PATB_SIZE_SHIFT - 12));
+
 		radix_init_amor();
 	}
 
Quoted 13 lines

arch/powerpc/mm/book3s64/radix_pgtable.c void radix__mmu_cleanup_all(void) · line 682

Quoted 10 lines
arch/powerpc/mm/book3s64/radix_pgtable.c +2 -1
@@ -673,7 +675,8 @@ void radix__mmu_cleanup_all(void)
 	if (!firmware_has_feature(FW_FEATURE_LPAR)) {
 		lpcr = mfspr(SPRN_LPCR);
 		mtspr(SPRN_LPCR, lpcr & ~LPCR_UPRT);
-		mtspr(SPRN_PTCR, 0);
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			mtspr(SPRN_PTCR, 0);
 		powernv_set_nmmu_ptcr(0);
 		radix__flush_tlb_all();
 	}

There's four of these case where we skip touching the PTCR, which is right on the borderline of warranting an accessor. I guess we can do it as a cleanup later.

I agree.

Since the kernel doesn't need to access a big number of ultravisor privileged registers, maybe we can define mtspr_<reg> and mfspr_<reg> inline functions that in ultravisor.h that skip touching the register if an ultravisor is present and and the register is ultravisor privileged. Thus, we don't need to replicate comments and that also would make it easier for developers to know what are the ultravisor privileged registers.

Something like this:

arch/powerpc/include/asm/ultravisor.h +0 -0
@@ -10,10 +10,21 @@


 #include <asm/ultravisor-api.h>
 #include <asm/asm-prototypes.h>
+#include <asm/reg.h>

 int early_init_dt_scan_ultravisor(unsigned long node, const char *uname, int depth, void *data);

+static inline void mtspr_ptcr(unsigned long val)
+{
+       /*
+        * If the ultravisor firmware is present, it maintains the partition +        * table. PTCR becomes ultravisor privileged only for writing. +        */
+       if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+               mtspr(SPRN_PTCR, val);
+}
+
 static inline int uv_register_pate(u64 lpid, u64 dw0, u64 dw1) {
        return ucall_norets(UV_WRITE_PATE, lpid, dw0, dw1);

(quoted diff) +0 -0

b/arch/powerpc/mm/book3s64/pgtable.c
index e1bbc48e730f..25156f9dfde8 100644

arch/powerpc/mm/book3s64/pgtable.c +0 -0
@@ -220,7 +220,7 @@ void __init mmu_partition_table_init(void)

         * 64 K size.
         */
        ptcr = __pa(partition_tb) | (PATB_SIZE_SHIFT - 12); -       mtspr(SPRN_PTCR, ptcr);
+       mtspr_ptcr(ptcr); powernv_set_nmmu_ptcr(ptcr);
 }

What do you think?
An alternative could be to change the mtspr() and mfspr() macros as we proposed in the v1, but access to non-ultravisor privileged registers would be performance impacted because we always would need to check if the register is one of the few ultravisor registers that the kernel needs to access.

Thanks,
Claudio

Quoted 2 lines

cheers

#24 lore raw

Re: [PATCH v4 4/8] KVM: PPC: Ultravisor: Use UV_WRITE_PATE ucall to register a PATE

Quoted 55 lines · Claudio Carvalho
Quoted 18 lines · Michael Ellerman
Quoted 18 lines

static pmd_t *get_pmd_from_cache(struct mm_struct *mm)

arch/powerpc/mm/book3s64/radix_pgtable.c +4 -2
index 8904aa1243d8..da6a6b76a040 100644
@@ -656,8 +656,10 @@ void radix__early_init_mmu_secondary(void)
 		lpcr = mfspr(SPRN_LPCR);
 		mtspr(SPRN_LPCR, lpcr | LPCR_UPRT | LPCR_HR);
 
-		mtspr(SPRN_PTCR,
-		      __pa(partition_tb) | (PATB_SIZE_SHIFT - 12));
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			mtspr(SPRN_PTCR, __pa(partition_tb) |
+			      (PATB_SIZE_SHIFT - 12));
+
 		radix_init_amor();
 	}
 
Quoted 13 lines

arch/powerpc/mm/book3s64/radix_pgtable.c void radix__mmu_cleanup_all(void) · line 682

Quoted 10 lines
arch/powerpc/mm/book3s64/radix_pgtable.c +2 -1
@@ -673,7 +675,8 @@ void radix__mmu_cleanup_all(void)
 	if (!firmware_has_feature(FW_FEATURE_LPAR)) {
 		lpcr = mfspr(SPRN_LPCR);
 		mtspr(SPRN_LPCR, lpcr & ~LPCR_UPRT);
-		mtspr(SPRN_PTCR, 0);
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			mtspr(SPRN_PTCR, 0);
 		powernv_set_nmmu_ptcr(0);
 		radix__flush_tlb_all();
 	}

There's four of these case where we skip touching the PTCR, which is right on the borderline of warranting an accessor. I guess we can do it as a cleanup later.

I agree.

Since the kernel doesn't need to access a big number of ultravisor privileged registers, maybe we can define mtspr_<reg> and mfspr_<reg> inline functions that in ultravisor.h that skip touching the register if an ultravisor is present and and the register is ultravisor privileged. Thus, we don't need to replicate comments and that also would make it easier for developers to know what are the ultravisor privileged registers.

Something like this:

arch/powerpc/include/asm/ultravisor.h +0 -0
@@ -10,10 +10,21 @@


 #include <asm/ultravisor-api.h>
 #include <asm/asm-prototypes.h>
+#include <asm/reg.h>

 int early_init_dt_scan_ultravisor(unsigned long node, const char *uname, int depth, void *data);

arch/powerpc/include/asm/ultravisor.h

Quoted 9 lines
arch/powerpc/include/asm/ultravisor.h +9 -0
+static inline void mtspr_ptcr(unsigned long val)
+{
+       /*
+        * If the ultravisor firmware is present, it maintains the partition
+        * table. PTCR becomes ultravisor privileged only for writing.
+        */
+       if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+               mtspr(SPRN_PTCR, val);
+}

+

Quoted 17 lines

 static inline int uv_register_pate(u64 lpid, u64 dw0, u64 dw1) {
        return ucall_norets(UV_WRITE_PATE, lpid, dw0, dw1);

arch/powerpc/mm/book3s64/pgtable.c +1 -1
index e1bbc48e730f..25156f9dfde8 100644
@@ -220,7 +220,7 @@ void __init mmu_partition_table_init(void)
          * 64 K size.
          */
         ptcr = __pa(partition_tb) | (PATB_SIZE_SHIFT - 12);
-       mtspr(SPRN_PTCR, ptcr);
+       mtspr_ptcr(ptcr);
         powernv_set_nmmu_ptcr(ptcr);
  }
 
Quoted 1 line

What do you think?

I don't think that's actually clearer.

If the logic was always:

if (ultravisor)

     do_ucall()
  else
     mtspr()

Then a wrapper called eg. set_ptcr() would make sense.

But because in some cases you do a ucall and some you don't, I don't think it helps to hide that in an accessor like above.

That is confusing to a reader who sees all this code to setup a value and then the write to PTCR does nothing.

And in fact you didn't explain why it's OK for those cases to not do the write at all.

Quoted 5 lines

An alternative could be to change the mtspr() and mfspr() macros as we proposed in the v1, but access to non-ultravisor privileged registers would be performance impacted because we always would need to check if the register is one of the few ultravisor registers that the kernel needs to access.

Yeah that and it would be very confusing to a reader who sees:

    ptcr = ...;
    mtspr(SPRN_PTCR, ptcr);
    ...

And then they discover the mtspr does *nothing* when the Ultravisor is enabled.

cheers

#25 lore raw

PATCH v4 5/8 KVM: PPC: Ultravisor: Restrict flush of the partition tlb cache

In reply to Claudio Carvalho

From: Ram Pai <[email protected]>

Ultravisor is responsible for flushing the tlb cache, since it manages the PATE entries. Hence skip tlb flush, if the ultravisor firmware is available.

---
 arch/powerpc/mm/book3s64/pgtable.c | 33 +++++++++++++++++-------------
 1 file changed, 19 insertions(+), 14 deletions(-)
arch/powerpc/mm/book3s64/pgtable.c +19 -14
index 224c5c7c2e3d..bc8eb2bf9810 100644
@@ -224,6 +224,23 @@ void __init mmu_partition_table_init(void)
 	powernv_set_nmmu_ptcr(ptcr);
 }
 
+static void flush_partition(unsigned int lpid, unsigned long dw0)
+{
+	if (dw0 & PATB_HR) {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 1, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
+	} else {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 0) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
+	}
+	/* do we need fixup here ?*/
+	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
+}
+
 static void __mmu_partition_table_set_entry(unsigned int lpid,
 					    unsigned long dw0,
 					    unsigned long dw1)
@@ -238,20 +255,8 @@ static void __mmu_partition_table_set_entry(unsigned int lpid,
 	 * The type of flush (hash or radix) depends on what the previous
 	 * use of this partition ID was, not the new use.
 	 */
-	asm volatile("ptesync" : : : "memory");
-	if (old & PATB_HR) {
-		asm volatile(PPC_TLBIE_5(%0,%1,2,0,1) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
-	} else {
-		asm volatile(PPC_TLBIE_5(%0,%1,2,0,0) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
-	}
-	/* do we need fixup here ?*/
-	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		flush_partition(lpid, old);
 }
 
 void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
Signature · 1 line
2.20.1
#26 lore raw

Re: [PATCH v4 5/8] KVM: PPC: Ultravisor: Restrict flush of the partition tlb cache

Quoted 49 lines · Claudio Carvalho

From: Ram Pai <[email protected]>

Ultravisor is responsible for flushing the tlb cache, since it manages the PATE entries. Hence skip tlb flush, if the ultravisor firmware is available.

---
 arch/powerpc/mm/book3s64/pgtable.c | 33 +++++++++++++++++-------------
 1 file changed, 19 insertions(+), 14 deletions(-)
arch/powerpc/mm/book3s64/pgtable.c +17 -5
index 224c5c7c2e3d..bc8eb2bf9810 100644
@@ -224,6 +224,23 @@ void __init mmu_partition_table_init(void)
 	powernv_set_nmmu_ptcr(ptcr);
 }
 
+static void flush_partition(unsigned int lpid, unsigned long dw0)
+{
+	if (dw0 & PATB_HR) {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 1, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
+	} else {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 0) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
+	}
+	/* do we need fixup here ?*/
+	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
+}
+
 static void __mmu_partition_table_set_entry(unsigned int lpid,
 					    unsigned long dw0,
 					    unsigned long dw1)
@@ -238,20 +255,8 @@ static void __mmu_partition_table_set_entry(unsigned int lpid,
 	 * The type of flush (hash or radix) depends on what the previous
 	 * use of this partition ID was, not the new use.
 	 */
-	asm volatile("ptesync" : : : "memory");
-	if (old & PATB_HR) {
-		asm volatile(PPC_TLBIE_5(%0,%1,2,0,1) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : :

arch/powerpc/mm/book3s64/pgtable.c static void __mmu_partition_table_set_entry(unsigned int lpid, · line 258

Quoted 10 lines
arch/powerpc/mm/book3s64/pgtable.c +1 -9
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
-	} else {
-		asm volatile(PPC_TLBIE_5(%0,%1,2,0,0) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
-	}
-	/* do we need fixup here ?*/
-	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))

__mmu_partition_table_set_entry() checks for UV and mmu_partition_table_set_entry() (the caller) checks for UV and the whole point of having separate flush_partition() and __mmu_partition_table_set_entry() is not really clear.

4/8 and 5/8 make more sense as one patch imho.

Quoted 5 lines
arch/powerpc/mm/book3s64/pgtable.c +1 -0
+		flush_partition(lpid, old);
 }
 
 void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
Signature · 1 line
Alexey
#27 lore raw

Re: [PATCH v4 5/8] KVM: PPC: Ultravisor: Restrict flush of the partition tlb cache

Quoted 68 lines · Alexey Kardashevskiy
Quoted 49 lines · Claudio Carvalho

From: Ram Pai <[email protected]>

Ultravisor is responsible for flushing the tlb cache, since it manages the PATE entries. Hence skip tlb flush, if the ultravisor firmware is available.

---
 arch/powerpc/mm/book3s64/pgtable.c | 33 +++++++++++++++++-------------
 1 file changed, 19 insertions(+), 14 deletions(-)
arch/powerpc/mm/book3s64/pgtable.c +17 -5
index 224c5c7c2e3d..bc8eb2bf9810 100644
@@ -224,6 +224,23 @@ void __init mmu_partition_table_init(void)
 	powernv_set_nmmu_ptcr(ptcr);
 }
 
+static void flush_partition(unsigned int lpid, unsigned long dw0)
+{
+	if (dw0 & PATB_HR) {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 1, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
+	} else {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 0) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
+	}
+	/* do we need fixup here ?*/
+	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
+}
+
 static void __mmu_partition_table_set_entry(unsigned int lpid,
 					    unsigned long dw0,
 					    unsigned long dw1)
@@ -238,20 +255,8 @@ static void __mmu_partition_table_set_entry(unsigned int lpid,
 	 * The type of flush (hash or radix) depends on what the previous
 	 * use of this partition ID was, not the new use.
 	 */
-	asm volatile("ptesync" : : : "memory");
-	if (old & PATB_HR) {
-		asm volatile(PPC_TLBIE_5(%0,%1,2,0,1) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : :

arch/powerpc/mm/book3s64/pgtable.c static void __mmu_partition_table_set_entry(unsigned int lpid, · line 258

Quoted 10 lines
arch/powerpc/mm/book3s64/pgtable.c +1 -9
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
-	} else {
-		asm volatile(PPC_TLBIE_5(%0,%1,2,0,0) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
-	}
-	/* do we need fixup here ?*/
-	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))

__mmu_partition_table_set_entry() checks for UV and mmu_partition_table_set_entry() (the caller) checks for UV and the whole point of having separate flush_partition() and __mmu_partition_table_set_entry() is not really clear.

Quoted 1 line

4/8 and 5/8 make more sense as one patch imho.

Makes sense. I merged them in the next version. Thanks.

Claudio

Quoted 7 lines
Quoted 7 lines
arch/powerpc/mm/book3s64/pgtable.c +1 -0
+		flush_partition(lpid, old);
 }
 
 void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
#28 lore raw

Re: [PATCH v4 5/8] KVM: PPC: Ultravisor: Restrict flush of the partition tlb cache

In reply to Claudio Carvalho
Quoted 42 lines · Claudio Carvalho

From: Ram Pai <[email protected]>

Ultravisor is responsible for flushing the tlb cache, since it manages the PATE entries. Hence skip tlb flush, if the ultravisor firmware is available.

---
 arch/powerpc/mm/book3s64/pgtable.c | 33 +++++++++++++++++-------------
 1 file changed, 19 insertions(+), 14 deletions(-)
arch/powerpc/mm/book3s64/pgtable.c +17 -0
index 224c5c7c2e3d..bc8eb2bf9810 100644
@@ -224,6 +224,23 @@ void __init mmu_partition_table_init(void)
 	powernv_set_nmmu_ptcr(ptcr);
 }
 
+static void flush_partition(unsigned int lpid, unsigned long dw0)
+{
+	if (dw0 & PATB_HR) {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 1, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
+	} else {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 0) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
+	}
+	/* do we need fixup here ?*/
+	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
+}
+
 static void __mmu_partition_table_set_entry(unsigned int lpid,
 					    unsigned long dw0,
 					    unsigned long dw1)

arch/powerpc/mm/book3s64/pgtable.c static void · line 242

Quoted 5 lines
arch/powerpc/mm/book3s64/pgtable.c +0 -1
@@ -238,20 +255,8 @@ static void
 __mmu_partition_table_set_entry(unsigned int lpid,
 	 * The type of flush (hash or radix) depends on what the previous
 	 * use of this partition ID was, not the new use.
 	 */
-	asm volatile("ptesync" : : : "memory");

Doesn't the line above that was deleted need to be added to the
beginning of flush_partition()

Quoted 19 lines
arch/powerpc/mm/book3s64/pgtable.c +2 -13
-	if (old & PATB_HR) {
-		asm volatile(PPC_TLBIE_5(%0,%1,2,0,1) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
-	} else {
-		asm volatile(PPC_TLBIE_5(%0,%1,2,0,0) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
-	}
-	/* do we need fixup here ?*/
-	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		flush_partition(lpid, old);
 }
 

void mmu_partition_table_set_entry(unsigned int lpid, unsigned long
dw0,

#29 lore raw

Re: [PATCH v4 5/8] KVM: PPC: Ultravisor: Restrict flush of the partition tlb cache

Quoted 42 lines · janani
Quoted 41 lines · Claudio Carvalho

From: Ram Pai <[email protected]>

Ultravisor is responsible for flushing the tlb cache, since it manages the PATE entries. Hence skip tlb flush, if the ultravisor firmware is available.

---
arch/powerpc/mm/book3s64/pgtable.c | 33 +++++++++++++++++-------------

1 file changed, 19 insertions(+), 14 deletions(-)
arch/powerpc/mm/book3s64/pgtable.c +17 -0
index 224c5c7c2e3d..bc8eb2bf9810 100644
@@ -224,6 +224,23 @@ void __init mmu_partition_table_init(void)
 	powernv_set_nmmu_ptcr(ptcr);
 }
 
+static void flush_partition(unsigned int lpid, unsigned long dw0)
+{
+	if (dw0 & PATB_HR) {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 1, 1) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
+	} else {
+		asm volatile(PPC_TLBIE_5(%0, %1, 2, 0, 0) : :
+			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
+	}
+	/* do we need fixup here ?*/
+	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
+}
+
 static void __mmu_partition_table_set_entry(unsigned int lpid,
 					    unsigned long dw0,
 					    unsigned long dw1)
Quoted 8 lines

arch/powerpc/mm/book3s64/pgtable.c static void · line 242

Quoted 6 lines
arch/powerpc/mm/book3s64/pgtable.c +0 -1
@@ -238,20 +255,8 @@ static void
 __mmu_partition_table_set_entry(unsigned int lpid,
 	 * The type of flush (hash or radix) depends on what the previous
 	 * use of this partition ID was, not the new use.
 	 */
-	asm volatile("ptesync" : : : "memory");

Doesn't the line above that was deleted need to be added to the
beginning of flush_partition()

It has to. It got dropped erroneously.

This is a good catch!

Thanks,
RP

#30 lore raw

PATCH v4 6/8 KVM: PPC: Ultravisor: Restrict LDBAR access

In reply to Claudio Carvalho

When the ultravisor firmware is available, it takes control over the LDBAR register. In this case, thread-imc updates and save/restore operations on the LDBAR register are handled by ultravisor.

---
 arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 2 ++
 arch/powerpc/platforms/powernv/idle.c     | 6 ++++--
 arch/powerpc/platforms/powernv/opal-imc.c | 4 ++++
 3 files changed, 10 insertions(+), 2 deletions(-)
arch/powerpc/kvm/book3s_hv_rmhandlers.S +2 -0
index f9b2620fbecd..cffb365d9d02 100644
@@ -375,8 +375,10 @@ BEGIN_FTR_SECTION
 	mtspr	SPRN_RPR, r0
 	ld	r0, KVM_SPLIT_PMMAR(r6)
 	mtspr	SPRN_PMMAR, r0
+BEGIN_FW_FTR_SECTION_NESTED(70)
 	ld	r0, KVM_SPLIT_LDBAR(r6)
 	mtspr	SPRN_LDBAR, r0
+END_FW_FTR_SECTION_NESTED(FW_FEATURE_ULTRAVISOR, 0, 70)
 	isync
 FTR_SECTION_ELSE
 	/* On P9 we use the split_info for coordinating LPCR changes */
arch/powerpc/platforms/powernv/idle.c +4 -2
index 77f2e0a4ee37..5593a2d55959 100644
@@ -679,7 +679,8 @@ static unsigned long power9_idle_stop(unsigned long psscr, bool mmu_on)
 		sprs.ptcr	= mfspr(SPRN_PTCR);
 		sprs.rpr	= mfspr(SPRN_RPR);
 		sprs.tscr	= mfspr(SPRN_TSCR);
-		sprs.ldbar	= mfspr(SPRN_LDBAR);
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			sprs.ldbar	= mfspr(SPRN_LDBAR);
 
 		sprs_saved = true;
 
@@ -762,7 +763,8 @@ static unsigned long power9_idle_stop(unsigned long psscr, bool mmu_on)
 	mtspr(SPRN_PTCR,	sprs.ptcr);
 	mtspr(SPRN_RPR,		sprs.rpr);
 	mtspr(SPRN_TSCR,	sprs.tscr);
-	mtspr(SPRN_LDBAR,	sprs.ldbar);
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		mtspr(SPRN_LDBAR,	sprs.ldbar);
 
 	if (pls >= pnv_first_tb_loss_level) {
 		/* TB loss */
arch/powerpc/platforms/powernv/opal-imc.c +4 -0
index 1b6932890a73..5fe2d4526cbc 100644
@@ -254,6 +254,10 @@ static int opal_imc_counters_probe(struct platform_device *pdev)
 	bool core_imc_reg = false, thread_imc_reg = false;
 	u32 type;
 
+	/* Disable IMC devices, when Ultravisor is enabled. */
+	if (firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		return -EACCES;
+
 	/*
 	 * Check whether this is kdump kernel. If yes, force the engines to
 	 * stop and return.
Signature · 1 line
2.20.1
#31 lore raw

Re: [PATCH v4 6/8] KVM: PPC: Ultravisor: Restrict LDBAR access

Quoted 3 lines · Claudio Carvalho

When the ultravisor firmware is available, it takes control over the LDBAR register. In this case, thread-imc updates and save/restore operations on the LDBAR register are handled by ultravisor.

What does LDBAR do? "Power ISA™ Version 3.0 B" or "User’s Manual POWER9 Processor" do not tell.

Quoted 67 lines
---
 arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 2 ++
 arch/powerpc/platforms/powernv/idle.c     | 6 ++++--
 arch/powerpc/platforms/powernv/opal-imc.c | 4 ++++
 3 files changed, 10 insertions(+), 2 deletions(-)
arch/powerpc/kvm/book3s_hv_rmhandlers.S +2 -0
index f9b2620fbecd..cffb365d9d02 100644
@@ -375,8 +375,10 @@ BEGIN_FTR_SECTION
 	mtspr	SPRN_RPR, r0
 	ld	r0, KVM_SPLIT_PMMAR(r6)
 	mtspr	SPRN_PMMAR, r0
+BEGIN_FW_FTR_SECTION_NESTED(70)
 	ld	r0, KVM_SPLIT_LDBAR(r6)
 	mtspr	SPRN_LDBAR, r0
+END_FW_FTR_SECTION_NESTED(FW_FEATURE_ULTRAVISOR, 0, 70)
 	isync
 FTR_SECTION_ELSE
 	/* On P9 we use the split_info for coordinating LPCR changes */
arch/powerpc/platforms/powernv/idle.c +4 -2
index 77f2e0a4ee37..5593a2d55959 100644
@@ -679,7 +679,8 @@ static unsigned long power9_idle_stop(unsigned long psscr, bool mmu_on)
 		sprs.ptcr	= mfspr(SPRN_PTCR);
 		sprs.rpr	= mfspr(SPRN_RPR);
 		sprs.tscr	= mfspr(SPRN_TSCR);
-		sprs.ldbar	= mfspr(SPRN_LDBAR);
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			sprs.ldbar	= mfspr(SPRN_LDBAR);
 
 		sprs_saved = true;
 
@@ -762,7 +763,8 @@ static unsigned long power9_idle_stop(unsigned long psscr, bool mmu_on)
 	mtspr(SPRN_PTCR,	sprs.ptcr);
 	mtspr(SPRN_RPR,		sprs.rpr);
 	mtspr(SPRN_TSCR,	sprs.tscr);
-	mtspr(SPRN_LDBAR,	sprs.ldbar);
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		mtspr(SPRN_LDBAR,	sprs.ldbar);
 
 	if (pls >= pnv_first_tb_loss_level) {
 		/* TB loss */
arch/powerpc/platforms/powernv/opal-imc.c +4 -0
index 1b6932890a73..5fe2d4526cbc 100644
@@ -254,6 +254,10 @@ static int opal_imc_counters_probe(struct platform_device *pdev)
 	bool core_imc_reg = false, thread_imc_reg = false;
 	u32 type;
 
+	/* Disable IMC devices, when Ultravisor is enabled. */
+	if (firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		return -EACCES;
+
 	/*
 	 * Check whether this is kdump kernel. If yes, force the engines to
 	 * stop and return.
Signature · 1 line
Alexey
#32 lore raw

Re: [PATCH v4 6/8] KVM: PPC: Ultravisor: Restrict LDBAR access

Quoted 7 lines · Alexey Kardashevskiy
Quoted 3 lines · Claudio Carvalho

When the ultravisor firmware is available, it takes control over the LDBAR register. In this case, thread-imc updates and save/restore operations on the LDBAR register are handled by ultravisor.

What does LDBAR do? "Power ISA™ Version 3.0 B" or "User’s Manual POWER9 Processor" do not tell.

LDBAR is a per-thread SPR used by thread-imc pmu to dump the counter data into memory. LDBAR contains memory address along with few other configuration bits (it is populated by the thread-imc pmu driver). It is populated and enabled only when any of the thread imc pmu events are monitored.

Maddy

Quoted 68 lines
Quoted 68 lines
---
  arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 2 ++
  arch/powerpc/platforms/powernv/idle.c     | 6 ++++--
  arch/powerpc/platforms/powernv/opal-imc.c | 4 ++++
  3 files changed, 10 insertions(+), 2 deletions(-)
arch/powerpc/kvm/book3s_hv_rmhandlers.S +2 -0
index f9b2620fbecd..cffb365d9d02 100644
@@ -375,8 +375,10 @@ BEGIN_FTR_SECTION
  	mtspr	SPRN_RPR, r0
  	ld	r0, KVM_SPLIT_PMMAR(r6)
  	mtspr	SPRN_PMMAR, r0
+BEGIN_FW_FTR_SECTION_NESTED(70)
  	ld	r0, KVM_SPLIT_LDBAR(r6)
  	mtspr	SPRN_LDBAR, r0
+END_FW_FTR_SECTION_NESTED(FW_FEATURE_ULTRAVISOR, 0, 70)
  	isync
  FTR_SECTION_ELSE
  	/* On P9 we use the split_info for coordinating LPCR changes */
arch/powerpc/platforms/powernv/idle.c +4 -2
index 77f2e0a4ee37..5593a2d55959 100644
@@ -679,7 +679,8 @@ static unsigned long power9_idle_stop(unsigned long psscr, bool mmu_on)
  		sprs.ptcr	= mfspr(SPRN_PTCR);
  		sprs.rpr	= mfspr(SPRN_RPR);
  		sprs.tscr	= mfspr(SPRN_TSCR);
-		sprs.ldbar	= mfspr(SPRN_LDBAR);
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			sprs.ldbar	= mfspr(SPRN_LDBAR);
 
  		sprs_saved = true;
 
@@ -762,7 +763,8 @@ static unsigned long power9_idle_stop(unsigned long psscr, bool mmu_on)
  	mtspr(SPRN_PTCR,	sprs.ptcr);
  	mtspr(SPRN_RPR,		sprs.rpr);
  	mtspr(SPRN_TSCR,	sprs.tscr);
-	mtspr(SPRN_LDBAR,	sprs.ldbar);
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		mtspr(SPRN_LDBAR,	sprs.ldbar);
 
  	if (pls >= pnv_first_tb_loss_level) {
  		/* TB loss */
arch/powerpc/platforms/powernv/opal-imc.c +4 -0
index 1b6932890a73..5fe2d4526cbc 100644
@@ -254,6 +254,10 @@ static int opal_imc_counters_probe(struct platform_device *pdev)
  	bool core_imc_reg = false, thread_imc_reg = false;
  	u32 type;
 
+	/* Disable IMC devices, when Ultravisor is enabled. */
+	if (firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		return -EACCES;
+
  	/*
  	 * Check whether this is kdump kernel. If yes, force the engines to
  	 * stop and return.
#33 lore raw

Re: [PATCH v4 6/8] KVM: PPC: Ultravisor: Restrict LDBAR access

Quoted 9 lines · maddy
Quoted 7 lines · Alexey Kardashevskiy
Quoted 3 lines · Claudio Carvalho

When the ultravisor firmware is available, it takes control over the LDBAR register. In this case, thread-imc updates and save/restore operations on the LDBAR register are handled by ultravisor.

What does LDBAR do? "Power ISA™ Version 3.0 B" or "User’s Manual POWER9 Processor" do not tell.

Quoted 7 lines

LDBAR is a per-thread SPR used by thread-imc pmu to dump the counter data into memory.
LDBAR contains memory address along with few other configuration bits (it is populated
by the thread-imc pmu driver). It is populated and enabled only when any of the thread
imc pmu events are monitored.

I was actually looking for a spec for this register, what is the document name?

Quoted 74 lines

Maddy

Quoted 71 lines
Quoted 71 lines

---
  arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 2 ++ arch/powerpc/platforms/powernv/idle.c     | 6 ++++-- arch/powerpc/platforms/powernv/opal-imc.c | 4 ++++

  3 files changed, 10 insertions(+), 2 deletions(-)
arch/powerpc/kvm/book3s_hv_rmhandlers.S +0 -0
index f9b2620fbecd..cffb365d9d02 100644
@@ -375,8 +375,10 @@ BEGIN_FTR_SECTION

      mtspr    SPRN_RPR, r0
      ld    r0, KVM_SPLIT_PMMAR(r6)
      mtspr    SPRN_PMMAR, r0 +BEGIN_FW_FTR_SECTION_NESTED(70)
      ld    r0, KVM_SPLIT_LDBAR(r6)
      mtspr    SPRN_LDBAR, r0 +END_FW_FTR_SECTION_NESTED(FW_FEATURE_ULTRAVISOR, 0, 70) isync
  FTR_SECTION_ELSE
      /* On P9 we use the split_info for coordinating LPCR changes */

arch/powerpc/platforms/powernv/idle.c +2 -1
index 77f2e0a4ee37..5593a2d55959 100644
@@ -679,7 +679,8 @@ static unsigned long power9_idle_stop(unsigned
 long psscr, bool mmu_on)
           sprs.ptcr    = mfspr(SPRN_PTCR);
           sprs.rpr    = mfspr(SPRN_RPR);
           sprs.tscr    = mfspr(SPRN_TSCR);
-        sprs.ldbar    = mfspr(SPRN_LDBAR);
+        if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+            sprs.ldbar    = mfspr(SPRN_LDBAR);
             sprs_saved = true;

  @@ -762,7 +763,8 @@ static unsigned long power9_idle_stop(unsigned long psscr, bool mmu_on)
      mtspr(SPRN_PTCR,    sprs.ptcr);
      mtspr(SPRN_RPR,        sprs.rpr);
      mtspr(SPRN_TSCR,    sprs.tscr);
-    mtspr(SPRN_LDBAR,    sprs.ldbar);
+    if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+        mtspr(SPRN_LDBAR,    sprs.ldbar);
        if (pls >= pnv_first_tb_loss_level) {
          /* TB loss */

arch/powerpc/platforms/powernv/opal-imc.c +0 -0
index 1b6932890a73..5fe2d4526cbc 100644
@@ -254,6 +254,10 @@ static int opal_imc_counters_probe(struct
 platform_device *pdev)
       bool core_imc_reg = false, thread_imc_reg = false;
       u32 type;

  +    /* Disable IMC devices, when Ultravisor is enabled. */ +    if (firmware_has_feature(FW_FEATURE_ULTRAVISOR)) +        return -EACCES;
+
      /*
       * Check whether this is kdump kernel. If yes, force the engines to
       * stop and return.

Signature · 1 line
Alexey
#34 lore raw

Re: Re: [PATCH v4 6/8] KVM: PPC: Ultravisor: Restrict LDBAR access

Quoted 21 lines · Alexey Kardashevskiy
Quoted 9 lines · maddy
Quoted 7 lines · Alexey Kardashevskiy
Quoted 3 lines · Claudio Carvalho

When the ultravisor firmware is available, it takes control over the LDBAR register. In this case, thread-imc updates and save/restore operations on the LDBAR register are handled by ultravisor.

What does LDBAR do? "Power ISA™ Version 3.0 B" or "User’s Manual POWER9 Processor" do not tell.

Quoted 7 lines

LDBAR is a per-thread SPR used by thread-imc pmu to dump the counter data into memory.
LDBAR contains memory address along with few other configuration bits (it is populated
by the thread-imc pmu driver). It is populated and enabled only when any of the thread
imc pmu events are monitored.

Quoted 2 lines

I was actually looking for a spec for this register, what is the document name?

Its not a architected register. Its documented in the Power9 workbook.

RP

#35 lore raw

Re: [PATCH v4 6/8] KVM: PPC: Ultravisor: Restrict LDBAR access

Quoted 18 lines · Ram Pai
Quoted 17 lines · Alexey Kardashevskiy
Quoted 14 lines · maddy
Quoted 6 lines · Alexey Kardashevskiy
Quoted 3 lines · Claudio Carvalho

When the ultravisor firmware is available, it takes control over the LDBAR register. In this case, thread-imc updates and save/restore operations on the LDBAR register are handled by ultravisor.

What does LDBAR do? "Power ISA™ Version 3.0 B" or "User’s Manual POWER9 Processor" do not tell.

LDBAR is a per-thread SPR used by thread-imc pmu to dump the counter data into memory.
LDBAR contains memory address along with few other configuration bits (it is populated
by the thread-imc pmu driver). It is populated and enabled only when any of the thread
imc pmu events are monitored.

Quoted 4 lines
Quoted 2 lines

I was actually looking for a spec for this register, what is the document name?

Its not a architected register. Its documented in the Power9 workbook.

I also found some information about the LDBAR in
arch/powerpc/perf/imc-pmu.c

Claudio

Quoted 3 lines

RP

Reviewed-by on v4 6/8
#36 lore raw

Re: [PATCH v4 6/8] KVM: PPC: Ultravisor: Restrict LDBAR access

In reply to Claudio Carvalho
Quoted 9 lines · Claudio Carvalho

When the ultravisor firmware is available, it takes control over the LDBAR register. In this case, thread-imc updates and save/restore operations on the LDBAR register are handled by ultravisor.

Quoted 66 lines
---
 arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 2 ++
 arch/powerpc/platforms/powernv/idle.c     | 6 ++++--
 arch/powerpc/platforms/powernv/opal-imc.c | 4 ++++
 3 files changed, 10 insertions(+), 2 deletions(-)
arch/powerpc/kvm/book3s_hv_rmhandlers.S +2 -0
index f9b2620fbecd..cffb365d9d02 100644
@@ -375,8 +375,10 @@ BEGIN_FTR_SECTION
 	mtspr	SPRN_RPR, r0
 	ld	r0, KVM_SPLIT_PMMAR(r6)
 	mtspr	SPRN_PMMAR, r0
+BEGIN_FW_FTR_SECTION_NESTED(70)
 	ld	r0, KVM_SPLIT_LDBAR(r6)
 	mtspr	SPRN_LDBAR, r0
+END_FW_FTR_SECTION_NESTED(FW_FEATURE_ULTRAVISOR, 0, 70)
 	isync
 FTR_SECTION_ELSE
 	/* On P9 we use the split_info for coordinating LPCR changes */
arch/powerpc/platforms/powernv/idle.c +4 -2
index 77f2e0a4ee37..5593a2d55959 100644
@@ -679,7 +679,8 @@ static unsigned long power9_idle_stop(unsigned
 long psscr, bool mmu_on)
 		sprs.ptcr	= mfspr(SPRN_PTCR);
 		sprs.rpr	= mfspr(SPRN_RPR);
 		sprs.tscr	= mfspr(SPRN_TSCR);
-		sprs.ldbar	= mfspr(SPRN_LDBAR);
+		if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+			sprs.ldbar	= mfspr(SPRN_LDBAR);
 
 		sprs_saved = true;
@@ -762,7 +763,8 @@ static unsigned long power9_idle_stop(unsigned
 long psscr, bool mmu_on)
 	mtspr(SPRN_PTCR,	sprs.ptcr);
 	mtspr(SPRN_RPR,		sprs.rpr);
 	mtspr(SPRN_TSCR,	sprs.tscr);
-	mtspr(SPRN_LDBAR,	sprs.ldbar);
+	if (!firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		mtspr(SPRN_LDBAR,	sprs.ldbar);
 
 	if (pls >= pnv_first_tb_loss_level) {
arch/powerpc/platforms/powernv/idle.c +0 -0
 		/* TB loss */
arch/powerpc/platforms/powernv/opal-imc.c +4 -0
index 1b6932890a73..5fe2d4526cbc 100644
@@ -254,6 +254,10 @@ static int opal_imc_counters_probe(struct
 platform_device *pdev)
 	bool core_imc_reg = false, thread_imc_reg = false;
 	u32 type;
 
+	/* Disable IMC devices, when Ultravisor is enabled. */
+	if (firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		return -EACCES;
+
 	/*
 	 * Check whether this is kdump kernel. If yes, force the engines to

* stop and return.

#37 lore raw

Re: [PATCH v4 6/8] KVM: PPC: Ultravisor: Restrict LDBAR access

In reply to Claudio Carvalho
Quoted 3 lines · Claudio Carvalho

When the ultravisor firmware is available, it takes control over the LDBAR register. In this case, thread-imc updates and save/restore operations on the LDBAR register are handled by ultravisor.

Please roll up the replies to Alexey's question about LDBAR into the change log.

arch/powerpc/kvm/book3s_hv_rmhandlers.S BEGIN_FTR_SECTION · line 381

Quoted 12 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +2 -0
index f9b2620fbecd..cffb365d9d02 100644
@@ -375,8 +375,10 @@ BEGIN_FTR_SECTION
 	mtspr	SPRN_RPR, r0
 	ld	r0, KVM_SPLIT_PMMAR(r6)
 	mtspr	SPRN_PMMAR, r0
+BEGIN_FW_FTR_SECTION_NESTED(70)
 	ld	r0, KVM_SPLIT_LDBAR(r6)
 	mtspr	SPRN_LDBAR, r0
+END_FW_FTR_SECTION_NESTED(FW_FEATURE_ULTRAVISOR, 0, 70)

That's in Power8 code isn't it? Which will never have an ultravisor.

arch/powerpc/platforms/powernv/opal-imc.c static int opal_imc_counters_probe(struct platform_device *pdev) · line 259

Quoted 11 lines
arch/powerpc/platforms/powernv/opal-imc.c +3 -0
index 1b6932890a73..5fe2d4526cbc 100644
@@ -254,6 +254,10 @@ static int opal_imc_counters_probe(struct platform_device *pdev)
 	bool core_imc_reg = false, thread_imc_reg = false;
 	u32 type;
 
+	/* Disable IMC devices, when Ultravisor is enabled. */
+	if (firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		return -EACCES;

I don't mind taking this change. But at the same time should the IMC stuff just be omitted from the device tree when we're in ultravisor mode?

cheers

#38 lore raw

Re: [PATCH v4 6/8] KVM: PPC: Ultravisor: Restrict LDBAR access

Quoted 7 lines · Michael Ellerman
Quoted 3 lines · Claudio Carvalho

When the ultravisor firmware is available, it takes control over the LDBAR register. In this case, thread-imc updates and save/restore operations on the LDBAR register are handled by ultravisor.

Please roll up the replies to Alexey's question about LDBAR into the change log.

Quoted 13 lines

arch/powerpc/kvm/book3s_hv_rmhandlers.S BEGIN_FTR_SECTION · line 381

Quoted 12 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +2 -0
index f9b2620fbecd..cffb365d9d02 100644
@@ -375,8 +375,10 @@ BEGIN_FTR_SECTION
 	mtspr	SPRN_RPR, r0
 	ld	r0, KVM_SPLIT_PMMAR(r6)
 	mtspr	SPRN_PMMAR, r0
+BEGIN_FW_FTR_SECTION_NESTED(70)
 	ld	r0, KVM_SPLIT_LDBAR(r6)
 	mtspr	SPRN_LDBAR, r0
+END_FW_FTR_SECTION_NESTED(FW_FEATURE_ULTRAVISOR, 0, 70)

That's in Power8 code isn't it? Which will never have an ultravisor.

IIUC, it might be executed in Power9 as well, but I can double check that.

Quoted 9 lines
Quoted 9 lines
arch/powerpc/platforms/powernv/opal-imc.c +0 -0
index 1b6932890a73..5fe2d4526cbc 100644
@@ -254,6 +254,10 @@ static int opal_imc_counters_probe(struct platform_device *pdev)
 	bool core_imc_reg = false, thread_imc_reg = false;
 	u32 type;
 
Quoted 5 lines

arch/powerpc/platforms/powernv/opal-imc.c static int opal_imc_counters_probe(struct platform_device *pdev) · line 259

Quoted 3 lines
arch/powerpc/platforms/powernv/opal-imc.c +3 -0
+	/* Disable IMC devices, when Ultravisor is enabled. */
+	if (firmware_has_feature(FW_FEATURE_ULTRAVISOR))
+		return -EACCES;

I don't mind taking this change. But at the same time should the IMC stuff just be omitted from the device tree when we're in ultravisor mode?

Yes. Maddy said that he will patch skiboot to remove the IMC nodes if ultravisor is present.

I added this check just to protect the kernel in case skiboot is not in the right level for some
reason.

Thanks,
Claudio

Quoted 3 lines

cheers

#39 lore raw

PATCH v4 7/8 KVM: PPC: Ultravisor: Enter a secure guest

In reply to Claudio Carvalho

From: Sukadev Bhattiprolu <[email protected]>

To enter a secure guest, we have to go through the ultravisor, therefore we do a ucall when we are entering a secure guest.

This change is needed for any sort of entry to the secure guest from the hypervisor, whether it is a return from an hcall, a return from a hypervisor interrupt, or the first time that a secure guest vCPU is run.

If we are returning from an hcall, the results are already in the appropriate registers R3:12, except for R3, R6 and R7. R3 has the status of the reflected hcall, therefore we move it to R0 for the ultravisor and set R3 to the UV_RETURN ucall number. R6,7 were used as temporary registers, hence we restore them.

Have fast_guest_return check the kvm_arch.secure_guest field so that a new CPU enters UV when started (in response to a RTAS start-cpu call).

Thanks to input from Paul Mackerras, Ram Pai and Mike Anderson.

[ Pass SRR1 in r11 for UV_RETURN, fix kvmppc_msr_interrupt to preserve
the MSR_S bit ]

[ Fix UV_RETURN ucall number and arch.secure_guest check ]

[ Save the actual R3 in R0 for the ultravisor and use R3 for the
UV_RETURN ucall number. Update commit message and ret_to_ultra comment ]

---
 arch/powerpc/include/asm/kvm_host.h       |  1 +
 arch/powerpc/include/asm/ultravisor-api.h |  1 +
 arch/powerpc/kernel/asm-offsets.c         |  1 +
 arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 40 +++++++++++++++++++----
 4 files changed, 37 insertions(+), 6 deletions(-)
arch/powerpc/include/asm/kvm_host.h +1 -0
index 013c76a0a03e..184becb62ea4 100644
@@ -294,6 +294,7 @@ struct kvm_arch {
 	cpumask_t cpu_in_guest;
 	u8 radix;
 	u8 fwnmi_enabled;
+	u8 secure_guest;
 	bool threads_indep;
 	bool nested_enable;
 	pgd_t *pgtable;
arch/powerpc/include/asm/ultravisor-api.h +1 -0
index 141940771add..7c4d0b4ced12 100644
@@ -19,5 +19,6 @@
 
 /* opcodes */
 #define UV_WRITE_PATE			0xF104
+#define UV_RETURN			0xF11C
 
 #endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
arch/powerpc/kernel/asm-offsets.c +1 -0
index 8e02444e9d3d..44742724513e 100644
@@ -508,6 +508,7 @@ int main(void)
 	OFFSET(KVM_VRMA_SLB_V, kvm, arch.vrma_slb_v);
 	OFFSET(KVM_RADIX, kvm, arch.radix);
 	OFFSET(KVM_FWNMI, kvm, arch.fwnmi_enabled);
+	OFFSET(KVM_SECURE_GUEST, kvm, arch.secure_guest);
 	OFFSET(VCPU_DSISR, kvm_vcpu, arch.shregs.dsisr);
 	OFFSET(VCPU_DAR, kvm_vcpu, arch.shregs.dar);
 	OFFSET(VCPU_VPA, kvm_vcpu, arch.vpa.pinned_addr);
arch/powerpc/kvm/book3s_hv_rmhandlers.S +34 -6
index cffb365d9d02..89813ca987c2 100644
@@ -36,6 +36,7 @@
 #include <asm/asm-compat.h>
 #include <asm/feature-fixups.h>
 #include <asm/cpuidle.h>
+#include <asm/ultravisor-api.h>
 
 /* Sign-extend HDEC if not on POWER9 */
 #define EXTEND_HDEC(reg)			\
@@ -1092,16 +1093,12 @@ BEGIN_FTR_SECTION
 END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
 
 	ld	r5, VCPU_LR(r4)
-	ld	r6, VCPU_CR(r4)
 	mtlr	r5
-	mtcr	r6
 
 	ld	r1, VCPU_GPR(R1)(r4)
 	ld	r2, VCPU_GPR(R2)(r4)
 	ld	r3, VCPU_GPR(R3)(r4)
 	ld	r5, VCPU_GPR(R5)(r4)
-	ld	r6, VCPU_GPR(R6)(r4)
-	ld	r7, VCPU_GPR(R7)(r4)
 	ld	r8, VCPU_GPR(R8)(r4)
 	ld	r9, VCPU_GPR(R9)(r4)
 	ld	r10, VCPU_GPR(R10)(r4)
@@ -1119,10 +1116,38 @@ BEGIN_FTR_SECTION
 	mtspr	SPRN_HDSISR, r0
 END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
 
+	ld	r6, VCPU_KVM(r4)
+	lbz	r7, KVM_SECURE_GUEST(r6)
+	cmpdi	r7, 0
+	bne	ret_to_ultra
+
+	lwz	r6, VCPU_CR(r4)
+	mtcr	r6
+
+	ld	r7, VCPU_GPR(R7)(r4)
+	ld	r6, VCPU_GPR(R6)(r4)
 	ld	r0, VCPU_GPR(R0)(r4)
 	ld	r4, VCPU_GPR(R4)(r4)
 	HRFI_TO_GUEST
 	b	.
+/*
+ * We are entering a secure guest, so we have to invoke the ultravisor to do
+ * that. If we are returning from a hcall, the results are already in the
+ * appropriate registers R3:12, except for R3, R6 and R7. R3 has the status of
+ * the reflected hcall, therefore we move it to R0 for the ultravisor and set
+ * R3 to the UV_RETURN ucall number. R6,7 were used as temporary registers
+ * above, hence we restore them.
+ */
+ret_to_ultra:
+	lwz	r6, VCPU_CR(r4)
+	mtcr	r6
+	mfspr	r11, SPRN_SRR1
+	mr	r0, r3
+	LOAD_REG_IMMEDIATE(r3, UV_RETURN)
+	ld	r7, VCPU_GPR(R7)(r4)
+	ld	r6, VCPU_GPR(R6)(r4)
+	ld	r4, VCPU_GPR(R4)(r4)
+	sc	2
 
 /*
  * Enter the guest on a P9 or later system where we have exactly
@@ -3318,13 +3343,16 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_TYPE_RADIX)
  *   r0 is used as a scratch register
  */
 kvmppc_msr_interrupt:
+	andis.	r0, r11, MSR_S@h
 	rldicl	r0, r11, 64 - MSR_TS_S_LG, 62
-	cmpwi	r0, 2 /* Check if we are in transactional state..  */
+	cmpwi	cr1, r0, 2 /* Check if we are in transactional state..  */
 	ld	r11, VCPU_INTR_MSR(r9)
-	bne	1f
+	bne	cr1, 1f
 	/* ... if transactional, change to suspended */
 	li	r0, 1
 1:	rldimi	r11, r0, MSR_TS_S_LG, 63 - MSR_TS_T_LG
+	beqlr
+	oris	r11, r11, MSR_S@h		/* preserve MSR_S bit setting */
 	blr
 
 /*
Signature · 1 line
2.20.1
Reviewed-by on v4 7/8
#40 lore raw

Re: [PATCH v4 7/8] KVM: PPC: Ultravisor: Enter a secure guest

Quoted 23 lines · Claudio Carvalho

From: Sukadev Bhattiprolu <[email protected]>

To enter a secure guest, we have to go through the ultravisor, therefore we do a ucall when we are entering a secure guest.

This change is needed for any sort of entry to the secure guest from the hypervisor, whether it is a return from an hcall, a return from a hypervisor interrupt, or the first time that a secure guest vCPU is run.

If we are returning from an hcall, the results are already in the appropriate registers R3:12, except for R3, R6 and R7. R3 has the status of the reflected hcall, therefore we move it to R0 for the ultravisor and set R3 to the UV_RETURN ucall number. R6,7 were used as temporary registers, hence we restore them.

Have fast_guest_return check the kvm_arch.secure_guest field so that a new CPU enters UV when started (in response to a RTAS start-cpu call).

Thanks to input from Paul Mackerras, Ram Pai and Mike Anderson.

[ Pass SRR1 in r11 for UV_RETURN, fix kvmppc_msr_interrupt to preserve
the MSR_S bit ]

Quoted 7 lines
[ Fix UV_RETURN ucall number and arch.secure_guest check ]

[ Save the actual R3 in R0 for the ultravisor and use R3 for the
UV_RETURN ucall number. Update commit message and ret_to_ultra
comment ]

Quoted 138 lines
---
 arch/powerpc/include/asm/kvm_host.h       |  1 +
 arch/powerpc/include/asm/ultravisor-api.h |  1 +
 arch/powerpc/kernel/asm-offsets.c         |  1 +
 arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 40 +++++++++++++++++++----
 4 files changed, 37 insertions(+), 6 deletions(-)
arch/powerpc/include/asm/kvm_host.h +1 -0
index 013c76a0a03e..184becb62ea4 100644
@@ -294,6 +294,7 @@ struct kvm_arch {
 	cpumask_t cpu_in_guest;
 	u8 radix;
 	u8 fwnmi_enabled;
+	u8 secure_guest;
 	bool threads_indep;
 	bool nested_enable;
 	pgd_t *pgtable;
arch/powerpc/include/asm/ultravisor-api.h +1 -0
index 141940771add..7c4d0b4ced12 100644
@@ -19,5 +19,6 @@
 
 /* opcodes */
 #define UV_WRITE_PATE			0xF104
+#define UV_RETURN			0xF11C
 
 #endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
arch/powerpc/kernel/asm-offsets.c +1 -0
index 8e02444e9d3d..44742724513e 100644
@@ -508,6 +508,7 @@ int main(void)
 	OFFSET(KVM_VRMA_SLB_V, kvm, arch.vrma_slb_v);
 	OFFSET(KVM_RADIX, kvm, arch.radix);
 	OFFSET(KVM_FWNMI, kvm, arch.fwnmi_enabled);
+	OFFSET(KVM_SECURE_GUEST, kvm, arch.secure_guest);
 	OFFSET(VCPU_DSISR, kvm_vcpu, arch.shregs.dsisr);
 	OFFSET(VCPU_DAR, kvm_vcpu, arch.shregs.dar);
 	OFFSET(VCPU_VPA, kvm_vcpu, arch.vpa.pinned_addr);
arch/powerpc/kvm/book3s_hv_rmhandlers.S +13 -4
index cffb365d9d02..89813ca987c2 100644
@@ -36,6 +36,7 @@
 #include <asm/asm-compat.h>
 #include <asm/feature-fixups.h>
 #include <asm/cpuidle.h>
+#include <asm/ultravisor-api.h>
 
 /* Sign-extend HDEC if not on POWER9 */
 #define EXTEND_HDEC(reg)			\
@@ -1092,16 +1093,12 @@ BEGIN_FTR_SECTION
 END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
 
 	ld	r5, VCPU_LR(r4)
-	ld	r6, VCPU_CR(r4)
 	mtlr	r5
-	mtcr	r6
 
 	ld	r1, VCPU_GPR(R1)(r4)
 	ld	r2, VCPU_GPR(R2)(r4)
 	ld	r3, VCPU_GPR(R3)(r4)
 	ld	r5, VCPU_GPR(R5)(r4)
-	ld	r6, VCPU_GPR(R6)(r4)
-	ld	r7, VCPU_GPR(R7)(r4)
 	ld	r8, VCPU_GPR(R8)(r4)
 	ld	r9, VCPU_GPR(R9)(r4)
 	ld	r10, VCPU_GPR(R10)(r4)
@@ -1119,10 +1116,38 @@ BEGIN_FTR_SECTION
 	mtspr	SPRN_HDSISR, r0
 END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
 
+	ld	r6, VCPU_KVM(r4)
+	lbz	r7, KVM_SECURE_GUEST(r6)
+	cmpdi	r7, 0
+	bne	ret_to_ultra
+
+	lwz	r6, VCPU_CR(r4)
+	mtcr	r6
+
+	ld	r7, VCPU_GPR(R7)(r4)
+	ld	r6, VCPU_GPR(R6)(r4)
 	ld	r0, VCPU_GPR(R0)(r4)
 	ld	r4, VCPU_GPR(R4)(r4)
 	HRFI_TO_GUEST
 	b	.
+/*
+ * We are entering a secure guest, so we have to invoke the ultravisor 

to do
+ * that. If we are returning from a hcall, the results are already in the
+ * appropriate registers R3:12, except for R3, R6 and R7. R3 has the status of
+ * the reflected hcall, therefore we move it to R0 for the ultravisor and set
+ * R3 to the UV_RETURN ucall number. R6,7 were used as temporary registers
+ * above, hence we restore them.
+ */

arch/powerpc/kvm/book3s_hv_rmhandlers.S +15 -2
+ret_to_ultra:
+	lwz	r6, VCPU_CR(r4)
+	mtcr	r6
+	mfspr	r11, SPRN_SRR1
+	mr	r0, r3
+	LOAD_REG_IMMEDIATE(r3, UV_RETURN)
+	ld	r7, VCPU_GPR(R7)(r4)
+	ld	r6, VCPU_GPR(R6)(r4)
+	ld	r4, VCPU_GPR(R4)(r4)
+	sc	2
 
 /*
  * Enter the guest on a P9 or later system where we have exactly
@@ -3318,13 +3343,16 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_TYPE_RADIX)
  *   r0 is used as a scratch register
  */
 kvmppc_msr_interrupt:
+	andis.	r0, r11, MSR_S@h
 	rldicl	r0, r11, 64 - MSR_TS_S_LG, 62
-	cmpwi	r0, 2 /* Check if we are in transactional state..  */
+	cmpwi	cr1, r0, 2 /* Check if we are in transactional state..  */
 	ld	r11, VCPU_INTR_MSR(r9)
-	bne	1f
+	bne	cr1, 1f
 	/* ... if transactional, change to suspended */
 	li	r0, 1
 1:	rldimi	r11, r0, MSR_TS_S_LG, 63 - MSR_TS_T_LG
+	beqlr
+	oris	r11, r11, MSR_S@h		/* preserve MSR_S bit setting */
 	blr
 
 /*
#41 lore raw

Re: [PATCH v4 7/8] KVM: PPC: Ultravisor: Enter a secure guest

Quoted 30 lines · janani
Quoted 21 lines · Claudio Carvalho

From: Sukadev Bhattiprolu <[email protected]>

To enter a secure guest, we have to go through the ultravisor, therefore we do a ucall when we are entering a secure guest.

This change is needed for any sort of entry to the secure guest from the hypervisor, whether it is a return from an hcall, a return from a hypervisor interrupt, or the first time that a secure guest vCPU is run.

If we are returning from an hcall, the results are already in the appropriate registers R3:12, except for R3, R6 and R7. R3 has the status of the reflected hcall, therefore we move it to R0 for the ultravisor and set R3 to the UV_RETURN ucall number. R6,7 were used as temporary registers, hence we restore them.

Have fast_guest_return check the kvm_arch.secure_guest field so that a new CPU enters UV when started (in response to a RTAS start-cpu call).

Thanks to input from Paul Mackerras, Ram Pai and Mike Anderson.

Quoted 8 lines

[ Pass SRR1 in r11 for UV_RETURN, fix kvmppc_msr_interrupt to preserve the MSR_S bit ]

[ Fix UV_RETURN ucall number and arch.secure_guest check ]

[ Save the actual R3 in R0 for the ultravisor and use R3 for the UV_RETURN ucall number. Update commit message and ret_to_ultra comment ]

Quoted 1 line

Thanks Janani for reviewing the patchset.

Claudio

Quoted 137 lines
Quoted 136 lines

---
 arch/powerpc/include/asm/kvm_host.h       |  1 + arch/powerpc/include/asm/ultravisor-api.h |  1 + arch/powerpc/kernel/asm-offsets.c         |  1 + arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 40 +++++++++++++++++++----

 4 files changed, 37 insertions(+), 6 deletions(-)
arch/powerpc/include/asm/kvm_host.h +1 -0
index 013c76a0a03e..184becb62ea4 100644
@@ -294,6 +294,7 @@ struct kvm_arch {
      cpumask_t cpu_in_guest;
      u8 radix;
      u8 fwnmi_enabled;
+    u8 secure_guest;
      bool threads_indep;
      bool nested_enable;
      pgd_t *pgtable;
arch/powerpc/include/asm/ultravisor-api.h +1 -0
index 141940771add..7c4d0b4ced12 100644
@@ -19,5 +19,6 @@
 
  /* opcodes */
  #define UV_WRITE_PATE            0xF104
+#define UV_RETURN            0xF11C
 
  #endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
arch/powerpc/kernel/asm-offsets.c +1 -0
index 8e02444e9d3d..44742724513e 100644
@@ -508,6 +508,7 @@ int main(void)
      OFFSET(KVM_VRMA_SLB_V, kvm, arch.vrma_slb_v);
      OFFSET(KVM_RADIX, kvm, arch.radix);
      OFFSET(KVM_FWNMI, kvm, arch.fwnmi_enabled);
+    OFFSET(KVM_SECURE_GUEST, kvm, arch.secure_guest);
      OFFSET(VCPU_DSISR, kvm_vcpu, arch.shregs.dsisr);
      OFFSET(VCPU_DAR, kvm_vcpu, arch.shregs.dar);
      OFFSET(VCPU_VPA, kvm_vcpu, arch.vpa.pinned_addr);
arch/powerpc/kvm/book3s_hv_rmhandlers.S +1 -1
index cffb365d9d02..89813ca987c2 100644
@@ -36,6 +36,7 @@
  #include <asm/asm-compat.h>
  #include <asm/feature-fixups.h>
  #include <asm/cpuidle.h>
+#include <asm/ultravisor-api.h>
 
  /* Sign-extend HDEC if not on POWER9 */
  #define EXTEND_HDEC(reg)            \
@@ -1092,16 +1093,12 @@ BEGIN_FTR_SECTION
  END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
 
      ld    r5, VCPU_LR(r4)
-    ld    r6, VCPU_CR(r4)

     mtlr    r5
-    mtcr    r6

     ld    r1, VCPU_GPR(R1)(r4)
     ld    r2, VCPU_GPR(R2)(r4)
     ld    r3, VCPU_GPR(R3)(r4)
     ld    r5, VCPU_GPR(R5)(r4)
-    ld    r6, VCPU_GPR(R6)(r4)
-    ld    r7, VCPU_GPR(R7)(r4)
     ld    r8, VCPU_GPR(R8)(r4)
     ld    r9, VCPU_GPR(R9)(r4)
     ld    r10, VCPU_GPR(R10)(r4)

arch/powerpc/kvm/book3s_hv_rmhandlers.S +0 -0
@@ -1119,10 +1116,38 @@ BEGIN_FTR_SECTION

     mtspr    SPRN_HDSISR, r0
 END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)

+    ld    r6, VCPU_KVM(r4)
+    lbz    r7, KVM_SECURE_GUEST(r6)
+    cmpdi    r7, 0
+    bne    ret_to_ultra

arch/powerpc/kvm/book3s_hv_rmhandlers.S +6 -0
+
+    lwz    r6, VCPU_CR(r4)
+    mtcr    r6
+
+    ld    r7, VCPU_GPR(R7)(r4)
+    ld    r6, VCPU_GPR(R6)(r4)

     ld    r0, VCPU_GPR(R0)(r4)
     ld    r4, VCPU_GPR(R4)(r4)
     HRFI_TO_GUEST
     b    .
+/*
+ * We are entering a secure guest, so we have to invoke the ultravisor to do
+ * that. If we are returning from a hcall, the results are already in the
+ * appropriate registers R3:12, except for R3, R6 and R7. R3 has the status of
+ * the reflected hcall, therefore we move it to R0 for the ultravisor and set
+ * R3 to the UV_RETURN ucall number. R6,7 were used as temporary registers
+ * above, hence we restore them.
+ */
+ret_to_ultra:
+    lwz    r6, VCPU_CR(r4)
+    mtcr    r6
+    mfspr    r11, SPRN_SRR1
+    mr    r0, r3
+    LOAD_REG_IMMEDIATE(r3, UV_RETURN)
+    ld    r7, VCPU_GPR(R7)(r4)
+    ld    r6, VCPU_GPR(R6)(r4)
+    ld    r4, VCPU_GPR(R4)(r4)
+    sc    2

 /*
  * Enter the guest on a P9 or later system where we have exactly

arch/powerpc/kvm/book3s_hv_rmhandlers.S +0 -0
@@ -3318,13 +3343,16 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_TYPE_RADIX)
   *   r0 is used as a scratch register
   */

 kvmppc_msr_interrupt:
+    andis.    r0, r11, MSR_S@h
     rldicl    r0, r11, 64 - MSR_TS_S_LG, 62
-    cmpwi    r0, 2 /* Check if we are in transactional state..  */ +    cmpwi    cr1, r0, 2 /* Check if we are in transactional state..  */ ld    r11, VCPU_INTR_MSR(r9)
-    bne    1f
+    bne    cr1, 1f
     /* ... if transactional, change to suspended */
     li    r0, 1
 1:    rldimi    r11, r0, MSR_TS_S_LG, 63 - MSR_TS_T_LG
+    beqlr
+    oris    r11, r11, MSR_S@h        /* preserve MSR_S bit setting */ blr

 /*

#42 lore raw

Re: [PATCH v4 7/8] KVM: PPC: Ultravisor: Enter a secure guest

In reply to Claudio Carvalho
Quoted 9 lines · Claudio Carvalho

From: Sukadev Bhattiprolu <[email protected]>

To enter a secure guest, we have to go through the ultravisor, therefore we do a ucall when we are entering a secure guest.

This change is needed for any sort of entry to the secure guest from the hypervisor, whether it is a return from an hcall, a return from a hypervisor interrupt, or the first time that a secure guest vCPU is run.

Quoted 5 lines

If we are returning from an hcall, the results are already in the appropriate registers R3:12, except for R3, R6 and R7. R3 has the status of the reflected hcall, therefore we move it to R0 for the ultravisor and set R3 to the UV_RETURN ucall number. R6,7 were used as temporary registers, hence we restore them.

This is another case where some documentation would help people to review the code.

Quoted 51 lines

Have fast_guest_return check the kvm_arch.secure_guest field so that a new CPU enters UV when started (in response to a RTAS start-cpu call).

Thanks to input from Paul Mackerras, Ram Pai and Mike Anderson.

[ Pass SRR1 in r11 for UV_RETURN, fix kvmppc_msr_interrupt to preserve
the MSR_S bit ]

[ Fix UV_RETURN ucall number and arch.secure_guest check ]

[ Save the actual R3 in R0 for the ultravisor and use R3 for the
UV_RETURN ucall number. Update commit message and ret_to_ultra comment ]

---
 arch/powerpc/include/asm/kvm_host.h       |  1 +
 arch/powerpc/include/asm/ultravisor-api.h |  1 +
 arch/powerpc/kernel/asm-offsets.c         |  1 +
 arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 40 +++++++++++++++++++----
 4 files changed, 37 insertions(+), 6 deletions(-)
arch/powerpc/kvm/book3s_hv_rmhandlers.S +1 -4
index cffb365d9d02..89813ca987c2 100644
@@ -36,6 +36,7 @@
 #include <asm/asm-compat.h>
 #include <asm/feature-fixups.h>
 #include <asm/cpuidle.h>
+#include <asm/ultravisor-api.h>
 
 /* Sign-extend HDEC if not on POWER9 */
 #define EXTEND_HDEC(reg)			\
@@ -1092,16 +1093,12 @@ BEGIN_FTR_SECTION
 END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
 
 	ld	r5, VCPU_LR(r4)
-	ld	r6, VCPU_CR(r4)
 	mtlr	r5
-	mtcr	r6
 
 	ld	r1, VCPU_GPR(R1)(r4)
 	ld	r2, VCPU_GPR(R2)(r4)
 	ld	r3, VCPU_GPR(R3)(r4)
 	ld	r5, VCPU_GPR(R5)(r4)
-	ld	r6, VCPU_GPR(R6)(r4)
-	ld	r7, VCPU_GPR(R7)(r4)
 	ld	r8, VCPU_GPR(R8)(r4)
 	ld	r9, VCPU_GPR(R9)(r4)
 	ld	r10, VCPU_GPR(R10)(r4)

arch/powerpc/kvm/book3s_hv_rmhandlers.S BEGIN_FTR_SECTION · line 1121

Quoted 6 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +3 -0
@@ -1119,10 +1116,38 @@ BEGIN_FTR_SECTION
 	mtspr	SPRN_HDSISR, r0
 END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
 
+	ld	r6, VCPU_KVM(r4)
+	lbz	r7, KVM_SECURE_GUEST(r6)
+	cmpdi	r7, 0

You could hoist the load of r6 and r7 to here?

Quoted 15 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +11 -0
+	bne	ret_to_ultra
+
+	lwz	r6, VCPU_CR(r4)
+	mtcr	r6
+
+	ld	r7, VCPU_GPR(R7)(r4)
+	ld	r6, VCPU_GPR(R6)(r4)
 	ld	r0, VCPU_GPR(R0)(r4)
 	ld	r4, VCPU_GPR(R4)(r4)
 	HRFI_TO_GUEST
 	b	.
+/*
+ * We are entering a secure guest, so we have to invoke the ultravisor to do
+ * that. If we are returning from a hcall, the results are already in the
+ * appropriate registers R3:12, except for R3, R6 and R7. R3 has the status of

arch/powerpc/kvm/book3s_hv_rmhandlers.S BEGIN_FTR_SECTION · line 1146

Quoted 10 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +10 -0
+ * the reflected hcall, therefore we move it to R0 for the ultravisor and set
+ * R3 to the UV_RETURN ucall number. R6,7 were used as temporary registers
+ * above, hence we restore them.
+ */
+ret_to_ultra:
+	lwz	r6, VCPU_CR(r4)
+	mtcr	r6
+	mfspr	r11, SPRN_SRR1
+	mr	r0, r3
+	LOAD_REG_IMMEDIATE(r3, UV_RETURN)

Worth open coding to save three instructions?

Quoted 14 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +5 -1
+	ld	r7, VCPU_GPR(R7)(r4)
+	ld	r6, VCPU_GPR(R6)(r4)
+	ld	r4, VCPU_GPR(R4)(r4)
+	sc	2
 
 /*
  * Enter the guest on a P9 or later system where we have exactly
@@ -3318,13 +3343,16 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_TYPE_RADIX)
  *   r0 is used as a scratch register
  */
 kvmppc_msr_interrupt:
+	andis.	r0, r11, MSR_S@h
 	rldicl	r0, r11, 64 - MSR_TS_S_LG, 62
-	cmpwi	r0, 2 /* Check if we are in transactional state..  */

arch/powerpc/kvm/book3s_hv_rmhandlers.S END_MMU_FTR_SECTION_IFSET(MMU_FTR_TYPE_RADIX) · line 3356

Quoted 10 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +4 -1
+	cmpwi	cr1, r0, 2 /* Check if we are in transactional state..  */
 	ld	r11, VCPU_INTR_MSR(r9)
-	bne	1f
+	bne	cr1, 1f
 	/* ... if transactional, change to suspended */
 	li	r0, 1
 1:	rldimi	r11, r0, MSR_TS_S_LG, 63 - MSR_TS_T_LG
+	beqlr
+	oris	r11, r11, MSR_S@h		/* preserve MSR_S bit setting */
 	blr

I don't see this part mentioned in the change log?

It's also pretty subtle, a comment might be helpful.

cheers

#43 lore raw

Re: [PATCH v4 7/8] KVM: PPC: Ultravisor: Enter a secure guest

Quoted 73 lines · Michael Ellerman [[email protected]] wrote:
Quoted 9 lines · Claudio Carvalho

From: Sukadev Bhattiprolu <[email protected]>

To enter a secure guest, we have to go through the ultravisor, therefore we do a ucall when we are entering a secure guest.

This change is needed for any sort of entry to the secure guest from the hypervisor, whether it is a return from an hcall, a return from a hypervisor interrupt, or the first time that a secure guest vCPU is run.

Quoted 5 lines

If we are returning from an hcall, the results are already in the appropriate registers R3:12, except for R3, R6 and R7. R3 has the status of the reflected hcall, therefore we move it to R0 for the ultravisor and set R3 to the UV_RETURN ucall number. R6,7 were used as temporary registers, hence we restore them.

This is another case where some documentation would help people to review the code.

Quoted 54 lines

Have fast_guest_return check the kvm_arch.secure_guest field so that a new CPU enters UV when started (in response to a RTAS start-cpu call).

Thanks to input from Paul Mackerras, Ram Pai and Mike Anderson.

[ Pass SRR1 in r11 for UV_RETURN, fix kvmppc_msr_interrupt to preserve
the MSR_S bit ]

[ Fix UV_RETURN ucall number and arch.secure_guest check ]

[ Save the actual R3 in R0 for the ultravisor and use R3 for the
UV_RETURN ucall number. Update commit message and ret_to_ultra comment ]

---
 arch/powerpc/include/asm/kvm_host.h       |  1 +
 arch/powerpc/include/asm/ultravisor-api.h |  1 +
 arch/powerpc/kernel/asm-offsets.c         |  1 +
 arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 40 +++++++++++++++++++----
 4 files changed, 37 insertions(+), 6 deletions(-)
arch/powerpc/kvm/book3s_hv_rmhandlers.S +1 -4
index cffb365d9d02..89813ca987c2 100644
@@ -36,6 +36,7 @@
 #include <asm/asm-compat.h>
 #include <asm/feature-fixups.h>
 #include <asm/cpuidle.h>
+#include <asm/ultravisor-api.h>
 
 /* Sign-extend HDEC if not on POWER9 */
 #define EXTEND_HDEC(reg)			\
@@ -1092,16 +1093,12 @@ BEGIN_FTR_SECTION
 END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
 
 	ld	r5, VCPU_LR(r4)
-	ld	r6, VCPU_CR(r4)
 	mtlr	r5
-	mtcr	r6
 
 	ld	r1, VCPU_GPR(R1)(r4)
 	ld	r2, VCPU_GPR(R2)(r4)
 	ld	r3, VCPU_GPR(R3)(r4)
 	ld	r5, VCPU_GPR(R5)(r4)
-	ld	r6, VCPU_GPR(R6)(r4)
-	ld	r7, VCPU_GPR(R7)(r4)
 	ld	r8, VCPU_GPR(R8)(r4)
 	ld	r9, VCPU_GPR(R9)(r4)
 	ld	r10, VCPU_GPR(R10)(r4)
@@ -1119,10 +1116,38 @@ BEGIN_FTR_SECTION
 	mtspr	SPRN_HDSISR, r0
 END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
 
Quoted 5 lines

arch/powerpc/kvm/book3s_hv_rmhandlers.S BEGIN_FTR_SECTION · line 1121

Quoted 3 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +3 -0
+	ld	r6, VCPU_KVM(r4)
+	lbz	r7, KVM_SECURE_GUEST(r6)
+	cmpdi	r7, 0

You could hoist the load of r6 and r7 to here?

we could move 'ld r7' here. r6 is used to restore CR below so it (r6) has to stay there?

Quoted 20 lines
Quoted 20 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +15 -0
+	bne	ret_to_ultra
+
+	lwz	r6, VCPU_CR(r4)
+	mtcr	r6
+
+	ld	r7, VCPU_GPR(R7)(r4)
+	ld	r6, VCPU_GPR(R6)(r4)
 	ld	r0, VCPU_GPR(R0)(r4)
 	ld	r4, VCPU_GPR(R4)(r4)
 	HRFI_TO_GUEST
 	b	.
+/*
+ * We are entering a secure guest, so we have to invoke the ultravisor to do
+ * that. If we are returning from a hcall, the results are already in the
+ * appropriate registers R3:12, except for R3, R6 and R7. R3 has the status of
+ * the reflected hcall, therefore we move it to R0 for the ultravisor and set
+ * R3 to the UV_RETURN ucall number. R6,7 were used as temporary registers
+ * above, hence we restore them.
+ */
Quoted 8 lines

arch/powerpc/kvm/book3s_hv_rmhandlers.S BEGIN_FTR_SECTION · line 1146

Quoted 6 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +6 -0
+ret_to_ultra:
+	lwz	r6, VCPU_CR(r4)
+	mtcr	r6
+	mfspr	r11, SPRN_SRR1
+	mr	r0, r3
+	LOAD_REG_IMMEDIATE(r3, UV_RETURN)

Worth open coding to save three instructions?

Yes, good point:

- LOAD_REG_IMMEDIATE(r3, UV_RETURN)
+

+       li      r3, 0
+       oris    r3, r3, (UV_RETURN)@__AS_ATHIGH
+       ori     r3, r3, (UV_RETURN)@l

+

Quoted 31 lines
Quoted 15 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +5 -1
+	ld	r7, VCPU_GPR(R7)(r4)
+	ld	r6, VCPU_GPR(R6)(r4)
+	ld	r4, VCPU_GPR(R4)(r4)
+	sc	2
 
 /*
  * Enter the guest on a P9 or later system where we have exactly
@@ -3318,13 +3343,16 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_TYPE_RADIX)
  *   r0 is used as a scratch register
  */
 kvmppc_msr_interrupt:
+	andis.	r0, r11, MSR_S@h
 	rldicl	r0, r11, 64 - MSR_TS_S_LG, 62
-	cmpwi	r0, 2 /* Check if we are in transactional state..  */

arch/powerpc/kvm/book3s_hv_rmhandlers.S END_MMU_FTR_SECTION_IFSET(MMU_FTR_TYPE_RADIX) · line 3356

Quoted 10 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +4 -1
+	cmpwi	cr1, r0, 2 /* Check if we are in transactional state..  */
 	ld	r11, VCPU_INTR_MSR(r9)
-	bne	1f
+	bne	cr1, 1f
 	/* ... if transactional, change to suspended */
 	li	r0, 1
 1:	rldimi	r11, r0, MSR_TS_S_LG, 63 - MSR_TS_T_LG
+	beqlr
+	oris	r11, r11, MSR_S@h		/* preserve MSR_S bit setting */
 	blr

I don't see this part mentioned in the change log?

It's also pretty subtle, a comment might be helpful.

cheers

#44 lore raw

Re: [PATCH v4 7/8] KVM: PPC: Ultravisor: Enter a secure guest

Quoted 74 lines · Sukadev Bhattiprolu
Quoted 73 lines · Michael Ellerman [[email protected]] wrote:
Quoted 9 lines · Claudio Carvalho

From: Sukadev Bhattiprolu <[email protected]>

To enter a secure guest, we have to go through the ultravisor, therefore we do a ucall when we are entering a secure guest.

This change is needed for any sort of entry to the secure guest from the hypervisor, whether it is a return from an hcall, a return from a hypervisor interrupt, or the first time that a secure guest vCPU is run.

Quoted 5 lines

If we are returning from an hcall, the results are already in the appropriate registers R3:12, except for R3, R6 and R7. R3 has the status of the reflected hcall, therefore we move it to R0 for the ultravisor and set R3 to the UV_RETURN ucall number. R6,7 were used as temporary registers, hence we restore them.

This is another case where some documentation would help people to review the code.

Quoted 54 lines

Have fast_guest_return check the kvm_arch.secure_guest field so that a new CPU enters UV when started (in response to a RTAS start-cpu call).

Thanks to input from Paul Mackerras, Ram Pai and Mike Anderson.

[ Pass SRR1 in r11 for UV_RETURN, fix kvmppc_msr_interrupt to preserve
the MSR_S bit ]

[ Fix UV_RETURN ucall number and arch.secure_guest check ]

[ Save the actual R3 in R0 for the ultravisor and use R3 for the
UV_RETURN ucall number. Update commit message and ret_to_ultra comment ]

---
 arch/powerpc/include/asm/kvm_host.h       |  1 +
 arch/powerpc/include/asm/ultravisor-api.h |  1 +
 arch/powerpc/kernel/asm-offsets.c         |  1 +
 arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 40 +++++++++++++++++++----
 4 files changed, 37 insertions(+), 6 deletions(-)
arch/powerpc/kvm/book3s_hv_rmhandlers.S +1 -4
index cffb365d9d02..89813ca987c2 100644
@@ -36,6 +36,7 @@
 #include <asm/asm-compat.h>
 #include <asm/feature-fixups.h>
 #include <asm/cpuidle.h>
+#include <asm/ultravisor-api.h>
 
 /* Sign-extend HDEC if not on POWER9 */
 #define EXTEND_HDEC(reg)			\
@@ -1092,16 +1093,12 @@ BEGIN_FTR_SECTION
 END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
 
 	ld	r5, VCPU_LR(r4)
-	ld	r6, VCPU_CR(r4)
 	mtlr	r5
-	mtcr	r6
 
 	ld	r1, VCPU_GPR(R1)(r4)
 	ld	r2, VCPU_GPR(R2)(r4)
 	ld	r3, VCPU_GPR(R3)(r4)
 	ld	r5, VCPU_GPR(R5)(r4)
-	ld	r6, VCPU_GPR(R6)(r4)
-	ld	r7, VCPU_GPR(R7)(r4)
 	ld	r8, VCPU_GPR(R8)(r4)
 	ld	r9, VCPU_GPR(R9)(r4)
 	ld	r10, VCPU_GPR(R10)(r4)
@@ -1119,10 +1116,38 @@ BEGIN_FTR_SECTION
 	mtspr	SPRN_HDSISR, r0
 END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
 
Quoted 8 lines
Quoted 5 lines

arch/powerpc/kvm/book3s_hv_rmhandlers.S BEGIN_FTR_SECTION · line 1121

Quoted 3 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +3 -0
+	ld	r6, VCPU_KVM(r4)
+	lbz	r7, KVM_SECURE_GUEST(r6)
+	cmpdi	r7, 0

You could hoist the load of r6 and r7 to here?

we could move 'ld r7' here. r6 is used to restore CR below so it (r6) has to stay there?

It's used to restore CR in both paths, so both paths load VCPU_CR(r4) into r6. So we could instead do that load once, before the branch?

Quoted 30 lines
Quoted 19 lines
Quoted 19 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +15 -0
+	bne	ret_to_ultra
+
+	lwz	r6, VCPU_CR(r4)
+	mtcr	r6
+
+	ld	r7, VCPU_GPR(R7)(r4)
+	ld	r6, VCPU_GPR(R6)(r4)
 	ld	r0, VCPU_GPR(R0)(r4)
 	ld	r4, VCPU_GPR(R4)(r4)
 	HRFI_TO_GUEST
 	b	.
+/*
+ * We are entering a secure guest, so we have to invoke the ultravisor to do
+ * that. If we are returning from a hcall, the results are already in the
+ * appropriate registers R3:12, except for R3, R6 and R7. R3 has the status of
+ * the reflected hcall, therefore we move it to R0 for the ultravisor and set
+ * R3 to the UV_RETURN ucall number. R6,7 were used as temporary registers
+ * above, hence we restore them.
+ */
Quoted 8 lines

arch/powerpc/kvm/book3s_hv_rmhandlers.S BEGIN_FTR_SECTION · line 1146

Quoted 6 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +6 -0
+ret_to_ultra:
+	lwz	r6, VCPU_CR(r4)
+	mtcr	r6
+	mfspr	r11, SPRN_SRR1
+	mr	r0, r3
+	LOAD_REG_IMMEDIATE(r3, UV_RETURN)

Worth open coding to save three instructions?

Yes, good point:

Review

Quoted 5 lines
(quoted diff) +4 -1
-       LOAD_REG_IMMEDIATE(r3, UV_RETURN)
+
+       li      r3, 0
+       oris    r3, r3, (UV_RETURN)@__AS_ATHIGH
+       ori     r3, r3, (UV_RETURN)@l

This should do it no?

li r3, 0
oris r3, r3, UV_RETURN

cheers

#45 lore raw

Re: [PATCH v4 7/8] KVM: PPC: Ultravisor: Enter a secure guest

In reply to Claudio Carvalho

Claudio Carvalho's on June 29, 2019 6:08 am:

Quoted 90 lines

From: Sukadev Bhattiprolu <[email protected]>

To enter a secure guest, we have to go through the ultravisor, therefore we do a ucall when we are entering a secure guest.

This change is needed for any sort of entry to the secure guest from the hypervisor, whether it is a return from an hcall, a return from a hypervisor interrupt, or the first time that a secure guest vCPU is run.

If we are returning from an hcall, the results are already in the appropriate registers R3:12, except for R3, R6 and R7. R3 has the status of the reflected hcall, therefore we move it to R0 for the ultravisor and set R3 to the UV_RETURN ucall number. R6,7 were used as temporary registers, hence we restore them.

Have fast_guest_return check the kvm_arch.secure_guest field so that a new CPU enters UV when started (in response to a RTAS start-cpu call).

Thanks to input from Paul Mackerras, Ram Pai and Mike Anderson.

[ Pass SRR1 in r11 for UV_RETURN, fix kvmppc_msr_interrupt to preserve
the MSR_S bit ]

[ Fix UV_RETURN ucall number and arch.secure_guest check ]

[ Save the actual R3 in R0 for the ultravisor and use R3 for the
UV_RETURN ucall number. Update commit message and ret_to_ultra comment ]

---
 arch/powerpc/include/asm/kvm_host.h       |  1 +
 arch/powerpc/include/asm/ultravisor-api.h |  1 +
 arch/powerpc/kernel/asm-offsets.c         |  1 +
 arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 40 +++++++++++++++++++----
 4 files changed, 37 insertions(+), 6 deletions(-)
arch/powerpc/include/asm/kvm_host.h +1 -0
index 013c76a0a03e..184becb62ea4 100644
@@ -294,6 +294,7 @@ struct kvm_arch {
 	cpumask_t cpu_in_guest;
 	u8 radix;
 	u8 fwnmi_enabled;
+	u8 secure_guest;
 	bool threads_indep;
 	bool nested_enable;
 	pgd_t *pgtable;
arch/powerpc/include/asm/ultravisor-api.h +1 -0
index 141940771add..7c4d0b4ced12 100644
@@ -19,5 +19,6 @@
 
 /* opcodes */
 #define UV_WRITE_PATE			0xF104
+#define UV_RETURN			0xF11C
 
 #endif /* _ASM_POWERPC_ULTRAVISOR_API_H */
arch/powerpc/kernel/asm-offsets.c +1 -0
index 8e02444e9d3d..44742724513e 100644
@@ -508,6 +508,7 @@ int main(void)
 	OFFSET(KVM_VRMA_SLB_V, kvm, arch.vrma_slb_v);
 	OFFSET(KVM_RADIX, kvm, arch.radix);
 	OFFSET(KVM_FWNMI, kvm, arch.fwnmi_enabled);
+	OFFSET(KVM_SECURE_GUEST, kvm, arch.secure_guest);
 	OFFSET(VCPU_DSISR, kvm_vcpu, arch.shregs.dsisr);
 	OFFSET(VCPU_DAR, kvm_vcpu, arch.shregs.dar);
 	OFFSET(VCPU_VPA, kvm_vcpu, arch.vpa.pinned_addr);
arch/powerpc/kvm/book3s_hv_rmhandlers.S +1 -2
index cffb365d9d02..89813ca987c2 100644
@@ -36,6 +36,7 @@
 #include <asm/asm-compat.h>
 #include <asm/feature-fixups.h>
 #include <asm/cpuidle.h>
+#include <asm/ultravisor-api.h>
 
 /* Sign-extend HDEC if not on POWER9 */
 #define EXTEND_HDEC(reg)			\
@@ -1092,16 +1093,12 @@ BEGIN_FTR_SECTION
 END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
 
 	ld	r5, VCPU_LR(r4)
-	ld	r6, VCPU_CR(r4)
 	mtlr	r5
-	mtcr	r6

arch/powerpc/kvm/book3s_hv_rmhandlers.S BEGIN_FTR_SECTION · line 1104

Quoted 10 lines
arch/powerpc/kvm/book3s_hv_rmhandlers.S +0 -2
 
 	ld	r1, VCPU_GPR(R1)(r4)
 	ld	r2, VCPU_GPR(R2)(r4)
 	ld	r3, VCPU_GPR(R3)(r4)
 	ld	r5, VCPU_GPR(R5)(r4)
-	ld	r6, VCPU_GPR(R6)(r4)
-	ld	r7, VCPU_GPR(R7)(r4)
 	ld	r8, VCPU_GPR(R8)(r4)
 	ld	r9, VCPU_GPR(R9)(r4)
 	ld	r10, VCPU_GPR(R10)(r4)

Just to try to be less arbitrary about things, could you use regs adjacent to r4? Generally good because then it has a chance to get our loads paired up (which may not help some CPUs).

Thanks,
Nick

#46 lore raw

PATCH v4 8/8 KVM: PPC: Ultravisor: Check for MSR_S during hv_reset_msr

In reply to Claudio Carvalho

From: Michael Anderson <[email protected]>

- Check for MSR_S so that kvmppc_set_msr will include it. Prior to this change return to guest would not have the S bit set.

- Patch based on comment from Paul Mackerras <[email protected]>

---
 arch/powerpc/kvm/book3s_64_mmu_hv.c | 1 +
 1 file changed, 1 insertion(+)
arch/powerpc/kvm/book3s_64_mmu_hv.c +1 -0
index ab3d484c5e2e..ab62a66f9b4e 100644
@@ -295,6 +295,7 @@ static void kvmppc_mmu_book3s_64_hv_reset_msr(struct kvm_vcpu *vcpu)
 		msr |= MSR_TS_S;
 	else
 		msr |= vcpu->arch.shregs.msr & MSR_TS_MASK;
+	msr |= vcpu->arch.shregs.msr & MSR_S;
 	kvmppc_set_msr(vcpu, msr);
 }
 
Signature · 1 line
2.20.1
Reviewed-by on v4 8/8
#47 lore raw

Re: [PATCH v4 8/8] KVM: PPC: Ultravisor: Check for MSR_S during hv_reset_msr

Quoted 11 lines · Claudio Carvalho

From: Michael Anderson <[email protected]>

- Check for MSR_S so that kvmppc_set_msr will include it. Prior to
this
change return to guest would not have the S bit set.

- Patch based on comment from Paul Mackerras <[email protected]>

Quoted 17 lines
---
 arch/powerpc/kvm/book3s_64_mmu_hv.c | 1 +
 1 file changed, 1 insertion(+)
arch/powerpc/kvm/book3s_64_mmu_hv.c +1 -0
index ab3d484c5e2e..ab62a66f9b4e 100644
@@ -295,6 +295,7 @@ static void
 kvmppc_mmu_book3s_64_hv_reset_msr(struct kvm_vcpu *vcpu)
 		msr |= MSR_TS_S;
 	else
 		msr |= vcpu->arch.shregs.msr & MSR_TS_MASK;
+	msr |= vcpu->arch.shregs.msr & MSR_S;
 	kvmppc_set_msr(vcpu, msr);
 }
#48 lore raw

Re: [PATCH v4 8/8] KVM: PPC: Ultravisor: Check for MSR_S during hv_reset_msr

In reply to Claudio Carvalho
Quoted 4 lines · Claudio Carvalho

From: Michael Anderson <[email protected]>

- Check for MSR_S so that kvmppc_set_msr will include it. Prior to this change return to guest would not have the S bit set.

That sounds like it would be bad?

Please spell out what the practical impact of the patch is, ie. somewhere on the spectrum from "without this patch everything catches fire", to "this is not a bug but makes things clearer because ..."

cheers

Quoted 23 lines

- Patch based on comment from Paul Mackerras <[email protected]>

---
 arch/powerpc/kvm/book3s_64_mmu_hv.c | 1 +
 1 file changed, 1 insertion(+)
arch/powerpc/kvm/book3s_64_mmu_hv.c +1 -0
index ab3d484c5e2e..ab62a66f9b4e 100644
@@ -295,6 +295,7 @@ static void kvmppc_mmu_book3s_64_hv_reset_msr(struct kvm_vcpu *vcpu)
 		msr |= MSR_TS_S;
 	else
 		msr |= vcpu->arch.shregs.msr & MSR_TS_MASK;
+	msr |= vcpu->arch.shregs.msr & MSR_S;
 	kvmppc_set_msr(vcpu, msr);
 }
 
Signature · 1 line
2.20.1