KVM: PPC: Nested PAPR guests
Subsystems (4)
v2 cover KVM: PPC: Nested PAPR guests
There is existing support for nested guests on powernv hosts however the hcall interface this uses is not support by other PAPR hosts. A set of new hcalls will be added to PAPR to facilitate creating and managing guests by a regular partition in the following way:
- L1 and L0 negotiate capabilities with
H_GUEST_{G,S}ET_CAPABILITIES
- L1 requests the L0 create a L2 with
H_GUEST_CREATE and receives a handle to use in future hcalls
- L1 requests the L0 create a L2 vCPU with
H_GUEST_CREATE_VCPU
- L1 sets up the L2 using H_GUEST_SET and the
H_GUEST_VCPU_RUN input buffer
- L1 requests the L0 runs the L2 vCPU using H_GUEST_VCPU_RUN
- L2 returns to L1 with an exit reason and L1 reads the H_GUEST_VCPU_RUN output buffer populated by the L0
- L1 handles the exit using H_GET_STATE if necessary
- L1 reruns L2 vCPU with H_GUEST_VCPU_RUN
- L1 frees the L2 in the L0 with H_GUEST_DELETE
Further details are available in Documentation/powerpc/kvm-nested.rst.
This series adds KVM support for using this hcall interface as a regular PAPR partition, i.e. the L1. It does not add support for running as the L0.
The new hcalls have been implemented in the spapr qemu model for testing.
This is available at https://github.com/mikey/qemu/tree/kvm-papr
There are scripts available to assist in setting up an environment for testing nested guests at https://github.com/mikey/kvm-powervm-test
A tree with this series is available at
https://github.com/iamjpn/linux/tree/features/kvm-papr
Thanks to Amit Machhiwal, Kautuk Consul, Vaibhav Jain, Michael Neuling, Shivaprasad Bhat, Harsh Prateek Bora, Paul Mackerras and Nicholas Piggin.
Change overview in v2:
- Rebase on top of kvm ppc prefix instruction support
- Make documentation an individual patch
- Move guest state buffer files from arch/powerpc/lib/ to arch/powerpc/kvm/
- Use kunit for testing guest state buffer
- Fix some build errors
- Change HEIR element from 4 bytes to 8 bytes
Previous revisions:
- v1: https://lore.kernel.org/linuxppc-dev/[email protected]/
Jordan Niethe (5):
KVM: PPC: Use getters and setters for vcpu register state
KVM: PPC: Add fpr getters and setters
KVM: PPC: Add vr getters and setters
KVM: PPC: Add helper library for Guest State Buffers
KVM: PPC: Add support for nested PAPR guests
Michael Neuling (1):
docs: powerpc: Document nested KVM on POWER
Documentation/powerpc/index.rst | 1 + Documentation/powerpc/kvm-nested.rst | 636 +++++++++++ arch/powerpc/Kconfig.debug | 12 + arch/powerpc/include/asm/guest-state-buffer.h | 988 ++++++++++++++++++ arch/powerpc/include/asm/hvcall.h | 30 + arch/powerpc/include/asm/kvm_book3s.h | 205 +++- arch/powerpc/include/asm/kvm_book3s_64.h | 6 + arch/powerpc/include/asm/kvm_booke.h | 10 + arch/powerpc/include/asm/kvm_host.h | 21 + arch/powerpc/include/asm/kvm_ppc.h | 80 +- arch/powerpc/include/asm/plpar_wrappers.h | 198 ++++ arch/powerpc/kvm/Makefile | 4 + arch/powerpc/kvm/book3s.c | 38 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 9 +- arch/powerpc/kvm/book3s_64_vio.c | 4 +- arch/powerpc/kvm/book3s_hv.c | 336 ++++-- arch/powerpc/kvm/book3s_hv.h | 65 ++ arch/powerpc/kvm/book3s_hv_builtin.c | 10 +- arch/powerpc/kvm/book3s_hv_nested.c | 38 +- arch/powerpc/kvm/book3s_hv_p9_entry.c | 4 +- arch/powerpc/kvm/book3s_hv_papr.c | 940 +++++++++++++++++ arch/powerpc/kvm/book3s_hv_ras.c | 5 +- arch/powerpc/kvm/book3s_hv_rm_mmu.c | 8 +- arch/powerpc/kvm/book3s_hv_rm_xics.c | 4 +- arch/powerpc/kvm/book3s_xive.c | 9 +- arch/powerpc/kvm/emulate_loadstore.c | 6 +- arch/powerpc/kvm/guest-state-buffer.c | 612 +++++++++++ arch/powerpc/kvm/powerpc.c | 76 +- arch/powerpc/kvm/test-guest-state-buffer.c | 321 ++++++ 30 files changed, 4467 insertions(+), 213 deletions(-) create mode 100644 Documentation/powerpc/kvm-nested.rst create mode 100644 arch/powerpc/include/asm/guest-state-buffer.h create mode 100644 arch/powerpc/kvm/book3s_hv_papr.c create mode 100644 arch/powerpc/kvm/guest-state-buffer.c create mode 100644 arch/powerpc/kvm/test-guest-state-buffer.c
Signature · 1 line
2.31.1
RFC PATCH v2 1/6 KVM: PPC: Use getters and setters for vcpu register state
There are already some getter and setter functions used for accessing vcpu register state, e.g. kvmppc_get_pc(). There are also more complicated examples that are generated by macros like kvmppc_get_sprg0() which are generated by the SHARED_SPRNG_WRAPPER() macro.
In the new PAPR API for nested guest partitions the L1 is required to communicate with the L0 to modify and read nested guest state.
Prepare to support this by replacing direct accesses to vcpu register state with wrapper functions. Follow the existing pattern of using macros to generate individual wrappers. These wrappers will be augmented for supporting PAPR nested guests later.
--- arch/powerpc/include/asm/kvm_book3s.h | 68 +++++++- arch/powerpc/include/asm/kvm_ppc.h | 48 ++++-- arch/powerpc/kvm/book3s.c | 22 +-- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 9 +- arch/powerpc/kvm/book3s_64_vio.c | 4 +- arch/powerpc/kvm/book3s_hv.c | 222 +++++++++++++------------ arch/powerpc/kvm/book3s_hv.h | 59 +++++++ arch/powerpc/kvm/book3s_hv_builtin.c | 10 +- arch/powerpc/kvm/book3s_hv_p9_entry.c | 4 +- arch/powerpc/kvm/book3s_hv_ras.c | 5 +- arch/powerpc/kvm/book3s_hv_rm_mmu.c | 8 +- arch/powerpc/kvm/book3s_hv_rm_xics.c | 4 +- arch/powerpc/kvm/book3s_xive.c | 9 +- arch/powerpc/kvm/powerpc.c | 4 +- 15 files changed, 322 insertions(+), 158 deletions(-)
arch/powerpc/include/asm/kvm_book3s.h +67 -1
@@ -392,6 +392,16 @@ static inline ulong kvmppc_get_pc(struct kvm_vcpu *vcpu)return vcpu->arch.regs.nip; } +static inline void kvmppc_set_pid (struct kvm_vcpu *vcpu,u32 val) +{ + vcpu->arch.pid = val; +} + +static inline u32 kvmppc_get_pid (struct kvm_vcpu *vcpu) +{ +return vcpu->arch.pid; +} +static inline u64 kvmppc_get_msr (struct kvm_vcpu *vcpu);static inline bool kvmppc_need_byteswap (struct kvm_vcpu *vcpu) {
@@ -403,10 +413,66 @@ static inline ulong kvmppc_get_fault_dar(struct kvm_vcpu *vcpu)return vcpu->arch.fault_dar; } +#define BOOK3S_WRAPPER_SET (reg, size) \ +static inline void kvmppc_set_##reg(struct kvm_vcpu *vcpu, u##size val) \ +{ \ + \ + vcpu->arch.reg = val; \ +} + +#define BOOK3S_WRAPPER_GET (reg, size) \ +static inline u##size kvmppc_get_##reg(struct kvm_vcpu *vcpu) \ +{ \ + return vcpu->arch.reg; \ +} + +#define BOOK3S_WRAPPER (reg, size) \ + BOOK3S_WRAPPER_SET(reg, size) \ + BOOK3S_WRAPPER_GET(reg, size) \ + +BOOK3S_WRAPPER(tar, 64) +BOOK3S_WRAPPER(ebbhr, 64) +BOOK3S_WRAPPER(ebbrr, 64) +BOOK3S_WRAPPER(bescr, 64) +BOOK3S_WRAPPER(ic, 64) +BOOK3S_WRAPPER(vrsave, 64) + + +#define VCORE_WRAPPER_SET (reg, size) \ +static inline void kvmppc_set_##reg ##_hv(struct kvm_vcpu *vcpu, u##size val) \ +{ \ + vcpu->arch.vcore->reg = val; \ +} + +#define VCORE_WRAPPER_GET (reg, size) \ +static inline u##size kvmppc_get_##reg ##_hv(struct kvm_vcpu *vcpu) \ +{ \ + return vcpu->arch.vcore->reg; \ +} + +#define VCORE_WRAPPER (reg, size) \ + VCORE_WRAPPER_SET(reg, size) \ + VCORE_WRAPPER_GET(reg, size) \ + + +VCORE_WRAPPER(vtb, 64) +VCORE_WRAPPER(tb_offset, 64) +VCORE_WRAPPER(lpcr, 64) + +static inline u64 kvmppc_get_dec_expires (struct kvm_vcpu *vcpu) +{ +return vcpu->arch.dec_expires; +} + +static inline void kvmppc_set_dec_expires (struct kvm_vcpu *vcpu,u64 val) +{ + vcpu->arch.dec_expires = val; +} +/* Expiry time of vcpu DEC relative to host TB */ static inline u64 kvmppc_dec_expires_host_tb (struct kvm_vcpu *vcpu) { -return vcpu->arch.dec_expires - vcpu->arch.vcore->tb_offset; +return kvmppc_get_dec_expires (vcpu) -kvmppc_get_tb_offset_hv (vcpu); }static inline bool is_kvmppc_resume_guest (int r)
arch/powerpc/include/asm/kvm_ppc.h +38 -10
85 lines not on this page · raw message
arch/powerpc/kvm/book3s.c +11 -11
@@ -565,7 +565,7 @@ int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) regs->msr =kvmppc_get_msr (vcpu); regs->srr0 =kvmppc_get_srr0 (vcpu); regs->srr1 =kvmppc_get_srr1 (vcpu); - regs->pid = vcpu->arch.pid; + regs->pid =kvmppc_get_pid (vcpu); regs->sprg0 =kvmppc_get_sprg0 (vcpu); regs->sprg1 =kvmppc_get_sprg1 (vcpu); regs->sprg2 =kvmppc_get_sprg2 (vcpu);
@@ -683,19 +683,19 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id, *val =get_reg_val (id, vcpu->arch.fscr);break ;case KVM_REG_PPC_TAR : - *val =get_reg_val (id, vcpu->arch.tar); + *val =get_reg_val (id,kvmppc_get_tar (vcpu));break ;case KVM_REG_PPC_EBBHR : - *val =get_reg_val (id, vcpu->arch.ebbhr); + *val =get_reg_val (id,kvmppc_get_ebbhr (vcpu));break ;case KVM_REG_PPC_EBBRR : - *val =get_reg_val (id, vcpu->arch.ebbrr); + *val =get_reg_val (id,kvmppc_get_ebbrr (vcpu));break ;case KVM_REG_PPC_BESCR : - *val =get_reg_val (id, vcpu->arch.bescr); + *val =get_reg_val (id,kvmppc_get_bescr (vcpu));break ;case KVM_REG_PPC_IC : - *val =get_reg_val (id, vcpu->arch.ic); + *val =get_reg_val (id,kvmppc_get_ic (vcpu));break ;default : r = -EINVAL ;
@@ -768,19 +768,19 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id, vcpu->arch.fscr =set_reg_val (id, *val);break ;case KVM_REG_PPC_TAR : - vcpu->arch.tar =set_reg_val (id, *val); +kvmppc_set_tar (vcpu,set_reg_val (id, *val));break ;case KVM_REG_PPC_EBBHR : - vcpu->arch.ebbhr =set_reg_val (id, *val); +kvmppc_set_ebbhr (vcpu,set_reg_val (id, *val));break ;case KVM_REG_PPC_EBBRR : - vcpu->arch.ebbrr =set_reg_val (id, *val); +kvmppc_set_ebbrr (vcpu,set_reg_val (id, *val));break ;case KVM_REG_PPC_BESCR : - vcpu->arch.bescr =set_reg_val (id, *val); +kvmppc_set_bescr (vcpu,set_reg_val (id, *val));break ;case KVM_REG_PPC_IC : - vcpu->arch.ic =set_reg_val (id, *val); +kvmppc_set_ic (vcpu,set_reg_val (id, *val));break ;default : r = -EINVAL ;
arch/powerpc/kvm/book3s_64_mmu_hv.c +2 -2
@@ -347,7 +347,7 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr,unsigned long v, orig_v, gr;__be64 *hptep;long int index; -int virtmode = vcpu->arch.shregs.msr & (data ?MSR_DR :MSR_IR ); +int virtmode =kvmppc_get_msr (vcpu) & (data ?MSR_DR :MSR_IR );if (kvm_is_radix (vcpu->kvm))return kvmppc_mmu_radix_xlate (vcpu, eaddr, gpte, data, iswrite);
@@ -385,7 +385,7 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr,/* Get PP bits and key for permission check */ pp = gr & (HPTE_R_PP0 |HPTE_R_PP ); - key = (vcpu->arch.shregs.msr &MSR_PR ) ?SLB_VSID_KP :SLB_VSID_KS ; + key = (kvmppc_get_msr (vcpu) &MSR_PR ) ?SLB_VSID_KP :SLB_VSID_KS ; key &= slb_v;/* Calculate permissions */
arch/powerpc/kvm/book3s_64_mmu_radix.c +5 -4
@@ -15,6 +15,7 @@#include <asm/kvm_ppc.h> #include <asm/kvm_book3s.h> +#include "book3s_hv.h" #include <asm/page.h> #include <asm/mmu.h> #include <asm/pgalloc.h>
@@ -96,7 +97,7 @@ static long kvmhv_copy_tofrom_guest_radix(struct kvm_vcpu *vcpu, gva_t eaddr, void *to, void *from, unsigned long n) { int lpid = vcpu->kvm->arch.lpid; -int pid = vcpu->arch.pid; +int pid =kvmppc_get_pid (vcpu);/* This would cause a data segment intr so don't allow the access */ if (eaddr & (0x3FFUL <<52 ))
@@ -270,7 +271,7 @@ int kvmppc_mmu_radix_xlate(struct kvm_vcpu *vcpu, gva_t eaddr,/* Work out effective PID */ switch (eaddr >>62 ) {case 0 : - pid = vcpu->arch.pid; + pid =kvmppc_get_pid (vcpu);break ;case 3 : pid =0 ;
@@ -294,9 +295,9 @@ int kvmppc_mmu_radix_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, }else {if (!(pte & _PAGE_PRIVILEGED)) {/* Check AMR/IAMR to see if strict mode is in force */ -if (vcpu->arch.amr & (1ul <<62 )) +if (kvmppc_get_amr_hv (vcpu) & (1ul <<62 )) gpte->may_read =0 ; -if (vcpu->arch.amr & (1ul <<63 )) +if (kvmppc_get_amr_hv (vcpu) & (1ul <<63 )) gpte->may_write =0 ;if (vcpu->arch.iamr & (1ul <<62 )) gpte->may_execute =0 ;
arch/powerpc/kvm/book3s_64_vio.c +2 -2
@@ -786,12 +786,12 @@ long kvmppc_h_get_tce(struct kvm_vcpu *vcpu, unsigned long liobn, idx = (ioba >> stt->page_shift) - stt->offset; page = stt->pages[idx /TCES_PER_PAGE ];if (!page) { - vcpu->arch.regs.gpr[4 ] =0 ; +kvmppc_set_gpr (vcpu,4 ,0 );return H_SUCCESS ; } tbl = (u64 *)page_address (page); - vcpu->arch.regs.gpr[4 ] = tbl[idx %TCES_PER_PAGE ]; +kvmppc_set_gpr (vcpu,4 , tbl[idx %TCES_PER_PAGE ]);return H_SUCCESS ; }
arch/powerpc/kvm/book3s_hv.c +114 -108
638 lines not on this page · raw message
arch/powerpc/kvm/book3s_hv.h +59 -0
62 lines not on this page · raw message
arch/powerpc/kvm/book3s_hv_builtin.c +7 -3
@@ -182,9 +182,13 @@ EXPORT_SYMBOL_GPL(kvmppc_hwrng_present);long kvmppc_rm_h_random (struct kvm_vcpu *vcpu) { +unsigned long rand; +if (ppc_md.get_random_seed && - ppc_md.get_random_seed (&vcpu->arch.regs.gpr[4 ])) + ppc_md.get_random_seed (&rand)) { +kvmppc_set_gpr (vcpu,4 , rand);return H_SUCCESS ; + }return H_HARDWARE ; }
@@ -510,7 +514,7 @@ void kvmppc_set_msr_hv(struct kvm_vcpu *vcpu, u64 msr)*/ if ((msr &MSR_TS_MASK ) =MSR_TS_MASK ) msr &= ~MSR_TS_MASK ; - vcpu->arch.shregs.msr = msr; +kvmppc_set_msr_fast (vcpu, msr);kvmppc_end_cede (vcpu); }EXPORT_SYMBOL_GPL (kvmppc_set_msr_hv);
@@ -548,7 +552,7 @@ static void inject_interrupt(struct kvm_vcpu *vcpu, int vec, u64 srr1_flags)kvmppc_set_srr0 (vcpu, pc);kvmppc_set_srr1 (vcpu, (msr &SRR1_MSR_BITS ) | srr1_flags);kvmppc_set_pc (vcpu, new_pc); - vcpu->arch.shregs.msr = new_msr; +kvmppc_set_msr_fast (vcpu, new_msr); }void kvmppc_inject_interrupt_hv (struct kvm_vcpu *vcpu,int vec,u64 srr1_flags)
arch/powerpc/kvm/book3s_hv_p9_entry.c +2 -2
@@ -305,7 +305,7 @@ static void switch_mmu_to_guest_radix(struct kvm *kvm, struct kvm_vcpu *vcpu, u6u32 pid; lpid = nested ? nested->shadow_lpid : kvm->arch.lpid; - pid = vcpu->arch.pid; + pid =kvmppc_get_pid (vcpu);/* * Prior memory accesses to host PID Q3 must be completed before we
@@ -330,7 +330,7 @@ static void switch_mmu_to_guest_hpt(struct kvm *kvm, struct kvm_vcpu *vcpu, u64int i; lpid = kvm->arch.lpid; - pid = vcpu->arch.pid; + pid =kvmppc_get_pid (vcpu);/* * See switch_mmu_to_guest_radix. ptesync should not be required here
arch/powerpc/kvm/book3s_hv_ras.c +3 -2
@@ -15,6 +15,7 @@#include <asm/cputhreads.h> #include <asm/hmi.h> #include <asm/kvm_ppc.h> +#include "book3s_hv.h" /* SRR1 bits for machine check on POWER7 */ #define SRR1_MC_LDSTERR (1ul << (63-42))
@@ -173,14 +174,14 @@ long kvmppc_p9_realmode_hmi_handler(struct kvm_vcpu *vcpu) ppc_md.hmi_exception_early (NULL ); out: -if (vc->tb_offset) { +if (kvmppc_get_tb_offset_hv (vcpu)) {u64 new_tb =mftb () + vc->tb_offset;mtspr (SPRN_TBU40 , new_tb);if ((mftb () &0xffffff ) < (new_tb &0xffffff )) { new_tb +=0x1000000 ;mtspr (SPRN_TBU40 , new_tb); } - vc->tb_offset_applied = vc->tb_offset; + vc->tb_offset_applied =kvmppc_get_tb_offset_hv (vcpu); }return ret;
arch/powerpc/kvm/book3s_hv_rm_mmu.c +4 -4
@@ -776,8 +776,8 @@ long kvmppc_h_read(struct kvm_vcpu *vcpu, unsigned long flags, r = rev[i].guest_rpte | (r & (HPTE_R_R |HPTE_R_C )); r &= ~HPTE_GR_RESERVED ; } - vcpu->arch.regs.gpr[4 + i *2 ] = v; - vcpu->arch.regs.gpr[5 + i *2 ] = r; +kvmppc_set_gpr (vcpu,4 + i *2 , v); +kvmppc_set_gpr (vcpu,5 + i *2 , r); }return H_SUCCESS ; }
@@ -824,7 +824,7 @@ long kvmppc_h_clear_ref(struct kvm_vcpu *vcpu, unsigned long flags, } } } - vcpu->arch.regs.gpr[4 ] = gr; +kvmppc_set_gpr (vcpu,4 , gr); ret =H_SUCCESS ; out:unlock_hpte (hpte, v & ~HPTE_V_HVLOCK );
@@ -872,7 +872,7 @@ long kvmppc_h_clear_mod(struct kvm_vcpu *vcpu, unsigned long flags,kvmppc_set_dirty_from_hpte (kvm, v, gr); } } - vcpu->arch.regs.gpr[4 ] = gr; +kvmppc_set_gpr (vcpu,4 , gr); ret =H_SUCCESS ; out:unlock_hpte (hpte, v & ~HPTE_V_HVLOCK );
arch/powerpc/kvm/book3s_hv_rm_xics.c +2 -2
@@ -481,7 +481,7 @@ static void icp_rm_down_cppr(struct kvmppc_xics *xics, struct kvmppc_icp *icp,unsigned long xics_rm_h_xirr_x (struct kvm_vcpu *vcpu) { - vcpu->arch.regs.gpr[5 ] =get_tb (); +kvmppc_set_gpr (vcpu,5 ,get_tb ());return xics_rm_h_xirr (vcpu); }
@@ -518,7 +518,7 @@ unsigned long xics_rm_h_xirr(struct kvm_vcpu *vcpu) }while (!icp_rm_try_update (icp, old_state, new_state));/* Return the result in GPR4 */ - vcpu->arch.regs.gpr[4 ] = xirr; +kvmppc_set_gpr (vcpu,4 , xirr);return check_too_hard (xics, icp); }
arch/powerpc/kvm/book3s_xive.c +4 -5
@@ -29,6 +29,7 @@#include <linux/seq_file.h> #include "book3s_xive.h" +#include "book3s_hv.h" #define __x_eoi_page (xd) ((void __iomem *)((xd)->eoi_mmio))#define __x_trig_page (xd) ((void __iomem *)((xd)->trig_mmio))
@@ -328,7 +329,7 @@ static unsigned long xive_vm_h_xirr(struct kvm_vcpu *vcpu)*/ /* Return interrupt and old CPPR in GPR4 */ - vcpu->arch.regs.gpr[4 ] = hirq | (old_cppr <<24 ); +kvmppc_set_gpr (vcpu,4 , hirq | (old_cppr <<24 ));return H_SUCCESS ; }
@@ -364,7 +365,7 @@ static unsigned long xive_vm_h_ipoll(struct kvm_vcpu *vcpu, unsigned long server hirq =xive_vm_scan_interrupts (xc, pending, scan_poll);/* Return interrupt and old CPPR in GPR4 */ - vcpu->arch.regs.gpr[4 ] = hirq | (xc->cppr <<24 ); +kvmppc_set_gpr (vcpu,4 , hirq | (xc->cppr <<24 ));return H_SUCCESS ; }
@@ -2779,8 +2780,6 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)int kvmppc_xive_xics_hcall (struct kvm_vcpu *vcpu,u32 req) { -struct kvmppc_vcore *vc = vcpu->arch.vcore; -/* The VM should have configured XICS mode before doing XICS hcalls. */ if (!kvmppc_xics_enabled (vcpu))return H_TOO_HARD ;
@@ -2799,7 +2798,7 @@ int kvmppc_xive_xics_hcall(struct kvm_vcpu *vcpu, u32 req)return xive_vm_h_ipoll (vcpu,kvmppc_get_gpr (vcpu,4 ));case H_XIRR_X :xive_vm_h_xirr (vcpu); -kvmppc_set_gpr (vcpu,5 ,get_tb () + vc->tb_offset); +kvmppc_set_gpr (vcpu,5 ,get_tb () +kvmppc_get_tb_offset_hv (vcpu));return H_SUCCESS ; }
arch/powerpc/kvm/powerpc.c +2 -2
@@ -1729,7 +1729,7 @@ int kvm_vcpu_ioctl_get_one_reg(struct kvm_vcpu *vcpu, struct kvm_one_reg *reg) val =get_reg_val (reg->id, vcpu->arch.vr.vscr.u[3 ]);break ;case KVM_REG_PPC_VRSAVE : - val =get_reg_val (reg->id, vcpu->arch.vrsave); + val =get_reg_val (reg->id,kvmppc_get_vrsave (vcpu));break ;#endif /* CONFIG_ALTIVEC */ default :
@@ -1784,7 +1784,7 @@ int kvm_vcpu_ioctl_set_one_reg(struct kvm_vcpu *vcpu, struct kvm_one_reg *reg) r = -ENXIO ; break; } - vcpu->arch.vrsave =set_reg_val (reg->id, val); +kvmppc_set_vrsave (vcpu,set_reg_val (reg->id, val));break ;#endif /* CONFIG_ALTIVEC */ default :
Signature · 1 line
2.31.1
Nicholas Piggin
reviewerRe: [RFC PATCH v2 1/6] KVM: PPC: Use getters and setters for vcpu register state
Quoted 93 lines · Jordan Niethe
There are already some getter and setter functions used for accessing vcpu register state, e.g.
kvmppc_get_pc(). There are also more complicated examples that are generated by macros likekvmppc_get_sprg0()which are generated by theSHARED_SPRNG_WRAPPER()macro.In the new PAPR API for nested guest partitions the L1 is required to communicate with the L0 to modify and read nested guest state.
Prepare to support this by replacing direct accesses to vcpu register state with wrapper functions. Follow the existing pattern of using macros to generate individual wrappers. These wrappers will be augmented for supporting PAPR nested guests later.
Signed-off-by Jordan Niethe <[email protected]>--- arch/powerpc/include/asm/kvm_book3s.h | 68 +++++++- arch/powerpc/include/asm/kvm_ppc.h | 48 ++++-- arch/powerpc/kvm/book3s.c | 22 +-- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 9 +- arch/powerpc/kvm/book3s_64_vio.c | 4 +- arch/powerpc/kvm/book3s_hv.c | 222 +++++++++++++------------ arch/powerpc/kvm/book3s_hv.h | 59 +++++++ arch/powerpc/kvm/book3s_hv_builtin.c | 10 +- arch/powerpc/kvm/book3s_hv_p9_entry.c | 4 +- arch/powerpc/kvm/book3s_hv_ras.c | 5 +- arch/powerpc/kvm/book3s_hv_rm_mmu.c | 8 +- arch/powerpc/kvm/book3s_hv_rm_xics.c | 4 +- arch/powerpc/kvm/book3s_xive.c | 9 +- arch/powerpc/kvm/powerpc.c | 4 +- 15 files changed, 322 insertions(+), 158 deletions(-)arch/powerpc/include/asm/kvm_book3s.h +45
-0@@ -392,6 +392,16 @@ static inline ulong kvmppc_get_pc(struct kvm_vcpu *vcpu)return vcpu->arch.regs.nip; } +static inline void kvmppc_set_pid (struct kvm_vcpu *vcpu,u32 val) +{ + vcpu->arch.pid = val; +} + +static inline u32 kvmppc_get_pid (struct kvm_vcpu *vcpu) +{ +return vcpu->arch.pid; +} +static inline u64 kvmppc_get_msr (struct kvm_vcpu *vcpu);static inline bool kvmppc_need_byteswap (struct kvm_vcpu *vcpu) {@@ -403,10 +413,66 @@ static inline ulong kvmppc_get_fault_dar(struct kvm_vcpu *vcpu)return vcpu->arch.fault_dar; } +#define BOOK3S_WRAPPER_SET (reg, size) \ +static inline void kvmppc_set_##reg(struct kvm_vcpu *vcpu, u##size val) \ +{ \ + \ + vcpu->arch.reg = val; \ +} + +#define BOOK3S_WRAPPER_GET (reg, size) \ +static inline u##size kvmppc_get_##reg(struct kvm_vcpu *vcpu) \ +{ \ + return vcpu->arch.reg; \ +} + +#define BOOK3S_WRAPPER (reg, size) \ + BOOK3S_WRAPPER_SET(reg, size) \ + BOOK3S_WRAPPER_GET(reg, size) \ + +BOOK3S_WRAPPER(tar, 64) +BOOK3S_WRAPPER(ebbhr, 64) +BOOK3S_WRAPPER(ebbrr, 64) +BOOK3S_WRAPPER(bescr, 64) +BOOK3S_WRAPPER(ic, 64) +BOOK3S_WRAPPER(vrsave, 64) + + +#define VCORE_WRAPPER_SET (reg, size) \ +static inline void kvmppc_set_##reg ##_hv(struct kvm_vcpu *vcpu, u##size val) \ +{ \ + vcpu->arch.vcore->reg = val; \ +} + +#define VCORE_WRAPPER_GET (reg, size) \ +static inline u##size kvmppc_get_##reg ##_hv(struct kvm_vcpu *vcpu) \ +{ \ + return vcpu->arch.vcore->reg; \
arch/powerpc/include/asm/kvm_book3s.h static inline ulong kvmppc_get_fault_dar(struct kvm_vcpu *vcpu) · line 460
Quoted 10 lines
arch/powerpc/include/asm/kvm_book3s.h +10
-0+} + +#define VCORE_WRAPPER (reg, size) \ + VCORE_WRAPPER_SET(reg, size) \ + VCORE_WRAPPER_GET(reg, size) \ + + +VCORE_WRAPPER(vtb, 64) +VCORE_WRAPPER(tb_offset, 64) +VCORE_WRAPPER (lpcr,64 )
The general idea is fine, some of the names could use a bit of improvement. What's a BOOK3S_WRAPPER for example, is it not a VCPU_WRAPPER, or alternatively why isn't a VCORE_WRAPPER Book3S as well?
Quoted 24 lines
arch/powerpc/include/asm/kvm_book3s.h +12
-1+ +static inline u64 kvmppc_get_dec_expires (struct kvm_vcpu *vcpu) +{ +return vcpu->arch.dec_expires; +} + +static inline void kvmppc_set_dec_expires (struct kvm_vcpu *vcpu,u64 val) +{ + vcpu->arch.dec_expires = val; +} +/* Expiry time of vcpu DEC relative to host TB */ static inline u64 kvmppc_dec_expires_host_tb (struct kvm_vcpu *vcpu) { -return vcpu->arch.dec_expires - vcpu->arch.vcore->tb_offset; +return kvmppc_get_dec_expires (vcpu) -kvmppc_get_tb_offset_hv (vcpu); }static inline bool is_kvmppc_resume_guest (int r)arch/powerpc/include/asm/kvm_ppc.h +0
-0
arch/powerpc/include/asm/kvm_ppc.h static inline ulong kvmppc_get_##reg(struct kvm_vcpu *vcpu) \ · line 942
Quoted 8 lines
arch/powerpc/include/asm/kvm_ppc.h +1
-1@@ -936,7 +936,7 @@ static inline ulong kvmppc_get_##reg(struct kvm_vcpu *vcpu) \#define SPRNG_WRAPPER_SET (reg, bookehv_spr) \ static inline void kvmppc_set_##reg(struct kvm_vcpu *vcpu, ulong val) \ { \ - mtspr(bookehv_spr, val); \ + mtspr(bookehv_spr, val); \ } \#define SHARED_WRAPPER_GET (reg, size) \
Stray hunk I think.
Quoted 19 lines
arch/powerpc/include/asm/kvm_ppc.h +15
-0@@ -957,10 +957,32 @@ static inline void kvmppc_set_##reg(struct kvm_vcpu *vcpu, u##size val) \ vcpu->arch.shared->reg = cpu_to_le##size(val); \ } \ +#define SHARED_CACHE_WRAPPER_GET (reg, size) \ +static inline u##size kvmppc_get_##reg(struct kvm_vcpu *vcpu) \ +{ \ + if (kvmppc_shared_big_endian(vcpu)) \ + return be##size##_to_cpu(vcpu->arch.shared->reg); \ + else \ + return le##size##_to_cpu(vcpu->arch.shared->reg); \ +} \ + +#define SHARED_CACHE_WRAPPER_SET (reg, size) \ +static inline void kvmppc_set_##reg(struct kvm_vcpu *vcpu, u##size val) \ +{ \ + if (kvmppc_shared_big_endian(vcpu)) \ + vcpu->arch.shared->reg = cpu_to_be##size(val); \ + else \
arch/powerpc/include/asm/kvm_ppc.h static inline void kvmppc_set_##reg(struct kvm_vcpu *vcpu, u##size val) \ · line 984
Quoted 10 lines
arch/powerpc/include/asm/kvm_ppc.h +6
-0+ vcpu->arch.shared->reg = cpu_to_le##size(val); \ +} \ +#define SHARED_WRAPPER (reg, size) \ SHARED_WRAPPER_GET(reg, size) \ SHARED_WRAPPER_SET(reg, size) \ +#define SHARED_CACHE_WRAPPER (reg, size) \ + SHARED_CACHE_WRAPPER_GET(reg, size) \ + SHARED_CACHE_WRAPPER_SET(reg, size) \
SHARED_CACHE_WRAPPER that does the same thing as SHARED_WRAPPER.
I know some of the names are a but crufty but it's probably a good time to rethink them a bit.
KVMPPC_VCPU_SHARED_REG_ACCESSOR or something like that. A few more keystrokes could help imensely.
arch/powerpc/kvm/book3s_hv_p9_entry.c static void switch_mmu_to_guest_radix(struct kvm *kvm, struct kvm_vcpu *vcpu, u6 · line 311
Quoted 13 lines
arch/powerpc/kvm/book3s_hv_p9_entry.c +1
-1@@ -305,7 +305,7 @@ static void switch_mmu_to_guest_radix(struct kvm *kvm, struct kvm_vcpu *vcpu, u6u32 pid; lpid = nested ? nested->shadow_lpid : kvm->arch.lpid; - pid = vcpu->arch.pid; + pid =kvmppc_get_pid (vcpu);/* * Prior memory accesses to host PID Q3 must be completed before we
Could add some accessors for get_lpid / get_guest_id which check for the correct KVM mode maybe.
Thanks,
Nick
Re: [RFC PATCH v2 1/6] KVM: PPC: Use getters and setters for vcpu register state
On Wed, Jun 7, 2023 at 5:53 PM Nicholas Piggin <[email protected]> wrote:
[snip]
Quoted 5 lines
The general idea is fine, some of the names could use a bit of improvement. What's a
BOOK3S_WRAPPERfor example, is it not aVCPU_WRAPPER, or alternatively why isn't aVCORE_WRAPPERBook3S as well?
Yeah the names are not great.
I didn't call it VCPU_WRAPPER because I wanted to keep separate BOOK3S_WRAPPER for book3s registersHV_WRAPPER for hv specific registers
I will change it to something like you suggested.
[snip]
Quoted 2 lines
Stray hunk I think.
Yep.
Quoted 27 lines
Quoted 27 lines
arch/powerpc/include/asm/kvm_ppc.h +18
-0@@ -957,10 +957,32 @@ static inline void kvmppc_set_##reg(struct kvm_vcpu *vcpu, u##size val) \ vcpu->arch.shared->reg = cpu_to_le##size(val); \ } \ +#define SHARED_CACHE_WRAPPER_GET (reg, size) \ +static inline u##size kvmppc_get_##reg(struct kvm_vcpu *vcpu) \ +{ \ + if (kvmppc_shared_big_endian(vcpu)) \ + return be##size##_to_cpu(vcpu->arch.shared->reg); \ + else \ + return le##size##_to_cpu(vcpu->arch.shared->reg); \ +} \ + +#define SHARED_CACHE_WRAPPER_SET (reg, size) \ +static inline void kvmppc_set_##reg(struct kvm_vcpu *vcpu, u##size val) \ +{ \ + if (kvmppc_shared_big_endian(vcpu)) \ + vcpu->arch.shared->reg = cpu_to_be##size(val); \ + else \ + vcpu->arch.shared->reg = cpu_to_le##size(val); \ +} \ +#define SHARED_WRAPPER (reg, size) \ SHARED_WRAPPER_GET(reg, size) \ SHARED_WRAPPER_SET(reg, size) \
Quoted 5 lines
arch/powerpc/include/asm/kvm_ppc.h static inline void kvmppc_set_##reg(struct kvm_vcpu *vcpu, u##size val) \ · line 984
Quoted 3 lines
arch/powerpc/include/asm/kvm_ppc.h +3
-0+#define SHARED_CACHE_WRAPPER (reg, size) \ + SHARED_CACHE_WRAPPER_GET(reg, size) \ + SHARED_CACHE_WRAPPER_SET(reg, size) \
SHARED_CACHE_WRAPPERthat does the same thing asSHARED_WRAPPER.
That changes once the guest state buffer IDs are included in a later patch.
Quoted 6 lines
I know some of the names are a but crufty but it's probably a good time to rethink them a bit.
KVMPPC_VCPU_SHARED_REG_ACCESSORor something like that. A few more keystrokes could help imensely.
Yes, I will do something like that, for the BOOK3S_WRAPPER and HV_WRAPPER
too.
Quoted 12 lines
Quoted 12 lines
arch/powerpc/kvm/book3s_hv_p9_entry.c +1
-1@@ -305,7 +305,7 @@ static void switch_mmu_to_guest_radix(struct kvm *kvm, struct kvm_vcpu *vcpu, u6u32 pid; lpid = nested ? nested->shadow_lpid : kvm->arch.lpid; - pid = vcpu->arch.pid; + pid =kvmppc_get_pid (vcpu);
Quoted 5 lines
arch/powerpc/kvm/book3s_hv_p9_entry.c static void switch_mmu_to_guest_radix(struct kvm *kvm, struct kvm_vcpu *vcpu, u6 · line 311
Quoted 2 lines
arch/powerpc/kvm/book3s_hv_p9_entry.c +0
-0/* * Prior memory accesses to host PID Q3 must be completed before we Could add some accessors for
get_lpid/get_guest_idwhich check for the correct KVM mode maybe.
True.
Thanks,
Jordan
Quoted 4 lines
Thanks,
Nick
RFC PATCH v2 2/6 KVM: PPC: Add fpr getters and setters
In reply to Jordan NietheAdd wrappers for fpr registers to prepare for supporting PAPR nested guests.
--- arch/powerpc/include/asm/kvm_book3s.h | 31 +++++++++++++++++++++++++++ arch/powerpc/include/asm/kvm_booke.h | 10 +++++++++ arch/powerpc/kvm/book3s.c | 16 +++++++------- arch/powerpc/kvm/emulate_loadstore.c | 2 +- arch/powerpc/kvm/powerpc.c | 22 +++++++++---------- 5 files changed, 61 insertions(+), 20 deletions(-)
arch/powerpc/include/asm/kvm_book3s.h +31 -0
@@ -413,6 +413,37 @@ static inline ulong kvmppc_get_fault_dar(struct kvm_vcpu *vcpu)return vcpu->arch.fault_dar; } +static inline u64 kvmppc_get_fpr (struct kvm_vcpu *vcpu,int i) +{ +return vcpu->arch.fp.fpr[i][TS_FPROFFSET ]; +} + +static inline void kvmppc_set_fpr (struct kvm_vcpu *vcpu,int i,u64 val) +{ + vcpu->arch.fp.fpr[i][TS_FPROFFSET ] = val; +} + +static inline u64 kvmppc_get_fpscr (struct kvm_vcpu *vcpu) +{ +return vcpu->arch.fp.fpscr; +} + +static inline void kvmppc_set_fpscr (struct kvm_vcpu *vcpu,u64 val) +{ + vcpu->arch.fp.fpscr = val; +} + + +static inline u64 kvmppc_get_vsx_fpr (struct kvm_vcpu *vcpu,int i,int j) +{ +return vcpu->arch.fp.fpr[i][j]; +} + +static inline void kvmppc_set_vsx_fpr (struct kvm_vcpu *vcpu,int i,int j,u64 val) +{ + vcpu->arch.fp.fpr[i][j] = val; +} +#define BOOK3S_WRAPPER_SET (reg, size) \ static inline void kvmppc_set_##reg(struct kvm_vcpu *vcpu, u##size val) \ { \
arch/powerpc/include/asm/kvm_booke.h +10 -0
@@ -89,6 +89,16 @@ static inline ulong kvmppc_get_pc(struct kvm_vcpu *vcpu)return vcpu->arch.regs.nip; } +static inline void kvmppc_set_fpr (struct kvm_vcpu *vcpu,int i,u64 val) +{ + vcpu->arch.fp.fpr[i][TS_FPROFFSET ] = val; +} + +static inline u64 kvmppc_get_fpr (struct kvm_vcpu *vcpu,int i) +{ +return vcpu->arch.fp.fpr[i][TS_FPROFFSET ]; +} +#ifdef CONFIG_BOOKE static inline ulong kvmppc_get_fault_dar (struct kvm_vcpu *vcpu) {
arch/powerpc/kvm/book3s.c +8 -8
@@ -636,17 +636,17 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,break ; case KVM_REG_PPC_FPR0 ... KVM_REG_PPC_FPR31: i = id -KVM_REG_PPC_FPR0 ; - *val =get_reg_val (id,VCPU_FPR (vcpu, i)); + *val =get_reg_val (id,kvmppc_get_fpr (vcpu, i));break ;case KVM_REG_PPC_FPSCR : - *val =get_reg_val (id, vcpu->arch.fp.fpscr); + *val =get_reg_val (id,kvmppc_get_fpscr (vcpu));break ;#ifdef CONFIG_VSX case KVM_REG_PPC_VSR0 ... KVM_REG_PPC_VSR31:if (cpu_has_feature (CPU_FTR_VSX )) { i = id -KVM_REG_PPC_VSR0 ; - val->vsxval[0 ] = vcpu->arch.fp.fpr[i][0 ]; - val->vsxval[1 ] = vcpu->arch.fp.fpr[i][1 ]; + val->vsxval[0 ] =kvmppc_get_vsx_fpr (vcpu, i,0 ); + val->vsxval[1 ] =kvmppc_get_vsx_fpr (vcpu, i,1 ); }else { r = -ENXIO ; }
@@ -724,7 +724,7 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,break ; case KVM_REG_PPC_FPR0 ... KVM_REG_PPC_FPR31: i = id -KVM_REG_PPC_FPR0 ; -VCPU_FPR (vcpu, i) =set_reg_val (id, *val); +kvmppc_set_fpr (vcpu, i,set_reg_val (id, *val));break ;case KVM_REG_PPC_FPSCR : vcpu->arch.fp.fpscr =set_reg_val (id, *val);
@@ -733,8 +733,8 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id, case KVM_REG_PPC_VSR0 ... KVM_REG_PPC_VSR31:if (cpu_has_feature (CPU_FTR_VSX )) { i = id -KVM_REG_PPC_VSR0 ; - vcpu->arch.fp.fpr[i][0 ] = val->vsxval[0 ]; - vcpu->arch.fp.fpr[i][1 ] = val->vsxval[1 ]; +kvmppc_set_vsx_fpr (vcpu, i,0 , val->vsxval[0 ]); +kvmppc_set_vsx_fpr (vcpu, i,1 , val->vsxval[1 ]); }else { r = -ENXIO ; }
@@ -765,7 +765,7 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,break ;#endif /* CONFIG_KVM_XIVE */ case KVM_REG_PPC_FSCR : - vcpu->arch.fscr =set_reg_val (id, *val); +kvmppc_set_fpscr (vcpu,set_reg_val (id, *val));break ;case KVM_REG_PPC_TAR :kvmppc_set_tar (vcpu,set_reg_val (id, *val));
arch/powerpc/kvm/emulate_loadstore.c +1 -1
@@ -250,7 +250,7 @@ int kvmppc_emulate_loadstore(struct kvm_vcpu *vcpu) vcpu->arch.mmio_sp64_extend =1 ; emulated =kvmppc_handle_store (vcpu, -VCPU_FPR (vcpu, op.reg), size,1 ); +kvmppc_get_fpr (vcpu, op.reg), size,1 );if ((op.type &UPDATE ) && (emulated !=EMULATE_FAIL ))kvmppc_set_gpr (vcpu, op.update_reg, op.ea);
arch/powerpc/kvm/powerpc.c +11 -11
@@ -938,7 +938,7 @@ static inline void kvmppc_set_vsr_dword(struct kvm_vcpu *vcpu, val.vsxval[offset] = gpr;VCPU_VSX_VR (vcpu, index -32 ) = val.vval; }else { -VCPU_VSX_FPR (vcpu, index, offset) = gpr; +kvmppc_set_vsx_fpr (vcpu, index, offset, gpr); } }
@@ -954,8 +954,8 @@ static inline void kvmppc_set_vsr_dword_dump(struct kvm_vcpu *vcpu, val.vsxval[1 ] = gpr;VCPU_VSX_VR (vcpu, index -32 ) = val.vval; }else { -VCPU_VSX_FPR (vcpu, index,0 ) = gpr; -VCPU_VSX_FPR (vcpu, index,1 ) = gpr; +kvmppc_set_vsx_fpr (vcpu, index,0 , gpr); +kvmppc_set_vsx_fpr (vcpu, index,1 , gpr); } }
@@ -974,8 +974,8 @@ static inline void kvmppc_set_vsr_word_dump(struct kvm_vcpu *vcpu, }else { val.vsx32val[0 ] = gpr; val.vsx32val[1 ] = gpr; -VCPU_VSX_FPR (vcpu, index,0 ) = val.vsxval[0 ]; -VCPU_VSX_FPR (vcpu, index,1 ) = val.vsxval[0 ]; +kvmppc_set_vsx_fpr (vcpu, index,0 , val.vsxval[0 ]); +kvmppc_set_vsx_fpr (vcpu, index,1 , val.vsxval[0 ]); } }
@@ -997,9 +997,9 @@ static inline void kvmppc_set_vsr_word(struct kvm_vcpu *vcpu, } else { dword_offset = offset /2 ; word_offset = offset %2 ; - val.vsxval[0 ] =VCPU_VSX_FPR (vcpu, index, dword_offset); + val.vsxval[0 ] =kvmppc_get_vsx_fpr (vcpu, index, dword_offset); val.vsx32val[word_offset] = gpr32; -VCPU_VSX_FPR (vcpu, index, dword_offset) = val.vsxval[0 ]; +kvmppc_set_vsx_fpr (vcpu, index, dword_offset, val.vsxval[0 ]); } }#endif /* CONFIG_VSX */
@@ -1194,14 +1194,14 @@ static void kvmppc_complete_mmio_load(struct kvm_vcpu *vcpu)if (vcpu->kvm->arch.kvm_ops->giveup_ext) vcpu->kvm->arch.kvm_ops->giveup_ext (vcpu,MSR_FP ); -VCPU_FPR (vcpu, vcpu->arch.io_gpr &KVM_MMIO_REG_MASK ) = gpr; +kvmppc_set_fpr (vcpu, vcpu->arch.io_gpr &KVM_MMIO_REG_MASK , gpr);break ;#ifdef CONFIG_PPC_BOOK3S case KVM_MMIO_REG_QPR : vcpu->arch.qpr[vcpu->arch.io_gpr &KVM_MMIO_REG_MASK ] = gpr;break ;case KVM_MMIO_REG_FQPR : -VCPU_FPR (vcpu, vcpu->arch.io_gpr &KVM_MMIO_REG_MASK ) = gpr; +kvmppc_set_fpr (vcpu, vcpu->arch.io_gpr &KVM_MMIO_REG_MASK , gpr); vcpu->arch.qpr[vcpu->arch.io_gpr &KVM_MMIO_REG_MASK ] = gpr;break ;#endif
@@ -1419,7 +1419,7 @@ static inline int kvmppc_get_vsr_data(struct kvm_vcpu *vcpu, int rs, u64 *val) }if (rs <32 ) { - *val =VCPU_VSX_FPR (vcpu, rs, vsx_offset); + *val =kvmppc_get_vsx_fpr (vcpu, rs, vsx_offset); }else { reg.vval =VCPU_VSX_VR (vcpu, rs -32 ); *val = reg.vsxval[vsx_offset];
@@ -1438,7 +1438,7 @@ static inline int kvmppc_get_vsr_data(struct kvm_vcpu *vcpu, int rs, u64 *val)if (rs <32 ) { dword_offset = vsx_offset /2 ; word_offset = vsx_offset %2 ; - reg.vsxval[0 ] =VCPU_VSX_FPR (vcpu, rs, dword_offset); + reg.vsxval[0 ] =kvmppc_get_vsx_fpr (vcpu, rs, dword_offset); *val = reg.vsx32val[word_offset]; }else { reg.vval =VCPU_VSX_VR (vcpu, rs -32 );
Signature · 1 line
2.31.1
Nicholas Piggin
reviewerRe: [RFC PATCH v2 2/6] KVM: PPC: Add fpr getters and setters
Quoted 148 lines · Jordan Niethe
Add wrappers for fpr registers to prepare for supporting PAPR nested guests.
Signed-off-by Jordan Niethe <[email protected]>--- arch/powerpc/include/asm/kvm_book3s.h | 31 +++++++++++++++++++++++++++ arch/powerpc/include/asm/kvm_booke.h | 10 +++++++++ arch/powerpc/kvm/book3s.c | 16 +++++++------- arch/powerpc/kvm/emulate_loadstore.c | 2 +- arch/powerpc/kvm/powerpc.c | 22 +++++++++---------- 5 files changed, 61 insertions(+), 20 deletions(-)arch/powerpc/include/asm/kvm_book3s.h +31
-0@@ -413,6 +413,37 @@ static inline ulong kvmppc_get_fault_dar(struct kvm_vcpu *vcpu)return vcpu->arch.fault_dar; } +static inline u64 kvmppc_get_fpr (struct kvm_vcpu *vcpu,int i) +{ +return vcpu->arch.fp.fpr[i][TS_FPROFFSET ]; +} + +static inline void kvmppc_set_fpr (struct kvm_vcpu *vcpu,int i,u64 val) +{ + vcpu->arch.fp.fpr[i][TS_FPROFFSET ] = val; +} + +static inline u64 kvmppc_get_fpscr (struct kvm_vcpu *vcpu) +{ +return vcpu->arch.fp.fpscr; +} + +static inline void kvmppc_set_fpscr (struct kvm_vcpu *vcpu,u64 val) +{ + vcpu->arch.fp.fpscr = val; +} + + +static inline u64 kvmppc_get_vsx_fpr (struct kvm_vcpu *vcpu,int i,int j) +{ +return vcpu->arch.fp.fpr[i][j]; +} + +static inline void kvmppc_set_vsx_fpr (struct kvm_vcpu *vcpu,int i,int j,u64 val) +{ + vcpu->arch.fp.fpr[i][j] = val; +} +#define BOOK3S_WRAPPER_SET (reg, size) \ static inline void kvmppc_set_##reg(struct kvm_vcpu *vcpu, u##size val) \ { \arch/powerpc/include/asm/kvm_booke.h +10
-0@@ -89,6 +89,16 @@ static inline ulong kvmppc_get_pc(struct kvm_vcpu *vcpu)return vcpu->arch.regs.nip; } +static inline void kvmppc_set_fpr (struct kvm_vcpu *vcpu,int i,u64 val) +{ + vcpu->arch.fp.fpr[i][TS_FPROFFSET ] = val; +} + +static inline u64 kvmppc_get_fpr (struct kvm_vcpu *vcpu,int i) +{ +return vcpu->arch.fp.fpr[i][TS_FPROFFSET ]; +} +#ifdef CONFIG_BOOKE static inline ulong kvmppc_get_fault_dar (struct kvm_vcpu *vcpu) {arch/powerpc/kvm/book3s.c +8
-8@@ -636,17 +636,17 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,break ; case KVM_REG_PPC_FPR0 ... KVM_REG_PPC_FPR31: i = id -KVM_REG_PPC_FPR0 ; - *val =get_reg_val (id,VCPU_FPR (vcpu, i)); + *val =get_reg_val (id,kvmppc_get_fpr (vcpu, i));break ;case KVM_REG_PPC_FPSCR : - *val =get_reg_val (id, vcpu->arch.fp.fpscr); + *val =get_reg_val (id,kvmppc_get_fpscr (vcpu));break ;#ifdef CONFIG_VSX case KVM_REG_PPC_VSR0 ... KVM_REG_PPC_VSR31:if (cpu_has_feature (CPU_FTR_VSX )) { i = id -KVM_REG_PPC_VSR0 ; - val->vsxval[0 ] = vcpu->arch.fp.fpr[i][0 ]; - val->vsxval[1 ] = vcpu->arch.fp.fpr[i][1 ]; + val->vsxval[0 ] =kvmppc_get_vsx_fpr (vcpu, i,0 ); + val->vsxval[1 ] =kvmppc_get_vsx_fpr (vcpu, i,1 ); }else { r = -ENXIO ; }@@ -724,7 +724,7 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,break ; case KVM_REG_PPC_FPR0 ... KVM_REG_PPC_FPR31: i = id -KVM_REG_PPC_FPR0 ; -VCPU_FPR (vcpu, i) =set_reg_val (id, *val); +kvmppc_set_fpr (vcpu, i,set_reg_val (id, *val));break ;case KVM_REG_PPC_FPSCR : vcpu->arch.fp.fpscr =set_reg_val (id, *val);@@ -733,8 +733,8 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id, case KVM_REG_PPC_VSR0 ... KVM_REG_PPC_VSR31:if (cpu_has_feature (CPU_FTR_VSX )) { i = id -KVM_REG_PPC_VSR0 ; - vcpu->arch.fp.fpr[i][0 ] = val->vsxval[0 ]; - vcpu->arch.fp.fpr[i][1 ] = val->vsxval[1 ]; +kvmppc_set_vsx_fpr (vcpu, i,0 , val->vsxval[0 ]); +kvmppc_set_vsx_fpr (vcpu, i,1 , val->vsxval[1 ]); }else { r = -ENXIO ; }@@ -765,7 +765,7 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,break ;#endif /* CONFIG_KVM_XIVE */ case KVM_REG_PPC_FSCR : - vcpu->arch.fscr =set_reg_val (id, *val); +kvmppc_set_fpscr (vcpu,set_reg_val (id, *val));break ;case KVM_REG_PPC_TAR :kvmppc_set_tar (vcpu,set_reg_val (id, *val));arch/powerpc/kvm/emulate_loadstore.c +1
-1@@ -250,7 +250,7 @@ int kvmppc_emulate_loadstore(struct kvm_vcpu *vcpu) vcpu->arch.mmio_sp64_extend =1 ; emulated =kvmppc_handle_store (vcpu, -VCPU_FPR (vcpu, op.reg), size,1 ); +kvmppc_get_fpr (vcpu, op.reg), size,1 );if ((op.type &UPDATE ) && (emulated !=EMULATE_FAIL ))kvmppc_set_gpr (vcpu, op.update_reg, op.ea);arch/powerpc/kvm/powerpc.c +0
-0
arch/powerpc/kvm/powerpc.c static inline void kvmppc_set_vsr_dword(struct kvm_vcpu *vcpu, · line 944
Quoted 8 lines
arch/powerpc/kvm/powerpc.c +1
-1@@ -938,7 +938,7 @@ static inline void kvmppc_set_vsr_dword(struct kvm_vcpu *vcpu, val.vsxval[offset] = gpr;VCPU_VSX_VR (vcpu, index -32 ) = val.vval; }else { -VCPU_VSX_FPR (vcpu, index, offset) = gpr; +kvmppc_set_vsx_fpr (vcpu, index, offset, gpr); } }
Is there a particular reason some reg sets are broken into their own patches? Looking at this hunk you think the VR one got missed, but it's in its own patch.
Not really a big deal but I wouldn't mind them all in one patch. Or at least the FP/VR/VSR ine one since they're quite regular and similar.
Thanks,
Nick
Re: [RFC PATCH v2 2/6] KVM: PPC: Add fpr getters and setters
On Wed, Jun 7, 2023 at 5:56 PM Nicholas Piggin <[email protected]> wrote:
[snip]
Quoted 7 lines
Is there a particular reason some reg sets are broken into their own patches? Looking at this hunk you think the VR one got missed, but it's in its own patch.
Not really a big deal but I wouldn't mind them all in one patch. Or at least the FP/VR/VSR ine one since they're quite regular and similar.
There's not really a reason,
Originally I had things even more broken apart but then thought one patch made
more sense. Part way through squashing the patches I had a change of heart
and thought I'd see if people had a preference.
I'll just finish the squashing for the next series.
Thanks,
Jordan
Quoted 4 lines
Thanks,
Nick
RFC PATCH v2 3/6 KVM: PPC: Add vr getters and setters
In reply to Jordan NietheAdd wrappers for vr registers to prepare for supporting PAPR nested guests.
--- arch/powerpc/include/asm/kvm_book3s.h | 20 +++++++++++ arch/powerpc/kvm/powerpc.c | 50 +++++++++++++-------------- 2 files changed, 45 insertions(+), 25 deletions(-)
arch/powerpc/include/asm/kvm_book3s.h +20 -0
@@ -444,6 +444,26 @@ static inline void kvmppc_set_vsx_fpr(struct kvm_vcpu *vcpu, int i, int j, u64 v vcpu->arch.fp.fpr[i][j] = val; } +static inline vector128 kvmppc_get_vsx_vr (struct kvm_vcpu *vcpu,int i) +{ +return vcpu->arch.vr.vr[i]; +} + +static inline void kvmppc_set_vsx_vr (struct kvm_vcpu *vcpu,int i,vector128 val) +{ + vcpu->arch.vr.vr[i] = val; +} + +static inline u32 kvmppc_get_vscr (struct kvm_vcpu *vcpu) +{ +return vcpu->arch.vr.vscr.u[3 ]; +} + +static inline void kvmppc_set_vscr (struct kvm_vcpu *vcpu,u32 val) +{ + vcpu->arch.vr.vscr.u[3 ] = val; +} +#define BOOK3S_WRAPPER_SET (reg, size) \ static inline void kvmppc_set_##reg(struct kvm_vcpu *vcpu, u##size val) \ { \
arch/powerpc/kvm/powerpc.c +25 -25
166 lines not on this page · raw message
Signature · 1 line
2.31.1
RFC PATCH v2 4/6 KVM: PPC: Add helper library for Guest State Buffers
In reply to Jordan NietheThe new PAPR nested guest API introduces the concept of a Guest State Buffer for communication about L2 guests between L1 and L0 hosts.
In the new API, the L0 manages the L2 on behalf of the L1. This means that if the L1 needs to change L2 state (e.g. GPRs, SPRs, partition table...), it must request the L0 perform the modification. If the nested host needs to read L2 state likewise this request must go through the L0.
The Guest State Buffer is a Type-Length-Value style data format defined in the PAPR which assigns all relevant partition state a unique identity. Unlike a typical TLV format the length is redundant as the length of each identity is fixed but is included for checking correctness.
A guest state buffer consists of an element count followed by a stream of elements, where elements are composed of an ID number, data length, then the data:
Header:
<---4 bytes--->
+----------------+----- | Element Count | Elements... +----------------+-----
Element:
<----2 bytes---> <-2 bytes-> <-Length bytes->
+----------------+-----------+----------------+ | Guest State ID | Length | Data | +----------------+-----------+----------------+
Guest State IDs have other attributes defined in the PAPR such as whether they are per thread or per guest, or read-only.
Introduce a library for using guest state buffers. This includes support for actions such as creating buffers, adding elements to buffers, reading the value of elements and parsing buffers. This will be used later by the PAPR nested guest support.
---
v2:
- Add missing #ifdef CONFIG_VSXs
- Move files from lib/ to kvm/
- Guard compilation on CONFIG_KVM_BOOK3S_HV_POSSIBLE
- Use kunit for guest state buffer tests
- Add configuration option for the tests
- Use macros for contiguous id ranges like GPRs
- Add some missing EXPORTs to functions
- HEIR element is a double word not a word
--- arch/powerpc/Kconfig.debug | 12 + arch/powerpc/include/asm/guest-state-buffer.h | 901 ++++++++++++++++++ arch/powerpc/include/asm/kvm_book3s.h | 2 + arch/powerpc/kvm/Makefile | 3 + arch/powerpc/kvm/guest-state-buffer.c | 563 +++++++++++ arch/powerpc/kvm/test-guest-state-buffer.c | 321 +++++++ 6 files changed, 1802 insertions(+) create mode 100644 arch/powerpc/include/asm/guest-state-buffer.h create mode 100644 arch/powerpc/kvm/guest-state-buffer.c create mode 100644 arch/powerpc/kvm/test-guest-state-buffer.c
arch/powerpc/Kconfig.debug +12 -0
@@ -82,6 +82,18 @@ config MSI_BITMAP_SELFTEST bool "Run self-tests of the MSI bitmap code" depends on DEBUG_KERNEL +config GUEST_STATE_BUFFER_TEST + def_tristate n + prompt "Enable Guest State Buffer unit tests" + depends on KUNIT + depends on KVM_BOOK3S_HV_POSSIBLE + default KUNIT_ALL_TESTS + help + The Guest State Buffer is a data format specified in the PAPR. + It is by hcalls to communicate the state of L2 guests between + the L1 and L0 hypervisors. Enable unit tests for the library + used to create and use guest state buffers. + config PPC_IRQ_SOFT_MASK_DEBUG bool "Include extra checks for powerpc irq soft masking" depends on PPC64
arch/powerpc/include/asm/guest-state-buffer.h new file +901 -0
901 lines not on this page · raw message
arch/powerpc/include/asm/kvm_book3s.h +2 -0
@@ -444,6 +444,7 @@ static inline void kvmppc_set_vsx_fpr(struct kvm_vcpu *vcpu, int i, int j, u64 v vcpu->arch.fp.fpr[i][j] = val; } +#ifdef CONFIG_VSX static inline vector128 kvmppc_get_vsx_vr (struct kvm_vcpu *vcpu,int i) {return vcpu->arch.vr.vr[i];
@@ -463,6 +464,7 @@ static inline void kvmppc_set_vscr(struct kvm_vcpu *vcpu, u32 val) { vcpu->arch.vr.vscr.u[3 ] = val; } +#endif #define BOOK3S_WRAPPER_SET (reg, size) \ static inline void kvmppc_set_##reg(struct kvm_vcpu *vcpu, u##size val) \
arch/powerpc/kvm/Makefile +3 -0
@@ -87,8 +87,11 @@ kvm-book3s_64-builtin-objs-$(CONFIG_KVM_BOOK3S_64_HANDLER) += \ book3s_hv_ras.o \ book3s_hv_builtin.o \ book3s_hv_p9_perf.o \ + guest-state-buffer.o \ $(kvm-book3s_64-builtin-tm-objs-y) \ $(kvm-book3s_64-builtin-xics-objs-y) + +obj-$(CONFIG_GUEST_STATE_BUFFER_TEST) += test-guest-state-buffer.o endif kvm-book3s_64-objs-$(CONFIG_KVM_XICS) += \
arch/powerpc/kvm/guest-state-buffer.c new file +563 -0
563 lines not on this page · raw message
arch/powerpc/kvm/test-guest-state-buffer.c new file +321 -0
321 lines not on this page · raw message
Signature · 1 line
2.31.1
Nicholas Piggin
reviewerRe: [RFC PATCH v2 4/6] KVM: PPC: Add helper library for Guest State Buffers
Quoted 33 lines · Jordan Niethe
The new PAPR nested guest API introduces the concept of a Guest State Buffer for communication about L2 guests between L1 and L0 hosts.
In the new API, the L0 manages the L2 on behalf of the L1. This means that if the L1 needs to change L2 state (e.g. GPRs, SPRs, partition table...), it must request the L0 perform the modification. If the nested host needs to read L2 state likewise this request must go through the L0.
The Guest State Buffer is a Type-Length-Value style data format defined in the PAPR which assigns all relevant partition state a unique identity. Unlike a typical TLV format the length is redundant as the length of each identity is fixed but is included for checking correctness.
A guest state buffer consists of an element count followed by a stream of elements, where elements are composed of an ID number, data length, then the data:
Header:
<---4 bytes---> +----------------+----- | Element Count | Elements... +----------------+-----Element:
<----2 bytes---> <-2 bytes-> <-Length bytes-> +----------------+-----------+----------------+ | Guest State ID | Length | Data | +----------------+-----------+----------------+
Quoted 7 lines
Guest State IDs have other attributes defined in the PAPR such as whether they are per thread or per guest, or read-only.
Introduce a library for using guest state buffers. This includes support for actions such as creating buffers, adding elements to buffers, reading the value of elements and parsing buffers. This will be used later by the PAPR nested guest support.
This is a tour de force in one of these things, so I hate to be the "me smash with club" guy, but what if you allocated buffers with enough room for all the state (or 99% of cases, in which case an overflow would make an hcall)?
What's actually a fast-path that we don't get from the interrupt return buffer? Getting and setting a few regs for MMIO emulation?
Quoted 57 lines
Signed-off-by Jordan Niethe <[email protected]>---
v2:
- Add missing #ifdefCONFIG_VSXs
- Move files fromlib/to kvm/
- Guard compilation onCONFIG_KVM_BOOK3S_HV_POSSIBLE
- Use kunit for guest state buffer tests
- Add configuration option for the tests
- Use macros for contiguous id ranges like GPRs
- Add some missing EXPORTs to functions
- HEIR element is a double word not a word--- arch/powerpc/Kconfig.debug | 12 + arch/powerpc/include/asm/guest-state-buffer.h | 901 ++++++++++++++++++ arch/powerpc/include/asm/kvm_book3s.h | 2 + arch/powerpc/kvm/Makefile | 3 + arch/powerpc/kvm/guest-state-buffer.c | 563 +++++++++++ arch/powerpc/kvm/test-guest-state-buffer.c | 321 +++++++ 6 files changed, 1802 insertions(+) create mode 100644 arch/powerpc/include/asm/guest-state-buffer.h create mode 100644 arch/powerpc/kvm/guest-state-buffer.c create mode 100644 arch/powerpc/kvm/test-guest-state-buffer.carch/powerpc/Kconfig.debug +12
-0@@ -82,6 +82,18 @@ config MSI_BITMAP_SELFTEST bool "Run self-tests of the MSI bitmap code" depends on DEBUG_KERNEL +config GUEST_STATE_BUFFER_TEST + def_tristate n + prompt "Enable Guest State Buffer unit tests" + depends on KUNIT + depends on KVM_BOOK3S_HV_POSSIBLE + default KUNIT_ALL_TESTS + help + The Guest State Buffer is a data format specified in the PAPR. + It is by hcalls to communicate the state of L2 guests between + the L1 and L0 hypervisors. Enable unit tests for the library + used to create and use guest state buffers. + config PPC_IRQ_SOFT_MASK_DEBUG bool "Include extra checks for powerpc irq soft masking" depends on PPC64arch/powerpc/include/asm/guest-state-buffer.h new file +5
-0@@ -0,0 +1,901 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Interface based on include/net/netlink.h +*/ +#ifndef _ASM_POWERPC_GUEST_STATE_BUFFER_H
arch/powerpc/include/asm/guest-state-buffer.h line 15
Quoted 10 lines
arch/powerpc/include/asm/guest-state-buffer.h new file +10
-0+#define _ASM_POWERPC_GUEST_STATE_BUFFER_H + +#include <linux/gfp.h> +#include <linux/bitmap.h> +#include <asm/plpar_wrappers.h> + +/************************************************************************** +* Guest State Buffer Constants +**************************************************************************/ +#define GSID_BLANK 0x0000
The namespaces are a little abbreviated. KVM_PAPR_ might be nice if you're calling the API that.
Quoted 14 lines
arch/powerpc/include/asm/guest-state-buffer.h new file +13
-0+ +#define GSID_HOST_STATE_SIZE 0x0001/* Size of Hypervisor Internal Format VCPU state */ +#define GSID_RUN_OUTPUT_MIN_SIZE 0x0002/* Minimum size of the Run VCPU output buffer */ +#define GSID_LOGICAL_PVR 0x0003/* Logical PVR */ +#define GSID_TB_OFFSET 0x0004/* Timebase Offset */ +#define GSID_PARTITION_TABLE 0x0005/* Partition Scoped Page Table */ +#define GSID_PROCESS_TABLE 0x0006/* Process Table */ + +#define GSID_RUN_INPUT 0x0C00/* Run VCPU Input Buffer */ +#define GSID_RUN_OUTPUT 0x0C01/* Run VCPU Out Buffer */ +#define GSID_VPA 0x0C02/* HRA to Guest VCPU VPA */ + +#define GSID_GPR (x) (0x1000 + (x))
arch/powerpc/include/asm/guest-state-buffer.h line 38
Quoted 10 lines
arch/powerpc/include/asm/guest-state-buffer.h new file +10
-0+#define GSID_HDEC_EXPIRY_TB 0x1020 +#define GSID_NIA 0x1021 +#define GSID_MSR 0x1022 +#define GSID_LR 0x1023 +#define GSID_XER 0x1024 +#define GSID_CTR 0x1025 +#define GSID_CFAR 0x1026 +#define GSID_SRR0 0x1027 +#define GSID_SRR1 0x1028 +#define GSID_DAR 0x1029
It's a shame you have to rip up all your wrapper functions now to shoehorn these in.
If you included names analogous to the reg field names in the kvm structures, the wrappers could do macro expansions that get them.
#define __GSID_WRAPPER_dar GSID_DAR
Or similar.
And since of course you have to explicitly enumerate all these, I wouldn't mind defining the types and lengths up-front rather than down in the type function. You'd like to be able to go through the spec and eyeball type, number, size.
[snip]
Quoted 28 lines
arch/powerpc/include/asm/guest-state-buffer.h new file +27
-0+/** +* gsb_paddress() - the physical address of buffer +* @gsb: guest state buffer +* +* Returns the physical address of the buffer. +*/ +static inline u64 gsb_paddress (struct gs_buff *gsb) +{ +return __pa (gsb_header (gsb)); +} +/** +* __gse_put_reg() - add a register type guest state element to a buffer +* @gsb: guest state buffer to add element to +* @iden: guest state ID +* @val: host endian value +* +* Adds a register type guest state element. Uses the guest state ID for +* determining the length of the guest element. If the guest state ID has +* bits that can not be set they will be cleared. +*/ +static inline int __gse_put_reg (struct gs_buff *gsb,u16 iden,u64 val) +{ + val &=gsid_mask (iden); +if (gsid_size (iden) ==sizeof (u64)) +return gse_put_u64 (gsb, iden, val); + +if (gsid_size (iden) ==sizeof (u32)) {
arch/powerpc/include/asm/guest-state-buffer.h line 428
Quoted 10 lines
arch/powerpc/include/asm/guest-state-buffer.h new file +10
-0+u32 tmp; + + tmp = (u32 )val; +if (tmp != val) +return -EINVAL ; + +return gse_put_u32 (gsb, iden, tmp); + } +return -EINVAL ; +}
There is a clever accessor that derives the length from the type, but then you fall back to this.
Quoted 17 lines
arch/powerpc/include/asm/guest-state-buffer.h new file +17
-0+/** +* gse_put - add a guest state element to a buffer +* @gsb: guest state buffer to add to +* @iden: guest state identity +* @v: generic value +*/ +#define gse_put (gsb, iden, v) \ + (_Generic((v), \ + u64 : __gse_put_reg, \ + long unsigned int : __gse_put_reg, \ + u32 : __gse_put_reg, \ + struct gs_buff_info : gse_put_buff_info, \ + struct gs_proc_table : gse_put_proc_table, \ + struct gs_part_table : gse_put_part_table, \ + vector128 : gse_put_vector128)(gsb, iden, v)) + +/**
arch/powerpc/include/asm/guest-state-buffer.h line 780
Quoted 10 lines
arch/powerpc/include/asm/guest-state-buffer.h new file +10
-0+* gse_get - return the data of a guest state element +* @gsb: guest state element to add to +* @v: generic value pointer to return in +*/ +#define gse_get (gse, v) \ + (*v = (_Generic((v), \ + u64 * : __gse_get_reg, \ + unsigned long * : __gse_get_reg, \ + u32 * : __gse_get_reg, \ + vector128 * : gse_get_vector128)(gse)))
I don't see the benefit of this. Caller always knows the type doesn't it? It seems like the right function could be called directly. It makes the calling convention a bit clunky too. I know there's similar precedent for uaccess functions, but not sure I like it for this.
Quoted 18 lines
arch/powerpc/include/asm/guest-state-buffer.h new file +18
-0+struct gs_buff *gsb_new (size_t size,unsigned long guest_id, +unsigned long vcpu_id,gfp_t flags) +{ +struct gs_buff *gsb; + + gsb =kzalloc (sizeof (*gsb), flags); +if (!gsb) +return NULL ; + + size =roundup_pow_of_two (size); + gsb->hdr =kzalloc (size,GFP_KERNEL ); +if (!gsb->hdr) + goto free; + + gsb->capacity = size; + gsb->len =sizeof (struct gs_header ); + gsb->vcpu_id = vcpu_id; + gsb->guest_id = guest_id;
arch/powerpc/include/asm/guest-state-buffer.h
Quoted 10 lines
arch/powerpc/include/asm/guest-state-buffer.h new file +10
-0+ + gsb->hdr->nelems =cpu_to_be32 (0 ); + +return gsb; + +free: +kfree (gsb); +return NULL ; +} +EXPORT_SYMBOL (gsb_new);
Should all be GPL exports.
Needs more namespace too, I reckon (not just exports but any kernel-wide name this short and non-descriptive needs a kvmppc or kvm_papr or something).
Thanks,
Nick
Re: [RFC PATCH v2 4/6] KVM: PPC: Add helper library for Guest State Buffers
On Wed, Jun 7, 2023 at 6:27 PM Nicholas Piggin <[email protected]> wrote:
[snip]
Quoted 8 lines
This is a tour de force in one of these things, so I hate to be the "me smash with club" guy, but what if you allocated buffers with enough room for all the state (or 99% of cases, in which case an overflow would make an hcall)?
What's actually a fast-path that we don't get from the interrupt return buffer? Getting and setting a few regs for MMIO emulation?
As it is a vcpu uses four buffers:
- One for registering it's input and output buffers
This is allocated just large enough for GSID_RUN_OUTPUT_MIN_SIZE, GSID_RUN_INPUT and GSID_RUN_OUTPUT.
Freed once the buffers are registered.
I suppose we could just make a buffer big enough to be used for the
vcpu run input buffer then have it register its own address.
- One for process and partition table entries
Because kvmhv_set_ptbl_entry() isn't associated with a vcpu. kvmhv_papr_set_ptbl_entry() allocates and frees a minimal sized
buffer on demand.
- The run vcpu input buffer
Persists over the lifetime of the vcpu after creation. Large enough
to hold all VCPU-wide elements. The same buffer is also reused for:
* GET state hcalls
* SET guest wide state hcalls (guest wide can not be passed into
the vcpu run buffer)
- The run vcpu output buffer
Persists over the lifetime of the vcpu after creation. This is
sized to be GSID_RUN_OUTPUT_MIN_SIZE as returned by the L0.
It's unlikely that it would be larger than the run vcpu buffer
size, so I guess you could make it that size too. Probably you could even use the run vcpu input buffer as the vcpu output buffer.
The buffers could all be that max size and could combine the configuration buffer, input and output buffers, but I feel it's more understandable like this.
[snip]
Quoted 3 lines
The namespaces are a little abbreviated.
KVM_PAPR_might be nice if you're calling the API that.
Will we go with KVM_NESTED_V2_ ?
Quoted 29 lines
Quoted 15 lines
arch/powerpc/include/asm/guest-state-buffer.h new file +13
-0+ +#define GSID_HOST_STATE_SIZE 0x0001/* Size of Hypervisor Internal Format VCPU state */ +#define GSID_RUN_OUTPUT_MIN_SIZE 0x0002/* Minimum size of the Run VCPU output buffer */ +#define GSID_LOGICAL_PVR 0x0003/* Logical PVR */ +#define GSID_TB_OFFSET 0x0004/* Timebase Offset */ +#define GSID_PARTITION_TABLE 0x0005/* Partition Scoped Page Table */ +#define GSID_PROCESS_TABLE 0x0006/* Process Table */ + +#define GSID_RUN_INPUT 0x0C00/* Run VCPU Input Buffer */ +#define GSID_RUN_OUTPUT 0x0C01/* Run VCPU Out Buffer */ +#define GSID_VPA 0x0C02/* HRA to Guest VCPU VPA */ + +#define GSID_GPR (x) (0x1000 + (x))arch/powerpc/include/asm/guest-state-buffer.h line 38
Quoted 10 lines
arch/powerpc/include/asm/guest-state-buffer.h new file +10
-0+#define GSID_HDEC_EXPIRY_TB 0x1020 +#define GSID_NIA 0x1021 +#define GSID_MSR 0x1022 +#define GSID_LR 0x1023 +#define GSID_XER 0x1024 +#define GSID_CTR 0x1025 +#define GSID_CFAR 0x1026 +#define GSID_SRR0 0x1027 +#define GSID_SRR1 0x1028 +#define GSID_DAR 0x1029It's a shame you have to rip up all your wrapper functions now to shoehorn these in.
Quoted 6 lines
If you included names analogous to the reg field names in the kvm structures, the wrappers could do macro expansions that get them.
#define
__GSID_WRAPPER_darGSID_DAROr similar.
Before I had something pretty hacky, in the macro accessors I had along the lines of
gsid_table[offsetof(vcpu, reg)]
to get the GSID for the register.
We can do the wrapper idea, I just worry if it is getting too magic.
Quoted 5 lines
And since of course you have to explicitly enumerate all these, I wouldn't mind defining the types and lengths up-front rather than down in the type function. You'd like to be able to go through the spec and eyeball type, number, size.
Something like
#define KVM_NESTED_V2_GS_NIA (KVM_NESTED_V2_GSID_NIA | VCPU_WIDE | READ_WRITE | DOUBLE_WORD)
etc
?
Quoted 36 lines
[snip]
Quoted 33 lines
(quoted diff) +32
-0+/** + * gsb_paddress() - the physical address of buffer + * @gsb: guest state buffer + * + * Returns the physical address of the buffer. + */ +static inline u64 gsb_paddress(struct gs_buff *gsb) +{ + return __pa(gsb_header(gsb)); +} +/** + * __gse_put_reg() - add a register type guest state element to a buffer + * @gsb: guest state buffer to add element to + * @iden: guest state ID + * @val: host endian value + * + * Adds a register type guest state element. Uses the guest state ID for + * determining the length of the guest element. If the guest state ID has + * bits that can not be set they will be cleared. + */ +static inline int __gse_put_reg(struct gs_buff *gsb, u16 iden, u64 val) +{ + val &= gsid_mask(iden); + if (gsid_size(iden) == sizeof(u64)) + return gse_put_u64(gsb, iden, val); + + if (gsid_size(iden) == sizeof(u32)) { + u32 tmp; + + tmp = (u32)val; + if (tmp != val) + return -EINVAL;
Quoted 8 lines
arch/powerpc/include/asm/guest-state-buffer.h line 601
Quoted 5 lines
arch/powerpc/include/asm/guest-state-buffer.h new file +5
-0+ +return gse_put_u32 (gsb, iden, tmp); + } +return -EINVAL ; +}There is a clever accessor that derives the length from the type, but then you fall back to this.
It's basically just to massage where we have a kvm representation and guest state buffer representation mismatch:
Like: unsigned long ccr; being 8 bytes and having 4 byte CR in the spec.
Quoted 25 lines
Quoted 25 lines
arch/powerpc/include/asm/guest-state-buffer.h new file +24
-0+/** +* gse_put - add a guest state element to a buffer +* @gsb: guest state buffer to add to +* @iden: guest state identity +* @v: generic value +*/ +#define gse_put (gsb, iden, v) \ + (_Generic((v), \ + u64 : __gse_put_reg, \ + long unsigned int : __gse_put_reg, \ + u32 : __gse_put_reg, \ + struct gs_buff_info : gse_put_buff_info, \ + struct gs_proc_table : gse_put_proc_table, \ + struct gs_part_table : gse_put_part_table, \ + vector128 : gse_put_vector128)(gsb, iden, v)) + +/** +* gse_get - return the data of a guest state element +* @gsb: guest state element to add to +* @v: generic value pointer to return in +*/ +#define gse_get (gse, v) \ + (*v = (_Generic((v), \ + u64 * : __gse_get_reg, \
Quoted 8 lines
arch/powerpc/include/asm/guest-state-buffer.h line 780
Quoted 3 lines
arch/powerpc/include/asm/guest-state-buffer.h new file +3
-0+ unsigned long * : __gse_get_reg, \ + u32 * : __gse_get_reg, \ + vector128 * : gse_get_vector128)(gse)))I don't see the benefit of this. Caller always knows the type doesn't it? It seems like the right function could be called directly. It makes the calling convention a bit clunky too. I know there's similar precedent for uaccess functions, but not sure I like it for this.
The compiler also knows so I just thought I'd save some typing. I agree it's kind of ugly, happy to drop it.
[snip]
Quoted 6 lines
Should all be GPL exports.
Needs more namespace too, I reckon (not just exports but any kernel-wide name this short and non-descriptive needs a kvmppc or
kvm_papror something).
Will do.
Thanks,
Jordan
Quoted 4 lines
Thanks,
Nick
RFC PATCH v2 5/6 KVM: PPC: Add support for nested PAPR guests
In reply to Jordan NietheA series of hcalls have been added to the PAPR which allow a regular guest partition to create and manage guest partitions of its own. Add support to KVM to utilize these hcalls to enable running nested guests.
Overview of the new hcall usage:
- L1 and L0 negotiate capabilities with
H_GUEST_{G,S}ET_CAPABILITIES()
- L1 requests the L0 create a L2 with
H_GUEST_CREATE() and receives a handle to use in future hcalls
- L1 requests the L0 create a L2 vCPU with
H_GUEST_CREATE_VCPU()
- L1 sets up the L2 using H_GUEST_SET and the
H_GUEST_VCPU_RUN input buffer
- L1 requests the L0 runs the L2 vCPU using H_GUEST_VCPU_RUN()
- L2 returns to L1 with an exit reason and L1 reads the H_GUEST_VCPU_RUN output buffer populated by the L0
- L1 handles the exit using H_GET_STATE if necessary
- L1 reruns L2 vCPU with H_GUEST_VCPU_RUN
- L1 frees the L2 in the L0 with H_GUEST_DELETE()
Support for the new API is determined by trying H_GUEST_GET_CAPABILITIES. On a successful return, the new API will then be used.
Use the vcpu register state setters for tracking modified guest state elements and copy the thread wide values into the H_GUEST_VCPU_RUN input buffer immediately before running a L2. The guest wide elements can not be added to the input buffer so send them with a separate H_GUEST_SET call if necessary.
Make the vcpu register getter load the corresponding value from the real host with H_GUEST_GET. To avoid unnecessarily calling H_GUEST_GET, track which values have already been loaded between H_GUEST_VCPU_RUN calls. If an element is present in the H_GUEST_VCPU_RUN output buffer it also does not need to be loaded again.
There is existing support for running nested guests on KVM with powernv. However the interface used for this is not supported by other PAPR hosts. This existing API is still supported.
---
v2:
- Declare op structs as static
- Use expressions in switch case with local variables
- Do not use the PVR for the LOGICAL PVR ID
- Handle emul_inst as now a double word
- Use new GPR(), etc macros
- Determine PAPR nested capabilities from cpu features
--- arch/powerpc/include/asm/guest-state-buffer.h | 105 +- arch/powerpc/include/asm/hvcall.h | 30 + arch/powerpc/include/asm/kvm_book3s.h | 122 ++- arch/powerpc/include/asm/kvm_book3s_64.h | 6 + arch/powerpc/include/asm/kvm_host.h | 21 + arch/powerpc/include/asm/kvm_ppc.h | 64 +- arch/powerpc/include/asm/plpar_wrappers.h | 198 ++++ arch/powerpc/kvm/Makefile | 1 + arch/powerpc/kvm/book3s_hv.c | 126 ++- arch/powerpc/kvm/book3s_hv.h | 74 +- arch/powerpc/kvm/book3s_hv_nested.c | 38 +- arch/powerpc/kvm/book3s_hv_papr.c | 940 ++++++++++++++++++ arch/powerpc/kvm/emulate_loadstore.c | 4 +- arch/powerpc/kvm/guest-state-buffer.c | 49 + 14 files changed, 1684 insertions(+), 94 deletions(-) create mode 100644 arch/powerpc/kvm/book3s_hv_papr.c
arch/powerpc/include/asm/guest-state-buffer.h +96 -9
@@ -5,6 +5,7 @@#ifndef _ASM_POWERPC_GUEST_STATE_BUFFER_H#define _ASM_POWERPC_GUEST_STATE_BUFFER_H +#include "asm/hvcall.h" #include <linux/gfp.h> #include <linux/bitmap.h> #include <asm/plpar_wrappers.h>
@@ -14,16 +15,16 @@**************************************************************************/ #define GSID_BLANK 0x0000 -#define GSID_HOST_STATE_SIZE 0x0001/* Size of Hypervisor Internal Format VCPU state */ -#define GSID_RUN_OUTPUT_MIN_SIZE 0x0002/* Minimum size of the Run VCPU output buffer */ -#define GSID_LOGICAL_PVR 0x0003/* Logical PVR */ -#define GSID_TB_OFFSET 0x0004/* Timebase Offset */ -#define GSID_PARTITION_TABLE 0x0005/* Partition Scoped Page Table */ -#define GSID_PROCESS_TABLE 0x0006/* Process Table */ +#define GSID_HOST_STATE_SIZE 0x0001 +#define GSID_RUN_OUTPUT_MIN_SIZE 0x0002 +#define GSID_LOGICAL_PVR 0x0003 +#define GSID_TB_OFFSET 0x0004 +#define GSID_PARTITION_TABLE 0x0005 +#define GSID_PROCESS_TABLE 0x0006 -#define GSID_RUN_INPUT 0x0C00/* Run VCPU Input Buffer */ -#define GSID_RUN_OUTPUT 0x0C01/* Run VCPU Out Buffer */ -#define GSID_VPA 0x0C02/* HRA to Guest VCPU VPA */ +#define GSID_RUN_INPUT 0x0C00 +#define GSID_RUN_OUTPUT 0x0C01 +#define GSID_VPA 0x0C02#define GSID_GPR (x) (0x1000 + (x))#define GSID_HDEC_EXPIRY_TB 0x1020
@@ -300,6 +301,8 @@ struct gs_buff *gsb_new(size_t size, unsigned long guest_id, unsigned long vcpu_id, gfp_t flags);void gsb_free (struct gs_buff *gsb);void *gsb_put (struct gs_buff *gsb,size_t size); +int gsb_send (struct gs_buff *gsb,unsigned long flags); +int gsb_recv (struct gs_buff *gsb,unsigned long flags);/** * gsb_header() - the header of a guest state buffer
@@ -898,4 +901,88 @@ static inline void gsm_reset(struct gs_msg *gsm) gsbm_zero(&gsm->bitmap); } +/** +* gsb_receive_data - flexibly update values from a guest state buffer +* @gsb: guest state buffer +* @gsm: guest state message +* +* Requests updated values for the guest state values included in the guest +* state message. The guest state message will then deserialize the guest state +* buffer. +*/ +static inline int gsb_receive_data (struct gs_buff *gsb,struct gs_msg *gsm) +{ +int rc; + + rc =gsm_fill_info (gsm, gsb); +if (rc <0 ) +return rc; + + rc =gsb_recv (gsb, gsm->flags); +if (rc <0 ) +return rc; + + rc =gsm_refresh_info (gsm, gsb); +if (rc <0 ) +return rc; +return 0 ; +} + +/** +* gsb_recv - receive a single guest state ID +* @gsb: guest state buffer +* @gsm: guest state message +* @iden: guest state identity +*/ +static inline int gsb_receive_datum (struct gs_buff *gsb,struct gs_msg *gsm, +u16 iden) +{ +int rc; + +gsm_include (gsm, iden); + rc =gsb_receive_data (gsb, gsm); +if (rc <0 ) +return rc; +gsm_reset (gsm); +return 0 ; +} + +/** +* gsb_send_data - flexibly send values from a guest state buffer +* @gsb: guest state buffer +* @gsm: guest state message +* +* Sends the guest state values included in the guest state message. +*/ +static inline int gsb_send_data (struct gs_buff *gsb,struct gs_msg *gsm) +{ +int rc; + + rc =gsm_fill_info (gsm, gsb); +if (rc <0 ) +return rc; + rc =gsb_send (gsb, gsm->flags); + +return rc; +} + +/** +* gsb_recv - send a single guest state ID +* @gsb: guest state buffer +* @gsm: guest state message +* @iden: guest state identity +*/ +static inline int gsb_send_datum (struct gs_buff *gsb,struct gs_msg *gsm, +u16 iden) +{ +int rc; + +gsm_include (gsm, iden); + rc =gsb_send_data (gsb, gsm); +if (rc <0 ) +return rc; +gsm_reset (gsm); +return 0 ; +} +#endif /* _ASM_POWERPC_GUEST_STATE_BUFFER_H */
arch/powerpc/include/asm/hvcall.h +30 -0
@@ -100,6 +100,18 @@#define H_COP_HW -74#define H_STATE -75#define H_IN_USE -77 + +#define H_INVALID_ELEMENT_ID -79 +#define H_INVALID_ELEMENT_SIZE -80 +#define H_INVALID_ELEMENT_VALUE -81 +#define H_INPUT_BUFFER_NOT_DEFINED -82 +#define H_INPUT_BUFFER_TOO_SMALL -83 +#define H_OUTPUT_BUFFER_NOT_DEFINED -84 +#define H_OUTPUT_BUFFER_TOO_SMALL -85 +#define H_PARTITION_PAGE_TABLE_NOT_DEFINED -86 +#define H_GUEST_VCPU_STATE_NOT_HV_OWNED -87 + +#define H_UNSUPPORTED_FLAG_START -256#define H_UNSUPPORTED_FLAG_END -511#define H_MULTI_THREADS_ACTIVE -9005
@@ -381,6 +393,15 @@#define H_ENTER_NESTED 0xF804#define H_TLB_INVALIDATE 0xF808#define H_COPY_TOFROM_GUEST 0xF80C +#define H_GUEST_GET_CAPABILITIES 0x460 +#define H_GUEST_SET_CAPABILITIES 0x464 +#define H_GUEST_CREATE 0x470 +#define H_GUEST_CREATE_VCPU 0x474 +#define H_GUEST_GET_STATE 0x478 +#define H_GUEST_SET_STATE 0x47C +#define H_GUEST_RUN_VCPU 0x480 +#define H_GUEST_COPY_MEMORY 0x484 +#define H_GUEST_DELETE 0x488/* Flags for H_SVM_PAGE_IN */ #define H_PAGE_IN_SHARED 0x1
@@ -467,6 +488,15 @@#define H_RPTI_PAGE_1G 0x08#define H_RPTI_PAGE_ALL (-1UL) +/* Flags for H_GUEST_{S,G}_STATE */ +#define H_GUEST_FLAGS_WIDE (1UL<<(63-0)) + +/* Flag values used for H_{S,G}SET_GUEST_CAPABILITIES */ +#define H_GUEST_CAP_COPY_MEM (1UL<<(63-0)) +#define H_GUEST_CAP_POWER9 (1UL<<(63-1)) +#define H_GUEST_CAP_POWER10 (1UL<<(63-2)) +#define H_GUEST_CAP_BITMAP2 (1UL<<(63-63)) +#ifndef __ASSEMBLY__#include <linux/types.h>
arch/powerpc/include/asm/kvm_book3s.h +103 -19
296 lines not on this page · raw message
arch/powerpc/include/asm/kvm_book3s_64.h +6 -0
@@ -677,6 +677,12 @@ static inline pte_t *find_kvm_host_pte(struct kvm *kvm, unsigned long mmu_seq,extern pte_t *find_kvm_nested_guest_pte (struct kvm *kvm,unsigned long lpid,unsigned long ea,unsigned *hshift); +int kvmhv_papr_vcpu_create (struct kvm_vcpu *vcpu,struct kvmhv_papr_host *nested_state); +void kvmhv_papr_vcpu_free (struct kvm_vcpu *vcpu,struct kvmhv_papr_host *nested_state); +int kvmhv_papr_flush_vcpu (struct kvm_vcpu *vcpu,u64 time_limit); +int kvmhv_papr_set_ptbl_entry (u64 lpid,u64 dw0,u64 dw1); +int kvmhv_papr_parse_output (struct kvm_vcpu *vcpu); +#endif /* CONFIG_KVM_BOOK3S_HV_POSSIBLE */ #endif /* __ASM_KVM_BOOK3S_64_H__ */
arch/powerpc/include/asm/kvm_host.h +21 -0
@@ -25,6 +25,7 @@#include <asm/cacheflush.h> #include <asm/hvcall.h> #include <asm/mce.h> +#include <asm/guest-state-buffer.h> #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
@@ -509,6 +510,23 @@ union xive_tma_w01 {__be64 w01; }; +/* Nested PAPR host H_GUEST_RUN_VCPU configuration */ +struct kvmhv_papr_config { +struct gs_buff_info vcpu_run_output_cfg; +struct gs_buff_info vcpu_run_input_cfg; +u64 vcpu_run_output_size; +}; + +/* Nested PAPR host state */ +struct kvmhv_papr_host { +struct kvmhv_papr_config cfg; +struct gs_buff *vcpu_run_output; +struct gs_buff *vcpu_run_input; +struct gs_msg *vcpu_message; +struct gs_msg *vcore_message; +struct gs_bitmap valids; +}; +struct kvm_vcpu_arch {ulong host_stack;u32 host_pid;
@@ -575,6 +593,7 @@ struct kvm_vcpu_arch {ulong dscr;ulong amr;ulong uamor; +ulong amor;ulong iamr;u32 ctrl;u32 dabrx;
@@ -829,6 +848,8 @@ struct kvm_vcpu_arch {u64 nested_hfscr;/* HFSCR that the L1 requested for the nested guest */ u32 nested_vcpu_id;gpa_t nested_io_gpr; +/* For nested APIv2 guests*/ +struct kvmhv_papr_host papr_host;#endif #ifdef CONFIG_KVM_BOOK3S_HV_EXIT_TIMING
arch/powerpc/include/asm/kvm_ppc.h +48 -16
114 lines not on this page · raw message
arch/powerpc/include/asm/plpar_wrappers.h +198 -0
210 lines not on this page · raw message
arch/powerpc/kvm/Makefile +1 -0
@@ -87,6 +87,7 @@ kvm-book3s_64-builtin-objs-$(CONFIG_KVM_BOOK3S_64_HANDLER) += \ book3s_hv_ras.o \ book3s_hv_builtin.o \ book3s_hv_p9_perf.o \ + book3s_hv_papr.o \ guest-state-buffer.o \ $(kvm-book3s_64-builtin-tm-objs-y) \ $(kvm-book3s_64-builtin-xics-objs-y)
arch/powerpc/kvm/book3s_hv.c +116 -10
218 lines not on this page · raw message
arch/powerpc/kvm/book3s_hv.h +40 -34
107 lines not on this page · raw message
arch/powerpc/kvm/book3s_hv_nested.c +33 -5
@@ -428,10 +428,12 @@ long kvmhv_enter_nested_guest(struct kvm_vcpu *vcpu)return vcpu->arch.trap; } +static unsigned long nested_capabilities; +long kvmhv_nested_init (void ) {long int ptb_order; -unsigned long ptcr; +unsigned long ptcr, host_capabilities;long rc;if (!kvmhv_on_pseries ())
@@ -439,6 +441,27 @@ long kvmhv_nested_init(void)if (!radix_enabled ())return -ENODEV ; + rc =plpar_guest_get_capabilities (0 , &host_capabilities); +if (rc =H_SUCCESS ) { +unsigned long capabilities =0 ; + +if (cpu_has_feature (CPU_FTR_ARCH_31 )) + capabilities |=H_GUEST_CAP_POWER10 ; +if (cpu_has_feature (CPU_FTR_ARCH_300 )) + capabilities |=H_GUEST_CAP_POWER9 ; + + nested_capabilities = capabilities & host_capabilities; + rc =plpar_guest_set_capabilities (0 , nested_capabilities); +if (rc !=H_SUCCESS ) { +pr_err ("kvm-hv: Could not configure parent hypervisor capabilities (rc=%ld)" , + rc); +return -ENODEV ; + } + + __kvmhv_on_papr = true; +return 0 ; + } +/* Partition table entry is 1<<4 bytes in size, hence the 4. */ ptb_order =KVM_MAX_NESTED_GUESTS_SHIFT +4 ;/* Minimum partition table size is 1<<12 bytes */
@@ -507,10 +530,15 @@ void kvmhv_set_ptbl_entry(unsigned int lpid, u64 dw0, u64 dw1)return ; } - pseries_partition_tb[lpid].patb0 =cpu_to_be64 (dw0); - pseries_partition_tb[lpid].patb1 =cpu_to_be64 (dw1); -/* L0 will do the necessary barriers */ -kvmhv_flush_lpid (lpid); +if (!kvmhv_on_papr ()) { + pseries_partition_tb[lpid].patb0 =cpu_to_be64 (dw0); + pseries_partition_tb[lpid].patb1 =cpu_to_be64 (dw1); +/* L0 will do the necessary barriers */ +kvmhv_flush_lpid (lpid); + } + +if (kvmhv_on_papr ()) +kvmhv_papr_set_ptbl_entry (lpid, dw0, dw1); }static void kvmhv_set_nested_ptbl (struct kvm_nested_guest *gp)
arch/powerpc/kvm/book3s_hv_papr.c new file +937 -0
937 lines not on this page · raw message
arch/powerpc/kvm/emulate_loadstore.c +3 -1
@@ -92,7 +92,8 @@ int kvmppc_emulate_loadstore(struct kvm_vcpu *vcpu) vcpu->arch.mmio_host_swabbed =0 ; emulated =EMULATE_FAIL ; - vcpu->arch.regs.msr = vcpu->arch.shared->msr; + vcpu->arch.regs.msr =kvmppc_get_msr (vcpu); +kvmhv_papr_reload_ptregs (vcpu, &vcpu->arch.regs);if (analyse_instr (&op, &vcpu->arch.regs, inst) =0 ) {int type = op.type &INSTR_TYPE_MASK ;int size =GETSIZE (op.type);
@@ -357,6 +358,7 @@ int kvmppc_emulate_loadstore(struct kvm_vcpu *vcpu) }trace_kvm_ppc_instr (ppc_inst_val (inst),kvmppc_get_pc (vcpu), emulated); +kvmhv_papr_mark_dirty_ptregs (vcpu, &vcpu->arch.regs);/* Advance past emulated instruction. */ if (emulated !=EMULATE_FAIL )
arch/powerpc/kvm/guest-state-buffer.c +49 -0
52 lines not on this page · raw message
Signature · 1 line
2.31.1
Nicholas Piggin
reviewerRe: [RFC PATCH v2 5/6] KVM: PPC: Add support for nested PAPR guests
Quoted 94 lines · Jordan Niethe
A series of hcalls have been added to the PAPR which allow a regular guest partition to create and manage guest partitions of its own. Add support to KVM to utilize these hcalls to enable running nested guests.
Overview of the new hcall usage:
- L1 and L0 negotiate capabilities with
H_GUEST_{G,S}ET_CAPABILITIES()- L1 requests the L0 create a L2 with
H_GUEST_CREATE()and receives a handle to use in future hcalls- L1 requests the L0 create a L2 vCPU with
H_GUEST_CREATE_VCPU()- L1 sets up the L2 using
H_GUEST_SETand the
H_GUEST_VCPU_RUNinput buffer- L1 requests the L0 runs the L2 vCPU using
H_GUEST_VCPU_RUN()- L2 returns to L1 with an exit reason and L1 reads the
H_GUEST_VCPU_RUNoutput buffer populated by the L0- L1 handles the exit using
H_GET_STATEif necessary- L1 reruns L2 vCPU with
H_GUEST_VCPU_RUN- L1 frees the L2 in the L0 with
H_GUEST_DELETE()Support for the new API is determined by trying
H_GUEST_GET_CAPABILITIES. On a successful return, the new API will then be used.Use the vcpu register state setters for tracking modified guest state elements and copy the thread wide values into the
H_GUEST_VCPU_RUNinput buffer immediately before running a L2. The guest wide elements can not be added to the input buffer so send them with a separateH_GUEST_SETcall if necessary.Make the vcpu register getter load the corresponding value from the real host with
H_GUEST_GET. To avoid unnecessarily callingH_GUEST_GET, track which values have already been loaded betweenH_GUEST_VCPU_RUNcalls. If an element is present in theH_GUEST_VCPU_RUNoutput buffer it also does not need to be loaded again.There is existing support for running nested guests on KVM with powernv. However the interface used for this is not supported by other PAPR hosts. This existing API is still supported.
Signed-off-by Jordan Niethe <[email protected]>---
v2:
- Declare op structs as static
- Use expressions in switch case with local variables
- Do not use the PVR for the LOGICAL PVR ID
- Handleemul_instas now a double word
- Use newGPR(), etc macros
- Determine PAPR nested capabilities from cpu features--- arch/powerpc/include/asm/guest-state-buffer.h | 105 +- arch/powerpc/include/asm/hvcall.h | 30 + arch/powerpc/include/asm/kvm_book3s.h | 122 ++- arch/powerpc/include/asm/kvm_book3s_64.h | 6 + arch/powerpc/include/asm/kvm_host.h | 21 + arch/powerpc/include/asm/kvm_ppc.h | 64 +- arch/powerpc/include/asm/plpar_wrappers.h | 198 ++++ arch/powerpc/kvm/Makefile | 1 + arch/powerpc/kvm/book3s_hv.c | 126 ++- arch/powerpc/kvm/book3s_hv.h | 74 +- arch/powerpc/kvm/book3s_hv_nested.c | 38 +- arch/powerpc/kvm/book3s_hv_papr.c | 940 ++++++++++++++++++ arch/powerpc/kvm/emulate_loadstore.c | 4 +- arch/powerpc/kvm/guest-state-buffer.c | 49 + 14 files changed, 1684 insertions(+), 94 deletions(-) create mode 100644 arch/powerpc/kvm/book3s_hv_papr.carch/powerpc/include/asm/guest-state-buffer.h +1
-2@@ -5,6 +5,7 @@#ifndef _ASM_POWERPC_GUEST_STATE_BUFFER_H#define _ASM_POWERPC_GUEST_STATE_BUFFER_H +#include "asm/hvcall.h" #include <linux/gfp.h> #include <linux/bitmap.h> #include <asm/plpar_wrappers.h> @@ -14,16 +15,16 @@**************************************************************************/ #define GSID_BLANK 0x0000 -#define GSID_HOST_STATE_SIZE 0x0001/* Size of Hypervisor Internal Format VCPU state */ -#define GSID_RUN_OUTPUT_MIN_SIZE 0x0002/* Minimum size of the Run VCPU output buffer */
arch/powerpc/include/asm/guest-state-buffer.h line 23
Quoted 10 lines
arch/powerpc/include/asm/guest-state-buffer.h +6
-4-#define GSID_LOGICAL_PVR 0x0003/* Logical PVR */ -#define GSID_TB_OFFSET 0x0004/* Timebase Offset */ -#define GSID_PARTITION_TABLE 0x0005/* Partition Scoped Page Table */ -#define GSID_PROCESS_TABLE 0x0006/* Process Table */ +#define GSID_HOST_STATE_SIZE 0x0001 +#define GSID_RUN_OUTPUT_MIN_SIZE 0x0002 +#define GSID_LOGICAL_PVR 0x0003 +#define GSID_TB_OFFSET 0x0004 +#define GSID_PARTITION_TABLE 0x0005 +#define GSID_PROCESS_TABLE 0x0006
You lost your comments.
Quoted 15 lines
arch/powerpc/include/asm/kvm_book3s.h +1
-0@@ -12,6 +12,7 @@#include <linux/types.h> #include <linux/kvm_host.h> #include <asm/kvm_book3s_asm.h> +#include <asm/guest-state-buffer.h> struct kvmppc_bat {u64 raw;@@ -316,6 +317,57 @@ long int kvmhv_nested_page_fault(struct kvm_vcpu *vcpu);void kvmppc_giveup_fac (struct kvm_vcpu *vcpu,ulong fac);
arch/powerpc/include/asm/kvm_book3s.h long int kvmhv_nested_page_fault(struct kvm_vcpu *vcpu); · line 328
Quoted 10 lines
arch/powerpc/include/asm/kvm_book3s.h +9
-0+ +#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE + +extern bool __kvmhv_on_papr; + +static inline bool kvmhv_on_papr (void ) +{ +return __kvmhv_on_papr; +}
It's a nitpick, but kvmhv_on_pseries() is because we're runnning KVM-HV on a pseries guest kernel. Which is a papr guest kernel. So this kind of doesn't make sense if you read it the same way.
kvmhv_nested_using_papr() or something like that might read a bit better.
This could be a static key too.
arch/powerpc/include/asm/kvm_book3s.h struct kvm_vcpu_arch { · line 599
Quoted 8 lines
arch/powerpc/include/asm/kvm_book3s.h +1
-0@@ -575,6 +593,7 @@ struct kvm_vcpu_arch {ulong dscr;ulong amr;ulong uamor; +ulong amor;ulong iamr;u32 ctrl;u32 dabrx;
This belongs somewhere else.
arch/powerpc/include/asm/kvm_book3s.h struct kvm_vcpu_arch { · line 853
Quoted 7 lines
arch/powerpc/include/asm/kvm_book3s.h +2
-0@@ -829,6 +848,8 @@ struct kvm_vcpu_arch {u64 nested_hfscr;/* HFSCR that the L1 requested for the nested guest */ u32 nested_vcpu_id;gpa_t nested_io_gpr; +/* For nested APIv2 guests*/ +struct kvmhv_papr_host papr_host;#endif
This is not exactly a papr host. Might have to come up with a better name especially if we implement a L0 things could get confusing.
Quoted 17 lines
arch/powerpc/include/asm/kvm_book3s.h +13
-0@@ -342,6 +343,203 @@ static inline long plpar_get_cpu_characteristics(struct h_cpu_char_result *p)return rc; } +static inline long plpar_guest_create (unsigned long flags,unsigned long *guest_id) +{ +unsigned long retbuf[PLPAR_HCALL_BUFSIZE ]; +unsigned long token; +long rc; + + token =-1UL ; +while (true) { + rc =plpar_hcall (H_GUEST_CREATE , retbuf, flags, token); +if (rc ==H_SUCCESS ) { + *guest_id = retbuf[0 ]; +break ; + }
arch/powerpc/include/asm/kvm_book3s.h static inline long plpar_get_cpu_characteristics(struct h_cpu_char_result *p) · line 368
Quoted 10 lines
arch/powerpc/include/asm/kvm_book3s.h +10
-0+ +if (rc ==H_BUSY ) { + token = retbuf[0 ]; +cpu_relax (); +continue ; + } + +if (H_IS_LONG_BUSY (rc)) { + token = retbuf[0 ]; +mdelay (get_longbusy_msecs (rc));
All of these things need a non-sleeping delay? Can we sleep instead? Or if not, might have to think about going back to the caller and it can retry.
get/set state might be a bit inconvenient, although I don't expect that should potentially take so long as guest and vcpu create/delete, so at least those ones would be good if they're called while preemptable.
arch/powerpc/kvm/book3s_hv.c static void kvmppc_core_vcpu_put_hv(struct kvm_vcpu *vcpu) · line 389
Quoted 12 lines
arch/powerpc/kvm/book3s_hv.c +4
-0@@ -383,6 +383,11 @@ static void kvmppc_core_vcpu_put_hv(struct kvm_vcpu *vcpu)spin_unlock_irqrestore (&vcpu->arch.tbacct_lock, flags); } +static void kvmppc_set_pvr_hv (struct kvm_vcpu *vcpu,u32 pvr) +{ + vcpu->arch.pvr = pvr; +}
Didn't you lose this in a previous patch? I thought it must have moved to a header but it reappears.
Quoted 10 lines
arch/powerpc/kvm/book3s_hv.c +2
-1+/* Dummy value used in computing PCR value below */ #define PCR_ARCH_31 (PCR_ARCH_300 << 1)@@ -1262,13 +1267,14 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)return RESUME_HOST ;break ;#endif -case H_RANDOM : +case H_RANDOM : {
arch/powerpc/kvm/book3s_hv.c int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu) · line 1280
Quoted 10 lines
arch/powerpc/kvm/book3s_hv.c +1
-0unsigned long rand;if (!arch_get_random_seed_longs (&rand,1 )) ret =H_HARDWARE ;kvmppc_set_gpr (vcpu,4 , rand);break ; + }case H_RPT_INVALIDATE : ret = kvmppc_h_rpt_invalidate(vcpu, kvmppc_get_gpr(vcpu, 4), kvmppc_get_gpr(vcpu, 5),
Compile fix for a previous patch.
Quoted 10 lines
arch/powerpc/kvm/book3s_hv.c +4
-1@@ -2921,14 +2927,21 @@ static int kvmppc_core_vcpu_create_hv(struct kvm_vcpu *vcpu) vcpu->arch.shared_big_endian = false;#endif #endif -kvmppc_set_mmcr_hv (vcpu,0 ,MMCR0_FC ); +if (kvmhv_on_papr ()) { + err =kvmhv_papr_vcpu_create (vcpu, &vcpu->arch.papr_host); +if (err <0 ) +return err;
arch/powerpc/kvm/book3s_hv.c static int kvmppc_core_vcpu_create_hv(struct kvm_vcpu *vcpu) · line 2944
Quoted 10 lines
arch/powerpc/kvm/book3s_hv.c +4
-0+ } + +kvmppc_set_mmcr_hv (vcpu,0 ,MMCR0_FC );if (cpu_has_feature (CPU_FTR_ARCH_31 )) {kvmppc_set_mmcr_hv (vcpu,0 ,kvmppc_get_mmcr_hv (vcpu,0 ) |MMCR0_PMCCEXT );kvmppc_set_mmcra_hv (vcpu,MMCRA_BHRB_DISABLE ); }kvmppc_set_ctrl_hv (vcpu,CTRL_RUNLATCH ); +kvmppc_set_amor_hv (vcpu, ~0 );
This AMOR thing should go somewhere else. Not actually sure why it needs to be added to the vcpu since it's always ~0. Maybe just put that in a #define somewhere and use that.
Quoted 24 lines
arch/powerpc/kvm/book3s_hv.c +20
-0@@ -4042,6 +4059,50 @@ static void vcpu_vpa_increment_dispatch(struct kvm_vcpu *vcpu) } } +static int kvmhv_vcpu_entry_papr (struct kvm_vcpu *vcpu,u64 time_limit,unsigned long lpcr,u64 *tb) +{ +struct kvmhv_papr_host *ph; +unsigned long msr, i; +int trap; +long rc; + + ph = &vcpu->arch.papr_host; + + msr =mfmsr (); +kvmppc_msr_hard_disable_set_facilities (vcpu, msr); +if (lazy_irq_pending ()) +return 0 ; + +kvmhv_papr_flush_vcpu (vcpu, time_limit); + +accumulate_time (vcpu, &vcpu->arch.in_guest); + rc =plpar_guest_run_vcpu (0 , vcpu->kvm->arch.lpid, vcpu->vcpu_id, + &trap, &i); +
arch/powerpc/kvm/book3s_hv.c static void vcpu_vpa_increment_dispatch(struct kvm_vcpu *vcpu) · line 4091
Quoted 10 lines
arch/powerpc/kvm/book3s_hv.c +10
-0+if (rc !=H_SUCCESS ) { +pr_err ("KVM Guest Run VCPU hcall failed\n" ); +if (rc ==H_INVALID_ELEMENT_ID ) +pr_err ("KVM: Guest Run VCPU invalid element id at %ld\n" , i); +else if (rc ==H_INVALID_ELEMENT_SIZE ) +pr_err ("KVM: Guest Run VCPU invalid element size at %ld\n" , i); +else if (rc ==H_INVALID_ELEMENT_VALUE ) +pr_err ("KVM: Guest Run VCPU invalid element value at %ld\n" , i); +return 0 ; + }
This needs the proper error handling. Were you going to wait until I sent that up for existing code?
Quoted 9 lines
arch/powerpc/kvm/book3s_hv.c +1
-0@@ -5119,6 +5183,7 @@ static void kvmppc_core_commit_memory_region_hv(struct kvm *kvm,*/ void kvmppc_update_lpcr (struct kvm *kvm,unsigned long lpcr,unsigned long mask) { +struct kvm_vcpu *vcpu;long int i;u32 cores_done =0 ;
arch/powerpc/kvm/book3s_hv.c void kvmppc_update_lpcr(struct kvm *kvm, unsigned long lpcr, unsigned long mask) · line 5213
Quoted 10 lines
arch/powerpc/kvm/book3s_hv.c +6
-0@@ -5139,6 +5204,12 @@ void kvmppc_update_lpcr(struct kvm *kvm, unsigned long lpcr, unsigned long mask)if (++cores_done >= kvm->arch.online_vcores)break ; } + +if (kvmhv_on_papr ()) { +kvm_for_each_vcpu (i, vcpu, kvm) { +kvmppc_set_lpcr_hv (vcpu, vcpu->arch.vcore->lpcr); + } + } }
vcpu define could go in that scope I guess.
Quoted 33 lines
arch/powerpc/kvm/book3s_hv.c +20
-4@@ -5405,15 +5476,43 @@ static int kvmppc_core_init_vm_hv(struct kvm *kvm)/* Allocate the guest's logical partition ID */ - lpid =kvmppc_alloc_lpid (); -if ((long )lpid <0 ) -return -ENOMEM ; - kvm->arch.lpid = lpid; +if (!kvmhv_on_papr ()) { + lpid =kvmppc_alloc_lpid (); +if ((long )lpid <0 ) +return -ENOMEM ; + kvm->arch.lpid = lpid; + }kvmppc_alloc_host_rm_ops ();kvmhv_vm_nested_init (kvm); +if (kvmhv_on_papr ()) { +long rc; +unsigned long guest_id; + + rc =plpar_guest_create (0 , &guest_id); + +if (rc !=H_SUCCESS ) +pr_err ("KVM: Create Guest hcall failed, rc=%ld\n" , rc); + +switch (rc) { +case H_PARAMETER : +case H_FUNCTION : +case H_STATE : +return -EINVAL ;
arch/powerpc/kvm/book3s_hv.c static int kvmppc_core_init_vm_hv(struct kvm *kvm) · line 5513
Quoted 10 lines
arch/powerpc/kvm/book3s_hv.c +10
-0+case H_NOT_ENOUGH_RESOURCES : +case H_ABORTED : +return -ENOMEM ; +case H_AUTHORITY : +return -EPERM ; +case H_NOT_AVAILABLE : +return -EBUSY ; + } + kvm->arch.lpid = guest_id; + }
I wouldn't mind putting lpid/guest_id in different variables. guest_id is 64-bit isn't it? LPIDR is 32. If nothing else that could cause issues if the hypervisor does something clever with the token.
arch/powerpc/kvm/book3s_hv.c static void kvmppc_core_destroy_vm_hv(struct kvm *kvm) · line 5680
Quoted 8 lines
arch/powerpc/kvm/book3s_hv.c +2
-1@@ -5573,10 +5675,14 @@ static void kvmppc_core_destroy_vm_hv(struct kvm *kvm) kvm->arch.process_table =0 ;if (kvm->arch.secure_guest)uv_svm_terminate (kvm->arch.lpid); -kvmhv_set_ptbl_entry (kvm->arch.lpid,0 ,0 ); +if (!kvmhv_on_papr ()) +kvmhv_set_ptbl_entry (kvm->arch.lpid,0 ,0 ); }
Would be nice to have a +ve test for the "existing" API. All we have to do is think of a name for it.
Thanks,
Nick
Re: [RFC PATCH v2 5/6] KVM: PPC: Add support for nested PAPR guests
On Wed, Jun 7, 2023 at 7:09 PM Nicholas Piggin <[email protected]> wrote:
[snip]
Quoted 2 lines
You lost your comments.
Thanks
Quoted 31 lines
Quoted 16 lines
arch/powerpc/include/asm/kvm_book3s.h +1
-0@@ -12,6 +12,7 @@#include <linux/types.h> #include <linux/kvm_host.h> #include <asm/kvm_book3s_asm.h> +#include <asm/guest-state-buffer.h> struct kvmppc_bat {u64 raw;@@ -316,6 +317,57 @@ long int kvmhv_nested_page_fault(struct kvm_vcpu *vcpu);void kvmppc_giveup_fac (struct kvm_vcpu *vcpu,ulong fac);arch/powerpc/include/asm/kvm_book3s.h long int kvmhv_nested_page_fault(struct kvm_vcpu *vcpu); · line 328
Quoted 10 lines
arch/powerpc/include/asm/kvm_book3s.h +9
-0+ +#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE + +extern bool __kvmhv_on_papr; + +static inline bool kvmhv_on_papr (void ) +{ +return __kvmhv_on_papr; +}It's a nitpick, but
kvmhv_on_pseries()is because we're runnning KVM-HV on a pseries guest kernel. Which is a papr guest kernel. So this kind of doesn't make sense if you read it the same way.
Quoted 2 lines
kvmhv_nested_using_papr()or something like that might read a bit better.
Will we go with kvmhv_using_nested_v2()?
Quoted 2 lines
This could be a static key too.
Will do.
Quoted 11 lines
arch/powerpc/include/asm/kvm_book3s.h struct kvm_vcpu_arch { · line 599
Quoted 9 lines
arch/powerpc/include/asm/kvm_book3s.h +1
-0@@ -575,6 +593,7 @@ struct kvm_vcpu_arch {ulong dscr;ulong amr;ulong uamor; +ulong amor;ulong iamr;u32 ctrl;u32 dabrx;This belongs somewhere else.
It can be dropped.
Quoted 11 lines
arch/powerpc/include/asm/kvm_book3s.h struct kvm_vcpu_arch { · line 853
Quoted 8 lines
arch/powerpc/include/asm/kvm_book3s.h +2
-0@@ -829,6 +848,8 @@ struct kvm_vcpu_arch {u64 nested_hfscr;/* HFSCR that the L1 requested for the nested guest */ u32 nested_vcpu_id;gpa_t nested_io_gpr; +/* For nested APIv2 guests*/ +struct kvmhv_papr_host papr_host;#endif This is not exactly a papr host. Might have to come up with a better name especially if we implement a L0 things could get confusing.
Any name ideas? nestedv2_state?
Quoted 33 lines
Quoted 18 lines
arch/powerpc/include/asm/kvm_book3s.h +13
-0@@ -342,6 +343,203 @@ static inline long plpar_get_cpu_characteristics(struct h_cpu_char_result *p)return rc; } +static inline long plpar_guest_create (unsigned long flags,unsigned long *guest_id) +{ +unsigned long retbuf[PLPAR_HCALL_BUFSIZE ]; +unsigned long token; +long rc; + + token =-1UL ; +while (true) { + rc =plpar_hcall (H_GUEST_CREATE , retbuf, flags, token); +if (rc ==H_SUCCESS ) { + *guest_id = retbuf[0 ]; +break ; + }arch/powerpc/include/asm/kvm_book3s.h static inline long plpar_get_cpu_characteristics(struct h_cpu_char_result *p) · line 368
Quoted 10 lines
arch/powerpc/include/asm/kvm_book3s.h +10
-0+ +if (rc ==H_BUSY ) { + token = retbuf[0 ]; +cpu_relax (); +continue ; + } + +if (H_IS_LONG_BUSY (rc)) { + token = retbuf[0 ]; +mdelay (get_longbusy_msecs (rc));All of these things need a non-sleeping delay? Can we sleep instead? Or if not, might have to think about going back to the caller and it can retry.
Quoted 4 lines
get/set state might be a bit inconvenient, although I don't expect that should potentially take so long as guest and vcpu create/delete, so at least those ones would be good if they're called while preemptable.
Yeah no reason not to sleep except for get/set, let me try it out.
Quoted 9 lines
Quoted 9 lines
arch/powerpc/kvm/book3s_hv.c +0
-0@@ -383,6 +383,11 @@ static void kvmppc_core_vcpu_put_hv(struct kvm_vcpu *vcpu)spin_unlock_irqrestore (&vcpu->arch.tbacct_lock, flags); }
Quoted 7 lines
arch/powerpc/kvm/book3s_hv.c static void kvmppc_core_vcpu_put_hv(struct kvm_vcpu *vcpu) · line 389
Quoted 4 lines
arch/powerpc/kvm/book3s_hv.c +4
-0+static void kvmppc_set_pvr_hv (struct kvm_vcpu *vcpu,u32 pvr) +{ + vcpu->arch.pvr = pvr; +}Didn't you lose this in a previous patch? I thought it must have moved to a header but it reappears.
Yes, that was meant to stay put.
Quoted 13 lines
Quoted 13 lines
arch/powerpc/kvm/book3s_hv.c +2
-1+/* Dummy value used in computing PCR value below */ #define PCR_ARCH_31 (PCR_ARCH_300 << 1)@@ -1262,13 +1267,14 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)return RESUME_HOST ;break ;#endif -case H_RANDOM : +case H_RANDOM : {unsigned long rand;
Quoted 10 lines
arch/powerpc/kvm/book3s_hv.c int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu) · line 1280
Quoted 8 lines
arch/powerpc/kvm/book3s_hv.c +1
-0if (!arch_get_random_seed_longs (&rand,1 )) ret =H_HARDWARE ;kvmppc_set_gpr (vcpu,4 , rand);break ; + }case H_RPT_INVALIDATE : ret = kvmppc_h_rpt_invalidate(vcpu, kvmppc_get_gpr(vcpu, 4), kvmppc_get_gpr(vcpu, 5),Compile fix for a previous patch.
Thanks.
Quoted 19 lines
Quoted 19 lines
arch/powerpc/kvm/book3s_hv.c +7
-1@@ -2921,14 +2927,21 @@ static int kvmppc_core_vcpu_create_hv(struct kvm_vcpu *vcpu) vcpu->arch.shared_big_endian = false;#endif #endif -kvmppc_set_mmcr_hv (vcpu,0 ,MMCR0_FC ); +if (kvmhv_on_papr ()) { + err =kvmhv_papr_vcpu_create (vcpu, &vcpu->arch.papr_host); +if (err <0 ) +return err; + } + +kvmppc_set_mmcr_hv (vcpu,0 ,MMCR0_FC );if (cpu_has_feature (CPU_FTR_ARCH_31 )) {kvmppc_set_mmcr_hv (vcpu,0 ,kvmppc_get_mmcr_hv (vcpu,0 ) |MMCR0_PMCCEXT );kvmppc_set_mmcra_hv (vcpu,MMCRA_BHRB_DISABLE ); }
Quoted 6 lines
arch/powerpc/kvm/book3s_hv.c static int kvmppc_core_vcpu_create_hv(struct kvm_vcpu *vcpu) · line 2944
Quoted 2 lines
arch/powerpc/kvm/book3s_hv.c +1
-0kvmppc_set_ctrl_hv (vcpu,CTRL_RUNLATCH ); +kvmppc_set_amor_hv (vcpu, ~0 );This AMOR thing should go somewhere else. Not actually sure why it needs to be added to the vcpu since it's always ~0. Maybe just put that in a #define somewhere and use that.
Yes, you are right, just can get rid of it from the vcpu entirely.
Quoted 30 lines
Quoted 30 lines
arch/powerpc/kvm/book3s_hv.c +25
-0@@ -4042,6 +4059,50 @@ static void vcpu_vpa_increment_dispatch(struct kvm_vcpu *vcpu) } } +static int kvmhv_vcpu_entry_papr (struct kvm_vcpu *vcpu,u64 time_limit,unsigned long lpcr,u64 *tb) +{ +struct kvmhv_papr_host *ph; +unsigned long msr, i; +int trap; +long rc; + + ph = &vcpu->arch.papr_host; + + msr =mfmsr (); +kvmppc_msr_hard_disable_set_facilities (vcpu, msr); +if (lazy_irq_pending ()) +return 0 ; + +kvmhv_papr_flush_vcpu (vcpu, time_limit); + +accumulate_time (vcpu, &vcpu->arch.in_guest); + rc =plpar_guest_run_vcpu (0 , vcpu->kvm->arch.lpid, vcpu->vcpu_id, + &trap, &i); + +if (rc !=H_SUCCESS ) { +pr_err ("KVM Guest Run VCPU hcall failed\n" ); +if (rc ==H_INVALID_ELEMENT_ID ) +pr_err ("KVM: Guest Run VCPU invalid element id at %ld\n" , i); +else if (rc ==H_INVALID_ELEMENT_SIZE )
Quoted 8 lines
arch/powerpc/kvm/book3s_hv.c static void vcpu_vpa_increment_dispatch(struct kvm_vcpu *vcpu) · line 4091
Quoted 5 lines
arch/powerpc/kvm/book3s_hv.c +5
-0+pr_err ("KVM: Guest Run VCPU invalid element size at %ld\n" , i); + else if (rc == H_INVALID_ELEMENT_VALUE) + pr_err("KVM: Guest Run VCPU invalid element value at %ld\n", i); +return 0 ; + }This needs the proper error handling. Were you going to wait until I sent that up for existing code?
Overall the unhappy paths need to be tightened up in the next revision. But yeah this hits the same thing as the v1 API.
Quoted 9 lines
Quoted 9 lines
arch/powerpc/kvm/book3s_hv.c +1
-0@@ -5119,6 +5183,7 @@ static void kvmppc_core_commit_memory_region_hv(struct kvm *kvm,*/ void kvmppc_update_lpcr (struct kvm *kvm,unsigned long lpcr,unsigned long mask) { +struct kvm_vcpu *vcpu;long int i;u32 cores_done =0 ;
Quoted 13 lines
arch/powerpc/kvm/book3s_hv.c void kvmppc_update_lpcr(struct kvm *kvm, unsigned long lpcr, unsigned long mask) · line 5213
Quoted 11 lines
arch/powerpc/kvm/book3s_hv.c +6
-0@@ -5139,6 +5204,12 @@ void kvmppc_update_lpcr(struct kvm *kvm, unsigned long lpcr, unsigned long mask)if (++cores_done >= kvm->arch.online_vcores)break ; } + +if (kvmhv_on_papr ()) { +kvm_for_each_vcpu (i, vcpu, kvm) { +kvmppc_set_lpcr_hv (vcpu, vcpu->arch.vcore->lpcr); + } + } }vcpu define could go in that scope I guess.
True.
Quoted 40 lines
Quoted 40 lines
arch/powerpc/kvm/book3s_hv.c +26
-4@@ -5405,15 +5476,43 @@ static int kvmppc_core_init_vm_hv(struct kvm *kvm)/* Allocate the guest's logical partition ID */ - lpid =kvmppc_alloc_lpid (); -if ((long )lpid <0 ) -return -ENOMEM ; - kvm->arch.lpid = lpid; +if (!kvmhv_on_papr ()) { + lpid =kvmppc_alloc_lpid (); +if ((long )lpid <0 ) +return -ENOMEM ; + kvm->arch.lpid = lpid; + }kvmppc_alloc_host_rm_ops ();kvmhv_vm_nested_init (kvm); +if (kvmhv_on_papr ()) { +long rc; +unsigned long guest_id; + + rc =plpar_guest_create (0 , &guest_id); + +if (rc !=H_SUCCESS ) +pr_err ("KVM: Create Guest hcall failed, rc=%ld\n" , rc); + +switch (rc) { +case H_PARAMETER : +case H_FUNCTION : +case H_STATE : +return -EINVAL ; +case H_NOT_ENOUGH_RESOURCES : +case H_ABORTED : +return -ENOMEM ; +case H_AUTHORITY : +return -EPERM ; +case H_NOT_AVAILABLE :
Quoted 8 lines
arch/powerpc/kvm/book3s_hv.c static int kvmppc_core_init_vm_hv(struct kvm *kvm) · line 5513
Quoted 4 lines
arch/powerpc/kvm/book3s_hv.c +4
-0+return -EBUSY ; + } + kvm->arch.lpid = guest_id; + }I wouldn't mind putting lpid/guest_id in different variables.
guest_idis 64-bit isn't it? LPIDR is 32. If nothing else that could cause issues if the hypervisor does something clever with the token.
I was trying to get rid of a difference between this API and the others, but I'd forgotten about the 64bit / 32bit difference. Will put it back in its own variable.
Quoted 12 lines
arch/powerpc/kvm/book3s_hv.c static void kvmppc_core_destroy_vm_hv(struct kvm *kvm) · line 5680
Quoted 9 lines
arch/powerpc/kvm/book3s_hv.c +2
-1@@ -5573,10 +5675,14 @@ static void kvmppc_core_destroy_vm_hv(struct kvm *kvm) kvm->arch.process_table =0 ;if (kvm->arch.secure_guest)uv_svm_terminate (kvm->arch.lpid); -kvmhv_set_ptbl_entry (kvm->arch.lpid,0 ,0 ); +if (!kvmhv_on_papr ()) +kvmhv_set_ptbl_entry (kvm->arch.lpid,0 ,0 ); }Would be nice to have a +ve test for the "existing" API. All we have to do is think of a name for it.
Will we go with nestedv1?
Thanks,
Jordan
Quoted 4 lines
Thanks,
Nick
RFC PATCH v2 6/6 docs: powerpc: Document nested KVM on POWER
In reply to Jordan NietheFrom: Michael Neuling <[email protected]>
Document support for nested KVM on POWER using the existing API as well as the new PAPR API. This includes the new HCALL interface and how it used by KVM.
---
v2:
- Separated into individual patch
--- Documentation/powerpc/index.rst | 1 + Documentation/powerpc/kvm-nested.rst | 636 +++++++++++++++++++++++++++ 2 files changed, 637 insertions(+) create mode 100644 Documentation/powerpc/kvm-nested.rst
Documentation/powerpc/index.rst +1 -0
@@ -25,6 +25,7 @@ powerpc isa-versions kaslr-booke32 mpc52xx + kvm-nested papr_hcalls pci_iov_resource_on_powernv pmu-ebb
Documentation/powerpc/kvm-nested.rst new file +631 -0
631 lines not on this page · raw message
Signature · 1 line
2.31.1
Re: [PATCH RFC v2 6/6] docs: powerpc: Document nested KVM on POWER
Quoted 1 line · Jordan Niethe
From: Michael Neuling <[email protected]>
Hi,
There are some minor typos in the documentation pointed out below
Documentation/powerpc/kvm-nested.rst line 185
Quoted 8 lines
Documentation/powerpc/kvm-nested.rst new file +8
-0+H_GUEST_GET_STATE() +------------------- + +This is called to get state associated with an L2 (Guest-wide or vCPU specific). +This info is passed via the Guest State Buffer (GSB), a standard format as +explained later in this doc, necessary details below: + +This can set either L2 wide or vcpu specific information. Examples of
We are getting the info about vcpu here : s/set/get
Documentation/powerpc/kvm-nested.rst line 280
Quoted 5 lines
Documentation/powerpc/kvm-nested.rst new file +5
-0+H_GUEST_RUN_VCPU() +------------------ + +This is called to run an L2 vCPU. The L2 and vCPU IDs are passed in as +parameters. The vCPU run with the state set previously using
Minor nit : s/run/runs
Quoted 14 lines
Documentation/powerpc/kvm-nested.rst new file +10
-0+H_GUEST_SET_STATE(). When the L2 exits, the L1 will resume from this +hcall. + +This hcall also has associated input and output GSBs. Unlike +H_GUEST_{S,G}ET_STATE(), these GSB pointers are not passed in as +parameters to the hcall (This was done in the interest of +performance). The locations of these GSBs must be preregistered using +the H_GUEST_SET_STATE() call with ID 0x0c00 and 0x0c01 (see table +below). +Signature · 2 lines
-- 2.31.1
Re: [PATCH RFC v2 6/6] docs: powerpc: Document nested KVM on POWER
Quoted 6 lines · On Wed, Jun 7, 2023 at 3:38 PM Gautam Menghani <[email protected]> wrote:
Quoted 1 line · Jordan Niethe
From: Michael Neuling <[email protected]>
Hi,
There are some minor typos in the documentation pointed out below
Thank you, will correct in the next revision.
Jordan
Quoted 36 lines
Documentation/powerpc/kvm-nested.rst line 185
Quoted 10 lines
Documentation/powerpc/kvm-nested.rst new file +8
-0+H_GUEST_GET_STATE() +------------------- + +This is called to get state associated with an L2 (Guest-wide or vCPU specific). +This info is passed via the Guest State Buffer (GSB), a standard format as +explained later in this doc, necessary details below: + +This can set either L2 wide or vcpu specific information. Examples ofWe are getting the info about vcpu here : s/set/get
Documentation/powerpc/kvm-nested.rst line 280
Quoted 5 lines
Documentation/powerpc/kvm-nested.rst new file +5
-0+H_GUEST_RUN_VCPU() +------------------ + +This is called to run an L2 vCPU. The L2 and vCPU IDs are passed in as +parameters. The vCPU run with the state set previously usingMinor nit : s/run/runs
Quoted 14 lines
Documentation/powerpc/kvm-nested.rst new file +10
-1+H_GUEST_SET_STATE(). When the L2 exits, the L1 will resume from this +hcall. + +This hcall also has associated input and output GSBs. Unlike +H_GUEST_{S,G}ET_STATE(), these GSB pointers are not passed in as +parameters to the hcall (This was done in the interest of +performance). The locations of these GSBs must be preregistered using +the H_GUEST_SET_STATE() call with ID 0x0c00 and 0x0c01 (see table +below). + --2.31.1
Nicholas Piggin
reviewerQuoted 2 lines · Jordan Niethe
There is existing support for nested guests on powernv hosts however the hcall interface this uses is not support by other PAPR hosts.
I kind of liked it being called nested-HV v1 and v2 APIs as short and to the point, but I suppose that's ambiguous with version 2 of the v1 API, so papr is okay. What's the old API called in this scheme, then? "Existing API" is not great after patches go upstream.
And, you've probably explained it pretty well but slightly more of a background first up could be helpful. E.g.,
A nested-HV API for PAPR has been developed based on the KVM-specific nested-HV API that is upstream in Linux/KVM and QEMU. The PAPR API had to break compatibility to accommodate implementation in other hypervisors and partitioning firmware.
And key overall differences
The control flow and interrupt processing between L0, L1, and L2 in the new PAPR API are conceptually unchanged. Where the old API is almost stateless, the PAPR API is stateful, with the L1 registering L2 virtual machines and vCPUs with the L0. Supervisor-privileged register switching duty is now the responsibility for the L0, which holds canonical L2 register state and handles all switching. This new register handling motivates the "getters and setters" wrappers
...
Thanks,
Nick
Quoted 9 lines · On Wed, Jun 7, 2023 at 3:54 PM Nicholas Piggin <[email protected]> wrote:
Quoted 2 lines · Jordan Niethe
There is existing support for nested guests on powernv hosts however the hcall interface this uses is not support by other PAPR hosts.
I kind of liked it being called nested-HV v1 and v2 APIs as short and to the point, but I suppose that's ambiguous with version 2 of the v1 API, so papr is okay. What's the old API called in this scheme, then? "Existing API" is not great after patches go upstream.
Yes I was trying for a more descriptive name but it is just more confusing and I'm struggling for a better alternative.
In the next revision I'll use v1 and v2. For version 2 of v1 we now call it v1.2 or something like that?
Quoted 11 lines
And, you've probably explained it pretty well but slightly more of a background first up could be helpful. E.g.,
A nested-HV API for PAPR has been developed based on the KVM-specific nested-HV API that is upstream in Linux/KVM and QEMU. The PAPR API had to break compatibility to accommodate implementation in other hypervisors and partitioning firmware.
And key overall differences
Quoted 8 lines
The control flow and interrupt processing between L0, L1, and L2 in the new PAPR API are conceptually unchanged. Where the old API is almost stateless, the PAPR API is stateful, with the L1 registering L2 virtual machines and vCPUs with the L0. Supervisor-privileged register switching duty is now the responsibility for the L0, which holds canonical L2 register state and handles all switching. This new register handling motivates the "getters and setters" wrappers
...
I'll include something along those lines.
Thanks,
Jordan
Quoted 4 lines
Thanks,
Nick