KVM: PPC: In-kernel PAPR interrupt controller emulation
Subsystems (4)
cover KVM: PPC: In-kernel PAPR interrupt controller emulation
This series implements in-kernel emulation of the XICS interrupt controller specified in the PAPR specification and used by pseries guests. Since the XICS is organized a bit differently to the interrupt controllers used on x86 machines, I have defined some new ioctls for setting up the XICS and for saving and restoring its state. It may be possible to use some of the currently x86-specific ioctls instead.
Paul.
RFC PATCH 1/9 KVM: PPC: Remove unused argument to kvmppc_core_dequeue_external
Currently kvmppc_core_dequeue_external() takes a struct kvm_interrupt * argument and does nothing with it, in any of its implementations. This removes it in order to make things easier for forthcoming in-kernel interrupt controller emulation code.
--- arch/powerpc/include/asm/kvm_ppc.h | 3 +-- arch/powerpc/kvm/book3s.c | 3 +-- arch/powerpc/kvm/booke.c | 3 +-- arch/powerpc/kvm/powerpc.c | 2 +- 4 files changed, 4 insertions(+), 7 deletions(-)
arch/powerpc/include/asm/kvm_ppc.h +1 -2
@@ -105,8 +105,7 @@ extern void kvmppc_core_queue_dec(struct kvm_vcpu *vcpu);extern void kvmppc_core_dequeue_dec (struct kvm_vcpu *vcpu);extern void kvmppc_core_queue_external (struct kvm_vcpu *vcpu,struct kvm_interrupt *irq); -extern void kvmppc_core_dequeue_external (struct kvm_vcpu *vcpu, -struct kvm_interrupt *irq); +extern void kvmppc_core_dequeue_external (struct kvm_vcpu *vcpu);extern void kvmppc_core_flush_tlb (struct kvm_vcpu *vcpu); extern int kvmppc_core_emulate_op(struct kvm_run *run, struct kvm_vcpu *vcpu,
arch/powerpc/kvm/book3s.c +1 -2
@@ -160,8 +160,7 @@ void kvmppc_core_queue_external(struct kvm_vcpu *vcpu,kvmppc_book3s_queue_irqprio (vcpu, vec); } -void kvmppc_core_dequeue_external (struct kvm_vcpu *vcpu, -struct kvm_interrupt *irq) +void kvmppc_core_dequeue_external (struct kvm_vcpu *vcpu) {kvmppc_book3s_dequeue_irqprio (vcpu,BOOK3S_INTERRUPT_EXTERNAL );kvmppc_book3s_dequeue_irqprio (vcpu,BOOK3S_INTERRUPT_EXTERNAL_LEVEL );
arch/powerpc/kvm/booke.c +1 -2
@@ -213,8 +213,7 @@ void kvmppc_core_queue_external(struct kvm_vcpu *vcpu,kvmppc_booke_queue_irqprio (vcpu, prio); } -void kvmppc_core_dequeue_external (struct kvm_vcpu *vcpu, -struct kvm_interrupt *irq) +void kvmppc_core_dequeue_external (struct kvm_vcpu *vcpu) {clear_bit (BOOKE_IRQPRIO_EXTERNAL , &vcpu->arch.pending_exceptions);clear_bit (BOOKE_IRQPRIO_EXTERNAL_LEVEL , &vcpu->arch.pending_exceptions);
arch/powerpc/kvm/powerpc.c +1 -1
@@ -707,7 +707,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)int kvm_vcpu_ioctl_interrupt (struct kvm_vcpu *vcpu,struct kvm_interrupt *irq) {if (irq->irq ==KVM_INTERRUPT_UNSET ) { -kvmppc_core_dequeue_external (vcpu, irq); +kvmppc_core_dequeue_external (vcpu);return 0 ; }
Signature · 1 line
1.7.10.4
RFC PATCH 2/9 KVM: PPC: Book3S: Add infrastructure to implement kernel-side RTAS calls
In reply to Paul MackerrasFrom: Michael Ellerman <[email protected]>
For pseries machine emulation, in order to move the interrupt controller code to the kernel, we need to intercept some RTAS calls in the kernel itself. This adds an infrastructure to allow in-kernel handlers to be registered for RTAS services by name. A new ioctl, KVM_PPC_RTAS_DEFINE_TOKEN, then allows userspace to associate token values with those service names. Then, when the guest requests an RTAS service with one of those token values, it will be handled by the relevant in-kernel handler rather than being passed up to userspace as at present.
--- Documentation/virtual/kvm/api.txt | 19 ++++ arch/powerpc/include/asm/hvcall.h | 3 + arch/powerpc/include/asm/kvm_host.h | 1 + arch/powerpc/include/asm/kvm_ppc.h | 4 + arch/powerpc/include/uapi/asm/kvm.h | 6 ++ arch/powerpc/kvm/Makefile | 1 + arch/powerpc/kvm/book3s_hv.c | 18 +++- arch/powerpc/kvm/book3s_pr_papr.c | 7 ++ arch/powerpc/kvm/book3s_rtas.c | 182 +++++++++++++++++++++++++++++++++++ arch/powerpc/kvm/powerpc.c | 9 +- include/uapi/linux/kvm.h | 4 + 11 files changed, 252 insertions(+), 2 deletions(-) create mode 100644 arch/powerpc/kvm/book3s_rtas.c
Documentation/virtual/kvm/api.txt +19 -0
@@ -2071,6 +2071,25 @@ KVM_S390_INT_EXTERNAL_CALL (vcpu) - sigp external call; source cpu in parm Note that the vcpu ioctl is asynchronous to vcpu execution. +4.78 KVM_PPC_RTAS_DEFINE_TOKEN + +Capability: KVM_CAP_PPC_RTAS +Architectures: ppc +Type: vm ioctl +Parameters: struct kvm_rtas_token_args +Returns: 0 on success, -1 on error + +Defines a token value for a RTAS (Run Time Abstraction Services) +service in order to allow it to be handled in the kernel. The +argument struct gives the name of the service, which must be the name +of a service that has a kernel-side implementation. If the token +value is non-zero, it will be associated with that service, and +subsequent RTAS calls by the guest specifying that token will be +handled by the kernel. If the token value is 0, then any token +associated with the service will be forgotten, and subsequent RTAS +calls by the guest for that service will be passed to userspace to be +handled. + 5. The kvm_run structure ------------------------
arch/powerpc/include/asm/hvcall.h +3 -0
@@ -269,6 +269,9 @@#define H_GET_MPP_X 0x314#define MAX_HCALL_OPCODE H_GET_MPP_X +/* Platform specific hcalls, used by KVM */ +#define H_RTAS 0xf000 +#ifndef __ASSEMBLY__/**
arch/powerpc/include/asm/kvm_host.h +1 -0
@@ -255,6 +255,7 @@ struct kvm_arch {#endif /* CONFIG_KVM_BOOK3S_64_HV */ #ifdef CONFIG_PPC_BOOK3S_64 struct list_head spapr_tce_tables; +struct list_head rtas_tokens;#endif };
arch/powerpc/include/asm/kvm_ppc.h +4 -0
@@ -163,6 +163,10 @@ extern void kvmppc_bookehv_exit(void);extern int kvmppc_prepare_to_enter (struct kvm_vcpu *vcpu); +extern int kvm_vm_ioctl_rtas_define_token (struct kvm *kvm,void __user *argp); +extern int kvmppc_rtas_hcall (struct kvm_vcpu *vcpu); +extern void kvmppc_rtas_tokens_free (struct kvm *kvm); +/* * Cuts out inst bits with ordering according to spec. * That means the leftmost bit is zero. All given bits are included.
arch/powerpc/include/uapi/asm/kvm.h +6 -0
@@ -296,6 +296,12 @@ struct kvm_allocate_rma {__u64 rma_size; }; +/* for KVM_CAP_PPC_RTAS */ +struct kvm_rtas_token_args { +char name[120 ]; +__u64 token;/* Use a token of 0 to undefine a mapping */ +}; +struct kvm_book3e_206_tlb_entry {__u32 mas8;__u32 mas1;
arch/powerpc/kvm/Makefile +1 -0
@@ -80,6 +80,7 @@ kvm-book3s_64-module-objs := \ emulate.o \ book3s.o \ book3s_64_vio.o \ + book3s_rtas.o \ $(kvm-book3s_64-objs-y) kvm-objs-$(CONFIG_KVM_BOOK3S_64) := $(kvm-book3s_64-module-objs)
arch/powerpc/kvm/book3s_hv.c +17 -1
@@ -479,7 +479,7 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)unsigned long req =kvmppc_get_gpr (vcpu,3 );unsigned long target, ret =H_SUCCESS ;struct kvm_vcpu *tvcpu; -int idx; +int idx, rc;switch (req) {case H_ENTER :
@@ -515,6 +515,19 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)kvmppc_get_gpr (vcpu,5 ), kvmppc_get_gpr(vcpu, 6));break ; +case H_RTAS : +if (list_empty (&vcpu->kvm->arch.rtas_tokens)) +return RESUME_HOST ; + + rc =kvmppc_rtas_hcall (vcpu); + +if (rc = -ENOENT ) +return RESUME_HOST ; +else if (rc =0 ) +break ; + +/* Send the error out to userspace via KVM_RUN */ +return rc;default :return RESUME_HOST ; }
@@ -1815,6 +1828,7 @@ int kvmppc_core_init_vm(struct kvm *kvm) kvm->arch.lpid = lpid;INIT_LIST_HEAD (&kvm->arch.spapr_tce_tables); +INIT_LIST_HEAD (&kvm->arch.rtas_tokens); kvm->arch.rma =NULL ;
@@ -1860,6 +1874,8 @@ void kvmppc_core_destroy_vm(struct kvm *kvm) kvm->arch.rma =NULL ; } +kvmppc_rtas_tokens_free (kvm); +kvmppc_free_hpt (kvm);WARN_ON (!list_empty (&kvm->arch.spapr_tce_tables)); }
arch/powerpc/kvm/book3s_pr_papr.c +7 -0
@@ -246,6 +246,13 @@ int kvmppc_h_pr(struct kvm_vcpu *vcpu, unsigned long cmd)clear_bit (KVM_REQ_UNHALT , &vcpu->requests); vcpu->stat.halt_wakeup++;return EMULATE_DONE ; +case H_RTAS : +if (list_empty (&vcpu->kvm->arch.rtas_tokens)) +return RESUME_HOST ; +if (kvmppc_rtas_hcall (vcpu)) +break ; +kvmppc_set_gpr (vcpu,3 ,0 ); +return EMULATE_DONE ; }return EMULATE_FAIL ;
arch/powerpc/kvm/book3s_rtas.c new file +182 -0
182 lines not on this page · raw message
arch/powerpc/kvm/powerpc.c +8 -1
@@ -332,6 +332,7 @@ int kvm_dev_ioctl_check_extension(long ext)#ifdef CONFIG_PPC_BOOK3S_64 case KVM_CAP_SPAPR_TCE :case KVM_CAP_PPC_ALLOC_HTAB : +case KVM_CAP_PPC_RTAS : r =1 ;break ;#endif /* CONFIG_PPC_BOOK3S_64 */
@@ -909,8 +910,8 @@ long kvm_arch_vm_ioctl(struct file *filp,#ifdef CONFIG_KVM_BOOK3S_64_HV case KVM_ALLOCATE_RMA : { -struct kvm *kvm = filp->private_data;struct kvm_allocate_rma rma; +struct kvm *kvm = filp->private_data; r =kvm_vm_ioctl_allocate_rma (kvm, &rma);if (r >=0 &©_to_user (argp, &rma,sizeof (rma)))
@@ -947,6 +948,12 @@ long kvm_arch_vm_ioctl(struct file *filp, r = -EFAULT ; break; } +case KVM_PPC_RTAS_DEFINE_TOKEN : { +struct kvm *kvm = filp->private_data; + + r =kvm_vm_ioctl_rtas_define_token (kvm, argp); +break ; + }#endif /* CONFIG_PPC_BOOK3S_64 */ default : r = -ENOTTY ;
include/uapi/linux/kvm.h +4 -0
@@ -634,6 +634,7 @@ struct kvm_ppc_smmu_info {#endif #define KVM_CAP_IRQFD_RESAMPLE 82#define KVM_CAP_PPC_BOOKE_WATCHDOG 83 +#define KVM_CAP_PPC_RTAS 84#ifdef KVM_CAP_IRQ_ROUTING
@@ -859,6 +860,9 @@ struct kvm_s390_ucas_mapping {#define KVM_CREATE_SPAPR_TCE _IOW(KVMIO, 0xa8, struct kvm_create_spapr_tce)/* Available with KVM_CAP_RMA */ #define KVM_ALLOCATE_RMA _IOR(KVMIO, 0xa9, struct kvm_allocate_rma) +/* Available with KVM_CAP_PPC_RTAS */ +#define KVM_PPC_RTAS_DEFINE_TOKEN _IOW(KVMIO, 0xaa, struct kvm_rtas_token_args) +/* * ioctls for vcpu fds
Signature · 1 line
1.7.10.4
RFC PATCH 3/9 KVM: PPC: Book3S: Add kernel emulation for the XICS interrupt controller
In reply to Paul MackerrasFrom: Benjamin Herrenschmidt <[email protected]>
This adds in-kernel emulation of the XICS (eXternal Interrupt Controller Specification) interrupt controller specified by PAPR, for both HV and PR KVM guests.
It supports up to 4095 "BUIDs" (blocks of interrupts) of up to 4096 interrupts each. Each vcpu gets one ICP (interrupt controller presentation) entity, and each BUID gets one ICS (interrupt controller source) entity. These entities are created with the new KVM_CREATE_IRQCHIP_ARGS ioctl, which takes a structure indicating which type of controller to create and the parameters pertaining to it (flags for both, BUID and number of interrupt sources for ICS). We don't use the existing KVM_CREATE_IRQCHIP ioctl because it doesn't allow us to pass any parameters defining the new interrupt controller.
This is based on an initial implementation by Michael Ellerman <[email protected]> reworked by myself.
--- Documentation/virtual/kvm/api.txt | 17 + arch/powerpc/include/asm/kvm_book3s.h | 1 + arch/powerpc/include/asm/kvm_host.h | 8 + arch/powerpc/include/asm/kvm_ppc.h | 18 + arch/powerpc/include/uapi/asm/kvm.h | 32 + arch/powerpc/kvm/Makefile | 1 + arch/powerpc/kvm/book3s.c | 2 +- arch/powerpc/kvm/book3s_hv.c | 20 + arch/powerpc/kvm/book3s_pr.c | 13 + arch/powerpc/kvm/book3s_pr_papr.c | 16 + arch/powerpc/kvm/book3s_rtas.c | 51 +- arch/powerpc/kvm/book3s_xics.c | 1112 +++++++++++++++++++++++++++++++++ arch/powerpc/kvm/powerpc.c | 21 + include/uapi/linux/kvm.h | 6 +- 14 files changed, 1315 insertions(+), 3 deletions(-) create mode 100644 arch/powerpc/kvm/book3s_xics.c
Documentation/virtual/kvm/api.txt +17 -0
@@ -2090,6 +2090,23 @@ associated with the service will be forgotten, and subsequent RTAS calls by the guest for that service will be passed to userspace to be handled. +4.79 KVM_CREATE_IRQCHIP_ARGS + +Capability: KVM_CAP_SPAPR_XICS +Architectures: ppc +Type: vm ioctl +Parameters: struct kvm_irqchip_args +Returns: 0 on success, -1 on error + +Creates an interrupt controller model in the kernel. This is +currently only implemented for the XICS (eXternal Interrupt Controller +Specification) model defined in PAPR. This ioctl creates either an +interrupt controller presentation (ICP) or an interrupt controller +source (ICS) entity, depending on the type field of the argument +struct. When creating an ICS, the argument struct further indicates +the BUID (Bus Unit ID) and number of interrupt sources for the new +ICS. + 5. The kvm_run structure ------------------------
arch/powerpc/include/asm/kvm_book3s.h +1 -0
@@ -140,6 +140,7 @@ extern int kvmppc_mmu_hv_init(void);extern int kvmppc_ld (struct kvm_vcpu *vcpu,ulong *eaddr,int size,void *ptr,bool data);extern int kvmppc_st (struct kvm_vcpu *vcpu,ulong *eaddr,int size,void *ptr,bool data);extern void kvmppc_book3s_queue_irqprio (struct kvm_vcpu *vcpu,unsigned int vec); +extern void kvmppc_book3s_dequeue_irqprio (struct kvm_vcpu *vcpu,unsigned int vec);extern void kvmppc_inject_interrupt (struct kvm_vcpu *vcpu,int vec,u64 flags);extern void kvmppc_set_bat (struct kvm_vcpu *vcpu,struct kvmppc_bat *bat,bool upper,u32 val);
arch/powerpc/include/asm/kvm_host.h +8 -0
@@ -190,6 +190,10 @@ struct kvmppc_linear_info {int type; }; +/* XICS components, defined in boo3s_xics.c */ +struct kvmppc_xics ; +struct kvmppc_icp ; +/* * The reverse mapping array has one entry for each HPTE, * which stores the guest's view of the second word of the HPTE
@@ -256,6 +260,7 @@ struct kvm_arch {#ifdef CONFIG_PPC_BOOK3S_64 struct list_head spapr_tce_tables;struct list_head rtas_tokens; +struct kvmppc_xics *xics;#endif };
@@ -565,6 +570,9 @@ struct kvm_vcpu_arch {u64 busy_stolen;u64 busy_preempt;#endif +#ifdef CONFIG_PPC_BOOK3S_64 +struct kvmppc_icp *icp;/* XICS presentation controller */ +#endif };/* Values for vcpu->arch.state */
arch/powerpc/include/asm/kvm_ppc.h +18 -0
@@ -131,6 +131,12 @@ extern long kvmppc_prepare_vrma(struct kvm *kvm,extern void kvmppc_map_vrma (struct kvm_vcpu *vcpu,struct kvm_memory_slot *memslot,unsigned long porder);extern int kvmppc_pseries_do_hcall (struct kvm_vcpu *vcpu); +extern int kvmppc_xics_hcall (struct kvm_vcpu *vcpu,u32 cmd); +extern int kvmppc_xics_ioctl (struct kvm *kvm,unsigned ioctl,unsigned long arg); +extern void kvmppc_xics_free_icp (struct kvm_vcpu *vcpu); +extern int kvmppc_xics_create_icp (struct kvm_vcpu *vcpu); +extern void kvmppc_xics_free (struct kvm *kvm); +extern long kvm_vm_ioctl_create_spapr_tce (struct kvm *kvm,struct kvm_create_spapr_tce *args); extern long kvmppc_h_put_tce(struct kvm_vcpu *vcpu, unsigned long liobn,
@@ -166,6 +172,8 @@ extern int kvmppc_prepare_to_enter(struct kvm_vcpu *vcpu);extern int kvm_vm_ioctl_rtas_define_token (struct kvm *kvm,void __user *argp);extern int kvmppc_rtas_hcall (struct kvm_vcpu *vcpu);extern void kvmppc_rtas_tokens_free (struct kvm *kvm); +extern int kvmppc_xics_set_xive (struct kvm *kvm,u32 irq,u32 server,u32 priority); +extern int kvmppc_xics_get_xive (struct kvm *kvm,u32 irq,u32 *server,u32 *priority);/* * Cuts out inst bits with ordering according to spec.
@@ -262,6 +270,16 @@ static inline void kvmppc_set_xics_phys(int cpu, unsigned long addr)static inline void kvm_linear_init (void ) {} + +#endif + +#ifdef CONFIG_PPC_BOOK3S_64 +static inline int kvmppc_xics_enabled (struct kvm *kvm) +{ +return kvm->arch.xics !=NULL ; +} +#else +static inline int kvmppc_xics_enabled (struct kvm *kvm) {return 0 ; }#endif int kvm_vcpu_ioctl_config_tlb(struct kvm_vcpu *vcpu,
arch/powerpc/include/uapi/asm/kvm.h +32 -0
@@ -302,6 +302,38 @@ struct kvm_rtas_token_args {__u64 token;/* Use a token of 0 to undefine a mapping */ }; +/* for KVM_CAP_SPAPR_XICS */ +#define __KVM_HAVE_IRQCHIP_ARGS +struct kvm_irqchip_args { +#define KVM_IRQCHIP_TYPE_ICP 0/* XICS: ICP (presentation controller) */ +#define KVM_IRQCHIP_TYPE_ICS 1/* XICS: ICS (source controller) */ +__u32 type; +union { +/* XICS ICP arguments. This needs to be called once before +* creating any VCPU to initialize the main kernel XICS data +* structures. +*/ +struct { +__u32 flags; + } icp; + +/* XICS ICS arguments. You can call this for every BUID you +* want to make available. +* +* The BUID is 12 bits, the interrupt number within a BUID +* is up to 12 bits as well. The resulting interrupt numbers +* exposed to the guest are BUID || IRQ which is 24 bit +* +* BUID cannot be 0. +*/ +struct { +__u32 flags; +__u16 buid; +__u16 nr_irqs; + } ics; + }; +}; +struct kvm_book3e_206_tlb_entry {__u32 mas8;__u32 mas1;
arch/powerpc/kvm/Makefile +1 -0
@@ -81,6 +81,7 @@ kvm-book3s_64-module-objs := \ book3s.o \ book3s_64_vio.o \ book3s_rtas.o \ + book3s_xics.o \ $(kvm-book3s_64-objs-y) kvm-objs-$(CONFIG_KVM_BOOK3S_64) := $(kvm-book3s_64-module-objs)
arch/powerpc/kvm/book3s.c +1 -1
@@ -104,7 +104,7 @@ static int kvmppc_book3s_vec2irqprio(unsigned int vec)return prio; } -static void kvmppc_book3s_dequeue_irqprio (struct kvm_vcpu *vcpu, +void kvmppc_book3s_dequeue_irqprio (struct kvm_vcpu *vcpu,unsigned int vec) {unsigned long old_pending = vcpu->arch.pending_exceptions;
arch/powerpc/kvm/book3s_hv.c +20 -0
@@ -528,6 +528,14 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)/* Send the error out to userspace via KVM_RUN */ return rc; +case H_XIRR : +case H_CPPR : +case H_EOI : +case H_IPI : +if (kvmppc_xics_enabled (vcpu->kvm)) { + ret =kvmppc_xics_hcall (vcpu, req); +break ; + }/* fallthrough */ default :return RESUME_HOST ; }
@@ -876,6 +884,13 @@ struct kvm_vcpu *kvmppc_core_vcpu_create(struct kvm *kvm, unsigned int id)spin_lock_init (&vcpu->arch.tbacct_lock); vcpu->arch.busy_preempt =TB_NIL ; +/* Create the XICS */ +if (kvmppc_xics_enabled (kvm)) { + err =kvmppc_xics_create_icp (vcpu); +if (err <0 ) + goto free_vcpu; + } +kvmppc_mmu_book3s_hv_init (vcpu); vcpu->arch.state =KVMPPC_VCPU_NOTREADY ;
@@ -926,6 +941,8 @@ void kvmppc_core_vcpu_free(struct kvm_vcpu *vcpu)kvmppc_unpin_guest_page (vcpu->kvm, vcpu->arch.vpa.pinned_addr);spin_unlock (&vcpu->arch.vpa_update_lock);kvm_vcpu_uninit (vcpu); +if (kvmppc_xics_enabled (vcpu->kvm)) +kvmppc_xics_free_icp (vcpu);kmem_cache_free (kvm_vcpu_cache, vcpu); }
@@ -1876,6 +1893,9 @@ void kvmppc_core_destroy_vm(struct kvm *kvm)kvmppc_rtas_tokens_free (kvm); +if (kvmppc_xics_enabled (kvm)) +kvmppc_xics_free (kvm); +kvmppc_free_hpt (kvm);WARN_ON (!list_empty (&kvm->arch.spapr_tce_tables)); }
arch/powerpc/kvm/book3s_pr.c +13 -0
@@ -1052,6 +1052,13 @@ struct kvm_vcpu *kvmppc_core_vcpu_create(struct kvm *kvm, unsigned int id)if (err <0 ) goto uninit_vcpu; +/* Create the XICS */ +if (kvmppc_xics_enabled (kvm)) { + err =kvmppc_xics_create_icp (vcpu); +if (err <0 ) + goto free_vcpu; + } +return vcpu; uninit_vcpu:
@@ -1068,6 +1075,8 @@ void kvmppc_core_vcpu_free(struct kvm_vcpu *vcpu) {struct kvmppc_vcpu_book3s *vcpu_book3s =to_book3s (vcpu); +if (kvmppc_xics_enabled (vcpu->kvm)) +kvmppc_xics_free_icp (vcpu);free_page ((unsigned long )vcpu->arch.shared &PAGE_MASK );kvm_vcpu_uninit (vcpu);kfree (vcpu_book3s->shadow_vcpu);
@@ -1278,6 +1287,7 @@ int kvmppc_core_init_vm(struct kvm *kvm) {#ifdef CONFIG_PPC64 INIT_LIST_HEAD (&kvm->arch.spapr_tce_tables); +INIT_LIST_HEAD (&kvm->arch.rtas_tokens);#endif return 0 ;
@@ -1288,6 +1298,9 @@ void kvmppc_core_destroy_vm(struct kvm *kvm)#ifdef CONFIG_PPC64 WARN_ON (!list_empty (&kvm->arch.spapr_tce_tables));#endif +if (kvmppc_xics_enabled (kvm)) +kvmppc_xics_free (kvm); + }static int kvmppc_book3s_init (void )
arch/powerpc/kvm/book3s_pr_papr.c +16 -0
@@ -227,6 +227,15 @@ static int kvmppc_h_pr_put_tce(struct kvm_vcpu *vcpu)return EMULATE_DONE ; } +static int kvmppc_h_pr_xics_hcall (struct kvm_vcpu *vcpu,u32 cmd) +{ +long rc =kvmppc_xics_hcall (vcpu, cmd); +if (rc ==H_TOO_HARD ) +return EMULATE_FAIL ; +kvmppc_set_gpr (vcpu,3 , rc); +return EMULATE_DONE ; +} +int kvmppc_h_pr (struct kvm_vcpu *vcpu,unsigned long cmd) {switch (cmd) {
@@ -246,6 +255,13 @@ int kvmppc_h_pr(struct kvm_vcpu *vcpu, unsigned long cmd)clear_bit (KVM_REQ_UNHALT , &vcpu->requests); vcpu->stat.halt_wakeup++;return EMULATE_DONE ; +case H_XIRR : +case H_CPPR : +case H_EOI : +case H_IPI : +if (kvmppc_xics_enabled (vcpu->kvm)) +return kvmppc_h_pr_xics_hcall (vcpu, cmd); +break ;case H_RTAS :if (list_empty (&vcpu->kvm->arch.rtas_tokens))return RESUME_HOST ;
arch/powerpc/kvm/book3s_rtas.c +50 -1
62 lines not on this page · raw message
arch/powerpc/kvm/book3s_xics.c new file +1112 -0
1,112 lines not on this page · raw message
arch/powerpc/kvm/powerpc.c +21 -0
@@ -374,6 +374,7 @@ int kvm_dev_ioctl_check_extension(long ext)break ;#ifdef CONFIG_PPC_BOOK3S_64 case KVM_CAP_PPC_GET_SMMU_INFO : +case KVM_CAP_SPAPR_XICS : r =1 ;break ;#endif
@@ -954,6 +955,26 @@ long kvm_arch_vm_ioctl(struct file *filp, r =kvm_vm_ioctl_rtas_define_token (kvm, argp); break; } +case KVM_IRQ_LINE : { +struct kvm *kvm = filp->private_data; + + r = -ENOTTY ; +if (kvmppc_xics_enabled (kvm)) + r =kvmppc_xics_ioctl (kvm, ioctl, arg); +break ; + } +case KVM_CREATE_IRQCHIP_ARGS : { +struct kvm *kvm = filp->private_data; +u32 type; + + r = -EFAULT ; +if (get_user (type, (u32 __user *)argp)) +break ; + r = -EINVAL ; +if (type ==KVM_IRQCHIP_TYPE_ICP || type ==KVM_IRQCHIP_TYPE_ICS ) + r =kvmppc_xics_ioctl (kvm, ioctl, arg); +break ; + }#endif /* CONFIG_PPC_BOOK3S_64 */ default : r = -ENOTTY ;
include/uapi/linux/kvm.h +5 -1
@@ -115,6 +115,7 @@ struct kvm_irq_level {* ACPI gsi notion of irq. * For IA-64 (APIC model) IOAPIC0: irq 0-23; IOAPIC1: irq 24-47.. * For X86 (standard AT mode) PIC0/1: irq 0-15. IOAPIC0: 0-23.. +* On powerpc SPAPR, the ICS source number, level is ignored. */ union {__u32 irq;
@@ -635,6 +636,7 @@ struct kvm_ppc_smmu_info {#define KVM_CAP_IRQFD_RESAMPLE 82#define KVM_CAP_PPC_BOOKE_WATCHDOG 83#define KVM_CAP_PPC_RTAS 84 +#define KVM_CAP_SPAPR_XICS 85#ifdef KVM_CAP_IRQ_ROUTING
@@ -862,7 +864,9 @@ struct kvm_s390_ucas_mapping {#define KVM_ALLOCATE_RMA _IOR(KVMIO, 0xa9, struct kvm_allocate_rma)/* Available with KVM_CAP_PPC_RTAS */ #define KVM_PPC_RTAS_DEFINE_TOKEN _IOW(KVMIO, 0xaa, struct kvm_rtas_token_args) - +#ifdef __KVM_HAVE_IRQCHIP_ARGS +#define KVM_CREATE_IRQCHIP_ARGS _IOW(KVMIO, 0xab, struct kvm_irqchip_args) +#endif /* * ioctls for vcpu fds
Signature · 1 line
1.7.10.4
RFC PATCH 4/9 KVM: PPC: Book3S HV: Speed up wakeups of CPUs on HV KVM
In reply to Paul MackerrasFrom: Benjamin Herrenschmidt <[email protected]>
Currently, we wake up a CPU by sending a host IPI with smp_send_reschedule() to thread 0 of that core, which will take all threads out of the guest, and cause them to re-evaluate their interrupt status on the way back in.
This adds a mechanism to differentiate real host IPIs from IPIs sent by KVM for guest threads to poke each other, in order to target the guest threads precisely when possible and avoid that global switch of the core to host state.
We then use this new facility in the in-kernel XICS code.
--- arch/powerpc/include/asm/kvm_book3s_asm.h | 8 ++- arch/powerpc/include/asm/kvm_ppc.h | 29 ++++++++ arch/powerpc/kernel/asm-offsets.c | 2 + arch/powerpc/kvm/book3s_hv.c | 26 ++++++- arch/powerpc/kvm/book3s_hv_rmhandlers.S | 112 +++++++++++++++++++++++------ arch/powerpc/kvm/book3s_xics.c | 2 +- arch/powerpc/sysdev/xics/icp-native.c | 8 +++ 7 files changed, 163 insertions(+), 24 deletions(-)
arch/powerpc/include/asm/kvm_book3s_asm.h +7 -1
@@ -20,6 +20,11 @@#ifndef __ASM_KVM_BOOK3S_ASM_H__#define __ASM_KVM_BOOK3S_ASM_H__ +/* XICS ICP register offsets */ +#define XICS_XIRR 4 +#define XICS_MFRR 0xc +#define XICS_IPI 2/* interrupt source # for IPIs */ +#ifdef __ASSEMBLY__#ifdef CONFIG_KVM_BOOK3S_HANDLER
@@ -81,10 +86,11 @@ struct kvmppc_host_state {#ifdef CONFIG_KVM_BOOK3S_64_HV u8 hwthread_req;u8 hwthread_state; - +u8 host_ipi;struct kvm_vcpu *kvm_vcpu;struct kvmppc_vcore *kvm_vcore;unsigned long xics_phys; +u32 saved_xirr;u64 dabr;u64 host_mmcr[3 ];u32 host_pmc[8 ];
arch/powerpc/include/asm/kvm_ppc.h +29 -0
@@ -262,6 +262,21 @@ static inline void kvmppc_set_xics_phys(int cpu, unsigned long addr) paca[cpu].kvm_hstate.xics_phys = addr; } +static inline u32 kvmppc_get_xics_latch (void ) +{ +u32 xirr =get_paca ()->kvm_hstate.saved_xirr; + +get_paca ()->kvm_hstate.saved_xirr =0 ; + +return xirr; +} + +static inline void kvmppc_set_host_ipi (int cpu,u8 host_ipi) +{ + paca[cpu].kvm_hstate.host_ipi = host_ipi; +} + +extern void kvmppc_fast_vcpu_kick (struct kvm_vcpu *vcpu);extern void kvm_linear_init (void );#else
@@ -271,6 +286,18 @@ static inline void kvmppc_set_xics_phys(int cpu, unsigned long addr)static inline void kvm_linear_init (void ) {} +static inline u32 kvmppc_get_xics_latch (void ) +{ +return 0 ; +} + +static inline void kvmppc_set_host_ipi (int cpu,u8 host_ipi) +{} + +static inline void kvmppc_fast_vcpu_kick (struct kvm_vcpu *vcpu) +{ +kvm_vcpu_kick (vcpu); +}#endif #ifdef CONFIG_PPC_BOOK3S_64
@@ -314,4 +341,6 @@ static inline void kvmppc_lazy_ee_enable(void) #endif } +extern void xics_wake_cpu (int cpu); +#endif /* __POWERPC_KVM_PPC_H__ */
arch/powerpc/kernel/asm-offsets.c +2 -0
@@ -545,6 +545,8 @@ int main(void)HSTATE_FIELD (HSTATE_KVM_VCPU , kvm_vcpu);HSTATE_FIELD (HSTATE_KVM_VCORE , kvm_vcore);HSTATE_FIELD (HSTATE_XICS_PHYS , xics_phys); +HSTATE_FIELD (HSTATE_SAVED_XIRR , saved_xirr); +HSTATE_FIELD (HSTATE_HOST_IPI , host_ipi);HSTATE_FIELD (HSTATE_MMCR , host_mmcr);HSTATE_FIELD (HSTATE_PMC , host_pmc);HSTATE_FIELD (HSTATE_PURR , host_purr);
arch/powerpc/kvm/book3s_hv.c +25 -1
@@ -66,6 +66,31 @@static void kvmppc_end_cede (struct kvm_vcpu *vcpu);static int kvmppc_hv_setup_htab_rma (struct kvm_vcpu *vcpu); +void kvmppc_fast_vcpu_kick (struct kvm_vcpu *vcpu) +{ +int me; +int cpu = vcpu->cpu; +wait_queue_head_t *wqp; + + wqp =kvm_arch_vcpu_wq (vcpu); +if (waitqueue_active (wqp)) { +wake_up_interruptible (wqp); + ++vcpu->stat.halt_wakeup; + } + + me =get_cpu (); + +/* CPU points to the first thread of the core */ +if (cpu != me && cpu >=0 && cpu < nr_cpu_ids) { +int real_cpu = cpu + vcpu->arch.ptid; +if (paca[real_cpu].kvm_hstate.xics_phys) +xics_wake_cpu (real_cpu); +else if (cpu_online (cpu)) +smp_send_reschedule (cpu); + } +put_cpu (); +} +/* * We use the vcpu_load/put functions to measure stolen time. * Stolen time is counted as time when either the vcpu is able to
@@ -974,7 +999,6 @@ static void kvmppc_end_cede(struct kvm_vcpu *vcpu) }extern int __kvmppc_vcore_entry (struct kvm_run *kvm_run,struct kvm_vcpu *vcpu); -extern void xics_wake_cpu (int cpu);static void kvmppc_remove_runnable (struct kvmppc_vcore *vc,struct kvm_vcpu *vcpu)
arch/powerpc/kvm/book3s_hv_rmhandlers.S +91 -21
150 lines not on this page · raw message
arch/powerpc/kvm/book3s_xics.c +1 -1
@@ -318,7 +318,7 @@ static inline bool icp_try_update(struct kvmppc_icp *icp,kvmppc_book3s_queue_irqprio (icp->vcpu,BOOK3S_INTERRUPT_EXTERNAL_LEVEL );if (!change_self) -kvm_vcpu_kick (icp->vcpu); +kvmppc_fast_vcpu_kick (icp->vcpu); } bail:return success;
arch/powerpc/sysdev/xics/icp-native.c +8 -0
@@ -51,6 +51,12 @@ static struct icp_ipl __iomem *icp_native_regs[NR_CPUS];static inline unsigned int icp_native_get_xirr (void ) {int cpu =smp_processor_id (); +unsigned int xirr; + +/* Handled an interrupt latched by KVM */ + xirr =kvmppc_get_xics_latch (); +if (xirr) +return xirr;return in_be32 (&icp_native_regs[cpu]->xirr.word); }
@@ -138,6 +144,7 @@ static unsigned int icp_native_get_irq(void)static void icp_native_cause_ipi (int cpu,unsigned long data) { +kvmppc_set_host_ipi (cpu,1 );icp_native_set_qirr (cpu,IPI_PRIORITY ); }
@@ -151,6 +158,7 @@ static irqreturn_t icp_native_ipi_action(int irq, void *dev_id) {int cpu =smp_processor_id (); +kvmppc_set_host_ipi (cpu,0 );icp_native_set_qirr (cpu,0xff );return smp_ipi_demux ();
Signature · 1 line
1.7.10.4
RFC PATCH 5/9 KVM: PPC: Book3S HV: Add support for real mode ICP in XICS emulation
In reply to Paul MackerrasFrom: Benjamin Herrenschmidt <[email protected]>
This adds an implementation of the XICS hypercalls in real mode for HV KVM, which allows us to avoid exiting the guest MMU context on all threads for a variety of operations such as fetching a pending interrupt, EOI of messages, IPIs, etc.
For debugging purposes, the use of the real mode implementation can be disabled by setting the KVM_ICP_FLAG_NOREALMODE bit in the icp.flags field of struct kvm_irqchip_args.
--- arch/powerpc/include/uapi/asm/kvm.h | 1 + arch/powerpc/kvm/Makefile | 1 + arch/powerpc/kvm/book3s_hv_rm_xics.c | 402 ++++++++++++++++++++++++++++++++++ arch/powerpc/kvm/book3s_xics.c | 153 +++++-------- arch/powerpc/kvm/book3s_xics.h | 116 ++++++++++ 5 files changed, 572 insertions(+), 101 deletions(-) create mode 100644 arch/powerpc/kvm/book3s_hv_rm_xics.c create mode 100644 arch/powerpc/kvm/book3s_xics.h
arch/powerpc/include/uapi/asm/kvm.h +1 -0
@@ -314,6 +314,7 @@ struct kvm_irqchip_args {* structures. */ struct { +#define KVM_ICP_FLAG_NOREALMODE 0x00000001/* Disable real mode ICP */ __u32 flags; } icp;
arch/powerpc/kvm/Makefile +1 -0
@@ -72,6 +72,7 @@ kvm-book3s_64-builtin-objs-$(CONFIG_KVM_BOOK3S_64_HV) := \ book3s_hv_rmhandlers.o \ book3s_hv_rm_mmu.o \ book3s_64_vio_hv.o \ + book3s_hv_rm_xics.o \ book3s_hv_builtin.o kvm-book3s_64-module-objs := \
arch/powerpc/kvm/book3s_hv_rm_xics.c new file +402 -0
402 lines not on this page · raw message
arch/powerpc/kvm/book3s_xics.c +52 -101
224 lines not on this page · raw message
arch/powerpc/kvm/book3s_xics.h new file +116 -0
116 lines not on this page · raw message
Signature · 1 line
1.7.10.4
RFC PATCH 6/9 KVM: PPC: Book3S HV: Improve real-mode handling of external interrupts
In reply to Paul MackerrasThis streamlines our handling of external interrupts that come in while we're in the guest. First, when waking up a hardware thread that was napping, we split off the "napping due to H_CEDE" case earlier, and use the code that handles an external interrupt (0x500) in the guest to handle that too. Secondly, the code that handles those external interrupts now checks if any other thread is exiting to the host before bouncing an external interrupt to the guest, and also checks that there is actually an external interrupt pending for the guest before setting the LPCR MER bit (mediated external request).
This also makes sure that we clear the "ceded" flag when we handle a wakeup from cede in real mode, and fixes a potential infinite loop in kvmppc_run_vcpu() which can occur if we ever end up with the ceded flag set but MSR[EE] off.
--- arch/powerpc/include/asm/reg.h | 1 + arch/powerpc/kvm/book3s_hv.c | 5 +- arch/powerpc/kvm/book3s_hv_rmhandlers.S | 140 +++++++++++++++++-------------- 3 files changed, 81 insertions(+), 65 deletions(-)
arch/powerpc/include/asm/reg.h +1 -0
@@ -254,6 +254,7 @@#define LPCR_PECE1 0x00002000/* decrementer can cause exit */ #define LPCR_PECE2 0x00001000/* machine check etc can cause exit */ #define LPCR_MER 0x00000800/* Mediated External Exception */ +#define LPCR_MER_SH 11#define LPCR_LPES 0x0000000c#define LPCR_LPES0 0x00000008/* LPAR Env selector 0 */ #define LPCR_LPES1 0x00000004/* LPAR Env selector 1 */
arch/powerpc/kvm/book3s_hv.c +4 -1
@@ -1373,9 +1373,12 @@ static int kvmppc_run_vcpu(struct kvm_run *kvm_run, struct kvm_vcpu *vcpu)break ; vc->runner = vcpu; n_ceded =0 ; -list_for_each_entry (v, &vc->runnable_threads, arch.run_list) +list_for_each_entry (v, &vc->runnable_threads, arch.run_list) {if (!v->arch.pending_exceptions) n_ceded += v->arch.ceded; +else + v->arch.ceded =0 ; + }if (n_ceded == vc->n_runnable)kvmppc_vcore_blocked (vc);else
arch/powerpc/kvm/book3s_hv_rmhandlers.S +76 -64
234 lines not on this page · raw message
Signature · 1 line
1.7.10.4
RFC PATCH 7/9 KVM: PPC: Book3S: Add support for ibm,int-on/off RTAS calls
In reply to Paul MackerrasThis adds support for the ibm,int-on and ibm,int-off RTAS calls to the in-kernel XICS emulation and corrects the handling of the saved priority by the ibm,set-xive RTAS call. With this, ibm,int-off sets the specified interrupt's priority in its saved_priority field and sets the priority to 0xff (the least favoured value). ibm,int-on restores the saved_priority to the priority field, and ibm,set-xive sets both the priority and the saved_priority to the specified priority value.
--- arch/powerpc/include/asm/kvm_ppc.h | 2 + arch/powerpc/kvm/book3s_rtas.c | 40 +++++++++++++++++ arch/powerpc/kvm/book3s_xics.c | 86 +++++++++++++++++++++++++++++------- arch/powerpc/kvm/book3s_xics.h | 2 +- 4 files changed, 114 insertions(+), 16 deletions(-)
arch/powerpc/include/asm/kvm_ppc.h +2 -0
@@ -174,6 +174,8 @@ extern int kvmppc_rtas_hcall(struct kvm_vcpu *vcpu);extern void kvmppc_rtas_tokens_free (struct kvm *kvm);extern int kvmppc_xics_set_xive (struct kvm *kvm,u32 irq,u32 server,u32 priority);extern int kvmppc_xics_get_xive (struct kvm *kvm,u32 irq,u32 *server,u32 *priority); +extern int kvmppc_xics_int_on (struct kvm *kvm,u32 irq); +extern int kvmppc_xics_int_off (struct kvm *kvm,u32 irq);/* * Cuts out inst bits with ordering according to spec.
arch/powerpc/kvm/book3s_rtas.c +40 -0
@@ -64,6 +64,44 @@ out: args->rets[0 ] = rc; } +static void kvm_rtas_int_off (struct kvm_vcpu *vcpu,struct rtas_args *args) +{ +u32 irq; +int rc; + +if (args->nargs !=1 || args->nret !=1 ) { + rc =-3 ; + goto out; + } + + irq = args->args[0 ]; + + rc =kvmppc_xics_int_off (vcpu->kvm, irq); +if (rc) + rc =-3 ; +out: + args->rets[0 ] = rc; +} + +static void kvm_rtas_int_on (struct kvm_vcpu *vcpu,struct rtas_args *args) +{ +u32 irq; +int rc; + +if (args->nargs !=1 || args->nret !=1 ) { + rc =-3 ; + goto out; + } + + irq = args->args[0 ]; + + rc =kvmppc_xics_int_on (vcpu->kvm, irq); +if (rc) + rc =-3 ; +out: + args->rets[0 ] = rc; +} +struct rtas_handler {void (*handler)(struct kvm_vcpu *vcpu,struct rtas_args *args);char *name;
@@ -72,6 +110,8 @@ struct rtas_handler {static struct rtas_handler rtas_handlers[] = { { .name ="ibm,set-xive" , .handler = kvm_rtas_set_xive }, { .name ="ibm,get-xive" , .handler = kvm_rtas_get_xive }, + { .name ="ibm,int-off" , .handler = kvm_rtas_int_off }, + { .name ="ibm,int-on" , .handler = kvm_rtas_int_on }, };struct rtas_token_definition {
arch/powerpc/kvm/book3s_xics.c +71 -15
120 lines not on this page · raw message
arch/powerpc/kvm/book3s_xics.h +1 -1
@@ -22,7 +22,7 @@ struct ics_irq_state {u32 number;u32 server;u8 priority; -u8 saved_priority;/* currently unused */ +u8 saved_priority;u8 resend;u8 masked_pending;u8 asserted;/* Only for LSI */
Signature · 1 line
1.7.10.4
RFC PATCH 8/9 KVM: PPC: Book3S: Facilities to save/restore XICS presentation ctrler state
In reply to Paul MackerrasThis adds the ability for userspace to save and restore the state of the XICS interrupt presentation controllers (ICPs) via the KVM_GET/SET_ONE_REG interface. Since there is one ICP per vcpu, we simply define a new 64-bit register in the ONE_REG space for the ICP state. The state includes the CPU priority setting, the pending IPI priority, and the priority and source number of any pending external interrupt.
--- arch/powerpc/include/asm/kvm_ppc.h | 2 + arch/powerpc/include/uapi/asm/kvm.h | 11 +++++ arch/powerpc/kvm/book3s.c | 18 +++++++ arch/powerpc/kvm/book3s_xics.c | 90 +++++++++++++++++++++++++++++++++++ 4 files changed, 121 insertions(+)
arch/powerpc/include/asm/kvm_ppc.h +2 -0
@@ -176,6 +176,8 @@ extern int kvmppc_xics_set_xive(struct kvm *kvm, u32 irq, u32 server, u32 prioriextern int kvmppc_xics_get_xive (struct kvm *kvm,u32 irq,u32 *server,u32 *priority);extern int kvmppc_xics_int_on (struct kvm *kvm,u32 irq);extern int kvmppc_xics_int_off (struct kvm *kvm,u32 irq); +extern u64 kvmppc_xics_get_icp (struct kvm_vcpu *vcpu); +extern int kvmppc_xics_set_icp (struct kvm_vcpu *vcpu,u64 icpval);/* * Cuts out inst bits with ordering according to spec.
arch/powerpc/include/uapi/asm/kvm.h +11 -0
@@ -425,4 +425,15 @@ struct kvm_book3e_206_tlb_params {#define KVM_REG_PPC_VPA_SLB (KVM_REG_PPC | KVM_REG_SIZE_U128 | 0x83)#define KVM_REG_PPC_VPA_DTL (KVM_REG_PPC | KVM_REG_SIZE_U128 | 0x84) +/* Per-vcpu interrupt controller state */ +#define KVM_REG_PPC_ICP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U64 | 0x85) +#define KVM_REG_PPC_ICP_CPPR_SHIFT 56/* current proc priority */ +#define KVM_REG_PPC_ICP_CPPR_MASK 0xff +#define KVM_REG_PPC_ICP_XISR_SHIFT 32/* interrupt status field */ +#define KVM_REG_PPC_ICP_XISR_MASK 0xffffff +#define KVM_REG_PPC_ICP_MFRR_SHIFT 24/* pending IPI priority */ +#define KVM_REG_PPC_ICP_MFRR_MASK 0xff +#define KVM_REG_PPC_ICP_PPRI_SHIFT 16/* pending irq priority */ +#define KVM_REG_PPC_ICP_PPRI_MASK 0xff +#endif /* __LINUX_KVM_POWERPC_H */
arch/powerpc/kvm/book3s.c +18 -0
@@ -529,6 +529,15 @@ int kvm_vcpu_ioctl_get_one_reg(struct kvm_vcpu *vcpu, struct kvm_one_reg *reg) val =get_reg_val (reg->id, vcpu->arch.vscr.u[3 ]);break ;#endif /* CONFIG_ALTIVEC */ +#ifdef CONFIG_PPC_BOOK3S_64 +case KVM_REG_PPC_ICP_STATE : +if (!vcpu->arch.icp) { + r = -ENXIO ; +break ; + } + val =get_reg_val (reg->id,kvmppc_xics_get_icp (vcpu)); +break ; +#endif default : r = -EINVAL ;break ;
@@ -591,6 +600,15 @@ int kvm_vcpu_ioctl_set_one_reg(struct kvm_vcpu *vcpu, struct kvm_one_reg *reg) vcpu->arch.vscr.u[3 ] =set_reg_val (reg->id, val);break ;#endif /* CONFIG_ALTIVEC */ +#ifdef CONFIG_PPC_BOOK3S_64 +case KVM_REG_PPC_ICP_STATE : +if (!vcpu->arch.icp) { + r = -ENXIO ; +break ; + } + r =kvmppc_xics_set_icp (vcpu,set_reg_val (reg->id, val)); +break ; +#endif default : r = -EINVAL ;break ;
arch/powerpc/kvm/book3s_xics.c +90 -0
96 lines not on this page · raw message
Signature · 1 line
1.7.10.4
RFC PATCH 9/9 KVM: PPC: Book 3S: Facilities to save/restore XICS source controller state
In reply to Paul MackerrasThis adds the ability for userspace to save and restore the state of the XICS interrupt source controllers (ICS) via two new VM ioctls, KVM_IRQCHIP_GET_SOURCES and KVM_IRQCHIP_SET_SOURCES. Both take an argument struct that gives the starting interrupt source number, the number of interrupt sources to be processed, and a pointer to an array with space for 64 bits per source where the state of the interrupt sources are to be stored or loaded. The interrupt sources are required to all be within one ICS.
--- Documentation/virtual/kvm/api.txt | 33 ++++++++++++ arch/powerpc/kvm/book3s_xics.c | 108 +++++++++++++++++++++++++++++++++++++ arch/powerpc/kvm/powerpc.c | 4 +- include/uapi/linux/kvm.h | 17 ++++++ 4 files changed, 161 insertions(+), 1 deletion(-)
Documentation/virtual/kvm/api.txt +33 -0
@@ -2107,6 +2107,39 @@ struct. When creating an ICS, the argument struct further indicates the BUID (Bus Unit ID) and number of interrupt sources for the new ICS. +4.80 KVM_IRQCHIP_GET_SOURCES + +Capability: KVM_CAP_SPAPR_XICS +Architectures: ppc +Type: vm ioctl +Parameters: struct kvm_irq_sources +Returns: 0 on success, -1 on error + +Copies configuration and status information about a range of interrupt +sources into a user-supplied buffer. The argument struct gives the +starting interrupt source number and the number of interrupt sources. +All of the interrupt sources must be within the range of a single +interrupt source controller (ICS). The user buffer is an array of +64-bit quantities, one per interrupt source, with (from the least- +significant bit) 32 bits of interrupt server number, 8 bits of +priority, and 1 bit each for a level-sensitive indicator, a masked +indicator, and a pending indicator. + +4.81 KVM_IRQCHIP_SET_SOURCES + +Capability: KVM_CAP_SPAPR_XICS +Architectures: ppc +Type: vm ioctl +Parameters: struct kvm_irq_sources +Returns: 0 on success, -1 on error + +Sets the configuration and status for a range of interrupt sources +from information supplied in a user-supplied buffer. The argument +struct gives the starting interrupt source number and the number of +interrupt sources. All of the interrupt sources must be within the +range of a single interrupt source controller (ICS). The user buffer +is formatted as for KVM_IRQCHIP_GET_SOURCES. + 5. The kvm_run structure ------------------------
arch/powerpc/kvm/book3s_xics.c +108 -0
120 lines not on this page · raw message
arch/powerpc/kvm/powerpc.c +3 -1
@@ -955,7 +955,9 @@ long kvm_arch_vm_ioctl(struct file *filp, r =kvm_vm_ioctl_rtas_define_token (kvm, argp);break ; } -case KVM_IRQ_LINE : { +case KVM_IRQ_LINE : +case KVM_IRQCHIP_GET_SOURCES : +case KVM_IRQCHIP_SET_SOURCES : {struct kvm *kvm = filp->private_data; r = -ENOTTY ;
include/uapi/linux/kvm.h +17 -0
@@ -779,6 +779,21 @@ struct kvm_msi {__u8 pad[16 ]; }; +struct kvm_irq_sources { +__u32 irq; +__u32 nr_irqs; +__u64 __user *irqbuf; +}; + +/* irqbuf entries are laid out like this: */ +#define KVM_IRQ_SERVER_SHIFT 0 +#define KVM_IRQ_SERVER_MASK 0xffffffffULL +#define KVM_IRQ_PRIORITY_SHIFT 32 +#define KVM_IRQ_PRIORITY_MASK 0xff +#define KVM_IRQ_LEVEL_SENSITIVE (1ULL << 40) +#define KVM_IRQ_MASKED (1ULL << 41) +#define KVM_IRQ_PENDING (1ULL << 42) +/* * ioctls for VM fds */
@@ -867,6 +882,8 @@ struct kvm_s390_ucas_mapping {#ifdef __KVM_HAVE_IRQCHIP_ARGS#define KVM_CREATE_IRQCHIP_ARGS _IOW(KVMIO, 0xab, struct kvm_irqchip_args)#endif +#define KVM_IRQCHIP_GET_SOURCES _IOW(KVMIO, 0xac, struct kvm_irq_sources) +#define KVM_IRQCHIP_SET_SOURCES _IOW(KVMIO, 0xad, struct kvm_irq_sources)/* * ioctls for vcpu fds
Signature · 1 line
1.7.10.4
Quoted 7 lines · Paul Mackerras
This series implements in-kernel emulation of the XICS interrupt controller specified in the PAPR specification and used by pseries guests. Since the XICS is organized a bit differently to the interrupt controllers used on x86 machines, I have defined some new ioctls for setting up the XICS and for saving and restoring its state. It may be possible to use some of the currently x86-specific ioctls instead.
Is this one already following the new world order that we discussed during KVM Forum? :)
Alex
Quoted 6 lines
Paul.
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Alexander Graf
Quoted 7 lines · Paul Mackerras
This series implements in-kernel emulation of the XICS interrupt controller specified in the PAPR specification and used by pseries guests. Since the XICS is organized a bit differently to the interrupt controllers used on x86 machines, I have defined some new ioctls for setting up the XICS and for saving and restoring its state. It may be possible to use some of the currently x86-specific ioctls instead.
Is this one already following the new world order that we discussed during KVM Forum? :)
The "new world order" .... (sorry, looks like nobody took notes and people expect me to do a write up from memory now ... :-)
Well, mostly we agreed that the x86 stuff wasn't going to work for us.
So basically what we discussed boils down to:
- Move the existing "generic" KVM ioctl to create the kernel APIC to
x86 since it's no as-is useful for other archs who, among other things, might need to pass argument based on the machine type (type of interrupt subsystem for example, etc...)
- Once that's done, well, instanciating interrupt controller objects
becomes pretty much an arch specific matter. We could try to keep some ioctls somewhat common though it's not *that* useful because the various types & arguments are going to be fairly arch specific, so goes for configuration. Examples of what could be kept common are:
* Create irq chip, takes at least a "type" argument, possibly a few
more type-specific args (or a union, but then let's keep space in there since we can't change the size of the struct later as it would impact the ioctl number afaik).
* Assigning the address of an irq chip when it can change (see ARM
patches)
- The existing business with irqfd etc... is mostly ok, except the
interfaces messing around with MSIs (see virtio-pci use of kvm functions directly). The assignment of an irq number for an MSI must be a hook, probably a PCI controller hook, so x86 can get it done via its existing kernel interfaces and sane architectures can keep the assignment in qemu where it belongs.
Cheers,
Ben.
Quoted 16 lines · Benjamin Herrenschmidt
Quoted 11 lines · Alexander Graf
Quoted 7 lines · Paul Mackerras
This series implements in-kernel emulation of the XICS interrupt controller specified in the PAPR specification and used by pseries guests. Since the XICS is organized a bit differently to the interrupt controllers used on x86 machines, I have defined some new ioctls for setting up the XICS and for saving and restoring its state. It may be possible to use some of the currently x86-specific ioctls instead.
Is this one already following the new world order that we discussed during KVM Forum? :)
The "new world order" .... (sorry, looks like nobody took notes and people expect me to do a write up from memory now ... :-)
Quoted 8 lines
Well, mostly we agreed that the x86 stuff wasn't going to work for us.
So basically what we discussed boils down to:
- Move the existing "generic" KVM ioctl to create the kernel APIC to
x86 since it's no as-is useful for other archs who, among other things, might need to pass argument based on the machine type (type of interrupt subsystem for example, etc...)
Assuming you're talking about KVM_CREATE_IRQCHIP, it is already handled entirely in arch code (arch/x86/kvm/x86.c and arch/ia64/kvm/kvm-ia64.c), along with KVM_GET_IRQCHIP and KVM_SET_IRQCHIP.
Quoted 10 lines
- Once that's done, well, instanciating interrupt controller objects
becomes pretty much an arch specific matter. We could try to keep some ioctls somewhat common though it's not *that* useful because the various types & arguments are going to be fairly arch specific, so goes for configuration. Examples of what could be kept common are:* Create irq chip, takes at least a "type" argument, possibly a few
more type-specific args (or a union, but then let's keep space in there since we can't change the size of the struct later as it would impact the ioctl number afaik).
The existing KVM_CREATE_IRQCHIP is defined as _IO(...) which means that it doesn't read or write memory, but there is still the 3rd argument to the ioctl, which would give us 64 bits to indicate the type of the top-level IRQ controller (XICS, MPIC, ...), but not much more.
It sounds like the agreement at the meeting was more along the lines of the KVM_CREATE_IRQCHIP_ARGS ioctl (as in the patches I posted) which can be called multiple times to instantiate pieces of the interrupt framework, rather than having a KVM_CREATE_IRQCHIP that gets called once early on to say that there we are using in-kernel interrupt controller emulation, followed by other calls to configure the various parts of the framework. Is that accurate?
Quoted 9 lines
* Assigning the address of an irq chip when it can change (see ARM
patches)- The existing business with irqfd etc... is mostly ok, except the
interfaces messing around with MSIs (see virtio-pci use of kvm functions directly). The assignment of an irq number for an MSI must be a hook, probably a PCI controller hook, so x86 can get it done via its existing kernel interfaces and sane architectures can keep the assignment in qemu where it belongs.
So, if I've understood you correctly about what was agreed, the set of ioctls that is implemented in the patches I posted is in line with what was agreed, isn't it? If not, where does it differ? (To recap, I had KVM_CREATE_IRQCHIP_ARGS, KVM_IRQCHIP_GET_SOURCES and KVM_IRQCHIP_SET_SOURCES, plus a one-reg interface to get/set the vcpu-specific state.)
Paul.
Quoted 26 lines · Paul Mackerras
Quoted 16 lines · Benjamin Herrenschmidt
Quoted 11 lines · Alexander Graf
Quoted 7 lines · Paul Mackerras
This series implements in-kernel emulation of the XICS interrupt controller specified in the PAPR specification and used by pseries guests. Since the XICS is organized a bit differently to the interrupt controllers used on x86 machines, I have defined some new ioctls for setting up the XICS and for saving and restoring its state. It may be possible to use some of the currently x86-specific ioctls instead.
Is this one already following the new world order that we discussed during KVM Forum? :)
The "new world order" .... (sorry, looks like nobody took notes and people expect me to do a write up from memory now ... :-)
Quoted 8 lines
Well, mostly we agreed that the x86 stuff wasn't going to work for us.
So basically what we discussed boils down to:
- Move the existing "generic" KVM ioctl to create the kernel APIC to x86 since it's no as-is useful for other archs who, among other things, might need to pass argument based on the machine type (type of interrupt subsystem for example, etc...)
Quoted 4 lines
Assuming you're talking about
KVM_CREATE_IRQCHIP, it is already handled entirely in arch code (arch/x86/kvm/x86.candarch/ia64/kvm/kvm-ia64.c), along withKVM_GET_IRQCHIPandKVM_SET_IRQCHIP.
This part is about QEMU.
Quoted 18 lines
Quoted 11 lines
- Once that's done, well, instanciating interrupt controller objects becomes pretty much an arch specific matter. We could try to keep some ioctls somewhat common though it's not *that* useful because the various types & arguments are going to be fairly arch specific, so goes for configuration. Examples of what could be kept common are:
* Create irq chip, takes at least a "type" argument, possibly a few
more type-specific args (or a union, but then let's keep space in there since we can't change the size of the struct later as it would impact the ioctl number afaik).The existing
KVM_CREATE_IRQCHIPis defined as_IO(...)which means that it doesn't read or write memory, but there is still the 3rd argument to the ioctl, which would give us 64 bits to indicate the type of the top-level IRQ controller (XICS, MPIC, ...), but not much more.
Quoted 7 lines
It sounds like the agreement at the meeting was more along the lines of the
KVM_CREATE_IRQCHIP_ARGSioctl (as in the patches I posted) which can be called multiple times to instantiate pieces of the interrupt framework, rather than having aKVM_CREATE_IRQCHIPthat gets called once early on to say that there we are using in-kernel interrupt controller emulation, followed by other calls to configure the various parts of the framework. Is that accurate?
KVM_CREATE_IRQCHIP should get a parameter indicating the type of IRQCHIP (XICS / MPIC / APIC / VGIC / ...). The parameter-less version defaults to an arch specific default (APIC for x86, VGIC for arm, error on PPC).
I'm not 100% sure yet whether we want to support models with multiple IRQCHIPs. If so, we need to return a device token from the KVM_CREATE_IRQCHIP ioctl. Maybe it makes more sense to model specific cases like this as separate type though with specific, explicit subdevice ids. Not sure.
Other instantiation attributes we don't know that early on should be settable between the time frame of vm creation and first execution. An example for this are device addresses. Check out the threads "KVM: ARM: Rename KVM_SET_DEVICE_ADDRESS" and "KVM: ARM: Introduce KVM_SET_DEVICE_ADDRESS ioctl" for more information on this particular bit.
Alex
Quoted 19 lines
Quoted 10 lines
* Assigning the address of an irq chip when it can change (see ARM
patches)- The existing business with irqfd etc... is mostly ok, except the interfaces messing around with MSIs (see virtio-pci use of kvm functions directly). The assignment of an irq number for an MSI must be a hook, probably a PCI controller hook, so x86 can get it done via its existing kernel interfaces and sane architectures can keep the assignment in qemu where it belongs.
So, if I've understood you correctly about what was agreed, the set of ioctls that is implemented in the patches I posted is in line with what was agreed, isn't it? If not, where does it differ? (To recap, I had
KVM_CREATE_IRQCHIP_ARGS,KVM_IRQCHIP_GET_SOURCESandKVM_IRQCHIP_SET_SOURCES, plus a one-reg interface to get/set the vcpu-specific state.)Paul.
Quoted 1 line · Alexander Graf
Other instantiation attributes we don't know that early on should be settable between the time frame of vm creation and first execution. An example for this are device addresses. Check out the threads "KVM: ARM: Rename
KVM_SET_DEVICE_ADDRESS" and "KVM: ARM: IntroduceKVM_SET_DEVICE_ADDRESSioctl" for more information on this particular bit.
FWIW, on Freescale chips the MPIC base address can move at runtime if the guest writes to CCSRBAR (though QEMU doesn't currently implement this).
-Scott
Quoted 4 lines · Alexander Graf
Quoted 1 line · Paul Mackerras
On Sat, Dec 15, 2012 at 01:06:13PM +1100, Benjamin Herrenschmidt wrote:
[snip]
Quoted 11 lines
Quoted 9 lines
Quoted 4 lines
- Move the existing "generic" KVM ioctl to create the kernel APIC to x86 since it's no as-is useful for other archs who, among other things, might need to pass argument based on the machine type (type of interrupt subsystem for example, etc...)
Assuming you're talking about
KVM_CREATE_IRQCHIP, it is already handled entirely in arch code (arch/x86/kvm/x86.candarch/ia64/kvm/kvm-ia64.c), along withKVM_GET_IRQCHIPandKVM_SET_IRQCHIP.This part is about QEMU.
OK, that makes sense, but it presumably can't be done until the new kernel interface exists, right?
Quoted 26 lines
Quoted 18 lines
Quoted 11 lines
- Once that's done, well, instanciating interrupt controller objects becomes pretty much an arch specific matter. We could try to keep some ioctls somewhat common though it's not *that* useful because the various types & arguments are going to be fairly arch specific, so goes for configuration. Examples of what could be kept common are:
* Create irq chip, takes at least a "type" argument, possibly a few
more type-specific args (or a union, but then let's keep space in there since we can't change the size of the struct later as it would impact the ioctl number afaik).The existing
KVM_CREATE_IRQCHIPis defined as_IO(...)which means that it doesn't read or write memory, but there is still the 3rd argument to the ioctl, which would give us 64 bits to indicate the type of the top-level IRQ controller (XICS, MPIC, ...), but not much more.Quoted 7 lines
It sounds like the agreement at the meeting was more along the lines of the
KVM_CREATE_IRQCHIP_ARGSioctl (as in the patches I posted) which can be called multiple times to instantiate pieces of the interrupt framework, rather than having aKVM_CREATE_IRQCHIPthat gets called once early on to say that there we are using in-kernel interrupt controller emulation, followed by other calls to configure the various parts of the framework. Is that accurate?
Quoted 1 line
KVM_CREATE_IRQCHIPshould get a parameter indicating the type of IRQCHIP (XICS / MPIC / APIC / VGIC / ...). The parameter-less version defaults to an arch specific default (APIC for x86, VGIC for arm, error on PPC).
So, I propose KVM_CREATE_IRQCHIP_ARGS as the version of the ioctl that indicates the type of irqchip. The parameterless version already gives an error on PPC.
Quoted 1 line
I'm not 100% sure yet whether we want to support models with multiple IRQCHIPs. If so, we need to return a device token from the
KVM_CREATE_IRQCHIPioctl. Maybe it makes more sense to model specific cases like this as separate type though with specific, explicit subdevice ids. Not sure.
I think it depends on what you mean by IRQCHIP. If you mean the overall interrupt controller architecture, then no it doesn't make sense to have more than one. The overall architecture might well contain multiple identifiable pieces which could be considered IRQCHIPs. This is the case on x86 already, where they have two i8259-style PICs and one or more IOAPICs (and presumably emulated LAPICs as well). It's the case for the PAPR-style XICS controller, where we have a presentation controller (ICP) per CPU (like the x86 LAPIC), plus one or more source controllers (ICS) (like the x86 IOAPIC).
Which meaning of IRQCHIP do you prefer?
My series of patches uses KVM_CREATE_IRQCHIP_ARGS to indicate that the overall architecture is XICS (by creating the ICPs) and to create the source controllers (ICS), so it gets called multiple times. It would be possible to have KVM_CREATE_IRQCHIP_ARGS called just once with a parameter saying "XICS", and then create the source controllers implicitly as a side effect of the first KVM_IRQ_SET_SOURCES ioctl that refers to it.
Would that be preferable?
Quoted 1 line
Other instantiation attributes we don't know that early on should be settable between the time frame of vm creation and first execution. An example for this are device addresses. Check out the threads "KVM: ARM: Rename
KVM_SET_DEVICE_ADDRESS" and "KVM: ARM: IntroduceKVM_SET_DEVICE_ADDRESSioctl" for more information on this particular bit.
That ioctl looks like it should be generic, and would be useful for MPIC (OpenPIC) emulation. I don't need it for PAPR since the interrupt controllers are accessed through hcalls, not MMIO.
Paul.
Quoted 16 lines · Paul Mackerras
Quoted 4 lines · Alexander Graf
Quoted 1 line · Paul Mackerras
On Sat, Dec 15, 2012 at 01:06:13PM +1100, Benjamin Herrenschmidt wrote:
[snip]
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
- Move the existing "generic" KVM ioctl to create the kernel APIC to x86 since it's no as-is useful for other archs who, among other things, might need to pass argument based on the machine type (type of interrupt subsystem for example, etc...)
Assuming you're talking about
KVM_CREATE_IRQCHIP, it is already handled entirely in arch code (arch/x86/kvm/x86.candarch/ia64/kvm/kvm-ia64.c), along withKVM_GET_IRQCHIPandKVM_SET_IRQCHIP.
Quoted 4 lines
Quoted 1 line
This part is about QEMU.
OK, that makes sense, but it presumably can't be done until the new kernel interface exists, right?
Well, moving an x86 specific call into x86 specific code should be possible without kernel changes, no? :)
I'm not sure how useful it is without having a new interface though.
Quoted 27 lines
Quoted 27 lines
Quoted 26 lines
Quoted 8 lines
- Once that's done, well, instanciating interrupt controller objects becomes pretty much an arch specific matter. We could try to keep some ioctls somewhat common though it's not *that* useful because the various types & arguments are going to be fairly arch specific, so goes for configuration. Examples of what could be kept common are:
Quoted 4 lines
* Create irq chip, takes at least a "type" argument, possibly a few
more type-specific args (or a union, but then let's keep space in there since we can't change the size of the struct later as it would impact the ioctl number afaik).The existing
KVM_CREATE_IRQCHIPis defined as_IO(...)which means that it doesn't read or write memory, but there is still the 3rd argument to the ioctl, which would give us 64 bits to indicate the type of the top-level IRQ controller (XICS, MPIC, ...), but not much more.It sounds like the agreement at the meeting was more along the lines of the
KVM_CREATE_IRQCHIP_ARGSioctl (as in the patches I posted) which can be called multiple times to instantiate pieces of the interrupt framework, rather than having aKVM_CREATE_IRQCHIPthat gets called once early on to say that there we are using in-kernel interrupt controller emulation, followed by other calls to configure the various parts of the framework. Is that accurate?
Quoted 5 lines
Quoted 1 line
KVM_CREATE_IRQCHIPshould get a parameter indicating the type of IRQCHIP (XICS / MPIC / APIC / VGIC / ...). The parameter-less version defaults to an arch specific default (APIC for x86, VGIC for arm, error on PPC).So, I propose
KVM_CREATE_IRQCHIP_ARGSas the version of the ioctl that indicates the type of irqchip. The parameterless version already gives an error on PPC.
Works for me. I needs to get the type of irqchip as argument, maybe some more (like MPIC version, though that _could_ be handled through a different type too)
Quoted 14 lines
Quoted 2 lines
I'm not 100% sure yet whether we want to support models with multiple IRQCHIPs. If so, we need to return a device token from the
KVM_CREATE_IRQCHIPioctl. Maybe it makes more sense to model specific cases like this as separate type though with specific, explicit subdevice ids. Not sure.I think it depends on what you mean by IRQCHIP. If you mean the overall interrupt controller architecture, then no it doesn't make sense to have more than one. The overall architecture might well contain multiple identifiable pieces which could be considered IRQCHIPs. This is the case on x86 already, where they have two i8259-style PICs and one or more IOAPICs (and presumably emulated LAPICs as well). It's the case for the PAPR-style XICS controller, where we have a presentation controller (ICP) per CPU (like the x86 LAPIC), plus one or more source controllers (ICS) (like the x86 IOAPIC).
Quoted 1 line
Which meaning of IRQCHIP do you prefer?
Yeah, I'm always getting confused on all this too. We have 2 things we talk about here
* overall architecture
* individual devices hanging off of that architecture
We need some link from "architecture" to "devices" because we need to be able to set MMIO offsets for the MPIC for example. On x86 and ARM today, this is implicit. The device IDs simply exist when we create a certain type of "system wide IRQCHIP".
I'm not sure if that's a good idea or not. At least it needs to be properly documented and the linkage between "system wide type is X" -> "devices 1,2,3 exist" has to be clear.
Quoted 10 lines
My series of patches uses
KVM_CREATE_IRQCHIP_ARGSto indicate that the overall architecture is XICS (by creating the ICPs) and to create the source controllers (ICS), so it gets called multiple times. It would be possible to haveKVM_CREATE_IRQCHIP_ARGScalled just once with a parameter saying "XICS", and then create the source controllers implicitly as a side effect of the firstKVM_IRQ_SET_SOURCESioctl that refers to it.Would that be preferable?
I like the latter better, yes. In fact, instead of KVM_IRQ_SET_SOURCES you could use something like CREATE_DEVICE and create a source controller device, no? I wonder if that'd be any improvement over a SET_SOURCES ioctl. Hrm.
Quoted 5 lines
Quoted 1 line
Other instantiation attributes we don't know that early on should be settable between the time frame of vm creation and first execution. An example for this are device addresses. Check out the threads "KVM: ARM: Rename
KVM_SET_DEVICE_ADDRESS" and "KVM: ARM: IntroduceKVM_SET_DEVICE_ADDRESSioctl" for more information on this particular bit.That ioctl looks like it should be generic, and would be useful for MPIC (OpenPIC) emulation. I don't need it for PAPR since the interrupt controllers are accessed through hcalls, not MMIO.
Yes. And it will be. We need it for in-kernel MPIC emulation too.
Alex