KVM: iommu: Overhaul device posted IRQs support
Sean Christopherson
maintainerv2 cover KVM: iommu: Overhaul device posted IRQs support
TL;DR: Overhaul device posted interrupts in KVM and IOMMU, and AVIC in
general.
This applies on the series to add CONFIG_KVM_IOAPIC (and to kill irq_comm.c):
https://lore.kernel.org/all/[email protected]
Fix a variety of bugs related to device posted IRQs, especially on the AMD side, and clean up KVM's implementation (this series actually removes more code than it adds).
Stating the obvious, this series is comically large. Though it's smaller than v1! (Ignoring that I cheated by moving 15 patches to a prep series, and that Paolo already grabbed several patches).
Sairaj, I applied your Tested-by somewhat sparingly, as some of the patches changed (most notably "Consolidate IRTE update when toggling AVIC on/off"). Please holler if you want me to remove/add any tags. And when you get time, I'd greatly appreciate a sanity check!
Batch #1 is mostly SVM specific:
- Cleans up various warts and bugs in the IRTE tracking
- Fixes AVIC to not reject large VMs (honor KVM's ABI)
- Wire up AVIC to enable_ipiv to support disabling IPI virtualization while still utilizing device posted interrupts, and to workaround erratum #1235.
Batch #3 overhauls the guts of IRQ bypass in KVM, and moves the vast majority of the logic to common x86; only the code that needs to communicate with the IOMMU is truly vendor specific.
Batch #4 is more SVM/AVIC cleanups that are made possible by batch #3.
Batch #5 adds WARNs and drops dead code after all the previous cleanups and fixes (I don't want to add the WARNs earlier; I don't see any point in adding WARNs in code that's known to be broken).
Batch #6 is yet more SVM/AVIC cleanups, with the specific goal of configuring IRTEs to generate GA log interrupts if and only if KVM actually needs a wake event.
v2:
- Drop patches that were already merged.
- Move code into irq.c, not x86.c. [Paolo]
- Collect review/testing tags. [Sairaj, Vasant]
- Sqaush fixup for a comment that was added in the prior patch. [Sairaj]
- Rewrote the changelog for "Delete IRTE link from previous vCPU irrespective of new routing". [Sairaj]
- Actually drop "struct amd_svm_iommu_ir" and all usage in "Track per-vCPU IRTEs using kvm_kernel_irqfd structure" (the previous version was getting hilarious lucky with struct offsets). [Sairaj]
- Drop unused params from kvm_pi_update_irte() and pi_update_irte(). [Sairaj]
- Document the rules and behavior of amd_iommu_update_ga(). [Joerg]
- Fix a changelog typo. [Paolo]
- Document that GALogIntr isn't cached, i.e. can be safely updated without an invalidation. [Joao, Vasant]
- Rework avic_vcpu_{load,put}() to use an enumerated parameter instead of a series of booleans. [Paolo]
- Drop a redundant "&& new". [Francesco]
- Drop the *** DO NOT MERGE *** testing hack patches.
v1: https://lore.kernel.org/all/[email protected]
Maxim Levitsky (2):
KVM: SVM: Add enable_ipiv param, never set IsRunning if disabled
KVM: SVM: Disable (x2)AVIC IPI virtualization if CPU has erratum #1235
Sean Christopherson (57):
KVM: x86: Pass new routing entries and irqfd when updating IRTEs
KVM: SVM: Track per-vCPU IRTEs using kvm_kernel_irqfd structure
KVM: SVM: Delete IRTE link from previous vCPU before setting new IRTE iommu/amd: KVM: SVM: Delete now-unused cached/previous GA tag fields
KVM: SVM: Delete IRTE link from previous vCPU irrespective of new
routing
KVM: SVM: Drop pointless masking of default APIC base when setting
V_APIC_BAR
KVM: SVM: Drop pointless masking of kernel page pa's with AVIC HPA
masks
KVM: SVM: Add helper to deduplicate code for getting AVIC backing page
KVM: SVM: Drop vcpu_svm's pointless avic_backing_page field
KVM: SVM: Inhibit AVIC if ID is too big instead of rejecting vCPU
creation
KVM: SVM: Drop redundant check in AVIC code on ID during vCPU creation
KVM: SVM: Track AVIC tables as natively sized pointers, not "struct
pages"
KVM: SVM: Drop superfluous "cache" of AVIC Physical ID entry pointer
KVM: VMX: Move enable_ipiv knob to common x86
KVM: VMX: Suppress PI notifications whenever the vCPU is put
KVM: SVM: Add a comment to explain why avic_vcpu_blocking() ignores
IRQ blocking
iommu/amd: KVM: SVM: Use pi_desc_addr to derive ga_root_ptr iommu/amd: KVM: SVM: Pass NULL @vcpu_info to indicate "not guest mode"
KVM: SVM: Stop walking list of routing table entries when updating
IRTE
KVM: VMX: Stop walking list of routing table entries when updating
IRTE
KVM: SVM: Extract SVM specific code out of get_pi_vcpu_info()
KVM: x86: Move IRQ routing/delivery APIs from x86.c => irq.c
KVM: x86: Nullify irqfd->producer after updating IRTEs
KVM: x86: Dedup AVIC vs. PI code for identifying target vCPU
KVM: x86: Move posted interrupt tracepoint to common code
KVM: SVM: Clean up return handling in avic_pi_update_irte()
iommu: KVM: Split "struct vcpu_data" into separate AMD vs. Intel
structs
KVM: Don't WARN if updating IRQ bypass route fails
KVM: Fold kvm_arch_irqfd_route_changed() into
kvm_arch_update_irqfd_routing()
KVM: x86: Track irq_bypass_vcpu in common x86 code
KVM: x86: Skip IOMMU IRTE updates if there's no old or new vCPU being
targeted
KVM: x86: Don't update IRTE entries when old and new routes were !MSI
KVM: SVM: Revert IRTE to legacy mode if IOMMU doesn't provide IR
metadata
KVM: SVM: Take and hold ir_list_lock across IRTE updates in IOMMU iommu/amd: Document which IRTE fields amd_iommu_update_ga() can modify iommu/amd: KVM: SVM: Infer IsRun from validity of pCPU destination iommu/amd: Factor out helper for manipulating IRTE GA/CPU info iommu/amd: KVM: SVM: Set pCPU info in IRTE when setting vCPU affinity iommu/amd: KVM: SVM: Add IRTE metadata to affined vCPU's list if AVIC
is inhibited
KVM: SVM: Don't check for assigned device(s) when updating affinity
KVM: SVM: Don't check for assigned device(s) when activating AVIC
KVM: SVM: WARN if (de)activating guest mode in IOMMU fails
KVM: SVM: Process all IRTEs on affinity change even if one update
fails
KVM: SVM: WARN if updating IRTE GA fields in IOMMU fails
KVM: x86: Drop superfluous "has assigned device" check in
kvm_pi_update_irte()
KVM: x86: WARN if IRQ bypass isn't supported in kvm_pi_update_irte()
KVM: x86: WARN if IRQ bypass routing is updated without in-kernel
local APIC
KVM: SVM: WARN if ir_list is non-empty at vCPU free
KVM: x86: Decouple device assignment from IRQ bypass
KVM: VMX: WARN if VT-d Posted IRQs aren't possible when starting IRQ
bypass
KVM: SVM: Use vcpu_idx, not vcpu_id, for GA log tag/metadata iommu/amd: WARN if KVM calls GA IRTE helpers without virtual APIC
support
KVM: SVM: Fold avic_set_pi_irte_mode() into its sole caller
KVM: SVM: Don't check vCPU's blocking status when toggling AVIC on/off
KVM: SVM: Consolidate IRTE update when toggling AVIC on/off iommu/amd: KVM: SVM: Allow KVM to control need for GA log interrupts
KVM: SVM: Generate GA log IRQs only if the associated vCPUs is
blocking
arch/x86/include/asm/irq_remapping.h | 17 +- arch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 20 +- arch/x86/include/asm/svm.h | 13 +- arch/x86/kvm/irq.c | 140 ++++++ arch/x86/kvm/svm/avic.c | 702 ++++++++++++--------------- arch/x86/kvm/svm/svm.c | 4 + arch/x86/kvm/svm/svm.h | 32 +- arch/x86/kvm/trace.h | 19 +- arch/x86/kvm/vmx/capabilities.h | 1 - arch/x86/kvm/vmx/main.c | 2 +- arch/x86/kvm/vmx/posted_intr.c | 140 ++---- arch/x86/kvm/vmx/posted_intr.h | 10 +- arch/x86/kvm/vmx/vmx.c | 2 - arch/x86/kvm/x86.c | 90 +--- drivers/iommu/amd/amd_iommu_types.h | 1 - drivers/iommu/amd/iommu.c | 125 +++-- drivers/iommu/intel/irq_remapping.c | 10 +- include/linux/amd-iommu.h | 25 +- include/linux/kvm_host.h | 9 +- include/linux/kvm_irqfd.h | 4 + virt/kvm/eventfd.c | 22 +- 22 files changed, 672 insertions(+), 718 deletions(-)
base-commit: 3debd5461fba1dcb33e732b16153da0cf5d0c251
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 1/59 KVM: x86: Pass new routing entries and irqfd when updating IRTEs
When updating IRTEs in response to a GSI routing or IRQ bypass change, pass the new/current routing information along with the associated irqfd. This will allow KVM x86 to harden, simplify, and deduplicate its code.
Since adding/removing a bypass producer is now conveniently protected with irqfds.lock, i.e. can't run concurrently with kvm_irq_routing_update(), use the routing information cached in the irqfd instead of looking up the information in the current GSI routing tables.
Opportunistically convert an existing printk() to pr_info() and put its string onto a single line (old code that strictly adhered to 80 chars).
--- arch/x86/include/asm/kvm_host.h | 6 ++++-- arch/x86/kvm/svm/avic.c | 18 +++++++---------- arch/x86/kvm/svm/svm.h | 5 +++-- arch/x86/kvm/vmx/posted_intr.c | 19 ++++++++--------- arch/x86/kvm/vmx/posted_intr.h | 8 ++++++-- arch/x86/kvm/x86.c | 36 ++++++++++++++++++--------------- include/linux/kvm_host.h | 7 +++++-- virt/kvm/eventfd.c | 11 +++++----- 8 files changed, 58 insertions(+), 52 deletions(-)
arch/x86/include/asm/kvm_host.h +4 -2
@@ -296,6 +296,7 @@ enum x86_intercept_stage;*/ #define KVM_APIC_PV_EOI_PENDING 1 +struct kvm_kernel_irqfd ;struct kvm_kernel_irq_routing_entry ;/*
@@ -1844,8 +1845,9 @@ struct kvm_x86_ops {void (*vcpu_blocking)(struct kvm_vcpu *vcpu);void (*vcpu_unblocking)(struct kvm_vcpu *vcpu); -int (*pi_update_irte)(struct kvm *kvm,unsigned int host_irq, -uint32_t guest_irq,bool set); +int (*pi_update_irte)(struct kvm_kernel_irqfd *irqfd,struct kvm *kvm, +unsigned int host_irq,uint32_t guest_irq, +struct kvm_kernel_irq_routing_entry *new);void (*pi_start_assignment)(struct kvm *kvm);void (*apicv_pre_state_restore)(struct kvm_vcpu *vcpu);void (*apicv_post_state_restore)(struct kvm_vcpu *vcpu);
arch/x86/kvm/svm/avic.c +7 -11
@@ -18,6 +18,7 @@#include <linux/hashtable.h> #include <linux/amd-iommu.h> #include <linux/kvm_host.h> +#include <linux/kvm_irqfd.h> #include <asm/irq_remapping.h>
@@ -885,21 +886,14 @@ get_pi_vcpu_info(struct kvm *kvm, struct kvm_kernel_irq_routing_entry *e,return 0 ; } -/* -* avic_pi_update_irte - set IRTE for Posted-Interrupts -* -* @kvm: kvm -* @host_irq: host irq of the interrupt -* @guest_irq: gsi of the interrupt -* @set: set or unset PI -* returns 0 on success, < 0 on failure -*/ -int avic_pi_update_irte (struct kvm *kvm,unsigned int host_irq, -uint32_t guest_irq,bool set) +int avic_pi_update_irte (struct kvm_kernel_irqfd *irqfd,struct kvm *kvm, +unsigned int host_irq,uint32_t guest_irq, +struct kvm_kernel_irq_routing_entry *new) {struct kvm_kernel_irq_routing_entry *e;struct kvm_irq_routing_table *irq_rt;bool enable_remapped_mode = true; +bool set = !!new;int idx, ret =0 ;if (!kvm_arch_has_assigned_device (kvm) || !kvm_arch_has_irq_bypass ())
@@ -925,6 +919,8 @@ int avic_pi_update_irte(struct kvm *kvm, unsigned int host_irq,if (e->type !=KVM_IRQ_ROUTING_MSI )continue ; +WARN_ON_ONCE (new &&memcmp (e, new,sizeof (*new))); +/** * Here, we setup with legacy mode in the following cases: * 1. When cannot target interrupt to a specific vcpu.
arch/x86/kvm/svm/svm.h +3 -2
@@ -736,8 +736,9 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu);void avic_vcpu_put (struct kvm_vcpu *vcpu);void avic_apicv_post_state_restore (struct kvm_vcpu *vcpu);void avic_refresh_apicv_exec_ctrl (struct kvm_vcpu *vcpu); -int avic_pi_update_irte (struct kvm *kvm,unsigned int host_irq, -uint32_t guest_irq,bool set); +int avic_pi_update_irte (struct kvm_kernel_irqfd *irqfd,struct kvm *kvm, +unsigned int host_irq,uint32_t guest_irq, +struct kvm_kernel_irq_routing_entry *new);void avic_vcpu_blocking (struct kvm_vcpu *vcpu);void avic_vcpu_unblocking (struct kvm_vcpu *vcpu);void avic_ring_doorbell (struct kvm_vcpu *vcpu);
arch/x86/kvm/vmx/posted_intr.c +8 -11
@@ -2,6 +2,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <linux/kvm_host.h> +#include <linux/kvm_irqfd.h> #include <asm/irq_remapping.h> #include <asm/cpu.h>
@@ -294,17 +295,9 @@ void vmx_pi_start_assignment(struct kvm *kvm)kvm_make_all_cpus_request (kvm,KVM_REQ_UNBLOCK ); } -/* -* vmx_pi_update_irte - set IRTE for Posted-Interrupts -* -* @kvm: kvm -* @host_irq: host irq of the interrupt -* @guest_irq: gsi of the interrupt -* @set: set or unset PI -* returns 0 on success, < 0 on failure -*/ -int vmx_pi_update_irte (struct kvm *kvm,unsigned int host_irq, -uint32_t guest_irq,bool set) +int vmx_pi_update_irte (struct kvm_kernel_irqfd *irqfd,struct kvm *kvm, +unsigned int host_irq,uint32_t guest_irq, +struct kvm_kernel_irq_routing_entry *new) {struct kvm_kernel_irq_routing_entry *e;struct kvm_irq_routing_table *irq_rt;
@@ -312,6 +305,7 @@ int vmx_pi_update_irte(struct kvm *kvm, unsigned int host_irq,struct kvm_lapic_irq irq;struct kvm_vcpu *vcpu;struct vcpu_data vcpu_info; +bool set = !!new;int idx, ret =0 ;if (!vmx_can_use_vtd_pi (kvm))
@@ -329,6 +323,9 @@ int vmx_pi_update_irte(struct kvm *kvm, unsigned int host_irq,hlist_for_each_entry (e, &irq_rt->map[guest_irq], link) {if (e->type !=KVM_IRQ_ROUTING_MSI )continue ; + +WARN_ON_ONCE (new &&memcmp (e, new,sizeof (*new))); +/* * VT-d PI cannot support posting multicast/broadcast * interrupts to a vCPU, we still use interrupt remapping
arch/x86/kvm/vmx/posted_intr.h +6 -2
@@ -3,6 +3,9 @@#define __KVM_X86_VMX_POSTED_INTR_H#include <linux/bitmap.h> +#include <linux/find.h> +#include <linux/kvm_host.h> +#include <asm/posted_intr.h> void vmx_vcpu_pi_load (struct kvm_vcpu *vcpu,int cpu);
@@ -11,8 +14,9 @@ void pi_wakeup_handler(void);void __init pi_init_cpu (int cpu);void pi_apicv_pre_state_restore (struct kvm_vcpu *vcpu);bool pi_has_pending_interrupt (struct kvm_vcpu *vcpu); -int vmx_pi_update_irte (struct kvm *kvm,unsigned int host_irq, -uint32_t guest_irq,bool set); +int vmx_pi_update_irte (struct kvm_kernel_irqfd *irqfd,struct kvm *kvm, +unsigned int host_irq,uint32_t guest_irq, +struct kvm_kernel_irq_routing_entry *new);void vmx_pi_start_assignment (struct kvm *kvm);static inline int pi_find_highest_vector (struct pi_desc *pi_desc)
arch/x86/kvm/x86.c +20 -16
@@ -13630,31 +13630,31 @@ int kvm_arch_irq_bypass_add_producer(struct irq_bypass_consumer *cons,struct kvm_kernel_irqfd *irqfd = container_of(cons, struct kvm_kernel_irqfd, consumer);struct kvm *kvm = irqfd->kvm; -int ret; +int ret =0 ;kvm_arch_start_assignment (irqfd->kvm);spin_lock_irq (&kvm->irqfds.lock); irqfd->producer = prod; - ret =kvm_x86_call (pi_update_irte)(irqfd->kvm, - prod->irq, irqfd->gsi,1 ); -if (ret) -kvm_arch_end_assignment (irqfd->kvm); - +if (irqfd->irq_entry.type ==KVM_IRQ_ROUTING_MSI ) { + ret =kvm_x86_call (pi_update_irte)(irqfd, irqfd->kvm, prod->irq, + irqfd->gsi, &irqfd->irq_entry); +if (ret) +kvm_arch_end_assignment (irqfd->kvm); + }spin_unlock_irq (&kvm->irqfds.lock); -return ret; }void kvm_arch_irq_bypass_del_producer (struct irq_bypass_consumer *cons,struct irq_bypass_producer *prod) { -int ret;struct kvm_kernel_irqfd *irqfd = container_of(cons, struct kvm_kernel_irqfd, consumer);struct kvm *kvm = irqfd->kvm; +int ret;WARN_ON (irqfd->producer != prod);
@@ -13667,11 +13667,13 @@ void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *cons,spin_lock_irq (&kvm->irqfds.lock); irqfd->producer =NULL ; - ret =kvm_x86_call (pi_update_irte)(irqfd->kvm, - prod->irq, irqfd->gsi,0 ); -if (ret) -printk (KERN_INFO "irq bypass consumer (token %p) unregistration" -" fails: %d\n" , irqfd->consumer.token, ret); +if (irqfd->irq_entry.type ==KVM_IRQ_ROUTING_MSI ) { + ret =kvm_x86_call (pi_update_irte)(irqfd, irqfd->kvm, prod->irq, + irqfd->gsi,NULL ); +if (ret) +pr_info ("irq bypass consumer (token %p) unregistration fails: %d\n" , + irqfd->consumer.token, ret); + }spin_unlock_irq (&kvm->irqfds.lock);
@@ -13679,10 +13681,12 @@ void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *cons,kvm_arch_end_assignment (irqfd->kvm); } -int kvm_arch_update_irqfd_routing (struct kvm *kvm,unsigned int host_irq, -uint32_t guest_irq,bool set) +int kvm_arch_update_irqfd_routing (struct kvm_kernel_irqfd *irqfd, +struct kvm_kernel_irq_routing_entry *old, +struct kvm_kernel_irq_routing_entry *new) { -return kvm_x86_call (pi_update_irte)(kvm, host_irq, guest_irq, set); +return kvm_x86_call (pi_update_irte)(irqfd, irqfd->kvm, irqfd->producer->irq, + irqfd->gsi, new); } bool kvm_arch_irqfd_route_changed(struct kvm_kernel_irq_routing_entry *old,
include/linux/kvm_host.h +5 -2
@@ -2397,6 +2397,8 @@ struct kvm_vcpu *kvm_get_running_vcpu(void);struct kvm_vcpu *__percpu *kvm_get_running_vcpus (void );#if IS_ENABLED (CONFIG_HAVE_KVM_IRQ_BYPASS ) +struct kvm_kernel_irqfd ; +bool kvm_arch_has_irq_bypass (void );int kvm_arch_irq_bypass_add_producer (struct irq_bypass_consumer *,struct irq_bypass_producer *);
@@ -2404,8 +2406,9 @@ void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *, struct irq_bypass_producer *);void kvm_arch_irq_bypass_stop (struct irq_bypass_consumer *);void kvm_arch_irq_bypass_start (struct irq_bypass_consumer *); -int kvm_arch_update_irqfd_routing (struct kvm *kvm,unsigned int host_irq, -uint32_t guest_irq,bool set); +int kvm_arch_update_irqfd_routing (struct kvm_kernel_irqfd *irqfd, +struct kvm_kernel_irq_routing_entry *old, +struct kvm_kernel_irq_routing_entry *new);bool kvm_arch_irqfd_route_changed (struct kvm_kernel_irq_routing_entry *,struct kvm_kernel_irq_routing_entry *);#endif /* CONFIG_HAVE_KVM_IRQ_BYPASS */
virt/kvm/eventfd.c +5 -6
@@ -285,9 +285,9 @@ void __attribute__((weak)) kvm_arch_irq_bypass_start( { } -int __attribute__((weak))kvm_arch_update_irqfd_routing ( -struct kvm *kvm,unsigned int host_irq, -uint32_t guest_irq,bool set) +int __weak kvm_arch_update_irqfd_routing (struct kvm_kernel_irqfd *irqfd, +struct kvm_kernel_irq_routing_entry *old, +struct kvm_kernel_irq_routing_entry *new) {return 0 ; }
@@ -619,9 +619,8 @@ void kvm_irq_routing_update(struct kvm *kvm)#if IS_ENABLED (CONFIG_HAVE_KVM_IRQ_BYPASS )if (irqfd->producer &&kvm_arch_irqfd_route_changed (&old, &irqfd->irq_entry)) { -int ret =kvm_arch_update_irqfd_routing ( - irqfd->kvm, irqfd->producer->irq, - irqfd->gsi,1 ); +int ret =kvm_arch_update_irqfd_routing (irqfd, &old, &irqfd->irq_entry); +WARN_ON (ret); }#endif
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 2/59 KVM: SVM: Track per-vCPU IRTEs using kvm_kernel_irqfd structure
In reply to Sean ChristophersonTrack the IRTEs that are posting to an SVM vCPU via the associated irqfd structure and GSI routing instead of dynamically allocating a separate data structure. In addition to eliminating an atomic allocation, this will allow hoisting much of the IRTE update logic to common x86.
--- arch/x86/kvm/svm/avic.c | 71 +++++++++++++++------------------------ arch/x86/kvm/svm/svm.h | 10 +++--- include/linux/kvm_irqfd.h | 3 ++ 3 files changed, 36 insertions(+), 48 deletions(-)
arch/x86/kvm/svm/avic.c +27 -44
@@ -75,14 +75,6 @@ static bool next_vm_id_wrapped = 0;static DEFINE_SPINLOCK (svm_vm_data_hash_lock);bool x2avic_enabled; -/* -* This is a wrapper of struct amd_iommu_ir_data. -*/ -struct amd_svm_iommu_ir { -struct list_head node;/* Used by SVM for per-vcpu ir_list */ -void *data;/* Storing pointer to struct amd_ir_data */ -}; -static void avic_activate_vmcb (struct vcpu_svm *svm) {struct vmcb *vmcb = svm->vmcb01.ptr;
@@ -746,8 +738,8 @@ static int avic_set_pi_irte_mode(struct kvm_vcpu *vcpu, bool activate) {int ret =0 ;unsigned long flags; -struct amd_svm_iommu_ir *ir;struct vcpu_svm *svm =to_svm (vcpu); +struct kvm_kernel_irqfd *irqfd;if (!kvm_arch_has_assigned_device (vcpu->kvm))return 0 ;
@@ -761,11 +753,11 @@ static int avic_set_pi_irte_mode(struct kvm_vcpu *vcpu, bool activate)if (list_empty (&svm->ir_list)) goto out; -list_for_each_entry (ir, &svm->ir_list, node) { +list_for_each_entry (irqfd, &svm->ir_list, vcpu_list) {if (activate) - ret =amd_iommu_activate_guest_mode (ir->data); + ret =amd_iommu_activate_guest_mode (irqfd->irq_bypass_data);else - ret =amd_iommu_deactivate_guest_mode (ir->data); + ret =amd_iommu_deactivate_guest_mode (irqfd->irq_bypass_data);if (ret)break ; }
@@ -774,27 +766,30 @@ static int avic_set_pi_irte_mode(struct kvm_vcpu *vcpu, bool activate)return ret; } -static void svm_ir_list_del (struct vcpu_svm *svm,struct amd_iommu_pi_data *pi) +static void svm_ir_list_del (struct vcpu_svm *svm, +struct kvm_kernel_irqfd *irqfd, +struct amd_iommu_pi_data *pi) {unsigned long flags; -struct amd_svm_iommu_ir *cur; +struct kvm_kernel_irqfd *cur;spin_lock_irqsave (&svm->ir_list_lock, flags); -list_for_each_entry (cur, &svm->ir_list, node) { -if (cur->data != pi->ir_data) +list_for_each_entry (cur, &svm->ir_list, vcpu_list) { +if (cur->irq_bypass_data != pi->ir_data)continue ; -list_del (&cur->node); -kfree (cur); +if (WARN_ON_ONCE (cur != irqfd)) +continue ; +list_del (&irqfd->vcpu_list);break ; }spin_unlock_irqrestore (&svm->ir_list_lock, flags); } -static int svm_ir_list_add (struct vcpu_svm *svm,struct amd_iommu_pi_data *pi) +static int svm_ir_list_add (struct vcpu_svm *svm, +struct kvm_kernel_irqfd *irqfd, +struct amd_iommu_pi_data *pi) { -int ret =0 ;unsigned long flags; -struct amd_svm_iommu_ir *ir;u64 entry;if (WARN_ON_ONCE (!pi->ir_data))
@@ -811,25 +806,14 @@ static int svm_ir_list_add(struct vcpu_svm *svm, struct amd_iommu_pi_data *pi)struct kvm_vcpu *prev_vcpu =kvm_get_vcpu_by_id (kvm, vcpu_id);struct vcpu_svm *prev_svm; -if (!prev_vcpu) { - ret = -EINVAL ; - goto out; - } +if (!prev_vcpu) +return -EINVAL ; prev_svm =to_svm (prev_vcpu); -svm_ir_list_del (prev_svm, pi); +svm_ir_list_del (prev_svm, irqfd, pi); } -/** -* Allocating new amd_iommu_pi_data, which will get -* add to the per-vcpu ir_list. -*/ - ir =kzalloc (sizeof (struct amd_svm_iommu_ir ),GFP_ATOMIC | __GFP_ACCOUNT); -if (!ir) { - ret = -ENOMEM ; - goto out; - } - ir->data = pi->ir_data; + irqfd->irq_bypass_data = pi->ir_data;spin_lock_irqsave (&svm->ir_list_lock, flags);
@@ -844,10 +828,9 @@ static int svm_ir_list_add(struct vcpu_svm *svm, struct amd_iommu_pi_data *pi)amd_iommu_update_ga (entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK , true, pi->ir_data); -list_add (&ir->node, &svm->ir_list); +list_add (&irqfd->vcpu_list, &svm->ir_list);spin_unlock_irqrestore (&svm->ir_list_lock, flags); -out: -return ret; +return 0 ; }/*
@@ -951,7 +934,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,* scheduling information in IOMMU irte. */ if (!ret && pi.is_guest_mode) -svm_ir_list_add (svm, &pi); +svm_ir_list_add (svm, irqfd, &pi); }if (!ret && svm) {
@@ -992,7 +975,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, vcpu =kvm_get_vcpu_by_id (kvm, id);if (vcpu) -svm_ir_list_del (to_svm (vcpu), &pi); +svm_ir_list_del (to_svm (vcpu), irqfd, &pi); } } out:
@@ -1004,8 +987,8 @@ static inline int avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu, bool r) { int ret = 0; -struct amd_svm_iommu_ir *ir;struct vcpu_svm *svm =to_svm (vcpu); +struct kvm_kernel_irqfd *irqfd;lockdep_assert_held (&svm->ir_list_lock);
@@ -1019,8 +1002,8 @@ avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu, bool r)if (list_empty (&svm->ir_list))return 0 ; -list_for_each_entry (ir, &svm->ir_list, node) { - ret =amd_iommu_update_ga (cpu, r, ir->data); +list_for_each_entry (irqfd, &svm->ir_list, vcpu_list) { + ret =amd_iommu_update_ga (cpu, r, irqfd->irq_bypass_data);if (ret)return ret; }
arch/x86/kvm/svm/svm.h +6 -4
@@ -310,10 +310,12 @@ struct vcpu_svm {u64 *avic_physical_id_cache;/* -* Per-vcpu list of struct amd_svm_iommu_ir: -* This is used mainly to store interrupt remapping information used -* when update the vcpu affinity. This avoids the need to scan for -* IRTE and try to match ga_tag in the IOMMU driver. +* Per-vCPU list of irqfds that are eligible to post IRQs directly to +* the vCPU (a.k.a. device posted IRQs, a.k.a. IRQ bypass). The list +* is used to reconfigure IRTEs when the vCPU is loaded/put (to set the +* target pCPU), when AVIC is toggled on/off (to (de)activate bypass), +* and if the irqfd becomes ineligible for posting (to put the IRTE +* back into remapped mode). */ struct list_head ir_list;spinlock_t ir_list_lock;
include/linux/kvm_irqfd.h +3 -0
@@ -59,6 +59,9 @@ struct kvm_kernel_irqfd {struct work_struct shutdown;struct irq_bypass_consumer consumer;struct irq_bypass_producer *producer; + +struct list_head vcpu_list; + void *irq_bypass_data; };#endif /* __LINUX_KVM_IRQFD_H */
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 3/59 KVM: SVM: Delete IRTE link from previous vCPU before setting new IRTE
In reply to Sean ChristophersonDelete the previous per-vCPU IRTE link prior to modifying the IRTE. If forcing the IRTE back to remapped mode fails, the IRQ is already broken; keeping stale metadata won't change that, and the IOMMU should be sufficiently paranoid to sanitize the IRTE when the IRQ is freed and reallocated.
This will allow hoisting the vCPU tracking to common x86, which in turn will allow most of the IRTE update code to be deduplicated.
--- arch/x86/kvm/svm/avic.c | 60 +++++++++------------------------------ include/linux/kvm_irqfd.h | 1 + 2 files changed, 14 insertions(+), 47 deletions(-)
arch/x86/kvm/svm/avic.c +13 -47
@@ -766,23 +766,19 @@ static int avic_set_pi_irte_mode(struct kvm_vcpu *vcpu, bool activate)return ret; } -static void svm_ir_list_del (struct vcpu_svm *svm, -struct kvm_kernel_irqfd *irqfd, -struct amd_iommu_pi_data *pi) +static void svm_ir_list_del (struct kvm_kernel_irqfd *irqfd) { +struct kvm_vcpu *vcpu = irqfd->irq_bypass_vcpu;unsigned long flags; -struct kvm_kernel_irqfd *cur; -spin_lock_irqsave (&svm->ir_list_lock, flags); -list_for_each_entry (cur, &svm->ir_list, vcpu_list) { -if (cur->irq_bypass_data != pi->ir_data) -continue ; -if (WARN_ON_ONCE (cur != irqfd)) -continue ; -list_del (&irqfd->vcpu_list); -break ; - } -spin_unlock_irqrestore (&svm->ir_list_lock, flags); +if (!vcpu) +return ; + +spin_lock_irqsave (&to_svm (vcpu)->ir_list_lock, flags); +list_del (&irqfd->vcpu_list); +spin_unlock_irqrestore (&to_svm (vcpu)->ir_list_lock, flags); + + irqfd->irq_bypass_vcpu =NULL ; } static int svm_ir_list_add(struct vcpu_svm *svm,
@@ -795,24 +791,7 @@ static int svm_ir_list_add(struct vcpu_svm *svm,if (WARN_ON_ONCE (!pi->ir_data))return -EINVAL ; -/** -* In some cases, the existing irte is updated and re-set, -* so we need to check here if it's already been * added -* to the ir_list. -*/ -if (pi->prev_ga_tag) { -struct kvm *kvm = svm->vcpu.kvm; -u32 vcpu_id =AVIC_GATAG_TO_VCPUID (pi->prev_ga_tag); -struct kvm_vcpu *prev_vcpu =kvm_get_vcpu_by_id (kvm, vcpu_id); -struct vcpu_svm *prev_svm; - -if (!prev_vcpu) -return -EINVAL ; - - prev_svm =to_svm (prev_vcpu); -svm_ir_list_del (prev_svm, irqfd, pi); - } - + irqfd->irq_bypass_vcpu = &svm->vcpu; irqfd->irq_bypass_data = pi->ir_data;spin_lock_irqsave (&svm->ir_list_lock, flags);
@@ -904,6 +883,8 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,WARN_ON_ONCE (new &&memcmp (e, new,sizeof (*new))); +svm_ir_list_del (irqfd); +/** * Here, we setup with legacy mode in the following cases: * 1. When cannot target interrupt to a specific vcpu.
@@ -962,21 +943,6 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, pi.prev_ga_tag =0 ; pi.is_guest_mode = false; ret =irq_set_vcpu_affinity (host_irq, &pi); - -/** -* Check if the posted interrupt was previously -* setup with the guest_mode by checking if the ga_tag -* was cached. If so, we need to clean up the per-vcpu -* ir_list. -*/ -if (!ret && pi.prev_ga_tag) { -int id =AVIC_GATAG_TO_VCPUID (pi.prev_ga_tag); -struct kvm_vcpu *vcpu; - - vcpu =kvm_get_vcpu_by_id (kvm, id); -if (vcpu) -svm_ir_list_del (to_svm (vcpu), irqfd, &pi); - } } out:srcu_read_unlock (&kvm->irq_srcu, idx);
include/linux/kvm_irqfd.h +1 -0
@@ -60,6 +60,7 @@ struct kvm_kernel_irqfd {struct irq_bypass_consumer consumer;struct irq_bypass_producer *producer; +struct kvm_vcpu *irq_bypass_vcpu;struct list_head vcpu_list;void *irq_bypass_data; };
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 4/59 iommu/amd: KVM: SVM: Delete now-unused cached/previous GA tag fields
In reply to Sean ChristophersonDelete the amd_ir_data.prev_ga_tag field now that all usage is superfluous.
--- arch/x86/kvm/svm/avic.c | 2 -- drivers/iommu/amd/amd_iommu_types.h | 1 - drivers/iommu/amd/iommu.c | 10 ---------- include/linux/amd-iommu.h | 2 +- 4 files changed, 1 insertion(+), 14 deletions(-)
arch/x86/kvm/svm/avic.c +0 -2
@@ -938,9 +938,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,/** * Here, pi is used to: * - Tell IOMMU to use legacy mode for this interrupt. -* - Retrieve ga_tag of prior interrupt remapping data. */ - pi.prev_ga_tag =0 ; pi.is_guest_mode = false; ret =irq_set_vcpu_affinity (host_irq, &pi); }
drivers/iommu/amd/amd_iommu_types.h +0 -1
@@ -1060,7 +1060,6 @@ struct irq_2_irte { };struct amd_ir_data { -u32 cached_ga_tag;struct amd_iommu *iommu;struct irq_2_irte irq_2_irte;struct msi_msg msi_entry;
drivers/iommu/amd/iommu.c +0 -10
@@ -3887,23 +3887,13 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *vcpu_info) ir_data->cfg =irqd_cfg (data); pi_data->ir_data = ir_data; - pi_data->prev_ga_tag = ir_data->cached_ga_tag;if (pi_data->is_guest_mode) { ir_data->ga_root_ptr = (pi_data->base >>12 ); ir_data->ga_vector = vcpu_pi_info->vector; ir_data->ga_tag = pi_data->ga_tag; ret =amd_iommu_activate_guest_mode (ir_data); -if (!ret) - ir_data->cached_ga_tag = pi_data->ga_tag; }else { ret =amd_iommu_deactivate_guest_mode (ir_data); - -/* -* This communicates the ga_tag back to the caller -* so that it can do all the necessary clean up. -*/ -if (!ret) - ir_data->cached_ga_tag =0 ; }return ret;
include/linux/amd-iommu.h +1 -1
@@ -19,8 +19,8 @@ struct amd_iommu;*/ struct amd_iommu_pi_data {u32 ga_tag; -u32 prev_ga_tag;u64 base; +bool is_guest_mode;struct vcpu_data *vcpu_data;void *ir_data;
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 5/59 KVM: SVM: Delete IRTE link from previous vCPU irrespective of new routing
In reply to Sean ChristophersonDelete the IRTE link from the previous vCPU irrespective of the new routing state, i.e. even if the IRTE won't be configured to post IRQs to a vCPU. Whether or not the new route is postable as no bearing on the *old* route. Failure to delete the link can result in KVM incorrectly updating the IRTE, e.g. if the "old" vCPU is scheduled in/out.
--- arch/x86/kvm/svm/avic.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
arch/x86/kvm/svm/avic.c +6 -2
@@ -861,6 +861,12 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,if (!kvm_arch_has_assigned_device (kvm) || !kvm_arch_has_irq_bypass ())return 0 ; +/* +* If the IRQ was affined to a different vCPU, remove the IRTE metadata +* from the *previous* vCPU's list. +*/ +svm_ir_list_del (irqfd); +pr_debug ("SVM: %s: host_irq=%#x, guest_irq=%#x, set=%#x\n" , __func__, host_irq, guest_irq, set);
@@ -883,8 +889,6 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,WARN_ON_ONCE (new &&memcmp (e, new,sizeof (*new))); -svm_ir_list_del (irqfd); -/** * Here, we setup with legacy mode in the following cases: * 1. When cannot target interrupt to a specific vcpu.
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 6/59 KVM: SVM: Drop pointless masking of default APIC base when setting V_APIC_BAR
In reply to Sean ChristophersonDrop VMCB_AVIC_APIC_BAR_MASK, it's just a regurgitation of the maximum theoretical 4KiB-aligned physical address, i.e. is not novel in any way, and its only usage is to mask the default APIC base, which is 4KiB aligned and (obviously) a legal physical address.
No functional change intended.
--- arch/x86/include/asm/svm.h | 2 -- arch/x86/kvm/svm/avic.c | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-)
arch/x86/include/asm/svm.h +0 -2
@@ -260,8 +260,6 @@ struct __attribute__ ((__packed__)) vmcb_control_area {#define AVIC_DOORBELL_PHYSICAL_ID_MASK GENMASK_ULL(11, 0) -#define VMCB_AVIC_APIC_BAR_MASK 0xFFFFFFFFFF000ULL -#define AVIC_UNACCEL_ACCESS_WRITE_MASK 1#define AVIC_UNACCEL_ACCESS_OFFSET_MASK 0xFF0#define AVIC_UNACCEL_ACCESS_VECTOR_MASK 0xFFFFFFFF
arch/x86/kvm/svm/avic.c +1 -1
@@ -244,7 +244,7 @@ void avic_init_vmcb(struct vcpu_svm *svm, struct vmcb *vmcb) vmcb->control.avic_backing_page = bpa &AVIC_HPA_MASK ; vmcb->control.avic_logical_id = lpa &AVIC_HPA_MASK ; vmcb->control.avic_physical_id = ppa &AVIC_HPA_MASK ; - vmcb->control.avic_vapic_bar =APIC_DEFAULT_PHYS_BASE &VMCB_AVIC_APIC_BAR_MASK ; + vmcb->control.avic_vapic_bar =APIC_DEFAULT_PHYS_BASE ;if (kvm_apicv_activated (svm->vcpu.kvm))avic_activate_vmcb (svm);
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 7/59 KVM: SVM: Drop pointless masking of kernel page pa's with AVIC HPA masks
In reply to Sean ChristophersonDrop AVIC_HPA_MASK and all its users, the mask is just the 4KiB-aligned maximum theoretical physical address for x86-64 CPUs, as x86-64 is currently defined (going beyond PA52 would require an entirely new paging mode, which would arguably create a new, different architecture).
All usage in KVM masks the result of page_to_phys(), which on x86-64 is guaranteed to be 4KiB aligned and a legal physical address; if either of those requirements doesn't hold true, KVM has far bigger problems.
Drop masking the avic_backing_page with AVIC_PHYSICAL_ID_ENTRY_BACKING_PAGE_MASK for all the same reasons, but keep the macro even though it's unused in functional code. It's a distinct architectural define, and having the definition in software helps visualize the layout of an entry. And to be hyper-paranoid about MAXPA going beyond 52, add a compile-time assert to ensure the kernel's maximum supported physical address stays in bounds.
The unnecessary masking in avic_init_vmcb() also incorrectly assumes that SME's C-bit resides between bits 51:11; that holds true for current CPUs, but isn't required by AMD's architecture:
In some implementations, the bit used may be a physical address bit
Key word being "may".
Opportunistically use the GENMASK_ULL() version for AVIC_PHYSICAL_ID_ENTRY_BACKING_PAGE_MASK, which is far more readable than a set of repeating Fs.
--- arch/x86/include/asm/svm.h | 4 +--- arch/x86/kvm/svm/avic.c | 18 ++++++++++-------- 2 files changed, 11 insertions(+), 11 deletions(-)
arch/x86/include/asm/svm.h +1 -3
@@ -253,7 +253,7 @@ struct __attribute__ ((__packed__)) vmcb_control_area {#define AVIC_LOGICAL_ID_ENTRY_VALID_MASK (1 << 31)#define AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK GENMASK_ULL(11, 0) -#define AVIC_PHYSICAL_ID_ENTRY_BACKING_PAGE_MASK (0xFFFFFFFFFFULL << 12) +#define AVIC_PHYSICAL_ID_ENTRY_BACKING_PAGE_MASK GENMASK_ULL(51, 12)#define AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK (1ULL << 62)#define AVIC_PHYSICAL_ID_ENTRY_VALID_MASK (1ULL << 63)#define AVIC_PHYSICAL_ID_TABLE_SIZE_MASK (0xFFULL)
@@ -288,8 +288,6 @@ enum avic_ipi_failure_cause {static_assert ((AVIC_MAX_PHYSICAL_ID &AVIC_PHYSICAL_MAX_INDEX_MASK ) ==AVIC_MAX_PHYSICAL_ID );static_assert ((X2AVIC_MAX_PHYSICAL_ID &AVIC_PHYSICAL_MAX_INDEX_MASK ) ==X2AVIC_MAX_PHYSICAL_ID ); -#define AVIC_HPA_MASK ~((0xFFFULL << 52) | 0xFFF) -#define SVM_SEV_FEAT_SNP_ACTIVE BIT(0)#define SVM_SEV_FEAT_RESTRICTED_INJECTION BIT(3)#define SVM_SEV_FEAT_ALTERNATE_INJECTION BIT(4)
arch/x86/kvm/svm/avic.c +10 -8
@@ -241,9 +241,9 @@ void avic_init_vmcb(struct vcpu_svm *svm, struct vmcb *vmcb)phys_addr_t lpa =__sme_set (page_to_phys (kvm_svm->avic_logical_id_table_page));phys_addr_t ppa =__sme_set (page_to_phys (kvm_svm->avic_physical_id_table_page)); - vmcb->control.avic_backing_page = bpa &AVIC_HPA_MASK ; - vmcb->control.avic_logical_id = lpa &AVIC_HPA_MASK ; - vmcb->control.avic_physical_id = ppa &AVIC_HPA_MASK ; + vmcb->control.avic_backing_page = bpa; + vmcb->control.avic_logical_id = lpa; + vmcb->control.avic_physical_id = ppa; vmcb->control.avic_vapic_bar =APIC_DEFAULT_PHYS_BASE ;if (kvm_apicv_activated (svm->vcpu.kvm))
@@ -301,9 +301,12 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu)if (!entry)return -EINVAL ; - new_entry =__sme_set ((page_to_phys (svm->avic_backing_page) & -AVIC_PHYSICAL_ID_ENTRY_BACKING_PAGE_MASK ) | -AVIC_PHYSICAL_ID_ENTRY_VALID_MASK ); +/* Note, fls64() returns the bit position, +1. */ +BUILD_BUG_ON (__PHYSICAL_MASK_SHIFT > +fls64 (AVIC_PHYSICAL_ID_ENTRY_BACKING_PAGE_MASK )); + + new_entry =__sme_set (page_to_phys (svm->avic_backing_page)) | +AVIC_PHYSICAL_ID_ENTRY_VALID_MASK ;WRITE_ONCE (*entry, new_entry); svm->avic_physical_id_cache = entry;
@@ -903,8 +906,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, enable_remapped_mode = false;/* Try to enable guest_mode in IRTE */ - pi.base =__sme_set (page_to_phys (svm->avic_backing_page) & -AVIC_HPA_MASK ); + pi.base =__sme_set (page_to_phys (svm->avic_backing_page)); pi.ga_tag =AVIC_GATAG (to_kvm_svm (kvm)->avic_vm_id, svm->vcpu.vcpu_id); pi.is_guest_mode = true;
Signature · 1 line
2.49.0.1151.ga128411c76-goog
kernel test robot
build botRe: [PATCH v2 07/59] KVM: SVM: Drop pointless masking of kernel page pa's with AVIC HPA masks
kernel test robot noticed the following build warnings:
Hi Sean,
kernel test robot noticed the following build warnings:
[auto build test WARNING on 3debd5461fba1dcb33e732b16153da0cf5d0c251]
url: https://github.com/intel-lab-lkp/linux/commits/Sean-Christopherson/KVM-x86-Pass-new-routing-entries-and-irqfd-when-updating-IRTEs/20250523-091959 base: 3debd5461fba1dcb33e732b16153da0cf5d0c251 patch link: https://lore.kernel.org/r/20250523010004.3240643-8-seanjc%40google.com patch subject: [PATCH v2 07/59] KVM: SVM: Drop pointless masking of kernel page pa's with AVIC HPA masks config: i386-randconfig-061-20250523 (https://download.01.org/0day-ci/archive/20250524/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250524/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
arch/x86/kvm/svm/avic.c: note: in included file (through arch/x86/include/asm/bitops.h, include/linux/bitops.h, include/linux/kernel.h, ...):
>> include/asm-generic/bitops/fls64.h:24:20: sparse: sparse: cast truncates bits from constant value (ffffffffff000 becomes fffff000)
vim +24 include/asm-generic/bitops/fls64.h
a54baa1487c51c Akinobu Mita 2006-03-26 6
d57594c203b1e7 Alexander van Heukelum 2008-03-15 7 /**
d57594c203b1e7 Alexander van Heukelum 2008-03-15 8 * fls64 - find last set bit in a 64-bit word
d57594c203b1e7 Alexander van Heukelum 2008-03-15 9 * @x: the word to search
d57594c203b1e7 Alexander van Heukelum 2008-03-15 10 *
d57594c203b1e7 Alexander van Heukelum 2008-03-15 11 * This is defined in a similar way as the libc and compiler builtin
d57594c203b1e7 Alexander van Heukelum 2008-03-15 12 * ffsll, but returns the position of the most significant set bit.
d57594c203b1e7 Alexander van Heukelum 2008-03-15 13 *
d57594c203b1e7 Alexander van Heukelum 2008-03-15 14 * fls64(value) returns 0 if value is 0 or the position of the last
d57594c203b1e7 Alexander van Heukelum 2008-03-15 15 * set bit if value is nonzero. The last (most significant) bit is
d57594c203b1e7 Alexander van Heukelum 2008-03-15 16 * at position 64.
d57594c203b1e7 Alexander van Heukelum 2008-03-15 17 */
d57594c203b1e7 Alexander van Heukelum 2008-03-15 18 #if BITS_PER_LONG == 32
c8399943bdb70f Andi Kleen 2009-01-12 19 static __always_inline int fls64(__u64 x)
2dfc383ad587bb Akinobu Mita 2006-03-26 20 {
2dfc383ad587bb Akinobu Mita 2006-03-26 21 __u32 h = x >> 32;
2dfc383ad587bb Akinobu Mita 2006-03-26 22 if (h)
2dfc383ad587bb Akinobu Mita 2006-03-26 23 return fls(h) + 32;
2dfc383ad587bb Akinobu Mita 2006-03-26 @24 return fls(x);
2dfc383ad587bb Akinobu Mita 2006-03-26 25 }
d57594c203b1e7 Alexander van Heukelum 2008-03-15 26 #elif BITS_PER_LONG == 64
c8399943bdb70f Andi Kleen 2009-01-12 27 static __always_inline int fls64(__u64 x)
d57594c203b1e7 Alexander van Heukelum 2008-03-15 28 {
d57594c203b1e7 Alexander van Heukelum 2008-03-15 29 if (x == 0)
d57594c203b1e7 Alexander van Heukelum 2008-03-15 30 return 0;
d57594c203b1e7 Alexander van Heukelum 2008-03-15 31 return __fls(x) + 1;
d57594c203b1e7 Alexander van Heukelum 2008-03-15 32 }
d57594c203b1e7 Alexander van Heukelum 2008-03-15 33 #else
d57594c203b1e7 Alexander van Heukelum 2008-03-15 34 #error BITS_PER_LONG not 32 or 64
d57594c203b1e7 Alexander van Heukelum 2008-03-15 35 #endif2dfc383ad587bb Akinobu Mita 2006-03-26 36
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Sean Christopherson
maintainerPATCH v2 8/59 KVM: SVM: Add helper to deduplicate code for getting AVIC backing page
In reply to Sean ChristophersonAdd a helper to get the physical address of the AVIC backing page, both to deduplicate code and to prepare for getting the address directly from apic->regs, at which point it won't be all that obvious that the address in question is what SVM calls the AVIC backing page.
No functional change intended.
--- arch/x86/kvm/svm/avic.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
arch/x86/kvm/svm/avic.c +9 -5
@@ -234,14 +234,18 @@ int avic_vm_init(struct kvm *kvm)return err; } +static phys_addr_t avic_get_backing_page_address (struct vcpu_svm *svm) +{ +return __sme_set (page_to_phys (svm->avic_backing_page)); +} +void avic_init_vmcb (struct vcpu_svm *svm,struct vmcb *vmcb) {struct kvm_svm *kvm_svm =to_kvm_svm (svm->vcpu.kvm); -phys_addr_t bpa =__sme_set (page_to_phys (svm->avic_backing_page));phys_addr_t lpa =__sme_set (page_to_phys (kvm_svm->avic_logical_id_table_page));phys_addr_t ppa =__sme_set (page_to_phys (kvm_svm->avic_physical_id_table_page)); - vmcb->control.avic_backing_page = bpa; + vmcb->control.avic_backing_page =avic_get_backing_page_address (svm); vmcb->control.avic_logical_id = lpa; vmcb->control.avic_physical_id = ppa; vmcb->control.avic_vapic_bar =APIC_DEFAULT_PHYS_BASE ;
@@ -305,7 +309,7 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu)BUILD_BUG_ON (__PHYSICAL_MASK_SHIFT >fls64 (AVIC_PHYSICAL_ID_ENTRY_BACKING_PAGE_MASK )); - new_entry =__sme_set (page_to_phys (svm->avic_backing_page)) | + new_entry =avic_get_backing_page_address (svm) |AVIC_PHYSICAL_ID_ENTRY_VALID_MASK ;WRITE_ONCE (*entry, new_entry);
@@ -845,7 +849,7 @@ get_pi_vcpu_info(struct kvm *kvm, struct kvm_kernel_irq_routing_entry *e,pr_debug ("SVM: %s: use GA mode for irq %u\n" , __func__, irq.vector); *svm =to_svm (vcpu); - vcpu_info->pi_desc_addr =__sme_set (page_to_phys ((*svm)->avic_backing_page)); + vcpu_info->pi_desc_addr =avic_get_backing_page_address (*svm); vcpu_info->vector = irq.vector;return 0 ;
@@ -906,7 +910,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, enable_remapped_mode = false;/* Try to enable guest_mode in IRTE */ - pi.base =__sme_set (page_to_phys (svm->avic_backing_page)); + pi.base =avic_get_backing_page_address (svm); pi.ga_tag =AVIC_GATAG (to_kvm_svm (kvm)->avic_vm_id, svm->vcpu.vcpu_id); pi.is_guest_mode = true;
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 9/59 KVM: SVM: Drop vcpu_svm's pointless avic_backing_page field
In reply to Sean ChristophersonDrop vcpu_svm's avic_backing_page pointer and instead grab the physical address of KVM's vAPIC page directly from the source. Getting a physical address from a kernel virtual address is not an expensive operation, and getting the physical address from a struct page is *more* expensive for CONFIG_SPARSEMEM=y kernels. Regardless, none of the paths that consume the address are hot paths, i.e. shaving cycles is not a priority.
Eliminating the "cache" means KVM doesn't have to worry about the cache being invalid, which will simplify a future fix when dealing with vCPU IDs that are too big.
WARN if KVM attempts to allocate a vCPU's AVIC backing page without an in-kernel local APIC. avic_init_vcpu() bails early if the APIC is not in-kernel, and KVM disallows enabling an in-kernel APIC after vCPUs have been created, i.e. it should be impossible to reach avic_init_backing_page() without the vAPIC being allocated.
--- arch/x86/kvm/svm/avic.c | 6 ++---- arch/x86/kvm/svm/svm.h | 1 - 2 files changed, 2 insertions(+), 5 deletions(-)
arch/x86/kvm/svm/avic.c +2 -4
@@ -236,7 +236,7 @@ int avic_vm_init(struct kvm *kvm)static phys_addr_t avic_get_backing_page_address (struct vcpu_svm *svm) { -return __sme_set (page_to_phys (svm->avic_backing_page)); +return __sme_set (__pa (svm->vcpu.arch.apic->regs)); }void avic_init_vmcb (struct vcpu_svm *svm,struct vmcb *vmcb)
@@ -281,7 +281,7 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu) (id > X2AVIC_MAX_PHYSICAL_ID))return -EINVAL ; -if (!vcpu->arch.apic->regs) +if (WARN_ON_ONCE (!vcpu->arch.apic->regs))return -EINVAL ;if (kvm_apicv_activated (vcpu->kvm)) {
@@ -298,8 +298,6 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu)return ret; } - svm->avic_backing_page =virt_to_page (vcpu->arch.apic->regs); -/* Setting AVIC backing page address in the phy APIC ID table */ entry =avic_get_physical_id_entry (vcpu, id);if (!entry)
arch/x86/kvm/svm/svm.h +0 -1
@@ -306,7 +306,6 @@ struct vcpu_svm {u32 ldr_reg;u32 dfr_reg; -struct page *avic_backing_page;u64 *avic_physical_id_cache;/*
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 10/59 KVM: SVM: Inhibit AVIC if ID is too big instead of rejecting vCPU creation
In reply to Sean ChristophersonInhibit AVIC with a new "ID too big" flag if userspace creates a vCPU with an ID that is too big, but otherwise allow vCPU creation to succeed. Rejecting KVM_CREATE_VCPU with EINVAL violates KVM's ABI as KVM advertises that the max vCPU ID is 4095, but disallows creating vCPUs with IDs bigger than 254 (AVIC) or 511 (x2AVIC).
Alternatively, KVM could advertise an accurate value depending on which AVIC mode is in use, but that wouldn't really solve the underlying problem, e.g. would be a breaking change if KVM were to ever try and enable AVIC or x2AVIC by default.
--- arch/x86/include/asm/kvm_host.h | 9 ++++++++- arch/x86/kvm/svm/avic.c | 14 ++++++++++++-- arch/x86/kvm/svm/svm.h | 3 ++- 3 files changed, 22 insertions(+), 4 deletions(-)
arch/x86/include/asm/kvm_host.h +8 -1
@@ -1314,6 +1314,12 @@ enum kvm_apicv_inhibit {*/ APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED , +/* +* AVIC is disabled because the vCPU's APIC ID is beyond the max +* supported by AVIC/x2AVIC, i.e. the vCPU is unaddressable. +*/ +APICV_INHIBIT_REASON_PHYSICAL_ID_TOO_BIG , + NR_APICV_INHIBIT_REASONS, };
@@ -1332,7 +1338,8 @@ enum kvm_apicv_inhibit {__APICV_INHIBIT_REASON (IRQWIN ), \__APICV_INHIBIT_REASON (PIT_REINJ ), \__APICV_INHIBIT_REASON (SEV ), \ - __APICV_INHIBIT_REASON(LOGICAL_ID_ALIASED) +__APICV_INHIBIT_REASON (LOGICAL_ID_ALIASED ), \ + __APICV_INHIBIT_REASON(PHYSICAL_ID_TOO_BIG) struct kvm_arch { unsigned long n_used_mmu_pages;
arch/x86/kvm/svm/avic.c +12 -2
@@ -277,9 +277,19 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu)int id = vcpu->vcpu_id;struct vcpu_svm *svm =to_svm (vcpu); +/* +* Inhibit AVIC if the vCPU ID is bigger than what is supported by AVIC +* hardware. Immediately clear apicv_active, i.e. don't wait until the +* KVM_REQ_APICV_UPDATE request is processed on the first KVM_RUN, as +* avic_vcpu_load() expects to be called if and only if the vCPU has +* fully initialized AVIC. +*/ if ((!x2avic_enabled && id >AVIC_MAX_PHYSICAL_ID ) || - (id >X2AVIC_MAX_PHYSICAL_ID )) -return -EINVAL ; + (id >X2AVIC_MAX_PHYSICAL_ID )) { +kvm_set_apicv_inhibit (vcpu->kvm,APICV_INHIBIT_REASON_PHYSICAL_ID_TOO_BIG ); + vcpu->arch.apic->apicv_active = false; +return 0 ; + }if (WARN_ON_ONCE (!vcpu->arch.apic->regs))return -EINVAL ;
arch/x86/kvm/svm/svm.h +2 -1
@@ -722,7 +722,8 @@ extern struct kvm_x86_nested_ops svm_nested_ops; BIT(APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED) | \ BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) | \ BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED) | \ - BIT(APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED) \ + BIT(APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED) | \ + BIT(APICV_INHIBIT_REASON_PHYSICAL_ID_TOO_BIG) \ )bool avic_hardware_setup (void );
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 11/59 KVM: SVM: Drop redundant check in AVIC code on ID during vCPU creation
In reply to Sean ChristophersonDrop avic_get_physical_id_entry()'s compatibility check on the incoming ID, as its sole caller, avic_init_backing_page(), performs the exact same check. Drop avic_get_physical_id_entry() entirely as the only remaining functionality is getting the address of the Physical ID table, and accessing the array without an immediate bounds check is kludgy.
Opportunistically add a compile-time assertion to ensure the vcpu_id can't result in a bounds overflow, e.g. if KVM (really) messed up a maximum physical ID #define, as well as run-time assertions so that a NULL pointer dereference is morphed into a safer WARN().
No functional change intended.
--- arch/x86/kvm/svm/avic.c | 37 +++++++++++++++---------------------- 1 file changed, 15 insertions(+), 22 deletions(-)
arch/x86/kvm/svm/avic.c +15 -22
@@ -256,26 +256,12 @@ void avic_init_vmcb(struct vcpu_svm *svm, struct vmcb *vmcb)avic_deactivate_vmcb (svm); } -static u64 *avic_get_physical_id_entry (struct kvm_vcpu *vcpu, -unsigned int index) -{ -u64 *avic_physical_id_table; -struct kvm_svm *kvm_svm =to_kvm_svm (vcpu->kvm); - -if ((!x2avic_enabled && index >AVIC_MAX_PHYSICAL_ID ) || - (index >X2AVIC_MAX_PHYSICAL_ID )) -return NULL ; - - avic_physical_id_table =page_address (kvm_svm->avic_physical_id_table_page); - -return &avic_physical_id_table[index]; -} -static int avic_init_backing_page (struct kvm_vcpu *vcpu) { -u64 *entry, new_entry; -int id = vcpu->vcpu_id; +struct kvm_svm *kvm_svm =to_kvm_svm (vcpu->kvm);struct vcpu_svm *svm =to_svm (vcpu); +u32 id = vcpu->vcpu_id; +u64 *table, new_entry;/* * Inhibit AVIC if the vCPU ID is bigger than what is supported by AVIC
@@ -291,6 +277,9 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu)return 0 ; } +BUILD_BUG_ON ((AVIC_MAX_PHYSICAL_ID +1 ) *sizeof (*table) >PAGE_SIZE || + (X2AVIC_MAX_PHYSICAL_ID +1 ) *sizeof (*table) >PAGE_SIZE ); +if (WARN_ON_ONCE (!vcpu->arch.apic->regs))return -EINVAL ;
@@ -309,9 +298,7 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu) }/* Setting AVIC backing page address in the phy APIC ID table */ - entry =avic_get_physical_id_entry (vcpu, id); -if (!entry) -return -EINVAL ; + table =page_address (kvm_svm->avic_physical_id_table_page);/* Note, fls64() returns the bit position, +1. */ BUILD_BUG_ON(__PHYSICAL_MASK_SHIFT >
@@ -319,9 +306,9 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu) new_entry =avic_get_backing_page_address (svm) |AVIC_PHYSICAL_ID_ENTRY_VALID_MASK ; -WRITE_ONCE (*entry, new_entry); +WRITE_ONCE (table[id], new_entry); - svm->avic_physical_id_cache = entry; + svm->avic_physical_id_cache = &table[id];return 0 ; }
@@ -1004,6 +991,9 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu)if (WARN_ON (h_physical_id & ~AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK ))return ; +if (WARN_ON_ONCE (!svm->avic_physical_id_cache)) +return ; +/* * No need to update anything if the vCPU is blocking, i.e. if the vCPU * is being scheduled in after being preempted. The CPU entries in the
@@ -1044,6 +1034,9 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu)lockdep_assert_preemption_disabled (); +if (WARN_ON_ONCE (!svm->avic_physical_id_cache)) +return ; +/* * Note, reading the Physical ID entry outside of ir_list_lock is safe * as only the pCPU that has loaded (or is loading) the vCPU is allowed
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 12/59 KVM: SVM: Track AVIC tables as natively sized pointers, not "struct pages"
In reply to Sean ChristophersonAllocate and track AVIC's logical and physical tables as u32 and u64 pointers respectively, as managing the pages as "struct page" pointers adds an almost absurd amount of boilerplate and complexity. E.g. with page_address() out of the way, svm->avic_physical_id_cache becomes completely superfluous, and will be removed in a future cleanup.
No functional change intended.
--- arch/x86/kvm/svm/avic.c | 49 ++++++++++++++--------------------------- arch/x86/kvm/svm/svm.h | 4 ++-- 2 files changed, 18 insertions(+), 35 deletions(-)
arch/x86/kvm/svm/avic.c +16 -33
@@ -172,10 +172,8 @@ void avic_vm_destroy(struct kvm *kvm)if (!enable_apicv)return ; -if (kvm_svm->avic_logical_id_table_page) -__free_page (kvm_svm->avic_logical_id_table_page); -if (kvm_svm->avic_physical_id_table_page) -__free_page (kvm_svm->avic_physical_id_table_page); +free_page ((unsigned long )kvm_svm->avic_logical_id_table); +free_page ((unsigned long )kvm_svm->avic_physical_id_table);spin_lock_irqsave (&svm_vm_data_hash_lock, flags);hash_del (&kvm_svm->hnode);
@@ -188,27 +186,19 @@ int avic_vm_init(struct kvm *kvm)int err = -ENOMEM ;struct kvm_svm *kvm_svm =to_kvm_svm (kvm);struct kvm_svm *k2; -struct page *p_page; -struct page *l_page;u32 vm_id;if (!enable_apicv)return 0 ; -/* Allocating physical APIC ID table (4KB) */ - p_page =alloc_page (GFP_KERNEL_ACCOUNT | __GFP_ZERO); -if (!p_page) + kvm_svm->avic_physical_id_table = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT ); +if (!kvm_svm->avic_physical_id_table) goto free_avic; - kvm_svm->avic_physical_id_table_page = p_page; - -/* Allocating logical APIC ID table (4KB) */ - l_page =alloc_page (GFP_KERNEL_ACCOUNT | __GFP_ZERO); -if (!l_page) + kvm_svm->avic_logical_id_table = (void *)get_zeroed_page (GFP_KERNEL_ACCOUNT ); +if (!kvm_svm->avic_logical_id_table) goto free_avic; - kvm_svm->avic_logical_id_table_page = l_page; -spin_lock_irqsave (&svm_vm_data_hash_lock, flags); again: vm_id = next_vm_id = (next_vm_id +1 ) &AVIC_VM_ID_MASK ;
@@ -242,12 +232,10 @@ static phys_addr_t avic_get_backing_page_address(struct vcpu_svm *svm)void avic_init_vmcb (struct vcpu_svm *svm,struct vmcb *vmcb) {struct kvm_svm *kvm_svm =to_kvm_svm (svm->vcpu.kvm); -phys_addr_t lpa =__sme_set (page_to_phys (kvm_svm->avic_logical_id_table_page)); -phys_addr_t ppa =__sme_set (page_to_phys (kvm_svm->avic_physical_id_table_page)); vmcb->control.avic_backing_page =avic_get_backing_page_address (svm); - vmcb->control.avic_logical_id = lpa; - vmcb->control.avic_physical_id = ppa; + vmcb->control.avic_logical_id =__sme_set (__pa (kvm_svm->avic_logical_id_table)); + vmcb->control.avic_physical_id =__sme_set (__pa (kvm_svm->avic_physical_id_table)); vmcb->control.avic_vapic_bar =APIC_DEFAULT_PHYS_BASE ;if (kvm_apicv_activated (svm->vcpu.kvm))
@@ -261,7 +249,7 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu)struct kvm_svm *kvm_svm =to_kvm_svm (vcpu->kvm);struct vcpu_svm *svm =to_svm (vcpu);u32 id = vcpu->vcpu_id; -u64 *table, new_entry; +u64 new_entry;/* * Inhibit AVIC if the vCPU ID is bigger than what is supported by AVIC
@@ -277,8 +265,8 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu)return 0 ; } -BUILD_BUG_ON ((AVIC_MAX_PHYSICAL_ID +1 ) *sizeof (*table) >PAGE_SIZE || - (X2AVIC_MAX_PHYSICAL_ID +1 ) *sizeof (*table) >PAGE_SIZE ); +BUILD_BUG_ON ((AVIC_MAX_PHYSICAL_ID +1 ) *sizeof (new_entry) >PAGE_SIZE || + (X2AVIC_MAX_PHYSICAL_ID +1 ) *sizeof (new_entry) >PAGE_SIZE );if (WARN_ON_ONCE (!vcpu->arch.apic->regs))return -EINVAL ;
@@ -297,18 +285,16 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu)return ret; } -/* Setting AVIC backing page address in the phy APIC ID table */ - table =page_address (kvm_svm->avic_physical_id_table_page); -/* Note, fls64() returns the bit position, +1. */ BUILD_BUG_ON (__PHYSICAL_MASK_SHIFT >fls64 (AVIC_PHYSICAL_ID_ENTRY_BACKING_PAGE_MASK )); +/* Setting AVIC backing page address in the phy APIC ID table */ new_entry =avic_get_backing_page_address (svm) |AVIC_PHYSICAL_ID_ENTRY_VALID_MASK ; -WRITE_ONCE (table[id], new_entry); +WRITE_ONCE (kvm_svm->avic_physical_id_table[id], new_entry); - svm->avic_physical_id_cache = &table[id]; + svm->avic_physical_id_cache = &kvm_svm->avic_physical_id_table[id];return 0 ; }
@@ -442,7 +428,7 @@ static int avic_kick_target_vcpus_fast(struct kvm *kvm, struct kvm_lapic *sourceif (apic_x2apic_mode (source)) avic_logical_id_table =NULL ;else - avic_logical_id_table =page_address (kvm_svm->avic_logical_id_table_page); + avic_logical_id_table = kvm_svm->avic_logical_id_table;/* * AVIC is inhibited if vCPUs aren't mapped 1:1 with logical
@@ -544,7 +530,6 @@ unsigned long avic_vcpu_get_apicv_inhibit_reasons(struct kvm_vcpu *vcpu)static u32 *avic_get_logical_id_entry (struct kvm_vcpu *vcpu,u32 ldr,bool flat) {struct kvm_svm *kvm_svm =to_kvm_svm (vcpu->kvm); -u32 *logical_apic_id_table;u32 cluster, index; ldr =GET_APIC_LOGICAL_ID (ldr);
@@ -565,9 +550,7 @@ static u32 *avic_get_logical_id_entry(struct kvm_vcpu *vcpu, u32 ldr, bool flat)return NULL ; index += (cluster <<2 ); - logical_apic_id_table = (u32 *)page_address (kvm_svm->avic_logical_id_table_page); - -return &logical_apic_id_table[index]; +return &kvm_svm->avic_logical_id_table[index]; }static void avic_ldr_write (struct kvm_vcpu *vcpu,u8 g_physical_id,u32 ldr)
arch/x86/kvm/svm/svm.h +2 -2
@@ -123,8 +123,8 @@ struct kvm_svm {/* Struct members for AVIC */ u32 avic_vm_id; -struct page *avic_logical_id_table_page; -struct page *avic_physical_id_table_page; +u32 *avic_logical_id_table; +u64 *avic_physical_id_table;struct hlist_node hnode;struct kvm_sev_info sev_info;
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 13/59 KVM: SVM: Drop superfluous "cache" of AVIC Physical ID entry pointer
In reply to Sean ChristophersonDrop the vCPU's pointer to its AVIC Physical ID entry, and simply index the table directly. Caching a pointer address is completely unnecessary for performance, and while the field technically caches the result of the pointer calculation, it's all too easy to misinterpret the name and think that the field somehow caches the _data_ in the table.
No functional change intended.
--- arch/x86/kvm/svm/avic.c | 27 +++++++++++++++------------ arch/x86/kvm/svm/svm.h | 1 - 2 files changed, 15 insertions(+), 13 deletions(-)
arch/x86/kvm/svm/avic.c +15 -12
@@ -294,8 +294,6 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu)AVIC_PHYSICAL_ID_ENTRY_VALID_MASK ;WRITE_ONCE (kvm_svm->avic_physical_id_table[id], new_entry); - svm->avic_physical_id_cache = &kvm_svm->avic_physical_id_table[id]; -return 0 ; }
@@ -770,13 +768,16 @@ static int svm_ir_list_add(struct vcpu_svm *svm,struct kvm_kernel_irqfd *irqfd, struct amd_iommu_pi_data *pi) { + struct kvm_vcpu *vcpu = &svm->vcpu; +struct kvm *kvm = vcpu->kvm; +struct kvm_svm *kvm_svm =to_kvm_svm (kvm);unsigned long flags;u64 entry;if (WARN_ON_ONCE (!pi->ir_data))return -EINVAL ; - irqfd->irq_bypass_vcpu = &svm->vcpu; + irqfd->irq_bypass_vcpu = vcpu; irqfd->irq_bypass_data = pi->ir_data;spin_lock_irqsave (&svm->ir_list_lock, flags);
@@ -787,7 +788,7 @@ static int svm_ir_list_add(struct vcpu_svm *svm,* will update the pCPU info when the vCPU awkened and/or scheduled in. * See also avic_vcpu_load(). */ - entry =READ_ONCE (*(svm->avic_physical_id_cache)); + entry =READ_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id]);if (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK )amd_iommu_update_ga (entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK , true, pi->ir_data);
@@ -964,17 +965,18 @@ avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu, bool r)void avic_vcpu_load (struct kvm_vcpu *vcpu,int cpu) { -u64 entry; +struct kvm_svm *kvm_svm =to_kvm_svm (vcpu->kvm);int h_physical_id =kvm_cpu_get_apicid (cpu);struct vcpu_svm *svm =to_svm (vcpu);unsigned long flags; +u64 entry;lockdep_assert_preemption_disabled ();if (WARN_ON (h_physical_id & ~AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK ))return ; -if (WARN_ON_ONCE (!svm->avic_physical_id_cache)) +if (WARN_ON_ONCE (vcpu->vcpu_id *sizeof (entry) >=PAGE_SIZE ))return ;/*
@@ -996,14 +998,14 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu)*/ spin_lock_irqsave (&svm->ir_list_lock, flags); - entry =READ_ONCE (*(svm->avic_physical_id_cache)); + entry =READ_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id]);WARN_ON_ONCE (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ); entry &= ~AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK ; entry |= (h_physical_id &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK ); entry |=AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ; -WRITE_ONCE (*(svm->avic_physical_id_cache), entry); +WRITE_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id], entry);avic_update_iommu_vcpu_affinity (vcpu, h_physical_id, true);spin_unlock_irqrestore (&svm->ir_list_lock, flags);
@@ -1011,13 +1013,14 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu)void avic_vcpu_put (struct kvm_vcpu *vcpu) { -u64 entry; +struct kvm_svm *kvm_svm =to_kvm_svm (vcpu->kvm);struct vcpu_svm *svm =to_svm (vcpu);unsigned long flags; +u64 entry;lockdep_assert_preemption_disabled (); -if (WARN_ON_ONCE (!svm->avic_physical_id_cache)) +if (WARN_ON_ONCE (vcpu->vcpu_id *sizeof (entry) >=PAGE_SIZE ))return ;/*
@@ -1027,7 +1030,7 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu)* can't be scheduled out and thus avic_vcpu_{put,load}() can't run * recursively. */ - entry =READ_ONCE (*(svm->avic_physical_id_cache)); + entry =READ_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id]);/* Nothing to do if IsRunning == '0' due to vCPU blocking. */ if (!(entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ))
@@ -1046,7 +1049,7 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu)avic_update_iommu_vcpu_affinity (vcpu,-1 ,0 ); entry &= ~AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ; -WRITE_ONCE (*(svm->avic_physical_id_cache), entry); +WRITE_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id], entry);spin_unlock_irqrestore (&svm->ir_list_lock, flags);
arch/x86/kvm/svm/svm.h +0 -1
@@ -306,7 +306,6 @@ struct vcpu_svm {u32 ldr_reg;u32 dfr_reg; -u64 *avic_physical_id_cache;/* * Per-vCPU list of irqfds that are eligible to post IRQs directly to
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 14/59 KVM: VMX: Move enable_ipiv knob to common x86
In reply to Sean ChristophersonMove enable_ipiv to common x86 so that it can be reused by SVM to control IPI virtualization when AVIC is enabled. SVM doesn't actually provide a way to truly disable IPI virtualization, but KVM can get close enough by skipping the necessary table programming.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/vmx/capabilities.h | 1 - arch/x86/kvm/vmx/vmx.c | 2 -- arch/x86/kvm/x86.c | 3 +++ 4 files changed, 4 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1950,6 +1950,7 @@ struct kvm_arch_async_pf {extern u32 __read_mostly kvm_nr_uret_msrs;extern bool __read_mostly allow_smaller_maxphyaddr;extern bool __read_mostly enable_apicv; +extern bool __read_mostly enable_ipiv;extern bool __read_mostly enable_device_posted_irqs;extern struct kvm_x86_ops kvm_x86_ops;
arch/x86/kvm/vmx/capabilities.h +0 -1
@@ -15,7 +15,6 @@ extern bool __read_mostly enable_ept;extern bool __read_mostly enable_unrestricted_guest;extern bool __read_mostly enable_ept_ad_bits;extern bool __read_mostly enable_pml; -extern bool __read_mostly enable_ipiv;extern int __read_mostly pt_mode;#define PT_MODE_SYSTEM 0
arch/x86/kvm/vmx/vmx.c +0 -2
@@ -112,8 +112,6 @@ static bool __read_mostly fasteoi = 1;module_param (fasteoi, bool,0444 );module_param (enable_apicv, bool,0444 ); - -bool __read_mostly enable_ipiv = true;module_param (enable_ipiv, bool,0444 );module_param (enable_device_posted_irqs, bool,0444 );
arch/x86/kvm/x86.c +3 -0
@@ -226,6 +226,9 @@ EXPORT_SYMBOL_GPL(allow_smaller_maxphyaddr);bool __read_mostly enable_apicv = true;EXPORT_SYMBOL_GPL (enable_apicv); +bool __read_mostly enable_ipiv = true; +EXPORT_SYMBOL_GPL (enable_ipiv); +bool __read_mostly enable_device_posted_irqs = true;EXPORT_SYMBOL_GPL (enable_device_posted_irqs);
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 15/59 KVM: SVM: Add enable_ipiv param, never set IsRunning if disabled
In reply to Sean ChristophersonFrom: Maxim Levitsky <[email protected]>
Let userspace "disable" IPI virtualization for AVIC via the enable_ipiv module param, by never setting IsRunning. SVM doesn't provide a way to disable IPI virtualization in hardware, but by ensuring CPUs never see IsRunning=1, every IPI in the guest (except for self-IPIs) will generate a VM-Exit.
To avoid setting the real IsRunning bit, while still allowing KVM to use each vCPU's entry to update GA log entries, simply maintain a shadow of the entry, without propagating IsRunning updates to the real table when IPI virtualization is disabled.
Providing a way to effectively disable IPI virtualization will allow KVM to safely enable AVIC on hardware that is susceptible to erratum #1235, which causes hardware to sometimes fail to detect that the IsRunning bit has been cleared by software.
Note, the table _must_ be fully populated, as broadcast IPIs skip invalid entries, i.e. won't generate VM-Exit if every entry is invalid, and so simply pointing the VMCB at a common dummy table won't work.
Alternatively, KVM could allocate a shadow of the entire table, but that'd be a waste of 4KiB since the per-vCPU entry doesn't actually consume an additional 8 bytes of memory (vCPU structures are large enough that they are backed by order-N pages).
--- arch/x86/kvm/svm/avic.c | 32 ++++++++++++++++++++++++++------ arch/x86/kvm/svm/svm.c | 2 ++ arch/x86/kvm/svm/svm.h | 8 ++++++++ 3 files changed, 36 insertions(+), 6 deletions(-)
arch/x86/kvm/svm/avic.c +26 -6
@@ -292,6 +292,13 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu)/* Setting AVIC backing page address in the phy APIC ID table */ new_entry =avic_get_backing_page_address (svm) |AVIC_PHYSICAL_ID_ENTRY_VALID_MASK ; + svm->avic_physical_id_entry = new_entry; + +/* +* Initialize the real table, as vCPUs must have a valid entry in order +* for broadcast IPIs to function correctly (broadcast IPIs ignore +* invalid entries, i.e. aren't guaranteed to generate a VM-Exit). +*/ WRITE_ONCE (kvm_svm->avic_physical_id_table[id], new_entry);return 0 ;
@@ -769,8 +776,6 @@ static int svm_ir_list_add(struct vcpu_svm *svm, struct amd_iommu_pi_data *pi) {struct kvm_vcpu *vcpu = &svm->vcpu; -struct kvm *kvm = vcpu->kvm; -struct kvm_svm *kvm_svm =to_kvm_svm (kvm);unsigned long flags;u64 entry;
@@ -788,7 +793,7 @@ static int svm_ir_list_add(struct vcpu_svm *svm,* will update the pCPU info when the vCPU awkened and/or scheduled in. * See also avic_vcpu_load(). */ - entry =READ_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id]); + entry = svm->avic_physical_id_entry;if (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK )amd_iommu_update_ga (entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK , true, pi->ir_data);
@@ -998,14 +1003,26 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu)*/ spin_lock_irqsave (&svm->ir_list_lock, flags); - entry =READ_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id]); + entry = svm->avic_physical_id_entry;WARN_ON_ONCE (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ); entry &= ~AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK ; entry |= (h_physical_id &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK ); entry |=AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ; + svm->avic_physical_id_entry = entry; + +/* +* If IPI virtualization is disabled, clear IsRunning when updating the +* actual Physical ID table, so that the CPU never sees IsRunning=1. +* Keep the APIC ID up-to-date in the entry to minimize the chances of +* things going sideways if hardware peeks at the ID. +*/ +if (!enable_ipiv) + entry &= ~AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ; +WRITE_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id], entry); +avic_update_iommu_vcpu_affinity (vcpu, h_physical_id, true);spin_unlock_irqrestore (&svm->ir_list_lock, flags);
@@ -1030,7 +1047,7 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu)* can't be scheduled out and thus avic_vcpu_{put,load}() can't run * recursively. */ - entry =READ_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id]); + entry = svm->avic_physical_id_entry;/* Nothing to do if IsRunning == '0' due to vCPU blocking. */ if (!(entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ))
@@ -1049,7 +1066,10 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu)avic_update_iommu_vcpu_affinity (vcpu,-1 ,0 ); entry &= ~AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ; -WRITE_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id], entry); + svm->avic_physical_id_entry = entry; + +if (enable_ipiv) +WRITE_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id], entry);spin_unlock_irqrestore (&svm->ir_list_lock, flags);
arch/x86/kvm/svm/svm.c +2 -0
@@ -231,6 +231,7 @@ module_param(tsc_scaling, int, 0444);*/ static bool avic;module_param (avic, bool,0444 ); +module_param (enable_ipiv, bool,0444 );module_param (enable_device_posted_irqs, bool,0444 );
@@ -5594,6 +5595,7 @@ static __init int svm_hardware_setup(void) enable_apicv = avic = avic &&avic_hardware_setup ();if (!enable_apicv) { + enable_ipiv = false; svm_x86_ops.vcpu_blocking =NULL ; svm_x86_ops.vcpu_unblocking =NULL ; svm_x86_ops.vcpu_get_apicv_inhibit_reasons =NULL ;
arch/x86/kvm/svm/svm.h +8 -0
@@ -307,6 +307,14 @@ struct vcpu_svm {u32 ldr_reg;u32 dfr_reg; +/* This is essentially a shadow of the vCPU's actual entry in the +* Physical ID table that is programmed into the VMCB, i.e. that is +* seen by the CPU. If IPI virtualization is disabled, IsRunning is +* only ever set in the shadow, i.e. is never propagated to the "real" +* table, so that hardware never sees IsRunning=1. +*/ +u64 avic_physical_id_entry; +/* * Per-vCPU list of irqfds that are eligible to post IRQs directly to * the vCPU (a.k.a. device posted IRQs, a.k.a. IRQ bypass). The list
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 16/59 KVM: SVM: Disable (x2)AVIC IPI virtualization if CPU has erratum #1235
In reply to Sean ChristophersonFrom: Maxim Levitsky <[email protected]>
Disable IPI virtualization on AMD Family 17h CPUs (Zen2 and Zen1), as hardware doesn't reliably detect changes to the 'IsRunning' bit during ICR write emulation, and might fail to VM-Exit on the sending vCPU, if IsRunning was recently cleared.
The absence of the VM-Exit leads to KVM not waking (or triggering nested VM-Exit) of the target vCPU(s) of the IPI, which can lead to hung vCPUs, unbounded delays in L2 execution, etc.
To workaround the erratum, simply disable IPI virtualization, which prevents KVM from setting IsRunning and thus eliminates the race where hardware sees a stale IsRunning=1. As a result, all ICR writes (except when "Self" shorthand is used) will VM-Exit and therefore be correctly emulated by KVM.
Disabling IPI virtualization does carry a performance penalty, but benchmarkng shows that enabling AVIC without IPI virtualization is still much better than not using AVIC at all, because AVIC still accelerates posted interrupts and the receiving end of the IPIs.
Note, when virtualizaing Self-IPIs, the CPU skips reading the physical ID table and updates the vIRR directly (because the vCPU is by definition actively running), i.e. Self-IPI isn't susceptible to the erratum *and* is still accelerated by hardware.
--- arch/x86/kvm/svm/avic.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/x86/kvm/svm/avic.c +8 -0
@@ -1187,6 +1187,14 @@ bool avic_hardware_setup(void)if (x2avic_enabled)pr_info ("x2AVIC enabled\n" ); +/* +* Disable IPI virtualization for AMD Family 17h CPUs (Zen1 and Zen2) +* due to erratum 1235, which results in missed GA log events and thus +* missed wake events for blocking vCPUs due to the CPU failing to see +* a software update to clear IsRunning. +*/ + enable_ipiv = enable_ipiv && boot_cpu_data.x86 !=0x17 ; +amd_iommu_register_ga_log_notifier (&avic_ga_log_notifier);return true;
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 17/59 KVM: VMX: Suppress PI notifications whenever the vCPU is put
In reply to Sean ChristophersonSuppress posted interrupt notifications (set PID.SN=1) whenever the vCPU is put, i.e. unloaded, not just when the vCPU is preempted, as KVM doesn't do anything in response to a notification IRQ that arrives in the host, nor does KVM rely on the Outstanding Notification (PID.ON) flag when the vCPU is unloaded. And, the cost of scanning the PIR to manually set PID.ON when loading the vCPU is quite small, especially relative to the cost of loading (and unloading) a vCPU.
On the flip side, leaving SN clear means a notification for the vCPU will result in a spurious IRQ for the pCPU, even if vCPU task is scheduled out, running in userspace, etc. Even worse, if the pCPU is running a different vCPU, the spurious IRQ could trigger posted interrupt processing for the wrong vCPU, which is technically a violation of the architecture, as setting bits in PIR aren't supposed to be propagated to the vIRR until a notification IRQ is received.
The saving grace of the current behavior is that hardware sends notification interrupts if and only if PID.ON=0, i.e. only the first posted interrupt for a vCPU will trigger a spurious IRQ (for each window where the vCPU is unloaded).
Ideally, KVM would suppress notifications before enabling IRQs in the VM-Exit, but KVM relies on PID.ON as an indicator that there is a posted interrupt pending in PIR, e.g. in vmx_sync_pir_to_irr(), and sadly there is no way to ask hardware to set PID.ON, but not generate an interrupt. That could be solved by using pi_has_pending_interrupt() instead of checking only PID.ON, but it's not at all clear that would be a performance win, as KVM would end up scanning the entire PIR whenever an interrupt isn't pending.
And long term, the spurious IRQ window, i.e. where a vCPU is loaded with IRQs enabled, can effectively be made smaller for hot paths by moving performance critical VM-Exit handlers into the fastpath, i.e. by never enabling IRQs for hot path VM-Exits.
--- arch/x86/kvm/vmx/posted_intr.c | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-)
arch/x86/kvm/vmx/posted_intr.c +16 -13
@@ -73,13 +73,10 @@ void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)/* * If the vCPU wasn't on the wakeup list and wasn't migrated, then the * full update can be skipped as neither the vector nor the destination -* needs to be changed. +* needs to be changed. Clear SN even if there is no assigned device, +* again for simplicity. */ if (pi_desc->nv !=POSTED_INTR_WAKEUP_VECTOR && vcpu->cpu == cpu) { -/* -* Clear SN if it was set due to being preempted. Again, do -* this even if there is no assigned device for simplicity. -*/ if (pi_test_and_clear_sn (pi_desc)) goto after_clear_sn;return ;
@@ -225,17 +222,23 @@ void vmx_vcpu_pi_put(struct kvm_vcpu *vcpu)if (!vmx_needs_pi_wakeup (vcpu))return ; -if (kvm_vcpu_is_blocking (vcpu) && +/* +* If the vCPU is blocking with IRQs enabled and ISN'T being preempted, +* enable the wakeup handler so that notification IRQ wakes the vCPU as +* expected. There is no need to enable the wakeup handler if the vCPU +* is preempted between setting its wait state and manually scheduling +* out, as the task is still runnable, i.e. doesn't need a wake event +* from KVM to be scheduled in. +* +* If the wakeup handler isn't being enabled, Suppress Notifications as +* the cost of propagating PIR.IRR to PID.ON is negligible compared to +* the cost of a spurious IRQ, and vCPU put/load is a slow path. +*/ +if (!vcpu->preempted &&kvm_vcpu_is_blocking (vcpu) && ((is_td_vcpu (vcpu) &&tdx_interrupt_allowed (vcpu)) || (!is_td_vcpu (vcpu) && !vmx_interrupt_blocked (vcpu))))pi_enable_wakeup_handler (vcpu); - -/* -* Set SN when the vCPU is preempted. Note, the vCPU can both be seen -* as blocking and preempted, e.g. if it's preempted between setting -* its wait state and manually scheduling out. -*/ -if (vcpu->preempted) +else pi_set_sn (pi_desc); }
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 18/59 KVM: SVM: Add a comment to explain why avic_vcpu_blocking() ignores IRQ blocking
In reply to Sean ChristophersonAdd a comment to explain why KVM clears IsRunning when putting a vCPU, even though leaving IsRunning=1 would be ok from a functional perspective. Per Maxim's experiments, a misbehaving VM could spam the AVIC doorbell so fast as to induce a 50%+ loss in performance.
--- arch/x86/kvm/svm/avic.c | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-)
arch/x86/kvm/svm/avic.c +18 -13
@@ -1121,19 +1121,24 @@ void avic_vcpu_blocking(struct kvm_vcpu *vcpu)if (!kvm_vcpu_apicv_active (vcpu))return ; -/* -* Unload the AVIC when the vCPU is about to block, _before_ -* the vCPU actually blocks. -* -* Any IRQs that arrive before IsRunning=0 will not cause an -* incomplete IPI vmexit on the source, therefore vIRR will also -* be checked by kvm_vcpu_check_block() before blocking. The -* memory barrier implicit in set_current_state orders writing -* IsRunning=0 before reading the vIRR. The processor needs a -* matching memory barrier on interrupt delivery between writing -* IRR and reading IsRunning; the lack of this barrier might be -* the cause of errata #1235). -*/ +/* +* Unload the AVIC when the vCPU is about to block, _before_ the vCPU +* actually blocks. +* +* Note, any IRQs that arrive before IsRunning=0 will not cause an +* incomplete IPI vmexit on the source; kvm_vcpu_check_block() handles +* this by checking vIRR one last time before blocking. The memory +* barrier implicit in set_current_state orders writing IsRunning=0 +* before reading the vIRR. The processor needs a matching memory +* barrier on interrupt delivery between writing IRR and reading +* IsRunning; the lack of this barrier might be the cause of errata #1235). +* +* Clear IsRunning=0 even if guest IRQs are disabled, i.e. even if KVM +* doesn't need to detect events for scheduling purposes. The doorbell +* used to signal running vCPUs cannot be blocked, i.e. will perturb the +* CPU and cause noisy neighbor problems if the VM is sending interrupts +* to the vCPU while it's scheduled out. +*/ avic_vcpu_put (vcpu); }
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 19/59 iommu/amd: KVM: SVM: Use pi_desc_addr to derive ga_root_ptr
In reply to Sean ChristophersonUse vcpu_data.pi_desc_addr instead of amd_iommu_pi_data.base to get the GA root pointer. KVM is the only source of amd_iommu_pi_data.base, and KVM's one and only path for writing amd_iommu_pi_data.base computes the exact same value for vcpu_data.pi_desc_addr and amd_iommu_pi_data.base, and fills amd_iommu_pi_data.base if and only if vcpu_data.pi_desc_addr is valid, i.e. amd_iommu_pi_data.base is fully redundant.
--- arch/x86/kvm/svm/avic.c | 7 +++++-- drivers/iommu/amd/iommu.c | 2 +- include/linux/amd-iommu.h | 2 -- 3 files changed, 6 insertions(+), 5 deletions(-)
arch/x86/kvm/svm/avic.c +5 -2
@@ -893,8 +893,11 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, enable_remapped_mode = false; -/* Try to enable guest_mode in IRTE */ - pi.base =avic_get_backing_page_address (svm); +/* +* Try to enable guest_mode in IRTE. Note, the address +* of the vCPU's AVIC backing page is passed to the +* IOMMU via vcpu_info->pi_desc_addr. +*/ pi.ga_tag =AVIC_GATAG (to_kvm_svm (kvm)->avic_vm_id, svm->vcpu.vcpu_id); pi.is_guest_mode = true;
drivers/iommu/amd/iommu.c +1 -1
@@ -3888,7 +3888,7 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *vcpu_info) pi_data->ir_data = ir_data;if (pi_data->is_guest_mode) { - ir_data->ga_root_ptr = (pi_data->base >>12 ); + ir_data->ga_root_ptr = (vcpu_pi_info->pi_desc_addr >>12 ); ir_data->ga_vector = vcpu_pi_info->vector; ir_data->ga_tag = pi_data->ga_tag; ret =amd_iommu_activate_guest_mode (ir_data);
include/linux/amd-iommu.h +0 -2
@@ -19,8 +19,6 @@ struct amd_iommu;*/ struct amd_iommu_pi_data {u32 ga_tag; -u64 base; -bool is_guest_mode;struct vcpu_data *vcpu_data;void *ir_data;
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 20/59 iommu/amd: KVM: SVM: Pass NULL @vcpu_info to indicate "not guest mode"
In reply to Sean ChristophersonPass NULL to amd_ir_set_vcpu_affinity() to communicate "don't post to a vCPU" now that there's no need to communicate information back to KVM about the previous vCPU (KVM does its own tracking).
--- arch/x86/kvm/svm/avic.c | 16 ++++------------ drivers/iommu/amd/iommu.c | 10 +++++++--- 2 files changed, 11 insertions(+), 15 deletions(-)
arch/x86/kvm/svm/avic.c +4 -12
@@ -927,18 +927,10 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, } } - ret =0 ; -if (enable_remapped_mode) { -/* Use legacy mode in IRTE */ -struct amd_iommu_pi_data pi; - -/** -* Here, pi is used to: -* - Tell IOMMU to use legacy mode for this interrupt. -*/ - pi.is_guest_mode = false; - ret =irq_set_vcpu_affinity (host_irq, &pi); - } +if (enable_remapped_mode) + ret =irq_set_vcpu_affinity (host_irq,NULL ); +else + ret =0 ; out:srcu_read_unlock (&kvm->irq_srcu, idx);return ret;
drivers/iommu/amd/iommu.c +7 -3
@@ -3864,7 +3864,6 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *vcpu_info) {int ret;struct amd_iommu_pi_data *pi_data = vcpu_info; -struct vcpu_data *vcpu_pi_info = pi_data->vcpu_data;struct amd_ir_data *ir_data = data->chip_data;struct irq_2_irte *irte_info = &ir_data->irq_2_irte;struct iommu_dev_data *dev_data;
@@ -3885,9 +3884,14 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *vcpu_info)return -EINVAL ; ir_data->cfg =irqd_cfg (data); - pi_data->ir_data = ir_data; -if (pi_data->is_guest_mode) { +if (pi_data) { +struct vcpu_data *vcpu_pi_info = pi_data->vcpu_data; + + pi_data->ir_data = ir_data; + +WARN_ON_ONCE (!pi_data->is_guest_mode); + ir_data->ga_root_ptr = (vcpu_pi_info->pi_desc_addr >>12 ); ir_data->ga_vector = vcpu_pi_info->vector; ir_data->ga_tag = pi_data->ga_tag;
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 21/59 KVM: SVM: Stop walking list of routing table entries when updating IRTE
In reply to Sean ChristophersonNow that KVM explicitly passes the new/current GSI routing to pi_update_irte(), simply use the provided routing entry and stop walking the routing table to find that entry. KVM, via setup_routing_entry() and sanity checked by kvm_get_msi_route(), disallows having a GSI configured to trigger multiple MSIs.
I.e. this is subtly a glorified nop, as KVM allows at most one MSI per GSI, the for-loop can only ever process one entry, and that entry is the new/current entry (see the WARN_ON_ONCE() added by "KVM: x86: Pass new routing entries and irqfd when updating IRTEs" to ensure @new matches the entry found in the routing table).
--- arch/x86/kvm/svm/avic.c | 109 ++++++++++++++++------------------------ 1 file changed, 44 insertions(+), 65 deletions(-)
arch/x86/kvm/svm/avic.c +44 -65
@@ -843,11 +843,10 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, unsigned int host_irq, uint32_t guest_irq, struct kvm_kernel_irq_routing_entry *new) { - struct kvm_kernel_irq_routing_entry *e; -struct kvm_irq_routing_table *irq_rt; bool enable_remapped_mode = true; -bool set = !!new; -int idx, ret =0 ; +struct vcpu_data vcpu_info; +struct vcpu_svm *svm =NULL ; +int ret =0 ;if (!kvm_arch_has_assigned_device (kvm) || !kvm_arch_has_irq_bypass ())return 0 ;
@@ -859,72 +858,53 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,svm_ir_list_del (irqfd);pr_debug ("SVM: %s: host_irq=%#x, guest_irq=%#x, set=%#x\n" , - __func__, host_irq, guest_irq, set); + __func__, host_irq, guest_irq, !!new); - idx =srcu_read_lock (&kvm->irq_srcu); - irq_rt =srcu_dereference (kvm->irq_routing, &kvm->irq_srcu); +/** +* Here, we setup with legacy mode in the following cases: +* 1. When cannot target interrupt to a specific vcpu. +* 2. Unsetting posted interrupt. +* 3. APIC virtualization is disabled for the vcpu. +* 4. IRQ has incompatible delivery mode (SMI, INIT, etc) +*/ +if (new && new->type ==KVM_IRQ_ROUTING_MSI && + !get_pi_vcpu_info (kvm, new, &vcpu_info, &svm) && +kvm_vcpu_apicv_active (&svm->vcpu)) { +struct amd_iommu_pi_data pi; -if (guest_irq >= irq_rt->nr_rt_entries || -hlist_empty (&irq_rt->map[guest_irq])) { -pr_warn_once ("no route for guest_irq %u/%u (broken user space?)\n" , - guest_irq, irq_rt->nr_rt_entries); - goto out; - } + enable_remapped_mode = false; -hlist_for_each_entry (e, &irq_rt->map[guest_irq], link) { -struct vcpu_data vcpu_info; -struct vcpu_svm *svm =NULL ; - -if (e->type !=KVM_IRQ_ROUTING_MSI ) -continue ; - -WARN_ON_ONCE (new &&memcmp (e, new,sizeof (*new))); +/* +* Try to enable guest_mode in IRTE. Note, the address +* of the vCPU's AVIC backing page is passed to the +* IOMMU via vcpu_info->pi_desc_addr. +*/ + pi.ga_tag =AVIC_GATAG (to_kvm_svm (kvm)->avic_vm_id, + svm->vcpu.vcpu_id); + pi.is_guest_mode = true; + pi.vcpu_data = &vcpu_info; + ret =irq_set_vcpu_affinity (host_irq, &pi);/** -* Here, we setup with legacy mode in the following cases: -* 1. When cannot target interrupt to a specific vcpu. -* 2. Unsetting posted interrupt. -* 3. APIC virtualization is disabled for the vcpu. -* 4. IRQ has incompatible delivery mode (SMI, INIT, etc) +* Here, we successfully setting up vcpu affinity in +* IOMMU guest mode. Now, we need to store the posted +* interrupt information in a per-vcpu ir_list so that +* we can reference to them directly when we update vcpu +* scheduling information in IOMMU irte. */ -if (!get_pi_vcpu_info (kvm, e, &vcpu_info, &svm) && set && -kvm_vcpu_apicv_active (&svm->vcpu)) { -struct amd_iommu_pi_data pi; - - enable_remapped_mode = false; - -/* -* Try to enable guest_mode in IRTE. Note, the address -* of the vCPU's AVIC backing page is passed to the -* IOMMU via vcpu_info->pi_desc_addr. -*/ - pi.ga_tag =AVIC_GATAG (to_kvm_svm (kvm)->avic_vm_id, - svm->vcpu.vcpu_id); - pi.is_guest_mode = true; - pi.vcpu_data = &vcpu_info; - ret =irq_set_vcpu_affinity (host_irq, &pi); - -/** -* Here, we successfully setting up vcpu affinity in -* IOMMU guest mode. Now, we need to store the posted -* interrupt information in a per-vcpu ir_list so that -* we can reference to them directly when we update vcpu -* scheduling information in IOMMU irte. -*/ -if (!ret && pi.is_guest_mode) -svm_ir_list_add (svm, irqfd, &pi); - } - -if (!ret && svm) { -trace_kvm_pi_irte_update (host_irq, svm->vcpu.vcpu_id, - e->gsi, vcpu_info.vector, - vcpu_info.pi_desc_addr, set); - } - -if (ret <0 ) { -pr_err ("%s: failed to update PI IRTE\n" , __func__); - goto out; - } +if (!ret) + ret =svm_ir_list_add (svm, irqfd, &pi); + } + +if (!ret && svm) { +trace_kvm_pi_irte_update (host_irq, svm->vcpu.vcpu_id, + guest_irq, vcpu_info.vector, + vcpu_info.pi_desc_addr, !!new); + } + +if (ret <0 ) { +pr_err ("%s: failed to update PI IRTE\n" , __func__); + goto out; }if (enable_remapped_mode)
@@ -932,7 +912,6 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,else ret =0 ; out: -srcu_read_unlock (&kvm->irq_srcu, idx);return ret; }
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 22/59 KVM: VMX: Stop walking list of routing table entries when updating IRTE
In reply to Sean ChristophersonNow that KVM provides the to-be-updated routing entry, stop walking the routing table to find that entry. KVM, via setup_routing_entry() and sanity checked by kvm_get_msi_route(), disallows having a GSI configured to trigger multiple MSIs, i.e. the for-loop can only process one entry.
--- arch/x86/kvm/vmx/posted_intr.c | 100 +++++++++++---------------------- 1 file changed, 33 insertions(+), 67 deletions(-)
arch/x86/kvm/vmx/posted_intr.c +33 -67
@@ -302,78 +302,44 @@ int vmx_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, unsigned int host_irq, uint32_t guest_irq, struct kvm_kernel_irq_routing_entry *new) { - struct kvm_kernel_irq_routing_entry *e; -struct kvm_irq_routing_table *irq_rt; -bool enable_remapped_mode = true; struct kvm_lapic_irq irq;struct kvm_vcpu *vcpu;struct vcpu_data vcpu_info; -bool set = !!new; -int idx, ret =0 ;if (!vmx_can_use_vtd_pi (kvm))return 0 ; - idx =srcu_read_lock (&kvm->irq_srcu); - irq_rt =srcu_dereference (kvm->irq_routing, &kvm->irq_srcu); -if (guest_irq >= irq_rt->nr_rt_entries || -hlist_empty (&irq_rt->map[guest_irq])) { -pr_warn_once ("no route for guest_irq %u/%u (broken user space?)\n" , - guest_irq, irq_rt->nr_rt_entries); - goto out; - } - -hlist_for_each_entry (e, &irq_rt->map[guest_irq], link) { -if (e->type !=KVM_IRQ_ROUTING_MSI ) -continue ; - -WARN_ON_ONCE (new &&memcmp (e, new,sizeof (*new))); - -/* -* VT-d PI cannot support posting multicast/broadcast -* interrupts to a vCPU, we still use interrupt remapping -* for these kind of interrupts. -* -* For lowest-priority interrupts, we only support -* those with single CPU as the destination, e.g. user -* configures the interrupts via /proc/irq or uses -* irqbalance to make the interrupts single-CPU. -* -* We will support full lowest-priority interrupt later. -* -* In addition, we can only inject generic interrupts using -* the PI mechanism, refuse to route others through it. -*/ - -kvm_set_msi_irq (kvm, e, &irq); -if (!kvm_intr_is_single_vcpu (kvm, &irq, &vcpu) || - !kvm_irq_is_postable (&irq)) -continue ; - - vcpu_info.pi_desc_addr =__pa (vcpu_to_pi_desc (vcpu)); - vcpu_info.vector = irq.vector; - -trace_kvm_pi_irte_update (host_irq, vcpu->vcpu_id, e->gsi, - vcpu_info.vector, vcpu_info.pi_desc_addr, set); - -if (!set) -continue ; - - enable_remapped_mode = false; - - ret =irq_set_vcpu_affinity (host_irq, &vcpu_info); -if (ret <0 ) { -printk (KERN_INFO "%s: failed to update PI IRTE\n" , - __func__); - goto out; - } - } - -if (enable_remapped_mode) - ret =irq_set_vcpu_affinity (host_irq,NULL ); - - ret =0 ; -out: -srcu_read_unlock (&kvm->irq_srcu, idx); -return ret; +/* +* VT-d PI cannot support posting multicast/broadcast +* interrupts to a vCPU, we still use interrupt remapping +* for these kind of interrupts. +* +* For lowest-priority interrupts, we only support +* those with single CPU as the destination, e.g. user +* configures the interrupts via /proc/irq or uses +* irqbalance to make the interrupts single-CPU. +* +* We will support full lowest-priority interrupt later. +* +* In addition, we can only inject generic interrupts using +* the PI mechanism, refuse to route others through it. +*/ +if (!new || new->type !=KVM_IRQ_ROUTING_MSI ) + goto do_remapping; + +kvm_set_msi_irq (kvm, new, &irq); + +if (!kvm_intr_is_single_vcpu (kvm, &irq, &vcpu) || + !kvm_irq_is_postable (&irq)) + goto do_remapping; + + vcpu_info.pi_desc_addr =__pa (vcpu_to_pi_desc (vcpu)); + vcpu_info.vector = irq.vector; + +trace_kvm_pi_irte_update (host_irq, vcpu->vcpu_id, guest_irq, + vcpu_info.vector, vcpu_info.pi_desc_addr, true); + +return irq_set_vcpu_affinity (host_irq, &vcpu_info); +do_remapping: +return irq_set_vcpu_affinity (host_irq,NULL ); }
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 23/59 KVM: SVM: Extract SVM specific code out of get_pi_vcpu_info()
In reply to Sean ChristophersonGenericize SVM's get_pi_vcpu_info() so that it can be shared with VMX. The only SVM specific information it provides is the AVIC back page, and that can be trivially retrieved by its sole caller.
No functional change intended.
--- arch/x86/kvm/svm/avic.c | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-)
arch/x86/kvm/svm/avic.c +12 -13
@@ -816,14 +816,14 @@ static int svm_ir_list_add(struct vcpu_svm *svm,*/ static int get_pi_vcpu_info (struct kvm *kvm,struct kvm_kernel_irq_routing_entry *e, -struct vcpu_data *vcpu_info,struct vcpu_svm **svm) +struct vcpu_data *vcpu_info,struct kvm_vcpu **vcpu) {struct kvm_lapic_irq irq; -struct kvm_vcpu *vcpu =NULL ; + *vcpu =NULL ;kvm_set_msi_irq (kvm, e, &irq); -if (!kvm_intr_is_single_vcpu (kvm, &irq, &vcpu) || +if (!kvm_intr_is_single_vcpu (kvm, &irq, vcpu) || !kvm_irq_is_postable (&irq)) {pr_debug ("SVM: %s: use legacy intr remap mode for irq %u\n" , __func__, irq.vector);
@@ -832,8 +832,6 @@ get_pi_vcpu_info(struct kvm *kvm, struct kvm_kernel_irq_routing_entry *e,pr_debug ("SVM: %s: use GA mode for irq %u\n" , __func__, irq.vector); - *svm =to_svm (vcpu); - vcpu_info->pi_desc_addr =avic_get_backing_page_address (*svm); vcpu_info->vector = irq.vector;return 0 ;
@@ -845,7 +843,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, {bool enable_remapped_mode = true;struct vcpu_data vcpu_info; -struct vcpu_svm *svm =NULL ; +struct kvm_vcpu *vcpu =NULL ;int ret =0 ;if (!kvm_arch_has_assigned_device (kvm) || !kvm_arch_has_irq_bypass ())
@@ -868,19 +866,20 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,* 4. IRQ has incompatible delivery mode (SMI, INIT, etc) */ if (new && new->type ==KVM_IRQ_ROUTING_MSI && - !get_pi_vcpu_info (kvm, new, &vcpu_info, &svm) && -kvm_vcpu_apicv_active (&svm->vcpu)) { + !get_pi_vcpu_info (kvm, new, &vcpu_info, &vcpu) && +kvm_vcpu_apicv_active (vcpu)) {struct amd_iommu_pi_data pi; enable_remapped_mode = false; + vcpu_info.pi_desc_addr =avic_get_backing_page_address (to_svm (vcpu)); +/* * Try to enable guest_mode in IRTE. Note, the address * of the vCPU's AVIC backing page is passed to the * IOMMU via vcpu_info->pi_desc_addr. */ - pi.ga_tag =AVIC_GATAG (to_kvm_svm (kvm)->avic_vm_id, - svm->vcpu.vcpu_id); + pi.ga_tag =AVIC_GATAG (to_kvm_svm (kvm)->avic_vm_id, vcpu->vcpu_id); pi.is_guest_mode = true; pi.vcpu_data = &vcpu_info; ret =irq_set_vcpu_affinity (host_irq, &pi);
@@ -893,11 +892,11 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,* scheduling information in IOMMU irte. */ if (!ret) - ret =svm_ir_list_add (svm, irqfd, &pi); + ret =svm_ir_list_add (to_svm (vcpu), irqfd, &pi); } -if (!ret && svm) { -trace_kvm_pi_irte_update (host_irq, svm->vcpu.vcpu_id, +if (!ret && vcpu) { +trace_kvm_pi_irte_update (host_irq, vcpu->vcpu_id, guest_irq, vcpu_info.vector, vcpu_info.pi_desc_addr, !!new); }
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 24/59 KVM: x86: Move IRQ routing/delivery APIs from x86.c => irq.c
In reply to Sean ChristophersonMove a bunch of IRQ routing and delivery APIs from x86.c to irq.c. x86.c has grown quite fat, and irq.c is the perfect landing spot.
Opportunistically rewrite kvm_arch_irq_bypass_del_producer()'s comment, as the existing comment has several typos and is rather confusing.
--- arch/x86/kvm/irq.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 87 --------------------------------------------- 2 files changed, 88 insertions(+), 87 deletions(-)
arch/x86/kvm/irq.c +88 -0
@@ -11,6 +11,7 @@#include <linux/export.h> #include <linux/kvm_host.h> +#include <linux/kvm_irqfd.h> #include "hyperv.h" #include "ioapic.h"
@@ -332,6 +333,18 @@ int kvm_arch_set_irq_inatomic(struct kvm_kernel_irq_routing_entry *e,return -EWOULDBLOCK ; } +int kvm_vm_ioctl_irq_line (struct kvm *kvm,struct kvm_irq_level *irq_event, +bool line_status) +{ +if (!irqchip_in_kernel (kvm)) +return -ENXIO ; + + irq_event->status =kvm_set_irq (kvm,KVM_USERSPACE_IRQ_SOURCE_ID , + irq_event->irq, irq_event->level, + line_status); +return 0 ; +} +bool kvm_arch_can_set_irq_routing (struct kvm *kvm) {return irqchip_in_kernel (kvm);
@@ -494,3 +507,78 @@ void kvm_arch_irq_routing_update(struct kvm *kvm)if (irqchip_split (kvm))kvm_make_scan_ioapic_request (kvm); } + +int kvm_arch_irq_bypass_add_producer (struct irq_bypass_consumer *cons, +struct irq_bypass_producer *prod) +{ +struct kvm_kernel_irqfd *irqfd = + container_of(cons, struct kvm_kernel_irqfd, consumer); +struct kvm *kvm = irqfd->kvm; +int ret =0 ; + +kvm_arch_start_assignment (irqfd->kvm); + +spin_lock_irq (&kvm->irqfds.lock); + irqfd->producer = prod; + +if (irqfd->irq_entry.type ==KVM_IRQ_ROUTING_MSI ) { + ret =kvm_x86_call (pi_update_irte)(irqfd, irqfd->kvm, prod->irq, + irqfd->gsi, &irqfd->irq_entry); +if (ret) +kvm_arch_end_assignment (irqfd->kvm); + } +spin_unlock_irq (&kvm->irqfds.lock); + +return ret; +} + +void kvm_arch_irq_bypass_del_producer (struct irq_bypass_consumer *cons, +struct irq_bypass_producer *prod) +{ +struct kvm_kernel_irqfd *irqfd = + container_of(cons, struct kvm_kernel_irqfd, consumer); +struct kvm *kvm = irqfd->kvm; +int ret; + +WARN_ON (irqfd->producer != prod); + +/* +* If the producer of an IRQ that is currently being posted to a vCPU +* is unregistered, change the associated IRTE back to remapped mode as +* the IRQ has been released (or repurposed) by the device driver, i.e. +* KVM must relinquish control of the IRTE. +*/ +spin_lock_irq (&kvm->irqfds.lock); + irqfd->producer =NULL ; + +if (irqfd->irq_entry.type ==KVM_IRQ_ROUTING_MSI ) { + ret =kvm_x86_call (pi_update_irte)(irqfd, irqfd->kvm, prod->irq, + irqfd->gsi,NULL ); +if (ret) +pr_info ("irq bypass consumer (token %p) unregistration fails: %d\n" , + irqfd->consumer.token, ret); + } + +spin_unlock_irq (&kvm->irqfds.lock); + + +kvm_arch_end_assignment (irqfd->kvm); +} + +int kvm_arch_update_irqfd_routing (struct kvm_kernel_irqfd *irqfd, +struct kvm_kernel_irq_routing_entry *old, +struct kvm_kernel_irq_routing_entry *new) +{ +return kvm_x86_call (pi_update_irte)(irqfd, irqfd->kvm, irqfd->producer->irq, + irqfd->gsi, new); +} + +bool kvm_arch_irqfd_route_changed (struct kvm_kernel_irq_routing_entry *old, +struct kvm_kernel_irq_routing_entry *new) +{ +if (old->type !=KVM_IRQ_ROUTING_MSI || + new->type !=KVM_IRQ_ROUTING_MSI ) +return true; + +return !!memcmp (&old->msi, &new->msi,sizeof (new->msi)); +}
arch/x86/kvm/x86.c +0 -87
99 lines not on this page · raw message
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 25/59 KVM: x86: Nullify irqfd->producer after updating IRTEs
In reply to Sean ChristophersonNullify irqfd->producer (when it's going away) _after_ updating IRTEs so that the producer can be queried during the update.
--- arch/x86/kvm/irq.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/kvm/irq.c +1 -2
@@ -549,7 +549,6 @@ void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *cons,* KVM must relinquish control of the IRTE. */ spin_lock_irq (&kvm->irqfds.lock); - irqfd->producer =NULL ;if (irqfd->irq_entry.type ==KVM_IRQ_ROUTING_MSI ) { ret = kvm_x86_call(pi_update_irte)(irqfd, irqfd->kvm, prod->irq,
@@ -558,10 +557,10 @@ void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *cons,pr_info ("irq bypass consumer (token %p) unregistration fails: %d\n" , irqfd->consumer.token, ret); } + irqfd->producer =NULL ;spin_unlock_irq (&kvm->irqfds.lock); -kvm_arch_end_assignment (irqfd->kvm); }
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 26/59 KVM: x86: Dedup AVIC vs. PI code for identifying target vCPU
In reply to Sean ChristophersonHoist the logic for identifying the target vCPU for a posted interrupt into common x86. The code is functionally identical between Intel and AMD.
--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/irq.c | 45 +++++++++++++++--- arch/x86/kvm/svm/avic.c | 82 ++++++++------------------------- arch/x86/kvm/svm/svm.h | 2 +- arch/x86/kvm/vmx/posted_intr.c | 55 ++++++---------------- arch/x86/kvm/vmx/posted_intr.h | 2 +- 6 files changed, 75 insertions(+), 113 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -1854,7 +1854,7 @@ struct kvm_x86_ops {int (*pi_update_irte)(struct kvm_kernel_irqfd *irqfd,struct kvm *kvm,unsigned int host_irq,uint32_t guest_irq, -struct kvm_kernel_irq_routing_entry *new); +struct kvm_vcpu *vcpu,u32 vector);void (*pi_start_assignment)(struct kvm *kvm);void (*apicv_pre_state_restore)(struct kvm_vcpu *vcpu);void (*apicv_post_state_restore)(struct kvm_vcpu *vcpu);
arch/x86/kvm/irq.c +39 -6
69 lines not on this page · raw message
arch/x86/kvm/svm/avic.c +19 -63
123 lines not on this page · raw message
arch/x86/kvm/svm/svm.h +1 -1
@@ -747,7 +747,7 @@ void avic_apicv_post_state_restore(struct kvm_vcpu *vcpu);void avic_refresh_apicv_exec_ctrl (struct kvm_vcpu *vcpu);int avic_pi_update_irte (struct kvm_kernel_irqfd *irqfd,struct kvm *kvm,unsigned int host_irq,uint32_t guest_irq, -struct kvm_kernel_irq_routing_entry *new); +struct kvm_vcpu *vcpu,u32 vector);void avic_vcpu_blocking (struct kvm_vcpu *vcpu);void avic_vcpu_unblocking (struct kvm_vcpu *vcpu);void avic_ring_doorbell (struct kvm_vcpu *vcpu);
arch/x86/kvm/vmx/posted_intr.c +14 -41
60 lines not on this page · raw message
arch/x86/kvm/vmx/posted_intr.h +1 -1
@@ -16,7 +16,7 @@ void pi_apicv_pre_state_restore(struct kvm_vcpu *vcpu);bool pi_has_pending_interrupt (struct kvm_vcpu *vcpu);int vmx_pi_update_irte (struct kvm_kernel_irqfd *irqfd,struct kvm *kvm,unsigned int host_irq,uint32_t guest_irq, -struct kvm_kernel_irq_routing_entry *new); +struct kvm_vcpu *vcpu,u32 vector);void vmx_pi_start_assignment (struct kvm *kvm);static inline int pi_find_highest_vector (struct pi_desc *pi_desc)
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 27/59 KVM: x86: Move posted interrupt tracepoint to common code
In reply to Sean ChristophersonMove the pi_irte_update tracepoint to common x86, and call it whenever the IRTE is modified. Tracing only the modifications that result in an IRQ being posted to a vCPU makes the tracepoint useless for debugging.
Drop the vendor specific address; plumbing that into common code isn't worth the trouble, as the address is meaningless without a whole pile of other information that isn't provided in any tracepoint.
--- arch/x86/kvm/irq.c | 12 ++++++++++-- arch/x86/kvm/svm/avic.c | 6 ------ arch/x86/kvm/trace.h | 19 +++++++------------ arch/x86/kvm/vmx/posted_intr.c | 3 --- arch/x86/kvm/x86.c | 1 - 5 files changed, 17 insertions(+), 24 deletions(-)
arch/x86/kvm/irq.c +10 -2
@@ -511,9 +511,11 @@ void kvm_arch_irq_routing_update(struct kvm *kvm)static int kvm_pi_update_irte (struct kvm_kernel_irqfd *irqfd,struct kvm_kernel_irq_routing_entry *entry) { +unsigned int host_irq = irqfd->producer->irq;struct kvm *kvm = irqfd->kvm;struct kvm_vcpu *vcpu =NULL ;struct kvm_lapic_irq irq; +int r; if (!irqchip_in_kernel(kvm) || !kvm_arch_has_irq_bypass() ||
@@ -540,8 +542,13 @@ static int kvm_pi_update_irte(struct kvm_kernel_irqfd *irqfd, vcpu =NULL ; } -return kvm_x86_call (pi_update_irte)(irqfd, irqfd->kvm, irqfd->producer->irq, - irqfd->gsi, vcpu, irq.vector); + r =kvm_x86_call (pi_update_irte)(irqfd, irqfd->kvm, host_irq, irqfd->gsi, + vcpu, irq.vector); +if (r) +return r; + +trace_kvm_pi_irte_update (host_irq, vcpu, irqfd->gsi, irq.vector, !!vcpu); +return 0 ; } int kvm_arch_irq_bypass_add_producer(struct irq_bypass_consumer *cons,
@@ -595,6 +602,7 @@ void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *cons,spin_unlock_irq (&kvm->irqfds.lock); +kvm_arch_end_assignment (irqfd->kvm); }
arch/x86/kvm/svm/avic.c +0 -6
@@ -815,9 +815,6 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,*/ svm_ir_list_del (irqfd); -pr_debug ("SVM: %s: host_irq=%#x, guest_irq=%#x, set=%#x\n" , - __func__, host_irq, guest_irq, !!vcpu); -/** * Here, we setup with legacy mode in the following cases: * 1. When cannot target interrupt to a specific vcpu.
@@ -853,9 +850,6 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,*/ if (!ret) ret =svm_ir_list_add (to_svm (vcpu), irqfd, &pi); - -trace_kvm_pi_irte_update (host_irq, vcpu->vcpu_id, guest_irq, - vector, vcpu_info.pi_desc_addr, true); }else { ret =irq_set_vcpu_affinity (host_irq,NULL ); }
arch/x86/kvm/trace.h +7 -12
@@ -1176,37 +1176,32 @@ TRACE_EVENT(kvm_smm_transition,* Tracepoint for VT-d posted-interrupts and AMD-Vi Guest Virtual APIC. */ TRACE_EVENT (kvm_pi_irte_update, - TP_PROTO(unsigned int host_irq, unsigned int vcpu_id, - unsigned int gsi, unsigned int gvec, - u64 pi_desc_addr, bool set), -TP_ARGS (host_irq, vcpu_id, gsi, gvec, pi_desc_addr, set), + TP_PROTO(unsigned int host_irq, struct kvm_vcpu *vcpu, + unsigned int gsi, unsigned int gvec, bool set), +TP_ARGS (host_irq, vcpu, gsi, gvec, set),TP_STRUCT__entry ( __field( unsigned int, host_irq ) - __field( unsigned int, vcpu_id ) + __field( int, vcpu_id ) __field( unsigned int, gsi ) __field( unsigned int, gvec ) - __field( u64, pi_desc_addr )__field ( bool, set ) ),TP_fast_assign ( __entry->host_irq = host_irq; - __entry->vcpu_id = vcpu_id; + __entry->vcpu_id = vcpu ? vcpu->vcpu_id : -1; __entry->gsi = gsi; __entry->gvec = gvec; - __entry->pi_desc_addr = pi_desc_addr; __entry->set = set; ), -TP_printk ("PI is %s for irq %u, vcpu %u, gsi: 0x%x, " -"gvec: 0x%x, pi_desc_addr: 0x%llx" , +TP_printk ("PI is %s for irq %u, vcpu %d, gsi: 0x%x, gvec: 0x%x" , __entry->set ?"enabled and being updated" :"disabled" , __entry->host_irq, __entry->vcpu_id, __entry->gsi, - __entry->gvec, - __entry->pi_desc_addr) + __entry->gvec) );/*
arch/x86/kvm/vmx/posted_intr.c +0 -3
@@ -308,9 +308,6 @@ int vmx_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, .vector = vector, }; -trace_kvm_pi_irte_update (host_irq, vcpu->vcpu_id, guest_irq, - vcpu_info.vector, vcpu_info.pi_desc_addr, true); -return irq_set_vcpu_affinity (host_irq, &vcpu_info); }else {return irq_set_vcpu_affinity (host_irq,NULL );
arch/x86/kvm/x86.c +0 -1
@@ -14014,7 +14014,6 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intercepts);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_write_tsc_offset);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_ple_window_update);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_pml_full); -EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_pi_irte_update);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_avic_unaccelerated_access);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_avic_incomplete_ipi);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_avic_ga_log);
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 28/59 KVM: SVM: Clean up return handling in avic_pi_update_irte()
In reply to Sean ChristophersonClean up the return paths for avic_pi_update_irte() now that the refactoring dust has settled.
Opportunistically drop the pr_err() on IRTE update failures. Logging that a failure occurred without _any_ context is quite useless.
--- arch/x86/kvm/svm/avic.c | 20 +++++--------------- 1 file changed, 5 insertions(+), 15 deletions(-)
arch/x86/kvm/svm/avic.c +5 -15
@@ -807,8 +807,6 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, unsigned int host_irq, uint32_t guest_irq, struct kvm_vcpu *vcpu, u32 vector) { - int ret = 0; -/* * If the IRQ was affined to a different vCPU, remove the IRTE metadata * from the *previous* vCPU's list.
@@ -838,8 +836,11 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, .is_guest_mode = true, .vcpu_data = &vcpu_info, }; +int ret; ret =irq_set_vcpu_affinity (host_irq, &pi); +if (ret) +return ret;/** * Here, we successfully setting up vcpu affinity in
@@ -848,20 +849,9 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,* we can reference to them directly when we update vcpu * scheduling information in IOMMU irte. */ -if (!ret) - ret =svm_ir_list_add (to_svm (vcpu), irqfd, &pi); - }else { - ret =irq_set_vcpu_affinity (host_irq,NULL ); +return svm_ir_list_add (to_svm (vcpu), irqfd, &pi); } - -if (ret <0 ) { -pr_err ("%s: failed to update PI IRTE\n" , __func__); - goto out; - } - - ret =0 ; -out: -return ret; +return irq_set_vcpu_affinity (host_irq,NULL ); } static inline int
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 29/59 iommu: KVM: Split "struct vcpu_data" into separate AMD vs. Intel structs
In reply to Sean ChristophersonSplit the vcpu_data structure that serves as a handoff from KVM to IOMMU drivers into vendor specific structures. Overloading a single structure makes the code hard to read and maintain, is *very* misleading as it suggests that mixing vendors is actually supported, and bastardizing Intel's posted interrupt descriptor address when AMD's IOMMU already has its own structure is quite unnecessary.
--- arch/x86/include/asm/irq_remapping.h | 15 ++++++++++++++- arch/x86/kvm/svm/avic.c | 21 ++++++++------------- arch/x86/kvm/vmx/posted_intr.c | 4 ++-- drivers/iommu/amd/iommu.c | 12 ++++-------- drivers/iommu/intel/irq_remapping.c | 10 +++++----- include/linux/amd-iommu.h | 12 ------------ 6 files changed, 33 insertions(+), 41 deletions(-)
arch/x86/include/asm/irq_remapping.h +14 -1
@@ -26,7 +26,20 @@ enum { IRQ_REMAP_X2APIC_MODE, }; -struct vcpu_data { +/* +* This is mainly used to communicate information back-and-forth +* between SVM and IOMMU for setting up and tearing down posted +* interrupt +*/ +struct amd_iommu_pi_data { +u64 vapic_addr;/* Physical address of the vCPU's vAPIC. */ +u32 ga_tag; +u32 vector;/* Guest vector of the interrupt */ +bool is_guest_mode; +void *ir_data; +}; + +struct intel_iommu_pi_data {u64 pi_desc_addr;/* Physical address of PI Descriptor */ u32 vector;/* Guest vector of the interrupt */ };
arch/x86/kvm/svm/avic.c +8 -13
@@ -822,23 +822,18 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,*/ if (vcpu &&kvm_vcpu_apicv_active (vcpu)) {/* -* Try to enable guest_mode in IRTE. Note, the address -* of the vCPU's AVIC backing page is passed to the -* IOMMU via vcpu_info->pi_desc_addr. +* Try to enable guest_mode in IRTE. */ -struct vcpu_data vcpu_info = { - .pi_desc_addr =avic_get_backing_page_address (to_svm (vcpu)), - .vector = vector, - }; - -struct amd_iommu_pi_data pi = { - .ga_tag =AVIC_GATAG (to_kvm_svm (kvm)->avic_vm_id, vcpu->vcpu_id), +struct amd_iommu_pi_data pi_data = { + .ga_tag =AVIC_GATAG (to_kvm_svm (kvm)->avic_vm_id, + vcpu->vcpu_id), .is_guest_mode = true, - .vcpu_data = &vcpu_info, + .vapic_addr =avic_get_backing_page_address (to_svm (vcpu)), + .vector = vector, };int ret; - ret =irq_set_vcpu_affinity (host_irq, &pi); + ret =irq_set_vcpu_affinity (host_irq, &pi_data);if (ret)return ret;
@@ -849,7 +844,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,* we can reference to them directly when we update vcpu * scheduling information in IOMMU irte. */ -return svm_ir_list_add (to_svm (vcpu), irqfd, &pi); +return svm_ir_list_add (to_svm (vcpu), irqfd, &pi_data); }return irq_set_vcpu_affinity (host_irq,NULL ); }
arch/x86/kvm/vmx/posted_intr.c +2 -2
@@ -303,12 +303,12 @@ int vmx_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, struct kvm_vcpu *vcpu, u32 vector) {if (vcpu) { -struct vcpu_data vcpu_info = { +struct intel_iommu_pi_data pi_data = { .pi_desc_addr =__pa (vcpu_to_pi_desc (vcpu)), .vector = vector, }; -return irq_set_vcpu_affinity (host_irq, &vcpu_info); +return irq_set_vcpu_affinity (host_irq, &pi_data); }else {return irq_set_vcpu_affinity (host_irq,NULL ); }
drivers/iommu/amd/iommu.c +4 -8
@@ -3860,10 +3860,10 @@ int amd_iommu_deactivate_guest_mode(void *data) }EXPORT_SYMBOL (amd_iommu_deactivate_guest_mode); -static int amd_ir_set_vcpu_affinity (struct irq_data *data,void *vcpu_info) +static int amd_ir_set_vcpu_affinity (struct irq_data *data,void *info) {int ret; -struct amd_iommu_pi_data *pi_data = vcpu_info; +struct amd_iommu_pi_data *pi_data = info;struct amd_ir_data *ir_data = data->chip_data;struct irq_2_irte *irte_info = &ir_data->irq_2_irte;struct iommu_dev_data *dev_data;
@@ -3886,14 +3886,10 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *vcpu_info) ir_data->cfg =irqd_cfg (data);if (pi_data) { -struct vcpu_data *vcpu_pi_info = pi_data->vcpu_data; - pi_data->ir_data = ir_data; -WARN_ON_ONCE (!pi_data->is_guest_mode); - - ir_data->ga_root_ptr = (vcpu_pi_info->pi_desc_addr >>12 ); - ir_data->ga_vector = vcpu_pi_info->vector; + ir_data->ga_root_ptr = (pi_data->vapic_addr >>12 ); + ir_data->ga_vector = pi_data->vector; ir_data->ga_tag = pi_data->ga_tag; ret =amd_iommu_activate_guest_mode (ir_data); }else {
drivers/iommu/intel/irq_remapping.c +5 -5
@@ -1244,10 +1244,10 @@ static void intel_ir_compose_msi_msg(struct irq_data *irq_data,static int intel_ir_set_vcpu_affinity (struct irq_data *data,void *info) {struct intel_ir_data *ir_data = data->chip_data; -struct vcpu_data *vcpu_pi_info = info; +struct intel_iommu_pi_data *pi_data = info;/* stop posting interrupts, back to the default mode */ -if (!vcpu_pi_info) { +if (!pi_data) {__intel_ir_reconfigure_irte (data, true); }else {struct irte irte_pi;
@@ -1265,10 +1265,10 @@ static int intel_ir_set_vcpu_affinity(struct irq_data *data, void *info)/* Update the posted mode fields */ irte_pi.p_pst =1 ; irte_pi.p_urgent =0 ; - irte_pi.p_vector = vcpu_pi_info->vector; - irte_pi.pda_l = (vcpu_pi_info->pi_desc_addr >> + irte_pi.p_vector = pi_data->vector; + irte_pi.pda_l = (pi_data->pi_desc_addr >> (32 -PDA_LOW_BIT )) & ~(-1UL <<PDA_LOW_BIT ); - irte_pi.pda_h = (vcpu_pi_info->pi_desc_addr >>32 ) & + irte_pi.pda_h = (pi_data->pi_desc_addr >>32 ) & ~(-1UL <<PDA_HIGH_BIT ); ir_data->irq_2_iommu.posted_vcpu = true;
include/linux/amd-iommu.h +0 -12
@@ -12,18 +12,6 @@struct amd_iommu ; -/* -* This is mainly used to communicate information back-and-forth -* between SVM and IOMMU for setting up and tearing down posted -* interrupt -*/ -struct amd_iommu_pi_data { -u32 ga_tag; -bool is_guest_mode; -struct vcpu_data *vcpu_data; -void *ir_data; -}; -#ifdef CONFIG_AMD_IOMMU struct task_struct ;
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 30/59 KVM: Don't WARN if updating IRQ bypass route fails
In reply to Sean ChristophersonDon't bother WARNing if updating an IRTE route fails now that vendor code provides much more precise WARNs. The generic WARN doesn't provide enough information to actually debug the problem, and has obviously done nothing to surface the myriad bugs in KVM's implementation.
--- arch/x86/kvm/irq.c | 8 ++++---- include/linux/kvm_host.h | 6 +++--- virt/kvm/eventfd.c | 15 ++++++--------- 3 files changed, 13 insertions(+), 16 deletions(-)
arch/x86/kvm/irq.c +4 -4
@@ -606,11 +606,11 @@ void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *cons,kvm_arch_end_assignment (irqfd->kvm); } -int kvm_arch_update_irqfd_routing (struct kvm_kernel_irqfd *irqfd, -struct kvm_kernel_irq_routing_entry *old, -struct kvm_kernel_irq_routing_entry *new) +void kvm_arch_update_irqfd_routing (struct kvm_kernel_irqfd *irqfd, +struct kvm_kernel_irq_routing_entry *old, +struct kvm_kernel_irq_routing_entry *new) { -return kvm_pi_update_irte (irqfd, new); +kvm_pi_update_irte (irqfd, new); } bool kvm_arch_irqfd_route_changed(struct kvm_kernel_irq_routing_entry *old,
include/linux/kvm_host.h +3 -3
@@ -2406,9 +2406,9 @@ void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *, struct irq_bypass_producer *);void kvm_arch_irq_bypass_stop (struct irq_bypass_consumer *);void kvm_arch_irq_bypass_start (struct irq_bypass_consumer *); -int kvm_arch_update_irqfd_routing (struct kvm_kernel_irqfd *irqfd, -struct kvm_kernel_irq_routing_entry *old, -struct kvm_kernel_irq_routing_entry *new); +void kvm_arch_update_irqfd_routing (struct kvm_kernel_irqfd *irqfd, +struct kvm_kernel_irq_routing_entry *old, +struct kvm_kernel_irq_routing_entry *new);bool kvm_arch_irqfd_route_changed (struct kvm_kernel_irq_routing_entry *,struct kvm_kernel_irq_routing_entry *);#endif /* CONFIG_HAVE_KVM_IRQ_BYPASS */
virt/kvm/eventfd.c +6 -9
@@ -285,11 +285,11 @@ void __attribute__((weak)) kvm_arch_irq_bypass_start( { } -int __weak kvm_arch_update_irqfd_routing (struct kvm_kernel_irqfd *irqfd, -struct kvm_kernel_irq_routing_entry *old, -struct kvm_kernel_irq_routing_entry *new) +void __weak kvm_arch_update_irqfd_routing (struct kvm_kernel_irqfd *irqfd, +struct kvm_kernel_irq_routing_entry *old, +struct kvm_kernel_irq_routing_entry *new) { -return 0 ; + } bool __attribute__((weak)) kvm_arch_irqfd_route_changed(
@@ -618,11 +618,8 @@ void kvm_irq_routing_update(struct kvm *kvm)#if IS_ENABLED (CONFIG_HAVE_KVM_IRQ_BYPASS )if (irqfd->producer && -kvm_arch_irqfd_route_changed (&old, &irqfd->irq_entry)) { -int ret =kvm_arch_update_irqfd_routing (irqfd, &old, &irqfd->irq_entry); - -WARN_ON (ret); - } +kvm_arch_irqfd_route_changed (&old, &irqfd->irq_entry)) +kvm_arch_update_irqfd_routing (irqfd, &old, &irqfd->irq_entry);#endif }
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 31/59 KVM: Fold kvm_arch_irqfd_route_changed() into kvm_arch_update_irqfd_routing()
In reply to Sean ChristophersonFold kvm_arch_irqfd_route_changed() into kvm_arch_update_irqfd_routing(). Calling arch code to know whether or not to call arch code is absurd.
--- arch/x86/kvm/irq.c | 15 +++++---------- include/linux/kvm_host.h | 2 -- virt/kvm/eventfd.c | 10 +--------- 3 files changed, 6 insertions(+), 21 deletions(-)
arch/x86/kvm/irq.c +5 -10
@@ -610,15 +610,10 @@ void kvm_arch_update_irqfd_routing(struct kvm_kernel_irqfd *irqfd, struct kvm_kernel_irq_routing_entry *old, struct kvm_kernel_irq_routing_entry *new) { +if (old->type ==KVM_IRQ_ROUTING_MSI && + new->type ==KVM_IRQ_ROUTING_MSI && + !memcmp (&old->msi, &new->msi,sizeof (new->msi))) +return ; +kvm_pi_update_irte (irqfd, new); } - -bool kvm_arch_irqfd_route_changed (struct kvm_kernel_irq_routing_entry *old, -struct kvm_kernel_irq_routing_entry *new) -{ -if (old->type !=KVM_IRQ_ROUTING_MSI || - new->type !=KVM_IRQ_ROUTING_MSI ) -return true; - -return !!memcmp (&old->msi, &new->msi,sizeof (new->msi)); -}
include/linux/kvm_host.h +0 -2
@@ -2409,8 +2409,6 @@ void kvm_arch_irq_bypass_start(struct irq_bypass_consumer *);void kvm_arch_update_irqfd_routing (struct kvm_kernel_irqfd *irqfd,struct kvm_kernel_irq_routing_entry *old,struct kvm_kernel_irq_routing_entry *new); -bool kvm_arch_irqfd_route_changed (struct kvm_kernel_irq_routing_entry *, -struct kvm_kernel_irq_routing_entry *);#endif /* CONFIG_HAVE_KVM_IRQ_BYPASS */ #ifdef CONFIG_HAVE_KVM_INVALID_WAKEUPS
virt/kvm/eventfd.c +1 -9
@@ -291,13 +291,6 @@ void __weak kvm_arch_update_irqfd_routing(struct kvm_kernel_irqfd *irqfd, { } - -bool __attribute__((weak))kvm_arch_irqfd_route_changed ( -struct kvm_kernel_irq_routing_entry *old, -struct kvm_kernel_irq_routing_entry *new) -{ -return true; -}#endif static int
@@ -617,8 +610,7 @@ void kvm_irq_routing_update(struct kvm *kvm)irqfd_update (kvm, irqfd);#if IS_ENABLED (CONFIG_HAVE_KVM_IRQ_BYPASS ) -if (irqfd->producer && -kvm_arch_irqfd_route_changed (&old, &irqfd->irq_entry)) +if (irqfd->producer)kvm_arch_update_irqfd_routing (irqfd, &old, &irqfd->irq_entry);#endif }
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 32/59 KVM: x86: Track irq_bypass_vcpu in common x86 code
In reply to Sean ChristophersonTrack the vCPU that is being targeted for IRQ bypass, a.k.a. for a posted IRQ, in common x86 code. This will allow for additional consolidation of the SVM and VMX code.
--- arch/x86/kvm/irq.c | 7 ++++++- arch/x86/kvm/svm/avic.c | 4 ---- 2 files changed, 6 insertions(+), 5 deletions(-)
arch/x86/kvm/irq.c +6 -1
@@ -544,8 +544,13 @@ static int kvm_pi_update_irte(struct kvm_kernel_irqfd *irqfd, r =kvm_x86_call (pi_update_irte)(irqfd, irqfd->kvm, host_irq, irqfd->gsi, vcpu, irq.vector); -if (r) +if (r) { +WARN_ON_ONCE (irqfd->irq_bypass_vcpu && !vcpu); + irqfd->irq_bypass_vcpu =NULL ;return r; + } + + irqfd->irq_bypass_vcpu = vcpu;trace_kvm_pi_irte_update (host_irq, vcpu, irqfd->gsi, irq.vector, !!vcpu);return 0 ;
arch/x86/kvm/svm/avic.c +0 -4
@@ -767,22 +767,18 @@ static void svm_ir_list_del(struct kvm_kernel_irqfd *irqfd)spin_lock_irqsave (&to_svm (vcpu)->ir_list_lock, flags);list_del (&irqfd->vcpu_list);spin_unlock_irqrestore (&to_svm (vcpu)->ir_list_lock, flags); - - irqfd->irq_bypass_vcpu =NULL ; }static int svm_ir_list_add (struct vcpu_svm *svm,struct kvm_kernel_irqfd *irqfd,struct amd_iommu_pi_data *pi) { -struct kvm_vcpu *vcpu = &svm->vcpu;unsigned long flags;u64 entry;if (WARN_ON_ONCE (!pi->ir_data))return -EINVAL ; - irqfd->irq_bypass_vcpu = vcpu; irqfd->irq_bypass_data = pi->ir_data;spin_lock_irqsave (&svm->ir_list_lock, flags);
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 33/59 KVM: x86: Skip IOMMU IRTE updates if there's no old or new vCPU being targeted
In reply to Sean ChristophersonDon't "reconfigure" an IRTE into host controlled mode when it's already in the state, i.e. if KVM's GSI routing changes but the IRQ wasn't and still isn't being posted to a vCPU.
--- arch/x86/kvm/irq.c | 3 +++ 1 file changed, 3 insertions(+)
arch/x86/kvm/irq.c +3 -0
@@ -542,6 +542,9 @@ static int kvm_pi_update_irte(struct kvm_kernel_irqfd *irqfd, vcpu =NULL ; } +if (!irqfd->irq_bypass_vcpu && !vcpu) +return 0 ; + r =kvm_x86_call (pi_update_irte)(irqfd, irqfd->kvm, host_irq, irqfd->gsi, vcpu, irq.vector);if (r) {
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 34/59 KVM: x86: Don't update IRTE entries when old and new routes were !MSI
In reply to Sean ChristophersonSkip the entirety of IRTE updates on a GSI routing change if neither the old nor the new routing is for an MSI, i.e. if the neither routing setup allows for posting to a vCPU. If the IRTE isn't already host controlled, KVM has bigger problems.
--- arch/x86/kvm/irq.c | 4 ++++ 1 file changed, 4 insertions(+)
arch/x86/kvm/irq.c +4 -0
@@ -618,6 +618,10 @@ void kvm_arch_update_irqfd_routing(struct kvm_kernel_irqfd *irqfd, struct kvm_kernel_irq_routing_entry *old, struct kvm_kernel_irq_routing_entry *new) { +if (new->type !=KVM_IRQ_ROUTING_MSI && + old->type !=KVM_IRQ_ROUTING_MSI ) +return ; +if (old->type ==KVM_IRQ_ROUTING_MSI && new->type ==KVM_IRQ_ROUTING_MSI && !memcmp (&old->msi, &new->msi,sizeof (new->msi)))
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 35/59 KVM: SVM: Revert IRTE to legacy mode if IOMMU doesn't provide IR metadata
In reply to Sean ChristophersonRevert the IRTE back to remapping mode if the AMD IOMMU driver mucks up and doesn't provide the necessary metadata. Returning an error up the stack without actually handling the error is useless and confusing.
--- arch/x86/kvm/svm/avic.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-)
arch/x86/kvm/svm/avic.c +15 -8
@@ -769,16 +769,13 @@ static void svm_ir_list_del(struct kvm_kernel_irqfd *irqfd)spin_unlock_irqrestore (&to_svm (vcpu)->ir_list_lock, flags); } -static int svm_ir_list_add (struct vcpu_svm *svm, -struct kvm_kernel_irqfd *irqfd, -struct amd_iommu_pi_data *pi) +static void svm_ir_list_add (struct vcpu_svm *svm, +struct kvm_kernel_irqfd *irqfd, +struct amd_iommu_pi_data *pi) {unsigned long flags;u64 entry; -if (WARN_ON_ONCE (!pi->ir_data)) -return -EINVAL ; - irqfd->irq_bypass_data = pi->ir_data;spin_lock_irqsave (&svm->ir_list_lock, flags);
@@ -796,7 +793,6 @@ static int svm_ir_list_add(struct vcpu_svm *svm,list_add (&irqfd->vcpu_list, &svm->ir_list);spin_unlock_irqrestore (&svm->ir_list_lock, flags); -return 0 ; } int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,
@@ -833,6 +829,16 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,if (ret)return ret; +/* +* Revert to legacy mode if the IOMMU didn't provide metadata +* for the IRTE, which KVM needs to keep the IRTE up-to-date, +* e.g. if the vCPU is migrated or AVIC is disabled. +*/ +if (WARN_ON_ONCE (!pi_data.ir_data)) { +irq_set_vcpu_affinity (host_irq,NULL ); +return -EIO ; + } +/** * Here, we successfully setting up vcpu affinity in * IOMMU guest mode. Now, we need to store the posted
@@ -840,7 +846,8 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,* we can reference to them directly when we update vcpu * scheduling information in IOMMU irte. */ -return svm_ir_list_add (to_svm (vcpu), irqfd, &pi_data); +svm_ir_list_add (to_svm (vcpu), irqfd, &pi_data); +return 0 ; }return irq_set_vcpu_affinity (host_irq,NULL ); }
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 36/59 KVM: SVM: Take and hold ir_list_lock across IRTE updates in IOMMU
In reply to Sean ChristophersonNow that svm_ir_list_add() isn't overloaded with all manner of weird things, fold it into avic_pi_update_irte(), and more importantly take ir_list_lock across the irq_set_vcpu_affinity() calls to ensure the info that's shoved into the IRTE is fresh. While preemption (and IRQs) is disabled on the task performing the IRTE update, thanks to irqfds.lock, that task doesn't hold the vCPU's mutex, i.e. preemption being disabled is irrelevant.
--- arch/x86/kvm/svm/avic.c | 55 +++++++++++++++++------------------------ 1 file changed, 22 insertions(+), 33 deletions(-)
arch/x86/kvm/svm/avic.c +22 -33
@@ -769,32 +769,6 @@ static void svm_ir_list_del(struct kvm_kernel_irqfd *irqfd)spin_unlock_irqrestore (&to_svm (vcpu)->ir_list_lock, flags); } -static void svm_ir_list_add (struct vcpu_svm *svm, -struct kvm_kernel_irqfd *irqfd, -struct amd_iommu_pi_data *pi) -{ -unsigned long flags; -u64 entry; - - irqfd->irq_bypass_data = pi->ir_data; - -spin_lock_irqsave (&svm->ir_list_lock, flags); - -/* -* Update the target pCPU for IOMMU doorbells if the vCPU is running. -* If the vCPU is NOT running, i.e. is blocking or scheduled out, KVM -* will update the pCPU info when the vCPU awkened and/or scheduled in. -* See also avic_vcpu_load(). -*/ - entry = svm->avic_physical_id_entry; -if (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ) -amd_iommu_update_ga (entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK , - true, pi->ir_data); - -list_add (&irqfd->vcpu_list, &svm->ir_list); -spin_unlock_irqrestore (&svm->ir_list_lock, flags); -} -int avic_pi_update_irte (struct kvm_kernel_irqfd *irqfd,struct kvm *kvm,unsigned int host_irq,uint32_t guest_irq,struct kvm_vcpu *vcpu,u32 vector)
@@ -823,8 +797,18 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, .vapic_addr = avic_get_backing_page_address(to_svm(vcpu)), .vector = vector, }; +struct vcpu_svm *svm =to_svm (vcpu); +u64 entry;int ret; +/* +* Prevent the vCPU from being scheduled out or migrated until +* the IRTE is updated and its metadata has been added to the +* list of IRQs being posted to the vCPU, to ensure the IRTE +* isn't programmed with stale pCPU/IsRunning information. +*/ +guard (spinlock_irqsave)(&svm->ir_list_lock); + ret =irq_set_vcpu_affinity (host_irq, &pi_data);if (ret)return ret;
@@ -839,14 +823,19 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,return -EIO ; } -/** -* Here, we successfully setting up vcpu affinity in -* IOMMU guest mode. Now, we need to store the posted -* interrupt information in a per-vcpu ir_list so that -* we can reference to them directly when we update vcpu -* scheduling information in IOMMU irte. +/* +* Update the target pCPU for IOMMU doorbells if the vCPU is +* running. If the vCPU is NOT running, i.e. is blocking or +* scheduled out, KVM will update the pCPU info when the vCPU +* is awakened and/or scheduled in. See also avic_vcpu_load(). */ -svm_ir_list_add (to_svm (vcpu), irqfd, &pi_data); + entry = svm->avic_physical_id_entry; +if (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ) +amd_iommu_update_ga (entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK , + true, pi_data.ir_data); + + irqfd->irq_bypass_data = pi_data.ir_data; +list_add (&irqfd->vcpu_list, &svm->ir_list);return 0 ; }return irq_set_vcpu_affinity (host_irq,NULL );
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 37/59 iommu/amd: Document which IRTE fields amd_iommu_update_ga() can modify
In reply to Sean ChristophersonAdd a comment to amd_iommu_update_ga() to document what fields it can safely modify without issuing an invalidation of the IRTE, and to explain its role in keeping GA IRTEs up-to-date.
Per page 93 of the IOMMU spec dated Feb 2025:
When virtual interrupts are enabled by setting MMIO Offset 0018h[GAEn] and IRTE[GuestMode=1], IRTE[IsRun], IRTE[Destination], and if present IRTE[GATag], are not cached by the IOMMU. Modifications to these fields do not require an invalidation of the Interrupt Remapping Table.
--- drivers/iommu/amd/iommu.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
drivers/iommu/amd/iommu.c +12 -0
@@ -3986,6 +3986,18 @@ int amd_iommu_create_irq_domain(struct amd_iommu *iommu)return 0 ; } +/* +* Update the pCPU information for an IRTE that is configured to post IRQs to +* a vCPU, without issuing an IOMMU invalidation for the IRTE. +* +* This API is intended to be used when a vCPU is scheduled in/out (or stops +* running for any reason), to do a fast update of IsRun and (conditionally) +* Destination. +* +* Per the IOMMU spec, the Destination, IsRun, and GATag fields are not cached +* and thus don't require an invalidation to ensure the IOMMU consumes fresh +* information. +*/ int amd_iommu_update_ga (int cpu,bool is_run,void *data) {struct amd_ir_data *ir_data = (struct amd_ir_data *)data;
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 38/59 iommu/amd: KVM: SVM: Infer IsRun from validity of pCPU destination
In reply to Sean ChristophersonInfer whether or not a vCPU should be marked running from the validity of the pCPU on which it is running. amd_iommu_update_ga() already skips the IRTE update if the pCPU is invalid, i.e. passing %true for is_run with an invalid pCPU would be a blatant and egregrious KVM bug.
--- arch/x86/kvm/svm/avic.c | 11 +++++------ drivers/iommu/amd/iommu.c | 14 +++++++++----- include/linux/amd-iommu.h | 6 ++---- 3 files changed, 16 insertions(+), 15 deletions(-)
arch/x86/kvm/svm/avic.c +5 -6
@@ -832,7 +832,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, entry = svm->avic_physical_id_entry;if (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK )amd_iommu_update_ga (entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK , - true, pi_data.ir_data); + pi_data.ir_data); irqfd->irq_bypass_data = pi_data.ir_data;list_add (&irqfd->vcpu_list, &svm->ir_list);
@@ -841,8 +841,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,return irq_set_vcpu_affinity (host_irq,NULL ); } -static inline int -avic_update_iommu_vcpu_affinity (struct kvm_vcpu *vcpu,int cpu,bool r) +static inline int avic_update_iommu_vcpu_affinity (struct kvm_vcpu *vcpu,int cpu) {int ret =0 ;struct vcpu_svm *svm =to_svm (vcpu);
@@ -861,7 +860,7 @@ avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu, bool r)return 0 ;list_for_each_entry (irqfd, &svm->ir_list, vcpu_list) { - ret =amd_iommu_update_ga (cpu, r, irqfd->irq_bypass_data); + ret =amd_iommu_update_ga (cpu, irqfd->irq_bypass_data);if (ret)return ret; }
@@ -923,7 +922,7 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu)WRITE_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id], entry); -avic_update_iommu_vcpu_affinity (vcpu, h_physical_id, true); +avic_update_iommu_vcpu_affinity (vcpu, h_physical_id);spin_unlock_irqrestore (&svm->ir_list_lock, flags); }
@@ -963,7 +962,7 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu)*/ spin_lock_irqsave (&svm->ir_list_lock, flags); -avic_update_iommu_vcpu_affinity (vcpu,-1 ,0 ); +avic_update_iommu_vcpu_affinity (vcpu,-1 ); entry &= ~AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ; svm->avic_physical_id_entry = entry;
drivers/iommu/amd/iommu.c +9 -5
@@ -3990,15 +3990,17 @@ int amd_iommu_create_irq_domain(struct amd_iommu *iommu)* Update the pCPU information for an IRTE that is configured to post IRQs to * a vCPU, without issuing an IOMMU invalidation for the IRTE. * -* This API is intended to be used when a vCPU is scheduled in/out (or stops -* running for any reason), to do a fast update of IsRun and (conditionally) -* Destination. +* If the vCPU is associated with a pCPU (@cpu >= 0), configure the Destination +* with the pCPU's APIC ID and set IsRun, else clear IsRun. I.e. treat vCPUs +* that are associated with a pCPU as running. This API is intended to be used +* when a vCPU is scheduled in/out (or stops running for any reason), to do a +* fast update of IsRun and (conditionally) Destination. * * Per the IOMMU spec, the Destination, IsRun, and GATag fields are not cached * and thus don't require an invalidation to ensure the IOMMU consumes fresh * information. */ -int amd_iommu_update_ga (int cpu,bool is_run,void *data) +int amd_iommu_update_ga (int cpu,void *data) {struct amd_ir_data *ir_data = (struct amd_ir_data *)data;struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -4015,8 +4017,10 @@ int amd_iommu_update_ga(int cpu, bool is_run, void *data)APICID_TO_IRTE_DEST_LO (cpu); entry->hi.fields.destination =APICID_TO_IRTE_DEST_HI (cpu); + entry->lo.fields_vapic.is_run = true; + }else { + entry->lo.fields_vapic.is_run = false; } - entry->lo.fields_vapic.is_run = is_run;return __modify_irte_ga (ir_data->iommu, ir_data->irq_2_irte.devid, ir_data->irq_2_irte.index, entry);
include/linux/amd-iommu.h +2 -4
@@ -30,8 +30,7 @@ static inline void amd_iommu_detect(void) { }/* IOMMU AVIC Function */ extern int amd_iommu_register_ga_log_notifier (int (*notifier)(u32 )); -extern int -amd_iommu_update_ga (int cpu,bool is_run,void *data); +extern int amd_iommu_update_ga (int cpu,void *data);extern int amd_iommu_activate_guest_mode (void *data);extern int amd_iommu_deactivate_guest_mode (void *data);
@@ -44,8 +43,7 @@ amd_iommu_register_ga_log_notifier(int (*notifier)(u32))return 0 ; } -static inline int -amd_iommu_update_ga (int cpu,bool is_run,void *data) +static inline int amd_iommu_update_ga (int cpu,void *data) {return 0 ; }
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Re: [PATCH v2 38/59] iommu/amd: KVM: SVM: Infer IsRun from validity of pCPU destination
Quoted 27 lines · Sean Christopherson
Infer whether or not a vCPU should be marked running from the validity of the pCPU on which it is running.
amd_iommu_update_ga()already skips the IRTE update if the pCPU is invalid, i.e. passing %true foris_runwith an invalid pCPU would be a blatant and egregrious KVM bug.Tested-by Sairaj Kodilkar <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/svm/avic.c | 11 +++++------ drivers/iommu/amd/iommu.c | 14 +++++++++----- include/linux/amd-iommu.h | 6 ++---- 3 files changed, 16 insertions(+), 15 deletions(-)arch/x86/kvm/svm/avic.c +1
-1@@ -832,7 +832,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, entry = svm->avic_physical_id_entry;if (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK )amd_iommu_update_ga (entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK , - true, pi_data.ir_data); + pi_data.ir_data); irqfd->irq_bypass_data = pi_data.ir_data;list_add (&irqfd->vcpu_list, &svm->ir_list);
arch/x86/kvm/svm/avic.c int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, · line 845
Quoted 7 lines
arch/x86/kvm/svm/avic.c +1
-2@@ -841,8 +841,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,return irq_set_vcpu_affinity (host_irq,NULL ); } -static inline int -avic_update_iommu_vcpu_affinity (struct kvm_vcpu *vcpu,int cpu,bool r) +static inline int avic_update_iommu_vcpu_affinity (struct kvm_vcpu *vcpu,int cpu) {
Hi sean
What if define cpu as "unsigned int" instead of "int" and use nr_cpu_ids as invalid cpu id ? I see that it is common in the other subsystems to use nr_cpu_ids instead of -1.
Thanks
Sairaj
Sean Christopherson
maintainerRe: [PATCH v2 38/59] iommu/amd: KVM: SVM: Infer IsRun from validity of pCPU destination
Quoted 36 lines · Sairaj Kodilkar
Quoted 26 lines · Sean Christopherson
Infer whether or not a vCPU should be marked running from the validity of the pCPU on which it is running.
amd_iommu_update_ga()already skips the IRTE update if the pCPU is invalid, i.e. passing %true foris_runwith an invalid pCPU would be a blatant and egregrious KVM bug.Tested-by Sairaj Kodilkar <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/svm/avic.c | 11 +++++------ drivers/iommu/amd/iommu.c | 14 +++++++++----- include/linux/amd-iommu.h | 6 ++---- 3 files changed, 16 insertions(+), 15 deletions(-)arch/x86/kvm/svm/avic.c +1
-1@@ -832,7 +832,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, entry = svm->avic_physical_id_entry;if (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK )amd_iommu_update_ga (entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK , - true, pi_data.ir_data); + pi_data.ir_data); irqfd->irq_bypass_data = pi_data.ir_data;list_add (&irqfd->vcpu_list, &svm->ir_list);arch/x86/kvm/svm/avic.c int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, · line 844
Quoted 6 lines
arch/x86/kvm/svm/avic.c +1
-2@@ -841,8 +841,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,return irq_set_vcpu_affinity (host_irq,NULL ); } -static inline int -avic_update_iommu_vcpu_affinity (struct kvm_vcpu *vcpu,int cpu,bool r) +static inline int avic_update_iommu_vcpu_affinity (struct kvm_vcpu *vcpu,int cpu) {Hi sean
Quoted 3 lines
What if define cpu as "unsigned int" instead of "int" and use
nr_cpu_idsas invalid cpu id ? I see that it is common in the other subsystems to usenr_cpu_idsinstead of -1.
My vote is for -1, as it makes the KVM side of things much more intuitive
E.g. this is pretty obviously saying "no associated CPU"
avic_update_iommu_vcpu_affinity(vcpu, -1);
whereas this honestly just looks a bit weird.
avic_update_iommu_vcpu_affinity(vcpu, nr_cpu_ids);
It also requires knowing what cpu numbers are strictly packed in the kernel, i.e. that nr_cpu_ids is guaranteed to be greater than the cpu numbers themselves (e.g. the the kernel can't have nr_cpu_ids=2 with CPU0 and CPU2 being the two CPUs).
I also don't love that nr_cpu_ids is __read_mostly, i.e. isn't const post-boot (though at a glance, it looks like it could be __ro_after_init on x86 at least).
Sean Christopherson
maintainerPATCH v2 39/59 iommu/amd: Factor out helper for manipulating IRTE GA/CPU info
In reply to Sean ChristophersonSplit the guts of amd_iommu_update_ga() to a dedicated helper so that the logic can be shared with flows that put the IRTE into posted mode.
Opportunistically move amd_iommu_update_ga() and its new helper above amd_iommu_activate_guest_mode() so that it's all co-located.
--- drivers/iommu/amd/iommu.c | 87 +++++++++++++++++++++------------------ 1 file changed, 46 insertions(+), 41 deletions(-)
drivers/iommu/amd/iommu.c +46 -41
@@ -3804,6 +3804,52 @@ static const struct irq_domain_ops amd_ir_domain_ops = { .deactivate = irq_remapping_deactivate, }; +static void __amd_iommu_update_ga (struct irte_ga *entry,int cpu) +{ +if (cpu >=0 ) { + entry->lo.fields_vapic.destination = +APICID_TO_IRTE_DEST_LO (cpu); + entry->hi.fields.destination = +APICID_TO_IRTE_DEST_HI (cpu); + entry->lo.fields_vapic.is_run = true; + }else { + entry->lo.fields_vapic.is_run = false; + } +} + +/* +* Update the pCPU information for an IRTE that is configured to post IRQs to +* a vCPU, without issuing an IOMMU invalidation for the IRTE. +* +* If the vCPU is associated with a pCPU (@cpu >= 0), configure the Destination +* with the pCPU's APIC ID and set IsRun, else clear IsRun. I.e. treat vCPUs +* that are associated with a pCPU as running. This API is intended to be used +* when a vCPU is scheduled in/out (or stops running for any reason), to do a +* fast update of IsRun and (conditionally) Destination. +* +* Per the IOMMU spec, the Destination, IsRun, and GATag fields are not cached +* and thus don't require an invalidation to ensure the IOMMU consumes fresh +* information. +*/ +int amd_iommu_update_ga (int cpu,void *data) +{ +struct amd_ir_data *ir_data = (struct amd_ir_data *)data; +struct irte_ga *entry = (struct irte_ga *) ir_data->entry; + +if (!AMD_IOMMU_GUEST_IR_VAPIC (amd_iommu_guest_ir) || + !entry || !entry->lo.fields_vapic.guest_mode) +return 0 ; + +if (!ir_data->iommu) +return -ENODEV ; + +__amd_iommu_update_ga (entry, cpu); + +return __modify_irte_ga (ir_data->iommu, ir_data->irq_2_irte.devid, + ir_data->irq_2_irte.index, entry); +} +EXPORT_SYMBOL (amd_iommu_update_ga); +int amd_iommu_activate_guest_mode (void *data) {struct amd_ir_data *ir_data = (struct amd_ir_data *)data;
@@ -3985,45 +4031,4 @@ int amd_iommu_create_irq_domain(struct amd_iommu *iommu) return 0; } - -/* -* Update the pCPU information for an IRTE that is configured to post IRQs to -* a vCPU, without issuing an IOMMU invalidation for the IRTE. -* -* If the vCPU is associated with a pCPU (@cpu >= 0), configure the Destination -* with the pCPU's APIC ID and set IsRun, else clear IsRun. I.e. treat vCPUs -* that are associated with a pCPU as running. This API is intended to be used -* when a vCPU is scheduled in/out (or stops running for any reason), to do a -* fast update of IsRun and (conditionally) Destination. -* -* Per the IOMMU spec, the Destination, IsRun, and GATag fields are not cached -* and thus don't require an invalidation to ensure the IOMMU consumes fresh -* information. -*/ -int amd_iommu_update_ga (int cpu,void *data) -{ -struct amd_ir_data *ir_data = (struct amd_ir_data *)data; -struct irte_ga *entry = (struct irte_ga *) ir_data->entry; - -if (!AMD_IOMMU_GUEST_IR_VAPIC (amd_iommu_guest_ir) || - !entry || !entry->lo.fields_vapic.guest_mode) -return 0 ; - -if (!ir_data->iommu) -return -ENODEV ; - -if (cpu >=0 ) { - entry->lo.fields_vapic.destination = -APICID_TO_IRTE_DEST_LO (cpu); - entry->hi.fields.destination = -APICID_TO_IRTE_DEST_HI (cpu); - entry->lo.fields_vapic.is_run = true; - }else { - entry->lo.fields_vapic.is_run = false; - } - -return __modify_irte_ga (ir_data->iommu, ir_data->irq_2_irte.devid, - ir_data->irq_2_irte.index, entry); -} -EXPORT_SYMBOL (amd_iommu_update_ga);#endif
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 40/59 iommu/amd: KVM: SVM: Set pCPU info in IRTE when setting vCPU affinity
In reply to Sean ChristophersonNow that setting vCPU affinity is guarded with ir_list_lock, i.e. now that avic_physical_id_entry can be safely accessed, set the pCPU info straight-away when setting vCPU affinity. Putting the IRTE into posted mode, and then immediately updating the IRTE a second time if the target vCPU is running is wasteful and confusing.
This also fixes a flaw where a posted IRQ that arrives between putting the IRTE into guest_mode and setting the correct destination could cause the IOMMU to ring the doorbell on the wrong pCPU.
--- arch/x86/include/asm/irq_remapping.h | 1 + arch/x86/kvm/svm/avic.c | 26 ++++++++++++++------------ drivers/iommu/amd/iommu.c | 6 ++++-- include/linux/amd-iommu.h | 4 ++-- 4 files changed, 21 insertions(+), 16 deletions(-)
arch/x86/include/asm/irq_remapping.h +1 -0
@@ -35,6 +35,7 @@ struct amd_iommu_pi_data {u64 vapic_addr;/* Physical address of the vCPU's vAPIC. */ u32 ga_tag;u32 vector;/* Guest vector of the interrupt */ +int cpu;bool is_guest_mode;void *ir_data; };
arch/x86/kvm/svm/avic.c +14 -12
@@ -726,6 +726,7 @@ void avic_apicv_post_state_restore(struct kvm_vcpu *vcpu)static int avic_set_pi_irte_mode (struct kvm_vcpu *vcpu,bool activate) { +int apic_id =kvm_cpu_get_apicid (vcpu->cpu);int ret =0 ;unsigned long flags;struct vcpu_svm *svm =to_svm (vcpu);
@@ -745,7 +746,7 @@ static int avic_set_pi_irte_mode(struct kvm_vcpu *vcpu, bool activate)list_for_each_entry (irqfd, &svm->ir_list, vcpu_list) {if (activate) - ret =amd_iommu_activate_guest_mode (irqfd->irq_bypass_data); + ret =amd_iommu_activate_guest_mode (irqfd->irq_bypass_data, apic_id);else ret =amd_iommu_deactivate_guest_mode (irqfd->irq_bypass_data);if (ret)
@@ -809,6 +810,18 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,*/ guard (spinlock_irqsave)(&svm->ir_list_lock); +/* +* Update the target pCPU for IOMMU doorbells if the vCPU is +* running. If the vCPU is NOT running, i.e. is blocking or +* scheduled out, KVM will update the pCPU info when the vCPU +* is awakened and/or scheduled in. See also avic_vcpu_load(). +*/ + entry = svm->avic_physical_id_entry; +if (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ) + pi_data.cpu = entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK ; +else + pi_data.cpu =-1 ; + ret =irq_set_vcpu_affinity (host_irq, &pi_data);if (ret)return ret;
@@ -823,17 +836,6 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,return -EIO ; } -/* -* Update the target pCPU for IOMMU doorbells if the vCPU is -* running. If the vCPU is NOT running, i.e. is blocking or -* scheduled out, KVM will update the pCPU info when the vCPU -* is awakened and/or scheduled in. See also avic_vcpu_load(). -*/ - entry = svm->avic_physical_id_entry; -if (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ) -amd_iommu_update_ga (entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK , - pi_data.ir_data); - irqfd->irq_bypass_data = pi_data.ir_data;list_add (&irqfd->vcpu_list, &svm->ir_list);return 0 ;
drivers/iommu/amd/iommu.c +4 -2
@@ -3850,7 +3850,7 @@ int amd_iommu_update_ga(int cpu, void *data) }EXPORT_SYMBOL (amd_iommu_update_ga); -int amd_iommu_activate_guest_mode (void *data) +int amd_iommu_activate_guest_mode (void *data,int cpu) {struct amd_ir_data *ir_data = (struct amd_ir_data *)data;struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -3871,6 +3871,8 @@ int amd_iommu_activate_guest_mode(void *data) entry->hi.fields.vector = ir_data->ga_vector; entry->lo.fields_vapic.ga_tag = ir_data->ga_tag; +__amd_iommu_update_ga (entry, cpu); +return modify_irte_ga (ir_data->iommu, ir_data->irq_2_irte.devid, ir_data->irq_2_irte.index, entry); }
@@ -3937,7 +3939,7 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *info) ir_data->ga_root_ptr = (pi_data->vapic_addr >>12 ); ir_data->ga_vector = pi_data->vector; ir_data->ga_tag = pi_data->ga_tag; - ret =amd_iommu_activate_guest_mode (ir_data); + ret =amd_iommu_activate_guest_mode (ir_data, pi_data->cpu); }else { ret =amd_iommu_deactivate_guest_mode (ir_data); }
include/linux/amd-iommu.h +2 -2
@@ -32,7 +32,7 @@ extern int amd_iommu_register_ga_log_notifier(int (*notifier)(u32));extern int amd_iommu_update_ga (int cpu,void *data); -extern int amd_iommu_activate_guest_mode (void *data); +extern int amd_iommu_activate_guest_mode (void *data,int cpu);extern int amd_iommu_deactivate_guest_mode (void *data);#else /* defined(CONFIG_AMD_IOMMU) && defined(CONFIG_IRQ_REMAP) */
@@ -48,7 +48,7 @@ static inline int amd_iommu_update_ga(int cpu, void *data)return 0 ; } -static inline int amd_iommu_activate_guest_mode (void *data) +static inline int amd_iommu_activate_guest_mode (void *data,int cpu) {return 0 ; }
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 41/59 iommu/amd: KVM: SVM: Add IRTE metadata to affined vCPU's list if AVIC is inhibited
In reply to Sean ChristophersonIf an IRQ can be posted to a vCPU, but AVIC is currently inhibited on the vCPU, go through the dance of "affining" the IRTE to the vCPU, but leave the actual IRTE in remapped mode. KVM already handles the case where AVIC is inhibited => uninhibited with posted IRQs (see avic_set_pi_irte_mode()), but doesn't handle the scenario where a postable IRQ comes along while AVIC is inhibited.
--- arch/x86/kvm/svm/avic.c | 16 ++++++---------- drivers/iommu/amd/iommu.c | 5 ++++- 2 files changed, 10 insertions(+), 11 deletions(-)
arch/x86/kvm/svm/avic.c +6 -10
@@ -780,21 +780,17 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,*/ svm_ir_list_del (irqfd); -/** -* Here, we setup with legacy mode in the following cases: -* 1. When cannot target interrupt to a specific vcpu. -* 2. Unsetting posted interrupt. -* 3. APIC virtualization is disabled for the vcpu. -* 4. IRQ has incompatible delivery mode (SMI, INIT, etc) -*/ -if (vcpu &&kvm_vcpu_apicv_active (vcpu)) { +if (vcpu) {/* -* Try to enable guest_mode in IRTE. +* Try to enable guest_mode in IRTE, unless AVIC is inhibited, +* in which case configure the IRTE for legacy mode, but track +* the IRTE metadata so that it can be converted to guest mode +* if AVIC is enabled/uninhibited in the future. */ struct amd_iommu_pi_data pi_data = { .ga_tag =AVIC_GATAG (to_kvm_svm (kvm)->avic_vm_id, vcpu->vcpu_id), - .is_guest_mode = true, + .is_guest_mode =kvm_vcpu_apicv_active (vcpu), .vapic_addr =avic_get_backing_page_address (to_svm (vcpu)), .vector = vector, };
drivers/iommu/amd/iommu.c +4 -1
@@ -3939,7 +3939,10 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *info) ir_data->ga_root_ptr = (pi_data->vapic_addr >>12 ); ir_data->ga_vector = pi_data->vector; ir_data->ga_tag = pi_data->ga_tag; - ret =amd_iommu_activate_guest_mode (ir_data, pi_data->cpu); +if (pi_data->is_guest_mode) + ret =amd_iommu_activate_guest_mode (ir_data, pi_data->cpu); +else + ret =amd_iommu_deactivate_guest_mode (ir_data); }else { ret =amd_iommu_deactivate_guest_mode (ir_data); }
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Re: [PATCH v2 41/59] iommu/amd: KVM: SVM: Add IRTE metadata to affined vCPU's list if AVIC is inhibited
drivers/iommu/amd/iommu.c static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *info) · line 3945
Quoted 13 lines · Sean Christopherson
drivers/iommu/amd/iommu.c +4
-1@@ -3939,7 +3939,10 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *info) ir_data->ga_root_ptr = (pi_data->vapic_addr >>12 ); ir_data->ga_vector = pi_data->vector; ir_data->ga_tag = pi_data->ga_tag; - ret =amd_iommu_activate_guest_mode (ir_data, pi_data->cpu); +if (pi_data->is_guest_mode) + ret =amd_iommu_activate_guest_mode (ir_data, pi_data->cpu); +else + ret =amd_iommu_deactivate_guest_mode (ir_data);
Hi Sean,
Why the extra nesting here ?
Its much more cleaner to do..
if (pi_data && pi_data->is_guest_mode) {
ir_data->ga_root_ptr = (pi_data->vapic_addr >> 12);
ir_data->ga_vector = pi_data->vector;
ir_data->ga_tag = pi_data->ga_tag;
ret = amd_iommu_activate_guest_mode(ir_data, pi_data->cpu);
} else {
ret = amd_iommu_deactivate_guest_mode(ir_data);
}Thanks
Sairaj Kodilkar
Quoted 3 lines
drivers/iommu/amd/iommu.c +0
-0}else { ret =amd_iommu_deactivate_guest_mode (ir_data); }
Sean Christopherson
maintainerRe: [PATCH v2 41/59] iommu/amd: KVM: SVM: Add IRTE metadata to affined vCPU's list if AVIC is inhibited
Quoted 20 lines · Sairaj Kodilkar
drivers/iommu/amd/iommu.c static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *info) · line 3945
Quoted 13 lines · Sean Christopherson
drivers/iommu/amd/iommu.c +4
-1@@ -3939,7 +3939,10 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *info) ir_data->ga_root_ptr = (pi_data->vapic_addr >>12 ); ir_data->ga_vector = pi_data->vector; ir_data->ga_tag = pi_data->ga_tag; - ret =amd_iommu_activate_guest_mode (ir_data, pi_data->cpu); +if (pi_data->is_guest_mode) + ret =amd_iommu_activate_guest_mode (ir_data, pi_data->cpu); +else + ret =amd_iommu_deactivate_guest_mode (ir_data);Hi Sean,
Why the extra nesting here ?
Its much more cleaner to do..
Quoted 8 lines
if (pi_data && pi_data->is_guest_mode) {
ir_data->ga_root_ptr = (pi_data->vapic_addr >> 12);
ir_data->ga_vector = pi_data->vector;
ir_data->ga_tag = pi_data->ga_tag;
ret = amd_iommu_activate_guest_mode(ir_data, pi_data->cpu);
} else {
ret = amd_iommu_deactivate_guest_mode(ir_data);
}Because the intent of the change (and the long-term code) is to affine/bind the vCPU to the IRTE metadata, while leaving the actual IRTE in remapped mode. I.e. connect the passed in pi_data (@info) to the the chip data:
pi_data->ir_data = ir_data;
and set the GA root, vector and tag in the chip data.
ir_data->ga_root_ptr = (pi_data->vapic_addr >> 12); ir_data->ga_vector = pi_data->vector; ir_data->ga_tag = pi_data->ga_tag;
That way if KVM enables AVIC, KVM can call amd_iommu_activate_guest_mode() to switch the IRTE to vAPIC mode.
If KVM doesn't bind to the IRTE, KVM would need to track all host IRQs (Linux's "virtual" IRQ numbers) that can be posted to the vCPU in order to active vAPIC mode. It would also require taking VM-wide locks in KVM in order to guarantee accurate IRQ routing information.
FWIW, I don't love that KVM essentially backdoors into the AMD IOMMU via amd_iommu_(de)activate_guest_mode(), but I also don't see a better alternative. E.g. on Intel, KVM just leaves the IRTE in posted mode, and relies on the notification vector IRQ to kick the vCPU into host mode so that KVM can manually process the PIR.
But that trick doesn't work as well on AMD, because the "guest isn't running" IRQ will hit whatever CPU is handling the IOMMU interrupts, not the CPU that's running the vCPU. I.e. it _could_ functionally be made to work, but it would likely yield pretty poor performance (and would require a decent amount of new KVM code).
Sean Christopherson
maintainerPATCH v2 42/59 KVM: SVM: Don't check for assigned device(s) when updating affinity
In reply to Sean ChristophersonDon't bother checking if a VM has an assigned device when updating AVIC vCPU affinity, querying ir_list is just as cheap and nothing prevents racing with changes in device assignment.
--- arch/x86/kvm/svm/avic.c | 3 --- 1 file changed, 3 deletions(-)
arch/x86/kvm/svm/avic.c +0 -3
@@ -847,9 +847,6 @@ static inline int avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpulockdep_assert_held (&svm->ir_list_lock); -if (!kvm_arch_has_assigned_device (vcpu->kvm)) -return 0 ; -/* * Here, we go through the per-vcpu ir_list to update all existing * interrupt remapping table entry targeting this vcpu.
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 43/59 KVM: SVM: Don't check for assigned device(s) when activating AVIC
In reply to Sean ChristophersonDon't short-circuit IRTE updating when (de)activating AVIC based on the VM having assigned devices, as nothing prevents AVIC (de)activation from racing with device (de)assignment. And from a performance perspective, bailing early when there is no assigned device doesn't add much, as ir_list_lock will never be contended if there's no assigned device.
--- arch/x86/kvm/svm/avic.c | 3 --- 1 file changed, 3 deletions(-)
arch/x86/kvm/svm/avic.c +0 -3
@@ -732,9 +732,6 @@ static int avic_set_pi_irte_mode(struct kvm_vcpu *vcpu, bool activate)struct vcpu_svm *svm =to_svm (vcpu);struct kvm_kernel_irqfd *irqfd; -if (!kvm_arch_has_assigned_device (vcpu->kvm)) -return 0 ; -/* * Here, we go through the per-vcpu ir_list to update all existing * interrupt remapping table entry targeting this vcpu.
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 44/59 KVM: SVM: WARN if (de)activating guest mode in IOMMU fails
In reply to Sean ChristophersonWARN if (de)activating "guest mode" for an IRTE entry fails as modifying an IRTE should only fail if KVM is buggy, e.g. has stale metadata.
--- arch/x86/kvm/svm/avic.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
arch/x86/kvm/svm/avic.c +5 -7
@@ -724,10 +724,9 @@ void avic_apicv_post_state_restore(struct kvm_vcpu *vcpu)avic_handle_ldr_update (vcpu); } -static int avic_set_pi_irte_mode (struct kvm_vcpu *vcpu,bool activate) +static void avic_set_pi_irte_mode (struct kvm_vcpu *vcpu,bool activate) {int apic_id =kvm_cpu_get_apicid (vcpu->cpu); -int ret =0 ;unsigned long flags;struct vcpu_svm *svm =to_svm (vcpu);struct kvm_kernel_irqfd *irqfd;
@@ -742,16 +741,15 @@ static int avic_set_pi_irte_mode(struct kvm_vcpu *vcpu, bool activate) goto out;list_for_each_entry (irqfd, &svm->ir_list, vcpu_list) { +void *data = irqfd->irq_bypass_data; +if (activate) - ret =amd_iommu_activate_guest_mode (irqfd->irq_bypass_data, apic_id); +WARN_ON_ONCE (amd_iommu_activate_guest_mode (data, apic_id));else - ret =amd_iommu_deactivate_guest_mode (irqfd->irq_bypass_data); -if (ret) -break ; +WARN_ON_ONCE (amd_iommu_deactivate_guest_mode (data)); } out:spin_unlock_irqrestore (&svm->ir_list_lock, flags); -return ret; }static void svm_ir_list_del (struct kvm_kernel_irqfd *irqfd)
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 45/59 KVM: SVM: Process all IRTEs on affinity change even if one update fails
In reply to Sean ChristophersonWhen updating IRTE GA fields, keep processing all other IRTEs if an update fails, as not updating later entries risks making a bad situation worse.
--- arch/x86/kvm/svm/avic.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
arch/x86/kvm/svm/avic.c +3 -5
@@ -849,12 +849,10 @@ static inline int avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpuif (list_empty (&svm->ir_list))return 0 ; -list_for_each_entry (irqfd, &svm->ir_list, vcpu_list) { +list_for_each_entry (irqfd, &svm->ir_list, vcpu_list) ret =amd_iommu_update_ga (cpu, irqfd->irq_bypass_data); -if (ret) -return ret; - } -return 0 ; + +return ret; }void avic_vcpu_load (struct kvm_vcpu *vcpu,int cpu)
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 46/59 KVM: SVM: WARN if updating IRTE GA fields in IOMMU fails
In reply to Sean ChristophersonWARN if updating GA information for an IRTE entry fails as modifying an IRTE should only fail if KVM is buggy, e.g. has stale metadata, and because returning an error that is always ignored is pointless.
--- arch/x86/kvm/svm/avic.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-)
arch/x86/kvm/svm/avic.c +3 -6
@@ -834,9 +834,8 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,return irq_set_vcpu_affinity (host_irq,NULL ); } -static inline int avic_update_iommu_vcpu_affinity (struct kvm_vcpu *vcpu,int cpu) +static void avic_update_iommu_vcpu_affinity (struct kvm_vcpu *vcpu,int cpu) { -int ret =0 ;struct vcpu_svm *svm =to_svm (vcpu);struct kvm_kernel_irqfd *irqfd;
@@ -847,12 +846,10 @@ static inline int avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu* interrupt remapping table entry targeting this vcpu. */ if (list_empty (&svm->ir_list)) -return 0 ; +return ;list_for_each_entry (irqfd, &svm->ir_list, vcpu_list) - ret =amd_iommu_update_ga (cpu, irqfd->irq_bypass_data); - -return ret; +WARN_ON_ONCE (amd_iommu_update_ga (cpu, irqfd->irq_bypass_data)); }void avic_vcpu_load (struct kvm_vcpu *vcpu,int cpu)
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 47/59 KVM: x86: Drop superfluous "has assigned device" check in kvm_pi_update_irte()
In reply to Sean ChristophersonDon't bother checking if the VM has an assigned device when updating IRTE entries. kvm_arch_irq_bypass_add_producer() explicitly increments the assigned device count, kvm_arch_irq_bypass_del_producer() explicitly decrements the count before invoking kvm_pi_update_irte(), and kvm_irq_routing_update() only updates IRTE entries if there's an active IRQ bypass producer.
--- arch/x86/kvm/irq.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
arch/x86/kvm/irq.c +1 -3
@@ -517,9 +517,7 @@ static int kvm_pi_update_irte(struct kvm_kernel_irqfd *irqfd,struct kvm_lapic_irq irq;int r; -if (!irqchip_in_kernel (kvm) || - !kvm_arch_has_irq_bypass () || - !kvm_arch_has_assigned_device (kvm)) +if (!irqchip_in_kernel (kvm) || !kvm_arch_has_irq_bypass ())return 0 ;if (entry && entry->type ==KVM_IRQ_ROUTING_MSI ) {
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 48/59 KVM: x86: WARN if IRQ bypass isn't supported in kvm_pi_update_irte()
In reply to Sean ChristophersonWARN if kvm_pi_update_irte() is reached without IRQ bypass support, as the code is only reachable if the VM already has an IRQ bypass producer (see kvm_irq_routing_update()), or from kvm_arch_irq_bypass_{add,del}_producer(), which, stating the obvious, are called if and only if KVM enables its IRQ bypass hooks.
--- arch/x86/kvm/irq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/irq.c +1 -1
@@ -517,7 +517,7 @@ static int kvm_pi_update_irte(struct kvm_kernel_irqfd *irqfd,struct kvm_lapic_irq irq;int r; -if (!irqchip_in_kernel (kvm) || !kvm_arch_has_irq_bypass ()) +if (!irqchip_in_kernel (kvm) ||WARN_ON_ONCE (!kvm_arch_has_irq_bypass ()))return 0 ;if (entry && entry->type ==KVM_IRQ_ROUTING_MSI ) {
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 49/59 KVM: x86: WARN if IRQ bypass routing is updated without in-kernel local APIC
In reply to Sean ChristophersonYell if kvm_pi_update_irte() is reached without an in-kernel local APIC, as kvm_arch_irqfd_allowed() should prevent attaching an irqfd and thus any and all postable IRQs to an APIC-less VM.
--- arch/x86/kvm/irq.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/irq.c +2 -2
@@ -517,8 +517,8 @@ static int kvm_pi_update_irte(struct kvm_kernel_irqfd *irqfd,struct kvm_lapic_irq irq;int r; -if (!irqchip_in_kernel (kvm) ||WARN_ON_ONCE (!kvm_arch_has_irq_bypass ())) -return 0 ; +if (WARN_ON_ONCE (!irqchip_in_kernel (kvm) || !kvm_arch_has_irq_bypass ())) +return -EINVAL ;if (entry && entry->type ==KVM_IRQ_ROUTING_MSI ) {kvm_set_msi_irq (kvm, entry, &irq);
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 50/59 KVM: SVM: WARN if ir_list is non-empty at vCPU free
In reply to Sean ChristophersonNow that AVIC IRTE tracking is in a mostly sane state, WARN if a vCPU is freed with ir_list entries, i.e. if KVM leaves a dangling IRTE.
Initialize the per-vCPU interrupt remapping list and its lock even if AVIC is disabled so that the WARN doesn't hit false positives (and so that KVM doesn't need to call into AVIC code for a simple sanity check).
--- arch/x86/kvm/svm/avic.c | 5 +++-- arch/x86/kvm/svm/svm.c | 2 ++ 2 files changed, 5 insertions(+), 2 deletions(-)
arch/x86/kvm/svm/avic.c +3 -2
@@ -704,6 +704,9 @@ int avic_init_vcpu(struct vcpu_svm *svm)int ret;struct kvm_vcpu *vcpu = &svm->vcpu; +INIT_LIST_HEAD (&svm->ir_list); +spin_lock_init (&svm->ir_list_lock); +if (!enable_apicv || !irqchip_in_kernel (vcpu->kvm))return 0 ;
@@ -711,8 +714,6 @@ int avic_init_vcpu(struct vcpu_svm *svm)if (ret)return ret; -INIT_LIST_HEAD (&svm->ir_list); -spin_lock_init (&svm->ir_list_lock); svm->dfr_reg =APIC_DFR_FLAT ;return ret;
arch/x86/kvm/svm/svm.c +2 -0
@@ -1497,6 +1497,8 @@ static void svm_vcpu_free(struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu); +WARN_ON_ONCE (!list_empty (&svm->ir_list)); +svm_leave_nested (vcpu);svm_free_nested (svm);
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 51/59 KVM: x86: Decouple device assignment from IRQ bypass
In reply to Sean ChristophersonUse a dedicated counter to track the number of IRQs that can utilize IRQ bypass instead of piggybacking the assigned device count. As evidenced by commit 2edd9cb79fb3 ("kvm: detect assigned device via irqbypass manager"), it's possible for a device to be able to post IRQs to a vCPU without said device being assigned to a VM.
Leave the calls to kvm_arch_{start,end}_assignment() alone for the moment to avoid regressing the MMIO stale data mitigation. KVM is abusing the assigned device count when applying mmio_stale_data_clear, and it's not at all clear if vDPA devices rely on this behavior. This will hopefully be cleaned up in the future, as the number of assigned devices is a terrible heuristic for detecting if a VM has access to host MMIO.
--- arch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 4 +++- arch/x86/kvm/irq.c | 9 ++++++++- arch/x86/kvm/vmx/main.c | 2 +- arch/x86/kvm/vmx/posted_intr.c | 16 ++++++++++------ arch/x86/kvm/vmx/posted_intr.h | 2 +- arch/x86/kvm/x86.c | 3 +-- 7 files changed, 25 insertions(+), 13 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -1
@@ -112,7 +112,7 @@ KVM_X86_OP_OPTIONAL(update_cpu_dirty_logging) KVM_X86_OP_OPTIONAL(vcpu_blocking) KVM_X86_OP_OPTIONAL(vcpu_unblocking) KVM_X86_OP_OPTIONAL(pi_update_irte) -KVM_X86_OP_OPTIONAL(pi_start_assignment) +KVM_X86_OP_OPTIONAL(pi_start_bypass) KVM_X86_OP_OPTIONAL(apicv_pre_state_restore) KVM_X86_OP_OPTIONAL(apicv_post_state_restore) KVM_X86_OP_OPTIONAL_RET0(dy_apicv_has_pending_interrupt)
arch/x86/include/asm/kvm_host.h +3 -1
@@ -1382,6 +1382,8 @@ struct kvm_arch {atomic_t noncoherent_dma_count;#define __KVM_HAVE_ARCH_ASSIGNED_DEVICEatomic_t assigned_device_count; +unsigned long nr_possible_bypass_irqs; +#ifdef CONFIG_KVM_IOAPIC struct kvm_pic *vpic;struct kvm_ioapic *vioapic;
@@ -1855,7 +1857,7 @@ struct kvm_x86_ops {int (*pi_update_irte)(struct kvm_kernel_irqfd *irqfd,struct kvm *kvm,unsigned int host_irq,uint32_t guest_irq,struct kvm_vcpu *vcpu,u32 vector); -void (*pi_start_assignment)(struct kvm *kvm); +void (*pi_start_bypass)(struct kvm *kvm);void (*apicv_pre_state_restore)(struct kvm_vcpu *vcpu);void (*apicv_post_state_restore)(struct kvm_vcpu *vcpu);bool (*dy_apicv_has_pending_interrupt)(struct kvm_vcpu *vcpu);
arch/x86/kvm/irq.c +8 -1
@@ -570,10 +570,15 @@ int kvm_arch_irq_bypass_add_producer(struct irq_bypass_consumer *cons,spin_lock_irq (&kvm->irqfds.lock); irqfd->producer = prod; +if (!kvm->arch.nr_possible_bypass_irqs++) +kvm_x86_call (pi_start_bypass)(kvm); +if (irqfd->irq_entry.type ==KVM_IRQ_ROUTING_MSI ) { ret =kvm_pi_update_irte (irqfd, &irqfd->irq_entry); -if (ret) +if (ret) { + kvm->arch.nr_possible_bypass_irqs--;kvm_arch_end_assignment (irqfd->kvm); + } }spin_unlock_irq (&kvm->irqfds.lock);
@@ -606,6 +611,8 @@ void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *cons, } irqfd->producer =NULL ; + kvm->arch.nr_possible_bypass_irqs--; +spin_unlock_irq (&kvm->irqfds.lock);
arch/x86/kvm/vmx/main.c +1 -1
@@ -1014,7 +1014,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .nested_ops = &vmx_nested_ops, .pi_update_irte = vmx_pi_update_irte, - .pi_start_assignment = vmx_pi_start_assignment, + .pi_start_bypass = vmx_pi_start_bypass, #ifdef CONFIG_X86_64 .set_hv_timer = vt_op(set_hv_timer),
arch/x86/kvm/vmx/posted_intr.c +10 -6
@@ -146,8 +146,13 @@ void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)static bool vmx_can_use_vtd_pi (struct kvm *kvm) { +/* +* Note, reading the number of possible bypass IRQs can race with a +* bypass IRQ being attached to the VM. vmx_pi_start_bypass() ensures +* blockng vCPUs will see an elevated count or get KVM_REQ_UNBLOCK. +*/ return irqchip_in_kernel (kvm) &&kvm_arch_has_irq_bypass () && -kvm_arch_has_assigned_device (kvm); +READ_ONCE (kvm->arch.nr_possible_bypass_irqs); }/*
@@ -285,12 +290,11 @@ bool pi_has_pending_interrupt(struct kvm_vcpu *vcpu)/* -* Bail out of the block loop if the VM has an assigned -* device, but the blocking vCPU didn't reconfigure the -* PI.NV to the wakeup vector, i.e. the assigned device -* came along after the initial check in vmx_vcpu_pi_put(). +* Kick all vCPUs when the first possible bypass IRQ is attached to a VM, as +* blocking vCPUs may scheduled out without reconfiguring PID.NV to the wakeup +* vector, i.e. if the bypass IRQ came along after vmx_vcpu_pi_put(). */ -void vmx_pi_start_assignment (struct kvm *kvm) +void vmx_pi_start_bypass (struct kvm *kvm) {if (!kvm_arch_has_irq_bypass ())return ;
arch/x86/kvm/vmx/posted_intr.h +1 -1
@@ -17,7 +17,7 @@ bool pi_has_pending_interrupt(struct kvm_vcpu *vcpu);int vmx_pi_update_irte (struct kvm_kernel_irqfd *irqfd,struct kvm *kvm,unsigned int host_irq,uint32_t guest_irq,struct kvm_vcpu *vcpu,u32 vector); -void vmx_pi_start_assignment (struct kvm *kvm); +void vmx_pi_start_bypass (struct kvm *kvm);static inline int pi_find_highest_vector (struct pi_desc *pi_desc) {
arch/x86/kvm/x86.c +1 -2
@@ -13563,8 +13563,7 @@ bool kvm_arch_can_dequeue_async_page_present(struct kvm_vcpu *vcpu)void kvm_arch_start_assignment (struct kvm *kvm) { -if (atomic_inc_return (&kvm->arch.assigned_device_count) ==1 ) -kvm_x86_call (pi_start_assignment)(kvm); +atomic_inc (&kvm->arch.assigned_device_count); }EXPORT_SYMBOL_GPL (kvm_arch_start_assignment);
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 52/59 KVM: VMX: WARN if VT-d Posted IRQs aren't possible when starting IRQ bypass
In reply to Sean ChristophersonWARN if KVM attempts to "start" IRQ bypass when VT-d Posted IRQs are disabled, to make it obvious that the logic is a sanity check, and so that a bug related to nr_possible_bypass_irqs is more like to cause noisy failures, e.g. so that KVM doesn't silently fail to wake blocking vCPUs.
--- arch/x86/kvm/vmx/posted_intr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/vmx/posted_intr.c +1 -1
@@ -296,7 +296,7 @@ bool pi_has_pending_interrupt(struct kvm_vcpu *vcpu)*/ void vmx_pi_start_bypass (struct kvm *kvm) { -if (!kvm_arch_has_irq_bypass ()) +if (WARN_ON_ONCE (!vmx_can_use_vtd_pi (kvm)))return ;kvm_make_all_cpus_request (kvm,KVM_REQ_UNBLOCK );
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 53/59 KVM: SVM: Use vcpu_idx, not vcpu_id, for GA log tag/metadata
In reply to Sean ChristophersonUse a vCPU's index, not its ID, for the GA log tag/metadata that's used to find and kick vCPUs when a device posted interrupt serves as a wake event. Lookups on a vCPU index are O(fast) (not sure what xa_load() actually provides), whereas a vCPU ID lookup is O(n) if a vCPU's ID doesn't match its index.
Unlike the Physical APIC Table, which is accessed by hardware when virtualizing IPIs, hardware doesn't consume the GA tag, i.e. KVM _must_ use APIC IDs to fill the Physical APIC Table, but KVM has free rein over the format/meaning of the GA tag.
--- arch/x86/kvm/svm/avic.c | 37 ++++++++++++++++++++----------------- 1 file changed, 20 insertions(+), 17 deletions(-)
arch/x86/kvm/svm/avic.c +20 -17
@@ -29,36 +29,39 @@#include "svm.h" /* -* Encode the arbitrary VM ID and the vCPU's default APIC ID, i.e the vCPU ID, -* into the GATag so that KVM can retrieve the correct vCPU from a GALog entry -* if an interrupt can't be delivered, e.g. because the vCPU isn't running. +* Encode the arbitrary VM ID and the vCPU's _index_ into the GATag so that +* KVM can retrieve the correct vCPU from a GALog entry if an interrupt can't +* be delivered, e.g. because the vCPU isn't running. Use the vCPU's index +* instead of its ID (a.k.a. its default APIC ID), as KVM is guaranteed a fast +* lookup on the index, where as vCPUs whose index doesn't match their ID need +* to walk the entire xarray of vCPUs in the worst case scenario. * -* For the vCPU ID, use however many bits are currently allowed for the max +* For the vCPU index, use however many bits are currently allowed for the max * guest physical APIC ID (limited by the size of the physical ID table), and * use whatever bits remain to assign arbitrary AVIC IDs to VMs. Note, the * size of the GATag is defined by hardware (32 bits), but is an opaque value * as far as hardware is concerned. */ -#define AVIC_VCPU_ID_MASK AVIC_PHYSICAL_MAX_INDEX_MASK +#define AVIC_VCPU_IDX_MASK AVIC_PHYSICAL_MAX_INDEX_MASK#define AVIC_VM_ID_SHIFT HWEIGHT32(AVIC_PHYSICAL_MAX_INDEX_MASK)#define AVIC_VM_ID_MASK (GENMASK(31, AVIC_VM_ID_SHIFT) >> AVIC_VM_ID_SHIFT)#define AVIC_GATAG_TO_VMID (x) ((x >> AVIC_VM_ID_SHIFT) & AVIC_VM_ID_MASK) -#define AVIC_GATAG_TO_VCPUID (x) (x & AVIC_VCPU_ID_MASK) +#define AVIC_GATAG_TO_VCPUIDX (x) (x & AVIC_VCPU_IDX_MASK) -#define __AVIC_GATAG (vm_id, vcpu_id) ((((vm_id) & AVIC_VM_ID_MASK) << AVIC_VM_ID_SHIFT) | \ - ((vcpu_id) & AVIC_VCPU_ID_MASK)) -#define AVIC_GATAG (vm_id, vcpu_id) \ +#define __AVIC_GATAG (vm_id, vcpu_idx) ((((vm_id) & AVIC_VM_ID_MASK) << AVIC_VM_ID_SHIFT) | \ + ((vcpu_idx) & AVIC_VCPU_IDX_MASK)) +#define AVIC_GATAG (vm_id, vcpu_idx) \ ({ \ - u32 ga_tag = __AVIC_GATAG(vm_id, vcpu_id); \ + u32 ga_tag = __AVIC_GATAG(vm_id, vcpu_idx); \ \ - WARN_ON_ONCE(AVIC_GATAG_TO_VCPUID(ga_tag) != (vcpu_id)); \ + WARN_ON_ONCE(AVIC_GATAG_TO_VCPUIDX(ga_tag) != (vcpu_idx)); \ WARN_ON_ONCE(AVIC_GATAG_TO_VMID(ga_tag) != (vm_id)); \ ga_tag; \ }) -static_assert (__AVIC_GATAG (AVIC_VM_ID_MASK ,AVIC_VCPU_ID_MASK ) ==-1u ); +static_assert (__AVIC_GATAG (AVIC_VM_ID_MASK ,AVIC_VCPU_IDX_MASK ) ==-1u );static bool force_avic;module_param_unsafe (force_avic, bool,0444 );
@@ -139,16 +142,16 @@ int avic_ga_log_notifier(u32 ga_tag)struct kvm_svm *kvm_svm;struct kvm_vcpu *vcpu =NULL ;u32 vm_id =AVIC_GATAG_TO_VMID (ga_tag); -u32 vcpu_id =AVIC_GATAG_TO_VCPUID (ga_tag); +u32 vcpu_idx =AVIC_GATAG_TO_VCPUIDX (ga_tag); -pr_debug ("SVM: %s: vm_id=%#x, vcpu_id=%#x\n" , __func__, vm_id, vcpu_id); -trace_kvm_avic_ga_log (vm_id, vcpu_id); +pr_debug ("SVM: %s: vm_id=%#x, vcpu_idx=%#x\n" , __func__, vm_id, vcpu_idx); +trace_kvm_avic_ga_log (vm_id, vcpu_idx);spin_lock_irqsave (&svm_vm_data_hash_lock, flags);hash_for_each_possible (svm_vm_data_hash, kvm_svm, hnode, vm_id) {if (kvm_svm->avic_vm_id != vm_id)continue ; - vcpu =kvm_get_vcpu_by_id (&kvm_svm->kvm, vcpu_id); + vcpu =kvm_get_vcpu (&kvm_svm->kvm, vcpu_idx);break ; }spin_unlock_irqrestore (&svm_vm_data_hash_lock, flags);
@@ -785,7 +788,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,*/ struct amd_iommu_pi_data pi_data = { .ga_tag =AVIC_GATAG (to_kvm_svm (kvm)->avic_vm_id, - vcpu->vcpu_id), + vcpu->vcpu_idx), .is_guest_mode =kvm_vcpu_apicv_active (vcpu), .vapic_addr =avic_get_backing_page_address (to_svm (vcpu)), .vector = vector,
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 54/59 iommu/amd: WARN if KVM calls GA IRTE helpers without virtual APIC support
In reply to Sean ChristophersonWARN if KVM attempts to update IRTE entries when virtual APIC isn't fully supported, as KVM should guard all such calls on IRQ posting being enabled.
--- drivers/iommu/amd/iommu.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
drivers/iommu/amd/iommu.c +12 -5
@@ -3836,8 +3836,10 @@ int amd_iommu_update_ga(int cpu, void *data)struct amd_ir_data *ir_data = (struct amd_ir_data *)data;struct irte_ga *entry = (struct irte_ga *) ir_data->entry; -if (!AMD_IOMMU_GUEST_IR_VAPIC (amd_iommu_guest_ir) || - !entry || !entry->lo.fields_vapic.guest_mode) +if (WARN_ON_ONCE (!AMD_IOMMU_GUEST_IR_VAPIC (amd_iommu_guest_ir))) +return -EINVAL ; + +if (!entry || !entry->lo.fields_vapic.guest_mode)return 0 ;if (!ir_data->iommu)
@@ -3856,7 +3858,10 @@ int amd_iommu_activate_guest_mode(void *data, int cpu)struct irte_ga *entry = (struct irte_ga *) ir_data->entry;u64 valid; -if (!AMD_IOMMU_GUEST_IR_VAPIC (amd_iommu_guest_ir) || !entry) +if (WARN_ON_ONCE (!AMD_IOMMU_GUEST_IR_VAPIC (amd_iommu_guest_ir))) +return -EINVAL ; + +if (!entry)return 0 ; valid = entry->lo.fields_vapic.valid;
@@ -3885,8 +3890,10 @@ int amd_iommu_deactivate_guest_mode(void *data)struct irq_cfg *cfg = ir_data->cfg;u64 valid; -if (!AMD_IOMMU_GUEST_IR_VAPIC (amd_iommu_guest_ir) || - !entry || !entry->lo.fields_vapic.guest_mode) +if (WARN_ON_ONCE (!AMD_IOMMU_GUEST_IR_VAPIC (amd_iommu_guest_ir))) +return -EINVAL ; + +if (!entry || !entry->lo.fields_vapic.guest_mode)return 0 ; valid = entry->lo.fields_remap.valid;
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 55/59 KVM: SVM: Fold avic_set_pi_irte_mode() into its sole caller
In reply to Sean ChristophersonFold avic_set_pi_irte_mode() into avic_refresh_apicv_exec_ctrl() in anticipation of moving the __avic_vcpu_{load,put}() calls into the critical section, and because having a one-off helper with a name that's easily confused with avic_pi_update_irte() is unnecessary.
No functional change intended.
--- arch/x86/kvm/svm/avic.c | 52 ++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 29 deletions(-)
arch/x86/kvm/svm/avic.c +23 -29
@@ -728,34 +728,6 @@ void avic_apicv_post_state_restore(struct kvm_vcpu *vcpu)avic_handle_ldr_update (vcpu); } -static void avic_set_pi_irte_mode (struct kvm_vcpu *vcpu,bool activate) -{ -int apic_id =kvm_cpu_get_apicid (vcpu->cpu); -unsigned long flags; -struct vcpu_svm *svm =to_svm (vcpu); -struct kvm_kernel_irqfd *irqfd; - -/* -* Here, we go through the per-vcpu ir_list to update all existing -* interrupt remapping table entry targeting this vcpu. -*/ -spin_lock_irqsave (&svm->ir_list_lock, flags); - -if (list_empty (&svm->ir_list)) - goto out; - -list_for_each_entry (irqfd, &svm->ir_list, vcpu_list) { -void *data = irqfd->irq_bypass_data; - -if (activate) -WARN_ON_ONCE (amd_iommu_activate_guest_mode (data, apic_id)); -else -WARN_ON_ONCE (amd_iommu_deactivate_guest_mode (data)); - } -out: -spin_unlock_irqrestore (&svm->ir_list_lock, flags); -} -static void svm_ir_list_del (struct kvm_kernel_irqfd *irqfd) {struct kvm_vcpu *vcpu = irqfd->irq_bypass_vcpu;
@@ -990,6 +962,10 @@ void avic_refresh_virtual_apic_mode(struct kvm_vcpu *vcpu)void avic_refresh_apicv_exec_ctrl (struct kvm_vcpu *vcpu) {bool activated =kvm_vcpu_apicv_active (vcpu); +int apic_id =kvm_cpu_get_apicid (vcpu->cpu); +struct vcpu_svm *svm =to_svm (vcpu); +struct kvm_kernel_irqfd *irqfd; +unsigned long flags;if (!enable_apicv)return ;
@@ -1001,7 +977,25 @@ void avic_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu)else avic_vcpu_put (vcpu); -avic_set_pi_irte_mode (vcpu, activated); +/* +* Here, we go through the per-vcpu ir_list to update all existing +* interrupt remapping table entry targeting this vcpu. +*/ +spin_lock_irqsave (&svm->ir_list_lock, flags); + +if (list_empty (&svm->ir_list)) + goto out; + +list_for_each_entry (irqfd, &svm->ir_list, vcpu_list) { +void *data = irqfd->irq_bypass_data; + +if (activated) +WARN_ON_ONCE (amd_iommu_activate_guest_mode (data, apic_id)); +else +WARN_ON_ONCE (amd_iommu_deactivate_guest_mode (data)); + } +out: +spin_unlock_irqrestore (&svm->ir_list_lock, flags); }void avic_vcpu_blocking (struct kvm_vcpu *vcpu)
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 56/59 KVM: SVM: Don't check vCPU's blocking status when toggling AVIC on/off
In reply to Sean ChristophersonDon't query a vCPU's blocking status when toggling AVIC on/off; barring KVM bugs, the vCPU can't be blocking when refreshing AVIC controls. And if there are KVM bugs, ensuring the vCPU and its associated IRTEs are in the correct state is desirable, i.e. well worth any overhead in a buggy scenario.
Isolating the "real" load/put flows will allow moving the IOMMU IRTE (de)activation logic from avic_refresh_apicv_exec_ctrl() to avic_update_iommu_vcpu_affinity(), i.e. will allow updating the vCPU's physical ID entry and its IRTEs in a common path, under a single critical section of ir_list_lock.
--- arch/x86/kvm/svm/avic.c | 65 +++++++++++++++++++++++------------------ 1 file changed, 37 insertions(+), 28 deletions(-)
arch/x86/kvm/svm/avic.c +37 -28
@@ -828,7 +828,7 @@ static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu)WARN_ON_ONCE (amd_iommu_update_ga (cpu, irqfd->irq_bypass_data)); } -void avic_vcpu_load (struct kvm_vcpu *vcpu,int cpu) +static void __avic_vcpu_load (struct kvm_vcpu *vcpu,int cpu) {struct kvm_svm *kvm_svm =to_kvm_svm (vcpu->kvm);int h_physical_id =kvm_cpu_get_apicid (cpu);
@@ -844,16 +844,6 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu)if (WARN_ON_ONCE (vcpu->vcpu_id *sizeof (entry) >=PAGE_SIZE ))return ; -/* -* No need to update anything if the vCPU is blocking, i.e. if the vCPU -* is being scheduled in after being preempted. The CPU entries in the -* Physical APIC table and IRTE are consumed iff IsRun{ning} is '1'. -* If the vCPU was migrated, its new CPU value will be stuffed when the -* vCPU unblocks. -*/ -if (kvm_vcpu_is_blocking (vcpu)) -return ; -/* * Grab the per-vCPU interrupt remapping lock even if the VM doesn't * _currently_ have assigned devices, as that can change. Holding
@@ -888,31 +878,33 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu)spin_unlock_irqrestore (&svm->ir_list_lock, flags); } -void avic_vcpu_put (struct kvm_vcpu *vcpu) +void avic_vcpu_load (struct kvm_vcpu *vcpu,int cpu) +{ +/* +* No need to update anything if the vCPU is blocking, i.e. if the vCPU +* is being scheduled in after being preempted. The CPU entries in the +* Physical APIC table and IRTE are consumed iff IsRun{ning} is '1'. +* If the vCPU was migrated, its new CPU value will be stuffed when the +* vCPU unblocks. +*/ +if (kvm_vcpu_is_blocking (vcpu)) +return ; + +__avic_vcpu_load (vcpu, cpu); +} + +static void __avic_vcpu_put (struct kvm_vcpu *vcpu) {struct kvm_svm *kvm_svm =to_kvm_svm (vcpu->kvm);struct vcpu_svm *svm =to_svm (vcpu);unsigned long flags; -u64 entry; +u64 entry = svm->avic_physical_id_entry;lockdep_assert_preemption_disabled ();if (WARN_ON_ONCE (vcpu->vcpu_id *sizeof (entry) >=PAGE_SIZE ))return ; -/* -* Note, reading the Physical ID entry outside of ir_list_lock is safe -* as only the pCPU that has loaded (or is loading) the vCPU is allowed -* to modify the entry, and preemption is disabled. I.e. the vCPU -* can't be scheduled out and thus avic_vcpu_{put,load}() can't run -* recursively. -*/ - entry = svm->avic_physical_id_entry; - -/* Nothing to do if IsRunning == '0' due to vCPU blocking. */ -if (!(entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK )) -return ; -/* * Take and hold the per-vCPU interrupt remapping lock while updating * the Physical ID entry even though the lock doesn't protect against
@@ -932,7 +924,24 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu)WRITE_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id], entry);spin_unlock_irqrestore (&svm->ir_list_lock, flags); +} +void avic_vcpu_put (struct kvm_vcpu *vcpu) +{ +/* +* Note, reading the Physical ID entry outside of ir_list_lock is safe +* as only the pCPU that has loaded (or is loading) the vCPU is allowed +* to modify the entry, and preemption is disabled. I.e. the vCPU +* can't be scheduled out and thus avic_vcpu_{put,load}() can't run +* recursively. +*/ +u64 entry =to_svm (vcpu)->avic_physical_id_entry; + +/* Nothing to do if IsRunning == '0' due to vCPU blocking. */ +if (!(entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK )) +return ; + +__avic_vcpu_put (vcpu); }void avic_refresh_virtual_apic_mode (struct kvm_vcpu *vcpu)
@@ -973,9 +982,9 @@ void avic_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu)avic_refresh_virtual_apic_mode (vcpu);if (activated) -avic_vcpu_load (vcpu, vcpu->cpu); +__avic_vcpu_load (vcpu, vcpu->cpu);else -avic_vcpu_put (vcpu); +__avic_vcpu_put (vcpu);/* * Here, we go through the per-vcpu ir_list to update all existing
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 57/59 KVM: SVM: Consolidate IRTE update when toggling AVIC on/off
In reply to Sean ChristophersonFold the IRTE modification logic in avic_refresh_apicv_exec_ctrl() into __avic_vcpu_{load,put}(), and add a param to the helpers to communicate whether or not AVIC is being toggled, i.e. if IRTE needs a "full" update, or just a quick update to set the CPU and IsRun.
--- arch/x86/kvm/svm/avic.c | 85 ++++++++++++++++++++++------------------- 1 file changed, 46 insertions(+), 39 deletions(-)
arch/x86/kvm/svm/avic.c +46 -39
@@ -810,7 +810,28 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,return irq_set_vcpu_affinity (host_irq,NULL ); } -static void avic_update_iommu_vcpu_affinity (struct kvm_vcpu *vcpu,int cpu) +enum avic_vcpu_action { +/* +* There is no need to differentiate between activate and deactivate, +* as KVM only refreshes AVIC state when the vCPU is scheduled in and +* isn't blocking, i.e. the pCPU must always be (in)valid when AVIC is +* being (de)activated. +*/ +AVIC_TOGGLE_ON_OFF =BIT (0 ), +AVIC_ACTIVATE =AVIC_TOGGLE_ON_OFF , +AVIC_DEACTIVATE =AVIC_TOGGLE_ON_OFF , + +/* +* No unique action is required to deal with a vCPU that stops/starts +* running, as IRTEs are configured to generate GALog interrupts at all +* times. +*/ +AVIC_START_RUNNING =0 , +AVIC_STOP_RUNNING =0 , +}; + +static void avic_update_iommu_vcpu_affinity (struct kvm_vcpu *vcpu,int cpu, +enum avic_vcpu_action action) {struct vcpu_svm *svm =to_svm (vcpu);struct kvm_kernel_irqfd *irqfd;
@@ -824,11 +845,20 @@ static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu)if (list_empty (&svm->ir_list))return ; -list_for_each_entry (irqfd, &svm->ir_list, vcpu_list) -WARN_ON_ONCE (amd_iommu_update_ga (cpu, irqfd->irq_bypass_data)); +list_for_each_entry (irqfd, &svm->ir_list, vcpu_list) { +void *data = irqfd->irq_bypass_data; + +if (!(action &AVIC_TOGGLE_ON_OFF )) +WARN_ON_ONCE (amd_iommu_update_ga (cpu, data)); +else if (cpu >=0 ) +WARN_ON_ONCE (amd_iommu_activate_guest_mode (data, cpu)); +else +WARN_ON_ONCE (amd_iommu_deactivate_guest_mode (data)); + } } -static void __avic_vcpu_load (struct kvm_vcpu *vcpu,int cpu) +static void __avic_vcpu_load (struct kvm_vcpu *vcpu,int cpu, +enum avic_vcpu_action action) {struct kvm_svm *kvm_svm =to_kvm_svm (vcpu->kvm);int h_physical_id =kvm_cpu_get_apicid (cpu);
@@ -873,7 +903,7 @@ static void __avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu)WRITE_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id], entry); -avic_update_iommu_vcpu_affinity (vcpu, h_physical_id); +avic_update_iommu_vcpu_affinity (vcpu, h_physical_id, action);spin_unlock_irqrestore (&svm->ir_list_lock, flags); }
@@ -890,10 +920,10 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu)if (kvm_vcpu_is_blocking (vcpu))return ; -__avic_vcpu_load (vcpu, cpu); +__avic_vcpu_load (vcpu, cpu,AVIC_START_RUNNING ); } -static void __avic_vcpu_put (struct kvm_vcpu *vcpu) +static void __avic_vcpu_put (struct kvm_vcpu *vcpu,enum avic_vcpu_action action) {struct kvm_svm *kvm_svm =to_kvm_svm (vcpu->kvm);struct vcpu_svm *svm =to_svm (vcpu);
@@ -915,7 +945,7 @@ static void __avic_vcpu_put(struct kvm_vcpu *vcpu)*/ spin_lock_irqsave (&svm->ir_list_lock, flags); -avic_update_iommu_vcpu_affinity (vcpu,-1 ); +avic_update_iommu_vcpu_affinity (vcpu,-1 , action); entry &= ~AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ; svm->avic_physical_id_entry = entry;
@@ -941,7 +971,7 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu)if (!(entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ))return ; -__avic_vcpu_put (vcpu); +__avic_vcpu_put (vcpu,AVIC_STOP_RUNNING ); }void avic_refresh_virtual_apic_mode (struct kvm_vcpu *vcpu)
@@ -970,41 +1000,18 @@ void avic_refresh_virtual_apic_mode(struct kvm_vcpu *vcpu)void avic_refresh_apicv_exec_ctrl (struct kvm_vcpu *vcpu) { -bool activated =kvm_vcpu_apicv_active (vcpu); -int apic_id =kvm_cpu_get_apicid (vcpu->cpu); -struct vcpu_svm *svm =to_svm (vcpu); -struct kvm_kernel_irqfd *irqfd; -unsigned long flags; -if (!enable_apicv)return ; +/* APICv should only be toggled on/off while the vCPU is running. */ +WARN_ON_ONCE (kvm_vcpu_is_blocking (vcpu)); +avic_refresh_virtual_apic_mode (vcpu); -if (activated) -__avic_vcpu_load (vcpu, vcpu->cpu); +if (kvm_vcpu_apicv_active (vcpu)) +__avic_vcpu_load (vcpu, vcpu->cpu,AVIC_ACTIVATE );else -__avic_vcpu_put (vcpu); - -/* -* Here, we go through the per-vcpu ir_list to update all existing -* interrupt remapping table entry targeting this vcpu. -*/ -spin_lock_irqsave (&svm->ir_list_lock, flags); - -if (list_empty (&svm->ir_list)) - goto out; - -list_for_each_entry (irqfd, &svm->ir_list, vcpu_list) { -void *data = irqfd->irq_bypass_data; - -if (activated) -WARN_ON_ONCE (amd_iommu_activate_guest_mode (data, apic_id)); -else -WARN_ON_ONCE (amd_iommu_deactivate_guest_mode (data)); - } -out: -spin_unlock_irqrestore (&svm->ir_list_lock, flags); +__avic_vcpu_put (vcpu,AVIC_DEACTIVATE ); }void avic_vcpu_blocking (struct kvm_vcpu *vcpu)
@@ -1030,7 +1037,7 @@ void avic_vcpu_blocking(struct kvm_vcpu *vcpu)* CPU and cause noisy neighbor problems if the VM is sending interrupts * to the vCPU while it's scheduled out. */ -avic_vcpu_put (vcpu); +__avic_vcpu_put (vcpu,AVIC_STOP_RUNNING ); }void avic_vcpu_unblocking (struct kvm_vcpu *vcpu)
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 58/59 iommu/amd: KVM: SVM: Allow KVM to control need for GA log interrupts
In reply to Sean ChristophersonAdd plumbing to the AMD IOMMU driver to allow KVM to control whether or not an IRTE is configured to generate GA log interrupts. KVM only needs a notification if the target vCPU is blocking, so the vCPU can be awakened. If a vCPU is preempted or exits to userspace, KVM clears is_run, but will set the vCPU back to running when userspace does KVM_RUN and/or the vCPU task is scheduled back in, i.e. KVM doesn't need a notification.
Unconditionally pass "true" in all KVM paths to isolate the IOMMU changes from the KVM changes insofar as possible.
Opportunistically swap the ordering of parameters for amd_iommu_update_ga() so that the match amd_iommu_activate_guest_mode().
Note, as of this writing, the AMD IOMMU manual doesn't list GALogIntr as a non-cached field, but per AMD hardware architects, it's not cached and can be safely updated without an invalidation.
--- arch/x86/include/asm/irq_remapping.h | 1 + arch/x86/kvm/svm/avic.c | 10 ++++++---- drivers/iommu/amd/iommu.c | 28 +++++++++++++++++----------- include/linux/amd-iommu.h | 9 ++++----- 4 files changed, 28 insertions(+), 20 deletions(-)
arch/x86/include/asm/irq_remapping.h +1 -0
@@ -36,6 +36,7 @@ struct amd_iommu_pi_data {u32 ga_tag;u32 vector;/* Guest vector of the interrupt */ int cpu; +bool ga_log_intr;bool is_guest_mode;void *ir_data; };
arch/x86/kvm/svm/avic.c +6 -4
@@ -784,10 +784,12 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,* is awakened and/or scheduled in. See also avic_vcpu_load(). */ entry = svm->avic_physical_id_entry; -if (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ) +if (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ) { pi_data.cpu = entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK ; -else + }else { pi_data.cpu =-1 ; + pi_data.ga_log_intr = true; + } ret =irq_set_vcpu_affinity (host_irq, &pi_data);if (ret)
@@ -849,9 +851,9 @@ static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu,void *data = irqfd->irq_bypass_data;if (!(action &AVIC_TOGGLE_ON_OFF )) -WARN_ON_ONCE (amd_iommu_update_ga (cpu, data)); +WARN_ON_ONCE (amd_iommu_update_ga (data, cpu, true));else if (cpu >=0 ) -WARN_ON_ONCE (amd_iommu_activate_guest_mode (data, cpu)); +WARN_ON_ONCE (amd_iommu_activate_guest_mode (data, cpu, true));else WARN_ON_ONCE (amd_iommu_deactivate_guest_mode (data)); }
drivers/iommu/amd/iommu.c +17 -11
@@ -3804,7 +3804,8 @@ static const struct irq_domain_ops amd_ir_domain_ops = { .deactivate = irq_remapping_deactivate, }; -static void __amd_iommu_update_ga (struct irte_ga *entry,int cpu) +static void __amd_iommu_update_ga (struct irte_ga *entry,int cpu, +bool ga_log_intr) {if (cpu >=0 ) { entry->lo.fields_vapic.destination =
@@ -3812,8 +3813,10 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int cpu) entry->hi.fields.destination =APICID_TO_IRTE_DEST_HI (cpu); entry->lo.fields_vapic.is_run = true; + entry->lo.fields_vapic.ga_log_intr = false; }else { entry->lo.fields_vapic.is_run = false; + entry->lo.fields_vapic.ga_log_intr = ga_log_intr; } }
@@ -3822,16 +3825,19 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int cpu)* a vCPU, without issuing an IOMMU invalidation for the IRTE. * * If the vCPU is associated with a pCPU (@cpu >= 0), configure the Destination -* with the pCPU's APIC ID and set IsRun, else clear IsRun. I.e. treat vCPUs -* that are associated with a pCPU as running. This API is intended to be used -* when a vCPU is scheduled in/out (or stops running for any reason), to do a -* fast update of IsRun and (conditionally) Destination. +* with the pCPU's APIC ID, set IsRun, and clear GALogIntr. If the vCPU isn't +* associated with a pCPU (@cpu < 0), clear IsRun and set/clear GALogIntr based +* on input from the caller (e.g. KVM only requests GALogIntr when the vCPU is +* blocking and requires a notification wake event). I.e. treat vCPUs that are +* associated with a pCPU as running. This API is intended to be used when a +* vCPU is scheduled in/out (or stops running for any reason), to do a fast +* update of IsRun, GALogIntr, and (conditionally) Destination. * * Per the IOMMU spec, the Destination, IsRun, and GATag fields are not cached * and thus don't require an invalidation to ensure the IOMMU consumes fresh * information. */ -int amd_iommu_update_ga (int cpu,void *data) +int amd_iommu_update_ga (void *data,int cpu,bool ga_log_intr) {struct amd_ir_data *ir_data = (struct amd_ir_data *)data;struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -3845,14 +3851,14 @@ int amd_iommu_update_ga(int cpu, void *data)if (!ir_data->iommu)return -ENODEV ; -__amd_iommu_update_ga (entry, cpu); +__amd_iommu_update_ga (entry, cpu, ga_log_intr);return __modify_irte_ga (ir_data->iommu, ir_data->irq_2_irte.devid, ir_data->irq_2_irte.index, entry); }EXPORT_SYMBOL (amd_iommu_update_ga); -int amd_iommu_activate_guest_mode (void *data,int cpu) +int amd_iommu_activate_guest_mode (void *data,int cpu,bool ga_log_intr) {struct amd_ir_data *ir_data = (struct amd_ir_data *)data;struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -3871,12 +3877,11 @@ int amd_iommu_activate_guest_mode(void *data, int cpu) entry->lo.fields_vapic.valid = valid; entry->lo.fields_vapic.guest_mode =1 ; - entry->lo.fields_vapic.ga_log_intr =1 ; entry->hi.fields.ga_root_ptr = ir_data->ga_root_ptr; entry->hi.fields.vector = ir_data->ga_vector; entry->lo.fields_vapic.ga_tag = ir_data->ga_tag; -__amd_iommu_update_ga (entry, cpu); +__amd_iommu_update_ga (entry, cpu, ga_log_intr);return modify_irte_ga (ir_data->iommu, ir_data->irq_2_irte.devid, ir_data->irq_2_irte.index, entry);
@@ -3947,7 +3952,8 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *info) ir_data->ga_vector = pi_data->vector; ir_data->ga_tag = pi_data->ga_tag;if (pi_data->is_guest_mode) - ret =amd_iommu_activate_guest_mode (ir_data, pi_data->cpu); + ret =amd_iommu_activate_guest_mode (ir_data, pi_data->cpu, + pi_data->ga_log_intr);else ret =amd_iommu_deactivate_guest_mode (ir_data); }else {
include/linux/amd-iommu.h +4 -5
@@ -30,9 +30,8 @@ static inline void amd_iommu_detect(void) { }/* IOMMU AVIC Function */ extern int amd_iommu_register_ga_log_notifier (int (*notifier)(u32 )); -extern int amd_iommu_update_ga (int cpu,void *data); - -extern int amd_iommu_activate_guest_mode (void *data,int cpu); +extern int amd_iommu_update_ga (void *data,int cpu,bool ga_log_intr); +extern int amd_iommu_activate_guest_mode (void *data,int cpu,bool ga_log_intr);extern int amd_iommu_deactivate_guest_mode (void *data);#else /* defined(CONFIG_AMD_IOMMU) && defined(CONFIG_IRQ_REMAP) */
@@ -43,12 +42,12 @@ amd_iommu_register_ga_log_notifier(int (*notifier)(u32))return 0 ; } -static inline int amd_iommu_update_ga (int cpu,void *data) +static inline int amd_iommu_update_ga (void *data,int cpu,bool ga_log_intr) {return 0 ; } -static inline int amd_iommu_activate_guest_mode (void *data,int cpu) +static inline int amd_iommu_activate_guest_mode (void *data,int cpu,bool ga_log_intr) {return 0 ; }
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Sean Christopherson
maintainerPATCH v2 59/59 KVM: SVM: Generate GA log IRQs only if the associated vCPUs is blocking
In reply to Sean ChristophersonConfigure IRTEs to GA log interrupts for device posted IRQs that hit non-running vCPUs if and only if the target vCPU is blocking, i.e. actually needs a wake event. If the vCPU has exited to userspace or was preempted, generating GA log entries and interrupts is wasteful and unnecessary, as the vCPU will be re-loaded and/or scheduled back in irrespective of the GA log notification (avic_ga_log_notifier() is just a fancy wrapper for kvm_vcpu_wake_up()).
Use a should-be-zero bit in the vCPU's Physical APIC ID Table Entry to track whether or not the vCPU's associated IRTEs are configured to generate GA logs, but only set the synthetic bit in KVM's "cache", i.e. never set the should-be-zero bit in tables that are used by hardware. Use a synthetic bit instead of a dedicated boolean to minimize the odds of messing up the locking, i.e. so that all the existing rules that apply to avic_physical_id_entry for IS_RUNNING are reused verbatim for GA_LOG_INTR.
Note, because KVM (by design) "puts" AVIC state in a "pre-blocking" phase, using kvm_vcpu_is_blocking() to track the need for notifications isn't a viable option.
--- arch/x86/include/asm/svm.h | 7 +++++ arch/x86/kvm/svm/avic.c | 63 ++++++++++++++++++++++++++++++-------- 2 files changed, 58 insertions(+), 12 deletions(-)
arch/x86/include/asm/svm.h +7 -0
@@ -252,6 +252,13 @@ struct __attribute__ ((__packed__)) vmcb_control_area {#define AVIC_LOGICAL_ID_ENTRY_VALID_BIT 31#define AVIC_LOGICAL_ID_ENTRY_VALID_MASK (1 << 31) +/* +* GA_LOG_INTR is a synthetic flag that's never propagated to hardware-visible +* tables. GA_LOG_INTR is set if the vCPU needs device posted IRQs to generate +* GA log interrupts to wake the vCPU (because it's blocking or about to block). +*/ +#define AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR BIT_ULL(61) +#define AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK GENMASK_ULL(11, 0)#define AVIC_PHYSICAL_ID_ENTRY_BACKING_PAGE_MASK GENMASK_ULL(51, 12)#define AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK (1ULL << 62)
arch/x86/kvm/svm/avic.c +51 -12
120 lines not on this page · raw message
Signature · 1 line
2.49.0.1151.ga128411c76-goog
Paolo Bonzini
maintainerQuoted 33 lines · Sean Christopherson
TL;DR: Overhaul device posted interrupts in KVM and IOMMU, and AVIC in
general.This applies on the series to add
CONFIG_KVM_IOAPIC(and to killirq_comm.c):https://lore.kernel.org/all/[email protected]Fix a variety of bugs related to device posted IRQs, especially on the AMD side, and clean up KVM's implementation (this series actually removes more code than it adds).
Stating the obvious, this series is comically large. Though it's smaller than v1! (Ignoring that I cheated by moving 15 patches to a prep series, and that Paolo already grabbed several patches).
Sairaj, I applied your Tested-by somewhat sparingly, as some of the patches changed (most notably "Consolidate IRTE update when toggling AVIC on/off"). Please holler if you want me to remove/add any tags. And when you get time, I'd greatly appreciate a sanity check!
Batch #1 is mostly SVM specific:
- Cleans up various warts and bugs in the IRTE tracking
- Fixes AVIC to not reject large VMs (honor KVM's ABI)
- Wire up AVIC toenable_ipivto support disabling IPI virtualization while still utilizing device posted interrupts, and to workaround erratum #1235.Batch #3 overhauls the guts of IRQ bypass in KVM, and moves the vast majority of the logic to common x86; only the code that needs to communicate with the IOMMU is truly vendor specific.
Batch #4 is more SVM/AVIC cleanups that are made possible by batch #3.
Quoted 7 lines
Batch #5 adds WARNs and drops dead code after all the previous cleanups and fixes (I don't want to add the WARNs earlier; I don't see any point in adding WARNs in code that's known to be broken).
Batch #6 is yet more SVM/AVIC cleanups, with the specific goal of configuring IRTEs to generate GA log interrupts if and only if KVM actually needs a wake event.
Looks good - it's not even that different from v1. Thanks!
Paolo
Quoted 130 lines
v2:
- Drop patches that were already merged.
- Move code intoirq.c, notx86.c. [Paolo]
- Collect review/testing tags. [Sairaj, Vasant]
- Sqaush fixup for a comment that was added in the prior patch. [Sairaj]
- Rewrote the changelog for "Delete IRTE link from previous vCPU irrespective of new routing". [Sairaj]
- Actually drop "structamd_svm_iommu_ir" and all usage in "Track per-vCPU IRTEs usingkvm_kernel_irqfdstructure" (the previous version was getting hilarious lucky with struct offsets). [Sairaj]
- Drop unused params fromkvm_pi_update_irte()andpi_update_irte(). [Sairaj]
- Document the rules and behavior ofamd_iommu_update_ga(). [Joerg]
- Fix a changelog typo. [Paolo]
- Document that GALogIntr isn't cached, i.e. can be safely updated without an invalidation. [Joao, Vasant]
- Rework avic_vcpu_{load,put}() to use an enumerated parameter instead of a series of booleans. [Paolo]
- Drop a redundant "&& new". [Francesco]
- Drop the *** DO NOT MERGE *** testing hack patches.v1: https://lore.kernel.org/all/[email protected]
Maxim Levitsky (2):
KVM: SVM: Addenable_ipivparam, never set IsRunning if disabled
KVM: SVM: Disable (x2)AVIC IPI virtualization if CPU has erratum #1235Sean Christopherson (57):
KVM: x86: Pass new routing entries and irqfd when updating IRTEs
KVM: SVM: Track per-vCPU IRTEs usingkvm_kernel_irqfdstructure
KVM: SVM: Delete IRTE link from previous vCPU before setting new IRTE iommu/amd: KVM: SVM: Delete now-unused cached/previous GA tag fields
KVM: SVM: Delete IRTE link from previous vCPU irrespective of new
routing
KVM: SVM: Drop pointless masking of default APIC base when setting
V_APIC_BAR
KVM: SVM: Drop pointless masking of kernel page pa's with AVIC HPA
masks
KVM: SVM: Add helper to deduplicate code for getting AVIC backing page
KVM: SVM: Drop vcpu_svm's pointlessavic_backing_pagefield
KVM: SVM: Inhibit AVIC if ID is too big instead of rejecting vCPU
creation
KVM: SVM: Drop redundant check in AVIC code on ID during vCPU creation
KVM: SVM: Track AVIC tables as natively sized pointers, not "struct
pages"
KVM: SVM: Drop superfluous "cache" of AVIC Physical ID entry pointer
KVM: VMX: Moveenable_ipivknob to common x86
KVM: VMX: Suppress PI notifications whenever the vCPU is put
KVM: SVM: Add a comment to explain whyavic_vcpu_blocking()ignores
IRQ blocking
iommu/amd: KVM: SVM: Usepi_desc_addrto derivega_root_ptriommu/amd: KVM: SVM: Pass NULL @vcpu_info to indicate "not guest mode"
KVM: SVM: Stop walking list of routing table entries when updating
IRTE
KVM: VMX: Stop walking list of routing table entries when updating
IRTE
KVM: SVM: Extract SVM specific code out ofget_pi_vcpu_info()
KVM: x86: Move IRQ routing/delivery APIs fromx86.c=>irq.c
KVM: x86: Nullifyirqfd->producerafter updating IRTEs
KVM: x86: Dedup AVIC vs. PI code for identifying target vCPU
KVM: x86: Move posted interrupt tracepoint to common code
KVM: SVM: Clean up return handling inavic_pi_update_irte()
iommu: KVM: Split "structvcpu_data" into separate AMD vs. Intel
structs
KVM: Don't WARN if updating IRQ bypass route fails
KVM: Foldkvm_arch_irqfd_route_changed()into
kvm_arch_update_irqfd_routing()
KVM: x86: Trackirq_bypass_vcpuin common x86 code
KVM: x86: Skip IOMMU IRTE updates if there's no old or new vCPU being
targeted
KVM: x86: Don't update IRTE entries when old and new routes were !MSI
KVM: SVM: Revert IRTE to legacy mode if IOMMU doesn't provide IR
metadata
KVM: SVM: Take and holdir_list_lockacross IRTE updates in IOMMU iommu/amd: Document which IRTE fieldsamd_iommu_update_ga()can modify iommu/amd: KVM: SVM: Infer IsRun from validity of pCPU destination iommu/amd: Factor out helper for manipulating IRTE GA/CPU info iommu/amd: KVM: SVM: Set pCPU info in IRTE when setting vCPU affinity iommu/amd: KVM: SVM: Add IRTE metadata to affined vCPU's list if AVIC
is inhibited
KVM: SVM: Don't check for assigned device(s) when updating affinity
KVM: SVM: Don't check for assigned device(s) when activating AVIC
KVM: SVM: WARN if (de)activating guest mode in IOMMU fails
KVM: SVM: Process all IRTEs on affinity change even if one update
fails
KVM: SVM: WARN if updating IRTE GA fields in IOMMU fails
KVM: x86: Drop superfluous "has assigned device" check in
kvm_pi_update_irte()
KVM: x86: WARN if IRQ bypass isn't supported inkvm_pi_update_irte()
KVM: x86: WARN if IRQ bypass routing is updated without in-kernel
local APIC
KVM: SVM: WARN ifir_listis non-empty at vCPU free
KVM: x86: Decouple device assignment from IRQ bypass
KVM: VMX: WARN if VT-d Posted IRQs aren't possible when starting IRQ
bypass
KVM: SVM: Usevcpu_idx, notvcpu_id, for GA log tag/metadata iommu/amd: WARN if KVM calls GA IRTE helpers without virtual APIC
support
KVM: SVM: Foldavic_set_pi_irte_mode()into its sole caller
KVM: SVM: Don't check vCPU's blocking status when toggling AVIC on/off
KVM: SVM: Consolidate IRTE update when toggling AVIC on/off iommu/amd: KVM: SVM: Allow KVM to control need for GA log interrupts
KVM: SVM: Generate GA log IRQs only if the associated vCPUs is
blockingarch/x86/include/asm/irq_remapping.h | 17 +- arch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 20 +- arch/x86/include/asm/svm.h | 13 +- arch/x86/kvm/irq.c | 140 ++++++ arch/x86/kvm/svm/avic.c | 702 ++++++++++++--------------- arch/x86/kvm/svm/svm.c | 4 + arch/x86/kvm/svm/svm.h | 32 +- arch/x86/kvm/trace.h | 19 +- arch/x86/kvm/vmx/capabilities.h | 1 - arch/x86/kvm/vmx/main.c | 2 +- arch/x86/kvm/vmx/posted_intr.c | 140 ++---- arch/x86/kvm/vmx/posted_intr.h | 10 +- arch/x86/kvm/vmx/vmx.c | 2 - arch/x86/kvm/x86.c | 90 +--- drivers/iommu/amd/amd_iommu_types.h | 1 - drivers/iommu/amd/iommu.c | 125 +++-- drivers/iommu/intel/irq_remapping.c | 10 +- include/linux/amd-iommu.h | 25 +- include/linux/kvm_host.h | 9 +- include/linux/kvm_irqfd.h | 4 + virt/kvm/eventfd.c | 22 +- 22 files changed, 672 insertions(+), 718 deletions(-)base-commit:
3debd5461fba1dcb33e732b16153da0cf5d0c251
Hi Sean,
Sorry for the delay in testing. All sanity tests are OK. I reran the performance test on the V2 and noticed that V2 has significantly more GALOG entries than V1 for all three cases. I also noticed that the Guest Nvme interrupt rate has dropped for the 192 VCPUS.
I haven't figured out what is causing this. I will continue my investigation further.
VCPUS = 32, Jobs per NVME = 8
==============================================================================================
V2 V1 Percent change
----------------------------------------------------------------------------------------------
Guest Nvme interrupts 124,260,796 124,559,110 -0.20%
IOPS (in kilo) 4,790 4,796 -0.01%
GALOG entries 8117 169 4702%
----------------------------------------------------------------------------------------------
VCPUS = 64, Jobs per NVME = 16
==============================================================================================
V2 V1 Percent change
----------------------------------------------------------------------------------------------
Guest Nvme interrupts 102,394,358 99,800,056 2.00%
IOPS (in kilo) 4,796 4,798 -0.04%
GALOG entries 19,057 11,923 59.83%
----------------------------------------------------------------------------------------------
VCPUS = 192, Jobs per NVME = 48
==============================================================================================
V2 V1 Percent change
----------------------------------------------------------------------------------------------
Guest Nvme interrupts 68,363,232 78,066,512 -12.42%
IOPS (in kilo) 4,751 4,749 -0.04%
GALOG entries 62,768 56,215 11.66%
----------------------------------------------------------------------------------------------
Thanks
Sairaj
Sean Christopherson
maintainerQuoted 6 lines · Sairaj Kodilkar
Hi Sean,
Sorry for the delay in testing. All sanity tests are OK. I reran the performance test on the V2 and noticed that V2 has significantly more GALOG entries than V1 for all three cases. I also noticed that the Guest Nvme interrupt rate has dropped for the 192 VCPUS.
Hmm, I don't see any obvious bugs or differences (based on a code diffed between v1 and v2). I'll poke at the GALogIntr behavior just to double check, but my guess is that the differences are due to exernal factors, e.g. guest behavior, timing, scheduling, etc.
IOPS are all nearly identical, so I'm not terribly concerned.
Quoted 1 line
I haven't figured out what is causing this.
Might just be slight differences in guest behavior? E.g. did you change the guest kernel?
Quoted 8 lines
I will continue my investigation further.
VCPUS = 32, Jobs per NVME = 8
==============================================================================================
V2 V1 Percent change
----------------------------------------------------------------------------------------------
Guest Nvme interrupts 124,260,796 124,559,110 -0.20%
IOPS (in kilo) 4,790 4,796 -0.01%
Uber nit, the percent change should be -0.10%
Quoted 14 lines
GALOG entries 8117 169 4702%
----------------------------------------------------------------------------------------------VCPUS = 64, Jobs per NVME = 16
==============================================================================================
V2 V1 Percent change
----------------------------------------------------------------------------------------------
Guest Nvme interrupts 102,394,358 99,800,056 2.00%
IOPS (in kilo) 4,796 4,798 -0.04%
GALOG entries 19,057 11,923 59.83%
----------------------------------------------------------------------------------------------
Quoted 6 lines
VCPUS = 192, Jobs per NVME = 48==============================================================================================
V2 V1 Percent change
----------------------------------------------------------------------------------------------
Guest Nvme interrupts 68,363,232 78,066,512 -12.42%
IOPS (in kilo) 4,751 4,749 -0.04%
Uber nit #2, percent change should be postive 0.04%? 4,751 > 4,749.
Quoted 5 lines
GALOG entries 62,768 56,215 11.66%
----------------------------------------------------------------------------------------------Thanks
Sairaj
Quoted 8 lines · Sean Christopherson
Quoted 6 lines · Sairaj Kodilkar
Hi Sean,
Sorry for the delay in testing. All sanity tests are OK. I reran the performance test on the V2 and noticed that V2 has significantly more GALOG entries than V1 for all three cases. I also noticed that the Guest Nvme interrupt rate has dropped for the 192 VCPUS.
Quoted 8 lines
Hmm, I don't see any obvious bugs or differences (based on a code diffed between v1 and v2). I'll poke at the GALogIntr behavior just to double check, but my guess is that the differences are due to exernal factors, e.g. guest behavior, timing, scheduling, etc.
IOPS are all nearly identical, so I'm not terribly concerned.
Yep you are right. I was indeed using different guest kernel to test V2. Keeping it same, I can produce almost identical results for both V1 and V2.
Only one case that still stands out is with 32 vCPUS, where posted interrupt count has increased from 200 to 7000. But IOPS and NVME interrupt rate is identical hence I am not concerned about it as well.
Thanks
Sairaj