Add support for AMD IOMMU GAPPI
Subsystems (7)
v4 cover Add support for AMD IOMMU GAPPI
Introduction
============
On newer generations of AMD processors, IOMMU AVIC/x2AVIC guest-mode interrupt remapping can use Guest APIC Physical Processor Interrupt (GAPPI) as an alternative host-notification path when device interrupts target a vCPU that is not running (IRTE[IsRun] = 0).
With AVIC enabled, the IOMMU posts device interrupts into the guest virtual APIC backing page. When the vCPU is not running, KVM must additionally be notified so it can schedule the vCPU.
The legacy notification path is the GA log (GALOG): the IOMMU appends vCPU tags to a shared GA log buffer and raises a single GA log interrupt. KVM registers a notifier and scans the buffer to decide which vCPUs to wake. Under heavy interrupt load this adds latency and can overflow the buffer because all wakeups funnel through one interrupt and one shared log.
Guest APIC Physical Processor Interrupt (GAPPI), defined in section 2.2.5.4 of the AMD I/O Virtualization Technology (IOMMU) Specification [1], is an alternative. With GAPPI enabled, the IOMMU still updates the guest vAPIC backing page IRR, but may deliver a physical APIC interrupt directly to IRTE[Destination], using IRTE[GATag][7:0] as the vector. This distributes host wakeup notifications across CPUs instead of centralizing them in a log buffer.
This series programs guest-mode IRTEs accordingly: IRTE[Destination] carries the target host physical APIC ID, IRTE[GATag] is set to POSTED_INTR_WAKEUP_VECTOR, and IRTE[GAPPIDis] / IRTE[GALogIntr] are set based on whether KVM requests host wakeup. GAPPI is selected at boot via the amd_iommu=gappi kernel parameter on capable hardware, otherwise the existing GA log path is unchanged.
SVM/AMD IOMMU interface changes
===============================
The first four patches refactor the SVM/AMD IOMMU interface ahead of GAPPI.
The cpu field is renamed to apicid because it carries the host physical APIC ID for IRTE[Destination], not a Linux CPU number.
The ga_log_intr boolean is renamed to wakeup_intr (and the synthetic AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR shadow bit to AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR). wakeup_intr describes KVM's intent (request host wakeup while the vCPU is not running), not a specific hardware mechanism.
A separate is_running boolean is added to IOMMU interface because GAPPI requires a valid apicid in IRTE[Destination] even when the vCPU is not running. The prior encoding (apicid >= 0 means running, apicid == -1 means not running) no longer works once apicid carries the GAPPI destination while IRTE[IsRun] is clear. The IOMMU driver keys IRTE[IsRun] and destination programming off this explicit boolean instead of inferring running state from apicid.
KVM GAPPI wakeup scheme
=======================
SVM follows the Intel posted-interrupt wakeup model already used by VMX. Each pCPU maintains a list of blocked vCPUs that may be woken by a GAPPI delivery to that CPU. When a vCPU blocks while waiting for a device interrupt, SVM enqueues it on the wakeup list of the pCPU on which it was previously running (gappi_cpu) and passes that pCPU's physical APIC ID to the IOMMU to program IRTE[Destination]. The rationale is that the vCPU is likely to run again on the same pCPU, which is common when vCPUs are pinned; targeting GAPPI notifications there reduces unnecessary VMEXITs from GAPPI deliveries on other CPUs. When the vCPU is scheduled in again, it is removed from the list and IRTE[Destination] is updated to the current pCPU.
List maintenance is moved from VMX layer to common KVM layer so that both VMX and SVM layer can share the same list handling code.
Note
====kvm_pi_enable_wakeup_handler() and kvm_pi_disable_wakeup_handler() take the target pCPU explicitly instead of using vcpu->cpu, because on SVM the two can differ. avic_vcpu_load() returns early while the vCPU is blocking and leaves the target wakeup CPU untouched, but the generic layer still refreshes vcpu->cpu on every load. Using vcpu->cpu would then unlink the vCPU under the wrong per-pCPU lock, so SVM tracks the pCPU it last programmed in svm->gappi_cpu and passes that instead.
Changes since v3
================
https://lore.kernel.org/all/[email protected]/
Patch 3:
- Make amd_iommu_activate_guest_mode() as wrapper around __amd_iommu_activate_guest_mode() to in order to hardcode is_running flag to "true" for external calls
Patch 5:
- Newly introduced, moves wakeup list handling from VMX to the KVM layer
Patch 6:
- Use common list handling functions introduced in patch 5.
- Remove list addition during pi_update_irte by moving list handling
functions before list_empty(&svm->ir_list) check.
Changes since v2
================
https://lore.kernel.org/linux-iommu/[email protected]/
Patch[1-6]
- Expand commit messages to explain GAPPI, the interface changes, and the per-CPU wakeup list scheme [Sean].
Patch[1-3]
- Split the monolithic SVM/IOMMU API refactor into four preparatory patches [Sean]
- Rename posted_intr to wakeup_intr to reflect host wakeup intent, not guest interrupt posting [Sean]
- Pass vCPU running status with a extra parameter (is_running) instead of flags.
Patch[4,5]
- Move ga_tag=POSTED_INTR_WAKEUP_VECTOR setting from IOMMU to SVM layer.
Changes since V1:
================
https://lore.kernel.org/all/[email protected]/
Patch4
- Disable interrupts while holding wakeup list lock inside [sashiko] avic_add_vcpu_to_gappi_wakeup_list and avic_remove_vcpu_from_gappi_wakeup_list
- Unregister posted_intr_wakeup_handler during module unload [sashiko]
Patch5
- Disable GAPPI feature during kexec and suspend path [sashiko]
------
[1] https://docs.amd.com/v/u/en-US/48882_3.11_IOMMU_PUB
------
Base: 89516a1c6837156f43525ef49fdbd781145942f5
Sairaj Kodilkar (7):
iommu/amd: KVM: SVM: Rename cpu to apicid in IOMMU interface iommu/amd: KVM: SVM: Rename ga_log_intr to wakeup_intr in IOMMU
interface
iommu/amd: KVM: SVM: Add explicit vCPU running state to IOMMU
interface
iommu/amd: Program guest-mode IRTEs for GAPPI wakeup when IRTE[IsRun]
= 0
KVM: VMX: Factor out wakeup list handling code to KVM
KVM: SVM: Add support for AMD IOMMU Guest APIC Physical Processor
Interrupt (GAPPI)
iommu/amd: Provide kernel command line option to enable GAPPI
.../admin-guide/kernel-parameters.txt | 3 +- arch/x86/include/asm/irq_remapping.h | 5 +- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/include/asm/svm.h | 9 +- arch/x86/kvm/svm/avic.c | 109 +++++++++++++---- arch/x86/kvm/svm/svm.c | 3 + arch/x86/kvm/svm/svm.h | 5 + arch/x86/kvm/vmx/common.h | 3 - arch/x86/kvm/vmx/main.c | 6 + arch/x86/kvm/vmx/posted_intr.c | 82 +------------ arch/x86/kvm/vmx/tdx.c | 1 - arch/x86/kvm/vmx/vmx.c | 14 +-- arch/x86/kvm/vmx/x86_ops.h | 1 + arch/x86/kvm/x86.c | 111 ++++++++++++++++++ drivers/iommu/amd/amd_iommu_types.h | 6 +- drivers/iommu/amd/init.c | 31 ++++- drivers/iommu/amd/iommu.c | 65 ++++++---- include/linux/amd-iommu.h | 16 ++- include/linux/kvm_host.h | 6 + 20 files changed, 323 insertions(+), 155 deletions(-)
Signature · 1 line
2.34.1
PATCH v4 1/7 iommu/amd: KVM: SVM: Rename cpu to apicid in IOMMU interface
The cpu field passed from SVM to the AMD IOMMU driver does not hold a linux CPU number. It carries the host physical APIC ID used to program the IRTE Destination field when the target vCPU is running (IRTE[IsRun] = 1).
Rename the field to apicid everywhere in the IOMMU interface so the name matches the hardware semantics.
No functional change is intended.
--- arch/x86/include/asm/irq_remapping.h | 2 +- arch/x86/kvm/svm/avic.c | 12 +++++----- drivers/iommu/amd/iommu.c | 35 ++++++++++++++-------------- include/linux/amd-iommu.h | 8 +++---- 4 files changed, 29 insertions(+), 28 deletions(-)
arch/x86/include/asm/irq_remapping.h +1 -1
@@ -35,7 +35,7 @@ struct amd_iommu_pi_data {u64 vapic_addr;/* Physical address of the vCPU's vAPIC. */ u32 ga_tag;u32 vector;/* Guest vector of the interrupt */ -int cpu; +int apicid;bool ga_log_intr;bool is_guest_mode;void *ir_data;
arch/x86/kvm/svm/avic.c +6 -6
@@ -949,9 +949,9 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,*/ entry = svm->avic_physical_id_entry;if (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ) { - pi_data.cpu = entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK ; + pi_data.apicid = entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK ; }else { - pi_data.cpu =-1 ; + pi_data.apicid =-1 ; pi_data.ga_log_intr = entry &AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR ; }
@@ -1004,7 +1004,7 @@ enum avic_vcpu_action { AVIC_START_BLOCKING = BIT(1), }; -static void avic_update_iommu_vcpu_affinity (struct kvm_vcpu *vcpu,int cpu, +static void avic_update_iommu_vcpu_affinity (struct kvm_vcpu *vcpu,int apicid,enum avic_vcpu_action action) {bool ga_log_intr = (action &AVIC_START_BLOCKING );
@@ -1024,9 +1024,9 @@ static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu,void *data = irqfd->irq_bypass_data;if (!(action &AVIC_TOGGLE_ON_OFF )) -WARN_ON_ONCE (amd_iommu_update_ga (data, cpu, ga_log_intr)); -else if (cpu >=0 ) -WARN_ON_ONCE (amd_iommu_activate_guest_mode (data, cpu, ga_log_intr)); +WARN_ON_ONCE (amd_iommu_update_ga (data, apicid, ga_log_intr)); +else if (apicid >=0 ) +WARN_ON_ONCE (amd_iommu_activate_guest_mode (data, apicid, ga_log_intr));else WARN_ON_ONCE (amd_iommu_deactivate_guest_mode (data)); }
drivers/iommu/amd/iommu.c +18 -17
@@ -3959,14 +3959,14 @@ static const struct irq_domain_ops amd_ir_domain_ops = { .deactivate = irq_remapping_deactivate, }; -static void __amd_iommu_update_ga (struct irte_ga *entry,int cpu, +static void __amd_iommu_update_ga (struct irte_ga *entry,int apicid,bool ga_log_intr) { -if (cpu >=0 ) { +if (apicid >=0 ) { entry->lo.fields_vapic.destination = -APICID_TO_IRTE_DEST_LO (cpu); +APICID_TO_IRTE_DEST_LO (apicid); entry->hi.fields.destination = -APICID_TO_IRTE_DEST_HI (cpu); +APICID_TO_IRTE_DEST_HI (apicid); entry->lo.fields_vapic.is_run = true; entry->lo.fields_vapic.ga_log_intr = false; }else {
@@ -3979,20 +3979,21 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int cpu,* Update the pCPU information for an IRTE that is configured to post IRQs to * a vCPU, without issuing an IOMMU invalidation for the IRTE. * -* If the vCPU is associated with a pCPU (@cpu >= 0), configure the Destination -* with the pCPU's APIC ID, set IsRun, and clear GALogIntr. If the vCPU isn't -* associated with a pCPU (@cpu < 0), clear IsRun and set/clear GALogIntr based -* on input from the caller (e.g. KVM only requests GALogIntr when the vCPU is -* blocking and requires a notification wake event). I.e. treat vCPUs that are -* associated with a pCPU as running. This API is intended to be used when a -* vCPU is scheduled in/out (or stops running for any reason), to do a fast -* update of IsRun, GALogIntr, and (conditionally) Destination. +* If the vCPU is associated with a pCPU (@apicid >= 0), configure the +* Destination with the pCPU's APIC ID, set IsRun, and clear GALogIntr. If the +* vCPU isn't associated with a pCPU (@apicid < 0), clear IsRun and set/clear +* GALogIntr based on input from the caller (e.g. KVM only requests GALogIntr +* when the vCPU is blocking and requires a notification wake event). I.e. +* treat vCPUs that are associated with a pCPU as running. This API is +* intended to be used when a vCPU is scheduled in/out (or stops running for +* any reason), to do a fast update of IsRun, GALogIntr, and (conditionally) +* Destination. * * Per the IOMMU spec, the Destination, IsRun, and GATag fields are not cached * and thus don't require an invalidation to ensure the IOMMU consumes fresh * information. */ -int amd_iommu_update_ga (void *data,int cpu,bool ga_log_intr) +int amd_iommu_update_ga (void *data,int apicid,bool ga_log_intr) {struct amd_ir_data *ir_data = (struct amd_ir_data *)data;struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -4006,14 +4007,14 @@ int amd_iommu_update_ga(void *data, int cpu, bool ga_log_intr)if (!ir_data->iommu)return -ENODEV ; -__amd_iommu_update_ga (entry, cpu, ga_log_intr); +__amd_iommu_update_ga (entry, apicid, ga_log_intr);return __modify_irte_ga (ir_data->iommu, ir_data->irq_2_irte.devid, ir_data->irq_2_irte.index, entry); }EXPORT_SYMBOL (amd_iommu_update_ga); -int amd_iommu_activate_guest_mode (void *data,int cpu,bool ga_log_intr) +int amd_iommu_activate_guest_mode (void *data,int apicid,bool ga_log_intr) {struct amd_ir_data *ir_data = (struct amd_ir_data *)data;struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -4036,7 +4037,7 @@ int amd_iommu_activate_guest_mode(void *data, int cpu, bool ga_log_intr) entry->hi.fields.vector = ir_data->ga_vector; entry->lo.fields_vapic.ga_tag = ir_data->ga_tag; -__amd_iommu_update_ga (entry, cpu, ga_log_intr); +__amd_iommu_update_ga (entry, apicid, ga_log_intr);return modify_irte_ga (ir_data->iommu, ir_data->irq_2_irte.devid, ir_data->irq_2_irte.index, entry);
@@ -4107,7 +4108,7 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *info) ir_data->ga_vector = pi_data->vector; ir_data->ga_tag = pi_data->ga_tag;if (pi_data->is_guest_mode) - ret =amd_iommu_activate_guest_mode (ir_data, pi_data->cpu, + ret =amd_iommu_activate_guest_mode (ir_data, pi_data->apicid, pi_data->ga_log_intr);else ret =amd_iommu_deactivate_guest_mode (ir_data);
include/linux/amd-iommu.h +4 -4
@@ -30,8 +30,8 @@ static inline void amd_iommu_detect(void) { }/* IOMMU AVIC Function */ extern int amd_iommu_register_ga_log_notifier (int (*notifier)(u32 )); -extern int amd_iommu_update_ga (void *data,int cpu,bool ga_log_intr); -extern int amd_iommu_activate_guest_mode (void *data,int cpu,bool ga_log_intr); +extern int amd_iommu_update_ga (void *data,int apicid,bool ga_log_intr); +extern int amd_iommu_activate_guest_mode (void *data,int apicid,bool ga_log_intr);extern int amd_iommu_deactivate_guest_mode (void *data);#else /* defined(CONFIG_AMD_IOMMU) && defined(CONFIG_IRQ_REMAP) */
@@ -42,12 +42,12 @@ amd_iommu_register_ga_log_notifier(int (*notifier)(u32))return 0 ; } -static inline int amd_iommu_update_ga (void *data,int cpu,bool ga_log_intr) +static inline int amd_iommu_update_ga (void *data,int apicid,bool ga_log_intr) {return 0 ; } -static inline int amd_iommu_activate_guest_mode (void *data,int cpu,bool ga_log_intr) +static inline int amd_iommu_activate_guest_mode (void *data,int apicid,bool ga_log_intr) {return 0 ; }
Signature · 1 line
2.34.1
Re: [PATCH v4 1/7] iommu/amd: KVM: SVM: Rename cpu to apicid in IOMMU interface
Quoted 11 lines · Sairaj Kodilkar
The cpu field passed from SVM to the AMD IOMMU driver does not hold a linux CPU number. It carries the host physical APIC ID used to program the IRTE Destination field when the target vCPU is running (IRTE[IsRun] = 1).
Rename the field to apicid everywhere in the IOMMU interface so the name matches the hardware semantics.
No functional change is intended.
Signed-off-by Sairaj Kodilkar <[email protected]>
-Vasant
PATCH v4 2/7 iommu/amd: KVM: SVM: Rename ga_log_intr to wakeup_intr in IOMMU interface
In reply to Sairaj KodilkarWhen AVIC is enabled, device interrupts are posted into the guest virtual APIC backing page regardless of vCPU running state. When the vCPU is not running, KVM may additionally need the IOMMU to notify the host so the vCPU can be woken, historically via the GA log mechanism.
The existing ga_log_intr name is tied to one specific hardware mechanism. Guest APIC Physical Processor Interrupt (GAPPI) provides an alternative host-notification path when the vCPU is not running, i.e., IRTE[IsRun] = 0. The KVM-side intent is the same in both cases: request host wakeup notification while the vCPU is blocked.
Rename ga_log_intr to wakeup_intr in the IOMMU API to describe KVM's intent rather than a single hardware knob. The IOMMU driver still maps wakeup_intr to IRTE[GALogIntr] in this patch; GAPPI support is added in following patches.
Similarly rename the synthetic AVIC physical ID table shadow bit from AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR to AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR.
No functional change is intended.
--- arch/x86/include/asm/irq_remapping.h | 2 +- arch/x86/include/asm/svm.h | 9 +++++---- arch/x86/kvm/svm/avic.c | 26 +++++++++++++------------- drivers/iommu/amd/iommu.c | 16 ++++++++-------- include/linux/amd-iommu.h | 8 ++++---- 5 files changed, 31 insertions(+), 30 deletions(-)
arch/x86/include/asm/irq_remapping.h +1 -1
@@ -36,7 +36,7 @@ struct amd_iommu_pi_data {u32 ga_tag;u32 vector;/* Guest vector of the interrupt */ int apicid; -bool ga_log_intr; +bool wakeup_intr;bool is_guest_mode;void *ir_data; };
arch/x86/include/asm/svm.h +5 -4
@@ -259,11 +259,12 @@ struct __attribute__ ((__packed__)) vmcb_control_area {#define AVIC_LOGICAL_ID_ENTRY_VALID_MASK (1 << 31)/* -* GA_LOG_INTR is a synthetic flag that's never propagated to hardware-visible -* tables. GA_LOG_INTR is set if the vCPU needs device posted IRQs to generate -* GA log interrupts to wake the vCPU (because it's blocking or about to block). +* WAKEUP_INTR is a synthetic flag that's never propagated to hardware-visible +* tables. WAKEUP_INTR is set if the vCPU needs device posted IRQs to generate +* GA log/GAPPI interrupts to wake the vCPU (because it's blocking or about to +* block). */ -#define AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR BIT_ULL(61) +#define AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR BIT_ULL(61)#define AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK GENMASK_ULL(11, 0)#define AVIC_PHYSICAL_ID_ENTRY_BACKING_PAGE_MASK GENMASK_ULL(51, 12)
arch/x86/kvm/svm/avic.c +13 -13
@@ -952,7 +952,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, pi_data.apicid = entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK ; }else { pi_data.apicid =-1 ; - pi_data.ga_log_intr = entry &AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR ; + pi_data.wakeup_intr = entry &AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR ; } ret =irq_set_vcpu_affinity (host_irq, &pi_data);
@@ -991,15 +991,15 @@ enum avic_vcpu_action {* No unique action is required to deal with a vCPU that stops/starts * running. A vCPU that starts running by definition stops blocking as * well, and a vCPU that stops running can't have been blocking, i.e. -* doesn't need to toggle GALogIntr. +* doesn't need to toggle host wakeup notification. */ AVIC_START_RUNNING =0 ,AVIC_STOP_RUNNING =0 ,/* -* When a vCPU starts blocking, KVM needs to set the GALogIntr flag -* int all associated IRTEs so that KVM can wake the vCPU if an IRQ is -* sent to the vCPU. +* When a vCPU starts blocking, KVM needs to request host wakeup +* notification in all associated IRTEs so that KVM can wake the vCPU +* if an IRQ is sent to the vCPU. */ AVIC_START_BLOCKING = BIT(1), };
@@ -1007,7 +1007,7 @@ enum avic_vcpu_action {static void avic_update_iommu_vcpu_affinity (struct kvm_vcpu *vcpu,int apicid,enum avic_vcpu_action action) { -bool ga_log_intr = (action &AVIC_START_BLOCKING ); +bool wakeup_intr = (action &AVIC_START_BLOCKING );struct vcpu_svm *svm =to_svm (vcpu);struct kvm_kernel_irqfd *irqfd;
@@ -1024,9 +1024,9 @@ static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid,void *data = irqfd->irq_bypass_data;if (!(action &AVIC_TOGGLE_ON_OFF )) -WARN_ON_ONCE (amd_iommu_update_ga (data, apicid, ga_log_intr)); +WARN_ON_ONCE (amd_iommu_update_ga (data, apicid, wakeup_intr));else if (apicid >=0 ) -WARN_ON_ONCE (amd_iommu_activate_guest_mode (data, apicid, ga_log_intr)); +WARN_ON_ONCE (amd_iommu_activate_guest_mode (data, apicid, wakeup_intr));else WARN_ON_ONCE (amd_iommu_deactivate_guest_mode (data)); }
@@ -1063,7 +1063,7 @@ static void __avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu,WARN_ON_ONCE (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ); entry &= ~(AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK | -AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR ); +AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR ); entry |= (h_physical_id &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK ); entry |=AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ;
@@ -1125,7 +1125,7 @@ static void __avic_vcpu_put(struct kvm_vcpu *vcpu, enum avic_vcpu_action action)avic_update_iommu_vcpu_affinity (vcpu,-1 , action); -WARN_ON_ONCE (entry &AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR ); +WARN_ON_ONCE (entry &AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR );/* * Keep the previous APIC ID in the entry so that a rogue doorbell from
@@ -1137,11 +1137,11 @@ static void __avic_vcpu_put(struct kvm_vcpu *vcpu, enum avic_vcpu_action action)WRITE_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id], entry);/* -* Note! Don't set AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR in the table as +* Note! Don't set AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR in the table as * it's a synthetic flag that usurps an unused should-be-zero bit. */ if (action &AVIC_START_BLOCKING ) - entry |=AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR ; + entry |=AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR ; svm->avic_physical_id_entry = entry;
@@ -1173,7 +1173,7 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu)* The vCPU was preempted while blocking, ensure its IRTEs are * configured to generate GA Log Interrupts. */ -if (!(WARN_ON_ONCE (!(entry &AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR )))) +if (!(WARN_ON_ONCE (!(entry &AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR ))))return ; }
drivers/iommu/amd/iommu.c +8 -8
@@ -3960,7 +3960,7 @@ static const struct irq_domain_ops amd_ir_domain_ops = { };static void __amd_iommu_update_ga (struct irte_ga *entry,int apicid, -bool ga_log_intr) +bool wakeup_intr) {if (apicid >=0 ) { entry->lo.fields_vapic.destination =
@@ -3971,7 +3971,7 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid, entry->lo.fields_vapic.ga_log_intr = false; }else { entry->lo.fields_vapic.is_run = false; - entry->lo.fields_vapic.ga_log_intr = ga_log_intr; + entry->lo.fields_vapic.ga_log_intr = wakeup_intr; } }
@@ -3982,7 +3982,7 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,* If the vCPU is associated with a pCPU (@apicid >= 0), configure the * Destination with the pCPU's APIC ID, set IsRun, and clear GALogIntr. If the * vCPU isn't associated with a pCPU (@apicid < 0), clear IsRun and set/clear -* GALogIntr based on input from the caller (e.g. KVM only requests GALogIntr +* GALogIntr based on input from the caller (e.g. KVM only requests wakeup_intr * when the vCPU is blocking and requires a notification wake event). I.e. * treat vCPUs that are associated with a pCPU as running. This API is * intended to be used when a vCPU is scheduled in/out (or stops running for
@@ -3993,7 +3993,7 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,* and thus don't require an invalidation to ensure the IOMMU consumes fresh * information. */ -int amd_iommu_update_ga (void *data,int apicid,bool ga_log_intr) +int amd_iommu_update_ga (void *data,int apicid,bool wakeup_intr) {struct amd_ir_data *ir_data = (struct amd_ir_data *)data;struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -4007,14 +4007,14 @@ int amd_iommu_update_ga(void *data, int apicid, bool ga_log_intr)if (!ir_data->iommu)return -ENODEV ; -__amd_iommu_update_ga (entry, apicid, ga_log_intr); +__amd_iommu_update_ga (entry, apicid, wakeup_intr);return __modify_irte_ga (ir_data->iommu, ir_data->irq_2_irte.devid, ir_data->irq_2_irte.index, entry); }EXPORT_SYMBOL (amd_iommu_update_ga); -int amd_iommu_activate_guest_mode (void *data,int apicid,bool ga_log_intr) +int amd_iommu_activate_guest_mode (void *data,int apicid,bool wakeup_intr) {struct amd_ir_data *ir_data = (struct amd_ir_data *)data;struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -4037,7 +4037,7 @@ int amd_iommu_activate_guest_mode(void *data, int apicid, bool ga_log_intr) entry->hi.fields.vector = ir_data->ga_vector; entry->lo.fields_vapic.ga_tag = ir_data->ga_tag; -__amd_iommu_update_ga (entry, apicid, ga_log_intr); +__amd_iommu_update_ga (entry, apicid, wakeup_intr);return modify_irte_ga (ir_data->iommu, ir_data->irq_2_irte.devid, ir_data->irq_2_irte.index, entry);
@@ -4109,7 +4109,7 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *info) ir_data->ga_tag = pi_data->ga_tag;if (pi_data->is_guest_mode) ret =amd_iommu_activate_guest_mode (ir_data, pi_data->apicid, - pi_data->ga_log_intr); + pi_data->wakeup_intr);else ret =amd_iommu_deactivate_guest_mode (ir_data); }else {
include/linux/amd-iommu.h +4 -4
@@ -30,8 +30,8 @@ static inline void amd_iommu_detect(void) { }/* IOMMU AVIC Function */ extern int amd_iommu_register_ga_log_notifier (int (*notifier)(u32 )); -extern int amd_iommu_update_ga (void *data,int apicid,bool ga_log_intr); -extern int amd_iommu_activate_guest_mode (void *data,int apicid,bool ga_log_intr); +extern int amd_iommu_update_ga (void *data,int apicid,bool wakeup_intr); +extern int amd_iommu_activate_guest_mode (void *data,int apicid,bool wakeup_intr);extern int amd_iommu_deactivate_guest_mode (void *data);#else /* defined(CONFIG_AMD_IOMMU) && defined(CONFIG_IRQ_REMAP) */
@@ -42,12 +42,12 @@ amd_iommu_register_ga_log_notifier(int (*notifier)(u32))return 0 ; } -static inline int amd_iommu_update_ga (void *data,int apicid,bool ga_log_intr) +static inline int amd_iommu_update_ga (void *data,int apicid,bool wakeup_intr) {return 0 ; } -static inline int amd_iommu_activate_guest_mode (void *data,int apicid,bool ga_log_intr) +static inline int amd_iommu_activate_guest_mode (void *data,int apicid,bool wakeup_intr) {return 0 ; }
Signature · 1 line
2.34.1
Re: [PATCH v4 2/7] iommu/amd: KVM: SVM: Rename ga_log_intr to wakeup_intr in IOMMU interface
Quoted 16 lines · Sairaj Kodilkar
When AVIC is enabled, device interrupts are posted into the guest virtual APIC backing page regardless of vCPU running state. When the vCPU is not running, KVM may additionally need the IOMMU to notify the host so the vCPU can be woken, historically via the GA log mechanism.
The existing
ga_log_intrname is tied to one specific hardware mechanism. Guest APIC Physical Processor Interrupt (GAPPI) provides an alternative host-notification path when the vCPU is not running, i.e., IRTE[IsRun] = 0. The KVM-side intent is the same in both cases: request host wakeup notification while the vCPU is blocked.Rename
ga_log_intrtowakeup_intrin the IOMMU API to describe KVM's intent rather than a single hardware knob. The IOMMU driver still mapswakeup_intrto IRTE[GALogIntr] in this patch; GAPPI support is added in following patches.
Quoted 6 lines
Similarly rename the synthetic AVIC physical ID table shadow bit from
AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTRtoAVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR.No functional change is intended.
Signed-off-by Sairaj Kodilkar <[email protected]>
-Vasant
PATCH v4 3/7 iommu/amd: KVM: SVM: Add explicit vCPU running state to IOMMU interface
In reply to Sairaj KodilkarGuest APIC Physical Processor Interrupt (GAPPI) is an AMD IOMMU mechanism for notifying the host when a device interrupt targets a non-running vCPU (IRTE[IsRun] = 0). Unlike the GA log path, GAPPI delivers a physical APIC interrupt directly to the host CPU described by IRTE[Destination], with the vector carried in IRTE[GATag], while still posting the interrupt into the guest virtual APIC backing page as usual.
In GAPPI mode, the IOMMU delivers the notification to the host CPU using the physical APIC ID in IRTE[Destination] and the vector in IRTE[GATag]. KVM must therefore supply a valid apicid even when the vCPU is not running. The prior interface inferred running state from apicid: apicid >= 0 meant running and apicid == -1 meant not running. That encoding breaks once apicid carries the GAPPI destination while the vCPU is not running; the IOMMU driver can no longer derive vCPU running state from apicid alone and needs an explicit indication from KVM.
Add an explicit is_running indication to the IOMMU interface so that SVM can pass vCPU running state to the IOMMU.
SVM calls amd_iommu_activate_guest_mode() only when the vCPU is running, so keep is_running out of the exported prototype and make it a thin wrapper that hardcodes is_running = true. The IOMMU internal caller, amd_ir_set_vcpu_affinity(), calls __amd_iommu_activate_guest_mode() directly and passes the running state supplied by KVM, i.e. it handles both a running and a non-running vCPU.
No functional change is intended.
--- arch/x86/include/asm/irq_remapping.h | 1 + arch/x86/kvm/svm/avic.c | 7 ++++-- drivers/iommu/amd/iommu.c | 34 ++++++++++++++++------------ include/linux/amd-iommu.h | 6 +++-- 4 files changed, 30 insertions(+), 18 deletions(-)
arch/x86/include/asm/irq_remapping.h +1 -0
@@ -37,6 +37,7 @@ struct amd_iommu_pi_data {u32 vector;/* Guest vector of the interrupt */ int apicid;bool wakeup_intr; +bool is_running;bool is_guest_mode;void *ir_data; };
arch/x86/kvm/svm/avic.c +5 -2
@@ -950,9 +950,11 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, entry = svm->avic_physical_id_entry;if (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ) { pi_data.apicid = entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK ; + pi_data.is_running = true; }else { pi_data.apicid =-1 ; pi_data.wakeup_intr = entry &AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR ; + pi_data.is_running = false; } ret =irq_set_vcpu_affinity (host_irq, &pi_data);
@@ -1008,6 +1010,7 @@ static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid, enum avic_vcpu_action action) {bool wakeup_intr = (action &AVIC_START_BLOCKING ); +bool is_running = apicid >=0 ;struct vcpu_svm *svm =to_svm (vcpu);struct kvm_kernel_irqfd *irqfd;
@@ -1024,8 +1027,8 @@ static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid,void *data = irqfd->irq_bypass_data;if (!(action &AVIC_TOGGLE_ON_OFF )) -WARN_ON_ONCE (amd_iommu_update_ga (data, apicid, wakeup_intr)); -else if (apicid >=0 ) +WARN_ON_ONCE (amd_iommu_update_ga (data, apicid, wakeup_intr, is_running)); +else if (is_running)WARN_ON_ONCE (amd_iommu_activate_guest_mode (data, apicid, wakeup_intr));else WARN_ON_ONCE (amd_iommu_deactivate_guest_mode (data));
drivers/iommu/amd/iommu.c +20 -14
@@ -3960,9 +3960,9 @@ static const struct irq_domain_ops amd_ir_domain_ops = { };static void __amd_iommu_update_ga (struct irte_ga *entry,int apicid, -bool wakeup_intr) +bool wakeup_intr,bool is_running) { -if (apicid >=0 ) { +if (is_running) { entry->lo.fields_vapic.destination =APICID_TO_IRTE_DEST_LO (apicid); entry->hi.fields.destination =
@@ -3979,12 +3979,11 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,* Update the pCPU information for an IRTE that is configured to post IRQs to * a vCPU, without issuing an IOMMU invalidation for the IRTE. * -* If the vCPU is associated with a pCPU (@apicid >= 0), configure the -* Destination with the pCPU's APIC ID, set IsRun, and clear GALogIntr. If the -* vCPU isn't associated with a pCPU (@apicid < 0), clear IsRun and set/clear -* GALogIntr based on input from the caller (e.g. KVM only requests wakeup_intr -* when the vCPU is blocking and requires a notification wake event). I.e. -* treat vCPUs that are associated with a pCPU as running. This API is +* If the vCPU is scheduled to run on pCPU (@is_running = 1), configure the +* Destination with the pCPU's APIC ID, set IsRun, and clear GALogIntr. If the +* vCPU is scheduled out (@is_running = 0), clear IsRun and set/clear GALogIntr +* based on input from the caller (e.g. KVM only requests wakeup_intr when the +* vCPU is blocking and requires a notification wake event). This API is * intended to be used when a vCPU is scheduled in/out (or stops running for * any reason), to do a fast update of IsRun, GALogIntr, and (conditionally) * Destination.
@@ -3993,7 +3992,7 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,* and thus don't require an invalidation to ensure the IOMMU consumes fresh * information. */ -int amd_iommu_update_ga (void *data,int apicid,bool wakeup_intr) +int amd_iommu_update_ga (void *data,int apicid,bool wakeup_intr,bool is_running) {struct amd_ir_data *ir_data = (struct amd_ir_data *)data;struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -4007,14 +4006,15 @@ int amd_iommu_update_ga(void *data, int apicid, bool wakeup_intr)if (!ir_data->iommu)return -ENODEV ; -__amd_iommu_update_ga (entry, apicid, wakeup_intr); +__amd_iommu_update_ga (entry, apicid, wakeup_intr, is_running);return __modify_irte_ga (ir_data->iommu, ir_data->irq_2_irte.devid, ir_data->irq_2_irte.index, entry); }EXPORT_SYMBOL (amd_iommu_update_ga); -int amd_iommu_activate_guest_mode (void *data,int apicid,bool wakeup_intr) +static int __amd_iommu_activate_guest_mode (void *data,int apicid,bool wakeup_intr, +bool is_running) {struct amd_ir_data *ir_data = (struct amd_ir_data *)data;struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -4037,11 +4037,16 @@ int amd_iommu_activate_guest_mode(void *data, int apicid, bool wakeup_intr) entry->hi.fields.vector = ir_data->ga_vector; entry->lo.fields_vapic.ga_tag = ir_data->ga_tag; -__amd_iommu_update_ga (entry, apicid, wakeup_intr); +__amd_iommu_update_ga (entry, apicid, wakeup_intr, is_running);return modify_irte_ga (ir_data->iommu, ir_data->irq_2_irte.devid, ir_data->irq_2_irte.index, entry); } + +int amd_iommu_activate_guest_mode (void *data,int apicid,bool wakeup_intr) +{ +return __amd_iommu_activate_guest_mode (data, apicid, wakeup_intr, true); +}EXPORT_SYMBOL (amd_iommu_activate_guest_mode);int amd_iommu_deactivate_guest_mode (void *data)
@@ -4108,8 +4113,9 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *info) ir_data->ga_vector = pi_data->vector; ir_data->ga_tag = pi_data->ga_tag;if (pi_data->is_guest_mode) - ret =amd_iommu_activate_guest_mode (ir_data, pi_data->apicid, - pi_data->wakeup_intr); + ret =__amd_iommu_activate_guest_mode (ir_data, pi_data->apicid, + pi_data->wakeup_intr, + pi_data->is_running);else ret =amd_iommu_deactivate_guest_mode (ir_data); }else {
include/linux/amd-iommu.h +4 -2
@@ -30,7 +30,8 @@ static inline void amd_iommu_detect(void) { }/* IOMMU AVIC Function */ extern int amd_iommu_register_ga_log_notifier (int (*notifier)(u32 )); -extern int amd_iommu_update_ga (void *data,int apicid,bool wakeup_intr); +extern int amd_iommu_update_ga (void *data,int apicid,bool wakeup_intr, +bool is_running);extern int amd_iommu_activate_guest_mode (void *data,int apicid,bool wakeup_intr);extern int amd_iommu_deactivate_guest_mode (void *data);
@@ -42,7 +43,8 @@ amd_iommu_register_ga_log_notifier(int (*notifier)(u32))return 0 ; } -static inline int amd_iommu_update_ga (void *data,int apicid,bool wakeup_intr) +static inline int amd_iommu_update_ga (void *data,int apicid,bool wakeup_intr, +bool is_running) {return 0 ; }
Signature · 1 line
2.34.1
Re: [PATCH v4 3/7] iommu/amd: KVM: SVM: Add explicit vCPU running state to IOMMU interface
Quoted 26 lines · Sairaj Kodilkar
Guest APIC Physical Processor Interrupt (GAPPI) is an AMD IOMMU mechanism for notifying the host when a device interrupt targets a non-running vCPU (IRTE[IsRun] = 0). Unlike the GA log path, GAPPI delivers a physical APIC interrupt directly to the host CPU described by IRTE[Destination], with the vector carried in IRTE[GATag], while still posting the interrupt into the guest virtual APIC backing page as usual.
In GAPPI mode, the IOMMU delivers the notification to the host CPU using the physical APIC ID in IRTE[Destination] and the vector in IRTE[GATag]. KVM must therefore supply a valid apicid even when the vCPU is not running. The prior interface inferred running state from apicid: apicid >= 0 meant running and apicid == -1 meant not running. That encoding breaks once apicid carries the GAPPI destination while the vCPU is not running; the IOMMU driver can no longer derive vCPU running state from apicid alone and needs an explicit indication from KVM.
Add an explicit
is_runningindication to the IOMMU interface so that SVM can pass vCPU running state to the IOMMU.SVM calls
amd_iommu_activate_guest_mode()only when the vCPU is running, so keepis_runningout of the exported prototype and make it a thin wrapper that hardcodesis_running= true. The IOMMU internal caller,amd_ir_set_vcpu_affinity(), calls__amd_iommu_activate_guest_mode()directly and passes the running state supplied by KVM, i.e. it handles both a running and a non-running vCPU.
Quoted 3 lines
No functional change is intended.
Signed-off-by Sairaj Kodilkar <[email protected]>
-Vasant
PATCH v4 4/7 iommu/amd: Program guest-mode IRTEs for GAPPI wakeup when IRTE[IsRun] = 0
In reply to Sairaj KodilkarGuest APIC Physical Processor Interrupt (GAPPI) is an alternative to the GA log for notifying the host when a device interrupt targets a non-running vCPU (IRTE[IsRun] = 0). Per the AMD IOMMU specification [1] (section 2.2.5.4), with GAPPI enabled the IOMMU delivers a physical APIC interrupt to the CPU described by IRTE[Destination], using IRTE[GATag][7:0] as the vector, while still updating the guest vAPIC backing page IRR as usual.
The AMD IOMMU also allows suppressing GAPPI interrupts using the IRTE[GAPPIDis] bit when the GAPPIDisSup feature is available in extended feature register 2. The AMD IOMMU driver sets this bit when the wakeup_intr flag is not set by KVM.
Note: amd_iommu_gappi is currently false; a later patch enables it via the kernel command line.
[1] https://docs.amd.com/v/u/en-US/48882_3.11_IOMMU_PUB
--- drivers/iommu/amd/amd_iommu_types.h | 4 +++- drivers/iommu/amd/init.c | 3 +++ drivers/iommu/amd/iommu.c | 30 +++++++++++++++++++---------- include/linux/amd-iommu.h | 6 ++++++ 4 files changed, 32 insertions(+), 11 deletions(-)
drivers/iommu/amd/amd_iommu_types.h +3 -1
@@ -113,6 +113,7 @@/* Extended Feature 2 Bits */ #define FEATURE_SEVSNPIO_SUP BIT_ULL(1)#define FEATURE_GCR3TRPMODE BIT_ULL(3) +#define FEATURE_GAPPIDISSUP BIT_ULL(4)#define FEATURE_SNPAVICSUP GENMASK_ULL(7, 5)#define FEATURE_SNPAVICSUP_GAM (x) \ (FIELD_GET(FEATURE_SNPAVICSUP, x) == 0x1)
@@ -1004,7 +1005,8 @@ union irte_ga_lo { no_fault : 1, /* ------ */ ga_log_intr : 1, - rsvd1 : 3, + rsvd1 : 2, + gappi_dis : 1, is_run : 1, /* ------ */ guest_mode : 1,
drivers/iommu/amd/init.c +3 -0
@@ -160,6 +160,9 @@ u8 amd_iommu_hpt_level;/* Guest page table level */ int amd_iommu_gpt_level =PAGE_MODE_4_LEVEL ; +bool amd_iommu_gappi; +EXPORT_SYMBOL (amd_iommu_gappi); +int amd_iommu_guest_ir =AMD_IOMMU_GUEST_IR_VAPIC ;static int amd_iommu_xt_mode =IRQ_REMAP_XAPIC_MODE ;
drivers/iommu/amd/iommu.c +20 -10
@@ -3970,8 +3970,18 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid, entry->lo.fields_vapic.is_run = true; entry->lo.fields_vapic.ga_log_intr = false; }else { - entry->lo.fields_vapic.is_run = false; - entry->lo.fields_vapic.ga_log_intr = wakeup_intr; +if (amd_iommu_gappi) { + entry->lo.fields_vapic.gappi_dis = !wakeup_intr && +check_feature2 (FEATURE_GAPPIDISSUP ); + entry->lo.fields_vapic.is_run = false; + entry->lo.fields_vapic.destination = +APICID_TO_IRTE_DEST_LO (apicid); + entry->hi.fields.destination = +APICID_TO_IRTE_DEST_HI (apicid); + }else { + entry->lo.fields_vapic.is_run = false; + entry->lo.fields_vapic.ga_log_intr = wakeup_intr; + } } }
@@ -3982,15 +3992,15 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,* If the vCPU is scheduled to run on pCPU (@is_running = 1), configure the * Destination with the pCPU's APIC ID, set IsRun, and clear GALogIntr. If the * vCPU is scheduled out (@is_running = 0), clear IsRun and set/clear GALogIntr -* based on input from the caller (e.g. KVM only requests wakeup_intr when the -* vCPU is blocking and requires a notification wake event). This API is -* intended to be used when a vCPU is scheduled in/out (or stops running for -* any reason), to do a fast update of IsRun, GALogIntr, and (conditionally) -* Destination. +* and GAPPIDis based on input from the caller (e.g. KVM only requests +* wakeup_intr when the vCPU is blocking and requires a notification wake +* event). This API is intended to be used when a vCPU is scheduled in/out (or +* stops running for any reason), to do a fast update of IsRun, GALogIntr, +* GAPPIDis and (conditionally) Destination. * -* Per the IOMMU spec, the Destination, IsRun, and GATag fields are not cached -* and thus don't require an invalidation to ensure the IOMMU consumes fresh -* information. +* Per the IOMMU spec, the Destination, IsRun, GATag and GAPPIDis fields are +* not cached and thus don't require an invalidation to ensure the IOMMU +* consumes fresh information. */ int amd_iommu_update_ga (void *data,int apicid,bool wakeup_intr,bool is_running) {
include/linux/amd-iommu.h +6 -0
@@ -35,6 +35,9 @@ extern int amd_iommu_update_ga(void *data, int apicid, bool wakeup_intr,extern int amd_iommu_activate_guest_mode (void *data,int apicid,bool wakeup_intr);extern int amd_iommu_deactivate_guest_mode (void *data); +/* IOMMU AVIC Flag */ +extern bool amd_iommu_gappi; +#else /* defined(CONFIG_AMD_IOMMU) && defined(CONFIG_IRQ_REMAP) */ static inline int
@@ -58,6 +61,9 @@ static inline int amd_iommu_deactivate_guest_mode(void *data) {return 0 ; } + +#define amd_iommu_gappi false +#endif /* defined(CONFIG_AMD_IOMMU) && defined(CONFIG_IRQ_REMAP) */ int amd_iommu_get_num_iommus (void );
Signature · 1 line
2.34.1
Re: [PATCH v4 4/7] iommu/amd: Program guest-mode IRTEs for GAPPI wakeup when IRTE[IsRun] = 0
Quoted 13 lines · Sairaj Kodilkar
Guest APIC Physical Processor Interrupt (GAPPI) is an alternative to the GA log for notifying the host when a device interrupt targets a non-running vCPU (IRTE[IsRun] = 0). Per the AMD IOMMU specification [1] (section 2.2.5.4), with GAPPI enabled the IOMMU delivers a physical APIC interrupt to the CPU described by IRTE[Destination], using IRTE[GATag][7:0] as the vector, while still updating the guest vAPIC backing page IRR as usual.
The AMD IOMMU also allows suppressing GAPPI interrupts using the IRTE[GAPPIDis] bit when the GAPPIDisSup feature is available in extended feature register 2. The AMD IOMMU driver sets this bit when the
wakeup_intrflag is not set by KVM.
Quoted 6 lines
Note:
amd_iommu_gappiis currently false; a later patch enables it via the kernel command line.[1] https://docs.amd.com/v/u/en-US/48882_3.11_IOMMU_PUB
Signed-off-by Sairaj Kodilkar <[email protected]>
-Vasant
PATCH 1/1 iommu/amd: Reset IRTE notification bits on every GA update
In reply to Sairaj KodilkarFrom: Jinmeng Zhou <[email protected]>
__amd_iommu_update_ga() clears GALogIntr when the vCPU starts running, but leaves GAPPIDis untouched, so a GAPPIDis value programmed while the vCPU was scheduled out is carried over into the running IRTE. GAPPIDis is recomputed every time the vCPU is put, so the stale value is only ever visible while the vCPU is running. GALogIntr and GAPPIDis are never both in use, as GAPPI vs. GA log is selected when the IOMMU is initialized.
Clear both bits up front and let the not-running path set whichever one the current wakeup policy needs, so that each update fully derives the notification bits from the new state. The only resulting difference is that GAPPIDis is now clear, rather than stale, in IRTEs with IsRun set. Update the kernel-doc accordingly.
---
This is a suggested cleanup on top of v4 4/7; please feel free to squash it into that patch for the next revision.
drivers/iommu/amd/iommu.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/iommu/amd/iommu.c +14 -8
@@ -3962,13 +3962,19 @@ static const struct irq_domain_ops amd_ir_domain_ops = {static void __amd_iommu_update_ga (struct irte_ga *entry,int apicid,bool wakeup_intr,bool is_running) { +/* +* Derive the notification bits from the new state on every update so +* that an in-place update doesn't carry over bits from the old state. +*/ + entry->lo.fields_vapic.ga_log_intr = false; + entry->lo.fields_vapic.gappi_dis = false; +if (is_running) { entry->lo.fields_vapic.destination =APICID_TO_IRTE_DEST_LO (apicid); entry->hi.fields.destination =APICID_TO_IRTE_DEST_HI (apicid); entry->lo.fields_vapic.is_run = true; - entry->lo.fields_vapic.ga_log_intr = false; }else {if (amd_iommu_gappi) { entry->lo.fields_vapic.gappi_dis = !wakeup_intr &&
@@ -3990,13 +3996,13 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,* a vCPU, without issuing an IOMMU invalidation for the IRTE. * * If the vCPU is scheduled to run on pCPU (@is_running = 1), configure the -* Destination with the pCPU's APIC ID, set IsRun, and clear GALogIntr. If the -* vCPU is scheduled out (@is_running = 0), clear IsRun and set/clear GALogIntr -* and GAPPIDis based on input from the caller (e.g. KVM only requests -* wakeup_intr when the vCPU is blocking and requires a notification wake -* event). This API is intended to be used when a vCPU is scheduled in/out (or -* stops running for any reason), to do a fast update of IsRun, GALogIntr, -* GAPPIDis and (conditionally) Destination. +* Destination with the pCPU's APIC ID, set IsRun, and clear GALogIntr and +* GAPPIDis. If the vCPU is scheduled out (@is_running = 0), clear IsRun and +* set/clear GALogIntr and GAPPIDis based on input from the caller (e.g. KVM +* only requests wakeup_intr when the vCPU is blocking and requires a +* notification wake event). This API is intended to be used when a vCPU is +* scheduled in/out (or stops running for any reason), to do a fast update of +* IsRun, GALogIntr, GAPPIDis and (conditionally) Destination. * * Per the IOMMU spec, the Destination, IsRun, GATag and GAPPIDis fields are * not cached and thus don't require an invalidation to ensure the IOMMU
Signature · 1 line
2.39.5
PATCH v4 5/7 KVM: VMX: Factor out wakeup list handling code to KVM
In reply to Sairaj KodilkarVMX maintains a per-pCPU list of blocked vCPUs so that they can be woken when a device interrupt arrives while the vCPU is not running. The vCPU is queued on the list of the pCPU it last ran on and its notification vector is switched to POSTED_INTR_WAKEUP_VECTOR, so the handler for that vector walks the local list and wakes the vCPUs that have a notification pending.
Guest APIC Physical Processor Interrupt (GAPPI) needs the same infrastructure on AMD. When a device interrupt targets a non-running vCPU (IRTE[IsRun] = 0), the IOMMU delivers a physical APIC interrupt to IRTE[Destination] using IRTE[GATag][7:0] as the vector, which KVM programs to POSTED_INTR_WAKEUP_VECTOR. SVM therefore requires the same per-pCPU list of blocked vCPUs that VMX already has. Hence Move the per-pCPU list, its lock, the wakeup handler and the per-CPU initialization from VMX to common KVM x86 code.
Intel and AMD determine whether a queued vCPU has a notification pending in different ways. Hence, introduce a new kvm_x86_ops callback vcpu_irq_pending(), which the wakeup handler uses to decide whether a vCPU needs to be woken.
No functional change is intended.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/vmx/common.h | 3 - arch/x86/kvm/vmx/main.c | 6 ++ arch/x86/kvm/vmx/posted_intr.c | 82 +-------------------- arch/x86/kvm/vmx/tdx.c | 1 - arch/x86/kvm/vmx/vmx.c | 14 ++-- arch/x86/kvm/vmx/x86_ops.h | 1 + arch/x86/kvm/x86.c | 111 +++++++++++++++++++++++++++++ include/linux/kvm_host.h | 6 ++ 10 files changed, 135 insertions(+), 91 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -30,6 +30,7 @@ KVM_X86_OP(vcpu_reset) KVM_X86_OP(prepare_switch_to_guest) KVM_X86_OP(vcpu_load) KVM_X86_OP(vcpu_put) +KVM_X86_OP_OPTIONAL(vcpu_irq_pending) KVM_X86_OP(update_exception_bitmap) KVM_X86_OP(get_msr) KVM_X86_OP(set_msr)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1787,6 +1787,7 @@ struct kvm_x86_ops {void (*prepare_switch_to_guest)(struct kvm_vcpu *vcpu);void (*vcpu_load)(struct kvm_vcpu *vcpu,int cpu);void (*vcpu_put)(struct kvm_vcpu *vcpu); +bool (*vcpu_irq_pending)(struct kvm_vcpu *vcpu);/* * Mask of DEBUGCTL bits that are owned by the host, i.e. that need to
arch/x86/kvm/vmx/common.h +0 -3
@@ -34,9 +34,6 @@ struct vcpu_vt {/* Posted interrupt descriptor */ struct pi_desc pi_desc; -/* Used if this vCPU is waiting for PI notification wakeup. */ -struct list_head pi_wakeup_list; -union vmx_exit_reason exit_reason;unsigned long exit_qualification;
arch/x86/kvm/vmx/main.c +6 -0
@@ -108,6 +108,11 @@ static void vt_vcpu_load(struct kvm_vcpu *vcpu, int cpu)vmx_vcpu_load (vcpu, cpu); } +static bool vt_vcpu_irq_pending (struct kvm_vcpu *vcpu) +{ +return vmx_vcpu_irq_pending (vcpu); +} +static void vt_update_cpu_dirty_logging (struct kvm_vcpu *vcpu) {/*
@@ -896,6 +901,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .prepare_switch_to_guest = vt_op(prepare_switch_to_guest), .vcpu_load = vt_op(vcpu_load), .vcpu_put = vt_op(vcpu_put), + .vcpu_irq_pending = vt_op(vcpu_irq_pending), .HOST_OWNED_DEBUGCTL = VMX_HOST_OWNED_DEBUGCTL_BITS,
arch/x86/kvm/vmx/posted_intr.c +2 -80
114 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.c +0 -1
@@ -687,7 +687,6 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu)fpstate_set_confidential (&vcpu->arch.guest_fpu); vcpu->arch.apic->guest_apic_protected = true; -INIT_LIST_HEAD (&tdx->vt.pi_wakeup_list); vcpu->arch.efer =EFER_SCE |EFER_LME |EFER_LMA |EFER_NX ;
arch/x86/kvm/vmx/vmx.c +7 -7
@@ -1567,6 +1567,13 @@ void vmx_vcpu_put(struct kvm_vcpu *vcpu)vmx_prepare_switch_to_host (to_vmx (vcpu)); } +bool vmx_vcpu_irq_pending (struct kvm_vcpu *vcpu) +{ +struct vcpu_vt *vt =to_vt (vcpu); + +return pi_test_on (&vt->pi_desc); +} +static void vmx_switch_loaded_vmcs (struct kvm_vcpu *vcpu,struct loaded_vmcs *vmcs) {
@@ -7686,8 +7693,6 @@ int vmx_vcpu_create(struct kvm_vcpu *vcpu)BUILD_BUG_ON (offsetof(struct vcpu_vmx , vcpu) !=0 ); vmx =to_vmx (vcpu); -INIT_LIST_HEAD (&vmx->vt.pi_wakeup_list); - err = -ENOMEM ; vmx->vpid =allocate_vpid ();
@@ -8475,8 +8480,6 @@ void vmx_migrate_timers(struct kvm_vcpu *vcpu)void vmx_hardware_unsetup (void ) { -kvm_set_posted_intr_wakeup_handler (NULL ); -if (nested)nested_vmx_hardware_unsetup (); }
@@ -8780,7 +8783,6 @@ __init int vmx_hardware_setup(void)return r; } -kvm_set_posted_intr_wakeup_handler (pi_wakeup_handler);/* * On Intel CPUs that lack self-snoop feature, letting the guest control
@@ -8850,8 +8852,6 @@ int __init vmx_init(void)for_each_possible_cpu (cpu) {INIT_LIST_HEAD (&per_cpu (loaded_vmcss_on_cpu, cpu)); - -pi_init_cpu (cpu); }vmx_check_vmcs12_offsets ();
arch/x86/kvm/vmx/x86_ops.h +1 -0
@@ -26,6 +26,7 @@ void vmx_vcpu_free(struct kvm_vcpu *vcpu);void vmx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event);void vmx_vcpu_load (struct kvm_vcpu *vcpu,int cpu);void vmx_vcpu_put (struct kvm_vcpu *vcpu); +bool vmx_vcpu_irq_pending (struct kvm_vcpu *vcpu);int vmx_handle_exit (struct kvm_vcpu *vcpu,fastpath_t exit_fastpath);void vmx_handle_exit_irqoff (struct kvm_vcpu *vcpu);int vmx_skip_emulated_instruction (struct kvm_vcpu *vcpu);
arch/x86/kvm/x86.c +111 -0
151 lines not on this page · raw message
include/linux/kvm_host.h +6 -0
@@ -398,6 +398,9 @@ struct kvm_vcpu {*/ struct kvm_memory_slot *last_used_slot;u64 last_used_slot_gen; + +/* Used if this vCPU is waiting for PI notification wakeup. */ +struct list_head pi_wakeup_list; };/*
@@ -2609,4 +2612,7 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, struct kvm_pre_fault_memory *range);#endif +void kvm_pi_enable_wakeup_handler (struct kvm_vcpu *vcpu,int cpu); +void kvm_pi_disable_wakeup_handler (struct kvm_vcpu *vcpu,int cpu); +#endif
Signature · 1 line
2.34.1
PATCH v4 6/7 KVM: SVM: Add support for AMD IOMMU Guest APIC Physical Processor Interrupt (GAPPI)
In reply to Sairaj KodilkarWith AVIC guest-mode interrupt remapping, device interrupts are posted into the guest vAPIC backing page by the IOMMU. When the vCPU is not running (IRTE[IsRun] = 0), KVM must still be notified to schedule it. The legacy path uses the GA log.
GAPPI (Guest APIC Physical Processor Interrupt) is an alternative to the GA log mechanism provided by the AMD IOMMU. With GAPPI enabled, the IOMMU still updates the vAPIC backing page IRR, but the host wakeup notification is delivered as a physical APIC interrupt to IRTE[Destination], using IRTE[GATag][7:0] as the vector (POSTED_INTR_WAKEUP_VECTOR).
SVM follows the Intel posted-interrupt wakeup model. Each pCPU maintains a list of blocked vCPUs that may be woken by a GAPPI delivery to that CPU. When a vCPU blocks while waiting for an interrupt, SVM enqueues it on the wakeup list of the pCPU on which it was previously running and passes that same pCPU's physical APIC ID to the IOMMU to program IRTE[Destination]. The rationale is that the vCPU is likely to run again on the same pCPU, which is common when vCPUs are pinned; targeting GAPPI notifications there reduces unnecessary VMEXITs from GAPPI deliveries on other CPUs. SVM registers the GAPPI handler via kvm_set_posted_intr_wakeup_handler(). On delivery, it walks the local vCPU list and wakes vCPUs with a pending IRR.
All GAPPI logic is gated on amd_iommu_gappi. Without it, KVM and the IOMMU falls back to the legacy GA log mechanism for vCPU wakeup.
--- arch/x86/kvm/svm/avic.c | 72 ++++++++++++++++++++++++++++++++++++----- arch/x86/kvm/svm/svm.c | 3 ++ arch/x86/kvm/svm/svm.h | 5 +++ 3 files changed, 72 insertions(+), 8 deletions(-)
arch/x86/kvm/svm/avic.c +64 -8
@@ -874,6 +874,8 @@ int avic_init_vcpu(struct vcpu_svm *svm)INIT_LIST_HEAD (&svm->ir_list);raw_spin_lock_init (&svm->ir_list_lock); + svm->gappi_cpu =-1 ; +if (!enable_apicv || !irqchip_in_kernel (vcpu->kvm))return 0 ;
@@ -886,6 +888,20 @@ int avic_init_vcpu(struct vcpu_svm *svm)return ret; } +void avic_destroy_vcpu (struct vcpu_svm *svm) +{ +if (amd_iommu_gappi && svm->gappi_cpu !=-1 ) { +unsigned long flags; + +local_irq_save (flags); + +kvm_pi_disable_wakeup_handler (&svm->vcpu, svm->gappi_cpu); + svm->gappi_cpu =-1 ; + +local_irq_restore (flags); + } +} +void avic_apicv_post_state_restore (struct kvm_vcpu *vcpu) {avic_handle_dfr_update (vcpu);
@@ -923,8 +939,6 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,* if AVIC is enabled/uninhibited in the future. */ struct amd_iommu_pi_data pi_data = { - .ga_tag =AVIC_GATAG (to_kvm_svm (kvm)->avic_vm_id, - vcpu->vcpu_idx), .is_guest_mode =kvm_vcpu_apicv_active (vcpu), .vapic_addr =avic_get_backing_page_address (to_svm (vcpu)), .vector = vector,
@@ -947,6 +961,12 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,* scheduled out, KVM will update the pCPU info when the vCPU * is awakened and/or scheduled in. See also avic_vcpu_load(). */ +if (amd_iommu_gappi) + pi_data.ga_tag =POSTED_INTR_WAKEUP_VECTOR ; +else + pi_data.ga_tag =AVIC_GATAG (to_kvm_svm (kvm)->avic_vm_id, + vcpu->vcpu_idx); + entry = svm->avic_physical_id_entry;if (entry &AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK ) { pi_data.apicid = entry &AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK ;
@@ -955,6 +975,19 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm, pi_data.apicid =-1 ; pi_data.wakeup_intr = entry &AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR ; pi_data.is_running = false; + +if (amd_iommu_gappi) { +if (svm->gappi_cpu !=-1 ) + pi_data.apicid =kvm_cpu_get_apicid (svm->gappi_cpu); +else +/* +* If user calls KVM_IRQFD before vCPU +* run, use CPU 0. We don't need to add +* vCPU to the wakeup list as it is +* never been loaded. +*/ + pi_data.apicid =kvm_cpu_get_apicid (0 ); + } } ret =irq_set_vcpu_affinity (host_irq, &pi_data);
@@ -1007,7 +1040,7 @@ enum avic_vcpu_action { };static void avic_update_iommu_vcpu_affinity (struct kvm_vcpu *vcpu,int apicid, -enum avic_vcpu_action action) +int cpu,enum avic_vcpu_action action) {bool wakeup_intr = (action &AVIC_START_BLOCKING );bool is_running = apicid >=0 ;
@@ -1016,6 +1049,20 @@ static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid,lockdep_assert_held (&svm->ir_list_lock); +if (amd_iommu_gappi && is_running) { +if (svm->gappi_cpu !=-1 ) +/* +* Handle initial state when vCPU is loaded for the +* first time without any IRQ affinity. +*/ +kvm_pi_disable_wakeup_handler (vcpu, svm->gappi_cpu); + + svm->gappi_cpu = cpu;/* Store cpu number as target for GAPPI */ + }else if (amd_iommu_gappi) { + apicid =kvm_cpu_get_apicid (svm->gappi_cpu); +kvm_pi_enable_wakeup_handler (vcpu, svm->gappi_cpu); + } +/* * Here, we go through the per-vcpu ir_list to update all existing * interrupt remapping table entry targeting this vcpu.
@@ -1083,7 +1130,7 @@ static void __avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu,WRITE_ONCE (kvm_svm->avic_physical_id_table[vcpu->vcpu_id], entry); -avic_update_iommu_vcpu_affinity (vcpu, h_physical_id, action); +avic_update_iommu_vcpu_affinity (vcpu, h_physical_id, cpu, action);raw_spin_unlock_irqrestore (&svm->ir_list_lock, flags); }
@@ -1126,7 +1173,7 @@ static void __avic_vcpu_put(struct kvm_vcpu *vcpu, enum avic_vcpu_action action)*/ raw_spin_lock_irqsave (&svm->ir_list_lock, flags); -avic_update_iommu_vcpu_affinity (vcpu,-1 , action); +avic_update_iommu_vcpu_affinity (vcpu,-1 ,-1 , action);WARN_ON_ONCE (entry &AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR );
@@ -1174,7 +1221,7 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu)/* * The vCPU was preempted while blocking, ensure its IRTEs are -* configured to generate GA Log Interrupts. +* configured to request host wakeup notification. */ if (!(WARN_ON_ONCE (!(entry &AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR ))))return ;
@@ -1299,6 +1346,13 @@ static bool __init avic_want_avic_enabled(void)return true; } +bool avic_vcpu_irq_pending (struct kvm_vcpu *vcpu) +{ +struct vcpu_svm *svm =to_svm (vcpu); + +return kvm_lapic_find_highest_irr (&svm->vcpu) >=0 ; +} +/* * Note: * - The module param avic enable both xAPIC and x2APIC mode.
@@ -1342,6 +1396,8 @@ bool __init avic_hardware_setup(void)void avic_hardware_unsetup (void ) { -if (avic) -amd_iommu_register_ga_log_notifier (NULL ); +if (!avic) +return ; + +amd_iommu_register_ga_log_notifier (NULL ); }
arch/x86/kvm/svm/svm.c +3 -0
@@ -1356,6 +1356,8 @@ static void svm_vcpu_free(struct kvm_vcpu *vcpu)WARN_ON_ONCE (!list_empty (&svm->ir_list)); +avic_destroy_vcpu (svm); +svm_leave_nested (vcpu);svm_free_nested (svm);
@@ -5292,6 +5294,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .vcpu_put = svm_vcpu_put, .vcpu_blocking = avic_vcpu_blocking, .vcpu_unblocking = avic_vcpu_unblocking, + .vcpu_irq_pending = avic_vcpu_irq_pending, .update_exception_bitmap = svm_update_exception_bitmap, .get_feature_msr = svm_get_feature_msr,
arch/x86/kvm/svm/svm.h +5 -0
@@ -362,6 +362,9 @@ struct vcpu_svm {/* Guest GIF value, used when vGIF is not enabled */ bool guest_gif; + +/* GAPPI related fields */ +int gappi_cpu; };struct svm_cpu_data {
@@ -909,8 +912,10 @@ void avic_init_vmcb(struct vcpu_svm *svm, struct vmcb *vmcb);int avic_incomplete_ipi_interception (struct kvm_vcpu *vcpu);int avic_unaccelerated_access_interception (struct kvm_vcpu *vcpu);int avic_init_vcpu (struct vcpu_svm *svm); +void avic_destroy_vcpu (struct vcpu_svm *svm);void avic_vcpu_load (struct kvm_vcpu *vcpu,int cpu);void avic_vcpu_put (struct kvm_vcpu *vcpu); +bool avic_vcpu_irq_pending (struct kvm_vcpu *vcpu);void avic_apicv_post_state_restore (struct kvm_vcpu *vcpu);void avic_refresh_apicv_exec_ctrl (struct kvm_vcpu *vcpu); int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,
Signature · 1 line
2.34.1
Vasant Hegde
maintainerRe: [PATCH v4 6/7] KVM: SVM: Add support for AMD IOMMU Guest APIC Physical Processor Interrupt (GAPPI)
Quoted 37 lines · Sairaj Kodilkar
With AVIC guest-mode interrupt remapping, device interrupts are posted into the guest vAPIC backing page by the IOMMU. When the vCPU is not running (IRTE[IsRun] = 0), KVM must still be notified to schedule it. The legacy path uses the GA log.
GAPPI (Guest APIC Physical Processor Interrupt) is an alternative to the GA log mechanism provided by the AMD IOMMU. With GAPPI enabled, the IOMMU still updates the vAPIC backing page IRR, but the host wakeup notification is delivered as a physical APIC interrupt to IRTE[Destination], using IRTE[GATag][7:0] as the vector (
POSTED_INTR_WAKEUP_VECTOR).SVM follows the Intel posted-interrupt wakeup model. Each pCPU maintains a list of blocked vCPUs that may be woken by a GAPPI delivery to that CPU. When a vCPU blocks while waiting for an interrupt, SVM enqueues it on the wakeup list of the pCPU on which it was previously running and passes that same pCPU's physical APIC ID to the IOMMU to program IRTE[Destination]. The rationale is that the vCPU is likely to run again on the same pCPU, which is common when vCPUs are pinned; targeting GAPPI notifications there reduces unnecessary VMEXITs from GAPPI deliveries on other CPUs. SVM registers the GAPPI handler via
kvm_set_posted_intr_wakeup_handler(). On delivery, it walks the local vCPU list and wakes vCPUs with a pending IRR.All GAPPI logic is gated on
amd_iommu_gappi. Without it, KVM and the IOMMU falls back to the legacy GA log mechanism for vCPU wakeup.Signed-off-by Sairaj Kodilkar <[email protected]>--- arch/x86/kvm/svm/avic.c | 72 ++++++++++++++++++++++++++++++++++++----- arch/x86/kvm/svm/svm.c | 3 ++ arch/x86/kvm/svm/svm.h | 5 +++ 3 files changed, 72 insertions(+), 8 deletions(-)arch/x86/kvm/svm/avic.c +0
-0
arch/x86/kvm/svm/avic.c int avic_init_vcpu(struct vcpu_svm *svm) · line 877
Quoted 4 lines
arch/x86/kvm/svm/avic.c +1
-0@@ -874,6 +874,8 @@ int avic_init_vcpu(struct vcpu_svm *svm)INIT_LIST_HEAD (&svm->ir_list);raw_spin_lock_init (&svm->ir_list_lock); + svm->gappi_cpu =-1 ;
Can we initialize it to CPU0?
arch/x86/kvm/svm/avic.c int avic_init_vcpu(struct vcpu_svm *svm) · line 893
Quoted 11 lines
arch/x86/kvm/svm/avic.c +4
-0+if (!enable_apicv || !irqchip_in_kernel (vcpu->kvm))return 0 ;@@ -886,6 +888,20 @@ int avic_init_vcpu(struct vcpu_svm *svm)return ret; } +void avic_destroy_vcpu (struct vcpu_svm *svm) +{ +if (amd_iommu_gappi && svm->gappi_cpu !=-1 ) {
Redundant amd_iommu_gappi check?
arch/x86/kvm/svm/avic.c int avic_init_vcpu(struct vcpu_svm *svm) · line 899
Quoted 6 lines
arch/x86/kvm/svm/avic.c +6
-0+unsigned long flags; + +local_irq_save (flags); + +kvm_pi_disable_wakeup_handler (&svm->vcpu, svm->gappi_cpu); + svm->gappi_cpu =-1 ;
Redundant assignement?
-Vasant
Re: [PATCH v4 6/7] KVM: SVM: Add support for AMD IOMMU Guest APIC Physical Processor Interrupt (GAPPI)
Quoted 43 lines · Vasant Hegde
Quoted 40 lines · Sairaj Kodilkar
With AVIC guest-mode interrupt remapping, device interrupts are posted into the guest vAPIC backing page by the IOMMU. When the vCPU is not running (IRTE[IsRun] = 0), KVM must still be notified to schedule it. The legacy path uses the GA log.
GAPPI (Guest APIC Physical Processor Interrupt) is an alternative to the GA log mechanism provided by the AMD IOMMU. With GAPPI enabled, the IOMMU still updates the vAPIC backing page IRR, but the host wakeup notification is delivered as a physical APIC interrupt to IRTE[Destination], using IRTE[GATag][7:0] as the vector (
POSTED_INTR_WAKEUP_VECTOR).SVM follows the Intel posted-interrupt wakeup model. Each pCPU maintains a list of blocked vCPUs that may be woken by a GAPPI delivery to that CPU. When a vCPU blocks while waiting for an interrupt, SVM enqueues it on the wakeup list of the pCPU on which it was previously running and passes that same pCPU's physical APIC ID to the IOMMU to program IRTE[Destination]. The rationale is that the vCPU is likely to run again on the same pCPU, which is common when vCPUs are pinned; targeting GAPPI notifications there reduces unnecessary VMEXITs from GAPPI deliveries on other CPUs. SVM registers the GAPPI handler via
kvm_set_posted_intr_wakeup_handler(). On delivery, it walks the local vCPU list and wakes vCPUs with a pending IRR.All GAPPI logic is gated on
amd_iommu_gappi. Without it, KVM and the IOMMU falls back to the legacy GA log mechanism for vCPU wakeup.Signed-off-by Sairaj Kodilkar <[email protected]>--- arch/x86/kvm/svm/avic.c | 72 ++++++++++++++++++++++++++++++++++++----- arch/x86/kvm/svm/svm.c | 3 ++ arch/x86/kvm/svm/svm.h | 5 +++ 3 files changed, 72 insertions(+), 8 deletions(-)arch/x86/kvm/svm/avic.c +0
-0@@ -874,6 +874,8 @@ int avic_init_vcpu(struct vcpu_svm *svm)INIT_LIST_HEAD (&svm->ir_list);raw_spin_lock_init (&svm->ir_list_lock);
Quoted 3 lines
arch/x86/kvm/svm/avic.c int avic_init_vcpu(struct vcpu_svm *svm) · line 877
Quoted 1 line
arch/x86/kvm/svm/avic.c +1
-0+ svm->gappi_cpu =-1 ;Can we initialize it to CPU0?
Yeah, I think we can do this, I was worried about passing gappi_cpu=0 to kvm_pi_disable_wakeup_handler() when vCPU is not in list but I think it should be safe because of list_empty() check.
Quoted 8 lines
Quoted 8 lines
arch/x86/kvm/svm/avic.c +1
-0+if (!enable_apicv || !irqchip_in_kernel (vcpu->kvm))return 0 ;@@ -886,6 +888,20 @@ int avic_init_vcpu(struct vcpu_svm *svm)return ret; }
Quoted 5 lines
arch/x86/kvm/svm/avic.c int avic_init_vcpu(struct vcpu_svm *svm) · line 893
Quoted 3 lines
arch/x86/kvm/svm/avic.c +3
-0+void avic_destroy_vcpu (struct vcpu_svm *svm) +{ +if (amd_iommu_gappi && svm->gappi_cpu !=-1 ) {Redundant
amd_iommu_gappicheck?
I am not sure if we should call kvm_pi_disable_wakeup_handler() outside amd_iommu_gappi check. Although it is safe to do so, I want to have this call inside amd_iommu_gappi check.
Thanks
Sairaj.
Quoted 12 lines
arch/x86/kvm/svm/avic.c int avic_init_vcpu(struct vcpu_svm *svm) · line 899
Quoted 7 lines
arch/x86/kvm/svm/avic.c +6
-0+unsigned long flags; + +local_irq_save (flags); + +kvm_pi_disable_wakeup_handler (&svm->vcpu, svm->gappi_cpu); + svm->gappi_cpu =-1 ;Redundant assignement?
-Vasant
PATCH v4 7/7 iommu/amd: Provide kernel command line option to enable GAPPI
In reply to Sairaj KodilkarEnable GAPPI when the IOMMU GAPPISup extended feature bit is set and the kernel is booted with the amd_iommu=gappi kernel parameter.
Document the option in Documentation/admin-guide/kernel-parameters.txt.
--- .../admin-guide/kernel-parameters.txt | 3 +- drivers/iommu/amd/amd_iommu_types.h | 2 ++ drivers/iommu/amd/init.c | 28 +++++++++++++++++-- 3 files changed, 29 insertions(+), 4 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +2 -1
@@ -455,7 +455,8 @@ Kernel parameters to 4 KiB. v2_pgsizes_only - Limit page-sizes used for v1 page-tables to 4KiB/2Mib/1GiB. - + gappi - Use GAPPI instead of GALOG for vCPU + wakeup notification. amd_iommu_dump= [HW,X86-64] Enable AMD IOMMU driver option to dump the ACPI table
drivers/iommu/amd/amd_iommu_types.h +2 -0
@@ -94,6 +94,7 @@#define FEATURE_X2APIC BIT_ULL(2)#define FEATURE_NX BIT_ULL(3)#define FEATURE_GT BIT_ULL(4) +#define FEATURE_GAPPI BIT_ULL(5)#define FEATURE_IA BIT_ULL(6)#define FEATURE_GA BIT_ULL(7)#define FEATURE_HE BIT_ULL(8)
@@ -191,6 +192,7 @@#define CONTROL_EPH_EN 45#define CONTROL_XT_EN 50#define CONTROL_INTCAPXT_EN 51 +#define CONTROL_GAPPI_EN 55#define CONTROL_GCR3TRPMODE 58#define CONTROL_IRTCACHEDIS 59#define CONTROL_SNPAVIC_EN 61
drivers/iommu/amd/init.c +25 -3
@@ -465,6 +465,9 @@ static void iommu_disable(struct amd_iommu *iommu)iommu_feature_disable (iommu,CONTROL_GALOG_EN );iommu_feature_disable (iommu,CONTROL_GAINT_EN ); +/* Disable IOMMU GAPPI */ +iommu_feature_disable (iommu,CONTROL_GAPPI_EN ); +/* Disable IOMMU PPR logging */ iommu_feature_disable (iommu,CONTROL_PPRLOG_EN );iommu_feature_disable (iommu,CONTROL_PPRINT_EN );
@@ -2999,6 +3002,13 @@ static void enable_iommus_vapic(void)struct amd_iommu *iommu;for_each_iommu (iommu) { +/* +* Disable GAPPI, do not check amd_iommu_gappi as it may be +* false in new kexec kernel even though previous kernel has +* enabled it. +*/ +iommu_feature_disable (iommu,CONTROL_GAPPI_EN ); +/* * Disable GALog if already running. It could have been enabled * in the previous boot before kdump.
@@ -3038,10 +3048,19 @@ static void enable_iommus_vapic(void)return ; } +if (amd_iommu_gappi && + !(check_feature (FEATURE_GAPPI ) && +AMD_IOMMU_GUEST_IR_VAPIC (amd_iommu_guest_ir))) { +pr_warn ("GAPPI is not supported.\n" ); + amd_iommu_gappi = false; + } +/* Enabling GAM and SNPAVIC support */ for_each_iommu (iommu) { -if (iommu_init_ga_log (iommu) || -iommu_ga_log_enable (iommu)) +if (amd_iommu_gappi) +iommu_feature_enable (iommu,CONTROL_GAPPI_EN ); +else if (iommu_init_ga_log (iommu) || +iommu_ga_log_enable (iommu))return ;iommu_feature_enable (iommu,CONTROL_GAM_EN );
@@ -3050,7 +3069,8 @@ static void enable_iommus_vapic(void) } amd_iommu_irq_ops.capability |= (1 <<IRQ_POSTING_CAP ); -pr_info ("Virtual APIC enabled\n" ); +pr_info ("Virtual APIC enabled with %s\n" , + amd_iommu_gappi ?"GAPPI" :"GALOG" );#endif }
@@ -3741,6 +3761,8 @@ static int __init parse_amd_iommu_options(char *str) }else if (strncmp (str,"v2_pgsizes_only" ,15 ) ==0 ) {pr_info ("Restricting V1 page-sizes to 4KiB/2MiB/1GiB" ); amd_iommu_pgsize_bitmap =AMD_IOMMU_PGSIZES_V2 ; + }else if (strncmp (str,"gappi" ,5 ) ==0 ) { + amd_iommu_gappi = true; }else {pr_notice ("Unknown option - '%s'\n" , str); }
Signature · 1 line
2.34.1
Randy Dunlap
reviewerRe: [PATCH v4 7/7] iommu/amd: Provide kernel command line option to enable GAPPI
Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 459
Quoted 11 lines · Sairaj Kodilkar
Documentation/admin-guide/kernel-parameters.txt +2
-1@@ -455,7 +455,8 @@ Kernel parameters to 4 KiB. v2_pgsizes_only - Limit page-sizes used for v1 page-tables to 4KiB/2Mib/1GiB. - + gappi - Use GAPPI instead of GALOG for vCPU + wakeup notification.
Is GALOG documented somewhere? Should it be?
I can't find it...
thanks.
Signature · 1 line
~Randy
Re: [PATCH v4 7/7] iommu/amd: Provide kernel command line option to enable GAPPI
Quoted 17 lines · Randy Dunlap
Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 459
Quoted 11 lines · Sairaj Kodilkar
Documentation/admin-guide/kernel-parameters.txt +2
-1@@ -455,7 +455,8 @@ Kernel parameters to 4 KiB. v2_pgsizes_only - Limit page-sizes used for v1 page-tables to 4KiB/2Mib/1GiB. - + gappi - Use GAPPI instead of GALOG for vCPU + wakeup notification.Is GALOG documented somewhere? Should it be?
I can't find it...
Its not documented. Maybe we can add small documentation in arch/x86/iommu.rst
Thanks
Sairaj
Quoted 2 lines
thanks.
Quoted 132 lines · Sairaj Kodilkar
Introduction
============
On newer generations of AMD processors, IOMMU AVIC/x2AVIC guest-mode interrupt remapping can use Guest APIC Physical Processor Interrupt (GAPPI) as an alternative host-notification path when device interrupts target a vCPU that is not running (IRTE[IsRun] = 0).With AVIC enabled, the IOMMU posts device interrupts into the guest virtual APIC backing page. When the vCPU is not running, KVM must additionally be notified so it can schedule the vCPU.
The legacy notification path is the GA log (GALOG): the IOMMU appends vCPU tags to a shared GA log buffer and raises a single GA log interrupt. KVM registers a notifier and scans the buffer to decide which vCPUs to wake. Under heavy interrupt load this adds latency and can overflow the buffer because all wakeups funnel through one interrupt and one shared log.
Guest APIC Physical Processor Interrupt (GAPPI), defined in section 2.2.5.4 of the AMD I/O Virtualization Technology (IOMMU) Specification [1], is an alternative. With GAPPI enabled, the IOMMU still updates the guest vAPIC backing page IRR, but may deliver a physical APIC interrupt directly to IRTE[Destination], using IRTE[GATag][7:0] as the vector. This distributes host wakeup notifications across CPUs instead of centralizing them in a log buffer.
This series programs guest-mode IRTEs accordingly: IRTE[Destination] carries the target host physical APIC ID, IRTE[GATag] is set to
POSTED_INTR_WAKEUP_VECTOR, and IRTE[GAPPIDis] / IRTE[GALogIntr] are set based on whether KVM requests host wakeup. GAPPI is selected at boot via the amd_iommu=gappi kernel parameter on capable hardware, otherwise the existing GA log path is unchanged.SVM/AMD IOMMU interface changes
===============================
The first four patches refactor the SVM/AMD IOMMU interface ahead of GAPPI.The cpu field is renamed to apicid because it carries the host physical APIC ID for IRTE[Destination], not a Linux CPU number.
The
ga_log_intrboolean is renamed towakeup_intr(and the syntheticAVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTRshadow bit toAVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR).wakeup_intrdescribes KVM's intent (request host wakeup while the vCPU is not running), not a specific hardware mechanism.A separate
is_runningboolean is added to IOMMU interface because GAPPI requires a valid apicid in IRTE[Destination] even when the vCPU is not running. The prior encoding (apicid >= 0 means running, apicid == -1 means not running) no longer works once apicid carries the GAPPI destination while IRTE[IsRun] is clear. The IOMMU driver keys IRTE[IsRun] and destination programming off this explicit boolean instead of inferring running state from apicid.KVM GAPPI wakeup scheme
=======================
SVM follows the Intel posted-interrupt wakeup model already used by VMX. Each pCPU maintains a list of blocked vCPUs that may be woken by a GAPPI delivery to that CPU. When a vCPU blocks while waiting for a device interrupt, SVM enqueues it on the wakeup list of the pCPU on which it was previously running (gappi_cpu) and passes that pCPU's physical APIC ID to the IOMMU to program IRTE[Destination]. The rationale is that the vCPU is likely to run again on the same pCPU, which is common when vCPUs are pinned; targeting GAPPI notifications there reduces unnecessary VMEXITs from GAPPI deliveries on other CPUs. When the vCPU is scheduled in again, it is removed from the list and IRTE[Destination] is updated to the current pCPU.List maintenance is moved from VMX layer to common KVM layer so that both VMX and SVM layer can share the same list handling code.
Note
====kvm_pi_enable_wakeup_handler()andkvm_pi_disable_wakeup_handler()take the target pCPU explicitly instead of usingvcpu->cpu, because on SVM the two can differ.avic_vcpu_load()returns early while the vCPU is blocking and leaves the target wakeup CPU untouched, but the generic layer still refreshesvcpu->cpuon every load. Usingvcpu->cpuwould then unlink the vCPU under the wrong per-pCPU lock, so SVM tracks the pCPU it last programmed insvm->gappi_cpuand passes that instead.Changes since v3
================
https://lore.kernel.org/all/[email protected]/Patch 3:
- Makeamd_iommu_activate_guest_mode()as wrapper around__amd_iommu_activate_guest_mode()to in order to hardcodeis_runningflag to "true" for external callsPatch 5:
- Newly introduced, moves wakeup list handling from VMX to the KVM layerPatch 6:
- Use common list handling functions introduced in patch 5.
- Remove list addition duringpi_update_irteby moving list handlingfunctions before list_empty(&svm->ir_list) check.Changes since v2
================
https://lore.kernel.org/linux-iommu/[email protected]/Patch[1-6]
- Expand commit messages to explain GAPPI, the interface changes, and the per-CPU wakeup list scheme [Sean].Patch[1-3]
- Split the monolithic SVM/IOMMU API refactor into four preparatory patches [Sean]
- Renameposted_intrtowakeup_intrto reflect host wakeup intent, not guest interrupt posting [Sean]
- Pass vCPU running status with a extra parameter (is_running) instead of flags.Patch[4,5]
- Move ga_tag=POSTED_INTR_WAKEUP_VECTOR setting from IOMMU to SVM layer.Changes since V1:
================
https://lore.kernel.org/all/[email protected]/Patch4
- Disable interrupts while holding wakeup list lock inside [sashiko]avic_add_vcpu_to_gappi_wakeup_listandavic_remove_vcpu_from_gappi_wakeup_list
- Unregisterposted_intr_wakeup_handlerduring module unload [sashiko]Patch5
- Disable GAPPI feature during kexec and suspend path [sashiko]
Quoted 5 lines
------
[1] https://docs.amd.com/v/u/en-US/48882_3.11_IOMMU_PUB------
Base:89516a1c6837156f43525ef49fdbd781145942f5
Apologies for wrong base commit, Please use following base commit
Base: 8cd9520d35a6c38db6567e97dd93b1f11f185dc6
Quoted 36 lines
Sairaj Kodilkar (7):
iommu/amd: KVM: SVM: Rename cpu to apicid in IOMMU interface iommu/amd: KVM: SVM: Renamega_log_intrtowakeup_intrin IOMMU
interface
iommu/amd: KVM: SVM: Add explicit vCPU running state to IOMMU
interface
iommu/amd: Program guest-mode IRTEs for GAPPI wakeup when IRTE[IsRun]= 0KVM: VMX: Factor out wakeup list handling code to KVM
KVM: SVM: Add support for AMD IOMMU Guest APIC Physical Processor
Interrupt (GAPPI)
iommu/amd: Provide kernel command line option to enable GAPPI.../admin-guide/kernel-parameters.txt | 3 +- arch/x86/include/asm/irq_remapping.h | 5 +- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/include/asm/svm.h | 9 +- arch/x86/kvm/svm/avic.c | 109 +++++++++++++---- arch/x86/kvm/svm/svm.c | 3 + arch/x86/kvm/svm/svm.h | 5 + arch/x86/kvm/vmx/common.h | 3 - arch/x86/kvm/vmx/main.c | 6 + arch/x86/kvm/vmx/posted_intr.c | 82 +------------ arch/x86/kvm/vmx/tdx.c | 1 - arch/x86/kvm/vmx/vmx.c | 14 +-- arch/x86/kvm/vmx/x86_ops.h | 1 + arch/x86/kvm/x86.c | 111 ++++++++++++++++++ drivers/iommu/amd/amd_iommu_types.h | 6 +- drivers/iommu/amd/init.c | 31 ++++- drivers/iommu/amd/iommu.c | 65 ++++++---- include/linux/amd-iommu.h | 16 ++- include/linux/kvm_host.h | 6 + 20 files changed, 323 insertions(+), 155 deletions(-)