Skip to contentSkip to conversation

Add support for AMD IOMMU GAPPI

PATCH New Sairaj Kodilkar 2 participants 10 messages Nov 11

Subsystems (7)
Documentation x86 KVM KVM x86 +3 more subsystems
lore
Share & tools
lore
#0 lore raw

v7 cover Add support for AMD IOMMU GAPPI

Introduction
============
On newer generations of AMD processors, IOMMU AVIC/x2AVIC guest-mode interrupt remapping can use Guest APIC Physical Processor Interrupt (GAPPI) as an alternative host-notification path when device interrupts target a vCPU that is not running (IRTE[IsRun] = 0).

With AVIC enabled, the IOMMU posts device interrupts into the guest virtual APIC backing page. When the vCPU is not running, KVM must additionally be notified so it can schedule the vCPU.

The legacy notification path is the GA log (GALOG): the IOMMU appends vCPU tags to a shared GA log buffer and raises a single GA log interrupt. KVM registers a notifier and scans the buffer to decide which vCPUs to wake. Under heavy interrupt load this adds latency and can overflow the buffer because all wakeups funnel through one interrupt and one shared log.

Guest APIC Physical Processor Interrupt (GAPPI), defined in section 2.2.5.4 of the AMD I/O Virtualization Technology (IOMMU) Specification [1], is an alternative. With GAPPI enabled, the IOMMU still updates the guest vAPIC backing page IRR, but may deliver a physical APIC interrupt directly to IRTE[Destination], using IRTE[GATag][7:0] as the vector. This distributes host wakeup notifications across CPUs instead of centralizing them in a log buffer.

This series programs guest-mode IRTEs accordingly: IRTE[Destination] carries the target host physical APIC ID, IRTE[GATag] is set to POSTED_INTR_WAKEUP_VECTOR, and IRTE[GAPPIDis] / IRTE[GALogIntr] are set based on whether KVM requests host wakeup. GAPPI is selected at boot via the amd_iommu=gappi kernel parameter on capable hardware, otherwise the existing GA log path is unchanged.

SVM/AMD IOMMU interface changes
===============================
The first four patches refactor the SVM/AMD IOMMU interface ahead of GAPPI.

The cpu field is renamed to apicid because it carries the host physical APIC ID for IRTE[Destination], not a Linux CPU number.

The ga_log_intr boolean is renamed to wakeup_intr (and the synthetic AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR shadow bit to AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR). wakeup_intr describes KVM's intent (request host wakeup while the vCPU is not running), not a specific hardware mechanism.

Remove wakeup_intr flag from amd_iommu_activate_guest_mode() since it is always called when is_running flag is set and IOMMU never uses wakeup_intr flag when is_running is set.

A separate is_running boolean is added to IOMMU interface because GAPPI requires a valid apicid in IRTE[Destination] even when the vCPU is not running. The prior encoding (apicid >= 0 means running, apicid == -1 means not running) no longer works once apicid carries the GAPPI destination while IRTE[IsRun] is clear. The IOMMU driver keys IRTE[IsRun] and destination programming off this explicit boolean instead of inferring running state from apicid.

KVM GAPPI wakeup scheme
=======================
SVM follows the Intel posted-interrupt wakeup model already used by VMX. Each pCPU maintains a list of blocked vCPUs that may be woken by a GAPPI delivery to that CPU. When a vCPU blocks while waiting for a device interrupt, SVM enqueues it on the wakeup list of the pCPU on which it was previously running (gappi_cpu) and passes that pCPU's physical APIC ID to the IOMMU to program IRTE[Destination]. The rationale is that the vCPU is likely to run again on the same pCPU, which is common when vCPUs are pinned; targeting GAPPI notifications there reduces unnecessary VMEXITs from GAPPI deliveries on other CPUs. When the vCPU is scheduled in again, it is removed from the list and IRTE[Destination] is updated to the current pCPU.

List maintenance is moved from VMX layer to common KVM layer so that both VMX and SVM layer can share the same list handling code.

Note
====
kvm_pi_enable_wakeup_handler() and kvm_pi_disable_wakeup_handler() take the target pCPU explicitly instead of using vcpu->cpu, because on SVM the two can differ. avic_vcpu_load() returns early while the vCPU is blocking and leaves the target wakeup CPU untouched, but the generic layer still refreshes vcpu->cpu on every load. Using vcpu->cpu would then unlink the vCPU under the wrong per-pCPU lock, so SVM tracks the pCPU it last programmed in svm->gappi_cpu and passes that instead.

Changes since v6
================
https://lore.kernel.org/all/[email protected]/
Patch 6:
- Update svm->gappi_cpu = cpu before jumping to skip_wakeup_handler when AVIC is getting enabled. [Sashiko]
- Remove check for wakeup_intr while adding vCPU on wakeup list. [Sashiko]

Changes since v5
================
https://lore.kernel.org/all/[email protected]/
Patch 3:
- Remove wakeup_intr from the amd_iommu_activate_guest_mode() interface as it is never used when is_running flag is set. [Sashiko]

Patch 4:
- Set GAPPI dis bit to 0 when vCPU is transitioning to running state. [Sashiko]

Patch 5:
- Use KVM_X86_OP_OPTIONAL_RET0 instead of KVM_X86_OP_OPTIONAL to define vcpu_irq_pending callback. [Sashiko]

Patch 6:
- Do not add vCPU to wakeup list when avic is disabled or wakeup_intr flag is not set (i.e. vCPU is not blocking). [Sashiko]

Patch 7:
- Rename bit 5 in the feature list to "GAPPI". [Sashiko]

Changes since v4
================
https://lore.kernel.org/all/[email protected]/
Patch 6:
- Initialize gappi_cpu to 0 instead of -1, the list_empty() check handles the initial case when vCPU is not on the wakeup list of CPU 0 [Vasant]

Changes since v3
================
https://lore.kernel.org/all/[email protected]/

Patch 3:
- Make amd_iommu_activate_guest_mode() as wrapper around __amd_iommu_activate_guest_mode() to in order to hardcode is_running flag to "true" for external calls [Sean]

Patch 5:
- Newly introduced, moves wakeup list handling from VMX to the KVM layer

    [Sean]

Patch 6:
- Use common list handling functions introduced in patch 5.
- Remove list addition during pi_update_irte by moving list handling

    functions before list_empty(&svm->ir_list) check.

Changes since v2
================
https://lore.kernel.org/linux-iommu/[email protected]/

Patch[1-6]
- Expand commit messages to explain GAPPI, the interface changes, and the per-CPU wakeup list scheme [Sean].

Patch[1-3]
- Split the monolithic SVM/IOMMU API refactor into four preparatory patches [Sean]
- Rename posted_intr to wakeup_intr to reflect host wakeup intent, not guest interrupt posting [Sean]
- Pass vCPU running status with a extra parameter (is_running) instead of flags.

Patch[4,5]
- Move ga_tag=POSTED_INTR_WAKEUP_VECTOR setting from IOMMU to SVM layer.

Changes since V1:
================
https://lore.kernel.org/all/[email protected]/

Patch4
- Disable interrupts while holding wakeup list lock inside [sashiko] avic_add_vcpu_to_gappi_wakeup_list and avic_remove_vcpu_from_gappi_wakeup_list
- Unregister posted_intr_wakeup_handler during module unload [sashiko]

Patch5
- Disable GAPPI feature during kexec and suspend path [sashiko]

------

[1] https://docs.amd.com/v/u/en-US/48882_3.11_IOMMU_PUB

------

Sairaj Kodilkar (7):
iommu/amd: KVM: SVM: Rename cpu to apicid in IOMMU interface iommu/amd: KVM: SVM: Rename ga_log_intr to wakeup_intr in IOMMU
interface
iommu/amd: KVM: SVM: Add explicit vCPU running state to IOMMU
interface
iommu/amd: Program guest-mode IRTEs for GAPPI wakeup when IRTE[IsRun]

    = 0

KVM: VMX: Factor out wakeup list handling code to KVM
KVM: SVM: Add support for AMD IOMMU Guest APIC Physical Processor
Interrupt (GAPPI)
iommu/amd: Provide kernel command line option to enable GAPPI

 .../admin-guide/kernel-parameters.txt         |   3 +-
 Documentation/arch/x86/iommu.rst              |  22 ++++
 arch/x86/include/asm/irq_remapping.h          |   5 +-
 arch/x86/include/asm/kvm-x86-ops.h            |   1 +
 arch/x86/include/asm/kvm_host.h               |   1 +
 arch/x86/include/asm/svm.h                    |   9 +-
 arch/x86/kvm/svm/avic.c                       | 112 ++++++++++++++----
 arch/x86/kvm/svm/svm.c                        |   1 +
 arch/x86/kvm/svm/svm.h                        |   4 +
 arch/x86/kvm/vmx/common.h                     |   3 -
 arch/x86/kvm/vmx/main.c                       |   6 +
 arch/x86/kvm/vmx/posted_intr.c                |  82 +------------
 arch/x86/kvm/vmx/posted_intr.h                |   2 -
 arch/x86/kvm/vmx/tdx.c                        |   1 -
 arch/x86/kvm/vmx/vmx.c                        |  14 +--
 arch/x86/kvm/vmx/x86_ops.h                    |   1 +
 arch/x86/kvm/x86.c                            | 111 +++++++++++++++++
 drivers/iommu/amd/amd_iommu_types.h           |   6 +-
 drivers/iommu/amd/init.c                      |  38 +++++-
 drivers/iommu/amd/iommu.c                     |  66 +++++++----
 include/linux/amd-iommu.h                     |  16 ++-
 include/linux/kvm_host.h                      |  10 ++
 22 files changed, 356 insertions(+), 158 deletions(-)

base-commit: 067c5c24cef23d913b67eaaf71ae67e7169a6620

Signature · 1 line
2.34.1
#1 lore raw

PATCH v7 1/7 iommu/amd: KVM: SVM: Rename cpu to apicid in IOMMU interface

The cpu field passed from SVM to the AMD IOMMU driver does not hold a linux CPU number. It carries the host physical APIC ID used to program the IRTE Destination field when the target vCPU is running (IRTE[IsRun] = 1).

Rename the field to apicid everywhere in the IOMMU interface so the name matches the hardware semantics.

No functional change is intended.

---
 arch/x86/include/asm/irq_remapping.h |  2 +-
 arch/x86/kvm/svm/avic.c              | 12 +++++-----
 drivers/iommu/amd/iommu.c            | 35 ++++++++++++++--------------
 include/linux/amd-iommu.h            |  8 +++----
 4 files changed, 29 insertions(+), 28 deletions(-)
arch/x86/include/asm/irq_remapping.h +1 -1
index 37b94f484ef3..263f4df429d8 100644
@@ -35,7 +35,7 @@ struct amd_iommu_pi_data {
 	u64 vapic_addr;		/* Physical address of the vCPU's vAPIC. */
 	u32 ga_tag;
 	u32 vector;		/* Guest vector of the interrupt */
-	int cpu;
+	int apicid;
 	bool ga_log_intr;
 	bool is_guest_mode;
 	void *ir_data;
arch/x86/kvm/svm/avic.c +6 -6
index 96ca39c0045f..5ac5ce4039a6 100644
@@ -960,9 +960,9 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,
 		 */
 		entry = svm->avic_physical_id_entry;
 		if (entry & AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK) {
-			pi_data.cpu = entry & AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK;
+			pi_data.apicid = entry & AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK;
 		} else {
-			pi_data.cpu = -1;
+			pi_data.apicid = -1;
 			pi_data.ga_log_intr = entry & AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR;
 		}
 
@@ -1015,7 +1015,7 @@ enum avic_vcpu_action {
 	AVIC_START_BLOCKING	= BIT(1),
 };
 
-static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu,
+static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid,
 					    enum avic_vcpu_action action)
 {
 	bool ga_log_intr = (action & AVIC_START_BLOCKING);
@@ -1035,9 +1035,9 @@ static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu,
 		void *data = irqfd->irq_bypass_data;
 
 		if (!(action & AVIC_TOGGLE_ON_OFF))
-			WARN_ON_ONCE(amd_iommu_update_ga(data, cpu, ga_log_intr));
-		else if (cpu >= 0)
-			WARN_ON_ONCE(amd_iommu_activate_guest_mode(data, cpu, ga_log_intr));
+			WARN_ON_ONCE(amd_iommu_update_ga(data, apicid, ga_log_intr));
+		else if (apicid >= 0)
+			WARN_ON_ONCE(amd_iommu_activate_guest_mode(data, apicid, ga_log_intr));
 		else
 			WARN_ON_ONCE(amd_iommu_deactivate_guest_mode(data));
 	}
drivers/iommu/amd/iommu.c +18 -17
index 41ee4ad1dcf8..26afedfbf0bc 100644
@@ -4053,14 +4053,14 @@ static const struct irq_domain_ops amd_ir_domain_ops = {
 	.deactivate = irq_remapping_deactivate,
 };
 
-static void __amd_iommu_update_ga(struct irte_ga *entry, int cpu,
+static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,
 				  bool ga_log_intr)
 {
-	if (cpu >= 0) {
+	if (apicid >= 0) {
 		entry->lo.fields_vapic.destination =
-					APICID_TO_IRTE_DEST_LO(cpu);
+					APICID_TO_IRTE_DEST_LO(apicid);
 		entry->hi.fields.destination =
-					APICID_TO_IRTE_DEST_HI(cpu);
+					APICID_TO_IRTE_DEST_HI(apicid);
 		entry->lo.fields_vapic.is_run = true;
 		entry->lo.fields_vapic.ga_log_intr = false;
 	} else {
@@ -4073,20 +4073,21 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int cpu,
  * Update the pCPU information for an IRTE that is configured to post IRQs to
  * a vCPU, without issuing an IOMMU invalidation for the IRTE.
  *
- * If the vCPU is associated with a pCPU (@cpu >= 0), configure the Destination
- * with the pCPU's APIC ID, set IsRun, and clear GALogIntr.  If the vCPU isn't
- * associated with a pCPU (@cpu < 0), clear IsRun and set/clear GALogIntr based
- * on input from the caller (e.g. KVM only requests GALogIntr when the vCPU is
- * blocking and requires a notification wake event).  I.e. treat vCPUs that are
- * associated with a pCPU as running.  This API is intended to be used when a
- * vCPU is scheduled in/out (or stops running for any reason), to do a fast
- * update of IsRun, GALogIntr, and (conditionally) Destination.
+ * If the vCPU is associated with a pCPU (@apicid >= 0), configure the
+ * Destination with the pCPU's APIC ID, set IsRun, and clear GALogIntr.  If the
+ * vCPU isn't associated with a pCPU (@apicid < 0), clear IsRun and set/clear
+ * GALogIntr based on input from the caller (e.g. KVM only requests GALogIntr
+ * when the vCPU is blocking and requires a notification wake event).  I.e.
+ * treat vCPUs that are associated with a pCPU as running.  This API is
+ * intended to be used when a vCPU is scheduled in/out (or stops running for
+ * any reason), to do a fast update of IsRun, GALogIntr, and (conditionally)
+ * Destination.
  *
  * Per the IOMMU spec, the Destination, IsRun, and GATag fields are not cached
  * and thus don't require an invalidation to ensure the IOMMU consumes fresh
  * information.
  */
-int amd_iommu_update_ga(void *data, int cpu, bool ga_log_intr)
+int amd_iommu_update_ga(void *data, int apicid, bool ga_log_intr)
 {
 	struct amd_ir_data *ir_data = (struct amd_ir_data *)data;
 	struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -4100,14 +4101,14 @@ int amd_iommu_update_ga(void *data, int cpu, bool ga_log_intr)
 	if (!ir_data->iommu)
 		return -ENODEV;
 
-	__amd_iommu_update_ga(entry, cpu, ga_log_intr);
+	__amd_iommu_update_ga(entry, apicid, ga_log_intr);
 
 	return __modify_irte_ga(ir_data->iommu, ir_data->irq_2_irte.devid,
 				ir_data->irq_2_irte.index, entry);
 }
 EXPORT_SYMBOL(amd_iommu_update_ga);
 
-int amd_iommu_activate_guest_mode(void *data, int cpu, bool ga_log_intr)
+int amd_iommu_activate_guest_mode(void *data, int apicid, bool ga_log_intr)
 {
 	struct amd_ir_data *ir_data = (struct amd_ir_data *)data;
 	struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -4130,7 +4131,7 @@ int amd_iommu_activate_guest_mode(void *data, int cpu, bool ga_log_intr)
 	entry->hi.fields.vector            = ir_data->ga_vector;
 	entry->lo.fields_vapic.ga_tag      = ir_data->ga_tag;
 
-	__amd_iommu_update_ga(entry, cpu, ga_log_intr);
+	__amd_iommu_update_ga(entry, apicid, ga_log_intr);
 
 	return modify_irte_ga(ir_data->iommu, ir_data->irq_2_irte.devid,
 			      ir_data->irq_2_irte.index, entry);
@@ -4201,7 +4202,7 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *info)
 		ir_data->ga_vector = pi_data->vector;
 		ir_data->ga_tag = pi_data->ga_tag;
 		if (pi_data->is_guest_mode)
-			ret = amd_iommu_activate_guest_mode(ir_data, pi_data->cpu,
+			ret = amd_iommu_activate_guest_mode(ir_data, pi_data->apicid,
 							    pi_data->ga_log_intr);
 		else
 			ret = amd_iommu_deactivate_guest_mode(ir_data);
include/linux/amd-iommu.h +4 -4
index 5c9dcf720f84..a467e3e90acc 100644
@@ -30,8 +30,8 @@ static inline void amd_iommu_detect(void) { }
 /* IOMMU AVIC Function */
 extern int amd_iommu_register_ga_log_notifier(int (*notifier)(u32));
 
-extern int amd_iommu_update_ga(void *data, int cpu, bool ga_log_intr);
-extern int amd_iommu_activate_guest_mode(void *data, int cpu, bool ga_log_intr);
+extern int amd_iommu_update_ga(void *data, int apicid, bool ga_log_intr);
+extern int amd_iommu_activate_guest_mode(void *data, int apicid, bool ga_log_intr);
 extern int amd_iommu_deactivate_guest_mode(void *data);
 
 #else /* defined(CONFIG_AMD_IOMMU) && defined(CONFIG_IRQ_REMAP) */
@@ -42,12 +42,12 @@ amd_iommu_register_ga_log_notifier(int (*notifier)(u32))
 	return 0;
 }
 
-static inline int amd_iommu_update_ga(void *data, int cpu, bool ga_log_intr)
+static inline int amd_iommu_update_ga(void *data, int apicid, bool ga_log_intr)
 {
 	return 0;
 }
 
-static inline int amd_iommu_activate_guest_mode(void *data, int cpu, bool ga_log_intr)
+static inline int amd_iommu_activate_guest_mode(void *data, int apicid, bool ga_log_intr)
 {
 	return 0;
 }
Signature · 1 line
2.34.1
#2 lore raw

PATCH v7 2/7 iommu/amd: KVM: SVM: Rename ga_log_intr to wakeup_intr in IOMMU interface

In reply to Sairaj Kodilkar

When AVIC is enabled, device interrupts are posted into the guest virtual APIC backing page regardless of vCPU running state. When the vCPU is not running, KVM may additionally need the IOMMU to notify the host so the vCPU can be woken, historically via the GA log mechanism.

The existing ga_log_intr name is tied to one specific hardware mechanism. Guest APIC Physical Processor Interrupt (GAPPI) provides an alternative host-notification path when the vCPU is not running, i.e., IRTE[IsRun] = 0. The KVM-side intent is the same in both cases: request host wakeup notification while the vCPU is blocked.

Rename ga_log_intr to wakeup_intr in the IOMMU API to describe KVM's intent rather than a single hardware knob. The IOMMU driver still maps wakeup_intr to IRTE[GALogIntr] in this patch; GAPPI support is added in following patches.

Similarly rename the synthetic AVIC physical ID table shadow bit from AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR to AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR.

No functional change is intended.

---
 arch/x86/include/asm/irq_remapping.h |  2 +-
 arch/x86/include/asm/svm.h           |  9 +++++----
 arch/x86/kvm/svm/avic.c              | 26 +++++++++++++-------------
 drivers/iommu/amd/iommu.c            | 16 ++++++++--------
 include/linux/amd-iommu.h            |  8 ++++----
 5 files changed, 31 insertions(+), 30 deletions(-)
arch/x86/include/asm/irq_remapping.h +1 -1
index 263f4df429d8..789e3e154600 100644
@@ -36,7 +36,7 @@ struct amd_iommu_pi_data {
 	u32 ga_tag;
 	u32 vector;		/* Guest vector of the interrupt */
 	int apicid;
-	bool ga_log_intr;
+	bool wakeup_intr;
 	bool is_guest_mode;
 	void *ir_data;
 };
arch/x86/include/asm/svm.h +5 -4
index 83fd49b26a4b..3f38fafec138 100644
@@ -264,11 +264,12 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
 #define AVIC_LOGICAL_ID_ENTRY_VALID_MASK		(1 << 31)
 
 /*
- * GA_LOG_INTR is a synthetic flag that's never propagated to hardware-visible
- * tables.  GA_LOG_INTR is set if the vCPU needs device posted IRQs to generate
- * GA log interrupts to wake the vCPU (because it's blocking or about to block).
+ * WAKEUP_INTR is a synthetic flag that's never propagated to hardware-visible
+ * tables.  WAKEUP_INTR is set if the vCPU needs device posted IRQs to generate
+ * GA log/GAPPI interrupts to wake the vCPU (because it's blocking or about to
+ * block).
  */
-#define AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR		BIT_ULL(61)
+#define AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR		BIT_ULL(61)
 
 #define AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK	GENMASK_ULL(11, 0)
 #define AVIC_PHYSICAL_ID_ENTRY_BACKING_PAGE_MASK	GENMASK_ULL(51, 12)
arch/x86/kvm/svm/avic.c +13 -13
index 5ac5ce4039a6..edbdb5d75062 100644
@@ -963,7 +963,7 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,
 			pi_data.apicid = entry & AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK;
 		} else {
 			pi_data.apicid = -1;
-			pi_data.ga_log_intr = entry & AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR;
+			pi_data.wakeup_intr = entry & AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR;
 		}
 
 		ret = irq_set_vcpu_affinity(host_irq, &pi_data);
@@ -1002,15 +1002,15 @@ enum avic_vcpu_action {
 	 * No unique action is required to deal with a vCPU that stops/starts
 	 * running.  A vCPU that starts running by definition stops blocking as
 	 * well, and a vCPU that stops running can't have been blocking, i.e.
-	 * doesn't need to toggle GALogIntr.
+	 * doesn't need to toggle host wakeup notification.
 	 */
 	AVIC_START_RUNNING	= 0,
 	AVIC_STOP_RUNNING	= 0,
 
 	/*
-	 * When a vCPU starts blocking, KVM needs to set the GALogIntr flag
-	 * int all associated IRTEs so that KVM can wake the vCPU if an IRQ is
-	 * sent to the vCPU.
+	 * When a vCPU starts blocking, KVM needs to request host wakeup
+	 * notification in all associated IRTEs so that KVM can wake the vCPU
+	 * if an IRQ is sent to the vCPU.
 	 */
 	AVIC_START_BLOCKING	= BIT(1),
 };
@@ -1018,7 +1018,7 @@ enum avic_vcpu_action {
 static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid,
 					    enum avic_vcpu_action action)
 {
-	bool ga_log_intr = (action & AVIC_START_BLOCKING);
+	bool wakeup_intr = (action & AVIC_START_BLOCKING);
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct kvm_kernel_irqfd *irqfd;
 
@@ -1035,9 +1035,9 @@ static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid,
 		void *data = irqfd->irq_bypass_data;
 
 		if (!(action & AVIC_TOGGLE_ON_OFF))
-			WARN_ON_ONCE(amd_iommu_update_ga(data, apicid, ga_log_intr));
+			WARN_ON_ONCE(amd_iommu_update_ga(data, apicid, wakeup_intr));
 		else if (apicid >= 0)
-			WARN_ON_ONCE(amd_iommu_activate_guest_mode(data, apicid, ga_log_intr));
+			WARN_ON_ONCE(amd_iommu_activate_guest_mode(data, apicid, wakeup_intr));
 		else
 			WARN_ON_ONCE(amd_iommu_deactivate_guest_mode(data));
 	}
@@ -1073,7 +1073,7 @@ static void __avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu,
 	WARN_ON_ONCE(entry & AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK);
 
 	entry &= ~(AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK |
-		   AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR);
+		   AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR);
 	entry |= (h_physical_id & AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK);
 	entry |= AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK;
 
@@ -1134,7 +1134,7 @@ static void __avic_vcpu_put(struct kvm_vcpu *vcpu, enum avic_vcpu_action action)
 
 	avic_update_iommu_vcpu_affinity(vcpu, -1, action);
 
-	WARN_ON_ONCE(entry & AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR);
+	WARN_ON_ONCE(entry & AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR);
 
 	/*
 	 * Keep the previous APIC ID in the entry so that a rogue doorbell from
@@ -1146,11 +1146,11 @@ static void __avic_vcpu_put(struct kvm_vcpu *vcpu, enum avic_vcpu_action action)
 		WRITE_ONCE(kvm_svm->avic_physical_id_table[vcpu->vcpu_id], entry);
 
 	/*
-	 * Note!  Don't set AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR in the table as
+	 * Note!  Don't set AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR in the table as
 	 * it's a synthetic flag that usurps an unused should-be-zero bit.
 	 */
 	if (action & AVIC_START_BLOCKING)
-		entry |= AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR;
+		entry |= AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR;
 
 	svm->avic_physical_id_entry = entry;
 
@@ -1182,7 +1182,7 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu)
 		 * The vCPU was preempted while blocking, ensure its IRTEs are
 		 * configured to generate GA Log Interrupts.
 		 */
-		if (!(WARN_ON_ONCE(!(entry & AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR))))
+		if (!(WARN_ON_ONCE(!(entry & AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR))))
 			return;
 	}
 
drivers/iommu/amd/iommu.c +8 -8
index 26afedfbf0bc..fcf23ed4ce43 100644
@@ -4054,7 +4054,7 @@ static const struct irq_domain_ops amd_ir_domain_ops = {
 };
 
 static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,
-				  bool ga_log_intr)
+				  bool wakeup_intr)
 {
 	if (apicid >= 0) {
 		entry->lo.fields_vapic.destination =
@@ -4065,7 +4065,7 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,
 		entry->lo.fields_vapic.ga_log_intr = false;
 	} else {
 		entry->lo.fields_vapic.is_run = false;
-		entry->lo.fields_vapic.ga_log_intr = ga_log_intr;
+		entry->lo.fields_vapic.ga_log_intr = wakeup_intr;
 	}
 }
 
@@ -4076,7 +4076,7 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,
  * If the vCPU is associated with a pCPU (@apicid >= 0), configure the
  * Destination with the pCPU's APIC ID, set IsRun, and clear GALogIntr.  If the
  * vCPU isn't associated with a pCPU (@apicid < 0), clear IsRun and set/clear
- * GALogIntr based on input from the caller (e.g. KVM only requests GALogIntr
+ * GALogIntr based on input from the caller (e.g. KVM only requests wakeup_intr
  * when the vCPU is blocking and requires a notification wake event).  I.e.
  * treat vCPUs that are associated with a pCPU as running.  This API is
  * intended to be used when a vCPU is scheduled in/out (or stops running for
@@ -4087,7 +4087,7 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,
  * and thus don't require an invalidation to ensure the IOMMU consumes fresh
  * information.
  */
-int amd_iommu_update_ga(void *data, int apicid, bool ga_log_intr)
+int amd_iommu_update_ga(void *data, int apicid, bool wakeup_intr)
 {
 	struct amd_ir_data *ir_data = (struct amd_ir_data *)data;
 	struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -4101,14 +4101,14 @@ int amd_iommu_update_ga(void *data, int apicid, bool ga_log_intr)
 	if (!ir_data->iommu)
 		return -ENODEV;
 
-	__amd_iommu_update_ga(entry, apicid, ga_log_intr);
+	__amd_iommu_update_ga(entry, apicid, wakeup_intr);
 
 	return __modify_irte_ga(ir_data->iommu, ir_data->irq_2_irte.devid,
 				ir_data->irq_2_irte.index, entry);
 }
 EXPORT_SYMBOL(amd_iommu_update_ga);
 
-int amd_iommu_activate_guest_mode(void *data, int apicid, bool ga_log_intr)
+int amd_iommu_activate_guest_mode(void *data, int apicid, bool wakeup_intr)
 {
 	struct amd_ir_data *ir_data = (struct amd_ir_data *)data;
 	struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -4131,7 +4131,7 @@ int amd_iommu_activate_guest_mode(void *data, int apicid, bool ga_log_intr)
 	entry->hi.fields.vector            = ir_data->ga_vector;
 	entry->lo.fields_vapic.ga_tag      = ir_data->ga_tag;
 
-	__amd_iommu_update_ga(entry, apicid, ga_log_intr);
+	__amd_iommu_update_ga(entry, apicid, wakeup_intr);
 
 	return modify_irte_ga(ir_data->iommu, ir_data->irq_2_irte.devid,
 			      ir_data->irq_2_irte.index, entry);
@@ -4203,7 +4203,7 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *info)
 		ir_data->ga_tag = pi_data->ga_tag;
 		if (pi_data->is_guest_mode)
 			ret = amd_iommu_activate_guest_mode(ir_data, pi_data->apicid,
-							    pi_data->ga_log_intr);
+							    pi_data->wakeup_intr);
 		else
 			ret = amd_iommu_deactivate_guest_mode(ir_data);
 	} else {
include/linux/amd-iommu.h +4 -4
index a467e3e90acc..7253456331de 100644
@@ -30,8 +30,8 @@ static inline void amd_iommu_detect(void) { }
 /* IOMMU AVIC Function */
 extern int amd_iommu_register_ga_log_notifier(int (*notifier)(u32));
 
-extern int amd_iommu_update_ga(void *data, int apicid, bool ga_log_intr);
-extern int amd_iommu_activate_guest_mode(void *data, int apicid, bool ga_log_intr);
+extern int amd_iommu_update_ga(void *data, int apicid, bool wakeup_intr);
+extern int amd_iommu_activate_guest_mode(void *data, int apicid, bool wakeup_intr);
 extern int amd_iommu_deactivate_guest_mode(void *data);
 
 #else /* defined(CONFIG_AMD_IOMMU) && defined(CONFIG_IRQ_REMAP) */
@@ -42,12 +42,12 @@ amd_iommu_register_ga_log_notifier(int (*notifier)(u32))
 	return 0;
 }
 
-static inline int amd_iommu_update_ga(void *data, int apicid, bool ga_log_intr)
+static inline int amd_iommu_update_ga(void *data, int apicid, bool wakeup_intr)
 {
 	return 0;
 }
 
-static inline int amd_iommu_activate_guest_mode(void *data, int apicid, bool ga_log_intr)
+static inline int amd_iommu_activate_guest_mode(void *data, int apicid, bool wakeup_intr)
 {
 	return 0;
 }
Signature · 1 line
2.34.1
#3 lore raw

PATCH v7 3/7 iommu/amd: KVM: SVM: Add explicit vCPU running state to IOMMU interface

In reply to Sairaj Kodilkar

Guest APIC Physical Processor Interrupt (GAPPI) is an AMD IOMMU mechanism for notifying the host when a device interrupt targets a non-running vCPU (IRTE[IsRun] = 0). Unlike the GA log path, GAPPI delivers a physical APIC interrupt directly to the host CPU described by IRTE[Destination], with the vector carried in IRTE[GATag], while still posting the interrupt into the guest virtual APIC backing page as usual.

In GAPPI mode, the IOMMU delivers the notification to the host CPU using the physical APIC ID in IRTE[Destination] and the vector in IRTE[GATag]. KVM must therefore supply a valid apicid even when the vCPU is not running. The prior interface inferred running state from apicid: apicid >= 0 meant running and apicid == -1 meant not running. That encoding breaks once apicid carries the GAPPI destination while the vCPU is not running; the IOMMU driver can no longer derive vCPU running state from apicid alone and needs an explicit indication from KVM.

Add an explicit is_running indication to the IOMMU interface so that SVM can pass vCPU running state to the IOMMU.

SVM calls amd_iommu_activate_guest_mode() only when the vCPU is running, so keep is_running out of the exported prototype and make it a thin wrapper that hardcodes is_running = true. The IOMMU internal caller, amd_ir_set_vcpu_affinity(), calls __amd_iommu_activate_guest_mode() directly and passes the running state supplied by KVM, i.e. it handles both a running and a non-running vCPU.

No functional change is intended.

---
 arch/x86/include/asm/irq_remapping.h |  1 +
 arch/x86/kvm/svm/avic.c              |  9 +++++---
 drivers/iommu/amd/iommu.c            | 34 ++++++++++++++++------------
 include/linux/amd-iommu.h            | 10 ++++----
 4 files changed, 33 insertions(+), 21 deletions(-)
arch/x86/include/asm/irq_remapping.h +1 -0
index 789e3e154600..af0cdfa32c44 100644
@@ -37,6 +37,7 @@ struct amd_iommu_pi_data {
 	u32 vector;		/* Guest vector of the interrupt */
 	int apicid;
 	bool wakeup_intr;
+	bool is_running;
 	bool is_guest_mode;
 	void *ir_data;
 };
arch/x86/kvm/svm/avic.c +6 -3
index edbdb5d75062..b975f08825ce 100644
@@ -961,9 +961,11 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,
 		entry = svm->avic_physical_id_entry;
 		if (entry & AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK) {
 			pi_data.apicid = entry & AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK;
+			pi_data.is_running = true;
 		} else {
 			pi_data.apicid = -1;
 			pi_data.wakeup_intr = entry & AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR;
+			pi_data.is_running = false;
 		}
 
 		ret = irq_set_vcpu_affinity(host_irq, &pi_data);
@@ -1019,6 +1021,7 @@ static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid,
 					    enum avic_vcpu_action action)
 {
 	bool wakeup_intr = (action & AVIC_START_BLOCKING);
+	bool is_running = apicid >= 0;
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct kvm_kernel_irqfd *irqfd;
 
@@ -1035,9 +1038,9 @@ static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid,
 		void *data = irqfd->irq_bypass_data;
 
 		if (!(action & AVIC_TOGGLE_ON_OFF))
-			WARN_ON_ONCE(amd_iommu_update_ga(data, apicid, wakeup_intr));
-		else if (apicid >= 0)
-			WARN_ON_ONCE(amd_iommu_activate_guest_mode(data, apicid, wakeup_intr));
+			WARN_ON_ONCE(amd_iommu_update_ga(data, apicid, wakeup_intr, is_running));
+		else if (is_running)
+			WARN_ON_ONCE(amd_iommu_activate_guest_mode(data, apicid));
 		else
 			WARN_ON_ONCE(amd_iommu_deactivate_guest_mode(data));
 	}
drivers/iommu/amd/iommu.c +20 -14
index fcf23ed4ce43..c00a85b3a090 100644
@@ -4054,9 +4054,9 @@ static const struct irq_domain_ops amd_ir_domain_ops = {
 };
 
 static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,
-				  bool wakeup_intr)
+				  bool wakeup_intr, bool is_running)
 {
-	if (apicid >= 0) {
+	if (is_running) {
 		entry->lo.fields_vapic.destination =
 					APICID_TO_IRTE_DEST_LO(apicid);
 		entry->hi.fields.destination =
@@ -4073,12 +4073,11 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,
  * Update the pCPU information for an IRTE that is configured to post IRQs to
  * a vCPU, without issuing an IOMMU invalidation for the IRTE.
  *
- * If the vCPU is associated with a pCPU (@apicid >= 0), configure the
- * Destination with the pCPU's APIC ID, set IsRun, and clear GALogIntr.  If the
- * vCPU isn't associated with a pCPU (@apicid < 0), clear IsRun and set/clear
- * GALogIntr based on input from the caller (e.g. KVM only requests wakeup_intr
- * when the vCPU is blocking and requires a notification wake event).  I.e.
- * treat vCPUs that are associated with a pCPU as running.  This API is
+ * If the vCPU is scheduled to run on pCPU (@is_running = 1), configure the
+ * Destination with the pCPU's APIC ID, set IsRun, and clear GALogIntr. If the
+ * vCPU is scheduled out (@is_running = 0), clear IsRun and set/clear GALogIntr
+ * based on input from the caller (e.g. KVM only requests wakeup_intr when the
+ * vCPU is blocking and requires a notification wake event). This API is
  * intended to be used when a vCPU is scheduled in/out (or stops running for
  * any reason), to do a fast update of IsRun, GALogIntr, and (conditionally)
  * Destination.
@@ -4087,7 +4086,7 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,
  * and thus don't require an invalidation to ensure the IOMMU consumes fresh
  * information.
  */
-int amd_iommu_update_ga(void *data, int apicid, bool wakeup_intr)
+int amd_iommu_update_ga(void *data, int apicid, bool wakeup_intr, bool is_running)
 {
 	struct amd_ir_data *ir_data = (struct amd_ir_data *)data;
 	struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -4101,14 +4100,15 @@ int amd_iommu_update_ga(void *data, int apicid, bool wakeup_intr)
 	if (!ir_data->iommu)
 		return -ENODEV;
 
-	__amd_iommu_update_ga(entry, apicid, wakeup_intr);
+	__amd_iommu_update_ga(entry, apicid, wakeup_intr, is_running);
 
 	return __modify_irte_ga(ir_data->iommu, ir_data->irq_2_irte.devid,
 				ir_data->irq_2_irte.index, entry);
 }
 EXPORT_SYMBOL(amd_iommu_update_ga);
 
-int amd_iommu_activate_guest_mode(void *data, int apicid, bool wakeup_intr)
+static int __amd_iommu_activate_guest_mode(void *data, int apicid, bool wakeup_intr,
+					   bool is_running)
 {
 	struct amd_ir_data *ir_data = (struct amd_ir_data *)data;
 	struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
@@ -4131,11 +4131,16 @@ int amd_iommu_activate_guest_mode(void *data, int apicid, bool wakeup_intr)
 	entry->hi.fields.vector            = ir_data->ga_vector;
 	entry->lo.fields_vapic.ga_tag      = ir_data->ga_tag;
 
-	__amd_iommu_update_ga(entry, apicid, wakeup_intr);
+	__amd_iommu_update_ga(entry, apicid, wakeup_intr, is_running);
 
 	return modify_irte_ga(ir_data->iommu, ir_data->irq_2_irte.devid,
 			      ir_data->irq_2_irte.index, entry);
 }
+
+int amd_iommu_activate_guest_mode(void *data, int apicid)
+{
+	return __amd_iommu_activate_guest_mode(data, apicid, false, true);
+}
 EXPORT_SYMBOL(amd_iommu_activate_guest_mode);
 
 int amd_iommu_deactivate_guest_mode(void *data)
@@ -4202,8 +4207,9 @@ static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *info)
 		ir_data->ga_vector = pi_data->vector;
 		ir_data->ga_tag = pi_data->ga_tag;
 		if (pi_data->is_guest_mode)
-			ret = amd_iommu_activate_guest_mode(ir_data, pi_data->apicid,
-							    pi_data->wakeup_intr);
+			ret = __amd_iommu_activate_guest_mode(ir_data, pi_data->apicid,
+							      pi_data->wakeup_intr,
+							      pi_data->is_running);
 		else
 			ret = amd_iommu_deactivate_guest_mode(ir_data);
 	} else {
include/linux/amd-iommu.h +6 -4
index 7253456331de..2b340f0f5b31 100644
@@ -30,8 +30,9 @@ static inline void amd_iommu_detect(void) { }
 /* IOMMU AVIC Function */
 extern int amd_iommu_register_ga_log_notifier(int (*notifier)(u32));
 
-extern int amd_iommu_update_ga(void *data, int apicid, bool wakeup_intr);
-extern int amd_iommu_activate_guest_mode(void *data, int apicid, bool wakeup_intr);
+extern int amd_iommu_update_ga(void *data, int apicid, bool wakeup_intr,
+			       bool is_running);
+extern int amd_iommu_activate_guest_mode(void *data, int apicid);
 extern int amd_iommu_deactivate_guest_mode(void *data);
 
 #else /* defined(CONFIG_AMD_IOMMU) && defined(CONFIG_IRQ_REMAP) */
@@ -42,12 +43,13 @@ amd_iommu_register_ga_log_notifier(int (*notifier)(u32))
 	return 0;
 }
 
-static inline int amd_iommu_update_ga(void *data, int apicid, bool wakeup_intr)
+static inline int amd_iommu_update_ga(void *data, int apicid, bool wakeup_intr,
+				      bool is_running)
 {
 	return 0;
 }
 
-static inline int amd_iommu_activate_guest_mode(void *data, int apicid, bool wakeup_intr)
+static inline int amd_iommu_activate_guest_mode(void *data, int apicid)
 {
 	return 0;
 }
Signature · 1 line
2.34.1
#4 lore raw

PATCH v7 4/7 iommu/amd: Program guest-mode IRTEs for GAPPI wakeup when IRTE[IsRun] = 0

In reply to Sairaj Kodilkar

Guest APIC Physical Processor Interrupt (GAPPI) is an alternative to the GA log for notifying the host when a device interrupt targets a non-running vCPU (IRTE[IsRun] = 0). Per the AMD IOMMU specification [1] (section 2.2.5.4), with GAPPI enabled the IOMMU delivers a physical APIC interrupt to the CPU described by IRTE[Destination], using IRTE[GATag][7:0] as the vector, while still updating the guest vAPIC backing page IRR as usual.

The AMD IOMMU also allows suppressing GAPPI interrupts using the IRTE[GAPPIDis] bit when the GAPPIDisSup feature is available in extended feature register 2. The AMD IOMMU driver sets this bit when the wakeup_intr flag is not set by KVM.

Note: amd_iommu_gappi is currently false; a later patch enables it via the kernel command line.

[1] https://docs.amd.com/v/u/en-US/48882_3.11_IOMMU_PUB

---
 drivers/iommu/amd/amd_iommu_types.h |  4 +++-
 drivers/iommu/amd/init.c            |  5 +++++
 drivers/iommu/amd/iommu.c           | 31 +++++++++++++++++++----------
 include/linux/amd-iommu.h           |  6 ++++++
 4 files changed, 35 insertions(+), 11 deletions(-)
drivers/iommu/amd/amd_iommu_types.h +3 -1
index f98770db0090..4600674baf84 100644
@@ -100,6 +100,7 @@
 /* Extended Feature 2 Bits */
 #define FEATURE_SEVSNPIO_SUP	BIT_ULL(1)
 #define FEATURE_GCR3TRPMODE	BIT_ULL(3)
+#define FEATURE_GAPPIDISSUP	BIT_ULL(4)
 #define FEATURE_SNPAVICSUP	GENMASK_ULL(7, 5)
 #define FEATURE_SNPAVICSUP_GAM(x) \
 	(FIELD_GET(FEATURE_SNPAVICSUP, x) == 0x1)
@@ -992,7 +993,8 @@ union irte_ga_lo {
 		    no_fault	: 1,
 		    /* ------ */
 		    ga_log_intr	: 1,
-		    rsvd1	: 3,
+		    rsvd1	: 2,
+		    gappi_dis	: 1,
 		    is_run	: 1,
 		    /* ------ */
 		    guest_mode	: 1,
drivers/iommu/amd/init.c +5 -0
index 58f63298493b..054ef7262f3c 100644
@@ -160,6 +160,11 @@ u8 amd_iommu_hpt_vasize;
 /* Guest page table level */
 int amd_iommu_gpt_level = PAGE_MODE_4_LEVEL;
 
+#ifdef CONFIG_IRQ_REMAP
+bool amd_iommu_gappi;
+EXPORT_SYMBOL(amd_iommu_gappi);
+#endif
+
 int amd_iommu_guest_ir = AMD_IOMMU_GUEST_IR_VAPIC;
 static int amd_iommu_xt_mode = IRQ_REMAP_XAPIC_MODE;
 
drivers/iommu/amd/iommu.c +21 -10
index c00a85b3a090..3557d6e0d65c 100644
@@ -4063,9 +4063,20 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,
 					APICID_TO_IRTE_DEST_HI(apicid);
 		entry->lo.fields_vapic.is_run = true;
 		entry->lo.fields_vapic.ga_log_intr = false;
+		entry->lo.fields_vapic.gappi_dis = false;
 	} else {
-		entry->lo.fields_vapic.is_run = false;
-		entry->lo.fields_vapic.ga_log_intr = wakeup_intr;
+		if (amd_iommu_gappi) {
+			entry->lo.fields_vapic.gappi_dis = !wakeup_intr &&
+						check_feature2(FEATURE_GAPPIDISSUP);
+			entry->lo.fields_vapic.is_run = false;
+			entry->lo.fields_vapic.destination =
+						APICID_TO_IRTE_DEST_LO(apicid);
+			entry->hi.fields.destination =
+						APICID_TO_IRTE_DEST_HI(apicid);
+		} else {
+			entry->lo.fields_vapic.is_run = false;
+			entry->lo.fields_vapic.ga_log_intr = wakeup_intr;
+		}
 	}
 }
 
@@ -4076,15 +4087,15 @@ static void __amd_iommu_update_ga(struct irte_ga *entry, int apicid,
  * If the vCPU is scheduled to run on pCPU (@is_running = 1), configure the
  * Destination with the pCPU's APIC ID, set IsRun, and clear GALogIntr. If the
  * vCPU is scheduled out (@is_running = 0), clear IsRun and set/clear GALogIntr
- * based on input from the caller (e.g. KVM only requests wakeup_intr when the
- * vCPU is blocking and requires a notification wake event). This API is
- * intended to be used when a vCPU is scheduled in/out (or stops running for
- * any reason), to do a fast update of IsRun, GALogIntr, and (conditionally)
- * Destination.
+ * and GAPPIDis based on input from the caller (e.g. KVM only requests
+ * wakeup_intr when the vCPU is blocking and requires a notification wake
+ * event). This API is intended to be used when a vCPU is scheduled in/out (or
+ * stops running for any reason), to do a fast update of IsRun, GALogIntr,
+ * GAPPIDis and (conditionally) Destination.
  *
- * Per the IOMMU spec, the Destination, IsRun, and GATag fields are not cached
- * and thus don't require an invalidation to ensure the IOMMU consumes fresh
- * information.
+ * Per the IOMMU spec, the Destination, IsRun, GATag and GAPPIDis fields are
+ * not cached and thus don't require an invalidation to ensure the IOMMU
+ * consumes fresh information.
  */
 int amd_iommu_update_ga(void *data, int apicid, bool wakeup_intr, bool is_running)
 {
include/linux/amd-iommu.h +6 -0
index 2b340f0f5b31..14be1322b7b6 100644
@@ -35,6 +35,9 @@ extern int amd_iommu_update_ga(void *data, int apicid, bool wakeup_intr,
 extern int amd_iommu_activate_guest_mode(void *data, int apicid);
 extern int amd_iommu_deactivate_guest_mode(void *data);
 
+/* IOMMU AVIC Flag */
+extern bool amd_iommu_gappi;
+
 #else /* defined(CONFIG_AMD_IOMMU) && defined(CONFIG_IRQ_REMAP) */
 
 static inline int
@@ -58,6 +61,9 @@ static inline int amd_iommu_deactivate_guest_mode(void *data)
 {
 	return 0;
 }
+
+#define amd_iommu_gappi false
+
 #endif /* defined(CONFIG_AMD_IOMMU) && defined(CONFIG_IRQ_REMAP) */
 
 int amd_iommu_get_num_iommus(void);
Signature · 1 line
2.34.1
#5 lore raw

PATCH v7 5/7 KVM: VMX: Factor out wakeup list handling code to KVM

In reply to Sairaj Kodilkar

VMX maintains a per-pCPU list of blocked vCPUs so that they can be woken when a device interrupt arrives while the vCPU is not running. The vCPU is queued on the list of the pCPU it last ran on and its notification vector is switched to POSTED_INTR_WAKEUP_VECTOR, so the handler for that vector walks the local list and wakes the vCPUs that have a notification pending.

Guest APIC Physical Processor Interrupt (GAPPI) needs the same infrastructure on AMD. When a device interrupt targets a non-running vCPU (IRTE[IsRun] = 0), the IOMMU delivers a physical APIC interrupt to IRTE[Destination] using IRTE[GATag][7:0] as the vector, which KVM programs to POSTED_INTR_WAKEUP_VECTOR. SVM therefore requires the same per-pCPU list of blocked vCPUs that VMX already has. Hence Move the per-pCPU list, its lock, the wakeup handler and the per-CPU initialization from VMX to common KVM x86 code.

Intel and AMD determine whether a queued vCPU has a notification pending in different ways. Hence, introduce a new kvm_x86_ops callback vcpu_irq_pending(), which the wakeup handler uses to decide whether a vCPU needs to be woken.

---
 arch/x86/include/asm/kvm-x86-ops.h |   1 +
 arch/x86/include/asm/kvm_host.h    |   1 +
 arch/x86/kvm/vmx/common.h          |   3 -
 arch/x86/kvm/vmx/main.c            |   6 ++
 arch/x86/kvm/vmx/posted_intr.c     |  82 +--------------------
 arch/x86/kvm/vmx/posted_intr.h     |   2 -
 arch/x86/kvm/vmx/tdx.c             |   1 -
 arch/x86/kvm/vmx/vmx.c             |  14 ++--
 arch/x86/kvm/vmx/x86_ops.h         |   1 +
 arch/x86/kvm/x86.c                 | 111 +++++++++++++++++++++++++++++
 include/linux/kvm_host.h           |  10 +++
 11 files changed, 139 insertions(+), 93 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index 4dde4e3e608a..7c1f8ce7de9f 100644
@@ -30,6 +30,7 @@ KVM_X86_OP(vcpu_reset)
 KVM_X86_OP(prepare_switch_to_guest)
 KVM_X86_OP(vcpu_load)
 KVM_X86_OP(vcpu_put)
+KVM_X86_OP_OPTIONAL_RET0(vcpu_irq_pending)
 KVM_X86_OP(update_exception_bitmap)
 KVM_X86_OP(get_msr)
 KVM_X86_OP(set_msr)
arch/x86/include/asm/kvm_host.h +1 -0
index 1ee4b127004d..9750ace7d429 100644
@@ -1534,6 +1534,7 @@ struct kvm_x86_ops {
 	void (*prepare_switch_to_guest)(struct kvm_vcpu *vcpu);
 	void (*vcpu_load)(struct kvm_vcpu *vcpu, int cpu);
 	void (*vcpu_put)(struct kvm_vcpu *vcpu);
+	bool (*vcpu_irq_pending)(struct kvm_vcpu *vcpu);
 
 	/*
 	 * Mask of DEBUGCTL bits that are owned by the host, i.e. that need to
arch/x86/kvm/vmx/common.h +0 -3
index b2905335a2d2..c5b41e12df58 100644
@@ -37,9 +37,6 @@ struct vcpu_vt {
 	/* Posted interrupt descriptor */
 	struct pi_desc pi_desc;
 
-	/* Used if this vCPU is waiting for PI notification wakeup. */
-	struct list_head pi_wakeup_list;
-
 	union vmx_exit_reason exit_reason;
 
 	unsigned long	exit_qualification;
arch/x86/kvm/vmx/main.c +6 -0
index d75170a6199e..48a294364358 100644
@@ -113,6 +113,11 @@ static void vt_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 	vmx_vcpu_load(vcpu, cpu);
 }
 
+static bool vt_vcpu_irq_pending(struct kvm_vcpu *vcpu)
+{
+	return vmx_vcpu_irq_pending(vcpu);
+}
+
 static void vt_update_cpu_dirty_logging(struct kvm_vcpu *vcpu, bool enable)
 {
 	/*
@@ -1024,6 +1029,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.prepare_switch_to_guest = vt_op(prepare_switch_to_guest),
 	.vcpu_load = vt_op(vcpu_load),
 	.vcpu_put = vt_op(vcpu_put),
+	.vcpu_irq_pending = vt_op(vcpu_irq_pending),
 
 	.HOST_OWNED_DEBUGCTL = VMX_HOST_OWNED_DEBUGCTL_BITS,
 
arch/x86/kvm/vmx/posted_intr.c +2 -80

114 lines not on this page · raw message

arch/x86/kvm/vmx/posted_intr.h +0 -2
index c5860ea828ae..3180436049e8 100644
@@ -10,8 +10,6 @@
 
 void vt_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu);
 void vt_vcpu_pi_put(struct kvm_vcpu *vcpu);
-void pi_wakeup_handler(void);
-void __init pi_init_cpu(int cpu);
 void pi_apicv_pre_state_restore(struct kvm_vcpu *vcpu);
 bool pi_has_pending_interrupt(struct kvm_vcpu *vcpu);
 int vt_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,
arch/x86/kvm/vmx/tdx.c +0 -1
index c213c0eba919..7f44111d06cc 100644
@@ -683,7 +683,6 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu)
 
 	fpstate_set_confidential(&vcpu->arch.guest_fpu);
 	vcpu->arch.apic->guest_apic_protected = true;
-	INIT_LIST_HEAD(&tdx->vt.pi_wakeup_list);
 
 	vcpu->arch.efer = EFER_SCE | EFER_LME | EFER_LMA | EFER_NX;
 
arch/x86/kvm/vmx/vmx.c +7 -7
index 82eb891f684c..5c909069414d 100644
@@ -1587,6 +1587,13 @@ void vmx_vcpu_put(struct kvm_vcpu *vcpu)
 	vmx_prepare_switch_to_host(to_vmx(vcpu));
 }
 
+bool vmx_vcpu_irq_pending(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_vt *vt = to_vt(vcpu);
+
+	return pi_test_on(&vt->pi_desc);
+}
+
 static void vmx_switch_loaded_vmcs(struct kvm_vcpu *vcpu,
 				   struct loaded_vmcs *vmcs)
 {
@@ -7533,8 +7540,6 @@ int vmx_vcpu_create(struct kvm_vcpu *vcpu)
 	BUILD_BUG_ON(offsetof(struct vcpu_vmx, vcpu) != 0);
 	vmx = to_vmx(vcpu);
 
-	INIT_LIST_HEAD(&vmx->vt.pi_wakeup_list);
-
 	err = -ENOMEM;
 
 	vmx->vpid = allocate_vpid();
@@ -8413,8 +8418,6 @@ void vmx_migrate_timers(struct kvm_vcpu *vcpu)
 
 void vmx_hardware_unsetup(void)
 {
-	kvm_set_posted_intr_wakeup_handler(NULL);
-
 	if (nested)
 		nested_vmx_hardware_unsetup();
 }
@@ -8695,7 +8698,6 @@ __init int vmx_hardware_setup(void)
 	}
 	vmx_nested_ops.enabled = nested;
 
-	kvm_set_posted_intr_wakeup_handler(pi_wakeup_handler);
 
 	/*
 	 * On Intel CPUs that lack self-snoop feature, letting the guest control
@@ -8765,8 +8767,6 @@ int __init vmx_init(void)
 
 	for_each_possible_cpu(cpu) {
 		INIT_LIST_HEAD(&per_cpu(loaded_vmcss_on_cpu, cpu));
-
-		pi_init_cpu(cpu);
 	}
 
 	vmx_check_vmcs12_offsets();
arch/x86/kvm/vmx/x86_ops.h +1 -0
index cd72684d075d..767680731c0b 100644
@@ -26,6 +26,7 @@ void vmx_vcpu_free(struct kvm_vcpu *vcpu);
 void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);
 void vmx_vcpu_load(struct kvm_vcpu *vcpu, int cpu);
 void vmx_vcpu_put(struct kvm_vcpu *vcpu);
+bool vmx_vcpu_irq_pending(struct kvm_vcpu *vcpu);
 int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath);
 int vmx_skip_emulated_instruction(struct kvm_vcpu *vcpu);
 void vmx_update_emulated_instruction(struct kvm_vcpu *vcpu);
arch/x86/kvm/x86.c +111 -0

151 lines not on this page · raw message

include/linux/kvm_host.h +10 -0
index e7fc94e2910a..9d6b4794aaa0 100644
@@ -402,6 +402,11 @@ struct kvm_vcpu {
 	 */
 	struct kvm_memory_slot *last_used_slot;
 	u64 last_used_slot_gen;
+
+#ifdef CONFIG_X86
+	/* Used if this vCPU is waiting for PI notification wakeup. */
+	struct list_head pi_wakeup_list;
+#endif
 };
 
 /*
@@ -2689,4 +2694,9 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
 				    struct kvm_pre_fault_memory *range);
 #endif
 
+#ifdef CONFIG_X86
+void kvm_pi_enable_wakeup_handler(struct kvm_vcpu *vcpu, int cpu);
+void kvm_pi_disable_wakeup_handler(struct kvm_vcpu *vcpu, int cpu);
+#endif
+
 #endif
Signature · 1 line
2.34.1
#6 lore raw

PATCH v7 6/7 KVM: SVM: Add support for AMD IOMMU Guest APIC Physical Processor Interrupt (GAPPI)

In reply to Sairaj Kodilkar

With AVIC guest-mode interrupt remapping, device interrupts are posted into the guest vAPIC backing page by the IOMMU. When the vCPU is not running (IRTE[IsRun] = 0), KVM must still be notified to schedule it. The legacy path uses the GA log.

GAPPI (Guest APIC Physical Processor Interrupt) is an alternative to the GA log mechanism provided by the AMD IOMMU. With GAPPI enabled, the IOMMU still updates the vAPIC backing page IRR, but the host wakeup notification is delivered as a physical APIC interrupt to IRTE[Destination], using IRTE[GATag][7:0] as the vector (POSTED_INTR_WAKEUP_VECTOR).

SVM follows the Intel posted-interrupt wakeup model. Each pCPU maintains a list of blocked vCPUs that may be woken by a GAPPI delivery to that CPU. When a vCPU blocks while waiting for an interrupt, SVM enqueues it on the wakeup list of the pCPU on which it was previously running and passes that same pCPU's physical APIC ID to the IOMMU to program IRTE[Destination]. The rationale is that the vCPU is likely to run again on the same pCPU, which is common when vCPUs are pinned; targeting GAPPI notifications there reduces unnecessary VMEXITs from GAPPI deliveries on other CPUs. SVM registers the GAPPI handler via kvm_set_posted_intr_wakeup_handler(). On delivery, it walks the local vCPU list and wakes vCPUs with a pending IRR.

All GAPPI logic is gated on amd_iommu_gappi. Without it, KVM and the IOMMU falls back to the legacy GA log mechanism for vCPU wakeup.

---
 arch/x86/kvm/svm/avic.c | 75 ++++++++++++++++++++++++++++++++++++-----
 arch/x86/kvm/svm/svm.c  |  1 +
 arch/x86/kvm/svm/svm.h  |  4 +++
 3 files changed, 72 insertions(+), 8 deletions(-)
arch/x86/kvm/svm/avic.c +67 -8
index b975f08825ce..ebfd74adc4cb 100644
@@ -877,6 +877,8 @@ int avic_init_vcpu(struct vcpu_svm *svm)
 	INIT_LIST_HEAD(&svm->ir_list);
 	raw_spin_lock_init(&svm->ir_list_lock);
 
+	svm->gappi_cpu = 0;
+
 	if (!enable_apicv || !irqchip_in_kernel(vcpu->kvm))
 		return 0;
 
@@ -895,6 +897,17 @@ void avic_vcpu_free(struct kvm_vcpu *vcpu)
 
 	if (kvm_svm->avic_physical_id_table && avic_is_addressable_vcpu(vcpu))
 		WRITE_ONCE(kvm_svm->avic_physical_id_table[vcpu->vcpu_id], 0);
+
+	if (amd_iommu_gappi) {
+		struct vcpu_svm *svm = to_svm(vcpu);
+		unsigned long flags;
+
+		local_irq_save(flags);
+
+		kvm_pi_disable_wakeup_handler(vcpu, svm->gappi_cpu);
+
+		local_irq_restore(flags);
+	}
 }
 
 void avic_apicv_post_state_restore(struct kvm_vcpu *vcpu)
@@ -934,8 +947,6 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,
 		 * if AVIC is enabled/uninhibited in the future.
 		 */
 		struct amd_iommu_pi_data pi_data = {
-			.ga_tag = AVIC_GATAG(to_kvm_svm(kvm)->avic_vm_id,
-					     vcpu->vcpu_idx),
 			.is_guest_mode = kvm_vcpu_apicv_active(vcpu),
 			.vapic_addr = avic_get_backing_page_address(to_svm(vcpu)),
 			.vector = vector,
@@ -958,6 +969,12 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,
 		 * scheduled out, KVM will update the pCPU info when the vCPU
 		 * is awakened and/or scheduled in.  See also avic_vcpu_load().
 		 */
+		if (amd_iommu_gappi)
+			pi_data.ga_tag = POSTED_INTR_WAKEUP_VECTOR;
+		else
+			pi_data.ga_tag = AVIC_GATAG(to_kvm_svm(kvm)->avic_vm_id,
+						    vcpu->vcpu_idx);
+
 		entry = svm->avic_physical_id_entry;
 		if (entry & AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK) {
 			pi_data.apicid = entry & AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK;
@@ -966,6 +983,9 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,
 			pi_data.apicid = -1;
 			pi_data.wakeup_intr = entry & AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR;
 			pi_data.is_running = false;
+
+			if (amd_iommu_gappi)
+				pi_data.apicid = cpu_physical_id(svm->gappi_cpu);
 		}
 
 		ret = irq_set_vcpu_affinity(host_irq, &pi_data);
@@ -1018,7 +1038,7 @@ enum avic_vcpu_action {
 };
 
 static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid,
-					    enum avic_vcpu_action action)
+					    int cpu, enum avic_vcpu_action action)
 {
 	bool wakeup_intr = (action & AVIC_START_BLOCKING);
 	bool is_running = apicid >= 0;
@@ -1027,6 +1047,36 @@ static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid,
 
 	lockdep_assert_held(&svm->ir_list_lock);
 
+	if (!amd_iommu_gappi)
+		goto skip_wakeup_handler;
+
+	if (action & AVIC_TOGGLE_ON_OFF) {
+		/*
+		 * When AVIC is disabled, IOMMU switches to legacy IRTE and
+		 * does not generate wakeup notification. Hence there is no
+		 * need to add vCPU on wakeup list. This reduces the number of
+		 * vCPUs on the list, effectively reducing interrupt handler
+		 * time and avoiding spurious wakeup to the vCPU.
+		 */
+		if (is_running)
+			/*
+			 * Store the CPU so that next call with
+			 * AVIC_STOP_RUNNING can add the vCPU on to
+			 * appropriate wakeup list.
+			 */
+			svm->gappi_cpu = cpu;
+		goto skip_wakeup_handler;
+	}
+
+	if (is_running) {
+		kvm_pi_disable_wakeup_handler(vcpu, svm->gappi_cpu);
+		svm->gappi_cpu = cpu;
+	} else {
+		apicid = cpu_physical_id(svm->gappi_cpu);
+		kvm_pi_enable_wakeup_handler(vcpu, svm->gappi_cpu);
+	}
+
+skip_wakeup_handler:
 	/*
 	 * Here, we go through the per-vcpu ir_list to update all existing
 	 * interrupt remapping table entry targeting this vcpu.
@@ -1093,7 +1143,7 @@ static void __avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu,
 
 	WRITE_ONCE(kvm_svm->avic_physical_id_table[vcpu->vcpu_id], entry);
 
-	avic_update_iommu_vcpu_affinity(vcpu, h_physical_id, action);
+	avic_update_iommu_vcpu_affinity(vcpu, h_physical_id, cpu, action);
 
 	raw_spin_unlock_irqrestore(&svm->ir_list_lock, flags);
 }
@@ -1135,7 +1185,7 @@ static void __avic_vcpu_put(struct kvm_vcpu *vcpu, enum avic_vcpu_action action)
 	 */
 	raw_spin_lock_irqsave(&svm->ir_list_lock, flags);
 
-	avic_update_iommu_vcpu_affinity(vcpu, -1, action);
+	avic_update_iommu_vcpu_affinity(vcpu, -1, -1, action);
 
 	WARN_ON_ONCE(entry & AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR);
 
@@ -1183,7 +1233,7 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu)
 
 		/*
 		 * The vCPU was preempted while blocking, ensure its IRTEs are
-		 * configured to generate GA Log Interrupts.
+		 * configured to request host wakeup notification.
 		 */
 		if (!(WARN_ON_ONCE(!(entry & AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR))))
 			return;
@@ -1308,6 +1358,13 @@ static bool __init avic_want_avic_enabled(void)
 	return true;
 }
 
+bool avic_vcpu_irq_pending(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_svm *svm = to_svm(vcpu);
+
+	return kvm_lapic_find_highest_irr(&svm->vcpu) >= 0;
+}
+
 /*
  * Note:
  * - The module param avic enable both xAPIC and x2APIC mode.
@@ -1351,6 +1408,8 @@ bool __init avic_hardware_setup(void)
 
 void avic_hardware_unsetup(void)
 {
-	if (avic)
-		amd_iommu_register_ga_log_notifier(NULL);
+	if (!avic)
+		return;
+
+	amd_iommu_register_ga_log_notifier(NULL);
 }
arch/x86/kvm/svm/svm.c +1 -0
index e30ee80624b7..a8f99fb474f5 100644
@@ -5486,6 +5486,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
 	.vcpu_put = svm_vcpu_put,
 	.vcpu_blocking = avic_vcpu_blocking,
 	.vcpu_unblocking = avic_vcpu_unblocking,
+	.vcpu_irq_pending = avic_vcpu_irq_pending,
 
 	.update_exception_bitmap = svm_update_exception_bitmap,
 	.get_feature_msr = svm_get_feature_msr,
arch/x86/kvm/svm/svm.h +4 -0
index cd3242811e43..331a7c8fb5c6 100644
@@ -369,6 +369,9 @@ struct vcpu_svm {
 
 	/* Guest GIF value, used when vGIF is not enabled */
 	bool guest_gif;
+
+	/* GAPPI related fields */
+	int gappi_cpu;
 };
 
 struct svm_cpu_data {
@@ -969,6 +972,7 @@ int avic_init_vcpu(struct vcpu_svm *svm);
 void avic_vcpu_free(struct kvm_vcpu *vcpu);
 void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu);
 void avic_vcpu_put(struct kvm_vcpu *vcpu);
+bool avic_vcpu_irq_pending(struct kvm_vcpu *vcpu);
 void avic_apicv_post_state_restore(struct kvm_vcpu *vcpu);
 void avic_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu);
 int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,
Signature · 1 line
2.34.1
#7 lore raw

Re: [PATCH v7 6/7] KVM: SVM: Add support for AMD IOMMU Guest APIC Physical Processor Interrupt (GAPPI)

Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [Medium] KVM: SVM: circular lock dependency in __avic_vcpu_load() with GAPPI - [Medium] KVM: SVM…

Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [Medium] KVM: SVM: circular lock dependency in __avic_vcpu_load() with GAPPI
- [Medium] KVM: SVM: non-blocking vCPUs queued on wakeup list on AVIC_STOP_RUNNING
- [Low] KVM: SVM: stale commit message and cleanup churn in avic_hardware_unsetup()
--

commit 569af3da1a9a9cddca4d1455d68973d185c0555b
Author: Sairaj Kodilkar <[email protected]>

KVM: SVM: Add support for AMD IOMMU Guest APIC Physical Processor Interrupt (GAPPI)

This commit adds KVM SVM support for AMD IOMMU Guest APIC Physical Processor Interrupt (GAPPI) to wake blocked vCPUs via POSTED_INTR_WAKEUP_VECTOR instead of the legacy GA log mechanism.

Quoted 4 lines
arch/x86/kvm/svm/avic.c +0 -0
index b975f08825ce2..ebfd74adc4cbd 100644

[ ... ]

Quoted 27 lines
arch/x86/kvm/svm/avic.c +15 -1
@@ -1018,7 +1038,7 @@ enum avic_vcpu_action {
 };
 
 static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid,
-					    enum avic_vcpu_action action)
+					    int cpu, enum avic_vcpu_action action)
 {
 	bool wakeup_intr = (action & AVIC_START_BLOCKING);
 	bool is_running = apicid >= 0;
@@ -1027,6 +1047,36 @@ static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid,
 
 	lockdep_assert_held(&svm->ir_list_lock);
 
+	if (!amd_iommu_gappi)
+		goto skip_wakeup_handler;
+
+	if (action & AVIC_TOGGLE_ON_OFF) {
+		/*
+		 * When AVIC is disabled, IOMMU switches to legacy IRTE and
+		 * does not generate wakeup notification. Hence there is no
+		 * need to add vCPU on wakeup list. This reduces the number of
+		 * vCPUs on the list, effectively reducing interrupt handler
+		 * time and avoiding spurious wakeup to the vCPU.
+		 */
+		if (is_running)
+			/*
+			 * Store the CPU so that next call with

arch/x86/kvm/svm/avic.c static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid, · line 1073

Quoted 10 lines
arch/x86/kvm/svm/avic.c +10 -0
+			 * AVIC_STOP_RUNNING can add the vCPU on to
+			 * appropriate wakeup list.
+			 */
+			svm->gappi_cpu = cpu;
+		goto skip_wakeup_handler;
+	}
+
+	if (is_running) {
+		kvm_pi_disable_wakeup_handler(vcpu, svm->gappi_cpu);
+		svm->gappi_cpu = cpu;

[Severity: Medium]
Does calling kvm_pi_disable_wakeup_handler() while holding svm->ir_list_lock introduce a lockdep circular locking dependency with the scheduler runqueue lock?

CPU0 (<hardirq>)        CPU1                     CPU2
----                    ----                     ----
lock(wakeup_vcpus_on_cpu_lock)
                        lock(&rq->__lock)
                        lock(&svm->ir_list_lock)
                                                 lock(&svm->ir_list_lock)
                                                 lock(wakeup_vcpus_on_cpu_lock)

lock(&rq->__lock)

*** DEADLOCK ***

CPU0: kvm_pi_wakeup_handler() -> kvm_vcpu_wake_up() -> try_to_wake_up()
CPU1: schedule() -> kvm_sched_out() -> ... -> __avic_vcpu_put()
CPU2: __avic_vcpu_load() -> ... -> kvm_pi_disable_wakeup_handler()

kvm_pi_enable_wakeup_handler() uses PI_LOCK_SCHED_OUT to avoid a cycle when called under rq->__lock during kvm_sched_out(), while kvm_pi_disable_wakeup_handler() acquires wakeup_vcpus_on_cpu_lock at subclass 0 before acquiring PI_LOCK_SCHED_OUT. Because __avic_vcpu_put() acquires svm->ir_list_lock under rq->__lock when a running vCPU is preempted, calling kvm_pi_disable_wakeup_handler() under svm->ir_list_lock in __avic_vcpu_load() bridges rq->__lock to wakeup_vcpus_on_cpu_lock at subclass 0 and triggers a lockdep splat on CONFIG_PROVE_LOCKING kernels.

arch/x86/kvm/svm/avic.c static void avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int apicid, · line 1077

Quoted 4 lines
arch/x86/kvm/svm/avic.c +4 -0
+	} else {
+		apicid = cpu_physical_id(svm->gappi_cpu);
+		kvm_pi_enable_wakeup_handler(vcpu, svm->gappi_cpu);
+	}

[Severity: Medium]
Should kvm_pi_enable_wakeup_handler() only be called when wakeup_intr is true (action & AVIC_START_BLOCKING)?

When a vCPU is preempted or exits to userspace without blocking, avic_vcpu_put() calls __avic_vcpu_put(vcpu, AVIC_STOP_RUNNING), where is_running is false and wakeup_intr is false. Enqueuing the vCPU here puts non-blocking vCPUs onto per_cpu(wakeup_vcpus_on_cpu, svm->gappi_cpu) on every vcpu_put(), causing kvm_pi_wakeup_handler() to scan their vAPIC IRR bitmaps on every wakeup interrupt and requiring every subsequent __avic_vcpu_load() to take wakeup_vcpus_on_cpu_lock to remove them.

[ ... ]

Quoted 15 lines
arch/x86/kvm/svm/avic.c +7 -0
@@ -1308,6 +1358,13 @@ static bool __init avic_want_avic_enabled(void)
 	return true;
 }
 
+bool avic_vcpu_irq_pending(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_svm *svm = to_svm(vcpu);
+
+	return kvm_lapic_find_highest_irr(&svm->vcpu) >= 0;
+}
+
 /*
  * Note:
  * - The module param avic enable both xAPIC and x2APIC mode.

arch/x86/kvm/svm/avic.c bool __init avic_hardware_setup(void) · line 1415

Quoted 10 lines
arch/x86/kvm/svm/avic.c +4 -2
@@ -1351,6 +1408,8 @@ bool __init avic_hardware_setup(void)
 
 void avic_hardware_unsetup(void)
 {
-	if (avic)
-		amd_iommu_register_ga_log_notifier(NULL);
+	if (!avic)
+		return;
+
+	amd_iommu_register_ga_log_notifier(NULL);
 }

[Severity: Low]
This isn't a bug, but the change in avic_hardware_unsetup() looks like leftover churn from when wakeup handler unregistration lived in SVM rather than common x86 code. Along the same lines, the commit message states that SVM registers the GAPPI handler via kvm_set_posted_intr_wakeup_handler() and walks the local vCPU list, which was moved to common x86 code in earlier commit "KVM: VMX: Factor out wakeup list handling code to KVM". Also, avic_vcpu_irq_pending() can pass vcpu directly to kvm_lapic_find_highest_irr() without converting through to_svm(vcpu).

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
#8 lore raw

PATCH v7 7/7 iommu/amd: Provide kernel command line option to enable GAPPI

In reply to Sairaj Kodilkar

Enable GAPPI when the IOMMU GAPPISup extended feature bit is set and the kernel is booted with the amd_iommu=gappi kernel parameter.

Document the option in Documentation/admin-guide/kernel-parameters.txt and mechanism in Documentation/arch/x86/iommu.rst.

---
 .../admin-guide/kernel-parameters.txt         |  3 +-
 Documentation/arch/x86/iommu.rst              | 22 +++++++++++++
 drivers/iommu/amd/amd_iommu_types.h           |  2 ++
 drivers/iommu/amd/init.c                      | 33 ++++++++++++++++---
 4 files changed, 55 insertions(+), 5 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +2 -1
index 0b3c140babd9..54c01621fc92 100644
@@ -456,7 +456,8 @@ Kernel parameters
 				          to 4 KiB.
 			v2_pgsizes_only - Limit page-sizes used for v1 page-tables
 				          to 4KiB/2Mib/1GiB.
-
+			gappi           - Use GAPPI instead of GALOG for vCPU
+				          wakeup notification.
 
 	amd_iommu_dump=	[HW,X86-64]
 			Enable AMD IOMMU driver option to dump the ACPI table
Documentation/arch/x86/iommu.rst +22 -0
index 79c33560299b..6bac30f5abbd 100644
@@ -150,3 +150,25 @@ AMD Fault reporting
 
 	AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007 address=0xffffc02000 flags=0x0000]
 	AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0 domain=0x0007 address=0xffffc02000 flags=0x0000]
+
+
+AMD vCPU wakeup Mechanism
+--------------------------
+
+GALOG
+^^^^^
+
+GALOG is a legacy notification path to wakeup the vCPU waiting on an interrupt.
+When IOMMU receives an interrupt destined to vCPU which is scheduled out, it
+updates the vAPIC backing page IRR and appends vCPU tags to a shared GA log
+buffer and raises a single GA log interrupt. IOMMU driver scans this buffer to
+decide which vCPU to wakeup.
+
+GAPPI
+^^^^^
+
+GAPPI is an alternative to GALOG.  With GAPPI enabled, the IOMMU still updates
+the guest vAPIC backing page IRR, but may deliver a physical APIC interrupt
+directly to IRTE[Destination], using IRTE[GATag][7:0] as the vector. This
+distributes host wakeup notifications across CPUs instead of centralizing them
+in a log buffer. This can be enabled by adding option amd_iommu=gappi
drivers/iommu/amd/amd_iommu_types.h +2 -0
index 4600674baf84..47c7e7d28df4 100644
@@ -79,6 +79,7 @@
 #define FEATURE_X2APIC		BIT_ULL(2)
 #define FEATURE_NX		BIT_ULL(3)
 #define FEATURE_GT		BIT_ULL(4)
+#define FEATURE_GAPPI		BIT_ULL(5)
 #define FEATURE_IA		BIT_ULL(6)
 #define FEATURE_GA		BIT_ULL(7)
 #define FEATURE_HE		BIT_ULL(8)
@@ -181,6 +182,7 @@
 #define CONTROL_EPH_EN		45
 #define CONTROL_XT_EN		50
 #define CONTROL_INTCAPXT_EN	51
+#define CONTROL_GAPPI_EN	55
 #define CONTROL_GCR3TRPMODE	58
 #define CONTROL_IRTCACHEDIS	59
 #define CONTROL_SNPAVIC_EN	61
drivers/iommu/amd/init.c +29 -4
index 054ef7262f3c..6a248327047a 100644
@@ -486,6 +486,9 @@ static void iommu_disable(struct amd_iommu *iommu)
 	iommu_feature_disable(iommu, CONTROL_GALOG_EN);
 	iommu_feature_disable(iommu, CONTROL_GAINT_EN);
 
+	/* Disable IOMMU GAPPI */
+	iommu_feature_disable(iommu, CONTROL_GAPPI_EN);
+
 	/* Disable IOMMU PPR logging */
 	iommu_feature_disable(iommu, CONTROL_PPRLOG_EN);
 	iommu_feature_disable(iommu, CONTROL_PPRINT_EN);
@@ -2376,7 +2379,7 @@ static void print_iommu_info(void)
 {
 	int i;
 	static const char * const feat_str[] = {
-		"PreF", "PPR", "X2APIC", "NX", "GT", "[5]",
+		"PreF", "PPR", "X2APIC", "NX", "GT", "GAPPI",
 		"IA", "GA", "HE", "PC"
 	};
 
@@ -3102,6 +3105,13 @@ static void enable_iommus_vapic(void)
 	struct amd_iommu *iommu;
 
 	for_each_iommu(iommu) {
+		/*
+		 * Disable GAPPI, do not check amd_iommu_gappi as it may be
+		 * false in new kexec kernel even though previous kernel has
+		 * enabled it.
+		 */
+		iommu_feature_disable(iommu, CONTROL_GAPPI_EN);
+
 		/*
 		 * Disable GALog if already running. It could have been enabled
 		 * in the previous boot before kdump.
@@ -3131,18 +3141,27 @@ static void enable_iommus_vapic(void)
 	if (!AMD_IOMMU_GUEST_IR_VAPIC(amd_iommu_guest_ir))
 		return;
 
+	if (amd_iommu_gappi &&
+	    !(check_feature(FEATURE_GAPPI) &&
+	      AMD_IOMMU_GUEST_IR_VAPIC(amd_iommu_guest_ir))) {
+		pr_warn("GAPPI is not supported.\n");
+		amd_iommu_gappi = false;
+	}
+
 	/* Enabling GAM and SNPAVIC support */
 	for_each_iommu(iommu) {
-		if (!iommu_ga_log_enable(iommu))
+		if (amd_iommu_gappi)
+			iommu_feature_enable(iommu, CONTROL_GAPPI_EN);
+		else if (!iommu_ga_log_enable(iommu))
 			return;
-
 		iommu_feature_enable(iommu, CONTROL_GAM_EN);
 		if (amd_iommu_snp_en)
 			iommu_feature_enable(iommu, CONTROL_SNPAVIC_EN);
 	}
 
 	amd_iommu_irq_ops.capability |= (1 << IRQ_POSTING_CAP);
-	pr_info("Virtual APIC enabled\n");
+	pr_info("Virtual APIC enabled with %s\n",
+		amd_iommu_gappi ? "GAPPI" : "GALOG");
 #endif
 }
 
@@ -3893,6 +3912,12 @@ static int __init parse_amd_iommu_options(char *str)
 		} else if (strncmp(str, "v2_pgsizes_only", 15) == 0) {
 			pr_info("Restricting V1 page-sizes to 4KiB/2MiB/1GiB");
 			amd_iommu_pgsize_bitmap = AMD_IOMMU_PGSIZES_V2;
+		} else if (strncmp(str, "gappi", 5) == 0) {
+#ifdef CONFIG_IRQ_REMAP
+			amd_iommu_gappi = true;
+#else
+			pr_warn("amd_iommu=gappi ignored, IRQ remapping is not enabled\n");
+#endif
 		} else {
 			pr_notice("Unknown option - '%s'\n", str);
 		}
Signature · 1 line
2.34.1
#9 lore raw

Re: [PATCH v7 7/7] iommu/amd: Provide kernel command line option to enable GAPPI

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] iommu/amd: enable_iommus_vapic() leaves amd_iommu_gappi true on early return --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] iommu/amd: enable_iommus_vapic() leaves amd_iommu_gappi true on early return
--

commit c24e6331ec8e811c471b1e85ddce5dfb41ecad29
Author: Sairaj Kodilkar <[email protected]>

iommu/amd: Provide kernel command line option to enable GAPPI

This commit adds the amd_iommu=gappi kernel command-line option to enable GAPPI when the IOMMU GAPPISup extended feature bit is supported, and documents the option and mechanism.

Quoted 4 lines
drivers/iommu/amd/init.c +0 -0
index 054ef7262f3cc..6a248327047ac 100644

[ ... ]

drivers/iommu/amd/init.c static void enable_iommus_vapic(void) · line 3149

Quoted 10 lines
drivers/iommu/amd/init.c +6 -0
@@ -3131,18 +3141,27 @@ static void enable_iommus_vapic(void)
 	if (!AMD_IOMMU_GUEST_IR_VAPIC(amd_iommu_guest_ir))
 		return;
 
+	if (amd_iommu_gappi &&
+	    !(check_feature(FEATURE_GAPPI) &&
+	      AMD_IOMMU_GUEST_IR_VAPIC(amd_iommu_guest_ir))) {
+		pr_warn("GAPPI is not supported.\n");
+		amd_iommu_gappi = false;
+	}

[Severity: Medium]
Does the early return when !AMD_IOMMU_GUEST_IR_VAPIC(amd_iommu_guest_ir) make the AMD_IOMMU_GUEST_IR_VAPIC() check right below it unreachable when false and leave amd_iommu_gappi set to true?

parse_amd_iommu_options() sets amd_iommu_gappi = true before Virtual APIC support is validated. If the kernel is booted with amd_iommu=gappi and IOMMU Virtual APIC is disabled or downgraded (for example with amd_iommu_intr=legacy or when check_vapic_support() falls back to legacy GA mode), enable_iommus_vapic() returns before clearing amd_iommu_gappi (and likewise if the IOMMU is disabled or fails to initialize).

When CPU AVIC is active in KVM SVM, avic_update_iommu_vcpu_affinity() and avic_vcpu_free() check amd_iommu_gappi before checking whether the vCPU has any IOMMU IRTEs:

arch/x86/kvm/svm/avic.c:avic_update_iommu_vcpu_affinity() {
	...
	if (!amd_iommu_gappi)
		goto skip_wakeup_handler;
	...
	if (is_running) {
		kvm_pi_disable_wakeup_handler(vcpu, svm->gappi_cpu);
		svm->gappi_cpu = cpu;
	} else {
		apicid = cpu_physical_id(svm->gappi_cpu);
		kvm_pi_enable_wakeup_handler(vcpu, svm->gappi_cpu);
	}

skip_wakeup_handler:

	if (list_empty(&svm->ir_list))
		return;
	...
}

Leaving amd_iommu_gappi set to true when CONTROL_GAPPI_EN is not enabled causes KVM to register and unregister per-CPU wakeup handlers on every vCPU load, put, block, and free even though IOMMU Virtual APIC is not in use.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7