Coalesced Interrupt Delivery with posted MSI
Subsystems (8)
v3 cover Coalesced Interrupt Delivery with posted MSI
Hi Thomas and all,
This patch set is aimed to improve IRQ throughput on Intel Xeon by making use of posted interrupts.
There is a session at LPC2023 IOMMU/VFIO/PCI MC where I have presented this topic.
https://lpc.events/event/17/sessions/172/#20231115
Background
==========
On modern x86 server SoCs, interrupt remapping (IR) is required and turned on by default to support X2APIC. Two interrupt remapping modes can be supported by IOMMU/VT-d:
- Remappable (host)
- Posted (guest only so far)
With remappable mode, the device MSI to CPU process is a HW flow without system software touch points, it roughly goes as follows:
1. Devices issue interrupt requests with writes to 0xFEEx_xxxx
2. The system agent accepts and remaps/translates the IRQ
3. Upon receiving the translation response, the system agent notifies the
destination CPU with the translated MSI
4. CPU's local APIC accepts interrupts into its IRR/ISR registers
5. Interrupt delivered through IDT (MSI vector)
The above process can be inefficient under high IRQ rates. The notifications in step #3 are often unnecessary when the destination CPU is already overwhelmed with handling bursts of IRQs. On some architectures, such as Intel Xeon, step #3 is also expensive and requires strong ordering w.r.t DMA. As a result, slower IRQ rates can become a limiting factor for DMA I/O performance.
For example, on Intel Xeon Sapphire Rapids SoC, as more NVMe disks are attached to the same socket, FIO (libaio engine) 4K block random read performance per-disk drops quickly.
# of disks 2 4 8 ------------------------------------- IOPS(million) 1.991 1.136 0.834 (NVMe Gen 5 Samsung PM174x)
With posted mode enabled in interrupt remapping, the interrupt flow is divided into two parts: posting (storing pending IRQ vector information in memory) and CPU notification.
The above remappable IRQ flow becomes the following (1 and 2 unchanged):
3. Notifies the destination CPU with a notification vector
- IOMMU suppresses CPU notification
- IOMMU atomic swap/store IRQ status to memory-resident posted interrupt descriptor (PID)
4. CPU's local APIC accepts the notification interrupt into its IRR/ISR
registers
5. Interrupt delivered through IDT (notification vector handler)
System SW allows new notifications by clearing outstanding notification (ON) bit in PID.
(The above flow is not in Linux today since we only use posted mode for VM)
Note that the system software can now suppress CPU notifications at runtime as needed. This allows the system software to coalesce the expensive CPU notifications and in turn, improve IRQ throughput and DMA performance.
Consider the following scenario when MSIs arrive at a CPU in high-frequency bursts:
Time ----------------------------------------------------------------------->
^ ^ ^ ^ ^ ^ ^ ^ ^ MSIs A B C D E F G H I
RI N N' N' N N' N' N' N N
PI N N N N
RI: remappable interrupt; PI: posted interrupt;
N: interrupt notification, N': superfluous interrupt notification
With remappable interrupt (row titled RI), every MSI generates a notification event to the CPU.
With posted interrupts enabled in this patch set (row titled PI), CPU notifications are coalesced during IRQ bursts. N's are eliminated in the flow above. We refer to this mechanism Coalesced Interrupt Delivery (CID).
Post interrupts have existed for a long time, they have been used for virtualization where MSIs from directly assigned devices can be delivered to the guest kernel without VMM intervention. On x86 Intel platforms, posted interrupts can be used on the host as well. Only host physical address of Posted interrupt descriptor (PID) is used.
This patch set enables a new usage of posted interrupts on existing (and new hardware) for host kernel device MSIs. It is referred to as Posted MSIs throughout this patch set.
Performance (with this patch set):
==================================
Test #1. NVMe FIO
FIO libaio (million IOPS/sec/disk) Gen 5 NVMe Samsung PM174x disks on a single socket, Intel Xeon Sapphire Rapids. Random read with 4k block size. NVMe IRQ affinity is managed by the kernel with one vector per CPU.
#disks Before After %Gain
---------------------------------------------
8 0.834 1.943 132%
4 1.136 2.023 78%
Other observations:
- Increased block sizes shows diminishing benefits, e.g. with 4 NVME disks on one x16 PCIe slot, the combined IOPS looks like:
Block Size Baseline PostedMSI
-------------------------------------
4K 6475 8778
8K 5727 5896
16k 2864 2900
32k 1546 1520
128k 397 398- Submission/Completion latency (usec) also improved at 4K block size only FIO report SLAT
---------------------------------------
Block Size Baseline postedMSI
4k 2177 2282
8k 4416 3967
16k 2950 3053
32k 3453 3505
128k 5911 5801
FIO report CLAT
---------------------------------------
Block Size Baseline postedMSI
4k 313 230
8k 352 343
16k 711 702
32k 1320 1343
128k 5146 5137
Test #2. Intel Data Streaming Accelerator
Two dedicated workqueues from two PCI root complex integrated endpoint (RCIEP) devices, pin IRQ affinity of the two interrupts to a single CPU.
Before After %Gain -------------------------------------
DSA memfill (mil IRQs/sec) 5.157 8.987 74%
DMA throughput has similar improvements.
At lower IRQ rate (< 1 million/second), no performance benefits nor regression observed so far.
No harm tests also performed to ensure no performance regression on workloads that do not have high interrupt rate. These tests include:
- kernel compile time
- file copy
- FIO NVME random writes
Implementation choices:
======================
- Transparent to the device drivers
- System-wide option instead of per-device or per-IRQ opt-in, i.e. once enabled all device MSIs are posted. The benefit is that we only need to change IR irq_chip and domain layer. No change to PCI MSI.
Exceptions are: IOAPIC, HPET, and VT-d's own IRQs
- Limit the number of polling/demuxing loops per CPU notification event
- Only change Intel-IR in IRQ domain hierarchy VECTOR->INTEL-IR->PCI-MSI,
- X86 Intel only so far, can be extended to other architectures with posted interrupt support (ARM and AMD), RFC.
- Bare metal only, no posted interrupt capable virtual IOMMU.
Changes and implications (moving from remappable to posted mode)
===============================
1. All MSI vectors are multiplexed into a single notification vector for each CPU MSI vectors are then de-multiplexed by SW, no IDT delivery for MSIs
2. Losing the following features compared to the remappable mode (AFAIK, none of the below matters for device MSIs)
- Control of delivery mode, e.g. NMI for MSIs
- No logical destinations, posted interrupt destination is x2APIC physical APIC ID
- No per vector stack, since all MSI vectors are multiplexed into one
Runtime changes
===============
The IRQ runtime behavior has changed with this patch, here is a pseudo trace comparison for 3 MSIs of different vectors arriving in a burst on the same CPU. A system vector interrupt (e.g. timer) arrives randomly.
BEFORE:
interrupt(MSI)
irq_enter()
handler() /* EOI */
irq_exit()
process_softirq()interrupt(timer)
interrupt(MSI)
irq_enter()
handler() /* EOI */
irq_exit()
process_softirq()interrupt(MSI)
irq_enter()
handler() /* EOI */
irq_exit()
process_softirq()AFTER:
interrupt /* Posted MSI notification vector */
irq_enter()
atomic_xchg(PIR)
handler()
handler()
handler()
pi_clear_on()
apic_eoi()
irq_exit()interrupt(timer)
process_softirq()
With posted MSI (as pointed out by Thomas Gleixner), both high-priority interrupts (system interrupt vectors) and softIRQs are blocked during MSI vector demux loop. Some can be timing sensitive.
Here are the options I have attempted or still working on:
1. Use self-IPI to invoke MSI vector handler but that took away the majority of the performance benefits.
2. Limit the # of demuxing loops, this is implemented in this patch. Note that today, we already allow one low priority MSI to block system interrupts. System vector can preempt MSI vectors without waiting for EOI but we have IRQ disabled in the ISR.
Performance data (on DSA with MEMFILL) also shows that coalescing more than 3 loops yields diminishing benefits. Therefore, the max loops for coalescing is set to 3 in this patch.
MaxLoop IRQ/sec bandwidth Mbps
-------------------------------------------------------------------------
2 6157107 25219 3 6226611 25504 4 6557081 26857 5 6629683 27155 6 6662425 27289
3. limit the time that system interrupts can be blocked (WIP).
In addition, posted MSI uses atomic xchg from both CPU and IOMMU. Compared to remappable mode, there may be additional cache line ownership contention over PID. However, we have not observed performance regression at lower IRQ rates. At high interrupt rate, posted mode always wins.
Testing:
========
The following tests have been performed and continue to be evaluated.
- IRQ affinity change, migration
- CPU offlining
- Multi vector coalescing
- Low IRQ rate, general no-harm test
- VM device assignment via VFIO
- General no harm test, performance regressions have not been observed for low IRQ rate workload.
With the patch, a new entry in /proc/interrupts is added. cat /proc/interrupts | grep PMN
PMN: 13868907 Posted MSI notification event
No change to the device MSI accounting.
A new INTEL-IR-POST irq_chip is visible at IRQ debugfs, e.g.
domain: IR-PCI-MSIX-0000:6f:01.0-12
hwirq: 0x8
chip: IR-PCI-MSIX-0000:6f:01.0
flags: 0x430
IRQCHIP_SKIP_SET_WAKE IRQCHIP_ONESHOT_SAFE
parent:
domain: INTEL-IR-12-13
hwirq: 0x90000
chip: INTEL-IR-POST /* For posted MSIs */
flags: 0x0
parent:
domain: VECTOR
hwirq: 0x65
chip: APICAcknowledgment
==============
- Rajesh Sankaran and Ashok Raj for the original idea
- Thomas Gleixner for reviewing and guiding the upstream direction of PoC patches. Help correct my many misunderstandings of the IRQ subsystem.
- Jie J Yan(Jeff), Sebastien Lemarie, and Dan Liang for performance evaluation with NVMe and network workload
- Bernice Zhang and Scott Morris for functional validation
- Michael Prinke helped me understand how VT-d HW works
- Sanjay Kumar for providing the DSA IRQ test suite
Changelogs (details in each patch):
V3:
- Add Intel flexible return and event delivery (FRED) support
- Fix potential double EOI bug
- Fix a bug in removing posted interrupt descriptor bitfields
V2:
- Code change logs are in individual patches.
- Use "Originally-by" and "Suggested-by" tags to clarify credits/responsibilities.
- More performance evaluation done on FIO
4K rand read test. Four Samsung PM174x NVMe drives on a single x16 PCIe gen5 lane. Fixed CPU frequency at 2.7GHz (p1, highest non-turbo).
IOPS* CPU% sys% user% Ints/sec IOPS/CPU LAT**
AIO (before) 6231 55.5 39.7 15.8 5714721 112.2702703 328
AIO (after) 8936 71.5 51.5 20 7397543 124.979021 229
IOURING(before) 6880 43.7 30.3 13.4 6512402 157.4370709 149
IOURING(after) 8688 58.3 41.3 17 7625158 149.0222985 118
IOURING POLLEDQ 13100 100 85.1 14.9 8000 131 156
* x1000 4 drives combined
** 95% usec.
This patchset improves IOPS, IRQ throughput, and reduces latency for non-polled queues.
V1 (since RFC)
- Removed mentioning of wishful features, IRQ preemption, separate and full MSI vector space
- Refined MSI handler de-multiplexing loop based on suggestions from Peter and Thomas. Reduced xchg() usage and code duplication
- Assign the new posted IR irq_chip only to device MSI/x, avoid changing IO-APIC code
- Extract and use common code for preventing lost interrupt during affinity change
- Added more test results to the cover letter
Thanks,
Jacob
Jacob Pan (12):
KVM: VMX: Move posted interrupt descriptor out of vmx code x86/irq: Unionize PID.PIR for 64bit access w/o casting x86/irq: Remove bitfields in posted interrupt descriptor x86/irq: Add a Kconfig option for posted MSI
x86/irq: Reserve a per CPU IDT vector for posted MSIs x86/irq: Set up per host CPU posted interrupt descriptors x86/irq: Factor out calling ISR from common_interrupt x86/irq: Install posted MSI notification handler x86/irq: Factor out common code for checking pending interrupts x86/irq: Extend checks for pending vectors to posted interrupts iommu/vt-d: Make posted MSI an opt-in cmdline option iommu/vt-d: Enable posted mode for device MSIs
.../admin-guide/kernel-parameters.txt | 1 + arch/x86/Kconfig | 11 ++ arch/x86/entry/entry_fred.c | 2 + arch/x86/include/asm/apic.h | 12 ++ arch/x86/include/asm/hardirq.h | 6 + arch/x86/include/asm/idtentry.h | 6 + arch/x86/include/asm/irq_remapping.h | 11 ++ arch/x86/include/asm/irq_vectors.h | 8 +- arch/x86/include/asm/posted_intr.h | 118 ++++++++++++ arch/x86/kernel/apic/vector.c | 5 +- arch/x86/kernel/cpu/common.c | 3 + arch/x86/kernel/idt.c | 3 + arch/x86/kernel/irq.c | 172 ++++++++++++++++-- arch/x86/kvm/vmx/posted_intr.c | 4 +- arch/x86/kvm/vmx/posted_intr.h | 93 +--------- arch/x86/kvm/vmx/vmx.c | 3 +- arch/x86/kvm/vmx/vmx.h | 2 +- drivers/iommu/intel/irq_remapping.c | 113 +++++++++++- drivers/iommu/irq_remapping.c | 9 +- 19 files changed, 463 insertions(+), 119 deletions(-) create mode 100644 arch/x86/include/asm/posted_intr.h
Signature · 1 line
2.25.1
PATCH v3 1/12 KVM: VMX: Move posted interrupt descriptor out of vmx code
To prepare native usage of posted interrupt, move PID declaration out of VMX code such that they can be shared.
--- arch/x86/include/asm/posted_intr.h | 88 ++++++++++++++++++++++++++++ arch/x86/kvm/vmx/posted_intr.h | 93 +----------------------------- arch/x86/kvm/vmx/vmx.c | 1 + arch/x86/kvm/vmx/vmx.h | 2 +- 4 files changed, 91 insertions(+), 93 deletions(-) create mode 100644 arch/x86/include/asm/posted_intr.h
arch/x86/include/asm/posted_intr.h new file +88 -0
@@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _X86_POSTED_INTR_H +#define _X86_POSTED_INTR_H + +#define POSTED_INTR_ON 0 +#define POSTED_INTR_SN 1 + +#define PID_TABLE_ENTRY_VALID 1 + +/* Posted-Interrupt Descriptor */ +struct pi_desc { +u32 pir[8 ];/* Posted interrupt requested */ +union { +struct { +/* bit 256 - Outstanding Notification */ +u16 on :1 , +/* bit 257 - Suppress Notification */ + sn :1 , +/* bit 271:258 - Reserved */ + rsvd_1 :14 ; +/* bit 279:272 - Notification Vector */ +u8 nv; +/* bit 287:280 - Reserved */ +u8 rsvd_2; +/* bit 319:288 - Notification Destination */ +u32 ndst; + }; +u64 control; + }; +u32 rsvd[6 ]; +}__aligned(64) ; + +static inline bool pi_test_and_set_on (struct pi_desc *pi_desc) +{ +return test_and_set_bit (POSTED_INTR_ON , (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_and_clear_on (struct pi_desc *pi_desc) +{ +return test_and_clear_bit (POSTED_INTR_ON , (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_and_clear_sn (struct pi_desc *pi_desc) +{ +return test_and_clear_bit (POSTED_INTR_SN , (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_and_set_pir (int vector,struct pi_desc *pi_desc) +{ +return test_and_set_bit (vector, (unsigned long *)pi_desc->pir); +} + +static inline bool pi_is_pir_empty (struct pi_desc *pi_desc) +{ +return bitmap_empty ((unsigned long *)pi_desc->pir,NR_VECTORS ); +} + +static inline void pi_set_sn (struct pi_desc *pi_desc) +{ +set_bit (POSTED_INTR_SN , (unsigned long *)&pi_desc->control); +} + +static inline void pi_set_on (struct pi_desc *pi_desc) +{ +set_bit (POSTED_INTR_ON , (unsigned long *)&pi_desc->control); +} + +static inline void pi_clear_on (struct pi_desc *pi_desc) +{ +clear_bit (POSTED_INTR_ON , (unsigned long *)&pi_desc->control); +} + +static inline void pi_clear_sn (struct pi_desc *pi_desc) +{ +clear_bit (POSTED_INTR_SN , (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_on (struct pi_desc *pi_desc) +{ +return test_bit (POSTED_INTR_ON , (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_sn (struct pi_desc *pi_desc) +{ +return test_bit (POSTED_INTR_SN , (unsigned long *)&pi_desc->control); +} + +#endif /* _X86_POSTED_INTR_H */
arch/x86/kvm/vmx/posted_intr.h +1 -92
99 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.c +1 -0
@@ -70,6 +70,7 @@#include "x86.h" #include "smm.h" #include "vmx_onhyperv.h" +#include "posted_intr.h" MODULE_AUTHOR ("Qumranet" );MODULE_LICENSE ("GPL" );
arch/x86/kvm/vmx/vmx.h +1 -1
@@ -7,10 +7,10 @@#include <asm/kvm.h> #include <asm/intel_pt.h> #include <asm/perf_event.h> +#include <asm/posted_intr.h> #include "capabilities.h" #include "../kvm_cache_regs.h" -#include "posted_intr.h" #include "vmcs.h" #include "vmx_ops.h" #include "../cpuid.h"
Signature · 1 line
2.25.1
[tip: x86/irq] KVM: VMX: Move posted interrupt descriptor out of VMX code
The following commit has been merged into the x86/irq branch of tip:
The following commit has been merged into the x86/irq branch of tip:
Commit-ID: 699f67512f04cbaee965fad872702c06eaf440f6 Gitweb: https://git.kernel.org/tip/699f67512f04cbaee965fad872702c06eaf440f6 Author: Jacob Pan <[email protected]> AuthorDate: Tue, 23 Apr 2024 10:41:03 -07:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 30 Apr 2024 00:54:42 +02:00
KVM: VMX: Move posted interrupt descriptor out of VMX code
To prepare native usage of posted interrupts, move the PID declarations out of VMX code such that they can be shared.
--- arch/x86/include/asm/posted_intr.h | 88 +++++++++++++++++++++++++++- arch/x86/kvm/vmx/posted_intr.h | 93 +----------------------------- arch/x86/kvm/vmx/vmx.c | 1 +- arch/x86/kvm/vmx/vmx.h | 2 +- 4 files changed, 91 insertions(+), 93 deletions(-) create mode 100644 arch/x86/include/asm/posted_intr.h
arch/x86/include/asm/posted_intr.h new file +88 -0
@@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _X86_POSTED_INTR_H +#define _X86_POSTED_INTR_H + +#define POSTED_INTR_ON 0 +#define POSTED_INTR_SN 1 + +#define PID_TABLE_ENTRY_VALID 1 + +/* Posted-Interrupt Descriptor */ +struct pi_desc { +u32 pir[8 ];/* Posted interrupt requested */ +union { +struct { +/* bit 256 - Outstanding Notification */ +u16 on :1 , +/* bit 257 - Suppress Notification */ + sn :1 , +/* bit 271:258 - Reserved */ + rsvd_1 :14 ; +/* bit 279:272 - Notification Vector */ +u8 nv; +/* bit 287:280 - Reserved */ +u8 rsvd_2; +/* bit 319:288 - Notification Destination */ +u32 ndst; + }; +u64 control; + }; +u32 rsvd[6 ]; +}__aligned(64) ; + +static inline bool pi_test_and_set_on (struct pi_desc *pi_desc) +{ +return test_and_set_bit (POSTED_INTR_ON , (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_and_clear_on (struct pi_desc *pi_desc) +{ +return test_and_clear_bit (POSTED_INTR_ON , (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_and_clear_sn (struct pi_desc *pi_desc) +{ +return test_and_clear_bit (POSTED_INTR_SN , (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_and_set_pir (int vector,struct pi_desc *pi_desc) +{ +return test_and_set_bit (vector, (unsigned long *)pi_desc->pir); +} + +static inline bool pi_is_pir_empty (struct pi_desc *pi_desc) +{ +return bitmap_empty ((unsigned long *)pi_desc->pir,NR_VECTORS ); +} + +static inline void pi_set_sn (struct pi_desc *pi_desc) +{ +set_bit (POSTED_INTR_SN , (unsigned long *)&pi_desc->control); +} + +static inline void pi_set_on (struct pi_desc *pi_desc) +{ +set_bit (POSTED_INTR_ON , (unsigned long *)&pi_desc->control); +} + +static inline void pi_clear_on (struct pi_desc *pi_desc) +{ +clear_bit (POSTED_INTR_ON , (unsigned long *)&pi_desc->control); +} + +static inline void pi_clear_sn (struct pi_desc *pi_desc) +{ +clear_bit (POSTED_INTR_SN , (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_on (struct pi_desc *pi_desc) +{ +return test_bit (POSTED_INTR_ON , (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_sn (struct pi_desc *pi_desc) +{ +return test_bit (POSTED_INTR_SN , (unsigned long *)&pi_desc->control); +} + +#endif /* _X86_POSTED_INTR_H */
arch/x86/kvm/vmx/posted_intr.h +1 -92
99 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.c +1 -0
@@ -70,6 +70,7 @@#include "x86.h" #include "smm.h" #include "vmx_onhyperv.h" +#include "posted_intr.h" MODULE_AUTHOR ("Qumranet" );MODULE_LICENSE ("GPL" );
arch/x86/kvm/vmx/vmx.h +1 -1
@@ -7,10 +7,10 @@#include <asm/kvm.h> #include <asm/intel_pt.h> #include <asm/perf_event.h> +#include <asm/posted_intr.h> #include "capabilities.h" #include "../kvm_cache_regs.h" -#include "posted_intr.h" #include "vmcs.h" #include "vmx_ops.h" #include "../cpuid.h"
PATCH v3 2/12 x86/irq: Unionize PID.PIR for 64bit access w/o casting
In reply to Jacob PanMake PIR field into u64 such that atomic xchg64 can be used without ugly casting.
--- arch/x86/include/asm/posted_intr.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
arch/x86/include/asm/posted_intr.h +4 -1
@@ -9,7 +9,10 @@/* Posted-Interrupt Descriptor */ struct pi_desc { - u32 pir[8]; /* Posted interrupt requested */ + union { + u32 pir[8]; /* Posted interrupt requested */ + u64 pir64[4]; + }; union { struct { /* bit 256 - Outstanding Notification */
Signature · 1 line
2.25.1
[tip: x86/irq] x86/irq: Unionize PID.PIR for 64bit access w/o casting
The following commit has been merged into the x86/irq branch of tip:
The following commit has been merged into the x86/irq branch of tip:
Commit-ID: 4ec8fd037139a4d8afb2a5c7edb4a17f9449a035 Gitweb: https://git.kernel.org/tip/4ec8fd037139a4d8afb2a5c7edb4a17f9449a035 Author: Jacob Pan <[email protected]> AuthorDate: Tue, 23 Apr 2024 10:41:04 -07:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 30 Apr 2024 00:54:42 +02:00
x86/irq: Unionize PID.PIR for 64bit access w/o casting
Make the PIR field into u64 such that atomic xchg64 can be used without ugly casting.
--- arch/x86/include/asm/posted_intr.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
arch/x86/include/asm/posted_intr.h +4 -1
@@ -9,7 +9,10 @@/* Posted-Interrupt Descriptor */ struct pi_desc { - u32 pir[8]; /* Posted interrupt requested */ + union { + u32 pir[8]; /* Posted interrupt requested */ + u64 pir64[4]; + }; union { struct { /* bit 256 - Outstanding Notification */
PATCH v3 3/12 x86/irq: Remove bitfields in posted interrupt descriptor
In reply to Jacob PanMixture of bitfields and types is weird and really not intuitive, remove bitfields and use typed data exclusively. Bitfields often result in inferior machine code.
---
v3:
- Fix a bug where SN bit position was used as the mask, reported by Oliver Sang.
- Add and use non-atomic helpers to manipulate SN bit
- Use pi_test_sn() instead of open coding
v2:
- Replace bitfields, no more mix.
--- arch/x86/include/asm/posted_intr.h | 21 ++++++++++++--------- arch/x86/kvm/vmx/posted_intr.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 2 +- 3 files changed, 15 insertions(+), 12 deletions(-)
arch/x86/include/asm/posted_intr.h +12 -9
@@ -15,17 +15,9 @@ struct pi_desc { };union {struct { -/* bit 256 - Outstanding Notification */ -u16 on :1 , -/* bit 257 - Suppress Notification */ - sn :1 , -/* bit 271:258 - Reserved */ - rsvd_1 :14 ; -/* bit 279:272 - Notification Vector */ +u16 notifications;/* Suppress and outstanding bits */ u8 nv; -/* bit 287:280 - Reserved */ u8 rsvd_2; -/* bit 319:288 - Notification Destination */ u32 ndst; };u64 control;
@@ -88,4 +80,15 @@ static inline bool pi_test_sn(struct pi_desc *pi_desc) return test_bit(POSTED_INTR_SN, (unsigned long *)&pi_desc->control); } +/* Non-atomic helpers */ +static inline void __pi_set_sn (struct pi_desc *pi_desc) +{ + pi_desc->notifications |=BIT (POSTED_INTR_SN ); +} + +static inline void __pi_clear_sn (struct pi_desc *pi_desc) +{ + pi_desc->notifications &= ~BIT (POSTED_INTR_SN ); +} +#endif /* _X86_POSTED_INTR_H */
arch/x86/kvm/vmx/posted_intr.c +2 -2
@@ -107,7 +107,7 @@ void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)* handle task migration (@cpu != vcpu->cpu). */ new.ndst = dest; - new.sn =0 ; +__pi_clear_sn (&new);/* * Restore the notification vector; in the blocking case, the
@@ -157,7 +157,7 @@ static void pi_enable_wakeup_handler(struct kvm_vcpu *vcpu) &per_cpu(wakeup_vcpus_on_cpu, vcpu->cpu));raw_spin_unlock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu)); -WARN (pi_desc->sn,"PI descriptor SN field set before blocking" ); +WARN (pi_test_sn (pi_desc),"PI descriptor SN field set before blocking" ); old.control =READ_ONCE (pi_desc->control); do {
arch/x86/kvm/vmx/vmx.c +1 -1
@@ -4843,7 +4843,7 @@ static void __vmx_vcpu_reset(struct kvm_vcpu *vcpu)* or POSTED_INTR_WAKEUP_VECTOR. */ vmx->pi_desc.nv =POSTED_INTR_VECTOR ; - vmx->pi_desc.sn =1 ; +__pi_set_sn (&vmx->pi_desc); }static void vmx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event)
Signature · 1 line
2.25.1
[tip: x86/irq] x86/irq: Remove bitfields in posted interrupt descriptor
The following commit has been merged into the x86/irq branch of tip:
The following commit has been merged into the x86/irq branch of tip:
Commit-ID: 2254808b53d92c9fe7b645b2f43acc55f22cdce6 Gitweb: https://git.kernel.org/tip/2254808b53d92c9fe7b645b2f43acc55f22cdce6 Author: Jacob Pan <[email protected]> AuthorDate: Tue, 23 Apr 2024 10:41:05 -07:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 30 Apr 2024 00:54:42 +02:00
x86/irq: Remove bitfields in posted interrupt descriptor
Mixture of bitfields and types is weird and really not intuitive, remove bitfields and use typed data exclusively. Bitfields often result in inferior machine code.
--- arch/x86/include/asm/posted_intr.h | 21 ++++++++++++--------- arch/x86/kvm/vmx/posted_intr.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 2 +- 3 files changed, 15 insertions(+), 12 deletions(-)
arch/x86/include/asm/posted_intr.h +12 -9
@@ -15,17 +15,9 @@ struct pi_desc { };union {struct { -/* bit 256 - Outstanding Notification */ -u16 on :1 , -/* bit 257 - Suppress Notification */ - sn :1 , -/* bit 271:258 - Reserved */ - rsvd_1 :14 ; -/* bit 279:272 - Notification Vector */ +u16 notifications;/* Suppress and outstanding bits */ u8 nv; -/* bit 287:280 - Reserved */ u8 rsvd_2; -/* bit 319:288 - Notification Destination */ u32 ndst; };u64 control;
@@ -88,4 +80,15 @@ static inline bool pi_test_sn(struct pi_desc *pi_desc) return test_bit(POSTED_INTR_SN, (unsigned long *)&pi_desc->control); } +/* Non-atomic helpers */ +static inline void __pi_set_sn (struct pi_desc *pi_desc) +{ + pi_desc->notifications |=BIT (POSTED_INTR_SN ); +} + +static inline void __pi_clear_sn (struct pi_desc *pi_desc) +{ + pi_desc->notifications &= ~BIT (POSTED_INTR_SN ); +} +#endif /* _X86_POSTED_INTR_H */
arch/x86/kvm/vmx/posted_intr.c +2 -2
@@ -107,7 +107,7 @@ void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)* handle task migration (@cpu != vcpu->cpu). */ new.ndst = dest; - new.sn =0 ; +__pi_clear_sn (&new);/* * Restore the notification vector; in the blocking case, the
@@ -157,7 +157,7 @@ static void pi_enable_wakeup_handler(struct kvm_vcpu *vcpu) &per_cpu(wakeup_vcpus_on_cpu, vcpu->cpu));raw_spin_unlock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu)); -WARN (pi_desc->sn,"PI descriptor SN field set before blocking" ); +WARN (pi_test_sn (pi_desc),"PI descriptor SN field set before blocking" ); old.control =READ_ONCE (pi_desc->control); do {
arch/x86/kvm/vmx/vmx.c +1 -1
@@ -4845,7 +4845,7 @@ static void __vmx_vcpu_reset(struct kvm_vcpu *vcpu)* or POSTED_INTR_WAKEUP_VECTOR. */ vmx->pi_desc.nv =POSTED_INTR_VECTOR ; - vmx->pi_desc.sn =1 ; +__pi_set_sn (&vmx->pi_desc); }static void vmx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event)
Re: [PATCH v3 03/12] x86/irq: Remove bitfields in posted interrupt descriptor
In reply to Jacob Panhi, Jacob Pan,
Quoted 9 lines · Jacob Pan
Mixture of bitfields and types is weird and really not intuitive, remove bitfields and use typed data exclusively. Bitfields often result in inferior machine code.
Quoted 4 lines
---
v3:
- Fix a bug where SN bit position was used as the mask, reported by Oliver Sang.
we tested this verion, and confirmed the issue gone
below just for reference.
for previous version, we noticed a "WARNING:at_arch/x86/kvm/vmx/posted_intr.c:#pi_enable_wakeup_handler[kvm_intel]" in testcase: kernel-selftests
Call Trace like below:
[ 399.225452][ T8098] ------------[ cut here ]------------ [ 399.232475][ T8098] PI descriptor SN field set before blocking [ 399.232514][ T8098] WARNING: CPU: 184 PID: 8098 at arch/x86/kvm/vmx/posted_intr.c:160 pi_enable_wakeup_handler+0x421/0x5f0 [kvm_intel] [ 399.254685][ T8098] Modules linked in: openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 intel_rapl_msr intel_rapl_common btrfs
+blake2b_generic x86_pkg_temp_thermal xor intel_powerclamp zstd_compress coretemp raid6_pq libcrc32c kvm_intel kvm nvme crct10dif_pclmul crc32_pclmul +nvme_core crc32c_intel t10_pi ghash_clmulni_intel ast sha512_ssse3 rapl intel_cstate dax_hmem crc64_rocksoft_generic drm_shmem_helper mei_me i2c_i801 +crc64_rocksoft mei drm_kms_helper i2c_smbus crc64 i2c_ismt wmi ipmi_ssif acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter joydev +binfmt_misc loop fuse drm dm_mod ip_tables
[ 399.321529][ T8098] CPU: 184 PID: 8098 Comm: xapic_ipi_test Not tainted 6.9.0-rc1-00008-g037ccaed5dc5 #1 [ 399.333325][ T8098] RIP: 0010:pi_enable_wakeup_handler+0x421/0x5f0 [kvm_intel] [ 399.342631][ T8098] Code: e8 a4 5a 7b c0 e9 b2 fc ff ff e8 5a 5c 7b c0 fb eb 93 bf f1 00 00 00 e8 dd 9c 29 c0 eb 82 48 c7 c7 e0 c6 ee c0 e8 0f 2f 37 c0 +<0f> 0b e9 e7 fe ff ff 4c 89 f7 e8 00 6a c8 c0 e9 cd fe ff ff 4c 89 [ 399.365742][ T8098] RSP: 0018:ffa000003a527780 EFLAGS: 00010086 [ 399.373626][ T8098] RAX: 0000000000000000 RBX: ff1100247c23c740 RCX: 0000000000000027 [ 399.383668][ T8098] RDX: 0000000000000027 RSI: 0000000000000004 RDI: ff11003fc3030c08 [ 399.393759][ T8098] RBP: ffa000003a527908 R08: 0000000000000001 R09: ffe21c07f8606181 [ 399.403834][ T8098] R10: ff11003fc3030c0b R11: 0000000000000000 R12: 1ff40000074a4ef4 [ 399.413901][ T8098] R13: 0000000000000000 R14: ff1100247c23e4a0 R15: 00000000000000b8 [ 399.423969][ T8098] FS: 00007f10de2006c0(0000) GS:ff11003fc3000000(0000) knlGS:0000000000000000 [ 399.435126][ T8098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 399.443651][ T8098] CR2: 0000000000000000 CR3: 000000210b958004 CR4: 0000000000f73ef0 [ 399.453753][ T8098] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 399.463835][ T8098] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ 399.473912][ T8098] PKRU: 55555554 [ 399.479005][ T8098] Call Trace: [ 399.483812][ T8098] <TASK> [ 399.488224][ T8098] ? __warn+0xcc/0x2d0 [ 399.493903][ T8098] ? pi_enable_wakeup_handler+0x421/0x5f0 [kvm_intel] [ 399.502643][ T8098] ? report_bug+0x261/0x2c0 [ 399.508825][ T8098] ? handle_bug+0x3a/0x90 [ 399.514817][ T8098] ? exc_invalid_op+0x17/0x40 [ 399.521191][ T8098] ? asm_exc_invalid_op+0x1a/0x20 [ 399.527938][ T8098] ? pi_enable_wakeup_handler+0x421/0x5f0 [kvm_intel] [ 399.536677][ T8098] ? __pfx_pi_enable_wakeup_handler+0x10/0x10 [kvm_intel] [ 399.545747][ T8098] ? __pfx_lock_repin_lock+0x10/0x10 [ 399.552740][ T8098] ? newidle_balance+0xc85/0x1300 [ 399.559408][ T8098] ? __pfx_perf_event_context_sched_out+0x10/0x10 [ 399.567652][ T8098] ? lock_acquire+0x432/0x4e0 [ 399.573959][ T8098] ? vmx_get_rflags+0x26/0x2c0 [kvm_intel] [ 399.581527][ T8098] vmx_vcpu_pi_put+0x1d3/0x230 [kvm_intel] [ 399.589119][ T8098] vmx_vcpu_put+0x12/0x20 [kvm_intel] [ 399.596205][ T8098] kvm_arch_vcpu_put+0x49e/0x7b0 [kvm] [ 399.603463][ T8098] kvm_sched_out+0xb2/0xe0 [kvm] [ 399.610100][ T8098] prepare_task_switch+0x321/0xc40 [ 399.616863][ T8098] ? lock_release+0x1bf/0x240 [ 399.623146][ T8098] __schedule+0x5a6/0x20b0 [ 399.629125][ T8098] ? __pfx___schedule+0x10/0x10 [ 399.635581][ T8098] ? __pfx_lock_acquire+0x10/0x10 [ 399.642228][ T8098] ? kvm_apic_has_interrupt+0x9c/0x160 [kvm] [ 399.650034][ T8098] ? lock_acquire+0x432/0x4e0 [ 399.656267][ T8098] ? __pfx_lock_acquire+0x10/0x10 [ 399.662905][ T8098] schedule+0xe2/0x2a0 [ 399.668478][ T8098] kvm_vcpu_block+0xd1/0x1c0 [kvm] [ 399.675308][ T8098] kvm_vcpu_halt+0xee/0x900 [kvm] [ 399.682018][ T8098] vcpu_run+0x50a/0x9d0 [kvm] [ 399.688346][ T8098] kvm_arch_vcpu_ioctl_run+0x377/0x1430 [kvm] [ 399.696254][ T8098] ? lock_release+0xe5/0x240 [ 399.702428][ T8098] kvm_vcpu_ioctl+0x34c/0xc40 [kvm] [ 399.709341][ T8098] ? __pfx_kvm_vcpu_ioctl+0x10/0x10 [kvm] [ 399.716821][ T8098] ? __lock_release+0x103/0x440 [ 399.723960][ T8098] ? __fget_files+0x1c7/0x330 [ 399.730219][ T8098] ? __pfx___lock_release+0x10/0x10 [ 399.737750][ T8098] ? __fget_files+0x1cc/0x330 [ 399.744028][ T8098] ? __fget_files+0x1c7/0x330 [ 399.750289][ T8098] ? lock_release+0xe5/0x240 [ 399.756461][ T8098] ? __fget_files+0x1cc/0x330 [ 399.762719][ T8098] __x64_sys_ioctl+0x134/0x1b0 [ 399.769077][ T8098] do_syscall_64+0x93/0x170 [ 399.775146][ T8098] ? do_user_addr_fault+0x477/0xcb0 [ 399.781989][ T8098] ? lockdep_hardirqs_on_prepare+0x279/0x3e0 [ 399.789654][ T8098] entry_SYSCALL_64_after_hwframe+0x6c/0x74 [ 399.797192][ T8098] RIP: 0033:0x7f10de88bc5b [ 399.803079][ T8098] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 +<89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00 [ 399.826105][ T8098] RSP: 002b:00007f10de1ff9c0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 399.836565][ T8098] RAX: ffffffffffffffda RBX: 000000001a5418c0 RCX: 00007f10de88bc5b [ 399.846521][ T8098] R
Quoted 87 lines
- Add and use non-atomic helpers to manipulate SN bit
- Usepi_test_sn()instead of open coding
v2:
- Replace bitfields, no more mix.--- arch/x86/include/asm/posted_intr.h | 21 ++++++++++++--------- arch/x86/kvm/vmx/posted_intr.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 2 +- 3 files changed, 15 insertions(+), 12 deletions(-)arch/x86/include/asm/posted_intr.h +12
-9@@ -15,17 +15,9 @@ struct pi_desc { };union {struct { -/* bit 256 - Outstanding Notification */ -u16 on :1 , -/* bit 257 - Suppress Notification */ - sn :1 , -/* bit 271:258 - Reserved */ - rsvd_1 :14 ; -/* bit 279:272 - Notification Vector */ +u16 notifications;/* Suppress and outstanding bits */ u8 nv; -/* bit 287:280 - Reserved */ u8 rsvd_2; -/* bit 319:288 - Notification Destination */ u32 ndst; };u64 control;@@ -88,4 +80,15 @@ static inline bool pi_test_sn(struct pi_desc *pi_desc) return test_bit(POSTED_INTR_SN, (unsigned long *)&pi_desc->control); } +/* Non-atomic helpers */ +static inline void __pi_set_sn (struct pi_desc *pi_desc) +{ + pi_desc->notifications |=BIT (POSTED_INTR_SN ); +} + +static inline void __pi_clear_sn (struct pi_desc *pi_desc) +{ + pi_desc->notifications &= ~BIT (POSTED_INTR_SN ); +} +#endif /* _X86_POSTED_INTR_H */ arch/x86/kvm/vmx/posted_intr.c +2
-2@@ -107,7 +107,7 @@ void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)* handle task migration (@cpu != vcpu->cpu). */ new.ndst = dest; - new.sn =0 ; +__pi_clear_sn (&new);/* * Restore the notification vector; in the blocking case, the @@ -157,7 +157,7 @@ static void pi_enable_wakeup_handler(struct kvm_vcpu *vcpu) &per_cpu(wakeup_vcpus_on_cpu, vcpu->cpu));raw_spin_unlock (&per_cpu (wakeup_vcpus_on_cpu_lock, vcpu->cpu)); -WARN (pi_desc->sn,"PI descriptor SN field set before blocking" ); +WARN (pi_test_sn (pi_desc),"PI descriptor SN field set before blocking" ); old.control =READ_ONCE (pi_desc->control); do {arch/x86/kvm/vmx/vmx.c +1
-1@@ -4843,7 +4843,7 @@ static void __vmx_vcpu_reset(struct kvm_vcpu *vcpu)* or POSTED_INTR_WAKEUP_VECTOR. */ vmx->pi_desc.nv =POSTED_INTR_VECTOR ; - vmx->pi_desc.sn =1 ; +__pi_set_sn (&vmx->pi_desc); }static void vmx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event)Signature · 1 line
2.25.1
PATCH v3 4/12 x86/irq: Add a Kconfig option for posted MSI
In reply to Jacob PanThis option will be used to support delivering MSIs as posted interrupts. Interrupt remapping is required.
---
v2: Remove x2apic dependency
--- arch/x86/Kconfig | 11 +++++++++++ 1 file changed, 11 insertions(+)
arch/x86/Kconfig +11 -0
@@ -464,6 +464,17 @@ config X86_X2APIC If you don't know what to do here, say N. +config X86_POSTED_MSI + bool "Enable MSI and MSI-x delivery by posted interrupts" + depends on X86_64 && IRQ_REMAP + help + This enables MSIs that are under interrupt remapping to be delivered as + posted interrupts to the host kernel. Interrupt throughput can + potentially be improved by coalescing CPU notifications during high + frequency bursts. + + If you don't know what to do here, say N. + config X86_MPPARSE bool "Enable MPS table" if ACPI default y
Signature · 1 line
2.25.1
[tip: x86/irq] x86/irq: Add a Kconfig option for posted MSI
The following commit has been merged into the x86/irq branch of tip:
The following commit has been merged into the x86/irq branch of tip:
Commit-ID: 7fec07fd217800c5174f51d8869518807e9aa144 Gitweb: https://git.kernel.org/tip/7fec07fd217800c5174f51d8869518807e9aa144 Author: Jacob Pan <[email protected]> AuthorDate: Tue, 23 Apr 2024 10:41:06 -07:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 30 Apr 2024 00:54:42 +02:00
x86/irq: Add a Kconfig option for posted MSI
This option will be used to support delivering MSIs as posted interrupts. Interrupt remapping is required.
--- arch/x86/Kconfig | 11 +++++++++++ 1 file changed, 11 insertions(+)
arch/x86/Kconfig +11 -0
@@ -464,6 +464,17 @@ config X86_X2APIC If you don't know what to do here, say N. +config X86_POSTED_MSI + bool "Enable MSI and MSI-x delivery by posted interrupts" + depends on X86_64 && IRQ_REMAP + help + This enables MSIs that are under interrupt remapping to be delivered as + posted interrupts to the host kernel. Interrupt throughput can + potentially be improved by coalescing CPU notifications during high + frequency bursts. + + If you don't know what to do here, say N. + config X86_MPPARSE bool "Enable MPS table" if ACPI default y
PATCH v3 5/12 x86/irq: Reserve a per CPU IDT vector for posted MSIs
In reply to Jacob PanWhen posted MSI is enabled, all device MSIs are multiplexed into a single notification vector. MSI handlers will be de-multiplexed at run-time by system software without IDT delivery.
---
v2:
- Add missing CONFIG_ in #ifdef
- Extend changes to x86 tools
--- arch/x86/include/asm/irq_vectors.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
arch/x86/include/asm/irq_vectors.h +7 -1
@@ -97,10 +97,16 @@#define LOCAL_TIMER_VECTOR 0xec +/* +* Posted interrupt notification vector for all device MSIs delivered to +* the host kernel. +*/ +#define POSTED_MSI_NOTIFICATION_VECTOR 0xeb +#define NR_VECTORS 256#ifdef CONFIG_X86_LOCAL_APIC -#define FIRST_SYSTEM_VECTOR LOCAL_TIMER_VECTOR +#define FIRST_SYSTEM_VECTOR POSTED_MSI_NOTIFICATION_VECTOR#else #define FIRST_SYSTEM_VECTOR NR_VECTORS#endif
Signature · 1 line
2.25.1
[tip: x86/irq] x86/irq: Reserve a per CPU IDT vector for posted MSIs
The following commit has been merged into the x86/irq branch of tip:
The following commit has been merged into the x86/irq branch of tip:
Commit-ID: f5a3562ec9dd29e61735ccf098d8ba05cf6c7c72 Gitweb: https://git.kernel.org/tip/f5a3562ec9dd29e61735ccf098d8ba05cf6c7c72 Author: Jacob Pan <[email protected]> AuthorDate: Tue, 23 Apr 2024 10:41:07 -07:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 30 Apr 2024 00:54:42 +02:00
x86/irq: Reserve a per CPU IDT vector for posted MSIs
When posted MSI is enabled, all device MSIs are multiplexed into a single notification vector. MSI handlers will be de-multiplexed at run-time by system software without IDT delivery.
--- arch/x86/include/asm/irq_vectors.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
arch/x86/include/asm/irq_vectors.h +7 -1
@@ -97,10 +97,16 @@#define LOCAL_TIMER_VECTOR 0xec +/* +* Posted interrupt notification vector for all device MSIs delivered to +* the host kernel. +*/ +#define POSTED_MSI_NOTIFICATION_VECTOR 0xeb +#define NR_VECTORS 256#ifdef CONFIG_X86_LOCAL_APIC -#define FIRST_SYSTEM_VECTOR LOCAL_TIMER_VECTOR +#define FIRST_SYSTEM_VECTOR POSTED_MSI_NOTIFICATION_VECTOR#else #define FIRST_SYSTEM_VECTOR NR_VECTORS#endif
PATCH v3 6/12 x86/irq: Set up per host CPU posted interrupt descriptors
In reply to Jacob PanTo support posted MSIs, create a posted interrupt descriptor (PID) for each host CPU. Later on, when setting up IRQ CPU affinity, IOMMU's interrupt remapping table entry (IRTE) will point to the physical address of the matching CPU's PID.
Each PID is initialized with the owner CPU's physical APICID as the destination.
---
v3: Rename per CPU PID for posted MSI from pi_desc to posted_msi_pi_desc
v2: Fix xAPIC destination ID assignment, Oliver Sang reported failture on system with x2apic optout BIOS option.
--- arch/x86/include/asm/hardirq.h | 3 +++ arch/x86/include/asm/posted_intr.h | 6 ++++++ arch/x86/kernel/cpu/common.c | 3 +++ arch/x86/kernel/irq.c | 23 +++++++++++++++++++++++ 4 files changed, 35 insertions(+)
arch/x86/include/asm/hardirq.h +3 -0
@@ -48,6 +48,9 @@ typedef struct {DECLARE_PER_CPU_SHARED_ALIGNED (irq_cpustat_t, irq_stat); +#ifdef CONFIG_X86_POSTED_MSI +DECLARE_PER_CPU_ALIGNED(struct pi_desc, posted_msi_pi_desc); +#endif #define __ARCH_IRQ_STAT#define inc_irq_stat (member) this_cpu_inc(irq_stat.member)
arch/x86/include/asm/posted_intr.h +6 -0
@@ -91,4 +91,10 @@ static inline void __pi_clear_sn(struct pi_desc *pi_desc) pi_desc->notifications &= ~BIT(POSTED_INTR_SN); } +#ifdef CONFIG_X86_POSTED_MSI +extern void intel_posted_msi_init (void ); +#else +static inline void intel_posted_msi_init (void ) {}; +#endif /* X86_POSTED_MSI */ +#endif /* _X86_POSTED_INTR_H */
arch/x86/kernel/cpu/common.c +3 -0
@@ -68,6 +68,7 @@#include <asm/traps.h> #include <asm/sev.h> #include <asm/tdx.h> +#include <asm/posted_intr.h> #include "cpu.h"
@@ -2227,6 +2228,8 @@ void cpu_init(void)barrier ();x2apic_setup (); + +intel_posted_msi_init (); }mmgrab (&init_mm);
arch/x86/kernel/irq.c +23 -0
@@ -22,6 +22,8 @@#include <asm/desc.h> #include <asm/traps.h> #include <asm/thermal.h> +#include <asm/posted_intr.h> +#include <asm/irq_remapping.h> #define CREATE_TRACE_POINTS #include <asm/trace/irq_vectors.h>
@@ -334,6 +336,27 @@ DEFINE_IDTENTRY_SYSVEC_SIMPLE(sysvec_kvm_posted_intr_nested_ipi) }#endif +#ifdef CONFIG_X86_POSTED_MSI + +/* Posted Interrupt Descriptors for coalesced MSIs to be posted */ +DEFINE_PER_CPU_ALIGNED(struct pi_desc, posted_msi_pi_desc); + +void intel_posted_msi_init (void ) +{ +u32 destination; +u32 apic_id; + +this_cpu_write (posted_msi_pi_desc.nv,POSTED_MSI_NOTIFICATION_VECTOR ); + +/* +* APIC destination ID is stored in bit 8:15 while in XAPIC mode. +* VT-d spec. CH 9.11 +*/ + apic_id =this_cpu_read (x86_cpu_to_apicid); + destination =x2apic_enabled () ? apic_id : apic_id <<8 ; +this_cpu_write (posted_msi_pi_desc.ndst, destination); +} +#endif /* X86_POSTED_MSI */ #ifdef CONFIG_HOTPLUG_CPU /* A cpu has been removed from cpu_online_mask. Reset irq affinities. */
Signature · 1 line
2.25.1
[tip: x86/irq] x86/irq: Set up per host CPU posted interrupt descriptors
The following commit has been merged into the x86/irq branch of tip:
The following commit has been merged into the x86/irq branch of tip:
Commit-ID: 43650dcf6d6322ec2d0938bb51f755810ffa783a Gitweb: https://git.kernel.org/tip/43650dcf6d6322ec2d0938bb51f755810ffa783a Author: Jacob Pan <[email protected]> AuthorDate: Tue, 23 Apr 2024 10:41:08 -07:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 30 Apr 2024 00:54:42 +02:00
x86/irq: Set up per host CPU posted interrupt descriptors
To support posted MSIs, create a posted interrupt descriptor (PID) for each host CPU. Later on, when setting up interrupt affinity, the IOMMU's interrupt remapping table entry (IRTE) will point to the physical address of the matching CPU's PID.
Each PID is initialized with the owner CPU's physical APICID as the destination.
--- arch/x86/include/asm/hardirq.h | 3 +++ arch/x86/include/asm/posted_intr.h | 6 ++++++ arch/x86/kernel/cpu/common.c | 3 +++ arch/x86/kernel/irq.c | 23 +++++++++++++++++++++++ 4 files changed, 35 insertions(+)
arch/x86/include/asm/hardirq.h +3 -0
@@ -48,6 +48,9 @@ typedef struct {DECLARE_PER_CPU_SHARED_ALIGNED (irq_cpustat_t, irq_stat); +#ifdef CONFIG_X86_POSTED_MSI +DECLARE_PER_CPU_ALIGNED(struct pi_desc, posted_msi_pi_desc); +#endif #define __ARCH_IRQ_STAT#define inc_irq_stat (member) this_cpu_inc(irq_stat.member)
arch/x86/include/asm/posted_intr.h +6 -0
@@ -91,4 +91,10 @@ static inline void __pi_clear_sn(struct pi_desc *pi_desc) pi_desc->notifications &= ~BIT(POSTED_INTR_SN); } +#ifdef CONFIG_X86_POSTED_MSI +extern void intel_posted_msi_init (void ); +#else +static inline void intel_posted_msi_init (void ) {}; +#endif /* X86_POSTED_MSI */ +#endif /* _X86_POSTED_INTR_H */
arch/x86/kernel/cpu/common.c +3 -0
@@ -68,6 +68,7 @@#include <asm/traps.h> #include <asm/sev.h> #include <asm/tdx.h> +#include <asm/posted_intr.h> #include "cpu.h"
@@ -2227,6 +2228,8 @@ void cpu_init(void)barrier ();x2apic_setup (); + +intel_posted_msi_init (); }mmgrab (&init_mm);
arch/x86/kernel/irq.c +23 -0
@@ -22,6 +22,8 @@#include <asm/desc.h> #include <asm/traps.h> #include <asm/thermal.h> +#include <asm/posted_intr.h> +#include <asm/irq_remapping.h> #define CREATE_TRACE_POINTS #include <asm/trace/irq_vectors.h>
@@ -334,6 +336,27 @@ DEFINE_IDTENTRY_SYSVEC_SIMPLE(sysvec_kvm_posted_intr_nested_ipi) }#endif +#ifdef CONFIG_X86_POSTED_MSI + +/* Posted Interrupt Descriptors for coalesced MSIs to be posted */ +DEFINE_PER_CPU_ALIGNED(struct pi_desc, posted_msi_pi_desc); + +void intel_posted_msi_init (void ) +{ +u32 destination; +u32 apic_id; + +this_cpu_write (posted_msi_pi_desc.nv,POSTED_MSI_NOTIFICATION_VECTOR ); + +/* +* APIC destination ID is stored in bit 8:15 while in XAPIC mode. +* VT-d spec. CH 9.11 +*/ + apic_id =this_cpu_read (x86_cpu_to_apicid); + destination =x2apic_enabled () ? apic_id : apic_id <<8 ; +this_cpu_write (posted_msi_pi_desc.ndst, destination); +} +#endif /* X86_POSTED_MSI */ #ifdef CONFIG_HOTPLUG_CPU /* A cpu has been removed from cpu_online_mask. Reset irq affinities. */
PATCH v3 7/12 x86/irq: Factor out calling ISR from common_interrupt
In reply to Jacob PanPrepare for calling external IRQ handlers directly from the posted MSI demultiplexing loop. Extract the common code with common interrupt to avoid code duplication.
--- arch/x86/kernel/irq.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)
arch/x86/kernel/irq.c +14 -9
@@ -242,18 +242,10 @@ static __always_inline void handle_irq(struct irq_desc *desc,__handle_irq (desc, regs); } -/* -* common_interrupt() handles all normal device IRQ's (the special SMP -* cross-CPU interrupts have their own entry points). -*/ -DEFINE_IDTENTRY_IRQ (common_interrupt) +static __always_inline void call_irq_handler(int vector, struct pt_regs *regs) { -struct pt_regs *old_regs =set_irq_regs (regs);struct irq_desc *desc; -/* entry code tells RCU that we're not quiescent. Check it. */ -RCU_LOCKDEP_WARN (!rcu_is_watching (),"IRQ failed to wake up RCU" ); - desc =__this_cpu_read (vector_irq[vector]);if (likely (!IS_ERR_OR_NULL (desc))) {handle_irq (desc, regs);
@@ -268,7 +260,20 @@ DEFINE_IDTENTRY_IRQ(common_interrupt)__this_cpu_write (vector_irq[vector],VECTOR_UNUSED ); } } +} + +/* +* common_interrupt() handles all normal device IRQ's (the special SMP +* cross-CPU interrupts have their own entry points). +*/ +DEFINE_IDTENTRY_IRQ (common_interrupt) +{ +struct pt_regs *old_regs =set_irq_regs (regs); + +/* entry code tells RCU that we're not quiescent. Check it. */ +RCU_LOCKDEP_WARN (!rcu_is_watching (),"IRQ failed to wake up RCU" ); +call_irq_handler (vector, regs);set_irq_regs (old_regs); }
Signature · 1 line
2.25.1
[tip: x86/irq] x86/irq: Factor out handler invocation from common_interrupt()
The following commit has been merged into the x86/irq branch of tip:
The following commit has been merged into the x86/irq branch of tip:
Commit-ID: 6087c7f36ab293a06bc0bcf3857ed4d7eb1f9905 Gitweb: https://git.kernel.org/tip/6087c7f36ab293a06bc0bcf3857ed4d7eb1f9905 Author: Jacob Pan <[email protected]> AuthorDate: Tue, 23 Apr 2024 10:41:09 -07:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 30 Apr 2024 00:54:42 +02:00
x86/irq: Factor out handler invocation from common_interrupt()
Prepare for calling external interrupt handlers directly from the posted MSI demultiplexing loop. Extract the common code from common_interrupt() to avoid code duplication.
--- arch/x86/kernel/irq.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)
arch/x86/kernel/irq.c +14 -9
@@ -242,18 +242,10 @@ static __always_inline void handle_irq(struct irq_desc *desc,__handle_irq (desc, regs); } -/* -* common_interrupt() handles all normal device IRQ's (the special SMP -* cross-CPU interrupts have their own entry points). -*/ -DEFINE_IDTENTRY_IRQ (common_interrupt) +static __always_inline void call_irq_handler(int vector, struct pt_regs *regs) { -struct pt_regs *old_regs =set_irq_regs (regs);struct irq_desc *desc; -/* entry code tells RCU that we're not quiescent. Check it. */ -RCU_LOCKDEP_WARN (!rcu_is_watching (),"IRQ failed to wake up RCU" ); - desc =__this_cpu_read (vector_irq[vector]);if (likely (!IS_ERR_OR_NULL (desc))) {handle_irq (desc, regs);
@@ -268,7 +260,20 @@ DEFINE_IDTENTRY_IRQ(common_interrupt)__this_cpu_write (vector_irq[vector],VECTOR_UNUSED ); } } +} + +/* +* common_interrupt() handles all normal device IRQ's (the special SMP +* cross-CPU interrupts have their own entry points). +*/ +DEFINE_IDTENTRY_IRQ (common_interrupt) +{ +struct pt_regs *old_regs =set_irq_regs (regs); + +/* entry code tells RCU that we're not quiescent. Check it. */ +RCU_LOCKDEP_WARN (!rcu_is_watching (),"IRQ failed to wake up RCU" ); +call_irq_handler (vector, regs);set_irq_regs (old_regs); }
PATCH v3 8/12 x86/irq: Install posted MSI notification handler
In reply to Jacob PanAll MSI vectors are multiplexed into a single notification vector when posted MSI is enabled. It is the responsibility of the notification vector handler to demultiplex MSI vectors. In this handler, for each pending bit, MSI vector handlers are dispatched without IDT delivery.
For example, the interrupt flow will change as follows:
(3 MSIs of different vectors arrive in a a high frequency burst)
BEFORE:
interrupt(MSI)
irq_enter()
handler() /* EOI */
irq_exit()
process_softirq()
interrupt(MSI)
irq_enter()
handler() /* EOI */
irq_exit()
process_softirq()
interrupt(MSI)
irq_enter()
handler() /* EOI */
irq_exit()
process_softirq()AFTER:
interrupt /* Posted MSI notification vector */
irq_enter()
atomic_xchg(PIR)
handler()
handler()
handler()
pi_clear_on()
apic_eoi()
irq_exit()
process_softirq()Except for the leading MSI, CPU notifications are skipped/coalesced.
For MSIs arrive at a low frequency, the demultiplexing loop does not wait for more interrupts to coalesce. Therefore, there's no additional latency other than the processing time.
---
v3:
- Add support for FRED
- Fix double eoi if a vector is invalid (Kevin)
v2:
- Delete extra inline attribute
- Fix pir pointer in xchg (Zeng Guang)
--- arch/x86/entry/entry_fred.c | 2 + arch/x86/include/asm/hardirq.h | 3 + arch/x86/include/asm/idtentry.h | 6 ++ arch/x86/kernel/idt.c | 3 + arch/x86/kernel/irq.c | 125 +++++++++++++++++++++++++++++++- 5 files changed, 135 insertions(+), 4 deletions(-)
arch/x86/entry/entry_fred.c +2 -0
@@ -117,6 +117,8 @@ static idtentry_t sysvec_table[NR_SYSTEM_VECTORS] __ro_after_init = {SYSVEC (POSTED_INTR_VECTOR , kvm_posted_intr_ipi),SYSVEC (POSTED_INTR_WAKEUP_VECTOR , kvm_posted_intr_wakeup_ipi),SYSVEC (POSTED_INTR_NESTED_VECTOR , kvm_posted_intr_nested_ipi), + + SYSVEC(POSTED_MSI_NOTIFICATION_VECTOR, posted_msi_notification), };static bool fred_setup_done__initdata ;
arch/x86/include/asm/hardirq.h +3 -0
@@ -44,6 +44,9 @@ typedef struct {unsigned int irq_hv_reenlightenment_count;unsigned int hyperv_stimer0_count;#endif +#ifdef CONFIG_X86_POSTED_MSI +unsigned int posted_msi_notification_count; +#endif }____cacheline_aligned irq_cpustat_t;DECLARE_PER_CPU_SHARED_ALIGNED (irq_cpustat_t, irq_stat);
arch/x86/include/asm/idtentry.h +6 -0
@@ -751,6 +751,12 @@ DECLARE_IDTENTRY_SYSVEC(POSTED_INTR_NESTED_VECTOR, sysvec_kvm_posted_intr_nested# define fred_sysvec_kvm_posted_intr_nested_ipi NULL#endif +# ifdef CONFIG_X86_POSTED_MSI +DECLARE_IDTENTRY_SYSVEC (POSTED_MSI_NOTIFICATION_VECTOR , sysvec_posted_msi_notification); +#else +# define fred_sysvec_posted_msi_notification NULL +# endif +#if IS_ENABLED (CONFIG_HYPERV )DECLARE_IDTENTRY_SYSVEC (HYPERVISOR_CALLBACK_VECTOR , sysvec_hyperv_callback);DECLARE_IDTENTRY_SYSVEC (HYPERV_REENLIGHTENMENT_VECTOR , sysvec_hyperv_reenlightenment);
arch/x86/kernel/idt.c +3 -0
@@ -163,6 +163,9 @@ static const __initconst struct idt_data apic_idts[] = {# endif INTG(SPURIOUS_APIC_VECTOR, asm_sysvec_spurious_apic_interrupt), INTG(ERROR_APIC_VECTOR, asm_sysvec_error_interrupt), +# ifdef CONFIG_X86_POSTED_MSI + INTG(POSTED_MSI_NOTIFICATION_VECTOR, asm_sysvec_posted_msi_notification), +# endif #endif };
arch/x86/kernel/irq.c +121 -4
162 lines not on this page · raw message
Signature · 1 line
2.25.1
[tip: x86/irq] x86/irq: Install posted MSI notification handler
The following commit has been merged into the x86/irq branch of tip:
The following commit has been merged into the x86/irq branch of tip:
Commit-ID: 1b03d82ba15e895776f1f7da2bb56a9a60e6dfed Gitweb: https://git.kernel.org/tip/1b03d82ba15e895776f1f7da2bb56a9a60e6dfed Author: Jacob Pan <[email protected]> AuthorDate: Tue, 23 Apr 2024 10:41:10 -07:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 30 Apr 2024 00:54:42 +02:00
x86/irq: Install posted MSI notification handler
All MSI vectors are multiplexed into a single notification vector when posted MSI is enabled. It is the responsibility of the notification vector handler to demultiplex MSI vectors. In the handler the MSI vector handlers are dispatched without IDT delivery for each pending MSI interrupt.
For example, the interrupt flow will change as follows:
(3 MSIs of different vectors arrive in a a high frequency burst)
BEFORE:
interrupt(MSI)
irq_enter()
handler() /* EOI */
irq_exit()
process_softirq()
interrupt(MSI)
irq_enter()
handler() /* EOI */
irq_exit()
process_softirq()
interrupt(MSI)
irq_enter()
handler() /* EOI */
irq_exit()
process_softirq()AFTER:
interrupt /* Posted MSI notification vector */
irq_enter()
atomic_xchg(PIR)
handler()
handler()
handler()
pi_clear_on()
apic_eoi()
irq_exit()
process_softirq()Except for the leading MSI, CPU notifications are skipped/coalesced.
For MSIs which arrive at a low frequency, the demultiplexing loop does not wait for more interrupts to coalesce. Therefore, there's no additional latency other than the processing time.
--- arch/x86/entry/entry_fred.c | 2 +- arch/x86/include/asm/hardirq.h | 3 +- arch/x86/include/asm/idtentry.h | 6 +- arch/x86/kernel/idt.c | 3 +- arch/x86/kernel/irq.c | 125 ++++++++++++++++++++++++++++++- 5 files changed, 135 insertions(+), 4 deletions(-)
arch/x86/entry/entry_fred.c +2 -0
@@ -117,6 +117,8 @@ static idtentry_t sysvec_table[NR_SYSTEM_VECTORS] __ro_after_init = {SYSVEC (POSTED_INTR_VECTOR , kvm_posted_intr_ipi),SYSVEC (POSTED_INTR_WAKEUP_VECTOR , kvm_posted_intr_wakeup_ipi),SYSVEC (POSTED_INTR_NESTED_VECTOR , kvm_posted_intr_nested_ipi), + + SYSVEC(POSTED_MSI_NOTIFICATION_VECTOR, posted_msi_notification), };static bool fred_setup_done__initdata ;
arch/x86/include/asm/hardirq.h +3 -0
@@ -44,6 +44,9 @@ typedef struct {unsigned int irq_hv_reenlightenment_count;unsigned int hyperv_stimer0_count;#endif +#ifdef CONFIG_X86_POSTED_MSI +unsigned int posted_msi_notification_count; +#endif }____cacheline_aligned irq_cpustat_t;DECLARE_PER_CPU_SHARED_ALIGNED (irq_cpustat_t, irq_stat);
arch/x86/include/asm/idtentry.h +6 -0
@@ -751,6 +751,12 @@ DECLARE_IDTENTRY_SYSVEC(POSTED_INTR_NESTED_VECTOR, sysvec_kvm_posted_intr_nested# define fred_sysvec_kvm_posted_intr_nested_ipi NULL#endif +# ifdef CONFIG_X86_POSTED_MSI +DECLARE_IDTENTRY_SYSVEC (POSTED_MSI_NOTIFICATION_VECTOR , sysvec_posted_msi_notification); +#else +# define fred_sysvec_posted_msi_notification NULL +# endif +#if IS_ENABLED (CONFIG_HYPERV )DECLARE_IDTENTRY_SYSVEC (HYPERVISOR_CALLBACK_VECTOR , sysvec_hyperv_callback);DECLARE_IDTENTRY_SYSVEC (HYPERV_REENLIGHTENMENT_VECTOR , sysvec_hyperv_reenlightenment);
arch/x86/kernel/idt.c +3 -0
@@ -163,6 +163,9 @@ static const __initconst struct idt_data apic_idts[] = {# endif INTG(SPURIOUS_APIC_VECTOR, asm_sysvec_spurious_apic_interrupt), INTG(ERROR_APIC_VECTOR, asm_sysvec_error_interrupt), +# ifdef CONFIG_X86_POSTED_MSI + INTG(POSTED_MSI_NOTIFICATION_VECTOR, asm_sysvec_posted_msi_notification), +# endif #endif };
arch/x86/kernel/irq.c +121 -4
162 lines not on this page · raw message
PATCH v3 9/12 x86/irq: Factor out common code for checking pending interrupts
In reply to Jacob PanUse a common function for checking pending interrupt vector in APIC IRR instead of duplicated open coding them.
Additional checks for posted MSI vectors can then be contained in this function.
--- arch/x86/include/asm/apic.h | 11 +++++++++++ arch/x86/kernel/apic/vector.c | 5 ++--- arch/x86/kernel/irq.c | 5 ++--- 3 files changed, 15 insertions(+), 6 deletions(-)
arch/x86/include/asm/apic.h +11 -0
@@ -500,6 +500,17 @@ static inline bool lapic_vector_set_in_irr(unsigned int vector)return !!(irr & (1U << (vector %32 ))); } +static inline bool is_vector_pending (unsigned int vector) +{ +unsigned int irr; + + irr =apic_read (APIC_IRR + (vector /32 *0x10 )); +if (irr & (1 << (vector %32 ))) +return true; + +return false; +} +/* * Warm reset vector position: */
arch/x86/kernel/apic/vector.c +2 -3
@@ -965,7 +965,7 @@ static void __vector_cleanup(struct vector_cleanup *cl, bool check_irr)lockdep_assert_held (&vector_lock);hlist_for_each_entry_safe (apicd, tmp, &cl->head, clist) { -unsigned int irr, vector = apicd->prev_vector; +unsigned int vector = apicd->prev_vector;/* * Paranoia: Check if the vector that needs to be cleaned
@@ -979,8 +979,7 @@ static void __vector_cleanup(struct vector_cleanup *cl, bool check_irr)* fixup_irqs() was just called to scan IRR for set bits and * forward them to new destination CPUs via IPIs. */ - irr = check_irr ?apic_read (APIC_IRR + (vector /32 *0x10 )) :0 ; -if (irr & (1U << (vector %32 ))) { +if (check_irr &&is_vector_pending (vector)) {pr_warn_once ("Moved interrupt pending in old target APIC %u\n" , apicd->irq); rearm = true;continue ;
arch/x86/kernel/irq.c +2 -3
@@ -484,7 +484,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_posted_msi_notification)/* A cpu has been removed from cpu_online_mask. Reset irq affinities. */ void fixup_irqs (void ) { -unsigned int irr, vector; +unsigned int vector;struct irq_desc *desc;struct irq_data *data;struct irq_chip *chip;
@@ -511,8 +511,7 @@ void fixup_irqs(void)if (IS_ERR_OR_NULL (__this_cpu_read (vector_irq[vector])))continue ; - irr =apic_read (APIC_IRR + (vector /32 *0x10 )); -if (irr & (1 << (vector %32 ))) { +if (is_vector_pending (vector)) { desc =__this_cpu_read (vector_irq[vector]);raw_spin_lock (&desc->lock);
Signature · 1 line
2.25.1
[tip: x86/irq] x86/irq: Factor out common code for checking pending interrupts
The following commit has been merged into the x86/irq branch of tip:
The following commit has been merged into the x86/irq branch of tip:
Commit-ID: fef05a078b6fa1e9047e0486f1f6daf70664fd12 Gitweb: https://git.kernel.org/tip/fef05a078b6fa1e9047e0486f1f6daf70664fd12 Author: Jacob Pan <[email protected]> AuthorDate: Tue, 23 Apr 2024 10:41:11 -07:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 30 Apr 2024 00:54:43 +02:00
x86/irq: Factor out common code for checking pending interrupts
Use a common function for checking pending interrupt vector in APIC IRR instead of duplicated open coding them.
Additional checks for posted MSI vectors can then be contained in this function.
--- arch/x86/include/asm/apic.h | 11 +++++++++++ arch/x86/kernel/apic/vector.c | 5 ++--- arch/x86/kernel/irq.c | 5 ++--- 3 files changed, 15 insertions(+), 6 deletions(-)
arch/x86/include/asm/apic.h +11 -0
@@ -500,6 +500,17 @@ static inline bool lapic_vector_set_in_irr(unsigned int vector)return !!(irr & (1U << (vector %32 ))); } +static inline bool is_vector_pending (unsigned int vector) +{ +unsigned int irr; + + irr =apic_read (APIC_IRR + (vector /32 *0x10 )); +if (irr & (1 << (vector %32 ))) +return true; + +return false; +} +/* * Warm reset vector position: */
arch/x86/kernel/apic/vector.c +2 -3
@@ -965,7 +965,7 @@ static void __vector_cleanup(struct vector_cleanup *cl, bool check_irr)lockdep_assert_held (&vector_lock);hlist_for_each_entry_safe (apicd, tmp, &cl->head, clist) { -unsigned int irr, vector = apicd->prev_vector; +unsigned int vector = apicd->prev_vector;/* * Paranoia: Check if the vector that needs to be cleaned
@@ -979,8 +979,7 @@ static void __vector_cleanup(struct vector_cleanup *cl, bool check_irr)* fixup_irqs() was just called to scan IRR for set bits and * forward them to new destination CPUs via IPIs. */ - irr = check_irr ?apic_read (APIC_IRR + (vector /32 *0x10 )) :0 ; -if (irr & (1U << (vector %32 ))) { +if (check_irr &&is_vector_pending (vector)) {pr_warn_once ("Moved interrupt pending in old target APIC %u\n" , apicd->irq); rearm = true;continue ;
arch/x86/kernel/irq.c +2 -3
@@ -484,7 +484,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_posted_msi_notification)/* A cpu has been removed from cpu_online_mask. Reset irq affinities. */ void fixup_irqs (void ) { -unsigned int irr, vector; +unsigned int vector;struct irq_desc *desc;struct irq_data *data;struct irq_chip *chip;
@@ -511,8 +511,7 @@ void fixup_irqs(void)if (IS_ERR_OR_NULL (__this_cpu_read (vector_irq[vector])))continue ; - irr =apic_read (APIC_IRR + (vector /32 *0x10 )); -if (irr & (1 << (vector %32 ))) { +if (is_vector_pending (vector)) { desc =__this_cpu_read (vector_irq[vector]);raw_spin_lock (&desc->lock);
PATCH v3 10/12 x86/irq: Extend checks for pending vectors to posted interrupts
In reply to Jacob PanDuring interrupt affinity change, it is possible to have interrupts delivered to the old CPU after the affinity has changed to the new one. To prevent lost interrupts, local APIC IRR is checked on the old CPU. Similar checks must be done for posted MSIs given the same reason.
Consider the following scenario:
Device system agent iommu memory CPU/LAPIC
1 FEEX_XXXX
2 Interrupt request
3 Fetch IRTE ->
4 ->Atomic Swap PID.PIR(vec)
Push to Global Observable(GO)
5 if (ON*)
done;* else
6 send a notification ->
* ON: outstanding notification, 1 will suppress new notifications
If the affinity change happens between 3 and 5 in IOMMU, the old CPU's posted interrupt request (PIR) could have pending bit set for the vector being moved.
This patch adds a helper function to check individual vector status. Then use the helper to check for pending interrupts on the source CPU's PID.
---
v3: Fix a stray letter in the comment, no code change
v2: Fold in helper function patch.
--- arch/x86/include/asm/apic.h | 3 ++- arch/x86/include/asm/posted_intr.h | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-)
arch/x86/include/asm/apic.h +2 -1
@@ -14,6 +14,7 @@#include <asm/msr.h> #include <asm/hardirq.h> #include <asm/io.h> +#include <asm/posted_intr.h> #define ARCH_APICTIMER_STOPS_ON_C3 1
@@ -508,7 +509,7 @@ static inline bool is_vector_pending(unsigned int vector)if (irr & (1 << (vector %32 )))return true; -return false; +return pi_pending_this_cpu (vector); }/*
arch/x86/include/asm/posted_intr.h +18 -0
@@ -1,6 +1,7 @@/* SPDX-License-Identifier: GPL-2.0 */ #ifndef _X86_POSTED_INTR_H#define _X86_POSTED_INTR_H +#include <asm/irq_vectors.h> #define POSTED_INTR_ON 0#define POSTED_INTR_SN 1
@@ -92,8 +93,25 @@ static inline void __pi_clear_sn(struct pi_desc *pi_desc) }#ifdef CONFIG_X86_POSTED_MSI +/* +* Not all external vectors are subject to interrupt remapping, e.g. IOMMU's +* own interrupts. Here we do not distinguish them since those vector bits in +* PIR will always be zero. +*/ +static inline bool pi_pending_this_cpu (unsigned int vector) +{ +struct pi_desc *pid =this_cpu_ptr (&posted_msi_pi_desc); + +if (WARN_ON_ONCE (vector >NR_VECTORS || vector <FIRST_EXTERNAL_VECTOR )) +return false; + +return test_bit (vector, (unsigned long *)pid->pir); +} +extern void intel_posted_msi_init (void );#else +static inline bool pi_pending_this_cpu (unsigned int vector) {return false; } +static inline void intel_posted_msi_init (void ) {};#endif /* X86_POSTED_MSI */
Signature · 1 line
2.25.1
[tip: x86/irq] x86/irq: Extend checks for pending vectors to posted interrupts
The following commit has been merged into the x86/irq branch of tip:
The following commit has been merged into the x86/irq branch of tip:
Commit-ID: ce0a92871179f8ca58ae8e3cf50e726a163bf831 Gitweb: https://git.kernel.org/tip/ce0a92871179f8ca58ae8e3cf50e726a163bf831 Author: Jacob Pan <[email protected]> AuthorDate: Tue, 23 Apr 2024 10:41:12 -07:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 30 Apr 2024 00:54:43 +02:00
x86/irq: Extend checks for pending vectors to posted interrupts
During interrupt affinity change, it is possible to have interrupts delivered to the old CPU after the affinity has changed to the new one. To prevent lost interrupts, local APIC IRR is checked on the old CPU. Similar checks must be done for posted MSIs given the same reason.
Consider the following scenario:
Device system agent iommu memory CPU/LAPIC
1 FEEX_XXXX
2 Interrupt request
3 Fetch IRTE ->
4 ->Atomic Swap PID.PIR(vec)
Push to Global Observable(GO)
5 if (ON*)
done;* else
6 send a notification ->
* ON: outstanding notification, 1 will suppress new notifications
If the affinity change happens between 3 and 5 in the IOMMU, the old CPU's posted interrupt request (PIR) could have the pending bit set for the vector being moved.
Add a helper function to check individual vector status. Then use the helper to check for pending interrupts on the source CPU's PID.
--- arch/x86/include/asm/apic.h | 3 ++- arch/x86/include/asm/posted_intr.h | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-)
arch/x86/include/asm/apic.h +2 -1
@@ -14,6 +14,7 @@#include <asm/msr.h> #include <asm/hardirq.h> #include <asm/io.h> +#include <asm/posted_intr.h> #define ARCH_APICTIMER_STOPS_ON_C3 1
@@ -508,7 +509,7 @@ static inline bool is_vector_pending(unsigned int vector)if (irr & (1 << (vector %32 )))return true; -return false; +return pi_pending_this_cpu (vector); }/*
arch/x86/include/asm/posted_intr.h +18 -0
@@ -1,6 +1,7 @@/* SPDX-License-Identifier: GPL-2.0 */ #ifndef _X86_POSTED_INTR_H#define _X86_POSTED_INTR_H +#include <asm/irq_vectors.h> #define POSTED_INTR_ON 0#define POSTED_INTR_SN 1
@@ -92,8 +93,25 @@ static inline void __pi_clear_sn(struct pi_desc *pi_desc) }#ifdef CONFIG_X86_POSTED_MSI +/* +* Not all external vectors are subject to interrupt remapping, e.g. IOMMU's +* own interrupts. Here we do not distinguish them since those vector bits in +* PIR will always be zero. +*/ +static inline bool pi_pending_this_cpu (unsigned int vector) +{ +struct pi_desc *pid =this_cpu_ptr (&posted_msi_pi_desc); + +if (WARN_ON_ONCE (vector >NR_VECTORS || vector <FIRST_EXTERNAL_VECTOR )) +return false; + +return test_bit (vector, (unsigned long *)pid->pir); +} +extern void intel_posted_msi_init (void );#else +static inline bool pi_pending_this_cpu (unsigned int vector) {return false; } +static inline void intel_posted_msi_init (void ) {};#endif /* X86_POSTED_MSI */
PATCH v3 11/12 iommu/vt-d: Make posted MSI an opt-in cmdline option
In reply to Jacob PanAdd a command line opt-in option for posted MSI if CONFIG_X86_POSTED_MSI=y.
Also introduce a helper function for testing if posted MSI is supported on the platform.
---
v3: Delete unnecessary checks for disable_irq_post || disable_irq_remap.
(Kevin) --- Documentation/admin-guide/kernel-parameters.txt | 1 + arch/x86/include/asm/irq_remapping.h | 11 +++++++++++ drivers/iommu/irq_remapping.c | 9 ++++++++- 3 files changed, 20 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +1 -0
@@ -2251,6 +2251,7 @@ no_x2apic_optout BIOS x2APIC opt-out request will be ignored nopost disable Interrupt Posting + posted_msi enable MSIs delivered as posted interrupts iomem= Disable strict checking of access to MMIO memory strict regions from userspace.
arch/x86/include/asm/irq_remapping.h +11 -0
@@ -50,6 +50,17 @@ static inline struct irq_domain *arch_get_ir_parent_domain(void) return x86_vector_domain; } +#ifdef CONFIG_X86_POSTED_MSI +extern int enable_posted_msi; + +static inline bool posted_msi_supported (void ) +{ +return enable_posted_msi &&irq_remapping_cap (IRQ_POSTING_CAP ); +} +#else +static inline bool posted_msi_supported (void ) {return false; }; +#endif +#else /* CONFIG_IRQ_REMAP */ static inline bool irq_remapping_cap (enum irq_remap_cap cap) {return 0 ; }
drivers/iommu/irq_remapping.c +8 -1
@@ -24,6 +24,10 @@ int no_x2apic_optout;int disable_irq_post =0 ; +#ifdef CONFIG_X86_POSTED_MSI +int enable_posted_msi; +#endif +static int disable_irq_remap;static struct irq_remap_ops *remap_ops;
@@ -70,7 +74,10 @@ static __init int setup_irqremap(char *str) no_x2apic_optout =1 ; else if (!strncmp(str, "nopost", 6)) disable_irq_post =1 ; - +#ifdef CONFIG_X86_POSTED_MSI + else if (!strncmp(str, "posted_msi", 10)) + enable_posted_msi =1 ; +#endif str +=strcspn (str,"," );while (*str ==',' ) str++;
Signature · 1 line
2.25.1
[tip: x86/irq] iommu/vt-d: Make posted MSI an opt-in command line option
The following commit has been merged into the x86/irq branch of tip:
The following commit has been merged into the x86/irq branch of tip:
Commit-ID: be9be07b22c96dc03d0ecc76b5a5f21c2dcb05a1 Gitweb: https://git.kernel.org/tip/be9be07b22c96dc03d0ecc76b5a5f21c2dcb05a1 Author: Jacob Pan <[email protected]> AuthorDate: Tue, 23 Apr 2024 10:41:13 -07:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 30 Apr 2024 00:54:43 +02:00
iommu/vt-d: Make posted MSI an opt-in command line option
Add a command line opt-in option for posted MSI if CONFIG_X86_POSTED_MSI=y.
Also introduce a helper function for testing if posted MSI is supported on the platform.
--- Documentation/admin-guide/kernel-parameters.txt | 2 ++ arch/x86/include/asm/irq_remapping.h | 7 +++++++ drivers/iommu/irq_remapping.c | 5 ++++- 3 files changed, 13 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +2 -0
@@ -2251,6 +2251,8 @@ no_x2apic_optout BIOS x2APIC opt-out request will be ignored nopost disable Interrupt Posting + posted_msi + enable MSIs delivered as posted interrupts iomem= Disable strict checking of access to MMIO memory strict regions from userspace.
arch/x86/include/asm/irq_remapping.h +7 -0
@@ -50,6 +50,13 @@ static inline struct irq_domain *arch_get_ir_parent_domain(void) return x86_vector_domain; } +extern bool enable_posted_msi; + +static inline bool posted_msi_supported (void ) +{ +return enable_posted_msi &&irq_remapping_cap (IRQ_POSTING_CAP ); +} +#else /* CONFIG_IRQ_REMAP */ static inline bool irq_remapping_cap (enum irq_remap_cap cap) {return 0 ; }
drivers/iommu/irq_remapping.c +4 -1
@@ -24,6 +24,8 @@ int no_x2apic_optout;int disable_irq_post =0 ; +bool enable_posted_msi__ro_after_init ; +static int disable_irq_remap;static struct irq_remap_ops *remap_ops;
@@ -70,7 +72,8 @@ static __init int setup_irqremap(char *str) no_x2apic_optout =1 ; else if (!strncmp(str, "nopost", 6)) disable_irq_post =1 ; - + else if (IS_ENABLED(CONFIG_X86_POSTED_MSI) && !strncmp(str, "posted_msi", 10)) + enable_posted_msi = true; str +=strcspn (str,"," );while (*str ==',' ) str++;
PATCH v3 12/12 iommu/vt-d: Enable posted mode for device MSIs
In reply to Jacob PanWith posted MSI feature enabled on the CPU side, iommu interrupt remapping table entries (IRTEs) for device MSI/x can be allocated, activated, and programed in posted mode. This means that IRTEs are linked with their respective PIDs of the target CPU.
Handlers for the posted MSI notification vector will de-multiplex device MSI handlers. CPU notifications are coalesced if interrupts arrive at a high frequency.
Excluding the following:
- legacy devices IOAPIC, HPET (may be needed for booting, not a source of high MSIs)
A new irq_chip for posted MSIs is introduced, the key difference is in irq_ack where EOI is performed by the notification handler.
When posted MSI is enabled, MSI domain/chip hierarchy will look like this example:
domain: IR-PCI-MSIX-0000:50:00.0-12
hwirq: 0x29
chip: IR-PCI-MSIX-0000:50:00.0
flags: 0x430
IRQCHIP_SKIP_SET_WAKE
IRQCHIP_ONESHOT_SAFE
parent:
domain: INTEL-IR-10-13
hwirq: 0x2d0000
chip: INTEL-IR-POST
flags: 0x0
parent:
domain: VECTOR
hwirq: 0x77
chip: APICVT-d's own IRQs (not remappable).
---
v3: - Remove extra irq_enter() in comments
- Fold in the introduction of posted MSI IRQ chip patch
v2: Fold in helper function for retrieving PID address
v1: Added a warning if the effective affinity mask is not set up
--- drivers/iommu/intel/irq_remapping.c | 113 +++++++++++++++++++++++++++- 1 file changed, 109 insertions(+), 4 deletions(-)
drivers/iommu/intel/irq_remapping.c +109 -4
@@ -19,6 +19,7 @@#include <asm/cpu.h> #include <asm/irq_remapping.h> #include <asm/pci-direct.h> +#include <asm/posted_intr.h> #include "iommu.h" #include "../irq_remapping.h"
@@ -49,6 +50,7 @@ struct irq_2_iommu {u16 sub_handle;u8 irte_mask;enum irq_mode mode; +bool posted_msi; };struct intel_ir_data {
@@ -1118,6 +1120,14 @@ static void prepare_irte(struct irte *irte, int vector, unsigned int dest) irte->redir_hint =1 ; } +static void prepare_irte_posted (struct irte *irte) +{ +memset (irte,0 ,sizeof (*irte)); + + irte->present =1 ; + irte->p_pst =1 ; +} +struct irq_remap_ops intel_irq_remap_ops = { .prepare = intel_prepare_irq_remapping, .enable = intel_enable_irq_remapping,
@@ -1126,6 +1136,47 @@ struct irq_remap_ops intel_irq_remap_ops = { .enable_faulting = enable_drhd_fault_handling, }; +#ifdef CONFIG_X86_POSTED_MSI + +static phys_addr_t get_pi_desc_addr (struct irq_data *irqd) +{ +int cpu =cpumask_first (irq_data_get_effective_affinity_mask (irqd)); + +if (WARN_ON (cpu >= nr_cpu_ids)) +return 0 ; + +return __pa (per_cpu_ptr (&posted_msi_pi_desc, cpu)); +} + +static void intel_ir_reconfigure_irte_posted (struct irq_data *irqd) +{ +struct intel_ir_data *ir_data = irqd->chip_data; +struct irte *irte = &ir_data->irte_entry; +struct irte irte_pi; +u64 pid_addr; + + pid_addr =get_pi_desc_addr (irqd); + +if (!pid_addr) { +pr_warn ("Failed to setup IRQ %d for posted mode" , irqd->irq); +return ; + } + +memset (&irte_pi,0 ,sizeof (irte_pi)); + +/* The shared IRTE already be set up as posted during alloc_irte */ +dmar_copy_shared_irte (&irte_pi, irte); + + irte_pi.pda_l = (pid_addr >> (32 -PDA_LOW_BIT )) & ~(-1UL <<PDA_LOW_BIT ); + irte_pi.pda_h = (pid_addr >>32 ) & ~(-1UL <<PDA_HIGH_BIT ); + +modify_irte (&ir_data->irq_2_iommu, &irte_pi); +} + +#else +static inline void intel_ir_reconfigure_irte_posted (struct irq_data *irqd) {} +#endif +static void intel_ir_reconfigure_irte (struct irq_data *irqd,bool force) {struct intel_ir_data *ir_data = irqd->chip_data;
@@ -1139,8 +1190,9 @@ static void intel_ir_reconfigure_irte(struct irq_data *irqd, bool force) irte->vector = cfg->vector; irte->dest_id =IRTE_DEST (cfg->dest_apicid); -/* Update the hardware only if the interrupt is in remapped mode. */ -if (force || ir_data->irq_2_iommu.mode ==IRQ_REMAPPING ) +if (ir_data->irq_2_iommu.posted_msi) +intel_ir_reconfigure_irte_posted (irqd); +else if (force || ir_data->irq_2_iommu.mode ==IRQ_REMAPPING )modify_irte (&ir_data->irq_2_iommu, irte); }
@@ -1194,7 +1246,7 @@ static int intel_ir_set_vcpu_affinity(struct irq_data *data, void *info)struct intel_ir_data *ir_data = data->chip_data;struct vcpu_data *vcpu_pi_info = info; -/* stop posting interrupts, back to remapping mode */ +/* stop posting interrupts, back to the default mode */ if (!vcpu_pi_info) {modify_irte (&ir_data->irq_2_iommu, &ir_data->irte_entry); }else {
@@ -1233,6 +1285,50 @@ static struct irq_chip intel_ir_chip = { .irq_set_vcpu_affinity = intel_ir_set_vcpu_affinity, }; +static void dummy (struct irq_data *d) +{ +} + +/* +* With posted MSIs, all vectors are multiplexed into a single notification +* vector. Devices MSIs are then dispatched in a demux loop where +* EOIs can be coalesced as well. +* +* "INTEL-IR-POST" IRQ chip does not do EOI on ACK, thus the dummy irq_ack() +* function. Instead EOI is performed by the posted interrupt notification +* handler. +* +* For the example below, 3 MSIs are coalesced into one CPU notification. Only +* one apic_eoi() is needed. +* +* __sysvec_posted_msi_notification() +* irq_enter(); +* handle_edge_irq() +* irq_chip_ack_parent() +* dummy(); // No EOI +* handle_irq_event() +* driver_handler() +* handle_edge_irq() +* irq_chip_ack_parent() +* dummy(); // No EOI +* handle_irq_event() +* driver_handler() +* handle_edge_irq() +* irq_chip_ack_parent() +* dummy(); // No EOI +* handle_irq_event() +* driver_handler() +* apic_eoi() +* irq_exit() +*/ +static struct irq_chip intel_ir_chip_post_msi = { + .name ="INTEL-IR-POST" , + .irq_ack = dummy, + .irq_set_affinity = intel_ir_set_affinity, + .irq_compose_msi_msg = intel_ir_compose_msi_msg, + .irq_set_vcpu_affinity = intel_ir_set_vcpu_affinity, +}; +static void fill_msi_msg (struct msi_msg *msg,u32 index,u32 subhandle) {memset (msg,0 ,sizeof (*msg));
@@ -1274,6 +1370,11 @@ static void intel_irq_remapping_prepare_irte(struct intel_ir_data *data,break ;case X86_IRQ_ALLOC_TYPE_PCI_MSI :case X86_IRQ_ALLOC_TYPE_PCI_MSIX : +if (posted_msi_supported ()) { +prepare_irte_posted (irte); + data->irq_2_iommu.posted_msi =1 ; + } +set_msi_sid (irte,pci_real_dma_dev (msi_desc_to_pci_dev (info->desc)));break ;
@@ -1361,7 +1462,11 @@ static int intel_irq_remapping_alloc(struct irq_domain *domain, irq_data->hwirq = (index <<16 ) + i; irq_data->chip_data = ird; - irq_data->chip = &intel_ir_chip; +if (posted_msi_supported () && + ((info->type ==X86_IRQ_ALLOC_TYPE_PCI_MSI ) || (info->type ==X86_IRQ_ALLOC_TYPE_PCI_MSIX ))) + irq_data->chip = &intel_ir_chip_post_msi; +else + irq_data->chip = &intel_ir_chip;intel_irq_remapping_prepare_irte (ird, irq_cfg, info, index, i);irq_set_status_flags (virq + i,IRQ_MOVE_PCNTXT ); }
Signature · 1 line
2.25.1
[tip: x86/irq] iommu/vt-d: Enable posted mode for device MSIs
The following commit has been merged into the x86/irq branch of tip:
The following commit has been merged into the x86/irq branch of tip:
Commit-ID: ed1e48ea43703002dc202ac7f3b0b0b9981ec2f0 Gitweb: https://git.kernel.org/tip/ed1e48ea43703002dc202ac7f3b0b0b9981ec2f0 Author: Jacob Pan <[email protected]> AuthorDate: Tue, 23 Apr 2024 10:41:14 -07:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 30 Apr 2024 00:54:43 +02:00
iommu/vt-d: Enable posted mode for device MSIs
With posted MSI feature enabled on the CPU side, iommu interrupt remapping table entries (IRTEs) for device MSI/x can be allocated, activated, and programed in posted mode. This means that IRTEs are linked with their respective PIDs of the target CPU.
Handlers for the posted MSI notification vector will de-multiplex device MSI handlers. CPU notifications are coalesced if interrupts arrive at a high frequency.
Posted interrupts are only used for device MSI and not for legacy devices (IO/APIC, HPET).
Introduce a new irq_chip for posted MSIs, which has a dummy irq_ack() callback as EOI is performed in the notification handler once.
When posted MSI is enabled, MSI domain/chip hierarchy will look like this example:
domain: IR-PCI-MSIX-0000:50:00.0-12
hwirq: 0x29
chip: IR-PCI-MSIX-0000:50:00.0
flags: 0x430
IRQCHIP_SKIP_SET_WAKE
IRQCHIP_ONESHOT_SAFE
parent:
domain: INTEL-IR-10-13
hwirq: 0x2d0000
chip: INTEL-IR-POST
flags: 0x0
parent:
domain: VECTOR
hwirq: 0x77
chip: APIC--- drivers/iommu/intel/irq_remapping.c | 113 ++++++++++++++++++++++++++- 1 file changed, 109 insertions(+), 4 deletions(-)
drivers/iommu/intel/irq_remapping.c +109 -4
@@ -19,6 +19,7 @@#include <asm/cpu.h> #include <asm/irq_remapping.h> #include <asm/pci-direct.h> +#include <asm/posted_intr.h> #include "iommu.h" #include "../irq_remapping.h"
@@ -49,6 +50,7 @@ struct irq_2_iommu {u16 sub_handle;u8 irte_mask;enum irq_mode mode; +bool posted_msi; };struct intel_ir_data {
@@ -1118,6 +1120,14 @@ static void prepare_irte(struct irte *irte, int vector, unsigned int dest) irte->redir_hint =1 ; } +static void prepare_irte_posted (struct irte *irte) +{ +memset (irte,0 ,sizeof (*irte)); + + irte->present =1 ; + irte->p_pst =1 ; +} +struct irq_remap_ops intel_irq_remap_ops = { .prepare = intel_prepare_irq_remapping, .enable = intel_enable_irq_remapping,
@@ -1126,6 +1136,47 @@ struct irq_remap_ops intel_irq_remap_ops = { .enable_faulting = enable_drhd_fault_handling, }; +#ifdef CONFIG_X86_POSTED_MSI + +static phys_addr_t get_pi_desc_addr (struct irq_data *irqd) +{ +int cpu =cpumask_first (irq_data_get_effective_affinity_mask (irqd)); + +if (WARN_ON (cpu >= nr_cpu_ids)) +return 0 ; + +return __pa (per_cpu_ptr (&posted_msi_pi_desc, cpu)); +} + +static void intel_ir_reconfigure_irte_posted (struct irq_data *irqd) +{ +struct intel_ir_data *ir_data = irqd->chip_data; +struct irte *irte = &ir_data->irte_entry; +struct irte irte_pi; +u64 pid_addr; + + pid_addr =get_pi_desc_addr (irqd); + +if (!pid_addr) { +pr_warn ("Failed to setup IRQ %d for posted mode" , irqd->irq); +return ; + } + +memset (&irte_pi,0 ,sizeof (irte_pi)); + +/* The shared IRTE already be set up as posted during alloc_irte */ +dmar_copy_shared_irte (&irte_pi, irte); + + irte_pi.pda_l = (pid_addr >> (32 -PDA_LOW_BIT )) & ~(-1UL <<PDA_LOW_BIT ); + irte_pi.pda_h = (pid_addr >>32 ) & ~(-1UL <<PDA_HIGH_BIT ); + +modify_irte (&ir_data->irq_2_iommu, &irte_pi); +} + +#else +static inline void intel_ir_reconfigure_irte_posted (struct irq_data *irqd) {} +#endif +static void intel_ir_reconfigure_irte (struct irq_data *irqd,bool force) {struct intel_ir_data *ir_data = irqd->chip_data;
@@ -1139,8 +1190,9 @@ static void intel_ir_reconfigure_irte(struct irq_data *irqd, bool force) irte->vector = cfg->vector; irte->dest_id =IRTE_DEST (cfg->dest_apicid); -/* Update the hardware only if the interrupt is in remapped mode. */ -if (force || ir_data->irq_2_iommu.mode ==IRQ_REMAPPING ) +if (ir_data->irq_2_iommu.posted_msi) +intel_ir_reconfigure_irte_posted (irqd); +else if (force || ir_data->irq_2_iommu.mode ==IRQ_REMAPPING )modify_irte (&ir_data->irq_2_iommu, irte); }
@@ -1194,7 +1246,7 @@ static int intel_ir_set_vcpu_affinity(struct irq_data *data, void *info)struct intel_ir_data *ir_data = data->chip_data;struct vcpu_data *vcpu_pi_info = info; -/* stop posting interrupts, back to remapping mode */ +/* stop posting interrupts, back to the default mode */ if (!vcpu_pi_info) {modify_irte (&ir_data->irq_2_iommu, &ir_data->irte_entry); }else {
@@ -1233,6 +1285,49 @@ static struct irq_chip intel_ir_chip = { .irq_set_vcpu_affinity = intel_ir_set_vcpu_affinity, }; +/* +* With posted MSIs, all vectors are multiplexed into a single notification +* vector. Devices MSIs are then dispatched in a demux loop where +* EOIs can be coalesced as well. +* +* "INTEL-IR-POST" IRQ chip does not do EOI on ACK, thus the dummy irq_ack() +* function. Instead EOI is performed by the posted interrupt notification +* handler. +* +* For the example below, 3 MSIs are coalesced into one CPU notification. Only +* one apic_eoi() is needed. +* +* __sysvec_posted_msi_notification() +* irq_enter(); +* handle_edge_irq() +* irq_chip_ack_parent() +* dummy(); // No EOI +* handle_irq_event() +* driver_handler() +* handle_edge_irq() +* irq_chip_ack_parent() +* dummy(); // No EOI +* handle_irq_event() +* driver_handler() +* handle_edge_irq() +* irq_chip_ack_parent() +* dummy(); // No EOI +* handle_irq_event() +* driver_handler() +* apic_eoi() +* irq_exit() +*/ + +static void dummy_ack (struct irq_data *d) { } + +static struct irq_chip intel_ir_chip_post_msi = { + .name ="INTEL-IR-POST" , + .irq_ack = dummy_ack, + .irq_set_affinity = intel_ir_set_affinity, + .irq_compose_msi_msg = intel_ir_compose_msi_msg, + .irq_set_vcpu_affinity = intel_ir_set_vcpu_affinity, +}; +static void fill_msi_msg (struct msi_msg *msg,u32 index,u32 subhandle) {memset (msg,0 ,sizeof (*msg));
@@ -1274,6 +1369,11 @@ static void intel_irq_remapping_prepare_irte(struct intel_ir_data *data,break ;case X86_IRQ_ALLOC_TYPE_PCI_MSI :case X86_IRQ_ALLOC_TYPE_PCI_MSIX : +if (posted_msi_supported ()) { +prepare_irte_posted (irte); + data->irq_2_iommu.posted_msi =1 ; + } +set_msi_sid (irte,pci_real_dma_dev (msi_desc_to_pci_dev (info->desc)));break ;
@@ -1361,7 +1461,12 @@ static int intel_irq_remapping_alloc(struct irq_domain *domain, irq_data->hwirq = (index <<16 ) + i; irq_data->chip_data = ird; - irq_data->chip = &intel_ir_chip; +if (posted_msi_supported () && + ((info->type ==X86_IRQ_ALLOC_TYPE_PCI_MSI ) || + (info->type ==X86_IRQ_ALLOC_TYPE_PCI_MSIX ))) + irq_data->chip = &intel_ir_chip_post_msi; +else + irq_data->chip = &intel_ir_chip;intel_irq_remapping_prepare_irte (ird, irq_cfg, info, index, i);irq_set_status_flags (virq + i,IRQ_MOVE_PCNTXT ); }
I think there is an inherent race condition when intremap=posted_msi and the IRQ subsystem resends pending interrupts via __apic_send_IPI().
In detail:
intremap=posted_msi does not process vectors for which the corresponding bit in the PIR register is set.
Now say that, for whatever reason, the IRQ infrastructure intercepts an interrupt marking it as PENDING. . handle_edge_irq() and many other places in kernel/irq have sections of code like this:
if (!irq_may_run(desc)) {
desc->istate |= IRQS_PENDING;
mask_ack_irq(desc);
goto out_unlock;
}Then eventually check_irq_resend() will try to resend pending interrupts
desc->istate &= ~IRQS_PENDING;
if (!try_retrigger(desc))
err = irq_sw_resend(desc);try_retrigger() on x86 eventually calls apic_retrigger_irq() which uses __apic_send_IPI(). Unfortunately the latter does not seem to set the 'vector' bit in the PIR (nor sends the POSTED_MSI interrupt) thus potentially causing a lost interrupt unless there is some other spontaneous interrupt coming from the device.
I could verify the stall (forcing the path that sets IRQS_PENDING), and could verify that the patch below fixes the problem
static int apic_retrigger_irq(struct irq_data *irqd)
{
struct apic_chip_data *apicd = apic_chip_data(irqd);
unsigned long flags;+ uint vec;
raw_spin_lock_irqsave(&vector_lock, flags);
+ vec = apicd->vector;
+ if (posted_msi_supported() &&
+ vec >= FIRST_EXTERNAL_VECTOR && vec < FIRST_SYSTEM_VECTOR) {
+ struct pi_desc *pid = per_cpu_ptr(&posted_msi_pi_desc, apicd->cpu);
+ set_bit(vec, (unsigned long *)pid->pir64);
+ __apic_send_IPI(apicd->cpu, POSTED_MSI_NOTIFICATION_VECTOR);
+ } else {
__apic_send_IPI(apicd->cpu, apicd->vector);
+ }
raw_spin_unlock_irqrestore(&vector_lock, flags);
return 1;
}Am I missing something ? any better fix ?
cheers
luigi
Quoted 23 lines · Luigi Rizzo
I think there is an inherent race condition when intremap=posted_msi and the IRQ subsystem resends pending interrupts via
__apic_send_IPI().In detail:
intremap=posted_msi does not process vectors for which the corresponding bit in the PIR register is set.Now say that, for whatever reason, the IRQ infrastructure intercepts an interrupt marking it as PENDING. .
handle_edge_irq()and many other places in kernel/irq have sections of code like this:if (!irq_may_run(desc)) { desc->istate |= IRQS_PENDING; mask_ack_irq(desc); goto out_unlock; }Then eventually
check_irq_resend()will try to resend pending interruptsdesc->istate &= ~IRQS_PENDING; if (!try_retrigger(desc)) err = irq_sw_resend(desc);
Quoted 5 lines
try_retrigger()on x86 eventually callsapic_retrigger_irq()which uses__apic_send_IPI(). Unfortunately the latter does not seem to set the 'vector' bit in the PIR (nor sends thePOSTED_MSIinterrupt) thus potentially causing a lost interrupt unless there is some other spontaneous interrupt coming from the device.
It sends an IPI to the actual vector, which invokes the handler directly. That works only once because the remap interrupt chip does not issue an EOI, so the vector becomes stale.... Clearly nobody ever tested that code.
Quoted 2 lines
I could verify the stall (forcing the path that sets
IRQS_PENDING), and could verify that the patch below fixes the problem
You can do the same with
echo trigger > /sys/kernel/debug/irq/irqs/$IRQNR
Review
Quoted 11 lines
static int
apic_retrigger_irq(struct irq_data *irqd){ struct apic_chip_data *apicd = apic_chip_data(irqd); unsigned long flags;(quoted diff) +5
-0+ uint vec; raw_spin_lock_irqsave(&vector_lock, flags); + vec = apicd->vector; + if (posted_msi_supported() && + vec >= FIRST_EXTERNAL_VECTOR && vec < FIRST_SYSTEM_VECTOR) { + struct pi_desc *pid = per_cpu_ptr(&posted_msi_pi_desc, apicd->cpu);
Won't build with CONFIG_X86_POSTED_MSI=n
Review
Quoted 2 lines
(quoted diff) +2
-0+ set_bit(vec, (unsigned long *)pid->pir64); + __apic_send_IPI(apicd->cpu, POSTED_MSI_NOTIFICATION_VECTOR);
That's wrong as it affects all interrupts and not just the MSI ones which are going through the posted vector. As the interrupt chip of those other ones is different, that retriggered handler would issue two EOIs, which is not what we want.
That's what the interrupt hierarchy is for.
While that _is_ solvable, there is another issue which is not solvable by that at all.
If there is ever a stray interrupt on such a vector, then the related APIC ISR bit becomes stale due to the lack of EOI, which means all vectors below that vector are never serviced again. Unlikely to happen, but if it happens it's not debuggable at all.
So instead of playing games with the PIR, this can be actually solved for both cases. See below.
Thanks,
tglx
--- arch/x86/include/asm/irq_remapping.h | 7 +++++++ arch/x86/kernel/irq.c | 28 +++++++++++++++++++++++----- drivers/iommu/intel/irq_remapping.c | 8 ++++---- 3 files changed, 34 insertions(+), 9 deletions(-)
arch/x86/include/asm/irq_remapping.h +7 -0
@@ -87,4 +87,11 @@ static inline void panic_if_irq_remap(co }#endif /* CONFIG_IRQ_REMAP */ + +#ifdef CONFIG_X86_POSTED_MSI +void intel_ack_posted_msi_irq (struct irq_data *irqd); +#else +#define intel_ack_posted_msi_irq (irqd) irq_move_irq(irdq) +#endif +#endif /* __X86_IRQ_REMAPPING_H */
arch/x86/kernel/irq.c +23 -5
@@ -396,6 +396,7 @@ DEFINE_IDTENTRY_SYSVEC_SIMPLE(sysvec_kvm/* Posted Interrupt Descriptors for coalesced MSIs to be posted */ DEFINE_PER_CPU_ALIGNED(struct pi_desc, posted_msi_pi_desc); +static DEFINE_PER_CPU_CACHE_HOT(bool, posted_msi_handler_active);void intel_posted_msi_init (void ) {
@@ -413,6 +414,24 @@ void intel_posted_msi_init(void)this_cpu_write (posted_msi_pi_desc.ndst, destination); } +void intel_ack_posted_msi_irq (struct irq_data *irqd) +{ +irq_move_irq (irqd); +/* +* Handle the rare case that irq_retrigger() raised the actual +* assigned vector on the target CPU, which means that it was not +* invoked via the posted MSI handler below. In that case APIC EOI +* is required as otherwise the ISR entry becomes stale and lower +* priority interrupts are never going to be delivered after that. +* +* If the posted handler invoked the device interrupt handler then +* the EOI would be premature because it would acknowledge the +* posted vector. +*/ +if (unlikely (!this_cpu_read (posted_msi_handler_active))) +apic_eoi (); +} + static __always_inline bool handle_pending_pir(unsigned long *pir, struct pt_regs *regs) {unsigned long pir_copy[NR_PIR_WORDS ];
@@ -439,12 +458,10 @@ static __always_inline bool handle_pendi*/ DEFINE_IDTENTRY_SYSVEC (sysvec_posted_msi_notification) { +struct pi_desc *pid =this_cpu_ptr (&posted_msi_pi_desc);struct pt_regs *old_regs =set_irq_regs (regs); -struct pi_desc *pid; -int i =0 ; - - pid =this_cpu_ptr (&posted_msi_pi_desc); +this_cpu_write (posted_msi_handler_active, true);inc_irq_stat (posted_msi_notification_count);irq_enter ();
@@ -453,7 +470,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_posted_msi* after clearing the outstanding notification bit. Hence, at most * MAX_POSTED_MSI_COALESCING_LOOP - 1 loops are executed here. */ -while (++i <MAX_POSTED_MSI_COALESCING_LOOP ) { +for (int i =1 ; i <MAX_POSTED_MSI_COALESCING_LOOP ; i++) {if (!handle_pending_pir (pid->pir, regs))break ; }
@@ -473,6 +490,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_posted_msiapic_eoi ();irq_exit (); +this_cpu_write (posted_msi_handler_active, false); set_irq_regs(old_regs); }#endif /* X86_POSTED_MSI */
drivers/iommu/intel/irq_remapping.c +4 -4
@@ -1303,17 +1303,17 @@ static struct irq_chip intel_ir_chip = {* irq_enter(); * handle_edge_irq() * irq_chip_ack_parent() -* irq_move_irq(); // No EOI +* intel_ack_posted_msi_irq(); // No EOI * handle_irq_event() * driver_handler() * handle_edge_irq() * irq_chip_ack_parent() -* irq_move_irq(); // No EOI +* intel_ack_posted_msi_irq(); // No EOI * handle_irq_event() * driver_handler() * handle_edge_irq() * irq_chip_ack_parent() -* irq_move_irq(); // No EOI +* intel_ack_posted_msi_irq(); // No EOI * handle_irq_event() * driver_handler() * apic_eoi()
@@ -1322,7 +1322,7 @@ static struct irq_chip intel_ir_chip = {*/ static struct irq_chip intel_ir_chip_post_msi = { .name ="INTEL-IR-POST" , - .irq_ack = irq_move_irq, + .irq_ack = intel_ack_posted_msi_irq, .irq_set_affinity = intel_ir_set_affinity, .irq_compose_msi_msg = intel_ir_compose_msi_msg, .irq_set_vcpu_affinity = intel_ir_set_vcpu_affinity,
Luigi Rizzo
Tested-byQuoted 5 lines · Thomas Gleixner
Quoted 3 lines · Luigi Rizzo
I think there is an inherent race condition when intremap=posted_msi and the IRQ subsystem resends pending interrupts via
__apic_send_IPI().
...
...
Quoted 4 lines
It sends an IPI to the actual vector, which invokes the handler directly. That works only once because the remap interrupt chip does not issue an EOI, so the vector becomes stale.... Clearly nobody ever tested that code.
Thanks for clarifying that the problem is the missing EOI.
Quoted 3 lines
...
So instead of playing games with the PIR, this can be actually solved for both cases. See below.
Thanks, I verified that your patch fixes the problem (and also works with software moderation)
cheers
luigi