PCI: MSI parent domain conversion
Subsystems (19)
cover PCI: MSI parent domain conversion
The initial implementation of PCI/MSI interrupt domains in the hierarchical interrupt domain model used a shortcut by providing a global PCI/MSI domain.
This works because the PCI/MSI[X] hardware is standardized and uniform, but it violates the basic design principle of hierarchical interrupt domains:
Each hardware block involved in the interrupt delivery chain should have a separate interrupt domain.
For PCI/MSI[X], the interrupt controller is per PCI device and not a global made-up entity.
Unsurprisingly, the shortcut turned out to have downsides as it does not allow dynamic allocation of interrupt vectors after initialization and it prevents supporting IMS on PCI. For further details, see:
https://lore.kernel.org/lkml/[email protected]/
The solution is implementing per device MSI domains, this means the entities which provide global PCI/MSI domain so far have to implement MSI parent domain functionality instead.
This series converts the PCI controller drivers to implement MSI parent domain.
drivers/pci/Kconfig | 1 + drivers/pci/controller/Kconfig | 11 + drivers/pci/controller/dwc/Kconfig | 1 + .../pci/controller/dwc/pcie-designware-host.c | 68 ++---- drivers/pci/controller/dwc/pcie-designware.h | 1 - drivers/pci/controller/mobiveil/Kconfig | 1 + .../controller/mobiveil/pcie-mobiveil-host.c | 42 ++-- .../pci/controller/mobiveil/pcie-mobiveil.h | 1 - drivers/pci/controller/pci-aardvark.c | 59 ++--- drivers/pci/controller/pci-hyperv.c | 98 ++++++-- drivers/pci/controller/pcie-altera-msi.c | 44 ++-- drivers/pci/controller/pcie-brcmstb.c | 44 ++-- drivers/pci/controller/pcie-iproc-msi.c | 45 ++-- drivers/pci/controller/pcie-mediatek-gen3.c | 67 ++--- drivers/pci/controller/pcie-mediatek.c | 46 ++-- drivers/pci/controller/pcie-rcar-host.c | 69 ++---- drivers/pci/controller/pcie-xilinx-dma-pl.c | 48 ++-- drivers/pci/controller/pcie-xilinx-nwl.c | 45 ++-- drivers/pci/controller/pcie-xilinx.c | 55 +++-- drivers/pci/controller/plda/Kconfig | 1 + drivers/pci/controller/plda/pcie-plda-host.c | 44 ++-- drivers/pci/controller/plda/pcie-plda.h | 1 - drivers/pci/controller/vmd.c | 229 +++++++++--------- 23 files changed, 504 insertions(+), 517 deletions(-)
Signature · 1 line
2.39.5
PATCH 1/16 PCI: dwc: Switch to msi_create_parent_irq_domain()
Move away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: Jingoo Han <[email protected]>
--- drivers/pci/controller/dwc/Kconfig | 1 + .../pci/controller/dwc/pcie-designware-host.c | 68 +++++++------------ drivers/pci/controller/dwc/pcie-designware.h | 1 - 3 files changed, 25 insertions(+), 45 deletions(-)
drivers/pci/controller/dwc/Kconfig +1 -0
@@ -19,6 +19,7 @@ config PCIE_DW_DEBUGFS config PCIE_DW_HOST bool select PCIE_DW + select IRQ_MSI_LIB config PCIE_DW_EP bool
drivers/pci/controller/dwc/pcie-designware-host.c +24 -44
97 lines not on this page · raw message
drivers/pci/controller/dwc/pcie-designware.h +0 -1
@@ -417,7 +417,6 @@ struct dw_pcie_rp {const struct dw_pcie_host_ops *ops;int msi_irq[MAX_MSI_CTRLS ];struct irq_domain *irq_domain; -struct irq_domain *msi_domain;dma_addr_t msi_data;struct irq_chip *msi_irq_chip;u32 num_vectors;
Signature · 1 line
2.39.5
Re: [PATCH 01/16] PCI: dwc: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
PATCH 2/16 PCI: mobiveil: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoMove away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: Karthikeyan Mitran <[email protected]>
Cc: Hou Zhiqiang <[email protected]>
--- drivers/pci/controller/mobiveil/Kconfig | 1 + .../controller/mobiveil/pcie-mobiveil-host.c | 42 ++++++++++--------- .../pci/controller/mobiveil/pcie-mobiveil.h | 1 - 3 files changed, 23 insertions(+), 21 deletions(-)
drivers/pci/controller/mobiveil/Kconfig +1 -0
@@ -9,6 +9,7 @@ config PCIE_MOBIVEIL config PCIE_MOBIVEIL_HOST bool depends on PCI_MSI + select IRQ_MSI_LIB select PCIE_MOBIVEIL config PCIE_LAYERSCAPE_GEN4
drivers/pci/controller/mobiveil/pcie-mobiveil-host.c +22 -20
63 lines not on this page · raw message
drivers/pci/controller/mobiveil/pcie-mobiveil.h +0 -1
@@ -135,7 +135,6 @@struct mobiveil_msi {/* MSI information */ struct mutex lock;/* protect bitmap variable */ -struct irq_domain *msi_domain;struct irq_domain *dev_domain;phys_addr_t msi_pages_phys;int num_of_vectors;
Signature · 1 line
2.39.5
Re: [PATCH 02/16] PCI: mobiveil: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
PATCH 3/16 PCI: aardvark: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoMove away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: Thomas Petazzoni <[email protected]>
Cc: "Pali Rohár" <[email protected]>
Cc: [email protected]
--- drivers/pci/controller/Kconfig | 1 + drivers/pci/controller/pci-aardvark.c | 59 +++++++++++---------------- 2 files changed, 24 insertions(+), 36 deletions(-)
drivers/pci/controller/Kconfig +1 -0
@@ -13,6 +13,7 @@ config PCI_AARDVARK depends on OF depends on PCI_MSI select PCI_BRIDGE_EMUL + select IRQ_MSI_LIB help Add support for Aardvark 64bit PCIe Host Controller. This controller is part of the South Bridge of the Marvel Armada
drivers/pci/controller/pci-aardvark.c +23 -36
97 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH 03/16] PCI: aardvark: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
PATCH 4/16 PCI: altera-msi: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoMove away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: Joyce Ooi <[email protected]>
--- drivers/pci/controller/Kconfig | 1 + drivers/pci/controller/pcie-altera-msi.c | 44 +++++++++++------------- 2 files changed, 22 insertions(+), 23 deletions(-)
drivers/pci/controller/Kconfig +1 -0
@@ -30,6 +30,7 @@ config PCIE_ALTERA_MSI tristate "Altera PCIe MSI feature" depends on PCIE_ALTERA depends on PCI_MSI + select IRQ_MSI_LIB help Say Y here if you want PCIe MSI support for the Altera FPGA. This MSI driver supports Altera MSI to GIC controller IP.
drivers/pci/controller/pcie-altera-msi.c +21 -23
78 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH 04/16] PCI: altera-msi: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
PATCH 5/16 PCI: brcmstb: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoMove away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: Florian Fainelli <[email protected]>
Cc: Broadcom internal kernel review list <[email protected]>
Cc: Jim Quinlan <[email protected]>
Cc: Nicolas Saenz Julienne <[email protected]>
Cc: [email protected]
Cc: [email protected]
--- drivers/pci/controller/Kconfig | 1 + drivers/pci/controller/pcie-brcmstb.c | 44 +++++++++++++-------------- 2 files changed, 23 insertions(+), 22 deletions(-)
drivers/pci/controller/Kconfig +1 -0
@@ -64,6 +64,7 @@ config PCIE_BRCMSTB BMIPS_GENERIC || COMPILE_TEST depends on OF depends on PCI_MSI + select IRQ_MSI_LIB default ARCH_BRCMSTB || BMIPS_GENERIC help Say Y here to enable PCIe host controller support for
drivers/pci/controller/pcie-brcmstb.c +22 -22
77 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH 05/16] PCI: brcmstb: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
Signature · 1 line
Florian
Re: [PATCH 05/16] PCI: brcmstb: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoQuoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
PATCH 6/16 PCI: iproc: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoMove away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: Ray Jui <[email protected]>
Cc: Scott Branden <[email protected]>
Cc: Broadcom internal kernel review list <[email protected]>
Cc: [email protected]
--- drivers/pci/controller/Kconfig | 1 + drivers/pci/controller/pcie-iproc-msi.c | 45 ++++++++++++------------- 2 files changed, 22 insertions(+), 24 deletions(-)
drivers/pci/controller/Kconfig +1 -0
@@ -101,6 +101,7 @@ config PCIE_IPROC_MSI bool "Broadcom iProc PCIe MSI support" depends on PCIE_IPROC_PLATFORM || PCIE_IPROC_BCMA depends on PCI_MSI + select IRQ_MSI_LIB default ARCH_BCM_IPROC help Say Y here if you want to enable MSI support for Broadcom's iProc
drivers/pci/controller/pcie-iproc-msi.c +21 -24
84 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH 06/16] PCI: iproc: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
Signature · 1 line
Florian
Re: [PATCH 06/16] PCI: iproc: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoQuoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
PATCH 7/16 PCI: mediatek-gen3: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoMove away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: Ryder Lee <[email protected]>
Cc: Jianjun Wang <[email protected]>
Cc: Matthias Brugger <[email protected]>
Cc: AngeloGioacchino Del Regno <[email protected]>
Cc: [email protected]
Cc: [email protected]
--- drivers/pci/controller/Kconfig | 1 + drivers/pci/controller/pcie-mediatek-gen3.c | 67 ++++++++------------- 2 files changed, 26 insertions(+), 42 deletions(-)
drivers/pci/controller/Kconfig +1 -0
@@ -203,6 +203,7 @@ config PCIE_MEDIATEK_GEN3 tristate "MediaTek Gen3 PCIe controller" depends on ARCH_AIROHA || ARCH_MEDIATEK || COMPILE_TEST depends on PCI_MSI + select IRQ_MSI_LIB help Adds support for PCIe Gen3 MAC controller for MediaTek SoCs. This PCIe controller is compatible with Gen3, Gen2 and Gen1 speed,
drivers/pci/controller/pcie-mediatek-gen3.c +25 -42
112 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH 07/16] PCI: mediatek-gen3: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
PATCH 8/16 PCI: mediatek: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoMove away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: Ryder Lee <[email protected]>
Cc: Jianjun Wang <[email protected]>
Cc: Matthias Brugger <[email protected]>
Cc: AngeloGioacchino Del Regno <[email protected]>
Cc: [email protected]
Cc: [email protected]
--- drivers/pci/controller/Kconfig | 1 + drivers/pci/controller/pcie-mediatek.c | 46 ++++++++++++-------------- 2 files changed, 23 insertions(+), 24 deletions(-)
drivers/pci/controller/Kconfig +1 -0
@@ -195,6 +195,7 @@ config PCIE_MEDIATEK depends on ARCH_AIROHA || ARCH_MEDIATEK || COMPILE_TEST depends on OF depends on PCI_MSI + select IRQ_MSI_LIB help Say Y here if you want to enable PCIe controller support on MediaTek SoCs.
drivers/pci/controller/pcie-mediatek.c +22 -24
88 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH 08/16] PCI: mediatek: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
PATCH 9/16 PCI: rcar-host: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoMove away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: Marek Vasut <[email protected]>
Cc: Yoshihiro Shimoda <[email protected]>
Cc: Geert Uytterhoeven <[email protected]>
Cc: Magnus Damm <[email protected]>
Cc: [email protected]
--- drivers/pci/controller/Kconfig | 1 + drivers/pci/controller/pcie-rcar-host.c | 69 +++++++++---------------- 2 files changed, 26 insertions(+), 44 deletions(-)
drivers/pci/controller/Kconfig +1 -0
@@ -243,6 +243,7 @@ config PCIE_RCAR_HOST bool "Renesas R-Car PCIe controller (host mode)" depends on ARCH_RENESAS || COMPILE_TEST depends on PCI_MSI + select IRQ_MSI_LIB help Say Y here if you want PCIe controller support on R-Car SoCs in host mode.
drivers/pci/controller/pcie-rcar-host.c +25 -44
101 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH 09/16] PCI: rcar-host: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
PATCH 10/16 PCI: xilinx-xdma: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoMove away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: Michal Simek <[email protected]>
Cc: [email protected]
--- drivers/pci/controller/Kconfig | 1 + drivers/pci/controller/pcie-xilinx-dma-pl.c | 48 +++++++++------------ 2 files changed, 22 insertions(+), 27 deletions(-)
drivers/pci/controller/Kconfig +1 -0
@@ -331,6 +331,7 @@ config PCIE_XILINX_DMA_PL depends on ARCH_ZYNQMP || COMPILE_TEST depends on PCI_MSI select PCI_HOST_COMMON + select IRQ_MSI_LIB help Say 'Y' here if you want kernel support for the Xilinx PL DMA PCIe host bridge. The controller is a Soft IP which can act as
drivers/pci/controller/pcie-xilinx-dma-pl.c +21 -27
83 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH 10/16] PCI: xilinx-xdma: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
PATCH 11/16 PCI: xilinx-nwl: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoMove away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: Michal Simek <[email protected]>
Cc: [email protected]
--- drivers/pci/controller/Kconfig | 1 + drivers/pci/controller/pcie-xilinx-nwl.c | 45 ++++++++++++------------ 2 files changed, 24 insertions(+), 22 deletions(-)
drivers/pci/controller/Kconfig +1 -0
@@ -342,6 +342,7 @@ config PCIE_XILINX_NWL bool "Xilinx NWL PCIe controller" depends on ARCH_ZYNQMP || COMPILE_TEST depends on PCI_MSI + select IRQ_MSI_LIB help Say 'Y' here if you want kernel support for Xilinx NWL PCIe controller. The controller can act as Root Port
drivers/pci/controller/pcie-xilinx-nwl.c +23 -22
75 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH 11/16] PCI: xilinx-nwl: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
PATCH 12/16 PCI: xilinx: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoMove away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: Michal Simek <[email protected]>
Cc: [email protected]
--- drivers/pci/controller/Kconfig | 1 + drivers/pci/controller/pcie-xilinx.c | 55 ++++++++++++++++------------ 2 files changed, 32 insertions(+), 24 deletions(-)
drivers/pci/controller/Kconfig +1 -0
@@ -322,6 +322,7 @@ config PCIE_XILINX bool "Xilinx AXI PCIe controller" depends on OF depends on PCI_MSI + select IRQ_MSI_LIB help Say 'Y' here if you want kernel to support the Xilinx AXI PCIe Host Bridge driver.
drivers/pci/controller/pcie-xilinx.c +31 -24
87 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH 12/16] PCI: xilinx: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
PATCH 13/16 PCI: plda: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoMove away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: Daire McNamara <[email protected]>
--- drivers/pci/controller/plda/Kconfig | 1 + drivers/pci/controller/plda/pcie-plda-host.c | 44 ++++++++++---------- drivers/pci/controller/plda/pcie-plda.h | 1 - 3 files changed, 23 insertions(+), 23 deletions(-)
drivers/pci/controller/plda/Kconfig +1 -0
@@ -5,6 +5,7 @@ menu "PLDA-based PCIe controllers" config PCIE_PLDA_HOST bool + select IRQ_MSI_LIB config PCIE_MICROCHIP_HOST tristate "Microchip AXI PCIe controller"
drivers/pci/controller/plda/pcie-plda-host.c +22 -22
73 lines not on this page · raw message
drivers/pci/controller/plda/pcie-plda.h +0 -1
@@ -164,7 +164,6 @@ struct plda_pcie_host_ops {struct plda_msi {struct mutex lock;/* Protect used bitmap */ -struct irq_domain *msi_domain;struct irq_domain *dev_domain;u32 num_vectors;u64 vector_phy;
Signature · 1 line
2.39.5
Re: [PATCH 13/16] PCI: plda: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
PATCH 14/16 PCI: hv: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoMove away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: K. Y. Srinivasan <[email protected]>
Cc: Haiyang Zhang <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Dexuan Cui <[email protected]>
Cc: [email protected]
--- drivers/pci/Kconfig | 1 + drivers/pci/controller/pci-hyperv.c | 98 +++++++++++++++++++++++------ 2 files changed, 80 insertions(+), 19 deletions(-)
drivers/pci/Kconfig +1 -0
@@ -223,6 +223,7 @@ config PCI_HYPERV tristate "Hyper-V PCI Frontend" depends on ((X86 && X86_64) || ARM64) && HYPERV && PCI_MSI && SYSFS select PCI_HYPERV_INTERFACE + select IRQ_MSI_LIB help The PCI device frontend driver allows the kernel to import arbitrary PCI devices from a PCI backend to support PCI driver domains.
drivers/pci/controller/pci-hyperv.c +79 -19
152 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
RE: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoFrom: Nam Cao <[email protected]> Sent: Thursday, June 26, 2025 7:48 AM
Quoted 3 lines
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().
From a build standpoint, this patch does not apply cleanly to linux-next20250630. See also an issue below where a needed irq function isn't exported.
At runtime, I've done basic smoke testing on an x86 VM in the Azure cloud that has a Mellanox NIC VF and two NVMe devices as PCI devices. So far everything looks good. But I'm still doing additional testing, and I want to also test on an ARM64 VM. Please give me another day or two to be completely satisfied.
Michael Kelley
Quoted 148 lines
Signed-off-by Nam Cao <[email protected]>---
Cc: K. Y. Srinivasan <[email protected]>
Cc: Haiyang Zhang <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Dexuan Cui <[email protected]>
Cc: [email protected]--- drivers/pci/Kconfig | 1 + drivers/pci/controller/pci-hyperv.c | 98 +++++++++++++++++++++++------ 2 files changed, 80 insertions(+), 19 deletions(-)drivers/pci/Kconfig +1
-0@@ -223,6 +223,7 @@ config PCI_HYPERV tristate "Hyper-V PCI Frontend" depends on ((X86 && X86_64) || ARM64) && HYPERV && PCI_MSI && SYSFS select PCI_HYPERV_INTERFACE + select IRQ_MSI_LIB help The PCI device frontend driver allows the kernel to import arbitrary PCI devices from a PCI backend to support PCI driver domains.drivers/pci/controller/pci-hyperv.c +40
-4@@ -44,6 +44,7 @@#include <linux/delay.h> #include <linux/semaphore.h> #include <linux/irq.h> +#include <linux/irqchip/irq-msi-lib.h> #include <linux/msi.h> #include <linux/hyperv.h> #include <linux/refcount.h> @@ -508,7 +509,6 @@ struct hv_pcibus_device {struct list_head children;struct list_head dr_list; -struct msi_domain_info msi_info;struct irq_domain *irq_domain;struct workqueue_struct *wq;@@ -1687,7 +1687,7 @@ static void hv_msi_free(struct irq_domain *domain, struct msi_domain_info *info,struct msi_desc *msi =irq_data_get_msi_desc (irq_data); pdev =msi_desc_to_pci_dev (msi); - hbus = info->data; + hbus = domain->host_data; int_desc =irq_data_get_irq_chip_data (irq_data);if (!int_desc)return ;@@ -1705,7 +1705,6 @@ static void hv_msi_free(struct irq_domain *domain, struct msi_domain_info *info,static void hv_irq_mask (struct irq_data *data) { -pci_msi_mask_irq (data);if (data->parent_data->chip->irq_mask)irq_chip_mask_parent (data); }@@ -1716,7 +1715,6 @@ static void hv_irq_unmask(struct irq_data *data)if (data->parent_data->chip->irq_unmask)irq_chip_unmask_parent (data); -pci_msi_unmask_irq (data); }struct compose_comp_ctxt {@@ -2101,6 +2099,44 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) msg->data =0 ; } +static bool hv_pcie_init_dev_msi_info (struct device *dev,struct irq_domain *domain, +struct irq_domain *real_parent,struct msi_domain_info *info) +{ +struct irq_chip *chip = info->chip; + +if (!msi_lib_init_dev_msi_info (dev, domain, real_parent, info)) +return false; + + info->ops->msi_prepare = hv_msi_prepare; + + chip->irq_set_affinity = irq_chip_set_affinity_parent; + +if (IS_ENABLED (CONFIG_X86 )) + chip->flags |=IRQCHIP_MOVE_DEFERRED ; + +return true; +} + +#define HV_PCIE_MSI_FLAGS_REQUIRED (MSI_FLAG_USE_DEF_DOM_OPS | \ + MSI_FLAG_USE_DEF_CHIP_OPS | \ + MSI_FLAG_PCI_MSI_MASK_PARENT) +#define HV_PCIE_MSI_FLAGS_SUPPORTED (MSI_FLAG_MULTI_PCI_MSI | \ + MSI_FLAG_PCI_MSIX | \ + MSI_GENERIC_FLAGS_MASK) + +static const struct msi_parent_ops hv_pcie_msi_parent_ops = { + .required_flags =HV_PCIE_MSI_FLAGS_REQUIRED , + .supported_flags =HV_PCIE_MSI_FLAGS_SUPPORTED , + .bus_select_token =DOMAIN_BUS_PCI_MSI , +#ifdef CONFIG_X86 + .chip_flags =MSI_CHIP_FLAG_SET_ACK , +#elif defined(CONFIG_ARM64) + .chip_flags =MSI_CHIP_FLAG_SET_EOI , +#endif + .prefix ="HV-" , + .init_dev_msi_info = hv_pcie_init_dev_msi_info, +}; +/* HW Interrupt Chip Descriptor */ static struct
irq_chiphv_msi_irq_chip= {drivers/pci/controller/pci-hyperv.c +16
-4.name = "Hyper-V PCIe MSI",@@ -2108,7 +2144,6 @@ static struct irq_chip hv_msi_irq_chip = { .irq_set_affinity = irq_chip_set_affinity_parent,#ifdef CONFIG_X86 .irq_ack = irq_chip_ack_parent, - .flags = IRQCHIP_MOVE_DEFERRED,#elif defined(CONFIG_ARM64 ) .irq_eoi = irq_chip_eoi_parent,#endif @@ -2116,9 +2151,37 @@ static struct irq_chip hv_msi_irq_chip = { .irq_unmask = hv_irq_unmask, }; -static struct msi_domain_ops hv_msi_ops = { - .msi_prepare = hv_msi_prepare, - .msi_free = hv_msi_free, +static int hv_pcie_domain_alloc (struct irq_domain *d,unsigned int virq,unsigned int nr_irqs, +void *arg) +{ +/* TODO: move the content of hv_compose_msi_msg() in here */ +int ret; + + ret =irq_domain_alloc_irqs_parent (d, virq, nr_irqs, arg); +if (ret <0 ) +return ret; + +for (int i =0 ; i < nr_irqs; i++) { +irq_domain_set_info (d, virq + i,0 , &hv_msi_irq_chip,NULL ,FLOW_HANDLER ,NULL , +FLOW_NAME ); + } + +return 0 ;
drivers/pci/controller/pci-hyperv.c static struct irq_chip hv_msi_irq_chip = { · line 2179
Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10
-0+} + +static void hv_pcie_domain_free (struct irq_domain *d,unsigned int virq,unsigned int nr_irqs) +{ +struct msi_domain_info *info = d->host_data; + +for (int i =0 ; i < nr_irqs; i++) +hv_msi_free (d, info, virq + i); + +irq_domain_free_irqs_top (d, virq, nr_irqs);
This code can be built as a module, so irq_domain_free_irqs_top() needs to be exported, which it currently is not.
Quoted 37 lines
drivers/pci/controller/pci-hyperv.c +13
-12+} + +static const struct irq_domain_ops hv_pcie_domain_ops = { + .alloc = hv_pcie_domain_alloc, + .free = hv_pcie_domain_free, };/** @@ -2136,17 +2199,14 @@ static struct msi_domain_ops hv_msi_ops = {*/ static int hv_pcie_init_irq_domain (struct hv_pcibus_device *hbus) { - hbus->msi_info.chip = &hv_msi_irq_chip; - hbus->msi_info.ops = &hv_msi_ops; - hbus->msi_info.flags = (MSI_FLAG_USE_DEF_DOM_OPS | -MSI_FLAG_USE_DEF_CHIP_OPS |MSI_FLAG_MULTI_PCI_MSI | -MSI_FLAG_PCI_MSIX ); - hbus->msi_info.handler =FLOW_HANDLER ; - hbus->msi_info.handler_name =FLOW_NAME ; - hbus->msi_info.data = hbus; - hbus->irq_domain =pci_msi_create_irq_domain (hbus->fwnode, - &hbus->msi_info, -hv_pci_get_root_domain ()); +struct irq_domain_info info = { + .fwnode = hbus->fwnode, + .ops = &hv_pcie_domain_ops, + .host_data = hbus, + .parent =hv_pci_get_root_domain (), + }; + + hbus->irq_domain =msi_create_parent_irq_domain (&info, &hv_pcie_msi_parent_ops);if (!hbus->irq_domain) {dev_err (&hbus->hdev->device,"Failed to build an MSI IRQ domain\n" ); --2.39.5
RE: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
Quoted 8 lines · Michael Kelley
From: Nam Cao <[email protected]> Sent: Thursday, June 26, 2025 7:48 AM
Quoted 3 lines
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().From a build standpoint, this patch does not apply cleanly to linux-next20250630. See also an issue below where a needed irq function isn't exported.
Does it conflict against the PCI tree?
Quoted 5 lines
At runtime, I've done basic smoke testing on an x86 VM in the Azure cloud that has a Mellanox NIC VF and two NVMe devices as PCI devices. So far everything looks good. But I'm still doing additional testing, and I want to also test on an ARM64 VM. Please give me another day or two to be completely satisfied.
Sure.
Quoted 11 lines
drivers/pci/controller/pci-hyperv.c static struct irq_chip hv_msi_irq_chip = { · line 2179
Quoted 8 lines
drivers/pci/controller/pci-hyperv.c +8
-0+static void hv_pcie_domain_free (struct irq_domain *d,unsigned int virq,unsigned int nr_irqs) +{ +struct msi_domain_info *info = d->host_data; + +for (int i =0 ; i < nr_irqs; i++) +hv_msi_free (d, info, virq + i); + +irq_domain_free_irqs_top (d, virq, nr_irqs);This code can be built as a module, so
irq_domain_free_irqs_top()needs to be exported, which it currently is not.
Nam, can you please create a seperate patch, which exports this and take care of the conflict?
Thanks,
tglx
RE: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
From: Thomas Gleixner <[email protected]> Sent: Thursday, July 3, 2025 1:00 PM
Quoted 12 lines · Michael Kelley
Quoted 8 lines
From: Nam Cao <[email protected]> Sent: Thursday, June 26, 2025 7:48 AM
Quoted 3 lines
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().From a build standpoint, this patch does not apply cleanly to linux-next20250630. See also an issue below where a needed irq function isn't exported.
Does it conflict against the PCI tree?
There's no conflict in the "next" or "for-linus" tags in
https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git/.
The conflict is with Patch 2 of this series:
which is in netdev/net-next.
Michael
Quoted 26 lines
Quoted 6 lines
At runtime, I've done basic smoke testing on an x86 VM in the Azure cloud that has a Mellanox NIC VF and two NVMe devices as PCI devices. So far everything looks good. But I'm still doing additional testing, and I want to also test on an ARM64 VM. Please give me another day or two to be completely satisfied.
Sure.
Quoted 11 lines
drivers/pci/controller/pci-hyperv.c static struct irq_chip hv_msi_irq_chip = { · line 2179
Quoted 8 lines
drivers/pci/controller/pci-hyperv.c +8
-0+static void hv_pcie_domain_free (struct irq_domain *d,unsigned int virq,unsigned int nr_irqs) +{ +struct msi_domain_info *info = d->host_data; + +for (int i =0 ; i < nr_irqs; i++) +hv_msi_free (d, info, virq + i); + +irq_domain_free_irqs_top (d, virq, nr_irqs);This code can be built as a module, so
irq_domain_free_irqs_top()needs to be exported, which it currently is not.Nam, can you please create a seperate patch, which exports this and take care of the conflict?
Thanks,
tglx
Re: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
Quoted 14 lines · Michael Kelley
From: Thomas Gleixner <[email protected]> Sent: Thursday, July 3, 2025 1:00 PM
Quoted 12 lines · Michael Kelley
Quoted 8 lines
From: Nam Cao <[email protected]> Sent: Thursday, June 26, 2025 7:48 AM
Quoted 3 lines
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().From a build standpoint, this patch does not apply cleanly to linux-next20250630. See also an issue below where a needed irq function isn't exported.
Does it conflict against the PCI tree?
Quoted 8 lines
There's no conflict in the "next" or "for-linus" tags in
https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git/.The conflict is with Patch 2 of this series:
which is in netdev/net-next.
I need some guidance here. If I make it apply cleanly to linux-next, it won't apply to pci tree.
I saw this type of conflict being resolved during merging to Linus's tree. Shouldn't we do the same for this case?
Quoted 8 lines
Michael
Quoted 6 lines
Quoted 6 lines
At runtime, I've done basic smoke testing on an x86 VM in the Azure cloud that has a Mellanox NIC VF and two NVMe devices as PCI devices. So far everything looks good. But I'm still doing additional testing, and I want to also test on an ARM64 VM. Please give me another day or two to be completely satisfied.
Good to hear, thanks!
Quoted 13 lines
Quoted 13 lines
Sure.
Quoted 11 lines
drivers/pci/controller/pci-hyperv.c static struct irq_chip hv_msi_irq_chip = { · line 2179
Quoted 8 lines
drivers/pci/controller/pci-hyperv.c +8
-0+static void hv_pcie_domain_free (struct irq_domain *d,unsigned int virq,unsigned int nr_irqs) +{ +struct msi_domain_info *info = d->host_data; + +for (int i =0 ; i < nr_irqs; i++) +hv_msi_free (d, info, virq + i); + +irq_domain_free_irqs_top (d, virq, nr_irqs);This code can be built as a module, so
irq_domain_free_irqs_top()needs to be exported, which it currently is not.
Quoted 2 lines
Quoted 2 lines
Nam, can you please create a seperate patch, which exports this and take care of the conflict?
Will do.
Best regards,
Nam
Re: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
Quoted 8 lines · Nam Cao
Quoted 1 line · Michael Kelley
which is in netdev/net-next.
I need some guidance here. If I make it apply cleanly to linux-next, it won't apply to pci tree.
I saw this type of conflict being resolved during merging to Linus's tree. Shouldn't we do the same for this case?
There are many ways to skin a cat. See my other reply.
RE: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
In reply to Michael KelleyQuoted 11 lines · Michael Kelley
From: Thomas Gleixner <[email protected]> Sent: Thursday, July 3, 2025 1:00 PM
Quoted 1 line
Does it conflict against the PCI tree?
There's no conflict in the "next" or "for-linus" tags in
https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git/.The conflict is with Patch 2 of this series:
which is in netdev/net-next.
That's a trivial one. There are two ways to handle it:
1) Take it through the PCI tree and provide a conflict resolution for linux-next and later for Linus as reference.
2) Route it through the net-next tree with an updated patch.
As there are no further dependencies (aside of the missing export which is needed anyway) it's obvious to pick #2 as it creates the least headaches. Assumed that the PCI folks have no objections.
Michael, as you have resolved the conflict already, can you please either take care of it yourself or provide the resolution here as reference for Nam?
Thanks,
tglx
RE: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
From: Thomas Gleixner <[email protected]> Sent: Thursday, July 3, 2025 2:22 PM
Quoted 21 lines · Michael Kelley
Quoted 11 lines
From: Thomas Gleixner <[email protected]> Sent: Thursday, July 3, 2025 1:00 PM
Quoted 1 line
Does it conflict against the PCI tree?
There's no conflict in the "next" or "for-linus" tags in
https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git/The conflict is with Patch 2 of this series:
which is in netdev/net-next.
That's a trivial one. There are two ways to handle it:
1) Take it through the PCI tree and provide a conflict resolution for linux-next and later for Linus as reference.
2) Route it through the net-next tree with an updated patch.
Quoted 7 lines
As there are no further dependencies (aside of the missing export which is needed anyway) it's obvious to pick #2 as it creates the least headaches. Assumed that the PCI folks have no objections.
Michael, as you have resolved the conflict already, can you please either take care of it yourself or provide the resolution here as reference for Nam?
I haven't resolved the conflict. As a shortcut for testing I just removed the conflicting patch since it is for a Microsoft custom NIC ("MANA") that's not in the configuration I'm testing with. I'll have to look more closely to figure out the resolution.
Separately, this patch (the switch to misc_create_parent_irq_domain) isn't working for Linux VMs on Hyper-V on ARM64. The initial symptom is that interrupts from the NVMe controller aren't getting handled and everything hangs. Here's the dmesg output:
[ 84.463419] hv_vmbus: registering driver hv_pci [ 84.463875] hv_pci abee639e-0b9d-49b7-9a07-c54ba8cd5734: PCI VMBus probing: Using version 0x10004 [ 84.464518] hv_pci abee639e-0b9d-49b7-9a07-c54ba8cd5734: PCI host bridge to bus 0b9d:00 [ 84.464529] pci_bus 0b9d:00: root bus resource [mem 0xfc0000000-0xfc00fffff window] [ 84.464531] pci_bus 0b9d:00: No busn resource found for root bus, will use [bus 00-ff] [ 84.465211] pci 0b9d:00:00.0: [1414:b111] type 00 class 0x010802 PCIe Endpoint [ 84.466657] pci 0b9d:00:00.0: BAR 0 [mem 0xfc0000000-0xfc00fffff 64bit] [ 84.481923] pci_bus 0b9d:00: busn_res: [bus 00-ff] end is updated to 00 [ 84.481936] pci 0b9d:00:00.0: BAR 0 [mem 0xfc0000000-0xfc00fffff 64bit]: assigned [ 84.482413] nvme nvme0: pci function 0b9d:00:00.0 [ 84.482513] nvme 0b9d:00:00.0: enabling device (0000 -> 0002) [ 84.556871] irq 17, desc: 00000000e8529819, depth: 0, count: 0, unhandled: 0 [ 84.556883] ->handle_irq(): 0000000062fa78bc, handle_bad_irq+0x0/0x270 [ 84.556892] ->irq_data.chip(): 00000000ba07832f, 0xffff00011469dc30 [ 84.556895] ->action(): 0000000069f160b3 [ 84.556896] ->action->handler(): 00000000e15d8191, nvme_irq+0x0/0x3e8 [ 172.307920] watchdog: BUG: soft lockup - CPU#6 stuck for 26s! [kworker/6:1H:195]
Everything looks normal up to the "irq 17" line. Meanwhile, the device probe path is waiting with this stack trace, which I suspect is the first Interaction with the NVMe controller:
[<0>] blk_execute_rq+0x1ec/0x348 [<0>] nvme_execute_rq+0x20/0x68 [<0>] __nvme_submit_sync_cmd+0xc8/0x170 [<0>] nvme_identify_ctrl.isra.0+0x90/0xf0 [<0>] nvme_init_identify+0x44/0xee0 [<0>] nvme_init_ctrl_finish+0x84/0x370 [<0>] nvme_probe+0x668/0x7d8 [<0>] local_pci_probe+0x48/0xd0 [<0>] pci_device_probe+0xd0/0x248 [<0>] really_probe+0xd4/0x388 [<0>] __driver_probe_device+0x90/0x1a8 [<0>] driver_probe_device+0x48/0x150 [<0>] __device_attach_driver+0xe0/0x1b8 [<0>] bus_for_each_drv+0x8c/0xf8 [<0>] __device_attach+0x104/0x1e8 [<0>] device_attach+0x1c/0x30 [<0>] pci_bus_add_device+0xe0/0x188 [<0>] pci_bus_add_devices+0x40/0x98 [<0>] hv_pci_probe+0x4b0/0x690 [pci_hyperv] [<0>] vmbus_probe+0x4c/0xb0 [hv_vmbus] [<0>] really_probe+0xd4/0x388 [<0>] __driver_probe_device+0x90/0x1a8 [<0>] driver_probe_device+0x48/0x150 [<0>] __driver_attach+0xe8/0x1c8 [<0>] bus_for_each_dev+0x80/0xf0 [<0>] driver_attach+0x2c/0x40 [<0>] bus_add_driver+0x118/0x260 [<0>] driver_register+0x68/0x138 [<0>] __vmbus_driver_register+0x70/0x98 [hv_vmbus] [<0>] init_hv_pci_drv+0x1b8/0xfff8 [pci_hyperv] [<0>] do_one_initcall+0x4c/0x2c8 [<0>] do_init_module+0x60/0x280 [<0>] load_module+0x2318/0x2448 [<0>] init_module_from_file+0x94/0xe0 [<0>] __arm64_sys_finit_module+0x228/0x3e8 [<0>] invoke_syscall+0x6c/0xf8 [<0>] el0_svc_common.constprop.0+0xc8/0xf0 [<0>] do_el0_svc+0x24/0x38 [<0>] el0_svc+0x40/0x1a0 [<0>] el0t_64_sync_handler+0xd0/0xe8 [<0>] el0t_64_sync+0x1b0/0x1b8
I'll try to figure out what's going wrong.
Michael
Re: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
Quoted 19 lines · Michael Kelley
I haven't resolved the conflict. As a shortcut for testing I just removed the conflicting patch since it is for a Microsoft custom NIC ("MANA") that's not in the configuration I'm testing with. I'll have to look more closely to figure out the resolution.
Separately, this patch (the switch to
misc_create_parent_irq_domain) isn't working for Linux VMs on Hyper-V on ARM64. The initial symptom is that interrupts from the NVMe controller aren't getting handled and everything hangs. Here's the dmesg output:[ 84.463419] hv_vmbus: registering driver hv_pci [ 84.463875] hv_pci abee639e-0b9d-49b7-9a07-c54ba8cd5734: PCI VMBus probing: Using version 0x10004 [ 84.464518] hv_pci abee639e-0b9d-49b7-9a07-c54ba8cd5734: PCI host bridge to bus 0b9d:00 [ 84.464529] pci_bus 0b9d:00: root bus resource [mem 0xfc0000000-0xfc00fffff window] [ 84.464531] pci_bus 0b9d:00: No busn resource found for root bus, will use [bus 00-ff] [ 84.465211] pci 0b9d:00:00.0: [1414:b111] type 00 class 0x010802 PCIe Endpoint [ 84.466657] pci 0b9d:00:00.0: BAR 0 [mem 0xfc0000000-0xfc00fffff 64bit] [ 84.481923] pci_bus 0b9d:00: busn_res: [bus 00-ff] end is updated to 00 [ 84.481936] pci 0b9d:00:00.0: BAR 0 [mem 0xfc0000000-0xfc00fffff 64bit]: assigned
Quoted 8 lines
[ 84.482413] nvme nvme0: pci function 0b9d:00:00.0 [ 84.482513] nvme 0b9d:00:00.0: enabling device (0000 -> 0002) [ 84.556871] irq 17, desc: 00000000e8529819, depth: 0, count: 0, unhandled: 0 [ 84.556883] ->handle_irq(): 0000000062fa78bc, handle_bad_irq+0x0/0x270 [ 84.556892] ->irq_data.chip(): 00000000ba07832f, 0xffff00011469dc30 [ 84.556895] ->action(): 0000000069f160b3 [ 84.556896] ->action->handler(): 00000000e15d8191, nvme_irq+0x0/0x3e8 [ 172.307920] watchdog: BUG: soft lockup - CPU#6 stuck for 26s! [kworker/6:1H:195]
Thanks for the report.
On arm64, this driver relies on the parent irq domain to set handler. So the driver must not overwrite it to NULL.
This should cures it:
drivers/pci/controller/pci-hyperv.c +3 -6
@@ -577,8 +577,6 @@ static void hv_pci_onchannelcallback(void *context);#ifdef CONFIG_X86 #define DELIVERY_MODE APIC_DELIVERY_MODE_FIXED -#define FLOW_HANDLER handle_edge_irq -#define FLOW_NAME "edge"static int hv_pci_irqchip_init (void ) {
@@ -723,8 +721,6 @@ static void hv_arch_irq_unmask(struct irq_data *data)#define HV_PCI_MSI_SPI_START 64#define HV_PCI_MSI_SPI_NR (1020 - HV_PCI_MSI_SPI_START)#define DELIVERY_MODE 0 -#define FLOW_HANDLER NULL -#define FLOW_NAME NULL#define hv_msi_prepare NULLstruct hv_pci_chip_data {
@@ -2162,8 +2158,9 @@ static int hv_pcie_domain_alloc(struct irq_domain *d, unsigned int virq, unsignereturn ret;for (int i =0 ; i < nr_irqs; i++) { -irq_domain_set_info (d, virq + i,0 , &hv_msi_irq_chip,NULL ,FLOW_HANDLER ,NULL , -FLOW_NAME ); +irq_domain_set_hwirq_and_chip (d, virq + i,0 , &hv_msi_irq_chip,NULL ); +if (IS_ENABLED (CONFIG_X86 )) +__irq_set_handler (virq + i, handle_edge_irq,0 ,"edge" ); }return 0 ;
RE: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
From: Nam Cao <[email protected]> Sent: Thursday, July 3, 2025 9:33 PM
Quoted 62 lines · Michael Kelley
Quoted 19 lines
I haven't resolved the conflict. As a shortcut for testing I just removed the conflicting patch since it is for a Microsoft custom NIC ("MANA") that's not in the configuration I'm testing with. I'll have to look more closely to figure out the resolution.
Separately, this patch (the switch to
misc_create_parent_irq_domain) isn't working for Linux VMs on Hyper-V on ARM64. The initial symptom is that interrupts from the NVMe controller aren't getting handled and everything hangs. Here's the dmesg output:[ 84.463419] hv_vmbus: registering driver hv_pci [ 84.463875] hv_pci abee639e-0b9d-49b7-9a07-c54ba8cd5734: PCI VMBus probing: Using version 0x10004 [ 84.464518] hv_pci abee639e-0b9d-49b7-9a07-c54ba8cd5734: PCI host bridge to bus 0b9d:00 [ 84.464529] pci_bus 0b9d:00: root bus resource [mem 0xfc0000000-0xfc00fffff window] [ 84.464531] pci_bus 0b9d:00: No busn resource found for root bus, will use [bus 00-ff] [ 84.465211] pci 0b9d:00:00.0: [1414:b111] type 00 class 0x010802 PCIe Endpoint [ 84.466657] pci 0b9d:00:00.0: BAR 0 [mem 0xfc0000000-0xfc00fffff 64bit] [ 84.481923] pci_bus 0b9d:00: busn_res: [bus 00-ff] end is updated to 00 [ 84.481936] pci 0b9d:00:00.0: BAR 0 [mem 0xfc0000000-0xfc00fffff 64bit]: assignedQuoted 8 lines
[ 84.482413] nvme nvme0: pci function 0b9d:00:00.0 [ 84.482513] nvme 0b9d:00:00.0: enabling device (0000 -> 0002) [ 84.556871] irq 17, desc: 00000000e8529819, depth: 0, count: 0, unhandled: 0 [ 84.556883] ->handle_irq(): 0000000062fa78bc, handle_bad_irq+0x0/0x270 [ 84.556892] ->irq_data.chip(): 00000000ba07832f, 0xffff00011469dc30 [ 84.556895] ->action(): 0000000069f160b3 [ 84.556896] ->action->handler(): 00000000e15d8191, nvme_irq+0x0/0x3e8 [ 172.307920] watchdog: BUG: soft lockup - CPU#6 stuck for 26s! [kworker/6:1H:195]Thanks for the report.
On arm64, this driver relies on the parent irq domain to set handler. So the driver must not overwrite it to NULL.
This should cures it:
drivers/pci/controller/pci-hyperv.c +0
-4@@ -577,8 +577,6 @@ static void hv_pci_onchannelcallback(void *context);#ifdef CONFIG_X86 #define DELIVERY_MODE APIC_DELIVERY_MODE_FIXED -#define FLOW_HANDLER handle_edge_irq -#define FLOW_NAME "edge"static int hv_pci_irqchip_init (void ) {@@ -723,8 +721,6 @@ static void hv_arch_irq_unmask(struct irq_data *data)#define HV_PCI_MSI_SPI_START 64#define HV_PCI_MSI_SPI_NR (1020 - HV_PCI_MSI_SPI_START)#define DELIVERY_MODE 0 -#define FLOW_HANDLER NULL -#define FLOW_NAME NULL#define hv_msi_prepare NULLstruct hv_pci_chip_data {@@ -2162,8 +2158,9 @@ static int hv_pcie_domain_alloc(struct irq_domain *d,unsigned int virq, unsigne
drivers/pci/controller/pci-hyperv.c +0
-0return ret;
drivers/pci/controller/pci-hyperv.c static int hv_pcie_domain_alloc(struct irq_domain *d, unsigned int virq, unsigne · line 2166
Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +3
-2for (int i =0 ; i < nr_irqs; i++) { -irq_domain_set_info (d, virq + i,0 , &hv_msi_irq_chip,NULL ,FLOW_HANDLER ,NULL , -FLOW_NAME ); +irq_domain_set_hwirq_and_chip (d, virq + i,0 , &hv_msi_irq_chip,NULL ); +if (IS_ENABLED (CONFIG_X86 )) +__irq_set_handler (virq + i, handle_edge_irq,0 ,"edge" ); }return 0 ;
Yes, that fixes the problem. Linux now boots with the PCI NIC VF and two NVMe controllers being visible and operational. Thanks for the fix! It would have taken me a while to figure it out.
I want to do some additional testing tomorrow, and look more closely at the code, but now I have something that works well enough to make further progress.
Michael
kernel test robot
build botRe: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
In reply to Nam Caokernel test robot noticed the following build errors:
Hi Nam,
kernel test robot noticed the following build errors:
[auto build test ERROR on pci/next]
[cannot apply to pci/for-linus linus/master v6.16-rc4 next-20250704] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Nam-Cao/PCI-dwc-Switch-to-msi_create_parent_irq_domain/20250626-230029 base: https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git next patch link: https://lore.kernel.org/r/024f0122314198fe0a42fef01af53e8953a687ec.1750858083.git.namcao%40linutronix.de patch subject: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain() config: x86_64-rhel-9.4 (https://download.01.org/0day-ci/archive/20250705/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14+deb12u1) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250705/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>, old ones prefixed by <<):
>> ERROR: modpost: "irq_domain_free_irqs_top" [drivers/pci/controller/pci-hyperv.ko] undefined!
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
RE: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoFrom: Nam Cao <[email protected]> Sent: Thursday, June 26, 2025 7:48 AM
Quoted 3 lines
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().
With the additional tweak to this patch that you supplied separately, everything in my testing on both x86 and arm64 seems to work OK. So that's all good.
On arm64, I did notice the following IRQ domain information from /sys/kernel/debug/irq/domains:
# cat HV-PCI-MSIX-1e03\:00\:00.0-12
name: HV-PCI-MSIX-1e03:00:00.0-12
size: 0
mapped: 7
flags: 0x00000213
IRQ_DOMAIN_FLAG_HIERARCHY
IRQ_DOMAIN_NAME_ALLOCATED
IRQ_DOMAIN_FLAG_MSI
IRQ_DOMAIN_FLAG_MSI_DEVICE
parent: 5D202AA8-1E03-4F0F-A786-390A0D2749E9-3
name: 5D202AA8-1E03-4F0F-A786-390A0D2749E9-3
size: 0
mapped: 7
flags: 0x00000103
IRQ_DOMAIN_FLAG_HIERARCHY
IRQ_DOMAIN_NAME_ALLOCATED
IRQ_DOMAIN_FLAG_MSI_PARENT
parent: hv_vpci_arm64
name: hv_vpci_arm64
size: 956
mapped: 31
flags: 0x00000003
IRQ_DOMAIN_FLAG_HIERARCHY
IRQ_DOMAIN_NAME_ALLOCATED
parent: irqchip@0x00000000ffff0000-1
name: irqchip@0x00000000ffff0000-1
size: 0
mapped: 47
flags: 0x00000003
IRQ_DOMAIN_FLAG_HIERARCHY
IRQ_DOMAIN_NAME_ALLOCATEDThe 5D202AA8-1E03-4F0F-A786-390A0D2749E9-3 domain has IRQ_DOMAIN_FLAG_MSI_PARENT set. But the hv_vpci_arm64 and irqchip@... domains do not. Is that a problem? On x86, the output is this, with IRQ_DOMAIN_FLAG_MSI_PARENT set in the next level up VECTOR domain:
# cat HV-PCI-MSIX-6b71\:00\:02.0-12
name: HV-PCI-MSIX-6b71:00:02.0-12
size: 0
mapped: 17
flags: 0x00000213
IRQ_DOMAIN_FLAG_HIERARCHY
IRQ_DOMAIN_NAME_ALLOCATED
IRQ_DOMAIN_FLAG_MSI
IRQ_DOMAIN_FLAG_MSI_DEVICE
parent: 8564CB14-6B71-477C-B189-F175118E6FF0-3
name: 8564CB14-6B71-477C-B189-F175118E6FF0-3
size: 0
mapped: 17
flags: 0x00000103
IRQ_DOMAIN_FLAG_HIERARCHY
IRQ_DOMAIN_NAME_ALLOCATED
IRQ_DOMAIN_FLAG_MSI_PARENT
parent: VECTOR
name: VECTOR
size: 0
mapped: 67
flags: 0x00000103
IRQ_DOMAIN_FLAG_HIERARCHY
IRQ_DOMAIN_NAME_ALLOCATED
IRQ_DOMAIN_FLAG_MSI_PARENTFinally, I've noted a couple of code review comments below. These comments may reflect my lack of fully understanding the MSI IRQ handling, in which case, please set me straight. Thanks,
Michael
Quoted 116 lines
Signed-off-by Nam Cao <[email protected]>---
Cc: K. Y. Srinivasan <[email protected]>
Cc: Haiyang Zhang <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Dexuan Cui <[email protected]>
Cc: [email protected]--- drivers/pci/Kconfig | 1 + drivers/pci/controller/pci-hyperv.c | 98 +++++++++++++++++++++++------ 2 files changed, 80 insertions(+), 19 deletions(-)drivers/pci/Kconfig +1
-0@@ -223,6 +223,7 @@ config PCI_HYPERV tristate "Hyper-V PCI Frontend" depends on ((X86 && X86_64) || ARM64) && HYPERV && PCI_MSI && SYSFS select PCI_HYPERV_INTERFACE + select IRQ_MSI_LIB help The PCI device frontend driver allows the kernel to import arbitrary PCI devices from a PCI backend to support PCI driver domains.drivers/pci/controller/pci-hyperv.c +40
-4@@ -44,6 +44,7 @@#include <linux/delay.h> #include <linux/semaphore.h> #include <linux/irq.h> +#include <linux/irqchip/irq-msi-lib.h> #include <linux/msi.h> #include <linux/hyperv.h> #include <linux/refcount.h> @@ -508,7 +509,6 @@ struct hv_pcibus_device {struct list_head children;struct list_head dr_list; -struct msi_domain_info msi_info;struct irq_domain *irq_domain;struct workqueue_struct *wq;@@ -1687,7 +1687,7 @@ static void hv_msi_free(struct irq_domain *domain, struct msi_domain_info *info,struct msi_desc *msi =irq_data_get_msi_desc (irq_data); pdev =msi_desc_to_pci_dev (msi); - hbus = info->data; + hbus = domain->host_data; int_desc =irq_data_get_irq_chip_data (irq_data);if (!int_desc)return ;@@ -1705,7 +1705,6 @@ static void hv_msi_free(struct irq_domain *domain, struct msi_domain_info *info,static void hv_irq_mask (struct irq_data *data) { -pci_msi_mask_irq (data);if (data->parent_data->chip->irq_mask)irq_chip_mask_parent (data); }@@ -1716,7 +1715,6 @@ static void hv_irq_unmask(struct irq_data *data)if (data->parent_data->chip->irq_unmask)irq_chip_unmask_parent (data); -pci_msi_unmask_irq (data); }struct compose_comp_ctxt {@@ -2101,6 +2099,44 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) msg->data =0 ; } +static bool hv_pcie_init_dev_msi_info (struct device *dev,struct irq_domain *domain, +struct irq_domain *real_parent,struct msi_domain_info *info) +{ +struct irq_chip *chip = info->chip; + +if (!msi_lib_init_dev_msi_info (dev, domain, real_parent, info)) +return false; + + info->ops->msi_prepare = hv_msi_prepare; + + chip->irq_set_affinity = irq_chip_set_affinity_parent; + +if (IS_ENABLED (CONFIG_X86 )) + chip->flags |=IRQCHIP_MOVE_DEFERRED ; + +return true; +} + +#define HV_PCIE_MSI_FLAGS_REQUIRED (MSI_FLAG_USE_DEF_DOM_OPS | \ + MSI_FLAG_USE_DEF_CHIP_OPS | \ + MSI_FLAG_PCI_MSI_MASK_PARENT) +#define HV_PCIE_MSI_FLAGS_SUPPORTED (MSI_FLAG_MULTI_PCI_MSI | \ + MSI_FLAG_PCI_MSIX | \ + MSI_GENERIC_FLAGS_MASK) + +static const struct msi_parent_ops hv_pcie_msi_parent_ops = { + .required_flags =HV_PCIE_MSI_FLAGS_REQUIRED , + .supported_flags =HV_PCIE_MSI_FLAGS_SUPPORTED , + .bus_select_token =DOMAIN_BUS_PCI_MSI , +#ifdef CONFIG_X86 + .chip_flags =MSI_CHIP_FLAG_SET_ACK , +#elif defined(CONFIG_ARM64) + .chip_flags =MSI_CHIP_FLAG_SET_EOI , +#endif + .prefix ="HV-" , + .init_dev_msi_info = hv_pcie_init_dev_msi_info, +}; +/* HW Interrupt Chip Descriptor */ static struct irq_chip hv_msi_irq_chip = { .name ="Hyper-V PCIe MSI" ,
drivers/pci/controller/pci-hyperv.c static struct irq_chip hv_msi_irq_chip = { · line 2149
Quoted 7 lines
drivers/pci/controller/pci-hyperv.c +0
-1@@ -2108,7 +2144,6 @@ static struct irq_chip hv_msi_irq_chip = { .irq_set_affinity = irq_chip_set_affinity_parent,#ifdef CONFIG_X86 .irq_ack = irq_chip_ack_parent, - .flags = IRQCHIP_MOVE_DEFERRED,#elif defined(CONFIG_ARM64 ) .irq_eoi = irq_chip_eoi_parent,#endif
Would it work to drop the #ifdef's and always set both .irq_ack and .irq_eoi on x86 and on ARM64? Is which one gets called controlled by the child HV-PCI-MSIX- ... domain, based on the .chip_flags? I'm trying to reduce the #ifdef clutter. I tested without the #ifdefs on both x86 and arm64, and everything works, but I know that doesn't prove that it's OK.
If the #ifdefs can go away, then I'd like to see a tweak to the way .chip_flags is set. Rather than do an #ifdef inline for struct msi_parent_ops hv_pcie_msi_parent_ops, add a #define HV_MSI_CHIP_FLAGS in the existing #ifdef X86 and #ifdef ARM64 sections respectively near the top of this source file, and then use HV_MSI_CHIP_FLAGS in struct msi_parent_ops hv_pcie_msi_parent_ops. As much as is reasonable, I'd like to not clutter the code with #ifdef X86 #elseif ARM64, but instead group all the differences under the existing #ifdefs near the top. There are some places where this isn't practical, but this seems like a place that is practical.
drivers/pci/controller/pci-hyperv.c static struct irq_chip hv_msi_irq_chip = { · line 2157
Quoted 11 lines
drivers/pci/controller/pci-hyperv.c +4
-3@@ -2116,9 +2151,37 @@ static struct irq_chip hv_msi_irq_chip = { .irq_unmask = hv_irq_unmask, }; -static struct msi_domain_ops hv_msi_ops = { - .msi_prepare = hv_msi_prepare, - .msi_free = hv_msi_free, +static int hv_pcie_domain_alloc (struct irq_domain *d,unsigned int virq,unsigned int nr_irqs, +void *arg) +{ +/* TODO: move the content of hv_compose_msi_msg() in here */
Could you elaborate on this TODO? Is the idea to loop through all the IRQs and generate the MSI message for each one? What is the advantage to doing it here? I noticed in Patch 3 of the series, the Aardvark controller has advk_msi_irq_compose_msi_msg(), but you had not moved it into the domain allocation path.
Also, is there some point in the time in the future where the "TODO" is likely to become a "MUST DO"?
Quoted 59 lines
drivers/pci/controller/pci-hyperv.c +35
-12+int ret; + + ret =irq_domain_alloc_irqs_parent (d, virq, nr_irqs, arg); +if (ret <0 ) +return ret; + +for (int i =0 ; i < nr_irqs; i++) { +irq_domain_set_info (d, virq + i,0 , &hv_msi_irq_chip,NULL ,FLOW_HANDLER ,NULL , +FLOW_NAME ); + } + +return 0 ; +} + +static void hv_pcie_domain_free (struct irq_domain *d,unsigned int virq,unsigned int nr_irqs) +{ +struct msi_domain_info *info = d->host_data; + +for (int i =0 ; i < nr_irqs; i++) +hv_msi_free (d, info, virq + i); + +irq_domain_free_irqs_top (d, virq, nr_irqs); +} + +static const struct irq_domain_ops hv_pcie_domain_ops = { + .alloc = hv_pcie_domain_alloc, + .free = hv_pcie_domain_free, };/** @@ -2136,17 +2199,14 @@ static struct msi_domain_ops hv_msi_ops = {*/ static int hv_pcie_init_irq_domain (struct hv_pcibus_device *hbus) { - hbus->msi_info.chip = &hv_msi_irq_chip; - hbus->msi_info.ops = &hv_msi_ops; - hbus->msi_info.flags = (MSI_FLAG_USE_DEF_DOM_OPS | -MSI_FLAG_USE_DEF_CHIP_OPS |MSI_FLAG_MULTI_PCI_MSI | -MSI_FLAG_PCI_MSIX ); - hbus->msi_info.handler =FLOW_HANDLER ; - hbus->msi_info.handler_name =FLOW_NAME ; - hbus->msi_info.data = hbus; - hbus->irq_domain =pci_msi_create_irq_domain (hbus->fwnode, - &hbus->msi_info, -hv_pci_get_root_domain ()); +struct irq_domain_info info = { + .fwnode = hbus->fwnode, + .ops = &hv_pcie_domain_ops, + .host_data = hbus, + .parent =hv_pci_get_root_domain (), + }; + + hbus->irq_domain =msi_create_parent_irq_domain (&info, &hv_pcie_msi_parent_ops);if (!hbus->irq_domain) {dev_err (&hbus->hdev->device,"Failed to build an MSI IRQ domain\n" ); --2.39.5
Re: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
Quoted 44 lines · Michael Kelley
From: Nam Cao <[email protected]> Sent: Thursday, June 26, 2025 7:48 AM
Quoted 3 lines
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().With the additional tweak to this patch that you supplied separately, everything in my testing on both x86 and arm64 seems to work OK. So that's all good.
On arm64, I did notice the following IRQ domain information from
/sys/kernel/debug/irq/domains:# cat HV-PCI-MSIX-1e03\:00\:00.0-12 name: HV-PCI-MSIX-1e03:00:00.0-12 size: 0 mapped: 7 flags: 0x00000213 IRQ_DOMAIN_FLAG_HIERARCHY IRQ_DOMAIN_NAME_ALLOCATED IRQ_DOMAIN_FLAG_MSI IRQ_DOMAIN_FLAG_MSI_DEVICE parent: 5D202AA8-1E03-4F0F-A786-390A0D2749E9-3 name: 5D202AA8-1E03-4F0F-A786-390A0D2749E9-3 size: 0 mapped: 7 flags: 0x00000103 IRQ_DOMAIN_FLAG_HIERARCHY IRQ_DOMAIN_NAME_ALLOCATED IRQ_DOMAIN_FLAG_MSI_PARENT parent: hv_vpci_arm64 name: hv_vpci_arm64 size: 956 mapped: 31 flags: 0x00000003 IRQ_DOMAIN_FLAG_HIERARCHY IRQ_DOMAIN_NAME_ALLOCATED parent: irqchip@0x00000000ffff0000-1 name: irqchip@0x00000000ffff0000-1 size: 0 mapped: 47 flags: 0x00000003 IRQ_DOMAIN_FLAG_HIERARCHY IRQ_DOMAIN_NAME_ALLOCATED
Quoted 5 lines
The 5D202AA8-1E03-4F0F-A786-390A0D2749E9-3 domain has
IRQ_DOMAIN_FLAG_MSI_PARENTset. But thehv_vpci_arm64and irqchip@... domains do not. Is that a problem? On x86, the output is this, withIRQ_DOMAIN_FLAG_MSI_PARENTset in the next level up VECTOR domain:
That looks normal. IRQ_DOMAIN_FLAG_MSI_PARENT is set for domains which provide MSI parent domain capability, which happens to be the case for x86 vector.
Quoted 160 lines
# cat HV-PCI-MSIX-6b71\:00\:02.0-12 name: HV-PCI-MSIX-6b71:00:02.0-12 size: 0 mapped: 17 flags: 0x00000213 IRQ_DOMAIN_FLAG_HIERARCHY IRQ_DOMAIN_NAME_ALLOCATED IRQ_DOMAIN_FLAG_MSI IRQ_DOMAIN_FLAG_MSI_DEVICE parent: 8564CB14-6B71-477C-B189-F175118E6FF0-3 name: 8564CB14-6B71-477C-B189-F175118E6FF0-3 size: 0 mapped: 17 flags: 0x00000103 IRQ_DOMAIN_FLAG_HIERARCHY IRQ_DOMAIN_NAME_ALLOCATED IRQ_DOMAIN_FLAG_MSI_PARENT parent: VECTOR name: VECTOR size: 0 mapped: 67 flags: 0x00000103 IRQ_DOMAIN_FLAG_HIERARCHY IRQ_DOMAIN_NAME_ALLOCATED IRQ_DOMAIN_FLAG_MSI_PARENTFinally, I've noted a couple of code review comments below. These comments may reflect my lack of fully understanding the MSI IRQ handling, in which case, please set me straight. Thanks,
Michael
Quoted 116 lines
Signed-off-by Nam Cao <[email protected]>---
Cc: K. Y. Srinivasan <[email protected]>
Cc: Haiyang Zhang <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Dexuan Cui <[email protected]>
Cc: [email protected]--- drivers/pci/Kconfig | 1 + drivers/pci/controller/pci-hyperv.c | 98 +++++++++++++++++++++++------ 2 files changed, 80 insertions(+), 19 deletions(-)drivers/pci/Kconfig +1
-0@@ -223,6 +223,7 @@ config PCI_HYPERV tristate "Hyper-V PCI Frontend" depends on ((X86 && X86_64) || ARM64) && HYPERV && PCI_MSI && SYSFS select PCI_HYPERV_INTERFACE + select IRQ_MSI_LIB help The PCI device frontend driver allows the kernel to import arbitrary PCI devices from a PCI backend to support PCI driver domains.drivers/pci/controller/pci-hyperv.c +40
-4@@ -44,6 +44,7 @@#include <linux/delay.h> #include <linux/semaphore.h> #include <linux/irq.h> +#include <linux/irqchip/irq-msi-lib.h> #include <linux/msi.h> #include <linux/hyperv.h> #include <linux/refcount.h> @@ -508,7 +509,6 @@ struct hv_pcibus_device {struct list_head children;struct list_head dr_list; -struct msi_domain_info msi_info;struct irq_domain *irq_domain;struct workqueue_struct *wq;@@ -1687,7 +1687,7 @@ static void hv_msi_free(struct irq_domain *domain, struct msi_domain_info *info,struct msi_desc *msi =irq_data_get_msi_desc (irq_data); pdev =msi_desc_to_pci_dev (msi); - hbus = info->data; + hbus = domain->host_data; int_desc =irq_data_get_irq_chip_data (irq_data);if (!int_desc)return ;@@ -1705,7 +1705,6 @@ static void hv_msi_free(struct irq_domain *domain, struct msi_domain_info *info,static void hv_irq_mask (struct irq_data *data) { -pci_msi_mask_irq (data);if (data->parent_data->chip->irq_mask)irq_chip_mask_parent (data); }@@ -1716,7 +1715,6 @@ static void hv_irq_unmask(struct irq_data *data)if (data->parent_data->chip->irq_unmask)irq_chip_unmask_parent (data); -pci_msi_unmask_irq (data); }struct compose_comp_ctxt {@@ -2101,6 +2099,44 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) msg->data =0 ; } +static bool hv_pcie_init_dev_msi_info (struct device *dev,struct irq_domain *domain, +struct irq_domain *real_parent,struct msi_domain_info *info) +{ +struct irq_chip *chip = info->chip; + +if (!msi_lib_init_dev_msi_info (dev, domain, real_parent, info)) +return false; + + info->ops->msi_prepare = hv_msi_prepare; + + chip->irq_set_affinity = irq_chip_set_affinity_parent; + +if (IS_ENABLED (CONFIG_X86 )) + chip->flags |=IRQCHIP_MOVE_DEFERRED ; + +return true; +} + +#define HV_PCIE_MSI_FLAGS_REQUIRED (MSI_FLAG_USE_DEF_DOM_OPS | \ + MSI_FLAG_USE_DEF_CHIP_OPS | \ + MSI_FLAG_PCI_MSI_MASK_PARENT) +#define HV_PCIE_MSI_FLAGS_SUPPORTED (MSI_FLAG_MULTI_PCI_MSI | \ + MSI_FLAG_PCI_MSIX | \ + MSI_GENERIC_FLAGS_MASK) + +static const struct msi_parent_ops hv_pcie_msi_parent_ops = { + .required_flags =HV_PCIE_MSI_FLAGS_REQUIRED , + .supported_flags =HV_PCIE_MSI_FLAGS_SUPPORTED , + .bus_select_token =DOMAIN_BUS_PCI_MSI , +#ifdef CONFIG_X86 + .chip_flags =MSI_CHIP_FLAG_SET_ACK , +#elif defined(CONFIG_ARM64) + .chip_flags =MSI_CHIP_FLAG_SET_EOI , +#endif + .prefix ="HV-" , + .init_dev_msi_info = hv_pcie_init_dev_msi_info, +}; +/* HW Interrupt Chip Descriptor */ static struct irq_chip hv_msi_irq_chip = { .name ="Hyper-V PCIe MSI" ,drivers/pci/controller/pci-hyperv.c static struct irq_chip hv_msi_irq_chip = { · line 2149
Quoted 7 lines
drivers/pci/controller/pci-hyperv.c +0
-1@@ -2108,7 +2144,6 @@ static struct irq_chip hv_msi_irq_chip = { .irq_set_affinity = irq_chip_set_affinity_parent,#ifdef CONFIG_X86 .irq_ack = irq_chip_ack_parent, - .flags = IRQCHIP_MOVE_DEFERRED,#elif defined(CONFIG_ARM64 ) .irq_eoi = irq_chip_eoi_parent,#endif Would it work to drop the #ifdef's and always set both .irq_ack and .irq_eoi on x86 and on ARM64? Is which one gets called controlled by the child HV-PCI-MSIX- ... domain, based on the .chip_flags?
Quoted 4 lines
I'm trying to reduce the #ifdef clutter. I tested without the #ifdefs on both x86 and arm64, and everything works, but I know that doesn't prove that it's OK.
Nothing is wrong with that, as far as I can tell.
Quoted 11 lines
If the #ifdefs can go away, then I'd like to see a tweak to the way .chip_flags is set. Rather than do an #ifdef inline for struct
msi_parent_opshv_pcie_msi_parent_ops, add a #defineHV_MSI_CHIP_FLAGSin the existing #ifdef X86 and #ifdef ARM64 sections respectively near the top of this source file, and then useHV_MSI_CHIP_FLAGSin structmsi_parent_opshv_pcie_msi_parent_ops. As much as is reasonable, I'd like to not clutter the code with #ifdef X86 #elseif ARM64, but instead group all the differences under the existing #ifdefs near the top. There are some places where this isn't practical, but this seems like a place that is practical.
Yes, that would be better. I will do it in v2.
Quoted 4 lines
Quoted 4 lines
drivers/pci/controller/pci-hyperv.c +0
-0@@ -2116,9 +2151,37 @@ static struct irq_chip hv_msi_irq_chip = { .irq_unmask = hv_irq_unmask, };
Quoted 13 lines
drivers/pci/controller/pci-hyperv.c static struct irq_chip hv_msi_irq_chip = { · line 2157
Quoted 7 lines
drivers/pci/controller/pci-hyperv.c +4
-3-static struct msi_domain_ops hv_msi_ops = { - .msi_prepare = hv_msi_prepare, - .msi_free = hv_msi_free, +static int hv_pcie_domain_alloc (struct irq_domain *d,unsigned int virq,unsigned int nr_irqs, +void *arg) +{ +/* TODO: move the content of hv_compose_msi_msg() in here */ Could you elaborate on this TODO? Is the idea to loop through all the IRQs and generate the MSI message for each one? What is the advantage to doing it here? I noticed in Patch 3 of the series, the Aardvark controller has
advk_msi_irq_compose_msi_msg(), but you had not moved it into the domain allocation path.
Sorry for being unclear. hv_compose_msi_msg() should not be moved here entirely. Let me elaborate this in v2.
What I meant is that, hv_compose_msi_msg() is doing more than what this callback is supposed to do (composing message). It works, but it is not correct. Interrupt allocation is the responsibility of irq_domain_ops::alloc(). Allocating and populating int_desc should be in hv_pcie_domain_alloc() instead.
irq_domain_ops's .alloc() and .free() should be asymmetric.
Quoted 3 lines
Also, is there some point in the time in the future where the "TODO" is likely to become a "MUST DO"?
There's nothing planned that would make this non-functional, as far as I know.
Thanks so much for examining the patch,
Nam
Re: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
Quoted 11 lines · Nam Cao
Quoted 8 lines · Michael Kelley
From: Nam Cao <[email protected]> Sent: Thursday, June 26, 2025 7:48 AM
Quoted 3 lines
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().With the additional tweak to this patch that you supplied separately, everything in my testing on both x86 and arm64 seems to work OK. So that's all good.
Thanks so much for examining the patch,
Btw, you probably would also be interested in
https://lore.kernel.org/lkml/[email protected]/
which does a similar conversion for the other hyperv driver.
Nam
RE: [PATCH 14/16] PCI: hv: Switch to msi_create_parent_irq_domain()
From: Nam Cao <[email protected]> Sent: Saturday, July 5, 2025 3:02 AM
Quoted 12 lines · Nam Cao
Quoted 10 lines · Michael Kelley
Quoted 8 lines
From: Nam Cao <[email protected]> Sent: Thursday, June 26, 2025 7:48 AM
Quoted 3 lines
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().With the additional tweak to this patch that you supplied separately, everything in my testing on both x86 and arm64 seems to work OK. So that's all good.
Quoted 7 lines
Quoted 1 line
Thanks so much for examining the patch,
Btw, you probably would also be interested in
https://lore.kernel.org/lkml/[email protected]/which does a similar conversion for the other hyperv driver.
Thanks. I had seen this other small series for the driver used by Linux when running as the root partition (i.e., "dom0") on the Hyper-V hypervisor. Unfortunately, I have less familiarity with that configuration, and I lack access to hardware needed to run and test it. Someone at Microsoft would need to look at the patches and give them a basic smoke test with Linux-as-root.
FWIW, I was a Microsoft employee for many years, working on Linux guests on Hyper-V. But I retired about 2 years ago. Though I've kept my hands in Linux kernel work, my access to hardware configs is more limited than when I was an employee -- now it's guest VMs on my personal laptop and Azure VMs.
Michael
PATCH 15/16 PCI: vmd: Convert to lock guards
In reply to Nam CaoConvert lock/unlock pairs to lock guard and tidy up the code.
---
Cc: Nirmal Patel <[email protected]>
Cc: Jonathan Derrick <[email protected]>
--- drivers/pci/controller/vmd.c | 73 ++++++++++++++---------------------- 1 file changed, 29 insertions(+), 44 deletions(-)
drivers/pci/controller/vmd.c +29 -44
@@ -180,13 +180,12 @@ static void vmd_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)static void vmd_irq_enable (struct irq_data *data) {struct vmd_irq *vmdirq = data->chip_data; -unsigned long flags; -raw_spin_lock_irqsave (&list_lock, flags); -WARN_ON (vmdirq->enabled); -list_add_tail_rcu (&vmdirq->node, &vmdirq->irq->irq_list); - vmdirq->enabled = true; -raw_spin_unlock_irqrestore (&list_lock, flags); + scoped_guard(raw_spinlock_irqsave, &list_lock) { +WARN_ON (vmdirq->enabled); +list_add_tail_rcu (&vmdirq->node, &vmdirq->irq->irq_list); + vmdirq->enabled = true; + } data->chip->irq_unmask (data); }
@@ -194,16 +193,15 @@ static void vmd_irq_enable(struct irq_data *data)static void vmd_irq_disable (struct irq_data *data) {struct vmd_irq *vmdirq = data->chip_data; -unsigned long flags; data->chip->irq_mask (data); -raw_spin_lock_irqsave (&list_lock, flags); -if (vmdirq->enabled) { -list_del_rcu (&vmdirq->node); - vmdirq->enabled = false; + scoped_guard(raw_spinlock_irqsave, &list_lock) { +if (vmdirq->enabled) { +list_del_rcu (&vmdirq->node); + vmdirq->enabled = false; + } } -raw_spin_unlock_irqrestore (&list_lock, flags); }static struct irq_chip vmd_msi_controller = {
@@ -225,7 +223,6 @@ static irq_hw_number_t vmd_get_hwirq(struct msi_domain_info *info,*/ static struct vmd_irq_list *vmd_next_irq (struct vmd_dev *vmd,struct msi_desc *desc) { -unsigned long flags;int i, best;if (vmd->msix_count ==1 + vmd->first_vec)
@@ -242,13 +239,13 @@ static struct vmd_irq_list *vmd_next_irq(struct vmd_dev *vmd, struct msi_desc *dreturn &vmd->irqs[vmd->first_vec]; } -raw_spin_lock_irqsave (&list_lock, flags); - best = vmd->first_vec +1 ; -for (i = best; i < vmd->msix_count; i++) -if (vmd->irqs[i].count < vmd->irqs[best].count) - best = i; - vmd->irqs[best].count++; -raw_spin_unlock_irqrestore (&list_lock, flags); + scoped_guard(raw_spinlock_irq, &list_lock) { + best = vmd->first_vec +1 ; +for (i = best; i < vmd->msix_count; i++) +if (vmd->irqs[i].count < vmd->irqs[best].count) + best = i; + vmd->irqs[best].count++; + }return &vmd->irqs[best]; }
@@ -277,14 +274,12 @@ static void vmd_msi_free(struct irq_domain *domain, struct msi_domain_info *info, unsigned int virq) {struct vmd_irq *vmdirq =irq_get_chip_data (virq); -unsigned long flags;synchronize_srcu (&vmdirq->irq->srcu);/* XXX: Potential optimization to rebalance */ -raw_spin_lock_irqsave (&list_lock, flags); - vmdirq->irq->count--; -raw_spin_unlock_irqrestore (&list_lock, flags); +scoped_guard (raw_spinlock_irq, &list_lock) + vmdirq->irq->count--;kfree (vmdirq); }
@@ -387,29 +382,24 @@ static int vmd_pci_read(struct pci_bus *bus, unsigned int devfn, int reg, {struct vmd_dev *vmd =vmd_from_bus (bus);void __iomem *addr =vmd_cfg_addr (vmd, bus, devfn, reg, len); -unsigned long flags; -int ret =0 ;if (!addr)return -EFAULT ; -raw_spin_lock_irqsave (&vmd->cfg_lock, flags); +guard (raw_spinlock_irqsave)(&vmd->cfg_lock);switch (len) {case 1 : *value =readb (addr); -break ; +return 0 ;case 2 : *value =readw (addr); -break ; +return 0 ;case 4 : *value =readl (addr); -break ; +return 0 ;default : - ret = -EINVAL ; -break ; +return -EINVAL ; } -raw_spin_unlock_irqrestore (&vmd->cfg_lock, flags); -return ret; }/*
@@ -422,32 +412,27 @@ static int vmd_pci_write(struct pci_bus *bus, unsigned int devfn, int reg, {struct vmd_dev *vmd =vmd_from_bus (bus);void __iomem *addr =vmd_cfg_addr (vmd, bus, devfn, reg, len); -unsigned long flags; -int ret =0 ;if (!addr)return -EFAULT ; -raw_spin_lock_irqsave (&vmd->cfg_lock, flags); +guard (raw_spinlock_irqsave)(&vmd->cfg_lock);switch (len) {case 1 :writeb (value, addr);readb (addr); -break ; +return 0 ;case 2 :writew (value, addr);readw (addr); -break ; +return 0 ;case 4 :writel (value, addr);readl (addr); -break ; +return 0 ;default : - ret = -EINVAL ; -break ; +return -EINVAL ; } -raw_spin_unlock_irqrestore (&vmd->cfg_lock, flags); -return ret; }static struct pci_ops vmd_ops = {
Signature · 1 line
2.39.5
Re: [PATCH 15/16] PCI: vmd: Convert to lock guards
Quoted 3 lines · Nam Cao
Convert lock/unlock pairs to lock guard and tidy up the code.
Signed-off-by Nam Cao <[email protected]>
PATCH 16/16 PCI: vmd: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoMove away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
---
Cc: Nirmal Patel <[email protected]>
Cc: Jonathan Derrick <[email protected]>
--- drivers/pci/controller/Kconfig | 1 + drivers/pci/controller/vmd.c | 160 +++++++++++++++++---------------- 2 files changed, 82 insertions(+), 79 deletions(-)
drivers/pci/controller/Kconfig +1 -0
@@ -156,6 +156,7 @@ config PCI_IXP4XX config VMD depends on PCI_MSI && X86_64 && !UML tristate "Intel Volume Management Device Driver" + select IRQ_MSI_LIB help Adds support for the Intel Volume Management Device (VMD). VMD is a secondary PCI host bridge that allows PCI Express root ports,
drivers/pci/controller/vmd.c +81 -79
242 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH 16/16] PCI: vmd: Switch to msi_create_parent_irq_domain()
Quoted 4 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>
Nirmal Patel
maintainerRe: [PATCH 16/16] PCI: vmd: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoQuoted 108 lines · On Thu, 26 Jun 2025 16:48:06 +0200 Nam Cao <[email protected]> wrote:
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().Signed-off-by Nam Cao <[email protected]>---
Cc: Nirmal Patel <[email protected]>
Cc: Jonathan Derrick <[email protected]>--- drivers/pci/controller/Kconfig | 1 + drivers/pci/controller/vmd.c | 160+++++++++++++++++---------------- 2 files changed, 82 insertions(+), 79 deletions(-)
(quoted diff) +0
-0b/drivers/pci/controller/Kconfig index
8f56ffd029ba2..41748d083b933100644 --- a/drivers/pci/controller/Kconfig
+++ b/drivers/pci/controller/Kconfigdrivers/pci/controller/Kconfig +1
-0@@ -156,6 +156,7 @@ config PCI_IXP4XX config VMD depends on PCI_MSI && X86_64 && !UML tristate "Intel Volume Management Device Driver" + select IRQ_MSI_LIB help Adds support for the Intel Volume Management Device (VMD). VMD is a secondary PCI host bridge that allows PCI Express root ports,(quoted diff) +0
-0
b/drivers/pci/controller/vmd.cindexd9b893bf4e456..38693a9487d9b100644 ---a/drivers/pci/controller/vmd.c
+++b/drivers/pci/controller/vmd.cdrivers/pci/controller/vmd.c +1
-2@@ -7,6 +7,7 @@#include <linux/device.h> #include <linux/interrupt.h> #include <linux/irq.h> +#include <linux/irqchip/irq-msi-lib.h> #include <linux/kernel.h> #include <linux/module.h> #include <linux/msi.h> @@ -174,9 +175,6 @@ static void vmd_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) msg->arch_addr_lo.destid_0_7 =index_from_irqs (vmd, irq); } -/* -* We rely on MSI_FLAG_USE_DEF_CHIP_OPS to set the IRQ mask/unmask ops.
- */
static voidvmd_irq_enable(struct irq_data *data){drivers/pci/controller/vmd.c +10
-10struct vmd_irq *vmdirq = data->chip_data;@@ -186,7 +184,11 @@ static void vmd_irq_enable(struct irq_data *data)list_add_tail_rcu (&vmdirq->node, &vmdirq->irq->irq_list); vmdirq->enabled = true; } +} +static void vmd_pci_msi_enable (struct irq_data *data) +{ +vmd_irq_enable (data->parent_data); data->chip->irq_unmask (data); }@@ -194,8 +196,6 @@ static void vmd_irq_disable(struct irq_data *data) {struct vmd_irq *vmdirq = data->chip_data; - data->chip->irq_mask (data); - scoped_guard(raw_spinlock_irqsave, &list_lock) {if (vmdirq->enabled) {list_del_rcu (&vmdirq->node);@@ -204,19 +204,17 @@ static void vmd_irq_disable(struct irq_data *data) } } +static void vmd_pci_msi_disable (struct irq_data *data) +{ + data->chip->irq_mask (data); +vmd_irq_disable (data->parent_data); +} +static struct irq_chip vmd_msi_controller = { .name ="VMD-MSI" , - .irq_enable = vmd_irq_enable, - .irq_disable = vmd_irq_disable, .irq_compose_msi_msg = vmd_compose_msi_msg, }; -static irq_hw_number_t vmd_get_hwirq (struct msi_domain_info *info, -msi_alloc_info_t *arg) -{ -return 0 ; -} -/* * XXX: We can be even smarter selecting the best IRQ once we solve the
* affinity problem.(quoted diff) +1
-3@@ -250,100 +248,110 @@ static struct vmd_irq_list *vmd_next_irq(struct vmd_dev *vmd, struct msi_desc *d return &vmd->irqs[best]; } -static int vmd_msi_init(struct irq_domain *domain, struct msi_domain_info *info, - unsigned int virq, irq_hw_number_t hwirq, - msi_alloc_info_t *arg) +static void vmd_msi_free(struct irq_domain *domain, unsigned intvirq, unsigned int
nr_irqs); +
+static int vmd_msi_alloc(structirq_domain*domain, unsigned int virq, unsigned intnr_irqs,
drivers/pci/controller/vmd.c static struct vmd_irq_list *vmd_next_irq(struct vmd_dev *vmd, struct msi_desc *d · line 254
Quoted 1 line
drivers/pci/controller/vmd.c +1
-0+ void *arg)
Is this wrapped in 80 columns? I can see few lines are more than 80. Disregard this if it is wrapped and it can be my claws mail client issue.
Quoted 188 lines
{
(quoted diff) +0
-2- struct msi_desc *desc = arg->desc; - struct vmd_dev *vmd =vmd_from_bus(msi_desc_to_pci_dev(desc)->bus);
(quoted diff) +0
-1- struct vmd_irq *vmdirq = kzalloc(sizeof(*vmdirq),
GFP_KERNEL);drivers/pci/controller/vmd.c +14
-5+struct msi_desc *desc = ((msi_alloc_info_t *)arg)->desc; +struct vmd_dev *vmd = domain->host_data; +struct vmd_irq *vmdirq; -if (!vmdirq) -return -ENOMEM ; +for (int i =0 ; i < nr_irqs; ++i) { + vmdirq =kzalloc (sizeof (*vmdirq),GFP_KERNEL ); +if (!vmdirq) { +vmd_msi_free (domain, virq, i); +return -ENOMEM ; + } -INIT_LIST_HEAD (&vmdirq->node); - vmdirq->irq =vmd_next_irq (vmd, desc); - vmdirq->virq = virq; +INIT_LIST_HEAD (&vmdirq->node); + vmdirq->irq =vmd_next_irq (vmd, desc); + vmdirq->virq = virq + i; + + irq_domain_set_info(domain, virq + i,
vmdirq->irq->virq,&vmd_msi_controller,(quoted diff) +1
-0+ vmdirq, handle_untracked_irq,vmd, NULL);
drivers/pci/controller/vmd.c +1
-1+ } - irq_domain_set_info(domain, virq, vmdirq->irq->virq,
info->chip, vmdirq,drivers/pci/controller/vmd.c +0
-3- handle_untracked_irq, vmd, NULL);return 0 ; } -static void vmd_msi_free (struct irq_domain *domain, - struct msi_domain_info *info, unsigned intvirq) +static void
vmd_msi_free(struct irq_domain *domain, unsigned int virq, unsigned int nr_irqs){drivers/pci/controller/vmd.c +7
-6struct vmd_irq *vmdirq =irq_get_chip_data (virq); -synchronize_srcu (&vmdirq->irq->srcu); +for (int i =0 ; i < nr_irqs; ++i) { +synchronize_srcu (&vmdirq->irq->srcu); -/* XXX: Potential optimization to rebalance */ -scoped_guard (raw_spinlock_irq, &list_lock) - vmdirq->irq->count--; +/* XXX: Potential optimization to rebalance */ +scoped_guard (raw_spinlock_irq, &list_lock) + vmdirq->irq->count--; -kfree (vmdirq); +kfree (vmdirq); + } } -static int vmd_msi_prepare(struct irq_domain *domain, struct device*dev,
drivers/pci/controller/vmd.c +6
-1- int nvec, msi_alloc_info_t *arg) +static const struct irq_domain_ops vmd_msi_domain_ops = { + .alloc = vmd_msi_alloc, + .free = vmd_msi_free, +}; + +static bool vmd_init_dev_msi_info(struct device *dev, struct
irq_domain*domain,(quoted diff) +1
-0+ struct irq_domain *real_parent,struct
msi_domain_info*info) {drivers/pci/controller/vmd.c +1
-2-struct pci_dev *pdev =to_pci_dev (dev); -struct vmd_dev *vmd =vmd_from_bus (pdev->bus); + if (WARN_ON_ONCE(info->bus_token !=
DOMAIN_BUS_PCI_DEVICE_MSIX))drivers/pci/controller/vmd.c +2
-2+return false; -if (nvec > vmd->msix_count) -return vmd->msix_count; + if (!msi_lib_init_dev_msi_info(dev, domain, real_parent,info))
drivers/pci/controller/vmd.c +4
-3+return false; -memset (arg,0 ,sizeof (*arg)); -return 0 ; + info->chip->irq_enable = vmd_pci_msi_enable; + info->chip->irq_disable = vmd_pci_msi_disable; +return true; } -static void vmd_set_desc(msi_alloc_info_t *arg, struct msi_desc*desc) -{
drivers/pci/controller/vmd.c +1
-2- arg->desc = desc; -} +#define VMD_MSI_FLAGS_SUPPORTED (
MSI_GENERIC_FLAGS_MASK|MSI_FLAG_PCI_MSIX) +#defineVMD_MSI_FLAGS_REQUIRED(MSI_FLAG_USE_DEF_DOM_OPS|MSI_FLAG_NO_AFFINITY) -static structmsi_domain_opsvmd_msi_domain_ops= {drivers/pci/controller/vmd.c +7
-7- .get_hwirq = vmd_get_hwirq, - .msi_init = vmd_msi_init, - .msi_free = vmd_msi_free, - .msi_prepare = vmd_msi_prepare, - .set_desc = vmd_set_desc, +static const struct msi_parent_ops vmd_msi_parent_ops = { + .supported_flags =VMD_MSI_FLAGS_SUPPORTED , + .required_flags =VMD_MSI_FLAGS_REQUIRED , + .bus_select_token =DOMAIN_BUS_VMD_MSI , + .bus_select_mask =MATCH_PCI_MSI , + .prefix ="VMD-" , + .init_dev_msi_info = vmd_init_dev_msi_info, }; -static struct msi_domain_info vmd_msi_domain_info = { - .flags = MSI_FLAG_USE_DEF_DOM_OPS |
MSI_FLAG_USE_DEF_CHIP_OPS|drivers/pci/controller/vmd.c +5
-17- MSI_FLAG_NO_AFFINITY | MSI_FLAG_PCI_MSIX, - .ops = &vmd_msi_domain_ops, - .chip = &vmd_msi_controller, -}; - -static void vmd_set_msi_remapping (struct vmd_dev *vmd,bool enable) -{ -u16 reg; - -pci_read_config_word (vmd->dev,PCI_REG_VMCONFIG , ®); - reg = enable ? (reg & ~VMCONFIG_MSI_REMAP ) : - (reg |VMCONFIG_MSI_REMAP ); -pci_write_config_word (vmd->dev,PCI_REG_VMCONFIG , reg); -} -static int vmd_create_irq_domain (struct vmd_dev *vmd) { -struct fwnode_handle *fn; +struct irq_domain_info info = { + .size = vmd->msix_count, + .ops = &vmd_msi_domain_ops, + .host_data = vmd, + }; - fn = irq_domain_alloc_named_id_fwnode("VMD-MSI",
vmd->sysdata.domain);(quoted diff) +1
-1- if (!fn) + info.fwnode = irq_domain_alloc_named_id_fwnode("VMD-MSI",
vmd->sysdata.domain);drivers/pci/controller/vmd.c +1
-1+if (!info.fwnode)return -ENODEV ; - vmd->irq_domain = pci_msi_create_irq_domain(fn,
&vmd_msi_domain_info, NULL);(quoted diff) +1
-0+ vmd->irq_domain = msi_create_parent_irq_domain(&info,
&vmd_msi_parent_ops); if (!vmd->irq_domain) {drivers/pci/controller/vmd.c +11
-4-irq_domain_free_fwnode (fn); +irq_domain_free_fwnode (info.fwnode);return -ENODEV ; }return 0 ; } +static void vmd_set_msi_remapping (struct vmd_dev *vmd,bool enable) +{ +u16 reg; + +pci_read_config_word (vmd->dev,PCI_REG_VMCONFIG , ®); + reg = enable ? (reg & ~VMCONFIG_MSI_REMAP ) : + (reg |VMCONFIG_MSI_REMAP ); +pci_write_config_word (vmd->dev,PCI_REG_VMCONFIG , reg); +} +static void vmd_remove_irq_domain (struct vmd_dev *vmd) {/* @@ -874,12 +882,6 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) ret = vmd_create_irq_domain(vmd);if (ret)return ret; - -/* -* Override the IRQ domain bus token so the domain can be
drivers/pci/controller/vmd.c +0
-3-* distinguished from a regular PCI/MSI domain. -*/ - irq_domain_update_bus_token(vmd->irq_domain,
DOMAIN_BUS_VMD_MSI); } else {drivers/pci/controller/vmd.c +0
-0vmd_set_msi_remapping (vmd, false); }
Re: [PATCH 16/16] PCI: vmd: Switch to msi_create_parent_irq_domain()
Quoted 6 lines · Nirmal Patel
Quoted 2 lines · On Thu, 26 Jun 2025 16:48:06 +0200 Nam Cao <[email protected]> wrote:
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().
Quoted 11 lines
drivers/pci/controller/vmd.c static struct vmd_irq_list *vmd_next_irq(struct vmd_dev *vmd, struct msi_desc *d · line 254
Quoted 7 lines
drivers/pci/controller/vmd.c +1
-2- unsigned int virq, irq_hw_number_t hwirq, - msi_alloc_info_t *arg) +static void vmd_msi_free(struct irq_domain *domain, unsigned intvirq, unsigned int
nr_irqs); +
+static int vmd_msi_alloc(structirq_domain*domain, unsigned int virq, unsigned intnr_irqs,drivers/pci/controller/vmd.c +1
-0+ void *arg)Is this wrapped in 80 columns? I can see few lines are more than 80. Disregard this if it is wrapped and it can be my claws mail client issue.
Wrapped locally, thanks Nirmal.
Bjorn
Re: [PATCH 16/16] PCI: vmd: Switch to msi_create_parent_irq_domain()
In reply to Nam CaoHi Nam,
On 26/06/2025 16:48, Nam Cao wrote:
[...]
Quoted 10 lines
drivers/pci/controller/vmd.c +3
-3-static void vmd_msi_free (struct irq_domain *domain, -struct msi_domain_info *info,unsigned int virq) +static void vmd_msi_free (struct irq_domain *domain,unsigned int virq,unsigned int nr_irqs) {struct vmd_irq *vmdirq =irq_get_chip_data (virq); -synchronize_srcu (&vmdirq->irq->srcu); +for (int i =0 ; i < nr_irqs; ++i) { +synchronize_srcu (&vmdirq->irq->srcu);
drivers/pci/controller/vmd.c static struct vmd_irq_list *vmd_next_irq(struct vmd_dev *vmd, struct msi_desc *d · line 290
Quoted 10 lines
drivers/pci/controller/vmd.c +5
-4-/* XXX: Potential optimization to rebalance */ -scoped_guard (raw_spinlock_irq, &list_lock) - vmdirq->irq->count--; +/* XXX: Potential optimization to rebalance */ +scoped_guard (raw_spinlock_irq, &list_lock) + vmdirq->irq->count--; -kfree (vmdirq); +kfree (vmdirq); + }
By introducing a for loop in this function, you are re-using vmdirq after free'ing it.
I can't send a patch because I am not faimliar with this API and I don't know how to fix it.
However, the issue was reported today by Coverity.
Any idea? :-)
Regards,
Signature · 5 lines
Antonio Quartulli CEO and Co-Founder Mandelbit Srl https://www.mandelbit.com
Re: [PATCH 16/16] PCI: vmd: Switch to msi_create_parent_irq_domain()
Quoted 1 line · Antonio Quartulli
Hi Nam,
Hi Antonio,
Quoted 23 lines · Nam Cao
[...]
Quoted 7 lines
(quoted diff) +2
-3-static void vmd_msi_free(struct irq_domain *domain, - struct msi_domain_info *info, unsigned int virq) +static void vmd_msi_free(struct irq_domain *domain, unsigned int virq, unsigned int nr_irqs) { struct vmd_irq *vmdirq = irq_get_chip_data(virq); - synchronize_srcu(&vmdirq->irq->srcu); + for (int i = 0; i < nr_irqs; ++i) {Review
Quoted 10 lines
(quoted diff) +6
-4+ synchronize_srcu(&vmdirq->irq->srcu); - /* XXX: Potential optimization to rebalance */ - scoped_guard(raw_spinlock_irq, &list_lock) - vmdirq->irq->count--; + /* XXX: Potential optimization to rebalance */ + scoped_guard(raw_spinlock_irq, &list_lock) + vmdirq->irq->count--; - kfree(vmdirq); + kfree(vmdirq); + }By introducing a for loop in this function, you are re-using vmdirq after free'ing it.
Quoted 6 lines
I can't send a patch because I am not faimliar with this API and I don't know how to fix it.
However, the issue was reported today by Coverity.
Any idea? :-)
Thanks for the report. That was indeed a mistake from my side.
I hope PCI maintainers don't mind squashing the below diff.
Sorry for the troubles so far,
Nam
drivers/pci/controller/vmd.c +3 -1
@@ -280,9 +280,11 @@ static int vmd_msi_alloc(struct irq_domain *domain, unsigned int virq,static void vmd_msi_free (struct irq_domain *domain,unsigned int virq,unsigned int nr_irqs) { -struct vmd_irq *vmdirq =irq_get_chip_data (virq); +struct vmd_irq *vmdirq;for (int i =0 ; i < nr_irqs; ++i) { + vmdirq =irq_get_chip_data (virq + i); +synchronize_srcu (&vmdirq->irq->srcu);/* XXX: Potential optimization to rebalance */
Re: [PATCH 16/16] PCI: vmd: Switch to msi_create_parent_irq_domain()
Quoted 30 lines · Nam Cao
Quoted 1 line · Antonio Quartulli
Hi Nam,
Hi Antonio,
Quoted 26 lines · Nam Cao
[...]
Quoted 7 lines
(quoted diff) +2
-3-static void vmd_msi_free(struct irq_domain *domain, - struct msi_domain_info *info, unsigned int virq) +static void vmd_msi_free(struct irq_domain *domain, unsigned int virq, unsigned int nr_irqs) { struct vmd_irq *vmdirq = irq_get_chip_data(virq); - synchronize_srcu(&vmdirq->irq->srcu); + for (int i = 0; i < nr_irqs; ++i) {Review
Quoted 10 lines
(quoted diff) +6
-4+ synchronize_srcu(&vmdirq->irq->srcu); - /* XXX: Potential optimization to rebalance */ - scoped_guard(raw_spinlock_irq, &list_lock) - vmdirq->irq->count--; + /* XXX: Potential optimization to rebalance */ + scoped_guard(raw_spinlock_irq, &list_lock) + vmdirq->irq->count--; - kfree(vmdirq); + kfree(vmdirq); + }By introducing a for loop in this function, you are re-using vmdirq after free'ing it.
I can't send a patch because I am not faimliar with this API and I don't know how to fix it.
Quoted 7 lines
Quoted 3 lines
However, the issue was reported today by Coverity.
Any idea? :-)
Thanks for the report. That was indeed a mistake from my side.
I hope PCI maintainers don't mind squashing the below diff.
Squashed, thanks! Updated commit:
https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git/commit/?id=4246b7fccf26
Quoted 18 lines
drivers/pci/controller/vmd.c +3
-1@@ -280,9 +280,11 @@ static int vmd_msi_alloc(struct irq_domain *domain, unsigned int virq,static void vmd_msi_free (struct irq_domain *domain,unsigned int virq,unsigned int nr_irqs) { -struct vmd_irq *vmdirq =irq_get_chip_data (virq); +struct vmd_irq *vmdirq;for (int i =0 ; i < nr_irqs; ++i) { + vmdirq =irq_get_chip_data (virq + i); +synchronize_srcu (&vmdirq->irq->srcu);/* XXX: Potential optimization to rebalance */
Quoted 41 lines · Nam Cao
The initial implementation of PCI/MSI interrupt domains in the hierarchical interrupt domain model used a shortcut by providing a global PCI/MSI domain.
This works because the PCI/MSI[X] hardware is standardized and uniform, but it violates the basic design principle of hierarchical interrupt domains:
Each hardware block involved in the interrupt delivery chain should have a separate interrupt domain.For PCI/MSI[X], the interrupt controller is per PCI device and not a global made-up entity.
Unsurprisingly, the shortcut turned out to have downsides as it does not allow dynamic allocation of interrupt vectors after initialization and it prevents supporting IMS on PCI. For further details, see:
https://lore.kernel.org/lkml/[email protected]/
The solution is implementing per device MSI domains, this means the entities which provide global PCI/MSI domain so far have to implement MSI parent domain functionality instead.
This series converts the PCI controller drivers to implement MSI parent domain.
drivers/pci/Kconfig | 1 + drivers/pci/controller/Kconfig | 11 + drivers/pci/controller/dwc/Kconfig | 1 + .../pci/controller/dwc/pcie-designware-host.c | 68 ++---- drivers/pci/controller/dwc/pcie-designware.h | 1 - drivers/pci/controller/mobiveil/Kconfig | 1 + .../controller/mobiveil/pcie-mobiveil-host.c | 42 ++-- .../pci/controller/mobiveil/pcie-mobiveil.h | 1 - drivers/pci/controller/pci-aardvark.c | 59 ++--- drivers/pci/controller/pci-hyperv.c | 98 ++++++-- drivers/pci/controller/pcie-altera-msi.c | 44 ++-- drivers/pci/controller/pcie-brcmstb.c | 44 ++-- drivers/pci/controller/pcie-iproc-msi.c | 45 ++-- drivers/pci/controller/pcie-mediatek-gen3.c | 67 ++--- drivers/pci/controller/pcie-mediatek.c | 46 ++-- drivers/pci/controller/pcie-rcar-host.c | 69 ++----
Quoted 8 lines
drivers/pci/controller/pcie-xilinx-dma-pl.c | 48 ++-- drivers/pci/controller/pcie-xilinx-nwl.c | 45 ++-- drivers/pci/controller/pcie-xilinx.c | 55 +++-- drivers/pci/controller/plda/Kconfig | 1 + drivers/pci/controller/plda/pcie-plda-host.c | 44 ++-- drivers/pci/controller/plda/pcie-plda.h | 1 - drivers/pci/controller/vmd.c | 229 +++++++++--------- 23 files changed, 504 insertions(+), 517 deletions(-)
Looks good to me, thanks! I think Mani will probably pick this up.
I might have included the specific "legacy MSI domain" thing you're replacing. It looks like you're replacing pci_msi_create_irq_domain() with msi_create_parent_irq_domain()?
Minor merge conflict in pcie-mediatek-gen3.c with dcbea1c7e94e ("PCI:
mediatek-gen3: Use dev_fwnode() for irq_domain_create_linear()"). No problem, we can easily fix that up.
The "++i" in vmd.c stuck out to me since "i++" is so much more common.
Quoted 51 lines · Bjorn Helgaas
Quoted 41 lines · Nam Cao
The initial implementation of PCI/MSI interrupt domains in the hierarchical interrupt domain model used a shortcut by providing a global PCI/MSI domain.
This works because the PCI/MSI[X] hardware is standardized and uniform, but it violates the basic design principle of hierarchical interrupt domains:
Each hardware block involved in the interrupt delivery chain should have a separate interrupt domain.For PCI/MSI[X], the interrupt controller is per PCI device and not a global made-up entity.
Unsurprisingly, the shortcut turned out to have downsides as it does not allow dynamic allocation of interrupt vectors after initialization and it prevents supporting IMS on PCI. For further details, see:
https://lore.kernel.org/lkml/[email protected]/
The solution is implementing per device MSI domains, this means the entities which provide global PCI/MSI domain so far have to implement MSI parent domain functionality instead.
This series converts the PCI controller drivers to implement MSI parent domain.
drivers/pci/Kconfig | 1 + drivers/pci/controller/Kconfig | 11 + drivers/pci/controller/dwc/Kconfig | 1 + .../pci/controller/dwc/pcie-designware-host.c | 68 ++---- drivers/pci/controller/dwc/pcie-designware.h | 1 - drivers/pci/controller/mobiveil/Kconfig | 1 + .../controller/mobiveil/pcie-mobiveil-host.c | 42 ++-- .../pci/controller/mobiveil/pcie-mobiveil.h | 1 - drivers/pci/controller/pci-aardvark.c | 59 ++--- drivers/pci/controller/pci-hyperv.c | 98 ++++++-- drivers/pci/controller/pcie-altera-msi.c | 44 ++-- drivers/pci/controller/pcie-brcmstb.c | 44 ++-- drivers/pci/controller/pcie-iproc-msi.c | 45 ++-- drivers/pci/controller/pcie-mediatek-gen3.c | 67 ++--- drivers/pci/controller/pcie-mediatek.c | 46 ++-- drivers/pci/controller/pcie-rcar-host.c | 69 ++----Quoted 8 lines
drivers/pci/controller/pcie-xilinx-dma-pl.c | 48 ++-- drivers/pci/controller/pcie-xilinx-nwl.c | 45 ++-- drivers/pci/controller/pcie-xilinx.c | 55 +++-- drivers/pci/controller/plda/Kconfig | 1 + drivers/pci/controller/plda/pcie-plda-host.c | 44 ++-- drivers/pci/controller/plda/pcie-plda.h | 1 - drivers/pci/controller/vmd.c | 229 +++++++++--------- 23 files changed, 504 insertions(+), 517 deletions(-)
Quoted 5 lines
Looks good to me, thanks! I think Mani will probably pick this up.
I might have included the specific "legacy MSI domain" thing you're replacing. It looks like you're replacing
pci_msi_create_irq_domain()withmsi_create_parent_irq_domain()?
Yes, pci_msi_create_irq_domain() is legacy. We will delete it once everything is converted.
Quoted 3 lines
Minor merge conflict in
pcie-mediatek-gen3.cwithdcbea1c7e94e("PCI:
mediatek-gen3: Usedev_fwnode()forirq_domain_create_linear()"). No problem, we can easily fix that up.
Thanks!
Quoted 1 line
The "++i" in
vmd.cstuck out to me since "i++" is so much more common.
I always do "++i", maybe I'm the weird one..
Best regards,
Nam
Manivannan Sadhasivam
maintainerQuoted 52 lines · Nam Cao
Quoted 51 lines · Bjorn Helgaas
Quoted 49 lines · Nam Cao
The initial implementation of PCI/MSI interrupt domains in the hierarchical interrupt domain model used a shortcut by providing a global PCI/MSI domain.
This works because the PCI/MSI[X] hardware is standardized and uniform, but it violates the basic design principle of hierarchical interrupt domains:
Each hardware block involved in the interrupt delivery chain should have a separate interrupt domain.For PCI/MSI[X], the interrupt controller is per PCI device and not a global made-up entity.
Unsurprisingly, the shortcut turned out to have downsides as it does not allow dynamic allocation of interrupt vectors after initialization and it prevents supporting IMS on PCI. For further details, see:
https://lore.kernel.org/lkml/[email protected]/
The solution is implementing per device MSI domains, this means the entities which provide global PCI/MSI domain so far have to implement MSI parent domain functionality instead.
This series converts the PCI controller drivers to implement MSI parent domain.
drivers/pci/Kconfig | 1 + drivers/pci/controller/Kconfig | 11 + drivers/pci/controller/dwc/Kconfig | 1 + .../pci/controller/dwc/pcie-designware-host.c | 68 ++---- drivers/pci/controller/dwc/pcie-designware.h | 1 - drivers/pci/controller/mobiveil/Kconfig | 1 + .../controller/mobiveil/pcie-mobiveil-host.c | 42 ++-- .../pci/controller/mobiveil/pcie-mobiveil.h | 1 - drivers/pci/controller/pci-aardvark.c | 59 ++--- drivers/pci/controller/pci-hyperv.c | 98 ++++++-- drivers/pci/controller/pcie-altera-msi.c | 44 ++-- drivers/pci/controller/pcie-brcmstb.c | 44 ++-- drivers/pci/controller/pcie-iproc-msi.c | 45 ++-- drivers/pci/controller/pcie-mediatek-gen3.c | 67 ++--- drivers/pci/controller/pcie-mediatek.c | 46 ++-- drivers/pci/controller/pcie-rcar-host.c | 69 ++---- drivers/pci/controller/pcie-xilinx-dma-pl.c | 48 ++-- drivers/pci/controller/pcie-xilinx-nwl.c | 45 ++-- drivers/pci/controller/pcie-xilinx.c | 55 +++-- drivers/pci/controller/plda/Kconfig | 1 + drivers/pci/controller/plda/pcie-plda-host.c | 44 ++-- drivers/pci/controller/plda/pcie-plda.h | 1 - drivers/pci/controller/vmd.c | 229 +++++++++--------- 23 files changed, 504 insertions(+), 517 deletions(-)
Quoted 9 lines
Quoted 5 lines
Looks good to me, thanks! I think Mani will probably pick this up.
I might have included the specific "legacy MSI domain" thing you're replacing. It looks like you're replacing
pci_msi_create_irq_domain()withmsi_create_parent_irq_domain()?Yes,
pci_msi_create_irq_domain()is legacy. We will delete it once everything is converted.
I'll amend the commit message to include pci_msi_create_irq_domain().
Quoted 10 lines
Quoted 3 lines
Minor merge conflict in
pcie-mediatek-gen3.cwithdcbea1c7e94e("PCI:
mediatek-gen3: Usedev_fwnode()forirq_domain_create_linear()"). No problem, we can easily fix that up.Thanks!
Quoted 1 line
The "++i" in
vmd.cstuck out to me since "i++" is so much more common.I always do "++i", maybe I'm the weird one..
Just for the sake of uniformity, I'll do 's/++i/i++' while applying.
And based on the discussion on patch 14/16, I'll drop it to be applied through netdev and apply the rest of the series through PCI tree.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Quoted 16 lines · Nam Cao
The initial implementation of PCI/MSI interrupt domains in the hierarchical interrupt domain model used a shortcut by providing a global PCI/MSI domain.
This works because the PCI/MSI[X] hardware is standardized and uniform, but it violates the basic design principle of hierarchical interrupt domains:
Each hardware block involved in the interrupt delivery chain should have a separate interrupt domain.For PCI/MSI[X], the interrupt controller is per PCI device and not a global made-up entity.
Unsurprisingly, the shortcut turned out to have downsides as it does not allow dynamic allocation of interrupt vectors after initialization and it prevents supporting IMS on PCI. For further details, see:
Quoted 9 lines
https://lore.kernel.org/lkml/[email protected]/
The solution is implementing per device MSI domains, this means the entities which provide global PCI/MSI domain so far have to implement MSI parent domain functionality instead.
This series converts the PCI controller drivers to implement MSI parent domain.
Applied the series except patch 14/16 to pci/controller/msi-parent, thanks!
- Mani
Quoted 28 lines
drivers/pci/Kconfig | 1 + drivers/pci/controller/Kconfig | 11 + drivers/pci/controller/dwc/Kconfig | 1 + .../pci/controller/dwc/pcie-designware-host.c | 68 ++---- drivers/pci/controller/dwc/pcie-designware.h | 1 - drivers/pci/controller/mobiveil/Kconfig | 1 + .../controller/mobiveil/pcie-mobiveil-host.c | 42 ++-- .../pci/controller/mobiveil/pcie-mobiveil.h | 1 - drivers/pci/controller/pci-aardvark.c | 59 ++--- drivers/pci/controller/pci-hyperv.c | 98 ++++++-- drivers/pci/controller/pcie-altera-msi.c | 44 ++-- drivers/pci/controller/pcie-brcmstb.c | 44 ++-- drivers/pci/controller/pcie-iproc-msi.c | 45 ++-- drivers/pci/controller/pcie-mediatek-gen3.c | 67 ++--- drivers/pci/controller/pcie-mediatek.c | 46 ++-- drivers/pci/controller/pcie-rcar-host.c | 69 ++---- drivers/pci/controller/pcie-xilinx-dma-pl.c | 48 ++-- drivers/pci/controller/pcie-xilinx-nwl.c | 45 ++-- drivers/pci/controller/pcie-xilinx.c | 55 +++-- drivers/pci/controller/plda/Kconfig | 1 + drivers/pci/controller/plda/pcie-plda-host.c | 44 ++-- drivers/pci/controller/plda/pcie-plda.h | 1 - drivers/pci/controller/vmd.c | 229 +++++++++--------- 23 files changed, 504 insertions(+), 517 deletions(-)Signature · 1 line
2.39.5
Signature · 1 line
மணிவண்ணன் சதாசிவம்