Skip to contentSkip to conversation

x86/PCI/MSI: Make sure that irq reservation mode works everywhere

PATCH In review Thomas Gleixner 4 participants 12 messages Oct 17, 2017 – Oct 21, 2017

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover x86/PCI/MSI: Make sure that irq reservation mode works everywhere

Dexuan reported that the recent rework of the vector allocation mode in x86 broke HyperV PCI passtrough because the rework missed to add the MSI_FLAG_MUST_REACTIVATE flag to the HyperV/PCI interrupt domain info.

The simple solution would be to set the flag in the HyperV/PCI driver but it's better to make this generic and let the PCI/MSI core code set the flag when reservation mode is enabled. That ensures that future users of this wont trip over the same problem.

Thanks,

tglx

---
 arch/x86/Kconfig           |    2 +-
 arch/x86/kernel/apic/msi.c |    5 ++---
 drivers/pci/msi.c          |    2 ++
 kernel/irq/Kconfig         |    3 +++
 4 files changed, 8 insertions(+), 4 deletions(-)
#1 lore raw

PATCH 1/3 genirq: Add config option for reservation mode

The interrupt reservation mode requires reactivation of PCI/MSI interrupts. Create a config option, so the PCI code can set the corresponding flag when required.

---
 kernel/irq/Kconfig |    3 +++
 1 file changed, 3 insertions(+)
kernel/irq/Kconfig +3 -0
@@ -100,6 +100,9 @@ config IRQ_TIMINGS
 config GENERIC_IRQ_MATRIX_ALLOCATOR
 	bool
 
+config GENERIC_IRQ_RESERVATION_MODE
+	bool
+
 config IRQ_DOMAIN_DEBUG
 	bool "Expose hardware/virtual IRQ mapping via debugfs"
 	depends on IRQ_DOMAIN && DEBUG_FS
#2 lore raw

[tip:x86/apic] genirq: Add config option for reservation mode

genirq: Add config option for reservation mode
Commit-ID:  2b5175c4fa974b6aa05bbd2ee8d443a8036a1714
Gitweb:     https://git.kernel.org/tip/2b5175c4fa974b6aa05bbd2ee8d443a8036a1714
Author:     Thomas Gleixner <[email protected]>
AuthorDate: Tue, 17 Oct 2017 09:54:57 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Wed, 18 Oct 2017 15:38:30 +0200

genirq: Add config option for reservation mode

The interrupt reservation mode requires reactivation of PCI/MSI interrupts. Create a config option, so the PCI code can set the corresponding flag when required.

Cc Josh Poulson <[email protected]>
Cc Mihai Costache <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Haiyang Zhang <[email protected]>
Cc Dexuan Cui <[email protected]>
Cc Simon Xiao <[email protected]>
Cc Saeed Mahameed <[email protected]>
Cc Jork Loeser <[email protected]>
Cc Bjorn Helgaas <[email protected]>
Cc KY Srinivasan <[email protected]>
---
 kernel/irq/Kconfig | 3 +++
 1 file changed, 3 insertions(+)
kernel/irq/Kconfig +3 -0
index ac1a3e2..89e3558 100644
@@ -100,6 +100,9 @@ config IRQ_TIMINGS
 config GENERIC_IRQ_MATRIX_ALLOCATOR
 	bool
 
+config GENERIC_IRQ_RESERVATION_MODE
+	bool
+
 config IRQ_DOMAIN_DEBUG
 	bool "Expose hardware/virtual IRQ mapping via debugfs"
 	depends on IRQ_DOMAIN && DEBUG_FS
#3 lore raw

PATCH 2/3 PCI/MSI: Set MSI_FLAG_MUST_REACTIVATE in core code

In reply to Thomas Gleixner

If interrupt reservation mode is enabled then the PCI/MSI interrupts must be reactivated after early activation.

Make sure that all callers of pci_msi_create_irq_domain() have the MSI_FLAG_MUST_REACTIVATE set when reservation mode is enabled.

---
 drivers/pci/msi.c |    2 ++
 1 file changed, 2 insertions(+)
drivers/pci/msi.c +2 -0
@@ -1441,6 +1441,8 @@ struct irq_domain *pci_msi_create_irq_do
 		pci_msi_domain_update_chip_ops(info);
 
 	info->flags |= MSI_FLAG_ACTIVATE_EARLY;
+	if (IS_ENABLED(CONFIG_GENERIC_IRQ_RESERVATION))
+		info->flags |= MSI_FLAG_MUST_REACTIVATE;
 
 	domain = msi_create_irq_domain(fwnode, info, parent);
 	if (!domain)
#4 lore raw

Re: [patch 2/3] PCI/MSI: Set MSI_FLAG_MUST_REACTIVATE in core code

Quoted 14 lines · Thomas Gleixner

If interrupt reservation mode is enabled then the PCI/MSI interrupts must be reactivated after early activation.

Make sure that all callers of pci_msi_create_irq_domain() have the MSI_FLAG_MUST_REACTIVATE set when reservation mode is enabled.

---
 drivers/pci/msi.c |    2 ++
 1 file changed, 2 insertions(+)

drivers/pci/msi.c struct irq_domain *pci_msi_create_irq_do · line 1444

Quoted 4 lines
drivers/pci/msi.c +1 -0
@@ -1441,6 +1441,8 @@ struct irq_domain *pci_msi_create_irq_do
 		pci_msi_domain_update_chip_ops(info);
 
 	info->flags |= MSI_FLAG_ACTIVATE_EARLY;
+	if (IS_ENABLED(CONFIG_GENERIC_IRQ_RESERVATION))

Bah. quilt refresh seems overrated. Correct patch below.

Thanks,

tglx

8<----------------

drivers/pci/msi.c +2 -0
@@ -1441,6 +1441,8 @@ struct irq_domain *pci_msi_create_irq_do
 		pci_msi_domain_update_chip_ops(info);
 
 	info->flags |= MSI_FLAG_ACTIVATE_EARLY;
+	if (IS_ENABLED(CONFIG_GENERIC_IRQ_RESERVATION_MODE))
+		info->flags |= MSI_FLAG_MUST_REACTIVATE;
 
 	domain = msi_create_irq_domain(fwnode, info, parent);
 	if (!domain)
#5 lore raw

[tip:x86/apic] PCI/MSI: Set MSI_FLAG_MUST_REACTIVATE in core code

In reply to Thomas Gleixner
PCI/MSI: Set MSI_FLAG_MUST_REACTIVATE in core code
Commit-ID:  25e960efc63852b84d1c3739aef586285b177395
Gitweb:     https://git.kernel.org/tip/25e960efc63852b84d1c3739aef586285b177395
Author:     Thomas Gleixner <[email protected]>
AuthorDate: Tue, 17 Oct 2017 09:54:58 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Wed, 18 Oct 2017 15:38:31 +0200

PCI/MSI: Set MSI_FLAG_MUST_REACTIVATE in core code

If interrupt reservation mode is enabled then the PCI/MSI interrupts must be reactivated after early activation.

Make sure that all callers of pci_msi_create_irq_domain() have the MSI_FLAG_MUST_REACTIVATE set when reservation mode is enabled.

Cc Josh Poulson <[email protected]>
Cc Mihai Costache <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Haiyang Zhang <[email protected]>
Cc Dexuan Cui <[email protected]>
Cc Simon Xiao <[email protected]>
Cc Saeed Mahameed <[email protected]>
Cc Jork Loeser <[email protected]>
Cc Bjorn Helgaas <[email protected]>
Cc KY Srinivasan <[email protected]>
---
 drivers/pci/msi.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/msi.c +2 -0
index 496ed91..e066071 100644
@@ -1441,6 +1441,8 @@ struct irq_domain *pci_msi_create_irq_domain(struct fwnode_handle *fwnode,
 		pci_msi_domain_update_chip_ops(info);
 
 	info->flags |= MSI_FLAG_ACTIVATE_EARLY;
+	if (IS_ENABLED(CONFIG_GENERIC_IRQ_RESERVATION_MODE))
+		info->flags |= MSI_FLAG_MUST_REACTIVATE;
 
 	domain = msi_create_irq_domain(fwnode, info, parent);
 	if (!domain)
#6 lore raw

PATCH 3/3 x86/vector/msi: Select CONFIG_GENERIC_IRQ_RESERVATION_MODE

In reply to Thomas Gleixner

Select CONFIG_GENERIC_IRQ_RESERVATION_MODE so PCI/MSI domains get the MSI_FLAG_MUST_REACTIVATE flag set in pci_msi_create_irq_domain().

Remove the explicit setters of this flag in the apic/msi code as they are not longer required.

Fixes 4900be83602b ("x86/vector/msi: Switch to global reservation mode")
---
 arch/x86/Kconfig           |    2 +-
 arch/x86/kernel/apic/msi.c |    5 ++---
 2 files changed, 3 insertions(+), 4 deletions(-)
arch/x86/Kconfig +1 -1
@@ -95,7 +95,7 @@ config X86
 	select GENERIC_IRQ_MATRIX_ALLOCATOR	if X86_LOCAL_APIC
 	select GENERIC_IRQ_MIGRATION		if SMP
 	select GENERIC_IRQ_PROBE
-	select GENERIC_IRQ_SHOW
+	select GENERIC_IRQ_RESERVATION_MODE
 	select GENERIC_PENDING_IRQ		if SMP
 	select GENERIC_SMP_IDLE_THREAD
 	select GENERIC_STRNCPY_FROM_USER
arch/x86/kernel/apic/msi.c +2 -3
@@ -129,7 +129,7 @@ static struct msi_domain_ops pci_msi_dom
 
 static struct msi_domain_info pci_msi_domain_info = {
 	.flags		= MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS |
-			  MSI_FLAG_PCI_MSIX | MSI_FLAG_MUST_REACTIVATE,
+			  MSI_FLAG_PCI_MSIX,
 	.ops		= &pci_msi_domain_ops,
 	.chip		= &pci_msi_controller,
 	.handler	= handle_edge_irq,
@@ -167,8 +167,7 @@ static struct irq_chip pci_msi_ir_contro
 
 static struct msi_domain_info pci_msi_ir_domain_info = {
 	.flags		= MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS |
-			  MSI_FLAG_MULTI_PCI_MSI | MSI_FLAG_PCI_MSIX |
-			  MSI_FLAG_MUST_REACTIVATE,
+			  MSI_FLAG_MULTI_PCI_MSI | MSI_FLAG_PCI_MSIX,
 	.ops		= &pci_msi_domain_ops,
 	.chip		= &pci_msi_ir_controller,
 	.handler	= handle_edge_irq,
#7 lore raw

RE: [patch 3/3] x86/vector/msi: Select CONFIG_GENERIC_IRQ_RESERVATION_MODE

arch/x86/Kconfig config X86 · line 101

Quoted 13 lines

From: Thomas Gleixner [mailto:[email protected]]
Sent: Tuesday, October 17, 2017 12:55 AM

arch/x86/Kconfig +1 -1
@@ -95,7 +95,7 @@ config X86
 	select GENERIC_IRQ_MATRIX_ALLOCATOR	if X86_LOCAL_APIC
 	select GENERIC_IRQ_MIGRATION		if SMP
 	select GENERIC_IRQ_PROBE
-	select GENERIC_IRQ_SHOW
+	select GENERIC_IRQ_RESERVATION_MODE
 	select GENERIC_PENDING_IRQ		if SMP
 	select GENERIC_SMP_IDLE_THREAD
 	select GENERIC_STRNCPY_FROM_USER

Hi Thomas,
It looks you removed GENERIC_IRQ_SHOW by accident?

I got this build error:
...
LD vmlinux.o
MODPOST vmlinux.o
fs/proc/interrupts.o:(.rodata+0x118): undefined reference to `show_interrup= ts'
Makefile:1000: recipe for target 'vmlinux' failed
make: *** [vmlinux] Error 1

Adding GENERIC_IRQ_SHOW back can fix the build failure, and with the pachset Hyper-V PCI pass-through can work fine.

Thanks,
-- Dexuan

#8 lore raw

RE: [patch 3/3] x86/vector/msi: Select CONFIG_GENERIC_IRQ_RESERVATION_MODE

Quoted 8 lines · Dexuan Cui
Quoted 8 lines

From: Thomas Gleixner [mailto:[email protected]]
Sent: Tuesday, October 17, 2017 12:55 AM

arch/x86/Kconfig +0 -0
@@ -95,7 +95,7 @@ config X86
 	select GENERIC_IRQ_MATRIX_ALLOCATOR	if X86_LOCAL_APIC
 	select GENERIC_IRQ_MIGRATION		if SMP
 	select GENERIC_IRQ_PROBE
Quoted 8 lines

arch/x86/Kconfig config X86 · line 101

Quoted 5 lines
arch/x86/Kconfig +1 -1
-	select GENERIC_IRQ_SHOW
+	select GENERIC_IRQ_RESERVATION_MODE
 	select GENERIC_PENDING_IRQ		if SMP
 	select GENERIC_SMP_IDLE_THREAD
 	select GENERIC_STRNCPY_FROM_USER

Hi Thomas,
It looks you removed GENERIC_IRQ_SHOW by accident?

Indeed. My patch foo today is more than bad....

Quoted 10 lines

I got this build error:
...
LD vmlinux.o
MODPOST vmlinux.o
fs/proc/interrupts.o:(.rodata+0x118): undefined reference to `show_interrupts' Makefile:1000: recipe for target 'vmlinux' failed
make: *** [vmlinux] Error 1

Adding GENERIC_IRQ_SHOW back can fix the build failure, and with the pachset Hyper-V PCI pass-through can work fine.

Thanks for testing.

tglx

#9 lore raw

[tip:x86/apic] x86/vector/msi: Select CONFIG_GENERIC_IRQ_RESERVATION_MODE

In reply to Thomas Gleixner
x86/vector/msi: Select CONFIG_GENERIC_IRQ_RESERVATION_MODE
Commit-ID:  c201c91799d687c0a6d8c3272950f51aad5ffebe
Gitweb:     https://git.kernel.org/tip/c201c91799d687c0a6d8c3272950f51aad5ffebe
Author:     Thomas Gleixner <[email protected]>
AuthorDate: Tue, 17 Oct 2017 09:54:59 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Wed, 18 Oct 2017 15:38:31 +0200

x86/vector/msi: Select CONFIG_GENERIC_IRQ_RESERVATION_MODE

Select CONFIG_GENERIC_IRQ_RESERVATION_MODE so PCI/MSI domains get the MSI_FLAG_MUST_REACTIVATE flag set in pci_msi_create_irq_domain().

Remove the explicit setters of this flag in the apic/msi code as they are not longer required.

Fixes 4900be83602b ("x86/vector/msi: Switch to global reservation mode")
Reported-and-tested-by Dexuan Cui <[email protected]>
Cc Josh Poulson <[email protected]>
Cc Mihai Costache <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Haiyang Zhang <[email protected]>
Cc Simon Xiao <[email protected]>
Cc Saeed Mahameed <[email protected]>
Cc Jork Loeser <[email protected]>
Cc Bjorn Helgaas <[email protected]>
Cc KY Srinivasan <[email protected]>
---
 arch/x86/Kconfig           | 1 +
 arch/x86/kernel/apic/msi.c | 5 ++---
 2 files changed, 3 insertions(+), 3 deletions(-)
arch/x86/Kconfig +1 -0
index 64e99d3..ea4beda 100644
@@ -95,6 +95,7 @@ config X86
 	select GENERIC_IRQ_MATRIX_ALLOCATOR	if X86_LOCAL_APIC
 	select GENERIC_IRQ_MIGRATION		if SMP
 	select GENERIC_IRQ_PROBE
+	select GENERIC_IRQ_RESERVATION_MODE
 	select GENERIC_IRQ_SHOW
 	select GENERIC_PENDING_IRQ		if SMP
 	select GENERIC_SMP_IDLE_THREAD
arch/x86/kernel/apic/msi.c +2 -3
index 5b6dd1a..9b18be7 100644
@@ -129,7 +129,7 @@ static struct msi_domain_ops pci_msi_domain_ops = {
 
 static struct msi_domain_info pci_msi_domain_info = {
 	.flags		= MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS |
-			  MSI_FLAG_PCI_MSIX | MSI_FLAG_MUST_REACTIVATE,
+			  MSI_FLAG_PCI_MSIX,
 	.ops		= &pci_msi_domain_ops,
 	.chip		= &pci_msi_controller,
 	.handler	= handle_edge_irq,
@@ -167,8 +167,7 @@ static struct irq_chip pci_msi_ir_controller = {
 
 static struct msi_domain_info pci_msi_ir_domain_info = {
 	.flags		= MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS |
-			  MSI_FLAG_MULTI_PCI_MSI | MSI_FLAG_PCI_MSIX |
-			  MSI_FLAG_MUST_REACTIVATE,
+			  MSI_FLAG_MULTI_PCI_MSI | MSI_FLAG_PCI_MSIX,
 	.ops		= &pci_msi_domain_ops,
 	.chip		= &pci_msi_ir_controller,
 	.handler	= handle_edge_irq,
Acked-by on the v1 series
#10 lore raw
In reply to Thomas Gleixner
Quoted 11 lines · Thomas Gleixner

Dexuan reported that the recent rework of the vector allocation mode in x86 broke HyperV PCI passtrough because the rework missed to add the MSI_FLAG_MUST_REACTIVATE flag to the HyperV/PCI interrupt domain info.

The simple solution would be to set the flag in the HyperV/PCI driver but it's better to make this generic and let the PCI/MSI core code set the flag when reservation mode is enabled. That ensures that future users of this wont trip over the same problem.

Thanks,

Quoted 8 lines

tglx

---
 arch/x86/Kconfig           |    2 +-
 arch/x86/kernel/apic/msi.c |    5 ++---
 drivers/pci/msi.c          |    2 ++
 kernel/irq/Kconfig         |    3 +++
 4 files changed, 8 insertions(+), 4 deletions(-)

This mentions 4900be83602b ("x86/vector/msi: Switch to global reservation mode"), which I don't have, so I assume it's an x86 thing. So I guess you'll probably merge this via the same tree?

Here's my ack for the PCI part:

#11 lore raw
Quoted 21 lines · Bjorn Helgaas
Quoted 11 lines · Thomas Gleixner

Dexuan reported that the recent rework of the vector allocation mode in x86 broke HyperV PCI passtrough because the rework missed to add the MSI_FLAG_MUST_REACTIVATE flag to the HyperV/PCI interrupt domain info.

The simple solution would be to set the flag in the HyperV/PCI driver but it's better to make this generic and let the PCI/MSI core code set the flag when reservation mode is enabled. That ensures that future users of this wont trip over the same problem.

Thanks,

Quoted 8 lines

tglx

---
 arch/x86/Kconfig           |    2 +-
 arch/x86/kernel/apic/msi.c |    5 ++---
 drivers/pci/msi.c          |    2 ++
 kernel/irq/Kconfig         |    3 +++
 4 files changed, 8 insertions(+), 4 deletions(-)
Quoted 3 lines

This mentions 4900be83602b ("x86/vector/msi: Switch to global reservation mode"), which I don't have, so I assume it's an x86 thing. So I guess you'll probably merge this via the same tree?

Yes, it's in tip/x86/apic branch and the fixups are applied there now as well.

Thanks,

tglx