Skip to contentSkip to conversation

powerpc/xive: Change IRQ domain to a tree domain

PATCH Applied Cédric Le Goater 4 participants 7 messages Nov 16, 2021 – Nov 21, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 powerpc/xive: Change IRQ domain to a tree domain

Commit 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive") introduced an IRQ_DOMAIN_FLAG_NO_MAP flag to isolate the 'nomap' domains still in use under the powerpc arch. With this new flag, the revmap_tree of the IRQ domain is not used anymore. This change broke the support of shared LSIs [1] in the XIVE driver because it was relying on a lookup in the revmap_tree to query previously mapped interrupts. Linux now creates two distinct IRQ mappings on the same HW IRQ which can lead to unexpected behavior in the drivers.

The XIVE IRQ domain is not a direct mapping domain and its HW IRQ interrupt number space is rather large : 1M/socket on POWER9 and POWER10, change the XIVE driver to use a 'tree' domain type instead.

[1] For instance, a linux KVM guest with virtio-rng and virtio-balloon devices.

Cc Marc Zyngier <[email protected]>
Cc [email protected] # v5.14+
Fixes 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive")

---

Marc,

The Fixes tag is there because the patch in question revealed that something was broken in XIVE. genirq is not in cause. However, I don't know for PS3 and Cell. May be less critical for now.

 arch/powerpc/sysdev/xive/common.c | 3 +--
 arch/powerpc/sysdev/xive/Kconfig  | 1 -
 2 files changed, 1 insertion(+), 3 deletions(-)
arch/powerpc/sysdev/xive/common.c +1 -2
index fed6fd16c8f4..9d0f0fe25598 100644
@@ -1536,8 +1536,7 @@ static const struct irq_domain_ops xive_irq_domain_ops = {
 
 static void __init xive_init_host(struct device_node *np)
 {
-	xive_irq_domain = irq_domain_add_nomap(np, XIVE_MAX_IRQ,
-					       &xive_irq_domain_ops, NULL);
+	xive_irq_domain = irq_domain_add_tree(np, &xive_irq_domain_ops, NULL);
 	if (WARN_ON(xive_irq_domain == NULL))
 		return;
 	irq_set_default_host(xive_irq_domain);
arch/powerpc/sysdev/xive/Kconfig +0 -1
index 97796c6b63f0..785c292d104b 100644
@@ -3,7 +3,6 @@ config PPC_XIVE
 	bool
 	select PPC_SMP_MUXED_IPI
 	select HARDIRQS_SW_RESEND
-	select IRQ_DOMAIN_NOMAP
 
 config PPC_XIVE_NATIVE
 	bool
Signature · 1 line
2.31.1
Tested-by on v1 1/1
#1 lore raw
Quoted 13 lines · On Tue, 16 Nov 2021 14:40:22 +0100 Cédric Le Goater <[email protected]> wrote:

Commit 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive") introduced an IRQ_DOMAIN_FLAG_NO_MAP flag to isolate the 'nomap' domains still in use under the powerpc arch. With this new flag, the revmap_tree of the IRQ domain is not used anymore. This change broke the support of shared LSIs [1] in the XIVE driver because it was relying on a lookup in the revmap_tree to query previously mapped interrupts. Linux now creates two distinct IRQ mappings on the same HW IRQ which can lead to unexpected behavior in the drivers.

The XIVE IRQ domain is not a direct mapping domain and its HW IRQ interrupt number space is rather large : 1M/socket on POWER9 and POWER10, change the XIVE driver to use a 'tree' domain type instead.

Quoted 9 lines

[1] For instance, a linux KVM guest with virtio-rng and virtio-balloon devices.

Cc Marc Zyngier <[email protected]>
Cc [email protected] # v5.14+
Fixes 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive")

---

with a KVM guest + virtio-rng + virtio-balloon on a POWER9 host.

Quoted 36 lines

Marc,

The Fixes tag is there because the patch in question revealed that something was broken in XIVE. genirq is not in cause. However, I don't know for PS3 and Cell. May be less critical for now.

 arch/powerpc/sysdev/xive/common.c | 3 +--
 arch/powerpc/sysdev/xive/Kconfig  | 1 -
 2 files changed, 1 insertion(+), 3 deletions(-)
arch/powerpc/sysdev/xive/common.c +1 -2
index fed6fd16c8f4..9d0f0fe25598 100644
@@ -1536,8 +1536,7 @@ static const struct irq_domain_ops xive_irq_domain_ops = {
 
 static void __init xive_init_host(struct device_node *np)
 {
-	xive_irq_domain = irq_domain_add_nomap(np, XIVE_MAX_IRQ,
-					       &xive_irq_domain_ops, NULL);
+	xive_irq_domain = irq_domain_add_tree(np, &xive_irq_domain_ops, NULL);
 	if (WARN_ON(xive_irq_domain == NULL))
 		return;
 	irq_set_default_host(xive_irq_domain);
arch/powerpc/sysdev/xive/Kconfig +0 -1
index 97796c6b63f0..785c292d104b 100644
@@ -3,7 +3,6 @@ config PPC_XIVE
 	bool
 	select PPC_SMP_MUXED_IPI
 	select HARDIRQS_SW_RESEND
-	select IRQ_DOMAIN_NOMAP
 
 config PPC_XIVE_NATIVE
 	bool
#2 lore raw
Quoted 26 lines · Greg Kurz
Quoted 13 lines · On Tue, 16 Nov 2021 14:40:22 +0100 Cédric Le Goater <[email protected]> wrote:

Commit 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive") introduced an IRQ_DOMAIN_FLAG_NO_MAP flag to isolate the 'nomap' domains still in use under the powerpc arch. With this new flag, the revmap_tree of the IRQ domain is not used anymore. This change broke the support of shared LSIs [1] in the XIVE driver because it was relying on a lookup in the revmap_tree to query previously mapped interrupts. Linux now creates two distinct IRQ mappings on the same HW IRQ which can lead to unexpected behavior in the drivers.

The XIVE IRQ domain is not a direct mapping domain and its HW IRQ interrupt number space is rather large : 1M/socket on POWER9 and POWER10, change the XIVE driver to use a 'tree' domain type instead.

Quoted 9 lines

[1] For instance, a linux KVM guest with virtio-rng and virtio-balloon devices.

Cc Marc Zyngier <[email protected]>
Cc [email protected] # v5.14+
Fixes 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive")

---

Quoted 3 lines

with a KVM guest + virtio-rng + virtio-balloon on a POWER9 host.

Did you test on a 5.14 backport or mainline ?

I am asking because a large change adding support for MSI domains to XIVE was merged in 5.15.

Thanks,

C.

Quoted 38 lines
Quoted 37 lines

Marc,

The Fixes tag is there because the patch in question revealed that something was broken in XIVE. genirq is not in cause. However, I don't know for PS3 and Cell. May be less critical for now.

  arch/powerpc/sysdev/xive/common.c | 3 +--
  arch/powerpc/sysdev/xive/Kconfig  | 1 -
  2 files changed, 1 insertion(+), 3 deletions(-)
arch/powerpc/sysdev/xive/common.c +1 -2
index fed6fd16c8f4..9d0f0fe25598 100644
@@ -1536,8 +1536,7 @@ static const struct irq_domain_ops xive_irq_domain_ops = {
 
  static void __init xive_init_host(struct device_node *np)
  {
-	xive_irq_domain = irq_domain_add_nomap(np, XIVE_MAX_IRQ,
-					       &xive_irq_domain_ops, NULL);
+	xive_irq_domain = irq_domain_add_tree(np, &xive_irq_domain_ops, NULL);
  	if (WARN_ON(xive_irq_domain == NULL))
  		return;
  	irq_set_default_host(xive_irq_domain);
arch/powerpc/sysdev/xive/Kconfig +0 -1
index 97796c6b63f0..785c292d104b 100644
@@ -3,7 +3,6 @@ config PPC_XIVE
  	bool
  	select PPC_SMP_MUXED_IPI
  	select HARDIRQS_SW_RESEND
-	select IRQ_DOMAIN_NOMAP
 
  config PPC_XIVE_NATIVE
  	bool
#3 lore raw
Quoted 27 lines · On Tue, 16 Nov 2021 15:49:13 +0100 Cédric Le Goater <[email protected]> wrote:
Quoted 26 lines · Greg Kurz
Quoted 22 lines · On Tue, 16 Nov 2021 14:40:22 +0100 Cédric Le Goater <[email protected]> wrote:

Commit 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive") introduced an IRQ_DOMAIN_FLAG_NO_MAP flag to isolate the 'nomap' domains still in use under the powerpc arch. With this new flag, the revmap_tree of the IRQ domain is not used anymore. This change broke the support of shared LSIs [1] in the XIVE driver because it was relying on a lookup in the revmap_tree to query previously mapped interrupts. Linux now creates two distinct IRQ mappings on the same HW IRQ which can lead to unexpected behavior in the drivers.

The XIVE IRQ domain is not a direct mapping domain and its HW IRQ interrupt number space is rather large : 1M/socket on POWER9 and POWER10, change the XIVE driver to use a 'tree' domain type instead.

[1] For instance, a linux KVM guest with virtio-rng and virtio-balloon devices.

Cc Marc Zyngier <[email protected]>
Cc [email protected] # v5.14+
Fixes 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive")

---

Quoted 6 lines
Quoted 3 lines

with a KVM guest + virtio-rng + virtio-balloon on a POWER9 host.

Did you test on a 5.14 backport or mainline ?

I've tested on a 5.14 backport only.

Quoted 47 lines

I am asking because a large change adding support for MSI domains to XIVE was merged in 5.15.

Thanks,

C.

Quoted 38 lines
Quoted 37 lines

Marc,

The Fixes tag is there because the patch in question revealed that something was broken in XIVE. genirq is not in cause. However, I don't know for PS3 and Cell. May be less critical for now.

  arch/powerpc/sysdev/xive/common.c | 3 +--
  arch/powerpc/sysdev/xive/Kconfig  | 1 -
  2 files changed, 1 insertion(+), 3 deletions(-)
arch/powerpc/sysdev/xive/common.c +1 -2
index fed6fd16c8f4..9d0f0fe25598 100644
@@ -1536,8 +1536,7 @@ static const struct irq_domain_ops xive_irq_domain_ops = {
 
  static void __init xive_init_host(struct device_node *np)
  {
-	xive_irq_domain = irq_domain_add_nomap(np, XIVE_MAX_IRQ,
-					       &xive_irq_domain_ops, NULL);
+	xive_irq_domain = irq_domain_add_tree(np, &xive_irq_domain_ops, NULL);
  	if (WARN_ON(xive_irq_domain == NULL))
  		return;
  	irq_set_default_host(xive_irq_domain);
arch/powerpc/sysdev/xive/Kconfig +0 -1
index 97796c6b63f0..785c292d104b 100644
@@ -3,7 +3,6 @@ config PPC_XIVE
  	bool
  	select PPC_SMP_MUXED_IPI
  	select HARDIRQS_SW_RESEND
-	select IRQ_DOMAIN_NOMAP
 
  config PPC_XIVE_NATIVE
  	bool
Acked-by on v1 1/1
#4 lore raw
In reply to Cédric Le Goater
Quoted 8 lines · Cédric Le Goater

Commit 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive") introduced an IRQ_DOMAIN_FLAG_NO_MAP flag to isolate the 'nomap' domains still in use under the powerpc arch. With this new flag, the revmap_tree of the IRQ domain is not used anymore. This change broke the support of shared LSIs [1] in the XIVE driver because it was relying on a lookup in the revmap_tree to query previously mapped interrupts.

Just a lookup? Surely there is more to it, no?

Quoted 14 lines

Linux now creates two distinct IRQ mappings on the
same HW IRQ which can lead to unexpected behavior in the drivers.

The XIVE IRQ domain is not a direct mapping domain and its HW IRQ interrupt number space is rather large : 1M/socket on POWER9 and POWER10, change the XIVE driver to use a 'tree' domain type instead.

[1] For instance, a linux KVM guest with virtio-rng and virtio-balloon devices.

Cc Marc Zyngier <[email protected]>
Cc [email protected] # v5.14+
Fixes 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive")
Quoted 7 lines

---

Marc,

The Fixes tag is there because the patch in question revealed that something was broken in XIVE. genirq is not in cause. However, I don't know for PS3 and Cell. May be less critical for now.

Depends if they expect something that a no-map domain cannot provide.

Quoted 24 lines
 
 arch/powerpc/sysdev/xive/common.c | 3 +--
 arch/powerpc/sysdev/xive/Kconfig  | 1 -
 2 files changed, 1 insertion(+), 3 deletions(-)
arch/powerpc/sysdev/xive/common.c +1 -2
index fed6fd16c8f4..9d0f0fe25598 100644
@@ -1536,8 +1536,7 @@ static const struct irq_domain_ops xive_irq_domain_ops = {
 
 static void __init xive_init_host(struct device_node *np)
 {
-	xive_irq_domain = irq_domain_add_nomap(np, XIVE_MAX_IRQ,
-					       &xive_irq_domain_ops, NULL);
+	xive_irq_domain = irq_domain_add_tree(np, &xive_irq_domain_ops, NULL);
 	if (WARN_ON(xive_irq_domain == NULL))
 		return;
 	irq_set_default_host(xive_irq_domain);
arch/powerpc/sysdev/xive/Kconfig +0 -0
index 97796c6b63f0..785c292d104b 100644

arch/powerpc/sysdev/xive/Kconfig config PPC_XIVE · line 8

Quoted 7 lines
arch/powerpc/sysdev/xive/Kconfig +0 -1
@@ -3,7 +3,6 @@ config PPC_XIVE
 	bool
 	select PPC_SMP_MUXED_IPI
 	select HARDIRQS_SW_RESEND
-	select IRQ_DOMAIN_NOMAP
 
 config PPC_XIVE_NATIVE
 	bool

As long as this works, I'm happy with one less no-map user.

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#5 lore raw
Quoted 11 lines · Marc Zyngier
Quoted 8 lines · Cédric Le Goater

Commit 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive") introduced an IRQ_DOMAIN_FLAG_NO_MAP flag to isolate the 'nomap' domains still in use under the powerpc arch. With this new flag, the revmap_tree of the IRQ domain is not used anymore. This change broke the support of shared LSIs [1] in the XIVE driver because it was relying on a lookup in the revmap_tree to query previously mapped interrupts.

Quoted 1 line

Just a lookup? Surely there is more to it, no?

nope. The HW IRQ for the INTx is defined in the DT. It is caught by of_irq_parse_and_map_pci() which simply adds an extra mapping on the same INTx since the previous one is not found.

Using an INTx is quite rare now days and a shared one is even more uncommon I guess, I could only reproduced on the baremetal platform with the QEMU PowerNV machine using the same virtio devices.

Thanks,

C.

Quoted 62 lines
Quoted 15 lines

Linux now creates two distinct IRQ mappings on the
same HW IRQ which can lead to unexpected behavior in the drivers.

The XIVE IRQ domain is not a direct mapping domain and its HW IRQ interrupt number space is rather large : 1M/socket on POWER9 and POWER10, change the XIVE driver to use a 'tree' domain type instead.

[1] For instance, a linux KVM guest with virtio-rng and virtio-balloon devices.

Cc Marc Zyngier <[email protected]>
Cc [email protected] # v5.14+
Fixes 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive")
Quoted 7 lines

---

Marc,

The Fixes tag is there because the patch in question revealed that something was broken in XIVE. genirq is not in cause. However, I don't know for PS3 and Cell. May be less critical for now.

Depends if they expect something that a no-map domain cannot provide.>

Quoted 24 lines
  
  arch/powerpc/sysdev/xive/common.c | 3 +--
  arch/powerpc/sysdev/xive/Kconfig  | 1 -
  2 files changed, 1 insertion(+), 3 deletions(-)
arch/powerpc/sysdev/xive/common.c +1 -2
index fed6fd16c8f4..9d0f0fe25598 100644
@@ -1536,8 +1536,7 @@ static const struct irq_domain_ops xive_irq_domain_ops = {
 
  static void __init xive_init_host(struct device_node *np)
  {
-	xive_irq_domain = irq_domain_add_nomap(np, XIVE_MAX_IRQ,
-					       &xive_irq_domain_ops, NULL);
+	xive_irq_domain = irq_domain_add_tree(np, &xive_irq_domain_ops, NULL);
  	if (WARN_ON(xive_irq_domain == NULL))
  		return;
  	irq_set_default_host(xive_irq_domain);
arch/powerpc/sysdev/xive/Kconfig +0 -0
index 97796c6b63f0..785c292d104b 100644

arch/powerpc/sysdev/xive/Kconfig config PPC_XIVE · line 8

Quoted 7 lines
arch/powerpc/sysdev/xive/Kconfig +0 -1
@@ -3,7 +3,6 @@ config PPC_XIVE
  	bool
  	select PPC_SMP_MUXED_IPI
  	select HARDIRQS_SW_RESEND
-	select IRQ_DOMAIN_NOMAP
 
  config PPC_XIVE_NATIVE
  	bool

As long as this works, I'm happy with one less no-map user.

	M.
#6 lore raw
In reply to Cédric Le Goater
Quoted 10 lines · Cédric Le Goater

Commit 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive") introduced an IRQ_DOMAIN_FLAG_NO_MAP flag to isolate the 'nomap' domains still in use under the powerpc arch. With this new flag, the revmap_tree of the IRQ domain is not used anymore. This change broke the support of shared LSIs [1] in the XIVE driver because it was relying on a lookup in the revmap_tree to query previously mapped interrupts. Linux now creates two distinct IRQ mappings on the same HW IRQ which can lead to unexpected behavior in the drivers.

[...]

Applied to powerpc/fixes.

[1/1] powerpc/xive: Change IRQ domain to a tree domain

      https://git.kernel.org/powerpc/c/8e80a73fa9a7747e3e8255cb149c543aabf65a24

cheers