powerpc/xive: Change IRQ domain to a tree domain
Subsystems (2)
PATCH 1/1 powerpc/xive: Change IRQ domain to a tree domain
Commit 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive") introduced an IRQ_DOMAIN_FLAG_NO_MAP flag to isolate the 'nomap' domains still in use under the powerpc arch. With this new flag, the revmap_tree of the IRQ domain is not used anymore. This change broke the support of shared LSIs [1] in the XIVE driver because it was relying on a lookup in the revmap_tree to query previously mapped interrupts. Linux now creates two distinct IRQ mappings on the same HW IRQ which can lead to unexpected behavior in the drivers.
The XIVE IRQ domain is not a direct mapping domain and its HW IRQ interrupt number space is rather large : 1M/socket on POWER9 and POWER10, change the XIVE driver to use a 'tree' domain type instead.
[1] For instance, a linux KVM guest with virtio-rng and virtio-balloon devices.
---
Marc,
The Fixes tag is there because the patch in question revealed that something was broken in XIVE. genirq is not in cause. However, I don't know for PS3 and Cell. May be less critical for now.
arch/powerpc/sysdev/xive/common.c | 3 +-- arch/powerpc/sysdev/xive/Kconfig | 1 - 2 files changed, 1 insertion(+), 3 deletions(-)
arch/powerpc/sysdev/xive/common.c +1 -2
@@ -1536,8 +1536,7 @@ static const struct irq_domain_ops xive_irq_domain_ops = {static void __init xive_init_host (struct device_node *np) { - xive_irq_domain =irq_domain_add_nomap (np,XIVE_MAX_IRQ , - &xive_irq_domain_ops,NULL ); + xive_irq_domain =irq_domain_add_tree (np, &xive_irq_domain_ops,NULL );if (WARN_ON (xive_irq_domain ==NULL ))return ;irq_set_default_host (xive_irq_domain);
arch/powerpc/sysdev/xive/Kconfig +0 -1
@@ -3,7 +3,6 @@ config PPC_XIVE bool select PPC_SMP_MUXED_IPI select HARDIRQS_SW_RESEND - select IRQ_DOMAIN_NOMAP config PPC_XIVE_NATIVE bool
Signature · 1 line
2.31.1
Quoted 13 lines · On Tue, 16 Nov 2021 14:40:22 +0100 Cédric Le Goater <[email protected]> wrote:
Commit
4f86a06e2d6e("irqdomain: Make normal and nomap irqdomains exclusive") introduced anIRQ_DOMAIN_FLAG_NO_MAPflag to isolate the 'nomap' domains still in use under the powerpc arch. With this new flag, therevmap_treeof the IRQ domain is not used anymore. This change broke the support of shared LSIs [1] in the XIVE driver because it was relying on a lookup in therevmap_treeto query previously mapped interrupts. Linux now creates two distinct IRQ mappings on the same HW IRQ which can lead to unexpected behavior in the drivers.The XIVE IRQ domain is not a direct mapping domain and its HW IRQ interrupt number space is rather large : 1M/socket on POWER9 and POWER10, change the XIVE driver to use a 'tree' domain type instead.
Quoted 9 lines
[1] For instance, a linux KVM guest with virtio-rng and virtio-balloon devices.
Cc Marc Zyngier <[email protected]>Cc [email protected] # v5.14+Fixes 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive")Signed-off-by Cédric Le Goater <[email protected]>---
with a KVM guest + virtio-rng + virtio-balloon on a POWER9 host.
Quoted 36 lines
Marc,
The Fixes tag is there because the patch in question revealed that something was broken in XIVE. genirq is not in cause. However, I don't know for PS3 and Cell. May be less critical for now.
arch/powerpc/sysdev/xive/common.c | 3 +-- arch/powerpc/sysdev/xive/Kconfig | 1 - 2 files changed, 1 insertion(+), 3 deletions(-)arch/powerpc/sysdev/xive/common.c +1
-2@@ -1536,8 +1536,7 @@ static const struct irq_domain_ops xive_irq_domain_ops = {static void __init xive_init_host (struct device_node *np) { - xive_irq_domain =irq_domain_add_nomap (np,XIVE_MAX_IRQ , - &xive_irq_domain_ops,NULL ); + xive_irq_domain =irq_domain_add_tree (np, &xive_irq_domain_ops,NULL );if (WARN_ON (xive_irq_domain ==NULL ))return ;irq_set_default_host (xive_irq_domain);arch/powerpc/sysdev/xive/Kconfig +0
-1@@ -3,7 +3,6 @@ config PPC_XIVE bool select PPC_SMP_MUXED_IPI select HARDIRQS_SW_RESEND - select IRQ_DOMAIN_NOMAP config PPC_XIVE_NATIVE bool
Quoted 26 lines · Greg Kurz
Quoted 13 lines · On Tue, 16 Nov 2021 14:40:22 +0100 Cédric Le Goater <[email protected]> wrote:
Commit
4f86a06e2d6e("irqdomain: Make normal and nomap irqdomains exclusive") introduced anIRQ_DOMAIN_FLAG_NO_MAPflag to isolate the 'nomap' domains still in use under the powerpc arch. With this new flag, therevmap_treeof the IRQ domain is not used anymore. This change broke the support of shared LSIs [1] in the XIVE driver because it was relying on a lookup in therevmap_treeto query previously mapped interrupts. Linux now creates two distinct IRQ mappings on the same HW IRQ which can lead to unexpected behavior in the drivers.The XIVE IRQ domain is not a direct mapping domain and its HW IRQ interrupt number space is rather large : 1M/socket on POWER9 and POWER10, change the XIVE driver to use a 'tree' domain type instead.
Quoted 9 lines
[1] For instance, a linux KVM guest with virtio-rng and virtio-balloon devices.
Cc Marc Zyngier <[email protected]>Cc [email protected] # v5.14+Fixes 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive")Signed-off-by Cédric Le Goater <[email protected]>---
Quoted 3 lines
Tested-by Greg Kurz <[email protected]>with a KVM guest + virtio-rng + virtio-balloon on a POWER9 host.
Did you test on a 5.14 backport or mainline ?
I am asking because a large change adding support for MSI domains to XIVE was merged in 5.15.
Thanks,
C.
Quoted 38 lines
Quoted 37 lines
Marc,
The Fixes tag is there because the patch in question revealed that something was broken in XIVE. genirq is not in cause. However, I don't know for PS3 and Cell. May be less critical for now.
arch/powerpc/sysdev/xive/common.c | 3 +-- arch/powerpc/sysdev/xive/Kconfig | 1 - 2 files changed, 1 insertion(+), 3 deletions(-)arch/powerpc/sysdev/xive/common.c +1
-2@@ -1536,8 +1536,7 @@ static const struct irq_domain_ops xive_irq_domain_ops = {static void __init xive_init_host (struct device_node *np) { - xive_irq_domain =irq_domain_add_nomap (np,XIVE_MAX_IRQ , - &xive_irq_domain_ops,NULL ); + xive_irq_domain =irq_domain_add_tree (np, &xive_irq_domain_ops,NULL );if (WARN_ON (xive_irq_domain ==NULL ))return ;irq_set_default_host (xive_irq_domain);arch/powerpc/sysdev/xive/Kconfig +0
-1@@ -3,7 +3,6 @@ config PPC_XIVE bool select PPC_SMP_MUXED_IPI select HARDIRQS_SW_RESEND - select IRQ_DOMAIN_NOMAP config PPC_XIVE_NATIVE bool
Quoted 27 lines · On Tue, 16 Nov 2021 15:49:13 +0100 Cédric Le Goater <[email protected]> wrote:
Quoted 26 lines · Greg Kurz
Quoted 22 lines · On Tue, 16 Nov 2021 14:40:22 +0100 Cédric Le Goater <[email protected]> wrote:
Commit
4f86a06e2d6e("irqdomain: Make normal and nomap irqdomains exclusive") introduced anIRQ_DOMAIN_FLAG_NO_MAPflag to isolate the 'nomap' domains still in use under the powerpc arch. With this new flag, therevmap_treeof the IRQ domain is not used anymore. This change broke the support of shared LSIs [1] in the XIVE driver because it was relying on a lookup in therevmap_treeto query previously mapped interrupts. Linux now creates two distinct IRQ mappings on the same HW IRQ which can lead to unexpected behavior in the drivers.The XIVE IRQ domain is not a direct mapping domain and its HW IRQ interrupt number space is rather large : 1M/socket on POWER9 and POWER10, change the XIVE driver to use a 'tree' domain type instead.
[1] For instance, a linux KVM guest with virtio-rng and virtio-balloon devices.
Cc Marc Zyngier <[email protected]>Cc [email protected] # v5.14+Fixes 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive")Signed-off-by Cédric Le Goater <[email protected]>---
Quoted 6 lines
Quoted 3 lines
Tested-by Greg Kurz <[email protected]>with a KVM guest + virtio-rng + virtio-balloon on a POWER9 host.
Did you test on a 5.14 backport or mainline ?
I've tested on a 5.14 backport only.
Quoted 47 lines
I am asking because a large change adding support for MSI domains to XIVE was merged in 5.15.
Thanks,
C.
Quoted 38 lines
Quoted 37 lines
Marc,
The Fixes tag is there because the patch in question revealed that something was broken in XIVE. genirq is not in cause. However, I don't know for PS3 and Cell. May be less critical for now.
arch/powerpc/sysdev/xive/common.c | 3 +-- arch/powerpc/sysdev/xive/Kconfig | 1 - 2 files changed, 1 insertion(+), 3 deletions(-)arch/powerpc/sysdev/xive/common.c +1
-2@@ -1536,8 +1536,7 @@ static const struct irq_domain_ops xive_irq_domain_ops = {static void __init xive_init_host (struct device_node *np) { - xive_irq_domain =irq_domain_add_nomap (np,XIVE_MAX_IRQ , - &xive_irq_domain_ops,NULL ); + xive_irq_domain =irq_domain_add_tree (np, &xive_irq_domain_ops,NULL );if (WARN_ON (xive_irq_domain ==NULL ))return ;irq_set_default_host (xive_irq_domain);arch/powerpc/sysdev/xive/Kconfig +0
-1@@ -3,7 +3,6 @@ config PPC_XIVE bool select PPC_SMP_MUXED_IPI select HARDIRQS_SW_RESEND - select IRQ_DOMAIN_NOMAP config PPC_XIVE_NATIVE bool
Quoted 8 lines · Cédric Le Goater
Commit
4f86a06e2d6e("irqdomain: Make normal and nomap irqdomains exclusive") introduced anIRQ_DOMAIN_FLAG_NO_MAPflag to isolate the 'nomap' domains still in use under the powerpc arch. With this new flag, therevmap_treeof the IRQ domain is not used anymore. This change broke the support of shared LSIs [1] in the XIVE driver because it was relying on a lookup in therevmap_treeto query previously mapped interrupts.
Just a lookup? Surely there is more to it, no?
Quoted 14 lines
Linux now creates two distinct IRQ mappings on the
same HW IRQ which can lead to unexpected behavior in the drivers.The XIVE IRQ domain is not a direct mapping domain and its HW IRQ interrupt number space is rather large : 1M/socket on POWER9 and POWER10, change the XIVE driver to use a 'tree' domain type instead.
[1] For instance, a linux KVM guest with virtio-rng and virtio-balloon devices.
Cc Marc Zyngier <[email protected]>Cc [email protected] # v5.14+Fixes 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive")Signed-off-by Cédric Le Goater <[email protected]>
Quoted 7 lines
---
Marc,
The Fixes tag is there because the patch in question revealed that something was broken in XIVE. genirq is not in cause. However, I don't know for PS3 and Cell. May be less critical for now.
Depends if they expect something that a no-map domain cannot provide.
Quoted 24 lines
arch/powerpc/sysdev/xive/common.c | 3 +-- arch/powerpc/sysdev/xive/Kconfig | 1 - 2 files changed, 1 insertion(+), 3 deletions(-)arch/powerpc/sysdev/xive/common.c +1
-2@@ -1536,8 +1536,7 @@ static const struct irq_domain_ops xive_irq_domain_ops = {static void __init xive_init_host (struct device_node *np) { - xive_irq_domain =irq_domain_add_nomap (np,XIVE_MAX_IRQ , - &xive_irq_domain_ops,NULL ); + xive_irq_domain =irq_domain_add_tree (np, &xive_irq_domain_ops,NULL );if (WARN_ON (xive_irq_domain ==NULL ))return ;irq_set_default_host (xive_irq_domain);arch/powerpc/sysdev/xive/Kconfig +0
-0
arch/powerpc/sysdev/xive/Kconfig config PPC_XIVE · line 8
Quoted 7 lines
arch/powerpc/sysdev/xive/Kconfig +0
-1@@ -3,7 +3,6 @@ config PPC_XIVE bool select PPC_SMP_MUXED_IPI select HARDIRQS_SW_RESEND - select IRQ_DOMAIN_NOMAP config PPC_XIVE_NATIVE bool
As long as this works, I'm happy with one less no-map user.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Quoted 11 lines · Marc Zyngier
Quoted 8 lines · Cédric Le Goater
Commit
4f86a06e2d6e("irqdomain: Make normal and nomap irqdomains exclusive") introduced anIRQ_DOMAIN_FLAG_NO_MAPflag to isolate the 'nomap' domains still in use under the powerpc arch. With this new flag, therevmap_treeof the IRQ domain is not used anymore. This change broke the support of shared LSIs [1] in the XIVE driver because it was relying on a lookup in therevmap_treeto query previously mapped interrupts.
Quoted 1 line
Just a lookup? Surely there is more to it, no?
nope. The HW IRQ for the INTx is defined in the DT. It is caught by of_irq_parse_and_map_pci() which simply adds an extra mapping on the same INTx since the previous one is not found.
Using an INTx is quite rare now days and a shared one is even more uncommon I guess, I could only reproduced on the baremetal platform with the QEMU PowerNV machine using the same virtio devices.
Thanks,
C.
Quoted 62 lines
Quoted 15 lines
Linux now creates two distinct IRQ mappings on the
same HW IRQ which can lead to unexpected behavior in the drivers.The XIVE IRQ domain is not a direct mapping domain and its HW IRQ interrupt number space is rather large : 1M/socket on POWER9 and POWER10, change the XIVE driver to use a 'tree' domain type instead.
[1] For instance, a linux KVM guest with virtio-rng and virtio-balloon devices.
Cc Marc Zyngier <[email protected]>Cc [email protected] # v5.14+Fixes 4f86a06e2d6e ("irqdomain: Make normal and nomap irqdomains exclusive")Signed-off-by Cédric Le Goater <[email protected]>Quoted 7 lines
---
Marc,
The Fixes tag is there because the patch in question revealed that something was broken in XIVE. genirq is not in cause. However, I don't know for PS3 and Cell. May be less critical for now.
Depends if they expect something that a no-map domain cannot provide.>
Quoted 24 lines
arch/powerpc/sysdev/xive/common.c | 3 +-- arch/powerpc/sysdev/xive/Kconfig | 1 - 2 files changed, 1 insertion(+), 3 deletions(-)arch/powerpc/sysdev/xive/common.c +1
-2@@ -1536,8 +1536,7 @@ static const struct irq_domain_ops xive_irq_domain_ops = {static void __init xive_init_host (struct device_node *np) { - xive_irq_domain =irq_domain_add_nomap (np,XIVE_MAX_IRQ , - &xive_irq_domain_ops,NULL ); + xive_irq_domain =irq_domain_add_tree (np, &xive_irq_domain_ops,NULL );if (WARN_ON (xive_irq_domain ==NULL ))return ;irq_set_default_host (xive_irq_domain);arch/powerpc/sysdev/xive/Kconfig +0
-0arch/powerpc/sysdev/xive/Kconfig config PPC_XIVE · line 8
Quoted 7 lines
arch/powerpc/sysdev/xive/Kconfig +0
-1@@ -3,7 +3,6 @@ config PPC_XIVE bool select PPC_SMP_MUXED_IPI select HARDIRQS_SW_RESEND - select IRQ_DOMAIN_NOMAP config PPC_XIVE_NATIVE boolAs long as this works, I'm happy with one less no-map user.
Acked-by Marc Zyngier <[email protected]>M.
Michael Ellerman
AppliedQuoted 10 lines · Cédric Le Goater
Commit
4f86a06e2d6e("irqdomain: Make normal and nomap irqdomains exclusive") introduced anIRQ_DOMAIN_FLAG_NO_MAPflag to isolate the 'nomap' domains still in use under the powerpc arch. With this new flag, therevmap_treeof the IRQ domain is not used anymore. This change broke the support of shared LSIs [1] in the XIVE driver because it was relying on a lookup in therevmap_treeto query previously mapped interrupts. Linux now creates two distinct IRQ mappings on the same HW IRQ which can lead to unexpected behavior in the drivers.[...]
Applied to powerpc/fixes.
[1/1] powerpc/xive: Change IRQ domain to a tree domain
https://git.kernel.org/powerpc/c/8e80a73fa9a7747e3e8255cb149c543aabf65a24
cheers