irqchip/gic-v3: Workaround for GIC-700 erratum 2941627
PATCH 1/1 irqchip/gic-v3: Workaround for GIC-700 erratum 2941627
GIC700 erratum 2941627 may cause GIC-700 missing SPIs wake requests when SPIs are deactivated while targeting a sleeping CPU - ie a CPU for which the redistributor:
GICR_WAKER.ProcessorSleep == 1
This runtime situation can happen if an SPI that has been activated on a core is retargeted to a different core, it becomes pending and the target core subsequently enters a power state quiescing the respective redistributor.
When this situation is hit, the de-activation carried out on the core that activated the SPI (through either ICC_EOIR1_EL1 or ICC_DIR_EL1 register writes) does not trigger a wake requests for the sleeping GIC redistributor even if the SPI is pending.
Fix the erratum by de-activating the SPI using the redistributor GICD_ICACTIVER register if the runtime conditions require it (ie the IRQ was retargeted between activation and de-activation).
--- Documentation/arm64/silicon-errata.rst | 3 ++ drivers/irqchip/irq-gic-v3.c | 71 +++++++++++++++++++++++++- 2 files changed, 73 insertions(+), 1 deletion(-)
Documentation/arm64/silicon-errata.rst +3 -0
@@ -141,6 +141,9 @@ stable kernels. | ARM | MMU-500 | #841119,826419 | N/A | +----------------+-----------------+-----------------+-----------------------------+ +----------------+-----------------+-----------------+-----------------------------+ +| ARM | GIC-700 | #2941627 | ARM64_ERRATUM_2941627 | ++----------------+-----------------+-----------------+-----------------------------+ ++----------------+-----------------+-----------------+-----------------------------+ | Broadcom | Brahma-B53 | N/A | ARM64_ERRATUM_845719 | +----------------+-----------------+-----------------+-----------------------------+ | Broadcom | Brahma-B53 | N/A | ARM64_ERRATUM_843419 |
drivers/irqchip/irq-gic-v3.c +70 -1
105 lines not on this page · raw message
Signature · 1 line
2.34.1
Marc Zyngier
maintainerLorenzo,
Quoted 12 lines · Lorenzo Pieralisi
GIC700 erratum 2941627 may cause GIC-700 missing SPIs wake requests when SPIs are deactivated while targeting a sleeping CPU - ie a CPU for which the redistributor:
GICR_WAKER.ProcessorSleep == 1
This runtime situation can happen if an SPI that has been activated on a core is retargeted to a different core, it becomes pending and the target core subsequently enters a power state quiescing the respective redistributor.
Quoted 7 lines
When this situation is hit, the de-activation carried out on the core that activated the SPI (through either
ICC_EOIR1_EL1orICC_DIR_EL1register writes) does not trigger a wake requests for the sleeping GIC redistributor even if the SPI is pending.Fix the erratum by de-activating the SPI using the
s/Fix/ Work around/
Quoted 45 lines
redistributor
GICD_ICACTIVERregister if the runtime conditions require it (ie the IRQ was retargeted between activation and de-activation).Signed-off-by Lorenzo Pieralisi <[email protected]>Cc Marc Zyngier <[email protected]>--- Documentation/arm64/silicon-errata.rst | 3 ++ drivers/irqchip/irq-gic-v3.c | 71 +++++++++++++++++++++++++- 2 files changed, 73 insertions(+), 1 deletion(-)Documentation/arm64/silicon-errata.rst +3
-0@@ -141,6 +141,9 @@ stable kernels. | ARM | MMU-500 | #841119,826419 | N/A | +----------------+-----------------+-----------------+-----------------------------+ +----------------+-----------------+-----------------+-----------------------------+ +| ARM | GIC-700 | #2941627 | ARM64_ERRATUM_2941627 | ++----------------+-----------------+-----------------+-----------------------------+ ++----------------+-----------------+-----------------+-----------------------------+ | Broadcom | Brahma-B53 | N/A | ARM64_ERRATUM_845719 | +----------------+-----------------+-----------------+-----------------------------+ | Broadcom | Brahma-B53 | N/A | ARM64_ERRATUM_843419 |drivers/irqchip/irq-gic-v3.c +5
-0@@ -68,6 +68,8 @@ struct gic_chip_data {static void __iomem *t241_dist_base_alias[T241_CHIPS_MAX ]__read_mostly ;static DEFINE_STATIC_KEY_FALSE (gic_nvidia_t241_erratum); +static DEFINE_STATIC_KEY_FALSE (gic_arm64_2941627_erratum); +static struct gic_chip_data gic_data__read_mostly ;static DEFINE_STATIC_KEY_TRUE (supports_deactivate_key);@@ -591,10 +593,35 @@ static void gic_irq_nmi_teardown(struct irq_data *d)gic_irq_set_prio (d,GICD_INT_DEF_PRI ); } +static bool gic_arm64_erratum_2941627_needed (struct irq_data *d) +{ +if (!static_branch_unlikely (&gic_arm64_2941627_erratum))
drivers/irqchip/irq-gic-v3.c static void gic_irq_nmi_teardown(struct irq_data *d) · line 608
Quoted 10 lines
drivers/irqchip/irq-gic-v3.c +10
-0+return false; + +/* +* The workaround is needed if the IRQ is an SPI and +* the target cpu is different from the one we are +* executing on. +*/ +return !((gic_irq_in_rdist (d)) ||gic_irq (d) >=8192 || +cpumask_equal (irq_data_get_effective_affinity_mask (d), +cpumask_of (smp_processor_id ())));
I dislike this statement for multiple reasons:
- it is written as a negation, making it harder than strictly necessary to parse as it is the opposite of the comment above
- gic_irq_in_rdist() and gic_irq(d) >= 8192 are two ways of checking the interrupt range -- maybe we should just do that
- cpumask_equal() is *slow* if you have more that 64 CPUs, something that is increasingly common -- a better option would be to check whether the current CPU is in the mask or not, which would be enough as we only have a single affinity bit set
- smp_processor_id() can check for preemption, which is pointless here, as we're doing things under the irq_desc raw spinlock.
I would expect something like:
enum gic_intid_range range = get_intid_range(d); return (range == SGI_RANGE || range == ESPI_RANGE) && !cpumask_test_cpu(raw_smp_processor_id(), irq_data_get_effective_affinity_mask(d));
Quoted 40 lines
drivers/irqchip/irq-gic-v3.c +21
-1+} +static void gic_eoi_irq (struct irq_data *d) {write_gicreg (gic_irq (d),ICC_EOIR1_EL1 );isb (); + +if (gic_arm64_erratum_2941627_needed (d)) { +/* +* Make sure the GIC stream deactivate packet +* issued by ICC_EOIR1_EL1 has completed before +* deactivating through GICD_IACTIVER. +*/ +dsb (sy); +gic_poke_irq (d,GICD_ICACTIVER ); + } }static void gic_eoimode1_eoi_irq (struct irq_data *d)@@ -605,7 +632,11 @@ static void gic_eoimode1_eoi_irq(struct irq_data *d)*/ if (gic_irq (d) >=8192 ||irqd_is_forwarded_to_vcpu (d))return ; -gic_write_dir (gic_irq (d)); + +if (!gic_arm64_erratum_2941627_needed (d)) +gic_write_dir (gic_irq (d)); +else +gic_poke_irq (d,GICD_ICACTIVER ); }static int gic_set_type (struct irq_data *d,unsigned int type)@@ -1796,6 +1827,25 @@ static bool gic_enable_quirk_nvidia_t241(void *data)return true; } +static bool gic_enable_quirk_arm64_2941627 (void *data) +{ +/* +* If CPUidle is not enabled the erratum runtime
drivers/irqchip/irq-gic-v3.c static bool gic_enable_quirk_nvidia_t241(void *data) · line 1843
Quoted 10 lines
drivers/irqchip/irq-gic-v3.c +10
-0+* conditions can't be hit, since that requires: +* +* - A core entering a deep power state with +* the associated GIC redistributor asleep +* and an IRQ active and pending targeted at it +* - A different core handling the IRQ and +* related GIC operations at the same time +*/ +if (!IS_ENABLED (CONFIG_CPU_IDLE )) +return false;
Could this still hit on a system that traps WFI to EL3 and uses this as a way to enter a low-power mode?
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Quoted 84 lines · Marc Zyngier
Lorenzo,
Quoted 12 lines · Lorenzo Pieralisi
GIC700 erratum 2941627 may cause GIC-700 missing SPIs wake requests when SPIs are deactivated while targeting a sleeping CPU - ie a CPU for which the redistributor:
GICR_WAKER.ProcessorSleep == 1
This runtime situation can happen if an SPI that has been activated on a core is retargeted to a different core, it becomes pending and the target core subsequently enters a power state quiescing the respective redistributor.
Quoted 7 lines
When this situation is hit, the de-activation carried out on the core that activated the SPI (through either
ICC_EOIR1_EL1orICC_DIR_EL1register writes) does not trigger a wake requests for the sleeping GIC redistributor even if the SPI is pending.Fix the erratum by de-activating the SPI using the
s/Fix/ Work around/
Quoted 45 lines
redistributor
GICD_ICACTIVERregister if the runtime conditions require it (ie the IRQ was retargeted between activation and de-activation).Signed-off-by Lorenzo Pieralisi <[email protected]>Cc Marc Zyngier <[email protected]>--- Documentation/arm64/silicon-errata.rst | 3 ++ drivers/irqchip/irq-gic-v3.c | 71 +++++++++++++++++++++++++- 2 files changed, 73 insertions(+), 1 deletion(-)Documentation/arm64/silicon-errata.rst +3
-0@@ -141,6 +141,9 @@ stable kernels. | ARM | MMU-500 | #841119,826419 | N/A | +----------------+-----------------+-----------------+-----------------------------+ +----------------+-----------------+-----------------+-----------------------------+ +| ARM | GIC-700 | #2941627 | ARM64_ERRATUM_2941627 | ++----------------+-----------------+-----------------+-----------------------------+ ++----------------+-----------------+-----------------+-----------------------------+ | Broadcom | Brahma-B53 | N/A | ARM64_ERRATUM_845719 | +----------------+-----------------+-----------------+-----------------------------+ | Broadcom | Brahma-B53 | N/A | ARM64_ERRATUM_843419 |drivers/irqchip/irq-gic-v3.c +5
-0@@ -68,6 +68,8 @@ struct gic_chip_data {static void __iomem *t241_dist_base_alias[T241_CHIPS_MAX ]__read_mostly ;static DEFINE_STATIC_KEY_FALSE (gic_nvidia_t241_erratum); +static DEFINE_STATIC_KEY_FALSE (gic_arm64_2941627_erratum); +static struct gic_chip_data gic_data__read_mostly ;static DEFINE_STATIC_KEY_TRUE (supports_deactivate_key);@@ -591,10 +593,35 @@ static void gic_irq_nmi_teardown(struct irq_data *d)gic_irq_set_prio (d,GICD_INT_DEF_PRI ); } +static bool gic_arm64_erratum_2941627_needed (struct irq_data *d) +{ +if (!static_branch_unlikely (&gic_arm64_2941627_erratum))drivers/irqchip/irq-gic-v3.c static void gic_irq_nmi_teardown(struct irq_data *d) · line 608
Quoted 10 lines
drivers/irqchip/irq-gic-v3.c +10
-0+return false; + +/* +* The workaround is needed if the IRQ is an SPI and +* the target cpu is different from the one we are +* executing on. +*/ +return !((gic_irq_in_rdist (d)) ||gic_irq (d) >=8192 || +cpumask_equal (irq_data_get_effective_affinity_mask (d), +cpumask_of (smp_processor_id ())));I dislike this statement for multiple reasons:
Quoted 2 lines
- it is written as a negation, making it harder than strictly necessary to parse as it is the opposite of the comment above
Yes, I agree.
Quoted 10 lines
-
gic_irq_in_rdist()andgic_irq(d)>= 8192 are two ways of checking the interrupt range -- maybe we should just do that-
cpumask_equal()is *slow* if you have more that 64 CPUs, something that is increasingly common -- a better option would be to check whether the current CPU is in the mask or not, which would be enough as we only have a single affinity bit set-
smp_processor_id()can check for preemption, which is pointless here, as we're doing things under theirq_descraw spinlock.
These are valid points and there is no reason why this should not be rewritten as you suggest below.
Quoted 7 lines
I would expect something like:
enum gic_intid_range range = get_intid_range(d); return (range == SGI_RANGE || range == ESPI_RANGE) && !cpumask_test_cpu(raw_smp_processor_id(), irq_data_get_effective_affinity_mask(d));
It should work (and it is easier to read in the process), thanks.
Quoted 45 lines
Quoted 45 lines
drivers/irqchip/irq-gic-v3.c +26
-1+} +static void gic_eoi_irq (struct irq_data *d) {write_gicreg (gic_irq (d),ICC_EOIR1_EL1 );isb (); + +if (gic_arm64_erratum_2941627_needed (d)) { +/* +* Make sure the GIC stream deactivate packet +* issued by ICC_EOIR1_EL1 has completed before +* deactivating through GICD_IACTIVER. +*/ +dsb (sy); +gic_poke_irq (d,GICD_ICACTIVER ); + } }static void gic_eoimode1_eoi_irq (struct irq_data *d)@@ -605,7 +632,11 @@ static void gic_eoimode1_eoi_irq(struct irq_data *d)*/ if (gic_irq (d) >=8192 ||irqd_is_forwarded_to_vcpu (d))return ; -gic_write_dir (gic_irq (d)); + +if (!gic_arm64_erratum_2941627_needed (d)) +gic_write_dir (gic_irq (d)); +else +gic_poke_irq (d,GICD_ICACTIVER ); }static int gic_set_type (struct irq_data *d,unsigned int type)@@ -1796,6 +1827,25 @@ static bool gic_enable_quirk_nvidia_t241(void *data)return true; } +static bool gic_enable_quirk_arm64_2941627 (void *data) +{ +/* +* If CPUidle is not enabled the erratum runtime +* conditions can't be hit, since that requires: +* +* - A core entering a deep power state with +* the associated GIC redistributor asleep +* and an IRQ active and pending targeted at it
Quoted 8 lines
drivers/irqchip/irq-gic-v3.c static bool gic_enable_quirk_nvidia_t241(void *data) · line 1843
Quoted 5 lines
drivers/irqchip/irq-gic-v3.c +5
-0+* - A different core handling the IRQ and +* related GIC operations at the same time +*/ +if (!IS_ENABLED (CONFIG_CPU_IDLE )) +return false;Could this still hit on a system that traps WFI to EL3 and uses this as a way to enter a low-power mode?
That's a valid point, I have not thought about that. If there are set-ups where this is allowed (and I think it *is* architecturally allowed with EL3 saving/restoring context and entering a deep power state - if you asked I suspect you have something concrete in mind :)) this "optimization" must be removed since we can still hit the bug; that's what I shall do for v2.
Thanks,
Lorenzo
Marc Zyngier
maintainerQuoted 25 lines · Lorenzo Pieralisi
Quoted 14 lines · Marc Zyngier
Lorenzo,
Quoted 10 lines · Lorenzo Pieralisi
drivers/irqchip/irq-gic-v3.c +9
-0+static bool gic_enable_quirk_arm64_2941627 (void *data) +{ +/* +* If CPUidle is not enabled the erratum runtime +* conditions can't be hit, since that requires: +* +* - A core entering a deep power state with +* the associated GIC redistributor asleep +* and an IRQ active and pending targeted at it Quoted 8 lines
drivers/irqchip/irq-gic-v3.c static bool gic_enable_quirk_nvidia_t241(void *data) · line 1843
Quoted 5 lines
drivers/irqchip/irq-gic-v3.c +5
-0+* - A different core handling the IRQ and +* related GIC operations at the same time +*/ +if (!IS_ENABLED (CONFIG_CPU_IDLE )) +return false;Could this still hit on a system that traps WFI to EL3 and uses this as a way to enter a low-power mode?
Quoted 6 lines
That's a valid point, I have not thought about that. If there are set-ups where this is allowed (and I think it *is* architecturally allowed with EL3 saving/restoring context and entering a deep power state - if you asked I suspect you have something concrete in mind :)) this "optimization" must be removed since we can still hit the bug; that's what I shall do for v2.
I do not have a concrete example of anyone doing that, but the fact that it is possible by the letter of the architecture makes me think that *someone* out there must be inventive enough to do it.
So I'd rather take the safe option and *always* enable the workaround. And then someone else can rock up and explain why they don't need it.
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
On Tue, Jul 04, 2023 at 03:44:50PM +0100, Marc Zyngier wrote:
[...]
Quoted 20 lines
drivers/irqchip/irq-gic-v3.c static void gic_irq_nmi_teardown(struct irq_data *d) · line 608
Quoted 3 lines
drivers/irqchip/irq-gic-v3.c +3
-0+return !((gic_irq_in_rdist (d)) ||gic_irq (d) >=8192 || +cpumask_equal (irq_data_get_effective_affinity_mask (d), +cpumask_of (smp_processor_id ())));I dislike this statement for multiple reasons:
- it is written as a negation, making it harder than strictly necessary to parse as it is the opposite of the comment above
-
gic_irq_in_rdist()andgic_irq(d)>= 8192 are two ways of checking the interrupt range -- maybe we should just do that-
cpumask_equal()is *slow* if you have more that 64 CPUs, something that is increasingly common -- a better option would be to check whether the current CPU is in the mask or not, which would be enough as we only have a single affinity bit set-
smp_processor_id()can check for preemption, which is pointless here, as we're doing things under theirq_descraw spinlock.
Quoted 8 lines
I would expect something like:
enum gic_intid_range range = get_intid_range(d); return (range == SGI_RANGE || range == ESPI_RANGE) && !cpumask_test_cpu(raw_smp_processor_id(), irq_data_get_effective_affinity_mask(d));
s/SGI/SPI - just noticed, for the records.
Thanks,
Lorenzo
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Marc Zyngier
maintainerQuoted 27 lines · Lorenzo Pieralisi
On Tue, Jul 04, 2023 at 03:44:50PM +0100, Marc Zyngier wrote:
[...]
Quoted 22 lines
drivers/irqchip/irq-gic-v3.c static void gic_irq_nmi_teardown(struct irq_data *d) · line 608
Quoted 3 lines
drivers/irqchip/irq-gic-v3.c +3
-0+return !((gic_irq_in_rdist (d)) ||gic_irq (d) >=8192 || +cpumask_equal (irq_data_get_effective_affinity_mask (d), +cpumask_of (smp_processor_id ())));I dislike this statement for multiple reasons:
- it is written as a negation, making it harder than strictly necessary to parse as it is the opposite of the comment above
-
gic_irq_in_rdist()andgic_irq(d)>= 8192 are two ways of checking the interrupt range -- maybe we should just do that-
cpumask_equal()is *slow* if you have more that 64 CPUs, something that is increasingly common -- a better option would be to check whether the current CPU is in the mask or not, which would be enough as we only have a single affinity bit set-
smp_processor_id()can check for preemption, which is pointless here, as we're doing things under theirq_descraw spinlock.I would expect something like:
Quoted 8 lines
Quoted 6 lines
enum gic_intid_range range = get_intid_range(d); return (range == SGI_RANGE || range == ESPI_RANGE) && !cpumask_test_cpu(raw_smp_processor_id(), irq_data_get_effective_affinity_mask(d));s/SGI/SPI - just noticed, for the records.
Indeed. As you can tell, I didn't really test the damn thing...
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
PATCH v2 1/1 irqchip/gic-v3: Workaround for GIC-700 erratum 2941627
In reply to Lorenzo PieralisiGIC700 erratum 2941627 may cause GIC-700 missing SPIs wake requests when SPIs are deactivated while targeting a sleeping CPU - ie a CPU for which the redistributor:
GICR_WAKER.ProcessorSleep == 1
This runtime situation can happen if an SPI that has been activated on a core is retargeted to a different core, it becomes pending and the target core subsequently enters a power state quiescing the respective redistributor.
When this situation is hit, the de-activation carried out on the core that activated the SPI (through either ICC_EOIR1_EL1 or ICC_DIR_EL1 register writes) does not trigger a wake requests for the sleeping GIC redistributor even if the SPI is pending.
Work around the erratum by de-activating the SPI using the redistributor GICD_ICACTIVER register if the runtime conditions require it (ie the IRQ was retargeted between activation and de-activation).
---
v1 -> v2:
- Updated trigger condition check in gic_arm64_erratum_2941627_needed() according to review
- Removed !CONFIG_CPU_IDLE workaround enablement check
Documentation/arm64/silicon-errata.rst | 3 ++ drivers/irqchip/irq-gic-v3.c | 62 +++++++++++++++++++++++++- 2 files changed, 64 insertions(+), 1 deletion(-)
Documentation/arm64/silicon-errata.rst +3 -0
@@ -141,6 +141,9 @@ stable kernels. | ARM | MMU-500 | #841119,826419 | N/A | +----------------+-----------------+-----------------+-----------------------------+ +----------------+-----------------+-----------------+-----------------------------+ +| ARM | GIC-700 | #2941627 | ARM64_ERRATUM_2941627 | ++----------------+-----------------+-----------------+-----------------------------+ ++----------------+-----------------+-----------------+-----------------------------+ | Broadcom | Brahma-B53 | N/A | ARM64_ERRATUM_845719 | +----------------+-----------------+-----------------+-----------------------------+ | Broadcom | Brahma-B53 | N/A | ARM64_ERRATUM_843419 |
drivers/irqchip/irq-gic-v3.c +61 -1
96 lines not on this page · raw message
Signature · 1 line
2.34.1
The following commit has been merged into the irq/irqchip-fixes branch of irqchip:
The following commit has been merged into the irq/irqchip-fixes branch of irqchip:
Commit-ID: 6fe5c68ee6a1aae0ef291a56001e7888de547fa2 Gitweb: https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms/6fe5c68ee6a1aae0ef291a56001e7888de547fa2 Author: Lorenzo Pieralisi <[email protected]> AuthorDate: Tue, 04 Jul 2023 17:50:34 +02:00 Committer: Marc Zyngier <[email protected]> CommitterDate: Tue, 11 Jul 2023 09:04:31 +01:00
irqchip/gic-v3: Workaround for GIC-700 erratum 2941627
GIC700 erratum 2941627 may cause GIC-700 missing SPIs wake requests when SPIs are deactivated while targeting a sleeping CPU - ie a CPU for which the redistributor:
GICR_WAKER.ProcessorSleep == 1
This runtime situation can happen if an SPI that has been activated on a core is retargeted to a different core, it becomes pending and the target core subsequently enters a power state quiescing the respective redistributor.
When this situation is hit, the de-activation carried out on the core that activated the SPI (through either ICC_EOIR1_EL1 or ICC_DIR_EL1 register writes) does not trigger a wake requests for the sleeping GIC redistributor even if the SPI is pending.
Work around the erratum by de-activating the SPI using the redistributor GICD_ICACTIVER register if the runtime conditions require it (ie the IRQ was retargeted between activation and de-activation).
--- Documentation/arm64/silicon-errata.rst | 3 +- drivers/irqchip/irq-gic-v3.c | 62 ++++++++++++++++++++++++- 2 files changed, 64 insertions(+), 1 deletion(-)
Documentation/arm64/silicon-errata.rst +3 -0
@@ -141,6 +141,9 @@ stable kernels. | ARM | MMU-500 | #841119,826419 | N/A | +----------------+-----------------+-----------------+-----------------------------+ +----------------+-----------------+-----------------+-----------------------------+ +| ARM | GIC-700 | #2941627 | ARM64_ERRATUM_2941627 | ++----------------+-----------------+-----------------+-----------------------------+ ++----------------+-----------------+-----------------+-----------------------------+ | Broadcom | Brahma-B53 | N/A | ARM64_ERRATUM_845719 | +----------------+-----------------+-----------------+-----------------------------+ | Broadcom | Brahma-B53 | N/A | ARM64_ERRATUM_843419 |
drivers/irqchip/irq-gic-v3.c +61 -1
96 lines not on this page · raw message