Skip to contentSkip to conversation

arm64: ACPI GTDT watchdog fixes

PATCH Applied Marc Zyngier 6 participants 13 messages Apr 21, 2021 – Apr 23, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover arm64: ACPI GTDT watchdog fixes

Dann recently reported that his ThunderX machine failed to boot since 64b499d8df40 ("irqchip/gic-v3: Configure SGIs as standard interrupts"), with a not so pretty crash while trying to send an IPI.

It turned out to be caused by a mix of broken firmware and a buggy GTDT watchdog driver. Both have forever been buggy, but the above commit revealed that the error handling path of the driver was probably the worse part of it all.

Anyway, this short series has two goals:
- handle broken firmware in a less broken way
- make sure that the route cause of the problem can be identified quickly

Thanks,

	M.

Marc Zyngier (2):
ACPI: GTDT: Don't corrupt interrupt mappings on watchdow probe failure
ACPI: irq: Prevent unregistering of GIC SGIs

 drivers/acpi/arm64/gtdt.c | 10 ++++++----
 drivers/acpi/irq.c        |  6 +++++-
 2 files changed, 11 insertions(+), 5 deletions(-)
Signature · 1 line
2.29.2
#1 lore raw

PATCH 1/2 ACPI: GTDT: Don't corrupt interrupt mappings on watchdow probe failure

When failing the driver probe because of invalid firmware properties, the GTDT driver unmaps the interrupt that it mapped earlier.

However, it never checks whether the mapping of the interrupt actially succeeded. Even more, should the firmware report an illegal interrupt number that overlaps with the GIC SGI range, this can result in an IPI being unmapped, and subsequent fireworks (as reported by Dann Frazier).

Rework the driver to have a slightly saner behaviour and actually check whether the interrupt has been mapped before unmapping things.

Fixes ca9ae5ec4ef0 ("acpi/arm64: Add SBSA Generic Watchdog support in GTDT driver")
Cc Fu Wei <[email protected]>
---
 drivers/acpi/arm64/gtdt.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/acpi/arm64/gtdt.c +6 -4
index f2d0e5915dab..0a0a982f9c28 100644
@@ -329,7 +329,7 @@ static int __init gtdt_import_sbsa_gwdt(struct acpi_gtdt_watchdog *wd,
 					int index)
 {
 	struct platform_device *pdev;
-	int irq = map_gt_gsi(wd->timer_interrupt, wd->timer_flags);
+	int irq;
 
 	/*
 	 * According to SBSA specification the size of refresh and control
@@ -338,7 +338,7 @@ static int __init gtdt_import_sbsa_gwdt(struct acpi_gtdt_watchdog *wd,
 	struct resource res[] = {
 		DEFINE_RES_MEM(wd->control_frame_address, SZ_4K),
 		DEFINE_RES_MEM(wd->refresh_frame_address, SZ_4K),
-		DEFINE_RES_IRQ(irq),
+		{},
 	};
 	int nr_res = ARRAY_SIZE(res);
 
@@ -348,10 +348,11 @@ static int __init gtdt_import_sbsa_gwdt(struct acpi_gtdt_watchdog *wd,
 
 	if (!(wd->refresh_frame_address && wd->control_frame_address)) {
 		pr_err(FW_BUG "failed to get the Watchdog base address.\n");
-		acpi_unregister_gsi(wd->timer_interrupt);
 		return -EINVAL;
 	}
 
+	irq = map_gt_gsi(wd->timer_interrupt, wd->timer_flags);
+	res[2] = (struct resource)DEFINE_RES_IRQ(irq);
 	if (irq <= 0) {
 		pr_warn("failed to map the Watchdog interrupt.\n");
 		nr_res--;
@@ -364,7 +365,8 @@ static int __init gtdt_import_sbsa_gwdt(struct acpi_gtdt_watchdog *wd,
 	 */
 	pdev = platform_device_register_simple("sbsa-gwdt", index, res, nr_res);
 	if (IS_ERR(pdev)) {
-		acpi_unregister_gsi(wd->timer_interrupt);
+		if (irq > 0)
+			acpi_unregister_gsi(wd->timer_interrupt);
 		return PTR_ERR(pdev);
 	}
 
Signature · 1 line
2.29.2
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] ACPI: GTDT: Don't corrupt interrupt mappings on watchdow probe failure

Quoted 12 lines · Marc Zyngier

When failing the driver probe because of invalid firmware properties, the GTDT driver unmaps the interrupt that it mapped earlier.

However, it never checks whether the mapping of the interrupt actially succeeded. Even more, should the firmware report an illegal interrupt number that overlaps with the GIC SGI range, this can result in an IPI being unmapped, and subsequent fireworks (as reported by Dann Frazier).

Rework the driver to have a slightly saner behaviour and actually check whether the interrupt has been mapped before unmapping things.

Signature · 2 lines
Regards,
Sudeep
#3 lore raw

Re: [PATCH 1/2] ACPI: GTDT: Don't corrupt interrupt mappings on watchdow probe failure

In reply to Marc Zyngier

(with stable email fixed, but may need separate posting anyways)

Quoted 9 lines · Marc Zyngier

When failing the driver probe because of invalid firmware properties, the GTDT driver unmaps the interrupt that it mapped earlier.

However, it never checks whether the mapping of the interrupt actially succeeded. Even more, should the firmware report an illegal interrupt number that overlaps with the GIC SGI range, this can result in an IPI being unmapped, and subsequent fireworks (as reported by Dann Frazier).

Quoted 8 lines

Rework the driver to have a slightly saner behaviour and actually check whether the interrupt has been mapped before unmapping things.

Fixes ca9ae5ec4ef0 ("acpi/arm64: Add SBSA Generic Watchdog support in GTDT driver")
                ^^^

I thought I messed up while replying with my fat fingers, but here is the culprit. I got mail delivery failure to my reply, thought I will let you know as it may be difficult to notice this when using git send-email.

Signature · 2 lines
Regards,
Sudeep
#4 lore raw

Re: [PATCH 1/2] ACPI: GTDT: Don't corrupt interrupt mappings on watchdow probe failure

Quoted 21 lines · Sudeep Holla

(with stable email fixed, but may need separate posting anyways)

Quoted 9 lines · Marc Zyngier

When failing the driver probe because of invalid firmware properties, the GTDT driver unmaps the interrupt that it mapped earlier.

However, it never checks whether the mapping of the interrupt actially succeeded. Even more, should the firmware report an illegal interrupt number that overlaps with the GIC SGI range, this can result in an IPI being unmapped, and subsequent fireworks (as reported by Dann Frazier).

Quoted 8 lines

Rework the driver to have a slightly saner behaviour and actually check whether the interrupt has been mapped before unmapping things.

Fixes ca9ae5ec4ef0 ("acpi/arm64: Add SBSA Generic Watchdog support in GTDT driver")
Quoted 5 lines
                ^^^

I thought I messed up while replying with my fat fingers, but here is the culprit. I got mail delivery failure to my reply, thought I will let you know as it may be difficult to notice this when using git send-email.

I'm definitely the one with very fat fingers. I'll fix the patch in my tree.

Thanks for the heads up,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#5 lore raw

PATCH 2/2 ACPI: irq: Prevent unregistering of GIC SGIs

In reply to Marc Zyngier

When using ACPI on arm64, which implies the GIC IRQ model, no table should ever provide a GSI number in the range [0:15], as these are reserved for IPIs.

However, drivers tend to call acpi_unregister_gsi() with any random GSI number provided by half baked tables, which results in an exploding kernel when its IPIs have been unconfigured.

In order to catch this, check for the silly case early, warn that something is going wrong and avoid the above disaster.

---
 drivers/acpi/irq.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/acpi/irq.c +5 -1
index e209081d644b..c68e694fca26 100644
@@ -75,8 +75,12 @@ void acpi_unregister_gsi(u32 gsi)
 {
 	struct irq_domain *d = irq_find_matching_fwnode(acpi_gsi_domain_id,
 							DOMAIN_BUS_ANY);
-	int irq = irq_find_mapping(d, gsi);
+	int irq;
 
+	if (WARN_ON(acpi_irq_model == ACPI_IRQ_MODEL_GIC && gsi < 16))
+		return;
+
+	irq = irq_find_mapping(d, gsi);
 	irq_dispose_mapping(irq);
 }
 EXPORT_SYMBOL_GPL(acpi_unregister_gsi);
Signature · 1 line
2.29.2
Reviewed-by on v1 2/2
#6 lore raw

Re: [PATCH 2/2] ACPI: irq: Prevent unregistering of GIC SGIs

Quoted 11 lines · Marc Zyngier

When using ACPI on arm64, which implies the GIC IRQ model, no table should ever provide a GSI number in the range [0:15], as these are reserved for IPIs.

However, drivers tend to call acpi_unregister_gsi() with any random GSI number provided by half baked tables, which results in an exploding kernel when its IPIs have been unconfigured.

In order to catch this, check for the silly case early, warn that something is going wrong and avoid the above disaster.

Just curious if this is just precaution or do we have a platform doing something stupid like this ?

--
Regards,
Sudeep

#7 lore raw

Re: [PATCH 2/2] ACPI: irq: Prevent unregistering of GIC SGIs

Quoted 10 lines · Sudeep Holla
Quoted 8 lines · Marc Zyngier

When using ACPI on arm64, which implies the GIC IRQ model, no table should ever provide a GSI number in the range [0:15], as these are reserved for IPIs.

However, drivers tend to call acpi_unregister_gsi() with any random GSI number provided by half baked tables, which results in an exploding kernel when its IPIs have been unconfigured.

Quoted 8 lines
Quoted 3 lines

In order to catch this, check for the silly case early, warn that something is going wrong and avoid the above disaster.

Just curious if this is just precaution or do we have a platform doing something stupid like this ?

Without this, it could be really hard to pinpoint which driver messes with IPIs. Having this in place would have caught the GTDT bug much earlier (several years ago actually).

The only reason I managed to track it down in a short amount of time is that the driver actually printed an error message before the kernel exploded while probing a completely unrelated driver. Without this message, I'd still be scratching my head.

The WARN_ON() would definitely point at the guilty party, and keep the kernel running.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Tested-by on the v1 series
#8 lore raw
In reply to Marc Zyngier
Quoted 18 lines · Marc Zyngier

Dann recently reported that his ThunderX machine failed to boot since 64b499d8df40 ("irqchip/gic-v3: Configure SGIs as standard interrupts"), with a not so pretty crash while trying to send an IPI.

It turned out to be caused by a mix of broken firmware and a buggy GTDT watchdog driver. Both have forever been buggy, but the above commit revealed that the error handling path of the driver was probably the worse part of it all.

Anyway, this short series has two goals:
- handle broken firmware in a less broken way
- make sure that the route cause of the problem can be identified quickly

Thanks,

	M.
Quoted 7 lines

Marc Zyngier (2):
ACPI: GTDT: Don't corrupt interrupt mappings on watchdow probe failure
ACPI: irq: Prevent unregistering of GIC SGIs

 drivers/acpi/arm64/gtdt.c | 10 ++++++----
 drivers/acpi/irq.c        |  6 +++++-
 2 files changed, 11 insertions(+), 5 deletions(-)

For the series:

maintainer Tested-by Reviewed-by on the v1 series
#9 lore raw
In reply to Marc Zyngier
Quoted 9 lines · Marc Zyngier

Dann recently reported that his ThunderX machine failed to boot since 64b499d8df40 ("irqchip/gic-v3: Configure SGIs as standard interrupts"), with a not so pretty crash while trying to send an IPI.

It turned out to be caused by a mix of broken firmware and a buggy GTDT watchdog driver. Both have forever been buggy, but the above commit revealed that the error handling path of the driver was probably the worse part of it all.

Quoted 4 lines

Anyway, this short series has two goals:
- handle broken firmware in a less broken way
- make sure that the route cause of the problem can be identified quickly

Tested on Kunpeng920 ARM64 server, didn't any issue after applying this patch set,

Thanks
Hanjun

Reviewed-by on the v1 series
#10 lore raw
In reply to Marc Zyngier
Quoted 18 lines · Marc Zyngier

Dann recently reported that his ThunderX machine failed to boot since 64b499d8df40 ("irqchip/gic-v3: Configure SGIs as standard interrupts"), with a not so pretty crash while trying to send an IPI.

It turned out to be caused by a mix of broken firmware and a buggy GTDT watchdog driver. Both have forever been buggy, but the above commit revealed that the error handling path of the driver was probably the worse part of it all.

Anyway, this short series has two goals:
- handle broken firmware in a less broken way
- make sure that the route cause of the problem can be identified quickly

Thanks,

	M.
Quoted 7 lines

Marc Zyngier (2):
ACPI: GTDT: Don't corrupt interrupt mappings on watchdow probe failure
ACPI: irq: Prevent unregistering of GIC SGIs

 drivers/acpi/arm64/gtdt.c | 10 ++++++----
 drivers/acpi/irq.c        |  6 +++++-
 2 files changed, 11 insertions(+), 5 deletions(-)

Patch(2) needs an ACK from Rafael - usually these patches go via the ARM64 tree but I don't think it is compulsory for this series.

Thank you !

maintainer
#11 lore raw
Quoted 27 lines · Lorenzo Pieralisi
Quoted 18 lines · Marc Zyngier

Dann recently reported that his ThunderX machine failed to boot since 64b499d8df40 ("irqchip/gic-v3: Configure SGIs as standard interrupts"), with a not so pretty crash while trying to send an IPI.

It turned out to be caused by a mix of broken firmware and a buggy GTDT watchdog driver. Both have forever been buggy, but the above commit revealed that the error handling path of the driver was probably the worse part of it all.

Anyway, this short series has two goals:
- handle broken firmware in a less broken way
- make sure that the route cause of the problem can be identified quickly

Thanks,

	M.
Quoted 7 lines

Marc Zyngier (2):
ACPI: GTDT: Don't corrupt interrupt mappings on watchdow probe failure
ACPI: irq: Prevent unregistering of GIC SGIs

 drivers/acpi/arm64/gtdt.c | 10 ++++++----
 drivers/acpi/irq.c        |  6 +++++-
 2 files changed, 11 insertions(+), 5 deletions(-)
Quoted 6 lines

Patch(2) needs an ACK from Rafael - usually these patches go via the ARM64 tree but I don't think it is compulsory for this series.

Thank you !

Thanks Lorenzo.

Rafael, if there are no objections, I'll take these two patches in the arm64 tree.

Signature · 1 line
Catalin
maintainer Applied
#12 lore raw
In reply to Marc Zyngier
Quoted 10 lines · Marc Zyngier

Dann recently reported that his ThunderX machine failed to boot since 64b499d8df40 ("irqchip/gic-v3: Configure SGIs as standard interrupts"), with a not so pretty crash while trying to send an IPI.

It turned out to be caused by a mix of broken firmware and a buggy GTDT watchdog driver. Both have forever been buggy, but the above commit revealed that the error handling path of the driver was probably the worse part of it all.

[...]

Applied to arm64 (for-next/core), thanks!

[1/2] ACPI: GTDT: Don't corrupt interrupt mappings on watchdow probe failure

      https://git.kernel.org/arm64/c/1ecd5b129252
[2/2] ACPI: irq: Prevent unregistering of GIC SGIs
      https://git.kernel.org/arm64/c/2a20b08f06e7
Signature · 1 line
Catalin