irqchip/gic-v3: Disable pseudo NMIs on Mediatek Chromebooks w/ bad FW

When trying to turn on the "pseudo NMI" kernel feature in Linux, it was discovered that all Mediatek-based Chromebooks that ever shipped (at least ones with GICv3) had a firmware bug where they wouldn't save certain GIC "GICR" registers properly. If a processor ever entered a suspend/idle mode where the GICR registers lost state then they'd be reset to their default state.

As a result of the bug, if you try to enable "pseudo NMIs" on the affected devices then certain interrupts will unexpectedly get promoted to be "pseudo NMIs" and cause crashes / freezes / general mayhem.

ChromeOS is looking to start turning on "pseudo NMIs" in production to make crash reports more actionable. To do so, we will release firmware updates for at least some of the affected Mediatek Chromebooks. However, even when we update the firmware of a Chromebook it's always possible that a user will end up booting with old firmware. We need to be able to detect when we're running with firmware that will crash and burn if pseudo NMIs are enabled.

The current plan is:
* Update the device trees of all affected Chromebooks to include the 'mediatek,broken-save-restore-fw' property. The kernel can use this to know not to enable certain features like "pseudo NMI". NOTE:
device trees for Chromebooks are never baked into the firmware but are bundled with the kernel. A kernel will never be configured to use "pseudo NMIs" and be bundled with an old device tree.
* When we get a fixed firmware for one of these Chromebooks, it will patch the device tree to remove this property.

For some details, you can also see the public bug
<https://issuetracker.google.com/281831288>

Reviewed-by Julius Werner <[email protected]>

Signed-off-by Douglas Anderson <[email protected]>

Some Chromebooks with Mediatek SoCs have a problem where the firmware doesn't properly save/restore certain GICR registers. Newer Chromebooks should fix this issue and we may be able to do firmware updates for old Chromebooks. At the moment, the only known issue with these Chromebooks is that we can't enable "pseudo NMIs" since the priority register can be lost. Enabling "pseudo NMIs" on Chromebooks with the problematic firmware causes crashes and freezes.

Let's detect devices with this problem and then disable "pseudo NMIs" on them. We'll detect the problem by looking for the presence of the "mediatek,broken-save-restore-fw" property in the GIC device tree node. Any devices with fixed firmware will not have this property.

Our detection plan works because we never bake a Chromebook's device tree into firmware. Instead, device trees are always bundled with the kernel. We'll update the device trees of all affected Chromebooks and then we'll never enable "pseudo NMI" on a kernel that is bundled with old device trees. When a firmware update is shipped that fixes this issue it will know to patch the device tree to remove the property.

In order to make this work, the quick detection mechanism of the GICv3 code is extended to be able to look for properties in addition to looking at "compatible".

Reviewed-by Julius Werner <[email protected]>

Signed-off-by Douglas Anderson <[email protected]>

Il 15/05/23 22:13, Douglas Anderson ha scritto:

Quoted 24 lines

Some Chromebooks with Mediatek SoCs have a problem where the firmware doesn't properly save/restore certain GICR registers. Newer Chromebooks should fix this issue and we may be able to do firmware updates for old Chromebooks. At the moment, the only known issue with these Chromebooks is that we can't enable "pseudo NMIs" since the priority register can be lost. Enabling "pseudo NMIs" on Chromebooks with the problematic firmware causes crashes and freezes.

Let's detect devices with this problem and then disable "pseudo NMIs" on them. We'll detect the problem by looking for the presence of the "mediatek,broken-save-restore-fw" property in the GIC device tree node. Any devices with fixed firmware will not have this property.

Our detection plan works because we never bake a Chromebook's device tree into firmware. Instead, device trees are always bundled with the kernel. We'll update the device trees of all affected Chromebooks and then we'll never enable "pseudo NMI" on a kernel that is bundled with old device trees. When a firmware update is shipped that fixes this issue it will know to patch the device tree to remove the property.

In order to make this work, the quick detection mechanism of the GICv3 code is extended to be able to look for properties in addition to looking at "compatible".

Quoted 2 lines

Reviewed-by Julius Werner <[email protected]>

Signed-off-by Douglas Anderson <[email protected]>

I don't like firmware removing properties from my devicetrees and I'd like this issue to get addressed in another way (use a scratch register? and check it in Linux drivers to determine if the issue is not present: if scratch contains BIT(x), do not parse the quirk) but that's a different discussion which is a bit out of context for this patch, so:

Reviewed-by AngeloGioacchino Del Regno <[email protected]>

Il 15/05/23 22:13, Douglas Anderson ha scritto:

Quoted 24 lines

Some Chromebooks with Mediatek SoCs have a problem where the firmware doesn't properly save/restore certain GICR registers. Newer Chromebooks should fix this issue and we may be able to do firmware updates for old Chromebooks. At the moment, the only known issue with these Chromebooks is that we can't enable "pseudo NMIs" since the priority register can be lost. Enabling "pseudo NMIs" on Chromebooks with the problematic firmware causes crashes and freezes.

Let's detect devices with this problem and then disable "pseudo NMIs" on them. We'll detect the problem by looking for the presence of the "mediatek,broken-save-restore-fw" property in the GIC device tree node. Any devices with fixed firmware will not have this property.

Our detection plan works because we never bake a Chromebook's device tree into firmware. Instead, device trees are always bundled with the kernel. We'll update the device trees of all affected Chromebooks and then we'll never enable "pseudo NMI" on a kernel that is bundled with old device trees. When a firmware update is shipped that fixes this issue it will know to patch the device tree to remove the property.

In order to make this work, the quick detection mechanism of the GICv3 code is extended to be able to look for properties in addition to looking at "compatible".

Quoted 2 lines

Reviewed-by Julius Werner <[email protected]>

Signed-off-by Douglas Anderson <[email protected]>

I don't like firmware removing properties from my devicetrees and I'd like this issue to get addressed in another way (use a scratch register? and check it in Linux drivers to determine if the issue is not present: if scratch contains BIT(x), do not parse the quirk) but that's a different discussion which is a bit out of context for this patch, so:

Some Chromebooks with Mediatek SoCs have a problem where the firmware doesn't properly save/restore certain GICR registers. Newer Chromebooks should fix this issue and we may be able to do firmware updates for old Chromebooks. At the moment, the only known issue with these Chromebooks is that we can't enable "pseudo NMIs" since the priority register can be lost. Enabling "pseudo NMIs" on Chromebooks with the problematic firmware causes crashes and freezes.

Let's detect devices with this problem and then disable "pseudo NMIs" on them. We'll detect the problem by looking for the presence of the "mediatek,broken-save-restore-fw" property in the GIC device tree node. Any devices with fixed firmware will not have this property.

Our detection plan works because we never bake a Chromebook's device tree into firmware. Instead, device trees are always bundled with the kernel. We'll update the device trees of all affected Chromebooks and then we'll never enable "pseudo NMI" on a kernel that is bundled with old device trees. When a firmware update is shipped that fixes this issue it will know to patch the device tree to remove the property.

In order to make this work, the quick detection mechanism of the GICv3 code is extended to be able to look for properties in addition to looking at "compatible".

Reviewed-by Julius Werner <[email protected]>

Signed-off-by Douglas Anderson <[email protected]>

---

Changes in v2:
- mediatek,gicr-save-quirk => mediatek,broken-save-restore-fw

drivers/irqchip/irq-gic-common.c +1 -1

                if (quirks->init(data))
                        pr_info("GIC: enabling workaround for %s\n",

@@ -28,7 +32,7 @@ void gic_enable_quirks(u32 iidr, const struct gic_quirk *quirks,
                void *data)
 {
        for (; quirks->desc; quirks++) {
-               if (quirks->compatible)
+               if (quirks->compatible || quirks->property)
                        continue;
                if (quirks->iidr != (quirks->mask & iidr))
                        continue;

drivers/irqchip/irq-gic-common.h +1 -0

index 27e3d4ed4f32..3db4592cda1c 100644
@@ -13,6 +13,7 @@
 struct gic_quirk {
        const char *desc;
        const char *compatible;
+       const char *property;
        bool (*init)(void *data);
        u32 iidr;
        u32 mask;

drivers/irqchip/irq-gic-v3.c +20 -1

index 6fcee221f201..a605aa79435a 100644
@@ -39,6 +39,7 @@
 
 #define FLAGS_WORKAROUND_GICR_WAKER_MSM8996    (1ULL << 0)
 #define FLAGS_WORKAROUND_CAVIUM_ERRATUM_38539  (1ULL << 1)
+#define FLAGS_WORKAROUND_MTK_GICR_SAVE         (1ULL << 2)
 
 #define GIC_IRQ_TYPE_PARTITION (GIC_IRQ_TYPE_LPI + 1)
 

@@ -1720,6 +1721,15 @@ static bool gic_enable_quirk_msm8996(void *data)
        return true;
 }
 
+static bool gic_enable_quirk_mtk_gicr(void *data)
+{
+       struct gic_chip_data *d = data;
+
+       d->flags |= FLAGS_WORKAROUND_MTK_GICR_SAVE;
+
+       return true;
+}
+
 static bool gic_enable_quirk_cavium_38539(void *data)
 {
        struct gic_chip_data *d = data;

@@ -1792,6 +1802,11 @@ static const struct gic_quirk gic_quirks[] = {
                .compatible = "qcom,msm8996-gic-v3",
                .init   = gic_enable_quirk_msm8996,
        },
+       {
+               .desc   = "GICv3: Mediatek Chromebook GICR save problem",
+               .property = "mediatek,broken-save-restore-fw",
+               .init   = gic_enable_quirk_mtk_gicr,
+       },
        {
                .desc   = "GICv3: HIP06 erratum 161010803",
                .iidr   = 0x0204043b,

@@ -1834,6 +1849,11 @@ static void gic_enable_nmi_support(void)
        if (!gic_prio_masking_enabled())
                return;
 
+       if (gic_data.flags & FLAGS_WORKAROUND_MTK_GICR_SAVE) {
+               pr_warn("Skipping NMI enable due to firmware issues\n");
+               return;
+       }
+
        ppi_nmi_refs = kcalloc(gic_data.ppi_nr, sizeof(*ppi_nmi_refs), GFP_KERNEL);
        if (!ppi_nmi_refs)
                return;
--

2.40.1.606.ga4b1b128d6-goog

Hi Douglas,

Quoted 24 lines · Douglas Anderson

Some Chromebooks with Mediatek SoCs have a problem where the firmware doesn't properly save/restore certain GICR registers. Newer Chromebooks should fix this issue and we may be able to do firmware updates for old Chromebooks. At the moment, the only known issue with these Chromebooks is that we can't enable "pseudo NMIs" since the priority register can be lost. Enabling "pseudo NMIs" on Chromebooks with the problematic firmware causes crashes and freezes.

Let's detect devices with this problem and then disable "pseudo NMIs" on them. We'll detect the problem by looking for the presence of the "mediatek,broken-save-restore-fw" property in the GIC device tree node. Any devices with fixed firmware will not have this property.

Our detection plan works because we never bake a Chromebook's device tree into firmware. Instead, device trees are always bundled with the kernel. We'll update the device trees of all affected Chromebooks and then we'll never enable "pseudo NMI" on a kernel that is bundled with old device trees. When a firmware update is shipped that fixes this issue it will know to patch the device tree to remove the property.

In order to make this work, the quick detection mechanism of the GICv3 code is extended to be able to look for properties in addition to looking at "compatible".

Quoted 6 lines

Reviewed-by Julius Werner <[email protected]>

Signed-off-by Douglas Anderson <[email protected]>

---

Changes in v2:
- mediatek,gicr-save-quirk => mediatek,broken-save-restore-fw

Thanks for your patch, which is now commit 44bd78dd2b8897f5 ("irqchip/gic-v3: Disable pseudo NMIs on Mediatek devices w/ firmware issues") in v6.4-rc4.

This causes enabling an unrelated workaround on R-Car V4H:

GIC: enabling workaround for GICv3: Cavium erratum 38539

drivers/irqchip/irq-gic-common.c void gic_enable_of_quirks(const struct device_node *np, · line 24

Quoted 13 lines

drivers/irqchip/irq-gic-common.c +5 -1

@@ -16,7 +16,11 @@ void gic_enable_of_quirks(const struct device_node *np,
                          const struct gic_quirk *quirks, void *data)
 {
        for (; quirks->desc; quirks++) {
-               if (!of_device_is_compatible(np, quirks->compatible))
+               if (quirks->compatible &&
+                   !of_device_is_compatible(np, quirks->compatible))
+                       continue;
+               if (quirks->property &&
+                   !of_property_read_bool(np, quirks->property))
                        continue;

Presumably the loop should continue if none of quirks-compatible or quirks->property is set?

Quoted 40 lines · Geert Uytterhoeven

Quoted 24 lines · Douglas Anderson

Some Chromebooks with Mediatek SoCs have a problem where the firmware doesn't properly save/restore certain GICR registers. Newer Chromebooks should fix this issue and we may be able to do firmware updates for old Chromebooks. At the moment, the only known issue with these Chromebooks is that we can't enable "pseudo NMIs" since the priority register can be lost. Enabling "pseudo NMIs" on Chromebooks with the problematic firmware causes crashes and freezes.

Let's detect devices with this problem and then disable "pseudo NMIs" on them. We'll detect the problem by looking for the presence of the "mediatek,broken-save-restore-fw" property in the GIC device tree node. Any devices with fixed firmware will not have this property.

Our detection plan works because we never bake a Chromebook's device tree into firmware. Instead, device trees are always bundled with the kernel. We'll update the device trees of all affected Chromebooks and then we'll never enable "pseudo NMI" on a kernel that is bundled with old device trees. When a firmware update is shipped that fixes this issue it will know to patch the device tree to remove the property.

In order to make this work, the quick detection mechanism of the GICv3 code is extended to be able to look for properties in addition to looking at "compatible".

Quoted 6 lines

Reviewed-by Julius Werner <[email protected]>

Signed-off-by Douglas Anderson <[email protected]>

---

Changes in v2:
- mediatek,gicr-save-quirk => mediatek,broken-save-restore-fw

Thanks for your patch, which is now commit 44bd78dd2b8897f5 ("irqchip/gic-v3: Disable pseudo NMIs on Mediatek devices w/ firmware issues") in v6.4-rc4.

This causes enabling an unrelated workaround on R-Car V4H:

GIC: enabling workaround for GICv3: Cavium erratum 38539

Quoted 16 lines

drivers/irqchip/irq-gic-common.c void gic_enable_of_quirks(const struct device_node *np, · line 24

Quoted 13 lines

drivers/irqchip/irq-gic-common.c +5 -1

@@ -16,7 +16,11 @@ void gic_enable_of_quirks(const struct device_node *np,
                          const struct gic_quirk *quirks, void *data)
 {
        for (; quirks->desc; quirks++) {
-               if (!of_device_is_compatible(np, quirks->compatible))
+               if (quirks->compatible &&
+                   !of_device_is_compatible(np, quirks->compatible))
+                       continue;
+               if (quirks->property &&
+                   !of_property_read_bool(np, quirks->property))
                        continue;

Presumably the loop should continue if none of quirks-compatible or quirks->property is set?

Indeed, thanks for pointing that out. Can you give the following hack a go (compile tested only)?

drivers/irqchip/irq-gic-common.c +2 -0

index de47b51cdadb..7b591736ab58 100644
@@ -16,6 +16,8 @@ void gic_enable_of_quirks(const struct device_node *np,
                          const struct gic_quirk *quirks, void *data)
 {
        for (; quirks->desc; quirks++) {
+               if (!quirks->compatible && !quirks->property)
+                       continue;
                if (quirks->compatible &&
                    !of_device_is_compatible(np, quirks->compatible))
                        continue;

If that works for you, I'll queue it ASAP.

Quoted 43 lines · Geert Uytterhoeven

Hi Douglas,

Quoted 24 lines · Douglas Anderson

Some Chromebooks with Mediatek SoCs have a problem where the firmware doesn't properly save/restore certain GICR registers. Newer Chromebooks should fix this issue and we may be able to do firmware updates for old Chromebooks. At the moment, the only known issue with these Chromebooks is that we can't enable "pseudo NMIs" since the priority register can be lost. Enabling "pseudo NMIs" on Chromebooks with the problematic firmware causes crashes and freezes.

Let's detect devices with this problem and then disable "pseudo NMIs" on them. We'll detect the problem by looking for the presence of the "mediatek,broken-save-restore-fw" property in the GIC device tree node. Any devices with fixed firmware will not have this property.

Our detection plan works because we never bake a Chromebook's device tree into firmware. Instead, device trees are always bundled with the kernel. We'll update the device trees of all affected Chromebooks and then we'll never enable "pseudo NMI" on a kernel that is bundled with old device trees. When a firmware update is shipped that fixes this issue it will know to patch the device tree to remove the property.

In order to make this work, the quick detection mechanism of the GICv3 code is extended to be able to look for properties in addition to looking at "compatible".

Quoted 6 lines

Reviewed-by Julius Werner <[email protected]>

Signed-off-by Douglas Anderson <[email protected]>

---

Changes in v2:
- mediatek,gicr-save-quirk => mediatek,broken-save-restore-fw

Thanks for your patch, which is now commit 44bd78dd2b8897f5 ("irqchip/gic-v3: Disable pseudo NMIs on Mediatek devices w/ firmware issues") in v6.4-rc4.

This causes enabling an unrelated workaround on R-Car V4H:

GIC: enabling workaround for GICv3: Cavium erratum 38539

Quoted 16 lines

drivers/irqchip/irq-gic-common.c void gic_enable_of_quirks(const struct device_node *np, · line 24

Quoted 13 lines

drivers/irqchip/irq-gic-common.c +5 -1

@@ -16,7 +16,11 @@ void gic_enable_of_quirks(const struct device_node *np,
                          const struct gic_quirk *quirks, void *data)
 {
        for (; quirks->desc; quirks++) {
-               if (!of_device_is_compatible(np, quirks->compatible))
+               if (quirks->compatible &&
+                   !of_device_is_compatible(np, quirks->compatible))
+                       continue;
+               if (quirks->property &&
+                   !of_property_read_bool(np, quirks->property))
                        continue;

Presumably the loop should continue if none of quirks-compatible or quirks->property is set?

Indeed, thanks for pointing that out. Can you give the following hack a go (compile tested only)?

drivers/irqchip/irq-gic-common.c +0 -0

index de47b51cdadb..7b591736ab58 100644

Firmware shipped on mt8183 Chromebooks is affected by the GICR save/restore issue as described by the patch ("dt-bindings:
interrupt-controller: arm,gic-v3: Add quirk for Mediatek SoCs w/ broken FW"). Add the quirk property.

Fixes cd894e274b74 ("arm64: dts: mt8183: Add krane-sku176 board")

Reviewed-by Julius Werner <[email protected]>

Signed-off-by Douglas Anderson <[email protected]>

Firmware shipped on mt8192 Chromebooks is affected by the GICR save/restore issue as described by the patch ("dt-bindings:
interrupt-controller: arm,gic-v3: Add quirk for Mediatek SoCs w/ broken FW"). Add the quirk property.

Fixes 331fae2fc922 ("arm64: dts: mediatek: Introduce MT8192-based Asurada board family")

Reviewed-by Julius Werner <[email protected]>

Signed-off-by Douglas Anderson <[email protected]>

Firmware shipped on mt8195 Chromebooks is affected by the GICR save/restore issue as described by the patch ("dt-bindings:
interrupt-controller: arm,gic-v3: Add quirk for Mediatek SoCs w/ broken FW"). Add the quirk property.

Fixes 5eb2e303ec6b ("arm64: dts: mediatek: Introduce MT8195 Cherry platform's Tomato")

Reviewed-by Julius Werner <[email protected]>

Signed-off-by Douglas Anderson <[email protected]>

As talked about in the bindings patch included in this series ("dt-bindings: interrupt-controller: arm,gic-v3: Add quirk for Mediatek SoCs w/ broken FW"), many Mediatek-based Chromebooks shipped with firmware that doesn't properly save/restore some GICR registers. This causes the system to crash if "pseudo NMIs" are turned on.

This series makes sure that we never allow turning on "pseudo NMIs" if we are running with the problematic firmware.

The patches in this series can land in any order and can go through entirely different trees. None of the patches are harmful on their own, but to get things fixed we need all of them.

v2 fixes the quirk name and also moves the quirk out of the SoC.dtsi file and into the Chromebook file. This, unfortunately, means that mt8186-based Chromebooks are no longer handled since they don't appear to be upstream yet. :(

Changes in v2:
- "when CPUs are powered" => "when the GIC redistributors are..."
- Changed "Fixes" tag.
- Moved from mt8183.dtsi to mt8183-kukui.dtsi
- Moved from mt8192.dtsi to mt8192-asurada.dtsi
- Moved from mt8195.dtsi to mt8195-cherry.dtsi
- mediatek,gicr-save-quirk => mediatek,broken-save-restore-fw

Douglas Anderson (5):
dt-bindings: interrupt-controller: arm,gic-v3: Add quirk for Mediatek
SoCs w/ broken FW
irqchip/gic-v3: Disable pseudo NMIs on Mediatek devices w/ firmware
issues
arm64: dts: mediatek: mt8183: Add mediatek,broken-save-restore-fw to
kukui
arm64: dts: mediatek: mt8192: Add mediatek,broken-save-restore-fw to
asurada
arm64: dts: mediatek: mt8195: Add mediatek,broken-save-restore-fw to
cherry

 .../interrupt-controller/arm,gic-v3.yaml      |  6 ++++++
 .../arm64/boot/dts/mediatek/mt8183-kukui.dtsi |  4 ++++
 .../boot/dts/mediatek/mt8192-asurada.dtsi     |  4 ++++
 .../boot/dts/mediatek/mt8195-cherry.dtsi      |  4 ++++
 drivers/irqchip/irq-gic-common.c              |  8 ++++++--
 drivers/irqchip/irq-gic-common.h              |  1 +
 drivers/irqchip/irq-gic-v3.c                  | 20 +++++++++++++++++++
 7 files changed, 45 insertions(+), 2 deletions(-)

Quoted 40 lines · Douglas Anderson

As talked about in the bindings patch included in this series ("dt-bindings: interrupt-controller: arm,gic-v3: Add quirk for Mediatek SoCs w/ broken FW"), many Mediatek-based Chromebooks shipped with firmware that doesn't properly save/restore some GICR registers. This causes the system to crash if "pseudo NMIs" are turned on.

This series makes sure that we never allow turning on "pseudo NMIs" if we are running with the problematic firmware.

The patches in this series can land in any order and can go through entirely different trees. None of the patches are harmful on their own, but to get things fixed we need all of them.

v2 fixes the quirk name and also moves the quirk out of the SoC.dtsi file and into the Chromebook file. This, unfortunately, means that mt8186-based Chromebooks are no longer handled since they don't appear to be upstream yet. :(

Changes in v2:
- "when CPUs are powered" => "when the GIC redistributors are..."
- Changed "Fixes" tag.
- Moved from mt8183.dtsi to mt8183-kukui.dtsi
- Moved from mt8192.dtsi to mt8192-asurada.dtsi
- Moved from mt8195.dtsi to mt8195-cherry.dtsi
- mediatek,gicr-save-quirk => mediatek,broken-save-restore-fw

Douglas Anderson (5):
dt-bindings: interrupt-controller: arm,gic-v3: Add quirk for Mediatek
SoCs w/ broken FW
irqchip/gic-v3: Disable pseudo NMIs on Mediatek devices w/ firmware
issues
arm64: dts: mediatek: mt8183: Add mediatek,broken-save-restore-fw to
kukui
arm64: dts: mediatek: mt8192: Add mediatek,broken-save-restore-fw to
asurada
arm64: dts: mediatek: mt8195: Add mediatek,broken-save-restore-fw to
cherry

Quoted 8 lines

  .../interrupt-controller/arm,gic-v3.yaml      |  6 ++++++
  .../arm64/boot/dts/mediatek/mt8183-kukui.dtsi |  4 ++++
  .../boot/dts/mediatek/mt8192-asurada.dtsi     |  4 ++++
  .../boot/dts/mediatek/mt8195-cherry.dtsi      |  4 ++++
  drivers/irqchip/irq-gic-common.c              |  8 ++++++--
  drivers/irqchip/irq-gic-common.h              |  1 +
  drivers/irqchip/irq-gic-v3.c                  | 20 +++++++++++++++++++
  7 files changed, 45 insertions(+), 2 deletions(-)

I'll take the first two patches as fixes. The rest can be merged via the soc tree as required.

	M.