Skip to contentSkip to conversation

tpm_tis: Disable interrupts on ThinkPad T490s

PATCH Applied Jerry Snitselaar 6 participants 22 messages Oct 15, 2020 – Dec 2, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tpm_tis: Disable interrupts on ThinkPad T490s

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

---
 drivers/char/tpm/tpm_tis.c | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)
drivers/char/tpm/tpm_tis.c +27 -2
index 0b214963539d..4ed6e660273a 100644
@@ -27,6 +27,7 @@
 #include <linux/of.h>
 #include <linux/of_device.h>
 #include <linux/kernel.h>
+#include <linux/dmi.h>
 #include "tpm.h"
 #include "tpm_tis_core.h"
 
@@ -49,8 +50,8 @@ static inline struct tpm_tis_tcg_phy *to_tpm_tis_tcg_phy(struct tpm_tis_data *da
 	return container_of(data, struct tpm_tis_tcg_phy, priv);
 }
 
-static bool interrupts = true;
-module_param(interrupts, bool, 0444);
+static int interrupts = -1;
+module_param(interrupts, int, 0444);
 MODULE_PARM_DESC(interrupts, "Enable interrupts");
 
 static bool itpm;
@@ -63,6 +64,28 @@ module_param(force, bool, 0444);
 MODULE_PARM_DESC(force, "Force device probe rather than using ACPI entry");
 #endif
 
+static int tpm_tis_disable_irq(const struct dmi_system_id *d)
+{
+	if (interrupts == -1) {
+		pr_notice("tpm_tis: %s detected: disabling interrupts.\n", d->ident);
+		interrupts = 0;
+	}
+
+	return 0;
+}
+
+static const struct dmi_system_id tpm_tis_dmi_table[] = {
+	{
+		.callback = tpm_tis_disable_irq,
+		.ident = "ThinkPad T490s",
+		.matches = {
+			DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
+			DMI_MATCH(DMI_PRODUCT_VERSION, "ThinkPad T490s"),
+		},
+	},
+	{}
+};
+
 #if defined(CONFIG_PNP) && defined(CONFIG_ACPI)
 static int has_hid(struct acpi_device *dev, const char *hid)
 {
@@ -192,6 +215,8 @@ static int tpm_tis_init(struct device *dev, struct tpm_info *tpm_info)
 	int irq = -1;
 	int rc;
 
+	dmi_check_system(tpm_tis_dmi_table);
+
 	rc = check_acpi_tpm2(dev);
 	if (rc)
 		return rc;
Signature · 1 line
2.28.0
#1 lore raw

James should this get tacked on the end of your patchset?

Regards,
Jerry

#2 lore raw
In reply to Jerry Snitselaar
Quoted 13 lines · Jerry Snitselaar

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

#3 lore raw

Hi,

Quoted 15 lines · Matthew Garrett
Quoted 13 lines · Jerry Snitselaar

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 10 lines

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

IIRC then at least on the T490 the irq storm caused systems to not boot in some cases. I guess if we detect the storm and disable the irq we might fix that... OTOH this problem seems to only hit a certain generation of Thinkpads so with some luck the denylist should not be too big and the denylist approach should work.

Regards,

Hans

#4 lore raw
In reply to Matthew Garrett

Matthew Garrett @ 2020-10-15 15:39 MST:

Quoted 15 lines · Jerry Snitselaar
Quoted 13 lines

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 10 lines

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

I believe Matthew is correct here. I found another system today with completely different vendor for both the system and the tpm chip. In addition another Lenovo model, the L490, has the issue.

This initial attempt at a solution like Matthew suggested works on the system I found today, but I imagine it is all sorts of wrong. In the 2 systems where I've seen it, there are about 100000 interrupts in around 1.5 seconds, and then the irq code shuts down the interrupt because they aren't being handled.

drivers/char/tpm/tpm_tis_core.c +37 -5
index 49ae09ac604f..478e9d02a3fa 100644
@@ -27,6 +27,11 @@
 #include "tpm.h"
 #include "tpm_tis_core.h"
 
+static unsigned int time_start = 0;
+static bool storm_check = true;
+static bool storm_killed = false;
+static u32 irqs_fired = 0;
+
 static void tpm_tis_clkrun_enable(struct tpm_chip *chip, bool value);
 
 static void tpm_tis_enable_interrupt(struct tpm_chip *chip, u8 mask)
@@ -464,25 +469,31 @@ static int tpm_tis_send_data(struct tpm_chip *chip, const u8 *buf, size_t len)
        return rc;
 }
 
-static void disable_interrupts(struct tpm_chip *chip)
+static void __disable_interrupts(struct tpm_chip *chip)
 {
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
        u32 intmask;
        int rc;
 
-       if (priv->irq == 0)
-               return;
-
        rc = tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask);
        if (rc < 0)
                intmask = 0;
 
        intmask &= ~TPM_GLOBAL_INT_ENABLE;
        rc = tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask);
+       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
+}
+
+static void disable_interrupts(struct tpm_chip *chip)
+{
+       struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (priv->irq == 0)
+               return;
+
+       __disable_interrupts(chip);
        devm_free_irq(chip->dev.parent, priv->irq, chip);
        priv->irq = 0;
-       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
 }
 
 /*
@@ -528,6 +539,12 @@ static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len)
        int rc, irq;
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (unlikely(storm_killed)) {
+               devm_free_irq(chip->dev.parent, priv->irq, chip);
+               priv->irq = 0;
+               storm_killed = false;
+       }
+
        if (!(chip->flags & TPM_CHIP_FLAG_IRQ) || priv->irq_tested)
                return tpm_tis_send_main(chip, buf, len);
 
@@ -748,6 +765,21 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id)
        u32 interrupt;
        int i, rc;
 
+       if (storm_check) {
+               irqs_fired++;
+
+               if (!time_start) {
+                       time_start = jiffies_to_msecs(jiffies);
+               } else if ((irqs_fired > 1000) && (jiffies_to_msecs(jiffies) - jiffies < 500)) {
+                       __disable_interrupts(chip);
+                       storm_check = false;
+                       storm_killed = true;
+                       return IRQ_HANDLED;
+               } else if ((jiffies_to_msecs(jiffies) - time_start > 500) && (irqs_fired < 1000)) {
+                       storm_check = false;
+               }
+       }
+
        rc = tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt);
        if (rc < 0)
                return IRQ_NONE;
#5 lore raw

Hi,

Quoted 33 lines · Jerry Snitselaar

Matthew Garrett @ 2020-10-15 15:39 MST:

Quoted 15 lines · Jerry Snitselaar
Quoted 13 lines

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 10 lines

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

I believe Matthew is correct here. I found another system today with completely different vendor for both the system and the tpm chip. In addition another Lenovo model, the L490, has the issue.

Quoted 5 lines

This initial attempt at a solution like Matthew suggested works on the system I found today, but I imagine it is all sorts of wrong. In the 2 systems where I've seen it, there are about 100000 interrupts in around 1.5 seconds, and then the irq code shuts down the interrupt because they aren't being handled.

Is that with your patch? The IRQ should be silenced as soon as devm_free_irq(chip->dev.parent, priv->irq, chip); is called.

Depending on if we can get your storm-detection to work or not, we might also choose to just never try to use the IRQ (at least on x86 systems). AFAIK the TPM is never used for high-throughput stuff so the polling overhead should not be a big deal (and I'm getting the feeling that Windows always polls).

Regards,

Hans

Quoted 89 lines
drivers/char/tpm/tpm_tis_core.c +37 -5
index 49ae09ac604f..478e9d02a3fa 100644
@@ -27,6 +27,11 @@
 #include "tpm.h"
 #include "tpm_tis_core.h"
 
+static unsigned int time_start = 0;
+static bool storm_check = true;
+static bool storm_killed = false;
+static u32 irqs_fired = 0;
+
 static void tpm_tis_clkrun_enable(struct tpm_chip *chip, bool value);
 
 static void tpm_tis_enable_interrupt(struct tpm_chip *chip, u8 mask)
@@ -464,25 +469,31 @@ static int tpm_tis_send_data(struct tpm_chip *chip, const u8 *buf, size_t len)
        return rc;
 }
 
-static void disable_interrupts(struct tpm_chip *chip)
+static void __disable_interrupts(struct tpm_chip *chip)
 {
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
        u32 intmask;
        int rc;
 
-       if (priv->irq == 0)
-               return;
-
        rc = tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask);
        if (rc < 0)
                intmask = 0;
 
        intmask &= ~TPM_GLOBAL_INT_ENABLE;
        rc = tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask);
+       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
+}
+
+static void disable_interrupts(struct tpm_chip *chip)
+{
+       struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (priv->irq == 0)
+               return;
+
+       __disable_interrupts(chip);
        devm_free_irq(chip->dev.parent, priv->irq, chip);
        priv->irq = 0;
-       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
 }
 
 /*
@@ -528,6 +539,12 @@ static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len)
        int rc, irq;
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (unlikely(storm_killed)) {
+               devm_free_irq(chip->dev.parent, priv->irq, chip);
+               priv->irq = 0;
+               storm_killed = false;
+       }
+
        if (!(chip->flags & TPM_CHIP_FLAG_IRQ) || priv->irq_tested)
                return tpm_tis_send_main(chip, buf, len);
 
@@ -748,6 +765,21 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id)
        u32 interrupt;
        int i, rc;
 
+       if (storm_check) {
+               irqs_fired++;
+
+               if (!time_start) {
+                       time_start = jiffies_to_msecs(jiffies);
+               } else if ((irqs_fired > 1000) && (jiffies_to_msecs(jiffies) - jiffies < 500)) {
+                       __disable_interrupts(chip);
+                       storm_check = false;
+                       storm_killed = true;
+                       return IRQ_HANDLED;
+               } else if ((jiffies_to_msecs(jiffies) - time_start > 500) && (irqs_fired < 1000)) {
+                       storm_check = false;
+               }
+       }
+
        rc = tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt);
        if (rc < 0)
                return IRQ_NONE;
#6 lore raw

Hans de Goede @ 2020-11-19 07:42 MST:

Quoted 36 lines

Hi,

Quoted 33 lines · Jerry Snitselaar

Matthew Garrett @ 2020-10-15 15:39 MST:

Quoted 15 lines · Jerry Snitselaar
Quoted 13 lines

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 10 lines

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

I believe Matthew is correct here. I found another system today with completely different vendor for both the system and the tpm chip. In addition another Lenovo model, the L490, has the issue.

Quoted 9 lines
Quoted 5 lines

This initial attempt at a solution like Matthew suggested works on the system I found today, but I imagine it is all sorts of wrong. In the 2 systems where I've seen it, there are about 100000 interrupts in around 1.5 seconds, and then the irq code shuts down the interrupt because they aren't being handled.

Is that with your patch? The IRQ should be silenced as soon as devm_free_irq(chip->dev.parent, priv->irq, chip); is called.

No that is just with James' patchset that enables interrupts for tpm_tis. It looks like the irq is firing, but the tpm's int_status register is clear, so the handler immediately returns IRQ_NONE. After that happens 100000 times the core irq code shuts down the irq, but it isn't released so I could still see the stats in /proc/interrupts. With my attempt below on top of that patchset it releases the irq. I had to stick the check prior to it checking the int_status register because it is cleared and the handler returns, and I couldn't do the devm_free_irq directly in tis_int_handler, so I tried sticking it in tpm_tis_send where the other odd irq testing code was already located. I'm not sure if there is another place that would work better for calling the devm_free_irq.

Quoted 6 lines

Depending on if we can get your storm-detection to work or not, we might also choose to just never try to use the IRQ (at least on x86 systems). AFAIK the TPM is never used for high-throughput stuff so the polling overhead should not be a big deal (and I'm getting the feeling that Windows always polls).

I was wondering about Windows as well. In addition to the Lenovo systems which the T490s had Nuvoton tpm, the system I found yesterday was a development system we have from a partner with an Infineon tpm. Dan Williams has seen it internally at Intel as well on some system.

Quoted 95 lines

Regards,

Hans

Quoted 89 lines
drivers/char/tpm/tpm_tis_core.c +37 -5
index 49ae09ac604f..478e9d02a3fa 100644
@@ -27,6 +27,11 @@
 #include "tpm.h"
 #include "tpm_tis_core.h"
 
+static unsigned int time_start = 0;
+static bool storm_check = true;
+static bool storm_killed = false;
+static u32 irqs_fired = 0;
+
 static void tpm_tis_clkrun_enable(struct tpm_chip *chip, bool value);
 
 static void tpm_tis_enable_interrupt(struct tpm_chip *chip, u8 mask)
@@ -464,25 +469,31 @@ static int tpm_tis_send_data(struct tpm_chip *chip, const u8 *buf, size_t len)
        return rc;
 }
 
-static void disable_interrupts(struct tpm_chip *chip)
+static void __disable_interrupts(struct tpm_chip *chip)
 {
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
        u32 intmask;
        int rc;
 
-       if (priv->irq == 0)
-               return;
-
        rc = tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask);
        if (rc < 0)
                intmask = 0;
 
        intmask &= ~TPM_GLOBAL_INT_ENABLE;
        rc = tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask);
+       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
+}
+
+static void disable_interrupts(struct tpm_chip *chip)
+{
+       struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (priv->irq == 0)
+               return;
+
+       __disable_interrupts(chip);
        devm_free_irq(chip->dev.parent, priv->irq, chip);
        priv->irq = 0;
-       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
 }
 
 /*
@@ -528,6 +539,12 @@ static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len)
        int rc, irq;
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (unlikely(storm_killed)) {
+               devm_free_irq(chip->dev.parent, priv->irq, chip);
+               priv->irq = 0;
+               storm_killed = false;
+       }
+
        if (!(chip->flags & TPM_CHIP_FLAG_IRQ) || priv->irq_tested)
                return tpm_tis_send_main(chip, buf, len);
 
@@ -748,6 +765,21 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id)
        u32 interrupt;
        int i, rc;
 
+       if (storm_check) {
+               irqs_fired++;
+
+               if (!time_start) {
+                       time_start = jiffies_to_msecs(jiffies);
+               } else if ((irqs_fired > 1000) && (jiffies_to_msecs(jiffies) - jiffies < 500)) {
+                       __disable_interrupts(chip);
+                       storm_check = false;
+                       storm_killed = true;
+                       return IRQ_HANDLED;
+               } else if ((jiffies_to_msecs(jiffies) - time_start > 500) && (irqs_fired < 1000)) {
+                       storm_check = false;
+               }
+       }
+
        rc = tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt);
        if (rc < 0)
                return IRQ_NONE;
#7 lore raw

Hi,

Quoted 49 lines · Jerry Snitselaar

Hans de Goede @ 2020-11-19 07:42 MST:

Quoted 36 lines

Hi,

Quoted 33 lines · Jerry Snitselaar

Matthew Garrett @ 2020-10-15 15:39 MST:

Quoted 15 lines · Jerry Snitselaar
Quoted 13 lines

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 10 lines

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

I believe Matthew is correct here. I found another system today with completely different vendor for both the system and the tpm chip. In addition another Lenovo model, the L490, has the issue.

Quoted 9 lines
Quoted 5 lines

This initial attempt at a solution like Matthew suggested works on the system I found today, but I imagine it is all sorts of wrong. In the 2 systems where I've seen it, there are about 100000 interrupts in around 1.5 seconds, and then the irq code shuts down the interrupt because they aren't being handled.

Is that with your patch? The IRQ should be silenced as soon as devm_free_irq(chip->dev.parent, priv->irq, chip); is called.

Quoted 5 lines

No that is just with James' patchset that enables interrupts for tpm_tis. It looks like the irq is firing, but the tpm's int_status register is clear, so the handler immediately returns IRQ_NONE. After that happens 100000 times the core irq code shuts down the irq, but it isn't released so I could still see the stats in /proc/interrupts.

I see, yes I have seen this behavior on the X1C8 with a pre-production BIOS.

Quoted 4 lines

With
my attempt below on top of that patchset it releases the irq. I had to stick the check prior to it checking the int_status register because it is cleared and the handler returns,

Ack.

Quoted 5 lines

and I couldn't do the devm_free_irq
directly in tis_int_handler, so I tried sticking it in tpm_tis_send where the other odd irq testing code was already located. I'm not sure if there is another place that would work better for calling the devm_free_irq.

Adding it together with the other IRQ testing related code seems fine to me.

Quoted 11 lines
Quoted 6 lines

Depending on if we can get your storm-detection to work or not, we might also choose to just never try to use the IRQ (at least on x86 systems). AFAIK the TPM is never used for high-throughput stuff so the polling overhead should not be a big deal (and I'm getting the feeling that Windows always polls).

I was wondering about Windows as well. In addition to the Lenovo systems which the T490s had Nuvoton tpm, the system I found yesterday was a development system we have from a partner with an Infineon tpm. Dan Williams has seen it internally at Intel as well on some system.

Sounds to me like Windows never uses the IRQ, so we get the fun of finding all the untested firmware bugs.

So if we cannot get this detection code to work reliable, maybe we should just not use the IRQ at all, at least on X86 + ACPI systems?

Anyways lets try this storm-detection thing first, but I have the feeling we should not spend too much time on this. Just outright disabling IRQ support might be better.

REgards,

Hans

maintainer
#8 lore raw
In reply to Hans de Goede
Quoted 51 lines · Hans de Goede

Hi,

Quoted 33 lines · Jerry Snitselaar

Matthew Garrett @ 2020-10-15 15:39 MST:

Quoted 15 lines · Jerry Snitselaar
Quoted 13 lines

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 10 lines

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

I believe Matthew is correct here. I found another system today with completely different vendor for both the system and the tpm chip. In addition another Lenovo model, the L490, has the issue.

Quoted 5 lines

This initial attempt at a solution like Matthew suggested works on the system I found today, but I imagine it is all sorts of wrong. In the 2 systems where I've seen it, there are about 100000 interrupts in around 1.5 seconds, and then the irq code shuts down the interrupt because they aren't being handled.

Is that with your patch? The IRQ should be silenced as soon as devm_free_irq(chip->dev.parent, priv->irq, chip); is called.

Depending on if we can get your storm-detection to work or not, we might also choose to just never try to use the IRQ (at least on x86 systems). AFAIK the TPM is never used for high-throughput stuff so the polling overhead should not be a big deal (and I'm getting the feeling that Windows always polls).

Quoted 3 lines

Regards,

Hans

Yeah, this is what I've been wondering for a while. Why could not we just strip off IRQ code? Why does it matter?

/Jarkko

maintainer
#9 lore raw
Quoted 52 lines · Jarkko Sakkinen
Quoted 51 lines · Hans de Goede

Hi,

Quoted 33 lines · Jerry Snitselaar

Matthew Garrett @ 2020-10-15 15:39 MST:

Quoted 15 lines · Jerry Snitselaar
Quoted 13 lines

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 10 lines

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

I believe Matthew is correct here. I found another system today with completely different vendor for both the system and the tpm chip. In addition another Lenovo model, the L490, has the issue.

Quoted 5 lines

This initial attempt at a solution like Matthew suggested works on the system I found today, but I imagine it is all sorts of wrong. In the 2 systems where I've seen it, there are about 100000 interrupts in around 1.5 seconds, and then the irq code shuts down the interrupt because they aren't being handled.

Is that with your patch? The IRQ should be silenced as soon as devm_free_irq(chip->dev.parent, priv->irq, chip); is called.

Depending on if we can get your storm-detection to work or not, we might also choose to just never try to use the IRQ (at least on x86 systems). AFAIK the TPM is never used for high-throughput stuff so the polling overhead should not be a big deal (and I'm getting the feeling that Windows always polls).

Quoted 6 lines
Quoted 3 lines

Regards,

Hans

Yeah, this is what I've been wondering for a while. Why could not we just strip off IRQ code? Why does it matter?

And we DO NOT use interrupts in tpm_crb and nobody has ever complained.

/Jarkko

maintainer
#10 lore raw
In reply to Jerry Snitselaar
Quoted 45 lines · Jerry Snitselaar

Matthew Garrett @ 2020-10-15 15:39 MST:

Quoted 15 lines · Jerry Snitselaar
Quoted 13 lines

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 10 lines

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

I believe Matthew is correct here. I found another system today with completely different vendor for both the system and the tpm chip. In addition another Lenovo model, the L490, has the issue.

This initial attempt at a solution like Matthew suggested works on the system I found today, but I imagine it is all sorts of wrong. In the 2 systems where I've seen it, there are about 100000 interrupts in around 1.5 seconds, and then the irq code shuts down the interrupt because they aren't being handled.

drivers/char/tpm/tpm_tis_core.c +0 -0
index 49ae09ac604f..478e9d02a3fa 100644

drivers/char/tpm/tpm_tis_core.c line 33

Quoted 7 lines
drivers/char/tpm/tpm_tis_core.c +4 -0
@@ -27,6 +27,11 @@
 #include "tpm.h"
 #include "tpm_tis_core.h"
 
+static unsigned int time_start = 0;
+static bool storm_check = true;
+static bool storm_killed = false;
+static u32 irqs_fired = 0;

Maybe kstat_irqs() would be a better idea than ad hoc stats.

Quoted 42 lines
drivers/char/tpm/tpm_tis_core.c +12 -5
+
 static void tpm_tis_clkrun_enable(struct tpm_chip *chip, bool value);
 
 static void tpm_tis_enable_interrupt(struct tpm_chip *chip, u8 mask)
@@ -464,25 +469,31 @@ static int tpm_tis_send_data(struct tpm_chip *chip, const u8 *buf, size_t len)
        return rc;
 }
 
-static void disable_interrupts(struct tpm_chip *chip)
+static void __disable_interrupts(struct tpm_chip *chip)
 {
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
        u32 intmask;
        int rc;
 
-       if (priv->irq == 0)
-               return;
-
        rc = tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask);
        if (rc < 0)
                intmask = 0;
 
        intmask &= ~TPM_GLOBAL_INT_ENABLE;
        rc = tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask);
+       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
+}
+
+static void disable_interrupts(struct tpm_chip *chip)
+{
+       struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (priv->irq == 0)
+               return;
+
+       __disable_interrupts(chip);
        devm_free_irq(chip->dev.parent, priv->irq, chip);
        priv->irq = 0;
-       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
 }
 
 /*

drivers/char/tpm/tpm_tis_core.c static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len) · line 546

Quoted 8 lines
drivers/char/tpm/tpm_tis_core.c +5 -0
@@ -528,6 +539,12 @@ static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len)
        int rc, irq;
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (unlikely(storm_killed)) {
+               devm_free_irq(chip->dev.parent, priv->irq, chip);
+               priv->irq = 0;
+               storm_killed = false;
+       }

OK this kind of bad solution because if tpm_tis_send() is not called, then IRQ is never freed. AFAIK, devres_* do not sleep but use spin lock, i.e. you could render out both storm_check and storm_killed.

Quoted 18 lines
drivers/char/tpm/tpm_tis_core.c +9 -0
+
        if (!(chip->flags & TPM_CHIP_FLAG_IRQ) || priv->irq_tested)
                return tpm_tis_send_main(chip, buf, len);
 
@@ -748,6 +765,21 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id)
        u32 interrupt;
        int i, rc;
 
+       if (storm_check) {
+               irqs_fired++;
+
+               if (!time_start) {
+                       time_start = jiffies_to_msecs(jiffies);
+               } else if ((irqs_fired > 1000) && (jiffies_to_msecs(jiffies) - jiffies < 500)) {
+                       __disable_interrupts(chip);
+                       storm_check = false;

drivers/char/tpm/tpm_tis_core.c static irqreturn_t tis_int_handler(int dummy, void *dev_id) · line 785

Quoted 10 lines
drivers/char/tpm/tpm_tis_core.c +7 -0
+                       storm_killed = true;
+                       return IRQ_HANDLED;
+               } else if ((jiffies_to_msecs(jiffies) - time_start > 500) && (irqs_fired < 1000)) {
+                       storm_check = false;
+               }
+       }
+
        rc = tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt);
        if (rc < 0)
                return IRQ_NONE;

/Jarkko

#11 lore raw

Jarkko Sakkinen @ 2020-11-23 20:26 MST:

Quoted 49 lines · Jerry Snitselaar
Quoted 48 lines

Matthew Garrett @ 2020-10-15 15:39 MST:

Quoted 15 lines · Jerry Snitselaar
Quoted 13 lines

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 10 lines

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

I believe Matthew is correct here. I found another system today with completely different vendor for both the system and the tpm chip. In addition another Lenovo model, the L490, has the issue.

This initial attempt at a solution like Matthew suggested works on the system I found today, but I imagine it is all sorts of wrong. In the 2 systems where I've seen it, there are about 100000 interrupts in around 1.5 seconds, and then the irq code shuts down the interrupt because they aren't being handled.

drivers/char/tpm/tpm_tis_core.c +0 -0
index 49ae09ac604f..478e9d02a3fa 100644
@@ -27,6 +27,11 @@
 #include "tpm.h"
 #include "tpm_tis_core.h"
 
Quoted 7 lines

drivers/char/tpm/tpm_tis_core.c line 33

Quoted 4 lines
drivers/char/tpm/tpm_tis_core.c +4 -0
+static unsigned int time_start = 0;
+static bool storm_check = true;
+static bool storm_killed = false;
+static u32 irqs_fired = 0;

Maybe kstat_irqs() would be a better idea than ad hoc stats.

Thanks, yes that would be better.

Quoted 45 lines
Quoted 45 lines
drivers/char/tpm/tpm_tis_core.c +12 -5
+
 static void tpm_tis_clkrun_enable(struct tpm_chip *chip, bool value);
 
 static void tpm_tis_enable_interrupt(struct tpm_chip *chip, u8 mask)
@@ -464,25 +469,31 @@ static int tpm_tis_send_data(struct tpm_chip *chip, const u8 *buf, size_t len)
        return rc;
 }
 
-static void disable_interrupts(struct tpm_chip *chip)
+static void __disable_interrupts(struct tpm_chip *chip)
 {
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
        u32 intmask;
        int rc;
 
-       if (priv->irq == 0)
-               return;
-
        rc = tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask);
        if (rc < 0)
                intmask = 0;
 
        intmask &= ~TPM_GLOBAL_INT_ENABLE;
        rc = tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask);
+       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
+}
+
+static void disable_interrupts(struct tpm_chip *chip)
+{
+       struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (priv->irq == 0)
+               return;
+
+       __disable_interrupts(chip);
        devm_free_irq(chip->dev.parent, priv->irq, chip);
        priv->irq = 0;
-       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
 }
 
 /*
@@ -528,6 +539,12 @@ static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len)
        int rc, irq;
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
Quoted 10 lines

drivers/char/tpm/tpm_tis_core.c static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len) · line 546

Quoted 5 lines
drivers/char/tpm/tpm_tis_core.c +5 -0
+       if (unlikely(storm_killed)) {
+               devm_free_irq(chip->dev.parent, priv->irq, chip);
+               priv->irq = 0;
+               storm_killed = false;
+       }

OK this kind of bad solution because if tpm_tis_send() is not called, then IRQ is never freed. AFAIK, devres_* do not sleep but use spin lock, i.e. you could render out both storm_check and storm_killed.

Is there a way to flag it for freeing later while in an interrupt context? I'm not sure where to clean it up since devm_free_irq can't be called in tis_int_handler.

Before diving further into that though, does anyone else have an opinion on ripping out the irq code, and just using polling? We've been only polling since 2015 anyways.

Quoted 30 lines
Quoted 18 lines
drivers/char/tpm/tpm_tis_core.c +9 -0
+
        if (!(chip->flags & TPM_CHIP_FLAG_IRQ) || priv->irq_tested)
                return tpm_tis_send_main(chip, buf, len);
 
@@ -748,6 +765,21 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id)
        u32 interrupt;
        int i, rc;
 
+       if (storm_check) {
+               irqs_fired++;
+
+               if (!time_start) {
+                       time_start = jiffies_to_msecs(jiffies);
+               } else if ((irqs_fired > 1000) && (jiffies_to_msecs(jiffies) - jiffies < 500)) {
+                       __disable_interrupts(chip);
+                       storm_check = false;

drivers/char/tpm/tpm_tis_core.c static irqreturn_t tis_int_handler(int dummy, void *dev_id) · line 785

Quoted 10 lines
drivers/char/tpm/tpm_tis_core.c +7 -0
+                       storm_killed = true;
+                       return IRQ_HANDLED;
+               } else if ((jiffies_to_msecs(jiffies) - time_start > 500) && (irqs_fired < 1000)) {
+                       storm_check = false;
+               }
+       }
+
        rc = tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt);
        if (rc < 0)
                return IRQ_NONE;

/Jarkko

maintainer
#12 lore raw
Quoted 3 lines · Jerry Snitselaar

Before diving further into that though, does anyone else have an opinion on ripping out the irq code, and just using polling? We've been only polling since 2015 anyways.

Well only a biased one, obviously: polling causes large amounts of busy waiting, which is a waste of CPU resources and does increase the time it takes us to do TPM operations ... not a concern if you're doing long computation ones, like signatures, but it is a problem for short operations like bulk updates of PCRs. The other potential issue, as we saw with atmel is that if you prod the chip too often (which you have to do with polling) you risk upsetting it. We've spent ages trying to tune the polling parameters to balance reduction of busy wait with chip upset and still, apparently, not quite got it right. If the TPM has a functioning IRQ then it gets us out of the whole polling mess entirely. The big question is how many chips that report an IRQ actually have a malfunctioning one?

James

maintainer
#13 lore raw
Quoted 18 lines · James Bottomley
Quoted 3 lines · Jerry Snitselaar

Before diving further into that though, does anyone else have an opinion on ripping out the irq code, and just using polling? We've been only polling since 2015 anyways.

Well only a biased one, obviously: polling causes large amounts of busy waiting, which is a waste of CPU resources and does increase the time it takes us to do TPM operations ... not a concern if you're doing long computation ones, like signatures, but it is a problem for short operations like bulk updates of PCRs. The other potential issue, as we saw with atmel is that if you prod the chip too often (which you have to do with polling) you risk upsetting it. We've spent ages trying to tune the polling parameters to balance reduction of busy wait with chip upset and still, apparently, not quite got it right. If the TPM has a functioning IRQ then it gets us out of the whole polling mess entirely. The big question is how many chips that report an IRQ actually have a malfunctioning one?

Quoted 1 line

James

Do we have a way to know is Windows TPM code using IRQ's?

/Jarkko

#14 lore raw
In reply to Jerry Snitselaar

Hi,

Quoted 118 lines · Jerry Snitselaar

Jarkko Sakkinen @ 2020-11-23 20:26 MST:

Quoted 49 lines · Jerry Snitselaar
Quoted 48 lines

Matthew Garrett @ 2020-10-15 15:39 MST:

Quoted 15 lines · Jerry Snitselaar
Quoted 13 lines

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 10 lines

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

I believe Matthew is correct here. I found another system today with completely different vendor for both the system and the tpm chip. In addition another Lenovo model, the L490, has the issue.

This initial attempt at a solution like Matthew suggested works on the system I found today, but I imagine it is all sorts of wrong. In the 2 systems where I've seen it, there are about 100000 interrupts in around 1.5 seconds, and then the irq code shuts down the interrupt because they aren't being handled.

drivers/char/tpm/tpm_tis_core.c +0 -0
index 49ae09ac604f..478e9d02a3fa 100644
@@ -27,6 +27,11 @@
 #include "tpm.h"
 #include "tpm_tis_core.h"
 
Quoted 7 lines

drivers/char/tpm/tpm_tis_core.c line 33

Quoted 4 lines
drivers/char/tpm/tpm_tis_core.c +4 -0
+static unsigned int time_start = 0;
+static bool storm_check = true;
+static bool storm_killed = false;
+static u32 irqs_fired = 0;

Maybe kstat_irqs() would be a better idea than ad hoc stats.

Thanks, yes that would be better.

Quoted 45 lines
Quoted 45 lines
drivers/char/tpm/tpm_tis_core.c +12 -5
+
 static void tpm_tis_clkrun_enable(struct tpm_chip *chip, bool value);
 
 static void tpm_tis_enable_interrupt(struct tpm_chip *chip, u8 mask)
@@ -464,25 +469,31 @@ static int tpm_tis_send_data(struct tpm_chip *chip, const u8 *buf, size_t len)
        return rc;
 }
 
-static void disable_interrupts(struct tpm_chip *chip)
+static void __disable_interrupts(struct tpm_chip *chip)
 {
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
        u32 intmask;
        int rc;
 
-       if (priv->irq == 0)
-               return;
-
        rc = tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask);
        if (rc < 0)
                intmask = 0;
 
        intmask &= ~TPM_GLOBAL_INT_ENABLE;
        rc = tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask);
+       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
+}
+
+static void disable_interrupts(struct tpm_chip *chip)
+{
+       struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (priv->irq == 0)
+               return;
+
+       __disable_interrupts(chip);
        devm_free_irq(chip->dev.parent, priv->irq, chip);
        priv->irq = 0;
-       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
 }
 
 /*
@@ -528,6 +539,12 @@ static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len)
        int rc, irq;
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
Quoted 10 lines

drivers/char/tpm/tpm_tis_core.c static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len) · line 546

Quoted 5 lines
drivers/char/tpm/tpm_tis_core.c +5 -0
+       if (unlikely(storm_killed)) {
+               devm_free_irq(chip->dev.parent, priv->irq, chip);
+               priv->irq = 0;
+               storm_killed = false;
+       }

OK this kind of bad solution because if tpm_tis_send() is not called, then IRQ is never freed. AFAIK, devres_* do not sleep but use spin lock, i.e. you could render out both storm_check and storm_killed.

Quoted 3 lines

Is there a way to flag it for freeing later while in an interrupt context? I'm not sure where to clean it up since devm_free_irq can't be called in tis_int_handler.

You could add a workqueue work-struct just for this and queue that up to do the free when you detect the storm. That will then run pretty much immediately, avoiding the storm going on for (much) longer.

Quoted 3 lines

Before diving further into that though, does anyone else have an opinion on ripping out the irq code, and just using polling? We've been only polling since 2015 anyways.

Given James Bottomley's reply I guess it would be worthwhile to get the storm detection to work.

Regards,

Hans

Quoted 32 lines
Quoted 31 lines
Quoted 19 lines
drivers/char/tpm/tpm_tis_core.c +9 -0
+
        if (!(chip->flags & TPM_CHIP_FLAG_IRQ) || priv->irq_tested)
                return tpm_tis_send_main(chip, buf, len);
 
@@ -748,6 +765,21 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id)
        u32 interrupt;
        int i, rc;
 
+       if (storm_check) {
+               irqs_fired++;
+
+               if (!time_start) {
+                       time_start = jiffies_to_msecs(jiffies);
+               } else if ((irqs_fired > 1000) && (jiffies_to_msecs(jiffies) - jiffies < 500)) {
+                       __disable_interrupts(chip);
+                       storm_check = false;

drivers/char/tpm/tpm_tis_core.c static irqreturn_t tis_int_handler(int dummy, void *dev_id) · line 785

Quoted 10 lines
drivers/char/tpm/tpm_tis_core.c +7 -0
+                       storm_killed = true;
+                       return IRQ_HANDLED;
+               } else if ((jiffies_to_msecs(jiffies) - time_start > 500) && (irqs_fired < 1000)) {
+                       storm_check = false;
+               }
+       }
+
        rc = tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt);
        if (rc < 0)
                return IRQ_NONE;

/Jarkko

maintainer
#15 lore raw
Quoted 121 lines · Hans de Goede

Hi,

Quoted 118 lines · Jerry Snitselaar

Jarkko Sakkinen @ 2020-11-23 20:26 MST:

Quoted 49 lines · Jerry Snitselaar
Quoted 48 lines

Matthew Garrett @ 2020-10-15 15:39 MST:

Quoted 15 lines · Jerry Snitselaar
Quoted 13 lines

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 10 lines

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

I believe Matthew is correct here. I found another system today with completely different vendor for both the system and the tpm chip. In addition another Lenovo model, the L490, has the issue.

This initial attempt at a solution like Matthew suggested works on the system I found today, but I imagine it is all sorts of wrong. In the 2 systems where I've seen it, there are about 100000 interrupts in around 1.5 seconds, and then the irq code shuts down the interrupt because they aren't being handled.

drivers/char/tpm/tpm_tis_core.c +0 -0
index 49ae09ac604f..478e9d02a3fa 100644
@@ -27,6 +27,11 @@
 #include "tpm.h"
 #include "tpm_tis_core.h"
 
Quoted 7 lines

drivers/char/tpm/tpm_tis_core.c line 33

Quoted 4 lines
drivers/char/tpm/tpm_tis_core.c +4 -0
+static unsigned int time_start = 0;
+static bool storm_check = true;
+static bool storm_killed = false;
+static u32 irqs_fired = 0;

Maybe kstat_irqs() would be a better idea than ad hoc stats.

Thanks, yes that would be better.

Quoted 55 lines
Quoted 42 lines
drivers/char/tpm/tpm_tis_core.c +12 -5
+
 static void tpm_tis_clkrun_enable(struct tpm_chip *chip, bool value);
 
 static void tpm_tis_enable_interrupt(struct tpm_chip *chip, u8 mask)
@@ -464,25 +469,31 @@ static int tpm_tis_send_data(struct tpm_chip *chip, const u8 *buf, size_t len)
        return rc;
 }
 
-static void disable_interrupts(struct tpm_chip *chip)
+static void __disable_interrupts(struct tpm_chip *chip)
 {
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
        u32 intmask;
        int rc;
 
-       if (priv->irq == 0)
-               return;
-
        rc = tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask);
        if (rc < 0)
                intmask = 0;
 
        intmask &= ~TPM_GLOBAL_INT_ENABLE;
        rc = tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask);
+       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
+}
+
+static void disable_interrupts(struct tpm_chip *chip)
+{
+       struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (priv->irq == 0)
+               return;
+
+       __disable_interrupts(chip);
        devm_free_irq(chip->dev.parent, priv->irq, chip);
        priv->irq = 0;
-       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
 }
 
 /*

drivers/char/tpm/tpm_tis_core.c static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len) · line 546

Quoted 8 lines
drivers/char/tpm/tpm_tis_core.c +5 -0
@@ -528,6 +539,12 @@ static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len)
        int rc, irq;
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (unlikely(storm_killed)) {
+               devm_free_irq(chip->dev.parent, priv->irq, chip);
+               priv->irq = 0;
+               storm_killed = false;
+       }

OK this kind of bad solution because if tpm_tis_send() is not called, then IRQ is never freed. AFAIK, devres_* do not sleep but use spin lock, i.e. you could render out both storm_check and storm_killed.

Quoted 7 lines
Quoted 3 lines

Is there a way to flag it for freeing later while in an interrupt context? I'm not sure where to clean it up since devm_free_irq can't be called in tis_int_handler.

You could add a workqueue work-struct just for this and queue that up to do the free when you detect the storm. That will then run pretty much immediately, avoiding the storm going on for (much) longer.

That's sounds feasible.

Quoted 6 lines
Quoted 3 lines

Before diving further into that though, does anyone else have an opinion on ripping out the irq code, and just using polling? We've been only polling since 2015 anyways.

Given James Bottomley's reply I guess it would be worthwhile to get the storm detection to work.

OK, agreed. I take my words back from a response few minutes ago :-)

Quoted 3 lines

Regards,

Hans

/Jarkko

#16 lore raw

Hi All,

Quoted 132 lines · Jarkko Sakkinen
Quoted 121 lines · Hans de Goede

Hi,

Quoted 118 lines · Jerry Snitselaar

Jarkko Sakkinen @ 2020-11-23 20:26 MST:

Quoted 49 lines · Jerry Snitselaar
Quoted 48 lines

Matthew Garrett @ 2020-10-15 15:39 MST:

Quoted 15 lines · Jerry Snitselaar
Quoted 13 lines

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 10 lines

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

I believe Matthew is correct here. I found another system today with completely different vendor for both the system and the tpm chip. In addition another Lenovo model, the L490, has the issue.

This initial attempt at a solution like Matthew suggested works on the system I found today, but I imagine it is all sorts of wrong. In the 2 systems where I've seen it, there are about 100000 interrupts in around 1.5 seconds, and then the irq code shuts down the interrupt because they aren't being handled.

drivers/char/tpm/tpm_tis_core.c +0 -0
index 49ae09ac604f..478e9d02a3fa 100644
@@ -27,6 +27,11 @@
 #include "tpm.h"
 #include "tpm_tis_core.h"
 
Quoted 7 lines

drivers/char/tpm/tpm_tis_core.c line 33

Quoted 4 lines
drivers/char/tpm/tpm_tis_core.c +4 -0
+static unsigned int time_start = 0;
+static bool storm_check = true;
+static bool storm_killed = false;
+static u32 irqs_fired = 0;

Maybe kstat_irqs() would be a better idea than ad hoc stats.

Thanks, yes that would be better.

Quoted 55 lines
Quoted 42 lines
drivers/char/tpm/tpm_tis_core.c +12 -5
+
 static void tpm_tis_clkrun_enable(struct tpm_chip *chip, bool value);
 
 static void tpm_tis_enable_interrupt(struct tpm_chip *chip, u8 mask)
@@ -464,25 +469,31 @@ static int tpm_tis_send_data(struct tpm_chip *chip, const u8 *buf, size_t len)
        return rc;
 }
 
-static void disable_interrupts(struct tpm_chip *chip)
+static void __disable_interrupts(struct tpm_chip *chip)
 {
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
        u32 intmask;
        int rc;
 
-       if (priv->irq == 0)
-               return;
-
        rc = tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask);
        if (rc < 0)
                intmask = 0;
 
        intmask &= ~TPM_GLOBAL_INT_ENABLE;
        rc = tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask);
+       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
+}
+
+static void disable_interrupts(struct tpm_chip *chip)
+{
+       struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (priv->irq == 0)
+               return;
+
+       __disable_interrupts(chip);
        devm_free_irq(chip->dev.parent, priv->irq, chip);
        priv->irq = 0;
-       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
 }
 
 /*

drivers/char/tpm/tpm_tis_core.c static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len) · line 546

Quoted 8 lines
drivers/char/tpm/tpm_tis_core.c +5 -0
@@ -528,6 +539,12 @@ static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len)
        int rc, irq;
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (unlikely(storm_killed)) {
+               devm_free_irq(chip->dev.parent, priv->irq, chip);
+               priv->irq = 0;
+               storm_killed = false;
+       }

OK this kind of bad solution because if tpm_tis_send() is not called, then IRQ is never freed. AFAIK, devres_* do not sleep but use spin lock, i.e. you could render out both storm_check and storm_killed.

Quoted 7 lines
Quoted 3 lines

Is there a way to flag it for freeing later while in an interrupt context? I'm not sure where to clean it up since devm_free_irq can't be called in tis_int_handler.

You could add a workqueue work-struct just for this and queue that up to do the free when you detect the storm. That will then run pretty much immediately, avoiding the storm going on for (much) longer.

That's sounds feasible.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Before diving further into that though, does anyone else have an opinion on ripping out the irq code, and just using polling? We've been only polling since 2015 anyways.

Given James Bottomley's reply I guess it would be worthwhile to get the storm detection to work.

OK, agreed. I take my words back from a response few minutes ago :-)

:)

To be clear, I think we should give the storm detection a go. Especially given the problems which James has seen with polling on some TPMs.

But if that turns out to not be feasible I agree we should just either disable IRQs by default on standard x86 platforms, or just remove the IRQ support all together.

Regards,

Hans

maintainer
#17 lore raw
Quoted 144 lines · Hans de Goede

Hi All,

Quoted 132 lines · Jarkko Sakkinen
Quoted 121 lines · Hans de Goede

Hi,

Quoted 118 lines · Jerry Snitselaar

Jarkko Sakkinen @ 2020-11-23 20:26 MST:

Quoted 49 lines · Jerry Snitselaar
Quoted 48 lines

Matthew Garrett @ 2020-10-15 15:39 MST:

Quoted 15 lines · Jerry Snitselaar
Quoted 13 lines

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 10 lines

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

I believe Matthew is correct here. I found another system today with completely different vendor for both the system and the tpm chip. In addition another Lenovo model, the L490, has the issue.

This initial attempt at a solution like Matthew suggested works on the system I found today, but I imagine it is all sorts of wrong. In the 2 systems where I've seen it, there are about 100000 interrupts in around 1.5 seconds, and then the irq code shuts down the interrupt because they aren't being handled.

drivers/char/tpm/tpm_tis_core.c +0 -0
index 49ae09ac604f..478e9d02a3fa 100644
@@ -27,6 +27,11 @@
 #include "tpm.h"
 #include "tpm_tis_core.h"
 
Quoted 7 lines

drivers/char/tpm/tpm_tis_core.c line 33

Quoted 4 lines
drivers/char/tpm/tpm_tis_core.c +4 -0
+static unsigned int time_start = 0;
+static bool storm_check = true;
+static bool storm_killed = false;
+static u32 irqs_fired = 0;

Maybe kstat_irqs() would be a better idea than ad hoc stats.

Thanks, yes that would be better.

Quoted 55 lines
Quoted 42 lines
drivers/char/tpm/tpm_tis_core.c +12 -5
+
 static void tpm_tis_clkrun_enable(struct tpm_chip *chip, bool value);
 
 static void tpm_tis_enable_interrupt(struct tpm_chip *chip, u8 mask)
@@ -464,25 +469,31 @@ static int tpm_tis_send_data(struct tpm_chip *chip, const u8 *buf, size_t len)
        return rc;
 }
 
-static void disable_interrupts(struct tpm_chip *chip)
+static void __disable_interrupts(struct tpm_chip *chip)
 {
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
        u32 intmask;
        int rc;
 
-       if (priv->irq == 0)
-               return;
-
        rc = tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask);
        if (rc < 0)
                intmask = 0;
 
        intmask &= ~TPM_GLOBAL_INT_ENABLE;
        rc = tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask);
+       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
+}
+
+static void disable_interrupts(struct tpm_chip *chip)
+{
+       struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (priv->irq == 0)
+               return;
+
+       __disable_interrupts(chip);
        devm_free_irq(chip->dev.parent, priv->irq, chip);
        priv->irq = 0;
-       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
 }
 
 /*

drivers/char/tpm/tpm_tis_core.c static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len) · line 546

Quoted 8 lines
drivers/char/tpm/tpm_tis_core.c +5 -0
@@ -528,6 +539,12 @@ static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len)
        int rc, irq;
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (unlikely(storm_killed)) {
+               devm_free_irq(chip->dev.parent, priv->irq, chip);
+               priv->irq = 0;
+               storm_killed = false;
+       }

OK this kind of bad solution because if tpm_tis_send() is not called, then IRQ is never freed. AFAIK, devres_* do not sleep but use spin lock, i.e. you could render out both storm_check and storm_killed.

Quoted 7 lines
Quoted 3 lines

Is there a way to flag it for freeing later while in an interrupt context? I'm not sure where to clean it up since devm_free_irq can't be called in tis_int_handler.

You could add a workqueue work-struct just for this and queue that up to do the free when you detect the storm. That will then run pretty much immediately, avoiding the storm going on for (much) longer.

That's sounds feasible.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Before diving further into that though, does anyone else have an opinion on ripping out the irq code, and just using polling? We've been only polling since 2015 anyways.

Given James Bottomley's reply I guess it would be worthwhile to get the storm detection to work.

OK, agreed. I take my words back from a response few minutes ago :-)

Quoted 8 lines

:)

To be clear, I think we should give the storm detection a go. Especially given the problems which James has seen with polling on some TPMs.

But if that turns out to not be feasible I agree we should just either disable IRQs by default on standard x86 platforms, or just remove the IRQ support all together.

Just for completeness: one option is also to whitelist IRQ's.

Quoted 3 lines

Regards,

Hans

/Jarkko

maintainer
#18 lore raw
In reply to Jerry Snitselaar
Quoted 118 lines · Jerry Snitselaar

Jarkko Sakkinen @ 2020-11-23 20:26 MST:

Quoted 49 lines · Jerry Snitselaar
Quoted 48 lines

Matthew Garrett @ 2020-10-15 15:39 MST:

Quoted 15 lines · Jerry Snitselaar
Quoted 13 lines

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 10 lines

I think an implication of this is that systems haven't been well-tested with interrupts enabled. In general when we've found a firmware issue in one place it ends up happening elsewhere as well, so it wouldn't surprise me if there are other machines that will also be unhappy with interrupts enabled. Would it be possible to automatically detect this case (eg, if we get more than a certain number of interrupts in a certain timeframe immediately after enabling the interrupt) and automatically fall back to polling in that case? It would also mean that users with fixed firmware wouldn't need to pass a parameter.

I believe Matthew is correct here. I found another system today with completely different vendor for both the system and the tpm chip. In addition another Lenovo model, the L490, has the issue.

This initial attempt at a solution like Matthew suggested works on the system I found today, but I imagine it is all sorts of wrong. In the 2 systems where I've seen it, there are about 100000 interrupts in around 1.5 seconds, and then the irq code shuts down the interrupt because they aren't being handled.

drivers/char/tpm/tpm_tis_core.c +0 -0
index 49ae09ac604f..478e9d02a3fa 100644
@@ -27,6 +27,11 @@
 #include "tpm.h"
 #include "tpm_tis_core.h"
 
Quoted 7 lines

drivers/char/tpm/tpm_tis_core.c line 33

Quoted 4 lines
drivers/char/tpm/tpm_tis_core.c +4 -0
+static unsigned int time_start = 0;
+static bool storm_check = true;
+static bool storm_killed = false;
+static u32 irqs_fired = 0;

Maybe kstat_irqs() would be a better idea than ad hoc stats.

Thanks, yes that would be better.

Quoted 45 lines
Quoted 45 lines
drivers/char/tpm/tpm_tis_core.c +12 -5
+
 static void tpm_tis_clkrun_enable(struct tpm_chip *chip, bool value);
 
 static void tpm_tis_enable_interrupt(struct tpm_chip *chip, u8 mask)
@@ -464,25 +469,31 @@ static int tpm_tis_send_data(struct tpm_chip *chip, const u8 *buf, size_t len)
        return rc;
 }
 
-static void disable_interrupts(struct tpm_chip *chip)
+static void __disable_interrupts(struct tpm_chip *chip)
 {
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
        u32 intmask;
        int rc;
 
-       if (priv->irq == 0)
-               return;
-
        rc = tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask);
        if (rc < 0)
                intmask = 0;
 
        intmask &= ~TPM_GLOBAL_INT_ENABLE;
        rc = tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask);
+       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
+}
+
+static void disable_interrupts(struct tpm_chip *chip)
+{
+       struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
+       if (priv->irq == 0)
+               return;
+
+       __disable_interrupts(chip);
        devm_free_irq(chip->dev.parent, priv->irq, chip);
        priv->irq = 0;
-       chip->flags &= ~TPM_CHIP_FLAG_IRQ;
 }
 
 /*
@@ -528,6 +539,12 @@ static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len)
        int rc, irq;
        struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
 
Quoted 10 lines

drivers/char/tpm/tpm_tis_core.c static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len) · line 546

Quoted 5 lines
drivers/char/tpm/tpm_tis_core.c +5 -0
+       if (unlikely(storm_killed)) {
+               devm_free_irq(chip->dev.parent, priv->irq, chip);
+               priv->irq = 0;
+               storm_killed = false;
+       }

OK this kind of bad solution because if tpm_tis_send() is not called, then IRQ is never freed. AFAIK, devres_* do not sleep but use spin lock, i.e. you could render out both storm_check and storm_killed.

Quoted 7 lines

Is there a way to flag it for freeing later while in an interrupt context? I'm not sure where to clean it up since devm_free_irq can't be called in tis_int_handler.

Before diving further into that though, does anyone else have an opinion on ripping out the irq code, and just using polling? We've been only polling since 2015 anyways.

Given these all these issues, it's quite obvious that Windows side is just polling. I'll ack a patch that removes the IRQ code for sure.

/Jarkko

Reviewed-by on v1 1/1
#19 lore raw
In reply to Jerry Snitselaar

Hi,

Quoted 13 lines · Jerry Snitselaar

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 6 lines

Patch looks good to me:

Regards,

Hans

Quoted 66 lines
---
  drivers/char/tpm/tpm_tis.c | 29 +++++++++++++++++++++++++++--
  1 file changed, 27 insertions(+), 2 deletions(-)
drivers/char/tpm/tpm_tis.c +27 -2
index 0b214963539d..4ed6e660273a 100644
@@ -27,6 +27,7 @@
  #include <linux/of.h>
  #include <linux/of_device.h>
  #include <linux/kernel.h>
+#include <linux/dmi.h>
  #include "tpm.h"
  #include "tpm_tis_core.h"
 
@@ -49,8 +50,8 @@ static inline struct tpm_tis_tcg_phy *to_tpm_tis_tcg_phy(struct tpm_tis_data *da
  	return container_of(data, struct tpm_tis_tcg_phy, priv);
  }
 
-static bool interrupts = true;
-module_param(interrupts, bool, 0444);
+static int interrupts = -1;
+module_param(interrupts, int, 0444);
  MODULE_PARM_DESC(interrupts, "Enable interrupts");
 
  static bool itpm;
@@ -63,6 +64,28 @@ module_param(force, bool, 0444);
  MODULE_PARM_DESC(force, "Force device probe rather than using ACPI entry");
  #endif
 
+static int tpm_tis_disable_irq(const struct dmi_system_id *d)
+{
+	if (interrupts == -1) {
+		pr_notice("tpm_tis: %s detected: disabling interrupts.\n", d->ident);
+		interrupts = 0;
+	}
+
+	return 0;
+}
+
+static const struct dmi_system_id tpm_tis_dmi_table[] = {
+	{
+		.callback = tpm_tis_disable_irq,
+		.ident = "ThinkPad T490s",
+		.matches = {
+			DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
+			DMI_MATCH(DMI_PRODUCT_VERSION, "ThinkPad T490s"),
+		},
+	},
+	{}
+};
+
  #if defined(CONFIG_PNP) && defined(CONFIG_ACPI)
  static int has_hid(struct acpi_device *dev, const char *hid)
  {
@@ -192,6 +215,8 @@ static int tpm_tis_init(struct device *dev, struct tpm_info *tpm_info)
  	int irq = -1;
  	int rc;
 
+	dmi_check_system(tpm_tis_dmi_table);
+
  	rc = check_acpi_tpm2(dev);
  	if (rc)
  		return rc;
Reviewed-by on v1 1/1
#20 lore raw
In reply to Jerry Snitselaar
Quoted 13 lines · Jerry Snitselaar

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 6 lines

I'll apply this and make it available in linux-next.

/Jarkko

Quoted 69 lines
---
 drivers/char/tpm/tpm_tis.c | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)
drivers/char/tpm/tpm_tis.c +27 -2
index 0b214963539d..4ed6e660273a 100644
@@ -27,6 +27,7 @@
 #include <linux/of.h>
 #include <linux/of_device.h>
 #include <linux/kernel.h>
+#include <linux/dmi.h>
 #include "tpm.h"
 #include "tpm_tis_core.h"
 
@@ -49,8 +50,8 @@ static inline struct tpm_tis_tcg_phy *to_tpm_tis_tcg_phy(struct tpm_tis_data *da
 	return container_of(data, struct tpm_tis_tcg_phy, priv);
 }
 
-static bool interrupts = true;
-module_param(interrupts, bool, 0444);
+static int interrupts = -1;
+module_param(interrupts, int, 0444);
 MODULE_PARM_DESC(interrupts, "Enable interrupts");
 
 static bool itpm;
@@ -63,6 +64,28 @@ module_param(force, bool, 0444);
 MODULE_PARM_DESC(force, "Force device probe rather than using ACPI entry");
 #endif
 
+static int tpm_tis_disable_irq(const struct dmi_system_id *d)
+{
+	if (interrupts == -1) {
+		pr_notice("tpm_tis: %s detected: disabling interrupts.\n", d->ident);
+		interrupts = 0;
+	}
+
+	return 0;
+}
+
+static const struct dmi_system_id tpm_tis_dmi_table[] = {
+	{
+		.callback = tpm_tis_disable_irq,
+		.ident = "ThinkPad T490s",
+		.matches = {
+			DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
+			DMI_MATCH(DMI_PRODUCT_VERSION, "ThinkPad T490s"),
+		},
+	},
+	{}
+};
+
 #if defined(CONFIG_PNP) && defined(CONFIG_ACPI)
 static int has_hid(struct acpi_device *dev, const char *hid)
 {
@@ -192,6 +215,8 @@ static int tpm_tis_init(struct device *dev, struct tpm_info *tpm_info)
 	int irq = -1;
 	int rc;
 
+	dmi_check_system(tpm_tis_dmi_table);
+
 	rc = check_acpi_tpm2(dev);
 	if (rc)
 		return rc;
Signature · 1 line
2.28.0
#21 lore raw
Quoted 21 lines · Jarkko Sakkinen
Quoted 13 lines · Jerry Snitselaar

There is a misconfiguration in the bios of the gpio pin used for the interrupt in the T490s. When interrupts are enabled in the tpm_tis driver code this results in an interrupt storm. This was initially reported when we attempted to enable the interrupt code in the tpm_tis driver, which previously wasn't setting a flag to enable it. Due to the reports of the interrupt storm that code was reverted and we went back to polling instead of using interrupts. Now that we know the T490s problem is a firmware issue, add code to check if the system is a T490s and disable interrupts if that is the case. This will allow us to enable interrupts for everyone else. If the user has a fixed bios they can force the enabling of interrupts with tpm_tis.interrupts=1 on the kernel command line.

Quoted 6 lines
Quoted 3 lines

I'll apply this and make it available in linux-next.

Applied.

Thank you.

/Jarkko