Kernel 6.4 failing to access TPM on Framework Laptop 12th gen
Subsystems (4)
Hi, Thorsten here, the Linux kernel's regression tracker.
I noticed a regression report in bugzilla.kernel.org. As many (most?) kernel developers don't keep an eye on it, I decided to forward it by mail.
Lino Sanfilippo, apparently it's cause by a change of yours.
Note, you have to use bugzilla to reach the reporters, as I sadly[1] can not CCed them in mails like this.
Quoting from https://bugzilla.kernel.org/show_bug.cgi?id=217631 :
Quoted 18 lines
After updating to linux-6.4.1 several users of Framework Laptop running Intel 12th gen CPUs reported that their machine fails to boot (in my case i5-1240P). It seems it is working fine for the model using 11th gen CPUs. See
https://community.frame.work/t/boot-and-shutdown-hangs-with-arch-linux-kernel-6-4-1-mainline-and-arch/33118* When using TPM to unlock a LUKS volume it will wait infinitely to start systemd-cryptsetup. When forcing a reboot, it will still wait for this service to stop and print the following message after waiting for some time (see attached cryptsetup_stack.jpg) * When disabling TPM unlock, systemd-cryptsetup will start successfully and allow to unlock the volume using the passphrase. However, after that the boot will also wait indefinitely for systemd-pcrphase to start. When force rebooting it will still wait for this service to stop (see pcrphase_stack.jpg).
Quoted 7 lines
I was able to boot successfully by either reverting
e644b2f498d297a928efcb7ff6f900c27f8b788eor disabling interrupts for my model like it has been done here:
https://lore.kernel.org/linux-integrity/20230620-flo-lenovo-l590-tpm-fix-v1-1-16032a8b5a1d%40bezdeka.de/
DMI_MATCH(DMI_SYS_VENDOR, "Framework"),DMI_MATCH(DMI_PRODUCT_VERSION, "A4"),
See the ticket and the linked forum post for more details.
[TLDR for the rest of this mail: I'm adding this report to the list of tracked Linux kernel regressions; the text you find below is based on a few templates paragraphs you might have encountered already in similar form.]
BTW, let me use this mail to also add the report to the list of tracked regressions to ensure it's doesn't fall through the cracks:
#regzbot introduced: e644b2f498d297a
https://bugzilla.kernel.org/show_bug.cgi?id=217631
#regzbot title: tpm: boot problems with Framework Laptop 12th gen #regzbot ignore-activity
This isn't a regression? This issue or a fix for it are already discussed somewhere else? It was fixed already? You want to clarify when the regression started to happen? Or point out I got the title or something else totally wrong? Then just reply and tell me -- ideally while also telling regzbot about it, as explained by the page listed in the footer of this mail.
Developers: When fixing the issue, remember to add 'Link:' tags pointing to the report (e.g. the buzgzilla ticket and maybe this mail as well, if this thread sees some discussion). See page linked in footer for details.
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.
[1] because bugzilla.kernel.org tells users upon registration their "email address will never be displayed to logged out users"
PATCH 1/2 tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
This device suffer an irq storm, so add it in tpm_tis_dmi_table to force polling.
https://bugs.archlinux.org/user/38129
https://bugzilla.kernel.org/show_bug.cgi?id=217631
--- drivers/char/tpm/tpm_tis.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/char/tpm/tpm_tis.c +8 -0
@@ -114,6 +114,14 @@ static int tpm_tis_disable_irq(const struct dmi_system_id *d) }static const struct dmi_system_id tpm_tis_dmi_table[] = { + { + .callback = tpm_tis_disable_irq, + .ident ="Framework Laptop Intel 12th gen" , + .matches = { +DMI_MATCH (DMI_SYS_VENDOR ,"Framework" ), +DMI_MATCH (DMI_PRODUCT_VERSION ,"A4" ), + }, + }, { .callback = tpm_tis_disable_irq, .ident ="ThinkPad T490s" ,
Signature · 1 line
2.41.0
PATCH 2/2 tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 13th gen
This device suffer an irq storm, so add it in tpm_tis_dmi_table to force polling.
https://bugs.archlinux.org/user/38129
https://bugzilla.kernel.org/show_bug.cgi?id=217631
--- drivers/char/tpm/tpm_tis.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/char/tpm/tpm_tis.c +8 -0
@@ -122,6 +122,14 @@ static const struct dmi_system_id tpm_tis_dmi_table[] = { DMI_MATCH(DMI_PRODUCT_VERSION, "A4"), }, }, + { + .callback = tpm_tis_disable_irq, + .ident = "Framework Laptop Intel 13th gen", + .matches = { +DMI_MATCH (DMI_SYS_VENDOR ,"Framework" ), + DMI_MATCH(DMI_PRODUCT_VERSION, "A6"), + }, + }, { .callback = tpm_tis_disable_irq, .ident = "ThinkPad T490s",
Signature · 1 line
2.41.0
Re: [PATCH 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to Christian HesseHi, Thorsten here, the Linux kernel's regression tracker. I'm not a proper reviewer for patches like this, but nevertheless two small things:
Quoted 2 lines · Christian Hesse
This device suffer an irq storm, so add it in
tpm_tis_dmi_tableto force polling.
Thx for working on this!
Quoted 1 line
Not sure why this link might be helpful. Did you maybe mean to include another one? Like this one?
Quoted 3 lines
https://bugzilla.kernel.org/show_bug.cgi?id=217631
Fixes e644b2f498d297a928efcb7ff6f900c27f8b788e
This should be:
Quoted 2 lines
Reported-by <[email protected]>
The bugzilla link from above above should be here like this instead:
BTW, checkpatch.pl should have mentioned the two latter things.
HTH, Ciao, Thorsten
Quoted 24 lines
Signed-off-by Christian Hesse <[email protected]>--- drivers/char/tpm/tpm_tis.c | 8 ++++++++ 1 file changed, 8 insertions(+)drivers/char/tpm/tpm_tis.c +8
-0@@ -114,6 +114,14 @@ static int tpm_tis_disable_irq(const struct dmi_system_id *d) }static const struct dmi_system_id tpm_tis_dmi_table[] = { + { + .callback = tpm_tis_disable_irq, + .ident ="Framework Laptop Intel 12th gen" , + .matches = { +DMI_MATCH (DMI_SYS_VENDOR ,"Framework" ), +DMI_MATCH (DMI_PRODUCT_VERSION ,"A4" ), + }, + }, { .callback = tpm_tis_disable_irq, .ident ="ThinkPad T490s" ,
Greg KH
maintainerRe: [PATCH 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to Christian HesseQuoted 5 lines · Christian Hesse
This device suffer an irq storm, so add it in
tpm_tis_dmi_tableto force polling.https://bugs.archlinux.org/user/38129
https://bugzilla.kernel.org/show_bug.cgi?id=217631
Add "Link:" tags?
Quoted 2 lines
Fixes e644b2f498d297a928efcb7ff6f900c27f8b788e
As per the kernel documentation, this should be:
otherwise you're going to get some emails from the linux-next bot.
thanks,
greg k-h
PATCH v2 1/2 tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to Christian HesseThis device suffer an irq storm, so add it in tpm_tis_dmi_table to force polling.
--- drivers/char/tpm/tpm_tis.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/char/tpm/tpm_tis.c +8 -0
@@ -114,6 +114,14 @@ static int tpm_tis_disable_irq(const struct dmi_system_id *d) }static const struct dmi_system_id tpm_tis_dmi_table[] = { + { + .callback = tpm_tis_disable_irq, + .ident ="Framework Laptop Intel 12th gen" , + .matches = { +DMI_MATCH (DMI_SYS_VENDOR ,"Framework" ), +DMI_MATCH (DMI_PRODUCT_VERSION ,"A4" ), + }, + }, { .callback = tpm_tis_disable_irq, .ident ="ThinkPad T490s" ,
Signature · 1 line
2.41.0
PATCH v2 2/2 tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 13th gen
This device suffer an irq storm, so add it in tpm_tis_dmi_table to force polling.
--- drivers/char/tpm/tpm_tis.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/char/tpm/tpm_tis.c +8 -0
@@ -122,6 +122,14 @@ static const struct dmi_system_id tpm_tis_dmi_table[] = { DMI_MATCH(DMI_PRODUCT_VERSION, "A4"), }, }, + { + .callback = tpm_tis_disable_irq, + .ident = "Framework Laptop Intel 13th gen", + .matches = { +DMI_MATCH (DMI_SYS_VENDOR ,"Framework" ), + DMI_MATCH(DMI_PRODUCT_VERSION, "A6"), + }, + }, { .callback = tpm_tis_disable_irq, .ident = "ThinkPad T490s",
Signature · 1 line
2.41.0
Re: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to Christian HesseQuoted 8 lines · Christian Hesse
This device suffer an irq storm, so add it in
tpm_tis_dmi_tableto force polling.Fixes e644b2f498d2 ("tpm, tpm_tis: Enable interrupt test")Reported-by <[email protected]>
Thx again for working on this. FWIW, a quote from a recent comment there FYI:
``` mapleleaf 2023-07-10 16:37:03 UTC I have the same problem and I own a Framework 12th-gen, but for whatever reason my DMI_PRODUCT_VERSION is A8 instead of A6... $ sudo dmidecode -s baseboard-version A8 [tag] [reply] [−] Private Comment 13 mapleleaf 2023-07-10 16:41:29 UTC And also: $ sudo dmidecode -s system-version A8 ```
Quoted 23 lines
--- drivers/char/tpm/tpm_tis.c | 8 ++++++++ 1 file changed, 8 insertions(+)drivers/char/tpm/tpm_tis.c +8
-0@@ -114,6 +114,14 @@ static int tpm_tis_disable_irq(const struct dmi_system_id *d) }static const struct dmi_system_id tpm_tis_dmi_table[] = { + { + .callback = tpm_tis_disable_irq, + .ident ="Framework Laptop Intel 12th gen" , + .matches = { +DMI_MATCH (DMI_SYS_VENDOR ,"Framework" ), +DMI_MATCH (DMI_PRODUCT_VERSION ,"A4" ), + }, + }, { .callback = tpm_tis_disable_irq, .ident ="ThinkPad T490s" ,
Jason Gunthorpe
reviewerRe: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to Christian HesseQuoted 2 lines · Christian Hesse
This device suffer an irq storm, so add it in
tpm_tis_dmi_tableto force polling.
I can't help but feel like we are doing something wrong in the Linux driver that we keep having IRQ problems.
Surely Windows uses the IRQ on these devices? How does it work reliably there?
Jason
Jarkko Sakkinen
maintainerRe: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Quoted 11 lines · Jason Gunthorpe
Quoted 2 lines · Christian Hesse
This device suffer an irq storm, so add it in
tpm_tis_dmi_tableto force polling.I can't help but feel like we are doing something wrong in the Linux driver that we keep having IRQ problems.
Surely Windows uses the IRQ on these devices? How does it work reliably there?
Jason
I'm about to send a PR to Linus with a pile of IRQ fixes for v6.4 feature.
BR, Jarkko
James Bottomley
maintainerRe: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to Jason GunthorpeQuoted 9 lines · Jason Gunthorpe
Quoted 2 lines · Christian Hesse
This device suffer an irq storm, so add it in
tpm_tis_dmi_tableto force polling.I can't help but feel like we are doing something wrong in the Linux driver that we keep having IRQ problems.
Surely Windows uses the IRQ on these devices? How does it work reliably there?
I've had some friends in the MS Linux team ask around to find out if anyone in the windows group is willing to divulge a little on this. My suspicion is that if interrupts are enabled at all on Windows TIS, it's via a whitelist.
James
Re: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to Jason GunthorpeQuoted 11 lines · Jason Gunthorpe
Quoted 2 lines · Christian Hesse
This device suffer an irq storm, so add it in
tpm_tis_dmi_tableto force polling.I can't help but feel like we are doing something wrong in the Linux driver that we keep having IRQ problems.
Surely Windows uses the IRQ on these devices? How does it work reliably there?
Jason
I worry about it as well, especially as more vendors get added to the list. On the other hand it seems like every 6-12 months I am interacting with vendors that repeatedly forget to have reserved memory regions added to IVRS and DMAR tables for devices that need them. So I guess it is possible for the problem to be on their end as well. For at least one case that someone looked at back in May, it looked like he could see the issue looking at a schematic for the system.
Regards,
Jerry
Re: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to Christian HesseChristian Hesse <[email protected]> on Mon, 2023/07/10 16:28:
Quoted 2 lines
This device suffer an irq storm, so add it in
tpm_tis_dmi_tableto force polling.
Uh, looks like this is not quite right. The product version specifies the CPU "level" across generations (for example "i5-1240P" vs. "i7-1260P" vs. "i7-1280P"). So I guess we should match on the product name instead...
I will send an updated set of patches. Would be nice if anybody could verify this...
Signature · 3 lines
main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH"
"CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];)
putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}PATCH v3 1/2 tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
This device suffer an irq storm, so add it in tpm_tis_dmi_table to force polling.
--- drivers/char/tpm/tpm_tis.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/char/tpm/tpm_tis.c +8 -0
@@ -114,6 +114,14 @@ static int tpm_tis_disable_irq(const struct dmi_system_id *d) }static const struct dmi_system_id tpm_tis_dmi_table[] = { + { + .callback = tpm_tis_disable_irq, + .ident ="Framework Laptop (12th Gen Intel Core)" , + .matches = { +DMI_MATCH (DMI_SYS_VENDOR ,"Framework" ), +DMI_MATCH (DMI_PRODUCT_NAME ,"Laptop (12th Gen Intel Core)" ), + }, + }, { .callback = tpm_tis_disable_irq, .ident ="ThinkPad T490s" ,
Signature · 1 line
2.41.0
PATCH v3 2/2 tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 13th gen
This device suffer an irq storm, so add it in tpm_tis_dmi_table to force polling.
--- drivers/char/tpm/tpm_tis.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/char/tpm/tpm_tis.c +8 -0
@@ -122,6 +122,14 @@ static const struct dmi_system_id tpm_tis_dmi_table[] = { DMI_MATCH(DMI_PRODUCT_NAME, "Laptop (12th Gen Intel Core)"), }, }, + { + .callback = tpm_tis_disable_irq, + .ident = "Framework Laptop (13th Gen Intel Core)", + .matches = { +DMI_MATCH (DMI_SYS_VENDOR ,"Framework" ), + DMI_MATCH(DMI_PRODUCT_NAME, "Laptop (13th Gen Intel Core)"), + }, + }, { .callback = tpm_tis_disable_irq, .ident = "ThinkPad T490s",
Signature · 1 line
2.41.0
Jarkko Sakkinen
maintainerRe: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to Christian HesseQuoted 10 lines · Christian Hesse
Christian Hesse <[email protected]> on Mon, 2023/07/10 16:28:
Quoted 2 lines
This device suffer an irq storm, so add it in
tpm_tis_dmi_tableto force polling.Uh, looks like this is not quite right. The product version specifies the CPU "level" across generations (for example "i5-1240P" vs. "i7-1260P" vs. "i7-1280P"). So I guess we should match on the product name instead...
I will send an updated set of patches. Would be nice if anybody could verify this...
OK, this good to hear! I've been late with my pull request (past rc1) because of kind of conflicting timing with Finnish holiday season and relocating my home office.
I'll replace v2 patches with v3 and send the PR for rc2 after that. So unluck turned into luck this time :-)
Thank you for spotting this!
BR, Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Quoted 12 lines · Jarkko Sakkinen
Quoted 10 lines · Christian Hesse
Christian Hesse <[email protected]> on Mon, 2023/07/10 16:28:
Quoted 2 lines
This device suffer an irq storm, so add it in
tpm_tis_dmi_tableto force polling.Uh, looks like this is not quite right. The product version specifies the CPU "level" across generations (for example "i5-1240P" vs. "i7-1260P" vs. "i7-1280P"). So I guess we should match on the product name instead...
I will send an updated set of patches. Would be nice if anybody could verify this...
Quoted 8 lines
OK, this good to hear! I've been late with my pull request (past rc1) because of kind of conflicting timing with Finnish holiday season and relocating my home office.
I'll replace v2 patches with v3 and send the PR for rc2 after that. So unluck turned into luck this time :-)
Thank you for spotting this!
Please, sanity check before I send the PR for rc2:
https://git.kernel.org/pub/scm/linux/kernel/git/jarkko/linux-tpmdd.git/
BR, Jarkko
Re: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Jarkko Sakkinen <[email protected]> on Tue, 2023/07/11 00:51:
Quoted 5 lines · Jarkko Sakkinen
Quoted 4 lines
OK, this good to hear! I've been late with my pull request (past rc1) because of kind of conflicting timing with Finnish holiday season and relocating my home office.
Quoted 8 lines
Quoted 4 lines
I'll replace v2 patches with v3 and send the PR for rc2 after that. So unluck turned into luck this time :-)
Thank you for spotting this!
Please, sanity check before I send the PR for rc2:
https://git.kernel.org/pub/scm/linux/kernel/git/jarkko/linux-tpmdd.git/
Looks good and works as expected on Framework Laptop 12th Gen, verified by me and someone in the linked bugzilla ticket. I do not have Framework Laptop 13th Gen available for testing.
Looks like there are general workarounds by disabling interrupts after a number of unhandled IRQs. Will this still go in?
Signature · 3 lines
main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH"
"CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];)
putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}Jarkko Sakkinen
maintainerRe: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Quoted 13 lines · Christian Hesse
Jarkko Sakkinen <[email protected]> on Tue, 2023/07/11 00:51:
Quoted 12 lines · Jarkko Sakkinen
Quoted 8 lines
OK, this good to hear! I've been late with my pull request (past rc1) because of kind of conflicting timing with Finnish holiday season and relocating my home office.
I'll replace v2 patches with v3 and send the PR for rc2 after that. So unluck turned into luck this time :-)
Thank you for spotting this!
Please, sanity check before I send the PR for rc2:
Quoted 12 lines
Looks good and works as expected on Framework Laptop 12th Gen, verified by me and someone in the linked bugzilla ticket. I do not have Framework Laptop 13th Gen available for testing.
Looks like there are general workarounds by disabling interrupts after a number of unhandled IRQs. Will this still go in?
Signature · 3 lines
main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}
So I think it should, what is deny listed now will stay like that even if Lino's patch is included.
BR, Jarkko
Re: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to Jarkko SakkinenQuoted 11 lines · Jarkko Sakkinen
On Mon, 2023-07-10 at 23:13 +0200, Christian Hesse wrote:
OK, this good to hear! I've been late with my pull request (past rc1) because of kind of conflicting timing with Finnish holiday season and relocating my home office.
I'll replace v2 patches with v3 and send the PR for rc2 after that. So unluck turned into luck this time :-)
Thank you for spotting this!
I want to say: this issue is NOT limited to Framework laptops.
For example this MSI gen12 i5-1240P laptop also suffers from same problem:
Manufacturer: Micro-Star International Co., Ltd. Product Name: Summit E13FlipEvo A12MT
Version: REV:1.0
SKU Number: 13P3.1
Family: Summit
So, probably just blacklisting affected models is not the best solution...
Re: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Quoted 23 lines · Grundik
Quoted 11 lines · Jarkko Sakkinen
On Mon, 2023-07-10 at 23:13 +0200, Christian Hesse wrote:
OK, this good to hear! I've been late with my pull request (past rc1) because of kind of conflicting timing with Finnish holiday season and relocating my home office.
I'll replace v2 patches with v3 and send the PR for rc2 after that. So unluck turned into luck this time :-)
Thank you for spotting this!
I want to say: this issue is NOT limited to Framework laptops.
For example this MSI gen12 i5-1240P laptop also suffers from same problem:
Manufacturer: Micro-Star International Co., Ltd. Product Name: Summit E13FlipEvo A12MT
Version: REV:1.0
SKU Number: 13P3.1
Family: Summit
Quoted 2 lines
So, probably just blacklisting affected models is not the best solution...
Hmmm. Jarkko, you earlier in the thread said: "I'm about to send a PR to Linus with a pile of IRQ fixes for v6.4 feature.". Could any of those potentially help Grundik or those with the Framework laptop? Is that worth testing?
Ciao, Thorsten
Jarkko Sakkinen
maintainerRe: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to GrundikQuoted 23 lines · Grundik
Quoted 11 lines · Jarkko Sakkinen
On Mon, 2023-07-10 at 23:13 +0200, Christian Hesse wrote:
OK, this good to hear! I've been late with my pull request (past rc1) because of kind of conflicting timing with Finnish holiday season and relocating my home office.
I'll replace v2 patches with v3 and send the PR for rc2 after that. So unluck turned into luck this time :-)
Thank you for spotting this!
I want to say: this issue is NOT limited to Framework laptops.
For example this MSI gen12 i5-1240P laptop also suffers from same problem:
Manufacturer: Micro-Star International Co., Ltd. Product Name: Summit E13FlipEvo A12MT
Version: REV:1.0
SKU Number: 13P3.1
Family: Summit
Quoted 2 lines
So, probably just blacklisting affected models is not the best solution...
It will be supplemented with
Together they should fairly sustainable framework.
Lino, can you add the same fixes tag as for this. It would probably ignore inline comments to keep the patch minimal since it is a critical fix. Just do the renames, remove inline comments and send v3.
For tpm_tis_check_for_interrupt_storm(), you can could rename it simply as tpm_tis_update_unhandle_irqs() as that it what it does (my review did not include a suggestion for this).
This way I think it should be fairly trivial to get a version that can be landed.
To put short:
1. Do the renames as suggested, they are good enough for me.
2. Drop inline comments, their usefulness is somewhat questionable and they increase the diff.
3. Generally aim for minimal diff but I think this should be good enough if you do steps 1 and 2.
If you don't have the time at hand, I can carefully do these cleanups and apply the patch. If you have the time and motivation, go ahead and send v3.
BR, Jarkko
Re: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Hi,
Quoted 52 lines · Jarkko Sakkinen
Quoted 23 lines · Grundik
Quoted 11 lines · Jarkko Sakkinen
On Mon, 2023-07-10 at 23:13 +0200, Christian Hesse wrote:
OK, this good to hear! I've been late with my pull request (past rc1) because of kind of conflicting timing with Finnish holiday season and relocating my home office.
I'll replace v2 patches with v3 and send the PR for rc2 after that. So unluck turned into luck this time :-)
Thank you for spotting this!
I want to say: this issue is NOT limited to Framework laptops.
For example this MSI gen12 i5-1240P laptop also suffers from same problem:
Manufacturer: Micro-Star International Co., Ltd. Product Name: Summit E13FlipEvo A12MT
Version: REV:1.0
SKU Number: 13P3.1
Family: SummitQuoted 2 lines
So, probably just blacklisting affected models is not the best solution...
It will be supplemented with
Together they should fairly sustainable framework.
Lino, can you add the same fixes tag as for this. It would probably ignore inline comments to keep the patch minimal since it is a critical fix. Just do the renames, remove inline comments and send v3.
For
tpm_tis_check_for_interrupt_storm(), you can could rename it simply astpm_tis_update_unhandle_irqs()as that it what it does (my review did not include a suggestion for this).This way I think it should be fairly trivial to get a version that can be landed.
To put short:
1. Do the renames as suggested, they are good enough for me.
2. Drop inline comments, their usefulness is somewhat questionable and they increase the diff.
3. Generally aim for minimal diff but I think this should be good enough if you do steps 1 and 2.
Quoted 4 lines
If you don't have the time at hand, I can carefully do these cleanups and apply the patch. If you have the time and motivation, go ahead and send v3.
Quoted 1 line
BR, Jarkko
No problem, I can do this. But note that the next version will be v4 not v3 (as v3 is the version you recently reviewed).
Regards,
Lino
Re: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to Jarkko SakkinenQuoted 10 lines · Jarkko Sakkinen
Quoted 10 lines
I want to say: this issue is NOT limited to Framework laptops.
For example this MSI gen12 i5-1240P laptop also suffers from same problem:
Manufacturer: Micro-Star International Co., Ltd. Product Name: Summit E13FlipEvo A12MT
Version: REV:1.0
SKU Number: 13P3.1
Family: Summit
Quoted 9 lines
Quoted 2 lines
So, probably just blacklisting affected models is not the best solution...
It will be supplemented with
Together they should fairly sustainable framework.
Unfortunately, they dont. Problem still occurs in debian 6.5-rc4 kernel, with forementioned laptop. According to sources, these patches are applied in that kernel version.
What can I do to assist fixing it?..
Re: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Quoted 9 lines · Grundik
Quoted 6 lines · Jarkko Sakkinen
Quoted 6 lines
I want to say: this issue is NOT limited to Framework laptops.
For example this MSI gen12 i5-1240P laptop also suffers from same problem:
Manufacturer: Micro-Star International Co., Ltd. Product Name: Summit E13FlipEvo A12MT[...]
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
It will be supplemented with
https://lore.kernel.org/linux-integrity/CTYXI8TL7C36.2SCWH82FAZWBO@suppilovahvero/T/#me895f1920ca6983f791b58a6fa0c157161a33849Together they should fairly sustainable framework.
Unfortunately, they dont. Problem still occurs in debian 6.5-rc4 kernel, with forementioned laptop. According to sources, these patches are applied in that kernel version.
Jarkko & Lino, did you see this msg Grundik posted that about a week ago? It looks like there is still something wrong there that need attention. Or am I missing something?
FWIW, two more users reported that they still see similar problems with recent 6.4.y kernels that contain the "tpm,tpm_tis: Disable interrupts after 1000 unhandled IRQs" patch. Both also with MSI laptops:
https://bugzilla.kernel.org/show_bug.cgi?id=217631#c18
https://bugzilla.kernel.org/show_bug.cgi?id=217631#c20
No reply either afaics.
Ciao, Thorsten
Re: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Quoted 10 lines · Thorsten Leemhuis
Jarkko & Lino, did you see this msg Grundik posted that about a week ago? It looks like there is still something wrong there that need attention. Or am I missing something?
FWIW, two more users reported that they still see similar problems with
recent 6.4.y kernels that contain the "tpm,tpm_tis: Disable interrupts
after 1000 unhandled IRQs" patch. Both also with MSI laptops:
Quoted 4 lines
https://bugzilla.kernel.org/show_bug.cgi?id=217631#c18
https://bugzilla.kernel.org/show_bug.cgi?id=217631#c20No reply either afaics.
As I said before: it does not looks like blacklisting is a good solution at all.
If there are general fix, then blacklisting only makes testing of that fix more difficult. If general fix works, why blacklist? If it does not work and its impossible to figure out why — maybe there should be kernel boot option to select between polling/irq instead of/in addition to hard-coded blacklist.
Unfortunately, its very hard to test this fixes on my side: since TPM is not a module, but compiled into kernel itself, it requires recompiling a whole kernel, which is quite a task for a laptop. But I will try my best, if needed.
Jarkko Sakkinen
maintainerRe: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Quoted 25 lines · Grundik
Quoted 10 lines · Thorsten Leemhuis
Jarkko & Lino, did you see this msg Grundik posted that about a week ago? It looks like there is still something wrong there that need attention. Or am I missing something?
FWIW, two more users reported that they still see similar problems with
recent 6.4.y kernels that contain the "tpm,tpm_tis: Disable interrupts
after 1000 unhandled IRQs" patch. Both also with MSI laptops:Quoted 4 lines
https://bugzilla.kernel.org/show_bug.cgi?id=217631#c18
https://bugzilla.kernel.org/show_bug.cgi?id=217631#c20No reply either afaics.
As I said before: it does not looks like blacklisting is a good solution at all.
If there are general fix, then blacklisting only makes testing of that fix more difficult. If general fix works, why blacklist? If it does not work and its impossible to figure out why — maybe there should be kernel boot option to select between polling/irq instead of/in addition to hard-coded blacklist.
Quoted 4 lines
Unfortunately, its very hard to test this fixes on my side: since TPM is not a module, but compiled into kernel itself, it requires recompiling a whole kernel, which is quite a task for a laptop. But I will try my best, if needed.
I agree with this.
Can you check my response to Thorsten and share your opinions?
BR, Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to Thorsten LeemhuisQuoted 27 lines · Thorsten Leemhuis
Quoted 9 lines · Grundik
Quoted 6 lines · Jarkko Sakkinen
Quoted 6 lines
I want to say: this issue is NOT limited to Framework laptops.
For example this MSI gen12 i5-1240P laptop also suffers from same problem:
Manufacturer: Micro-Star International Co., Ltd. Product Name: Summit E13FlipEvo A12MT[...]
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
It will be supplemented with
https://lore.kernel.org/linux-integrity/CTYXI8TL7C36.2SCWH82FAZWBO@suppilovahvero/T/#me895f1920ca6983f791b58a6fa0c157161a33849Together they should fairly sustainable framework.
Unfortunately, they dont. Problem still occurs in debian 6.5-rc4 kernel, with forementioned laptop. According to sources, these patches are applied in that kernel version.
Jarkko & Lino, did you see this msg Grundik posted that about a week ago? It looks like there is still something wrong there that need attention. Or am I missing something?
FWIW, two more users reported that they still see similar problems with recent 6.4.y kernels that contain the "tpm,tpm_tis: Disable interrupts after 1000 unhandled IRQs" patch. Both also with MSI laptops:
Quoted 6 lines
https://bugzilla.kernel.org/show_bug.cgi?id=217631#c18
https://bugzilla.kernel.org/show_bug.cgi?id=217631#c20No reply either afaics.
Ciao, Thorsten
I was planning to send a PR to Linus with a quirk for MSI GS66 Stealth 11UG, and apparently this bug report would add two additional MSI entries. This is becoming quickly a maintenance hell.
For Lenovo, I also added patch that will categorically disable irqs for all with vendor "LENOVO" because there was already six entries, and I got patch for 7th.
Now we at least know that IRQs TPMs are somewhat broken so it is time to make decisions to make this converge to something.
I see two long-standing options:
A. Move from deny list to allow list when considering using IRQs. This
can be supplemented with a kernel command-line parameter to enforce IRQs and ignore the allow list (and IRQ storm detection provides additional measure in case you try to enforce)
B. Change deny list to match only vendors for the time being. This can
be supplemented with a allow list that is processed after the deny list for models where IRQs are known to work.
This would the implementation of (B):
static const struct dmi_system_id tpm_tis_dmi_table[] = {
{
.callback = tpm_tis_disable_irq,
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Framework"),
},
},
{
.callback = tpm_tis_disable_irq,
.ident = "MSI GS66 Stealth 11UG",
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Micro-Star International Co., Ltd."),
},
},
{
.callback = tpm_tis_disable_irq,
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
},
},
{
.callback = tpm_tis_disable_irq,
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "TUXEDO"),
},
},
{
.callback = tpm_tis_disable_irq,
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "AAEON"),
},
},
{}
};BR, Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Quoted 93 lines · Jarkko Sakkinen
Quoted 27 lines · Thorsten Leemhuis
Quoted 9 lines · Grundik
Quoted 6 lines · Jarkko Sakkinen
Quoted 6 lines
I want to say: this issue is NOT limited to Framework laptops.
For example this MSI gen12 i5-1240P laptop also suffers from same problem:
Manufacturer: Micro-Star International Co., Ltd. Product Name: Summit E13FlipEvo A12MT[...]
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
It will be supplemented with
https://lore.kernel.org/linux-integrity/CTYXI8TL7C36.2SCWH82FAZWBO@suppilovahvero/T/#me895f1920ca6983f791b58a6fa0c157161a33849Together they should fairly sustainable framework.
Unfortunately, they dont. Problem still occurs in debian 6.5-rc4 kernel, with forementioned laptop. According to sources, these patches are applied in that kernel version.
Jarkko & Lino, did you see this msg Grundik posted that about a week ago? It looks like there is still something wrong there that need attention. Or am I missing something?
FWIW, two more users reported that they still see similar problems with recent 6.4.y kernels that contain the "tpm,tpm_tis: Disable interrupts after 1000 unhandled IRQs" patch. Both also with MSI laptops:
Quoted 6 lines
https://bugzilla.kernel.org/show_bug.cgi?id=217631#c18
https://bugzilla.kernel.org/show_bug.cgi?id=217631#c20No reply either afaics.
Ciao, Thorsten
I was planning to send a PR to Linus with a quirk for MSI GS66 Stealth 11UG, and apparently this bug report would add two additional MSI entries. This is becoming quickly a maintenance hell.
For Lenovo, I also added patch that will categorically disable irqs for all with vendor "LENOVO" because there was already six entries, and I got patch for 7th.
Now we at least know that IRQs TPMs are somewhat broken so it is time to make decisions to make this converge to something.
I see two long-standing options:
A. Move from deny list to allow list when considering using IRQs. This
can be supplemented with a kernel command-line parameter to enforce IRQs and ignore the allow list (and IRQ storm detection provides additional measure in case you try to enforce)
B. Change deny list to match only vendors for the time being. This can
be supplemented with a allow list that is processed after the deny list for models where IRQs are known to work.This would the implementation of (B):
static const struct dmi_system_id tpm_tis_dmi_table[] = { { .callback = tpm_tis_disable_irq, .matches = { DMI_MATCH(DMI_SYS_VENDOR, "Framework"), }, }, { .callback = tpm_tis_disable_irq, .ident = "MSI GS66 Stealth 11UG", .matches = { DMI_MATCH(DMI_SYS_VENDOR, "Micro-Star International Co., Ltd."), }, }, { .callback = tpm_tis_disable_irq, .matches = { DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"), }, }, { .callback = tpm_tis_disable_irq, .matches = { DMI_MATCH(DMI_SYS_VENDOR, "TUXEDO"), }, }, { .callback = tpm_tis_disable_irq, .matches = { DMI_MATCH(DMI_SYS_VENDOR, "AAEON"), }, }, {} };
Quoted 1 line
BR, Jarkko
I.e. this is a non-sustainable solution:
if (vendor && product) {
dev_info(&chip->dev,
"Consider adding the following entry to tpm_tis_dmi_table:\n");
dev_info(&chip->dev, "\tDMI_SYS_VENDOR: %s\n", vendor);
dev_info(&chip->dev, "\tDMI_PRODUCT_VERSION: %s\n", product);
}This is also super time consuming and takes the focus away from more important matters (like most likely the AMD rng fix would have gone smoother without these getting in the way all the time).
BR, Jarkko
Re: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Quoted 17 lines · Jarkko Sakkinen
Quoted 16 lines · Jarkko Sakkinen
On Fri Aug 11, 2023 at 11:18 AM EEST, Thorsten Leemhuis wrote:
I see two long-standing options:
A. Move from deny list to allow list when considering using IRQs. This
can be supplemented with a kernel command-line parameter to enforce
IRQs and ignore the allow list (and IRQ storm detection provides additional measure in case you try to enforce)
B. Change deny list to match only vendors for the time being. This can
be supplemented with a allow list that is processed after the deny
list for models where IRQs are known to work.
[...]
Quoted 4 lines
This is also super time consuming and takes the focus away from more important matters (like most likely the AMD rng fix would have gone smoother without these getting in the way all the time).
Main problem of any list is maintaining of them. So, I think there should not be any black or white lists at all. Module should work with reasonable default (polling is the one, which lived without problems for years and years due to bug, as I understand), and probably a boot option to force IRQ. Maybe module should warn user to try that option.
I don't know: is it even worth it to use IRQ, if it so problematic? Are there any significant advantages of that? I understand, polling is a resource consumer, but its just TPM, which is used mainly at the boot time, is it worth it?
Jarkko Sakkinen
maintainerRe: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Quoted 30 lines · Grundik
Quoted 17 lines · Jarkko Sakkinen
Quoted 16 lines · Jarkko Sakkinen
On Fri Aug 11, 2023 at 11:18 AM EEST, Thorsten Leemhuis wrote:
I see two long-standing options:
A. Move from deny list to allow list when considering using IRQs. This
can be supplemented with a kernel command-line parameter to enforce
IRQs and ignore the allow list (and IRQ storm detection provides additional measure in case you try to enforce)
B. Change deny list to match only vendors for the time being. This can
be supplemented with a allow list that is processed after the deny
list for models where IRQs are known to work.[...]
Quoted 4 lines
This is also super time consuming and takes the focus away from more important matters (like most likely the AMD rng fix would have gone smoother without these getting in the way all the time).
Main problem of any list is maintaining of them. So, I think there should not be any black or white lists at all. Module should work with reasonable default (polling is the one, which lived without problems for years and years due to bug, as I understand), and probably a boot option to force IRQ. Maybe module should warn user to try that option.
Quoted 4 lines
I don't know: is it even worth it to use IRQ, if it so problematic? Are there any significant advantages of that? I understand, polling is a resource consumer, but its just TPM, which is used mainly at the boot time, is it worth it?
+1
Thanks for sharing your opinion. I'll take the necessary steps.
BR, Jarkko
Re: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Quoted 4 lines · Jarkko Sakkinen
Thanks for sharing your opinion. I'll take the necessary steps.
I was thinking... Maybe I'm wrong, maybe I mistaking, but isnt this TPM located inside of the CPU chip? So that issue is not specific to laptop vendor or model, but its CPU-specific.
My MSI A12MT laptop has i5-1240P, Framework laptops mentioned in this thread, also has i5-1240P CPU. Unfortunately there are no such information about other affected models, but could it be just that CPU line?
Jarkko Sakkinen
maintainerRe: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Quoted 9 lines · Grundik
Quoted 4 lines · Jarkko Sakkinen
Thanks for sharing your opinion. I'll take the necessary steps.
I was thinking... Maybe I'm wrong, maybe I mistaking, but isnt this TPM located inside of the CPU chip? So that issue is not specific to laptop vendor or model, but its CPU-specific.
Nope, tpm_tis is MMIO interface for so called FIFO register interface, it could be backed e.g. by SPI [*]. It is a physical layer for MMIO. There's other backends too such as tpm_tis_i2c where kernel is directly exposed to the physical layer.
Quoted 4 lines
My MSI A12MT laptop has i5-1240P, Framework laptops mentioned in this thread, also has i5-1240P CPU. Unfortunately there are no such information about other affected models, but could it be just that CPU line?
[*] https://lkml.org/lkml/2023/8/14/1065
BR, Jarkko
Re: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to Jarkko SakkinenQuoted 4 lines · Jarkko Sakkinen
I was planning to send a PR to Linus with a quirk for MSI GS66 Stealth 11UG, and apparently this bug report would add two additional MSI entries. This is becoming quickly a maintenance hell.
Honestly, what would be the immediate effects of just not enabling the TPM irq by default at all, and making it an explicit opt-in?
When a common solution is to just disable the TPM in the BIOS entirely, and the end result is a working system, I really get the teeling that this is all pain for very very little gain.
Would anybody even notice if we just disabled it by default and added a "if you really want it, use 'tpm=irq' kernel command line"?
Hmm?
Linus
Jarkko Sakkinen
maintainerRe: [PATCH v2 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
Quoted 13 lines · Linus Torvalds
Quoted 4 lines · Jarkko Sakkinen
I was planning to send a PR to Linus with a quirk for MSI GS66 Stealth 11UG, and apparently this bug report would add two additional MSI entries. This is becoming quickly a maintenance hell.
Honestly, what would be the immediate effects of just not enabling the TPM irq by default at all, and making it an explicit opt-in?
When a common solution is to just disable the TPM in the BIOS entirely, and the end result is a working system, I really get the teeling that this is all pain for very very little gain.
Quoted 6 lines
Would anybody even notice if we just disabled it by default and added a "if you really want it, use 'tpm=irq' kernel command line"?
Hmm?
Linus
It would be in line with what I proposed in [1] (option A), and also what Grundik said in this thread.
Right now IRQs should be only enabled by a command-line option, and what "allow list" would mean in practice should really be something that would be advertised e.g. Device Tree or ACPI, not really part of the solution implemented right now.
I'll send a patch ASAP, which makes this feature opt-in.
[1] https://lore.kernel.org/linux-integrity/CUPW0XP1RFXI.162GZ78E46TBJ@suppilovahvero/
BR, Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH 1/2] tpm/tpm_tis: Disable interrupts for Framework Laptop Intel 12th gen
In reply to Christian HesseQuoted 23 lines · Christian Hesse
This device suffer an irq storm, so add it in
tpm_tis_dmi_tableto force polling.https://bugs.archlinux.org/user/38129
https://bugzilla.kernel.org/show_bug.cgi?id=217631Fixes e644b2f498d297a928efcb7ff6f900c27f8b788eReported-by <[email protected]>Signed-off-by Christian Hesse <[email protected]>--- drivers/char/tpm/tpm_tis.c | 8 ++++++++ 1 file changed, 8 insertions(+)drivers/char/tpm/tpm_tis.c +1
-0@@ -114,6 +114,14 @@ static int tpm_tis_disable_irq(const struct dmi_system_id *d) }static const struct dmi_system_id tpm_tis_dmi_table[] = { + {
drivers/char/tpm/tpm_tis.c static int tpm_tis_disable_irq(const struct dmi_system_id *d) · line 127
Quoted 10 lines
drivers/char/tpm/tpm_tis.c +7
-0+ .callback = tpm_tis_disable_irq, + .ident = "Framework Laptop Intel 12th gen", + .matches = { +DMI_MATCH (DMI_SYS_VENDOR ,"Framework" ), + DMI_MATCH(DMI_PRODUCT_VERSION, "A4"), + }, + }, { .callback = tpm_tis_disable_irq, .ident = "ThinkPad T490s",
Hi, no other issues except fixes tag should be in both:
I.e.
# $1 = commit ID
function git-fixes {
git --no-pager log --format='Fixes: %h ("%s")' --abbrev=12 -1 $1;
}
I'll update them, thank you.
BR, Jarkko