Skip to contentSkip to conversation

TDX host: kexec() support

PATCH Superseded Kai Huang 4 participants 22 messages Apr 7, 2024 – Apr 15, 2024

Subsystems (2)

v3 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Kai Huang

author
#0 lore raw

v3 cover TDX host: kexec() support

Currently kexec() support and TDX host are muturally exclusive in the Kconfig. This series adds the TDX host kexec support so that they can work together and can be enabled at the same time in the Kconfig.

v2 -> v3:
- Change to only do WBINVD for bare-metal, as Kirill/Tom pointed out WBINVD in TDX guests and SEV-ES/SEV-SNP guests triggers #VE.

v2: https://lore.kernel.org/linux-kernel/[email protected]/

v1 -> v2:
- Do unconditional WBINVD during kexec() -- Boris
- Change to cover crash kexec() -- Rick
- Add a new patch (last one) to add a mechanism to reset all TDX private pages due to having to cover crash kexec().
- Other code improvements -- Dave
- Rebase to latest tip/master.

v1: https://lore.kernel.org/linux-kernel/[email protected]/

Hi Dave, Sean, Paolo,

The last patch provides a new mechanism to handle all other TDX private pages when they become possible to exist, e.g., when KVM is ready to run TDX guests. It's not mandatory at this stage because currently we only have PAMT as private pages, but if we agree it's the right way to do then it can be applied together with rest patches too.

Hi Tom, Ashish,

This series touches AMD SME code too, and I don't have AMD machine to test. I appreciate if you can help to review and/or test.

Kai Huang (5):
x86/kexec: do unconditional WBINVD for bare-metal in stop_this_cpu() x86/kexec: do unconditional WBINVD for bare-metal in relocate_kernel() x86/kexec: Reset TDX private memory on platforms with TDX erratum x86/virt/tdx: Remove the !KEXEC_CORE dependency x86/virt/tdx: Add TDX memory reset notifier to reset other private
pages

 arch/x86/Kconfig                     |   1 -
 arch/x86/include/asm/kexec.h         |   2 +-
 arch/x86/include/asm/tdx.h           |  16 +++++
 arch/x86/kernel/machine_kexec_64.c   |  29 ++++++--
 arch/x86/kernel/process.c            |  18 +++--
 arch/x86/kernel/relocate_kernel_64.S |  14 ++--
 arch/x86/virt/vmx/tdx/tdx.c          | 100 +++++++++++++++++++++++++++
 7 files changed, 158 insertions(+), 22 deletions(-)

base-commit: 6569784d306a8d356bb2517c1528c1d611a51922

Signature · 1 line
2.43.2
author
#1 lore raw

PATCH v3 1/5 x86/kexec: do unconditional WBINVD for bare-metal in stop_this_cpu()

TL;DR:

Change to do unconditional WBINVD in stop_this_cpu() for bare metal to cover kexec support for both AMD SME and Intel TDX, despite there _was_ some issue preventing from doing so but now has it got fixed.

Long version:

Both SME and TDX can leave caches in incoherent state due to memory encryption. During kexec, the caches must be flushed before jumping to the second kernel to avoid silent memory corruption to the second kernel.

Currently, for SME the kernel only does WBINVD in stop_this_cpu() when the kernel determines the hardware supports SME. To support TDX, one option is to extend that specific check to cover both SME and TDX.

However, instead of sprinkling around vendor-specific checks, it's better to just do unconditional WBINVD. Kexec() is a slow path, and it is acceptable to have an additional WBINVD in order to have simple and easy to maintain code.

But only do WBINVD for bare-metal because TDX guests and SEV-ES/SEV-SNP guests will get unexpected (and yet unnecessary) #VE and may not be able to handle (e.g., TDX guest panics when it gets #VE due to WBINVD).

Note:

Historically, there _was_ an issue preventing doing unconditional WBINVD but that has been fixed.

When SME kexec() support was initially added in commit

bba4ed011a52: ("x86/mm, kexec: Allow kexec to be used with SME")

WBINVD was done unconditionally. However since then some issues were reported that different Intel systems would hang or reset due to that commit.

To try to fix, a later commit

f23d74f6c66c: ("x86/mm: Rework wbinvd, hlt operation in stop_this_cpu()")

then changed to only do WBINVD when hardware supports SME.

While this commit made the reported issues go away, it didn't pinpoint the root cause. Also, it didn't handle a corner case[*] correctly, which resulted in the reveal of the root cause and the final fix by commit

1f5e7eb7868e: ("x86/smp: Make stop_other_cpus() more robust")

See [1][2] for more information.

Further testing of doing unconditional WBINVD based on the above fix on the problematic machines (that issues were originally reported) confirmed the issues couldn't be reproduced.

See [3][4] for more information.

Therefore, it is safe to do unconditional WBINVD now.

[*] The commit didn't check whether the CPUID leaf is available or not. Making unsupported CPUID leaf on Intel returns garbage resulting in unintended WBINVD which caused some issue (followed by the analysis and the reveal of the final root cause). The corner case was independently fixed by commit

9b040453d444: ("x86/smp: Dont access non-existing CPUID leaf")

[1]: https://lore.kernel.org/lkml/CALu+AoQKmeixJdkO07t7BtttN7v3RM4_aBKi642bQ3fTBbSAVg@mail.gmail.com/T/#m300f3f9790850b5daa20a71abcc200ae8d94a12a
[2]: https://lore.kernel.org/lkml/CALu+AoQKmeixJdkO07t7BtttN7v3RM4_aBKi642bQ3fTBbSAVg@mail.gmail.com/T/#ma7263a7765483db0dabdeef62a1110940e634846
[3]: https://lore.kernel.org/lkml/CALu+AoQKmeixJdkO07t7BtttN7v3RM4_aBKi642bQ3fTBbSAVg@mail.gmail.com/T/#mc043191f2ff860d649c8466775dc61ac1e0ae320
[4]: https://lore.kernel.org/lkml/CALu+AoQKmeixJdkO07t7BtttN7v3RM4_aBKi642bQ3fTBbSAVg@mail.gmail.com/T/#md23f1a8f6afcc59fa2b0ac1967f18e418e24347c

---

v2 -> v3:
- Change to only do WBINVD for bare metal

---
 arch/x86/kernel/process.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)
arch/x86/kernel/process.c +8 -10
index b8441147eb5e..5ba8a9c1e47a 100644
@@ -813,18 +813,16 @@ void __noreturn stop_this_cpu(void *dummy)
 	mcheck_cpu_clear(c);
 
 	/*
-	 * Use wbinvd on processors that support SME. This provides support
-	 * for performing a successful kexec when going from SME inactive
-	 * to SME active (or vice-versa). The cache must be cleared so that
-	 * if there are entries with the same physical address, both with and
-	 * without the encryption bit, they don't race each other when flushed
-	 * and potentially end up with the wrong entry being committed to
-	 * memory.
+	 * The kernel could leave caches in incoherent state on SME/TDX
+	 * capable platforms.  Flush cache to avoid silent memory
+	 * corruption for these platforms.
 	 *
-	 * Test the CPUID bit directly because the machine might've cleared
-	 * X86_FEATURE_SME due to cmdline options.
+	 * stop_this_cpu() is not a fast path, just do unconditional
+	 * WBINVD for simplicity.  But only do WBINVD for bare-metal
+	 * as TDX guests and SEV-ES/SEV-SNP guests will get unexpected
+	 * (and unnecessary) #VE and may unable to handle.
 	 */
-	if (c->extended_cpuid_level >= 0x8000001f && (cpuid_eax(0x8000001f) & BIT(0)))
+	if (!boot_cpu_has(X86_FEATURE_HYPERVISOR))
 		native_wbinvd();
 
 	/*
Signature · 1 line
2.43.2
#2 lore raw

Re: [PATCH v3 1/5] x86/kexec: do unconditional WBINVD for bare-metal in stop_this_cpu()

Quoted 1 line · Kai Huang

TL;DR:

The commit message is waaay too verbose for no good reason. You don't really need to repeat all the history around this code.

Quoted 19 lines
---
 arch/x86/kernel/process.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)
arch/x86/kernel/process.c +0 -7
index b8441147eb5e..5ba8a9c1e47a 100644
@@ -813,18 +813,16 @@ void __noreturn stop_this_cpu(void *dummy)
 	mcheck_cpu_clear(c);
 
 	/*
-	 * Use wbinvd on processors that support SME. This provides support
-	 * for performing a successful kexec when going from SME inactive
-	 * to SME active (or vice-versa). The cache must be cleared so that
-	 * if there are entries with the same physical address, both with and
-	 * without the encryption bit, they don't race each other when flushed
-	 * and potentially end up with the wrong entry being committed to
-	 * memory.

arch/x86/kernel/process.c void __noreturn stop_this_cpu(void *dummy) · line 823

Quoted 10 lines
arch/x86/kernel/process.c +7 -2
+	 * The kernel could leave caches in incoherent state on SME/TDX
+	 * capable platforms.  Flush cache to avoid silent memory
+	 * corruption for these platforms.
 	 *
-	 * Test the CPUID bit directly because the machine might've cleared
-	 * X86_FEATURE_SME due to cmdline options.
+	 * stop_this_cpu() is not a fast path, just do unconditional
+	 * WBINVD for simplicity.  But only do WBINVD for bare-metal
+	 * as TDX guests and SEV-ES/SEV-SNP guests will get unexpected
+	 * (and unnecessary) #VE and may unable to handle.

s/#VE/exception/

On SEV it is #VC, not #VE.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
author
#3 lore raw

Re: [PATCH v3 1/5] x86/kexec: do unconditional WBINVD for bare-metal in stop_this_cpu()

Quoted 5 lines · Kirill A. Shutemov
Quoted 1 line · Kai Huang

TL;DR:

The commit message is waaay too verbose for no good reason. You don't really need to repeat all the history around this code.

Could you be more specific?

I was following Boris's suggestion to summerize all the discussion around the "unconditional WBINVD" issue.

https://lore.kernel.org/linux-kernel/20240228110207.GCZd8Sr8mXHA2KTiLz@fat_crate.local/

I can try to improve if I can know specifically what should be trimmed down.

Quoted 33 lines
Quoted 20 lines
---
  arch/x86/kernel/process.c | 18 ++++++++----------
  1 file changed, 8 insertions(+), 10 deletions(-)
arch/x86/kernel/process.c +0 -7
index b8441147eb5e..5ba8a9c1e47a 100644
@@ -813,18 +813,16 @@ void __noreturn stop_this_cpu(void *dummy)
  	mcheck_cpu_clear(c);
 
  	/*
-	 * Use wbinvd on processors that support SME. This provides support
-	 * for performing a successful kexec when going from SME inactive
-	 * to SME active (or vice-versa). The cache must be cleared so that
-	 * if there are entries with the same physical address, both with and
-	 * without the encryption bit, they don't race each other when flushed
-	 * and potentially end up with the wrong entry being committed to
-	 * memory.

arch/x86/kernel/process.c void __noreturn stop_this_cpu(void *dummy) · line 823

Quoted 10 lines
arch/x86/kernel/process.c +7 -2
+	 * The kernel could leave caches in incoherent state on SME/TDX
+	 * capable platforms.  Flush cache to avoid silent memory
+	 * corruption for these platforms.
  	 *
-	 * Test the CPUID bit directly because the machine might've cleared
-	 * X86_FEATURE_SME due to cmdline options.
+	 * stop_this_cpu() is not a fast path, just do unconditional
+	 * WBINVD for simplicity.  But only do WBINVD for bare-metal
+	 * as TDX guests and SEV-ES/SEV-SNP guests will get unexpected
+	 * (and unnecessary) #VE and may unable to handle.

s/#VE/exception/

Quoted 2 lines

On SEV it is #VC, not #VE.

Thanks. I think I'll use "exception (#VE or #VC)" which is clearer, as Tom typed in the comments to patch 2.

#4 lore raw

Re: [PATCH v3 1/5] x86/kexec: do unconditional WBINVD for bare-metal in stop_this_cpu()

Quoted 9 lines · Kai
Quoted 5 lines · Kirill A. Shutemov
Quoted 1 line · Kai Huang

TL;DR:

The commit message is waaay too verbose for no good reason. You don't really need to repeat all the history around this code.

Quoted 8 lines

Could you be more specific?

I was following Boris's suggestion to summerize all the discussion around the "unconditional WBINVD" issue.

https://lore.kernel.org/linux-kernel/20240228110207.GCZd8Sr8mXHA2KTiLz@fat_crate.local/

I can try to improve if I can know specifically what should be trimmed down.

What about something like this:

x86/mm: Do unconditional WBINVD in stop_this_cpu() for bare metal

Both AMD SME and Intel TDX can leave caches in an incoherent state due to memory encryption, which can lead to silent memory corruption during kexec. To address this issue, it is necessary to flush the caches before jumping to the second kernel.

Previously, the kernel only performed WBINVD in stop_this_cpu() when SME support was detected. To support TDX as well, instead of adding vendor-specific checks, it is proposed to unconditionally perform WBINVD. Kexec() is a slow path, and the additional WBINVD is acceptable for the sake of simplicity and maintainability.

It is important to note that WBINVD should only be done for bare-metal scenarios, as TDX guests and SEV-ES/SEV-SNP guests may not handle unexpected exceptions (#VE or #VC) caused by WBINVD.

Historically, there were issues with unconditional WBINVD, leading to system hangs or resets on different Intel systems. These issues were addressed by a series of commits, culminating in the fix provided by commit 1f5e7eb7868e ("x86/smp: Make stop_other_cpus() more robust").

Further testing on problematic machines confirmed that the issues could not be reproduced after applying the fix. Therefore, it is now safe to unconditionally perform WBINVD in stop_this_cpu().

You can also add links to relevant threads as Link: tags.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
author
#5 lore raw

Re: [PATCH v3 1/5] x86/kexec: do unconditional WBINVD for bare-metal in stop_this_cpu()

Quoted 43 lines · Kirill A. Shutemov
Quoted 9 lines · Kai
Quoted 5 lines · Kirill A. Shutemov
Quoted 1 line · Kai Huang

TL;DR:

The commit message is waaay too verbose for no good reason. You don't really need to repeat all the history around this code.

Quoted 8 lines

Could you be more specific?

I was following Boris's suggestion to summerize all the discussion around the "unconditional WBINVD" issue.

https://lore.kernel.org/linux-kernel/20240228110207.GCZd8Sr8mXHA2KTiLz@fat_crate.local/

I can try to improve if I can know specifically what should be trimmed down.

What about something like this:

x86/mm: Do unconditional WBINVD in stop_this_cpu() for bare metal

Both AMD SME and Intel TDX can leave caches in an incoherent state due to memory encryption, which can lead to silent memory corruption during kexec. To address this issue, it is necessary to flush the caches before jumping to the second kernel.

Previously, the kernel only performed WBINVD in stop_this_cpu() when SME support was detected. To support TDX as well, instead of adding vendor-specific checks, it is proposed to unconditionally perform WBINVD. Kexec() is a slow path, and the additional WBINVD is acceptable for the sake of simplicity and maintainability.

It is important to note that WBINVD should only be done for bare-metal scenarios, as TDX guests and SEV-ES/SEV-SNP guests may not handle unexpected exceptions (#VE or #VC) caused by WBINVD.

Historically, there were issues with unconditional WBINVD, leading to system hangs or resets on different Intel systems. These issues were addressed by a series of commits, culminating in the fix provided by commit 1f5e7eb7868e ("x86/smp: Make stop_other_cpus() more robust").

Quoted 6 lines

Further testing on problematic machines confirmed that the issues could not be reproduced after applying the fix. Therefore, it is now safe to unconditionally perform WBINVD in stop_this_cpu().

You can also add links to relevant threads as Link: tags.

Hmm.. The last two paragraphs doesn't tell the background that the "unconditional WBINVD" was the original way to do etc. The changelog of commit 1f5e7eb7868e ("x86/smp: Make stop_other_cpus() more robust" (and the commit IDs that it mentions) doesn't tell the full story either.

That means people will need to open all the Links to get the full information. I think it is against what Boris suggested.

Yeah I agree having a lengthy changelog is annoying sometimes, but for this particular case we have a "TL;DR" so doesn't seem that bad to me. :-)

So for now I would like to keep the text after the "Note:" in my original changelog, but I will use your first 3 paragraphs above to replace mine.

maintainer
#6 lore raw

Re: [PATCH v3 1/5] x86/kexec: do unconditional WBINVD for bare-metal in stop_this_cpu()

In reply to Huang, Kai
Quoted 8 lines · Kai

Could you be more specific?

I was following Boris's suggestion to summerize all the discussion around the "unconditional WBINVD" issue.

https://lore.kernel.org/linux-kernel/20240228110207.GCZd8Sr8mXHA2KTiLz@fat_crate.local/

I can try to improve if I can know specifically what should be trimmed down.

No, keep it this way. I've yet to see someone complaining from too verbose commit message while doing git archeology.

If it is too verbose to a reader, then that reader can jump over the paragraphs.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author
#7 lore raw

Re: [PATCH v3 1/5] x86/kexec: do unconditional WBINVD for bare-metal in stop_this_cpu()

Quoted 8 lines · Borislav Petkov
Quoted 7 lines · Kai

Could you be more specific?

I was following Boris's suggestion to summerize all the discussion around the "unconditional WBINVD" issue.

https://lore.kernel.org/linux-kernel/20240228110207.GCZd8Sr8mXHA2KTiLz@fat_crate.local/

Quoted 8 lines
Quoted 1 line

I can try to improve if I can know specifically what should be trimmed down.

No, keep it this way. I've yet to see someone complaining from too verbose commit message while doing git archeology.

If it is too verbose to a reader, then that reader can jump over the paragraphs.

Yeah as replied to Kirill I am keeping it. Thanks for the feedback.

#8 lore raw

Re: [PATCH v3 1/5] x86/kexec: do unconditional WBINVD for bare-metal in stop_this_cpu()

In reply to Kai Huang
Quoted 15 lines · Kai Huang
arch/x86/kernel/process.c +0 -7
index b8441147eb5e..5ba8a9c1e47a 100644
@@ -813,18 +813,16 @@ void __noreturn stop_this_cpu(void *dummy)
  	mcheck_cpu_clear(c);
 
  	/*
-	 * Use wbinvd on processors that support SME. This provides support
-	 * for performing a successful kexec when going from SME inactive
-	 * to SME active (or vice-versa). The cache must be cleared so that
-	 * if there are entries with the same physical address, both with and
-	 * without the encryption bit, they don't race each other when flushed
-	 * and potentially end up with the wrong entry being committed to
-	 * memory.

arch/x86/kernel/process.c void __noreturn stop_this_cpu(void *dummy) · line 823

Quoted 10 lines
arch/x86/kernel/process.c +7 -2
+	 * The kernel could leave caches in incoherent state on SME/TDX
+	 * capable platforms.  Flush cache to avoid silent memory
+	 * corruption for these platforms.
  	 *
-	 * Test the CPUID bit directly because the machine might've cleared
-	 * X86_FEATURE_SME due to cmdline options.
+	 * stop_this_cpu() is not a fast path, just do unconditional
+	 * WBINVD for simplicity.  But only do WBINVD for bare-metal
+	 * as TDX guests and SEV-ES/SEV-SNP guests will get unexpected
+	 * (and unnecessary) #VE and may unable to handle.

In addition to Kirill's comment on #VE...

This last part of the comment reads a bit odd since you say unconditional and then say only do WBINVD for bare-metal. Maybe something like this makes it a bit clearer?:

For TDX and SEV-ES/SEV-SNP guests, a WBINVD may cause an exception (#VE or #VC). However, all exception handling has been torn down at this point, so this would cause the guest to crash. Since memory within these types of guests is coherent only issue the WBINVD on bare-metal.

And you can expand the comment block out to at least 80 characters to make it more compact.

Thanks,
Tom

Quoted 6 lines
arch/x86/kernel/process.c +1 -1
  	 */
-	if (c->extended_cpuid_level >= 0x8000001f && (cpuid_eax(0x8000001f) & BIT(0)))
+	if (!boot_cpu_has(X86_FEATURE_HYPERVISOR))
  		native_wbinvd();
 
  	/*
#9 lore raw

Re: [PATCH v3 1/5] x86/kexec: do unconditional WBINVD for bare-metal in stop_this_cpu()

Quoted 31 lines · Tom Lendacky
Quoted 18 lines · Kai Huang
arch/x86/kernel/process.c +0 -4
index b8441147eb5e..5ba8a9c1e47a 100644
@@ -813,18 +813,16 @@ void __noreturn stop_this_cpu(void *dummy)
       mcheck_cpu_clear(c);
       /*
-     * Use wbinvd on processors that support SME. This provides support
-     * for performing a successful kexec when going from SME inactive
-     * to SME active (or vice-versa). The cache must be cleared so that
-     * if there are entries with the same physical address, both with 

and
-     * without the encryption bit, they don't race each other when flushed
-     * and potentially end up with the wrong entry being committed to -     * memory.
+     * The kernel could leave caches in incoherent state on SME/TDX +     * capable platforms.  Flush cache to avoid silent memory

Quoted 8 lines

+     * corruption for these platforms.
       *
-     * Test the CPUID bit directly because the machine might've cleared -     * X86_FEATURE_SME due to cmdline options.
+     * stop_this_cpu() is not a fast path, just do unconditional +     * WBINVD for simplicity.  But only do WBINVD for bare-metal +     * as TDX guests and SEV-ES/SEV-SNP guests will get unexpected +     * (and unnecessary) #VE and may unable to handle.

In addition to Kirill's comment on #VE...

Quoted 3 lines

This last part of the comment reads a bit odd since you say unconditional and then say only do WBINVD for bare-metal. Maybe something like this makes it a bit clearer?:

For TDX and SEV-ES/SEV-SNP guests, a WBINVD may cause an exception (#VE or #VC). However, all exception handling has been torn down at this point, so this would cause the guest to crash. Since memory within these types of guests is coherent only issue the WBINVD on bare-metal.

Hmmm... actually it was the other WBINVD in patch #2 that caused the crash, so what I wrote above isn't accurate. You might want to re-word as appropriate.

Thanks,
Tom

Quoted 12 lines

And you can expand the comment block out to at least 80 characters to make it more compact.

Thanks,
Tom

Quoted 6 lines

       */
-    if (c->extended_cpuid_level >= 0x8000001f &&
(cpuid_eax(0x8000001f) & BIT(0)))
+    if (!boot_cpu_has(X86_FEATURE_HYPERVISOR))
          native_wbinvd();
      /*

author
#10 lore raw

Re: [PATCH v3 1/5] x86/kexec: do unconditional WBINVD for bare-metal in stop_this_cpu()

Quoted 32 lines · Tom Lendacky
Quoted 31 lines · Tom Lendacky
Quoted 18 lines · Kai Huang
arch/x86/kernel/process.c +0 -4
index b8441147eb5e..5ba8a9c1e47a 100644
@@ -813,18 +813,16 @@ void __noreturn stop_this_cpu(void *dummy)
       mcheck_cpu_clear(c);
       /*
-     * Use wbinvd on processors that support SME. This provides support
-     * for performing a successful kexec when going from SME inactive
-     * to SME active (or vice-versa). The cache must be cleared so that
-     * if there are entries with the same physical address, both 

with and
-     * without the encryption bit, they don't race each other when flushed
-     * and potentially end up with the wrong entry being committed to -     * memory.
+     * The kernel could leave caches in incoherent state on SME/TDX +     * capable platforms.  Flush cache to avoid silent memory

Quoted 8 lines

+     * corruption for these platforms.
       *
-     * Test the CPUID bit directly because the machine might've cleared -     * X86_FEATURE_SME due to cmdline options.
+     * stop_this_cpu() is not a fast path, just do unconditional +     * WBINVD for simplicity.  But only do WBINVD for bare-metal +     * as TDX guests and SEV-ES/SEV-SNP guests will get unexpected +     * (and unnecessary) #VE and may unable to handle.

In addition to Kirill's comment on #VE...

Quoted 5 lines
Quoted 3 lines

This last part of the comment reads a bit odd since you say unconditional and then say only do WBINVD for bare-metal. Maybe something like this makes it a bit clearer?:

For TDX and SEV-ES/SEV-SNP guests, a WBINVD may cause an exception (#VE or #VC). However, all exception handling has been torn down at this point, so this would cause the guest to crash. Since memory within these types of guests is coherent only issue the WBINVD on bare-metal.

Hmmm... actually it was the other WBINVD in patch #2 that caused the crash, so what I wrote above isn't accurate. You might want to re-word as appropriate.

Yeah that's why I used "may unable to handle" in the comment, as I thought there's no need to be that specific?

I tend not to mention "memory within these types of guests is coherent".
I mean the current upstream kernel _ONLY_ does WBINVD for SME, that
means for all non-SME environment there's no concern to do WBINVD here.

Here we only extend to do WBINVD on more environments, so as long as there's no harm to do WBINVD for them it's OK.

How about below?

/*
* The kernel could leave caches in incoherent state on SME/TDX
* capable platforms. Flush cache to avoid silent memory
* corruption for these platforms.
*
* For TDX and SEV-ES/SEV-SNP guests, a WBINVD causes an
* exception (#VE or #VC), and the kernel may not be able
* to handle such exception (e.g., TDX guest panics if it
* sees #VE). Since stop_this_cpu() isn't a fast path, just
* issue the WBINVD on bare-metal instead of sprinkling
* around vendor-specific checks.
*/

Quoted 6 lines

Thanks,
Tom

Quoted 2 lines

And you can expand the comment block out to at least 80 characters to make it more compact.

OK I can do. I guess I have to change my vim setting to do so, though :-)

#11 lore raw

Re: [PATCH v3 1/5] x86/kexec: do unconditional WBINVD for bare-metal in stop_this_cpu()

Quoted 36 lines · Kai
Quoted 35 lines · Tom Lendacky
Quoted 34 lines · Tom Lendacky
Quoted 21 lines · Kai Huang
arch/x86/kernel/process.c +0 -1
index b8441147eb5e..5ba8a9c1e47a 100644
@@ -813,18 +813,16 @@ void __noreturn stop_this_cpu(void *dummy)
       mcheck_cpu_clear(c);
       /*
-     * Use wbinvd on processors that support SME. This provides 

support
-     * for performing a successful kexec when going from SME inactive -     * to SME active (or vice-versa). The cache must be cleared so that
-     * if there are entries with the same physical address, both with and
-     * without the encryption bit, they don't race each other when flushed
-     * and potentially end up with the wrong entry being committed to -     * memory.
+     * The kernel could leave caches in incoherent state on SME/TDX +     * capable platforms.  Flush cache to avoid silent memory +     * corruption for these platforms.

Quoted 8 lines

       *
-     * Test the CPUID bit directly because the machine might've cleared
-     * X86_FEATURE_SME due to cmdline options.
+     * stop_this_cpu() is not a fast path, just do unconditional +     * WBINVD for simplicity.  But only do WBINVD for bare-metal +     * as TDX guests and SEV-ES/SEV-SNP guests will get unexpected +     * (and unnecessary) #VE and may unable to handle.

In addition to Kirill's comment on #VE...

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

This last part of the comment reads a bit odd since you say unconditional and then say only do WBINVD for bare-metal. Maybe something like this makes it a bit clearer?:

For TDX and SEV-ES/SEV-SNP guests, a WBINVD may cause an exception (#VE or #VC). However, all exception handling has been torn down at this point, so this would cause the guest to crash. Since memory within these types of guests is coherent only issue the WBINVD on bare-metal.

Hmmm... actually it was the other WBINVD in patch #2 that caused the crash, so what I wrote above isn't accurate. You might want to re-word as appropriate.

Yeah that's why I used "may unable to handle" in the comment, as I thought there's no need to be that specific?

Yes, makes sense.

Quoted 9 lines

I tend not to mention "memory within these types of guests is coherent".
 I mean the current upstream kernel _ONLY_ does WBINVD for SME, that
means for all non-SME environment there's no concern to do WBINVD here.

Here we only extend to do WBINVD on more environments, so as long as there's no harm to do WBINVD for them it's OK.

How about below?

Quoted 12 lines

    /*
     * The kernel could leave caches in incoherent state on SME/TDX
     * capable platforms.  Flush cache to avoid silent memory
     * corruption for these platforms.
     *
     * For TDX and SEV-ES/SEV-SNP guests, a WBINVD causes an
     * exception (#VE or #VC), and the kernel may not be able
     * to handle such exception (e.g., TDX guest panics if it
     * sees #VE).  Since stop_this_cpu() isn't a fast path, just
     * issue the WBINVD on bare-metal instead of sprinkling
     * around vendor-specific checks.
     */

I think that's ok, but maybe just adding that the WBINVD is not necessary for TDX and SEV-ES/SEV-SNP would make it clearer. Just my opinion, though.

Thanks,
Tom

Quoted 8 lines
Quoted 6 lines

Thanks,
Tom

Quoted 2 lines

And you can expand the comment block out to at least 80 characters to make it more compact.

OK I can do.  I guess I have to change my vim setting to do so, though :-)

author
#12 lore raw

Re: [PATCH v3 1/5] x86/kexec: do unconditional WBINVD for bare-metal in stop_this_cpu()

Quoted 79 lines · Tom Lendacky
Quoted 46 lines · Kai
Quoted 45 lines · Tom Lendacky
Quoted 43 lines · Tom Lendacky
Quoted 21 lines · Kai Huang
arch/x86/kernel/process.c +0 -1
index b8441147eb5e..5ba8a9c1e47a 100644
@@ -813,18 +813,16 @@ void __noreturn stop_this_cpu(void *dummy)
       mcheck_cpu_clear(c);
       /*
-     * Use wbinvd on processors that support SME. This provides

support
-     * for performing a successful kexec when going from SME inactive -     * to SME active (or vice-versa). The cache must be cleared so that
-     * if there are entries with the same physical address, both with and
-     * without the encryption bit, they don't race each other when flushed
-     * and potentially end up with the wrong entry being committed to -     * memory.
+     * The kernel could leave caches in incoherent state on SME/TDX +     * capable platforms.  Flush cache to avoid silent memory +     * corruption for these platforms.

Quoted 8 lines

       *
-     * Test the CPUID bit directly because the machine might've cleared
-     * X86_FEATURE_SME due to cmdline options.
+     * stop_this_cpu() is not a fast path, just do unconditional +     * WBINVD for simplicity.  But only do WBINVD for bare-metal +     * as TDX guests and SEV-ES/SEV-SNP guests will get unexpected +     * (and unnecessary) #VE and may unable to handle.

In addition to Kirill's comment on #VE...

This last part of the comment reads a bit odd since you say unconditional and then say only do WBINVD for bare-metal. Maybe something like this makes it a bit clearer?:

For TDX and SEV-ES/SEV-SNP guests, a WBINVD may cause an exception (#VE or #VC). However, all exception handling has been torn down at this point, so this would cause the guest to crash. Since memory within these types of guests is coherent only issue the WBINVD on bare-metal.

Quoted 6 lines
Quoted 3 lines

Hmmm... actually it was the other WBINVD in patch #2 that caused the crash, so what I wrote above isn't accurate. You might want to re-word as appropriate.

Yeah that's why I used "may unable to handle" in the comment, as I thought there's no need to be that specific?

Yes, makes sense.

Quoted 10 lines

I tend not to mention "memory within these types of guests is coherent".
 I mean the current upstream kernel _ONLY_ does WBINVD for SME, that
means for all non-SME environment there's no concern to do WBINVD here.

Here we only extend to do WBINVD on more environments, so as long as there's no harm to do WBINVD for them it's OK.

How about below?

Quoted 12 lines

    /*
     * The kernel could leave caches in incoherent state on SME/TDX
     * capable platforms.  Flush cache to avoid silent memory
     * corruption for these platforms.
     *
     * For TDX and SEV-ES/SEV-SNP guests, a WBINVD causes an
     * exception (#VE or #VC), and the kernel may not be able
     * to handle such exception (e.g., TDX guest panics if it
     * sees #VE).  Since stop_this_cpu() isn't a fast path, just
     * issue the WBINVD on bare-metal instead of sprinkling
     * around vendor-specific checks.
     */

Quoted 4 lines

I think that's ok, but maybe just adding that the WBINVD is not necessary for TDX and SEV-ES/SEV-SNP would make it clearer. Just my opinion, though.

Yeah can do that.

Will add "WBINVD is not necessary for TDX and SEV-ES/SEV-SNP guests" before starting the "Since stop_this_cpu() ...".

Thanks for the feedback.

author
#13 lore raw

PATCH v3 2/5 x86/kexec: do unconditional WBINVD for bare-metal in relocate_kernel()

In reply to Kai Huang

Both SME and TDX can leave caches in incoherent state due to memory encryption. During kexec, the caches must be flushed before jumping to the second kernel to avoid silent memory corruption to the second kernel.

During kexec, the WBINVD in stop_this_cpu() flushes caches for all remote cpus when they are being stopped. For SME, the WBINVD in relocate_kernel() flushes the cache for the last running cpu (which is executing the kexec).

Similarly, to support kexec for TDX host, after stopping all remote cpus with cache flushed, the kernel needs to flush cache for the last running cpu.

Use the existing WBINVD in relocate_kernel() to cover TDX host as well.

However, instead of sprinkling around vendor-specific checks, just do unconditional WBINVD to cover both SME and TDX. Kexec is not a fast path so having one additional WBINVD for platforms w/o SME/TDX is acceptable.

But only do WBINVD for bare-metal because TDX guests and SEV-ES/SEV-SNP guests will get unexpected (and yet unnecessary) #VE which the kernel is unable to handle at this stage.

Cc Tom Lendacky <[email protected]>
Cc Dave Young <[email protected]>

---

v2 -> v3:
- Change to only do WBINVD for bare metal

---
 arch/x86/include/asm/kexec.h         |  2 +-
 arch/x86/kernel/machine_kexec_64.c   |  2 +-
 arch/x86/kernel/relocate_kernel_64.S | 14 +++++++++-----
 3 files changed, 11 insertions(+), 7 deletions(-)
arch/x86/include/asm/kexec.h +1 -1
index 91ca9a9ee3a2..455f8a6c66a9 100644
@@ -128,7 +128,7 @@ relocate_kernel(unsigned long indirection_page,
 		unsigned long page_list,
 		unsigned long start_address,
 		unsigned int preserve_context,
-		unsigned int host_mem_enc_active);
+		unsigned int bare_metal);
 #endif
 
 #define ARCH_HAS_KIMAGE_ARCH
arch/x86/kernel/machine_kexec_64.c +1 -1
index b180d8e497c3..a454477b7b4c 100644
@@ -358,7 +358,7 @@ void machine_kexec(struct kimage *image)
 				       (unsigned long)page_list,
 				       image->start,
 				       image->preserve_context,
-				       cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT));
+				       !boot_cpu_has(X86_FEATURE_HYPERVISOR));
 
 #ifdef CONFIG_KEXEC_JUMP
 	if (image->preserve_context)
arch/x86/kernel/relocate_kernel_64.S +9 -5
index 56cab1bb25f5..3e04c5e5687f 100644
@@ -50,7 +50,7 @@ SYM_CODE_START_NOALIGN(relocate_kernel)
 	 * %rsi page_list
 	 * %rdx start address
 	 * %rcx preserve_context
-	 * %r8  host_mem_enc_active
+	 * %r8  bare_metal
 	 */
 
 	/* Save the CPU context, used for jumping back */
@@ -78,7 +78,7 @@ SYM_CODE_START_NOALIGN(relocate_kernel)
 	pushq $0
 	popfq
 
-	/* Save SME active flag */
+	/* Save the bare_metal */
 	movq	%r8, %r12
 
 	/*
@@ -160,9 +160,13 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)
 	movq	%r9, %cr3
 
 	/*
-	 * If SME is active, there could be old encrypted cache line
-	 * entries that will conflict with the now unencrypted memory
-	 * used by kexec. Flush the caches before copying the kernel.
+	 * The kernel could leave caches in incoherent state on SME/TDX
+	 * capable platforms.  Just do unconditional WBINVD to avoid
+	 * silent memory corruption to the new kernel for these platforms.
+	 *
+	 * But only do WBINVD for bare-metal because TDX guests and
+	 * SEV-ES/SEV-SNP guests will get #VE which the kernel is unable
+	 * to handle at this stage.
 	 */
 	testq	%r12, %r12
 	jz 1f
Signature · 1 line
2.43.2
Reviewed-by on v3 2/5
#14 lore raw

Re: [PATCH v3 2/5] x86/kexec: do unconditional WBINVD for bare-metal in relocate_kernel()

Quoted 19 lines · Kai Huang

Both SME and TDX can leave caches in incoherent state due to memory encryption. During kexec, the caches must be flushed before jumping to the second kernel to avoid silent memory corruption to the second kernel.

During kexec, the WBINVD in stop_this_cpu() flushes caches for all remote cpus when they are being stopped. For SME, the WBINVD in relocate_kernel() flushes the cache for the last running cpu (which is executing the kexec).

Similarly, to support kexec for TDX host, after stopping all remote cpus with cache flushed, the kernel needs to flush cache for the last running cpu.

Use the existing WBINVD in relocate_kernel() to cover TDX host as well.

However, instead of sprinkling around vendor-specific checks, just do unconditional WBINVD to cover both SME and TDX. Kexec is not a fast path so having one additional WBINVD for platforms w/o SME/TDX is acceptable.

Quoted 7 lines

But only do WBINVD for bare-metal because TDX guests and SEV-ES/SEV-SNP guests will get unexpected (and yet unnecessary) #VE which the kernel is unable to handle at this stage.

Cc Tom Lendacky <[email protected]>
Cc Dave Young <[email protected]>
Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#15 lore raw

Re: [PATCH v3 2/5] x86/kexec: do unconditional WBINVD for bare-metal in relocate_kernel()

In reply to Kai Huang
Quoted 13 lines · Kai Huang

Both SME and TDX can leave caches in incoherent state due to memory encryption. During kexec, the caches must be flushed before jumping to the second kernel to avoid silent memory corruption to the second kernel.

During kexec, the WBINVD in stop_this_cpu() flushes caches for all remote cpus when they are being stopped. For SME, the WBINVD in relocate_kernel() flushes the cache for the last running cpu (which is executing the kexec).

Similarly, to support kexec for TDX host, after stopping all remote cpus with cache flushed, the kernel needs to flush cache for the last running cpu.

Quoted 8 lines

Use the existing WBINVD in relocate_kernel() to cover TDX host as well.

However, instead of sprinkling around vendor-specific checks, just do unconditional WBINVD to cover both SME and TDX. Kexec is not a fast path so having one additional WBINVD for platforms w/o SME/TDX is acceptable.

But only do WBINVD for bare-metal because TDX guests and SEV-ES/SEV-SNP guests will get unexpected (and yet unnecessary) #VE which the kernel is

s/#VE/#VE or #VC/

Quoted 56 lines

unable to handle at this stage.

Cc Tom Lendacky <[email protected]>
Cc Dave Young <[email protected]>

---

v2 -> v3:
- Change to only do WBINVD for bare metal

---
  arch/x86/include/asm/kexec.h         |  2 +-
  arch/x86/kernel/machine_kexec_64.c   |  2 +-
  arch/x86/kernel/relocate_kernel_64.S | 14 +++++++++-----
  3 files changed, 11 insertions(+), 7 deletions(-)
arch/x86/include/asm/kexec.h +1 -1
index 91ca9a9ee3a2..455f8a6c66a9 100644
@@ -128,7 +128,7 @@ relocate_kernel(unsigned long indirection_page,
  		unsigned long page_list,
  		unsigned long start_address,
  		unsigned int preserve_context,
-		unsigned int host_mem_enc_active);
+		unsigned int bare_metal);
  #endif
 
  #define ARCH_HAS_KIMAGE_ARCH
arch/x86/kernel/machine_kexec_64.c +1 -1
index b180d8e497c3..a454477b7b4c 100644
@@ -358,7 +358,7 @@ void machine_kexec(struct kimage *image)
  				       (unsigned long)page_list,
  				       image->start,
  				       image->preserve_context,
-				       cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT));
+				       !boot_cpu_has(X86_FEATURE_HYPERVISOR));
 
  #ifdef CONFIG_KEXEC_JUMP
  	if (image->preserve_context)
arch/x86/kernel/relocate_kernel_64.S +1 -1
index 56cab1bb25f5..3e04c5e5687f 100644
@@ -50,7 +50,7 @@ SYM_CODE_START_NOALIGN(relocate_kernel)
  	 * %rsi page_list
  	 * %rdx start address
  	 * %rcx preserve_context
-	 * %r8  host_mem_enc_active
+	 * %r8  bare_metal
  	 */
 
  	/* Save the CPU context, used for jumping back */

arch/x86/kernel/relocate_kernel_64.S SYM_CODE_START_NOALIGN(relocate_kernel) · line 81

Quoted 5 lines
arch/x86/kernel/relocate_kernel_64.S +1 -1
@@ -78,7 +78,7 @@ SYM_CODE_START_NOALIGN(relocate_kernel)
  	pushq $0
  	popfq
 
-	/* Save SME active flag */
+	/* Save the bare_metal */

Either "Save bare_metal" or "Save the bare_metal flag"

Quoted 7 lines
arch/x86/kernel/relocate_kernel_64.S +0 -0
  	movq	%r8, %r12
 
  	/*
@@ -160,9 +160,13 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)
  	movq	%r9, %cr3
 
  	/*

arch/x86/kernel/relocate_kernel_64.S SYM_CODE_START_LOCAL_NOALIGN(identity_mapped) · line 169

Quoted 10 lines
arch/x86/kernel/relocate_kernel_64.S +7 -3
-	 * If SME is active, there could be old encrypted cache line
-	 * entries that will conflict with the now unencrypted memory
-	 * used by kexec. Flush the caches before copying the kernel.
+	 * The kernel could leave caches in incoherent state on SME/TDX
+	 * capable platforms.  Just do unconditional WBINVD to avoid
+	 * silent memory corruption to the new kernel for these platforms.
+	 *
+	 * But only do WBINVD for bare-metal because TDX guests and
+	 * SEV-ES/SEV-SNP guests will get #VE which the kernel is unable
+	 * to handle at this stage.

Similar comment here about doing an unconditional WBINVD, but then qualifying that as only on guests. This is where talking about how exception handling has been torn down would be good.

So for the confusion around the WBINVD in patches 1 and 2.

Thanks,
Tom

Quoted 3 lines
arch/x86/kernel/relocate_kernel_64.S +0 -0
  	 */
  	testq	%r12, %r12
  	jz 1f
author
#16 lore raw

Re: [PATCH v3 2/5] x86/kexec: do unconditional WBINVD for bare-metal in relocate_kernel()

Quoted 16 lines · Tom Lendacky
Quoted 15 lines · Kai Huang

Both SME and TDX can leave caches in incoherent state due to memory encryption.  During kexec, the caches must be flushed before jumping to the second kernel to avoid silent memory corruption to the second kernel.

During kexec, the WBINVD in stop_this_cpu() flushes caches for all remote cpus when they are being stopped.  For SME, the WBINVD in relocate_kernel() flushes the cache for the last running cpu (which is executing the kexec).

Similarly, to support kexec for TDX host, after stopping all remote cpus with cache flushed, the kernel needs to flush cache for the last running cpu.

Use the existing WBINVD in relocate_kernel() to cover TDX host as well.

Quoted 8 lines
Quoted 6 lines

However, instead of sprinkling around vendor-specific checks, just do unconditional WBINVD to cover both SME and TDX.  Kexec is not a fast path so having one additional WBINVD for platforms w/o SME/TDX is acceptable.

But only do WBINVD for bare-metal because TDX guests and SEV-ES/SEV-SNP guests will get unexpected (and yet unnecessary) #VE which the kernel is

s/#VE/#VE or #VC/

Will do "exception (#VE or #VC)".

Quoted 52 lines
Quoted 52 lines

unable to handle at this stage.

Cc Tom Lendacky <[email protected]>
Cc Dave Young <[email protected]>

---

v2 -> v3:
  - Change to only do WBINVD for bare metal

---
  arch/x86/include/asm/kexec.h         |  2 +- arch/x86/kernel/machine_kexec_64.c   |  2 +- arch/x86/kernel/relocate_kernel_64.S | 14 +++++++++-----

  3 files changed, 11 insertions(+), 7 deletions(-)
arch/x86/include/asm/kexec.h +1 -1
index 91ca9a9ee3a2..455f8a6c66a9 100644
@@ -128,7 +128,7 @@ relocate_kernel(unsigned long indirection_page,
           unsigned long page_list,
           unsigned long start_address,
           unsigned int preserve_context,
-        unsigned int host_mem_enc_active);
+        unsigned int bare_metal);
   #endif
   #define ARCH_HAS_KIMAGE_ARCH
arch/x86/kernel/machine_kexec_64.c +1 -1
index b180d8e497c3..a454477b7b4c 100644
@@ -358,7 +358,7 @@ void machine_kexec(struct kimage *image)
                          (unsigned long)page_list,
                          image->start,
                          image->preserve_context,
-                       cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT));
+                       !boot_cpu_has(X86_FEATURE_HYPERVISOR));
   #ifdef CONFIG_KEXEC_JUMP
       if (image->preserve_context)
arch/x86/kernel/relocate_kernel_64.S +1 -1
index 56cab1bb25f5..3e04c5e5687f 100644
@@ -50,7 +50,7 @@ SYM_CODE_START_NOALIGN(relocate_kernel)
        * %rsi page_list
        * %rdx start address
        * %rcx preserve_context
-     * %r8  host_mem_enc_active
+     * %r8  bare_metal
        */
Quoted 8 lines
Quoted 6 lines
arch/x86/kernel/relocate_kernel_64.S +0 -0
       /* Save the CPU context, used for jumping back */
@@ -78,7 +78,7 @@ SYM_CODE_START_NOALIGN(relocate_kernel)

      pushq $0
      popfq
-    /* Save SME active flag */
+    /* Save the bare_metal */

Either "Save bare_metal" or "Save the bare_metal flag"

Will use the latter. Thanks.

Quoted 10 lines
Quoted 10 lines

      movq    %r8, %r12
      /*

arch/x86/kernel/relocate_kernel_64.S +0 -0
@@ -160,9 +160,13 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)

      movq    %r9, %cr3
      /*
-     * If SME is active, there could be old encrypted cache line -     * entries that will conflict with the now unencrypted memory -     * used by kexec. Flush the caches before copying the kernel. +     * The kernel could leave caches in incoherent state on SME/TDX

Quoted 9 lines
Quoted 6 lines

+     * capable platforms.  Just do unconditional WBINVD to avoid +     * silent memory corruption to the new kernel for these platforms. +     *
+     * But only do WBINVD for bare-metal because TDX guests and +     * SEV-ES/SEV-SNP guests will get #VE which the kernel is unable +     * to handle at this stage.

Similar comment here about doing an unconditional WBINVD, but then qualifying that as only on guests. This is where talking about how exception handling has been torn down would be good.

OK.

Thinking again, also it might be a good idea to not lose the existing comment for SME, because it somehow justifies why we do WBINVD _HERE_ I suppose?

How about below?

/*
* The kernel could leave caches in incoherent state on SME/TDX
* capable platforms. Flush cache to avoid silent memory
* corruption for these platforms.
*
* For SME, need to flush cache here before copying the kernel.
* When it is active, there could be old encrypted cache line
* entries that will conflict with the now unencrypted memory
* used by kexec.
*
* Do WBINVD for bare-metal to cover both SME and TDX, as it's
* not safe to do WBINVD for TDX and SEV-ES/SEV-SNP guests.
* WBINVD results in exception (#VE or #VC) for these guests, and
* at this stage kernel is not able to handle such exception any
* more because the kernel has torn down IDT.
*/

#17 lore raw

Re: [PATCH v3 2/5] x86/kexec: do unconditional WBINVD for bare-metal in relocate_kernel()

Quoted 26 lines · Kai
Quoted 18 lines · Tom Lendacky
Quoted 5 lines · Kai Huang
(quoted diff) +0 -0
@@ -160,9 +160,13 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)

      movq    %r9, %cr3
      /*
-     * If SME is active, there could be old encrypted cache line -     * entries that will conflict with the now unencrypted memory

Quoted 8 lines

-     * used by kexec. Flush the caches before copying the kernel. +     * The kernel could leave caches in incoherent state on SME/TDX +     * capable platforms.  Just do unconditional WBINVD to avoid +     * silent memory corruption to the new kernel for these platforms. +     *
+     * But only do WBINVD for bare-metal because TDX guests and +     * SEV-ES/SEV-SNP guests will get #VE which the kernel is unable +     * to handle at this stage.

Similar comment here about doing an unconditional WBINVD, but then qualifying that as only on guests. This is where talking about how exception handling has been torn down would be good.

OK.

Thinking again, also it might be a good idea to not lose the existing comment for SME, because it somehow justifies why we do WBINVD _HERE_ I suppose?

How about below?

Quoted 16 lines

    /*
     * The kernel could leave caches in incoherent state on SME/TDX
     * capable platforms.  Flush cache to avoid silent memory
     * corruption for these platforms.
     *
     * For SME, need to flush cache here before copying the kernel.
     * When it is active, there could be old encrypted cache line
     * entries that will conflict with the now unencrypted memory
     * used by kexec.
     *
     * Do WBINVD for bare-metal to cover both SME and TDX, as it's
     * not safe to do WBINVD for TDX and SEV-ES/SEV-SNP guests.
     * WBINVD results in exception (#VE or #VC) for these guests, and
     * at this stage kernel is not able to handle such exception any
     * more because the kernel has torn down IDT.
     */

Similar to my comment in the other patch, just adding something that indicates that the WBINVD isn't necessary for TDX and SEV-ES/SEV-SNP (and maybe guests in general) would help. Maybe something like:

* Do WBINVD for bare-metal only to cover both SME and TDX. It
* isn't necessary to perform a WBINVD in a guest and performing
* one could result in an exception (#VE or #VC) for a TDX or
* SEV-ES/SEV-SNP guest that can crash the guest since, at this
* stage, the kernel has torn down the IDT.

This is all just my opinion, though, and others may read it differently. Probably not worth bike-shedding it.

Thanks,
Tom

author
#18 lore raw

Re: [PATCH v3 2/5] x86/kexec: do unconditional WBINVD for bare-metal in relocate_kernel()

Quoted 52 lines · Tom Lendacky
Quoted 30 lines · Kai
Quoted 11 lines · Tom Lendacky
Quoted 9 lines · Kai Huang
arch/x86/kernel/relocate_kernel_64.S +3 -3
@@ -160,9 +160,13 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)
       movq    %r9, %cr3
       /*
-     * If SME is active, there could be old encrypted cache line
-     * entries that will conflict with the now unencrypted memory
-     * used by kexec. Flush the caches before copying the kernel.
+     * The kernel could leave caches in incoherent state on SME/TDX
+     * capable platforms.  Just do unconditional WBINVD to avoid
+     * silent memory corruption to the new kernel for these platforms.
Quoted 9 lines

arch/x86/kernel/relocate_kernel_64.S SYM_CODE_START_LOCAL_NOALIGN(identity_mapped) · line 169

Quoted 4 lines
arch/x86/kernel/relocate_kernel_64.S +4 -0
+     *
+     * But only do WBINVD for bare-metal because TDX guests and
+     * SEV-ES/SEV-SNP guests will get #VE which the kernel is unable
+     * to handle at this stage.

Similar comment here about doing an unconditional WBINVD, but then qualifying that as only on guests. This is where talking about how exception handling has been torn down would be good.

OK.

Thinking again, also it might be a good idea to not lose the existing comment for SME, because it somehow justifies why we do WBINVD _HERE_ I suppose?

How about below?

Quoted 16 lines

    /*
     * The kernel could leave caches in incoherent state on SME/TDX
     * capable platforms.  Flush cache to avoid silent memory
     * corruption for these platforms.
     *
     * For SME, need to flush cache here before copying the kernel.
     * When it is active, there could be old encrypted cache line
     * entries that will conflict with the now unencrypted memory
     * used by kexec.
     *
     * Do WBINVD for bare-metal to cover both SME and TDX, as it's
     * not safe to do WBINVD for TDX and SEV-ES/SEV-SNP guests.
     * WBINVD results in exception (#VE or #VC) for these guests, and
     * at this stage kernel is not able to handle such exception any
     * more because the kernel has torn down IDT.
     */

Similar to my comment in the other patch, just adding something that indicates that the WBINVD isn't necessary for TDX and SEV-ES/SEV-SNP (and maybe guests in general) would help. Maybe something like:

Quoted 6 lines

* Do WBINVD for bare-metal only to cover both SME and TDX. It
* isn't necessary to perform a WBINVD in a guest and performing
* one could result in an exception (#VE or #VC) for a TDX or
* SEV-ES/SEV-SNP guest that can crash the guest since, at this
* stage, the kernel has torn down the IDT.

Ah sure. I will replace the 3rd paragraph in my original comment with yours (assuming that is what you meant).

Thanks again for providing the feedback!

author
#19 lore raw

PATCH v3 3/5 x86/kexec: Reset TDX private memory on platforms with TDX erratum

In reply to Kai Huang

TL;DR:

On the platforms with TDX "partial write machine check" erratum, during kexec, convert TDX private memory back to normal before jumping to the second kernel to avoid the second kernel seeing potential unexpected machine check.

Long version:

The first few generations of TDX hardware have an erratum. A partial write to a TDX private memory cacheline will silently "poison" the line. Subsequent reads will consume the poison and generate a machine check. According to the TDX hardware spec, neither of these things should have happened.

== Background ==

Virtually all kernel memory accesses operations happen in full cachelines. In practice, writing a "byte" of memory usually reads a 64 byte cacheline of memory, modifies it, then writes the whole line back. Those operations do not trigger this problem.

This problem is triggered by "partial" writes where a write transaction of less than cacheline lands at the memory controller. The CPU does these via non-temporal write instructions (like MOVNTI), or through UC/WC memory mappings. The issue can also be triggered away from the CPU by devices doing partial writes via DMA.

== Problem ==

A fast warm reset doesn't reset TDX private memory. Kexec() can also boot into the new kernel directly. Thus if the old kernel has left any TDX private pages on the platform with this erratum, the new kernel might get unexpected machine check.

Note that w/o this erratum any kernel read/write on TDX private memory should never cause machine check, thus it's OK for the old kernel to leave TDX private pages as is.

== Solution ==

In short, with this erratum, the kernel needs to explicitly convert all TDX private pages back to normal to give the new kernel a clean slate after kexec(). The BIOS is also expected to disable fast warm reset as a workaround to this erratum, thus this implementation doesn't try to reset TDX private memory for the reboot case in the kernel but depends on the BIOS to enable the workaround.

Convert TDX private pages back to normal (using MOVDIR64B to clear these pages) after all remote cpus have been stopped and cache flush has been done on all cpus, when no more TDX activity can happen further.

Do it in machine_kexec() to cover both normal kexec, and crash kexec.

For now TDX private memory can only be PAMT pages. It would be ideal to cover all types of TDX private memory here, but there are practical problems to do so:

1) There's no existing infrastructure to track TDX private pages;
2) It's not feasible to query the TDX module about page type, because VMX, which making SEAMCALL requires, has already been disabled;
3) Even if it is feasible to query the TDX module, the result may not be accurate. E.g., the remote CPU could be stopped right before MOVDIR64B.

One temporary solution is to blindly convert all memory pages, but it's problematic to do so too, because not all pages are mapped as writable in the direct mapping. It can be done by switching to the identical mapping created for kexec(), or a new page table, but the complexity looks overkill.

Therefore, rather than doing something dramatic, only reset PAMT pages here.

Leave resetting other TDX private pages as a future work when they become possible to exist.

---

v2 -> v3:
- No change

v1 -> v2:
- Remove using reboot notifier to stop TDX module as it doesn't cover crash kexec. Change to use a variable with barrier instead. (Rick)
- Introduce kexec_save_processor_start() to make code better, and make the comment around calling site of tdx_reset_memory() more concise. (Dave)
- Mention cache for all other cpus have been flushed around native_wbinvd() in tdx_reset_memory(). (Dave)
- Remove the extended alternaties discussion from the comment, but leave it in the changelog. Point out what does current code do and point out risk. (Dave)

---
 arch/x86/include/asm/tdx.h         |  2 +
 arch/x86/kernel/machine_kexec_64.c | 27 ++++++++--
 arch/x86/virt/vmx/tdx/tdx.c        | 79 ++++++++++++++++++++++++++++++
 3 files changed, 104 insertions(+), 4 deletions(-)
arch/x86/include/asm/tdx.h +2 -0
index eba178996d84..ed3ac9a8a079 100644
@@ -116,11 +116,13 @@ static inline u64 sc_retry(sc_func_t func, u64 fn,
 int tdx_cpu_enable(void);
 int tdx_enable(void);
 const char *tdx_dump_mce_info(struct mce *m);
+void tdx_reset_memory(void);
 #else
 static inline void tdx_init(void) { }
 static inline int tdx_cpu_enable(void) { return -ENODEV; }
 static inline int tdx_enable(void)  { return -ENODEV; }
 static inline const char *tdx_dump_mce_info(struct mce *m) { return NULL; }
+static inline void tdx_reset_memory(void) { }
 #endif	/* CONFIG_INTEL_TDX_HOST */
 
 #endif /* !__ASSEMBLY__ */
arch/x86/kernel/machine_kexec_64.c +23 -4
index a454477b7b4c..ba5a66bf724e 100644
@@ -28,6 +28,7 @@
 #include <asm/setup.h>
 #include <asm/set_memory.h>
 #include <asm/cpu.h>
+#include <asm/tdx.h>
 
 #ifdef CONFIG_ACPI
 /*
@@ -288,6 +289,14 @@ void machine_kexec_cleanup(struct kimage *image)
 	free_transition_pgtable(image);
 }
 
+static void kexec_save_processor_start(struct kimage *image)
+{
+#ifdef CONFIG_KEXEC_JUMP
+	if (image->preserve_context)
+		save_processor_state();
+#endif
+}
+
 /*
  * Do not allocate memory (or fail in any way) in machine_kexec().
  * We are past the point of no return, committed to rebooting now.
@@ -298,10 +307,20 @@ void machine_kexec(struct kimage *image)
 	void *control_page;
 	int save_ftrace_enabled;
 
-#ifdef CONFIG_KEXEC_JUMP
-	if (image->preserve_context)
-		save_processor_state();
-#endif
+	kexec_save_processor_start(image);
+
+	/*
+	 * Convert TDX private memory back to normal (when needed) to
+	 * avoid the second kernel potentially seeing unexpected machine
+	 * check.
+	 *
+	 * However skip this when preserve_context is on.  By reaching
+	 * here, TDX (if ever got enabled by the kernel) has survived
+	 * from the suspend when preserve_context is on, and it can
+	 * continue to work after jumping back from the second kernel.
+	 */
+	if (!image->preserve_context)
+		tdx_reset_memory();
 
 	save_ftrace_enabled = __ftrace_enabled_save();
 
arch/x86/virt/vmx/tdx/tdx.c +79 -0

106 lines not on this page · raw message

Signature · 1 line
2.43.2
author
#20 lore raw

PATCH v3 4/5 x86/virt/tdx: Remove the !KEXEC_CORE dependency

In reply to Kai Huang

Now TDX host can work with kexec(). Remove the !KEXEC_CORE dependency.

---
 arch/x86/Kconfig | 1 -
 1 file changed, 1 deletion(-)
arch/x86/Kconfig +0 -1
index c62db6b853d7..bfafc8a16a07 100644
@@ -1967,7 +1967,6 @@ config INTEL_TDX_HOST
 	depends on X86_X2APIC
 	select ARCH_KEEP_MEMBLOCK
 	depends on CONTIG_ALLOC
-	depends on !KEXEC_CORE
 	depends on X86_MCE
 	help
 	  Intel Trust Domain Extensions (TDX) protects guest VMs from malicious
Signature · 1 line
2.43.2
author
#21 lore raw

PATCH v3 5/5 x86/virt/tdx: Add TDX memory reset notifier to reset other private pages

In reply to Kai Huang

TL;DR:

To cover both normal kexec and crash kexec, add a TDX specific memory reset notifier to let "in-kernel TDX users" use their own way to convert TDX private pages (that they manage respectively) in tdx_reset_memory().

Long version:

On the platforms with TDX "partial write machine check" erratum, during kexec, the kernel needs to convert TDX private memory back to normal before jumping to the second kernel to avoid the second kernel seeing potential machine check.

For now tdx_reset_memory() only resets PAMT pages. KVM will be the first in-kernel TDX user to support running TDX guests, and by then other TDX private pages will start to exist. They need to be covered too.

Currently the kernel doesn't have a unified way to tell whether a given page is TDX private page or not. One choice is to add such unified way, and there are couple of options to do it:

1) Use a bitmap, or Xarray, etc to track TDX private page for all PFNs;
2) Use a "software-only" bit in the direct-mapping PTE to mark a given

    page is TDX private page;

3) Use a new flag in 'struct page' to mark TDX private page;
4) ... potential other ways.

Option 1) consumes additional memory. E.g., if using bitmap, the overhead is "number of total RAM pages / 8" bytes.

Option 2) would cause splitting large-page mapping to 4K mapping in the direct mapping when one page is allocated as TDX private page, and cause additional TLB flush etc. It's not ideal for such use case.

Option 3) apparently contradicts to the effort to reduce the use of the flags of 'struct page'.

None of above is ideal.

Therefore, instead of providing a unified way to tell whether a given page is TDX private page or not, leave "resetting TDX private pages" to the "in-kernel user" of TDX.

This is motivated by the fact that KVM is already maintaining an Xarray to track "memory attributes (e.g., private or shared)" for each GFN for each guest. Thus KVM can use its own way to find all TDX private pages that it manages and convert them back to normal.

For the normal kexec the reboot notifier could be used, but it doesn't cover the cash kexec.

Add a TDX specific memory reset notifier to achieve this. The in-kernel TDX users will need to register their own notifiers to reset TDX private pages. Call these notifiers in tdx_reset_memory() right before resetting PAMT pages.

KVM will be the first user of this notifier. Export the "register" and "unregister" APIs for KVM to use.

---
 arch/x86/include/asm/tdx.h  | 14 ++++++++++++
 arch/x86/virt/vmx/tdx/tdx.c | 45 +++++++++++++++++++++++++++----------
 2 files changed, 47 insertions(+), 12 deletions(-)
arch/x86/include/asm/tdx.h +14 -0
index ed3ac9a8a079..7c2c0a0b9754 100644
@@ -117,12 +117,26 @@ int tdx_cpu_enable(void);
 int tdx_enable(void);
 const char *tdx_dump_mce_info(struct mce *m);
 void tdx_reset_memory(void);
+
+struct notifier_block;
+
+int tdx_register_memory_reset_notifier(struct notifier_block *nb);
+void tdx_unregister_memory_reset_notifier(struct notifier_block *nb);
 #else
 static inline void tdx_init(void) { }
 static inline int tdx_cpu_enable(void) { return -ENODEV; }
 static inline int tdx_enable(void)  { return -ENODEV; }
 static inline const char *tdx_dump_mce_info(struct mce *m) { return NULL; }
 static inline void tdx_reset_memory(void) { }
+
+struct notifier_block;
+
+static inline int tdx_register_memory_reset_notifier(struct notifier_block *nb)
+{
+	return -EOPNOTSUPP;
+}
+static inline void tdx_unregister_memory_reset_notifier(
+		struct notifier_block *nb) { }
 #endif	/* CONFIG_INTEL_TDX_HOST */
 
 #endif /* !__ASSEMBLY__ */
arch/x86/virt/vmx/tdx/tdx.c +33 -12

69 lines not on this page · raw message

Signature · 1 line
2.43.2