Add RMPOPT support.

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] x86/sev: CPU hotplug disablement bypassed on early kexec errors
--

commit ac084f2d7188c520f2ee3ced2ab76e4c068df91f
Author: Ashish Kalra <[email protected]>

x86/sev: Disable CPU hotplug while SNP is active

This commit disables CPU hotplug when SNP is enabled to prevent SNP guest memory from being broken by CPUs coming online without the SnpEn bit set. It ties CPU hotplug to the SNP-enable bit, disabling it in snp_prepare() and re-enabling it in snp_shutdown() when SNP is disabled. It also explicitly disables CPU hotplug during kexec if SNP is already enabled.

Quoted 4 lines

arch/x86/virt/svm/sev.c +0 -0

index cff285d8ad8e3..558f7924a3f85 100644

[ ... ]

Quoted 11 lines

arch/x86/virt/svm/sev.c +2 -0

@@ -577,6 +582,8 @@ EXPORT_SYMBOL_FOR_MODULES(snp_shutdown, "ccp");
  */
 int __init snp_rmptable_init(void)
 {
+	u64 val;
+
 	if (WARN_ON_ONCE(!cc_platform_has(CC_ATTR_HOST_SEV_SNP)))
 		return -ENOSYS;
 

@@ -586,6 +593,15 @@ int __init snp_rmptable_init(void)
 	if (!setup_rmptable())

arch/x86/virt/svm/sev.c int __init snp_rmptable_init(void) · line 603

Quoted 10 lines

arch/x86/virt/svm/sev.c +8 -0

 		return -ENOSYS;
 
+	/*
+	 * On a kexec boot SNP may already be enabled (legacy firmware leaves
+	 * SnpEn set across shutdown), in which case snp_prepare() bails without
+	 * disabling CPU hotplug, so disable it here.
+	 */
+	rdmsrq(MSR_AMD64_SYSCFG, val);
+	if (val & MSR_AMD64_SYSCFG_SNP_EN)
+		cpu_hotplug_disable();

[Severity: High]
If a kexec kernel boots into a system where the previous kernel left MSR_AMD64_SYSCFG_SNP_EN active, and an early initialization step fails (such as passing amd_iommu=off on the kernel command line resulting in !amd_iommu_snp_en, or setup_rmptable() failing), snp_rmptable_init() will return -ENOSYS before checking the physical MSR hardware state and disabling CPU hotplug.

Because hotplug remains enabled and the CCP driver skips initialization on failure (bypassing snp_prepare()), could a subsequently hotplugged CPU come online without the SNP_EN bit set?

If this results in a mixed hardware state, does it violate firmware constraints and cause fatal RMP faults when the new CPU accesses memory?

Should the hardware state be checked and hotplug disabled unconditionally before the software configuration checks?

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] x86/sev: CPU hotplug disablement bypassed on early kexec errors
--

commit ac084f2d7188c520f2ee3ced2ab76e4c068df91f
Author: Ashish Kalra <[email protected]>

x86/sev: Disable CPU hotplug while SNP is active

This commit disables CPU hotplug when SNP is enabled to prevent SNP guest memory from being broken by CPUs coming online without the SnpEn bit set. It ties CPU hotplug to the SNP-enable bit, disabling it in snp_prepare() and re-enabling it in snp_shutdown() when SNP is disabled. It also explicitly disables CPU hotplug during kexec if SNP is already enabled.

Quoted 4 lines

arch/x86/virt/svm/sev.c +0 -0

index cff285d8ad8e3..558f7924a3f85 100644

[ ... ]

Quoted 13 lines

arch/x86/virt/svm/sev.c +2 -0

@@ -577,6 +582,8 @@ EXPORT_SYMBOL_FOR_MODULES(snp_shutdown, "ccp");
  */
 int __init snp_rmptable_init(void)
 {
+	u64 val;
+
 	if (WARN_ON_ONCE(!cc_platform_has(CC_ATTR_HOST_SEV_SNP)))
 		return -ENOSYS;
 

@@ -586,6 +593,15 @@ int __init snp_rmptable_init(void)
 	if (!setup_rmptable())
 		return -ENOSYS;
 

arch/x86/virt/svm/sev.c int __init snp_rmptable_init(void) · line 603

Quoted 8 lines

arch/x86/virt/svm/sev.c +8 -0

+	/*
+	 * On a kexec boot SNP may already be enabled (legacy firmware leaves
+	 * SnpEn set across shutdown), in which case snp_prepare() bails without
+	 * disabling CPU hotplug, so disable it here.
+	 */
+	rdmsrq(MSR_AMD64_SYSCFG, val);
+	if (val & MSR_AMD64_SYSCFG_SNP_EN)
+		cpu_hotplug_disable();

[Severity: High]
If a kexec kernel boots into a system where the previous kernel left MSR_AMD64_SYSCFG_SNP_EN active, and an early initialization step fails (such as passing amd_iommu=off

on the kernel command line resulting in !amd_iommu_snp_en, or setup_rmptable() failing),

snp_rmptable_init() will
return -ENOSYS before checking the physical MSR hardware state and disabling CPU hotplug.

Because hotplug remains enabled and the CCP driver skips initialization on failure (bypassing snp_prepare()), could a subsequently hotplugged CPU come online without the SNP_EN bit set?

If this results in a mixed hardware state, does it violate firmware constraints and cause fatal RMP faults when the new CPU accesses memory?

Should the hardware state be checked and hotplug disabled unconditionally before the software configuration checks?

Signature · 1 line

Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2

Quoted 35 lines · On Wed, Sep 16, 2026 at 10:36:30PM +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] x86/sev: CPU hotplug disablement bypassed on early kexec errors
--

commit ac084f2d7188c520f2ee3ced2ab76e4c068df91f
Author: Ashish Kalra <[email protected]>

x86/sev: Disable CPU hotplug while SNP is active

This commit disables CPU hotplug when SNP is enabled to prevent SNP guest memory from being broken by CPUs coming online without the SnpEn bit set. It ties CPU hotplug to the SNP-enable bit, disabling it in snp_prepare() and re-enabling it in snp_shutdown() when SNP is disabled. It also explicitly disables CPU hotplug during kexec if SNP is already enabled.

Quoted 4 lines

arch/x86/virt/svm/sev.c +0 -0

index cff285d8ad8e3..558f7924a3f85 100644

[ ... ]

Quoted 13 lines

arch/x86/virt/svm/sev.c +2 -0

@@ -577,6 +582,8 @@ EXPORT_SYMBOL_FOR_MODULES(snp_shutdown, "ccp");
  */
 int __init snp_rmptable_init(void)
 {
+	u64 val;
+
 	if (WARN_ON_ONCE(!cc_platform_has(CC_ATTR_HOST_SEV_SNP)))
 		return -ENOSYS;
 

@@ -586,6 +593,15 @@ int __init snp_rmptable_init(void)
 	if (!setup_rmptable())
 		return -ENOSYS;
 

Quoted 13 lines

arch/x86/virt/svm/sev.c int __init snp_rmptable_init(void) · line 603

Quoted 8 lines

arch/x86/virt/svm/sev.c +8 -0

+	/*
+	 * On a kexec boot SNP may already be enabled (legacy firmware leaves
+	 * SnpEn set across shutdown), in which case snp_prepare() bails without
+	 * disabling CPU hotplug, so disable it here.
+	 */
+	rdmsrq(MSR_AMD64_SYSCFG, val);
+	if (val & MSR_AMD64_SYSCFG_SNP_EN)
+		cpu_hotplug_disable();

[Severity: High]
If a kexec kernel boots into a system where the previous kernel left MSR_AMD64_SYSCFG_SNP_EN active, and an early initialization step fails (such as passing amd_iommu=off

amd_iommu=off means no SNP.

Quoted 2 lines

on the kernel command line resulting in !amd_iommu_snp_en, or setup_rmptable() failing),

The same happens in that case:

        if (snp_rmptable_init()) {
                pr_warn("SNP: RMP initialization failed, SNP cannot be supported.\n");
                goto disable_snp;
		     ^^^^^^^^^^^

Does Sashiko not see that or somehow it ate a lot of credits so it cannot use enough now to "swap in" the whole context... if that is the AI that is going to destroy the world, then I'm not too worried. :)

Or is my AI failing me right now?

:-P

Leaving in the rest.

Quoted 8 lines

snp_rmptable_init() will
return -ENOSYS before checking the physical MSR hardware state and disabling CPU hotplug.

Because hotplug remains enabled and the CCP driver skips initialization on failure (bypassing snp_prepare()), could a subsequently hotplugged CPU come online without the SNP_EN bit set?

Yes, but the above disabled all our intention of running SNP in the first place...

Quoted 8 lines

If this results in a mixed hardware state, does it violate firmware constraints and cause fatal RMP faults when the new CPU accesses memory?

Should the hardware state be checked and hotplug disabled unconditionally before the software configuration checks?

Signature · 1 line

Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] x86/sev: missing MSR configuration on CPU hotplug cycle due to local check --

commit f73eea5b303a29d64f9efe740eeba6e570bac844
Author: Ashish Kalra <[email protected]>

x86/sev: Initialize RMPOPT configuration MSRs

The new RMPOPT instruction helps manage per-CPU RMP optimization structures inside the CPU. Initialize the per-CPU RMPOPT table base to the starting physical address to enable RMP optimization for up to 2 TB of system RAM on all CPUs.

Quoted 4 lines

arch/x86/virt/svm/sev.c +0 -0

index 558f7924a3f85..25cb486ff61b6 100644

[ ... ]

Quoted 27 lines

arch/x86/virt/svm/sev.c +23 -0

@@ -575,6 +577,42 @@ void snp_shutdown(void)
 }
 EXPORT_SYMBOL_FOR_MODULES(snp_shutdown, "ccp");
 
+static bool rmpopt_capable(void)
+{
+	return cpu_feature_enabled(X86_FEATURE_RMPOPT) &&
+	       cc_platform_has(CC_ATTR_HOST_SEV_SNP);
+}
+
+void snp_enable_rmpopt(void)
+{
+	u64 base;
+	int cpu;
+
+	if (!rmpopt_capable())
+		return;
+
+	rmpopt_pa_start = ALIGN_DOWN(PFN_PHYS(min_low_pfn), SZ_1G);
+
+	/*
+	 * Per-CPU RMPOPT tables cover at most 2 TB.  Program each core's
+	 * RMPOPT_BASE with the start of RAM to optimize up to 2 TB.  The MSR
+	 * can only be written after SNP is enabled, so this runs from the ccp
+	 * SNP init path (and again on guest teardown) rather than an initcall.
+	 *
+	 * The loop below programs RMPOPT_BASE on all primary threads.  RMPOPT_EN

arch/x86/virt/svm/sev.c void snp_shutdown(void) · line 612

Quoted 10 lines

arch/x86/virt/svm/sev.c +10 -0

+	 * cannot be cleared while SNP is enabled, and CPU hotplug is disabled
+	 * while SNP is active, so once programmed the MSRs stay set on all CPUs
+	 * until SNP is disabled.  A set RMPOPT_EN on the local CPU therefore
+	 * means the programming has already been done, so skip it.
+	 */
+	rdmsrq(MSR_AMD64_RMPOPT_BASE, base);
+	if (!(base & MSR_AMD64_RMPOPT_ENABLE))
+		for_each_cpu(cpu, cpu_primary_thread_mask)
+			wrmsrq_on_cpu(cpu, MSR_AMD64_RMPOPT_BASE,
+				      rmpopt_pa_start | MSR_AMD64_RMPOPT_ENABLE);

[Severity: High]
Could this local CPU check cause hotplugged CPUs to miss MSR configuration?

If SNP is shut down, CPU hotplug is re-enabled but the MSR remains intact on existing CPUs. If a CPU is offlined and brought back online, its MSR is reset to the default.

Later, when SNP is re-enabled, the local CPU executing snp_enable_rmpopt() might still have its own RMPOPT_ENABLE bit set. Will it erroneously assume all CPUs are programmed and skip the wrmsrq_on_cpu() loop, leaving the hotplugged CPU unconfigured?

Quoted 3 lines

arch/x86/virt/svm/sev.c +3 -0

+}
+EXPORT_SYMBOL_FOR_MODULES(snp_enable_rmpopt, "ccp");
+

Changes in v15:
- Rename snp_setup_rmpopt() to snp_enable_rmpopt().
- Program each core's RMPOPT_BASE only when it is not already set.

 arch/x86/include/asm/msr-index.h |  3 ++
 arch/x86/include/asm/sev.h       |  2 +
 arch/x86/virt/svm/sev.c          | 66 +++++++++++++++++++++++++++++---
 drivers/crypto/ccp/sev-dev.c     |  2 +
 4 files changed, 68 insertions(+), 5 deletions(-)

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] x86/sev: fatal #GP on blind RMPOPT scan across MMIO holes
- [High] x86/sev: soft lockup via IRQ-disabled IPI for cold RMPOPT cache
- [High] x86/sev: fatal #UD race between RMPOPT and firmware SNP shutdown --

commit 62479bc981175bad2d66a2cbd0e76b01cb89e84b
Author: Ashish Kalra <[email protected]>

x86/sev: Perform RMP optimizations asynchronously

This patch adds support for performing RMP optimizations asynchronously using a dedicated per-CPU workqueue. It scans physical memory up to 2TB and executes RMPOPT to skip RMP checks for 1GB regions that do not contain SNP guest memory.

Quoted 4 lines

arch/x86/virt/svm/sev.c +0 -0

index 25cb486ff61b6..bd97135cacc22 100644

[ ... ]

Quoted 14 lines

arch/x86/virt/svm/sev.c +6 -0

@@ -557,6 +576,12 @@ int snp_prepare(void)
 }
 EXPORT_SYMBOL_FOR_MODULES(snp_prepare, "ccp");
 
+static void rmpopt_disable(void)
+{
+	if (rmpopt_wq)
+		cancel_delayed_work_sync(&rmpopt_delayed_work);
+}
+
 void snp_shutdown(void)
 {
 	u64 syscfg;

arch/x86/virt/svm/sev.c void snp_shutdown(void) · line 593

Quoted 4 lines

arch/x86/virt/svm/sev.c +1 -0

@@ -565,6 +590,8 @@ void snp_shutdown(void)
 	if (syscfg & MSR_AMD64_SYSCFG_SNP_EN)
 		return;
 
+	rmpopt_disable();

[Severity: High]
Is it safe to cancel the worker here?

When SNP is disabled, the CCP driver sends SEV_CMD_SNP_SHUTDOWN_EX to the firmware, which clears the SnpEn hardware state. The driver then calls snp_shutdown(), which calls cancel_delayed_work_sync() here.

Between the firmware clearing SnpEn and the work being cancelled, the background worker or an IPI can concurrently execute the RMPOPT instruction.

Executing RMPOPT while SnpEn is clear causes a #UD exception, leading to a kernel panic.

Quoted 4 lines

arch/x86/virt/svm/sev.c +1 -0

+
 	clear_rmp();
 	on_each_cpu(mfd_reconfigure, NULL, 1);
 

[ ... ]

Quoted 21 lines

arch/x86/virt/svm/sev.c +16 -1

@@ -580,8 +607,69 @@ EXPORT_SYMBOL_FOR_MODULES(snp_shutdown, "ccp");
 static bool rmpopt_capable(void)
 {
 	return cpu_feature_enabled(X86_FEATURE_RMPOPT) &&
-	       cc_platform_has(CC_ATTR_HOST_SEV_SNP);
+	       cc_platform_has(CC_ATTR_HOST_SEV_SNP) && rmpopt_enabled;
+}
+
+/*
+ * RMPOPT optimizations skip RMP checks at 1GB granularity if this range of
+ * memory does not contain any SNP guest memory.
+ *
+ * @pa is a system physical address; RMPOPT operates on the containing 1GB.
+ */
+static void rmpopt(u64 pa)
+{
+	enum rmpopt_op_type op = RMPOPT_OP_VERIFY_AND_REPORT_STATUS;
+	u64 pa_start = ALIGN_DOWN(pa, SZ_1G);
+
+	/* Supported by binutils 2.48+ */
+	asm volatile(".byte 0xf2, 0x0f, 0x01, 0xfc"

arch/x86/virt/svm/sev.c EXPORT_SYMBOL_FOR_MODULES(snp_shutdown, "ccp"); · line 635

Quoted 10 lines

arch/x86/virt/svm/sev.c +10 -0

+		     :: "a" (pa_start), "c" (op)
+		     : "memory", "cc");
+}
+
+static void rmpopt_scan_range(void *arg)
+{
+	u64 pa;
+
+	for (pa = rmpopt_pa_start; pa < rmpopt_pa_end; pa += SZ_1G)
+		rmpopt(pa);

[Severity: High]
Does this iteration skip non-system RAM such as the PCI MMIO hole (e.g., 3GB-4GB)?

This iterates over the entire physical address range from min_low_pfn to max_pfn in 1GB steps. Executing the RMPOPT instruction on addresses not backed by valid system memory causes a #GP.

Since the rmpopt() inline assembly lacks an exception handler, will this result in a fatal kernel panic when executing on MMIO holes?

arch/x86/virt/svm/sev.c EXPORT_SYMBOL_FOR_MODULES(snp_shutdown, "ccp"); · line 646

Quoted 11 lines

arch/x86/virt/svm/sev.c +11 -0

+}
+
+static void do_rmpopt_work(struct work_struct *work)
+{
+	/*
+	 * Warm up the RMPOPT cache on this pinned per-CPU worker with interrupts
+	 * enabled, so the IRQ-disabled fan-out below only issues cache-hit RMPOPTs.
+	 */
+	rmpopt_scan_range(NULL);
+
+	on_each_cpu_mask(cpu_primary_thread_mask, rmpopt_scan_range, NULL, true);

[Severity: High]
Could this IPI cause severe scheduling latency or potential soft lockups?

do_rmpopt_work() calls rmpopt_scan_range() locally to warm up the RMPOPT cache, then uses on_each_cpu_mask() to run it on all other CPUs via IPI.

Since the RMPOPT table is per-CPU, local warmup on the worker CPU does not affect remote CPUs. The IPI forces all other CPUs to execute the scan with a cold cache.

Executing up to 2048 cache-miss hardware instructions inside an IRQ-disabled IPI handler seems likely to cause severe latency issues.

Quoted 2 lines

arch/x86/virt/svm/sev.c +1 -0

+}

Changes in v15:
- Move the workqueue allocation and the (fixed) optimization range computation to an initcall; snp_enable_rmpopt() now only programs the RMPOPT_BASE MSRs and queues the optimization pass.
- Gate rmpopt_capable() on a static rmpopt_enabled bool set when the workqueue is allocated, instead of an if (rmpopt_wq) check, and drop rmpopt_wq_mutex.
- Queue both the initial and the teardown pass with mod_delayed_work().

 arch/x86/virt/svm/sev.c | 99 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 95 insertions(+), 4 deletions(-)

Quoted 10 lines · Ashish Kalra

Changes in v15:
- Move the workqueue allocation and the (fixed) optimization range computation to an initcall; snp_enable_rmpopt() now only programs the RMPOPT_BASE MSRs and queues the optimization pass.
- Gate rmpopt_capable() on a static rmpopt_enabled bool set when the workqueue is allocated, instead of an if (rmpopt_wq) check, and drop rmpopt_wq_mutex.
- Queue both the initial and the teardown pass with mod_delayed_work().

 arch/x86/virt/svm/sev.c | 99 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 95 insertions(+), 4 deletions(-)

Some scrubbing to this one too:

Author: Ashish Kalra <[email protected]>
Date:   Wed Sep 16 22:15:14 2026 +0000

x86/sev: Perform RMP optimizations asynchronously

When SNP is enabled, all writes to memory are checked to ensure memory integrity. This imposes performance overhead on the whole system.

RMPOPT is a new instruction that minimizes the performance overhead of RMP checks on the hypervisor and on non-SNP guests by allowing such checks to be skipped for 1GB regions of memory that are known not to contain any SNP guest memory.

Add support for performing RMP optimizations asynchronously using a dedicated per-CPU workqueue. Shortly after SNP initialization, run an optimization pass over all physical memory.

As SNP guests are launched, RMPUPDATE assigns their private pages to guest-owned state; when such a page falls within an optimized 1GB region, the hardware clears that region's RMPOPT optimization and RMP checks resume there to protect the guest memory.

Since launching SNP guests clears these optimizations, perform them again asynchronously using the dedicated workqueue.

[ bp:
- Massage commit message and comments
- simplify code
- redo some of the logic ]

Suggested-by Thomas Lendacky <[email protected]>

Suggested-by Dave Hansen <[email protected]>

Signed-off-by Ashish Kalra <[email protected]>

Signed-off-by Borislav Petkov (AMD) <[email protected]>

Link https://patch.msgid.link/daa513fbe8f0672b00bc5e026e4399244be6dc72.1789594774.git.ashish.kalra@amd.com open here

arch/x86/virt/svm/sev.c +89 -6

index 1fecd246ce5f..94754f2986d3 100644
@@ -19,6 +19,7 @@
 #include <linux/iommu.h>
 #include <linux/amd-iommu.h>
 #include <linux/nospec.h>
+#include <linux/workqueue.h>
 
 #include <asm/sev.h>
 #include <asm/processor.h>

@@ -124,7 +125,27 @@ static void *rmp_bookkeeping __ro_after_init;
 
 static u64 probed_rmp_base, probed_rmp_size;
 
-static phys_addr_t rmpopt_pa_start;
+static u64 rmpopt_pa_start, rmpopt_pa_end;
+
+enum rmpopt_op_type {
+	RMPOPT_OP_VERIFY_AND_REPORT_STATUS,
+	RMPOPT_OP_REPORT_STATUS
+};
+
+static struct workqueue_struct *rmpopt_wq;
+static struct delayed_work rmpopt_delayed_work;
+
+/* Software RMPOPT facilities initialized */
+static bool rmpopt_soft_init;
+
+/*
+ * Delay, in milliseconds, before the RMP re-optimization pass runs after an
+ * SNP guest is torn down. This coalesces a burst of teardowns into a single
+ * scan and gives each guest's pages time to be converted back to the shared,
+ * hypervisor-owned state.  The 10 second value is a heuristic trading
+ * re-optimization latency against scanning too eagerly.
+ */
+#define RMPOPT_WORK_TIMEOUT	(10 * MSEC_PER_SEC)
 
 static LIST_HEAD(snp_leaked_pages_list);
 static DEFINE_SPINLOCK(snp_leaked_pages_list_lock);

@@ -565,6 +586,9 @@ void snp_shutdown(void)
 	if (syscfg & MSR_AMD64_SYSCFG_SNP_EN)
 		return;
 
+	if (rmpopt_soft_init)
+		cancel_delayed_work_sync(&rmpopt_delayed_work);
+
 	clear_rmp();
 	on_each_cpu(mfd_reconfigure, NULL, 1);
 

@@ -577,21 +601,80 @@ void snp_shutdown(void)
 }
 EXPORT_SYMBOL_FOR_MODULES(snp_shutdown, "ccp");
 
-static bool rmpopt_capable(void)
+/*
+ * RMPOPT optimizations skip RMP checks at 1GB granularity if this range of
+ * memory does not contain any SNP guest memory.
+ *
+ * @pa is a system physical address; RMPOPT operates on the containing 1GB.
+ */
+static void rmpopt(u64 pa)
 {
-	return cpu_feature_enabled(X86_FEATURE_RMPOPT) &&
-	       cc_platform_has(CC_ATTR_HOST_SEV_SNP);
+	enum rmpopt_op_type op = RMPOPT_OP_VERIFY_AND_REPORT_STATUS;
+	u64 pa_start = ALIGN_DOWN(pa, SZ_1G);
+
+	/* Supported by binutils 2.48+ */
+	asm volatile(".byte 0xf2, 0x0f, 0x01, 0xfc"
+		     :: "a" (pa_start), "c" (op)
+		     : "memory", "cc");
+}
+
+static void rmpopt_scan_range(void *arg)
+{
+	u64 pa;
+
+	for (pa = rmpopt_pa_start; pa < rmpopt_pa_end; pa += SZ_1G)
+		rmpopt(pa);
 }
 
+static void do_rmpopt_work(struct work_struct *work)
+{
+	/*
+	 * Warm up the RMPOPT cache on this pinned per-CPU worker with interrupts
+	 * enabled, so the IRQ-disabled fan-out below only issues cache-hit RMPOPTs.
+	 */
+	rmpopt_scan_range(NULL);
+
+	on_each_cpu_mask(cpu_primary_thread_mask, rmpopt_scan_range, NULL, true);
+}
+
+static int __init rmpopt_init(void)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_RMPOPT))
+		return -ENODEV;
+
+	rmpopt_wq = alloc_workqueue("rmpopt_wq", WQ_PERCPU, 1);
+	if (!rmpopt_wq) {
+		pr_err("Failed to allocate RMPOPT workqueue\n");
+		return -ENOMEM;
+	}
+
+	INIT_DELAYED_WORK(&rmpopt_delayed_work, do_rmpopt_work);
+
+	/* The optimization range is fixed at boot; compute it once. */
+	rmpopt_pa_start = ALIGN_DOWN(PFN_PHYS(min_low_pfn), SZ_1G);
+	rmpopt_pa_end = ALIGN(PFN_PHYS(max_pfn), SZ_1G);
+	if ((rmpopt_pa_end - rmpopt_pa_start) > SZ_2T)
+		rmpopt_pa_end = rmpopt_pa_start + SZ_2T;
+
+	rmpopt_soft_init = true;
+
+	return 0;
+}
+device_initcall(rmpopt_init);
+
 void snp_enable_rmpopt(void)
 {
 	u64 base;
 	int cpu;
 
-	if (!rmpopt_capable())
+	if (!cpu_feature_enabled(X86_FEATURE_RMPOPT))
 		return;
 
-	rmpopt_pa_start = ALIGN_DOWN(PFN_PHYS(min_low_pfn), SZ_1G);
+	if (!cc_platform_has(CC_ATTR_HOST_SEV_SNP))
+		return;
+
+	if (!rmpopt_soft_init)
+		return;
 
 	/*
 	 * Per-CPU RMPOPT tables cover at most 2 TB.  Program each core's

arch/x86/virt/svm/sev.c +0 -0

 }
 EXPORT_SYMBOL_FOR_MODULES(snp_enable_rmpopt, "ccp");
 

Looks good.

One comment here, i originally had pr_info_once() for printing this once during SNP init, if you change it to pr_info(), it will get printed again at every SNP guest teardown too, i am fine with it and actually it keeps the user aware that RMPOPT optimizations are being re-applied/re-enabled, but there may be just too many kernel messages dumped.

From: Ashish Kalra <[email protected]>

In the SEV-SNP architecture, hypervisor and non-SNP guests are subject to RMP checks on writes to provide integrity of SEV-SNP guest memory.

The RMPOPT architecture enables optimizations whereby the RMP checks can be skipped if 1GB regions of memory are known to not contain any SNP guest memory.

RMPOPT is a new instruction designed to minimize the performance overhead of RMP checks for the hypervisor and non-SNP guests.

RMPOPT instruction currently supports two functions. In case of the verify and report status function the CPU will read the RMP contents, verify the entire 1GB region starting at the provided SPA is HV-owned. For the entire 1GB region it checks that all RMP entries in this region are HV-owned (i.e, not in assigned state) and then accordingly updates the RMPOPT table to indicate if optimization has been enabled and provide indication to software if the optimization was successful.

In case of report status function, the CPU returns the optimization status for the 1GB region.

The RMPOPT table is managed by a combination of software and hardware. Software uses the RMPOPT instruction to set bits in the table, indicating that regions of memory are entirely HV-owned. Hardware automatically clears bits in the RMPOPT table when RMP contents are changed during RMPUPDATE instruction.

For more information on the RMPOPT instruction, see the AMD64 RMPOPT technical documentation.

As SNP is enabled by default the hypervisor and non-SNP guests are subject to RMP write checks to provide integrity of SNP guest memory.

This patch-series adds support to enable RMP optimizations for up to 2TB of system RAM across the system and allow RMPUPDATE to disable those optimizations as SNP guests are launched.

Support for RAM larger than 2 TB will be added in follow-on series.

This series also adds support to disable CPU hotplug while SNP is active, as the SEV firmware enumerates CPUs at SNP initialization and is not aware of the OS bringing CPUs online or offline afterwards. This also keeps the set of CPUs stable for the asynchronous RMPOPT scan, so the per-core RMPOPT_BASE MSRs programmed during setup remain valid.

This series also introduces support to re-enable RMP optimizations during SNP guest termination, after guest pages have been converted back to shared.

RMP optimizations are performed asynchronously by queuing work on a dedicated workqueue after a 10 second delay.

Delaying work allows batching of multiple SNP guest terminations.

Once 1GB hugetlb guest_memfd support is merged, support for re-enabling RMPOPT optimizations during 1GB page cleanup will be added in follow-on series.

v16:
- 3/5 (Initialize RMPOPT MSRs): clear X86_FEATURE_RMPOPT when segmented RMP is enabled but its probe fails, so the initcall does not set up RMPOPT for a system that will not bring up SNP. Program the RMPOPT_BASE MSRs only when RMPOPT_EN is not already set on the local CPU, so guest teardown skips reprogramming rather than reprogramming the MSRs on every teardown.

Review feedback from Sashiko AI.

v15:
- 3/5 (Initialize RMPOPT MSRs): rename snp_setup_rmpopt() to snp_enable_rmpopt(); program each core's RMPOPT_BASE only when it is not already set.
- 4/5 (async RMPOPT): split setup from enable -- an initcall allocates the workqueue and computes the (fixed) optimization range once, and snp_enable_rmpopt() programs the RMPOPT_BASE MSRs and queues the optimization pass. Gate rmpopt_capable() on a static rmpopt_enabled bool instead of an if (rmpopt_wq) check and drop rmpopt_wq_mutex. Queue both the initial and the teardown pass with mod_delayed_work().
- 5/5 (Re-enable on guest shutdown): call snp_enable_rmpopt() on guest teardown instead of a separate snp_rmpopt_all_physmem().

This is a full resend rather than a 4/5-5/5 reply because patch 3 changed as part of the setup/enable split.

Review feedback from Borislav Petkov.

v14:
- 3/5 (Initialize RMPOPT MSRs): remove rmpopt_disable() and the RMPOPT_BASE teardown from this patch. RMPOPT is now set up once and not torn down, so the RMPOPT_BASE MSRs are left in place on shutdown; the (now minimal) disable path moves to 4/5.
- 4/5 (async RMPOPT): rework the setup/teardown -- initialize once and do not tear it down. rmpopt_disable() now only cancels the pending re-optimization pass. Use a dedicated per-CPU workqueue (WQ_PERCPU) and drop the migrate_disable()/migrate_enable() around the local warm-up scan. Add a binutils-version comment above the RMPOPT .byte encoding and trim redundant comments and the 2 TB pr_info(). Reword the subject to "Perform RMP optimizations asynchronously".
- 5/5 (Re-enable on guest shutdown): move the call-site comment onto the snp_rmpopt_all_physmem() definition and simplify the RMPOPT_WORK_TIMEOUT comment.
- Correct the Suggested-by/Reviewed-by tags across the series.

Review feedback from Borislav Petkov.

v13:
- 3/5 (Initialize RMPOPT MSRs): use cpu_primary_thread_mask directly instead of building a local rmpopt_cpumask -- all primary threads are online while SNP is active, so the two are equivalent. Rename snp_cleanup_rmpopt() to rmpopt_disable().
- 4/5 (async RMPOPT): drop the leader/follower bookkeeping. Warm the RMP scan cache once, then fan the RMPOPT out to cpu_primary_thread_mask; re-running RMPOPT on the warm-up CPU is a cache hit, so the separate follower cpumask and the cpumask_andnot() are no longer needed. Rename rmpopt_work_handler() to do_rmpopt_work() and drop the now-unused this_cpu.
- Add Reviewed-by: Tom Lendacky <[email protected]> to the series.

Review feedback from Borislav Petkov.

v12:
- Merged the former "Add interface to re-enable RMP optimizations" and "KVM: SEV: Perform RMP optimizations on SNP guest shutdown" patches into a single patch (now 5/5), since the interface has no user without the KVM caller. The series is now 5 patches.
- 2/5 (Disable CPU hotplug): drop a stray reference to later patches from the commit message and trim the snp_prepare() comment. Move cpu_hotplug_enable() to the end of snp_shutdown(), after clear_rmp() and the mfd_reconfigure() IPI, so no all-CPU operation runs after hotplug is re-enabled.
- 3/5 (Initialize RMPOPT MSRs): restructure snp_probe_rmptable_info() so the contiguous probe (which clears X86_FEATURE_RMPOPT) is the common fall-through, also covering X86_FEATURE_SEGMENTED_RMP advertised but disabled by firmware. Move snp_setup_rmpopt() to the end of __sev_snp_init_locked(). Give snp_cleanup_rmpopt() its own short comment in snp_shutdown().
- 4/5 (async RMPOPT): document the optimization trigger points in the commit message. Rename enum rmpopt_function to rmpopt_op_type. Merge __rmpopt() into rmpopt() and replace rmpopt_smp() with a new rmpopt_scan_range() that loops as the on_each_cpu() callback, so the follower scan issues one IPI per core rather than one per 1GB. Use u64 for physical addresses. Allocate the follower cpumask once at setup. Document rmpopt_wq as the setup sentinel. Move the RMPOPT_WORK_TIMEOUT define to the guest-shutdown patch (its only user) as (10 * MSEC_PER_SEC). Remove a spammy pr_info() and reword comments.
- 5/5 (Re-enable RMP optimizations on SNP guest shutdown): document why re-optimization is driven by guest teardown (the only event that returns guest memory to hypervisor ownership); reword the commit message (s/clear/disable/ for optimizations, drop "Conversely").

Review feedback from Borislav Petkov.

v11:
- Reordered so "Disable CPU hotplug while SNP is active" (2/6) precedes "Initialize RMPOPT configuration MSRs" (3/6): the RMPOPT setup/cleanup code is then introduced with CPU hotplug already disabled and never takes cpus_read_lock().
- 1/6 (cpufeatures): drop the tools/arch/x86/include/asm/cpufeatures.h change and adopt the commit message as applied by Boris.
- 2/6 (Disable CPU hotplug): reword the commit message. Drop the redundant cpus_read_lock()/cpus_read_unlock() in snp_prepare() in this same patch -- with hotplug disabled the online CPU mask is stable, so the read lock is not needed and hotplug handling has no hole when bisecting.
- 3/6 (Initialize RMPOPT MSRs): replace the rmpopt_capable bool with a small helper local to arch/x86/virt/svm/sev.c -- cpu_feature_enabled(X86_FEATURE_RMPOPT) && cc_platform_has(CC_ATTR_HOST_SEV_SNP) -- clearing X86_FEATURE_RMPOPT for a contiguous (non-segmented) RMP in snp_probe_rmptable_info() (runs at BSP init, before alternatives). Rename rmpopt_cleanup() to snp_cleanup_rmpopt() to match snp_setup_rmpopt(). Both are introduced without cpus_read_lock() (hotplug is already disabled by 2/6). Simplify the RMPOPT_BASE comments. Drop the now-unused <asm/sev.h> include from core.c.
- 4/6 (async RMPOPT): the follower scan is likewise introduced without cpus_read_lock(). Drop the cond_resched() calls (nops on x86). On re-initialization after a legacy SNP shutdown, re-queue the optimization pass instead of skipping it. pr_warn() on cpumask allocation failure.

Review feedback from Borislav Petkov and K Prateek Nayak.

v10:
- Rework the CPU-hotplug patch (3/6): disable CPU hotplug in snp_prepare(), before SnpEn is set, instead of late in __sev_snp_init_locked(), so no CPU can come online without SnpEn during SNP initialization (per upstream review). Tie hotplug to SnpEn: it stays disabled while SnpEn is set -- including across a failed SNP_INIT and across the legacy SNP_SHUTDOWN_EX path -- and is re-enabled only once the firmware clears SnpEn on the x86_snp_shutdown path. Drop the separate idempotent flag: snp_prepare() re-enables hotplug on its own early failure, and a kexec target that boots with SnpEn already set disables hotplug once in snp_rmptable_init(). Reword the commit log and comments accordingly.
- Emit a pr_warn() in rmpopt_work_handler() (4/6) when the follower cpumask allocation fails, instead of silently skipping the optimization pass.

Sashiko AI upstream review identified several of the above issues.

v9:
- Rename rmpopt_configured to rmpopt_capable.
- Make rmpopt_cpumask a cpumask_var_t (allocated/freed at setup/cleanup) instead of a static cpumask_t.
- Drop the v8 WARN_ON_ONCE() on the RMPOPT_BASE writes; use a plain wrmsrq_on_cpu(), matching the SNP MSR-write convention in this file.
- Disable CPU hotplug with cpu_hotplug_disable()/cpu_hotplug_enable() (per tglx); re-enable only on the full x86_snp_shutdown path.
- Simplify rmpopt_work_handler() to a single leader-then-followers path:
with CPU hotplug disabled while SNP is active and snp_prepare() requiring all CPUs online when RMPOPT_BASE is programmed, every core is always programmed, so the explicit-leader fallback is now unreachable. Drop it along with the v8 work_on_cpu()/rmpopt_leader_fn() helper.
- Drop the debugfs interface (was patch 7/7) and its report-only plumbing; observability will be revisited after this series is merged.
- Restrict snp_rmpopt_all_physmem()'s export to the kvm-amd module.
- Use scoped_guard(cpus_read_lock) for the per-CPU MSR and follower loops.

Sashiko AI upstream review identified several of the above issues.

v8:
- Add a new patch to disable CPU hotplug while SNP is active, keeping the CPU set stable for the RMPOPT work handler.
- Drop the setup_clear_cpu_cap(X86_FEATURE_RMPOPT) calls; the rmpopt_configured bool is the runtime guard.
- WARN_ON_ONCE() on the RMPOPT_BASE MSR writes that previously ignored their return value.
- Simplify rmpopt_work_handler() by removing the explicit-leader
fallback: with CPU hotplug disabled while SNP is active and snp_prepare() requiring all CPUs online when RMPOPT_BASE is programmed, every core is always programmed, so the running CPU can always be the leader. This drops the smp_call_function_single() fallback (and with it the AB-BA deadlock and IRQ-latency concerns) and collapses the leader selection into a single leader-then-followers path.
- Use mod_delayed_work() in snp_rmpopt_all_physmem() so the batching delay tracks the last SNP guest termination.

Sashiko AI code review identified several of the above issues.

v7:
- Sync tools/arch/x86/include/asm/cpufeatures.h to mirror the kernel header for X86_FEATURE_RMPOPT.
- Fix commit title to use X86_FEATURE_RMPOPT to match the code (was X86_FEATURE_AMD_RMPOPT).
- Add static bool rmpopt_configured, set only when segmented RMP setup succeeds in setup_rmptable(). Check rmpopt_configured alongside cpu_feature_enabled(X86_FEATURE_RMPOPT) in snp_setup_rmpopt() and snp_rmpopt_all_physmem(), because setup_clear_cpu_cap() is unreliable after alternatives are patched. Add snp_clear_rmpopt_configured() called from amd_cc_platform_clear() when CC_ATTR_HOST_SEV_SNP is cleared. Do not use __ro_after_init on rmpopt_configured since the writer snp_clear_rmpopt_configured() is not __init.
- Add cond_resched() to all three leader loops in rmpopt_work_handler() to prevent soft lockups on systems with up to 2TB of RAM.
- Add comment above __rmpopt() documenting the RMPOPT instruction encoding (F2 0F 01 FC) and register interface (RAX = system physical address input, RCX = operation type input, RFLAGS.CF = output).
Note: RMPOPT does not modify RAX unlike PVALIDATE/RMPUPDATE, so the existing "a" (input-only) constraint is correct.

Sashiko AI code review identified several of the above issues.

v6:
- Drop wrmsrq_on_cpus() helper; use for_each_cpu() with wrmsrq_on_cpu() instead, as RMPOPT_BASE MSR programming is not performance-critical.
- Rewrite rmpopt_work_handler() leader selection to use a local follower_mask copy instead of modifying the global rmpopt_cpumask. This eliminates the current_cpu_cleared tracking and the restore at the end, and removes the need for synchronization comments about transient cpumask inconsistency.
- Add three-way leader selection in rmpopt_work_handler():
1. Current CPU is a primary thread in cpumask: run leader locally.
2. Current CPU is a sibling thread whose primary is in cpumask:
run leader locally (RMPOPT_BASE MSR is per-core), remove the primary from followers via cpumask_andnot(topology_sibling_cpumask).
3. Current CPU's core has no RMPOPT_BASE MSR programmed: pick an explicit leader via cpumask_first() + smp_call_function_single() to avoid #UD, with cpus_read_lock() around the IPI loop.
- Add WARN_ON_ONCE guard for empty cpumask in the explicit leader fallback path, with migrate_enable() before goto out.
- Add .llseek = seq_lseek to rmpopt_table_fops for consistency with other seq_file-based debugfs files and to support tools like "less".
- Change debugfs file permissions from 0444 to 0400 to restrict access to root only.
- Add comment in rmpopt_table_seq_show() explaining why cpu_online_mask is safe: RMPOPT_BASE MSR is per-core and snp_prepare() ensures all CPUs are online when the MSR is programmed.

Sashiko AI code review identified several of the above issues.

v5:
- Introduce rmpopt_cleanup() to tear down workqueue, debugfs, cpumask, and MSR state, called from snp_shutdown().
- Introduce rmpopt_wq_mutex to serialize snp_setup_rmpopt(), snp_rmpopt_all_physmem(), and rmpopt_cleanup().
- Introduce rmpopt_show_mutex to serialize debugfs reporting of rmpopt_report_cpumask.
- Move snp_rmpopt_all_physmem() call after SNP DECOMMISSION during guest shutdown.
- Use migrate_disable()/migrate_enable() for CPU pinning in the rmpopt_work_handler() leader loop to maintain CPU affinity without disabling preemption for the entire RMPOPT scan.
- Add cpus_read_lock()/cpus_read_unlock() around the follower on_each_cpu_mask() loop in rmpopt_work_handler().
- Guard snp_setup_rmpopt() against re-initialization when SNP_SHUTDOWN_EX with x86_snp_shutdown=0 skips rmpopt_cleanup() but clears snp_initialized, preventing workqueue and resource leaks on repeated init/shutdown cycles.
- Replace setup_clear_cpu_cap() with pr_err() on alloc_workqueue() failure in snp_setup_rmpopt(), as setup_clear_cpu_cap() cannot be used after alternatives are patched; callers check rmpopt_wq != NULL as the runtime guard instead.
- Add pr_info() when RMPOPT coverage is capped at 2TB.
- Add comments noting CPU hotplug is not supported with SNP enabled and only online primary threads are covered by rmpopt_cpumask.
- Add comment in setup_rmptable() noting Segmented RMP must be enabled to enable RMPOPT.
- Simplify cpumask setup loop to set if primary thread rather than skip if not primary.
- Improve grammar and clarity in snp_setup_rmpopt() comments.
- Added Reviewed-by's.

Sashiko AI code review identified several of the above issues.

v4:
- Add new wrmsrq_on_cpus() helper to write same u64 value to a per-CPU MSR across a cpumask without per-cpu struct allocation overhead.
- Rename configure_and_enable_rmpopt() to snp_setup_rmpopt().
- Use wrmsrq_on_cpus() instead of wrmsrq_on_cpu() loop for programming RMPOPT_BASE MSRs.
- Add setup_clear_cpu_cap(X86_FEATURE_RMPOPT) if segmented RMP setup fails or workqueue allocation fails.
- Add X86_FEATURE_RMPOPT feature clear logic in amd_cc_platform_clear() for CC_ATTR_HOST_SEV_SNP.
- All of the above allow checking for only X86_FEATURE_RMPOPT for both RMPOPT setup/enable and RMP re-optimizations.
- Rename snp_perform_rmp_optimization() to snp_rmpopt_all_physmem().
- Split rmpopt() into rmpopt() and rmpopt_smp() for SMP callback use.
- Introduce separate rmpopt_report_cpumask for debugfs reporting, distinct from rmpopt_cpumask used for primary thread tracking.
- Remove snp_perform_rmp_optimization() call from __sev_snp_init_locked() and instead setup and enable RMPOPT after SNP is enabled and initialized.

v3:
- Drop all RMPOPT kthread support and introduce adding custom and dedicated workqueue to schedule delayed and asynchronous RMPOPT work.
- Drop the guest_memfd inode cleanup interface and add support to re-enable RMP optimizations during guest shutdown using the asynchronous and delayed workqueue interface.
- Introduce new __rmpopt() helper and rmpopt() and rmpopt_report_status() wrappers on top which use rax and rcx parameters to closely match RMPOPT specs.
- Use new optimized RMPOPT loop to issue RMPOPT instructions on all system RAM upto 2TB and all CPUs, by optimizing each range on one CPU first, then let other CPUs execute RMPOPT in parallel so they can skip most work as the range has already been optimized.
- Also add support for running the optimized RMPOPT loop only on one thread per core.
- Replace all PUD_SIZE references with SZ_1G to conform to 1GB regions as specified by RMPOPT specifications and not be dependent on PUD_SIZE which makes the RMPOPT patch-set independent of x86 page table sizes.
- Use wrmsrq_on_cpu() to program the RMPOPT_BASE MSR registers on all CPUs that removes all ugly casting to use on_each_cpu_mask().
- Fix inline commits and patch commit messages

v2:
- Drop all NUMA and Socket configuration and enablement support and enable RMPOPT support for up to 2TB of system RAM.
- Drop get_cpumask_of_primary_threads() and enable per-core RMPOPT base MSRs and issue RMPOPT instruction on all CPUs.
- Drop the configfs interface to manually re-enable RMP optimizations.
- Add new guest_memfd cleanup interface to automatically re-enable RMP optimizations during guest shutdown.
- Include references to the public RMPOPT documentation.
- Move debugfs directory for RMPOPT under architecuture specific parent directory.

Ashish Kalra (5):
x86/cpufeatures: Add X86_FEATURE_RMPOPT feature flag x86/sev: Disable CPU hotplug while SNP is active x86/sev: Initialize RMPOPT configuration MSRs x86/sev: Perform RMP optimizations asynchronously x86/sev: Re-enable RMP optimizations on SNP guest shutdown

 arch/x86/include/asm/cpufeatures.h |   2 +-
 arch/x86/include/asm/msr-index.h   |   3 +
 arch/x86/include/asm/sev.h         |   2 +
 arch/x86/kernel/cpu/scattered.c    |   1 +
 arch/x86/kvm/svm/sev.c             |   2 +
 arch/x86/virt/svm/sev.c            | 193 ++++++++++++++++++++++++++---
 drivers/crypto/ccp/sev-dev.c       |   2 +
 7 files changed, 189 insertions(+), 16 deletions(-)