Skip to contentSkip to conversation

SEV-SNP fix for cpu soft lockup on 1TB+ guests

PATCH Applied Liam Merwick 3 participants 6 messages Jun 9, 2025 – Jun 24, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover SEV-SNP fix for cpu soft lockup on 1TB+ guests

When creating SEV-SNP guests with a large amount of memory (940GB or greater) the host experiences a soft cpu lockup while setting the per-page memory attributes on the whole range of memory in the guest.

The underlying issue is that the implementation of setting the memory attributes using an Xarray implementation is a time-consuming operation (e.g. a 1.9TB guest takes over 30 seconds to set the attributes)

Fix the lockup by modifying kvm_vm_set_mem_attributes() to call cond_resched() during the loops to reserve and store the Xarray entries to give the scheduler a chance to run a higher priority task on the runqueue if necessary and avoid staying in kernel mode long enough to trigger the lockup.

Tested with VMs up to 1900GB in size (the limit of hardware available to me)

The functionality was introduced in v6.8 but I tagged as just needing backporting as far as linux-6.12.y (applies cleanly)

Based on tag: kvm-6.16-1

v1 -> v2
Implement suggestion by Sean to use cond_resched() rather than splitting operations into batches.

v1: https://lore.kernel.org/all/[email protected]

Liam Merwick (3):
KVM: Allow CPU to reschedule while setting per-page memory attributes
KVM: Add trace_kvm_vm_set_mem_attributes()
KVM: fix typo in kvm_vm_set_mem_attributes() comment

 include/trace/events/kvm.h | 27 +++++++++++++++++++++++++++
 virt/kvm/kvm_main.c        |  7 ++++++-
 2 files changed, 33 insertions(+), 1 deletion(-)
Signature · 1 line
2.47.1
#1 lore raw

PATCH v2 1/3 KVM: Allow CPU to reschedule while setting per-page memory attributes

When running an SEV-SNP guest with a sufficiently large amount of memory (1TB+), the host can experience CPU soft lockups when running an operation in kvm_vm_set_mem_attributes() to set memory attributes on the whole range of guest memory.

watchdog: BUG: soft lockup - CPU#8 stuck for 26s! [qemu-kvm:6372]
CPU: 8 UID: 0 PID: 6372 Comm: qemu-kvm Kdump: loaded Not tainted 6.15.0-rc7.20250520.el9uek.rc1.x86_64 #1 PREEMPT(voluntary)
Hardware name: Oracle Corporation ORACLE SERVER E4-2c/Asm,MB Tray,2U,E4-2c, BIOS 78016600 11/13/2024
RIP: 0010:xas_create+0x78/0x1f0
Code: 00 00 00 41 80 fc 01 0f 84 82 00 00 00 ba 06 00 00 00 bd 06 00 00 00 49 8b 45 08 4d 8d 65 08 41 39 d6 73 20 83 ed 06 48 85 c0 <74> 67 48 89 c2 83 e2 03 48 83 fa 02 75 0c 48 3d 00 10 00 00 0f 87
RSP: 0018:ffffad890a34b940 EFLAGS: 00000286
RAX: ffff96f30b261daa RBX: ffffad890a34b9c8 RCX: 0000000000000000
RDX: 000000000000001e RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000018 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffad890a356868
R13: ffffad890a356860 R14: 0000000000000000 R15: ffffad890a356868
FS:  00007f5578a2a400(0000) GS:ffff97ed317e1000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f015c70fb18 CR3: 00000001109fd006 CR4: 0000000000f70ef0
PKRU: 55555554
Call Trace:
 <TASK>
 xas_store+0x58/0x630
 __xa_store+0xa5/0x130
 xa_store+0x2c/0x50
 kvm_vm_set_mem_attributes+0x343/0x710 [kvm]
 kvm_vm_ioctl+0x796/0xab0 [kvm]
 __x64_sys_ioctl+0xa3/0xd0
 do_syscall_64+0x8c/0x7a0
 entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7f5578d031bb
Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 2d 4c 0f 00 f7 d8 64 89 01 48
RSP: 002b:00007ffe0a742b88 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 000000004020aed2 RCX: 00007f5578d031bb
RDX: 00007ffe0a742c80 RSI: 000000004020aed2 RDI: 000000000000000b
RBP: 0000010000000000 R08: 0000010000000000 R09: 0000017680000000
R10: 0000000000000080 R11: 0000000000000246 R12: 00005575e5f95120
R13: 00007ffe0a742c80 R14: 0000000000000008 R15: 00005575e5f961e0

While looping through the range of memory setting the attributes, call cond_resched() to give the scheduler a chance to run a higher priority task on the runqueue if necessary and avoid staying in kernel mode long enough to trigger the lockup.

Fixes 5a475554db1e ("KVM: Introduce per-page memory attributes")
Cc [email protected] # 6.12.x
---
 virt/kvm/kvm_main.c | 3 +++
 1 file changed, 3 insertions(+)
virt/kvm/kvm_main.c +3 -0
index b99de3b5ffbc..aba4078ae225 100644
@@ -2557,6 +2557,8 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 		r = xa_reserve(&kvm->mem_attr_array, i, GFP_KERNEL_ACCOUNT);
 		if (r)
 			goto out_unlock;
+
+		cond_resched();
 	}
 
 	kvm_handle_gfn_range(kvm, &pre_set_range);
@@ -2565,6 +2567,7 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 		r = xa_err(xa_store(&kvm->mem_attr_array, i, entry,
 				    GFP_KERNEL_ACCOUNT));
 		KVM_BUG_ON(r, kvm);
+		cond_resched();
 	}
 
 	kvm_handle_gfn_range(kvm, &post_set_range);
Signature · 1 line
2.47.1
#2 lore raw

PATCH v2 2/3 KVM: Add trace_kvm_vm_set_mem_attributes()

In reply to Liam Merwick

Add a tracing function that, for a guest memory range, displays the start and end addresses plus the per-page attributes being set.

---
 include/trace/events/kvm.h | 27 +++++++++++++++++++++++++++
 virt/kvm/kvm_main.c        |  2 ++
 2 files changed, 29 insertions(+)
include/trace/events/kvm.h +27 -0
index 74e40d5d4af4..dfaad67d2c0a 100644
@@ -438,6 +438,33 @@ TRACE_EVENT(kvm_dirty_ring_exit,
 	TP_printk("vcpu %d", __entry->vcpu_id)
 );
 
+#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+/*
+ * @start:	Starting address of guest memory range
+ * @end:	End address of guest memory range
+ * @attr:	The value of the attribute being set.
+ */
+TRACE_EVENT(kvm_vm_set_mem_attributes,
+	TP_PROTO(gfn_t start, gfn_t end, unsigned long attr),
+	TP_ARGS(start, end, attr),
+
+	TP_STRUCT__entry(
+		__field(gfn_t,		start)
+		__field(gfn_t,		end)
+		__field(unsigned long,	attr)
+	),
+
+	TP_fast_assign(
+		__entry->start		= start;
+		__entry->end		= end;
+		__entry->attr		= attr;
+	),
+
+	TP_printk("%#016llx -- %#016llx [0x%lx]",
+		  __entry->start, __entry->end, __entry->attr)
+);
+#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
+
 TRACE_EVENT(kvm_unmap_hva_range,
 	TP_PROTO(unsigned long start, unsigned long end),
 	TP_ARGS(start, end),
virt/kvm/kvm_main.c +2 -0
index aba4078ae225..09d4217410fd 100644
@@ -2543,6 +2543,8 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 
 	entry = attributes ? xa_mk_value(attributes) : NULL;
 
+	trace_kvm_vm_set_mem_attributes(start, end, attributes);
+
 	mutex_lock(&kvm->slots_lock);
 
 	/* Nothing to do if the entire range as the desired attributes. */
Signature · 1 line
2.47.1
#3 lore raw

PATCH v2 3/3 KVM: fix typo in kvm_vm_set_mem_attributes() comment

In reply to Liam Merwick

It should be 'has' in the sentence and not 'as'.

---
 virt/kvm/kvm_main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
virt/kvm/kvm_main.c +1 -1
index 09d4217410fd..d030ebbdc91c 100644
@@ -2547,7 +2547,7 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 
 	mutex_lock(&kvm->slots_lock);
 
-	/* Nothing to do if the entire range as the desired attributes. */
+	/* Nothing to do if the entire range has the desired attributes. */
 	if (kvm_range_has_memory_attributes(kvm, start, end, ~0, attributes))
 		goto out_unlock;
 
Signature · 1 line
2.47.1
Reviewed-by on the v2 series
#4 lore raw
In reply to Liam Merwick
Quoted 26 lines · Liam Merwick

When creating SEV-SNP guests with a large amount of memory (940GB or greater) the host experiences a soft cpu lockup while setting the per-page memory attributes on the whole range of memory in the guest.

The underlying issue is that the implementation of setting the memory attributes using an Xarray implementation is a time-consuming operation (e.g. a 1.9TB guest takes over 30 seconds to set the attributes)

Fix the lockup by modifying kvm_vm_set_mem_attributes() to call cond_resched() during the loops to reserve and store the Xarray entries to give the scheduler a chance to run a higher priority task on the runqueue if necessary and avoid staying in kernel mode long enough to trigger the lockup.

Tested with VMs up to 1900GB in size (the limit of hardware available to me)

The functionality was introduced in v6.8 but I tagged as just needing backporting as far as linux-6.12.y (applies cleanly)

Based on tag: kvm-6.16-1

v1 -> v2
Implement suggestion by Sean to use cond_resched() rather than splitting operations into batches.

v1: https://lore.kernel.org/all/[email protected]

Quoted 9 lines

Liam Merwick (3):
KVM: Allow CPU to reschedule while setting per-page memory attributes
KVM: Add trace_kvm_vm_set_mem_attributes()
KVM: fix typo in kvm_vm_set_mem_attributes() comment

  include/trace/events/kvm.h | 27 +++++++++++++++++++++++++++
  virt/kvm/kvm_main.c        |  7 ++++++-
  2 files changed, 33 insertions(+), 1 deletion(-)

For the series:

reviewer Applied
#5 lore raw
In reply to Liam Merwick
Quoted 9 lines · Liam Merwick

When creating SEV-SNP guests with a large amount of memory (940GB or greater) the host experiences a soft cpu lockup while setting the per-page memory attributes on the whole range of memory in the guest.

The underlying issue is that the implementation of setting the memory attributes using an Xarray implementation is a time-consuming operation (e.g. a 1.9TB guest takes over 30 seconds to set the attributes)

[...]

Applied patch 1 to kvm-x86 fixes, and the others to 'kvm-x86 generic'. Thanks!

[1/3] KVM: Allow CPU to reschedule while setting per-page memory attributes

      https://github.com/kvm-x86/linux/commit/47bb584237cc
[2/3] KVM: Add trace_kvm_vm_set_mem_attributes()
      https://github.com/kvm-x86/linux/commit/741e595f02fe
[3/3] KVM: fix typo in kvm_vm_set_mem_attributes() comment
      https://github.com/kvm-x86/linux/commit/aa006b2e5159

--
https://github.com/kvm-x86/kvm-unit-tests/tree/next