Skip to contentSkip to conversation

efi/unaccepted: touch soft lockup during memory accept

PATCH Applied Chen Yu 4 participants 7 messages Apr 11, 2024 – May 6, 2024 EFI

lore
Share & tools
lore

Chen Yu

author
#0 lore raw

PATCH v2 1/1 efi/unaccepted: touch soft lockup during memory accept

Commit 50e782a86c98 ("efi/unaccepted: Fix soft lockups caused by parallel memory acceptance") has released the spinlock so other CPUs can do memory acceptance in parallel and not triggers softlockup on other CPUs.

However the softlock up was intermittent shown up if the memory of the TD guest is large, and the timeout of softlockup is set to 1 second.

The symptom is:
When the local irq is enabled at the end of accept_memory(), the softlockup detects that the watchdog on single CPU has not been fed for a while. That is to say, even other CPUs will not be blocked by spinlock, the current CPU might be stunk with local irq disabled for a while, which hurts not only nmi watchdog but also softlockup.

Chao Gao pointed out that the memory accept could be time costly and there was similar report before. Thus to avoid any softlocup detection during this stage, give the softlockup a flag to skip the timeout check at the end of accept_memory(), by invoking touch_softlockup_watchdog().

Fixes 50e782a86c98 ("efi/unaccepted: Fix soft lockups caused by parallel memory acceptance")

---
v1 -> v2:
Refine the commit log and add fixes tag/reviewed-by tag from Kirill.

---
 drivers/firmware/efi/unaccepted_memory.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/firmware/efi/unaccepted_memory.c +4 -0
index 5b439d04079c..50f6503fe49f 100644
@@ -4,6 +4,7 @@
 #include <linux/memblock.h>
 #include <linux/spinlock.h>
 #include <linux/crash_dump.h>
+#include <linux/nmi.h>
 #include <asm/unaccepted_memory.h>
 
 /* Protects unaccepted memory bitmap and accepting_list */
@@ -149,6 +150,9 @@ void accept_memory(phys_addr_t start, phys_addr_t end)
 	}
 
 	list_del(&range.list);
+
+	touch_softlockup_watchdog();
+
 	spin_unlock_irqrestore(&unaccepted_memory_lock, flags);
 }
 
Signature · 1 line
2.25.1
author
#1 lore raw
Quoted 23 lines · Chen Yu

Commit 50e782a86c98 ("efi/unaccepted: Fix soft lockups caused by parallel memory acceptance") has released the spinlock so other CPUs can do memory acceptance in parallel and not triggers softlockup on other CPUs.

However the softlock up was intermittent shown up if the memory of the TD guest is large, and the timeout of softlockup is set to 1 second.

The symptom is:
When the local irq is enabled at the end of accept_memory(), the softlockup detects that the watchdog on single CPU has not been fed for a while. That is to say, even other CPUs will not be blocked by spinlock, the current CPU might be stunk with local irq disabled for a while, which hurts not only nmi watchdog but also softlockup.

Chao Gao pointed out that the memory accept could be time costly and there was similar report before. Thus to avoid any softlocup detection during this stage, give the softlockup a flag to skip the timeout check at the end of accept_memory(), by invoking touch_softlockup_watchdog().

Quoted 7 lines
Fixes 50e782a86c98 ("efi/unaccepted: Fix soft lockups caused by parallel memory acceptance")

---
v1 -> v2:
Refine the commit log and add fixes tag/reviewed-by tag from Kirill.

Gently pinging about this patch.

thanks,
Chenyu

maintainer Applied
#2 lore raw
Quoted 33 lines · Chen Yu
Quoted 23 lines · Chen Yu

Commit 50e782a86c98 ("efi/unaccepted: Fix soft lockups caused by parallel memory acceptance") has released the spinlock so other CPUs can do memory acceptance in parallel and not triggers softlockup on other CPUs.

However the softlock up was intermittent shown up if the memory of the TD guest is large, and the timeout of softlockup is set to 1 second.

The symptom is:
When the local irq is enabled at the end of accept_memory(), the softlockup detects that the watchdog on single CPU has not been fed for a while. That is to say, even other CPUs will not be blocked by spinlock, the current CPU might be stunk with local irq disabled for a while, which hurts not only nmi watchdog but also softlockup.

Chao Gao pointed out that the memory accept could be time costly and there was similar report before. Thus to avoid any softlocup detection during this stage, give the softlockup a flag to skip the timeout check at the end of accept_memory(), by invoking touch_softlockup_watchdog().

Quoted 7 lines
Fixes 50e782a86c98 ("efi/unaccepted: Fix soft lockups caused by parallel memory acceptance")

---
v1 -> v2:
Refine the commit log and add fixes tag/reviewed-by tag from Kirill.

Quoted 2 lines

Gently pinging about this patch.

Queued up in efi/urgent now, thanks.

maintainer
#3 lore raw
Quoted 35 lines · Ard Biesheuvel
Quoted 33 lines · Chen Yu
Quoted 30 lines · Chen Yu

Commit 50e782a86c98 ("efi/unaccepted: Fix soft lockups caused by parallel memory acceptance") has released the spinlock so other CPUs can do memory acceptance in parallel and not triggers softlockup on other CPUs.

However the softlock up was intermittent shown up if the memory of the TD guest is large, and the timeout of softlockup is set to 1 second.

The symptom is:
When the local irq is enabled at the end of accept_memory(), the softlockup detects that the watchdog on single CPU has not been fed for a while. That is to say, even other CPUs will not be blocked by spinlock, the current CPU might be stunk with local irq disabled for a while, which hurts not only nmi watchdog but also softlockup.

Chao Gao pointed out that the memory accept could be time costly and there was similar report before. Thus to avoid any softlocup detection during this stage, give the softlockup a flag to skip the timeout check at the end of accept_memory(), by invoking touch_softlockup_watchdog().

Fixes 50e782a86c98 ("efi/unaccepted: Fix soft lockups caused by parallel memory acceptance")

---
v1 -> v2:
Refine the commit log and add fixes tag/reviewed-by tag from Kirill.

Quoted 4 lines
Quoted 2 lines

Gently pinging about this patch.

Queued up in efi/urgent now, thanks.

OK, I was about to send this patch to Linus (and I am still going to).

However, I do wonder if sprinkling touch_softlockup_watchdog() left and right is really the right solution here.

Looking at the backtrace, this is a page fault originating in user space. So why do we end up calling into the hypervisor to accept a chunk of memory large enough to trigger the softlockup watchdog? Or is the hypercall simply taking a disproportionate amount of time?

And AIUI, touch_softlockup_watchdog() hides the fact that we are hogging the CPU for way too long - is there any way we can at least yield the CPU on this condition?

#4 lore raw
Quoted 43 lines · Ard Biesheuvel
Quoted 35 lines · Ard Biesheuvel
Quoted 33 lines · Chen Yu
Quoted 30 lines · Chen Yu

Commit 50e782a86c98 ("efi/unaccepted: Fix soft lockups caused by parallel memory acceptance") has released the spinlock so other CPUs can do memory acceptance in parallel and not triggers softlockup on other CPUs.

However the softlock up was intermittent shown up if the memory of the TD guest is large, and the timeout of softlockup is set to 1 second.

The symptom is:
When the local irq is enabled at the end of accept_memory(), the softlockup detects that the watchdog on single CPU has not been fed for a while. That is to say, even other CPUs will not be blocked by spinlock, the current CPU might be stunk with local irq disabled for a while, which hurts not only nmi watchdog but also softlockup.

Chao Gao pointed out that the memory accept could be time costly and there was similar report before. Thus to avoid any softlocup detection during this stage, give the softlockup a flag to skip the timeout check at the end of accept_memory(), by invoking touch_softlockup_watchdog().

Fixes 50e782a86c98 ("efi/unaccepted: Fix soft lockups caused by parallel memory acceptance")

---
v1 -> v2:
Refine the commit log and add fixes tag/reviewed-by tag from Kirill.

Quoted 4 lines
Quoted 2 lines

Gently pinging about this patch.

Queued up in efi/urgent now, thanks.

OK, I was about to send this patch to Linus (and I am still going to).

Quoted 7 lines

However, I do wonder if sprinkling touch_softlockup_watchdog() left and right is really the right solution here.

Looking at the backtrace, this is a page fault originating in user space. So why do we end up calling into the hypervisor to accept a chunk of memory large enough to trigger the softlockup watchdog? Or is the hypercall simply taking a disproportionate amount of time?

Note that softlockup timeout was set to 1 second to trigger this. So this is exaggerated case.

Quoted 3 lines

And AIUI, touch_softlockup_watchdog() hides the fact that we are hogging the CPU for way too long - is there any way we can at least yield the CPU on this condition?

Not really. There's no magic entity that handles accept. It is done by CPU.

There's a feature in pipeline that makes page accept interruptable in TDX guest. It can help some cases. But if ended up in this codepath from non-preemptable context, it won't help.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
author
#5 lore raw
Quoted 59 lines · Kirill A. Shutemov
Quoted 43 lines · Ard Biesheuvel
Quoted 35 lines · Ard Biesheuvel
Quoted 33 lines · Chen Yu
Quoted 30 lines · Chen Yu

Commit 50e782a86c98 ("efi/unaccepted: Fix soft lockups caused by parallel memory acceptance") has released the spinlock so other CPUs can do memory acceptance in parallel and not triggers softlockup on other CPUs.

However the softlock up was intermittent shown up if the memory of the TD guest is large, and the timeout of softlockup is set to 1 second.

The symptom is:
When the local irq is enabled at the end of accept_memory(), the softlockup detects that the watchdog on single CPU has not been fed for a while. That is to say, even other CPUs will not be blocked by spinlock, the current CPU might be stunk with local irq disabled for a while, which hurts not only nmi watchdog but also softlockup.

Chao Gao pointed out that the memory accept could be time costly and there was similar report before. Thus to avoid any softlocup detection during this stage, give the softlockup a flag to skip the timeout check at the end of accept_memory(), by invoking touch_softlockup_watchdog().

Fixes 50e782a86c98 ("efi/unaccepted: Fix soft lockups caused by parallel memory acceptance")

---
v1 -> v2:
Refine the commit log and add fixes tag/reviewed-by tag from Kirill.

Quoted 4 lines
Quoted 2 lines

Gently pinging about this patch.

Queued up in efi/urgent now, thanks.

OK, I was about to send this patch to Linus (and I am still going to).

Quoted 7 lines

However, I do wonder if sprinkling touch_softlockup_watchdog() left and right is really the right solution here.

Looking at the backtrace, this is a page fault originating in user space. So why do we end up calling into the hypervisor to accept a chunk of memory large enough to trigger the softlockup watchdog? Or is the hypercall simply taking a disproportionate amount of time?

Note that softlockup timeout was set to 1 second to trigger this. So this is exaggerated case.

Quoted 3 lines

And AIUI, touch_softlockup_watchdog() hides the fact that we are hogging the CPU for way too long - is there any way we can at least yield the CPU on this condition?

Quoted 7 lines

Not really. There's no magic entity that handles accept. It is done by CPU.

There's a feature in pipeline that makes page accept interruptable in TDX guest. It can help some cases. But if ended up in this codepath from non-preemptable context, it won't help.

Is it possible to enable the local irq for a little while after each arch_accept_memory(phys_start, phys_end),
and even split the [phys_start,phys_end] to smaller regions? so the watchdog can be fed on time/tick is normal. But currently the softlock fed at the end seems to be more easier to implement.

thanks,
Chenyu

#6 lore raw
Quoted 68 lines · Chen Yu
Quoted 59 lines · Kirill A. Shutemov
Quoted 43 lines · Ard Biesheuvel
Quoted 35 lines · Ard Biesheuvel
Quoted 33 lines · Chen Yu
Quoted 30 lines · Chen Yu

Commit 50e782a86c98 ("efi/unaccepted: Fix soft lockups caused by parallel memory acceptance") has released the spinlock so other CPUs can do memory acceptance in parallel and not triggers softlockup on other CPUs.

However the softlock up was intermittent shown up if the memory of the TD guest is large, and the timeout of softlockup is set to 1 second.

The symptom is:
When the local irq is enabled at the end of accept_memory(), the softlockup detects that the watchdog on single CPU has not been fed for a while. That is to say, even other CPUs will not be blocked by spinlock, the current CPU might be stunk with local irq disabled for a while, which hurts not only nmi watchdog but also softlockup.

Chao Gao pointed out that the memory accept could be time costly and there was similar report before. Thus to avoid any softlocup detection during this stage, give the softlockup a flag to skip the timeout check at the end of accept_memory(), by invoking touch_softlockup_watchdog().

Fixes 50e782a86c98 ("efi/unaccepted: Fix soft lockups caused by parallel memory acceptance")

---
v1 -> v2:
Refine the commit log and add fixes tag/reviewed-by tag from Kirill.

Quoted 4 lines
Quoted 2 lines

Gently pinging about this patch.

Queued up in efi/urgent now, thanks.

OK, I was about to send this patch to Linus (and I am still going to).

Quoted 7 lines

However, I do wonder if sprinkling touch_softlockup_watchdog() left and right is really the right solution here.

Looking at the backtrace, this is a page fault originating in user space. So why do we end up calling into the hypervisor to accept a chunk of memory large enough to trigger the softlockup watchdog? Or is the hypercall simply taking a disproportionate amount of time?

Note that softlockup timeout was set to 1 second to trigger this. So this is exaggerated case.

Quoted 3 lines

And AIUI, touch_softlockup_watchdog() hides the fact that we are hogging the CPU for way too long - is there any way we can at least yield the CPU on this condition?

Quoted 7 lines

Not really. There's no magic entity that handles accept. It is done by CPU.

There's a feature in pipeline that makes page accept interruptable in TDX guest. It can help some cases. But if ended up in this codepath from non-preemptable context, it won't help.

Quoted 5 lines

Is it possible to enable the local irq for a little while after each arch_accept_memory(phys_start, phys_end),
and even split the [phys_start,phys_end] to smaller regions? so the watchdog can be fed on time/tick is normal. But currently the softlock fed at the end seems to be more easier to implement.

That's what I did initially. But Vlastimil correctly pointed that it will lead to deadlock.

https://lore.kernel.org/all/[email protected]/

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov