efi/runtime-wrappers: bound the wait for EFI runtime service calls
v3 cover efi/runtime-wrappers: bound the wait for EFI runtime service calls
When an EFI runtime service call hangs in firmware, the kworker on efi_rts_wq is stuck inside the firmware call and cannot be cancelled. The kernel currently waits indefinitely on the completion, and the caller holds efi_runtime_lock for the duration, so every subsequent EFI runtime caller (efivarfs, NVRAM writes, set_wakeup_time, ACPI PRM handlers, ...) is wedged until reboot. The only externally visible symptom is a "workqueue lockup" message and userspace processes piling up uninterruptibly on the semaphore.
A real example from one of our NVIDIA Grace hosts:
BUG: workqueue lockup - pool cpus=28 node=0 flags=0x0 nice=0 stuck for 127s! ... CPU: 28 PID: 590 Comm: kworker/u288:6 Workqueue: efi_rts_wq efi_call_rts Call trace: 0x4052f11ecc (P) 0x4052f10ed4 ... __efi_rt_asm_wrapper+0x50/0x78 efi_call_rts+0x178/0x240 process_scheduled_works+0x17c/0x420 worker_thread+0x184/0x4d8 kthread+0xcc/0x1f8 ret_from_fork+0x10/0x20
PC and LR are inside EFI runtime services firmware memory; firmware never returned; the worker stayed stuck across the 127s / 157s / 188s "workqueue lockup" reports until external monitoring eventually rebooted the host.
This series doesn't fix the firmware bug - that's vendor territory - but it stops one stuck EFI call from taking the rest of userspace down with it, and turns a generic stalled-task mystery into an unambiguous "EFI firmware is at fault" signal in dmesg, which is especially valuable at fleet scale where the same symptom could otherwise be attributed to dozens of unrelated stalls.
---
Changes in v3:
- Fix two review-flagged races (entry-time worker park vs. freed args; non-blocking EFI_RUNTIME_SERVICES check now under efi_runtime_lock)
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Drop v1's efi_rts_dead flag; reuse the existing EFI_RUNTIME_SERVICES bit (cleared on timeout) and return EFI_ABORTED instead of EFI_TIMEOUT (per Ard).
- Also guard the non-blocking paths (set_variable/query_variable_info/reset_system) and park the leaked worker via a shared efi_rts_park_worker() reused by x86's
page-fault handler;
- Split into smaller prep patches.
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Breno Leitao (7):
efi: fix stale reference to efi_recover_from_page_fault() efi/runtime-wrappers: factor out efi_rts_park_worker() efi/runtime-wrappers: handle queue_work() failure with goto exit efi/runtime-wrappers: check EFI_RUNTIME_SERVICES before using efi_rts_work efi/runtime-wrappers: bound the wait for EFI runtime service calls efi/runtime-wrappers: honour EFI_RUNTIME_SERVICES in the non-blocking paths efi/runtime-wrappers: retire the worker if a wedged call ever returns
arch/x86/platform/efi/quirks.c | 9 +---- drivers/firmware/efi/runtime-wrappers.c | 68 ++++++++++++++++++++++++++++----- include/linux/efi.h | 6 ++- 3 files changed, 64 insertions(+), 19 deletions(-)
---
base-commit: a87737435cfa134f9cdcc696ba3080759d04cf72
change-id: 20260609-efi_timeout-6f51d5bbcfb7
Best regards,
Signature · 1 line
Breno Leitao <[email protected]>
PATCH v3 1/7 efi: fix stale reference to efi_recover_from_page_fault()
efi_recover_from_page_fault() was renamed to efi_crash_gracefully_on_page_fault(), but the comment above enum efi_rts_ids was not updated. Use the current name.
--- include/linux/efi.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/efi.h +2 -2
@@ -1212,8 +1212,8 @@ efi_call_acpi_prm_handler(efi_status_t (__efiapi *handler_addr)(u64, void *),/* * efi_runtime_service() function identifiers. -* "NONE" is used by efi_recover_from_page_fault() to check if the page -* fault happened while executing an efi runtime service. +* "NONE" is used by efi_crash_gracefully_on_page_fault() to check if the +* page fault happened while executing an efi runtime service. */ enum efi_rts_ids {EFI_NONE ,
Signature · 1 line
2.53.0-Meta
PATCH v3 2/7 efi/runtime-wrappers: factor out efi_rts_park_worker()
In reply to Breno Leitaox86's efi_crash_gracefully_on_page_fault() ends in an infinite schedule() loop so the kworker that faulted in firmware never runs efi_rts_wq again. A later change needs the same "park this worker forever" primitive on the runtime service timeout path, so factor the loop into a shared efi_rts_park_worker() and call it from the x86 page-fault handler.
No functional change.
--- arch/x86/platform/efi/quirks.c | 9 +-------- drivers/firmware/efi/runtime-wrappers.c | 13 +++++++++++++ include/linux/efi.h | 2 ++ 3 files changed, 16 insertions(+), 8 deletions(-)
arch/x86/platform/efi/quirks.c +1 -8
@@ -832,12 +832,5 @@ void efi_crash_gracefully_on_page_fault(unsigned long phys_addr,clear_bit (EFI_RUNTIME_SERVICES , &efi.flags);pr_info ("Froze efi_rts_wq and disabled EFI Runtime Services\n" ); -/* -* Call schedule() in an infinite loop, so that any spurious wake ups -* will never run efi_rts_wq again. -*/ -for (;;) { -set_current_state (TASK_IDLE ); -schedule (); - } +efi_rts_park_worker (); }
drivers/firmware/efi/runtime-wrappers.c +13 -0
@@ -211,6 +211,19 @@ static struct task_struct *efi_runtime_lock_owner; extern struct semaphore __efi_uv_runtime_lock __alias(efi_runtime_lock);#endif +/* +* Park a worker that must never run efi_rts_wq again: EFI runtime services +* have been disabled and its efi_rts_work is abandoned. Loop in schedule() +* so a spurious wakeup cannot resume it. +*/ +void efi_rts_park_worker (void ) +{ +for (;;) { +set_current_state (TASK_IDLE ); +schedule (); + } +} +/* * Calls the appropriate efi_runtime_service() with the appropriate * arguments.
include/linux/efi.h +2 -0
@@ -1256,6 +1256,8 @@ extern struct efi_runtime_work efi_rts_work;/* Workqueue to queue EFI Runtime Services */ extern struct workqueue_struct *efi_rts_wq; +void efi_rts_park_worker (void ); +struct linux_efi_memreserve {int size;// allocated size of the array atomic_t count;// number of entries used
Signature · 1 line
2.53.0-Meta
PATCH v3 3/7 efi/runtime-wrappers: handle queue_work() failure with goto exit
In reply to Breno LeitaoConvert the queue_work() failure path in __efi_queue_work() to a goto exit instead of falling through to the wait and the WARN_ON_ONCE(status == EFI_ABORTED) below it. A failed queue_work() leaves the status at its initial EFI_ABORTED, so that warning would fire even though no call ran; it is meant for a completed call that returned EFI_ABORTED.
No change for the common (successful enqueue) path. This also prepares __efi_queue_work() for the timeout handling added later.
--- drivers/firmware/efi/runtime-wrappers.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
drivers/firmware/efi/runtime-wrappers.c +5 -3
@@ -351,10 +351,12 @@ static efi_status_t __efi_queue_work(enum efi_rts_ids id,* queue_work() returns 0 if work was already on queue, * _ideally_ this should never happen. */ -if (queue_work (efi_rts_wq, &efi_rts_work.work)) -wait_for_completion (&efi_rts_work.efi_rts_comp); -else +if (!queue_work (efi_rts_wq, &efi_rts_work.work)) {pr_err ("Failed to queue work to efi_rts_wq.\n" ); + goto exit; + } + +wait_for_completion (&efi_rts_work.efi_rts_comp);WARN_ON_ONCE (efi_rts_work.status ==EFI_ABORTED ); exit:
Signature · 1 line
2.53.0-Meta
PATCH v3 4/7 efi/runtime-wrappers: check EFI_RUNTIME_SERVICES before using efi_rts_work
In reply to Breno LeitaoMove the EFI_RUNTIME_SERVICES check to the top of __efi_queue_work() and return directly, so a caller that finds runtime services disabled returns without touching the shared efi_rts_work. No functional change.
This prepares for bounding the wait, where a timeout will clear EFI_RUNTIME_SERVICES while the leaked worker still owns efi_rts_work; a later caller must then bail out before reinitialising it.
--- drivers/firmware/efi/runtime-wrappers.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
drivers/firmware/efi/runtime-wrappers.c +5 -6
@@ -333,17 +333,16 @@ static void __nocfi efi_call_rts(struct work_struct *work)static efi_status_t __efi_queue_work (enum efi_rts_ids id,union efi_rts_args *args) { +if (!efi_enabled (EFI_RUNTIME_SERVICES )) { +pr_warn_once ("EFI Runtime Services are disabled!\n" ); +return EFI_DEVICE_ERROR ; + } + efi_rts_work.efi_rts_id = id; efi_rts_work.args = args; efi_rts_work.caller =__builtin_return_address (0 ); efi_rts_work.status =EFI_ABORTED ; -if (!efi_enabled (EFI_RUNTIME_SERVICES )) { -pr_warn_once ("EFI Runtime Services are disabled!\n" ); - efi_rts_work.status =EFI_DEVICE_ERROR ; - goto exit; - } -init_completion (&efi_rts_work.efi_rts_comp);INIT_WORK (&efi_rts_work.work, efi_call_rts);
Signature · 1 line
2.53.0-Meta
PATCH v3 5/7 efi/runtime-wrappers: bound the wait for EFI runtime service calls
In reply to Breno LeitaoWhen an EFI runtime service hangs in firmware, the efi_rts_wq worker is stuck inside the call and cannot be cancelled. __efi_queue_work() then waits on the completion forever while holding efi_runtime_lock, so every later EFI caller is wedged until reboot; the only symptom is a "workqueue lockup" and tasks piling up on the semaphore.
Replace wait_for_completion() with wait_for_completion_timeout() bounded by EFI_RTS_TIMEOUT (120 seconds). On timeout, clear EFI_RUNTIME_SERVICES and return EFI_ABORTED so later callers fail fast at the entry check instead of each paying another 120 seconds. The wedged worker is intentionally leaked and keeps ownership of efi_rts_work.
A worker that only starts running after the timeout would otherwise dereference efi_rts_work.args, now pointing into the caller's freed stack frame, and hand stale pointers to firmware. Park it with efi_rts_park_worker() at the entry of efi_call_rts() when runtime services are already disabled, before it touches args or enters firmware.
Known limitation: a worker already inside firmware when the timeout fires still holds efi_rts_args pointing into the caller's stack frame; if firmware unblocks afterwards and writes the output buffers, they land in reused memory. Firmware hung this long rarely recovers; a follow-up could bounce the buffers through kmalloc.
--- drivers/firmware/efi/runtime-wrappers.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-)
drivers/firmware/efi/runtime-wrappers.c +18 -1
@@ -118,6 +118,14 @@ union efi_rts_args {struct efi_runtime_work efi_rts_work; +/* +* Upper bound on how long we wait for a single EFI runtime service +* call to finish before declaring firmware wedged. Chosen to be longer +* than any plausible legitimate call (including UpdateCapsule on slow +* SPI-NOR) while still bounding userspace wait time. +*/ +#define EFI_RTS_TIMEOUT (120 * HZ) +/* * efi_queue_work: Queue EFI runtime service call and wait for completion * @_rts: EFI runtime service function identifier
@@ -234,6 +242,9 @@ static void __nocfi efi_call_rts(struct work_struct *work)efi_status_t status =EFI_NOT_FOUND ;unsigned long flags; +if (!efi_enabled (EFI_RUNTIME_SERVICES )) +efi_rts_park_worker (); + efi_runtime_lock_owner = current;arch_efi_call_virt_setup ();
@@ -355,7 +366,13 @@ static efi_status_t __efi_queue_work(enum efi_rts_ids id, goto exit; } -wait_for_completion (&efi_rts_work.efi_rts_comp); +if (!wait_for_completion_timeout (&efi_rts_work.efi_rts_comp, +EFI_RTS_TIMEOUT )) { +pr_err ("EFI runtime service %d wedged in firmware; disabling EFI runtime services\n" , + id); +clear_bit (EFI_RUNTIME_SERVICES , &efi.flags); +return EFI_ABORTED ; + }WARN_ON_ONCE (efi_rts_work.status ==EFI_ABORTED ); exit:
Signature · 1 line
2.53.0-Meta
Hi Berno,
Quoted 60 lines
When an EFI runtime service hangs in firmware, the
efi_rts_wqworker is stuck inside the call and cannot be cancelled.__efi_queue_work()then waits on the completion forever while holdingefi_runtime_lock, so every later EFI caller is wedged until reboot; the only symptom is a "workqueue lockup" and tasks piling up on the semaphore.Replace
wait_for_completion()withwait_for_completion_timeout()bounded byEFI_RTS_TIMEOUT(120 seconds). On timeout, clearEFI_RUNTIME_SERVICESand returnEFI_ABORTEDso later callers fail fast at the entry check instead of each paying another 120 seconds. The wedged worker is intentionally leaked and keeps ownership ofefi_rts_work.A worker that only starts running after the timeout would otherwise dereference efi_rts_work.args, now pointing into the caller's freed stack frame, and hand stale pointers to firmware. Park it with
efi_rts_park_worker()at the entry ofefi_call_rts()when runtime services are already disabled, before it touches args or enters firmware.Known limitation: a worker already inside firmware when the timeout fires still holds
efi_rts_argspointing into the caller's stack frame; if firmware unblocks afterwards and writes the output buffers, they land in reused memory. Firmware hung this long rarely recovers; a follow-up could bounce the buffers through kmalloc.Signed-off-by Breno Leitao <[email protected]>--- drivers/firmware/efi/runtime-wrappers.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-)drivers/firmware/efi/runtime-wrappers.c +11
-0@@ -118,6 +118,14 @@ union efi_rts_args {struct efi_runtime_work efi_rts_work; +/* +* Upper bound on how long we wait for a single EFI runtime service +* call to finish before declaring firmware wedged. Chosen to be longer +* than any plausible legitimate call (including UpdateCapsule on slow +* SPI-NOR) while still bounding userspace wait time. +*/ +#define EFI_RTS_TIMEOUT (120 * HZ) +/* * efi_queue_work: Queue EFI runtime service call and wait for completion * @_rts: EFI runtime service function identifier @@ -234,6 +242,9 @@ static void __nocfi efi_call_rts(struct work_struct *work)efi_status_t status =EFI_NOT_FOUND ;unsigned long flags; +if (!efi_enabled (EFI_RUNTIME_SERVICES )) +efi_rts_park_worker (); + efi_runtime_lock_owner = current;arch_efi_call_virt_setup ();@@ -355,7 +366,13 @@ static efi_status_t __efi_queue_work(enum efi_rts_ids id, goto exit;
drivers/firmware/efi/runtime-wrappers.c static efi_status_t __efi_queue_work(enum efi_rts_ids id, · line 375
Quoted 10 lines
drivers/firmware/efi/runtime-wrappers.c +7
-1} -wait_for_completion (&efi_rts_work.efi_rts_comp); +if (!wait_for_completion_timeout (&efi_rts_work.efi_rts_comp, +EFI_RTS_TIMEOUT )) { +pr_err ("EFI runtime service %d wedged in firmware; disabling EFI runtime services\n" , + id); +clear_bit (EFI_RUNTIME_SERVICES , &efi.flags); +return EFI_ABORTED ; + }
One of my concern is for the UpdateCapsule runtime service. It might have a case where takes more then 2 mins and there's one RFC to support a runtime capsule update for arm platform [1].
But I'm not sure how we can discern whether it's a time consuming or for the bug on the firmware.
At least, should EFI_RTS be configurable via Kconfig and boot param (timeout or keep the former behavior -- wait_for_completion without timeout)?
Thanks.
Signature · 2 lines
Sincerely, Yeoreum Yun
Hello Yeoreum,
Quoted 4 lines · Yeoreum Yun
One of my concern is for the UpdateCapsule runtime service. It might have a case where takes more then 2 mins and there's one RFC to support a runtime capsule update for arm platform [1].
I'm not familiar with UpdateCapsule. Are you suggesting that a 2-minute wait for an EFI call should be acceptable (?!)
Wouldn't that trigger stalls and hung-task/wq warnings throughout the kernel?
Quoted 8 lines
Hello Yeoreum,
Quoted 4 lines · Yeoreum Yun
One of my concern is for the UpdateCapsule runtime service. It might have a case where takes more then 2 mins and there's one RFC to support a runtime capsule update for arm platform [1].
Quoted 5 lines
I'm not familiar with UpdateCapsule. Are you suggesting that a 2-minute wait for an EFI call should be acceptable (?!)
Wouldn't that trigger stalls and hung-task/wq warnings throughout the kernel?
Might for default config timeout for hung-task. However at least its timeout configurable.
TBH, normally it wouldn't task more than 2 mins. But for a in case of *slow firmware*, would be good to make runtime-service timeout configurable at least?
Signature · 2 lines
Sincerely, Yeoreum Yun
Quoted 14 lines · Yeoreum Yun
Quoted 8 lines
Hello Yeoreum,
Quoted 4 lines · Yeoreum Yun
One of my concern is for the UpdateCapsule runtime service. It might have a case where takes more then 2 mins and there's one RFC to support a runtime capsule update for arm platform [1].
Quoted 5 lines
I'm not familiar with UpdateCapsule. Are you suggesting that a 2-minute wait for an EFI call should be acceptable (?!)
Wouldn't that trigger stalls and hung-task/wq warnings throughout the kernel?
Quoted 6 lines
Might for default config timeout for hung-task. However at least its timeout configurable.
TBH, normally it wouldn't task more than 2 mins. But for a in case of *slow firmware*, would be good to make runtime-service timeout configurable at least?
OK, that is fair request.
I would suggest we do an additional patch with this change, if Ard is positive on the idea.
Yeoreum, would you like to do it, or should I do it?
Quoted 22 lines · Yeoreum Yun
Quoted 14 lines
Quoted 8 lines
Hello Yeoreum,
Quoted 4 lines · Yeoreum Yun
One of my concern is for the UpdateCapsule runtime service. It might have a case where takes more then 2 mins and there's one RFC to support a runtime capsule update for arm platform [1].
Quoted 5 lines
I'm not familiar with UpdateCapsule. Are you suggesting that a 2-minute wait for an EFI call should be acceptable (?!)
Wouldn't that trigger stalls and hung-task/wq warnings throughout the kernel?
Quoted 6 lines
Might for default config timeout for hung-task. However at least its timeout configurable.
TBH, normally it wouldn't task more than 2 mins. But for a in case of *slow firmware*, would be good to make runtime-service timeout configurable at least?
Quoted 6 lines
OK, that is fair request.
I would suggest we do an additional patch with this change, if Ard is positive on the idea.
Yeoreum, would you like to do it, or should I do it?
It's up to you. I don't care who will do this :)
But if you have no bandwidth, I'll take once your patch is merged and Ard agrees for this.
Thanks!
Signature · 2 lines
Sincerely, Yeoreum Yun
Quoted 24 lines · Yeoreum Yun
Quoted 24 lines · Yeoreum Yun
Quoted 14 lines
Quoted 8 lines
Hello Yeoreum,
Quoted 4 lines · Yeoreum Yun
One of my concern is for the UpdateCapsule runtime service. It might have a case where takes more then 2 mins and there's one RFC to support a runtime capsule update for arm platform [1].
Quoted 5 lines
I'm not familiar with UpdateCapsule. Are you suggesting that a 2-minute wait for an EFI call should be acceptable (?!)
Wouldn't that trigger stalls and hung-task/wq warnings throughout the kernel?
Quoted 6 lines
Might for default config timeout for hung-task. However at least its timeout configurable.
TBH, normally it wouldn't task more than 2 mins. But for a in case of *slow firmware*, would be good to make runtime-service timeout configurable at least?
OK, that is fair request.
Quoted 8 lines
Quoted 4 lines
I would suggest we do an additional patch with this change, if Ard is positive on the idea.
Yeoreum, would you like to do it, or should I do it?
It's up to you. I don't care who will do this :)
But if you have no bandwidth, I'll take once your patch is merged and Ard agrees for this.
Sounds good. Let's wait for Ard's feedback on the additional patches [1], and we can decide how to proceed from there.
Ard Biesheuvel
maintainerQuoted 30 lines · Breno Leitao
Quoted 29 lines · Yeoreum Yun
Quoted 28 lines · Yeoreum Yun
Quoted 14 lines
Quoted 8 lines
Hello Yeoreum,
Quoted 4 lines · Yeoreum Yun
One of my concern is for the UpdateCapsule runtime service. It might have a case where takes more then 2 mins and there's one RFC to support a runtime capsule update for arm platform [1].
Quoted 5 lines
I'm not familiar with UpdateCapsule. Are you suggesting that a 2-minute wait for an EFI call should be acceptable (?!)
Wouldn't that trigger stalls and hung-task/wq warnings throughout the kernel?
Quoted 6 lines
Might for default config timeout for hung-task. However at least its timeout configurable.
TBH, normally it wouldn't task more than 2 mins. But for a in case of *slow firmware*, would be good to make runtime-service timeout configurable at least?
OK, that is fair request.
I would suggest we do an additional patch with this change, if Ard is positive on the idea.
Yeoreum, would you like to do it, or should I do it?
Quoted 8 lines
Quoted 3 lines
It's up to you. I don't care who will do this :)
But if you have no bandwidth, I'll take once your patch is merged and Ard agrees for this.Sounds good. Let's wait for Ard's feedback on the additional patches [1], and we can decide how to proceed from there.
Please send patches against efi/next, I will apply them promptly.
UpdateCapsule is a special case that we can deal with separately
Hello Ard,
Quoted 6 lines · Ard Biesheuvel
Quoted 4 lines
Sounds good. Let's wait for Ard's feedback on the additional patches [1], and we can decide how to proceed from there.
Please send patches against efi/next, I will apply them promptly.
Acknowledged. I'll rebase the fixes on efi/next and send them shortly.
Quoted 1 line
UpdateCapsule is a special case that we can deal with separately
Understood. I understand I should defer the configurable timeout work for now.
Hi Breno,
Quoted 9 lines
Hello Ard,
Quoted 5 lines · Ard Biesheuvel
Quoted 4 lines
Sounds good. Let's wait for Ard's feedback on the additional patches [1], and we can decide how to proceed from there.
Quoted 8 lines
Quoted 1 line
Please send patches against efi/next, I will apply them promptly.
Acknowledged. I'll rebase the fixes on efi/next and send them shortly.
Quoted 1 line
UpdateCapsule is a special case that we can deal with separately
Understood. I understand I should defer the configurable timeout work for now.
If you don't mind, Could I send the patch?
TBH, If we handle the UpdateCapsule seperately,
I think it seems good to make a exception for capsule update to wait unlimit.
Signature · 2 lines
Sincerely, Yeoreum Yun
Ard Biesheuvel
maintainerQuoted 15 lines · Yeoreum Yun
Hi Breno,
Quoted 13 lines
Hello Ard,
Quoted 6 lines · Ard Biesheuvel
Quoted 4 lines
Sounds good. Let's wait for Ard's feedback on the additional patches [1], and we can decide how to proceed from there.
Please send patches against efi/next, I will apply them promptly.
Acknowledged. I'll rebase the fixes on efi/next and send them shortly.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
UpdateCapsule is a special case that we can deal with separately
Understood. I understand I should defer the configurable timeout work for now.
If you don't mind, Could I send the patch?
TBH, If we handle the UpdateCapsule seperately,
I think it seems good to make a exception for capsule update to wait unlimit.
Yes, feel free to send a patch. We'll continue the discussion there.
PATCH v3 6/7 efi/runtime-wrappers: honour EFI_RUNTIME_SERVICES in the non-blocking paths
In reply to Breno LeitaoThree wrappers call firmware directly instead of going through __efi_queue_work(), and none of them check whether runtime services are still enabled: virt_efi_set_variable_nb(), virt_efi_query_variable_info_nb() and virt_efi_reset_system(). Once a hang has cleared EFI_RUNTIME_SERVICES - or efi_recover_from_page_fault() has cleared it on a firmware page fault - these paths still enter the (possibly wedged) firmware, e.g. an EFI pstore write through the non-blocking SetVariable() variant, in violation of UEFI's non-reentrancy rules. reset_system() is reachable too: efi_reboot() only gates it on the static efi_rt_services_supported() mask, which does not track the runtime disable.
Check efi_enabled(EFI_RUNTIME_SERVICES) in each before calling into firmware. Test it after taking efi_runtime_lock rather than before: the bit is only ever cleared at runtime while that lock is held, so checking it under the lock avoids racing with a concurrent timeout that clears the bit and drops the lock.
--- drivers/firmware/efi/runtime-wrappers.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
drivers/firmware/efi/runtime-wrappers.c +16 -0
@@ -480,6 +480,11 @@ virt_efi_set_variable_nb(efi_char16_t *name, efi_guid_t *vendor, u32 attr,if (down_trylock (&efi_runtime_lock))return EFI_NOT_READY ; +if (!efi_enabled (EFI_RUNTIME_SERVICES )) { +up (&efi_runtime_lock); +return EFI_DEVICE_ERROR ; + } + efi_runtime_lock_owner = current; status =efi_call_virt_pointer (efi.runtime, set_variable, name, vendor, attr, data_size, data);
@@ -519,6 +524,11 @@ virt_efi_query_variable_info_nb(u32 attr, u64 *storage_space,if (down_trylock (&efi_runtime_lock))return EFI_NOT_READY ; +if (!efi_enabled (EFI_RUNTIME_SERVICES )) { +up (&efi_runtime_lock); +return EFI_DEVICE_ERROR ; + } + efi_runtime_lock_owner = current; status = efi_call_virt_pointer(efi.runtime, query_variable_info, attr, storage_space, remaining_space,
@@ -549,6 +559,12 @@ virt_efi_reset_system(int reset_type, efi_status_t status,return ; } +if (!efi_enabled (EFI_RUNTIME_SERVICES )) { +pr_warn ("EFI Runtime Services are disabled, not invoking reset_system()\n" ); +up (&efi_runtime_lock); +return ; + } + efi_runtime_lock_owner = current;arch_efi_call_virt_setup (); efi_rts_work.efi_rts_id =EFI_RESET_SYSTEM ;
Signature · 1 line
2.53.0-Meta
PATCH v3 7/7 efi/runtime-wrappers: retire the worker if a wedged call ever returns
In reply to Breno LeitaoWhen __efi_queue_work() times out it disables runtime services and returns, but the kworker is still blocked inside firmware. If the firmware eventually unblocks, efi_call_rts() would run its tail on an efi_rts_work that the timed-out caller has long abandoned: signalling a stale completion and clearing efi_runtime_lock_owner that may by then belong to another caller.
If runtime services have been disabled by the time the call returns, park the worker with efi_rts_park_worker() instead, so it never touches efi_rts_work again or returns to the workqueue.
--- drivers/firmware/efi/runtime-wrappers.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/firmware/efi/runtime-wrappers.c +3 -0
@@ -336,6 +336,9 @@ static void __nocfi efi_call_rts(struct work_struct *work)efi_call_virt_check_flags (flags, efi_rts_work.caller);arch_efi_call_virt_teardown (); +if (!efi_enabled (EFI_RUNTIME_SERVICES )) +efi_rts_park_worker (); + efi_rts_work.status = status;complete (&efi_rts_work.efi_rts_comp); efi_runtime_lock_owner =NULL ;
Signature · 1 line
2.53.0-Meta
Hello Ard,
Quoted 8 lines · Breno Leitao
When an EFI runtime service call hangs in firmware, the kworker on
efi_rts_wqis stuck inside the firmware call and cannot be cancelled. The kernel currently waits indefinitely on the completion, and the caller holdsefi_runtime_lockfor the duration, so every subsequent EFI runtime caller (efivarfs, NVRAM writes,set_wakeup_time, ACPI PRM handlers, ...) is wedged until reboot. The only externally visible symptom is a "workqueue lockup" message and userspace processes piling up uninterruptibly on the semaphore.
I'd appreciate your thoughts on this approach. Is there anything you'd like me to adjust or improve before we move forward?
Thanks,
--breno
Ard Biesheuvel
maintainerQuoted 12 lines · Breno Leitao
Hello Ard,
Quoted 8 lines · Breno Leitao
When an EFI runtime service call hangs in firmware, the kworker on
efi_rts_wqis stuck inside the firmware call and cannot be cancelled. The kernel currently waits indefinitely on the completion, and the caller holdsefi_runtime_lockfor the duration, so every subsequent EFI runtime caller (efivarfs, NVRAM writes,set_wakeup_time, ACPI PRM handlers, ...) is wedged until reboot. The only externally visible symptom is a "workqueue lockup" message and userspace processes piling up uninterruptibly on the semaphore.
Quoted 3 lines
I'd appreciate your thoughts on this approach. Is there anything you'd like me to adjust or improve before we move forward?
This looks fine now, apologies for the delayed response.
I've queued this up for v7.3
Quoted 15 lines · Ard Biesheuvel
Quoted 12 lines · Breno Leitao
Hello Ard,
Quoted 8 lines · Breno Leitao
When an EFI runtime service call hangs in firmware, the kworker on
efi_rts_wqis stuck inside the firmware call and cannot be cancelled. The kernel currently waits indefinitely on the completion, and the caller holdsefi_runtime_lockfor the duration, so every subsequent EFI runtime caller (efivarfs, NVRAM writes,set_wakeup_time, ACPI PRM handlers, ...) is wedged until reboot. The only externally visible symptom is a "workqueue lockup" message and userspace processes piling up uninterruptibly on the semaphore.
Quoted 7 lines
Quoted 3 lines
I'd appreciate your thoughts on this approach. Is there anything you'd like me to adjust or improve before we move forward?
This looks fine now, apologies for the delayed response.
I've queued this up for v7.3
Thanks, Ard.
We got two reports that I would like to address:
1) efi_rts_park_worker() should be tagged as __noreturn
https://lore.kernel.org/all/[email protected]/
2) efi_runtime_lock_owner() is undeclared in some kernel config
https://lore.kernel.org/all/[email protected]/
Do you want me to respin this series, or, some fixes on top of current tree?
Thanks,
--breno