arm64: efi: Disable only the misbehaving runtime service on sync exceptions
Ard Biesheuvel
maintainerPATCH 1/1 arm64: efi: Disable only the misbehaving runtime service on sync exceptions
Alexandru reports that his Ampere Altra machine, whose buggy firmware triggers a synchronous exception in its implementation of SetTime() when called without SetVirtualAddressMap() having been called first, doesn't quite recover from this, and starts spewing error messages into the log that are unrelated to the buggy runtime service.
The driver in question is the EFI RTC driver, which should be fixed in any case, as flooding the log like that (or doing any logging to the kernel log at all on something whuch is not a severe issue) is not ok.
However, in this particular case, it would be beneficial for both ordinary use as well as diagnostics regarding broken firmware if we only prevent the broken runtime service from being called again, and permit others (such as GetTime() which triggers the logging or the variable services) from being used as normal.
So wire up the existing efi.runtime_supported_mask, and clear the service's bit in the mask if the generic runtime service wrapper observes a return value of EFI_ABORTED, which only happens if a service call is aborted due to an exception. (EFI_ABORTED is not documented as a valid error code for any of the EFI runtime services).
--- arch/arm64/kernel/efi.c | 1 - arch/x86/platform/efi/quirks.c | 2 +- drivers/firmware/efi/runtime-wrappers.c | 6 ++++-- include/linux/efi.h | 24 ++++++++++++------------ 4 files changed, 17 insertions(+), 16 deletions(-)
arch/arm64/kernel/efi.c +0 -1
@@ -153,7 +153,6 @@ asmlinkage efi_status_t __efi_rt_asm_recover(void); asmlinkage efi_status_t efi_handle_runtime_exception(const char *f) {pr_err (FW_BUG "Synchronous exception occurred in EFI runtime service %s()\n" , f); -clear_bit (EFI_RUNTIME_SERVICES , &efi.flags);return EFI_ABORTED ; }
arch/x86/platform/efi/quirks.c +1 -1
@@ -757,9 +757,9 @@ void efi_crash_gracefully_on_page_fault(unsigned long phys_addr)/* Signal error status to the efi caller process */ efi_rts_work.status =EFI_ABORTED ; + efi.runtime_supported_mask =0 ;complete (&efi_rts_work.efi_rts_comp); -clear_bit (EFI_RUNTIME_SERVICES , &efi.flags);pr_info ("Froze efi_rts_wq and disabled EFI Runtime Services\n" );/*
drivers/firmware/efi/runtime-wrappers.c +4 -2
@@ -60,8 +60,8 @@ struct efi_runtime_work efi_rts_work; ({ \ efi_rts_work.status = EFI_ABORTED; \ \ - if (!efi_enabled(EFI_RUNTIME_SERVICES)) { \ - pr_warn_once("EFI Runtime Services are disabled!\n"); \ + if (!efi_rt_services_supported(efi_rts_work.efi_rts_id)) { \ + pr_warn_ratelimited("EFI Runtime Service is disabled!\n");\ goto exit; \ } \ \
@@ -83,6 +83,8 @@ struct efi_runtime_work efi_rts_work; else \ pr_err("Failed to queue work to efi_rts_wq.\n"); \ \ +if (efi_rts_work.status ==EFI_ABORTED ) \ + efi.runtime_supported_mask &= ~efi_rts_work.efi_rts_id; \ exit: \ efi_rts_work.efi_rts_id =EFI_NONE ; \ efi_rts_work.status; \
include/linux/efi.h +12 -12
@@ -1254,18 +1254,18 @@ extern unsigned long rci2_table_phys;*/ enum efi_rts_ids {EFI_NONE , -EFI_GET_TIME , -EFI_SET_TIME , -EFI_GET_WAKEUP_TIME , -EFI_SET_WAKEUP_TIME , -EFI_GET_VARIABLE , -EFI_GET_NEXT_VARIABLE , -EFI_SET_VARIABLE , -EFI_QUERY_VARIABLE_INFO , -EFI_GET_NEXT_HIGH_MONO_COUNT , -EFI_RESET_SYSTEM , -EFI_UPDATE_CAPSULE , -EFI_QUERY_CAPSULE_CAPS , +EFI_GET_TIME =EFI_RT_SUPPORTED_GET_TIME , +EFI_SET_TIME =EFI_RT_SUPPORTED_SET_TIME , +EFI_GET_WAKEUP_TIME =EFI_RT_SUPPORTED_GET_WAKEUP_TIME , +EFI_SET_WAKEUP_TIME =EFI_RT_SUPPORTED_SET_WAKEUP_TIME , +EFI_GET_VARIABLE =EFI_RT_SUPPORTED_GET_VARIABLE , +EFI_GET_NEXT_VARIABLE =EFI_RT_SUPPORTED_GET_NEXT_VARIABLE_NAME , +EFI_SET_VARIABLE =EFI_RT_SUPPORTED_SET_VARIABLE , +EFI_GET_NEXT_HIGH_MONO_COUNT =EFI_RT_SUPPORTED_GET_NEXT_HIGH_MONOTONIC_COUNT , +EFI_RESET_SYSTEM =EFI_RT_SUPPORTED_RESET_SYSTEM , +EFI_UPDATE_CAPSULE =EFI_RT_SUPPORTED_UPDATE_CAPSULE , +EFI_QUERY_CAPSULE_CAPS =EFI_RT_SUPPORTED_QUERY_CAPSULE_CAPABILITIES , +EFI_QUERY_VARIABLE_INFO =EFI_RT_SUPPORTED_QUERY_VARIABLE_INFO , };/*
Signature · 1 line
2.35.1
Hi,
Quoted 16 lines · Ard Biesheuvel
Alexandru reports that his Ampere Altra machine, whose buggy firmware triggers a synchronous exception in its implementation of
SetTime()when called withoutSetVirtualAddressMap()having been called first, doesn't quite recover from this, and starts spewing error messages into the log that are unrelated to the buggy runtime service.The driver in question is the EFI RTC driver, which should be fixed in any case, as flooding the log like that (or doing any logging to the kernel log at all on something whuch is not a severe issue) is not ok.
However, in this particular case, it would be beneficial for both ordinary use as well as diagnostics regarding broken firmware if we only prevent the broken runtime service from being called again, and permit others (such as
GetTime()which triggers the logging or the variable services) from being used as normal.
Quoted 5 lines
So wire up the existing efi.runtime_supported_mask, and clear the service's bit in the mask if the generic runtime service wrapper observes a return value of
EFI_ABORTED, which only happens if a service call is aborted due to an exception. (EFI_ABORTEDis not documented as a valid error code for any of the EFI runtime services).
With a kernel built from v6.1-rc4, when doing efibootmgr after the EFI panic happens (so with runtime services disabled), this is what I get:
# efibootmgr Skipping unreadable variable "Boot0001": Interrupted system call Skipping unreadable variable "Boot0002": Interrupted system call show_order(): Interrupted system call
And dmesg shows:
[ 55.941312] efi: EFI Runtime Services are disabled!
With this patch on top of v6.1-rc4:
# efibootmgr Skipping unreadable variable "Boot0001": Invalid argument Skipping unreadable variable "Boot0002": Invalid argument show_order(): Invalid argument
Same thing happens if I cat the Boot001 efivars file. Nothing is printed on dmesg.
Changed efi_call_rts() to print the return value, status is 0x8000_0000_0000_000f (or 15 in decimal if casted into an int). Tried to debug further, but I'm not familiar with all the structs and what they represent (for example, efi_call_virt(get_variable, args) calls efi_call_virt_pointer(efi.runtime, get_variable, args), does it end up as __efi_rt_asm_wrapper((efi.runtime)->get_variable, "get_variable", args?) As an aside, it would be really helpful to document the arguments for __efi_rt_asm_wrapper. Pointers here how to debug further would be very welcome.
Thanks,
Alex
Quoted 99 lines
Cc Alexandru Elisei <[email protected]>Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/kernel/efi.c | 1 - arch/x86/platform/efi/quirks.c | 2 +- drivers/firmware/efi/runtime-wrappers.c | 6 ++++-- include/linux/efi.h | 24 ++++++++++++------------ 4 files changed, 17 insertions(+), 16 deletions(-)arch/arm64/kernel/efi.c +0
-1@@ -153,7 +153,6 @@ asmlinkage efi_status_t __efi_rt_asm_recover(void); asmlinkage efi_status_t efi_handle_runtime_exception(const char *f) {pr_err (FW_BUG "Synchronous exception occurred in EFI runtime service %s()\n" , f); -clear_bit (EFI_RUNTIME_SERVICES , &efi.flags);return EFI_ABORTED ; }arch/x86/platform/efi/quirks.c +1
-1@@ -757,9 +757,9 @@ void efi_crash_gracefully_on_page_fault(unsigned long phys_addr)/* Signal error status to the efi caller process */ efi_rts_work.status =EFI_ABORTED ; + efi.runtime_supported_mask =0 ;complete (&efi_rts_work.efi_rts_comp); -clear_bit (EFI_RUNTIME_SERVICES , &efi.flags);pr_info ("Froze efi_rts_wq and disabled EFI Runtime Services\n" );/* drivers/firmware/efi/runtime-wrappers.c +4
-2@@ -60,8 +60,8 @@ struct efi_runtime_work efi_rts_work; ({ \ efi_rts_work.status = EFI_ABORTED; \ \ - if (!efi_enabled(EFI_RUNTIME_SERVICES)) { \ - pr_warn_once("EFI Runtime Services are disabled!\n"); \ + if (!efi_rt_services_supported(efi_rts_work.efi_rts_id)) { \ + pr_warn_ratelimited("EFI Runtime Service is disabled!\n");\ goto exit; \ } \ \@@ -83,6 +83,8 @@ struct efi_runtime_work efi_rts_work; else \ pr_err("Failed to queue work to efi_rts_wq.\n"); \ \ +if (efi_rts_work.status ==EFI_ABORTED ) \ + efi.runtime_supported_mask &= ~efi_rts_work.efi_rts_id; \ exit: \ efi_rts_work.efi_rts_id =EFI_NONE ; \ efi_rts_work.status; \include/linux/efi.h +12
-12@@ -1254,18 +1254,18 @@ extern unsigned long rci2_table_phys;*/ enum efi_rts_ids {EFI_NONE , -EFI_GET_TIME , -EFI_SET_TIME , -EFI_GET_WAKEUP_TIME , -EFI_SET_WAKEUP_TIME , -EFI_GET_VARIABLE , -EFI_GET_NEXT_VARIABLE , -EFI_SET_VARIABLE , -EFI_QUERY_VARIABLE_INFO , -EFI_GET_NEXT_HIGH_MONO_COUNT , -EFI_RESET_SYSTEM , -EFI_UPDATE_CAPSULE , -EFI_QUERY_CAPSULE_CAPS , +EFI_GET_TIME =EFI_RT_SUPPORTED_GET_TIME , +EFI_SET_TIME =EFI_RT_SUPPORTED_SET_TIME , +EFI_GET_WAKEUP_TIME =EFI_RT_SUPPORTED_GET_WAKEUP_TIME , +EFI_SET_WAKEUP_TIME =EFI_RT_SUPPORTED_SET_WAKEUP_TIME , +EFI_GET_VARIABLE =EFI_RT_SUPPORTED_GET_VARIABLE , +EFI_GET_NEXT_VARIABLE =EFI_RT_SUPPORTED_GET_NEXT_VARIABLE_NAME , +EFI_SET_VARIABLE =EFI_RT_SUPPORTED_SET_VARIABLE , +EFI_GET_NEXT_HIGH_MONO_COUNT =EFI_RT_SUPPORTED_GET_NEXT_HIGH_MONOTONIC_COUNT , +EFI_RESET_SYSTEM =EFI_RT_SUPPORTED_RESET_SYSTEM , +EFI_UPDATE_CAPSULE =EFI_RT_SUPPORTED_UPDATE_CAPSULE , +EFI_QUERY_CAPSULE_CAPS =EFI_RT_SUPPORTED_QUERY_CAPSULE_CAPABILITIES , +EFI_QUERY_VARIABLE_INFO =EFI_RT_SUPPORTED_QUERY_VARIABLE_INFO , };/* Signature · 1 line
2.35.1
Ard Biesheuvel
maintainerQuoted 40 lines · Alexandru Elisei
Hi,
Quoted 16 lines · Ard Biesheuvel
Alexandru reports that his Ampere Altra machine, whose buggy firmware triggers a synchronous exception in its implementation of
SetTime()when called withoutSetVirtualAddressMap()having been called first, doesn't quite recover from this, and starts spewing error messages into the log that are unrelated to the buggy runtime service.The driver in question is the EFI RTC driver, which should be fixed in any case, as flooding the log like that (or doing any logging to the kernel log at all on something whuch is not a severe issue) is not ok.
However, in this particular case, it would be beneficial for both ordinary use as well as diagnostics regarding broken firmware if we only prevent the broken runtime service from being called again, and permit others (such as
GetTime()which triggers the logging or the variable services) from being used as normal.Quoted 5 lines
So wire up the existing efi.runtime_supported_mask, and clear the service's bit in the mask if the generic runtime service wrapper observes a return value of
EFI_ABORTED, which only happens if a service call is aborted due to an exception. (EFI_ABORTEDis not documented as a valid error code for any of the EFI runtime services).With a kernel built from v6.1-rc4, when doing efibootmgr after the EFI panic happens (so with runtime services disabled), this is what I get:
# efibootmgr Skipping unreadable variable "Boot0001": Interrupted system call Skipping unreadable variable "Boot0002": Interrupted system call show_order(): Interrupted system callAnd dmesg shows:
[ 55.941312] efi: EFI Runtime Services are disabled!With this patch on top of v6.1-rc4:
Quoted 8 lines
# efibootmgr Skipping unreadable variable "Boot0001": Invalid argument Skipping unreadable variable "Boot0002": Invalid argument show_order(): Invalid argumentSame thing happens if I cat the Boot001 efivars file. Nothing is printed on dmesg.
OK, this strongly suggests that the EFI runtime services end up in a funny state after the crash of SetTime(), and subsequent calls to any of them no longer work as expected.
Quoted 6 lines
Changed
efi_call_rts()to print the return value, status is 0x8000_0000_0000_000f (or 15 in decimal if casted into an int). Tried to debug further, but I'm not familiar with all the structs and what they represent (for example,efi_call_virt(get_variable, args)callsefi_call_virt_pointer(efi.runtime, get_variable, args), does it end up as__efi_rt_asm_wrapper((efi.runtime)->get_variable, "get_variable", args?)
Indeed.
The value of the function pointer is used to make the indirect call, and the string is only used if an error occurs, so we can print it to the log. The remaining arguments are simply the arguments to the firmware call.
Quoted 5 lines
As
an aside, it would be really helpful to document the arguments for__efi_rt_asm_wrapper. Pointers here how to debug further would be very welcome.
If the log is completely silent, there is not a lot to debug, really.
The error value you are observing is EFI_ACCESS_DENIED, and looking at the open source version of the Mt.Jade firmware, this might be the value returned from the secure world helper.
One other thing I would like to try is disabling set_time specifically using a command line parameter.
Btw could you share the output of dmidecode as well?
Hi,
Quoted 61 lines · Ard Biesheuvel
Quoted 40 lines · Alexandru Elisei
Hi,
Quoted 16 lines · Ard Biesheuvel
Alexandru reports that his Ampere Altra machine, whose buggy firmware triggers a synchronous exception in its implementation of
SetTime()when called withoutSetVirtualAddressMap()having been called first, doesn't quite recover from this, and starts spewing error messages into the log that are unrelated to the buggy runtime service.The driver in question is the EFI RTC driver, which should be fixed in any case, as flooding the log like that (or doing any logging to the kernel log at all on something whuch is not a severe issue) is not ok.
However, in this particular case, it would be beneficial for both ordinary use as well as diagnostics regarding broken firmware if we only prevent the broken runtime service from being called again, and permit others (such as
GetTime()which triggers the logging or the variable services) from being used as normal.Quoted 5 lines
So wire up the existing efi.runtime_supported_mask, and clear the service's bit in the mask if the generic runtime service wrapper observes a return value of
EFI_ABORTED, which only happens if a service call is aborted due to an exception. (EFI_ABORTEDis not documented as a valid error code for any of the EFI runtime services).With a kernel built from v6.1-rc4, when doing efibootmgr after the EFI panic happens (so with runtime services disabled), this is what I get:
# efibootmgr Skipping unreadable variable "Boot0001": Interrupted system call Skipping unreadable variable "Boot0002": Interrupted system call show_order(): Interrupted system callAnd dmesg shows:
[ 55.941312] efi: EFI Runtime Services are disabled!With this patch on top of v6.1-rc4:
Quoted 8 lines
# efibootmgr Skipping unreadable variable "Boot0001": Invalid argument Skipping unreadable variable "Boot0002": Invalid argument show_order(): Invalid argumentSame thing happens if I cat the Boot001 efivars file. Nothing is printed on dmesg.
OK, this strongly suggests that the EFI runtime services end up in a funny state after the crash of
SetTime(), and subsequent calls to any of them no longer work as expected.Quoted 6 lines
Changed
efi_call_rts()to print the return value, status is 0x8000_0000_0000_000f (or 15 in decimal if casted into an int). Tried to debug further, but I'm not familiar with all the structs and what they represent (for example,efi_call_virt(get_variable, args)callsefi_call_virt_pointer(efi.runtime, get_variable, args), does it end up as__efi_rt_asm_wrapper((efi.runtime)->get_variable, "get_variable", args?)
Quoted 6 lines
Indeed.
The value of the function pointer is used to make the indirect call, and the string is only used if an error occurs, so we can print it to the log. The remaining arguments are simply the arguments to the firmware call.
Got it, thanks for the explanation.
Quoted 7 lines
Quoted 6 lines
As
an aside, it would be really helpful to document the arguments for__efi_rt_asm_wrapper. Pointers here how to debug further would be very welcome.
Quoted 8 lines
If the log is completely silent, there is not a lot to debug, really.
The error value you are observing is
EFI_ACCESS_DENIED, and looking at the open source version of the Mt.Jade firmware, this might be the value returned from the secure world helper.One other thing I would like to try is disabling
set_timespecifically using a command line parameter.
Hm... had a look at kernel_parameters.txt and the rtc-efi driver and couldn't figure out how to do that. I could just return 0 early from efi_set_time(), I suppose. Is that what you had in mind?
Quoted 2 lines
Btw could you share the output of dmidecode as well?
Sure, you can find it at [1] (expires in 6 months, pastebin is unlisted). If it makes a difference, dmidecode was run with the stock Ubuntu kernel (v5.15).
[1] https://pastebin.com/eaL2LRCf
Thanks,
Alex
Ard Biesheuvel
maintainerQuoted 87 lines · Alexandru Elisei
Hi,
Quoted 61 lines · Ard Biesheuvel
Quoted 40 lines · Alexandru Elisei
Hi,
Quoted 16 lines · Ard Biesheuvel
Alexandru reports that his Ampere Altra machine, whose buggy firmware triggers a synchronous exception in its implementation of
SetTime()when called withoutSetVirtualAddressMap()having been called first, doesn't quite recover from this, and starts spewing error messages into the log that are unrelated to the buggy runtime service.The driver in question is the EFI RTC driver, which should be fixed in any case, as flooding the log like that (or doing any logging to the kernel log at all on something whuch is not a severe issue) is not ok.
However, in this particular case, it would be beneficial for both ordinary use as well as diagnostics regarding broken firmware if we only prevent the broken runtime service from being called again, and permit others (such as
GetTime()which triggers the logging or the variable services) from being used as normal.Quoted 5 lines
So wire up the existing efi.runtime_supported_mask, and clear the service's bit in the mask if the generic runtime service wrapper observes a return value of
EFI_ABORTED, which only happens if a service call is aborted due to an exception. (EFI_ABORTEDis not documented as a valid error code for any of the EFI runtime services).With a kernel built from v6.1-rc4, when doing efibootmgr after the EFI panic happens (so with runtime services disabled), this is what I get:
# efibootmgr Skipping unreadable variable "Boot0001": Interrupted system call Skipping unreadable variable "Boot0002": Interrupted system call show_order(): Interrupted system callAnd dmesg shows:
[ 55.941312] efi: EFI Runtime Services are disabled!With this patch on top of v6.1-rc4:
Quoted 8 lines
# efibootmgr Skipping unreadable variable "Boot0001": Invalid argument Skipping unreadable variable "Boot0002": Invalid argument show_order(): Invalid argumentSame thing happens if I cat the Boot001 efivars file. Nothing is printed on dmesg.
OK, this strongly suggests that the EFI runtime services end up in a funny state after the crash of
SetTime(), and subsequent calls to any of them no longer work as expected.Quoted 6 lines
Changed
efi_call_rts()to print the return value, status is 0x8000_0000_0000_000f (or 15 in decimal if casted into an int). Tried to debug further, but I'm not familiar with all the structs and what they represent (for example,efi_call_virt(get_variable, args)callsefi_call_virt_pointer(efi.runtime, get_variable, args), does it end up as__efi_rt_asm_wrapper((efi.runtime)->get_variable, "get_variable", args?)Quoted 6 lines
Indeed.
The value of the function pointer is used to make the indirect call, and the string is only used if an error occurs, so we can print it to the log. The remaining arguments are simply the arguments to the firmware call.
Got it, thanks for the explanation.
Quoted 13 lines
Quoted 6 lines
As
an aside, it would be really helpful to document the arguments for__efi_rt_asm_wrapper. Pointers here how to debug further would be very welcome.If the log is completely silent, there is not a lot to debug, really.
The error value you are observing is
EFI_ACCESS_DENIED, and looking at the open source version of the Mt.Jade firmware, this might be the value returned from the secure world helper.
Quoted 7 lines
Quoted 2 lines
One other thing I would like to try is disabling
set_timespecifically using a command line parameter.Hm... had a look at kernel_parameters.txt and the rtc-efi driver and couldn't figure out how to do that. I could just return 0 early from
efi_set_time(), I suppose. Is that what you had in mind?
What I had in mind is introduce a way to override efi.runtime_supported_mask via either a kernel command line option or a EFI variable.
I'll code up the latter and send it out so you can give it a spin.
Note: just to narrow down the firmware behavior a bit - I fully anticipate having to revert the novamap patch ultimately.
Quoted 9 lines
Quoted 2 lines
Btw could you share the output of dmidecode as well?
Sure, you can find it at [1] (expires in 6 months, pastebin is unlisted). If it makes a difference, dmidecode was run with the stock Ubuntu kernel (v5.15).
Thanks.