Skip to contentSkip to conversation

arm64: efi: Disable only the misbehaving runtime service on sync exceptions

PATCH In review Ard Biesheuvel 2 participants 5 messages Nov 8, 2022 – Nov 9, 2022

Subsystems (3)
lore
Share & tools
lore

Ard Biesheuvel

author maintainer
#0 lore raw

PATCH 1/1 arm64: efi: Disable only the misbehaving runtime service on sync exceptions

Alexandru reports that his Ampere Altra machine, whose buggy firmware triggers a synchronous exception in its implementation of SetTime() when called without SetVirtualAddressMap() having been called first, doesn't quite recover from this, and starts spewing error messages into the log that are unrelated to the buggy runtime service.

The driver in question is the EFI RTC driver, which should be fixed in any case, as flooding the log like that (or doing any logging to the kernel log at all on something whuch is not a severe issue) is not ok.

However, in this particular case, it would be beneficial for both ordinary use as well as diagnostics regarding broken firmware if we only prevent the broken runtime service from being called again, and permit others (such as GetTime() which triggers the logging or the variable services) from being used as normal.

So wire up the existing efi.runtime_supported_mask, and clear the service's bit in the mask if the generic runtime service wrapper observes a return value of EFI_ABORTED, which only happens if a service call is aborted due to an exception. (EFI_ABORTED is not documented as a valid error code for any of the EFI runtime services).

Cc Alexandru Elisei <[email protected]>
---
 arch/arm64/kernel/efi.c                 |  1 -
 arch/x86/platform/efi/quirks.c          |  2 +-
 drivers/firmware/efi/runtime-wrappers.c |  6 ++++--
 include/linux/efi.h                     | 24 ++++++++++++------------
 4 files changed, 17 insertions(+), 16 deletions(-)
arch/arm64/kernel/efi.c +0 -1
index ee53f2a0aa03f54a..a3b8852f2698d9f7 100644
@@ -153,7 +153,6 @@ asmlinkage efi_status_t __efi_rt_asm_recover(void);
 asmlinkage efi_status_t efi_handle_runtime_exception(const char *f)
 {
 	pr_err(FW_BUG "Synchronous exception occurred in EFI runtime service %s()\n", f);
-	clear_bit(EFI_RUNTIME_SERVICES, &efi.flags);
 	return EFI_ABORTED;
 }
 
arch/x86/platform/efi/quirks.c +1 -1
index b0b848d6933afbcf..3f48fdf42d97dc1f 100644
@@ -757,9 +757,9 @@ void efi_crash_gracefully_on_page_fault(unsigned long phys_addr)
 
 	/* Signal error status to the efi caller process */
 	efi_rts_work.status = EFI_ABORTED;
+	efi.runtime_supported_mask = 0;
 	complete(&efi_rts_work.efi_rts_comp);
 
-	clear_bit(EFI_RUNTIME_SERVICES, &efi.flags);
 	pr_info("Froze efi_rts_wq and disabled EFI Runtime Services\n");
 
 	/*
drivers/firmware/efi/runtime-wrappers.c +4 -2
index f3e54f6616f02475..336b8bcec86d0127 100644
@@ -60,8 +60,8 @@ struct efi_runtime_work efi_rts_work;
 ({									\
 	efi_rts_work.status = EFI_ABORTED;				\
 									\
-	if (!efi_enabled(EFI_RUNTIME_SERVICES)) {			\
-		pr_warn_once("EFI Runtime Services are disabled!\n");	\
+	if (!efi_rt_services_supported(efi_rts_work.efi_rts_id)) {	\
+		pr_warn_ratelimited("EFI Runtime Service is disabled!\n");\
 		goto exit;						\
 	}								\
 									\
@@ -83,6 +83,8 @@ struct efi_runtime_work efi_rts_work;
 	else								\
 		pr_err("Failed to queue work to efi_rts_wq.\n");	\
 									\
+	if (efi_rts_work.status == EFI_ABORTED)				\
+		efi.runtime_supported_mask &= ~efi_rts_work.efi_rts_id;	\
 exit:									\
 	efi_rts_work.efi_rts_id = EFI_NONE;				\
 	efi_rts_work.status;						\
include/linux/efi.h +12 -12
index 929d559ad41d29c6..61b252386d61cc4d 100644
@@ -1254,18 +1254,18 @@ extern unsigned long rci2_table_phys;
  */
 enum efi_rts_ids {
 	EFI_NONE,
-	EFI_GET_TIME,
-	EFI_SET_TIME,
-	EFI_GET_WAKEUP_TIME,
-	EFI_SET_WAKEUP_TIME,
-	EFI_GET_VARIABLE,
-	EFI_GET_NEXT_VARIABLE,
-	EFI_SET_VARIABLE,
-	EFI_QUERY_VARIABLE_INFO,
-	EFI_GET_NEXT_HIGH_MONO_COUNT,
-	EFI_RESET_SYSTEM,
-	EFI_UPDATE_CAPSULE,
-	EFI_QUERY_CAPSULE_CAPS,
+	EFI_GET_TIME			= EFI_RT_SUPPORTED_GET_TIME,
+	EFI_SET_TIME			= EFI_RT_SUPPORTED_SET_TIME,
+	EFI_GET_WAKEUP_TIME		= EFI_RT_SUPPORTED_GET_WAKEUP_TIME,
+	EFI_SET_WAKEUP_TIME		= EFI_RT_SUPPORTED_SET_WAKEUP_TIME,
+	EFI_GET_VARIABLE		= EFI_RT_SUPPORTED_GET_VARIABLE,
+	EFI_GET_NEXT_VARIABLE		= EFI_RT_SUPPORTED_GET_NEXT_VARIABLE_NAME,
+	EFI_SET_VARIABLE		= EFI_RT_SUPPORTED_SET_VARIABLE,
+	EFI_GET_NEXT_HIGH_MONO_COUNT	= EFI_RT_SUPPORTED_GET_NEXT_HIGH_MONOTONIC_COUNT,
+	EFI_RESET_SYSTEM		= EFI_RT_SUPPORTED_RESET_SYSTEM,
+	EFI_UPDATE_CAPSULE		= EFI_RT_SUPPORTED_UPDATE_CAPSULE,
+	EFI_QUERY_CAPSULE_CAPS		= EFI_RT_SUPPORTED_QUERY_CAPSULE_CAPABILITIES,
+	EFI_QUERY_VARIABLE_INFO		= EFI_RT_SUPPORTED_QUERY_VARIABLE_INFO,
 };
 
 /*
Signature · 1 line
2.35.1
#1 lore raw

Hi,

Quoted 16 lines · Ard Biesheuvel

Alexandru reports that his Ampere Altra machine, whose buggy firmware triggers a synchronous exception in its implementation of SetTime() when called without SetVirtualAddressMap() having been called first, doesn't quite recover from this, and starts spewing error messages into the log that are unrelated to the buggy runtime service.

The driver in question is the EFI RTC driver, which should be fixed in any case, as flooding the log like that (or doing any logging to the kernel log at all on something whuch is not a severe issue) is not ok.

However, in this particular case, it would be beneficial for both ordinary use as well as diagnostics regarding broken firmware if we only prevent the broken runtime service from being called again, and permit others (such as GetTime() which triggers the logging or the variable services) from being used as normal.

Quoted 5 lines

So wire up the existing efi.runtime_supported_mask, and clear the service's bit in the mask if the generic runtime service wrapper observes a return value of EFI_ABORTED, which only happens if a service call is aborted due to an exception. (EFI_ABORTED is not documented as a valid error code for any of the EFI runtime services).

With a kernel built from v6.1-rc4, when doing efibootmgr after the EFI panic happens (so with runtime services disabled), this is what I get:

# efibootmgr
Skipping unreadable variable "Boot0001": Interrupted system call
Skipping unreadable variable "Boot0002": Interrupted system call
show_order(): Interrupted system call

And dmesg shows:

[   55.941312] efi: EFI Runtime Services are disabled!

With this patch on top of v6.1-rc4:

# efibootmgr
Skipping unreadable variable "Boot0001": Invalid argument
Skipping unreadable variable "Boot0002": Invalid argument
show_order(): Invalid argument

Same thing happens if I cat the Boot001 efivars file. Nothing is printed on dmesg.

Changed efi_call_rts() to print the return value, status is 0x8000_0000_0000_000f (or 15 in decimal if casted into an int). Tried to debug further, but I'm not familiar with all the structs and what they represent (for example, efi_call_virt(get_variable, args) calls efi_call_virt_pointer(efi.runtime, get_variable, args), does it end up as __efi_rt_asm_wrapper((efi.runtime)->get_variable, "get_variable", args?) As an aside, it would be really helpful to document the arguments for __efi_rt_asm_wrapper. Pointers here how to debug further would be very welcome.

Thanks,
Alex

Quoted 99 lines
Cc Alexandru Elisei <[email protected]>
---
 arch/arm64/kernel/efi.c                 |  1 -
 arch/x86/platform/efi/quirks.c          |  2 +-
 drivers/firmware/efi/runtime-wrappers.c |  6 ++++--
 include/linux/efi.h                     | 24 ++++++++++++------------
 4 files changed, 17 insertions(+), 16 deletions(-)
arch/arm64/kernel/efi.c +0 -1
index ee53f2a0aa03f54a..a3b8852f2698d9f7 100644
@@ -153,7 +153,6 @@ asmlinkage efi_status_t __efi_rt_asm_recover(void);
 asmlinkage efi_status_t efi_handle_runtime_exception(const char *f)
 {
 	pr_err(FW_BUG "Synchronous exception occurred in EFI runtime service %s()\n", f);
-	clear_bit(EFI_RUNTIME_SERVICES, &efi.flags);
 	return EFI_ABORTED;
 }
 
arch/x86/platform/efi/quirks.c +1 -1
index b0b848d6933afbcf..3f48fdf42d97dc1f 100644
@@ -757,9 +757,9 @@ void efi_crash_gracefully_on_page_fault(unsigned long phys_addr)
 
 	/* Signal error status to the efi caller process */
 	efi_rts_work.status = EFI_ABORTED;
+	efi.runtime_supported_mask = 0;
 	complete(&efi_rts_work.efi_rts_comp);
 
-	clear_bit(EFI_RUNTIME_SERVICES, &efi.flags);
 	pr_info("Froze efi_rts_wq and disabled EFI Runtime Services\n");
 
 	/*
drivers/firmware/efi/runtime-wrappers.c +4 -2
index f3e54f6616f02475..336b8bcec86d0127 100644
@@ -60,8 +60,8 @@ struct efi_runtime_work efi_rts_work;
 ({									\
 	efi_rts_work.status = EFI_ABORTED;				\
 									\
-	if (!efi_enabled(EFI_RUNTIME_SERVICES)) {			\
-		pr_warn_once("EFI Runtime Services are disabled!\n");	\
+	if (!efi_rt_services_supported(efi_rts_work.efi_rts_id)) {	\
+		pr_warn_ratelimited("EFI Runtime Service is disabled!\n");\
 		goto exit;						\
 	}								\
 									\
@@ -83,6 +83,8 @@ struct efi_runtime_work efi_rts_work;
 	else								\
 		pr_err("Failed to queue work to efi_rts_wq.\n");	\
 									\
+	if (efi_rts_work.status == EFI_ABORTED)				\
+		efi.runtime_supported_mask &= ~efi_rts_work.efi_rts_id;	\
 exit:									\
 	efi_rts_work.efi_rts_id = EFI_NONE;				\
 	efi_rts_work.status;						\
include/linux/efi.h +12 -12
index 929d559ad41d29c6..61b252386d61cc4d 100644
@@ -1254,18 +1254,18 @@ extern unsigned long rci2_table_phys;
  */
 enum efi_rts_ids {
 	EFI_NONE,
-	EFI_GET_TIME,
-	EFI_SET_TIME,
-	EFI_GET_WAKEUP_TIME,
-	EFI_SET_WAKEUP_TIME,
-	EFI_GET_VARIABLE,
-	EFI_GET_NEXT_VARIABLE,
-	EFI_SET_VARIABLE,
-	EFI_QUERY_VARIABLE_INFO,
-	EFI_GET_NEXT_HIGH_MONO_COUNT,
-	EFI_RESET_SYSTEM,
-	EFI_UPDATE_CAPSULE,
-	EFI_QUERY_CAPSULE_CAPS,
+	EFI_GET_TIME			= EFI_RT_SUPPORTED_GET_TIME,
+	EFI_SET_TIME			= EFI_RT_SUPPORTED_SET_TIME,
+	EFI_GET_WAKEUP_TIME		= EFI_RT_SUPPORTED_GET_WAKEUP_TIME,
+	EFI_SET_WAKEUP_TIME		= EFI_RT_SUPPORTED_SET_WAKEUP_TIME,
+	EFI_GET_VARIABLE		= EFI_RT_SUPPORTED_GET_VARIABLE,
+	EFI_GET_NEXT_VARIABLE		= EFI_RT_SUPPORTED_GET_NEXT_VARIABLE_NAME,
+	EFI_SET_VARIABLE		= EFI_RT_SUPPORTED_SET_VARIABLE,
+	EFI_GET_NEXT_HIGH_MONO_COUNT	= EFI_RT_SUPPORTED_GET_NEXT_HIGH_MONOTONIC_COUNT,
+	EFI_RESET_SYSTEM		= EFI_RT_SUPPORTED_RESET_SYSTEM,
+	EFI_UPDATE_CAPSULE		= EFI_RT_SUPPORTED_UPDATE_CAPSULE,
+	EFI_QUERY_CAPSULE_CAPS		= EFI_RT_SUPPORTED_QUERY_CAPSULE_CAPABILITIES,
+	EFI_QUERY_VARIABLE_INFO		= EFI_RT_SUPPORTED_QUERY_VARIABLE_INFO,
 };
 
 /*
Signature · 1 line
2.35.1
author maintainer
#2 lore raw
Quoted 40 lines · Alexandru Elisei

Hi,

Quoted 16 lines · Ard Biesheuvel

Alexandru reports that his Ampere Altra machine, whose buggy firmware triggers a synchronous exception in its implementation of SetTime() when called without SetVirtualAddressMap() having been called first, doesn't quite recover from this, and starts spewing error messages into the log that are unrelated to the buggy runtime service.

The driver in question is the EFI RTC driver, which should be fixed in any case, as flooding the log like that (or doing any logging to the kernel log at all on something whuch is not a severe issue) is not ok.

However, in this particular case, it would be beneficial for both ordinary use as well as diagnostics regarding broken firmware if we only prevent the broken runtime service from being called again, and permit others (such as GetTime() which triggers the logging or the variable services) from being used as normal.

Quoted 5 lines

So wire up the existing efi.runtime_supported_mask, and clear the service's bit in the mask if the generic runtime service wrapper observes a return value of EFI_ABORTED, which only happens if a service call is aborted due to an exception. (EFI_ABORTED is not documented as a valid error code for any of the EFI runtime services).

With a kernel built from v6.1-rc4, when doing efibootmgr after the EFI panic happens (so with runtime services disabled), this is what I get:

# efibootmgr
Skipping unreadable variable "Boot0001": Interrupted system call
Skipping unreadable variable "Boot0002": Interrupted system call
show_order(): Interrupted system call

And dmesg shows:

[   55.941312] efi: EFI Runtime Services are disabled!

With this patch on top of v6.1-rc4:

Quoted 8 lines
# efibootmgr
Skipping unreadable variable "Boot0001": Invalid argument
Skipping unreadable variable "Boot0002": Invalid argument
show_order(): Invalid argument

Same thing happens if I cat the Boot001 efivars file. Nothing is printed on dmesg.

OK, this strongly suggests that the EFI runtime services end up in a funny state after the crash of SetTime(), and subsequent calls to any of them no longer work as expected.

Quoted 6 lines

Changed efi_call_rts() to print the return value, status is 0x8000_0000_0000_000f (or 15 in decimal if casted into an int). Tried to debug further, but I'm not familiar with all the structs and what they represent (for example, efi_call_virt(get_variable, args) calls efi_call_virt_pointer(efi.runtime, get_variable, args), does it end up as __efi_rt_asm_wrapper((efi.runtime)->get_variable, "get_variable", args?)

Indeed.

The value of the function pointer is used to make the indirect call, and the string is only used if an error occurs, so we can print it to the log. The remaining arguments are simply the arguments to the firmware call.

Quoted 5 lines

As
an aside, it would be really helpful to document the arguments for __efi_rt_asm_wrapper. Pointers here how to debug further would be very welcome.

If the log is completely silent, there is not a lot to debug, really.

The error value you are observing is EFI_ACCESS_DENIED, and looking at the open source version of the Mt.Jade firmware, this might be the value returned from the secure world helper.

One other thing I would like to try is disabling set_time specifically using a command line parameter.

Btw could you share the output of dmidecode as well?

#3 lore raw

Hi,

Quoted 61 lines · Ard Biesheuvel
Quoted 40 lines · Alexandru Elisei

Hi,

Quoted 16 lines · Ard Biesheuvel

Alexandru reports that his Ampere Altra machine, whose buggy firmware triggers a synchronous exception in its implementation of SetTime() when called without SetVirtualAddressMap() having been called first, doesn't quite recover from this, and starts spewing error messages into the log that are unrelated to the buggy runtime service.

The driver in question is the EFI RTC driver, which should be fixed in any case, as flooding the log like that (or doing any logging to the kernel log at all on something whuch is not a severe issue) is not ok.

However, in this particular case, it would be beneficial for both ordinary use as well as diagnostics regarding broken firmware if we only prevent the broken runtime service from being called again, and permit others (such as GetTime() which triggers the logging or the variable services) from being used as normal.

Quoted 5 lines

So wire up the existing efi.runtime_supported_mask, and clear the service's bit in the mask if the generic runtime service wrapper observes a return value of EFI_ABORTED, which only happens if a service call is aborted due to an exception. (EFI_ABORTED is not documented as a valid error code for any of the EFI runtime services).

With a kernel built from v6.1-rc4, when doing efibootmgr after the EFI panic happens (so with runtime services disabled), this is what I get:

# efibootmgr
Skipping unreadable variable "Boot0001": Interrupted system call
Skipping unreadable variable "Boot0002": Interrupted system call
show_order(): Interrupted system call

And dmesg shows:

[   55.941312] efi: EFI Runtime Services are disabled!

With this patch on top of v6.1-rc4:

Quoted 8 lines
# efibootmgr
Skipping unreadable variable "Boot0001": Invalid argument
Skipping unreadable variable "Boot0002": Invalid argument
show_order(): Invalid argument

Same thing happens if I cat the Boot001 efivars file. Nothing is printed on dmesg.

OK, this strongly suggests that the EFI runtime services end up in a funny state after the crash of SetTime(), and subsequent calls to any of them no longer work as expected.

Quoted 6 lines

Changed efi_call_rts() to print the return value, status is 0x8000_0000_0000_000f (or 15 in decimal if casted into an int). Tried to debug further, but I'm not familiar with all the structs and what they represent (for example, efi_call_virt(get_variable, args) calls efi_call_virt_pointer(efi.runtime, get_variable, args), does it end up as __efi_rt_asm_wrapper((efi.runtime)->get_variable, "get_variable", args?)

Quoted 6 lines

Indeed.

The value of the function pointer is used to make the indirect call, and the string is only used if an error occurs, so we can print it to the log. The remaining arguments are simply the arguments to the firmware call.

Got it, thanks for the explanation.

Quoted 7 lines
Quoted 6 lines

As
an aside, it would be really helpful to document the arguments for __efi_rt_asm_wrapper. Pointers here how to debug further would be very welcome.

Quoted 8 lines

If the log is completely silent, there is not a lot to debug, really.

The error value you are observing is EFI_ACCESS_DENIED, and looking at the open source version of the Mt.Jade firmware, this might be the value returned from the secure world helper.

One other thing I would like to try is disabling set_time specifically using a command line parameter.

Hm... had a look at kernel_parameters.txt and the rtc-efi driver and couldn't figure out how to do that. I could just return 0 early from efi_set_time(), I suppose. Is that what you had in mind?

Quoted 2 lines

Btw could you share the output of dmidecode as well?

Sure, you can find it at [1] (expires in 6 months, pastebin is unlisted). If it makes a difference, dmidecode was run with the stock Ubuntu kernel (v5.15).

[1] https://pastebin.com/eaL2LRCf

Thanks,
Alex

author maintainer
#4 lore raw
Quoted 87 lines · Alexandru Elisei

Hi,

Quoted 61 lines · Ard Biesheuvel
Quoted 40 lines · Alexandru Elisei

Hi,

Quoted 16 lines · Ard Biesheuvel

Alexandru reports that his Ampere Altra machine, whose buggy firmware triggers a synchronous exception in its implementation of SetTime() when called without SetVirtualAddressMap() having been called first, doesn't quite recover from this, and starts spewing error messages into the log that are unrelated to the buggy runtime service.

The driver in question is the EFI RTC driver, which should be fixed in any case, as flooding the log like that (or doing any logging to the kernel log at all on something whuch is not a severe issue) is not ok.

However, in this particular case, it would be beneficial for both ordinary use as well as diagnostics regarding broken firmware if we only prevent the broken runtime service from being called again, and permit others (such as GetTime() which triggers the logging or the variable services) from being used as normal.

Quoted 5 lines

So wire up the existing efi.runtime_supported_mask, and clear the service's bit in the mask if the generic runtime service wrapper observes a return value of EFI_ABORTED, which only happens if a service call is aborted due to an exception. (EFI_ABORTED is not documented as a valid error code for any of the EFI runtime services).

With a kernel built from v6.1-rc4, when doing efibootmgr after the EFI panic happens (so with runtime services disabled), this is what I get:

# efibootmgr
Skipping unreadable variable "Boot0001": Interrupted system call
Skipping unreadable variable "Boot0002": Interrupted system call
show_order(): Interrupted system call

And dmesg shows:

[   55.941312] efi: EFI Runtime Services are disabled!

With this patch on top of v6.1-rc4:

Quoted 8 lines
# efibootmgr
Skipping unreadable variable "Boot0001": Invalid argument
Skipping unreadable variable "Boot0002": Invalid argument
show_order(): Invalid argument

Same thing happens if I cat the Boot001 efivars file. Nothing is printed on dmesg.

OK, this strongly suggests that the EFI runtime services end up in a funny state after the crash of SetTime(), and subsequent calls to any of them no longer work as expected.

Quoted 6 lines

Changed efi_call_rts() to print the return value, status is 0x8000_0000_0000_000f (or 15 in decimal if casted into an int). Tried to debug further, but I'm not familiar with all the structs and what they represent (for example, efi_call_virt(get_variable, args) calls efi_call_virt_pointer(efi.runtime, get_variable, args), does it end up as __efi_rt_asm_wrapper((efi.runtime)->get_variable, "get_variable", args?)

Quoted 6 lines

Indeed.

The value of the function pointer is used to make the indirect call, and the string is only used if an error occurs, so we can print it to the log. The remaining arguments are simply the arguments to the firmware call.

Got it, thanks for the explanation.

Quoted 13 lines
Quoted 6 lines

As
an aside, it would be really helpful to document the arguments for __efi_rt_asm_wrapper. Pointers here how to debug further would be very welcome.

If the log is completely silent, there is not a lot to debug, really.

The error value you are observing is EFI_ACCESS_DENIED, and looking at the open source version of the Mt.Jade firmware, this might be the value returned from the secure world helper.

Quoted 7 lines
Quoted 2 lines

One other thing I would like to try is disabling set_time specifically using a command line parameter.

Hm... had a look at kernel_parameters.txt and the rtc-efi driver and couldn't figure out how to do that. I could just return 0 early from efi_set_time(), I suppose. Is that what you had in mind?

What I had in mind is introduce a way to override efi.runtime_supported_mask via either a kernel command line option or a EFI variable.

I'll code up the latter and send it out so you can give it a spin.

Note: just to narrow down the firmware behavior a bit - I fully anticipate having to revert the novamap patch ultimately.

Quoted 9 lines
Quoted 2 lines

Btw could you share the output of dmidecode as well?

Sure, you can find it at [1] (expires in 6 months, pastebin is unlisted). If it makes a difference, dmidecode was run with the stock Ubuntu kernel (v5.15).

[1] https://pastebin.com/eaL2LRCf

Thanks.