Skip to contentSkip to conversation

Compiler attributes: Introduce the __preserve_most function attribute

PATCH In review Marco Elver 3 participants 7 messages Aug 2, 2023

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/3 Compiler attributes: Introduce the __preserve_most function attribute

[1]: "On X86-64 and AArch64 targets, this attribute changes the calling convention of a function. The preserve_most calling convention attempts to make the code in the caller as unintrusive as possible. This convention behaves identically to the C calling convention on how arguments and return values are passed, but it uses a different set of caller/callee-saved registers. This alleviates the burden of saving and recovering a large register set before and after the call in the caller."

[1] https://clang.llvm.org/docs/AttributeReference.html#preserve-most

Use of this attribute results in better code generation for calls to very rarely called functions, such as error-reporting functions, or rarely executed slow paths.

Introduce the attribute to compiler_attributes.h.

---
 include/linux/compiler_attributes.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
include/linux/compiler_attributes.h +11 -0
index 00efa35c350f..615a63ecfcf6 100644
@@ -321,6 +321,17 @@
 # define __pass_object_size(type)
 #endif
 
+/*
+ * Optional: not supported by gcc.
+ *
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#preserve-most
+ */
+#if __has_attribute(__preserve_most__)
+# define __preserve_most __attribute__((__preserve_most__))
+#else
+# define __preserve_most
+#endif
+
 /*
  *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-pure-function-attribute
  */
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
#1 lore raw

PATCH 2/3 list_debug: Introduce inline wrappers for debug checks

Turn the list debug checking functions __list_*_valid() into inline functions that wrap the out-of-line functions. Care is taken to ensure the inline wrappers are always inlined, so that additional compiler instrumentation (such as sanitizers) does not result in redundant outlining.

This change is preparation for performing checks in the inline wrappers.

No functional change intended.

---
 arch/arm64/kvm/hyp/nvhe/list_debug.c |  6 +++---
 include/linux/list.h                 | 15 +++++++++++++--
 lib/list_debug.c                     | 11 +++++------
 3 files changed, 21 insertions(+), 11 deletions(-)
arch/arm64/kvm/hyp/nvhe/list_debug.c +3 -3
index d68abd7ea124..589284496ac5 100644
@@ -26,8 +26,8 @@ static inline __must_check bool nvhe_check_data_corruption(bool v)
 
 /* The predicates checked here are taken from lib/list_debug.c. */
 
-bool __list_add_valid(struct list_head *new, struct list_head *prev,
-		      struct list_head *next)
+bool ___list_add_valid(struct list_head *new, struct list_head *prev,
+		       struct list_head *next)
 {
 	if (NVHE_CHECK_DATA_CORRUPTION(next->prev != prev) ||
 	    NVHE_CHECK_DATA_CORRUPTION(prev->next != next) ||
@@ -37,7 +37,7 @@ bool __list_add_valid(struct list_head *new, struct list_head *prev,
 	return true;
 }
 
-bool __list_del_entry_valid(struct list_head *entry)
+bool ___list_del_entry_valid(struct list_head *entry)
 {
 	struct list_head *prev, *next;
 
include/linux/list.h +13 -2
index f10344dbad4d..e0b2cf904409 100644
@@ -39,10 +39,21 @@ static inline void INIT_LIST_HEAD(struct list_head *list)
 }
 
 #ifdef CONFIG_DEBUG_LIST
-extern bool __list_add_valid(struct list_head *new,
+extern bool ___list_add_valid(struct list_head *new,
 			      struct list_head *prev,
 			      struct list_head *next);
-extern bool __list_del_entry_valid(struct list_head *entry);
+static __always_inline bool __list_add_valid(struct list_head *new,
+					     struct list_head *prev,
+					     struct list_head *next)
+{
+	return ___list_add_valid(new, prev, next);
+}
+
+extern bool ___list_del_entry_valid(struct list_head *entry);
+static __always_inline bool __list_del_entry_valid(struct list_head *entry)
+{
+	return ___list_del_entry_valid(entry);
+}
 #else
 static inline bool __list_add_valid(struct list_head *new,
 				struct list_head *prev,
lib/list_debug.c +5 -6
index d98d43f80958..fd69009cc696 100644
@@ -17,8 +17,8 @@
  * attempt).
  */
 
-bool __list_add_valid(struct list_head *new, struct list_head *prev,
-		      struct list_head *next)
+bool ___list_add_valid(struct list_head *new, struct list_head *prev,
+		       struct list_head *next)
 {
 	if (CHECK_DATA_CORRUPTION(prev == NULL,
 			"list_add corruption. prev is NULL.\n") ||
@@ -37,9 +37,9 @@ bool __list_add_valid(struct list_head *new, struct list_head *prev,
 
 	return true;
 }
-EXPORT_SYMBOL(__list_add_valid);
+EXPORT_SYMBOL(___list_add_valid);
 
-bool __list_del_entry_valid(struct list_head *entry)
+bool ___list_del_entry_valid(struct list_head *entry)
 {
 	struct list_head *prev, *next;
 
@@ -65,6 +65,5 @@ bool __list_del_entry_valid(struct list_head *entry)
 		return false;
 
 	return true;
-
 }
-EXPORT_SYMBOL(__list_del_entry_valid);
+EXPORT_SYMBOL(___list_del_entry_valid);
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
#2 lore raw

PATCH 3/3 list_debug: Introduce CONFIG_DEBUG_LIST_MINIMAL

In reply to Marco Elver

Numerous production kernel configs (see [1, 2]) are choosing to enable CONFIG_DEBUG_LIST, which is also being recommended by KSPP for hardened configs [3]. The feature has never been designed with performance in mind, yet common list manipulation is happening across hot paths all over the kernel.

Introduce CONFIG_DEBUG_LIST_MINIMAL, which performs list pointer checking inline, and only upon list corruption delegates to the reporting slow path.

To generate optimal machine code with CONFIG_DEBUG_LIST_MINIMAL:

1. Elide checking for pointer values which upon dereference would result in an immediate access fault -- therefore "minimal" checks. The trade-off is lower-quality error reports.

2. Use the newly introduced __preserve_most function attribute (available with Clang, but not yet with GCC) to minimize the code footprint for calling the reporting slow path. As a result, function size of callers is reduced by avoiding saving registers before calling the rarely called reporting slow path.

3. Because the inline checks are a subset of the full set of checks in ___list_*_valid(), always return false if the inline checks failed. This avoids redundant compare and conditional branch right after return from the slow path.

As a side-effect of the checks being inline, if the compiler can prove some condition to always be true, it can completely elide some checks.

Running netperf with CONFIG_DEBUG_LIST_MINIMAL (using a Clang compiler with "preserve_most") shows throughput improvements, in my case of ~7% on average (up to 20-30% on some test cases).

---
 arch/arm64/kvm/hyp/nvhe/list_debug.c |  2 +
 include/linux/list.h                 | 56 +++++++++++++++++++++++++---
 lib/Kconfig.debug                    | 15 ++++++++
 lib/list_debug.c                     |  2 +
 4 files changed, 69 insertions(+), 6 deletions(-)
arch/arm64/kvm/hyp/nvhe/list_debug.c +2 -0
index 589284496ac5..df718e29f6d4 100644
@@ -26,6 +26,7 @@ static inline __must_check bool nvhe_check_data_corruption(bool v)
 
 /* The predicates checked here are taken from lib/list_debug.c. */
 
+__list_valid_slowpath
 bool ___list_add_valid(struct list_head *new, struct list_head *prev,
 		       struct list_head *next)
 {
@@ -37,6 +38,7 @@ bool ___list_add_valid(struct list_head *new, struct list_head *prev,
 	return true;
 }
 
+__list_valid_slowpath
 bool ___list_del_entry_valid(struct list_head *entry)
 {
 	struct list_head *prev, *next;
include/linux/list.h +50 -6

70 lines not on this page · raw message

lib/Kconfig.debug +15 -0
index fbc89baf7de6..e72cf08af0fa 100644
@@ -1680,6 +1680,21 @@ config DEBUG_LIST
 
 	  If unsure, say N.
 
+config DEBUG_LIST_MINIMAL
+	bool "Minimal linked list debug checks"
+	default !DEBUG_KERNEL
+	depends on DEBUG_LIST
+	help
+	  Only perform the minimal set of checks in the linked-list walking
+	  routines to catch corruptions that are not guaranteed to result in an
+	  immediate access fault.
+
+	  This trades lower quality error reports for improved performance: the
+	  generated code should be more optimal and provide trade-offs that may
+	  better serve safety- and performance- critical environments.
+
+	  If unsure, say Y.
+
 config DEBUG_PLIST
 	bool "Debug priority linked list manipulation"
 	depends on DEBUG_KERNEL
lib/list_debug.c +2 -0
index fd69009cc696..daad32855f0d 100644
@@ -17,6 +17,7 @@
  * attempt).
  */
 
+__list_valid_slowpath
 bool ___list_add_valid(struct list_head *new, struct list_head *prev,
 		       struct list_head *next)
 {
@@ -39,6 +40,7 @@ bool ___list_add_valid(struct list_head *new, struct list_head *prev,
 }
 EXPORT_SYMBOL(___list_add_valid);
 
+__list_valid_slowpath
 bool ___list_del_entry_valid(struct list_head *entry)
 {
 	struct list_head *prev, *next;
Signature · 1 line
2.41.0.585.gd2178a4bd4-goog
#3 lore raw
In reply to Marco Elver
Quoted 31 lines · Marco Elver

[1]: "On X86-64 and AArch64 targets, this attribute changes the calling convention of a function. The preserve_most calling convention attempts to make the code in the caller as unintrusive as possible. This convention behaves identically to the C calling convention on how arguments and return values are passed, but it uses a different set of caller/callee-saved registers. This alleviates the burden of saving and recovering a large register set before and after the call in the caller."

[1] https://clang.llvm.org/docs/AttributeReference.html#preserve-most

Use of this attribute results in better code generation for calls to very rarely called functions, such as error-reporting functions, or rarely executed slow paths.

Introduce the attribute to compiler_attributes.h.

---
 include/linux/compiler_attributes.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
include/linux/compiler_attributes.h +0 -0
index 00efa35c350f..615a63ecfcf6 100644
@@ -321,6 +321,17 @@
 # define __pass_object_size(type)
 #endif
 

include/linux/compiler_attributes.h line 333

Quoted 10 lines
include/linux/compiler_attributes.h +10 -0
+/*
+ * Optional: not supported by gcc.
+ *
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#preserve-most
+ */
+#if __has_attribute(__preserve_most__)
+# define __preserve_most __attribute__((__preserve_most__))
+#else
+# define __preserve_most
+#endif

Mark says that there may be an issue with using this in combination with ftrace because arm64 tracing relies on AAPCS. Probably not just arm64, but also other architectures (x86?).

To make this safe, I'm going to move __preserve_most to compiler_types.h and always pair it with notrace and some comments in v2.

#4 lore raw
Quoted 8 lines · Marco Elver

Mark says that there may be an issue with using this in combination with ftrace because arm64 tracing relies on AAPCS. Probably not just arm64, but also other architectures (x86?).

To make this safe, I'm going to move __preserve_most to compiler_types.h and always pair it with notrace and some comments in v2.

Sounds good, thanks! The patch here was otherwise good in terms of compiler_attributes.h.

I was also thinking about the implications for Rust. I guess if we need to call them, we will go through a C helper for the moment, and later on if we get cross-language LTO (or the LLVM IR hacks), it will hopefully not be a performance penalty.

Cheers,
Miguel

maintainer
#5 lore raw
In reply to Marco Elver
Quoted 9 lines · On Wed, 2 Aug 2023 17:06:37 +0200 Marco Elver <[email protected]> wrote:

[1]: "On X86-64 and AArch64 targets, this attribute changes the calling convention of a function. The preserve_most calling convention attempts to make the code in the caller as unintrusive as possible. This convention behaves identically to the C calling convention on how arguments and return values are passed, but it uses a different set of caller/callee-saved registers. This alleviates the burden of saving and recovering a large register set before and after the call in the caller."

Quoted 7 lines

[1] https://clang.llvm.org/docs/AttributeReference.html#preserve-most

Use of this attribute results in better code generation for calls to very rarely called functions, such as error-reporting functions, or rarely executed slow paths.

Introduce the attribute to compiler_attributes.h.

That sounds fairly radical. And no changes are needed for assembly code or asm statements?

I'll add "LLVM" to the patch title to make it clear that gcc isn't affected.

#6 lore raw
Quoted 14 lines · Andrew Morton
Quoted 11 lines · On Wed, 2 Aug 2023 17:06:37 +0200 Marco Elver <[email protected]> wrote:

[1]: "On X86-64 and AArch64 targets, this attribute changes the calling convention of a function. The preserve_most calling convention attempts to make the code in the caller as unintrusive as possible. This convention behaves identically to the C calling convention on how arguments and return values are passed, but it uses a different set of caller/callee-saved registers. This alleviates the burden of saving and recovering a large register set before and after the call in the caller."

[1] https://clang.llvm.org/docs/AttributeReference.html#preserve-most

Quoted 8 lines
Quoted 5 lines

Use of this attribute results in better code generation for calls to very rarely called functions, such as error-reporting functions, or rarely executed slow paths.

Introduce the attribute to compiler_attributes.h.

That sounds fairly radical. And no changes are needed for assembly code or asm statements?

The callee in this case is supposed to save the registers and restore them. If the caller (such as in asm) redundantly saves the registers as well, this would be safe although redundant. That being said, there are no plans to call functions marked with the attribute from asm. Only issue would be if someone implements such a function in asm with a C decl with the attribute - but also, there are no plans to do this.

I'll need to spin a v2 to always add notrace: one way this can go wrong if something inserts itself between the callers and callee, such as tracing would.