Skip to contentSkip to conversation

x86: Alias memset to __builtin_memset.

PATCH In review Clement Courbet 9 participants 19 messages Mar 23, 2020 – Mar 27, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86: Alias memset to __builtin_memset.

Recent compilers know the meaning of builtins (memset, memcpy, ...) and can replace calls by inline code when deemed better. For example, memset(p, 0, 4) will be lowered to a four-byte zero store.

When using -ffreestanding (this is the case e.g. building on clang), these optimizations are disabled. This means that **all** memsets, including those with small, constant sizes, will result in an actual call to memset.

We have identified several spots where we have high CPU usage because of this. For example, a single one of these memsets is responsible for about 0.3% of our total CPU usage in the kernel.

Aliasing memset to __builtin_memset allows the compiler to perform this optimization even when -ffreestanding is used. There is no change when -ffreestanding is not used.

Below is a diff (clang) for update_sg_lb_stats(), which includes the aforementionned hot memset:
memset(sgs, 0, sizeof(*sgs));

Diff:
movq %rsi, %rbx ~~~> movq $0x0, 0x40(%r8)
movq %rdi, %r15 ~~~> movq $0x0, 0x38(%r8)
movl $0x48, %edx ~~~> movq $0x0, 0x30(%r8)
movq %r8, %rdi ~~~> movq $0x0, 0x28(%r8)
xorl %esi, %esi ~~~> movq $0x0, 0x20(%r8)
callq <memset> ~~~> movq $0x0, 0x18(%r8)
~~~> movq $0x0, 0x10(%r8) ~~~> movq $0x0, 0x8(%r8) ~~~> movq $0x0, (%r8)

In terms of code size, this grows the clang-built kernel a bit (+0.022%):
440285512 vmlinux.clang.after
440383608 vmlinux.clang.before

---
 arch/x86/include/asm/string_64.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/x86/include/asm/string_64.h +10 -0
index 75314c3dbe47..7073c25aa4a3 100644
@@ -18,6 +18,15 @@ extern void *__memcpy(void *to, const void *from, size_t len);
 void *memset(void *s, int c, size_t n);
 void *__memset(void *s, int c, size_t n);
 
+/* Recent compilers can generate much better code for known size and/or
+ * fill values, and will fallback on `memset` if they fail.
+ * We alias `memset` to `__builtin_memset` explicitly to inform the compiler to
+ * perform this optimization even when -ffreestanding is used.
+ */
+#if (__GNUC__ >= 4)
+#define memset(s, c, count) __builtin_memset(s, c, count)
+#endif
+
 #define __HAVE_ARCH_MEMSET16
 static inline void *memset16(uint16_t *s, uint16_t v, size_t n)
 {
@@ -74,6 +83,7 @@ int strcmp(const char *cs, const char *ct);
 #undef memcpy
 #define memcpy(dst, src, len) __memcpy(dst, src, len)
 #define memmove(dst, src, len) __memmove(dst, src, len)
+#undef memset
 #define memset(s, c, n) __memset(s, c, n)
 
 #ifndef __NO_FORTIFY
Signature · 1 line
2.25.1.696.g5e7596f4ac-goog
#1 lore raw
Quoted 48 lines · Clement Courbet' via Clang Built Linux

Recent compilers know the meaning of builtins (memset, memcpy, ...) and can replace calls by inline code when deemed better. For example, memset(p, 0, 4) will be lowered to a four-byte zero store.

When using -ffreestanding (this is the case e.g. building on clang), these optimizations are disabled. This means that **all** memsets, including those with small, constant sizes, will result in an actual call to memset.

We have identified several spots where we have high CPU usage because of this. For example, a single one of these memsets is responsible for about 0.3% of our total CPU usage in the kernel.

Aliasing memset to __builtin_memset allows the compiler to perform this optimization even when -ffreestanding is used. There is no change when -ffreestanding is not used.

Below is a diff (clang) for update_sg_lb_stats(), which includes the aforementionned hot memset:
memset(sgs, 0, sizeof(*sgs));

Diff:
movq %rsi, %rbx ~~~> movq $0x0, 0x40(%r8)
movq %rdi, %r15 ~~~> movq $0x0, 0x38(%r8)
movl $0x48, %edx ~~~> movq $0x0, 0x30(%r8)
movq %r8, %rdi ~~~> movq $0x0, 0x28(%r8)
xorl %esi, %esi ~~~> movq $0x0, 0x20(%r8)
callq <memset> ~~~> movq $0x0, 0x18(%r8)
~~~> movq $0x0, 0x10(%r8) ~~~> movq $0x0, 0x8(%r8) ~~~> movq $0x0, (%r8)

In terms of code size, this grows the clang-built kernel a bit (+0.022%):
440285512 vmlinux.clang.after
440383608 vmlinux.clang.before

---
 arch/x86/include/asm/string_64.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/x86/include/asm/string_64.h +0 -0
index 75314c3dbe47..7073c25aa4a3 100644

arch/x86/include/asm/string_64.h extern void *__memcpy(void *to, const void *from, size_t len); · line 26

Quoted 9 lines
arch/x86/include/asm/string_64.h +6 -0
@@ -18,6 +18,15 @@ extern void *__memcpy(void *to, const void *from, size_t len);
 void *memset(void *s, int c, size_t n);
 void *__memset(void *s, int c, size_t n);
 
+/* Recent compilers can generate much better code for known size and/or
+ * fill values, and will fallback on `memset` if they fail.
+ * We alias `memset` to `__builtin_memset` explicitly to inform the compiler to
+ * perform this optimization even when -ffreestanding is used.
+ */
+#if (__GNUC__ >= 4)

This ifdef is unnecessary, this will always be true because the minimum supported GCC version is 4.6 and clang pretends it is 4.2.1. It appears the Intel compiler will pretend to be whatever whatever GCC version the host is using (no idea if it is still used by anyone or truly supported but still worth mentioning) so it would still always be true because of the GCC 4.6 requirement.

I cannot comment on the validity of the patch even though the reasoning seems sound to me.

Cheers,
Nathan

Quoted 17 lines
arch/x86/include/asm/string_64.h +4 -0
+#define memset(s, c, count) __builtin_memset(s, c, count)
+#endif
+
 #define __HAVE_ARCH_MEMSET16
 static inline void *memset16(uint16_t *s, uint16_t v, size_t n)
 {
@@ -74,6 +83,7 @@ int strcmp(const char *cs, const char *ct);
 #undef memcpy
 #define memcpy(dst, src, len) __memcpy(dst, src, len)
 #define memmove(dst, src, len) __memmove(dst, src, len)
+#undef memset
 #define memset(s, c, n) __memset(s, c, n)
 
 #ifndef __NO_FORTIFY
Signature · 1 line
2.25.1.696.g5e7596f4ac-goog
#2 lore raw
In reply to Clement Courbet
Thank you for the patch! Yet something to improve:

Hi Clement,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on tip/x86/core]
[also build test ERROR on v5.6-rc7 next-20200323]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Clement-Courbet/x86-Alias-memset-to-__builtin_memset/20200324-004007
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 248ed51048c40d36728e70914e38bffd7821da57
config: x86_64-randconfig-h003-20200323 (attached as .config)
compiler: clang version 11.0.0 (https://github.com/llvm/llvm-project 006244152d6c7dd6a390ff89b236cc7801834b46)

reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
COMPILER=clang make.cross ARCH=x86_64

If you fix the issue, kindly add following tag

All errors (new ones prefixed by >>):

In file included from scripts/mod/devicetable-offsets.c:3:
In file included from include/linux/mod_devicetable.h:13:
In file included from include/linux/uuid.h:12:

>> include/linux/string.h:358:24: error: definition of builtin function '__builtin_memset'
   __FORTIFY_INLINE void *memset(void *p, int c, __kernel_size_t size)
                          ^
   arch/x86/include/asm/string_64.h:27:29: note: expanded from macro 'memset'
   #define memset(s, c, count) __builtin_memset(s, c, count)
                               ^

1 error generated.
make[2]: *** [scripts/Makefile.build:99: scripts/mod/devicetable-offsets.s] Error 1 make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1112: prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:179: sub-make] Error 2
27 real 8 user 13 sys 79.89% cpu make prepare

vim +/__builtin_memset +358 include/linux/string.h

6974f0c4555e28 Daniel Micay 2017-07-12 357

6974f0c4555e28 Daniel Micay 2017-07-12 @358  __FORTIFY_INLINE void *memset(void *p, int c, __kernel_size_t size)
6974f0c4555e28 Daniel Micay 2017-07-12  359  {
6974f0c4555e28 Daniel Micay 2017-07-12  360  	size_t p_size = __builtin_object_size(p, 0);
6974f0c4555e28 Daniel Micay 2017-07-12  361  	if (__builtin_constant_p(size) && p_size < size)
6974f0c4555e28 Daniel Micay 2017-07-12  362  		__write_overflow();
6974f0c4555e28 Daniel Micay 2017-07-12  363  	if (p_size < size)
6974f0c4555e28 Daniel Micay 2017-07-12  364  		fortify_panic(__func__);
6974f0c4555e28 Daniel Micay 2017-07-12  365  	return __builtin_memset(p, c, size);
6974f0c4555e28 Daniel Micay 2017-07-12  366  }

6974f0c4555e28 Daniel Micay 2017-07-12 367

:::::: The code@line 358 was first introduced by commit
:::::: 6974f0c4555e285ab217cee58b6e874f776ff409 include/linux/string.h: add the option of fortified string.h functions

:::::: TO: Daniel Micay <[email protected]> :::::: CC: Linus Torvalds <[email protected]>

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#3 lore raw
Quoted 28 lines · kbuild test robot

Hi Clement,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on tip/x86/core]
[also build test ERROR on v5.6-rc7 next-20200323]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Clement-Courbet/x86-Alias-memset-to-__builtin_memset/20200324-004007
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 248ed51048c40d36728e70914e38bffd7821da57
config: x86_64-randconfig-h003-20200323 (attached as .config)
compiler: clang version 11.0.0 (https://github.com/llvm/llvm-project 006244152d6c7dd6a390ff89b236cc7801834b46)

reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
COMPILER=clang make.cross ARCH=x86_64

If you fix the issue, kindly add following tag

All errors (new ones prefixed by >>):

In file included from scripts/mod/devicetable-offsets.c:3:
In file included from include/linux/mod_devicetable.h:13:
In file included from include/linux/uuid.h:12:

Quoted 6 lines
Quoted 1 line
Quoted 1 line

include/linux/string.h:358:24: error: definition of builtin function '__builtin_memset'

__FORTIFY_INLINE void *memset(void *p, int c, __kernel_size_t size)

                          ^
   arch/x86/include/asm/string_64.h:27:29: note: expanded from macro 'memset'
   #define memset(s, c, count) __builtin_memset(s, c, count)
                               ^

This needs to be within #ifndef CONFIG_FORTIFY_SOURCE. FORTIFY already does this. Additionally, shouldn't this just be done universally, instead of in x86-specific code?

-Kees

Quoted 36 lines

1 error generated.
make[2]: *** [scripts/Makefile.build:99: scripts/mod/devicetable-offsets.s] Error 1 make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1112: prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:179: sub-make] Error 2
27 real 8 user 13 sys 79.89% cpu make prepare

vim +/__builtin_memset +358 include/linux/string.h

6974f0c4555e28 Daniel Micay 2017-07-12 357

6974f0c4555e28 Daniel Micay 2017-07-12 @358  __FORTIFY_INLINE void *memset(void *p, int c, __kernel_size_t size)
6974f0c4555e28 Daniel Micay 2017-07-12  359  {
6974f0c4555e28 Daniel Micay 2017-07-12  360  	size_t p_size = __builtin_object_size(p, 0);
6974f0c4555e28 Daniel Micay 2017-07-12  361  	if (__builtin_constant_p(size) && p_size < size)
6974f0c4555e28 Daniel Micay 2017-07-12  362  		__write_overflow();
6974f0c4555e28 Daniel Micay 2017-07-12  363  	if (p_size < size)
6974f0c4555e28 Daniel Micay 2017-07-12  364  		fortify_panic(__func__);
6974f0c4555e28 Daniel Micay 2017-07-12  365  	return __builtin_memset(p, c, size);
6974f0c4555e28 Daniel Micay 2017-07-12  366  }

6974f0c4555e28 Daniel Micay 2017-07-12 367

:::::: The code at line 358 was first introduced by commit
:::::: 6974f0c4555e285ab217cee58b6e874f776ff409 include/linux/string.h: add the option of fortified string.h functions

:::::: TO: Daniel Micay <[email protected]> :::::: CC: Linus Torvalds <[email protected]>

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Clang Built Linux" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clang-built-linux+unsubscribe(a)googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/202003240432.AGknaLzA%25lkp%40intel.com.
Signature · 1 line
Kees Cook
#4 lore raw
In reply to Clement Courbet
Quoted 10 lines · Clement Courbet' via Clang Built Linux

Recent compilers know the meaning of builtins (memset, memcpy, ...) and can replace calls by inline code when deemed better. For example, memset(p, 0, 4) will be lowered to a four-byte zero store.

When using -ffreestanding (this is the case e.g. building on clang), these optimizations are disabled. This means that **all** memsets, including those with small, constant sizes, will result in an actual call to memset.

Isn't this only added for 32b x86 (if I'm reading arch/x86/Makefile right)? Who's adding it for 64b?

arch/x86/Makefile has a comment:
88 # temporary until string.h is fixed
89 KBUILD_CFLAGS += -ffreestanding
Did you look into fixing that?

Quoted 20 lines
    We have identified several spots where we have high CPU usage
    because of this. For example, a single one of these memsets is
    responsible for about 0.3% of our total CPU usage in the kernel.

    Aliasing `memset` to `__builtin_memset` allows the compiler to
    perform this optimization even when -ffreestanding is used.
    There is no change when -ffreestanding is not used.

    Below is a diff (clang) for `update_sg_lb_stats()`, which
    includes the aforementionned hot memset:
        memset(sgs, 0, sizeof(*sgs));

    Diff:
        movq %rsi, %rbx        ~~~>  movq $0x0, 0x40(%r8)
        movq %rdi, %r15        ~~~>  movq $0x0, 0x38(%r8)
        movl $0x48, %edx       ~~~>  movq $0x0, 0x30(%r8)
        movq %r8, %rdi         ~~~>  movq $0x0, 0x28(%r8)
        xorl %esi, %esi        ~~~>  movq $0x0, 0x20(%r8)
        callq <memset>         ~~~>  movq $0x0, 0x18(%r8)
Quoted 8 lines
                               ~~~>  movq $0x0, 0x10(%r8)
                               ~~~>  movq $0x0, 0x8(%r8)
                               ~~~>  movq $0x0, (%r8)

    In terms of code size, this grows the clang-built kernel a
    bit (+0.022%):
    440285512 vmlinux.clang.after
    440383608 vmlinux.clang.before

The before number looks bigger? Did it shrink in size, or was the above mislabeled?

Quoted 35 lines
---
 arch/x86/include/asm/string_64.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/x86/include/asm/string_64.h +10 -1
index 75314c3dbe47..7073c25aa4a3 100644
@@ -18,6 +18,15 @@ extern void *__memcpy(void *to, const void *from, size_t len);
 void *memset(void *s, int c, size_t n);
 void *__memset(void *s, int c, size_t n);
 
+/* Recent compilers can generate much better code for known size and/or
+ * fill values, and will fallback on `memset` if they fail.
+ * We alias `memset` to `__builtin_memset` explicitly to inform the compiler to
+ * perform this optimization even when -ffreestanding is used.
+ */
+#if (__GNUC__ >= 4)
+#define memset(s, c, count) __builtin_memset(s, c, count)
+#endif
+
 #define __HAVE_ARCH_MEMSET16
 static inline void *memset16(uint16_t *s, uint16_t v, size_t n)
 {
@@ -74,6 +83,7 @@ int strcmp(const char *cs, const char *ct);
 #undef memcpy
 #define memcpy(dst, src, len) __memcpy(dst, src, len)
 #define memmove(dst, src, len) __memmove(dst, src, len)
+#undef memset
 #define memset(s, c, n) __memset(s, c, n)
 
 #ifndef __NO_FORTIFY
--
Signature · 2 lines
Thanks,
~Nick Desaulniers
#5 lore raw
Quoted 14 lines · Nick Desaulniers
Quoted 10 lines · Clement Courbet' via Clang Built Linux

Recent compilers know the meaning of builtins (memset, memcpy, ...) and can replace calls by inline code when deemed better. For example, memset(p, 0, 4) will be lowered to a four-byte zero store.

When using -ffreestanding (this is the case e.g. building on clang), these optimizations are disabled. This means that **all** memsets, including those with small, constant sizes, will result in an actual call to memset.

Quoted 20 lines

Isn't this only added for 32b x86 (if I'm reading arch/x86/Makefile right)? Who's adding it for 64b?

arch/x86/Makefile has a comment:
88 # temporary until string.h is fixed
89 KBUILD_CFLAGS += -ffreestanding
Did you look into fixing that?

Quoted 12 lines

We have identified several spots where we have high CPU usage because of this. For example, a single one of these memsets is responsible for about 0.3% of our total CPU usage in the kernel.

Aliasing memset to __builtin_memset allows the compiler to perform this optimization even when -ffreestanding is used. There is no change when -ffreestanding is not used.

Below is a diff (clang) for update_sg_lb_stats(), which includes the aforementionned hot memset:
memset(sgs, 0, sizeof(*sgs));

Further, make CC=clang -j71 kernel/sched/fair.o V=1 doesn't show -ffreestanding being used. Any idea where it's coming from in your build? Maybe a local modification to be removed?

Quoted 59 lines
Quoted 16 lines
    Diff:
        movq %rsi, %rbx        ~~~>  movq $0x0, 0x40(%r8)
        movq %rdi, %r15        ~~~>  movq $0x0, 0x38(%r8)
        movl $0x48, %edx       ~~~>  movq $0x0, 0x30(%r8)
        movq %r8, %rdi         ~~~>  movq $0x0, 0x28(%r8)
        xorl %esi, %esi        ~~~>  movq $0x0, 0x20(%r8)
        callq <memset>         ~~~>  movq $0x0, 0x18(%r8)
                               ~~~>  movq $0x0, 0x10(%r8)
                               ~~~>  movq $0x0, 0x8(%r8)
                               ~~~>  movq $0x0, (%r8)

    In terms of code size, this grows the clang-built kernel a
    bit (+0.022%):
    440285512 vmlinux.clang.after
    440383608 vmlinux.clang.before

The before number looks bigger? Did it shrink in size, or was the above mislabeled?

Quoted 27 lines
---
 arch/x86/include/asm/string_64.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/x86/include/asm/string_64.h +9 -0
index 75314c3dbe47..7073c25aa4a3 100644
@@ -18,6 +18,15 @@ extern void *__memcpy(void *to, const void *from, size_t len);
 void *memset(void *s, int c, size_t n);
 void *__memset(void *s, int c, size_t n);
 
+/* Recent compilers can generate much better code for known size and/or
+ * fill values, and will fallback on `memset` if they fail.
+ * We alias `memset` to `__builtin_memset` explicitly to inform the compiler to
+ * perform this optimization even when -ffreestanding is used.
+ */
+#if (__GNUC__ >= 4)
+#define memset(s, c, count) __builtin_memset(s, c, count)
+#endif
+
 #define __HAVE_ARCH_MEMSET16
 static inline void *memset16(uint16_t *s, uint16_t v, size_t n)
 {

arch/x86/include/asm/string_64.h int strcmp(const char *cs, const char *ct);

Quoted 8 lines
arch/x86/include/asm/string_64.h +1 -1
@@ -74,6 +83,7 @@ int strcmp(const char *cs, const char *ct);
 #undef memcpy
 #define memcpy(dst, src, len) __memcpy(dst, src, len)
 #define memmove(dst, src, len) __memmove(dst, src, len)
+#undef memset
 #define memset(s, c, n) __memset(s, c, n)
 
 #ifndef __NO_FORTIFY
--

--
Thanks,
~Nick Desaulniers

Signature · 2 lines
Thanks,
~Nick Desaulniers
#6 lore raw

PATCH 1/1 x86: Alias memset to __builtin_memset.

In reply to Clement Courbet

Thanks for the comments. Answers below.

Quoted 2 lines

This ifdef is unnecessary
This needs to be within #ifndef CONFIG_FORTIFY_SOURCE

Thanks, fixed.

Quoted 1 line

shouldn't this just be done universally ?

It looks like every architecture does its own magic with memory functions. I'm not very familiar with how the linux kernel is organized, do you have a pointer for where this would go if enabled globally ?

Quoted 3 lines

Who's adding it for 64b?
Any idea where it's coming from in your
build? Maybe a local modification to be removed?

Actually this is from our local build configuration. Apparently this is needed to build on some architectures that redefine common symbols, but the authors seemed to feel strongly that this should be on for all architectures. I've reached out to the authors for an extended rationale.
On the other hand I think that there is no reason to prevent people from building with freestanding if we can easily allow them to.

#7 lore raw
Quoted 15 lines · Clement Courbet

Thanks for the comments. Answers below.

Quoted 2 lines

This ifdef is unnecessary
This needs to be within #ifndef CONFIG_FORTIFY_SOURCE

Thanks, fixed.

Quoted 1 line

shouldn't this just be done universally ?

It looks like every architecture does its own magic with memory functions. I'm not very familiar with how the linux kernel is organized, do you have a pointer for where this would go if enabled globally ?

Quoted 5 lines
Quoted 3 lines

Who's adding it for 64b?
Any idea where it's coming from in your
build? Maybe a local modification to be removed?

Actually this is from our local build configuration. Apparently this

Not sure we should modify this for someone's downstream tree to work around an issue they introduced. If you want to file a bug internally, I'd be happy to comment on it.

Does __builtin_memset detect support for rep stosb, then patch the kernel to always use it or not? The kernel is limited in that we use -mno-sse and friends to avoid saving/restoring vector registers on context switch unless kernel_fpu_{begin|end}() is called, which the compiler doesn't insert for memcpy's.

Did you verify what this change does for other compilers?

Quoted 2 lines

is needed to build on some architectures that redefine common symbols, but the authors seemed to feel strongly that this should be

Sounds like a linkage error or issue with symbol visibility; I don't see how -ffreestanding should have any bearing on that.

Quoted 4 lines

on for all architectures. I've reached out to the authors for an extended rationale.
On the other hand I think that there is no reason to prevent people from building with freestanding if we can easily allow them to.

I disagree. The codegen in the kernel is very tricky to get right; it's somewhat an embedded system, yet provides many symbols that would have been provided by libc, yet it doesn't use vector operations for the general case. Adding -ffreestanding to optimize one hot memset in one function is using a really big hammer to solve the wrong problem.

Signature · 2 lines
Thanks,
~Nick Desaulniers
#8 lore raw

PATCH v2 1/1 x86: Alias memset to __builtin_memset.

In reply to Clement Courbet

Recent compilers know the meaning of builtins (memset, memcpy, ...) and can replace calls by inline code when deemed better. For example, memset(p, 0, 4) will be lowered to a four-byte zero store.

When using -ffreestanding (this is the case e.g. building on clang), these optimizations are disabled. This means that **all** memsets, including those with small, constant sizes, will result in an actual call to memset.

We have identified several spots where we have high CPU usage because of this. For example, a single one of these memsets is responsible for about 0.3% of our total CPU usage in the kernel.

Aliasing memset to __builtin_memset allows the compiler to perform this optimization even when -ffreestanding is used. There is no change when -ffreestanding is not used.

Below is a diff (clang) for update_sg_lb_stats(), which includes the aforementionned hot memset:
memset(sgs, 0, sizeof(*sgs));

Diff:
movq %rsi, %rbx ~~~> movq $0x0, 0x40(%r8)
movq %rdi, %r15 ~~~> movq $0x0, 0x38(%r8)
movl $0x48, %edx ~~~> movq $0x0, 0x30(%r8)
movq %r8, %rdi ~~~> movq $0x0, 0x28(%r8)
xorl %esi, %esi ~~~> movq $0x0, 0x20(%r8)
callq <memset> ~~~> movq $0x0, 0x18(%r8)
~~~> movq $0x0, 0x10(%r8) ~~~> movq $0x0, 0x8(%r8) ~~~> movq $0x0, (%r8)

In terms of code size, this grows the clang-built kernel a bit (+0.022%):
440285512 vmlinux.clang.after
440383608 vmlinux.clang.before

---

changes in v2:
- Removed ifdef(GNUC >= 4).
- Disabled change when CONFIG_FORTIFY_SOURCE is set.

---
 arch/x86/include/asm/string_64.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/x86/include/asm/string_64.h +10 -0
index 75314c3dbe47..9cfce0a840a4 100644
@@ -18,6 +18,15 @@ extern void *__memcpy(void *to, const void *from, size_t len);
 void *memset(void *s, int c, size_t n);
 void *__memset(void *s, int c, size_t n);
 
+/* Recent compilers can generate much better code for known size and/or
+ * fill values, and will fallback on `memset` if they fail.
+ * We alias `memset` to `__builtin_memset` explicitly to inform the compiler to
+ * perform this optimization even when -ffreestanding is used.
+ */
+#if !defined(CONFIG_FORTIFY_SOURCE)
+#define memset(s, c, count) __builtin_memset(s, c, count)
+#endif
+
 #define __HAVE_ARCH_MEMSET16
 static inline void *memset16(uint16_t *s, uint16_t v, size_t n)
 {
@@ -74,6 +83,7 @@ int strcmp(const char *cs, const char *ct);
 #undef memcpy
 #define memcpy(dst, src, len) __memcpy(dst, src, len)
 #define memmove(dst, src, len) __memmove(dst, src, len)
+#undef memset
 #define memset(s, c, n) __memset(s, c, n)
 
 #ifndef __NO_FORTIFY
Signature · 1 line
2.25.1.696.g5e7596f4ac-goog
#9 lore raw
Quoted 9 lines · Clement Courbet

Recent compilers know the meaning of builtins (memset, memcpy, ...) and can replace calls by inline code when deemed better. For example, memset(p, 0, 4) will be lowered to a four-byte zero store.

When using -ffreestanding (this is the case e.g. building on clang), these optimizations are disabled. This means that **all** memsets, including those with small, constant sizes, will result in an actual call to memset.

[]

Quoted 4 lines
    In terms of code size, this grows the clang-built kernel a
    bit (+0.022%):
    440285512 vmlinux.clang.after
    440383608 vmlinux.clang.before

This shows the kernel getting smaller not larger.
Is this still reversed or is this correct?

#10 lore raw

PATCH v3 1/1 x86: Alias memset to __builtin_memset.

In reply to Clement Courbet

Recent compilers know the meaning of builtins (memset, memcpy, ...) and can replace calls by inline code when deemed better. For example, memset(p, 0, 4) will be lowered to a four-byte zero store.

When using -ffreestanding (this is the case e.g. building on clang), these optimizations are disabled. This means that **all** memsets, including those with small, constant sizes, will result in an actual call to memset.

We have identified several spots where we have high CPU usage because of this. For example, a single one of these memsets is responsible for about 0.3% of our total CPU usage in the kernel.

Aliasing memset to __builtin_memset allows the compiler to perform this optimization even when -ffreestanding is used. There is no change when -ffreestanding is not used.

Below is a diff (clang) for update_sg_lb_stats(), which includes the aforementionned hot memset:
memset(sgs, 0, sizeof(*sgs));

Diff:
movq %rsi, %rbx ~~~> movq $0x0, 0x40(%r8)
movq %rdi, %r15 ~~~> movq $0x0, 0x38(%r8)
movl $0x48, %edx ~~~> movq $0x0, 0x30(%r8)
movq %r8, %rdi ~~~> movq $0x0, 0x28(%r8)
xorl %esi, %esi ~~~> movq $0x0, 0x20(%r8)
callq <memset> ~~~> movq $0x0, 0x18(%r8)
~~~> movq $0x0, 0x10(%r8) ~~~> movq $0x0, 0x8(%r8) ~~~> movq $0x0, (%r8)

In terms of code size, this shrins the clang-built kernel a bit (-0.022%):
440383608 vmlinux.clang.before
440285512 vmlinux.clang.after

---

changes in v2:
- Removed ifdef(GNUC >= 4).
- Disabled change when CONFIG_FORTIFY_SOURCE is set.

changes in v3:
- Fixed commit message: the kernel shrinks in size.

---
 arch/x86/include/asm/string_64.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/x86/include/asm/string_64.h +10 -0
index 75314c3dbe47..9cfce0a840a4 100644
@@ -18,6 +18,15 @@ extern void *__memcpy(void *to, const void *from, size_t len);
 void *memset(void *s, int c, size_t n);
 void *__memset(void *s, int c, size_t n);
 
+/* Recent compilers can generate much better code for known size and/or
+ * fill values, and will fallback on `memset` if they fail.
+ * We alias `memset` to `__builtin_memset` explicitly to inform the compiler to
+ * perform this optimization even when -ffreestanding is used.
+ */
+#if !defined(CONFIG_FORTIFY_SOURCE)
+#define memset(s, c, count) __builtin_memset(s, c, count)
+#endif
+
 #define __HAVE_ARCH_MEMSET16
 static inline void *memset16(uint16_t *s, uint16_t v, size_t n)
 {
@@ -74,6 +83,7 @@ int strcmp(const char *cs, const char *ct);
 #undef memcpy
 #define memcpy(dst, src, len) __memcpy(dst, src, len)
 #define memmove(dst, src, len) __memmove(dst, src, len)
+#undef memset
 #define memset(s, c, n) __memset(s, c, n)
 
 #ifndef __NO_FORTIFY
Signature · 1 line
2.25.1.696.g5e7596f4ac-goog
#11 lore raw

Hi all!

Sry for being late at the party:

On 24/03/2020 16:59, Clement Courbet wrote:
[...]

Quoted 9 lines
---
  arch/x86/include/asm/string_64.h | 10 ++++++++++
  1 file changed, 10 insertions(+)
arch/x86/include/asm/string_64.h +0 -0
index 75314c3dbe47..9cfce0a840a4 100644

arch/x86/include/asm/string_64.h extern void *__memcpy(void *to, const void *from, size_t len); · line 27

Quoted 10 lines
arch/x86/include/asm/string_64.h +7 -0
@@ -18,6 +18,15 @@ extern void *__memcpy(void *to, const void *from, size_t len);
  void *memset(void *s, int c, size_t n);
  void *__memset(void *s, int c, size_t n);
 
+/* Recent compilers can generate much better code for known size and/or
+ * fill values, and will fallback on `memset` if they fail.
+ * We alias `memset` to `__builtin_memset` explicitly to inform the compiler to
+ * perform this optimization even when -ffreestanding is used.
+ */
+#if !defined(CONFIG_FORTIFY_SOURCE)
+#define memset(s, c, count) __builtin_memset(s, c, count)

To be on the safe side, the usual way to write macros is like

#define memset(s, c, count) __builtin_memset((s), (c), (count))

as no one know what is passed as parameter to memset() and the extra pair of parentheses don't hurt.

And similar below (and I fear there are more places).

Or did I miss something in the Kernel?

arch/x86/include/asm/string_64.h int strcmp(const char *cs, const char *ct); · line 84

Quoted 8 lines
arch/x86/include/asm/string_64.h +2 -0
+#endif
+
  #define __HAVE_ARCH_MEMSET16
  static inline void *memset16(uint16_t *s, uint16_t v, size_t n)
  {
@@ -74,6 +83,7 @@ int strcmp(const char *cs, const char *ct);
  #undef memcpy
  #define memcpy(dst, src, len) __memcpy(dst, src, len)

#define memcpy(dst, src, len) __memcpy((dst), (src), (len))

arch/x86/include/asm/string_64.h int strcmp(const char *cs, const char *ct); · line 85

Quoted 1 line
arch/x86/include/asm/string_64.h +0 -0
  #define memmove(dst, src, len) __memmove(dst, src, len)

#define memmove(dst, src, len) __memmove((dst), (src), (len))

arch/x86/include/asm/string_64.h int strcmp(const char *cs, const char *ct); · line 87

Quoted 2 lines
arch/x86/include/asm/string_64.h +1 -0
+#undef memset
  #define memset(s, c, n) __memset(s, c, n)

#define memset(s, c, n) __memset((s), (c), (n))

arch/x86/include/asm/string_64.h int strcmp(const char *cs, const char *ct); · line 89

Quoted 3 lines
arch/x86/include/asm/string_64.h +0 -0
  #ifndef __NO_FORTIFY
 

MfG,
Bernd

Signature · 2 lines
Bernd Petrovitsch                  Email : [email protected]
                     LUGA : http://www.luga.at
#12 lore raw

PATCH 1/1 x86: Alias memset to __builtin_memset.

In reply to Clement Courbet
Quoted 2 lines

This shows the kernel getting smaller not larger.
Is this still reversed or is this correct?

Yes, sorry. This was wrong. The kernel actully shrinks. Fixed.

#13 lore raw

PATCH 1/1 x86: Alias memset to __builtin_memset.

In reply to Clement Courbet
Quoted 3 lines

Not sure we should modify this for someone's downstream tree to work around an issue they introduced. If you want to file a bug internally, I'd be happy to comment on it.

I don't have a strong opinion on that. I don't know the policy of the linux kernel w.r.t. this. There is an internal bug for this, where kernel maintainers suggested I upstream this patch.

Quoted 2 lines

Does __builtin_memset detect support for rep stosb, then patch the kernel to always use it or not?

__builtin_memset just allows the compiler to recognize that this has the semantics of a memset, exactly like it happens when -freestanding is not specified.

In terms of what compilers do when expanding the memset, it depends on the size.
gcc or clang obviously do not generate vector code when -no-sse is specified, as this would break promises.

clang inlines stores for small sizes and switches to memset as the size
increases: https://godbolt.org/z/_X16xt

gcc inlines stores for tiny sizes, then switches to repstos for larger
sizes: https://godbolt.org/z/m-G233 This behaviour is additionally configurable through command line flags: https://godbolt.org/z/wsoJpQ

Quoted 1 line

Did you verify what this change does for other compilers?

Are there other compilers are used to build the kernel on x86 ?

icc does the same as gcc and clang: https://godbolt.org/z/yCju_D

Quoted 1 line

yet it doesn't use vector operations for the general case

I'm not sure how vector operations relate to freestanding, or this change.

Quoted 3 lines

Adding -ffreestanding to optimize one hot memset in one function is using a really big hammer to solve the wrong problem.

-ffreestanding was not added to optimize anything. It was already there. If anything -ffreestanding actually pessimizes a lot of the code generation, as it prevents the compiler from recognizing the semantics of common primitives. This is what this change is trying to fix. Removing -ffreestanding from the options is another (easier?) way to fix the problem. This change is a stab at accomodating both those who want freestanding and those who want performance.

The hot memset I mentionned was just the hottest one. But as you can imagine there are many constant-size memsets spread across many functions, some of which are also very hot, the others constituting a long tail which is not negligible.

#14 lore raw

PATCH v4 1/1 x86: Alias memset to __builtin_memset.

In reply to Clement Courbet

Recent compilers know the meaning of builtins (memset, memcpy, ...) and can replace calls by inline code when deemed better. For example, memset(p, 0, 4) will be lowered to a four-byte zero store.

When using -ffreestanding (this is the case e.g. building on clang), these optimizations are disabled. This means that **all** memsets, including those with small, constant sizes, will result in an actual call to memset.

We have identified several spots where we have high CPU usage because of this. For example, a single one of these memsets is responsible for about 0.3% of our total CPU usage in the kernel.

Aliasing memset to __builtin_memset allows the compiler to perform this optimization even when -ffreestanding is used. There is no change when -ffreestanding is not used.

Below is a diff (clang) for update_sg_lb_stats(), which includes the aforementionned hot memset:
memset(sgs, 0, sizeof(*sgs));

Diff:
movq %rsi, %rbx ~~~> movq $0x0, 0x40(%r8)
movq %rdi, %r15 ~~~> movq $0x0, 0x38(%r8)
movl $0x48, %edx ~~~> movq $0x0, 0x30(%r8)
movq %r8, %rdi ~~~> movq $0x0, 0x28(%r8)
xorl %esi, %esi ~~~> movq $0x0, 0x20(%r8)
callq <memset> ~~~> movq $0x0, 0x18(%r8)
~~~> movq $0x0, 0x10(%r8) ~~~> movq $0x0, 0x8(%r8) ~~~> movq $0x0, (%r8)

In terms of code size, this shrins the clang-built kernel a bit (-0.022%):
440383608 vmlinux.clang.before
440285512 vmlinux.clang.after

---

changes in v2:
- Removed ifdef(GNUC >= 4).
- Disabled change when CONFIG_FORTIFY_SOURCE is set.

changes in v3:
- Fixed commit message: the kernel shrinks in size.

changes in v4:
- Properly parenthesize the macro.

---
 arch/x86/include/asm/string_64.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/x86/include/asm/string_64.h +10 -0
index 75314c3dbe47..1bfa825e9ad3 100644
@@ -18,6 +18,15 @@ extern void *__memcpy(void *to, const void *from, size_t len);
 void *memset(void *s, int c, size_t n);
 void *__memset(void *s, int c, size_t n);
 
+/* Recent compilers can generate much better code for known size and/or
+ * fill values, and will fallback on `memset` if they fail.
+ * We alias `memset` to `__builtin_memset` explicitly to inform the compiler to
+ * perform this optimization even when -ffreestanding is used.
+ */
+#if !defined(CONFIG_FORTIFY_SOURCE)
+#define memset(s, c, count) __builtin_memset((s), (c), (count))
+#endif
+
 #define __HAVE_ARCH_MEMSET16
 static inline void *memset16(uint16_t *s, uint16_t v, size_t n)
 {
@@ -74,6 +83,7 @@ int strcmp(const char *cs, const char *ct);
 #undef memcpy
 #define memcpy(dst, src, len) __memcpy(dst, src, len)
 #define memmove(dst, src, len) __memmove(dst, src, len)
+#undef memset
 #define memset(s, c, n) __memset(s, c, n)
 
 #ifndef __NO_FORTIFY
Signature · 1 line
2.25.1.696.g5e7596f4ac-goog
#15 lore raw

PATCH 1/1 x86: Alias memset to __builtin_memset.

In reply to Clement Courbet

I discussed with the original authors who added freestanding to our build. It turns out that it was added globally but this was just to to workaround powerpc not compiling under clang, but they felt the fix was appropriate globally.

Now Nick has dug up https://lkml.org/lkml/2019/8/29/1300, which advises against freestanding. Also, I've did some research and discovered that the original reason for using freestanding for powerpc has been fixed here:
https://lore.kernel.org/linuxppc-dev/[email protected]/

I'm going to remove -ffreestanding from downstream, so we don't really need this anymore, sorry for waisting people's time.

I wonder if the freestanding fix from the aforementioned patch is really needed though. I think that clang is actually right to point out the issue. I don't see any reason why setjmp()/longjmp() are declared as taking longs rather than ints. The implementation looks like it only ever propagates the value (in longjmp) or sets it to 1 (in setjmp), and we only ever call longjmp with integer parameters. But I'm not a PowerPC expert, so I might be misreading the code.

So it seems that we could just remove freestanding altogether and rewrite the code to:

arch/powerpc/include/asm/setjmp.h +4 -2
index 279d03a1eec6..7941ae68fe21 100644
@@ -12,7 +12,9 @@
 
 #define JMP_BUF_LEN    23
-extern long setjmp(long *);
-extern void longjmp(long *, long);
+typedef long * jmp_buf;
+
+extern int setjmp(jmp_buf);
+extern void longjmp(jmp_buf, int);
 

I'm happy to send a patch for this, and get rid of more -ffreestanding. Opinions ?

#16 lore raw
Quoted 41 lines · Clement Courbet

I discussed with the original authors who added freestanding to our build. It turns out that it was added globally but this was just to to workaround powerpc not compiling under clang, but they felt the fix was appropriate globally.

Now Nick has dug up https://lkml.org/lkml/2019/8/29/1300, which advises against freestanding. Also, I've did some research and discovered that the original reason for using freestanding for powerpc has been fixed here:
https://lore.kernel.org/linuxppc-dev/[email protected]/

I'm going to remove -ffreestanding from downstream, so we don't really need this anymore, sorry for waisting people's time.

I wonder if the freestanding fix from the aforementioned patch is really needed though. I think that clang is actually right to point out the issue. I don't see any reason why setjmp()/longjmp() are declared as taking longs rather than ints. The implementation looks like it only ever propagates the value (in longjmp) or sets it to 1 (in setjmp), and we only ever call longjmp with integer parameters. But I'm not a PowerPC expert, so I might be misreading the code.

So it seems that we could just remove freestanding altogether and rewrite the code to:

arch/powerpc/include/asm/setjmp.h +4 -2
index 279d03a1eec6..7941ae68fe21 100644
@@ -12,7 +12,9 @@
 
 #define JMP_BUF_LEN    23
-extern long setjmp(long *);
-extern void longjmp(long *, long);
+typedef long * jmp_buf;
+
+extern int setjmp(jmp_buf);
+extern void longjmp(jmp_buf, int);
 
Quoted 2 lines

I'm happy to send a patch for this, and get rid of more -ffreestanding. Opinions ?

Yes, I think the above diff and additionally cleaning up -ffreestanding from arch/powerpc/kernel/Makefile and arch/powerpc/xmon/Makefile would be a much better approach. If that's good enough to fix the warning, I kind of can't believe we didn't spot that in the original code review!

Actually, the god awful warning was:
./arch/powerpc/include/asm/setjmp.h:10:13: error: declaration of built-in function 'setjmp' requires the declaration of the 'jmp_buf' type, commonly provided in the header <setjmp.h>. [-Werror,-Wincomplete-setjmp-declaration]

extern long setjmp(long *) __attribute__((returns_twice));
            ^

So jmp_buf was missing, wasn't used, but also the long vs int confusion.

I tested the above diff, all calls to setjmp under arch/powerpc/ just compare the return value against 0. Callers of longjmp just pass 1 for the final parameter. So the above changes should be no functional change.

Signature · 2 lines
Thanks,
~Nick Desaulniers
#17 lore raw
In reply to Clement Courbet
Quoted 40 lines · Clement Courbet

I discussed with the original authors who added freestanding to our build. It turns out that it was added globally but this was just to to workaround powerpc not compiling under clang, but they felt the fix was appropriate globally.

Now Nick has dug up https://lkml.org/lkml/2019/8/29/1300, which advises against freestanding. Also, I've did some research and discovered that the original reason for using freestanding for powerpc has been fixed here:
https://lore.kernel.org/linuxppc-dev/[email protected]/

I'm going to remove -ffreestanding from downstream, so we don't really need this anymore, sorry for waisting people's time.

I wonder if the freestanding fix from the aforementioned patch is really needed though. I think that clang is actually right to point out the issue. I don't see any reason why setjmp()/longjmp() are declared as taking longs rather than ints. The implementation looks like it only ever propagates the value (in longjmp) or sets it to 1 (in setjmp), and we only ever call longjmp with integer parameters. But I'm not a PowerPC expert, so I might be misreading the code.

So it seems that we could just remove freestanding altogether and rewrite the code to:

arch/powerpc/include/asm/setjmp.h +4 -2
index 279d03a1eec6..7941ae68fe21 100644
@@ -12,7 +12,9 @@
 
 #define JMP_BUF_LEN    23
-extern long setjmp(long *);
-extern void longjmp(long *, long);
+typedef long * jmp_buf;
+
+extern int setjmp(jmp_buf);
+extern void longjmp(jmp_buf, int);
 
Quoted 2 lines

I'm happy to send a patch for this, and get rid of more -ffreestanding. Opinions ?

If it works then it looks like a much better fix than using -ffreestanding.

Please submit a patch with a change log etc. and I'd be happy to merge it.

cheers

#18 lore raw
In reply to Clement Courbet

Hi!

Quoted 12 lines · Clement Courbet
arch/powerpc/include/asm/setjmp.h +4 -2
@@ -12,7 +12,9 @@
 
 #define JMP_BUF_LEN    23
-extern long setjmp(long *);
-extern void longjmp(long *, long);
+typedef long * jmp_buf;
+
+extern int setjmp(jmp_buf);
+extern void longjmp(jmp_buf, int);
 
Quoted 2 lines

I'm happy to send a patch for this, and get rid of more -ffreestanding. Opinions ?

Pedantically, jmp_buf should be an array type. But, this will probably work fine, and it certainly is better than what we had before.

You could do
typedef long jmp_buf[JMP_BUF_LEN];
perhaps?

Thanks,

Segher