ilog2 vs. GCC inlining heuristics
Hi!
Based on the GCC PR97445 discussions, I'd like to propose following change, which should significantly decrease the amount of code in inline functions that use ilog2, but as I'm already two decades out of the Linux kernel development, I'd appreciate if some kernel developer could try that (all I have done is check that it gives the same results as before) and if it works submit it for inclusion into the kernel?
Thanks.
Improve ilog2 for constant arguments
As discussed in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97445
the const_ilog2 macro generates a lot of code which interferes badly with GCC inlining heuristics, until it can be proven that the ilog2 argument can or can't be simplified into a constant.
It can be expressed using __builtin_clzll builtin which is supported by GCC 3.4 and later and when used only in the __builtin_constant_p guarded code it ought to always fold back to a constant.
Other compilers support the same builtin for many years too.
Other option would be to change the const_ilog2 macro, though as the description says it is meant to be used also in C constant expressions, and while GCC will fold it to constant with constant argument even in those, perhaps it is better to avoid using extensions in that case.
include/linux/log2.h +2 -1
@@ -156,7 +156,8 @@ unsigned long __rounddown_pow_of_two(unsigned long n)#define ilog2 (n) \ ( \ __builtin_constant_p(n) ? \ - const_ilog2(n) : \ + ((n) < 2 ? 0 : \ + 63 - __builtin_clzll (n)) : \ (sizeof(n) <= 4) ? \ __ilog2_u32(n) : \ __ilog2_u64(n) \
Jakub
Le 21/10/2020 à 15:27, Jakub Jelinek a écrit :
Quoted 36 lines
Hi!
Based on the GCC PR97445 discussions, I'd like to propose following change, which should significantly decrease the amount of code in inline functions that use ilog2, but as I'm already two decades out of the Linux kernel development, I'd appreciate if some kernel developer could try that (all I have done is check that it gives the same results as before) and if it works submit it for inclusion into the kernel?
Thanks.
Improve ilog2 for constant arguments
As discussed in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97445
theconst_ilog2macro generates a lot of code which interferes badly with GCC inlining heuristics, until it can be proven that the ilog2 argument can or can't be simplified into a constant.It can be expressed using
__builtin_clzllbuiltin which is supported by GCC 3.4 and later and when used only in the__builtin_constant_pguarded code it ought to always fold back to a constant.
Other compilers support the same builtin for many years too.Other option would be to change the
const_ilog2macro, though as the description says it is meant to be used also in C constant expressions, and while GCC will fold it to constant with constant argument even in those, perhaps it is better to avoid using extensions in that case.Signed-off-by Jakub Jelinek <[email protected]>include/linux/log2.h +0
-0
include/linux/log2.h unsigned long __rounddown_pow_of_two(unsigned long n) · line 160
Quoted 6 lines
include/linux/log2.h +2
-1@@ -156,7 +156,8 @@ unsigned long __rounddown_pow_of_two(unsigned long n)#define ilog2 (n) \ ( \ __builtin_constant_p(n) ? \ - const_ilog2(n) : \ + ((n) < 2 ? 0 : \ + 63 - __builtin_clzll (n)) : \
That is likely to work only on 64 bits targets.
On 32 bits targets I guess it needs to be 31 - __builtin_clz (n), doesn't it ?
Quoted 6 lines
include/linux/log2.h +0
-0(sizeof (n) <=4 ) ? \__ilog2_u32 (n) : \__ilog2_u64 (n) \Jakub
Christophe
Quoted 11 lines · Christophe Leroy
include/linux/log2.h unsigned long __rounddown_pow_of_two(unsigned long n) · line 160
Quoted 9 lines
include/linux/log2.h +2
-1@@ -156,7 +156,8 @@ unsigned long __rounddown_pow_of_two(unsigned long n)#define ilog2 (n) \ ( \ __builtin_constant_p(n) ? \ - const_ilog2(n) : \ + ((n) < 2 ? 0 : \ + 63 - __builtin_clzll (n)) : \That is likely to work only on 64 bits targets.
No, it should work on all. __builtin_clzll argument is long long, which should be 64-bit on all architectures Linux kernel supports (and all architectures GCC supports - ok, avr with -mint8 doesn't, but it isn't C requirements compatible in that case).const_ilog2 also uses long long and will work with any long long or unsigned long long value (but e.g. not with __uint128_t values on 64-bit architectures, guess the kernel doesn't need that though). And similarly, for the non-constant operands the sizeof below ensures that __ilog2_u64 is used (even on 32-bit arches) and will handle that.
Quoted 8 lines
On 32 bits targets I guess it needs to be 31 -
__builtin_clz(n), doesn't it ?Quoted 6 lines
include/linux/log2.h +0
-0(sizeof (n) <=4 ) ? \__ilog2_u32 (n) : \__ilog2_u64 (n) \Jakub
Jakub
Quoted 8 lines · Jakub Jelinek
Hi!
Based on the GCC PR97445 discussions, I'd like to propose following change, which should significantly decrease the amount of code in inline functions that use ilog2, but as I'm already two decades out of the Linux kernel development, I'd appreciate if some kernel developer could try that (all I have done is check that it gives the same results as before) and if it works submit it for inclusion into the kernel?
I'll stick it in my queue and feed it to the robots.
Le 21/10/2020 à 17:19, Peter Zijlstra a écrit :
Quoted 12 lines · Jakub Jelinek
Quoted 8 lines
Hi!
Based on the GCC PR97445 discussions, I'd like to propose following change, which should significantly decrease the amount of code in inline functions that use ilog2, but as I'm already two decades out of the Linux kernel development, I'd appreciate if some kernel developer could try that (all I have done is check that it gives the same results as before) and if it works submit it for inclusion into the kernel?
I'll stick it in my queue and feed it to the robots.
I did a mpc885_ads_defconfig build with your patch. That's far better, even better than with gcc 9 without the patch.
I only have two instances of get_order() in vmlinux, one of it is used twice, the other is never user.
With -Winline, the reason for not inlining is for both because "the call is unlikely and the code size would grow"
Christophe
Quoted 16 lines · Christophe Leroy
Le 21/10/2020 à 17:19, Peter Zijlstra a écrit :
Quoted 12 lines · Jakub Jelinek
Quoted 8 lines
Hi!
Based on the GCC PR97445 discussions, I'd like to propose following change, which should significantly decrease the amount of code in inline functions that use ilog2, but as I'm already two decades out of the Linux kernel development, I'd appreciate if some kernel developer could try that (all I have done is check that it gives the same results as before) and if it works submit it for inclusion into the kernel?
I'll stick it in my queue and feed it to the robots.
Quoted 7 lines
I did a
mpc885_ads_defconfigbuild with your patch. That's far better, even better than with gcc 9 without the patch.I only have two instances of
get_order()in vmlinux, one of it is used twice, the other is never user.With -Winline, the reason for not inlining is for both because "the call is unlikely and the code size would grow"
Christophe
Do we want a lib/test_log2.c test? Although Jakub says that he already verified that is gives the same results as the previous code.
I ran a (new) lib/test_log2.ko before and after Jakub's patch and got the same results, although I am not claiming that it has an exhaustive set of tests in it. [it is attached]
Signature · 1 line
~Randy
From: Randy Dunlap <[email protected]>
// modified from lib/test_bitops.c
--- lib/Kconfig.debug | 11 +++ lib/Makefile | 1 lib/test_log2.c | 139 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 151 insertions(+)
lib/Kconfig.debug +11 -0
@@ -2136,6 +2136,17 @@ config TEST_BITOPS If unsure, say N. +config TEST_LOG2 + tristate "Test module for log2 interfaces" + depends on m + help + This builds the "test_log2" module that does basic testing of + all <linux/log2.h> interfaces. + It has no dependencies and doesn't run or load unless + explicitly requested by name. For example: modprobe test_log2. + + If unsure, say N. + config TEST_VMALLOC tristate "Test module for stress/performance analysis of vmalloc allocator" default n
lib/Makefile +1 -0
@@ -62,6 +62,7 @@ obj-$(CONFIG_TEST_BPF) += test_bpf.o obj-$(CONFIG_TEST_FIRMWARE) += test_firmware.o obj-$(CONFIG_TEST_BITOPS) += test_bitops.o CFLAGS_test_bitops.o += -Werror +obj-$(CONFIG_TEST_LOG2) += test_log2.o obj-$(CONFIG_TEST_SYSCTL) += test_sysctl.o obj-$(CONFIG_TEST_HASH) += test_hash.o test_siphash.o obj-$(CONFIG_TEST_IDA) += test_ida.o
lib/test_log2.c new file +139 -0
139 lines not on this page · raw message
Quoted 3 lines · Randy Dunlap
I ran a (new)
lib/test_log2.kobefore and after Jakub's patch and got the same results, although I am not claiming that it has an exhaustive set of tests in it. [it is attached]
The test most likely tests the non-constant paths (which is something I haven't changed), unless all the loops would be completely unrolled, all the arrays determined to be never written to and turned into const and all the loads from those arrays optimized into constants.
If you want to test both the non-constant and constant paths, it would be good to make sure to guarantee that. For non-constant, e.g. use those arrays but add some optimization barrier so that the compiler can't optimize them into constants (e.g. make the arrays volatile, or add __asm__ ("" : : "g" (test_power_of_2) : "memory");
etc. barriers at the start of test function etc.
And for constants, perhaps put the test values into macros, through which to place it both
#define TEST_ILOG2_UINT \
TEST (1, 0) TEST (2, 1) TEST (4, 2) TEST (~0U, 31) \
TEST (0x80000000U, 31)
and once define TEST(x, y) as { x, y }, then #undef TEST
and in another place as if (ilog2(x) != y) ...
Also, you do want to test even behavior of ilog2 at 0, e.g. the powerpc fls/fls64 implementations are incorrect for that value as could be seen with -fsanitize=undefined.
And you do want to test also unsigned long long.
Jakub
The following commit has been merged into the core/core branch of tip:
The following commit has been merged into the core/core branch of tip:
Commit-ID: ecbd43f6728a5cf79c8b50ed326658e9181531b1 Gitweb: https://git.kernel.org/tip/ecbd43f6728a5cf79c8b50ed326658e9181531b1 Author: Jakub Jelinek <[email protected]> AuthorDate: Wed, 21 Oct 2020 15:27:18 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 19 Nov 2020 11:26:18 +01:00
ilog2 vs. GCC inlining heuristics
Hi!
Based on the GCC PR97445 discussions, I'd like to propose following change, which should significantly decrease the amount of code in inline functions that use ilog2, but as I'm already two decades out of the Linux kernel development, I'd appreciate if some kernel developer could try that (all I have done is check that it gives the same results as before) and if it works submit it for inclusion into the kernel?
Thanks.
Improve ilog2 for constant arguments
As discussed in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97445
the const_ilog2 macro generates a lot of code which interferes badly with GCC inlining heuristics, until it can be proven that the ilog2 argument can or can't be simplified into a constant.
It can be expressed using __builtin_clzll builtin which is supported by GCC 3.4 and later and when used only in the __builtin_constant_p guarded code it ought to always fold back to a constant.
Other compilers support the same builtin for many years too.
Other option would be to change the const_ilog2 macro, though as the description says it is meant to be used also in C constant expressions, and while GCC will fold it to constant with constant argument even in those, perhaps it is better to avoid using extensions in that case.
--- include/linux/log2.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/log2.h +2 -1
@@ -156,7 +156,8 @@ unsigned long __rounddown_pow_of_two(unsigned long n)#define ilog2 (n) \ ( \ __builtin_constant_p(n) ? \ - const_ilog2(n) : \ + ((n) < 2 ? 0 : \ + 63 - __builtin_clzll (n)) : \ (sizeof(n) <= 4) ? \ __ilog2_u32(n) : \ __ilog2_u64(n) \
Sorry, I typoed the branch name. I'll make this branch go away.
Anyway, Jacub, your patch seems to not upset the robots, so I'll go post it properly for you.
Quoted 60 lines · On Fri, Nov 20, 2020 at 12:33:58PM -0000, tip-bot2 for Jakub Jelinek wrote:
The following commit has been merged into the core/core branch of tip:
Commit-ID: ecbd43f6728a5cf79c8b50ed326658e9181531b1 Gitweb: https://git.kernel.org/tip/ecbd43f6728a5cf79c8b50ed326658e9181531b1 Author: Jakub Jelinek <[email protected]> AuthorDate: Wed, 21 Oct 2020 15:27:18 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 19 Nov 2020 11:26:18 +01:00ilog2 vs. GCC inlining heuristics
Hi!
Based on the GCC PR97445 discussions, I'd like to propose following change, which should significantly decrease the amount of code in inline functions that use ilog2, but as I'm already two decades out of the Linux kernel development, I'd appreciate if some kernel developer could try that (all I have done is check that it gives the same results as before) and if it works submit it for inclusion into the kernel?
Thanks.
Improve ilog2 for constant arguments
As discussed in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97445
theconst_ilog2macro generates a lot of code which interferes badly with GCC inlining heuristics, until it can be proven that the ilog2 argument can or can't be simplified into a constant.It can be expressed using
__builtin_clzllbuiltin which is supported by GCC 3.4 and later and when used only in the__builtin_constant_pguarded code it ought to always fold back to a constant.
Other compilers support the same builtin for many years too.Other option would be to change the
const_ilog2macro, though as the description says it is meant to be used also in C constant expressions, and while GCC will fold it to constant with constant argument even in those, perhaps it is better to avoid using extensions in that case.Signed-off-by Jakub Jelinek <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- include/linux/log2.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/log2.h +2
-1@@ -156,7 +156,8 @@ unsigned long __rounddown_pow_of_two(unsigned long n)#define ilog2 (n) \ ( \ __builtin_constant_p(n) ? \ - const_ilog2(n) : \ + ((n) < 2 ? 0 : \ + 63 - __builtin_clzll (n)) : \ (sizeof(n) <= 4) ? \ __ilog2_u32(n) : \ __ilog2_u64(n) \