Skip to contentSkip to conversation

ilog2 vs. GCC inlining heuristics

PATCH Applied Jakub Jelinek 5 participants 9 messages Oct 21, 2020 – Nov 20, 2020 Library code

lore
Share & tools
lore
#0 lore raw

Hi!

Based on the GCC PR97445 discussions, I'd like to propose following change, which should significantly decrease the amount of code in inline functions that use ilog2, but as I'm already two decades out of the Linux kernel development, I'd appreciate if some kernel developer could try that (all I have done is check that it gives the same results as before) and if it works submit it for inclusion into the kernel?

Thanks.

Improve ilog2 for constant arguments

As discussed in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97445
the const_ilog2 macro generates a lot of code which interferes badly with GCC inlining heuristics, until it can be proven that the ilog2 argument can or can't be simplified into a constant.

It can be expressed using __builtin_clzll builtin which is supported by GCC 3.4 and later and when used only in the __builtin_constant_p guarded code it ought to always fold back to a constant.
Other compilers support the same builtin for many years too.

Other option would be to change the const_ilog2 macro, though as the description says it is meant to be used also in C constant expressions, and while GCC will fold it to constant with constant argument even in those, perhaps it is better to avoid using extensions in that case.

include/linux/log2.h +2 -1
index c619ec6eff4a..4307d3477642 100644
@@ -156,7 +156,8 @@ unsigned long __rounddown_pow_of_two(unsigned long n)
 #define ilog2(n) \
 ( \
 	__builtin_constant_p(n) ?	\
-	const_ilog2(n) :		\
+	((n) < 2 ? 0 :			\
+	 63 - __builtin_clzll (n)) :	\
 	(sizeof(n) <= 4) ?		\
 	__ilog2_u32(n) :		\
 	__ilog2_u64(n)			\

Jakub

#1 lore raw

Le 21/10/2020 à 15:27, Jakub Jelinek a écrit :

Quoted 36 lines

Hi!

Based on the GCC PR97445 discussions, I'd like to propose following change, which should significantly decrease the amount of code in inline functions that use ilog2, but as I'm already two decades out of the Linux kernel development, I'd appreciate if some kernel developer could try that (all I have done is check that it gives the same results as before) and if it works submit it for inclusion into the kernel?

Thanks.

Improve ilog2 for constant arguments

As discussed in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97445
the const_ilog2 macro generates a lot of code which interferes badly with GCC inlining heuristics, until it can be proven that the ilog2 argument can or can't be simplified into a constant.

It can be expressed using __builtin_clzll builtin which is supported by GCC 3.4 and later and when used only in the __builtin_constant_p guarded code it ought to always fold back to a constant.
Other compilers support the same builtin for many years too.

Other option would be to change the const_ilog2 macro, though as the description says it is meant to be used also in C constant expressions, and while GCC will fold it to constant with constant argument even in those, perhaps it is better to avoid using extensions in that case.

include/linux/log2.h +0 -0
index c619ec6eff4a..4307d3477642 100644

include/linux/log2.h unsigned long __rounddown_pow_of_two(unsigned long n) · line 160

Quoted 6 lines
include/linux/log2.h +2 -1
@@ -156,7 +156,8 @@ unsigned long __rounddown_pow_of_two(unsigned long n)
  #define ilog2(n) \
  ( \
  	__builtin_constant_p(n) ?	\
-	const_ilog2(n) :		\
+	((n) < 2 ? 0 :			\
+	 63 - __builtin_clzll (n)) :	\

That is likely to work only on 64 bits targets.
On 32 bits targets I guess it needs to be 31 - __builtin_clz (n), doesn't it ?

Quoted 6 lines
include/linux/log2.h +0 -0
  	(sizeof(n) <= 4) ?		\
  	__ilog2_u32(n) :		\
  	__ilog2_u64(n)			\

Jakub

Christophe

#2 lore raw
Quoted 11 lines · Christophe Leroy

include/linux/log2.h unsigned long __rounddown_pow_of_two(unsigned long n) · line 160

Quoted 9 lines
include/linux/log2.h +2 -1
@@ -156,7 +156,8 @@ unsigned long __rounddown_pow_of_two(unsigned long n)
  #define ilog2(n) \
  ( \
  	__builtin_constant_p(n) ?	\
-	const_ilog2(n) :		\
+	((n) < 2 ? 0 :			\
+	 63 - __builtin_clzll (n)) :	\

That is likely to work only on 64 bits targets.

No, it should work on all. __builtin_clzll argument is long long, which should be 64-bit on all architectures Linux kernel supports (and all architectures GCC supports - ok, avr with -mint8 doesn't, but it isn't C requirements compatible in that case).
const_ilog2 also uses long long and will work with any long long or unsigned long long value (but e.g. not with __uint128_t values on 64-bit architectures, guess the kernel doesn't need that though). And similarly, for the non-constant operands the sizeof below ensures that __ilog2_u64 is used (even on 32-bit arches) and will handle that.

Quoted 8 lines

On 32 bits targets I guess it needs to be 31 - __builtin_clz (n), doesn't it ?

Quoted 6 lines
include/linux/log2.h +0 -0
  	(sizeof(n) <= 4) ?		\
  	__ilog2_u32(n) :		\
  	__ilog2_u64(n)			\

Jakub

Jakub

#3 lore raw
In reply to Jakub Jelinek
Quoted 8 lines · Jakub Jelinek

Hi!

Based on the GCC PR97445 discussions, I'd like to propose following change, which should significantly decrease the amount of code in inline functions that use ilog2, but as I'm already two decades out of the Linux kernel development, I'd appreciate if some kernel developer could try that (all I have done is check that it gives the same results as before) and if it works submit it for inclusion into the kernel?

I'll stick it in my queue and feed it to the robots.

#4 lore raw

Le 21/10/2020 à 17:19, Peter Zijlstra a écrit :

Quoted 12 lines · Jakub Jelinek
Quoted 8 lines

Hi!

Based on the GCC PR97445 discussions, I'd like to propose following change, which should significantly decrease the amount of code in inline functions that use ilog2, but as I'm already two decades out of the Linux kernel development, I'd appreciate if some kernel developer could try that (all I have done is check that it gives the same results as before) and if it works submit it for inclusion into the kernel?

I'll stick it in my queue and feed it to the robots.

I did a mpc885_ads_defconfig build with your patch. That's far better, even better than with gcc 9 without the patch.

I only have two instances of get_order() in vmlinux, one of it is used twice, the other is never user.

With -Winline, the reason for not inlining is for both because "the call is unlikely and the code size would grow"

Christophe

#5 lore raw
Quoted 16 lines · Christophe Leroy

Le 21/10/2020 à 17:19, Peter Zijlstra a écrit :

Quoted 12 lines · Jakub Jelinek
Quoted 8 lines

Hi!

Based on the GCC PR97445 discussions, I'd like to propose following change, which should significantly decrease the amount of code in inline functions that use ilog2, but as I'm already two decades out of the Linux kernel development, I'd appreciate if some kernel developer could try that (all I have done is check that it gives the same results as before) and if it works submit it for inclusion into the kernel?

I'll stick it in my queue and feed it to the robots.

Quoted 7 lines

I did a mpc885_ads_defconfig build with your patch. That's far better, even better than with gcc 9 without the patch.

I only have two instances of get_order() in vmlinux, one of it is used twice, the other is never user.

With -Winline, the reason for not inlining is for both because "the call is unlikely and the code size would grow"

Christophe

Do we want a lib/test_log2.c test? Although Jakub says that he already verified that is gives the same results as the previous code.

I ran a (new) lib/test_log2.ko before and after Jakub's patch and got the same results, although I am not claiming that it has an exhaustive set of tests in it. [it is attached]

Signature · 1 line
~Randy

From: Randy Dunlap <[email protected]>

// modified from lib/test_bitops.c

---
 lib/Kconfig.debug |   11 +++
 lib/Makefile      |    1 
 lib/test_log2.c   |  139 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 151 insertions(+)
lib/Kconfig.debug +11 -0
@@ -2136,6 +2136,17 @@ config TEST_BITOPS
 
 	  If unsure, say N.
 
+config TEST_LOG2
+	tristate "Test module for log2 interfaces"
+	depends on m
+	help
+	  This builds the "test_log2" module that does basic testing of
+	  all <linux/log2.h> interfaces.
+	  It has no dependencies and doesn't run or load unless
+	  explicitly requested by name.  For example: modprobe test_log2.
+
+	  If unsure, say N.
+
 config TEST_VMALLOC
 	tristate "Test module for stress/performance analysis of vmalloc allocator"
 	default n
lib/Makefile +1 -0
@@ -62,6 +62,7 @@ obj-$(CONFIG_TEST_BPF) += test_bpf.o
 obj-$(CONFIG_TEST_FIRMWARE) += test_firmware.o
 obj-$(CONFIG_TEST_BITOPS) += test_bitops.o
 CFLAGS_test_bitops.o += -Werror
+obj-$(CONFIG_TEST_LOG2) += test_log2.o
 obj-$(CONFIG_TEST_SYSCTL) += test_sysctl.o
 obj-$(CONFIG_TEST_HASH) += test_hash.o test_siphash.o
 obj-$(CONFIG_TEST_IDA) += test_ida.o
lib/test_log2.c new file +139 -0

139 lines not on this page · raw message

#6 lore raw
Quoted 3 lines · Randy Dunlap

I ran a (new) lib/test_log2.ko before and after Jakub's patch and got the same results, although I am not claiming that it has an exhaustive set of tests in it. [it is attached]

The test most likely tests the non-constant paths (which is something I haven't changed), unless all the loops would be completely unrolled, all the arrays determined to be never written to and turned into const and all the loads from those arrays optimized into constants.

If you want to test both the non-constant and constant paths, it would be good to make sure to guarantee that. For non-constant, e.g. use those arrays but add some optimization barrier so that the compiler can't optimize them into constants (e.g. make the arrays volatile, or add __asm__ ("" : : "g" (test_power_of_2) : "memory");
etc. barriers at the start of test function etc.
And for constants, perhaps put the test values into macros, through which to place it both
#define TEST_ILOG2_UINT \
TEST (1, 0) TEST (2, 1) TEST (4, 2) TEST (~0U, 31) \
TEST (0x80000000U, 31)
and once define TEST(x, y) as { x, y }, then #undef TEST
and in another place as if (ilog2(x) != y) ...

Also, you do want to test even behavior of ilog2 at 0, e.g. the powerpc fls/fls64 implementations are incorrect for that value as could be seen with -fsanitize=undefined.

And you do want to test also unsigned long long.

Jakub

merge bot Applied
#7 lore raw
In reply to Jakub Jelinek
The following commit has been merged into the core/core branch of tip:

The following commit has been merged into the core/core branch of tip:

Commit-ID:     ecbd43f6728a5cf79c8b50ed326658e9181531b1
Gitweb:        https://git.kernel.org/tip/ecbd43f6728a5cf79c8b50ed326658e9181531b1
Author:        Jakub Jelinek <[email protected]>
AuthorDate:    Wed, 21 Oct 2020 15:27:18 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 19 Nov 2020 11:26:18 +01:00

ilog2 vs. GCC inlining heuristics

Hi!

Based on the GCC PR97445 discussions, I'd like to propose following change, which should significantly decrease the amount of code in inline functions that use ilog2, but as I'm already two decades out of the Linux kernel development, I'd appreciate if some kernel developer could try that (all I have done is check that it gives the same results as before) and if it works submit it for inclusion into the kernel?

Thanks.

Improve ilog2 for constant arguments

As discussed in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97445
the const_ilog2 macro generates a lot of code which interferes badly with GCC inlining heuristics, until it can be proven that the ilog2 argument can or can't be simplified into a constant.

It can be expressed using __builtin_clzll builtin which is supported by GCC 3.4 and later and when used only in the __builtin_constant_p guarded code it ought to always fold back to a constant.
Other compilers support the same builtin for many years too.

Other option would be to change the const_ilog2 macro, though as the description says it is meant to be used also in C constant expressions, and while GCC will fold it to constant with constant argument even in those, perhaps it is better to avoid using extensions in that case.

---
 include/linux/log2.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/log2.h +2 -1
index c619ec6..4307d34 100644
@@ -156,7 +156,8 @@ unsigned long __rounddown_pow_of_two(unsigned long n)
 #define ilog2(n) \
 ( \
 	__builtin_constant_p(n) ?	\
-	const_ilog2(n) :		\
+	((n) < 2 ? 0 :			\
+	 63 - __builtin_clzll (n)) :	\
 	(sizeof(n) <= 4) ?		\
 	__ilog2_u32(n) :		\
 	__ilog2_u64(n)			\
#8 lore raw

Sorry, I typoed the branch name. I'll make this branch go away.

Anyway, Jacub, your patch seems to not upset the robots, so I'll go post it properly for you.

Quoted 60 lines · On Fri, Nov 20, 2020 at 12:33:58PM -0000, tip-bot2 for Jakub Jelinek wrote:

The following commit has been merged into the core/core branch of tip:

Commit-ID:     ecbd43f6728a5cf79c8b50ed326658e9181531b1
Gitweb:        https://git.kernel.org/tip/ecbd43f6728a5cf79c8b50ed326658e9181531b1
Author:        Jakub Jelinek <[email protected]>
AuthorDate:    Wed, 21 Oct 2020 15:27:18 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 19 Nov 2020 11:26:18 +01:00

ilog2 vs. GCC inlining heuristics

Hi!

Based on the GCC PR97445 discussions, I'd like to propose following change, which should significantly decrease the amount of code in inline functions that use ilog2, but as I'm already two decades out of the Linux kernel development, I'd appreciate if some kernel developer could try that (all I have done is check that it gives the same results as before) and if it works submit it for inclusion into the kernel?

Thanks.

Improve ilog2 for constant arguments

As discussed in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97445
the const_ilog2 macro generates a lot of code which interferes badly with GCC inlining heuristics, until it can be proven that the ilog2 argument can or can't be simplified into a constant.

It can be expressed using __builtin_clzll builtin which is supported by GCC 3.4 and later and when used only in the __builtin_constant_p guarded code it ought to always fold back to a constant.
Other compilers support the same builtin for many years too.

Other option would be to change the const_ilog2 macro, though as the description says it is meant to be used also in C constant expressions, and while GCC will fold it to constant with constant argument even in those, perhaps it is better to avoid using extensions in that case.

---
 include/linux/log2.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/log2.h +2 -1
index c619ec6..4307d34 100644
@@ -156,7 +156,8 @@ unsigned long __rounddown_pow_of_two(unsigned long n)
 #define ilog2(n) \
 ( \
 	__builtin_constant_p(n) ?	\
-	const_ilog2(n) :		\
+	((n) < 2 ? 0 :			\
+	 63 - __builtin_clzll (n)) :	\
 	(sizeof(n) <= 4) ?		\
 	__ilog2_u32(n) :		\
 	__ilog2_u64(n)			\