MIPS: Make cpu_has_mips64r6 constant on 64-bit r6 kernels
PATCH 1/1 MIPS: Make cpu_has_mips64r6 constant on 64-bit r6 kernels
On 64-bit r6 kernels, __fls() and __ffs() never use dclz: they always run the C binary search, 37 and 39 instructions with GCC instead of 4 and 6. __ffs() calls __fls(), which uses dclz only if cpu_has_mips64 is a compile-time constant:
if (BITS_PER_LONG == 64 && !__builtin_constant_p(word) &&
__builtin_constant_p(cpu_has_mips64) && cpu_has_mips64) {cpu_has_mips64 ORs cpu_has_mips64r1/r2/r5/r6. On r6 kernels the r1, r2 and r5 terms fold to 0, but the r6 term tests cpu_data[0].isa_level, so 64-bit r6 kernels have never used dclz here.
The r1, r2 and r6 macros were run-time tests until commit 1aeba347b3a9 ("MIPS: Hardcode cpu_has_mips* where target ISA allows") made r1 and r2 constant in 2018; it kept for r6 the flag test that tells MIPS64r6 from MIPS32r6 CPUs. Commit a2ac81c6ef40 ("mips: Fix cpu_has_mips64r1/2 activation for MIPS32 CPUs") later added cpu_has_64bits to r1 and r2, and left r6 alone. Every other cpu_has_mips32rN and cpu_has_mips64rN macro is a constant on the kernels built for its ISA (the 64-bit ones for cpu_has_mips64rN); only cpu_has_mips64r6 still tests the flag at run time on 64-bit r6 kernels.
A 64-bit r6 kernel only runs on a MIPS64r6 CPU, so define cpu_has_mips64r6 as cpu_has_64bits && MIPS_ISA_REV >= 6, like r1/r2/r5 without the flag test. It folds to 1 on 64-bit r6 kernels, and pre-r6 kernels build to identical code. On 32-bit r6 kernels, cpu_has_64bits tests isa_level & MIPS_CPU_ISA_64BIT, and set_isa() gives r6 CPUs no such flag except MIPS_CPU_ISA_M64R6, so the run-time result is unchanged. No __isa_*_flag() helper fits: a pre-r6 kernel cannot run on an r6 CPU and vice versa, so the flag test adds nothing, yet a helper keeps it as a run-time load in r6 kernels (__isa_ge_and_flag(), the current form) or in pre-r6 ones (__isa_ge_or_flag()). The r1/r2/r5-like cpu_has_64bits && __isa_ge_or_flag(6, ...) grows 32r2el by 3352 bytes and 64r2el by 264 bytes with GCC 16.
Because __fls() and __ffs() are inlined into find_*_bit(), cpumask iteration and ilog2(), vmlinux of the 64-bit r6 defconfigs shrinks by 105-108 KB with GCC 16 and 124-147 KB with Clang 23 (scripts/bloat-o-meter -t). In the 64r6el and eyeq6 builds, the other users of cpu_has_mips64r6 change 3 functions, by 148-248 bytes in all; the rest comes through these two functions. __fls(0) now returns ~0UL instead of 0 on 64-bit r6 kernels, as its comment says and as on 64-bit r2 kernels.
---
No 64-bit r6 hardware was available: every run-time test below ran on QEMU. A run of the ffs and bitops KUnit tests and find_bit_benchmark on an EyeQ or Boston board, before and after, would be welcome.
Instructions of a function that only returns __fls(x) or __ffs(x) (return included), compiled alone with the lib/find_bit.o command of 64r6el, before -> after:
__fls __ffs GCC 8-16 37 -> 4 39 -> 6 Clang 17-19 62 -> 4 64 -> 6 Clang 20 61 -> 4 63 -> 6 Clang 21-23 55 -> 4 57 -> 6
The new __fls() with GCC 16:
li v0,63 dclz a0,a0 jr ra subu v0,v0,a0
find_bit_benchmark built into 64r6el on QEMU boston (I6400 model), random-filled bitmap, executed instructions per iteration counted by a TCG plugin, before -> after:
GCC 16 Clang 23 find_next_bit 53.1 -> 28.4 86.0 -> 34.5 find_next_zero_bit 55.1 -> 29.4 87.1 -> 35.5 find_last_bit 54.1 -> 29.4 76.9 -> 27.4 find_next_and_bit 60.5 -> 34.7 90.4 -> 38.8
bitmap_find_next_zero_area_off 3839.7 -> 2176.4 6045.6 -> 2625.9
The other three tests (find_first_bit, find_first_and_bit and find_nth_bit) spend their time scanning words or counting bits rather than in __fls(), and change by 4% or less. The helpers themselves shrink
alike: _find_next_bit() goes from 61 to 31 instructions with GCC 16 and from 85 to 34 with Clang 23.
bloat-o-meter -t on vmlinux of the 64r6, 64r6el, eyeq5, eyeq6 and eyeq6lplus defconfigs, bfe97f12a0a6 before and after with the same .config, kernel.org GCC 16.2.0 and Clang 23.1.3: -105340 to -108352 bytes with GCC 16, -124340 to -146692 bytes with Clang 23. Of the 105 KB of 64r6el with GCC 16, inlined bit search (find_*_bit(), mostly cpumask iteration) accounts for 74.4 KB, log2 math (ilog2(), get_order(), roundup_pow_of_two()) for 21.6 KB and direct __ffs()/__fls() calls (lib/find_bit.c among them) for 7.0 KB, and secondary inlining for most of the rest. With GCC 16, 13 symbols grow and one function is added (1712 bytes in all): 608 bytes are the alignment gap that bloat-o-meter assigns to the __unlikely_text_end marker, 64 bytes are two loops that GCC lays out differently, and the rest are callers into which a now smaller callee with a bit search is inlined, or changes from GCC's inline growth limit and IPA-RA; the ten largest growers with Clang 23 are callers into which a callee with a cpumask loop or a bit search is inlined.
__isa_ge_and_flag() has no user left after this; I left it next to the other __isa_*() helpers, but can drop it if you prefer.
Switching MIPS to the generic builtin helpers would fix this too, but was dropped in 2019 because the builtins become libgcc calls on pre-r1 ISAs:
https://lore.kernel.org/all/[email protected]/
Testing: built before and after (W=1, no new warnings) with GCC 8.1 to 16.2 and Clang 17 to 23 for 64r6el, eyeq5, eyeq6, 32r6el and 64r2el, and with GCC 16 and Clang 23 for 64r6, eyeq6lplus, malta_qemu_32r6 and a 64-bit r6 allmodconfig; Clang 17 to 20 cannot assemble genex.S before or after, so they were compared on objects. Pre-r6 kernels build to identical code (64r2el and 32r2el with every compiler; malta, ip27, loongson3 and cavium_octeon with GCC 16). 64r6el on QEMU boston (I6400) and malta_qemu_32r6 on QEMU malta (mips32r6-generic and I6400), built with GCC 16 and Clang 23, pass the ffs and bitops KUnit suites and test_bitmap and print the same "isa" line in /proc/cpuinfo before and after.
Against the mips-next branch of mips/linux.git (bfe97f12a0a6); also applies to next-20261029.
arch/mips/include/asm/cpu-features.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/mips/include/asm/cpu-features.h +1 -1
@@ -317,7 +317,7 @@ __isa_range_or_flag(5, 6, MIPS_CPU_ISA_M64R5))#endif #ifndef cpu_has_mips64r6 -# define cpu_has_mips64r6 __isa_ge_and_flag(6, MIPS_CPU_ISA_M64R6) +# define cpu_has_mips64r6 (cpu_has_64bits && MIPS_ISA_REV >= 6)#endif /*
base-commit: bfe97f12a0a64c3682be21a02404238e3b91e945
Signature · 1 line
2.34.1
arch/mips/include/asm/cpu-features.h line 320
Quoted 10 lines · Tzu-Ching Yang
arch/mips/include/asm/cpu-features.h +1
-1@@ -317,7 +317,7 @@ __isa_range_or_flag(5, 6, MIPS_CPU_ISA_M64R5))#endif #ifndef cpu_has_mips64r6 -# define cpu_has_mips64r6 __isa_ge_and_flag(6, MIPS_CPU_ISA_M64R6) +# define cpu_has_mips64r6 (cpu_has_64bits && MIPS_ISA_REV >= 6)
Umm, the wrong fix. You want __isa_ge_or_flag() instead, plus obviouslycpu_has_64bits' is also needed (and __isa_ge_and_flag() can be discarded; I guess it was meant to take care of what cpu_has_64bits' does and broke the whole idea by mistake). Your proposal precludes the use of R6 stuff with pre-R6 compilations -- not sure if that makes sense, but let's stay consistent with these macro definitions.
Maciej