Skip to contentSkip to conversation

x86/cpu: Use prefetcht0 for prefetch() on 64-bit

PATCH In review Martin Leitner-Ankerl 3 participants 6 messages Nov 3 – Nov 4 x86

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/cpu: Use prefetcht0 for prefetch() on 64-bit

x86-64 does not define ARCH_HAS_PREFETCH, so prefetch() from <linux/prefetch.h> is just __builtin_prefetch(). The kernel builds with -mno-sse, and clang then drops __builtin_prefetch() without a warning, while gcc emits prefetcht0. An x86-64 defconfig vmlinux has 8 prefetcht0 with clang 22.1.8, all from inline assembly, and 62 with gcc 16.2.1. Also prefetch_range() is empty on x86-64 with any compiler, it needs ARCH_HAS_PREFETCH.

Give x86-64 its own prefetch(), a prefetcht0 in inline assembly, which every x86-64 CPU has, and define ARCH_HAS_PREFETCH for 64-bit too. Not the 32-bit alternative: it becomes prefetchnta on CPUs with SSE, a non-temporal prefetch, which would change what gcc builds emit today.

Afterwards clang's vmlinux has 57 prefetcht0, the new ones in 34 functions, e.g. __blk_mq_alloc_requests(), prepend_path(), __alloc_skb(), tcp_ack() and the receive paths of e1000, tg3, sky2, r8169 and forcedeth. gcc's stay at 62. With gcc 16.2.1, prefetch_range() adds 16 to hfi1's receive functions and 8 to scsi_debug, which a defconfig does not build. Youling Tang proposed to define ARCH_HAS_PREFETCH in <linux/prefetch.h> instead [1].

KVM guest with one CPU on a Ryzen 9 7950X at a fixed 4.5 GHz, clang 22.1.8, 6 boots per kernel in turns, ns per call, median (range):

                    before                 after
  io_uring IOPOLL   680.1 (679.6..684.6)   664.9 (663.1..668.4)   -2.3%
  getcwd()          912.9 (912.0..1019.4)  901.4 (900.1..907.3)   -1.3%

IOPOLL reads 4 KiB from null_blk, 32 per submit; getcwd() is 40 directories deep. Built with gcc, where both kernels have the same prefetches, the two differ by +0.4% and 0.0%. pread() with O_DIRECT, UDP and TCP over loopback, times() and clock_gettime() change no more than that. Unfortunately the NIC receive paths need real hardware, so they are not measured.

Fixes ab483570a13b ("x86 & generic: change to __builtin_prefetch()")
Cc Youling Tang <[email protected]>
Cc Nikolay Borisov <[email protected]>

---

Notes:
Code generation, testing, compiling, analysis, measurements, etc. were all driven by an AI agent (Claude Code). I have reviewed the results and supervised the process.

A few minutes ago this went to the x86 maintainers only, without the lists, by mistake. Sorry for the duplicate.

 arch/x86/include/asm/processor.h | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
arch/x86/include/asm/processor.h +12 -1
index 77488ff5eb8c..64748b8705e3 100644
@@ -624,15 +624,25 @@ extern int			bootloader_version;
 extern char			ignore_fpu_irq;
 
 #define HAVE_ARCH_PICK_MMAP_LAYOUT 1
+#define ARCH_HAS_PREFETCH
 #define ARCH_HAS_PREFETCHW
 
 #ifdef CONFIG_X86_32
 # define BASE_PREFETCH		""
-# define ARCH_HAS_PREFETCH
 #else
 # define BASE_PREFETCH		"prefetcht0 %1"
 #endif
 
+#ifdef CONFIG_X86_64
+/*
+ * Every x86-64 CPU has prefetcht0. Not __builtin_prefetch(): the kernel
+ * builds with -mno-sse, and clang then drops the builtin without a warning.
+ */
+static inline void prefetch(const void *x)
+{
+	asm("prefetcht0 %0" : : "m" (*(const char *)x));
+}
+#else
 /*
  * Prefetch instructions for Pentium III (+) and AMD Athlon (+)
  *
@@ -645,6 +655,7 @@ static inline void prefetch(const void *x)
 			  X86_FEATURE_XMM,
 			  "m" (*(const char *)x));
 }
+#endif /* CONFIG_X86_64 */
 
 /*
  * 3dnow prefetch to get an exclusive cache line.

base-commit: 2305edc99ef7bbc11a8dcebf78853bc1b6a761bc

Signature · 1 line
2.56.0
Acked-by on v1 1/1
#1 lore raw
Quoted 70 lines · Martin Leitner-Ankerl

x86-64 does not define ARCH_HAS_PREFETCH, so prefetch() from <linux/prefetch.h> is just __builtin_prefetch(). The kernel builds with -mno-sse, and clang then drops __builtin_prefetch() without a warning, while gcc emits prefetcht0. An x86-64 defconfig vmlinux has 8 prefetcht0 with clang 22.1.8, all from inline assembly, and 62 with gcc 16.2.1. Also prefetch_range() is empty on x86-64 with any compiler, it needs ARCH_HAS_PREFETCH.

Give x86-64 its own prefetch(), a prefetcht0 in inline assembly, which every x86-64 CPU has, and define ARCH_HAS_PREFETCH for 64-bit too. Not the 32-bit alternative: it becomes prefetchnta on CPUs with SSE, a non-temporal prefetch, which would change what gcc builds emit today.

Afterwards clang's vmlinux has 57 prefetcht0, the new ones in 34 functions, e.g. __blk_mq_alloc_requests(), prepend_path(), __alloc_skb(), tcp_ack() and the receive paths of e1000, tg3, sky2, r8169 and forcedeth. gcc's stay at 62. With gcc 16.2.1, prefetch_range() adds 16 to hfi1's receive functions and 8 to scsi_debug, which a defconfig does not build. Youling Tang proposed to define ARCH_HAS_PREFETCH in <linux/prefetch.h> instead [1].

KVM guest with one CPU on a Ryzen 9 7950X at a fixed 4.5 GHz, clang 22.1.8, 6 boots per kernel in turns, ns per call, median (range):

                    before                 after
  io_uring IOPOLL   680.1 (679.6..684.6)   664.9 (663.1..668.4)   -2.3%
  getcwd()          912.9 (912.0..1019.4)  901.4 (900.1..907.3)   -1.3%

IOPOLL reads 4 KiB from null_blk, 32 per submit; getcwd() is 40 directories deep. Built with gcc, where both kernels have the same prefetches, the two differ by +0.4% and 0.0%. pread() with O_DIRECT, UDP and TCP over loopback, times() and clock_gettime() change no more than that. Unfortunately the NIC receive paths need real hardware, so they are not measured.

Fixes ab483570a13b ("x86 & generic: change to __builtin_prefetch()")
Cc Youling Tang <[email protected]>
Cc Nikolay Borisov <[email protected]>

---

Notes:
Code generation, testing, compiling, analysis, measurements, etc. were all driven by an AI agent (Claude Code). I have reviewed the results and supervised the process.

A few minutes ago this went to the x86 maintainers only, without the lists, by mistake. Sorry for the duplicate.

 arch/x86/include/asm/processor.h | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
arch/x86/include/asm/processor.h +1 -1
index 77488ff5eb8c..64748b8705e3 100644
@@ -624,15 +624,25 @@ extern int                        bootloader_version;
 extern char                    ignore_fpu_irq;
 
 #define HAVE_ARCH_PICK_MMAP_LAYOUT 1
+#define ARCH_HAS_PREFETCH
 #define ARCH_HAS_PREFETCHW
 
 #ifdef CONFIG_X86_32
 # define BASE_PREFETCH         ""
-# define ARCH_HAS_PREFETCH
 #else
 # define BASE_PREFETCH         "prefetcht0 %1"

arch/x86/include/asm/processor.h extern int bootloader_version; · line 643

Quoted 10 lines
arch/x86/include/asm/processor.h +8 -0
 #endif
 
+#ifdef CONFIG_X86_64
+/*
+ * Every x86-64 CPU has prefetcht0. Not __builtin_prefetch(): the kernel
+ * builds with -mno-sse, and clang then drops the builtin without a warning.
+ */
+static inline void prefetch(const void *x)
+{
+       asm("prefetcht0 %0" : : "m" (*(const char *)x));

Thanks for the patch!
Is there a reason to have the cast vs simply use "r" constraint and deref in the inline asm?
asm("prefetcht0 (%0)" : : "r" (x));
https://godbolt.org/z/xezh15oMe

Either way:

Quoted 18 lines
arch/x86/include/asm/processor.h +3 -0
+}
+#else
 /*
  * Prefetch instructions for Pentium III (+) and AMD Athlon (+)
  *
@@ -645,6 +655,7 @@ static inline void prefetch(const void *x)
                          X86_FEATURE_XMM,
                          "m" (*(const char *)x));
 }
+#endif /* CONFIG_X86_64 */
 
 /*
  * 3dnow prefetch to get an exclusive cache line.

base-commit: 2305edc99ef7bbc11a8dcebf78853bc1b6a761bc --
2.56.0

Signature · 2 lines
Thanks,
~Nick Desaulniers
#2 lore raw
Quoted 5 lines

Thanks for the patch!
Is there a reason to have the cast vs simply use "r" constraint and deref in the inline asm?
asm("prefetcht0 (%0)" : : "r" (x));
https://godbolt.org/z/xezh15oMe

I mostly did it to be consistent with the 32-bit prefetch() and prefetchw() in the same file, but it also has a codegen benefit. With "m" the compiler can put an offset or index in the prefetcht0 itself, while with "r" it needs to compute the address into a register first:
https://godbolt.org/z/j311eYzsY

Thanks for the ack!
Martin

#3 lore raw
Quoted 7 lines · Martin Leitner-Ankerl
Quoted 6 lines

Thanks for the patch!
Is there a reason to have the cast vs simply use "r" constraint and deref in the inline asm?
asm("prefetcht0 (%0)" : : "r" (x));
https://godbolt.org/z/xezh15oMe

Quoted 5 lines

I mostly did it to be consistent with the 32-bit prefetch() and prefetchw() in the same file, but it also has a codegen benefit. With "m" the compiler can put an offset or index in the prefetcht0 itself, while with "r" it needs to compute the address into a register first:
https://godbolt.org/z/j311eYzsY

Interesting! I learn more about inline asm everyday...

Quoted 3 lines

Thanks for the ack!
Martin

Signature · 2 lines
Thanks,
~Nick Desaulniers
#4 lore raw
In reply to Martin Leitner-Ankerl

arch/x86/include/asm/processor.h extern int bootloader_version; · line 645

Quoted 10 lines
arch/x86/include/asm/processor.h +10 -0
+#ifdef CONFIG_X86_64
+/*
+ * Every x86-64 CPU has prefetcht0. Not __builtin_prefetch(): the kernel
+ * builds with -mno-sse, and clang then drops the builtin without a warning.
+ */
+static inline void prefetch(const void *x)
+{
+	asm("prefetcht0 %0" : : "m" (*(const char *)x));
+}
+#else

One thing you should check if the inline assembly pessimizes the code on gcc. Normally builtins can be better optimized. Or keep using the builtin on gcc to be safe.

-Andi

#5 lore raw
Quoted 3 lines

One thing you should check if the inline assembly pessimizes the code on gcc. Normally builtins can be better optimized. Or keep using the builtin on gcc to be safe.

I built an x86-64 defconfig vmlinux with and without the patch with gcc 16.2.1, and
compared them function by function. The .text size is exactly the same, both have exactly 62 prefetcht0 in the binary, and instruction count differs by just
one: 7365619 without the patch, 7365620 with it

23 functions of 130789 actually differ: 21 only in the order of instructions, where prefetcht0 comes mostly 1-4 instructions earlier, and tcp_ack() and e1000_clean_rx_irq_ps() differ in register allocation, the latter has an extra mov.

I timed system calls that reach 8 of the 23 functions in a KVM guest with one CPU, 10 boots per kernel. The differences are very small and within the noise, in
range from -0.7% to 0.0%. So the asm might be slightly faster, but it's too close to
say for sure.

So I'd keep one prefetch() for both compilers. The other 15 functions that differ
are mostly NIC receive paths, which would require real hardware to measure.

Martin