Skip to contentSkip to conversation

x86: improve algorithm in clflush_cache_range

PATCH In review Ross Zwisler 2 participants 3 messages Mar 11, 2015 – Mar 13, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86: improve algorithm in clflush_cache_range

The current algorithm used in clflush_cache_range() can cause the last cache line of the buffer to be flushed twice.

Fix that algorithm so that each cache line will only be flushed once, and remove arithmetic on void pointers. Void pointer arithmetic is allowed by GCC extensions, but isn't part of the base C standards.

Cc H. Peter Anvin <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Dan Williams <[email protected]>
Cc Borislav Petkov <[email protected]>
---
 arch/x86/mm/pageattr.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
arch/x86/mm/pageattr.c +6 -7
index ae242a7c11c7..b75ecac859f2 100644
@@ -131,16 +131,15 @@ within(unsigned long addr, unsigned long start, unsigned long end)
  */
 void clflush_cache_range(void *vaddr, unsigned int size)
 {
-	void *vend = vaddr + size - 1;
+	unsigned long clflush_mask = boot_cpu_data.x86_clflush_size - 1;
+	char *vend = (char *)vaddr + size;
+	char *p;
 
 	mb();
 
-	for (; vaddr < vend; vaddr += boot_cpu_data.x86_clflush_size)
-		clflushopt(vaddr);
-	/*
-	 * Flush any possible final partial cacheline:
-	 */
-	clflushopt(vend);
+	for (p = (char *)((unsigned long)vaddr & ~clflush_mask);
+	     p < vend; p += boot_cpu_data.x86_clflush_size)
+		clflushopt(p);
 
 	mb();
 }
Signature · 1 line
1.9.3
#1 lore raw
Quoted 6 lines · Ross Zwisler

The current algorithm used in clflush_cache_range() can cause the last cache line of the buffer to be flushed twice.

Fix that algorithm so that each cache line will only be flushed once, and remove arithmetic on void pointers. Void pointer arithmetic is allowed by GCC extensions, but isn't part of the base C standards.

The optimization itself is fine, but that last argument is bogus: the Linux kernel very much relies on 'void *' arithmetics in a gazillion places.

Thanks,

Ingo

#2 lore raw
Quoted 5 lines · Ingo Molnar
Quoted 3 lines · Ross Zwisler

The current algorithm used in clflush_cache_range() can cause the last cache line of the buffer to be flushed twice.

Quoted 7 lines
Quoted 3 lines

Fix that algorithm so that each cache line will only be flushed once, and remove arithmetic on void pointers. Void pointer arithmetic is allowed by GCC extensions, but isn't part of the base C standards.

The optimization itself is fine, but that last argument is bogus: the Linux kernel very much relies on 'void *' arithmetics in a gazillion places.

Okay. Are you happy with the patch as is or would you like me to resubmit with that bit omitted from the change log?

- Ross