Skip to contentSkip to conversation

Add memcpy_cachebypass, a copy routine that tries to keep cache pressure down

PATCH In review Bryan O'Sullivan 6 participants 12 messages Jul 11, 2006 – Jul 12, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Add memcpy_cachebypass, a copy routine that tries to keep cache pressure down

This copy routine is memcpy-compatible, but on some architectures will use cache-bypassing loads to avoid bringing the source data into the cache.

One case where this is useful is when a device issues a DMA to a memory region, and the CPU must copy the DMAed data elsewhere before doing any work with it. Since the source data is read-once, write-never from the CPU's perspective, caching those addresses can only evict potentially useful data.

We provide an x86_64 implementation that uses SSE non-temporal loads, and a generic version that falls back to plain memcpy.

Implementors for other arches should not use cache-bypassing stores to the destination, as in most cases, the destination is accessed almost immediately after a copy finishes.

include/linux/string.h +1 -0
diff -r c5610179c494 -r da0cd816c4cb include/linux/string.h
@@ -85,6 +85,7 @@ extern void * memset(void *,int,__kernel
 #ifndef __HAVE_ARCH_MEMCPY
 extern void * memcpy(void *,const void *,__kernel_size_t);
 #endif
+extern void * memcpy_cachebypass(void *,const void *,__kernel_size_t);
 #ifndef __HAVE_ARCH_MEMMOVE
 extern void * memmove(void *,const void *,__kernel_size_t);
 #endif
lib/string.c +32 -0
diff -r c5610179c494 -r da0cd816c4cb lib/string.c
@@ -509,6 +509,38 @@ EXPORT_SYMBOL(memcpy);
 EXPORT_SYMBOL(memcpy);
 #endif
 
+void *memcpy_cachebypass(void *dest, const void *src, size_t count)
+	__attribute__((weak));
+
+/**
+ * memcpy_cachebypass - Copy one area of memory to another, if possible
+ * bypassing the CPU's cache when loading the copied-from data
+ * @dest: Where to copy to
+ * @src: Where to copy from (bypassing the CPU's cache, if possible)
+ * @count: The size of the area.
+ *
+ * This memcpy-compatible routine is intended for use when the CPU
+ * only reads the source data once.  It is useful when, for example, a
+ * hardware device writes to a memory region, and the CPU needs to
+ * copy this data somewhere else before working on it.  In such a
+ * case, caching the source addresses only serves to evict possibly
+ * useful data that will probably have to be reloaded.
+ *
+ * An arch-specific implementation should not attempt to bypass the
+ * cache when storing to the destination, as copied data is usually
+ * accessed almost immediately after a copy finishes.
+ *
+ * This routine does not *guarantee* that the source addresses won't
+ * be cached; a user of this code must not rely on this behaviour for
+ * correctness.  It should only be used in cases where it provides a
+ * measurable performance improvement.
+ */
+void *memcpy_cachebypass(void *dest, const void *src, size_t count)
+{
+	return memcpy(dest, src, count);
+}
+EXPORT_SYMBOL_GPL(memcpy_cachebypass);
+
 #ifndef __HAVE_ARCH_MEMMOVE
 /**
  * memmove - Copy one area of memory to another
arch/x86_64/lib/memcpy_cachebypass.S new file +142 -0

142 lines not on this page · raw message

Acked-by on v1 1/1
#1 lore raw
Quoted 12 lines · Bryan O'Sullivan

This copy routine is memcpy-compatible, but on some architectures will use cache-bypassing loads to avoid bringing the source data into the cache.

One case where this is useful is when a device issues a DMA to a memory region, and the CPU must copy the DMAed data elsewhere before doing any work with it. Since the source data is read-once, write-never from the CPU's perspective, caching those addresses can only evict potentially useful data.

We provide an x86_64 implementation that uses SSE non-temporal loads, and a generic version that falls back to plain memcpy.

Quoted 6 lines

Implementors for other arches should not use cache-bypassing stores to the destination, as in most cases, the destination is accessed almost immediately after a copy finishes.

#2 lore raw
In reply to Bryan O'Sullivan

From: Bryan O'Sullivan <[email protected]>
Date: Tue, 11 Jul 2006 13:50:55 -0700

Quoted 12 lines

This copy routine is memcpy-compatible, but on some architectures will use cache-bypassing loads to avoid bringing the source data into the cache.

One case where this is useful is when a device issues a DMA to a memory region, and the CPU must copy the DMAed data elsewhere before doing any work with it. Since the source data is read-once, write-never from the CPU's perspective, caching those addresses can only evict potentially useful data.

We provide an x86_64 implementation that uses SSE non-temporal loads, and a generic version that falls back to plain memcpy.

Quoted 6 lines

Implementors for other arches should not use cache-bypassing stores to the destination, as in most cases, the destination is accessed almost immediately after a copy finishes.

Please don't use a weak attribute, and instead use the same "__HAVE_ARCH_FOO" cpp test scheme used for the other string operations to allow a platform to override the default implementation in lib/string.x

#3 lore raw
Quoted 4 lines · David Miller

Please don't use a weak attribute, and instead use the same "__HAVE_ARCH_FOO" cpp test scheme used for the other string operations to allow a platform to override the default implementation in lib/string.x

I'm a bit confused.

The last time I tried submitting a patch that followed that style (for __iowrite_copy*), it got NAKed for propagating preprocessor abuse (Linus roundly flamed someone for a similar patch a few weeks before I submitted mine), and Andrew suggested that I use the same scheme that this patch uses.

So whose instructions do I follow? Yours of today, or Andrew's and Linus's of a few months ago?

	<b
#4 lore raw

From: Bryan O'Sullivan <[email protected]>
Date: Tue, 11 Jul 2006 14:30:01 -0700

Quoted 8 lines

The last time I tried submitting a patch that followed that style (for __iowrite_copy*), it got NAKed for propagating preprocessor abuse (Linus roundly flamed someone for a similar patch a few weeks before I submitted mine), and Andrew suggested that I use the same scheme that this patch uses.

So whose instructions do I follow? Yours of today, or Andrew's and Linus's of a few months ago?

I didn't realize there was change afoot in this area, sorry. I was just striving for consistency with current practice.

If Andrew suggested to use weak, that's fine, but it's kind of erroneous for something like lib/string.c because that gets built into a library lib.a file, which resolves any unresolved references.

When the kernel is linked, lib.a implementations only get brought in if they are not already resolved by definitions present in the other objects of the kernel image.

Weak makes more sense when dealing with object files, not archives.

#5 lore raw
Quoted 2 lines · David Miller

I didn't realize there was change afoot in this area, sorry. I was just striving for consistency with current practice.

Sure.

Quoted 3 lines

When the kernel is linked, lib.a implementations only get brought in if they are not already resolved by definitions present in the other objects of the kernel image.

Well, exactly this scheme seems to work for __iowrite_copy*. There's a weak generic version and a strong version in arch/x86_64/lib that overrides it, and it gets picked up at kernel link time.

It could be working by accident, I suppose, but it's at least consistent behaviour with what I'm used to from weak symbols.

	<b
#6 lore raw

From: Bryan O'Sullivan <[email protected]>
Date: Tue, 11 Jul 2006 15:05:08 -0700

Quoted 3 lines

Well, exactly this scheme seems to work for __iowrite_copy*. There's a weak generic version and a strong version in arch/x86_64/lib that overrides it, and it gets picked up at kernel link time.

It is linked in as an object, not into the library archive, that's why that one works like that.

That is why io.o is added to the "obj-y" variable instead of the "lib-y" variable. It is also necessary to link these things in as objects when module exports are present, because if there is no in-kernel reference to the function, you won't get the function nor it's module export :)

#7 lore raw
In reply to David Miller
Quoted 20 lines · David Miller

From: Bryan O'Sullivan <[email protected]>
Date: Tue, 11 Jul 2006 14:30:01 -0700

Quoted 8 lines

The last time I tried submitting a patch that followed that style (for __iowrite_copy*), it got NAKed for propagating preprocessor abuse (Linus roundly flamed someone for a similar patch a few weeks before I submitted mine), and Andrew suggested that I use the same scheme that this patch uses.

So whose instructions do I follow? Yours of today, or Andrew's and Linus's of a few months ago?

I didn't realize there was change afoot in this area, sorry. I was just striving for consistency with current practice.

If Andrew suggested to use weak, that's fine, but it's kind of erroneous for something like lib/string.c because that gets built into a library lib.a file, which resolves any unresolved references.

Quoted 5 lines

When the kernel is linked, lib.a implementations only get brought in if they are not already resolved by definitions present in the other objects of the kernel image.

Weak makes more sense when dealing with object files, not archives.

Weak is generally the wrong thing for what we do in kernel land. We don't even want to build the generic version if we have a better one. It also makes it really hard to find out what exactly in use. Linus only argued against __ARCH_HAVE_FOO, but his suggested replacemenet is:

#ifndef foo
#define foo generic_version
#endif

#8 lore raw
In reply to Bryan O'Sullivan

include/linux/string.h extern void * memset(void *,int,__kernel · line 88

Quoted 8 lines · On Tue, 11 Jul 2006 13:50:55 -0700 Bryan O'Sullivan wrote:

diff -r c5610179c494 -r da0cd816c4cb include/linux/string.h

include/linux/string.h +1 -0
@@ -85,6 +85,7 @@ extern void * memset(void *,int,__kernel
 #ifndef __HAVE_ARCH_MEMCPY
 extern void * memcpy(void *,const void *,__kernel_size_t);
 #endif
+extern void * memcpy_cachebypass(void *,const void *,__kernel_size_t);

space after commas, please.

Quoted 7 lines
include/linux/string.h +0 -0
 #ifndef __HAVE_ARCH_MEMMOVE
 extern void * memmove(void *,const void *,__kernel_size_t);
 #endif

diff -r c5610179c494 -r da0cd816c4cb lib/string.c

lib/string.c EXPORT_SYMBOL(memcpy); · line 517

Quoted 9 lines
lib/string.c +6 -0
@@ -509,6 +509,38 @@ EXPORT_SYMBOL(memcpy);
 EXPORT_SYMBOL(memcpy);
 #endif
 
+void *memcpy_cachebypass(void *dest, const void *src, size_t count)
+	__attribute__((weak));
+
+/**
+ * memcpy_cachebypass - Copy one area of memory to another, if possible
+ * bypassing the CPU's cache when loading the copied-from data

Currently kernel-doc function description is limited to one line. If you can't shorten it, just omit it completely and make it the first paragraph after the parameters.

Quoted 19 lines
lib/string.c +19 -0
+ * @dest: Where to copy to
+ * @src: Where to copy from (bypassing the CPU's cache, if possible)
+ * @count: The size of the area.
+ *
+ * This memcpy-compatible routine is intended for use when the CPU
+ * only reads the source data once.  It is useful when, for example, a
+ * hardware device writes to a memory region, and the CPU needs to
+ * copy this data somewhere else before working on it.  In such a
+ * case, caching the source addresses only serves to evict possibly
+ * useful data that will probably have to be reloaded.
+ *
+ * An arch-specific implementation should not attempt to bypass the
+ * cache when storing to the destination, as copied data is usually
+ * accessed almost immediately after a copy finishes.
+ *
+ * This routine does not *guarantee* that the source addresses won't
+ * be cached; a user of this code must not rely on this behaviour for
+ * correctness.  It should only be used in cases where it provides a
+ * measurable performance improvement.

lib/string.c EXPORT_SYMBOL(memcpy); · line 546

Quoted 10 lines
lib/string.c +7 -0
+ */
+void *memcpy_cachebypass(void *dest, const void *src, size_t count)
+{
+	return memcpy(dest, src, count);
+}
+EXPORT_SYMBOL_GPL(memcpy_cachebypass);
+
 #ifndef __HAVE_ARCH_MEMMOVE
 /**
  * memmove - Copy one area of memory to another

---
~Randy

#9 lore raw
Quoted 1 line · Randy.Dunlap

space after commas, please.

Yep.

Quoted 1 line

Currently kernel-doc function description is limited to one line.

Ugh, OK. What about "Memory copy, bypassing CPU cache for loads" for the one-liner? And a suitably modified first paragraph to make it clear that on some arches, it falls back to memcpy.

Thanks,

	<b
#10 lore raw
Quoted 11 lines · On Tue, 11 Jul 2006 14:35:19 -0700 Bryan O'Sullivan wrote:
Quoted 1 line · Randy.Dunlap

space after commas, please.

Yep.

Quoted 1 line

Currently kernel-doc function description is limited to one line.

Ugh, OK. What about "Memory copy, bypassing CPU cache for loads" for the one-liner? And a suitably modified first paragraph to make it clear that on some arches, it falls back to memcpy.

Yep, that sounds good.

Thanks.
---
~Randy

#11 lore raw
In reply to Bryan O'Sullivan
Quoted 9 lines · Bryan O'Sullivan

+ * memcpy_cachebypass - memcpy-compatible copy routine, using streaming loads
+ * @dest: destination address
+ * @src: source address (will not be cached)
+ * @count: number of bytes to copy
+ *
+ * Use streaming loads and normal stores for a special-case copy where
+ * we know we won't be reading the source again, but will be reading the
+ * destination again soon.
+ */

For what CPU did you optimize that function? Comment missing for that.
Also the comment should state that you're caching the target. Also I trust you ran it through a comprehensive memcpy-all-cases tester?

-Andi