Add memcpy_cachebypass, a copy routine that tries to keep cache pressure down
Subsystems (2)
PATCH 1/1 Add memcpy_cachebypass, a copy routine that tries to keep cache pressure down
This copy routine is memcpy-compatible, but on some architectures will use cache-bypassing loads to avoid bringing the source data into the cache.
One case where this is useful is when a device issues a DMA to a memory region, and the CPU must copy the DMAed data elsewhere before doing any work with it. Since the source data is read-once, write-never from the CPU's perspective, caching those addresses can only evict potentially useful data.
We provide an x86_64 implementation that uses SSE non-temporal loads, and a generic version that falls back to plain memcpy.
Implementors for other arches should not use cache-bypassing stores to the destination, as in most cases, the destination is accessed almost immediately after a copy finishes.
include/linux/string.h +1 -0
@@ -85,6 +85,7 @@ extern void * memset(void *,int,__kernel#ifndef __HAVE_ARCH_MEMCPYextern void *memcpy (void *,const void *,__kernel_size_t );#endif +extern void *memcpy_cachebypass (void *,const void *,__kernel_size_t );#ifndef __HAVE_ARCH_MEMMOVEextern void *memmove (void *,const void *,__kernel_size_t );#endif
lib/string.c +32 -0
@@ -509,6 +509,38 @@ EXPORT_SYMBOL(memcpy);EXPORT_SYMBOL (memcpy);#endif +void *memcpy_cachebypass (void *dest,const void *src,size_t count) + __attribute__((weak)); + +/** +* memcpy_cachebypass - Copy one area of memory to another, if possible +* bypassing the CPU's cache when loading the copied-from data +* @dest: Where to copy to +* @src: Where to copy from (bypassing the CPU's cache, if possible) +* @count: The size of the area. +* +* This memcpy-compatible routine is intended for use when the CPU +* only reads the source data once. It is useful when, for example, a +* hardware device writes to a memory region, and the CPU needs to +* copy this data somewhere else before working on it. In such a +* case, caching the source addresses only serves to evict possibly +* useful data that will probably have to be reloaded. +* +* An arch-specific implementation should not attempt to bypass the +* cache when storing to the destination, as copied data is usually +* accessed almost immediately after a copy finishes. +* +* This routine does not *guarantee* that the source addresses won't +* be cached; a user of this code must not rely on this behaviour for +* correctness. It should only be used in cases where it provides a +* measurable performance improvement. +*/ +void *memcpy_cachebypass (void *dest,const void *src,size_t count) +{ +return memcpy (dest, src, count); +} +EXPORT_SYMBOL_GPL (memcpy_cachebypass); +#ifndef __HAVE_ARCH_MEMMOVE/** * memmove - Copy one area of memory to another
arch/x86_64/lib/memcpy_cachebypass.S new file +142 -0
142 lines not on this page · raw message
Quoted 12 lines · Bryan O'Sullivan
This copy routine is memcpy-compatible, but on some architectures will use cache-bypassing loads to avoid bringing the source data into the cache.
One case where this is useful is when a device issues a DMA to a memory region, and the CPU must copy the DMAed data elsewhere before doing any work with it. Since the source data is read-once, write-never from the CPU's perspective, caching those addresses can only evict potentially useful data.
We provide an
x86_64implementation that uses SSE non-temporal loads, and a generic version that falls back to plain memcpy.
Quoted 6 lines
Implementors for other arches should not use cache-bypassing stores to the destination, as in most cases, the destination is accessed almost immediately after a copy finishes.
Signed-off-by Ralph Campbell <[email protected]>Signed-off-by Bryan O'Sullivan <[email protected]>
From: Bryan O'Sullivan <[email protected]>
Date: Tue, 11 Jul 2006 13:50:55 -0700
Quoted 12 lines
This copy routine is memcpy-compatible, but on some architectures will use cache-bypassing loads to avoid bringing the source data into the cache.
One case where this is useful is when a device issues a DMA to a memory region, and the CPU must copy the DMAed data elsewhere before doing any work with it. Since the source data is read-once, write-never from the CPU's perspective, caching those addresses can only evict potentially useful data.
We provide an
x86_64implementation that uses SSE non-temporal loads, and a generic version that falls back to plain memcpy.
Quoted 6 lines
Implementors for other arches should not use cache-bypassing stores to the destination, as in most cases, the destination is accessed almost immediately after a copy finishes.
Signed-off-by Ralph Campbell <[email protected]>Signed-off-by Bryan O'Sullivan <[email protected]>
Please don't use a weak attribute, and instead use the same "__HAVE_ARCH_FOO" cpp test scheme used for the other string operations to allow a platform to override the default implementation in lib/string.x
Quoted 4 lines · David Miller
Please don't use a weak attribute, and instead use the same "
__HAVE_ARCH_FOO" cpp test scheme used for the other string operations to allow a platform to override the default implementation inlib/string.x
I'm a bit confused.
The last time I tried submitting a patch that followed that style (for __iowrite_copy*), it got NAKed for propagating preprocessor abuse (Linus roundly flamed someone for a similar patch a few weeks before I submitted mine), and Andrew suggested that I use the same scheme that this patch uses.
So whose instructions do I follow? Yours of today, or Andrew's and Linus's of a few months ago?
<b
From: Bryan O'Sullivan <[email protected]>
Date: Tue, 11 Jul 2006 14:30:01 -0700
Quoted 8 lines
The last time I tried submitting a patch that followed that style (for __iowrite_copy*), it got NAKed for propagating preprocessor abuse (Linus roundly flamed someone for a similar patch a few weeks before I submitted mine), and Andrew suggested that I use the same scheme that this patch uses.
So whose instructions do I follow? Yours of today, or Andrew's and Linus's of a few months ago?
I didn't realize there was change afoot in this area, sorry. I was just striving for consistency with current practice.
If Andrew suggested to use weak, that's fine, but it's kind of erroneous for something like lib/string.c because that gets built into a library lib.a file, which resolves any unresolved references.
When the kernel is linked, lib.a implementations only get brought in if they are not already resolved by definitions present in the other objects of the kernel image.
Weak makes more sense when dealing with object files, not archives.
Quoted 2 lines · David Miller
I didn't realize there was change afoot in this area, sorry. I was just striving for consistency with current practice.
Sure.
Quoted 3 lines
When the kernel is linked, lib.a implementations only get brought in if they are not already resolved by definitions present in the other objects of the kernel image.
Well, exactly this scheme seems to work for __iowrite_copy*. There's a weak generic version and a strong version in arch/x86_64/lib that overrides it, and it gets picked up at kernel link time.
It could be working by accident, I suppose, but it's at least consistent behaviour with what I'm used to from weak symbols.
<b
From: Bryan O'Sullivan <[email protected]>
Date: Tue, 11 Jul 2006 15:05:08 -0700
Quoted 3 lines
Well, exactly this scheme seems to work for __iowrite_copy*. There's a weak generic version and a strong version in
arch/x86_64/libthat overrides it, and it gets picked up at kernel link time.
It is linked in as an object, not into the library archive, that's why that one works like that.
That is why io.o is added to the "obj-y" variable instead of the "lib-y" variable. It is also necessary to link these things in as objects when module exports are present, because if there is no in-kernel reference to the function, you won't get the function nor it's module export :)
Quoted 20 lines · David Miller
From: Bryan O'Sullivan <[email protected]>
Date: Tue, 11 Jul 2006 14:30:01 -0700Quoted 8 lines
The last time I tried submitting a patch that followed that style (for __iowrite_copy*), it got NAKed for propagating preprocessor abuse (Linus roundly flamed someone for a similar patch a few weeks before I submitted mine), and Andrew suggested that I use the same scheme that this patch uses.
So whose instructions do I follow? Yours of today, or Andrew's and Linus's of a few months ago?
I didn't realize there was change afoot in this area, sorry. I was just striving for consistency with current practice.
If Andrew suggested to use weak, that's fine, but it's kind of erroneous for something like
lib/string.cbecause that gets built into a library lib.a file, which resolves any unresolved references.
Quoted 5 lines
When the kernel is linked, lib.a implementations only get brought in if they are not already resolved by definitions present in the other objects of the kernel image.
Weak makes more sense when dealing with object files, not archives.
Weak is generally the wrong thing for what we do in kernel land. We don't even want to build the generic version if we have a better one. It also makes it really hard to find out what exactly in use. Linus only argued against __ARCH_HAVE_FOO, but his suggested replacemenet is:
#ifndef foo
#define foo generic_version
#endif
include/linux/string.h extern void * memset(void *,int,__kernel · line 88
Quoted 8 lines · On Tue, 11 Jul 2006 13:50:55 -0700 Bryan O'Sullivan wrote:
diff -r
c5610179c494-rda0cd816c4cbinclude/linux/string.hinclude/linux/string.h +1
-0@@ -85,6 +85,7 @@ extern void * memset(void *,int,__kernel#ifndef __HAVE_ARCH_MEMCPYextern void *memcpy (void *,const void *,__kernel_size_t );#endif +extern void *memcpy_cachebypass (void *,const void *,__kernel_size_t );
space after commas, please.
Quoted 7 lines
include/linux/string.h +0
-0#ifndef __HAVE_ARCH_MEMMOVEextern void *memmove (void *,const void *,__kernel_size_t );#endif diff -r
c5610179c494-rda0cd816c4cblib/string.c
lib/string.c EXPORT_SYMBOL(memcpy); · line 517
Quoted 9 lines
lib/string.c +6
-0@@ -509,6 +509,38 @@ EXPORT_SYMBOL(memcpy);EXPORT_SYMBOL (memcpy);#endif +void *memcpy_cachebypass (void *dest,const void *src,size_t count) + __attribute__((weak)); + +/** +* memcpy_cachebypass - Copy one area of memory to another, if possible +* bypassing the CPU's cache when loading the copied-from data
Currently kernel-doc function description is limited to one line. If you can't shorten it, just omit it completely and make it the first paragraph after the parameters.
Quoted 19 lines
lib/string.c +19
-0+* @dest: Where to copy to +* @src: Where to copy from (bypassing the CPU's cache, if possible) +* @count: The size of the area. +* +* This memcpy-compatible routine is intended for use when the CPU +* only reads the source data once. It is useful when, for example, a +* hardware device writes to a memory region, and the CPU needs to +* copy this data somewhere else before working on it. In such a +* case, caching the source addresses only serves to evict possibly +* useful data that will probably have to be reloaded. +* +* An arch-specific implementation should not attempt to bypass the +* cache when storing to the destination, as copied data is usually +* accessed almost immediately after a copy finishes. +* +* This routine does not *guarantee* that the source addresses won't +* be cached; a user of this code must not rely on this behaviour for +* correctness. It should only be used in cases where it provides a +* measurable performance improvement.
lib/string.c EXPORT_SYMBOL(memcpy); · line 546
Quoted 10 lines
lib/string.c +7
-0+*/ +void *memcpy_cachebypass (void *dest,const void *src,size_t count) +{ +return memcpy (dest, src, count); +} +EXPORT_SYMBOL_GPL (memcpy_cachebypass); +#ifndef __HAVE_ARCH_MEMMOVE/** * memmove - Copy one area of memory to another
---
~Randy
Quoted 1 line · Randy.Dunlap
space after commas, please.
Yep.
Quoted 1 line
Currently kernel-doc function description is limited to one line.
Ugh, OK. What about "Memory copy, bypassing CPU cache for loads" for the one-liner? And a suitably modified first paragraph to make it clear that on some arches, it falls back to memcpy.
Thanks,
<b
Quoted 11 lines · On Tue, 11 Jul 2006 14:35:19 -0700 Bryan O'Sullivan wrote:
Quoted 1 line · Randy.Dunlap
space after commas, please.
Yep.
Quoted 1 line
Currently kernel-doc function description is limited to one line.
Ugh, OK. What about "Memory copy, bypassing CPU cache for loads" for the one-liner? And a suitably modified first paragraph to make it clear that on some arches, it falls back to memcpy.
Yep, that sounds good.
Thanks.
---
~Randy
Quoted 9 lines · Bryan O'Sullivan
+ *
memcpy_cachebypass- memcpy-compatible copy routine, using streaming loads
+ * @dest: destination address
+ * @src: source address (will not be cached)
+ * @count: number of bytes to copy
+ *
+ * Use streaming loads and normal stores for a special-case copy where
+ * we know we won't be reading the source again, but will be reading the
+ * destination again soon.
+ */
For what CPU did you optimize that function? Comment missing for that.
Also the comment should state that you're caching the target. Also I trust you ran it through a comprehensive memcpy-all-cases tester?
-Andi