Add memcpy_cachebypass, a memcpy that doesn't cache reads
Subsystems (2)
PATCH 1/1 Add memcpy_cachebypass, a memcpy that doesn't cache reads
These patches add a memcpy that doesn't cache reads, and use it in the ipath driver's OpenIB receive path.
This version incorporates a few changes asked for last time around by DaveM.
<b
PATCH 1/1 Add memcpy_cachebypass, a memcpy that tries to reduce cache pressure
This copy routine is memcpy-compatible, but on some architectures will use cache-bypassing loads to avoid bringing the source data into the cache.
One case where this is useful is when a device issues a DMA to a memory region, and the CPU must copy the DMAed data elsewhere before doing any work with it. Since the source data is read-once, write-never from the CPU's perspective, caching the data at those addresses can only evict potentially useful data.
We provide an x86_64 implementation that uses SSE non-temporal loads, and a generic version that falls back to plain memcpy.
Implementors for other arches should not use cache-bypassing stores to the destination, as in most cases, the destination is accessed almost immediately after a copy finishes.
arch/x86_64/lib/Makefile +2 -1
@@ -9,4 +9,5 @@ lib-y := csum-partial.o csum-copy.o csum lib-y := csum-partial.o csum-copy.o csum-wrappers.o delay.o \ usercopy.o getuser.o putuser.o \ thunk.o clear_page.o copy_page.o bitstr.o bitops.o -lib-y += memcpy.o memmove.o memset.o copy_user.o rwlock.o +lib-y += memcpy.o memmove.o memset.o copy_user.o rwlock.o \ + memcpy_cachebypass.o
arch/x86_64/lib/memcpy_cachebypass.S new file +142 -0
142 lines not on this page · raw message
include/asm-x86_64/string.h +2 -0
@@ -39,6 +39,8 @@ extern void *__memcpy(void *to, const vo __ret =__builtin_memcpy ((dst),(src),__len); \ __ret; }) +#define __HAVE_ARCH_MEMCPY_CACHEBYPASS +extern void *memcpy_cachebypass (void *to,const void *from,size_t len);#define __HAVE_ARCH_MEMSETvoid *memset (void *s,int c,size_t n);
include/linux/string.h +3 -0
@@ -85,6 +85,9 @@ extern void * memset(void *,int,__kernel#ifndef __HAVE_ARCH_MEMCPYextern void *memcpy (void *,const void *,__kernel_size_t );#endif +#ifndef __HAVE_ARCH_MEMCPY_CACHEBYPASS +#define memcpy_cachebypass (dest, src, count) memcpy((dest), (src), (count)) +#endif #ifndef __HAVE_ARCH_MEMMOVEextern void *memmove (void *,const void *,__kernel_size_t );#endif
PATCH 1/1 IB/ipath - use memcpy_cachebypass in RDMA interrupt handler to reduce packet loss
In reply to Bryan O'SullivanIn cases where a large incoming RDMA is being received, we have to copy data inside the interrupt handler before we can ACK each packet. The source is DMAed to by the hardware, which means that the CPU won't have it cached. We only read the source this one time; using normal load instructions pollutes the dcache with useless data, reducing performance to the point where we can lose a significant number of packets.
We use memcpy_cachebypass to try to not fill the dcache with useless data. Avoiding the cache refill penalty lets us keep up better with the sender, resulting in many fewer dropped packets.
drivers/infiniband/hw/ipath/ipath_verbs.c +1 -1
@@ -167,7 +167,7 @@ void ipath_copy_sge(struct ipath_sge_staBUG_ON (len ==0 );if (len > length) len = length; -memcpy (sge->vaddr, data, len); +memcpy_cachebypass (sge->vaddr, data, len); sge->vaddr += len; sge->length -= len; sge->sge_length -= len;
Quoted 4 lines · On Wed, 29 Nov 2006 15:35:07 -0700 "Bryan O'Sullivan" <[email protected]> wrote:
These patches add a memcpy that doesn't cache reads, and use it in the ipath driver's OpenIB receive path.
This version incorporates a few changes asked for last time around by DaveM.
Everybody seems to be hiding.
The name memcpy_cachebypass() doesn't tell us whether it bypasses caching on the source, the dest or both. It'd be nice if it did.
<tries to remember what copy_user_zeroing_intel_nocache() does>
Quoted 3 lines · Andrew Morton
The name
memcpy_cachebypass()doesn't tell us whether it bypasses caching on the source, the dest or both. It'd be nice if it did.
Yep, I'll fix that and resubmit.
<b