Add memcpy_uncached_read, a memcpy that doesn't cache reads
Subsystems (2)
PATCH 1/1 IB/ipath - use memcpy_uncached_read in RDMA interrupt handler to reduce packet loss
In reply to Bryan O'SullivanIn cases where a large incoming RDMA is being received, we have to copy data inside the interrupt handler before we can ACK each packet. The source is DMAed to by the hardware, which means that the CPU won't have it cached. We only read the source this one time; using normal load instructions pollutes the dcache with useless data, reducing performance to the point where we can lose a significant number of packets.
We use memcpy_uncached_read to try to not fill the dcache with useless data. Avoiding the cache refill penalty lets us keep up better with the sender, resulting in many fewer dropped packets.
drivers/infiniband/hw/ipath/ipath_verbs.c +1 -1
@@ -167,7 +167,7 @@ void ipath_copy_sge(struct ipath_sge_staBUG_ON (len ==0 );if (len > length) len = length; -memcpy (sge->vaddr, data, len); +memcpy_uncached_read (sge->vaddr, data, len); sge->vaddr += len; sge->length -= len; sge->sge_length -= len;
PATCH 1/1 Add memcpy_uncached_read, a memcpy that doesn't cache reads
Hi, Andrew -
Here's a suitably renamed uncached-read memcpy. I hope the name is now self-explanatory.
<b
PATCH 1/1 Add memcpy_uncached_read, a memcpy that tries to reduce cache pressure
This copy routine is memcpy-compatible, but on some architectures will use cache-bypassing loads to avoid bringing the source data into the cache.
One case where this is useful is when a device issues a DMA to a memory region, and the CPU must copy the DMAed data elsewhere before doing any work with it. Since the source data is read-once, write-never from the CPU's perspective, caching the data at those addresses can only evict potentially useful data.
We provide an x86_64 implementation that uses SSE non-temporal loads, and a generic version that falls back to plain memcpy.
Implementors for other arches should not use cache-bypassing stores to the destination, as in most cases, the destination is accessed almost immediately after a copy finishes.
arch/x86_64/lib/Makefile +2 -1
@@ -9,4 +9,5 @@ lib-y := csum-partial.o csum-copy.o csum lib-y := csum-partial.o csum-copy.o csum-wrappers.o delay.o \ usercopy.o getuser.o putuser.o \ thunk.o clear_page.o copy_page.o bitstr.o bitops.o -lib-y += memcpy.o memmove.o memset.o copy_user.o rwlock.o +lib-y += memcpy.o memmove.o memset.o copy_user.o rwlock.o \ + memcpy_uncached_read.o
arch/x86_64/lib/memcpy_uncached_read.S new file +142 -0
142 lines not on this page · raw message
include/asm-x86_64/string.h +2 -0
@@ -39,6 +39,8 @@ extern void *__memcpy(void *to, const vo __ret =__builtin_memcpy ((dst),(src),__len); \ __ret; }) +#define __HAVE_ARCH_MEMCPY_UNCACHED_READ +extern void *memcpy_uncached_read (void *to,const void *from,size_t len);#define __HAVE_ARCH_MEMSETvoid *memset (void *s,int c,size_t n);
include/linux/string.h +3 -0
@@ -85,6 +85,9 @@ extern void * memset(void *,int,__kernel#ifndef __HAVE_ARCH_MEMCPYextern void *memcpy (void *,const void *,__kernel_size_t );#endif +#ifndef __HAVE_ARCH_MEMCPY_UNCACHED_READ +#define memcpy_uncached_read (dest, src, count) memcpy((dest), (src), (count)) +#endif #ifndef __HAVE_ARCH_MEMMOVEextern void *memmove (void *,const void *,__kernel_size_t );#endif