Skip to contentSkip to conversation

Add memcpy_uncached_read, a memcpy that doesn't cache reads

PATCH New Bryan O'Sullivan 1 participant 3 messages Dec 13, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Add memcpy_uncached_read, a memcpy that doesn't cache reads

Hi, Andrew -

Here's a suitably renamed uncached-read memcpy. I hope the name is now self-explanatory.

	<b
#1 lore raw

PATCH 1/1 IB/ipath - use memcpy_uncached_read in RDMA interrupt handler to reduce packet loss

In cases where a large incoming RDMA is being received, we have to copy data inside the interrupt handler before we can ACK each packet. The source is DMAed to by the hardware, which means that the CPU won't have it cached. We only read the source this one time; using normal load instructions pollutes the dcache with useless data, reducing performance to the point where we can lose a significant number of packets.

We use memcpy_uncached_read to try to not fill the dcache with useless data. Avoiding the cache refill penalty lets us keep up better with the sender, resulting in many fewer dropped packets.

drivers/infiniband/hw/ipath/ipath_verbs.c +1 -1
diff -r e7c3b265254b -r f25d77f76998 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -167,7 +167,7 @@ void ipath_copy_sge(struct ipath_sge_sta
 		BUG_ON(len == 0);
 		if (len > length)
 			len = length;
-		memcpy(sge->vaddr, data, len);
+		memcpy_uncached_read(sge->vaddr, data, len);
 		sge->vaddr += len;
 		sge->length -= len;
 		sge->sge_length -= len;
#2 lore raw

PATCH 1/1 Add memcpy_uncached_read, a memcpy that tries to reduce cache pressure

In reply to Bryan O'Sullivan

This copy routine is memcpy-compatible, but on some architectures will use cache-bypassing loads to avoid bringing the source data into the cache.

One case where this is useful is when a device issues a DMA to a memory region, and the CPU must copy the DMAed data elsewhere before doing any work with it. Since the source data is read-once, write-never from the CPU's perspective, caching the data at those addresses can only evict potentially useful data.

We provide an x86_64 implementation that uses SSE non-temporal loads, and a generic version that falls back to plain memcpy.

Implementors for other arches should not use cache-bypassing stores to the destination, as in most cases, the destination is accessed almost immediately after a copy finishes.

arch/x86_64/lib/Makefile +2 -1
diff -r 4a0c3ede5076 -r e7c3b265254b arch/x86_64/lib/Makefile
@@ -9,4 +9,5 @@ lib-y := csum-partial.o csum-copy.o csum
 lib-y := csum-partial.o csum-copy.o csum-wrappers.o delay.o \
 	usercopy.o getuser.o putuser.o  \
 	thunk.o clear_page.o copy_page.o bitstr.o bitops.o
-lib-y += memcpy.o memmove.o memset.o copy_user.o rwlock.o
+lib-y += memcpy.o memmove.o memset.o copy_user.o rwlock.o \
+	memcpy_uncached_read.o
arch/x86_64/lib/memcpy_uncached_read.S new file +142 -0

142 lines not on this page · raw message

include/asm-x86_64/string.h +2 -0
diff -r 4a0c3ede5076 -r e7c3b265254b include/asm-x86_64/string.h
@@ -39,6 +39,8 @@ extern void *__memcpy(void *to, const vo
 		 __ret = __builtin_memcpy((dst),(src),__len);	\
 	   __ret; }) 
 
+#define __HAVE_ARCH_MEMCPY_UNCACHED_READ
+extern void *memcpy_uncached_read(void *to, const void *from, size_t len); 
 
 #define __HAVE_ARCH_MEMSET
 void *memset(void *s, int c, size_t n);
include/linux/string.h +3 -0
diff -r 4a0c3ede5076 -r e7c3b265254b include/linux/string.h
@@ -85,6 +85,9 @@ extern void * memset(void *,int,__kernel
 #ifndef __HAVE_ARCH_MEMCPY
 extern void * memcpy(void *,const void *,__kernel_size_t);
 #endif
+#ifndef __HAVE_ARCH_MEMCPY_UNCACHED_READ
+#define memcpy_uncached_read(dest, src, count) memcpy((dest), (src), (count))
+#endif
 #ifndef __HAVE_ARCH_MEMMOVE
 extern void * memmove(void *,const void *,__kernel_size_t);
 #endif