Skip to contentSkip to conversation

Add memcpy_cachebypass, a memcpy that doesn't cache reads

PATCH In review Bryan O'Sullivan 2 participants 5 messages Nov 29, 2006 – Dec 1, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Add memcpy_cachebypass, a memcpy that doesn't cache reads

These patches add a memcpy that doesn't cache reads, and use it in the ipath driver's OpenIB receive path.

This version incorporates a few changes asked for last time around by DaveM.

	<b
#1 lore raw

PATCH 1/1 Add memcpy_cachebypass, a memcpy that tries to reduce cache pressure

This copy routine is memcpy-compatible, but on some architectures will use cache-bypassing loads to avoid bringing the source data into the cache.

One case where this is useful is when a device issues a DMA to a memory region, and the CPU must copy the DMAed data elsewhere before doing any work with it. Since the source data is read-once, write-never from the CPU's perspective, caching the data at those addresses can only evict potentially useful data.

We provide an x86_64 implementation that uses SSE non-temporal loads, and a generic version that falls back to plain memcpy.

Implementors for other arches should not use cache-bypassing stores to the destination, as in most cases, the destination is accessed almost immediately after a copy finishes.

arch/x86_64/lib/Makefile +2 -1
diff -r c76ed2f1387b -r 3300b7b66f46 arch/x86_64/lib/Makefile
@@ -9,4 +9,5 @@ lib-y := csum-partial.o csum-copy.o csum
 lib-y := csum-partial.o csum-copy.o csum-wrappers.o delay.o \
 	usercopy.o getuser.o putuser.o  \
 	thunk.o clear_page.o copy_page.o bitstr.o bitops.o
-lib-y += memcpy.o memmove.o memset.o copy_user.o rwlock.o
+lib-y += memcpy.o memmove.o memset.o copy_user.o rwlock.o \
+	memcpy_cachebypass.o
arch/x86_64/lib/memcpy_cachebypass.S new file +142 -0

142 lines not on this page · raw message

include/asm-x86_64/string.h +2 -0
diff -r c76ed2f1387b -r 3300b7b66f46 include/asm-x86_64/string.h
@@ -39,6 +39,8 @@ extern void *__memcpy(void *to, const vo
 		 __ret = __builtin_memcpy((dst),(src),__len);	\
 	   __ret; }) 
 
+#define __HAVE_ARCH_MEMCPY_CACHEBYPASS
+extern void *memcpy_cachebypass(void *to, const void *from, size_t len); 
 
 #define __HAVE_ARCH_MEMSET
 void *memset(void *s, int c, size_t n);
include/linux/string.h +3 -0
diff -r c76ed2f1387b -r 3300b7b66f46 include/linux/string.h
@@ -85,6 +85,9 @@ extern void * memset(void *,int,__kernel
 #ifndef __HAVE_ARCH_MEMCPY
 extern void * memcpy(void *,const void *,__kernel_size_t);
 #endif
+#ifndef __HAVE_ARCH_MEMCPY_CACHEBYPASS
+#define memcpy_cachebypass(dest, src, count) memcpy((dest), (src), (count))
+#endif
 #ifndef __HAVE_ARCH_MEMMOVE
 extern void * memmove(void *,const void *,__kernel_size_t);
 #endif
#2 lore raw

PATCH 1/1 IB/ipath - use memcpy_cachebypass in RDMA interrupt handler to reduce packet loss

In reply to Bryan O'Sullivan

In cases where a large incoming RDMA is being received, we have to copy data inside the interrupt handler before we can ACK each packet. The source is DMAed to by the hardware, which means that the CPU won't have it cached. We only read the source this one time; using normal load instructions pollutes the dcache with useless data, reducing performance to the point where we can lose a significant number of packets.

We use memcpy_cachebypass to try to not fill the dcache with useless data. Avoiding the cache refill penalty lets us keep up better with the sender, resulting in many fewer dropped packets.

drivers/infiniband/hw/ipath/ipath_verbs.c +1 -1
diff -r 3300b7b66f46 -r f9a929f40725 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -167,7 +167,7 @@ void ipath_copy_sge(struct ipath_sge_sta
 		BUG_ON(len == 0);
 		if (len > length)
 			len = length;
-		memcpy(sge->vaddr, data, len);
+		memcpy_cachebypass(sge->vaddr, data, len);
 		sge->vaddr += len;
 		sge->length -= len;
 		sge->sge_length -= len;
#3 lore raw
In reply to Bryan O'Sullivan
Quoted 4 lines · On Wed, 29 Nov 2006 15:35:07 -0700 "Bryan O'Sullivan" <[email protected]> wrote:

These patches add a memcpy that doesn't cache reads, and use it in the ipath driver's OpenIB receive path.

This version incorporates a few changes asked for last time around by DaveM.

Everybody seems to be hiding.

The name memcpy_cachebypass() doesn't tell us whether it bypasses caching on the source, the dest or both. It'd be nice if it did.

<tries to remember what copy_user_zeroing_intel_nocache() does>

#4 lore raw
Quoted 3 lines · Andrew Morton

The name memcpy_cachebypass() doesn't tell us whether it bypasses caching on the source, the dest or both. It'd be nice if it did.

Yep, I'll fix that and resubmit.

<b