Skip to contentSkip to conversation

Prefetches in memcpy

DISCUSSION Carsten Langgaard 8 participants 12 messages Nov 5, 2002 – Dec 8, 2002 MIPS

lore
Share & tools
lore
#0 lore raw

I have reported this before and it also gave a lot of responses, but nothing has been done about, unfortunately :-(

The problem is the prefetches in the memcpy function in the kernel. There is spread a number of PREF instructions in the memcpy function, but there is no check if we are prefetching out-side the areas we are copying to/from. This is extremely dangerous because we might prefetch out-side the physical memory area, causing e.g. a bus error or something even more nasty.

I recently found something even nastier, it could also hit a DMA buffer region, and thereby break the PCI DMA flushing scheme.
For example if the kernel is doing a memcpy from an area that's next to a DMA buffer area, we could end up in a situation where, we are prefetching
data into the cache from a memory location that is used for DMA transfer and owned by the device, but the DMA transfer has not yet completed. We then end up in a situation, where the memory and cache is out of sync and the cache is containing some old data.

So we definitely need to do something about the prefetches in the memcpy function.
We can either get rid of all the prefetches or make sure we don't prefetch out side the "memcpy" area.

/Carsten

--

_    _ ____  ___   Carsten Langgaard   Mailto:[email protected]
|\  /|||___)(___   MIPS Denmark        Direct: +45 4486 5527
| \/ |||    ____)  Lautrupvang 4B      Switch: +45 4486 5555
  TECHNOLOGIES     2750 Ballerup       Fax...: +45 4486 5556
                   Denmark             http://www.mips.com
#1 lore raw
Quoted 17 lines · Carsten Langgaard

The problem is the prefetches in the memcpy function in the kernel. There is spread a number of PREF instructions in the memcpy function, but there is no check if we are prefetching out-side the areas we are copying to/from. This is extremely dangerous because we might prefetch out-side the physical memory area, causing e.g. a bus error or something even more nasty.

I recently found something even nastier, it could also hit a DMA buffer region, and thereby break the PCI DMA flushing scheme.
For example if the kernel is doing a memcpy from an area that's next to a DMA buffer area, we could end up in a situation where, we are prefetching
data into the cache from a memory location that is used for DMA transfer and owned by the device, but the DMA transfer has not yet completed. We then end up in a situation, where the memory and cache is out of sync and the cache is containing some old data.

Quoted 3 lines

So we definitely need to do something about the prefetches in the memcpy function. We can either get rid of all the prefetches or make sure we don't prefetch out side the "memcpy" area.

We could fix the prefetch into DMA buffer problem with an extra flush but that's going to be expensive, I rather think we should avoid prefetches. As Kevin explained KSEG1 is a loophole in the spec so we can't really say what the behaviour of memcpy will be in KSEG1.

So I think the fix will have to be:

- Avoid prefetching beyond the end of the copy area in memcpy and memmove.
- Introduce a second variant of memcpy that never does prefetching. This one will be safe to use in KSEG1 / uncached XKPHYS also and will be used for memcpy_fromio, memcpy_toio and friends.

Ralf

#2 lore raw

From: "Ralf Baechle" <[email protected]>

Quoted 6 lines

So I think the fix will have to be:

- Avoid prefetching beyond the end of the copy area in memcpy and memmove.
- Introduce a second variant of memcpy that never does prefetching. This one will be safe to use in KSEG1 / uncached XKPHYS also and will be used for memcpy_fromio, memcpy_toio and friends.

Assuming we had a version that prefetched exactly to the end of the source memory block and no further, why would we need the second variant?

Kevin K.

#3 lore raw
Quoted 8 lines · Kevin D. Kissell
Quoted 4 lines

- Avoid prefetching beyond the end of the copy area in memcpy and memmove.
- Introduce a second variant of memcpy that never does prefetching. This one will be safe to use in KSEG1 / uncached XKPHYS also and will be used for memcpy_fromio, memcpy_toio and friends.

Assuming we had a version that prefetched exactly to the end of the source memory block and no further, why would we need the second variant?

Because the source of memcpy_fromio and the destination of memcpy_toio are some I/O address, typically something like a shared memory region on a network card, which is accessed uncached. The uncached address region might be mapped in KSEG2/KSEG3 or accessed through an uncached region of XKPHYS or KSEG1 where as I recall your statment the effect of prefetch instructions is undefined.

Ralf

#4 lore raw
In reply to Kevin D. Kissell
Quoted 8 lines · Kevin D. Kissell
Quoted 4 lines

- Avoid prefetching beyond the end of the copy area in memcpy and memmove.
- Introduce a second variant of memcpy that never does prefetching. This one will be safe to use in KSEG1 / uncached XKPHYS also and will be used for memcpy_fromio, memcpy_toio and friends.

Assuming we had a version that prefetched exactly to the end of the source memory block and no further, why would we need the second variant?

Uh... Maybe missread your mail the first time. We're also prefechting the destination area, of course with a different hint. The question is what the hints for destination are actually doing, the specs are fairly vague on that and seem to leave the decission to the implementor. Do we really want to try ...

Ralf

#5 lore raw
In reply to Kevin D. Kissell
Quoted 3 lines · Kevin D. Kissell

Assuming we had a version that prefetched exactly to the end of the source memory block and no further, why would we need the second variant?

memcpy_to/from_io needs to do strictly ordered fetches on some devices which fake a memory range with a fifo for performance (pci bursting dirty tricks)

#6 lore raw
In reply to Ralf Baechle

I would like to open this thread again, since nothing has been done about it. I think we should get rid of the prefetches until someone comes up with a version that doesn't prefetch beyond the copy destination/source area.

/Carsten

Quoted 28 lines · Ralf Baechle
Quoted 17 lines · Carsten Langgaard

The problem is the prefetches in the memcpy function in the kernel. There is spread a number of PREF instructions in the memcpy function, but there is no check if we are prefetching out-side the areas we are copying to/from. This is extremely dangerous because we might prefetch out-side the physical memory area, causing e.g. a bus error or something even more nasty.

I recently found something even nastier, it could also hit a DMA buffer region, and thereby break the PCI DMA flushing scheme.
For example if the kernel is doing a memcpy from an area that's next to a DMA buffer area, we could end up in a situation where, we are prefetching
data into the cache from a memory location that is used for DMA transfer and owned by the device, but the DMA transfer has not yet completed. We then end up in a situation, where the memory and cache is out of sync and the cache is containing some old data.

Quoted 3 lines

So we definitely need to do something about the prefetches in the memcpy function. We can either get rid of all the prefetches or make sure we don't prefetch out side the "memcpy" area.

We could fix the prefetch into DMA buffer problem with an extra flush but that's going to be expensive, I rather think we should avoid prefetches. As Kevin explained KSEG1 is a loophole in the spec so we can't really say what the behaviour of memcpy will be in KSEG1.

Quoted 8 lines

So I think the fix will have to be:

- Avoid prefetching beyond the end of the copy area in memcpy and memmove.
- Introduce a second variant of memcpy that never does prefetching. This one will be safe to use in KSEG1 / uncached XKPHYS also and will be used for memcpy_fromio, memcpy_toio and friends.

Ralf

--

_    _ ____  ___   Carsten Langgaard   Mailto:[email protected]
|\  /|||___)(___   MIPS Denmark        Direct: +45 4486 5527
| \/ |||    ____)  Lautrupvang 4B      Switch: +45 4486 5555
  TECHNOLOGIES     2750 Ballerup       Fax...: +45 4486 5556
                   Denmark             http://www.mips.com
#7 lore raw

Carsten,

Quoted 2 lines

I think we should get rid of the prefetches until someone comes up with a version that doesn't prefetch beyond the copy destination/source area.

I agree.

--
Dominic

#8 lore raw

Here's what I think we should do for now (attached patch).

/Carsten

Quoted 9 lines · Dominic Sweetman

Carsten,

Quoted 2 lines

I think we should get rid of the prefetches until someone comes up with a version that doesn't prefetch beyond the copy destination/source area.

I agree.

--
Dominic

--

_    _ ____  ___   Carsten Langgaard   Mailto:[email protected]
|\  /|||___)(___   MIPS Denmark        Direct: +45 4486 5527
| \/ |||    ____)  Lautrupvang 4B      Switch: +45 4486 5555
  TECHNOLOGIES     2750 Ballerup       Fax...: +45 4486 5556
                   Denmark             http://www.mips.com

Index: arch/mips/lib/memcpy.S
===================================================================
RCS file: /home/cvs/linux/arch/mips/lib/memcpy.S,v retrieving revision 1.6.2.4

mips/lib/memcpy.S +11 -0
diff -u -r1.6.2.4 memcpy.S
@@ -21,6 +21,17 @@
 #define src a1
 #define len a2
 
+/* 
+ * There is spread a number of PREF instructions in the memcpy function, but 
+ * there is no check if we are prefetching out-side the "memcpy" areas. 
+ * This is extremely dangerous because we might prefetch out-side the physical
+ * memory area causing e.g. a bus error or something even more nasty. 
+ * It could also hit a DMA buffer region, and there by break the PCI DMA 
+ * flushing scheme.
+ * So for now, we simply get rid of the PREFs here.
+ */
+#define PREF(hint,addr)
+	
 /*
  * Spec
  *

Index: arch/mips64/lib/memcpy.S
===================================================================
RCS file: /home/cvs/linux/arch/mips64/lib/memcpy.S,v retrieving revision 1.9.2.3

mips64/lib/memcpy.S +11 -0
diff -u -r1.9.2.3 memcpy.S
@@ -21,6 +21,17 @@
 #define src a1
 #define len a2
 
+/* 
+ * There is spread a number of PREF instructions in the memcpy function, but 
+ * there is no check if we are prefetching out-side the "memcpy" areas. 
+ * This is extremely dangerous because we might prefetch out-side the physical
+ * memory area causing e.g. a bus error or something even more nasty. 
+ * It could also hit a DMA buffer region, and there by break the PCI DMA 
+ * flushing scheme.
+ * So for now, we simply get rid of the PREFs here.
+ */
+#define PREF(hint,addr)
+	
 /*
  * Spec
  *
#9 lore raw
Quoted 1 line · Carsten Langgaard

Here's what I think we should do for now (attached patch).

Obviously you want to #undef them first...

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#10 lore raw
In reply to Carsten Langgaard

As someone rather new to this list, and to mips-linux in general, can someone explain the problem, as well as how one can avoid it or limit it's effects to a minimum ?

TIA

#11 lore raw
Quoted 3 lines

As someone rather new to this list, and to mips-linux in general, can someone explain the problem, as well as how one can avoid it or limit it's effects to a minimum ?

The problem is that memcpy() uses prefetch to pre-load the cache from the copy source, but does so very naively, such that it continues prefetching beyond the end of the block to be copied. This is a Bad Idea for a couple of reasons. The first to be commented upon was the fact that it can result in bus hangs/bus errors if the kernel is trying to copy from the last few hundred bytes of physical memory, but the really nasty one is that it screws up cache coherence for DMA I/O.

The no-brainer solution which has been proposed as a stopgap is simply to stop doing prefetch in memcpy() altogether. The "correct" solution would be to have a slightly more complex memcpy() loop which only does prefetch up to the end of the source block, which is what is now done in the x86 port.