Prefetches in memcpy
I have reported this before and it also gave a lot of responses, but nothing has been done about, unfortunately :-(
The problem is the prefetches in the memcpy function in the kernel. There is spread a number of PREF instructions in the memcpy function, but there is no check if we are prefetching out-side the areas we are copying to/from. This is extremely dangerous because we might prefetch out-side the physical memory area, causing e.g. a bus error or something even more nasty.
I recently found something even nastier, it could also hit a DMA buffer region, and thereby break the PCI DMA flushing scheme.
For example if the kernel is doing a memcpy from an area that's next to a DMA buffer area, we could end up in a situation where, we are prefetching
data into the cache from a memory location that is used for DMA transfer and owned by the device, but the DMA transfer has not yet completed. We then end up in a situation, where the memory and cache is out of sync and the cache is containing some old data.
So we definitely need to do something about the prefetches in the memcpy function.
We can either get rid of all the prefetches or make sure we don't prefetch out side the "memcpy" area.
/Carsten
--
_ _ ____ ___ Carsten Langgaard Mailto:[email protected] |\ /|||___)(___ MIPS Denmark Direct: +45 4486 5527 | \/ ||| ____) Lautrupvang 4B Switch: +45 4486 5555 TECHNOLOGIES 2750 Ballerup Fax...: +45 4486 5556 Denmark http://www.mips.com
Quoted 17 lines · Carsten Langgaard
The problem is the prefetches in the memcpy function in the kernel. There is spread a number of PREF instructions in the memcpy function, but there is no check if we are prefetching out-side the areas we are copying to/from. This is extremely dangerous because we might prefetch out-side the physical memory area, causing e.g. a bus error or something even more nasty.
I recently found something even nastier, it could also hit a DMA buffer region, and thereby break the PCI DMA flushing scheme.
For example if the kernel is doing a memcpy from an area that's next to a DMA buffer area, we could end up in a situation where, we are prefetching
data into the cache from a memory location that is used for DMA transfer and owned by the device, but the DMA transfer has not yet completed. We then end up in a situation, where the memory and cache is out of sync and the cache is containing some old data.
Quoted 3 lines
So we definitely need to do something about the prefetches in the memcpy function. We can either get rid of all the prefetches or make sure we don't prefetch out side the "memcpy" area.
We could fix the prefetch into DMA buffer problem with an extra flush but that's going to be expensive, I rather think we should avoid prefetches. As Kevin explained KSEG1 is a loophole in the spec so we can't really say what the behaviour of memcpy will be in KSEG1.
So I think the fix will have to be:
- Avoid prefetching beyond the end of the copy area in memcpy and memmove.
- Introduce a second variant of memcpy that never does prefetching. This one will be safe to use in KSEG1 / uncached XKPHYS also and will be used for memcpy_fromio, memcpy_toio and friends.
Ralf
From: "Ralf Baechle" <[email protected]>
Quoted 6 lines
So I think the fix will have to be:
- Avoid prefetching beyond the end of the copy area in memcpy and memmove.
- Introduce a second variant of memcpy that never does prefetching. This one will be safe to use in KSEG1 / uncached XKPHYS also and will be used formemcpy_fromio,memcpy_toioand friends.
Assuming we had a version that prefetched exactly to the end of the source memory block and no further, why would we need the second variant?
Kevin K.
Quoted 8 lines · Kevin D. Kissell
Quoted 4 lines
- Avoid prefetching beyond the end of the copy area in memcpy and memmove.
- Introduce a second variant of memcpy that never does prefetching. This one will be safe to use in KSEG1 / uncached XKPHYS also and will be used formemcpy_fromio,memcpy_toioand friends.Assuming we had a version that prefetched exactly to the end of the source memory block and no further, why would we need the second variant?
Because the source of memcpy_fromio and the destination of memcpy_toio are some I/O address, typically something like a shared memory region on a network card, which is accessed uncached. The uncached address region might be mapped in KSEG2/KSEG3 or accessed through an uncached region of XKPHYS or KSEG1 where as I recall your statment the effect of prefetch instructions is undefined.
Ralf
Quoted 8 lines · Kevin D. Kissell
Quoted 4 lines
- Avoid prefetching beyond the end of the copy area in memcpy and memmove.
- Introduce a second variant of memcpy that never does prefetching. This one will be safe to use in KSEG1 / uncached XKPHYS also and will be used formemcpy_fromio,memcpy_toioand friends.Assuming we had a version that prefetched exactly to the end of the source memory block and no further, why would we need the second variant?
Uh... Maybe missread your mail the first time. We're also prefechting the destination area, of course with a different hint. The question is what the hints for destination are actually doing, the specs are fairly vague on that and seem to leave the decission to the implementor. Do we really want to try ...
Ralf
Quoted 3 lines · Kevin D. Kissell
Assuming we had a version that prefetched exactly to the end of the source memory block and no further, why would we need the second variant?
memcpy_to/from_io needs to do strictly ordered fetches on some devices which fake a memory range with a fifo for performance (pci bursting dirty tricks)
I would like to open this thread again, since nothing has been done about it. I think we should get rid of the prefetches until someone comes up with a version that doesn't prefetch beyond the copy destination/source area.
/Carsten
Quoted 28 lines · Ralf Baechle
Quoted 17 lines · Carsten Langgaard
The problem is the prefetches in the memcpy function in the kernel. There is spread a number of PREF instructions in the memcpy function, but there is no check if we are prefetching out-side the areas we are copying to/from. This is extremely dangerous because we might prefetch out-side the physical memory area, causing e.g. a bus error or something even more nasty.
I recently found something even nastier, it could also hit a DMA buffer region, and thereby break the PCI DMA flushing scheme.
For example if the kernel is doing a memcpy from an area that's next to a DMA buffer area, we could end up in a situation where, we are prefetching
data into the cache from a memory location that is used for DMA transfer and owned by the device, but the DMA transfer has not yet completed. We then end up in a situation, where the memory and cache is out of sync and the cache is containing some old data.Quoted 3 lines
So we definitely need to do something about the prefetches in the memcpy function. We can either get rid of all the prefetches or make sure we don't prefetch out side the "memcpy" area.
We could fix the prefetch into DMA buffer problem with an extra flush but that's going to be expensive, I rather think we should avoid prefetches. As Kevin explained KSEG1 is a loophole in the spec so we can't really say what the behaviour of memcpy will be in KSEG1.
Quoted 8 lines
So I think the fix will have to be:
- Avoid prefetching beyond the end of the copy area in memcpy and memmove.
- Introduce a second variant of memcpy that never does prefetching. This one will be safe to use in KSEG1 / uncached XKPHYS also and will be used formemcpy_fromio,memcpy_toioand friends.Ralf
--
_ _ ____ ___ Carsten Langgaard Mailto:[email protected] |\ /|||___)(___ MIPS Denmark Direct: +45 4486 5527 | \/ ||| ____) Lautrupvang 4B Switch: +45 4486 5555 TECHNOLOGIES 2750 Ballerup Fax...: +45 4486 5556 Denmark http://www.mips.com
Carsten,
Quoted 2 lines
I think we should get rid of the prefetches until someone comes up with a version that doesn't prefetch beyond the copy destination/source area.
I agree.
--
Dominic
Here's what I think we should do for now (attached patch).
/Carsten
Quoted 9 lines · Dominic Sweetman
Carsten,
Quoted 2 lines
I think we should get rid of the prefetches until someone comes up with a version that doesn't prefetch beyond the copy destination/source area.
I agree.
--
Dominic
--
_ _ ____ ___ Carsten Langgaard Mailto:[email protected] |\ /|||___)(___ MIPS Denmark Direct: +45 4486 5527 | \/ ||| ____) Lautrupvang 4B Switch: +45 4486 5555 TECHNOLOGIES 2750 Ballerup Fax...: +45 4486 5556 Denmark http://www.mips.com
Index: arch/mips/lib/memcpy.S
===================================================================
RCS file: /home/cvs/linux/arch/mips/lib/memcpy.S,v retrieving revision 1.6.2.4
mips/lib/memcpy.S +11 -0
@@ -21,6 +21,17 @@ #define src a1 #define len a2 +/* + * There is spread a number of PREF instructions in the memcpy function, but + * there is no check if we are prefetching out-side the "memcpy" areas. + * This is extremely dangerous because we might prefetch out-side the physical + * memory area causing e.g. a bus error or something even more nasty. + * It could also hit a DMA buffer region, and there by break the PCI DMA + * flushing scheme. + * So for now, we simply get rid of the PREFs here. + */ +#define PREF(hint,addr) + /* * Spec *
Index: arch/mips64/lib/memcpy.S
===================================================================
RCS file: /home/cvs/linux/arch/mips64/lib/memcpy.S,v retrieving revision 1.9.2.3
mips64/lib/memcpy.S +11 -0
@@ -21,6 +21,17 @@ #define src a1 #define len a2 +/* + * There is spread a number of PREF instructions in the memcpy function, but + * there is no check if we are prefetching out-side the "memcpy" areas. + * This is extremely dangerous because we might prefetch out-side the physical + * memory area causing e.g. a bus error or something even more nasty. + * It could also hit a DMA buffer region, and there by break the PCI DMA + * flushing scheme. + * So for now, we simply get rid of the PREFs here. + */ +#define PREF(hint,addr) + /* * Spec *
Quoted 1 line · Carsten Langgaard
Here's what I think we should do for now (attached patch).
Obviously you want to #undef them first...
Signature · 3 lines
+ Maciej W. Rozycki, Technical University of Gdansk, Poland + +--------------------------------------------------------------+ + e-mail: [email protected], PGP key available +
As someone rather new to this list, and to mips-linux in general, can someone explain the problem, as well as how one can avoid it or limit it's effects to a minimum ?
TIA
Quoted 3 lines
As someone rather new to this list, and to mips-linux in general, can someone explain the problem, as well as how one can avoid it or limit it's effects to a minimum ?
The problem is that memcpy() uses prefetch to pre-load the cache from the copy source, but does so very naively, such that it continues prefetching beyond the end of the block to be copied. This is a Bad Idea for a couple of reasons. The first to be commented upon was the fact that it can result in bus hangs/bus errors if the kernel is trying to copy from the last few hundred bytes of physical memory, but the really nasty one is that it screws up cache coherence for DMA I/O.
The no-brainer solution which has been proposed as a stopgap is simply to stop doing prefetch in memcpy() altogether. The "correct" solution would be to have a slightly more complex memcpy() loop which only does prefetch up to the end of the source block, which is what is now done in the x86 port.