Skip to contentSkip to conversation

Promblem with PREF (prefetching) in memcpy

DISCUSSION Carsten Langgaard 8 participants 27 messages Oct 4, 2002 – Oct 5, 2002 1 message not in the archive MIPS

lore
Share & tools
lore
#0 lore raw

I think we have a problem with the PREF instructions spread out in the memcpy function.
We are prefetching outside the area we are copying. That's usually not a problem, but if we are prefetching outside the physical memory area (with an unmapped kseg address), anything could happen.
We could get a bus error (which we potentially could handle), but even worse we could have mapped the PCI space immediately following the the RAM area and then anything could happen.

So I think, we either need to make sure not to prefetch outside a page boundary or we make sure the last page in physical memory doesn't get use for unmapped kernel addresses.

Any comments ?

/Carsten

--

_    _ ____  ___   Carsten Langgaard   Mailto:[email protected]
|\  /|||___)(___   MIPS Denmark        Direct: +45 4486 5527
| \/ |||    ____)  Lautrupvang 4B      Switch: +45 4486 5555
  TECHNOLOGIES     2750 Ballerup       Fax...: +45 4486 5556
                   Denmark             http://www.mips.com
#1 lore raw
Quoted 2 lines · Carsten Langgaard

I think we have a problem with the PREF instructions spread out in the memcpy function.

Not really. The MIPS32 manual (for example):

"PREF does not cause addressing-related exceptions. If it does happen
to raise an exception condition, the exception condition is ignored. If an addressing-related exception condition is raised and ignored, no data movement occurs."

PREF never generates a memory operation for a location with an uncached memory access type."

For a Linux user program, at least, memory pages are "memory-like":
reads are guaranteed to be side-effect free, so any outlying prefetches are harmless. It's hard to see any circumstance where an accessible cacheable location would lead to bad side-effects on read.

Signature · 5 lines
Dominic Sweetman, 
MIPS Technologies (UK) - formerly Algorithmics
The Fruit Farm, Ely Road, Chittering, CAMBS CB5 9PH, ENGLAND
phone: +44 1223 706200 / fax: +44 1223 706250 / direct: +44 1223 706205
http://www.algor.co.uk
#2 lore raw
Quoted 11 lines · Dominic Sweetman
Quoted 2 lines · Carsten Langgaard

I think we have a problem with the PREF instructions spread out in the memcpy function.

Not really. The MIPS32 manual (for example):

"PREF does not cause addressing-related exceptions. If it does happen
to raise an exception condition, the exception condition is ignored. If an addressing-related exception condition is raised and ignored, no data movement occurs."

Is a bus error exception an address related exception ? I'm afraid some implementation think it's not.

Quoted 8 lines

PREF never generates a memory operation for a location with an uncached memory access type."

For a Linux user program, at least, memory pages are "memory-like":
reads are guaranteed to be side-effect free, so any outlying prefetches are harmless. It's hard to see any circumstance where an accessible cacheable location would lead to bad side-effects on read.

What about an UART RX register, we might loose a character ? You can also configure you system, so you get a external interrupt from you system controller in case of a bus error, there is no way the CPU can relate this interrupt to the prefetching.

Quoted 8 lines

--
Dominic Sweetman,
MIPS Technologies (UK) - formerly Algorithmics
The Fruit Farm, Ely Road, Chittering, CAMBS CB5 9PH, ENGLAND
phone: +44 1223 706200 / fax: +44 1223 706250 / direct: +44 1223 706205
http://www.algor.co.uk

--

_    _ ____  ___   Carsten Langgaard   Mailto:[email protected]
|\  /|||___)(___   MIPS Denmark        Direct: +45 4486 5527
| \/ |||    ____)  Lautrupvang 4B      Switch: +45 4486 5555
  TECHNOLOGIES     2750 Ballerup       Fax...: +45 4486 5556
                   Denmark             http://www.mips.com
#3 lore raw
Quoted 3 lines · Carsten Langgaard

Is a bus error exception an address related exception ? I'm afraid some implementation think it's not.

So you need an option for broken systems, no new news 8)

Quoted 4 lines

What about an UART RX register, we might loose a character ? You can also configure you system, so you get a external interrupt from you system controller in case of a bus error, there is no way the CPU can relate this interrupt to the prefetching.

The use of memcpy for I/O space isnt permitted in Linux, thats why we have memcpy_*_io stuff. Thus prefetches should never touch 'special' spaces. (On x86 the older Athlons corrupt their cache if you do this so its not a mips specific matter)

#4 lore raw
Quoted 12 lines · Alan Cox
Quoted 3 lines · Carsten Langgaard

Is a bus error exception an address related exception ? I'm afraid some implementation think it's not.

So you need an option for broken systems, no new news 8)

Quoted 4 lines

What about an UART RX register, we might loose a character ? You can also configure you system, so you get a external interrupt from you system controller in case of a bus error, there is no way the CPU can relate this interrupt to the prefetching.

Quoted 4 lines

The use of memcpy for I/O space isnt permitted in Linux, thats why we have memcpy_*_io stuff. Thus prefetches should never touch 'special' spaces. (On x86 the older Athlons corrupt their cache if you do this so its not a mips specific matter)

That's exactly the problem.
The actually loads and stores in memcpy is fine, it's the prefetching that prefetch too much.

/Carsten

--

_    _ ____  ___   Carsten Langgaard   Mailto:[email protected]
|\  /|||___)(___   MIPS Denmark        Direct: +45 4486 5527
| \/ |||    ____)  Lautrupvang 4B      Switch: +45 4486 5555
  TECHNOLOGIES     2750 Ballerup       Fax...: +45 4486 5556
                   Denmark             http://www.mips.com
#5 lore raw
Quoted 3 lines · Carsten Langgaard

That's exactly the problem.
The actually loads and stores in memcpy is fine, it's the prefetching that prefetch too much.

Then fix your prefetching code 8). We had problems with that on x86 too, prefetching off the last page into ISA space -> death. Was umm fun to debug

#6 lore raw

Yes, I think that must be the safest solution.

Any kernel routine which uses PREF instructions (but actually also user space code as Ralf's example pointed out), must guarantee that it does not issue prefetches to addresses which it is not specifically being asked to mess around with.

The other solutions seem too fragile, and will break at some point.

/Hartvig

Quoted 12 lines · Alan Cox
Quoted 3 lines · Carsten Langgaard

That's exactly the problem.
The actually loads and stores in memcpy is fine, it's the prefetching that prefetch too much.

Then fix your prefetching code 8). We had problems with that on x86 too, prefetching off the last page into ISA space -> death. Was umm fun to debug

Signature · 4 lines
 _    _   _____  ____     Hartvig Ekner        Mailto:[email protected]
 |\  /| | |____)(____                          Direct: +45 4486 5503
 | \/ | | |     _____)    MIPS Denmark         Switch: +45 4486 5555
T E C H N O L O G I E S   http://www.mips.com  Fax...: +45 4486 5556
#7 lore raw
In reply to Alan Cox
Quoted 5 lines · Carsten Langgaard
Quoted 2 lines

Is a bus error exception an address related exception ? I'm afraid some implementation think it's not.

So you need an option for broken systems, no new news 8)

Alas, it's not so simple. Bus errors are often not even precise exceptions, which is to say, they cannot be associated with a specific causing instruction.

Quoted 9 lines
Quoted 4 lines

What about an UART RX register, we might loose a character ? You can also configure you system, so you get a external interrupt from you system controller in case of a bus error, there is no way the CPU can relate this interrupt to the prefetching.

The use of memcpy for I/O space isnt permitted in Linux, thats why we have memcpy_*_io stuff. Thus prefetches should never touch 'special' spaces. (On x86 the older Athlons corrupt their cache if you do this so its not a mips specific matter)

The issue isn't that anyone would deliberately use memcpy() in I/O space. Rather, it's that memcpy() prefetches quite a ways ahead, and if one has I/O space assigned just after the end of physical memory, Bad Things might happen on a perfectly legal memcpy() that references the last couple hundred bytes of memory in a way that not even a clever and well-informed bus error handler could undo.

Kevin K.

#8 lore raw
Quoted 7 lines · Kevin D. Kissell

The issue isn't that anyone would deliberately use memcpy() in I/O space. Rather, it's that memcpy() prefetches quite a ways ahead, and if one has I/O space assigned just after the end of physical memory, Bad Things might happen on a perfectly legal memcpy() that references the last couple hundred bytes of memory in a way that not even a clever and well-informed bus error handler could undo.

Then your memcpy function is IMHO broken. Fix it to note prefetch beyond the end of the area you actually will copy and life should be a lot better

#9 lore raw

From: "Alan Cox" <[email protected]>

Quoted 9 lines · Kevin D. Kissell
Quoted 7 lines

The issue isn't that anyone would deliberately use memcpy() in I/O space. Rather, it's that memcpy() prefetches quite a ways ahead, and if one has I/O space assigned just after the end of physical memory, Bad Things might happen on a perfectly legal memcpy() that references the last couple hundred bytes of memory in a way that not even a clever and well-informed bus error handler could undo.

Quoted 3 lines

Then your memcpy function is IMHO broken. Fix it to note prefetch beyond the end of the area you actually will copy and life should be a lot better

Which is excatly the point that Carsten was raising when he started this thread!

The question is how, i.e. throttle memcpy or thow away a "guard band" of RAM?

Regards,

Kevin K.

#10 lore raw
Quoted 3 lines · Kevin D. Kissell

Which is excatly the point that Carsten was raising when he started this thread!

The question is how, i.e. throttle memcpy or thow away a "guard band" of RAM?

The x86 code basically says

	while(over 320 bytes left)
	{
		prefetch ahead
		copy bits
	}
	while(bytes left)
		copy bits

You also have to watch the prefetching in the tcp checksum code. We hit an x86 bug there also very recently. That one is new but shows up with the zero copy tcp nfs

#11 lore raw

From: "Alan Cox" <[email protected]>

Quoted 8 lines · Kevin D. Kissell
Quoted 3 lines

Which is excatly the point that Carsten was raising when he started this thread!

The question is how, i.e. throttle memcpy or thow away a "guard band" of RAM?

The x86 code basically says

Quoted 7 lines

while(over 320 bytes left)
{
prefetch ahead
copy bits
}
while(bytes left)
copy bits

Which is safe, simple, and efficient, but does seem to have the property that there's a "cursed" page in the system that can be randomly allocated and which will be curiously slow on memcopy(). That might or might not be a problem in the embedded application space.

Regards,

Kevin K.

#12 lore raw
Quoted 4 lines · Kevin D. Kissell

Which is safe, simple, and efficient, but does seem to have the property that there's a "cursed" page in the system that can be randomly allocated and which will be curiously slow on memcopy(). That might or might not be a problem in the embedded application space.

Im curious why you think that. Its the same speed for all pages and all copies. If anything it is more efficient because we don't drag 320 bytes of crap across the bus that we dont actually want anyway.

#13 lore raw
In reply to Alan Cox
Quoted 4 lines · Alan Cox
Quoted 2 lines

Is a bus error exception an address related exception ? I'm afraid some implementation think it's not.

So you need an option for broken systems, no new news 8)

For MIPS a bus error exception is not an address related exception by
definition (not surprising, anyway). Specifically, the BadVaddr register is not set upon one.

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#14 lore raw
Quoted 8 lines · Maciej W. Rozycki
Quoted 4 lines
Quoted 2 lines

Is a bus error exception an address related exception ? I'm afraid some implementation think it's not.

So you need an option for broken systems, no new news 8)

For MIPS a bus error exception is not an address related exception by
definition (not surprising, anyway). Specifically, the BadVaddr register is not set upon one.

A particular pain is the exception can be delayed almost arbitrarily due to posted writes. So quite frequently there is no easy association between the machine state at exception time and the cause ...

Ralf

#15 lore raw
Quoted 7 lines · Ralf Baechle
Quoted 3 lines

For MIPS a bus error exception is not an address related exception by
definition (not surprising, anyway). Specifically, the BadVaddr register is not set upon one.

A particular pain is the exception can be delayed almost arbitrarily due to posted writes. So quite frequently there is no easy association between the machine state at exception time and the cause ...

Hmm, I'm not sure if it's permitted to signal a bus exception for
asynchronous write cycles. At least the R3k and the R4000/R4400 specifications explictly forbid them and state that a general interrupt has to be used in such a case instead (and at least the DECstations get this circuitry right -- all of them have a dedicated gp interrupt input for bus errors on a write). Later chips might have relaxed this requirement though -- I don't know at the moment.

Anyway, the address which is involved in the cycle resulting in a bus
exception is always valid and the reason lies elsewhere -- e.g. it may be a bus timeout or a data ECC error.

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#16 lore raw
In reply to Carsten Langgaard
Quoted 9 lines · * Carsten Langgaard ([email protected]) wrote:
Quoted 4 lines

For a Linux user program, at least, memory pages are "memory-like":
reads are guaranteed to be side-effect free, so any outlying prefetches are harmless. It's hard to see any circumstance where an accessible cacheable location would lead to bad side-effects on read.

What about an UART RX register, we might loose a character ? You can also configure you system, so you get a external interrupt from you system controller in case of a bus error, there is no way the CPU can relate this interrupt to the prefetching.

Well those woudn't be cacheable (hmm what happens to a prefetch on none cached areas?) and also you could argue that you shouldn't be proding UARTs from user space (although you could equally argue that it is perfectly valid - but if you were doing it you probably wouldn't be attacking them with a memcpy).

Dave
---------------- Have a happy GNU millennium! ----------------------

/ Dr. David Alan Gilbert    | Running GNU/Linux on Alpha,68K| Happy  \ 
\ gro.gilbert @ treblig.org | MIPS,x86,ARM, SPARC and HP-PA | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/
#17 lore raw
In reply to Carsten Langgaard
Quoted 6 lines · Carsten Langgaard
Quoted 4 lines

"PREF does not cause addressing-related exceptions. If it does happen
to raise an exception condition, the exception condition is ignored. If an addressing-related exception condition is raised and ignored, no data movement occurs."

Is a bus error exception an address related exception ?

Yes.

I agree some MIPS documentation has been ambiguous on the subject, probably because from the chip designer's point of view the address is generated early and internally, and the data turns up later. It's ambiguous whether the BadVaddr register will be correctly set on a bus error (it is on most Big MIPS CPUs, I believe).

But that's beside the point. PREF should plainly never cause an exception because of a bus error.

Quoted 1 line

I'm afraid some implementation think it's not.

That would be a bug.

So let's suppose you've got that bug. And you've got a program running in cacheable memory, and PREF wanders off the edge of the region.

If you were running mapped, you surely wouldn't have a spurious cacheable mapping to single-cycle memory...

In kseg0, you might step into a different physical address region. It's good practice to reserve a 'guard' area of address space between general memory and I/O registers, of course. If the hardware doesn't do it, maybe the software could, by simply refusing to allocate the highest addressable 4Kbytes of memory for general purposes.

Some systems idly decode a cache refill in a non-memory region as one or a sequence of reads, causing a bus error.

I count that as one CPU bug and two system misfeatures. The solution, I guess, is a "bogus bus error" handler.

Quoted 1 line

What about an UART RX register, we might loose a character ?

Yes, without a guard region and with cache refills cheerfully decoded to bogus single-cycle reads, you can get bogus reads. With a careless-enough memory map, you might read something with a side effect.

Quoted 3 lines

You can also configure you system, so you get a external interrupt from you system controller in case of a bus error, there is no way the CPU can relate this interrupt to the prefetching.

Yes, that's true; interrupts on bus errors are vaguely useful for post-mortem diagnosis, but useless for recovery.

Signature · 5 lines
Dominic Sweetman, 
MIPS Technologies (UK) - formerly Algorithmics
The Fruit Farm, Ely Road, Chittering, CAMBS CB5 9PH, ENGLAND
phone: +44 1223 706200 / fax: +44 1223 706250 / direct: +44 1223 706205
http://www.algor.co.uk
#18 lore raw
Quoted 6 lines · Dominic Sweetman
Quoted 3 lines

You can also configure you system, so you get a external interrupt from you system controller in case of a bus error, there is no way the CPU can relate this interrupt to the prefetching.

Yes, that's true; interrupts on bus errors are vaguely useful for post-mortem diagnosis, but useless for recovery.

Why do you think so? Bus errors on reads are synchronous, so if the
handler can judge the error is recoverable (e.g. it was a correctable ECC error), you can just restart.

Bus errors on writes are asynchronous, but you may record the necessary
details (the cycle type, the address and the data involved) in chipset registers and if the handler judges the conditions are recoverable, it may fix up the error -- e.g. mark the page of physical memory as bad and substitute another one, copying all other valid data plus the bits that caused the failure.

A system has to be designed to handle such cases, though. And obviously
if you get one of these errors as a result of a severe hardware damage, you may not be able to recover, anyway.

Signature · 3 lines
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +
#19 lore raw
In reply to Dominic Sweetman

Hi Dom,

this problem occurs in kernel space (kseg0), not user space. In user space there is no problem due to the TLB "protection" of PREFs going outside the process working set, but that doesn't help in kernel mode.

/Hartvig

Quoted 28 lines · Dominic Sweetman
Quoted 2 lines · Carsten Langgaard

I think we have a problem with the PREF instructions spread out in the memcpy function.

Not really. The MIPS32 manual (for example):

"PREF does not cause addressing-related exceptions. If it does happen
to raise an exception condition, the exception condition is ignored. If an addressing-related exception condition is raised and ignored, no data movement occurs."

PREF never generates a memory operation for a location with an uncached memory access type."

For a Linux user program, at least, memory pages are "memory-like":
reads are guaranteed to be side-effect free, so any outlying prefetches are harmless. It's hard to see any circumstance where an accessible cacheable location would lead to bad side-effects on read.

Signature · 5 lines
Dominic Sweetman, 
MIPS Technologies (UK) - formerly Algorithmics
The Fruit Farm, Ely Road, Chittering, CAMBS CB5 9PH, ENGLAND
phone: +44 1223 706200 / fax: +44 1223 706250 / direct: +44 1223 706205
http://www.algor.co.uk
#20 lore raw
Quoted 3 lines · Hartvig Ekner

this problem occurs in kernel space (kseg0), not user space. In user space there is no problem due to the TLB "protection" of PREFs going outside the process working set, but that doesn't help in kernel mode.

Assume a userspace device driver having some hardware mapped immediately following a ordinary memory mapping. In that case it would be possible for prefetch to run from the one into the next mapping - boom.

Ralf

#21 lore raw
In reply to Dominic Sweetman

From: "Dominic Sweetman" <[email protected]>

Quoted 12 lines
Quoted 2 lines · Carsten Langgaard

I think we have a problem with the PREF instructions spread out in the memcpy function.

Not really. The MIPS32 manual (for example):

"PREF does not cause addressing-related exceptions. If it does happen
to raise an exception condition, the exception condition is ignored. If an addressing-related exception condition is raised and ignored, no data movement occurs."

Quoted 7 lines

PREF never generates a memory operation for a location with an uncached memory access type."

For a Linux user program, at least, memory pages are "memory-like":
reads are guaranteed to be side-effect free, so any outlying prefetches are harmless. It's hard to see any circumstance where an accessible cacheable location would lead to bad side-effects on read.

In case Carsten's reply wasn't clear enough, there is a loophole in the spec: kseg0. There is no TLB access to cause a TLB exception (which would suppress the operation and be nullifed), If the prefetch address is correctly aligned, so that there is no address exception. In typical use, kseg0 is cacheable, which means that the second paragraph you quote does not apply. A prefetch to a well-formed, cacheable kseg0 address which has no primary storage behind it (e.g. 0x04000000 on a system with 64M of physical memory) should, according to the spec, cause a cache fill to be initiated for the line at that address, which will result in a bus error, if not a flat-out system hang.

Regards,

Kevin K.

#22 lore raw
Quoted 5 lines

A prefetch to a well-formed, cacheable kseg0 address which has no primary storage behind it (e.g. 0x04000000 on a system with 64M of physical memory) should, according to the spec, cause a cache fill to be initiated for the line at that address,

True (at the CPU boundary).

The system, having decoded the prefetch address and discovered it accesses the 'launch at once on read' I/O register is not obliged to resolve its dilemma by reading it.

It's not even obliged to generate a bus error: it's often less harmful to just ignore it and pacify the CPU with some rubbish data...

... but let's suppose it doesn't:

Quoted 1 line

which will result in a bus error...

I assume that this is not theoretical, and that at least one MTI core fails to ignore bus-error on prefetch? (I take it that you're *not* arguing that this would be wrong). Oops.

If you enable bus errors, you must expect to have a bus error exception handler - and one of its jobs is to quietly ignore the bogus ones.

There are lots of other system hardware or software workarounds.

Quoted 1 line

... if not a flat-out system hang.

It's not good to design such suicide regions into the memory map... surely there's no MTI hardware which does *that*?

Dominic.

#23 lore raw
In reply to Kevin D. Kissell
Quoted 11 lines · Kevin D. Kissell

In case Carsten's reply wasn't clear enough, there is a loophole in the spec: kseg0. There is no TLB access to cause a TLB exception (which would suppress the operation and be nullifed), If the prefetch address is correctly aligned, so that there is no address exception. In typical use, kseg0 is cacheable, which means that the second paragraph you quote does not apply. A prefetch to a well-formed, cacheable kseg0 address which has no primary storage behind it (e.g. 0x04000000 on a system with 64M of physical memory) should, according to the spec, cause a cache fill to be initiated for the line at that address, which will result in a bus error, if not a flat-out system hang.

Traditionally the kernel leaves the last page of memory unused but this has been lost in the many changes to the memory initialization code. It's still a good idea with all the broken I/O chips and PCI bridges out there - and can avoid a hard to track bug.

That only leaves stuff usermode device drivers that are using cachable mappings in the dangerzone. That should be a rare case, if at all.

Ralf

#24 lore raw

I still say we should fix memcpy() (and everything else using prefetches) never to prefetch anything it hasn't been asked to touch.

Apart from the performance loss of the current practice which Alan pointed out, I think there is another very good reason to fix it: SW coherency.

Unless we want to do writeback/invalidate both before & after ownership of buffers are given/taken from DMA IO devices, we need to get rid of the prefetches to unwanted areas.

So that's two good arguments for fixing memcpy() as I see it. I haven't really heard any arguments against it?

Whether the last page of physical memory needs to be thrown away or not seems like a separate issue. Is this also done in x86 world? What are the issues you're thinking about Ralf? Are there devices which will read/write beyond what they have been asked to?

/Hartvig

Quoted 26 lines · Ralf Baechle
Quoted 11 lines · Kevin D. Kissell

In case Carsten's reply wasn't clear enough, there is a loophole in the spec: kseg0. There is no TLB access to cause a TLB exception (which would suppress the operation and be nullifed), If the prefetch address is correctly aligned, so that there is no address exception. In typical use, kseg0 is cacheable, which means that the second paragraph you quote does not apply. A prefetch to a well-formed, cacheable kseg0 address which has no primary storage behind it (e.g. 0x04000000 on a system with 64M of physical memory) should, according to the spec, cause a cache fill to be initiated for the line at that address, which will result in a bus error, if not a flat-out system hang.

Traditionally the kernel leaves the last page of memory unused but this has been lost in the many changes to the memory initialization code. It's still a good idea with all the broken I/O chips and PCI bridges out there - and can avoid a hard to track bug.

That only leaves stuff usermode device drivers that are using cachable mappings in the dangerzone. That should be a rare case, if at all.

Ralf

Signature · 4 lines
 _    _   _____  ____     Hartvig Ekner        Mailto:[email protected]
 |\  /| | |____)(____                          Direct: +45 4486 5503
 | \/ | | |     _____)    MIPS Denmark         Switch: +45 4486 5555
T E C H N O L O G I E S   http://www.mips.com  Fax...: +45 4486 5556
#25 lore raw
Quoted 4 lines · Hartvig Ekner

Whether the last page of physical memory needs to be thrown away or not seems like a separate issue. Is this also done in x86 world? What are the issues you're thinking about Ralf? Are there devices which will read/write beyond what they have been asked to?

For x86 it isnt done. There are no real wins for prefetching off the end of memory on x86. Also its important on x86 we don't prefetch into other memory as we may pull stuff out of the cache of other processors on an SMP box, and that is expensive

#26 lore raw
In reply to Dominic Sweetman

From: "Dominic Sweetman" <[email protected]>

Quoted 12 lines
Quoted 2 lines · Carsten Langgaard

I think we have a problem with the PREF instructions spread out in the memcpy function.

Not really. The MIPS32 manual (for example):

"PREF does not cause addressing-related exceptions. If it does happen
to raise an exception condition, the exception condition is ignored. If an addressing-related exception condition is raised and ignored, no data movement occurs."

Quoted 7 lines

PREF never generates a memory operation for a location with an uncached memory access type."

For a Linux user program, at least, memory pages are "memory-like":
reads are guaranteed to be side-effect free, so any outlying prefetches are harmless. It's hard to see any circumstance where an accessible cacheable location would lead to bad side-effects on read.

In case Carsten's reply wasn't clear enough, there is a loophole in the spec: kseg0. There is no TLB access to cause a TLB exception (which would suppress the operation and be nullifed), If the prefetch address is correctly aligned, so that there is no address exception. In typical use, kseg0 is cacheable, which means that the second paragraph you quote does not apply. A prefetch to a well-formed, cacheable kseg0 address which has no primary storage behind it (e.g. 0x04000000 on a system with 64M of physical memory) should, according to the spec, cause a cache fill to be initiated for the line at that address, which will result in a bus error, if not a flat-out system hang.

Regards,

Kevin K.