Anyone noticed that there are a lot of cache flushes after kunmap/kunmap_atomic is called?
On the MIPS processor, cache flushing is done based on virtual addresses. However, in the Linux kernel, there are a lot of places where memory is mapped with kmap or kmap_atomic, then unmapped with the corresponding kunmap or kunmap_atomic and only *then* is the cache flushed. In other words, we only flush the cache after we have dropped the mapping of memory into a virtual address. I think this is generally wrong.
This may really only affect those of us who have enabled high memory, but it's pretty prevalent in kernel code. We noted this before, but have apparently just been bitten by it. Is it just me or is there a fairly widespread problem for processors that flush the cache using virtual addresses?
- - - - - Cisco - - - - -
This e-mail and any attachments may contain information which is confidential, proprietary, privileged or otherwise protected by law. The information is solely intended for the named addressee (or a person responsible for delivering it to the addressee). If you are not the intended recipient of this message, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer.
Quoted 7 lines · David VomLehn
On the MIPS processor, cache flushing is done based on virtual addresses. However, in the Linux kernel, there are a lot of places where memory is mapped with kmap or
kmap_atomic, then unmapped with the corresponding kunmap orkunmap_atomicand only *then* is the cache flushed. In other words, we only flush the cache after we have dropped the mapping of memory into a virtual address. I think this is generally wrong.
Quoted 5 lines
This may really only affect those of us who have enabled high memory, but it's pretty prevalent in kernel code. We noted this before, but have apparently just been bitten by it. Is it just me or is there a fairly widespread problem for processors that flush the cache using virtual addresses?
Not all MIPS processors have virtually indexed caches; some have physically indexed caches or caches which behave effectivly like physically indexed and MIPS highmem only tries to support the latter kinds of caches. Which means most of the cache flushes turn into nops anyway.
Ralf
David,
I'm battling this now. Our mips 24k has a virtually indexed cache.
We're definitely seeing issues where the cache hasn't been flushed for highmem pages. I thought I had this fixed, but I'm still seeing some problems.
Jon
Quoted 23 lines · Ralf Baechle
Quoted 7 lines · David VomLehn
On the MIPS processor, cache flushing is done based on virtual addresses. However, in the Linux kernel, there are a lot of places where memory is mapped with kmap or
kmap_atomic, then unmapped with the corresponding kunmap orkunmap_atomicand only *then* is the cache flushed. In other words, we only flush the cache after we have dropped the mapping of memory into a virtual address. I think this is generally wrong.Quoted 5 lines
This may really only affect those of us who have enabled high memory, but it's pretty prevalent in kernel code. We noted this before, but have apparently just been bitten by it. Is it just me or is there a fairly widespread problem for processors that flush the cache using virtual addresses?
Not all MIPS processors have virtually indexed caches; some have physically indexed caches or caches which behave effectivly like physically indexed and MIPS highmem only tries to support the latter kinds of caches. Which means most of the cache flushes turn into nops anyway.
Ralf
Quoted 6 lines · Jon Fraser
David,
I'm battling this now. Our mips 24k has a virtually indexed cache.
We're definitely seeing issues where the cache hasn't been flushed for highmem pages. I thought I had this fixed, but I'm still seeing some problems.
As I understand it, you have the most difficult combination of things:
o You are using high memory
o You have a virtually indexed cache
o You have data cache aliases
Fortunately, we have only the first two of those in our system. We should probably put together a coherent set of patches for people who want high memory. So far as I can tell, the 32-bit MIPS architecture has a long way to go before it runs out of steam in the embedded world. I expect more people will need MIPS highmem support in the next few years.
David
David,
I agree. It's my number one work item this week. Well, so far, anyway :-)
Jon
Quoted 22 lines · David VomLehn
Quoted 6 lines · Jon Fraser
David,
I'm battling this now. Our mips 24k has a virtually indexed cache.
We're definitely seeing issues where the cache hasn't been flushed for highmem pages. I thought I had this fixed, but I'm still seeing some problems.As I understand it, you have the most difficult combination of things:
o You are using high memory
o You have a virtually indexed cache
o You have data cache aliasesFortunately, we have only the first two of those in our system. We should probably put together a coherent set of patches for people who want high memory. So far as I can tell, the 32-bit MIPS architecture has a long way to go before it runs out of steam in the embedded world. I expect more people will need MIPS highmem support in the next few years.
David
I remember dealing with analgous situations in System V many many years ago. If the cache flushes are virtual, then either there must be bullet-proof protection (hardware or software) against virtual aliasing, or they must be flushed on deallocation, before the mapping is destroyed. The good news is that, provided that DMA buffer management is done correctly, one should be able to use hit-invalidate instead of hit-writeback-invalidate on the deallocated pages. That won't save that many CPU cycles, but it will save some number of memory cycles and milliwatts.
Regards,
Kevin K.
Quoted 12 lines · David VomLehn
On the MIPS processor, cache flushing is done based on virtual addresses. However, in the Linux kernel, there are a lot of places where memory is mapped with kmap or
kmap_atomic, then unmapped with the corresponding kunmap orkunmap_atomicand only *then* is the cache flushed. In other words, we only flush the cache after we have dropped the mapping of memory into a virtual address. I think this is generally wrong.This may really only affect those of us who have enabled high memory, but it's pretty prevalent in kernel code. We noted this before, but have apparently just been bitten by it. Is it just me or is there a fairly widespread problem for processors that flush the cache using virtual addresses?
- - - - -
Cisco - - - - - This e-mail and any attachments may contain information which is confidential, proprietary, privileged or otherwise protected by law. The information is solely intended for the named addressee (or a person responsible for delivering it to the addressee). If you are not the intended recipient of this message, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer.