ARM: V6 MPCore v6_dma_inv_range RWFO fix
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.
--- arch/arm/mm/cache-v6.S | 16 ++++++++++++---- 1 files changed, 12 insertions(+), 4 deletions(-)
arch/arm/mm/cache-v6.S +12 -4
@@ -205,6 +205,10 @@ ENTRY(v6_flush_kern_dcache_area) v6_dma_inv_range: tst r0, #D_CACHE_LINE_SIZE - 1 bic r0, r0, #D_CACHE_LINE_SIZE - 1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldr r2, [r0] @ read for ownership + str r2, [r0] @ write for ownership +#endif #ifdef HARVARD_CACHE mcrne p15, 0, r0, c7, c10, 1 @ clean D line #else
@@ -212,16 +216,16 @@ v6_dma_inv_range: #endif tst r1, #D_CACHE_LINE_SIZE - 1 bic r1, r1, #D_CACHE_LINE_SIZE - 1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrne r2, [r1] @ read for ownership + strne r2, [r1] @ write for ownership +#endif #ifdef HARVARD_CACHE mcrne p15, 0, r1, c7, c14, 1 @ clean & invalidate D line #else mcrne p15, 0, r1, c7, c15, 1 @ clean & invalidate unified line #endif 1: -#ifdef CONFIG_DMA_CACHE_RWFO - ldr r2, [r0] @ read for ownership - str r2, [r0] @ write for ownership -#endif #ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c6, 1 @ invalidate D line #else
@@ -229,6 +233,10 @@ v6_dma_inv_range: #endif add r0, r0, #D_CACHE_LINE_SIZE cmp r0, r1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrlo r2, [r0] @ read for ownership + strlo r2, [r0] @ write for ownership +#endif blo 1b mov r0, #0 mcr p15, 0, r0, c7, c10, 4 @ drain write buffer
Signature · 1 line
1.6.0.6
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix
Quoted 8 lines · Valentine Barshak
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.
You should do this on the data _inside_ the requested buffer. We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix
Hi,
Quoted 10 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.
Quoted 3 lines
You should do this on the data _inside_ the requested buffer. We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.
OK, how about this (untested interdiff):
arch/arm/mm/cache-v6.S +2 -2
@@ -206,8 +206,8 @@ v6_dma_inv_range: tst r0, #D_CACHE_LINE_SIZE - 1 bic r0, r0, #D_CACHE_LINE_SIZE - 1 #ifdef CONFIG_DMA_CACHE_RWFO - ldr r2, [r0] @ read for ownership - str r2, [r0] @ write for ownership + ldr r2, [r0, #D_CACHE_LINE_SIZE - 4] @ read for ownership + str r2, [r0, #D_CACHE_LINE_SIZE - 4] @ write for ownership #endif #ifdef HARVARD_CACHE mcrne p15, 0, r0, c7, c10, 1 @ clean D line
I believe the above addresses "do this on the data _inside_ the requested buffer" and does no more harm to other data in shared cache lines than the preexisting code but still does not address issues of overlapping shared data within a single word but that seems rather asking for trouble in the first place.
Thanks!
--
Regards,
George
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix
Quoted 22 lines · George G. Davis
Quoted 10 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.Quoted 3 lines
You should do this on the data _inside_ the requested buffer. ?We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.
OK, how about this (untested interdiff):
arch/arm/mm/cache-v6.S +0
-0@@ -206,8 +206,8 @@ v6_dma_inv_range:
Quoted 8 lines
? ? ? ?tst ? ? r0, #D_CACHE_LINE_SIZE - 1
? ? ? ?bic ? ? r0, r0, #D_CACHE_LINE_SIZE - 1
?#ifdefCONFIG_DMA_CACHE_RWFO
- ? ? ? ldr ? ? r2, [r0] ? ? ? ? ? ? ? ? ? ? ? ?@ read for ownership
- ? ? ? str ? ? r2, [r0] ? ? ? ? ? ? ? ? ? ? ? ?@ write for ownership
+ ? ? ? ldr ? ? r2, [r0, #D_CACHE_LINE_SIZE - 4] ? ? ? ?@ read for ownership
+ ? ? ? str ? ? r2, [r0, #D_CACHE_LINE_SIZE - 4] ? ? ? ?@ write for ownership ?#endif
r0 here is already aligned to a cache line, so you still read from the same cache line, I don't see any difference.
Signature · 1 line
Catalin
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix
Quoted 18 lines · Catalin Marinas
Quoted 17 lines · George G. Davis
Quoted 10 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.Quoted 3 lines
You should do this on the data _inside_ the requested buffer. ?We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.
OK, how about this (untested interdiff):
Quoted 16 lines
Quoted 5 lines
arch/arm/mm/cache-v6.S +0
-0@@ -206,8 +206,8 @@ v6_dma_inv_range:Quoted 8 lines
? ? ? ?tst ? ? r0, #D_CACHE_LINE_SIZE - 1
? ? ? ?bic ? ? r0, r0, #D_CACHE_LINE_SIZE - 1
?#ifdefCONFIG_DMA_CACHE_RWFO
- ? ? ? ldr ? ? r2, [r0] ? ? ? ? ? ? ? ? ? ? ? ?@ read for ownership
- ? ? ? str ? ? r2, [r0] ? ? ? ? ? ? ? ? ? ? ? ?@ write for ownership
+ ? ? ? ldr ? ? r2, [r0, #D_CACHE_LINE_SIZE - 4] ? ? ? ?@ read for ownership
+ ? ? ? str ? ? r2, [r0, #D_CACHE_LINE_SIZE - 4] ? ? ? ?@ write for ownership ?#endifr0 here is already aligned to a cache line, so you still read from the same cache line, I don't see any difference.
Assuming "start" was not cache line aligned to begin with but let's say it was initially at cache offset 24, then the above meets the contraint that it is inside the buffer and doesn't touch any data which was outside the DMA buffer. I also had a paranoia that start and end may not even be word aligned which is why these RWFOs are placed after those BICs.
Thanks!
--
Regards,
George
Quoted 3 lines
Signature · 2 lines
-- Catalin
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix
In reply to Russell King - ARM LinuxQuoted 10 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.
Quoted 3 lines
You should do this on the data _inside_ the requested buffer. We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.
We could just use the boundary addresses to avoid writing beyond the buffer. Something like below (pretty much moving the BIC after the RFO, untested):
arch/arm/mm/cache-v6.S +8 -0
@@ -204,6 +204,10 @@ ENTRY(v6_flush_kern_dcache_area) */ v6_dma_inv_range: tst r0, #D_CACHE_LINE_SIZE - 1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrne r2, [r0] @ read for ownership + strne r2, [r0] @ write for ownership +#endif bic r0, r0, #D_CACHE_LINE_SIZE - 1 #ifdef HARVARD_CACHE mcrne p15, 0, r0, c7, c10, 1 @ clean D line
@@ -211,6 +215,10 @@ v6_dma_inv_range: mcrne p15, 0, r0, c7, c11, 1 @ clean unified line #endif tst r1, #D_CACHE_LINE_SIZE - 1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrne r2, [r1, #-4] @ read for ownership + strne r2, [r1, #-4] @ write for ownership +#endif bic r1, r1, #D_CACHE_LINE_SIZE - 1 #ifdef HARVARD_CACHE mcrne p15, 0, r1, c7, c14, 1 @ clean & invalidate D line
Signature · 1 line
Catalin
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix
Quoted 25 lines · Catalin Marinas
Quoted 10 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.Quoted 3 lines
You should do this on the data _inside_ the requested buffer. We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.
We could just use the boundary addresses to avoid writing beyond the buffer. Something like below (pretty much moving the BIC after the RFO, untested):
arch/arm/mm/cache-v6.S +0
-0
arch/arm/mm/cache-v6.S ENTRY(v6_flush_kern_dcache_area) · line 209
Quoted 6 lines
arch/arm/mm/cache-v6.S +3
-0@@ -204,6 +204,10 @@ ENTRY(v6_flush_kern_dcache_area) */ v6_dma_inv_range: tst r0, #D_CACHE_LINE_SIZE - 1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrne r2, [r0] @ read for ownership + strne r2, [r0] @ write for ownership
My concern doing RWFO prior to aligning "start" is that it could be non-word-aligned which may result in an alignment fault.
arch/arm/mm/cache-v6.S v6_dma_inv_range: · line 220
Quoted 11 lines
arch/arm/mm/cache-v6.S +4
-0+#endif bic r0, r0, #D_CACHE_LINE_SIZE - 1 #ifdef HARVARD_CACHE mcrne p15, 0, r0, c7, c10, 1 @ clean D line@@ -211,6 +215,10 @@ v6_dma_inv_range: mcrne p15, 0, r0, c7, c11, 1 @ clean unified line #endif tst r1, #D_CACHE_LINE_SIZE - 1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrne r2, [r1, #-4] @ read for ownership + strne r2, [r1, #-4] @ write for ownership
Same concern here with "end".
Are start and end reasonably expected to be word aligned?
Thanks!
--
Regards,
George
Quoted 11 lines
arch/arm/mm/cache-v6.S +1
-0+#endif bic r1, r1, #D_CACHE_LINE_SIZE - 1 #ifdef HARVARD_CACHE mcrne p15, 0, r1, c7, c14, 1 @ clean & invalidate D lineSignature · 2 lines
-- Catalin
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix
Quoted 21 lines · George G. Davis
Quoted 20 lines · Catalin Marinas
Quoted 10 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.Quoted 3 lines
You should do this on the data _inside_ the requested buffer. ?We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.
We could just use the boundary addresses to avoid writing beyond the buffer. Something like below (pretty much moving the BIC after the RFO, untested):
Quoted 14 lines
Quoted 11 lines
arch/arm/mm/cache-v6.S +0
-0@@ -204,6 +204,10 @@ ENTRY(v6_flush_kern_dcache_area)? */
?v6_dma_inv_range:
? ? ? tst ? ? r0, #D_CACHE_LINE_SIZE - 1
+#ifdefCONFIG_DMA_CACHE_RWFO
+ ? ? ldrne ? r2, [r0] ? ? ? ? ? ? ? ? ? ? ? ?@ read for ownership
+ ? ? strne ? r2, [r0] ? ? ? ? ? ? ? ? ? ? ? ?@ write for ownershipMy concern doing RWFO prior to aligning "start" is that it could be non-word-aligned which may result in an alignment fault.
True. Maybe you can use ldrbne/strbne.
Signature · 1 line
Catalin
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix
In reply to Catalin MarinasQuoted 11 lines · Catalin Marinas
Quoted 10 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.
Quoted 7 lines
Quoted 3 lines
You should do this on the data _inside_ the requested buffer. We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.
We could just use the boundary addresses to avoid writing beyond the buffer. Something like below (pretty much moving the BIC after the RFO, untested):
What if the pointer is not word aligned?
The safest thing to do is:
tst r0, #D_CACHE_LINE_SIZE - 1 bic r0, r0, #D_CACHE_LINE_SIZE - 1 ldrneb r2, [r0, #D_CACHE_LINE_SIZE - 1] strneb r2, [r0, #D_CACHE_LINE_SIZE - 1] ... tst r1, #D_CACHE_LINE_SIZE - 1 bic r1, r1, #D_CACHE_LINE_SIZE - 1 ldrneb r2, [r0] strneb r2, [r0]
so that we only touch the very last byte of the first cache line, and the very first byte of the last cache line. That provides for the most safe behaviour that we can manage in all scenarios.
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix
Quoted 16 lines · Russell King - ARM Linux
Quoted 15 lines · Catalin Marinas
Quoted 13 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.You should do this on the data _inside_ the requested buffer. We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.
Quoted 5 lines
Quoted 3 lines
We could just use the boundary addresses to avoid writing beyond the buffer. Something like below (pretty much moving the BIC after the RFO, untested):
What if the pointer is not word aligned?
I followed up in my reply to George.
Quoted 6 lines
The safest thing to do is:
tst r0, #D_CACHE_LINE_SIZE - 1 bic r0, r0, #D_CACHE_LINE_SIZE - 1 ldrneb r2, [r0, #D_CACHE_LINE_SIZE - 1] strneb r2, [r0, #D_CACHE_LINE_SIZE - 1]
Should we always assume that the buffer size is at least a cache line? I think the current code assumes this already.
Quoted 5 lines
...
tst r1, #D_CACHE_LINE_SIZE - 1 bic r1, r1, #D_CACHE_LINE_SIZE - 1 ldrneb r2, [r0] strneb r2, [r0]
Did you mean "ldrneb r2, [r1]"?
Since the interval is exclusive, what if r1 is already cache-line aligned?
Signature · 1 line
Catalin
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix
Quoted 7 lines · Catalin Marinas
Quoted 1 line · Russell King - ARM Linux
What if the pointer is not word aligned?
I followed up in my reply to George.
Quoted 2 lines
The safest thing to do is:
Quoted 7 lines
Quoted 4 lines
tst r0, #D_CACHE_LINE_SIZE - 1 bic r0, r0, #D_CACHE_LINE_SIZE - 1 ldrneb r2, [r0, #D_CACHE_LINE_SIZE - 1] strneb r2, [r0, #D_CACHE_LINE_SIZE - 1]Should we always assume that the buffer size is at least a cache line? I think the current code assumes this already.
Hmm, true. In that case:
tst r0, #D_CACHE_LINE_SIZE - 1 ldrneb r2, [r0] strneb r2, [r0] bic r0, r0, #D_CACHE_LINE_SIZE - 1
Quoted 6 lines
Quoted 4 lines
tst r1, #D_CACHE_LINE_SIZE - 1 bic r1, r1, #D_CACHE_LINE_SIZE - 1 ldrneb r2, [r0] strneb r2, [r0]Did you mean "ldrneb r2, [r1]"?
Yes. However, for the same reason above:
tst r1, #D_CACHE_LINE_SIZE - 1 ldrneb r2, [r1, #-1] strneb r2, [r1, #-1] bic r1, r1, #D_CACHE_LINE_SIZE - 1
Quoted 2 lines
Since the interval is exclusive, what if r1 is already cache-line aligned?
If r1 is cache line aligned, then we won't be touching that cache line except for the main loop, which should already do what is necessary.
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix
Quoted 29 lines · Russell King - ARM Linux
Quoted 7 lines · Catalin Marinas
Quoted 1 line · Russell King - ARM Linux
What if the pointer is not word aligned?
I followed up in my reply to George.
Quoted 2 lines
The safest thing to do is:
Quoted 7 lines
Quoted 4 lines
tst r0, #D_CACHE_LINE_SIZE - 1 bic r0, r0, #D_CACHE_LINE_SIZE - 1 ldrneb r2, [r0, #D_CACHE_LINE_SIZE - 1] strneb r2, [r0, #D_CACHE_LINE_SIZE - 1]Should we always assume that the buffer size is at least a cache line? I think the current code assumes this already.
Hmm, true. In that case:
tst r0, #D_CACHE_LINE_SIZE - 1 ldrneb r2, [r0] strneb r2, [r0] bic r0, r0, #D_CACHE_LINE_SIZE - 1Quoted 5 lines
Quoted 4 lines
tst r1, #D_CACHE_LINE_SIZE - 1 bic r1, r1, #D_CACHE_LINE_SIZE - 1 ldrneb r2, [r0] strneb r2, [r0]
Quoted 8 lines
Quoted 1 line
Did you mean "ldrneb r2, [r1]"?
Yes. However, for the same reason above:
tst r1, #D_CACHE_LINE_SIZE - 1 ldrneb r2, [r1, #-1] strneb r2, [r1, #-1] bic r1, r1, #D_CACHE_LINE_SIZE - 1
This looks OK to me (though not sure about Thumb-2, I haven't checked).
Signature · 1 line
Catalin
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix
Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fix v6_dma_flush_range to prevent RWFO outside the buffer.
--- arch/arm/mm/cache-v6.S | 40 ++++++++++++++++++++++++++-------------- 1 files changed, 26 insertions(+), 14 deletions(-)
arch/arm/mm/cache-v6.S +26 -14
@@ -203,25 +203,29 @@ ENTRY(v6_flush_kern_dcache_area) * - end - virtual end address of region */ v6_dma_inv_range: - tst r0, #D_CACHE_LINE_SIZE - 1 - bic r0, r0, #D_CACHE_LINE_SIZE - 1 -#ifdef HARVARD_CACHE - mcrne p15, 0, r0, c7, c10, 1 @ clean D line -#else - mcrne p15, 0, r0, c7, c11, 1 @ clean unified line -#endif tst r1, #D_CACHE_LINE_SIZE - 1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrneb r2, [r1, #-1] @ read for ownership + strneb r2, [r1, #-1] @ write for ownership +#endif bic r1, r1, #D_CACHE_LINE_SIZE - 1 #ifdef HARVARD_CACHE mcrne p15, 0, r1, c7, c14, 1 @ clean & invalidate D line #else mcrne p15, 0, r1, c7, c15, 1 @ clean & invalidate unified line #endif -1: #ifdef CONFIG_DMA_CACHE_RWFO - ldr r2, [r0] @ read for ownership - str r2, [r0] @ write for ownership + ldrb r2, [r0] @ read for ownership + strb r2, [r0] @ write for ownership +#endif + tst r0, #D_CACHE_LINE_SIZE - 1 + bic r0, r0, #D_CACHE_LINE_SIZE - 1 +#ifdef HARVARD_CACHE + mcrne p15, 0, r0, c7, c10, 1 @ clean D line +#else + mcrne p15, 0, r0, c7, c11, 1 @ clean unified line #endif +1: #ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c6, 1 @ invalidate D line #else
@@ -229,6 +233,10 @@ v6_dma_inv_range: #endif add r0, r0, #D_CACHE_LINE_SIZE cmp r0, r1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrlo r2, [r0] @ read for ownership + strlo r2, [r0] @ write for ownership +#endif blo 1b mov r0, #0 mcr p15, 0, r0, c7, c10, 4 @ drain write buffer
@@ -263,12 +271,12 @@ v6_dma_clean_range: * - end - virtual end address of region */ ENTRY(v6_dma_flush_range) - bic r0, r0, #D_CACHE_LINE_SIZE - 1 -1: #ifdef CONFIG_DMA_CACHE_RWFO - ldr r2, [r0] @ read for ownership - str r2, [r0] @ write for ownership + ldrb r2, [r0] @ read for ownership + strb r2, [r0] @ write for ownership #endif + bic r0, r0, #D_CACHE_LINE_SIZE - 1 +1: #ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c14, 1 @ clean & invalidate D line #else
@@ -276,6 +284,10 @@ ENTRY(v6_dma_flush_range) #endif add r0, r0, #D_CACHE_LINE_SIZE cmp r0, r1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrlob r2, [r0] @ read for ownership + strlob r2, [r0] @ write for ownership +#endif blo 1b mov r0, #0 mcr p15, 0, r0, c7, c10, 4 @ drain write buffer
Signature · 1 line
1.6.0.6
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix
Quoted 4 lines · Valentine Barshak
Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.
Any comments on this one?
Thanks,
Val.
Quoted 13 lines
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fixv6_dma_flush_rangeto prevent RWFO outside the buffer.Signed-off-by Valentine Barshak <[email protected]>Signed-off-by George G. Davis <[email protected]>
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix
In reply to Valentine BarshakQuoted 14 lines · Valentine Barshak
Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fixv6_dma_flush_rangeto prevent RWFO outside the buffer.
Quoted 2 lines
Signed-off-by Valentine Barshak <[email protected]>Signed-off-by George G. Davis <[email protected]>
I eventually found a bit of time to look at this. The patch looks fine to me:
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix
Quoted 16 lines · Catalin Marinas
Quoted 14 lines · Valentine Barshak
Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fixv6_dma_flush_rangeto prevent RWFO outside the buffer.
Quoted 8 lines
Quoted 3 lines
Signed-off-by Valentine Barshak <[email protected]>Signed-off-by George G. Davis <[email protected]>I eventually found a bit of time to look at this. The patch looks fine to me:
Acked-by Catalin Marinas <[email protected]>
Thanks!
Regards,
Val.
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix
In reply to Catalin MarinasQuoted 15 lines · Catalin Marinas
Quoted 14 lines · Valentine Barshak
Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fixv6_dma_flush_rangeto prevent RWFO outside the buffer.
Quoted 5 lines
Quoted 2 lines
Signed-off-by Valentine Barshak <[email protected]>Signed-off-by George G. Davis <[email protected]>I eventually found a bit of time to look at this. The patch looks fine to me:
Acked-by Catalin Marinas <[email protected]>
Is this also a stable candidate? At least it applies cleanly to .35
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix
Quoted 16 lines · Stephen Boyd
Quoted 15 lines · Catalin Marinas
Quoted 14 lines · Valentine Barshak
Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fixv6_dma_flush_rangeto prevent RWFO outside the buffer.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Signed-off-by Valentine Barshak <[email protected]>Signed-off-by George G. Davis <[email protected]>I eventually found a bit of time to look at this. The patch looks fine to me:
Acked-by Catalin Marinas <[email protected]>Is this also a stable candidate? At least it applies cleanly to .35
I'd say it makes sense to add "Cc: stable at kernel.org" to the commit header before forwarding this to the ARM Linux kernel Patch Tracking System [1] in this case.
--
Regards,
George
[1] http://www.arm.linux.org.uk/developer/patches/info.php
Quoted 4 lines
Signature · 3 lines
-- Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix
In reply to Valentine BarshakHi,
Apologies for the delayed reply.
Quoted 65 lines · Valentine Barshak
Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fixv6_dma_flush_rangeto prevent RWFO outside the buffer.Signed-off-by Valentine Barshak <[email protected]>Signed-off-by George G. Davis <[email protected]>--- arch/arm/mm/cache-v6.S | 40 ++++++++++++++++++++++++++-------------- 1 files changed, 26 insertions(+), 14 deletions(-)arch/arm/mm/cache-v6.S +14
-10@@ -203,25 +203,29 @@ ENTRY(v6_flush_kern_dcache_area) * - end - virtual end address of region */ v6_dma_inv_range: - tst r0, #D_CACHE_LINE_SIZE - 1 - bic r0, r0, #D_CACHE_LINE_SIZE - 1 -#ifdef HARVARD_CACHE - mcrne p15, 0, r0, c7, c10, 1 @ clean D line -#else - mcrne p15, 0, r0, c7, c11, 1 @ clean unified line -#endif tst r1, #D_CACHE_LINE_SIZE - 1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrneb r2, [r1, #-1] @ read for ownership + strneb r2, [r1, #-1] @ write for ownership +#endif bic r1, r1, #D_CACHE_LINE_SIZE - 1 #ifdef HARVARD_CACHE mcrne p15, 0, r1, c7, c14, 1 @ clean & invalidate D line #else mcrne p15, 0, r1, c7, c15, 1 @ clean & invalidate unified line #endif -1: #ifdef CONFIG_DMA_CACHE_RWFO - ldr r2, [r0] @ read for ownership - str r2, [r0] @ write for ownership + ldrb r2, [r0] @ read for ownership + strb r2, [r0] @ write for ownership +#endif + tst r0, #D_CACHE_LINE_SIZE - 1 + bic r0, r0, #D_CACHE_LINE_SIZE - 1 +#ifdef HARVARD_CACHE + mcrne p15, 0, r0, c7, c10, 1 @ clean D line +#else + mcrne p15, 0, r0, c7, c11, 1 @ clean unified line #endif +1: #ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c6, 1 @ invalidate D line #else
arch/arm/mm/cache-v6.S v6_dma_inv_range: · line 242
Quoted 10 lines
arch/arm/mm/cache-v6.S +4
-0@@ -229,6 +233,10 @@ v6_dma_inv_range: #endif add r0, r0, #D_CACHE_LINE_SIZE cmp r0, r1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrlo r2, [r0] @ read for ownership + strlo r2, [r0] @ write for ownership +#endif blo 1b mov r0, #0 mcr p15, 0, r0, c7, c10, 4 @ drain write buffer
Just a minor nit...
Any reason for rearranging the order of 'start'/'end' parameters in the above changes? It's not clear that there is any performance benefit by rearranging those parameters and it obfuscates the changes, e.g. this is equivalent to your change above and also much clearer:
arch/arm/mm/cache-v6.S +12 -4
@@ -203,6 +203,10 @@ ENTRY(v6_flush_kern_dcache_area) * - end - virtual end address of region */ v6_dma_inv_range: +#ifdef CONFIG_DMA_CACHE_RWFO + ldrb r2, [r0] @ read for ownership + strb r2, [r0] @ write for ownership +#endif tst r0, #D_CACHE_LINE_SIZE - 1 bic r0, r0, #D_CACHE_LINE_SIZE - 1 #ifdef HARVARD_CACHE
@@ -211,6 +215,10 @@ v6_dma_inv_range: mcrne p15, 0, r0, c7, c11, 1 @ clean unified line #endif tst r1, #D_CACHE_LINE_SIZE - 1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrneb r2, [r1, #-1] @ read for ownership + strneb r2, [r1, #-1] @ write for ownership +#endif bic r1, r1, #D_CACHE_LINE_SIZE - 1 #ifdef HARVARD_CACHE mcrne p15, 0, r1, c7, c14, 1 @ clean & invalidate D line
@@ -218,10 +226,6 @@ v6_dma_inv_range: mcrne p15, 0, r1, c7, c15, 1 @ clean & invalidate unified line #endif 1: -#ifdef CONFIG_DMA_CACHE_RWFO - ldr r2, [r0] @ read for ownership - str r2, [r0] @ write for ownership -#endif #ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c6, 1 @ invalidate D line #else
@@ -229,6 +233,10 @@ v6_dma_inv_range: #endif add r0, r0, #D_CACHE_LINE_SIZE cmp r0, r1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrlo r2, [r0] @ read for ownership + strlo r2, [r0] @ write for ownership +#endif blo 1b mov r0, #0 mcr p15, 0, r0, c7, c10, 4 @ drain write buffer
--
Regards,
George
Quoted 30 lines
arch/arm/mm/cache-v6.S +8
-4@@ -263,12 +271,12 @@ v6_dma_clean_range: * - end - virtual end address of region */ ENTRY(v6_dma_flush_range) - bic r0, r0, #D_CACHE_LINE_SIZE - 1 -1: #ifdef CONFIG_DMA_CACHE_RWFO - ldr r2, [r0] @ read for ownership - str r2, [r0] @ write for ownership + ldrb r2, [r0] @ read for ownership + strb r2, [r0] @ write for ownership #endif + bic r0, r0, #D_CACHE_LINE_SIZE - 1 +1: #ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c14, 1 @ clean & invalidate D line #else@@ -276,6 +284,10 @@ ENTRY(v6_dma_flush_range) #endif add r0, r0, #D_CACHE_LINE_SIZE cmp r0, r1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrlob r2, [r0] @ read for ownership + strlob r2, [r0] @ write for ownership +#endif blo 1b mov r0, #0 mcr p15, 0, r0, c7, c10, 4 @ drain write bufferSignature · 1 line
1.6.0.6
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix
Quoted 79 lines · George G. Davis
Quoted 65 lines · Valentine Barshak
Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fixv6_dma_flush_rangeto prevent RWFO outside the buffer.Signed-off-by Valentine Barshak <[email protected]>Signed-off-by George G. Davis <[email protected]>--- arch/arm/mm/cache-v6.S | 40 ++++++++++++++++++++++++++-------------- 1 files changed, 26 insertions(+), 14 deletions(-)arch/arm/mm/cache-v6.S +14
-10@@ -203,25 +203,29 @@ ENTRY(v6_flush_kern_dcache_area) * - end - virtual end address of region */ v6_dma_inv_range: - tst r0, #D_CACHE_LINE_SIZE - 1 - bic r0, r0, #D_CACHE_LINE_SIZE - 1 -#ifdef HARVARD_CACHE - mcrne p15, 0, r0, c7, c10, 1 @ clean D line -#else - mcrne p15, 0, r0, c7, c11, 1 @ clean unified line -#endif tst r1, #D_CACHE_LINE_SIZE - 1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrneb r2, [r1, #-1] @ read for ownership + strneb r2, [r1, #-1] @ write for ownership +#endif bic r1, r1, #D_CACHE_LINE_SIZE - 1 #ifdef HARVARD_CACHE mcrne p15, 0, r1, c7, c14, 1 @ clean & invalidate D line #else mcrne p15, 0, r1, c7, c15, 1 @ clean & invalidate unified line #endif -1: #ifdef CONFIG_DMA_CACHE_RWFO - ldr r2, [r0] @ read for ownership - str r2, [r0] @ write for ownership + ldrb r2, [r0] @ read for ownership + strb r2, [r0] @ write for ownership +#endif + tst r0, #D_CACHE_LINE_SIZE - 1 + bic r0, r0, #D_CACHE_LINE_SIZE - 1 +#ifdef HARVARD_CACHE + mcrne p15, 0, r0, c7, c10, 1 @ clean D line +#else + mcrne p15, 0, r0, c7, c11, 1 @ clean unified line #endif +1: #ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c6, 1 @ invalidate D line #elsearch/arm/mm/cache-v6.S v6_dma_inv_range: · line 242
Quoted 10 lines
arch/arm/mm/cache-v6.S +4
-0@@ -229,6 +233,10 @@ v6_dma_inv_range: #endif add r0, r0, #D_CACHE_LINE_SIZE cmp r0, r1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrlo r2, [r0] @ read for ownership + strlo r2, [r0] @ write for ownership +#endif blo 1b mov r0, #0 mcr p15, 0, r0, c7, c10, 4 @ drain write bufferJust a minor nit...
Quoted 4 lines
Any reason for rearranging the order of 'start'/'end' parameters in the above changes? It's not clear that there is any performance benefit by rearranging those parameters and it obfuscates the changes, e.g. this is equivalent to your change above and also much clearer:
I've been wondering about this as well. But I was too lazy to rewrite it and see whether there is a performance benefit in the original patch. I agree that yours looks cleaner.
Signature · 1 line
Catalin
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix
Quoted 82 lines · Catalin Marinas
Quoted 80 lines · George G. Davis
Quoted 66 lines · Valentine Barshak
Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fixv6_dma_flush_rangeto prevent RWFO outside the buffer.Signed-off-by Valentine Barshak <[email protected]>Signed-off-by George G. Davis <[email protected]>--- arch/arm/mm/cache-v6.S | 40 ++++++++++++++++++++++++++-------------- 1 files changed, 26 insertions(+), 14 deletions(-)arch/arm/mm/cache-v6.S +14
-10@@ -203,25 +203,29 @@ ENTRY(v6_flush_kern_dcache_area) * - end - virtual end address of region */ v6_dma_inv_range: - tst r0, #D_CACHE_LINE_SIZE - 1 - bic r0, r0, #D_CACHE_LINE_SIZE - 1 -#ifdef HARVARD_CACHE - mcrne p15, 0, r0, c7, c10, 1 @ clean D line -#else - mcrne p15, 0, r0, c7, c11, 1 @ clean unified line -#endif tst r1, #D_CACHE_LINE_SIZE - 1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrneb r2, [r1, #-1] @ read for ownership + strneb r2, [r1, #-1] @ write for ownership +#endif bic r1, r1, #D_CACHE_LINE_SIZE - 1 #ifdef HARVARD_CACHE mcrne p15, 0, r1, c7, c14, 1 @ clean & invalidate D line #else mcrne p15, 0, r1, c7, c15, 1 @ clean & invalidate unified line #endif -1: #ifdef CONFIG_DMA_CACHE_RWFO - ldr r2, [r0] @ read for ownership - str r2, [r0] @ write for ownership + ldrb r2, [r0] @ read for ownership + strb r2, [r0] @ write for ownership +#endif + tst r0, #D_CACHE_LINE_SIZE - 1 + bic r0, r0, #D_CACHE_LINE_SIZE - 1 +#ifdef HARVARD_CACHE + mcrne p15, 0, r0, c7, c10, 1 @ clean D line +#else + mcrne p15, 0, r0, c7, c11, 1 @ clean unified line #endif +1: #ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c6, 1 @ invalidate D line #elsearch/arm/mm/cache-v6.S v6_dma_inv_range: · line 242
Quoted 10 lines
arch/arm/mm/cache-v6.S +4
-0@@ -229,6 +233,10 @@ v6_dma_inv_range: #endif add r0, r0, #D_CACHE_LINE_SIZE cmp r0, r1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrlo r2, [r0] @ read for ownership + strlo r2, [r0] @ write for ownership +#endif blo 1b mov r0, #0 mcr p15, 0, r0, c7, c10, 4 @ drain write bufferJust a minor nit...
Quoted 7 lines
Quoted 5 lines
Any reason for rearranging the order of 'start'/'end' parameters in the above changes? It's not clear that there is any performance benefit by rearranging those parameters and it obfuscates the changes, e.g. this is equivalent to your change above and also much clearer:
It probably obfuscates the change, but I thought it made the code a bit cleaner and easier to read.
So that first the end (r1) is flushed and then we do all the operations needed on the first cache line (r0) in a row.
I'll resubmit the change and add "Cc: stable at kernel.org" to the header.
Thanks,
Val.
Quoted 5 lines
I've been wondering about this as well. But I was too lazy to rewrite it and see whether there is a performance benefit in the original patch. I agree that yours looks cleaner.
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix
Quoted 85 lines · Valentine Barshak
Quoted 79 lines · Catalin Marinas
Quoted 78 lines · George G. Davis
Quoted 65 lines · Valentine Barshak
Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fixv6_dma_flush_rangeto prevent RWFO outside the buffer.Signed-off-by Valentine Barshak <[email protected]>Signed-off-by George G. Davis <[email protected]>---
arch/arm/mm/cache-v6.S| 40 ++++++++++++++++++++++++++--------------1 files changed, 26 insertions(+), 14 deletions(-)arch/arm/mm/cache-v6.S +0
-0@@ -203,25 +203,29 @@ ENTRY(v6_flush_kern_dcache_area) * - end - virtual end address of region */
v6_dma_inv_range:arch/arm/mm/cache-v6.S +4
-7- tst r0, #D_CACHE_LINE_SIZE - 1 - bic r0, r0, #D_CACHE_LINE_SIZE - 1 -#ifdef HARVARD_CACHE - mcrne p15, 0, r0, c7, c10, 1 @ clean D line -#else - mcrne p15, 0, r0, c7, c11, 1 @ clean unified line -#endif tst r1, #D_CACHE_LINE_SIZE - 1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrneb r2, [r1, #-1] @ read for ownership + strneb r2, [r1, #-1] @ write for ownership +#endif bic r1, r1, #D_CACHE_LINE_SIZE - 1#ifdef
HARVARD_CACHE
mcrne p15, 0, r1, c7, c14, 1 @ clean & invalidate D line
#else
mcrne p15, 0, r1, c7, c15, 1 @ clean & invalidate unified line
#endif
-1:
#ifdefCONFIG_DMA_CACHE_RWFOarch/arm/mm/cache-v6.S +9
-2- ldr r2, [r0] @ read for ownership - str r2, [r0] @ write for ownership + ldrb r2, [r0] @ read for ownership + strb r2, [r0] @ write for ownership +#endif + tst r0, #D_CACHE_LINE_SIZE - 1 + bic r0, r0, #D_CACHE_LINE_SIZE - 1 +#ifdef HARVARD_CACHE + mcrne p15, 0, r0, c7, c10, 1 @ clean D line +#else + mcrne p15, 0, r0, c7, c11, 1 @ clean unified line#endif
+1:
#ifdefHARVARD_CACHE
mcr p15, 0, r0, c7, c6, 1 @ invalidate D line
#elsearch/arm/mm/cache-v6.S v6_dma_inv_range: · line 242
Quoted 10 lines
arch/arm/mm/cache-v6.S +4
-0@@ -229,6 +233,10 @@ v6_dma_inv_range: #endif add r0, r0, #D_CACHE_LINE_SIZE cmp r0, r1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrlo r2, [r0] @ read for ownership + strlo r2, [r0] @ write for ownership +#endif blo 1b mov r0, #0 mcr p15, 0, r0, c7, c10, 4 @ drain write bufferJust a minor nit...
Quoted 5 lines
Quoted 4 lines
Any reason for rearranging the order of 'start'/'end' parameters in the above changes? It's not clear that there is any performance benefit by rearranging those parameters and it obfuscates the changes, e.g. this is equivalent to your change above and also much clearer:
Quoted 4 lines
It probably obfuscates the change, but I thought it made the code a bit cleaner and easier to read.
So that first the end (r1) is flushed and then we do all the operations needed on the first cache line (r0) in a row.
OK, I see your rationale now for moving start (r0) alignment and RWFO fixups just prior to the "clean & invalidate" loop to make the code clearer to the casual reader that RWFO for the first line is done outside the loop and conditionally for remaining lines at the end of the loop. That's admittedly subtle but I'd still prefer to see small incremental changes for faster review (or more fully describe those subtle changes in the commit header). Alas, without comments in the code or commit header, the subtleness of those RWFOs won't be any clearer regardless of their spatial locality. You can always submit another patch to rearrange the code for better flow/readability later.
Quoted 1 line
I'll resubmit the change and add "Cc: stable at kernel.org" to the header.
Thanks!
--
Regards,
George
Quoted 7 lines
Thanks,
Val.Quoted 4 lines
I've been wondering about this as well. But I was too lazy to rewrite it and see whether there is a performance benefit in the original patch. I agree that yours looks cleaner.
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix
In reply to Catalin MarinasResending this one. Somehow it got lost and never appeared on the list.
Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fix v6_dma_flush_range to prevent RWFO outside the buffer.
arch/arm/mm/cache-v6.S +26 -14
@@ -203,25 +203,29 @@ ENTRY(v6_flush_kern_dcache_area) * - end - virtual end address of region */ v6_dma_inv_range: - tst r0, #D_CACHE_LINE_SIZE - 1 - bic r0, r0, #D_CACHE_LINE_SIZE - 1 -#ifdef HARVARD_CACHE - mcrne p15, 0, r0, c7, c10, 1 @ clean D line -#else - mcrne p15, 0, r0, c7, c11, 1 @ clean unified line -#endif tst r1, #D_CACHE_LINE_SIZE - 1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrneb r2, [r1, #-1] @ read for ownership + strneb r2, [r1, #-1] @ write for ownership +#endif bic r1, r1, #D_CACHE_LINE_SIZE - 1 #ifdef HARVARD_CACHE mcrne p15, 0, r1, c7, c14, 1 @ clean & invalidate D line #else mcrne p15, 0, r1, c7, c15, 1 @ clean & invalidate unified line #endif -1: #ifdef CONFIG_DMA_CACHE_RWFO - ldr r2, [r0] @ read for ownership - str r2, [r0] @ write for ownership + ldrb r2, [r0] @ read for ownership + strb r2, [r0] @ write for ownership +#endif + tst r0, #D_CACHE_LINE_SIZE - 1 + bic r0, r0, #D_CACHE_LINE_SIZE - 1 +#ifdef HARVARD_CACHE + mcrne p15, 0, r0, c7, c10, 1 @ clean D line +#else + mcrne p15, 0, r0, c7, c11, 1 @ clean unified line #endif +1: #ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c6, 1 @ invalidate D line #else
@@ -229,6 +233,10 @@ v6_dma_inv_range: #endif add r0, r0, #D_CACHE_LINE_SIZE cmp r0, r1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrlo r2, [r0] @ read for ownership + strlo r2, [r0] @ write for ownership +#endif blo 1b mov r0, #0 mcr p15, 0, r0, c7, c10, 4 @ drain write buffer
@@ -263,12 +271,12 @@ v6_dma_clean_range: * - end - virtual end address of region */ ENTRY(v6_dma_flush_range) - bic r0, r0, #D_CACHE_LINE_SIZE - 1 -1: #ifdef CONFIG_DMA_CACHE_RWFO - ldr r2, [r0] @ read for ownership - str r2, [r0] @ write for ownership + ldrb r2, [r0] @ read for ownership + strb r2, [r0] @ write for ownership #endif + bic r0, r0, #D_CACHE_LINE_SIZE - 1 +1: #ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c14, 1 @ clean & invalidate D line #else
@@ -276,6 +284,10 @@ ENTRY(v6_dma_flush_range) #endif add r0, r0, #D_CACHE_LINE_SIZE cmp r0, r1 +#ifdef CONFIG_DMA_CACHE_RWFO + ldrlob r2, [r0] @ read for ownership + strlob r2, [r0] @ write for ownership +#endif blo 1b mov r0, #0 mcr p15, 0, r0, c7, c10, 4 @ drain write buffer
PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix
In reply to Russell King - ARM LinuxQuoted 12 lines · Russell King - ARM Linux
Quoted 9 lines · Valentine Barshak
Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the
v6_dma_inv_rangefunction when cleaning the first line and flushing the last one, in case the address is not aligned toD_CACHE_LINE_SIZEboundary.
Fix this by reading/writing data if needed, before performing cache operations.
Quoted 4 lines
You should do this on the data _inside_ the requested buffer. We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.
The same problem could hit us in v6_dma_flush_range as well, when we do r/w after bic.
Thanks,
Val.