Skip to contentSkip to conversation

ARM: V6 MPCore v6_dma_inv_range RWFO fix

PATCH New Valentine Barshak 5 participants 24 messages Nov 23, 2010 – Dec 10, 2010 Arm (32-bit)

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.

---
 arch/arm/mm/cache-v6.S |   16 ++++++++++++----
 1 files changed, 12 insertions(+), 4 deletions(-)
arch/arm/mm/cache-v6.S +12 -4
index 99fa688..e0e5c6b 100644
@@ -205,6 +205,10 @@ ENTRY(v6_flush_kern_dcache_area)
 v6_dma_inv_range:
 	tst	r0, #D_CACHE_LINE_SIZE - 1
 	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldr	r2, [r0]			@ read for ownership
+	str	r2, [r0]			@ write for ownership
+#endif
 #ifdef HARVARD_CACHE
 	mcrne	p15, 0, r0, c7, c10, 1		@ clean D line
 #else
@@ -212,16 +216,16 @@ v6_dma_inv_range:
 #endif
 	tst	r1, #D_CACHE_LINE_SIZE - 1
 	bic	r1, r1, #D_CACHE_LINE_SIZE - 1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrne	r2, [r1]			@ read for ownership
+	strne	r2, [r1]			@ write for ownership
+#endif
 #ifdef HARVARD_CACHE
 	mcrne	p15, 0, r1, c7, c14, 1		@ clean & invalidate D line
 #else
 	mcrne	p15, 0, r1, c7, c15, 1		@ clean & invalidate unified line
 #endif
 1:
-#ifdef CONFIG_DMA_CACHE_RWFO
-	ldr	r2, [r0]			@ read for ownership
-	str	r2, [r0]			@ write for ownership
-#endif
 #ifdef HARVARD_CACHE
 	mcr	p15, 0, r0, c7, c6, 1		@ invalidate D line
 #else
@@ -229,6 +233,10 @@ v6_dma_inv_range:
 #endif
 	add	r0, r0, #D_CACHE_LINE_SIZE
 	cmp	r0, r1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrlo	r2, [r0]			@ read for ownership
+	strlo	r2, [r0]			@ write for ownership
+#endif
 	blo	1b
 	mov	r0, #0
 	mcr	p15, 0, r0, c7, c10, 4		@ drain write buffer
Signature · 1 line
1.6.0.6
#1 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix

Quoted 8 lines · Valentine Barshak

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.

You should do this on the data _inside_ the requested buffer. We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.

#2 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix

Hi,

Quoted 10 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.

Quoted 3 lines

You should do this on the data _inside_ the requested buffer. We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.

OK, how about this (untested interdiff):

arch/arm/mm/cache-v6.S +2 -2
index e0e5c6b..e778f2a 100644
@@ -206,8 +206,8 @@ v6_dma_inv_range:
 	tst	r0, #D_CACHE_LINE_SIZE - 1
 	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
 #ifdef CONFIG_DMA_CACHE_RWFO
-	ldr	r2, [r0]			@ read for ownership
-	str	r2, [r0]			@ write for ownership
+	ldr	r2, [r0, #D_CACHE_LINE_SIZE - 4]	@ read for ownership
+	str	r2, [r0, #D_CACHE_LINE_SIZE - 4]	@ write for ownership
 #endif
 #ifdef HARVARD_CACHE
 	mcrne	p15, 0, r0, c7, c10, 1		@ clean D line

I believe the above addresses "do this on the data _inside_ the requested buffer" and does no more harm to other data in shared cache lines than the preexisting code but still does not address issues of overlapping shared data within a single word but that seems rather asking for trouble in the first place.

Thanks!

--
Regards,
George

#3 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix

Quoted 22 lines · George G. Davis
Quoted 10 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.

Quoted 3 lines

You should do this on the data _inside_ the requested buffer. ?We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.

OK, how about this (untested interdiff):

arch/arm/mm/cache-v6.S +0 -0
index e0e5c6b..e778f2a 100644
@@ -206,8 +206,8 @@ v6_dma_inv_range:
Quoted 8 lines

? ? ? ?tst ? ? r0, #D_CACHE_LINE_SIZE - 1
? ? ? ?bic ? ? r0, r0, #D_CACHE_LINE_SIZE - 1
?#ifdef CONFIG_DMA_CACHE_RWFO
- ? ? ? ldr ? ? r2, [r0] ? ? ? ? ? ? ? ? ? ? ? ?@ read for ownership
- ? ? ? str ? ? r2, [r0] ? ? ? ? ? ? ? ? ? ? ? ?@ write for ownership
+ ? ? ? ldr ? ? r2, [r0, #D_CACHE_LINE_SIZE - 4] ? ? ? ?@ read for ownership
+ ? ? ? str ? ? r2, [r0, #D_CACHE_LINE_SIZE - 4] ? ? ? ?@ write for ownership ?#endif

r0 here is already aligned to a cache line, so you still read from the same cache line, I don't see any difference.

Signature · 1 line
Catalin
#4 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix

Quoted 18 lines · Catalin Marinas
Quoted 17 lines · George G. Davis
Quoted 10 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.

Quoted 3 lines

You should do this on the data _inside_ the requested buffer. ?We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.

OK, how about this (untested interdiff):

Quoted 16 lines
Quoted 5 lines
arch/arm/mm/cache-v6.S +0 -0
index e0e5c6b..e778f2a 100644
@@ -206,8 +206,8 @@ v6_dma_inv_range:
Quoted 8 lines

? ? ? ?tst ? ? r0, #D_CACHE_LINE_SIZE - 1
? ? ? ?bic ? ? r0, r0, #D_CACHE_LINE_SIZE - 1
?#ifdef CONFIG_DMA_CACHE_RWFO
- ? ? ? ldr ? ? r2, [r0] ? ? ? ? ? ? ? ? ? ? ? ?@ read for ownership
- ? ? ? str ? ? r2, [r0] ? ? ? ? ? ? ? ? ? ? ? ?@ write for ownership
+ ? ? ? ldr ? ? r2, [r0, #D_CACHE_LINE_SIZE - 4] ? ? ? ?@ read for ownership
+ ? ? ? str ? ? r2, [r0, #D_CACHE_LINE_SIZE - 4] ? ? ? ?@ write for ownership ?#endif

r0 here is already aligned to a cache line, so you still read from the same cache line, I don't see any difference.

Assuming "start" was not cache line aligned to begin with but let's say it was initially at cache offset 24, then the above meets the contraint that it is inside the buffer and doesn't touch any data which was outside the DMA buffer. I also had a paranoia that start and end may not even be word aligned which is why these RWFOs are placed after those BICs.

Thanks!

--
Regards,
George

Quoted 3 lines
Signature · 2 lines
-- 
Catalin
#5 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix

In reply to Russell King - ARM Linux
Quoted 10 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.

Quoted 3 lines

You should do this on the data _inside_ the requested buffer. We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.

We could just use the boundary addresses to avoid writing beyond the buffer. Something like below (pretty much moving the BIC after the RFO, untested):

arch/arm/mm/cache-v6.S +8 -0
index 99fa688..d63bb55 100644
@@ -204,6 +204,10 @@ ENTRY(v6_flush_kern_dcache_area)
  */
 v6_dma_inv_range:
 	tst	r0, #D_CACHE_LINE_SIZE - 1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrne	r2, [r0]			@ read for ownership
+	strne	r2, [r0]			@ write for ownership
+#endif
 	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
 #ifdef HARVARD_CACHE
 	mcrne	p15, 0, r0, c7, c10, 1		@ clean D line
@@ -211,6 +215,10 @@ v6_dma_inv_range:
 	mcrne	p15, 0, r0, c7, c11, 1		@ clean unified line
 #endif
 	tst	r1, #D_CACHE_LINE_SIZE - 1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrne	r2, [r1, #-4]			@ read for ownership
+	strne	r2, [r1, #-4]			@ write for ownership
+#endif
 	bic	r1, r1, #D_CACHE_LINE_SIZE - 1
 #ifdef HARVARD_CACHE
 	mcrne	p15, 0, r1, c7, c14, 1		@ clean & invalidate D line
Signature · 1 line
Catalin
#6 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix

Quoted 25 lines · Catalin Marinas
Quoted 10 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.

Quoted 3 lines

You should do this on the data _inside_ the requested buffer. We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.

We could just use the boundary addresses to avoid writing beyond the buffer. Something like below (pretty much moving the BIC after the RFO, untested):

arch/arm/mm/cache-v6.S +0 -0
index 99fa688..d63bb55 100644

arch/arm/mm/cache-v6.S ENTRY(v6_flush_kern_dcache_area) · line 209

Quoted 6 lines
arch/arm/mm/cache-v6.S +3 -0
@@ -204,6 +204,10 @@ ENTRY(v6_flush_kern_dcache_area)
  */
 v6_dma_inv_range:
 	tst	r0, #D_CACHE_LINE_SIZE - 1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrne	r2, [r0]			@ read for ownership
+	strne	r2, [r0]			@ write for ownership

My concern doing RWFO prior to aligning "start" is that it could be non-word-aligned which may result in an alignment fault.

arch/arm/mm/cache-v6.S v6_dma_inv_range: · line 220

Quoted 11 lines
arch/arm/mm/cache-v6.S +4 -0
+#endif
 	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
 #ifdef HARVARD_CACHE
 	mcrne	p15, 0, r0, c7, c10, 1		@ clean D line
@@ -211,6 +215,10 @@ v6_dma_inv_range:
 	mcrne	p15, 0, r0, c7, c11, 1		@ clean unified line
 #endif
 	tst	r1, #D_CACHE_LINE_SIZE - 1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrne	r2, [r1, #-4]			@ read for ownership
+	strne	r2, [r1, #-4]			@ write for ownership

Same concern here with "end".

Are start and end reasonably expected to be word aligned?

Thanks!

--
Regards,
George

Quoted 11 lines
arch/arm/mm/cache-v6.S +1 -0
+#endif
 	bic	r1, r1, #D_CACHE_LINE_SIZE - 1
 #ifdef HARVARD_CACHE
 	mcrne	p15, 0, r1, c7, c14, 1		@ clean & invalidate D line
Signature · 2 lines
-- 
Catalin
#7 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix

Quoted 21 lines · George G. Davis
Quoted 20 lines · Catalin Marinas
Quoted 10 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.

Quoted 3 lines

You should do this on the data _inside_ the requested buffer. ?We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.

We could just use the boundary addresses to avoid writing beyond the buffer. Something like below (pretty much moving the BIC after the RFO, untested):

Quoted 14 lines
Quoted 11 lines
arch/arm/mm/cache-v6.S +0 -0
index 99fa688..d63bb55 100644
@@ -204,6 +204,10 @@ ENTRY(v6_flush_kern_dcache_area)

? */
?v6_dma_inv_range:
? ? ? tst ? ? r0, #D_CACHE_LINE_SIZE - 1
+#ifdef CONFIG_DMA_CACHE_RWFO
+ ? ? ldrne ? r2, [r0] ? ? ? ? ? ? ? ? ? ? ? ?@ read for ownership
+ ? ? strne ? r2, [r0] ? ? ? ? ? ? ? ? ? ? ? ?@ write for ownership

My concern doing RWFO prior to aligning "start" is that it could be non-word-aligned which may result in an alignment fault.

True. Maybe you can use ldrbne/strbne.

Signature · 1 line
Catalin
#8 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix

In reply to Catalin Marinas
Quoted 11 lines · Catalin Marinas
Quoted 10 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.

Quoted 7 lines
Quoted 3 lines

You should do this on the data _inside_ the requested buffer. We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.

We could just use the boundary addresses to avoid writing beyond the buffer. Something like below (pretty much moving the BIC after the RFO, untested):

What if the pointer is not word aligned?

The safest thing to do is:

	tst	r0, #D_CACHE_LINE_SIZE - 1
	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
	ldrneb	r2, [r0, #D_CACHE_LINE_SIZE - 1]
	strneb	r2, [r0, #D_CACHE_LINE_SIZE - 1]
...
	tst	r1, #D_CACHE_LINE_SIZE - 1
	bic	r1, r1, #D_CACHE_LINE_SIZE - 1
	ldrneb	r2, [r0]
	strneb	r2, [r0]

so that we only touch the very last byte of the first cache line, and the very first byte of the last cache line. That provides for the most safe behaviour that we can manage in all scenarios.

#9 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix

Quoted 16 lines · Russell King - ARM Linux
Quoted 15 lines · Catalin Marinas
Quoted 13 lines · Russell King - ARM Linux
Quoted 8 lines · Valentine Barshak

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.

You should do this on the data _inside_ the requested buffer. We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.

Quoted 5 lines
Quoted 3 lines

We could just use the boundary addresses to avoid writing beyond the buffer. Something like below (pretty much moving the BIC after the RFO, untested):

What if the pointer is not word aligned?

I followed up in my reply to George.

Quoted 6 lines

The safest thing to do is:

        tst     r0, #D_CACHE_LINE_SIZE - 1
        bic     r0, r0, #D_CACHE_LINE_SIZE - 1
        ldrneb  r2, [r0, #D_CACHE_LINE_SIZE - 1]
        strneb  r2, [r0, #D_CACHE_LINE_SIZE - 1]

Should we always assume that the buffer size is at least a cache line? I think the current code assumes this already.

Quoted 5 lines

...

        tst     r1, #D_CACHE_LINE_SIZE - 1
        bic     r1, r1, #D_CACHE_LINE_SIZE - 1
        ldrneb  r2, [r0]
        strneb  r2, [r0]

Did you mean "ldrneb r2, [r1]"?

Since the interval is exclusive, what if r1 is already cache-line aligned?

Signature · 1 line
Catalin
#10 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix

Quoted 7 lines · Catalin Marinas
Quoted 1 line · Russell King - ARM Linux

What if the pointer is not word aligned?

I followed up in my reply to George.

Quoted 2 lines

The safest thing to do is:

Quoted 7 lines
Quoted 4 lines
        tst     r0, #D_CACHE_LINE_SIZE - 1
        bic     r0, r0, #D_CACHE_LINE_SIZE - 1
        ldrneb  r2, [r0, #D_CACHE_LINE_SIZE - 1]
        strneb  r2, [r0, #D_CACHE_LINE_SIZE - 1]

Should we always assume that the buffer size is at least a cache line? I think the current code assumes this already.

Hmm, true. In that case:

	tst     r0, #D_CACHE_LINE_SIZE - 1
	ldrneb  r2, [r0]
	strneb  r2, [r0]
	bic     r0, r0, #D_CACHE_LINE_SIZE - 1
Quoted 6 lines
Quoted 4 lines
        tst     r1, #D_CACHE_LINE_SIZE - 1
        bic     r1, r1, #D_CACHE_LINE_SIZE - 1
        ldrneb  r2, [r0]
        strneb  r2, [r0]

Did you mean "ldrneb r2, [r1]"?

Yes. However, for the same reason above:

	tst	r1, #D_CACHE_LINE_SIZE - 1
	ldrneb	r2, [r1, #-1]
	strneb	r2, [r1, #-1]
	bic	r1, r1, #D_CACHE_LINE_SIZE - 1
Quoted 2 lines

Since the interval is exclusive, what if r1 is already cache-line aligned?

If r1 is cache line aligned, then we won't be touching that cache line except for the main loop, which should already do what is necessary.

#11 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix

Quoted 29 lines · Russell King - ARM Linux
Quoted 7 lines · Catalin Marinas
Quoted 1 line · Russell King - ARM Linux

What if the pointer is not word aligned?

I followed up in my reply to George.

Quoted 2 lines

The safest thing to do is:

Quoted 7 lines
Quoted 4 lines
        tst     r0, #D_CACHE_LINE_SIZE - 1
        bic     r0, r0, #D_CACHE_LINE_SIZE - 1
        ldrneb  r2, [r0, #D_CACHE_LINE_SIZE - 1]
        strneb  r2, [r0, #D_CACHE_LINE_SIZE - 1]

Should we always assume that the buffer size is at least a cache line? I think the current code assumes this already.

Hmm, true. In that case:

        tst     r0, #D_CACHE_LINE_SIZE - 1
        ldrneb  r2, [r0]
        strneb  r2, [r0]
        bic     r0, r0, #D_CACHE_LINE_SIZE - 1
Quoted 5 lines
Quoted 4 lines
        tst     r1, #D_CACHE_LINE_SIZE - 1
        bic     r1, r1, #D_CACHE_LINE_SIZE - 1
        ldrneb  r2, [r0]
        strneb  r2, [r0]
Quoted 8 lines
Quoted 1 line

Did you mean "ldrneb r2, [r1]"?

Yes. However, for the same reason above:

        tst     r1, #D_CACHE_LINE_SIZE - 1
        ldrneb  r2, [r1, #-1]
        strneb  r2, [r1, #-1]
        bic     r1, r1, #D_CACHE_LINE_SIZE - 1

This looks OK to me (though not sure about Thumb-2, I haven't checked).

Signature · 1 line
Catalin
#12 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix

Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fix v6_dma_flush_range to prevent RWFO outside the buffer.

---
 arch/arm/mm/cache-v6.S |   40 ++++++++++++++++++++++++++--------------
 1 files changed, 26 insertions(+), 14 deletions(-)
arch/arm/mm/cache-v6.S +26 -14
index 99fa688..622f8a1 100644
@@ -203,25 +203,29 @@ ENTRY(v6_flush_kern_dcache_area)
  *	- end     - virtual end address of region
  */
 v6_dma_inv_range:
-	tst	r0, #D_CACHE_LINE_SIZE - 1
-	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
-#ifdef HARVARD_CACHE
-	mcrne	p15, 0, r0, c7, c10, 1		@ clean D line
-#else
-	mcrne	p15, 0, r0, c7, c11, 1		@ clean unified line
-#endif
 	tst	r1, #D_CACHE_LINE_SIZE - 1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrneb	r2, [r1, #-1]			@ read for ownership
+	strneb	r2, [r1, #-1]			@ write for ownership
+#endif
 	bic	r1, r1, #D_CACHE_LINE_SIZE - 1
 #ifdef HARVARD_CACHE
 	mcrne	p15, 0, r1, c7, c14, 1		@ clean & invalidate D line
 #else
 	mcrne	p15, 0, r1, c7, c15, 1		@ clean & invalidate unified line
 #endif
-1:
 #ifdef CONFIG_DMA_CACHE_RWFO
-	ldr	r2, [r0]			@ read for ownership
-	str	r2, [r0]			@ write for ownership
+	ldrb	r2, [r0]			@ read for ownership
+	strb	r2, [r0]			@ write for ownership
+#endif
+	tst	r0, #D_CACHE_LINE_SIZE - 1
+	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
+#ifdef HARVARD_CACHE
+	mcrne	p15, 0, r0, c7, c10, 1		@ clean D line
+#else
+	mcrne	p15, 0, r0, c7, c11, 1		@ clean unified line
 #endif
+1:
 #ifdef HARVARD_CACHE
 	mcr	p15, 0, r0, c7, c6, 1		@ invalidate D line
 #else
@@ -229,6 +233,10 @@ v6_dma_inv_range:
 #endif
 	add	r0, r0, #D_CACHE_LINE_SIZE
 	cmp	r0, r1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrlo	r2, [r0]			@ read for ownership
+	strlo	r2, [r0]			@ write for ownership
+#endif
 	blo	1b
 	mov	r0, #0
 	mcr	p15, 0, r0, c7, c10, 4		@ drain write buffer
@@ -263,12 +271,12 @@ v6_dma_clean_range:
  *	- end     - virtual end address of region
  */
 ENTRY(v6_dma_flush_range)
-	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
-1:
 #ifdef CONFIG_DMA_CACHE_RWFO
-	ldr	r2, [r0]			@ read for ownership
-	str	r2, [r0]			@ write for ownership
+	ldrb	r2, [r0]		@ read for ownership
+	strb	r2, [r0]		@ write for ownership
 #endif
+	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
+1:
 #ifdef HARVARD_CACHE
 	mcr	p15, 0, r0, c7, c14, 1		@ clean & invalidate D line
 #else
@@ -276,6 +284,10 @@ ENTRY(v6_dma_flush_range)
 #endif
 	add	r0, r0, #D_CACHE_LINE_SIZE
 	cmp	r0, r1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrlob	r2, [r0]			@ read for ownership
+	strlob	r2, [r0]			@ write for ownership
+#endif
 	blo	1b
 	mov	r0, #0
 	mcr	p15, 0, r0, c7, c10, 4		@ drain write buffer
Signature · 1 line
1.6.0.6
#13 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix

Quoted 4 lines · Valentine Barshak

Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.

Any comments on this one?

Thanks,
Val.

Quoted 13 lines

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fix v6_dma_flush_range to prevent RWFO outside the buffer.

#14 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix

In reply to Valentine Barshak
Quoted 14 lines · Valentine Barshak

Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fix v6_dma_flush_range to prevent RWFO outside the buffer.

Quoted 2 lines

I eventually found a bit of time to look at this. The patch looks fine to me:

#15 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix

Quoted 16 lines · Catalin Marinas
Quoted 14 lines · Valentine Barshak

Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fix v6_dma_flush_range to prevent RWFO outside the buffer.

Quoted 8 lines
Quoted 3 lines

I eventually found a bit of time to look at this. The patch looks fine to me:

Thanks!

Regards,
Val.

#16 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix

In reply to Catalin Marinas
Quoted 15 lines · Catalin Marinas
Quoted 14 lines · Valentine Barshak

Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fix v6_dma_flush_range to prevent RWFO outside the buffer.

Quoted 5 lines
Quoted 2 lines

I eventually found a bit of time to look at this. The patch looks fine to me:

Is this also a stable candidate? At least it applies cleanly to .35

Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
#17 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix

Quoted 16 lines · Stephen Boyd
Quoted 15 lines · Catalin Marinas
Quoted 14 lines · Valentine Barshak

Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fix v6_dma_flush_range to prevent RWFO outside the buffer.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I eventually found a bit of time to look at this. The patch looks fine to me:

Is this also a stable candidate? At least it applies cleanly to .35

I'd say it makes sense to add "Cc: stable at kernel.org" to the commit header before forwarding this to the ARM Linux kernel Patch Tracking System [1] in this case.

--
Regards,
George

[1] http://www.arm.linux.org.uk/developer/patches/info.php

Quoted 4 lines
Signature · 3 lines
-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
#18 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix

In reply to Valentine Barshak

Hi,

Apologies for the delayed reply.

Quoted 65 lines · Valentine Barshak

Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fix v6_dma_flush_range to prevent RWFO outside the buffer.

---
 arch/arm/mm/cache-v6.S |   40 ++++++++++++++++++++++++++--------------
 1 files changed, 26 insertions(+), 14 deletions(-)
arch/arm/mm/cache-v6.S +14 -10
index 99fa688..622f8a1 100644
@@ -203,25 +203,29 @@ ENTRY(v6_flush_kern_dcache_area)
  *	- end     - virtual end address of region
  */
 v6_dma_inv_range:
-	tst	r0, #D_CACHE_LINE_SIZE - 1
-	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
-#ifdef HARVARD_CACHE
-	mcrne	p15, 0, r0, c7, c10, 1		@ clean D line
-#else
-	mcrne	p15, 0, r0, c7, c11, 1		@ clean unified line
-#endif
 	tst	r1, #D_CACHE_LINE_SIZE - 1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrneb	r2, [r1, #-1]			@ read for ownership
+	strneb	r2, [r1, #-1]			@ write for ownership
+#endif
 	bic	r1, r1, #D_CACHE_LINE_SIZE - 1
 #ifdef HARVARD_CACHE
 	mcrne	p15, 0, r1, c7, c14, 1		@ clean & invalidate D line
 #else
 	mcrne	p15, 0, r1, c7, c15, 1		@ clean & invalidate unified line
 #endif
-1:
 #ifdef CONFIG_DMA_CACHE_RWFO
-	ldr	r2, [r0]			@ read for ownership
-	str	r2, [r0]			@ write for ownership
+	ldrb	r2, [r0]			@ read for ownership
+	strb	r2, [r0]			@ write for ownership
+#endif
+	tst	r0, #D_CACHE_LINE_SIZE - 1
+	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
+#ifdef HARVARD_CACHE
+	mcrne	p15, 0, r0, c7, c10, 1		@ clean D line
+#else
+	mcrne	p15, 0, r0, c7, c11, 1		@ clean unified line
 #endif
+1:
 #ifdef HARVARD_CACHE
 	mcr	p15, 0, r0, c7, c6, 1		@ invalidate D line
 #else

arch/arm/mm/cache-v6.S v6_dma_inv_range: · line 242

Quoted 10 lines
arch/arm/mm/cache-v6.S +4 -0
@@ -229,6 +233,10 @@ v6_dma_inv_range:
 #endif
 	add	r0, r0, #D_CACHE_LINE_SIZE
 	cmp	r0, r1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrlo	r2, [r0]			@ read for ownership
+	strlo	r2, [r0]			@ write for ownership
+#endif
 	blo	1b
 	mov	r0, #0
 	mcr	p15, 0, r0, c7, c10, 4		@ drain write buffer

Just a minor nit...

Any reason for rearranging the order of 'start'/'end' parameters in the above changes? It's not clear that there is any performance benefit by rearranging those parameters and it obfuscates the changes, e.g. this is equivalent to your change above and also much clearer:

arch/arm/mm/cache-v6.S +12 -4
index 99fa688..28e4799 100644
@@ -203,6 +203,10 @@ ENTRY(v6_flush_kern_dcache_area)
  *	- end     - virtual end address of region
  */
 v6_dma_inv_range:
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrb	r2, [r0]			@ read for ownership
+	strb	r2, [r0]			@ write for ownership
+#endif
 	tst	r0, #D_CACHE_LINE_SIZE - 1
 	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
 #ifdef HARVARD_CACHE
@@ -211,6 +215,10 @@ v6_dma_inv_range:
 	mcrne	p15, 0, r0, c7, c11, 1		@ clean unified line
 #endif
 	tst	r1, #D_CACHE_LINE_SIZE - 1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrneb	r2, [r1, #-1]			@ read for ownership
+	strneb	r2, [r1, #-1]			@ write for ownership
+#endif
 	bic	r1, r1, #D_CACHE_LINE_SIZE - 1
 #ifdef HARVARD_CACHE
 	mcrne	p15, 0, r1, c7, c14, 1		@ clean & invalidate D line
@@ -218,10 +226,6 @@ v6_dma_inv_range:
 	mcrne	p15, 0, r1, c7, c15, 1		@ clean & invalidate unified line
 #endif
 1:
-#ifdef CONFIG_DMA_CACHE_RWFO
-	ldr	r2, [r0]			@ read for ownership
-	str	r2, [r0]			@ write for ownership
-#endif
 #ifdef HARVARD_CACHE
 	mcr	p15, 0, r0, c7, c6, 1		@ invalidate D line
 #else
@@ -229,6 +233,10 @@ v6_dma_inv_range:
 #endif
 	add	r0, r0, #D_CACHE_LINE_SIZE
 	cmp	r0, r1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrlo	r2, [r0]			@ read for ownership
+	strlo	r2, [r0]			@ write for ownership
+#endif
 	blo	1b
 	mov	r0, #0
 	mcr	p15, 0, r0, c7, c10, 4		@ drain write buffer

--
Regards,
George

Quoted 30 lines
arch/arm/mm/cache-v6.S +8 -4
@@ -263,12 +271,12 @@ v6_dma_clean_range:
  *	- end     - virtual end address of region
  */
 ENTRY(v6_dma_flush_range)
-	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
-1:
 #ifdef CONFIG_DMA_CACHE_RWFO
-	ldr	r2, [r0]			@ read for ownership
-	str	r2, [r0]			@ write for ownership
+	ldrb	r2, [r0]		@ read for ownership
+	strb	r2, [r0]		@ write for ownership
 #endif
+	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
+1:
 #ifdef HARVARD_CACHE
 	mcr	p15, 0, r0, c7, c14, 1		@ clean & invalidate D line
 #else
@@ -276,6 +284,10 @@ ENTRY(v6_dma_flush_range)
 #endif
 	add	r0, r0, #D_CACHE_LINE_SIZE
 	cmp	r0, r1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrlob	r2, [r0]			@ read for ownership
+	strlob	r2, [r0]			@ write for ownership
+#endif
 	blo	1b
 	mov	r0, #0
 	mcr	p15, 0, r0, c7, c10, 4		@ drain write buffer
Signature · 1 line
1.6.0.6
#19 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix

Quoted 79 lines · George G. Davis
Quoted 65 lines · Valentine Barshak

Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fix v6_dma_flush_range to prevent RWFO outside the buffer.

---
 arch/arm/mm/cache-v6.S |   40 ++++++++++++++++++++++++++--------------
 1 files changed, 26 insertions(+), 14 deletions(-)
arch/arm/mm/cache-v6.S +14 -10
index 99fa688..622f8a1 100644
@@ -203,25 +203,29 @@ ENTRY(v6_flush_kern_dcache_area)
  *   - end     - virtual end address of region
  */
 v6_dma_inv_range:
-     tst     r0, #D_CACHE_LINE_SIZE - 1
-     bic     r0, r0, #D_CACHE_LINE_SIZE - 1
-#ifdef HARVARD_CACHE
-     mcrne   p15, 0, r0, c7, c10, 1          @ clean D line
-#else
-     mcrne   p15, 0, r0, c7, c11, 1          @ clean unified line
-#endif
      tst     r1, #D_CACHE_LINE_SIZE - 1
+#ifdef CONFIG_DMA_CACHE_RWFO
+     ldrneb  r2, [r1, #-1]                   @ read for ownership
+     strneb  r2, [r1, #-1]                   @ write for ownership
+#endif
      bic     r1, r1, #D_CACHE_LINE_SIZE - 1
 #ifdef HARVARD_CACHE
      mcrne   p15, 0, r1, c7, c14, 1          @ clean & invalidate D line
 #else
      mcrne   p15, 0, r1, c7, c15, 1          @ clean & invalidate unified line
 #endif
-1:
 #ifdef CONFIG_DMA_CACHE_RWFO
-     ldr     r2, [r0]                        @ read for ownership
-     str     r2, [r0]                        @ write for ownership
+     ldrb    r2, [r0]                        @ read for ownership
+     strb    r2, [r0]                        @ write for ownership
+#endif
+     tst     r0, #D_CACHE_LINE_SIZE - 1
+     bic     r0, r0, #D_CACHE_LINE_SIZE - 1
+#ifdef HARVARD_CACHE
+     mcrne   p15, 0, r0, c7, c10, 1          @ clean D line
+#else
+     mcrne   p15, 0, r0, c7, c11, 1          @ clean unified line
 #endif
+1:
 #ifdef HARVARD_CACHE
      mcr     p15, 0, r0, c7, c6, 1           @ invalidate D line
 #else

arch/arm/mm/cache-v6.S v6_dma_inv_range: · line 242

Quoted 10 lines
arch/arm/mm/cache-v6.S +4 -0
@@ -229,6 +233,10 @@ v6_dma_inv_range:
 #endif
      add     r0, r0, #D_CACHE_LINE_SIZE
      cmp     r0, r1
+#ifdef CONFIG_DMA_CACHE_RWFO
+     ldrlo   r2, [r0]                        @ read for ownership
+     strlo   r2, [r0]                        @ write for ownership
+#endif
      blo     1b
      mov     r0, #0
      mcr     p15, 0, r0, c7, c10, 4          @ drain write buffer

Just a minor nit...

Quoted 4 lines

Any reason for rearranging the order of 'start'/'end' parameters in the above changes? It's not clear that there is any performance benefit by rearranging those parameters and it obfuscates the changes, e.g. this is equivalent to your change above and also much clearer:

I've been wondering about this as well. But I was too lazy to rewrite it and see whether there is a performance benefit in the original patch. I agree that yours looks cleaner.

Signature · 1 line
Catalin
#20 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix

Quoted 82 lines · Catalin Marinas
Quoted 80 lines · George G. Davis
Quoted 66 lines · Valentine Barshak

Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fix v6_dma_flush_range to prevent RWFO outside the buffer.

---
 arch/arm/mm/cache-v6.S |   40 ++++++++++++++++++++++++++--------------
 1 files changed, 26 insertions(+), 14 deletions(-)
arch/arm/mm/cache-v6.S +14 -10
index 99fa688..622f8a1 100644
@@ -203,25 +203,29 @@ ENTRY(v6_flush_kern_dcache_area)
  *   - end     - virtual end address of region
  */
 v6_dma_inv_range:
-     tst     r0, #D_CACHE_LINE_SIZE - 1
-     bic     r0, r0, #D_CACHE_LINE_SIZE - 1
-#ifdef HARVARD_CACHE
-     mcrne   p15, 0, r0, c7, c10, 1          @ clean D line
-#else
-     mcrne   p15, 0, r0, c7, c11, 1          @ clean unified line
-#endif
      tst     r1, #D_CACHE_LINE_SIZE - 1
+#ifdef CONFIG_DMA_CACHE_RWFO
+     ldrneb  r2, [r1, #-1]                   @ read for ownership
+     strneb  r2, [r1, #-1]                   @ write for ownership
+#endif
      bic     r1, r1, #D_CACHE_LINE_SIZE - 1
 #ifdef HARVARD_CACHE
      mcrne   p15, 0, r1, c7, c14, 1          @ clean & invalidate D line
 #else
      mcrne   p15, 0, r1, c7, c15, 1          @ clean & invalidate unified line
 #endif
-1:
 #ifdef CONFIG_DMA_CACHE_RWFO
-     ldr     r2, [r0]                        @ read for ownership
-     str     r2, [r0]                        @ write for ownership
+     ldrb    r2, [r0]                        @ read for ownership
+     strb    r2, [r0]                        @ write for ownership
+#endif
+     tst     r0, #D_CACHE_LINE_SIZE - 1
+     bic     r0, r0, #D_CACHE_LINE_SIZE - 1
+#ifdef HARVARD_CACHE
+     mcrne   p15, 0, r0, c7, c10, 1          @ clean D line
+#else
+     mcrne   p15, 0, r0, c7, c11, 1          @ clean unified line
 #endif
+1:
 #ifdef HARVARD_CACHE
      mcr     p15, 0, r0, c7, c6, 1           @ invalidate D line
 #else

arch/arm/mm/cache-v6.S v6_dma_inv_range: · line 242

Quoted 10 lines
arch/arm/mm/cache-v6.S +4 -0
@@ -229,6 +233,10 @@ v6_dma_inv_range:
 #endif
      add     r0, r0, #D_CACHE_LINE_SIZE
      cmp     r0, r1
+#ifdef CONFIG_DMA_CACHE_RWFO
+     ldrlo   r2, [r0]                        @ read for ownership
+     strlo   r2, [r0]                        @ write for ownership
+#endif
      blo     1b
      mov     r0, #0
      mcr     p15, 0, r0, c7, c10, 4          @ drain write buffer

Just a minor nit...

Quoted 7 lines
Quoted 5 lines

Any reason for rearranging the order of 'start'/'end' parameters in the above changes? It's not clear that there is any performance benefit by rearranging those parameters and it obfuscates the changes, e.g. this is equivalent to your change above and also much clearer:

It probably obfuscates the change, but I thought it made the code a bit cleaner and easier to read.
So that first the end (r1) is flushed and then we do all the operations needed on the first cache line (r0) in a row.
I'll resubmit the change and add "Cc: stable at kernel.org" to the header.

Thanks,
Val.

Quoted 5 lines

I've been wondering about this as well. But I was too lazy to rewrite it and see whether there is a performance benefit in the original patch. I agree that yours looks cleaner.

#21 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix

Quoted 85 lines · Valentine Barshak
Quoted 79 lines · Catalin Marinas
Quoted 78 lines · George G. Davis
Quoted 65 lines · Valentine Barshak

Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fix v6_dma_flush_range to prevent RWFO outside the buffer.

---
arch/arm/mm/cache-v6.S | 40 ++++++++++++++++++++++++++--------------

1 files changed, 26 insertions(+), 14 deletions(-)
arch/arm/mm/cache-v6.S +0 -0
index 99fa688..622f8a1 100644
@@ -203,25 +203,29 @@ ENTRY(v6_flush_kern_dcache_area)
 *   - end     - virtual end address of region
 */

v6_dma_inv_range:

arch/arm/mm/cache-v6.S +4 -7
-     tst     r0, #D_CACHE_LINE_SIZE - 1
-     bic     r0, r0, #D_CACHE_LINE_SIZE - 1
-#ifdef HARVARD_CACHE
-     mcrne   p15, 0, r0, c7, c10, 1          @ clean D line
-#else
-     mcrne   p15, 0, r0, c7, c11, 1          @ clean unified line
-#endif
     tst     r1, #D_CACHE_LINE_SIZE - 1
+#ifdef CONFIG_DMA_CACHE_RWFO
+     ldrneb  r2, [r1, #-1]                   @ read for ownership
+     strneb  r2, [r1, #-1]                   @ write for ownership
+#endif
     bic     r1, r1, #D_CACHE_LINE_SIZE - 1

#ifdef HARVARD_CACHE
mcrne p15, 0, r1, c7, c14, 1 @ clean & invalidate D line
#else
mcrne p15, 0, r1, c7, c15, 1 @ clean & invalidate unified line
#endif
-1:
#ifdef CONFIG_DMA_CACHE_RWFO

arch/arm/mm/cache-v6.S +9 -2
-     ldr     r2, [r0]                        @ read for ownership
-     str     r2, [r0]                        @ write for ownership
+     ldrb    r2, [r0]                        @ read for ownership
+     strb    r2, [r0]                        @ write for ownership
+#endif
+     tst     r0, #D_CACHE_LINE_SIZE - 1
+     bic     r0, r0, #D_CACHE_LINE_SIZE - 1
+#ifdef HARVARD_CACHE
+     mcrne   p15, 0, r0, c7, c10, 1          @ clean D line
+#else
+     mcrne   p15, 0, r0, c7, c11, 1          @ clean unified line

#endif
+1:
#ifdef HARVARD_CACHE
mcr p15, 0, r0, c7, c6, 1 @ invalidate D line
#else

arch/arm/mm/cache-v6.S v6_dma_inv_range: · line 242

Quoted 10 lines
arch/arm/mm/cache-v6.S +4 -0
@@ -229,6 +233,10 @@ v6_dma_inv_range:
 #endif
     add     r0, r0, #D_CACHE_LINE_SIZE
     cmp     r0, r1
+#ifdef CONFIG_DMA_CACHE_RWFO
+     ldrlo   r2, [r0]                        @ read for ownership
+     strlo   r2, [r0]                        @ write for ownership
+#endif
     blo     1b
     mov     r0, #0
     mcr     p15, 0, r0, c7, c10, 4          @ drain write buffer

Just a minor nit...

Quoted 5 lines
Quoted 4 lines

Any reason for rearranging the order of 'start'/'end' parameters in the above changes? It's not clear that there is any performance benefit by rearranging those parameters and it obfuscates the changes, e.g. this is equivalent to your change above and also much clearer:

Quoted 4 lines

It probably obfuscates the change, but I thought it made the code a bit cleaner and easier to read.
So that first the end (r1) is flushed and then we do all the operations needed on the first cache line (r0) in a row.

OK, I see your rationale now for moving start (r0) alignment and RWFO fixups just prior to the "clean & invalidate" loop to make the code clearer to the casual reader that RWFO for the first line is done outside the loop and conditionally for remaining lines at the end of the loop. That's admittedly subtle but I'd still prefer to see small incremental changes for faster review (or more fully describe those subtle changes in the commit header). Alas, without comments in the code or commit header, the subtleness of those RWFOs won't be any clearer regardless of their spatial locality. You can always submit another patch to rearrange the code for better flow/readability later.

Quoted 1 line

I'll resubmit the change and add "Cc: stable at kernel.org" to the header.

Thanks!

--
Regards,
George

Quoted 7 lines

Thanks,
Val.

Quoted 4 lines

I've been wondering about this as well. But I was too lazy to rewrite it and see whether there is a performance benefit in the original patch. I agree that yours looks cleaner.

#22 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range and v6_dma_flush_range RWFO fix

In reply to Catalin Marinas

Resending this one. Somehow it got lost and never appeared on the list.

Updated according to the comments to avoid r/w outside the buffer and used byte r/w for the possible unaligned data. Seems to work fine.

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.
While at it, fix v6_dma_flush_range to prevent RWFO outside the buffer.

arch/arm/mm/cache-v6.S +26 -14
index 99fa688..622f8a1 100644
@@ -203,25 +203,29 @@ ENTRY(v6_flush_kern_dcache_area)
  *	- end     - virtual end address of region
  */
 v6_dma_inv_range:
-	tst	r0, #D_CACHE_LINE_SIZE - 1
-	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
-#ifdef HARVARD_CACHE
-	mcrne	p15, 0, r0, c7, c10, 1		@ clean D line
-#else
-	mcrne	p15, 0, r0, c7, c11, 1		@ clean unified line
-#endif
 	tst	r1, #D_CACHE_LINE_SIZE - 1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrneb	r2, [r1, #-1]			@ read for ownership
+	strneb	r2, [r1, #-1]			@ write for ownership
+#endif
 	bic	r1, r1, #D_CACHE_LINE_SIZE - 1
 #ifdef HARVARD_CACHE
 	mcrne	p15, 0, r1, c7, c14, 1		@ clean & invalidate D line
 #else
 	mcrne	p15, 0, r1, c7, c15, 1		@ clean & invalidate unified line
 #endif
-1:
 #ifdef CONFIG_DMA_CACHE_RWFO
-	ldr	r2, [r0]			@ read for ownership
-	str	r2, [r0]			@ write for ownership
+	ldrb	r2, [r0]			@ read for ownership
+	strb	r2, [r0]			@ write for ownership
+#endif
+	tst	r0, #D_CACHE_LINE_SIZE - 1
+	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
+#ifdef HARVARD_CACHE
+	mcrne	p15, 0, r0, c7, c10, 1		@ clean D line
+#else
+	mcrne	p15, 0, r0, c7, c11, 1		@ clean unified line
 #endif
+1:
 #ifdef HARVARD_CACHE
 	mcr	p15, 0, r0, c7, c6, 1		@ invalidate D line
 #else
@@ -229,6 +233,10 @@ v6_dma_inv_range:
 #endif
 	add	r0, r0, #D_CACHE_LINE_SIZE
 	cmp	r0, r1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrlo	r2, [r0]			@ read for ownership
+	strlo	r2, [r0]			@ write for ownership
+#endif
 	blo	1b
 	mov	r0, #0
 	mcr	p15, 0, r0, c7, c10, 4		@ drain write buffer
@@ -263,12 +271,12 @@ v6_dma_clean_range:
  *	- end     - virtual end address of region
  */
 ENTRY(v6_dma_flush_range)
-	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
-1:
 #ifdef CONFIG_DMA_CACHE_RWFO
-	ldr	r2, [r0]			@ read for ownership
-	str	r2, [r0]			@ write for ownership
+	ldrb	r2, [r0]		@ read for ownership
+	strb	r2, [r0]		@ write for ownership
 #endif
+	bic	r0, r0, #D_CACHE_LINE_SIZE - 1
+1:
 #ifdef HARVARD_CACHE
 	mcr	p15, 0, r0, c7, c14, 1		@ clean & invalidate D line
 #else
@@ -276,6 +284,10 @@ ENTRY(v6_dma_flush_range)
 #endif
 	add	r0, r0, #D_CACHE_LINE_SIZE
 	cmp	r0, r1
+#ifdef CONFIG_DMA_CACHE_RWFO
+	ldrlob	r2, [r0]			@ read for ownership
+	strlob	r2, [r0]			@ write for ownership
+#endif
 	blo	1b
 	mov	r0, #0
 	mcr	p15, 0, r0, c7, c10, 4		@ drain write buffer
#23 lore raw

PATCH 1/1 ARM: V6 MPCore v6_dma_inv_range RWFO fix

In reply to Russell King - ARM Linux
Quoted 12 lines · Russell King - ARM Linux
Quoted 9 lines · Valentine Barshak

Cache ownership must be acqired by reading/writing data from the cache line to make cache operation have the desired effect on the SMP MPCore CPU. However, the ownership is never aquired in the v6_dma_inv_range function when cleaning the first line and flushing the last one, in case the address is not aligned to D_CACHE_LINE_SIZE boundary.
Fix this by reading/writing data if needed, before performing cache operations.

Quoted 4 lines

You should do this on the data _inside_ the requested buffer. We don't know if the overlapping cache line shares itself with some atomic variable, and doing a read-write on it could undo other updates to it.

The same problem could hit us in v6_dma_flush_range as well, when we do r/w after bic.

Thanks,
Val.