Skip to contentSkip to conversation

arc: Flush and invalidate caches on start

PATCH In review Alexey Brodkin 2 participants 4 messages Aug 31, 2017 – Nov 15, 2017 Synopsys ARC architecture

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 arc: Flush and invalidate caches on start

This is useful to make sure no stale data exists in caches after bootloaders. The worst thing could be some lines of cache were locked in a bootloader for example during DDR recalibration and never unlocked. This may lead to really unpredictable issues later down the line.

Signed-off-by Alexey Brodkin <abrodkin at synopsys.com>
---
 arch/arc/kernel/head.S | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
arch/arc/kernel/head.S +16 -0
index 8b90d25a15cc..04e28b664183 100644
@@ -34,6 +34,10 @@
 #endif
 	sr	r5, [ARC_REG_IC_CTRL]
 
+	; Invalidate entire I$
+	mov	r5, 1
+	sr	r5, [ARC_REG_IC_IVIC]
+
 1:
 	lr	r5, [ARC_REG_DC_BCR]
 	breq    r5, 0, 1f		; D$ doesn't exist
@@ -46,6 +50,18 @@
 #endif
 	sr	r5, [ARC_REG_DC_CTRL]
 
+	; Flush entire D$
+	mov	r5, 1
+	sr	r5, [ARC_REG_DC_FLSH]
+	; Wait for flush operation to complete
+1:
+	lr	r5, [ARC_REG_DC_CTRL]
+	bbit1	r5, DC_CTRL_FLUSH_STATUS, 1b
+
+	; Invalidate entire D$
+	mov	r5, 1
+	sr	r5, [ARC_REG_DC_IVDC]
+
 1:
 .endm
 
Signature · 1 line
2.11.0
#1 lore raw
Quoted 31 lines · Alexey Brodkin

This is useful to make sure no stale data exists in caches after bootloaders. The worst thing could be some lines of cache were locked in a bootloader for example during DDR recalibration and never unlocked. This may lead to really unpredictable issues later down the line.

Signed-off-by Alexey Brodkin <abrodkin at synopsys.com>
---
  arch/arc/kernel/head.S | 16 ++++++++++++++++
  1 file changed, 16 insertions(+)
arch/arc/kernel/head.S +6 -0
index 8b90d25a15cc..04e28b664183 100644
@@ -34,6 +34,10 @@
  #endif
  	sr	r5, [ARC_REG_IC_CTRL]
 
+	; Invalidate entire I$
+	mov	r5, 1
+	sr	r5, [ARC_REG_IC_IVIC]
+
  1:
  	lr	r5, [ARC_REG_DC_BCR]
  	breq    r5, 0, 1f		; D$ doesn't exist
@@ -46,6 +50,18 @@
  #endif
  	sr	r5, [ARC_REG_DC_CTRL]
 
+	; Flush entire D$
+	mov	r5, 1

arch/arc/kernel/head.S line 64

Quoted 10 lines
arch/arc/kernel/head.S +10 -0
+	sr	r5, [ARC_REG_DC_FLSH]
+	; Wait for flush operation to complete
+1:
+	lr	r5, [ARC_REG_DC_CTRL]
+	bbit1	r5, DC_CTRL_FLUSH_STATUS, 1b
+
+	; Invalidate entire D$
+	mov	r5, 1
+	sr	r5, [ARC_REG_DC_IVDC]
+

AFAIK uboot already flushes the caches before handing control over to kernel - so why do we need it here.
If uboot is locking lines, it needs to fix that and not penalize the general case with or w/o uboot !

-Vineet

#2 lore raw

Hi Vineet,

Quoted 40 lines · Vineet Gupta
Quoted 39 lines · Alexey Brodkin

This is useful to make sure no stale data exists in caches after bootloaders. The worst thing could be some lines of cache were locked in a bootloader for example during DDR recalibration and never unlocked. This may lead to really unpredictable issues later down the line.

Signed-off-by Alexey Brodkin <abrodkin at synopsys.com>

---
? arch/arc/kernel/head.S | 16 ++++++++++++++++
? 1 file changed, 16 insertions(+)

arch/arc/kernel/head.S +0 -0
index 8b90d25a15cc..04e28b664183 100644
@@ -34,6 +34,10 @@

? #endif
?? sr r5, [ARC_REG_IC_CTRL]
??

arch/arc/kernel/head.S +4 -0
+	; Invalidate entire I$
+	mov	r5, 1
+	sr	r5, [ARC_REG_IC_IVIC]
+

? 1:
?? lr r5, [ARC_REG_DC_BCR]
?? breq????r5, 0, 1f ; D$ doesn't exist

arch/arc/kernel/head.S +0 -0
@@ -46,6 +50,18 @@

? #endif
?? sr r5, [ARC_REG_DC_CTRL]
??

arch/arc/kernel/head.S +9 -0
+	; Flush entire D$
+	mov	r5, 1
+	sr	r5, [ARC_REG_DC_FLSH]
+	; Wait for flush operation to complete
+1:
+	lr	r5, [ARC_REG_DC_CTRL]
+	bbit1	r5, DC_CTRL_FLUSH_STATUS, 1b
+
+	; Invalidate entire D$
Quoted 8 lines

arch/arc/kernel/head.S line 64

Quoted 3 lines
arch/arc/kernel/head.S +3 -0
+	mov	r5, 1
+	sr	r5, [ARC_REG_DC_IVDC]
+

AFAIK uboot already flushes the caches before handing control over to kernel - so? why do we need it here.
If uboot is locking lines, it needs to fix that and not penalize the general case? with or w/o uboot !

U-Boot indeed flushes caches.. but doesn't invalidate them! And only invalidation unlocks locked lines.

That indeed should be added in U-Boot but I'd say above stuff doesn't influence a lot code size or execution time while makes system more fool-proof.

-Alexey

#3 lore raw
Quoted 35 lines · Alexey Brodkin

Hi Vineet,

Quoted 32 lines · Vineet Gupta
Quoted 30 lines · Alexey Brodkin

This is useful to make sure no stale data exists in caches after bootloaders. The worst thing could be some lines of cache were locked in a bootloader for example during DDR recalibration and never unlocked. This may lead to really unpredictable issues later down the line.

Signed-off-by Alexey Brodkin <abrodkin at synopsys.com>

---
? arch/arc/kernel/head.S | 16 ++++++++++++++++
? 1 file changed, 16 insertions(+)

arch/arc/kernel/head.S +0 -0
index 8b90d25a15cc..04e28b664183 100644
@@ -34,6 +34,10 @@

? #endif
?? sr r5, [ARC_REG_IC_CTRL]
??

arch/arc/kernel/head.S +4 -0
+	; Invalidate entire I$
+	mov	r5, 1
+	sr	r5, [ARC_REG_IC_IVIC]
+

? 1:
?? lr r5, [ARC_REG_DC_BCR]
?? breq????r5, 0, 1f ; D$ doesn't exist

arch/arc/kernel/head.S +0 -0
@@ -46,6 +50,18 @@

? #endif
?? sr r5, [ARC_REG_DC_CTRL]

Quoted 8 lines
Quoted 8 lines

arch/arc/kernel/head.S line 59

Quoted 8 lines

??

arch/arc/kernel/head.S +7 -0
+	; Flush entire D$
+	mov	r5, 1
+	sr	r5, [ARC_REG_DC_FLSH]
+	; Wait for flush operation to complete
+1:
+	lr	r5, [ARC_REG_DC_CTRL]
+	bbit1	r5, DC_CTRL_FLUSH_STATUS, 1b

Looking more at this stuff I think that on that early stage there's no point in flushing of D$, moreover it could be dangerous as if there's some garbage in there it will contaminate DDR with good data.

Quoted 10 lines
Quoted 9 lines

arch/arc/kernel/head.S line 64

Quoted 4 lines
arch/arc/kernel/head.S +4 -0
+	; Invalidate entire D$
+	mov	r5, 1
+	sr	r5, [ARC_REG_DC_IVDC]
+

AFAIK uboot already flushes the caches before handing control over to kernel - so? why do we need it here.
If uboot is locking lines, it needs to fix that and not penalize the general case? with or w/o uboot !

Quoted 5 lines

U-Boot indeed flushes caches.. but doesn't invalidate them! And only invalidation unlocks locked lines.

That indeed should be added in U-Boot but I'd say above stuff doesn't influence a lot code size or execution time while makes system more fool-proof.

As for invalidation that IMHO is really important especially for I$.
See even if we do proper invalidation of both caches in say U-Boot but keep caches enabled at least the last I$ line that contained last "jump" instruction will left in the cache. And now if we're unlucky enough we may execute code that overlaps location of that last "jump" and instead of expected instruction freshly loaded from DDR we'll execute garbage left-over.

Note this is not just a theory, we hit that problem trying to execute U-Boot from u-Boot (not kidding - we do self-upgrade that way).

Similar idea could be valid for D$ even though not very likely but still... Given code in U-boot writen in C even in between dcache_flush() and icache_invalidate() might happen some data accesses hidden in implementations of those functions. In fact currently we do checks if SLC exist in those functions and that check is done via polling a global var, so for sure some LD instructions are executed there.

That said if my explanation now makes more sense I'd be willing to send a v2 with invalidation of all caches (I$, D$ and SLC).

-Alexey