ARM: cache-v7: Disable preemption when reading CCSIDR
Subsystems (2)
PATCH 1/1 ARM: cache-v7: Disable preemption when reading CCSIDR
armv7's flush_cache_all() flushes caches via set/way. To determine the cache attributes (line size, number of sets, etc.) the assembly first writes the CSSELR register to select a cache level and then reads the CCSIDR register. The CSSELR register is banked per-cpu and is used to determine which cache level CCSIDR reads. If the task is migrated between when the CSSELR is written and the CCSIDR is read the CCSIDR value may be for an unexpected cache level (for example L1 instead of L2) and incorrect cache flushing could occur.
Disable preemption across the write and read so that the correct cache attributes are read and used for the cache flushing routine. This fixes a problem we see in scm_call() when flush_cache_all() is called from preemptible context and sometimes the L2 cache is not properly flushed out.
---
Should we move get_thread_info into assembler.h? It seems odd to include entry-header.S but I saw that vfp was doing the same.
arch/arm/mm/cache-v7.S | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-)
arch/arm/mm/cache-v7.S +11 -0
@@ -16,6 +16,7 @@ #include <asm/unwind.h> #include "proc-macros.S" +#include "../kernel/entry-header.S" /* * v7_flush_icache_all()
@@ -54,9 +55,19 @@ loop1: and r1, r1, #7 @ mask of the bits for current cache only cmp r1, #2 @ see what cache we have at this level blt skip @ skip if no cache, or just i-cache +#ifdef CONFIG_PREEMPT + get_thread_info r9 + ldr r11, [r9, #TI_PREEMPT] @ get preempt count + add r11, r11, #1 @ increment it + str r11, [r9, #TI_PREEMPT] +#endif mcr p15, 2, r10, c0, c0, 0 @ select current cache level in cssr isb @ isb to sych the new cssr&csidr mrc p15, 1, r1, c0, c0, 0 @ read the new csidr +#ifdef CONFIG_PREEMPT + sub r11, r11, #1 @ decrement preempt count + str r11, [r9, #TI_PREEMPT] +#endif and r2, r1, #7 @ extract the length of the cache lines add r2, r2, #4 @ add 4 (line length offset) ldr r4, =0x3ff
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Quoted 10 lines · Stephen Boyd
armv7's
flush_cache_all()flushes caches via set/way. To determine the cache attributes (line size, number of sets, etc.) the assembly first writes the CSSELR register to select a cache level and then reads the CCSIDR register. The CSSELR register is banked per-cpu and is used to determine which cache level CCSIDR reads. If the task is migrated between when the CSSELR is written and the CCSIDR is read the CCSIDR value may be for an unexpected cache level (for example L1 instead of L2) and incorrect cache flushing could occur.
Quoted 5 lines
Disable preemption across the write and read so that the correct cache attributes are read and used for the cache flushing routine. This fixes a problem we see in
scm_call()whenflush_cache_all()is called from preemptible context and sometimes the L2 cache is not properly flushed out.
This isn't going to work for two reasons:
(1) (and the VFP code suffers from this) after we re-enable preemption, we really should check for a pending preemption event in every case.
(2) v7_flush_dcache_all() is called from __v7_setup() using a very small private stack. This doesn't have a thread info structure at the bottom.
So, if we need to disable preemption here, we need to find a different solution to it.
Quoted 2 lines
Should we move
get_thread_infointoassembler.h? It seems odd to includeentry-header.Sbut I saw that vfp was doing the same.
Probably yes, and probably also have preempt_disable and preempt_enable assembler macros. That's going to get rather icky if we have to explicitly call the scheduler though (to solve (1)).
Quoted 7 lines · Russell King - ARM Linux
Quoted 2 lines · Stephen Boyd
Should we move
get_thread_infointoassembler.h? It seems odd to includeentry-header.Sbut I saw that vfp was doing the same.Probably yes, and probably also have
preempt_disableandpreempt_enableassembler macros. That's going to get rather icky if we have to explicitly call the scheduler though (to solve (1)).
What about a pair of helpers written in C instead?
v7_flush_dcache_all() could be renamed, and a wrapper function called v7_flush_dcache_all() would call the preemption disable helper, call the former v7_flush_dcache_all code, then call the preemption enable helper.
Then __v7_setup() could still call the core cache flush code without issues.
Nicolas
Quoted 7 lines · Nicolas Pitre
On Thu, 2 Feb 2012, Russell King - ARM Linux wrote
Quoted 6 lines · Stephen Boyd
Quoted 2 lines
Should we move
get_thread_infointoassembler.h? It seems odd to includeentry-header.Sbut I saw that vfp was doing the same.Probably yes, and probably also have
preempt_disableandpreempt_enableassembler macros. That's going to get rather icky if we have to explicitly call the scheduler though (to solve (1)).
Quoted 8 lines
What about a pair of helpers written in C instead?
v7_flush_dcache_all()could be renamed, and a wrapper function calledv7_flush_dcache_all()would call the preemption disable helper, call the formerv7_flush_dcache_allcode, then call the preemption enable helper.Then
__v7_setup()could still call the core cache flush code without issues.
I tried to put the preemption disable/enable right around the place where it was needed. With this approach we would disable preemption during the entire cache flush. I'm not sure if we want to make this function worse for performance, do we? It certainly sounds easier than writing all the preempt macros in assembly though.
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Quoted 14 lines · Stephen Boyd
Quoted 13 lines · Nicolas Pitre
On Thu, 2 Feb 2012, Russell King - ARM Linux wrote
Quoted 6 lines · Stephen Boyd
Quoted 2 lines
Should we move
get_thread_infointoassembler.h? It seems odd to includeentry-header.Sbut I saw that vfp was doing the same.Probably yes, and probably also have
preempt_disableandpreempt_enableassembler macros. That's going to get rather icky if we have to explicitly call the scheduler though (to solve (1)).What about a pair of helpers written in C instead?
v7_flush_dcache_all()could be renamed, and a wrapper function calledv7_flush_dcache_all()would call the preemption disable helper, call the formerv7_flush_dcache_allcode, then call the preemption enable helper.
Quoted 8 lines
Quoted 2 lines
Then
__v7_setup()could still call the core cache flush code without issues.I tried to put the preemption disable/enable right around the place where it was needed. With this approach we would disable preemption during the entire cache flush. I'm not sure if we want to make this function worse for performance, do we? It certainly sounds easier than writing all the preempt macros in assembly though.
Err, why do you think it's a big task?
preempt disable is a case of incrementing the thread preempt count, while preempt enable is a case of decrementing it, testing for zero, if zero, then checking whether TIF_NEED_RESCHED is set and calling a function.
If that's too much, then the simple method in assembly to quickly disable preemption over a very few set of instructions is using mrs/msr and cpsid i. That'll be far cheaper than fiddling about with preempt counters or messing about with veneers in C code.
Quoted 24 lines · Russell King - ARM Linux
Quoted 14 lines · Stephen Boyd
Quoted 13 lines · Nicolas Pitre
On Thu, 2 Feb 2012, Russell King - ARM Linux wrote
Quoted 6 lines · Stephen Boyd
Quoted 2 lines
Should we move
get_thread_infointoassembler.h? It seems odd to includeentry-header.Sbut I saw that vfp was doing the same.Probably yes, and probably also have
preempt_disableandpreempt_enableassembler macros. That's going to get rather icky if we have to explicitly call the scheduler though (to solve (1)).What about a pair of helpers written in C instead?
v7_flush_dcache_all()could be renamed, and a wrapper function calledv7_flush_dcache_all()would call the preemption disable helper, call the formerv7_flush_dcache_allcode, then call the preemption enable helper.Quoted 7 lines
Quoted 2 lines
Then
__v7_setup()could still call the core cache flush code without issues.I tried to put the preemption disable/enable right around the place where it was needed. With this approach we would disable preemption during the entire cache flush. I'm not sure if we want to make this function worse for performance, do we? It certainly sounds easier than writing all the preempt macros in assembly though.
Err, why do you think it's a big task?
Quoted 8 lines
preempt disable is a case of incrementing the thread preempt count, while preempt enable is a case of decrementing it, testing for zero, if zero, then checking whether
TIF_NEED_RESCHEDis set and calling a function.If that's too much, then the simple method in assembly to quickly disable preemption over a very few set of instructions is using mrs/msr and cpsid i. That'll be far cheaper than fiddling about with preempt counters or messing about with veneers in C code.
I'll try the macros. So far it isn't bad, just the __v7_setup to resolve.
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Quoted 29 lines · Stephen Boyd
Quoted 28 lines · Russell King - ARM Linux
Quoted 14 lines · Stephen Boyd
Quoted 13 lines · Nicolas Pitre
On Thu, 2 Feb 2012, Russell King - ARM Linux wrote
Quoted 6 lines · Stephen Boyd
Quoted 2 lines
Should we move
get_thread_infointoassembler.h? It seems odd to includeentry-header.Sbut I saw that vfp was doing the same.Probably yes, and probably also have
preempt_disableandpreempt_enableassembler macros. That's going to get rather icky if we have to explicitly call the scheduler though (to solve (1)).What about a pair of helpers written in C instead?
v7_flush_dcache_all()could be renamed, and a wrapper function calledv7_flush_dcache_all()would call the preemption disable helper, call the formerv7_flush_dcache_allcode, then call the preemption enable helper.Quoted 7 lines
Quoted 2 lines
Then
__v7_setup()could still call the core cache flush code without issues.I tried to put the preemption disable/enable right around the place where it was needed. With this approach we would disable preemption during the entire cache flush. I'm not sure if we want to make this function worse for performance, do we? It certainly sounds easier than writing all the preempt macros in assembly though.
Err, why do you think it's a big task?
preempt disable is a case of incrementing the thread preempt count, while preempt enable is a case of decrementing it, testing for zero, if zero, then checking whether
TIF_NEED_RESCHEDis set and calling a function.
Quoted 6 lines
Quoted 4 lines
If that's too much, then the simple method in assembly to quickly disable preemption over a very few set of instructions is using mrs/msr and cpsid i. That'll be far cheaper than fiddling about with preempt counters or messing about with veneers in C code.
I'll try the macros. So far it isn't bad, just the
__v7_setupto resolve.
If you simply disable/restore IRQs around the critical region then you don't have to worry about __v7_setup. Plus this will allow for v7_flush_dcache_all to still be callable from atomic context.
Nicolas
PATCH 1/1 ARM: cache-v7: Disable preemption when reading CCSIDR
armv7's flush_cache_all() flushes caches via set/way. To determine the cache attributes (line size, number of sets, etc.) the assembly first writes the CSSELR register to select a cache level and then reads the CCSIDR register. The CSSELR register is banked per-cpu and is used to determine which cache level CCSIDR reads. If the task is migrated between when the CSSELR is written and the CCSIDR is read the CCSIDR value may be for an unexpected cache level (for example L1 instead of L2) and incorrect cache flushing could occur.
Disable interrupts across the write and read so that the correct cache attributes are read and used for the cache flushing routine. We disable interrupts instead of disabling preemption because the critical section is only 3 instructions and we want to call v7_dcache_flush_all from __v7_setup which doesn't have a full kernel stack with a struct thread_info.
This fixes a problem we see in scm_call() when flush_cache_all() is called from preemptible context and sometimes the L2 cache is not properly flushed out.
---
Quoted 10 lines · Nicolas Pitre
Quoted 6 lines
Quoted 4 lines
If that's too much, then the simple method in assembly to quickly disable preemption over a very few set of instructions is using mrs/msr and cpsid i. That'll be far cheaper than fiddling about with preempt counters or messing about with veneers in C code.
I'll try the macros. So far it isn't bad, just the
__v7_setupto resolve.If you simply disable/restore IRQs around the critical region then you don't have to worry about
__v7_setup. Plus this will allow forv7_flush_dcache_allto still be callable from atomic context.
Ok. Here's a patch. I still need to test it. I'll send another patch series to cleanup the get_thread_info stuff (there's two of them?).
arch/arm/mm/cache-v7.S | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)
arch/arm/mm/cache-v7.S +6 -0
@@ -54,9 +54,15 @@ loop1: and r1, r1, #7 @ mask of the bits for current cache only cmp r1, #2 @ see what cache we have at this level blt skip @ skip if no cache, or just i-cache +#ifdef CONFIG_PREEMPT + save_and_disable_irqs r9 @ make cssr&csidr read atomic +#endif mcr p15, 2, r10, c0, c0, 0 @ select current cache level in cssr isb @ isb to sych the new cssr&csidr mrc p15, 1, r1, c0, c0, 0 @ read the new csidr +#ifdef CONFIG_PREEMPT + restore_irqs r9 +#endif and r2, r1, #7 @ extract the length of the cache lines add r2, r2, #4 @ add 4 (line length offset) ldr r4, =0x3ff
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Quoted 18 lines · Stephen Boyd
Quoted 3 lines · Nicolas Pitre
If you simply disable/restore IRQs around the critical region then you don't have to worry about
__v7_setup. Plus this will allow forv7_flush_dcache_allto still be callable from atomic context.Ok. Here's a patch. I still need to test it. I'll send another patch series to cleanup the
get_thread_infostuff (there's two of them?).arch/arm/mm/cache-v7.S | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)arch/arm/mm/cache-v7.S +0
-0@@ -54,9 +54,15 @@ loop1: and r1, r1, #7 @ mask of the bits for current cache only cmp r1, #2 @ see what cache we have at this level
arch/arm/mm/cache-v7.S loop1: · line 65
Quoted 10 lines
arch/arm/mm/cache-v7.S +6
-0blt skip @ skip if no cache, or just i-cache +#ifdef CONFIG_PREEMPT + save_and_disable_irqs r9 @ make cssr&csidr read atomic +#endif mcr p15, 2, r10, c0, c0, 0 @ select current cache level in cssr isb @ isb to sych the new cssr&csidr mrc p15, 1, r1, c0, c0, 0 @ read the new csidr +#ifdef CONFIG_PREEMPT + restore_irqs r9 +#endif
I'd suggest using restore_irqs_notrace instead. The IRQ-off period is so small that there is no point tracing it.
Withthat change:
Nicolas
Quoted 22 lines · Nicolas Pitre
Quoted 21 lines · Stephen Boyd
Quoted 3 lines · Nicolas Pitre
If you simply disable/restore IRQs around the critical region then you don't have to worry about
__v7_setup. Plus this will allow forv7_flush_dcache_allto still be callable from atomic context.Ok. Here's a patch. I still need to test it. I'll send another patch series to cleanup the
get_thread_infostuff (there's two of them?).arch/arm/mm/cache-v7.S | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)arch/arm/mm/cache-v7.S +3
-0@@ -54,9 +54,15 @@ loop1: and r1, r1, #7 @ mask of the bits for current cache only cmp r1, #2 @ see what cache we have at this level blt skip @ skip if no cache, or just i-cache +#ifdef CONFIG_PREEMPT + save_and_disable_irqs r9 @ make cssr&csidr read atomic +#endif
Quoted 8 lines
arch/arm/mm/cache-v7.S loop1: · line 65
Quoted 6 lines
arch/arm/mm/cache-v7.S +3
-0mcr p15, 2, r10, c0, c0, 0 @ select current cache level in cssr isb @ isb to sych the new cssr&csidr mrc p15, 1, r1, c0, c0, 0 @ read the new csidr +#ifdef CONFIG_PREEMPT + restore_irqs r9 +#endifI'd suggest using
restore_irqs_notraceinstead. The IRQ-off period is so small that there is no point tracing it.
Thanks. I'll make sure to do that before uploading to the patch tracker.
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Quoted 32 lines · Stephen Boyd
Quoted 22 lines · Nicolas Pitre
Quoted 21 lines · Stephen Boyd
Quoted 3 lines · Nicolas Pitre
If you simply disable/restore IRQs around the critical region then you don't have to worry about
__v7_setup. Plus this will allow forv7_flush_dcache_allto still be callable from atomic context.Ok. Here's a patch. I still need to test it. I'll send another patch series to cleanup the
get_thread_infostuff (there's two of them?).arch/arm/mm/cache-v7.S | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)arch/arm/mm/cache-v7.S +3
-0@@ -54,9 +54,15 @@ loop1: and r1, r1, #7 @ mask of the bits for current cache only cmp r1, #2 @ see what cache we have at this level blt skip @ skip if no cache, or just i-cache +#ifdef CONFIG_PREEMPT + save_and_disable_irqs r9 @ make cssr&csidr read atomic +#endifQuoted 8 lines
arch/arm/mm/cache-v7.S loop1: · line 65
Quoted 6 lines
arch/arm/mm/cache-v7.S +3
-0mcr p15, 2, r10, c0, c0, 0 @ select current cache level in cssr isb @ isb to sych the new cssr&csidr mrc p15, 1, r1, c0, c0, 0 @ read the new csidr +#ifdef CONFIG_PREEMPT + restore_irqs r9 +#endifI'd suggest using
restore_irqs_notraceinstead. The IRQ-off period is so small that there is no point tracing it.
Quoted 1 line
Thanks. I'll make sure to do that before uploading to the patch tracker.
Might be worth flagging this for the stable kernels as well (CC: stable at kernel.org).
Nicolas
Hello.
Quoted 30 lines · Nicolas Pitre
Quoted 29 lines
Quoted 29 lines
Quoted 16 lines
Quoted 3 lines
If you simply disable/restore IRQs around the critical region then you don't have to worry about
__v7_setup. Plus this will allow forv7_flush_dcache_allto still be callable from atomic context.Ok. Here's a patch. I still need to test it. I'll send another patch series to cleanup the
get_thread_infostuff (there's two of them?).arch/arm/mm/cache-v7.S | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)arch/arm/mm/cache-v7.S +0
-0@@ -54,9 +54,15 @@ loop1: and r1, r1, #7 @ mask of the bits for current cache only cmp r1, #2 @ see what cache we have at this levelarch/arm/mm/cache-v7.S loop1: · line 65
Quoted 10 lines
arch/arm/mm/cache-v7.S +6
-0blt skip @ skip if no cache, or just i-cache +#ifdef CONFIG_PREEMPT + save_and_disable_irqs r9 @ make cssr&csidr read atomic +#endif mcr p15, 2, r10, c0, c0, 0 @ select current cache level in cssr isb @ isb to sych the new cssr&csidr mrc p15, 1, r1, c0, c0, 0 @ read the new csidr +#ifdef CONFIG_PREEMPT + restore_irqs r9 +#endifI'd suggest using
restore_irqs_notraceinstead. The IRQ-off period is so small that there is no point tracing it.
Quoted 4 lines
Quoted 1 line
Thanks. I'll make sure to do that before uploading to the patch tracker.
Might be worth flagging this for the stable kernels as well (CC: stable at kernel.org).
The new address is stable at vger.kernel.org as Greg KH wrote.
WBR, Sergei
Quoted 17 lines · Stephen Boyd
armv7's
flush_cache_all()flushes caches via set/way. To determine the cache attributes (line size, number of sets, etc.) the assembly first writes the CSSELR register to select a cache level and then reads the CCSIDR register. The CSSELR register is banked per-cpu and is used to determine which cache level CCSIDR reads. If the task is migrated between when the CSSELR is written and the CCSIDR is read the CCSIDR value may be for an unexpected cache level (for example L1 instead of L2) and incorrect cache flushing could occur.Disable interrupts across the write and read so that the correct cache attributes are read and used for the cache flushing routine. We disable interrupts instead of disabling preemption because the critical section is only 3 instructions and we want to call
v7_dcache_flush_allfrom__v7_setupwhich doesn't have a full kernel stack with a structthread_info.
Quoted 7 lines
This fixes a problem we see in
scm_call()whenflush_cache_all()is called from preemptible context and sometimes the L2 cache is not properly flushed out.Signed-off-by Stephen Boyd <[email protected]>Cc Catalin Marinas <[email protected]>Cc Nicolas Pitre <[email protected]>
Signature · 1 line
Catalin
Quoted 42 lines · Stephen Boyd
armv7's
flush_cache_all()flushes caches via set/way. To determine the cache attributes (line size, number of sets, etc.) the assembly first writes the CSSELR register to select a cache level and then reads the CCSIDR register. The CSSELR register is banked per-cpu and is used to determine which cache level CCSIDR reads. If the task is migrated between when the CSSELR is written and the CCSIDR is read the CCSIDR value may be for an unexpected cache level (for example L1 instead of L2) and incorrect cache flushing could occur.Disable interrupts across the write and read so that the correct cache attributes are read and used for the cache flushing routine. We disable interrupts instead of disabling preemption because the critical section is only 3 instructions and we want to call
v7_dcache_flush_allfrom__v7_setupwhich doesn't have a full kernel stack with a structthread_info.This fixes a problem we see in
scm_call()whenflush_cache_all()is called from preemptible context and sometimes the L2 cache is not properly flushed out.Signed-off-by Stephen Boyd <[email protected]>Cc Catalin Marinas <[email protected]>Cc Nicolas Pitre <[email protected]>---
arch/arm/mm/cache-v7.S +0
-0@@ -54,9 +54,15 @@ loop1:? ? ? ?and ? ? r1, r1, #7 ? ? ? ? ? ? ? ? ? ? ?@ mask of the bits for current cache only ? ? ? ?cmp ? ? r1, #2 ? ? ? ? ? ? ? ? ? ? ? ? ?@ see what cache we have at this level ? ? ? ?blt ? ? skip ? ? ? ? ? ? ? ? ? ? ? ? ? ?@ skip if no cache, or just i-cache
arch/arm/mm/cache-v7.S +3
-0+#ifdef CONFIG_PREEMPT + ? ? ? save_and_disable_irqs r9 ? ? ? ? ? ? ? ?@ make cssr&csidr read atomic +#endif? ? ? ?mcr ? ? p15, 2, r10, c0, c0, 0 ? ? ? ? ?@ select current cache level in cssr ? ? ? ?isb ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? @ isb to sych the new cssr&csidr ? ? ? ?mrc ? ? p15, 1, r1, c0, c0, 0 ? ? ? ? ? @ read the new csidr
arch/arm/mm/cache-v7.S +3
-0+#ifdef CONFIG_PREEMPT + ? ? ? restore_irqs r9 +#endif
Quoted 3 lines
? ? ? ?and ? ? r2, r1, #7 ? ? ? ? ? ? ? ? ? ? ?@ extract the length of the cache lines ? ? ? ?add ? ? r2, r2, #4 ? ? ? ? ? ? ? ? ? ? ?@ add 4 (line length offset) ? ? ? ?ldr ? ? r4, =0x3ff
This patch breaks the kernel boot when lockdep is enabled.
v7_setup (called before the MMU is enabled) calls v7_flush_dcache_all, and the save_and_disable_irqs added by this patch ends up calling into lockdep C code (trace_hardirqs_off()) when we are in no position to execute it (no stack, no MMU).
The following fixes it. Perhaps it can be folded in?
arch/arm/include/asm/assembler.h +5 -0
@@ -137,6 +137,11 @@ disable_irq .endm + .macro save_and_disable_irqs_notrace, oldcpsr + mrs \oldcpsr, cpsr + disable_irq_notrace + .endm +/* * Restore interrupt state previously stored in a register. We don't * guarantee that this will preserve the flags.
arch/arm/mm/cache-v7.S +1 -1
@@ -55,7 +55,7 @@ loop1: cmp r1, #2 @ see what cache we have at this level blt skip @ skip if no cache, or just i-cache #ifdef CONFIG_PREEMPT - save_and_disable_irqs r9 @ make cssr&csidr read atomic + save_and_disable_irqs_notrace r9 @ make cssr&csidr read atomic #endif mcr p15, 2, r10, c0, c0, 0 @ select current cache level in cssr isb @ isb to sych the new cssr&csidr
Quoted 49 lines · Rabin Vincent
Quoted 42 lines · Stephen Boyd
armv7's
flush_cache_all()flushes caches via set/way. To determine the cache attributes (line size, number of sets, etc.) the assembly first writes the CSSELR register to select a cache level and then reads the CCSIDR register. The CSSELR register is banked per-cpu and is used to determine which cache level CCSIDR reads. If the task is migrated between when the CSSELR is written and the CCSIDR is read the CCSIDR value may be for an unexpected cache level (for example L1 instead of L2) and incorrect cache flushing could occur.Disable interrupts across the write and read so that the correct cache attributes are read and used for the cache flushing routine. We disable interrupts instead of disabling preemption because the critical section is only 3 instructions and we want to call
v7_dcache_flush_allfrom__v7_setupwhich doesn't have a full kernel stack with a structthread_info.This fixes a problem we see in
scm_call()whenflush_cache_all()is called from preemptible context and sometimes the L2 cache is not properly flushed out.Signed-off-by Stephen Boyd <[email protected]>Cc Catalin Marinas <[email protected]>Cc Nicolas Pitre <[email protected]>---
arch/arm/mm/cache-v7.S +0
-0@@ -54,9 +54,15 @@ loop1:? ? ? ?and ? ? r1, r1, #7 ? ? ? ? ? ? ? ? ? ? ?@ mask of the bits for current cache only ? ? ? ?cmp ? ? r1, #2 ? ? ? ? ? ? ? ? ? ? ? ? ?@ see what cache we have at this level ? ? ? ?blt ? ? skip ? ? ? ? ? ? ? ? ? ? ? ? ? ?@ skip if no cache, or just i-cache
arch/arm/mm/cache-v7.S +3
-0+#ifdef CONFIG_PREEMPT + ? ? ? save_and_disable_irqs r9 ? ? ? ? ? ? ? ?@ make cssr&csidr read atomic +#endif? ? ? ?mcr ? ? p15, 2, r10, c0, c0, 0 ? ? ? ? ?@ select current cache level in cssr ? ? ? ?isb ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? @ isb to sych the new cssr&csidr ? ? ? ?mrc ? ? p15, 1, r1, c0, c0, 0 ? ? ? ? ? @ read the new csidr
arch/arm/mm/cache-v7.S +3
-0+#ifdef CONFIG_PREEMPT + ? ? ? restore_irqs r9 +#endifQuoted 3 lines
? ? ? ?and ? ? r2, r1, #7 ? ? ? ? ? ? ? ? ? ? ?@ extract the length of the cache lines ? ? ? ?add ? ? r2, r2, #4 ? ? ? ? ? ? ? ? ? ? ?@ add 4 (line length offset) ? ? ? ?ldr ? ? r4, =0x3ff
This patch breaks the kernel boot when lockdep is enabled.
Quoted 6 lines
v7_setup(called before the MMU is enabled) callsv7_flush_dcache_all, and thesave_and_disable_irqsadded by this patch ends up calling into lockdep C code (trace_hardirqs_off()) when we are in no position to execute it (no stack, no MMU).The following fixes it. Perhaps it can be folded in?
Absolutely.
No tracing what so ever should be involved here.
Quoted 31 lines
arch/arm/include/asm/assembler.h +5
-0@@ -137,6 +137,11 @@ disable_irq .endm + .macro save_and_disable_irqs_notrace, oldcpsr + mrs \oldcpsr, cpsr + disable_irq_notrace + .endm +/* * Restore interrupt state previously stored in a register. We don't * guarantee that this will preserve the flags. arch/arm/mm/cache-v7.S +1
-1@@ -55,7 +55,7 @@ loop1: cmp r1, #2 @ see what cache we have at this level blt skip @ skip if no cache, or just i-cache #ifdef CONFIG_PREEMPT - save_and_disable_irqs r9 @ make cssr&csidr read atomic + save_and_disable_irqs_notrace r9 @ make cssr&csidr read atomic #endif mcr p15, 2, r10, c0, c0, 0 @ select current cache level in cssr isb @ isb to sych the new cssr&csidr
Quoted 55 lines · Nicolas Pitre
Quoted 53 lines · Rabin Vincent
Quoted 35 lines · Stephen Boyd
armv7's
flush_cache_all()flushes caches via set/way. To determine the cache attributes (line size, number of sets, etc.) the assembly first writes the CSSELR register to select a cache level and then reads the CCSIDR register. The CSSELR register is banked per-cpu and is used to determine which cache level CCSIDR reads. If the task is migrated between when the CSSELR is written and the CCSIDR is read the CCSIDR value may be for an unexpected cache level (for example L1 instead of L2) and incorrect cache flushing could occur.Disable interrupts across the write and read so that the correct cache attributes are read and used for the cache flushing routine. We disable interrupts instead of disabling preemption because the critical section is only 3 instructions and we want to call
v7_dcache_flush_allfrom__v7_setupwhich doesn't have a full kernel stack with a structthread_info.This fixes a problem we see in
scm_call()whenflush_cache_all()is called from preemptible context and sometimes the L2 cache is not properly flushed out.Signed-off-by Stephen Boyd <[email protected]>Cc Catalin Marinas <[email protected]>Cc Nicolas Pitre <[email protected]>---
arch/arm/mm/cache-v7.S +2
-0@@ -54,9 +54,15 @@ loop1: and r1, r1, #7 @ mask of the bits for current cache only cmp r1, #2 @ see what cache we have at this level blt skip @ skip if no cache, or just i-cache +#ifdef CONFIG_PREEMPT + save_and_disable_irqs r9 @ make cssr&csidr read atomicarch/arm/mm/cache-v7.S loop1: · line 68
Quoted 10 lines
arch/arm/mm/cache-v7.S +4
-0+#endif mcr p15, 2, r10, c0, c0, 0 @ select current cache level in cssr isb @ isb to sych the new cssr&csidr mrc p15, 1, r1, c0, c0, 0 @ read the new csidr +#ifdef CONFIG_PREEMPT + restore_irqs r9 +#endif and r2, r1, #7 @ extract the length of the cache lines add r2, r2, #4 @ add 4 (line length offset) ldr r4, =0x3ffThis patch breaks the kernel boot when lockdep is enabled.
v7_setup(called before the MMU is enabled) callsv7_flush_dcache_all, and thesave_and_disable_irqsadded by this patch ends up calling into lockdep C code (trace_hardirqs_off()) when we are in no position to execute it (no stack, no MMU).
Quoted 5 lines
Quoted 1 line
The following fixes it. Perhaps it can be folded in?
Absolutely.
No tracing what so ever should be involved here.
Thanks. Russell has already merged the original patch to the fixes branch. Hopefully he can fold this one in.
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Quoted 2 lines · Stephen Boyd
Thanks. Russell has already merged the original patch to the fixes branch. Hopefully he can fold this one in.
Nope, I've asked Linus to pull it.
So do we conclude that the original patch wasn't properly tested? :P
Quoted 6 lines · Russell King - ARM Linux
Quoted 2 lines · Stephen Boyd
Thanks. Russell has already merged the original patch to the fixes branch. Hopefully he can fold this one in.
Nope, I've asked Linus to pull it.
So do we conclude that the original patch wasn't properly tested? :P
Sigh. Lockdep strikes again! I promise I tested it with lockdep disabled.
It looks like Linus' hasn't pulled yet but maybe he just hasn't published it.
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Quoted 4 lines · Stephen Boyd
Quoted 3 lines · Russell King - ARM Linux
Quoted 2 lines · Stephen Boyd
Thanks. Russell has already merged the original patch to the fixes branch. Hopefully he can fold this one in.
Quoted 8 lines
Quoted 3 lines
Nope, I've asked Linus to pull it.
So do we conclude that the original patch wasn't properly tested? :P
Sigh. Lockdep strikes again! I promise I tested it with lockdep disabled.
It looks like Linus' hasn't pulled yet but maybe he just hasn't published it.
It's not nice to change something after you've sent a pull request - there's no way of knowing when Linus actually pulls it before he's published it, and if he gets something different then it can raise questions.
So, it's gone in as-is, and, as I'm now intending asking for another pull request soo soon after my previous one, this is something that we will have to live with probably for the remainder of the week.
Note that you should, by default, build your development kernels with lockdep enabled, it's there as a debugging tool to help you find logical locking errors faster than people can provoke deadlocks.
Quoted 14 lines · Russell King - ARM Linux
Quoted 4 lines · Stephen Boyd
Quoted 3 lines · Russell King - ARM Linux
Quoted 2 lines · Stephen Boyd
Thanks. Russell has already merged the original patch to the fixes branch. Hopefully he can fold this one in.
Quoted 8 lines
Quoted 3 lines
Nope, I've asked Linus to pull it.
So do we conclude that the original patch wasn't properly tested? :P
Sigh. Lockdep strikes again! I promise I tested it with lockdep disabled.
It looks like Linus' hasn't pulled yet but maybe he just hasn't published it.
Quoted 8 lines
It's not nice to change something after you've sent a pull request - there's no way of knowing when Linus actually pulls it before he's published it, and if he gets something different then it can raise questions.
So, it's gone in as-is, and, as I'm now intending asking for another pull request soo soon after my previous one, this is something that we will have to live with probably for the remainder of the week.
OK, since it can't be folded in, here is a proper patch:
8<---------
Quoted 1 line
From
26f02624a20a61ed1997a4e8648e4c766a54d91dMon Sep 17 00:00:00 2001
From: Rabin Vincent <[email protected]>
Date: Tue, 14 Feb 2012 19:22:07 +0530
Subject: [PATCH] ARM: fix v7 boot with lockdep enabled
Bootup with lockdep enabled has been broken on v7 since b46c0f74657d ("ARM: 7321/1: cache-v7: Disable preemption when reading CCSIDR").
This is because v7_setup (which is called very early during boot) calls v7_flush_dcache_all, and the save_and_disable_irqs added by that patch ends up attempting to call into lockdep C code (trace_hardirqs_off()) when we are in no position to execute it (no stack, MMU off).
Fix this by using a notrace variant of save_and_disable_irqs. The code already uses the notrace variant of restore_irqs.
--- arch/arm/include/asm/assembler.h | 5 +++++ arch/arm/mm/cache-v7.S | 2 +- 2 files changed, 6 insertions(+), 1 deletions(-)
arch/arm/include/asm/assembler.h +5 -0
@@ -137,6 +137,11 @@ disable_irq .endm + .macro save_and_disable_irqs_notrace, oldcpsr + mrs \oldcpsr, cpsr + disable_irq_notrace + .endm +/* * Restore interrupt state previously stored in a register. We don't * guarantee that this will preserve the flags.
arch/arm/mm/cache-v7.S +1 -1
@@ -55,7 +55,7 @@ loop1: cmp r1, #2 @ see what cache we have at this level blt skip @ skip if no cache, or just i-cache #ifdef CONFIG_PREEMPT - save_and_disable_irqs r9 @ make cssr&csidr read atomic + save_and_disable_irqs_notrace r9 @ make cssr&csidr read atomic #endif mcr p15, 2, r10, c0, c0, 0 @ select current cache level in cssr isb @ isb to sych the new cssr&csidr
Signature · 1 line
1.7.9
Nicolas Pitre
Reviewed-byQuoted 40 lines · Rabin Vincent
Quoted 14 lines · Russell King - ARM Linux
Quoted 4 lines · Stephen Boyd
Quoted 3 lines · Russell King - ARM Linux
Quoted 2 lines · Stephen Boyd
Thanks. Russell has already merged the original patch to the fixes branch. Hopefully he can fold this one in.
Quoted 8 lines
Quoted 3 lines
Nope, I've asked Linus to pull it.
So do we conclude that the original patch wasn't properly tested? :P
Sigh. Lockdep strikes again! I promise I tested it with lockdep disabled.
It looks like Linus' hasn't pulled yet but maybe he just hasn't published it.
Quoted 8 lines
It's not nice to change something after you've sent a pull request - there's no way of knowing when Linus actually pulls it before he's published it, and if he gets something different then it can raise questions.
So, it's gone in as-is, and, as I'm now intending asking for another pull request soo soon after my previous one, this is something that we will have to live with probably for the remainder of the week.
OK, since it can't be folded in, here is a proper patch:
8<---------
Quoted 1 line
From
26f02624a20a61ed1997a4e8648e4c766a54d91dMon Sep 17 00:00:00 2001From: Rabin Vincent <[email protected]>
Date: Tue, 14 Feb 2012 19:22:07 +0530
Subject: [PATCH] ARM: fix v7 boot with lockdep enabledBootup with lockdep enabled has been broken on v7 since
b46c0f74657d("ARM: 7321/1: cache-v7: Disable preemption when reading CCSIDR").This is because
v7_setup(which is called very early during boot) callsv7_flush_dcache_all, and thesave_and_disable_irqsadded by that patch ends up attempting to call into lockdep C code (trace_hardirqs_off()) when we are in no position to execute it (no stack, MMU off).
Quoted 6 lines
Fix this by using a notrace variant of
save_and_disable_irqs. The code already uses the notrace variant ofrestore_irqs.Cc Stephen Boyd <[email protected]>Cc Catalin Marinas <[email protected]>Cc Nicolas Pitre <[email protected]>
Quoted 39 lines
Cc stable at vger.kernel.orgSigned-off-by Rabin Vincent <[email protected]>--- arch/arm/include/asm/assembler.h | 5 +++++ arch/arm/mm/cache-v7.S | 2 +- 2 files changed, 6 insertions(+), 1 deletions(-)arch/arm/include/asm/assembler.h +5
-0@@ -137,6 +137,11 @@ disable_irq .endm + .macro save_and_disable_irqs_notrace, oldcpsr + mrs \oldcpsr, cpsr + disable_irq_notrace + .endm +/* * Restore interrupt state previously stored in a register. We don't * guarantee that this will preserve the flags. arch/arm/mm/cache-v7.S +1
-1@@ -55,7 +55,7 @@ loop1: cmp r1, #2 @ see what cache we have at this level blt skip @ skip if no cache, or just i-cache #ifdef CONFIG_PREEMPT - save_and_disable_irqs r9 @ make cssr&csidr read atomic + save_and_disable_irqs_notrace r9 @ make cssr&csidr read atomic #endif mcr p15, 2, r10, c0, c0, 0 @ select current cache level in cssr isb @ isb to sych the new cssr&csidrSignature · 1 line
1.7.9
Stephen Boyd
Acked-byQuoted 37 lines · Rabin Vincent
Quoted 12 lines · Russell King - ARM Linux
Quoted 11 lines · Stephen Boyd
Quoted 6 lines · Russell King - ARM Linux
Quoted 2 lines · Stephen Boyd
Thanks. Russell has already merged the original patch to the fixes branch. Hopefully he can fold this one in.
Nope, I've asked Linus to pull it.
So do we conclude that the original patch wasn't properly tested? :P
Sigh. Lockdep strikes again! I promise I tested it with lockdep disabled.
It looks like Linus' hasn't pulled yet but maybe he just hasn't published it.
Quoted 8 lines
It's not nice to change something after you've sent a pull request - there's no way of knowing when Linus actually pulls it before he's published it, and if he gets something different then it can raise questions.
So, it's gone in as-is, and, as I'm now intending asking for another pull request soo soon after my previous one, this is something that we will have to live with probably for the remainder of the week.
OK, since it can't be folded in, here is a proper patch:
8<---------
From26f02624a20a61ed1997a4e8648e4c766a54d91dMon Sep 17 00:00:00 2001
From: Rabin Vincent <[email protected]>
Date: Tue, 14 Feb 2012 19:22:07 +0530
Subject: [PATCH] ARM: fix v7 boot with lockdep enabledBootup with lockdep enabled has been broken on v7 since
b46c0f74657d("ARM: 7321/1: cache-v7: Disable preemption when reading CCSIDR").This is because
v7_setup(which is called very early during boot) callsv7_flush_dcache_all, and thesave_and_disable_irqsadded by that patch ends up attempting to call into lockdep C code (trace_hardirqs_off()) when we are in no position to execute it (no stack, MMU off).
Quoted 4 lines
Fix this by using a notrace variant of
save_and_disable_irqs. The code already uses the notrace variant ofrestore_irqs.Cc Stephen Boyd <[email protected]>
Quoted 38 lines
Cc Catalin Marinas <[email protected]>Cc Nicolas Pitre <[email protected]>Cc stable at vger.kernel.orgSigned-off-by Rabin Vincent <[email protected]>--- arch/arm/include/asm/assembler.h | 5 +++++ arch/arm/mm/cache-v7.S | 2 +- 2 files changed, 6 insertions(+), 1 deletions(-)arch/arm/include/asm/assembler.h +5
-0@@ -137,6 +137,11 @@ disable_irq .endm + .macro save_and_disable_irqs_notrace, oldcpsr + mrs \oldcpsr, cpsr + disable_irq_notrace + .endm +/* * Restore interrupt state previously stored in a register. We don't * guarantee that this will preserve the flags. arch/arm/mm/cache-v7.S +1
-1@@ -55,7 +55,7 @@ loop1: cmp r1, #2 @ see what cache we have at this level blt skip @ skip if no cache, or just i-cache #ifdef CONFIG_PREEMPT - save_and_disable_irqs r9 @ make cssr&csidr read atomic + save_and_disable_irqs_notrace r9 @ make cssr&csidr read atomic #endif mcr p15, 2, r10, c0, c0, 0 @ select current cache level in cssr isb @ isb to sych the new cssr&csidr
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Quoted 17 lines · Russell King - ARM Linux
Quoted 7 lines · Stephen Boyd
Quoted 6 lines · Nicolas Pitre
What about a pair of helpers written in C instead?
v7_flush_dcache_all()could be renamed, and a wrapper function calledv7_flush_dcache_all()would call the preemption disable helper, call the formerv7_flush_dcache_allcode, then call the preemption enable helper.Quoted 8 lines
Quoted 2 lines
Then
__v7_setup()could still call the core cache flush code without issues.I tried to put the preemption disable/enable right around the place where it was needed. With this approach we would disable preemption during the entire cache flush. I'm not sure if we want to make this function worse for performance, do we? It certainly sounds easier than writing all the preempt macros in assembly though.
Quoted 5 lines
Err, why do you think it's a big task?
preempt disable is a case of incrementing the thread preempt count, while preempt enable is a case of decrementing it, testing for zero, if zero, then checking whether
TIF_NEED_RESCHEDis set and calling a function.
Oh certainly. And we already do just that in a few places already. I re-read your previous email to realize that I initially misread your remark about the ickness of explicitly calling the scheduler.
Quoted 4 lines
If that's too much, then the simple method in assembly to quickly disable preemption over a very few set of instructions is using mrs/msr and cpsid i. That'll be far cheaper than fiddling about with preempt counters or messing about with veneers in C code.
Indeed. And I think that would be plenty sufficient here as the protected region is really short. I don't think that warrants any macros.
Nicolas
Quoted 26 lines · Russell King - ARM Linux
Quoted 14 lines · Stephen Boyd
Quoted 13 lines · Nicolas Pitre
On Thu, 2 Feb 2012, Russell King - ARM Linux wrote
Quoted 6 lines · Stephen Boyd
Quoted 2 lines
Should we move
get_thread_infointoassembler.h? It seems odd to includeentry-header.Sbut I saw that vfp was doing the same.Probably yes, and probably also have
preempt_disableandpreempt_enableassembler macros. That's going to get rather icky if we have to explicitly call the scheduler though (to solve (1)).What about a pair of helpers written in C instead?
v7_flush_dcache_all()could be renamed, and a wrapper function calledv7_flush_dcache_all()would call the preemption disable helper, call the formerv7_flush_dcache_allcode, then call the preemption enable helper.Quoted 8 lines
Quoted 2 lines
Then
__v7_setup()could still call the core cache flush code without issues.I tried to put the preemption disable/enable right around the place where it was needed. With this approach we would disable preemption during the entire cache flush. I'm not sure if we want to make this function worse for performance, do we? It certainly sounds easier than writing all the preempt macros in assembly though.
Err, why do you think it's a big task?
Quoted 8 lines
preempt disable is a case of incrementing the thread preempt count, while preempt enable is a case of decrementing it, testing for zero, if zero, then checking whether
TIF_NEED_RESCHEDis set and calling a function.If that's too much, then the simple method in assembly to quickly disable preemption over a very few set of instructions is using mrs/msr and cpsid i. That'll be far cheaper than fiddling about with preempt counters or messing about with veneers in C code.
Russell,
I think you misunderstood Stephen's point about the performance. He isn't referring to the performance difference between a C call to preemt disable/enable vs. a few assembly level instructions.
I believe he is referring to the performance hit of having preemption disabled during the entirety of the cache flush operation vs. having preemption disabled only for the duration of writing to CSSELR and reading back CCSIDR.
I would think a cache flush is a fairly long operation and to have preemption disable across it doesn't sound appealing to me.
Thoughts?
Thanks,
Saravana
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Quoted 53 lines · Saravana Kannan
Quoted 34 lines · Russell King - ARM Linux
Quoted 17 lines · Stephen Boyd
Quoted 16 lines · Nicolas Pitre
On Thu, 2 Feb 2012, Russell King - ARM Linux wrote
Quoted 7 lines · Stephen Boyd
Quoted 2 lines
Should we move
get_thread_infointoassembler.h? It seems odd to includeentry-header.Sbut I saw that vfp was doing the same.Probably yes, and probably also have
preempt_disableandpreempt_enable
assembler macros. That's going to get rather icky if we have to explicitly call the scheduler though (to solve (1)).What about a pair of helpers written in C instead?
v7_flush_dcache_all()could be renamed, and a wrapper function calledv7_flush_dcache_all()would call the preemption disable helper, call the
formerv7_flush_dcache_allcode, then call the preemption enable helper.Quoted 8 lines
Quoted 2 lines
Then
__v7_setup()could still call the core cache flush code without issues.I tried to put the preemption disable/enable right around the place where it was needed. With this approach we would disable preemption during the entire cache flush. I'm not sure if we want to make this function worse for performance, do we? It certainly sounds easier than writing all the preempt macros in assembly though.
Err, why do you think it's a big task?
preempt disable is a case of incrementing the thread preempt count, while
preempt enable is a case of decrementing it, testing for zero, if zero, then checking whetherTIF_NEED_RESCHEDis set and calling a function.Quoted 6 lines
If that's too much, then the simple method in assembly to quickly disable
preemption over a very few set of instructions is using mrs/msr and cpsid i.
That'll be far cheaper than fiddling about with preempt counters or messing about with veneers in C code.Russell,
I think you misunderstood Stephen's point about the performance. He isn't referring to the performance difference between a C call to preemt disable/enable vs. a few assembly level instructions.
I believe he is referring to the performance hit of having preemption disabled during the entirety of the cache flush operation vs. having preemption disabled only for the duration of writing to CSSELR and reading back CCSIDR.
Quoted 5 lines
I would think a cache flush is a fairly long operation and to have preemption disable across it doesn't sound appealing to me.
Thoughts?
Sorry I messed up the headers for v2 of the patch. It didn't get sent to the msm list.
Anyway, disabling interrupts for those few instructions sounds like the best approach and so I sent that out in v2.
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.