ARM: augment cache flushing API

ARM v7 architecture introduced the concept of cache levels and related coherency requirements. New processors like A7 and A15 embed an L2 unified cache controller that becomes part of the cache level hierarchy. Some operations in the kernel like cpu_suspend and __cpu_disable does not require a flush of the entire cache hierarchy to DRAM but just the cache levels belonging to the Level of Unification Inner Shareable (LoUIS), which in most of ARM v7 systems correspond to L1.

The current cache flushing API used in cpu_suspend and __cpu_disable, flush_cache_all(), ends up flushing the whole cache hierarchy since for v7 it cleans and invalidates all cache levels up to Level of Coherency (LoC) which cripples system performance when used in hot paths like hotplug and cpuidle.

Therefore a new kernel cache maintenance API must be added to the cpu_cache_fns structure of pointers to cope with latest ARM system requirements. This patch adds flush_cache_louis() to the ARM kernel cache maintenance API.

This function cleans and invalidates all data cache levels up to the level of unification inner shareable (LoUIS) and invalidates the instruction cache.

The cpu_cache_fns struct reflects this change by adding a new function pointer that is initialized by arch specific assembly files.

By default, all existing ARM archs do not instantiate any cache LoUIS function pointer, and flush_dcache_louis just falls back to flush_kern_all.

Reviewed-by Santosh Shilimkar <[email protected]>

Signed-off-by Lorenzo Pieralisi <[email protected]>

---
 arch/arm/include/asm/cacheflush.h | 17 +++++++++++++++++
 arch/arm/mm/proc-macros.S         |  7 ++++++-
 2 files changed, 23 insertions(+), 1 deletion(-)

arch/arm/include/asm/cacheflush.h +8 -0

index c6e2ed9..7683316 100644
@@ -50,6 +50,13 @@
  *
  *		Unconditionally clean and invalidate the entire cache.
  *
+ *     flush_kern_cache_louis()
+ *
+ *             Flush data cache levels up to the level of unification
+ *             inner shareable and invalidate the I-cache.
+ *             Only needed from v7 onwards, falls back to flush_cache_all()
+ *             for all other processor versions.
+ *
  *	flush_user_all()
  *
  *		Clean and invalidate all user space cache entries

@@ -98,6 +105,7 @@
 struct cpu_cache_fns {
 	void (*flush_icache_all)(void);
 	void (*flush_kern_all)(void);
+	void (*flush_kern_cache_louis)(void);
 	void (*flush_user_all)(void);
 	void (*flush_user_range)(unsigned long, unsigned long, unsigned int);
 

@@ -200,6 +208,15 @@ extern void copy_to_user_page(struct vm_area_struct *, struct page *,
 #define __flush_icache_preferred	__flush_icache_all_generic

We could introduce something like CONFIG_ARM_HAVE_CACHEFLUSH_LOUIS, and do:

<asm/glue-cache.h>
#ifndef MULTI_CACHE
#ifdef CONFIG_HAVE_ARM_CACHEFLUSH_LOUIS
#define __cpuc_flush_kern_cache_louis __glue(_CACHE,_flush_kern_cache_louis) #else
#define __cpuc_flush_kern_cache_louis __glue(_CACHE,_flush_kern_all) #endif
#endif

<asm/cacheflush.h>
#ifdef MULTI_CACHE
#define flush_cache_louis() cpu_cache.flush_kern_cache_louis()
#else
#define flush_cache_louis() __cpuc_flush_kern_cache_louis()
#endif

Quoted 12 lines · Dave Martin

We could introduce something like CONFIG_ARM_HAVE_CACHEFLUSH_LOUIS, and do:

<asm/glue-cache.h>
#ifndef MULTI_CACHE
#ifdef CONFIG_HAVE_ARM_CACHEFLUSH_LOUIS
#define __cpuc_flush_kern_cache_louis __glue(_CACHE,_flush_kern_cache_louis) #else
#define __cpuc_flush_kern_cache_louis __glue(_CACHE,_flush_kern_all) #endif
#endif

Quoted 6 lines

<asm/cacheflush.h>
#ifdef MULTI_CACHE
#define flush_cache_louis() cpu_cache.flush_kern_cache_louis()
#else
#define flush_cache_louis() __cpuc_flush_kern_cache_louis()
#endif

No, this is stupidly complicated and is fragile. Just alias the functions, like we do in cache-v4wt.S:

.globl v4wt_dma_flush_range
.equ v4wt_dma_flush_range, v4wt_dma_inv_range

except, you'll need:

.globl v4wt_flush_kern_cache_louis
.equ v4wt_flush_kern_cache_louis, v4wt_flush_kern_cache_all

You can do it automatically, using the attached sedscript and this bit of shell:

$ for f in $(grep -l define_cache_functions arch/arm/mm/*.S ); do
    sed -if sedscript $f
    git add $f
done
$ git commit -s

Do that first, and then go over those which you need to add a real flush_kern_cache_louis function to.

1,/__INITDATA\|define struct cpu_cache_fns/ {
	/ENTRY.*flush_kern_cache_all/ {
        	h
	        s/.*(\([^_]*\)_.*/\t.globl\t\1_flush_kern_cache_louis\n\t.equ\t\1_flush_kern_cache_louis, \1_flush_kern_cache_all\n/
        	x
	}
	/__INITDATA\|define struct cpu_cache_fns/ {
        	H
	        g
	}
}

arch/arm/mm/proc-macros.S +1 -1

index 2d8ff3a..28e91f8 100644
@@ -293,12 +293,17 @@ ENTRY(\name\()_processor_functions)
 	.size	\name\()_processor_functions, . - \name\()_processor_functions
 .endm
 
-.macro define_cache_functions name:req
+.macro define_cache_functions name:req, cachelouis=0
 	.align 2
 	.type	\name\()_cache_fns, #object
 ENTRY(\name\()_cache_fns)

arch/arm/include/asm/cacheflush.h extern void copy_to_user_page(struct vm_area_struct *, struct page *, · line 218

Quoted 8 lines · Lorenzo Pieralisi

arch/arm/include/asm/cacheflush.h +8 -0

+/*
+ * Flush caches up to Level of Unification Inner Shareable
+ */
+#ifdef MULTI_CACHE
+#define flush_cache_louis()   cpu_cache.flush_kern_cache_louis()
+#else
+#define flush_cache_louis()   __cpuc_flush_kern_all()
+#endif

NAK. This is broken as you don't seem to understand what MULTI_CACHE actually means. MULTI_CACHE means that we _may_ support more than one type of cache, so it not being selected means nothing as far as whether to use flush_kern_all() or not.

Follow the pattern in the rest of the file - that's the _only_ way to do this.

Note that ARMv6 only and ARMv7 only kernels will not have MULTI_CACHE defined (I mentioned this on Monday in our call, though not explicitly by that name.)

Quoted 19 lines

arch/arm/mm/proc-macros.S +5 -1

index 2d8ff3a..28e91f8 100644
@@ -293,12 +293,17 @@ ENTRY(\name\()_processor_functions)
 	.size	\name\()_processor_functions, . - \name\()_processor_functions
 .endm
 
-.macro define_cache_functions name:req
+.macro define_cache_functions name:req, cachelouis=0
 	.align 2
 	.type	\name\()_cache_fns, #object
 ENTRY(\name\()_cache_fns)
 	.long	\name\()_flush_icache_all
 	.long	\name\()_flush_kern_cache_all
+	.if \cachelouis
+	.long   \name\()_flush_kern_cache_louis
+	.else
+	.long   \name\()_flush_kern_cache_all

arch/arm/mm/proc-macros.S ENTRY(\name\()_processor_functions) · line 309

Quoted 4 lines

arch/arm/mm/proc-macros.S +1 -0

+	.endif
 	.long	\name\()_flush_user_cache_all
 	.long	\name\()_flush_user_cache_range
 	.long	\name\()_coherent_kern_range

And what the above means is that _every_ cache support file must supply a XXX_flush_kern_cache_louis function name. If there is no separate implementation, then name it an alias for the XXX_flush_cache_all function.

In processors like A15/A7 L2 cache is unified and integrated within the processor cache hierarchy, so that it is not considered an outer cache anymore. For processors like A15/A7 flush_cache_all() ends up cleaning all cache levels up to Level of Coherency (LoC) that includes the L2 unified cache.

When a single CPU is suspended (CPU idle) a complete L2 clean is not required, so generic cpu_suspend code must clean the data cache using the newly introduced cache LoUIS function.

The context and stack pointer (context pointer) are cleaned to main memory using cache area functions that operate on MVA and guarantee that the data is written back to main memory (perform cache cleaning up to the Point of Coherency - PoC) so that the processor can fetch the context when the MMU is off in the cpu_resume code path.

outer_cache management remains unchanged.

Reviewed-by Santosh Shilimkar <[email protected]>

Signed-off-by Lorenzo Pieralisi <[email protected]>

---
 arch/arm/kernel/suspend.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

arch/arm/kernel/suspend.c +16 -1

index 1794cc3..358bca3 100644
@@ -17,6 +17,8 @@ extern void cpu_resume_mmu(void);
  */
 void __cpu_suspend_save(u32 *ptr, u32 ptrsz, u32 sp, u32 *save_ptr)
 {
+	u32 *ctx = ptr;
+
 	*save_ptr = virt_to_phys(ptr);
 
 	/* This must correspond to the LDM in cpu_resume() assembly */

@@ -26,7 +28,20 @@ void __cpu_suspend_save(u32 *ptr, u32 ptrsz, u32 sp, u32 *save_ptr)
 
 	cpu_do_suspend(ptr);
 
-	flush_cache_all();
+	flush_cache_louis();
+
+	/*
+	 * flush_cache_louis does not guarantee that
+	 * save_ptr and ptr are cleaned to main memory,
+	 * just up to the Level of Unification Inner Shareable.
+	 * Since the context pointer and context itself
+	 * are to be retrieved with the MMU off that
+	 * data must be cleaned from all cache levels
+	 * to main memory using "area" cache primitives.
+	*/
+	__cpuc_flush_dcache_area(ctx, ptrsz);
+	__cpuc_flush_dcache_area(save_ptr, sizeof(*save_ptr));
+
 	outer_clean_range(*save_ptr, *save_ptr + ptrsz);
 	outer_clean_range(virt_to_phys(save_ptr),
 			  virt_to_phys(save_ptr) + sizeof(*save_ptr));

Signature · 1 line

1.7.12

Quoted 10 lines · Lorenzo Pieralisi

In processors like A15/A7 L2 cache is unified and integrated within the processor cache hierarchy, so that it is not considered an outer cache anymore. For processors like A15/A7 flush_cache_all() ends up cleaning all cache levels up to Level of Coherency (LoC) that includes the L2 unified cache.

When a single CPU is suspended (CPU idle) a complete L2 clean is not required, so generic cpu_suspend code must clean the data cache using the newly introduced cache LoUIS function.

Quoted 7 lines

The context and stack pointer (context pointer) are cleaned to main memory using cache area functions that operate on MVA and guarantee that the data is written back to main memory (perform cache cleaning up to the Point of Coherency - PoC) so that the processor can fetch the context when the MMU is off in the cpu_resume code path.

outer_cache management remains unchanged.

LoUIS matches the power domain affected by turning a single CPU off on most (all?) current v7 SoCs where this matters, but only by coincidence. There is no guarantee of that.

The _louis() API is useful, because this is exactly what you need to to I-/D-/TLB synchronisation in an SMP OS. Using it as preparation for powering a CPU off feels like misuse, at least in theory.

For powerdown, we would logically need a separate function, flush_cache_cpu() or something, whose job is precisely to flush those levels which will be affected by the power-down if that single CPU.

In the series, there is patch "[PATCH 3/6]" which adds an API which let you operate on a specific level.

Quoted 2 lines · Santosh

In the series, there is patch "[PATCH 3/6]" which adds an API which let you operate on a specific level.

Which is introduced but as far as I can see, is never used in the patch set. Therefore, it shouldn't be introduced.

We've been here before many many many times, where people introduce stuff into the kernel, and then they never get around to using the damned stuff. It's happened far too many times to permit on a "but I will use it in the future" kind of arguments.

If you're going to introduce something new, include the users in the patch set, or don't bother submitting the new function in the vague hope that some day it will get used.

On Thu, Sep 13, 2012 at 6:23 PM, Dave Martin <[email protected]> wrote:

Quoted 8 lines

LoUIS matches the power domain affected by turning a single CPU off on most (all?) current v7 SoCs where this matters, but only by coincidence. There is no guarantee of that.

The _louis() API is useful, because this is exactly what you need to to I-/D-/TLB synchronisation in an SMP OS. Using it as preparation for powering a CPU off feels like misuse, at least in theory.

Quoted 4 lines

For powerdown, we would logically need a separate function, flush_cache_cpu() or something, whose job is precisely to flush those levels which will be affected by the power-down if that single CPU.

In the series, there is patch "[PATCH 3/6]" which adds an API which let you operate on a specific level.

Quoted 10 lines · Lorenzo Pieralisi

In processors like A15/A7 L2 cache is unified and integrated within the processor cache hierarchy, so that it is not considered an outer cache anymore. For processors like A15/A7 flush_cache_all() ends up cleaning all cache levels up to Level of Coherency (LoC) that includes the L2 unified cache.

When a single CPU is suspended (CPU idle) a complete L2 clean is not required, so generic cpu_suspend code must clean the data cache using the newly introduced cache LoUIS function.

Quoted 7 lines

The context and stack pointer (context pointer) are cleaned to main memory using cache area functions that operate on MVA and guarantee that the data is written back to main memory (perform cache cleaning up to the Point of Coherency - PoC) so that the processor can fetch the context when the MMU is off in the cpu_resume code path.

outer_cache management remains unchanged.

LoUIS matches the power domain affected by turning a single CPU off on most (all?) current v7 SoCs where this matters, but only by coincidence. There is no guarantee of that.

The _louis() API is useful, because this is exactly what you need to to I-/D-/TLB synchronisation in an SMP OS. Using it as preparation for powering a CPU off feels like misuse, at least in theory.

For powerdown, we would logically need a separate function, flush_cache_cpu() or something, whose job is precisely to flush those levels which will be affected by the powerdown if that single CPU.