Skip to contentSkip to conversation

Cache code changes

PATCH Ralf Baechle 4 participants 6 messages Mar 20, 2003 – Mar 31, 2003 MIPS

lore
Share & tools
lore
#0 lore raw

Thought I should drop a note about the recent cache code changes in the Linux 2.4 and 2.5 code in CVS to explain what's going on.

flush_page_to_ram() has long been a deprecated interface and been scheduled to be removed for years. It's considered a inefficient, badly designed interface. It's use it for dealing with virtual aliases in the primary cache. That is whenever the memory managment code creates or modifies a page that is mapped to userspace it has to writeback and invalidate the kernel mapping of this page to avoid virtual aliases.

flush_page_to_ram() turned out to be a rather ad-hoc interface; the obvious but inefficient interface approach. It's also not capable of fully dealing with all types of cache aliases like aliases between the page cache and user mappings. Which may lead to silent data corruption and that's the reason why I'm doing such intrusive kernel surgery for a supposedly stable kernel. So there now is an alternative interface available in the kernel, flush_dcache_page(). flush_dcache_page() is implements a two stage approach. It marks pages which are in the page cache and therefore could possibly alias with userspace as possibly residing in cache if it doesn't flush them immediately. This allows delaying cache flushes - possibly infinitely. Which quite obviously is a performance gain.

Side effect - some implementations of flush_icache_page() knew that it's invocations are always preceeded by flush_page_to_ram() so the D-cache flush can be omitted. This is no longer there case. Another bug fixed along the way (but not yet for all processors) was flush_cache_page() not flushing the instruction cache ...

Along with that I've also cleaned the cache code for R4000 and R4400 CPUs. Continuing the mess seemed to be plain unmaintainable and at the same time huge. The heavily changed code (for your amusement now using a few new code constructs :-) is now over 40% smaller meassure in LOCs and about 2/3 smaller in code size and should make it fairly easy to add support for strange beasts such as TX39, TX49 or R5432 style caches caches.

Why is it still not working? Well, below a kludge that will get the latest 2.4 code to work again for all processors that are suffering from cache aliases. It's an inefficient solution but good enough for now.

Ralf

Index: include/asm-mips/page.h
===================================================================
RCS file: /home/cvs/linux/include/asm-mips/page.h,v retrieving revision 1.14.2.11

asm-mips/page.h +4 -2
diff -u -r1.14.2.11 page.h
@@ -64,8 +64,10 @@
 
 #define clear_page(page)	_clear_page(page)
 #define copy_page(to, from)	_copy_page(to, from)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
+#define clear_user_page(page, vaddr) \
+	do { clear_page(page); flush_cache_all(); } while (0)
+#define copy_user_page(to, from, vaddr)	\
+	do { copy_page(to, from); flush_cache_all(); } while (0)
 
 /*
  * These are used to make use of C type-checking..
#1 lore raw

I am very glad to see this happens:)

Currently linux/mips is really far from efficient,I've some data for this declaration:
For most SPEC CPU 2000 programs we run on a 4-way superscalar CPU
simulator
,we get <0.20 IPC in kernel mode,while the IPCs for the whole execution are often much higher(0.5-1.5). We believe this has something to do with the overly used cache flushes.

BTW, for 2.4.17,it seems this path is still not safe for cache aliases:
copy_cow_page for newly forked process, we use kernel virtual address
to do the copy,but without flush first.
add a flush_page_to_ram before the copy fix the errors.

but i am not sure whether i am missing something

Quoted 67 lines · Ralf Baechle

Thought I should drop a note about the recent cache code changes in the Linux 2.4 and 2.5 code in CVS to explain what's going on.

flush_page_to_ram() has long been a deprecated interface and been scheduled to be removed for years. It's considered a inefficient, badly designed interface. It's use it for dealing with virtual aliases in the primary cache. That is whenever the memory managment code creates or modifies a page that is mapped to userspace it has to writeback and invalidate the kernel mapping of this page to avoid virtual aliases.

flush_page_to_ram() turned out to be a rather ad-hoc interface; the obvious but inefficient interface approach. It's also not capable of fully dealing with all types of cache aliases like aliases between the page cache and user mappings. Which may lead to silent data corruption and that's the reason why I'm doing such intrusive kernel surgery for a supposedly stable kernel. So there now is an alternative interface available in the kernel, flush_dcache_page(). flush_dcache_page() is implements a two stage approach. It marks pages which are in the page cache and therefore could possibly alias with userspace as possibly residing in cache if it doesn't flush them immediately. This allows delaying cache flushes - possibly infinitely. Which quite obviously is a performance gain.

Side effect - some implementations of flush_icache_page() knew that it's invocations are always preceeded by flush_page_to_ram() so the D-cache flush can be omitted. This is no longer there case. Another bug fixed along the way (but not yet for all processors) was flush_cache_page() not flushing the instruction cache ...

Along with that I've also cleaned the cache code for R4000 and R4400 CPUs. Continuing the mess seemed to be plain unmaintainable and at the same time huge. The heavily changed code (for your amusement now using a few new code constructs :-) is now over 40% smaller meassure in LOCs and about 2/3 smaller in code size and should make it fairly easy to add support for strange beasts such as TX39, TX49 or R5432 style caches caches.

Why is it still not working? Well, below a kludge that will get the latest 2.4 code to work again for all processors that are suffering from cache aliases. It's an inefficient solution but good enough for now.

Ralf

Index: include/asm-mips/page.h
===================================================================
RCS file: /home/cvs/linux/include/asm-mips/page.h,v retrieving revision 1.14.2.11
diff -u -r1.14.2.11 page.h

asm-mips/page.h +4 -2
@@ -64,8 +64,10 @@
 
 #define clear_page(page)	_clear_page(page)
 #define copy_page(to, from)	_copy_page(to, from)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
+#define clear_user_page(page, vaddr) \
+	do { clear_page(page); flush_cache_all(); } while (0)
+#define copy_user_page(to, from, vaddr)	\
+	do { copy_page(to, from); flush_cache_all(); } while (0)
 
 /*
 * These are used to make use of C type-checking..
#2 lore raw
Quoted 11 lines · Fuxin Zhang

I am very glad to see this happens:)

Currently linux/mips is really far from efficient,I've some data for this declaration:
For most SPEC CPU 2000 programs we run on a 4-way superscalar CPU
simulator
,we get <0.20 IPC in kernel mode,while the IPCs for the whole execution are often much
higher(0.5-1.5). We believe this has something to do with the overly used cache flushes.

What type of cache are you using in this simulation? Virtual/physical indexing/tagging, what associativity?

Linux/MIPS's handling of virtually indexed data caches isn't as good as it should be but that's what I'm working on.

Quoted 6 lines

BTW, for 2.4.17,it seems this path is still not safe for cache aliases:
copy_cow_page for newly forked process, we use kernel virtual address
to do the copy,but without flush first.
add a flush_page_to_ram before the copy fix the errors.

but i am not sure whether i am missing something

The functions clear_user_page and copy_user_page are supposed to take care of aliases in this case. That's what the little patch did in my previous mail did, just in a rather inefficient way.

Ralf

#4 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"dominic" == Dominic Sweetman <[email protected]> writes:

dominic> Ralf,

Quoted 1 line
Quoted 1 line

flush_page_to_ram()...

dominic> Can you remind me where I can find the specification of what these dominic> functions are supposed to do?

In the linux source:

linux/Documentation/cachetlb.txt

Later, Juan.

Signature · 2 lines
In theory, practice and theory are the same, but in practice they 
are different -- Larry McVoy
#5 lore raw
In reply to Ralf Baechle

So below a fix for the 32-bit 2.4 kernel to resolve the remaining cache problems. I tried another much more complex fix but it turned out to be not implementable with the memory managment infrastructure provided by Linux 2.4.

This patch also begins to exploit some of the optimizations enabled by the new flush_dcache_page interface. As the result this is the fastest 2.4 kernel so far. The benefits are entirely for virtually indexed data caches; the smaller the caches, the higher the gain.

Ralf

mips/kernel/syscall.c 31 Mar 2003 11:36:02 -0000 renamed from mips/kernel/syscall.c 13 Mar 2003 14:29:14 -0000 +6 -5
diff -u -r1.22.2.5 syscall.c
@@ -55,10 +55,11 @@
 	return res;
 }
 
-#define MMAP_SHARED_ALIGN 0x8000
-#define COLOUR_ALIGN(addr, pgoff)					\
-	((((addr)+ MMAP_SHARED_ALIGN - 1)&~(MMAP_SHARED_ALIGN - 1)) +	\
-	 (((pgoff) << PAGE_SHIFT) & (MMAP_SHARED_ALIGN - 1)))
+unsigned long shm_align_mask = PAGE_SIZE - 1;	/* Sane caches */
+
+#define COLOUR_ALIGN(addr,pgoff)				\
+	((((addr) + shm_align_mask) & ~shm_align_mask) +	\
+	 (((pgoff) << PAGE_SHIFT) & shm_align_mask))
 
 unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr,
 	unsigned long len, unsigned long pgoff, unsigned long flags)
@@ -71,7 +72,7 @@
 		 * We do not accept a shared mapping if it would violate
 		 * cache aliasing constraints.
 		 */
-		if ((flags & MAP_SHARED) && (addr & (MMAP_SHARED_ALIGN - 1)))
+		if ((flags & MAP_SHARED) && (addr & shm_align_mask))
 			return -EINVAL;
 		return addr;
 	}
mips/mm/Makefile 31 Mar 2003 11:36:02 -0000 renamed from mips/mm/Makefile 29 Mar 2003 04:02:49 -0000 +2 -1
diff -u -r1.27.2.11 Makefile
@@ -11,7 +11,8 @@
 O_TARGET := mm.o
 
 export-objs			+= ioremap.o loadmmu.o remap.o
-obj-y				+= extable.o init.o ioremap.o fault.o loadmmu.o
+obj-y				+= cache.o extable.o init.o ioremap.o fault.o \
+				   loadmmu.o
 
 obj-$(CONFIG_CPU_R3000)		+= pg-r3k.o c-r3k.o tlb-r3k.o tlbex-r3k.o
 obj-$(CONFIG_CPU_TX39XX)	+= pg-r3k.o c-tx39.o tlb-r3k.o tlbex-r3k.o
mips/mm/c-mips32.c 31 Mar 2003 11:36:04 -0000 renamed from mips/mm/c-mips32.c 25 Mar 2003 14:30:19 -0000 +10 -36

73 lines not on this page · raw message

mips/mm/c-r3k.c 31 Mar 2003 11:36:04 -0000 renamed from mips/mm/c-r3k.c 25 Mar 2003 03:26:20 -0000 +4 -9
diff -u -r1.4.2.6 c-r3k.c
@@ -258,7 +258,7 @@
 {
 }
 
-static void r3k_flush_dcache_page(struct page * page)
+static void r3k_flush_data_cache_page(unsigned long addr)
 {
 }
 
@@ -314,11 +314,6 @@
 	r3k_flush_dcache_range(start, start + size);
 }
 
-void __update_cache(struct vm_area_struct *vma, unsigned long address,
-	pte_t pte)
-{
-}
-
 void __init ld_mmu_r23000(void)
 {
 	unsigned long config;
@@ -333,16 +328,16 @@
 	_flush_cache_mm = r3k_flush_cache_mm;
 	_flush_cache_range = r3k_flush_cache_range;
 	_flush_cache_page = r3k_flush_cache_page;
-	_flush_cache_sigtramp = r3k_flush_cache_sigtramp;
-	_flush_dcache_page = r3k_flush_dcache_page;
 	_flush_icache_page = r3k_flush_icache_page;
 	_flush_icache_range = r3k_flush_icache_range;
 
+	_flush_cache_sigtramp = r3k_flush_cache_sigtramp;
+	_flush_data_cache_page = r3k_flush_data_cache_page;
+
 	_dma_cache_wback_inv = r3k_dma_cache_wback_inv;
 
 	printk("Primary instruction cache %dkb, linesize %d bytes\n",
 		(int) (icache_size >> 10), (int) icache_lsize);
 	printk("Primary data cache %dkb, linesize %d bytes\n",
 		(int) (dcache_size >> 10), (int) dcache_lsize);
-
 }
mips/mm/c-r4k.c 31 Mar 2003 11:36:04 -0000 renamed from mips/mm/c-r4k.c 29 Mar 2003 00:48:52 -0000 +12 -35

81 lines not on this page · raw message

mips/mm/c-r5432.c 31 Mar 2003 11:36:05 -0000 renamed from mips/mm/c-r5432.c 25 Mar 2003 01:11:34 -0000 +11 -36

80 lines not on this page · raw message

mips/mm/c-rm7k.c 31 Mar 2003 11:36:05 -0000 renamed from mips/mm/c-rm7k.c 24 Mar 2003 23:49:27 -0000 +6 -8
diff -u -r1.4.2.9 c-rm7k.c
@@ -90,7 +90,7 @@
 {
 }
 
-static void rm7k_flush_dcache_page(struct page * page)
+static void rm7k_flush_data_cache_page(unsigned long addr)
 {
 }
 
@@ -187,11 +187,6 @@
 	protected_flush_icache_line(addr & ~(ic_lsize - 1));
 }
 
-void __update_cache(struct vm_area_struct *vma, unsigned long address,
-	pte_t pte)
-{
-}
-
 /* Detect and size the caches. */
 static inline void probe_icache(unsigned long config)
 {
@@ -336,10 +331,13 @@
 	_flush_cache_mm = rm7k_flush_cache_mm_d32i32;
 	_flush_cache_range = rm7k_flush_cache_range_d32i32;
 	_flush_cache_page = rm7k_flush_cache_page_d32i32;
-	_flush_dcache_page = rm7k_flush_dcache_page;
-	_flush_cache_sigtramp = rm7k_flush_cache_sigtramp;
 	_flush_icache_range = rm7k_flush_icache_range;
 	_flush_icache_page = rm7k_flush_icache_page;
+
+	_flush_cache_sigtramp = rm7k_flush_cache_sigtramp;
+	_flush_data_cache_page = rm7k_flush_data_cache_page;
+
+	shm_align_mask = PAGE_SIZE - 1;
 
 	_dma_cache_wback_inv = rm7k_dma_cache_wback_inv;
 	_dma_cache_wback = rm7k_dma_cache_wback;
mips/mm/c-sb1.c 31 Mar 2003 11:36:07 -0000 renamed from mips/mm/c-sb1.c 12 Mar 2003 16:39:54 -0000 +2 -6
diff -u -r1.11.2.22 c-sb1.c
@@ -420,11 +420,6 @@
 {
 }
 
-void __update_cache(struct vm_area_struct *vma, unsigned long address,
-	pte_t pte)
-{
-}
-
 /*
  *  Cache set values (from the mips64 spec)
  * 0 - 64
@@ -534,13 +529,14 @@
 	_flush_cache_page = (void (*)(struct vm_area_struct *, unsigned long))sb1_nop;
 	_flush_cache_mm = (void (*)(struct mm_struct *))sb1_nop;
 	_flush_cache_all = sb1_nop;
-	_flush_dcache_page = (void *) sb1_nop;
 
 	/* These routines are for Icache coherence with the Dcache */
 	_flush_icache_range = sb1_flush_icache_range;
 	_flush_icache_page = sb1_flush_icache_page;
 	_flush_icache_all = __sb1_flush_icache_all; /* local only */
+
 	_flush_cache_sigtramp = sb1_flush_cache_sigtramp;
+	_flush_data_cache_page = (void *) sb1_nop;
 
 	/* Full flush */
 	___flush_cache_all = sb1___flush_cache_all;
mips/mm/c-tx39.c 31 Mar 2003 11:36:07 -0000 renamed from mips/mm/c-tx39.c 25 Mar 2003 14:30:19 -0000 +16 -40

99 lines not on this page · raw message

mips/mm/c-tx49.c 31 Mar 2003 11:36:07 -0000 renamed from mips/mm/c-tx49.c 25 Mar 2003 01:11:34 -0000 +9 -46

100 lines not on this page · raw message

mips/mm/cache.c 31 Mar 2003 11:36:07 -0000 renamed from mips/mm/cache.c 1 Jan 1970 00:00:00 -0000 +57 -0

57 lines not on this page · raw message

mips/mm/init.c 31 Mar 2003 11:36:10 -0000 renamed from mips/mm/init.c 5 Aug 2002 23:53:35 -0000 +2 -10
diff -u -r1.38.2.7 init.c
@@ -45,15 +45,6 @@
 
 extern void prom_free_prom_memory(void);
 
-
-asmlinkage int sys_cacheflush(void *addr, int bytes, int cache)
-{
-	/* This should flush more selectivly ...  */
-	__flush_cache_all();
-
-	return 0;
-}
-
 /*
  * We have upto 8 empty zeroed pages so we can map one of the right colour
  * when needed.  This is necessary only on R4000 / R4400 SC and MC versions
@@ -67,7 +58,8 @@
 {
 	unsigned long order, size;
 	struct page *page;
-	if(mips_cpu.options & MIPS_CPU_VCE)
+
+	if (mips_cpu.options & MIPS_CPU_VCE)
 		order = 3;
 	else
 		order = 0;
mips/mm/loadmmu.c 31 Mar 2003 11:36:10 -0000 renamed from mips/mm/loadmmu.c 12 Mar 2003 15:06:35 -0000 +2 -2
diff -u -r1.45.2.2 loadmmu.c
@@ -29,11 +29,11 @@
 void (*_flush_cache_range)(struct mm_struct *mm, unsigned long start,
 			  unsigned long end);
 void (*_flush_cache_page)(struct vm_area_struct *vma, unsigned long page);
-void (*_flush_cache_sigtramp)(unsigned long addr);
 void (*_flush_icache_range)(unsigned long start, unsigned long end);
 void (*_flush_icache_page)(struct vm_area_struct *vma, struct page *page);
 
-void (*_flush_dcache_page)(struct page * page);
+void (*_flush_cache_sigtramp)(unsigned long addr);
+void (*_flush_data_cache_page)(unsigned long addr);
 void (*_flush_icache_all)(void);
 
 #ifdef CONFIG_NONCOHERENT_IO
asm-mips/cacheflush.h 31 Mar 2003 11:36:30 -0000 renamed from asm-mips/cacheflush.h 1 Jan 1970 00:00:00 -0000 +69 -0

69 lines not on this page · raw message

asm-mips/page.h 31 Mar 2003 11:36:30 -0000 renamed from asm-mips/page.h 28 Mar 2003 19:29:53 -0000 +30 -4
diff -u -r1.14.2.12 page.h
@@ -10,8 +10,6 @@
 #ifndef __ASM_PAGE_H
 #define __ASM_PAGE_H
 
-#include <linux/config.h>
-
 /* PAGE_SHIFT determines the page size */
 #define PAGE_SHIFT	12
 #define PAGE_SIZE	(1L << PAGE_SHIFT)
@@ -21,6 +19,10 @@
 
 #ifndef __ASSEMBLY__
 
+#include <linux/config.h>
+
+#include <asm/cacheflush.h>
+
 #define BUG() do { printk("kernel BUG at %s:%d!\n", __FILE__, __LINE__); *(int *)0=0; } while (0)
 #define PAGE_BUG(page) do {  BUG(); } while (0)
 
@@ -64,8 +66,32 @@
 
 #define clear_page(page)	_clear_page(page)
 #define copy_page(to, from)	_copy_page(to, from)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
+
+extern unsigned long shm_align_mask;
+
+static inline unsigned long pages_do_alias(unsigned long addr1,
+	unsigned long addr2)
+{
+	return (addr1 ^ addr2) & shm_align_mask;
+}
+
+static inline void clear_user_page(void *page, unsigned long vaddr)
+{
+	unsigned long kaddr = (unsigned long) page;
+
+	clear_page(page);
+	if (pages_do_alias(kaddr, vaddr))
+		flush_data_cache_page(kaddr);
+}
+
+static inline void copy_user_page(void * to, void * from, unsigned long vaddr)
+{
+	unsigned long kto = (unsigned long) to;
+
+	copy_page(to, from);
+	if (pages_do_alias(kto, vaddr))
+		flush_data_cache_page(kto);
+}
 
 /*
  * These are used to make use of C type-checking..
asm-mips/pgtable.h 31 Mar 2003 11:36:31 -0000 renamed from asm-mips/pgtable.h 28 Mar 2003 19:29:53 -0000 +1 -47

56 lines not on this page · raw message