Skip to contentSkip to conversation

avoid unnecessary cache flushes

PATCH In review Paul Mackerras 5 participants 17 messages Aug 31, 2001 – Sep 4, 2001

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 avoid unnecessary cache flushes

On several RISC architectures the I-cache doesn't snoop, instead software is responsible for doing any I-cache flushing necessary to maintain coherency. On PowerPC for instance, the I-cache doesn't snoop at all; on SPARC-64 and IA64 the I-cache will snoop DMA activity but not the D-cache on the same CPU.

So we need to do a D-cache writeback and I-cache invalidate for a page under certain circumstances, and the current code does that. However, it does unnecessary flushes because it doesn't keep any state to indicate whether the flush has already been done for a page. Clearly, once the flush has been done for a page it doesn't need to be done again unless the page is modified.

The solution that Dave M. and others came up with is to use the PG_arch_1 bit in the page struct to indicate whether the page is "I-cache clean" or not. This can mostly be done in the arch-dependent code but we need a couple of changes in the generic code as well.

The main extra thing we need is for copy_user_page and clear_user_page to be able to mark the page as I-cache dirty. To do this these functions need the struct page * for the destination page. When I suggested adding the struct page * argument previously, Linus responded with some comments which I have taken into account.

So here is a proposed patch against 2.4.10-pre2. We get a significant performance increase on PPC with this patch. In summary, the changes it makes are:

* copy_user_page and clear_user_page take page * arguments instead of void *.

* If an architecture needs its own versions of copy/clear_user_page, it should define __HAVE_ARCH_USER_PAGE in asm/page.h and declare copy/clear_user_page there.

* There are default versions of copy/clear_user_page in include/linux/highmem.h which will be used if the architecture doesn't define __HAVE_ARCH_USER_PAGE.

* Some functions have been renamed:

	clear_user_highpage	-> clear_user_page
	clear_highpage		-> clear_mem_page
	copy_user_highpage	-> copy_user_page
	copy_highpage		-> copy_mem_page
	memclear_highpage	-> memclear_page
	memclear_highpage_flush -> memclear_page_flush

This was prompted by Linus' comment that the highpage versions should be renamed to the non-highpage names, since highmem is no longer a rare special case.

* access_one_page() in kernel/ptrace.c uses flush_icache_range in the write case rather than flush_icache_page. This is because (a) we now have flush_icache_page being a no-op on PPC and (b) flushing a whole page when we typically only modify one word is way overkill.

This patch should be completely benign on those architectures that don't need to worry about I-cache coherency (e.g. i386).

Any comments from the architecture maintainers? Linus, does this look OK to apply to your tree?

Paul.

Documentation/cachetlb.txt +8 -2
diff -urN linux/Documentation/cachetlb.txt linux-cfa/Documentation/cachetlb.txt
@@ -260,8 +260,9 @@
 
 Here is the new interface:
 
-  void copy_user_page(void *to, void *from, unsigned long address)
-  void clear_user_page(void *to, unsigned long address)
+  void copy_user_page(struct page *to, struct page *from,
+		      unsigned long address)
+  void clear_user_page(struct page *to, unsigned long address)
 
 	These two routines store data in user anonymous or COW
 	pages.  It allows a port to efficiently avoid D-cache alias
@@ -279,6 +280,11 @@
 
 	If D-cache aliasing is not an issue, these two routines may
 	simply call memcpy/memset directly and do nothing more.
+
+	There are default versions of these procedures supplied in
+	include/linux/highmem.h.  If a port does not want to use the
+	default versions it should declare them and define the symbol
+	__HAVE_ARCH_USER_PAGE in include/asm/page.h.
 
   void flush_dcache_page(struct page *page)
 
arch/ppc/kernel/misc.S +2 -24
diff -urN linux/arch/ppc/kernel/misc.S linux-cfa/arch/ppc/kernel/misc.S
@@ -475,9 +475,9 @@
  * snoop from the data cache.
  * This is a no-op on the 601 which has a unified cache.
  *
- *	void __flush_page_to_ram(void *page)
+ *	void __flush_dcache_icache(void *page)
  */
-_GLOBAL(__flush_page_to_ram)
+_GLOBAL(__flush_dcache_icache)
 	mfspr	r5,PVR
 	rlwinm	r5,r5,16,16,31
 	cmpi	0,r5,1
@@ -493,28 +493,6 @@
 	mtctr	r4
 1:	icbi	0,r6
 	addi	r6,r6,CACHE_LINE_SIZE
-	bdnz	1b
-	sync
-	isync
-	blr
-
-/*
- * Flush a particular page from the instruction cache.
- * Note: this is necessary because the instruction cache does *not*
- * snoop from the data cache.
- * This is a no-op on the 601 which has a unified cache.
- *
- *	void __flush_icache_page(void *page)
- */
-_GLOBAL(__flush_icache_page)
-	mfspr	r5,PVR
-	rlwinm	r5,r5,16,16,31
-	cmpi	0,r5,1
-	beqlr				/* for 601, do nothing */
-	li	r4,4096/CACHE_LINE_SIZE	/* Number of lines in a page */
-	mtctr	r4
-1:	icbi	0,r3
-	addi	r3,r3,CACHE_LINE_SIZE
 	bdnz	1b
 	sync
 	isync
arch/ppc/kernel/ppc_ksyms.c +1 -0
diff -urN linux/arch/ppc/kernel/ppc_ksyms.c linux-cfa/arch/ppc/kernel/ppc_ksyms.c
@@ -185,6 +185,7 @@
 EXPORT_SYMBOL(enable_kernel_fp);
 EXPORT_SYMBOL(flush_icache_range);
 EXPORT_SYMBOL(flush_dcache_range);
+EXPORT_SYMBOL(flush_dcache_page);
 EXPORT_SYMBOL(xchg_u32);
 #ifdef CONFIG_ALTIVEC
 EXPORT_SYMBOL(last_task_used_altivec);
arch/ppc/mm/init.c +31 -4
diff -urN linux/arch/ppc/mm/init.c linux-cfa/arch/ppc/mm/init.c
@@ -578,17 +578,23 @@
 	mem_pieces_remove(&phys_avail, start, size, 1);
 }
 
-void flush_page_to_ram(struct page *page)
+/*
+ * This is called when a page has been modified by the kernel.
+ * It just marks the page as not i-cache clean.  We do the i-cache
+ * flush later when the page is given to a user process, if necessary.
+ */
+void flush_dcache_page(struct page *page)
 {
-	unsigned long vaddr = (unsigned long) kmap(page);
-	__flush_page_to_ram(vaddr);
-	kunmap(page);
+	clear_bit(PG_arch_1, &page->flags);
 }
 
 /*
  * set_pte stores a linux PTE into the linux page table.
  * On machines which use an MMU hash table we avoid changing the
  * _PAGE_HASHPTE bit.
+ * If the new PTE has _PAGE_EXEC set, meaning that the user wants
+ * to be able to execute out of the page, we check if the page is
+ * i-cache dirty and flush it if so, and mark it clean.
  */
 void set_pte(pte_t *ptep, pte_t pte)
 {
@@ -597,4 +603,25 @@
 #else
 	*ptep = pte;
 #endif
+	if (mem_init_done && (pte_val(pte) & _PAGE_EXEC)
+	    && (pte_val(pte) & PAGE_MASK) < total_memory) {
+		struct page *page = pte_page(pte);
+		if (!test_bit(PG_arch_1, &page->flags)) {
+			__flush_dcache_icache((unsigned long)kmap(page));
+			kunmap(page);
+			set_bit(PG_arch_1, &page->flags);
+		}
+	}
+}
+
+void clear_user_page(struct page *page, unsigned long vaddr)
+{
+	clear_mem_page(page);
+	clear_bit(PG_arch_1, &page->flags);
+}
+
+void copy_user_page(struct page *to, struct page *from, unsigned long vaddr)
+{
+	copy_mem_page(to, from);
+	clear_bit(PG_arch_1, &to->flags);
 }
include/asm-alpha/page.h +0 -3
diff -urN linux/include/asm-alpha/page.h linux-cfa/include/asm-alpha/page.h
@@ -15,10 +15,7 @@
 #define STRICT_MM_TYPECHECKS
 
 extern void clear_page(void *page);
-#define clear_user_page(page, vaddr)	clear_page(page)
-
 extern void copy_page(void * _to, void * _from);
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
 
 #ifdef STRICT_MM_TYPECHECKS
 /*
include/asm-arm/page.h +0 -3
diff -urN linux/include/asm-arm/page.h linux-cfa/include/asm-arm/page.h
@@ -14,9 +14,6 @@
 #define clear_page(page)	memzero((void *)(page), PAGE_SIZE)
 extern void copy_page(void *to, void *from);
 
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
-
 #ifdef STRICT_MM_TYPECHECKS
 /*
  * These are used to make use of C type-checking..
include/asm-cris/page.h +0 -3
diff -urN linux/include/asm-cris/page.h linux-cfa/include/asm-cris/page.h
@@ -14,9 +14,6 @@
 #define clear_page(page)        memset((void *)(page), 0, PAGE_SIZE)
 #define copy_page(to,from)      memcpy((void *)(to), (void *)(from), PAGE_SIZE)
 
-#define clear_user_page(page, vaddr)    clear_page(page)
-#define copy_user_page(to, from, vaddr) copy_page(to, from)
-
 #define STRICT_MM_TYPECHECKS
 
 #ifdef STRICT_MM_TYPECHECKS
include/asm-ia64/pgalloc.h +7 -5
diff -urN linux/include/asm-ia64/pgalloc.h linux-cfa/include/asm-ia64/pgalloc.h
@@ -234,18 +234,20 @@
 	clear_bit(PG_arch_1, &page->flags);
 }
 
+#define __HAVE_ARCH_USER_PAGE
+
 static inline void
-clear_user_page (void *addr, unsigned long vaddr, struct page *page)
+clear_user_page (struct page *page, unsigned long vaddr)
 {
-	clear_page(addr);
+	clear_mem_page(page);
 	flush_dcache_page(page);
 }
 
 static inline void
-copy_user_page (void *to, void *from, unsigned long vaddr, struct page *page)
+copy_user_page (struct page *to, struct page *from, unsigned long vaddr)
 {
-	copy_page(to, from);
-	flush_dcache_page(page);
+	copy_mem_page(to, from);
+	flush_dcache_page(to);
 }
 
 /*
include/asm-m68k/page.h +0 -3
diff -urN linux/include/asm-m68k/page.h linux-cfa/include/asm-m68k/page.h
@@ -76,9 +76,6 @@
 #define copy_page(to,from)	memcpy((to), (from), PAGE_SIZE)
 #endif
 
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
-
 /*
  * These are used to make use of C type-checking..
  */
include/asm-mips/page.h +0 -2
diff -urN linux/include/asm-mips/page.h linux-cfa/include/asm-mips/page.h
@@ -28,8 +28,6 @@
 
 #define clear_page(page)	_clear_page(page)
 #define copy_page(to, from)	_copy_page(to, from)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
 
 /*
  * These are used to make use of C type-checking..
include/asm-mips64/page.h +0 -2
diff -urN linux/include/asm-mips64/page.h linux-cfa/include/asm-mips64/page.h
@@ -28,8 +28,6 @@
 
 #define clear_page(page)	_clear_page(page)
 #define copy_page(to, from)	_copy_page(to, from)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
 
 /*
  * These are used to make use of C type-checking..
include/asm-parisc/page.h +0 -3
diff -urN linux/include/asm-parisc/page.h linux-cfa/include/asm-parisc/page.h
@@ -12,9 +12,6 @@
 #define clear_page(page)	memset((void *)(page), 0, PAGE_SIZE)
 #define copy_page(to,from)	memcpy((void *)(to), (void *)(from), PAGE_SIZE)
 
-#define clear_user_page(page, vaddr) clear_page(page)
-#define copy_user_page(to, from, vaddr) copy_page(to, from)
-
 /*
  * These are used to make use of C type-checking..
  */
include/asm-ppc/page.h +5 -2
diff -urN linux/include/asm-ppc/page.h linux-cfa/include/asm-ppc/page.h
@@ -83,8 +83,11 @@
 
 extern void clear_page(void *page);
 extern void copy_page(void *to, void *from);
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
+
+#define __HAVE_ARCH_USER_PAGE
+struct page;
+extern void clear_user_page(struct page *page, unsigned long vaddr);
+extern void copy_user_page(struct page *to, struct page *from, unsigned long vaddr);
 
 /* map phys->virtual and virtual->phys for RAM pages */
 static inline unsigned long ___pa(unsigned long v)
include/asm-ppc/pgtable.h +7 -4
diff -urN linux/include/asm-ppc/pgtable.h linux-cfa/include/asm-ppc/pgtable.h
@@ -83,12 +83,11 @@
 #define flush_cache_range(mm, a, b)	do { } while (0)
 #define flush_cache_page(vma, p)	do { } while (0)
 #define flush_icache_page(vma, page)	do { } while (0)
+#define flush_page_to_ram(page)		do { } while (0)
 
 extern void flush_icache_range(unsigned long, unsigned long);
-extern void __flush_page_to_ram(unsigned long page_va);
-extern void flush_page_to_ram(struct page *page);
-
-#define flush_dcache_page(page)			do { } while (0)
+extern void __flush_dcache_icache(unsigned long page_va);
+extern void flush_dcache_page(struct page *page);
 
 extern unsigned long va_to_phys(unsigned long address);
 extern pte_t *va_to_pte(unsigned long address);
@@ -421,6 +420,10 @@
 }
 
 /*
+ * When a new value is written into a PTE, we may need to flush the
+ * i-cache for the page of memory that the PTE points to.
+ * (Note: machines with software TLB reloads could do the flush in
+ * the instruction TLB miss handler instead.)
  * Writing a new value into the PTE doesn't disturb the state of the
  * _PAGE_HASHPTE bit, on those machines which use an MMU hash table.
  */
include/asm-s390/page.h +0 -3
diff -urN linux/include/asm-s390/page.h linux-cfa/include/asm-s390/page.h
@@ -59,9 +59,6 @@
 			      : "memory" );
 }
 
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
-
 #define BUG() do { \
         printk("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \
         __asm__ __volatile__(".word 0x0000"); \
include/asm-s390x/page.h +0 -3
diff -urN linux/include/asm-s390x/page.h linux-cfa/include/asm-s390x/page.h
@@ -57,9 +57,6 @@
 			      : "memory" );
 }
 
-#define clear_user_page(page, vaddr)    clear_page(page)
-#define copy_user_page(to, from, vaddr) copy_page(to, from)
-
 #define BUG() do { \
         printk("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \
         __asm__ __volatile__(".long 0"); \
include/asm-sh/page.h +6 -6
diff -urN linux/include/asm-sh/page.h linux-cfa/include/asm-sh/page.h
@@ -27,12 +27,12 @@
 #define clear_page(page)	memset((void *)(page), 0, PAGE_SIZE)
 #define copy_page(to,from)	memcpy((void *)(to), (void *)(from), PAGE_SIZE)
 
-#if defined(__sh3__)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
-#elif defined(__SH4__)
-extern void clear_user_page(void *to, unsigned long address);
-extern void copy_user_page(void *to, void *from, unsigned long address);
+#if defined(__SH4__)
+#define __HAVE_ARCH_USER_PAGE
+struct page;
+extern void clear_user_page(struct page *to, unsigned long address);
+extern void copy_user_page(struct page *to, struct page *from,
+			   unsigned long address);
 #endif
 
 /*
include/asm-sparc/page.h +0 -2
diff -urN linux/include/asm-sparc/page.h linux-cfa/include/asm-sparc/page.h
@@ -54,8 +54,6 @@
 
 #define clear_page(page)	 memset((void *)(page), 0, PAGE_SIZE)
 #define copy_page(to,from) 	memcpy((void *)(to), (void *)(from), PAGE_SIZE)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
 
 /* The following structure is used to hold the physical
  * memory configuration of the machine.  This is filled in
include/asm-sparc64/page.h +5 -2
diff -urN linux/include/asm-sparc64/page.h linux-cfa/include/asm-sparc64/page.h
@@ -25,8 +25,11 @@
 extern void _copy_page(void *to, void *from);
 #define clear_page(X)	_clear_page((void *)(X))
 #define copy_page(X,Y)	_copy_page((void *)(X), (void *)(Y))
-extern void clear_user_page(void *page, unsigned long vaddr);
-extern void copy_user_page(void *to, void *from, unsigned long vaddr);
+
+#define __HAVE_ARCH_USER_PAGE
+struct page;
+extern void clear_user_page(struct page *page, unsigned long vaddr);
+extern void copy_user_page(struct page *to, struct page *from, unsigned long vaddr);
 
 /* GROSS, defining this makes gcc pass these types as aggregates,
  * and thus on the stack, turn this crap off... -DaveM
include/linux/highmem.h +12 -7
diff -urN linux/include/linux/highmem.h linux-cfa/include/linux/highmem.h
@@ -43,20 +43,23 @@
 #endif /* CONFIG_HIGHMEM */
 
 /* when CONFIG_HIGHMEM is not set these will be plain clear/copy_page */
-static inline void clear_user_highpage(struct page *page, unsigned long vaddr)
+
+#ifndef __HAVE_ARCH_USER_PAGE
+static inline void clear_user_page(struct page *page, unsigned long vaddr)
 {
 	void *addr = kmap_atomic(page, KM_USER0);
-	clear_user_page(addr, vaddr);
+	clear_page(addr, vaddr);
 	kunmap_atomic(addr, KM_USER0);
 }
+#endif /* __HAVE_ARCH_USER_PAGE */
 
-static inline void clear_highpage(struct page *page)
+static inline void clear_mem_page(struct page *page)
 {
 	clear_page(kmap(page));
 	kunmap(page);
 }
 
-static inline void memclear_highpage(struct page *page, unsigned int offset, unsigned int size)
+static inline void memclear_page(struct page *page, unsigned int offset, unsigned int size)
 {
 	char *kaddr;
 
@@ -70,7 +73,7 @@
 /*
  * Same but also flushes aliased cache contents to RAM.
  */
-static inline void memclear_highpage_flush(struct page *page, unsigned int offset, unsigned int size)
+static inline void memclear_page_flush(struct page *page, unsigned int offset, unsigned int size)
 {
 	char *kaddr;
 
@@ -82,7 +85,8 @@
 	kunmap(page);
 }
 
-static inline void copy_user_highpage(struct page *to, struct page *from, unsigned long vaddr)
+#ifndef __HAVE_ARCH_USER_PAGE
+static inline void copy_user_page(struct page *to, struct page *from, unsigned long vaddr)
 {
 	char *vfrom, *vto;
 
@@ -92,8 +96,9 @@
 	kunmap_atomic(vfrom, KM_USER0);
 	kunmap_atomic(vto, KM_USER1);
 }
+#endif
 
-static inline void copy_highpage(struct page *to, struct page *from)
+static inline void copy_mem_page(struct page *to, struct page *from)
 {
 	char *vfrom, *vto;
 
kernel/ptrace.c +4 -3
diff -urN linux/kernel/ptrace.c linux-cfa/kernel/ptrace.c
@@ -103,10 +103,11 @@
 	flush_cache_page(vma, addr);
 
 	if (write) {
-		maddr = kmap(page);
-		memcpy(maddr + (addr & ~PAGE_MASK), buf, len);
+		maddr = kmap(page) + (addr & ~PAGE_MASK);
+		memcpy(maddr, buf, len);
 		flush_page_to_ram(page);
-		flush_icache_page(vma, page);
+		flush_icache_range((unsigned long) maddr,
+				   (unsigned long) maddr + len);
 		kunmap(page);
 	} else {
 		maddr = kmap(page);
mm/filemap.c +2 -2
diff -urN linux/mm/filemap.c linux-cfa/mm/filemap.c
@@ -193,7 +193,7 @@
 
 static inline void truncate_partial_page(struct page *page, unsigned partial)
 {
-	memclear_highpage_flush(page, partial, PAGE_CACHE_SIZE-partial);
+	memclear_page_flush(page, partial, PAGE_CACHE_SIZE-partial);
 				
 	if (page->buffers)
 		block_flushpage(page, partial);
@@ -1477,7 +1477,7 @@
 		struct page *new_page = alloc_page(GFP_HIGHUSER);
 
 		if (new_page) {
-			copy_user_highpage(new_page, old_page, address);
+			copy_user_page(new_page, old_page, address);
 			flush_page_to_ram(new_page);
 		} else
 			new_page = NOPAGE_OOM;
mm/memory.c +3 -3
diff -urN linux/mm/memory.c linux-cfa/mm/memory.c
@@ -61,10 +61,10 @@
 static inline void copy_cow_page(struct page * from, struct page * to, unsigned long address)
 {
 	if (from == ZERO_PAGE(address)) {
-		clear_user_highpage(to, address);
+		clear_user_page(to, address);
 		return;
 	}
-	copy_user_highpage(to, from, address);
+	copy_user_page(to, from, address);
 }
 
 mem_map_t * mem_map;
@@ -1186,7 +1186,7 @@
 		page = alloc_page(GFP_HIGHUSER);
 		if (!page)
 			goto no_mem;
-		clear_user_highpage(page, addr);
+		clear_user_page(page, addr);
 
 		spin_lock(&mm->page_table_lock);
 		if (!pte_none(*page_table)) {
mm/page_alloc.c +2 -3
diff -urN linux/mm/page_alloc.c linux-cfa/mm/page_alloc.c
@@ -526,9 +526,8 @@
 
 	page = alloc_pages(gfp_mask, 0);
 	if (page) {
-		void *address = page_address(page);
-		clear_page(address);
-		return (unsigned long) address;
+		clear_mem_page(page);
+		return (unsigned long) page_address(page);
 	}
 	return 0;
 }
mm/shmem.c +2 -2
diff -urN linux/mm/shmem.c linux-cfa/mm/shmem.c
@@ -376,7 +376,7 @@
 		page = page_cache_alloc(mapping);
 		if (!page)
 			return ERR_PTR(-ENOMEM);
-		clear_highpage(page);
+		clear_mem_page(page);
 		inode->i_blocks += BLOCKS_PER_PAGE;
 		add_to_page_cache (page, mapping, idx);
 	}
@@ -439,7 +439,7 @@
 		struct page *new_page = page_cache_alloc(inode->i_mapping);
 
 		if (new_page) {
-			copy_user_highpage(new_page, page, address);
+			copy_user_page(new_page, page, address);
 			flush_page_to_ram(new_page);
 		} else
 			new_page = NOPAGE_OOM;
#1 lore raw

From: Paul Mackerras <[email protected]>
Date: Fri, 31 Aug 2001 21:18:50 +1000 (EST)

Any comments from the architecture maintainers? Linus, does this look OK to apply to your tree?

No comments other than it will silently break sparc64 as you've updated the declarations in the asm-sparc64 headers but failed to fixup the assembler routine itself to expect the page * arg.

I would suggest instead to change the name of the assembler routine to __copy_user_page et al. and make copy_user_page just an inline or define which plucks out page->address and passes that onto __copy_user_page. This way you require no knowledge of Sparc assembly whatsoever.

Later,
David S. Miller
[email protected]

#2 lore raw
Quoted 4 lines · David S. Miller

I would suggest instead to change the name of the assembler routine to __copy_user_page et al. and make copy_user_page just an inline or define which plucks out page->address and passes that onto __copy_user_page.

Good idea, thanks. Here is a new patch - does it look OK to you? Linus, any comment?

Quoted 2 lines

This way you require no knowledge
of Sparc assembly whatsoever.

Well, I have done sparc assembly in my time (remember Dave Sitsky and I did a port of the kernel to the ultrasparc running in 32-bit mode before you did the sparc64 port) but the stuff you're doing in there isn't just assembly, it's magic assembly. ;)

Regards,
Paul.

Documentation/cachetlb.txt +8 -2
diff -urN linux/Documentation/cachetlb.txt linux-cfa/Documentation/cachetlb.txt
@@ -260,8 +260,9 @@
 
 Here is the new interface:
 
-  void copy_user_page(void *to, void *from, unsigned long address)
-  void clear_user_page(void *to, unsigned long address)
+  void copy_user_page(struct page *to, struct page *from,
+		      unsigned long address)
+  void clear_user_page(struct page *to, unsigned long address)
 
 	These two routines store data in user anonymous or COW
 	pages.  It allows a port to efficiently avoid D-cache alias
@@ -279,6 +280,11 @@
 
 	If D-cache aliasing is not an issue, these two routines may
 	simply call memcpy/memset directly and do nothing more.
+
+	There are default versions of these procedures supplied in
+	include/linux/highmem.h.  If a port does not want to use the
+	default versions it should declare them and define the symbol
+	__HAVE_ARCH_USER_PAGE in include/asm/page.h.
 
   void flush_dcache_page(struct page *page)
 
arch/ppc/kernel/misc.S +2 -24
diff -urN linux/arch/ppc/kernel/misc.S linux-cfa/arch/ppc/kernel/misc.S
@@ -475,9 +475,9 @@
  * snoop from the data cache.
  * This is a no-op on the 601 which has a unified cache.
  *
- *	void __flush_page_to_ram(void *page)
+ *	void __flush_dcache_icache(void *page)
  */
-_GLOBAL(__flush_page_to_ram)
+_GLOBAL(__flush_dcache_icache)
 	mfspr	r5,PVR
 	rlwinm	r5,r5,16,16,31
 	cmpi	0,r5,1
@@ -493,28 +493,6 @@
 	mtctr	r4
 1:	icbi	0,r6
 	addi	r6,r6,CACHE_LINE_SIZE
-	bdnz	1b
-	sync
-	isync
-	blr
-
-/*
- * Flush a particular page from the instruction cache.
- * Note: this is necessary because the instruction cache does *not*
- * snoop from the data cache.
- * This is a no-op on the 601 which has a unified cache.
- *
- *	void __flush_icache_page(void *page)
- */
-_GLOBAL(__flush_icache_page)
-	mfspr	r5,PVR
-	rlwinm	r5,r5,16,16,31
-	cmpi	0,r5,1
-	beqlr				/* for 601, do nothing */
-	li	r4,4096/CACHE_LINE_SIZE	/* Number of lines in a page */
-	mtctr	r4
-1:	icbi	0,r3
-	addi	r3,r3,CACHE_LINE_SIZE
 	bdnz	1b
 	sync
 	isync
arch/ppc/kernel/ppc_ksyms.c +1 -0
diff -urN linux/arch/ppc/kernel/ppc_ksyms.c linux-cfa/arch/ppc/kernel/ppc_ksyms.c
@@ -185,6 +185,7 @@
 EXPORT_SYMBOL(enable_kernel_fp);
 EXPORT_SYMBOL(flush_icache_range);
 EXPORT_SYMBOL(flush_dcache_range);
+EXPORT_SYMBOL(flush_dcache_page);
 EXPORT_SYMBOL(xchg_u32);
 #ifdef CONFIG_ALTIVEC
 EXPORT_SYMBOL(last_task_used_altivec);
arch/ppc/mm/init.c +31 -4
diff -urN linux/arch/ppc/mm/init.c linux-cfa/arch/ppc/mm/init.c
@@ -578,17 +578,23 @@
 	mem_pieces_remove(&phys_avail, start, size, 1);
 }
 
-void flush_page_to_ram(struct page *page)
+/*
+ * This is called when a page has been modified by the kernel.
+ * It just marks the page as not i-cache clean.  We do the i-cache
+ * flush later when the page is given to a user process, if necessary.
+ */
+void flush_dcache_page(struct page *page)
 {
-	unsigned long vaddr = (unsigned long) kmap(page);
-	__flush_page_to_ram(vaddr);
-	kunmap(page);
+	clear_bit(PG_arch_1, &page->flags);
 }
 
 /*
  * set_pte stores a linux PTE into the linux page table.
  * On machines which use an MMU hash table we avoid changing the
  * _PAGE_HASHPTE bit.
+ * If the new PTE has _PAGE_EXEC set, meaning that the user wants
+ * to be able to execute out of the page, we check if the page is
+ * i-cache dirty and flush it if so, and mark it clean.
  */
 void set_pte(pte_t *ptep, pte_t pte)
 {
@@ -597,4 +603,25 @@
 #else
 	*ptep = pte;
 #endif
+	if (mem_init_done && (pte_val(pte) & _PAGE_EXEC)
+	    && (pte_val(pte) & PAGE_MASK) < total_memory) {
+		struct page *page = pte_page(pte);
+		if (!test_bit(PG_arch_1, &page->flags)) {
+			__flush_dcache_icache((unsigned long)kmap(page));
+			kunmap(page);
+			set_bit(PG_arch_1, &page->flags);
+		}
+	}
+}
+
+void clear_user_page(struct page *page, unsigned long vaddr)
+{
+	clear_mem_page(page);
+	clear_bit(PG_arch_1, &page->flags);
+}
+
+void copy_user_page(struct page *to, struct page *from, unsigned long vaddr)
+{
+	copy_mem_page(to, from);
+	clear_bit(PG_arch_1, &to->flags);
 }
include/asm-alpha/page.h +0 -3
diff -urN linux/include/asm-alpha/page.h linux-cfa/include/asm-alpha/page.h
@@ -15,10 +15,7 @@
 #define STRICT_MM_TYPECHECKS
 
 extern void clear_page(void *page);
-#define clear_user_page(page, vaddr)	clear_page(page)
-
 extern void copy_page(void * _to, void * _from);
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
 
 #ifdef STRICT_MM_TYPECHECKS
 /*
include/asm-arm/page.h +0 -3
diff -urN linux/include/asm-arm/page.h linux-cfa/include/asm-arm/page.h
@@ -14,9 +14,6 @@
 #define clear_page(page)	memzero((void *)(page), PAGE_SIZE)
 extern void copy_page(void *to, void *from);
 
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
-
 #ifdef STRICT_MM_TYPECHECKS
 /*
  * These are used to make use of C type-checking..
include/asm-cris/page.h +0 -3
diff -urN linux/include/asm-cris/page.h linux-cfa/include/asm-cris/page.h
@@ -14,9 +14,6 @@
 #define clear_page(page)        memset((void *)(page), 0, PAGE_SIZE)
 #define copy_page(to,from)      memcpy((void *)(to), (void *)(from), PAGE_SIZE)
 
-#define clear_user_page(page, vaddr)    clear_page(page)
-#define copy_user_page(to, from, vaddr) copy_page(to, from)
-
 #define STRICT_MM_TYPECHECKS
 
 #ifdef STRICT_MM_TYPECHECKS
include/asm-ia64/pgalloc.h +7 -5
diff -urN linux/include/asm-ia64/pgalloc.h linux-cfa/include/asm-ia64/pgalloc.h
@@ -234,18 +234,20 @@
 	clear_bit(PG_arch_1, &page->flags);
 }
 
+#define __HAVE_ARCH_USER_PAGE
+
 static inline void
-clear_user_page (void *addr, unsigned long vaddr, struct page *page)
+clear_user_page (struct page *page, unsigned long vaddr)
 {
-	clear_page(addr);
+	clear_mem_page(page);
 	flush_dcache_page(page);
 }
 
 static inline void
-copy_user_page (void *to, void *from, unsigned long vaddr, struct page *page)
+copy_user_page (struct page *to, struct page *from, unsigned long vaddr)
 {
-	copy_page(to, from);
-	flush_dcache_page(page);
+	copy_mem_page(to, from);
+	flush_dcache_page(to);
 }
 
 /*
include/asm-m68k/page.h +0 -3
diff -urN linux/include/asm-m68k/page.h linux-cfa/include/asm-m68k/page.h
@@ -76,9 +76,6 @@
 #define copy_page(to,from)	memcpy((to), (from), PAGE_SIZE)
 #endif
 
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
-
 /*
  * These are used to make use of C type-checking..
  */
include/asm-mips/page.h +0 -2
diff -urN linux/include/asm-mips/page.h linux-cfa/include/asm-mips/page.h
@@ -28,8 +28,6 @@
 
 #define clear_page(page)	_clear_page(page)
 #define copy_page(to, from)	_copy_page(to, from)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
 
 /*
  * These are used to make use of C type-checking..
include/asm-mips64/page.h +0 -2
diff -urN linux/include/asm-mips64/page.h linux-cfa/include/asm-mips64/page.h
@@ -28,8 +28,6 @@
 
 #define clear_page(page)	_clear_page(page)
 #define copy_page(to, from)	_copy_page(to, from)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
 
 /*
  * These are used to make use of C type-checking..
include/asm-parisc/page.h +0 -3
diff -urN linux/include/asm-parisc/page.h linux-cfa/include/asm-parisc/page.h
@@ -12,9 +12,6 @@
 #define clear_page(page)	memset((void *)(page), 0, PAGE_SIZE)
 #define copy_page(to,from)	memcpy((void *)(to), (void *)(from), PAGE_SIZE)
 
-#define clear_user_page(page, vaddr) clear_page(page)
-#define copy_user_page(to, from, vaddr) copy_page(to, from)
-
 /*
  * These are used to make use of C type-checking..
  */
include/asm-ppc/page.h +5 -2
diff -urN linux/include/asm-ppc/page.h linux-cfa/include/asm-ppc/page.h
@@ -83,8 +83,11 @@
 
 extern void clear_page(void *page);
 extern void copy_page(void *to, void *from);
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
+
+#define __HAVE_ARCH_USER_PAGE
+struct page;
+extern void clear_user_page(struct page *page, unsigned long vaddr);
+extern void copy_user_page(struct page *to, struct page *from, unsigned long vaddr);
 
 /* map phys->virtual and virtual->phys for RAM pages */
 static inline unsigned long ___pa(unsigned long v)
include/asm-ppc/pgtable.h +7 -4
diff -urN linux/include/asm-ppc/pgtable.h linux-cfa/include/asm-ppc/pgtable.h
@@ -83,12 +83,11 @@
 #define flush_cache_range(mm, a, b)	do { } while (0)
 #define flush_cache_page(vma, p)	do { } while (0)
 #define flush_icache_page(vma, page)	do { } while (0)
+#define flush_page_to_ram(page)		do { } while (0)
 
 extern void flush_icache_range(unsigned long, unsigned long);
-extern void __flush_page_to_ram(unsigned long page_va);
-extern void flush_page_to_ram(struct page *page);
-
-#define flush_dcache_page(page)			do { } while (0)
+extern void __flush_dcache_icache(unsigned long page_va);
+extern void flush_dcache_page(struct page *page);
 
 extern unsigned long va_to_phys(unsigned long address);
 extern pte_t *va_to_pte(unsigned long address);
@@ -421,6 +420,10 @@
 }
 
 /*
+ * When a new value is written into a PTE, we may need to flush the
+ * i-cache for the page of memory that the PTE points to.
+ * (Note: machines with software TLB reloads could do the flush in
+ * the instruction TLB miss handler instead.)
  * Writing a new value into the PTE doesn't disturb the state of the
  * _PAGE_HASHPTE bit, on those machines which use an MMU hash table.
  */
include/asm-s390/page.h +0 -3
diff -urN linux/include/asm-s390/page.h linux-cfa/include/asm-s390/page.h
@@ -59,9 +59,6 @@
 			      : "memory" );
 }
 
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
-
 #define BUG() do { \
         printk("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \
         __asm__ __volatile__(".word 0x0000"); \
include/asm-s390x/page.h +0 -3
diff -urN linux/include/asm-s390x/page.h linux-cfa/include/asm-s390x/page.h
@@ -57,9 +57,6 @@
 			      : "memory" );
 }
 
-#define clear_user_page(page, vaddr)    clear_page(page)
-#define copy_user_page(to, from, vaddr) copy_page(to, from)
-
 #define BUG() do { \
         printk("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \
         __asm__ __volatile__(".long 0"); \
include/asm-sh/page.h +6 -6
diff -urN linux/include/asm-sh/page.h linux-cfa/include/asm-sh/page.h
@@ -27,12 +27,12 @@
 #define clear_page(page)	memset((void *)(page), 0, PAGE_SIZE)
 #define copy_page(to,from)	memcpy((void *)(to), (void *)(from), PAGE_SIZE)
 
-#if defined(__sh3__)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
-#elif defined(__SH4__)
-extern void clear_user_page(void *to, unsigned long address);
-extern void copy_user_page(void *to, void *from, unsigned long address);
+#if defined(__SH4__)
+#define __HAVE_ARCH_USER_PAGE
+struct page;
+extern void clear_user_page(struct page *to, unsigned long address);
+extern void copy_user_page(struct page *to, struct page *from,
+			   unsigned long address);
 #endif
 
 /*
include/asm-sparc/page.h +0 -2
diff -urN linux/include/asm-sparc/page.h linux-cfa/include/asm-sparc/page.h
@@ -54,8 +54,6 @@
 
 #define clear_page(page)	 memset((void *)(page), 0, PAGE_SIZE)
 #define copy_page(to,from) 	memcpy((void *)(to), (void *)(from), PAGE_SIZE)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
 
 /* The following structure is used to hold the physical
  * memory configuration of the machine.  This is filled in
include/asm-sparc64/page.h +7 -2
diff -urN linux/include/asm-sparc64/page.h linux-cfa/include/asm-sparc64/page.h
@@ -25,8 +25,13 @@
 extern void _copy_page(void *to, void *from);
 #define clear_page(X)	_clear_page((void *)(X))
 #define copy_page(X,Y)	_copy_page((void *)(X), (void *)(Y))
-extern void clear_user_page(void *page, unsigned long vaddr);
-extern void copy_user_page(void *to, void *from, unsigned long vaddr);
+
+#define __HAVE_ARCH_USER_PAGE
+extern void __clear_user_page(void *page, unsigned long vaddr);
+extern void __copy_user_page(void *to, void *from, unsigned long vaddr);
+#define clear_user_page(pg, va)	__clear_user_page((pg)->address, (va))
+#define copy_user_page(to, from, va) \
+		__copy_user_page((to)->address, (from)->address, (va))
 
 /* GROSS, defining this makes gcc pass these types as aggregates,
  * and thus on the stack, turn this crap off... -DaveM
include/linux/highmem.h +12 -7
diff -urN linux/include/linux/highmem.h linux-cfa/include/linux/highmem.h
@@ -43,20 +43,23 @@
 #endif /* CONFIG_HIGHMEM */
 
 /* when CONFIG_HIGHMEM is not set these will be plain clear/copy_page */
-static inline void clear_user_highpage(struct page *page, unsigned long vaddr)
+
+#ifndef __HAVE_ARCH_USER_PAGE
+static inline void clear_user_page(struct page *page, unsigned long vaddr)
 {
 	void *addr = kmap_atomic(page, KM_USER0);
-	clear_user_page(addr, vaddr);
+	clear_page(addr, vaddr);
 	kunmap_atomic(addr, KM_USER0);
 }
+#endif /* __HAVE_ARCH_USER_PAGE */
 
-static inline void clear_highpage(struct page *page)
+static inline void clear_mem_page(struct page *page)
 {
 	clear_page(kmap(page));
 	kunmap(page);
 }
 
-static inline void memclear_highpage(struct page *page, unsigned int offset, unsigned int size)
+static inline void memclear_page(struct page *page, unsigned int offset, unsigned int size)
 {
 	char *kaddr;
 
@@ -70,7 +73,7 @@
 /*
  * Same but also flushes aliased cache contents to RAM.
  */
-static inline void memclear_highpage_flush(struct page *page, unsigned int offset, unsigned int size)
+static inline void memclear_page_flush(struct page *page, unsigned int offset, unsigned int size)
 {
 	char *kaddr;
 
@@ -82,7 +85,8 @@
 	kunmap(page);
 }
 
-static inline void copy_user_highpage(struct page *to, struct page *from, unsigned long vaddr)
+#ifndef __HAVE_ARCH_USER_PAGE
+static inline void copy_user_page(struct page *to, struct page *from, unsigned long vaddr)
 {
 	char *vfrom, *vto;
 
@@ -92,8 +96,9 @@
 	kunmap_atomic(vfrom, KM_USER0);
 	kunmap_atomic(vto, KM_USER1);
 }
+#endif
 
-static inline void copy_highpage(struct page *to, struct page *from)
+static inline void copy_mem_page(struct page *to, struct page *from)
 {
 	char *vfrom, *vto;
 
kernel/ptrace.c +4 -3
diff -urN linux/kernel/ptrace.c linux-cfa/kernel/ptrace.c
@@ -103,10 +103,11 @@
 	flush_cache_page(vma, addr);
 
 	if (write) {
-		maddr = kmap(page);
-		memcpy(maddr + (addr & ~PAGE_MASK), buf, len);
+		maddr = kmap(page) + (addr & ~PAGE_MASK);
+		memcpy(maddr, buf, len);
 		flush_page_to_ram(page);
-		flush_icache_page(vma, page);
+		flush_icache_range((unsigned long) maddr,
+				   (unsigned long) maddr + len);
 		kunmap(page);
 	} else {
 		maddr = kmap(page);
mm/filemap.c +2 -2
diff -urN linux/mm/filemap.c linux-cfa/mm/filemap.c
@@ -193,7 +193,7 @@
 
 static inline void truncate_partial_page(struct page *page, unsigned partial)
 {
-	memclear_highpage_flush(page, partial, PAGE_CACHE_SIZE-partial);
+	memclear_page_flush(page, partial, PAGE_CACHE_SIZE-partial);
 				
 	if (page->buffers)
 		block_flushpage(page, partial);
@@ -1477,7 +1477,7 @@
 		struct page *new_page = alloc_page(GFP_HIGHUSER);
 
 		if (new_page) {
-			copy_user_highpage(new_page, old_page, address);
+			copy_user_page(new_page, old_page, address);
 			flush_page_to_ram(new_page);
 		} else
 			new_page = NOPAGE_OOM;
mm/memory.c +3 -3
diff -urN linux/mm/memory.c linux-cfa/mm/memory.c
@@ -61,10 +61,10 @@
 static inline void copy_cow_page(struct page * from, struct page * to, unsigned long address)
 {
 	if (from == ZERO_PAGE(address)) {
-		clear_user_highpage(to, address);
+		clear_user_page(to, address);
 		return;
 	}
-	copy_user_highpage(to, from, address);
+	copy_user_page(to, from, address);
 }
 
 mem_map_t * mem_map;
@@ -1186,7 +1186,7 @@
 		page = alloc_page(GFP_HIGHUSER);
 		if (!page)
 			goto no_mem;
-		clear_user_highpage(page, addr);
+		clear_user_page(page, addr);
 
 		spin_lock(&mm->page_table_lock);
 		if (!pte_none(*page_table)) {
mm/page_alloc.c +2 -3
diff -urN linux/mm/page_alloc.c linux-cfa/mm/page_alloc.c
@@ -526,9 +526,8 @@
 
 	page = alloc_pages(gfp_mask, 0);
 	if (page) {
-		void *address = page_address(page);
-		clear_page(address);
-		return (unsigned long) address;
+		clear_mem_page(page);
+		return (unsigned long) page_address(page);
 	}
 	return 0;
 }
mm/shmem.c +2 -2
diff -urN linux/mm/shmem.c linux-cfa/mm/shmem.c
@@ -376,7 +376,7 @@
 		page = page_cache_alloc(mapping);
 		if (!page)
 			return ERR_PTR(-ENOMEM);
-		clear_highpage(page);
+		clear_mem_page(page);
 		inode->i_blocks += BLOCKS_PER_PAGE;
 		add_to_page_cache (page, mapping, idx);
 	}
@@ -439,7 +439,7 @@
 		struct page *new_page = page_cache_alloc(inode->i_mapping);
 
 		if (new_page) {
-			copy_user_highpage(new_page, page, address);
+			copy_user_page(new_page, page, address);
 			flush_page_to_ram(new_page);
 		} else
 			new_page = NOPAGE_OOM;
#3 lore raw

This still is a no-go. You updated the header file, but what about the symbol names in arch/sparc64/lib/blockops.S and the module exports in arch/sparc64/kernel/sparc64_ksyms.c There is no way this thing will build.

Please, always apply a:

egrep {copy,clear}_user_page find arch/ include/ -type f

pass when doing these kinds of platform API changes.

Later,
David S. Miller
[email protected]

#4 lore raw
Quoted 4 lines · David S. Miller

This still is a no-go. You updated the header file, but what about the symbol names in arch/sparc64/lib/blockops.S and the module exports in arch/sparc64/kernel/sparc64_ksyms.c There is no way this thing will build.

My mistake. FWIW I had updated those files but I didn't add them to the list to be diffed.

Here is a new patch. I also updated cachetlb.txt a bit more.

Paul.

Documentation/cachetlb.txt +15 -4
diff -urN linux/Documentation/cachetlb.txt linux-cfa/Documentation/cachetlb.txt
@@ -260,8 +260,9 @@
 
 Here is the new interface:
 
-  void copy_user_page(void *to, void *from, unsigned long address)
-  void clear_user_page(void *to, unsigned long address)
+  void copy_user_page(struct page *to, struct page *from,
+		      unsigned long address)
+  void clear_user_page(struct page *to, unsigned long address)
 
 	These two routines store data in user anonymous or COW
 	pages.  It allows a port to efficiently avoid D-cache alias
@@ -277,8 +278,18 @@
 	The "address" parameter tells the virtual address where the
 	user will ultimately this page mapped.
 
-	If D-cache aliasing is not an issue, these two routines may
-	simply call memcpy/memset directly and do nothing more.
+	The "to" parameter points to the page struct for the
+	destination page.  This allows a port to store information
+	about the cache status of the page in the page struct (for
+	example, by using the PG_arch_1 bit of the flags field) and
+	update that status to reflect the effect of the clear or copy.
+
+	If D-cache aliasing is not an issue, and a port does not need
+	to maintain any cache status information in the page struct,
+	then it can use the default versions of these procedures
+	supplied in include/linux/highmem.h.  If a port does not want
+	to use the default versions it should declare them and define
+	the symbol __HAVE_ARCH_USER_PAGE in include/asm/page.h.
 
   void flush_dcache_page(struct page *page)
 
arch/ppc/kernel/misc.S +2 -24
diff -urN linux/arch/ppc/kernel/misc.S linux-cfa/arch/ppc/kernel/misc.S
@@ -475,9 +475,9 @@
  * snoop from the data cache.
  * This is a no-op on the 601 which has a unified cache.
  *
- *	void __flush_page_to_ram(void *page)
+ *	void __flush_dcache_icache(void *page)
  */
-_GLOBAL(__flush_page_to_ram)
+_GLOBAL(__flush_dcache_icache)
 	mfspr	r5,PVR
 	rlwinm	r5,r5,16,16,31
 	cmpi	0,r5,1
@@ -493,28 +493,6 @@
 	mtctr	r4
 1:	icbi	0,r6
 	addi	r6,r6,CACHE_LINE_SIZE
-	bdnz	1b
-	sync
-	isync
-	blr
-
-/*
- * Flush a particular page from the instruction cache.
- * Note: this is necessary because the instruction cache does *not*
- * snoop from the data cache.
- * This is a no-op on the 601 which has a unified cache.
- *
- *	void __flush_icache_page(void *page)
- */
-_GLOBAL(__flush_icache_page)
-	mfspr	r5,PVR
-	rlwinm	r5,r5,16,16,31
-	cmpi	0,r5,1
-	beqlr				/* for 601, do nothing */
-	li	r4,4096/CACHE_LINE_SIZE	/* Number of lines in a page */
-	mtctr	r4
-1:	icbi	0,r3
-	addi	r3,r3,CACHE_LINE_SIZE
 	bdnz	1b
 	sync
 	isync
arch/ppc/kernel/ppc_ksyms.c +1 -0
diff -urN linux/arch/ppc/kernel/ppc_ksyms.c linux-cfa/arch/ppc/kernel/ppc_ksyms.c
@@ -185,6 +185,7 @@
 EXPORT_SYMBOL(enable_kernel_fp);
 EXPORT_SYMBOL(flush_icache_range);
 EXPORT_SYMBOL(flush_dcache_range);
+EXPORT_SYMBOL(flush_dcache_page);
 EXPORT_SYMBOL(xchg_u32);
 #ifdef CONFIG_ALTIVEC
 EXPORT_SYMBOL(last_task_used_altivec);
arch/ppc/mm/init.c +31 -4
diff -urN linux/arch/ppc/mm/init.c linux-cfa/arch/ppc/mm/init.c
@@ -578,17 +578,23 @@
 	mem_pieces_remove(&phys_avail, start, size, 1);
 }
 
-void flush_page_to_ram(struct page *page)
+/*
+ * This is called when a page has been modified by the kernel.
+ * It just marks the page as not i-cache clean.  We do the i-cache
+ * flush later when the page is given to a user process, if necessary.
+ */
+void flush_dcache_page(struct page *page)
 {
-	unsigned long vaddr = (unsigned long) kmap(page);
-	__flush_page_to_ram(vaddr);
-	kunmap(page);
+	clear_bit(PG_arch_1, &page->flags);
 }
 
 /*
  * set_pte stores a linux PTE into the linux page table.
  * On machines which use an MMU hash table we avoid changing the
  * _PAGE_HASHPTE bit.
+ * If the new PTE has _PAGE_EXEC set, meaning that the user wants
+ * to be able to execute out of the page, we check if the page is
+ * i-cache dirty and flush it if so, and mark it clean.
  */
 void set_pte(pte_t *ptep, pte_t pte)
 {
@@ -597,4 +603,25 @@
 #else
 	*ptep = pte;
 #endif
+	if (mem_init_done && (pte_val(pte) & _PAGE_EXEC)
+	    && (pte_val(pte) & PAGE_MASK) < total_memory) {
+		struct page *page = pte_page(pte);
+		if (!test_bit(PG_arch_1, &page->flags)) {
+			__flush_dcache_icache((unsigned long)kmap(page));
+			kunmap(page);
+			set_bit(PG_arch_1, &page->flags);
+		}
+	}
+}
+
+void clear_user_page(struct page *page, unsigned long vaddr)
+{
+	clear_mem_page(page);
+	clear_bit(PG_arch_1, &page->flags);
+}
+
+void copy_user_page(struct page *to, struct page *from, unsigned long vaddr)
+{
+	copy_mem_page(to, from);
+	clear_bit(PG_arch_1, &to->flags);
 }
arch/sh/mm/cache.c +9 -2
diff -urN linux/arch/sh/mm/cache.c linux-cfa/arch/sh/mm/cache.c
@@ -508,17 +508,24 @@
 /* Page is 4K, OC size is 16K, there are four lines. */
 #define CACHE_ALIAS 0x00003000
 
-void clear_user_page(void *to, unsigned long address)
+void clear_user_page(struct page *page, unsigned long address)
 {
+	void *to = kmap_atomic(page, KM_USER0);
 	clear_page(to);
 	if (((address ^ (unsigned long)to) & CACHE_ALIAS))
 		__flush_page_to_ram(to);
+	kunmap_atomic(to, KM_USER0);
 }
 
-void copy_user_page(void *to, void *from, unsigned long address)
+void copy_user_page(struct page *to_pg, struct page *from_pg,
+		    unsigned long address)
 {
+	void *from = kmap_atomic(from_pg, KM_USER0);
+	void *to = kmap_atomic(to_pg, KM_USER1);
 	copy_page(to, from);
 	if (((address ^ (unsigned long)to) & CACHE_ALIAS))
 		__flush_page_to_ram(to);
+	kunmap_atomic(from, KM_USER0);
+	kunmap_atomic(to, KM_USER1);
 }
 #endif
arch/sparc64/kernel/sparc64_ksyms.c +2 -2
diff -urN linux/arch/sparc64/kernel/sparc64_ksyms.c linux-cfa/arch/sparc64/kernel/sparc64_ksyms.c
@@ -316,8 +316,8 @@
 EXPORT_SYMBOL(__memset);
 EXPORT_SYMBOL(_clear_page);
 EXPORT_SYMBOL(_copy_page);
-EXPORT_SYMBOL(clear_user_page);
-EXPORT_SYMBOL(copy_user_page);
+EXPORT_SYMBOL(__clear_user_page);
+EXPORT_SYMBOL(__copy_user_page);
 EXPORT_SYMBOL(__bzero);
 EXPORT_SYMBOL(__memscan_zero);
 EXPORT_SYMBOL(__memscan_generic);
arch/sparc64/lib/blockops.S +6 -6
diff -urN linux/arch/sparc64/lib/blockops.S linux-cfa/arch/sparc64/lib/blockops.S
@@ -65,9 +65,9 @@
 	retl
 	 nop
 
-	.globl		copy_user_page
-	.type		copy_user_page,@function
-copy_user_page: /* %o0=dest, %o1=src, %o2=vaddr */
+	.globl		__copy_user_page
+	.type		__copy_user_page,@function
+__copy_user_page: /* %o0=dest, %o1=src, %o2=vaddr */
 	VISEntry
 	sethi		%hi(PAGE_SIZE), %g3
 	sub		%o0, %g4, %g1
@@ -322,9 +322,9 @@
 	 clr		%o4
 
 	.align		32
-	.globl		clear_user_page
-	.type		clear_user_page,@function
-clear_user_page:	/* %o0=dest, %o1=vaddr */
+	.globl		__clear_user_page
+	.type		__clear_user_page,@function
+__clear_user_page:	/* %o0=dest, %o1=vaddr */
 	VISEntryHalf
 	sethi		%hi(PAGE_SIZE), %g3
 	sub		%o0, %g4, %g1
include/asm-alpha/page.h +0 -3
diff -urN linux/include/asm-alpha/page.h linux-cfa/include/asm-alpha/page.h
@@ -15,10 +15,7 @@
 #define STRICT_MM_TYPECHECKS
 
 extern void clear_page(void *page);
-#define clear_user_page(page, vaddr)	clear_page(page)
-
 extern void copy_page(void * _to, void * _from);
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
 
 #ifdef STRICT_MM_TYPECHECKS
 /*
include/asm-arm/page.h +0 -3
diff -urN linux/include/asm-arm/page.h linux-cfa/include/asm-arm/page.h
@@ -14,9 +14,6 @@
 #define clear_page(page)	memzero((void *)(page), PAGE_SIZE)
 extern void copy_page(void *to, void *from);
 
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
-
 #ifdef STRICT_MM_TYPECHECKS
 /*
  * These are used to make use of C type-checking..
include/asm-cris/page.h +0 -3
diff -urN linux/include/asm-cris/page.h linux-cfa/include/asm-cris/page.h
@@ -14,9 +14,6 @@
 #define clear_page(page)        memset((void *)(page), 0, PAGE_SIZE)
 #define copy_page(to,from)      memcpy((void *)(to), (void *)(from), PAGE_SIZE)
 
-#define clear_user_page(page, vaddr)    clear_page(page)
-#define copy_user_page(to, from, vaddr) copy_page(to, from)
-
 #define STRICT_MM_TYPECHECKS
 
 #ifdef STRICT_MM_TYPECHECKS
include/asm-ia64/pgalloc.h +7 -5
diff -urN linux/include/asm-ia64/pgalloc.h linux-cfa/include/asm-ia64/pgalloc.h
@@ -234,18 +234,20 @@
 	clear_bit(PG_arch_1, &page->flags);
 }
 
+#define __HAVE_ARCH_USER_PAGE
+
 static inline void
-clear_user_page (void *addr, unsigned long vaddr, struct page *page)
+clear_user_page (struct page *page, unsigned long vaddr)
 {
-	clear_page(addr);
+	clear_mem_page(page);
 	flush_dcache_page(page);
 }
 
 static inline void
-copy_user_page (void *to, void *from, unsigned long vaddr, struct page *page)
+copy_user_page (struct page *to, struct page *from, unsigned long vaddr)
 {
-	copy_page(to, from);
-	flush_dcache_page(page);
+	copy_mem_page(to, from);
+	flush_dcache_page(to);
 }
 
 /*
include/asm-m68k/page.h +0 -3
diff -urN linux/include/asm-m68k/page.h linux-cfa/include/asm-m68k/page.h
@@ -76,9 +76,6 @@
 #define copy_page(to,from)	memcpy((to), (from), PAGE_SIZE)
 #endif
 
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
-
 /*
  * These are used to make use of C type-checking..
  */
include/asm-mips/page.h +0 -2
diff -urN linux/include/asm-mips/page.h linux-cfa/include/asm-mips/page.h
@@ -28,8 +28,6 @@
 
 #define clear_page(page)	_clear_page(page)
 #define copy_page(to, from)	_copy_page(to, from)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
 
 /*
  * These are used to make use of C type-checking..
include/asm-mips64/page.h +0 -2
diff -urN linux/include/asm-mips64/page.h linux-cfa/include/asm-mips64/page.h
@@ -28,8 +28,6 @@
 
 #define clear_page(page)	_clear_page(page)
 #define copy_page(to, from)	_copy_page(to, from)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
 
 /*
  * These are used to make use of C type-checking..
include/asm-parisc/page.h +0 -3
diff -urN linux/include/asm-parisc/page.h linux-cfa/include/asm-parisc/page.h
@@ -12,9 +12,6 @@
 #define clear_page(page)	memset((void *)(page), 0, PAGE_SIZE)
 #define copy_page(to,from)	memcpy((void *)(to), (void *)(from), PAGE_SIZE)
 
-#define clear_user_page(page, vaddr) clear_page(page)
-#define copy_user_page(to, from, vaddr) copy_page(to, from)
-
 /*
  * These are used to make use of C type-checking..
  */
include/asm-ppc/page.h +5 -2
diff -urN linux/include/asm-ppc/page.h linux-cfa/include/asm-ppc/page.h
@@ -83,8 +83,11 @@
 
 extern void clear_page(void *page);
 extern void copy_page(void *to, void *from);
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
+
+#define __HAVE_ARCH_USER_PAGE
+struct page;
+extern void clear_user_page(struct page *page, unsigned long vaddr);
+extern void copy_user_page(struct page *to, struct page *from, unsigned long vaddr);
 
 /* map phys->virtual and virtual->phys for RAM pages */
 static inline unsigned long ___pa(unsigned long v)
include/asm-ppc/pgtable.h +7 -4
diff -urN linux/include/asm-ppc/pgtable.h linux-cfa/include/asm-ppc/pgtable.h
@@ -83,12 +83,11 @@
 #define flush_cache_range(mm, a, b)	do { } while (0)
 #define flush_cache_page(vma, p)	do { } while (0)
 #define flush_icache_page(vma, page)	do { } while (0)
+#define flush_page_to_ram(page)		do { } while (0)
 
 extern void flush_icache_range(unsigned long, unsigned long);
-extern void __flush_page_to_ram(unsigned long page_va);
-extern void flush_page_to_ram(struct page *page);
-
-#define flush_dcache_page(page)			do { } while (0)
+extern void __flush_dcache_icache(unsigned long page_va);
+extern void flush_dcache_page(struct page *page);
 
 extern unsigned long va_to_phys(unsigned long address);
 extern pte_t *va_to_pte(unsigned long address);
@@ -421,6 +420,10 @@
 }
 
 /*
+ * When a new value is written into a PTE, we may need to flush the
+ * i-cache for the page of memory that the PTE points to.
+ * (Note: machines with software TLB reloads could do the flush in
+ * the instruction TLB miss handler instead.)
  * Writing a new value into the PTE doesn't disturb the state of the
  * _PAGE_HASHPTE bit, on those machines which use an MMU hash table.
  */
include/asm-s390/page.h +0 -3
diff -urN linux/include/asm-s390/page.h linux-cfa/include/asm-s390/page.h
@@ -59,9 +59,6 @@
 			      : "memory" );
 }
 
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
-
 #define BUG() do { \
         printk("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \
         __asm__ __volatile__(".word 0x0000"); \
include/asm-s390x/page.h +0 -3
diff -urN linux/include/asm-s390x/page.h linux-cfa/include/asm-s390x/page.h
@@ -57,9 +57,6 @@
 			      : "memory" );
 }
 
-#define clear_user_page(page, vaddr)    clear_page(page)
-#define copy_user_page(to, from, vaddr) copy_page(to, from)
-
 #define BUG() do { \
         printk("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \
         __asm__ __volatile__(".long 0"); \
include/asm-sh/page.h +6 -6
diff -urN linux/include/asm-sh/page.h linux-cfa/include/asm-sh/page.h
@@ -27,12 +27,12 @@
 #define clear_page(page)	memset((void *)(page), 0, PAGE_SIZE)
 #define copy_page(to,from)	memcpy((void *)(to), (void *)(from), PAGE_SIZE)
 
-#if defined(__sh3__)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
-#elif defined(__SH4__)
-extern void clear_user_page(void *to, unsigned long address);
-extern void copy_user_page(void *to, void *from, unsigned long address);
+#if defined(__SH4__)
+#define __HAVE_ARCH_USER_PAGE
+struct page;
+extern void clear_user_page(struct page *to, unsigned long address);
+extern void copy_user_page(struct page *to, struct page *from,
+			   unsigned long address);
 #endif
 
 /*
include/asm-sparc/page.h +0 -2
diff -urN linux/include/asm-sparc/page.h linux-cfa/include/asm-sparc/page.h
@@ -54,8 +54,6 @@
 
 #define clear_page(page)	 memset((void *)(page), 0, PAGE_SIZE)
 #define copy_page(to,from) 	memcpy((void *)(to), (void *)(from), PAGE_SIZE)
-#define clear_user_page(page, vaddr)	clear_page(page)
-#define copy_user_page(to, from, vaddr)	copy_page(to, from)
 
 /* The following structure is used to hold the physical
  * memory configuration of the machine.  This is filled in
include/asm-sparc64/page.h +7 -2
diff -urN linux/include/asm-sparc64/page.h linux-cfa/include/asm-sparc64/page.h
@@ -25,8 +25,13 @@
 extern void _copy_page(void *to, void *from);
 #define clear_page(X)	_clear_page((void *)(X))
 #define copy_page(X,Y)	_copy_page((void *)(X), (void *)(Y))
-extern void clear_user_page(void *page, unsigned long vaddr);
-extern void copy_user_page(void *to, void *from, unsigned long vaddr);
+
+#define __HAVE_ARCH_USER_PAGE
+extern void __clear_user_page(void *page, unsigned long vaddr);
+extern void __copy_user_page(void *to, void *from, unsigned long vaddr);
+#define clear_user_page(pg, va)	__clear_user_page((pg)->address, (va))
+#define copy_user_page(to, from, va) \
+		__copy_user_page((to)->address, (from)->address, (va))
 
 /* GROSS, defining this makes gcc pass these types as aggregates,
  * and thus on the stack, turn this crap off... -DaveM
include/linux/highmem.h +12 -7
diff -urN linux/include/linux/highmem.h linux-cfa/include/linux/highmem.h
@@ -43,20 +43,23 @@
 #endif /* CONFIG_HIGHMEM */
 
 /* when CONFIG_HIGHMEM is not set these will be plain clear/copy_page */
-static inline void clear_user_highpage(struct page *page, unsigned long vaddr)
+
+#ifndef __HAVE_ARCH_USER_PAGE
+static inline void clear_user_page(struct page *page, unsigned long vaddr)
 {
 	void *addr = kmap_atomic(page, KM_USER0);
-	clear_user_page(addr, vaddr);
+	clear_page(addr, vaddr);
 	kunmap_atomic(addr, KM_USER0);
 }
+#endif /* __HAVE_ARCH_USER_PAGE */
 
-static inline void clear_highpage(struct page *page)
+static inline void clear_mem_page(struct page *page)
 {
 	clear_page(kmap(page));
 	kunmap(page);
 }
 
-static inline void memclear_highpage(struct page *page, unsigned int offset, unsigned int size)
+static inline void memclear_page(struct page *page, unsigned int offset, unsigned int size)
 {
 	char *kaddr;
 
@@ -70,7 +73,7 @@
 /*
  * Same but also flushes aliased cache contents to RAM.
  */
-static inline void memclear_highpage_flush(struct page *page, unsigned int offset, unsigned int size)
+static inline void memclear_page_flush(struct page *page, unsigned int offset, unsigned int size)
 {
 	char *kaddr;
 
@@ -82,7 +85,8 @@
 	kunmap(page);
 }
 
-static inline void copy_user_highpage(struct page *to, struct page *from, unsigned long vaddr)
+#ifndef __HAVE_ARCH_USER_PAGE
+static inline void copy_user_page(struct page *to, struct page *from, unsigned long vaddr)
 {
 	char *vfrom, *vto;
 
@@ -92,8 +96,9 @@
 	kunmap_atomic(vfrom, KM_USER0);
 	kunmap_atomic(vto, KM_USER1);
 }
+#endif
 
-static inline void copy_highpage(struct page *to, struct page *from)
+static inline void copy_mem_page(struct page *to, struct page *from)
 {
 	char *vfrom, *vto;
 
kernel/ptrace.c +4 -3
diff -urN linux/kernel/ptrace.c linux-cfa/kernel/ptrace.c
@@ -103,10 +103,11 @@
 	flush_cache_page(vma, addr);
 
 	if (write) {
-		maddr = kmap(page);
-		memcpy(maddr + (addr & ~PAGE_MASK), buf, len);
+		maddr = kmap(page) + (addr & ~PAGE_MASK);
+		memcpy(maddr, buf, len);
 		flush_page_to_ram(page);
-		flush_icache_page(vma, page);
+		flush_icache_range((unsigned long) maddr,
+				   (unsigned long) maddr + len);
 		kunmap(page);
 	} else {
 		maddr = kmap(page);
mm/filemap.c +2 -2
diff -urN linux/mm/filemap.c linux-cfa/mm/filemap.c
@@ -193,7 +193,7 @@
 
 static inline void truncate_partial_page(struct page *page, unsigned partial)
 {
-	memclear_highpage_flush(page, partial, PAGE_CACHE_SIZE-partial);
+	memclear_page_flush(page, partial, PAGE_CACHE_SIZE-partial);
 				
 	if (page->buffers)
 		block_flushpage(page, partial);
@@ -1477,7 +1477,7 @@
 		struct page *new_page = alloc_page(GFP_HIGHUSER);
 
 		if (new_page) {
-			copy_user_highpage(new_page, old_page, address);
+			copy_user_page(new_page, old_page, address);
 			flush_page_to_ram(new_page);
 		} else
 			new_page = NOPAGE_OOM;
mm/memory.c +3 -3
diff -urN linux/mm/memory.c linux-cfa/mm/memory.c
@@ -61,10 +61,10 @@
 static inline void copy_cow_page(struct page * from, struct page * to, unsigned long address)
 {
 	if (from == ZERO_PAGE(address)) {
-		clear_user_highpage(to, address);
+		clear_user_page(to, address);
 		return;
 	}
-	copy_user_highpage(to, from, address);
+	copy_user_page(to, from, address);
 }
 
 mem_map_t * mem_map;
@@ -1186,7 +1186,7 @@
 		page = alloc_page(GFP_HIGHUSER);
 		if (!page)
 			goto no_mem;
-		clear_user_highpage(page, addr);
+		clear_user_page(page, addr);
 
 		spin_lock(&mm->page_table_lock);
 		if (!pte_none(*page_table)) {
mm/page_alloc.c +2 -3
diff -urN linux/mm/page_alloc.c linux-cfa/mm/page_alloc.c
@@ -526,9 +526,8 @@
 
 	page = alloc_pages(gfp_mask, 0);
 	if (page) {
-		void *address = page_address(page);
-		clear_page(address);
-		return (unsigned long) address;
+		clear_mem_page(page);
+		return (unsigned long) page_address(page);
 	}
 	return 0;
 }
mm/shmem.c +2 -2
diff -urN linux/mm/shmem.c linux-cfa/mm/shmem.c
@@ -376,7 +376,7 @@
 		page = page_cache_alloc(mapping);
 		if (!page)
 			return ERR_PTR(-ENOMEM);
-		clear_highpage(page);
+		clear_mem_page(page);
 		inode->i_blocks += BLOCKS_PER_PAGE;
 		add_to_page_cache (page, mapping, idx);
 	}
@@ -439,7 +439,7 @@
 		struct page *new_page = page_cache_alloc(inode->i_mapping);
 
 		if (new_page) {
-			copy_user_highpage(new_page, page, address);
+			copy_user_page(new_page, page, address);
 			flush_page_to_ram(new_page);
 		} else
 			new_page = NOPAGE_OOM;
#6 lore raw
In reply to Paul Mackerras

arch/ppc/mm/init.c line 612

Quoted 4 lines · Paul Mackerras
arch/ppc/mm/init.c +4 -0
+		if (!test_bit(PG_arch_1, &page->flags)) {
+			__flush_dcache_icache((unsigned long)kmap(page));
+			kunmap(page);
+			set_bit(PG_arch_1, &page->flags);

Race. Use test_and_set_bit.

As for Alpha, all we have is "flush entire icache", so there's not much interesting we can do by way of optimization I don't think.

r~

#7 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Richard Henderson
  Richard> On Fri, Aug 31, 2001 at 09:18:50PM +1000, Paul Mackerras wrote:
  >> +		if (!test_bit(PG_arch_1, &page->flags)) {
  >> +			__flush_dcache_icache((unsigned long)kmap(page));
  >> +			kunmap(page);
  >> +			set_bit(PG_arch_1, &page->flags);

Richard> Race. Use test_and_set_bit.

Nope, the old code is correct: you must turn on PG_arch_1 *after* flushing the cache. Yes, you might sometimes double flush, but that's both safe and rare.

	--david
#8 lore raw
In reply to Paul Mackerras
Quoted 10 lines · David Mosberger
Quoted 4 lines

arch/ppc/mm/init.c line 612

Quoted 4 lines
arch/ppc/mm/init.c +4 -0
+		if (!test_bit(PG_arch_1, &page->flags)) {
+			__flush_dcache_icache((unsigned long)kmap(page));
+			kunmap(page);
+			set_bit(PG_arch_1, &page->flags);

Richard> Race. Use test_and_set_bit.

Nope, the old code is correct: you must turn on PG_arch_1 *after* flushing the cache. Yes, you might sometimes double flush, but that's both safe and rare.

You can get a missed flush from

	bit == 0
	flush cache

				modify page
				bit = 0

	bit = 1

unless this is protected from some outer lock of which I am not aware.

I do see your point about the early set though.

r~

#9 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Richard Henderson

Richard> You can get a missed flush from

Richard> bit == 0 flush cache

Richard> modify page bit = 0

Richard> bit = 1

Richard> unless this is protected from some outer lock of which I am Richard> not aware.

Richard> I do see your point about the early set though.

I didn't think there was any path where the kernel would on its own update code after the fact, but I could be missing something. Note, that if it's the user's doing, the user is responsible for ensuring coherence.

	--david
#10 lore raw
In reply to Richard Henderson
Quoted 10 lines · Richard Henderson

You can get a missed flush from

	bit == 0
	flush cache

				modify page
				bit = 0

	bit = 1
Quoted 2 lines

unless this is protected from some outer lock of which I am not aware.

The page is question is one which is mapped (is being mapped) into a process's address space with execute permission. If the page is part of the page cache for a file, then I would have thought that attempts to write to the page would return an ETXTBSY error. If the page is a private page (private COW or anonymous page) then I don't see where the kernel would be modifying the page. If it is already mapped into another process's address space with a shared writable mapping, and that process is writing to the page, then it is up to the user-level stuff to do the necessary cache-flushing.

So I think it's OK at the moment, but I agree it does look like a race waiting to happen.

For alpha, the thing that my patch does that might hurt is the change from flush_icache_page to flush_icache_range in kernel/ptrace.c. Any comment on that?

Paul.

#11 lore raw
In reply to Paul Mackerras
Quoted 3 lines · Paul Mackerras

For alpha, the thing that my patch does that might hurt is the change from flush_icache_page to flush_icache_range in kernel/ptrace.c. Any comment on that?

For the alpha such change will imply a performance penablity (will throw away the whole icache, not only the one belonging to the ptraced task).

We cannot change flush_icache_range to bump the asn because there's no vma in the flush_icache_range API (flush_icache_range in short is for the kernel side, like with vmalloc and kernel modules where a vma/mm/tsk wouldn't make sense anyways).

what's the point of such change? The whole point of the patch is to work with pages not with virtual addresses so you can do the bitflag bookeeping on the page structure, so I don't see why would you want to do the opposite change there.

Andrea

#12 lore raw
Quoted 7 lines · Andrea Arcangeli
Quoted 3 lines · Paul Mackerras

For alpha, the thing that my patch does that might hurt is the change from flush_icache_page to flush_icache_range in kernel/ptrace.c. Any comment on that?

For the alpha such change will imply a performance penablity (will throw away the whole icache, not only the one belonging to the ptraced task).

OK then maybe we need a flush_icache_user_range or something.

Quoted 4 lines

We cannot change flush_icache_range to bump the asn because there's no vma in the flush_icache_range API (flush_icache_range in short is for the kernel side, like with vmalloc and kernel modules where a vma/mm/tsk wouldn't make sense anyways).

Yes, that is what Documentation/cachetlb.txt says. I note that currently flush_icache_range is used on user addresses in binfmt_aout.c, and in fs/binfmt_elf.c in the load_aout_interp() function. But nobody uses a.out these days so that doesn't matter. :)

Quoted 1 line

what's the point of such change? The whole point of the patch is to work

flush_icache_page is not a good function to use because it is called in do_swap_page and in do_no_page in mm/memory.c and in those cases the page might already be i-cache clean and so we don't want to do any flush. In those cases, if the page does actually get read in from disk then we do want to do the flush, if it was in the page cache or swap cache and has been flushed before then we don't want to do the flush. (Actually doesn't that mean that on alpha you are throwing away the whole icache for the process every time it faults in an executable page?)

flush_icache_page is also not a good choice because it is overkill to flush a whole page when you have just written one word, to put in a breakpoint or something.

Quoted 3 lines

with pages not with virtual addresses so you can do the bitflag bookeeping on the page structure, so I don't see why would you want to do the opposite change there.

I would be happy with an interface that took a struct page *, and preferably an offset and length within the page. That would be a new interface though. Note that the caller of access_process_vm can't easily do the flush without duplicating most of the logic of access_process_vm.

Paul.

#13 lore raw
In reply to Paul Mackerras
Quoted 9 lines · Paul Mackerras

flush_icache_page is not a good function to use because it is called in do_swap_page and in do_no_page in mm/memory.c and in those cases the page might already be i-cache clean and so we don't want to do any flush. In those cases, if the page does actually get read in from disk then we do want to do the flush, if it was in the page cache or swap cache and has been flushed before then we don't want to do the flush. (Actually doesn't that mean that on alpha you are throwing away the whole icache for the process every time it faults in an executable page?)

Actually alpha is wasting lots of asn at evey swapin or pagein! See what the specification (implementation independent) says:

Virtual instruction caches are not required to notice modifications of
the virtual I-stream (they need not be coherent with the rest of
memory). Software that creates or modifies the instruction stream must
execute a CALL_PAL IMB before trying to exe-cute the new instructions.
In this context, to "modify the virtual I-stream" means either: any
^^^
Store to the same physical address that is subsequently fetched as an ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
instruction by some corresponding (virtual address, ASN) pair, or any
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (ptrace case thus we need imb or bumping the asn there) change to the virtual-to-physical address mapping so that different
values are fetched. For example, if two different virtual addresses, VA1
and VA2, map to the same page frame, a store to VA1 modifies the virtual
I-stream fetched by VA2. However, the following sequence does not modify
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
the virtual I-stream (this might happen in soft page faults). 1. Change ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
the mapping of an I-stream page from valid to invalid. 2. Copy the ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
corresponding page frame to a new page frame. 3. Change the original
^^^^^^^^^^^^^^^^^^^
mapping to be valid and point to the new page frame. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

As far I can tell during pageins/swapins we really don't need to flush the icache (we literally always mark the pagetable invalid + flush the tlb before any swapin or pagein so I think it fully applies to our case). Infact 2.2 never do any asn change or imb during the swapins/pageins and we never had a problem.

On alpha we need to flush the icache only when we mess with the memory contents under the icache, not when we pageout pagein this memory.

I believe the point here is that swapins and pageins of .text segments aren't going to want to change the contents of the icache anyways.

Actually one could argue that if we map an executable segment and then the vm unmaps the page, then somebody else changes the page in the disk and pagecache writing to it, then we fault on the page again with the l1 dcache, in this case I guess the cpu could still exectue the old istream and not notice that it's changed but who cares about this weird case anyways? If anybody does self modifying code this way he should serialize in userspace.

So in short I'd prefer to undefine the flush_icache_page on alpha and to have it used in the common code only from the paging activity. I'd really like if you could arrange this modification in a new release of your anti-cache-flushes patch.

Quoted 3 lines

flush_icache_page is also not a good choice because it is overkill to flush a whole page when you have just written one word, to put in a breakpoint or something.

Ok I see why you changed it than (just to flush one word and not the whole page), but we cannot just flush one word of icache on alpha (I was biased and this is why I didn't seen the point of the change ;), either we flush the whole icache or we flush only the address space of the ptraced mm by bumping its asn, so yes, we'd like a kind of flush_icache_range_mm (or _vma or whatever that pass the mm somehow). If you could add this modification to your patch as well that would be fine!

Quoted 3 lines

I would be happy with an interface that took a struct page *, and preferably an offset and length within the page. That would be a new interface though. Note that the caller of access_process_vm can't

yes, as mentioned above a new flush_icache_range_mm/vma/user would be enough, the other flush_icache_range should be used only for the kernel side.

Andrea

#15 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Richard Henderson

Richard> On Mon, Sep 03, 2001 at 02:00:39PM -0700, David Mosberger

Quoted 2 lines · Richard> wrote:
Quoted 2 lines

I didn't think there was any path where the kernel would on its own update code after the fact, but I could be missing something.

Richard> ptrace?

ptrace() is handled separately (it's like a user in that sense: it takes care of establishing coherence by calling the appropriate flushing routine).

	--david
#16 lore raw
In reply to Paul Mackerras
Quoted 2 lines · Paul Mackerras

If the page is a private page (private COW or anonymous page) then I don't see where the kernel would be modifying the page.

ptrace set breakpoint?

Quoted 3 lines

For alpha, the thing that my patch does that might hurt is the change from flush_icache_page to flush_icache_range in kernel/ptrace.c. Any comment on that?

Hum. Yes. We need a way to distinguish between userspace and kernelspace icache flushes.

Previously, flush_icache_page was used exclusively for userspace and flush_icache_range exclusively for kernelspace. Since I _do_ have address space numbers, I can avoid the "flush all" by allocating a new ASN for the user process. Which doesn't work for the kernel of course; there I do have to flush all.

r~