Skip to contentSkip to conversation

FCSE

PATCH New Gilles Chanteperdrix 2 participants 13 messages Oct 1, 2009 – Oct 4, 2009 Arm (32-bit)

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 FCSE

Hi,

Here comes the FCSE patch for Linux 2.6.32-rc1.

To summarize a bit the story of this patch which you will find in various posts on this list, this patch adds an option to the kernel configuration which allows using the ARM FCSE in one of two modes:
- the guaranteed mode, where context switches are guaranteed to run without flushing the caches, but where the number of processes is limited to 95 with a virtual memory space limited to 32 MB;
- the best-effort mode, where the limitations are removed, but where the cache is flushed when needed during context switches.

The patch can be considered as relatively stable, it has not changed much since Linux 2.6.27, and has been validated by running LTP with the best-effort mode, with no differences in the LTP results compared to an unpatched kernel. Furthermore, it has been reported to work in the field.

As for the performance improvement, running the hackbench test with 400 processes (so, well above the 95 processes limit) on an AT91RM9200 showed an improvement in the runing time which can be visualized here:
http://sisyphus.hd.free.fr/~gilles/hackbench.png.

The user-space worst-case scheduling latency is also improved by the guaranteed mode, but this only interests people running an RT kernel of any kind.

The patch is relatively simple and only modifies ARM-specific code, but has been split to ease review, as I have been advised off-list to do. Some parts of it, however, could be implemented more elegantly (to say the least), especially patches 8 and 9 which intercept set_pte_at() to count pages belonging to some types of mappings whereas counting mappings would be enough but would probably require to modify generic code.

Thanks in advance for your review.
Regards

Signature · 1 line
					    Gilles.
#1 lore raw

PATCH 1/1 FCSE: pid life cycle.

Add a pid member to the mm_context_t structure, allocate this pid in the init_new_context, free it in the destroy_context, switch it in switch_mm.

A bitfield is maintained to know what pids are currently in use.

---
 arch/arm/include/asm/fcse.h        |   44 +++++++++++++++++++++++++++
 arch/arm/include/asm/mmu.h         |    3 ++
 arch/arm/include/asm/mmu_context.h |   24 ++++++++++++++-
 arch/arm/kernel/fcse.c             |   57 ++++++++++++++++++++++++++++++++++++
 4 files changed, 126 insertions(+), 2 deletions(-)
 create mode 100644 arch/arm/include/asm/fcse.h
 create mode 100644 arch/arm/kernel/fcse.c
arch/arm/include/asm/fcse.h new file +44 -0
new file mode 100644
index 0000000..b44e866
@@ -0,0 +1,44 @@
+/*
+ * Filename:    arch/arm/include/asm/fcse.h
+ * Description: ARM Process ID (PID) includes for Fast Address Space Switching
+ *              (FASS) in ARM Linux.
+ * Created:     14/10/2001
+ * Changes:     19/02/2002 - Macros added.
+ *              03/08/2007 - Adapted to kernel 2.6.21 (ssm)
+ *              Feb 2008   - Simplified a bit (rco)
+ *
+ * Copyright:   (C) 2001, 2002 Adam Wiggins <[email protected]>
+ *              (C) 2007 Sebastian Smolorz <[email protected]>
+ *              (C) 2008 Richard Cochran
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of teh GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+#ifndef __ASM_ARM_FCSE_H
+#define __ASM_ARM_FCSE_H
+
+#ifdef CONFIG_ARM_FCSE
+
+#include <linux/mm_types.h>	/* For struct mm_struct */
+
+#define FCSE_PID_SHIFT 25
+
+/* Size of PID relocation area */
+#define FCSE_PID_TASK_SIZE (1UL << FCSE_PID_SHIFT)
+
+/* Sets the CPU's PID Register */
+static inline void fcse_pid_set(unsigned long pid)
+{
+	__asm__ __volatile__ ("mcr p15, 0, %0, c13, c0, 0"
+			      : /* */: "r" (pid) : "memory");
+}
+
+int fcse_pid_alloc(void);
+void fcse_pid_free(unsigned pid);
+
+#else /* ! CONFIG_ARM_FCSE */
+#define fcse_pid_set(pid) do { } while (0)
+#endif /* ! CONFIG_ARM_FCSE */
+
+#endif /* __ASM_ARM_FCSE_H */
arch/arm/include/asm/mmu.h +3 -0
index b561584..4a5a8db 100644
@@ -7,6 +7,9 @@ typedef struct {
 #ifdef CONFIG_CPU_HAS_ASID
 	unsigned int id;
 #endif
+#ifdef CONFIG_ARM_FCSE
+	unsigned long pid;
+#endif /* CONFIG_ARM_FCSE */
 	unsigned int kvm_seq;
 } mm_context_t;
 
arch/arm/include/asm/mmu_context.h +22 -2
index de6cefb..ec7a84d 100644
@@ -19,6 +19,7 @@
 #include <asm/cachetype.h>
 #include <asm/proc-fns.h>
 #include <asm-generic/mm_hooks.h>
+#include <asm/fcse.h>
 
 void __check_kvm_seq(struct mm_struct *mm);
 
@@ -68,11 +69,29 @@ static inline void check_context(struct mm_struct *mm)
 #endif
 }
 
-#define init_new_context(tsk,mm)	0
+static inline int
+init_new_context(struct task_struct *tsk, struct mm_struct *mm)
+{
+#ifdef CONFIG_ARM_FCSE
+	int pid;
+
+	pid = fcse_pid_alloc();
+	if (pid < 0)
+		return pid;
+	mm->context.pid = pid << FCSE_PID_SHIFT;
+#endif /* CONFIG_ARM_FCSE */
+
+	return 0;
+}
 
 #endif
 
-#define destroy_context(mm)		do { } while(0)
+static inline void destroy_context(struct mm_struct *mm)
+{
+#ifdef CONFIG_ARM_FCSE
+	fcse_pid_free(mm->context.pid >> FCSE_PID_SHIFT);
+#endif /* CONFIG_ARM_FCSE */
+}
 
 /*
  * This is called when "tsk" is about to enter lazy TLB mode.
@@ -109,6 +128,7 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,
 #endif
 	if (!cpumask_test_and_set_cpu(cpu, mm_cpumask(next)) || prev != next) {
 		check_context(next);
+		fcse_pid_set(next->context.pid);
 		cpu_switch_mm(next->pgd, next);
 		if (cache_is_vivt())
 			cpumask_clear_cpu(cpu, mm_cpumask(prev));
arch/arm/kernel/fcse.c new file +57 -0

57 lines not on this page · raw message

Signature · 1 line
1.5.6.5
#2 lore raw

PATCH 1/1 FCSE: Conversions between VA and MVA.

Add to asm/fcse.h the functions needed to allow conversion between virtual addresses (a.k.a VA) and modified virtual addresses (a.k.a MVA), that is virtual addresses translated by the FCSE pid.

Though the whole kernel handles VA, and do not see MVA, there are a few exceptions:
- the function handling page table walking must use MVA, this is solved by getting the pgd_offset function to use fcse_va_to_mva, which turns out to be sufficient;
- the TLB and cache operations which may be passed user-space addresses must use MVA, so, they call fcse_va_to_mva too;
- the function do_DataAbort is passed an MVA which has to be converted into a VA, this is done by calling fcse_mva_to_va.

---
 arch/arm/include/asm/cacheflush.h |   26 ++++++++++++++++++++------
 arch/arm/include/asm/fcse.h       |   29 +++++++++++++++++++++++++++++
 arch/arm/include/asm/pgtable.h    |   10 ++++++++--
 arch/arm/include/asm/tlbflush.h   |   14 ++++++++++++--
 arch/arm/mm/fault.c               |    6 ++++++
 arch/arm/mm/flush.c               |   15 ++++++++++-----
 arch/arm/mm/pgd.c                 |    7 ++++++-
 7 files changed, 91 insertions(+), 16 deletions(-)
arch/arm/include/asm/cacheflush.h +20 -6
index fd03fb6..db43631 100644
@@ -15,6 +15,7 @@
 #include <asm/glue.h>
 #include <asm/shmparam.h>
 #include <asm/cachetype.h>
+#include <asm/fcse.h>
 
 #define CACHE_COLOUR(vaddr)	((vaddr & (SHMLBA - 1)) >> PAGE_SHIFT)
 
@@ -341,16 +342,20 @@ static inline void flush_cache_mm(struct mm_struct *mm)
 static inline void
 flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end)
 {
-	if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm)))
+	if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) {
+		start = fcse_va_to_mva(vma->vm_mm, start);
+		end = fcse_va_to_mva(vma->vm_mm, end);
 		__cpuc_flush_user_range(start & PAGE_MASK, PAGE_ALIGN(end),
 					vma->vm_flags);
+	}
 }
 
 static inline void
 flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr, unsigned long pfn)
 {
 	if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) {
-		unsigned long addr = user_addr & PAGE_MASK;
+		unsigned long addr;
+		addr = fcse_va_to_mva(vma->vm_mm, user_addr) & PAGE_MASK;
 		__cpuc_flush_user_range(addr, addr + PAGE_SIZE, vma->vm_flags);
 	}
 }
@@ -381,14 +386,22 @@ extern void flush_ptrace_access(struct vm_area_struct *vma, struct page *page,
  * Harvard caches are synchronised for the user space address range.
  * This is used for the ARM private sys_cacheflush system call.
  */
-#define flush_cache_user_range(vma,start,end) \
-	__cpuc_coherent_user_range((start) & PAGE_MASK, PAGE_ALIGN(end))
+#define flush_cache_user_range(vma, start, end)				\
+	({								\
+		struct mm_struct *_mm = (vma)->vm_mm;			\
+		unsigned long _start, _end;				\
+		_start = fcse_va_to_mva(_mm, start) & PAGE_MASK;	\
+		_end = PAGE_ALIGN(fcse_va_to_mva(_mm, end));		\
+		__cpuc_coherent_user_range(_start, _end);		\
+	})
 
 /*
  * Perform necessary cache operations to ensure that data previously
  * stored within this range of addresses can be executed by the CPU.
  */
-#define flush_icache_range(s,e)		__cpuc_coherent_kern_range(s,e)
+#define flush_icache_range(s,e)						\
+	__cpuc_coherent_kern_range(fcse_va_to_mva(current->mm, (s)),	\
+				   fcse_va_to_mva(current->mm, (e)))
 
 /*
  * Perform necessary cache operations to ensure that the TLB will
@@ -426,7 +439,8 @@ static inline void flush_anon_page(struct vm_area_struct *vma,
 	extern void __flush_anon_page(struct vm_area_struct *vma,
 				struct page *, unsigned long);
 	if (PageAnon(page))
-		__flush_anon_page(vma, page, vmaddr);
+		__flush_anon_page(vma, page,
+				  fcse_va_to_mva(vma->vm_mm, vmaddr));
 }
 
 #define ARCH_HAS_FLUSH_KERNEL_DCACHE_PAGE
arch/arm/include/asm/fcse.h +29 -0
index b44e866..e443278 100644
@@ -27,6 +27,9 @@
 /* Size of PID relocation area */
 #define FCSE_PID_TASK_SIZE (1UL << FCSE_PID_SHIFT)
 
+/* Mask to get rid of PID from relocated address */
+#define FCSE_PID_MASK (FCSE_PID_TASK_SIZE - 1)
+
 /* Sets the CPU's PID Register */
 static inline void fcse_pid_set(unsigned long pid)
 {
@@ -34,11 +37,37 @@ static inline void fcse_pid_set(unsigned long pid)
 			      : /* */: "r" (pid) : "memory");
 }
 
+/* Returns the state of the CPU's PID Register */
+static inline unsigned long fcse_pid_get(void)
+{
+	unsigned long pid;
+	__asm__ __volatile__("mrc p15, 0, %0, c13, c0, 0" : "=&r" (pid));
+	return pid & ~FCSE_PID_MASK;
+}
+
+static inline unsigned long fcse_mva_to_va(unsigned long mva)
+{
+	unsigned long pid = fcse_pid_get();
+	if (pid && (pid == (mva & ~FCSE_PID_MASK)))
+		return mva & FCSE_PID_MASK;
+	return mva;
+}
+
+static inline unsigned long
+fcse_va_to_mva(struct mm_struct *mm, unsigned long va)
+{
+	if (va < FCSE_PID_TASK_SIZE)
+		return mm->context.pid | va;
+	return va;
+}
+
 int fcse_pid_alloc(void);
 void fcse_pid_free(unsigned pid);
 
 #else /* ! CONFIG_ARM_FCSE */
 #define fcse_pid_set(pid) do { } while (0)
+#define fcse_mva_to_va(x) (x)
+#define fcse_va_to_mva(mm, x) ({ (void)(mm); (x); })
 #endif /* ! CONFIG_ARM_FCSE */
 
 #endif /* __ASM_ARM_FCSE_H */
arch/arm/include/asm/pgtable.h +8 -2
index 201ccaa..197c596 100644
@@ -112,6 +112,8 @@
 #define LIBRARY_TEXT_START	0x0c000000
 
 #ifndef __ASSEMBLY__
+#include <asm/fcse.h>
+
 extern void __pte_error(const char *file, int line, unsigned long val);
 extern void __pmd_error(const char *file, int line, unsigned long val);
 extern void __pgd_error(const char *file, int line, unsigned long val);
@@ -362,10 +364,14 @@ static inline pte_t *pmd_page_vaddr(pmd_t pmd)
 /* to find an entry in a page-table-directory */
 #define pgd_index(addr)		((addr) >> PGDIR_SHIFT)
 
-#define pgd_offset(mm, addr)	((mm)->pgd+pgd_index(addr))
+#define pgd_offset(mm, addr)						\
+	({								\
+		struct mm_struct *_mm = (mm);				\
+		(_mm->pgd + pgd_index(fcse_va_to_mva(_mm, (addr))));	\
+	})
 
 /* to find an entry in a kernel page-table-directory */
-#define pgd_offset_k(addr)	pgd_offset(&init_mm, addr)
+#define pgd_offset_k(addr)	(init_mm.pgd + pgd_index(addr))
 
 /* Find an entry in the second-level page table.. */
 #define pmd_offset(dir, addr)	((pmd_t *)(dir))
arch/arm/include/asm/tlbflush.h +12 -2
index a45ab5d..7e14a84 100644
@@ -210,6 +210,7 @@
 #ifndef __ASSEMBLY__
 
 #include <linux/sched.h>
+#include <asm/fcse.h>
 
 struct cpu_tlb_fns {
 	void (*flush_user_range)(unsigned long, unsigned long, struct vm_area_struct *);
@@ -383,7 +384,8 @@ local_flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr)
 	const int zero = 0;
 	const unsigned int __tlb_flag = __cpu_tlb_flags;
 
-	uaddr = (uaddr & PAGE_MASK) | ASID(vma->vm_mm);
+	uaddr = (fcse_va_to_mva(vma->vm_mm, uaddr) & PAGE_MASK)
+		| ASID(vma->vm_mm);
 
 	if (tlb_flag(TLB_WB))
 		dsb();
@@ -504,7 +506,15 @@ static inline void clean_pmd_entry(pmd_t *pmd)
 /*
  * Convert calls to our calling convention.
  */
-#define local_flush_tlb_range(vma,start,end)	__cpu_flush_user_tlb_range(start,end,vma)
+#define local_flush_tlb_range(vma, start, end)			\
+	({							\
+		struct mm_struct *_mm = (vma)->vm_mm;		\
+		unsigned long _start, _end;			\
+		_start = fcse_va_to_mva(_mm, start);		\
+		_end = fcse_va_to_mva(_mm, end);		\
+		__cpu_flush_user_tlb_range(_start, _end, vma);	\
+	})
+
 #define local_flush_tlb_kernel_range(s,e)	__cpu_flush_kern_tlb_range(s,e)
 
 #ifndef CONFIG_SMP
arch/arm/mm/fault.c +6 -0
index 379f785..493c980 100644
@@ -73,6 +73,10 @@ void show_pte(struct mm_struct *mm, unsigned long addr)
 	if (!mm)
 		mm = &init_mm;
 
+#ifdef CONFIG_ARM_FCSE
+	printk(KERN_ALERT "fcse pid: %ld, 0x%08lx\n",
+	       mm->context.pid >> FCSE_PID_SHIFT, mm->context.pid);
+#endif /* CONFIG_ARM_FCSE */
 	printk(KERN_ALERT "pgd = %p\n", mm->pgd);
 	pgd = pgd_offset(mm, addr);
 	printk(KERN_ALERT "[%08lx] *pgd=%08lx", addr, pgd_val(*pgd));
@@ -506,6 +510,8 @@ do_DataAbort(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
 	const struct fsr_info *inf = fsr_info + fsr_fs(fsr);
 	struct siginfo info;
 
+	addr = fcse_mva_to_va(addr);
+
 	if (!inf->fn(addr, fsr & ~FSR_LNX_PF, regs))
 		return;
 
arch/arm/mm/flush.c +10 -5
index b279429..ec6fcda 100644
@@ -73,9 +73,11 @@ void flush_cache_mm(struct mm_struct *mm)
 void flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end)
 {
 	if (cache_is_vivt()) {
-		if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm)))
-			__cpuc_flush_user_range(start & PAGE_MASK, PAGE_ALIGN(end),
-						vma->vm_flags);
+		if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) {
+			start = fcse_va_to_mva(vma->vm_mm, start) & PAGE_MASK;
+			end = PAGE_ALIGN(fcse_va_to_mva(vma->vm_mm, end));
+			__cpuc_flush_user_range(start, end, vma->vm_flags);
+		}
 		return;
 	}
 
@@ -98,8 +100,11 @@ void flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr, unsig
 {
 	if (cache_is_vivt()) {
 		if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) {
-			unsigned long addr = user_addr & PAGE_MASK;
-			__cpuc_flush_user_range(addr, addr + PAGE_SIZE, vma->vm_flags);
+			unsigned long addr;
+			addr = fcse_va_to_mva(vma->vm_mm, user_addr);
+			addr &= PAGE_MASK;
+			__cpuc_flush_user_range(addr, addr + PAGE_SIZE,
+						vma->vm_flags);
 		}
 		return;
 	}
arch/arm/mm/pgd.c +6 -1
index 2690146..9796798 100644
@@ -43,6 +43,11 @@ pgd_t *get_pgd_slow(struct mm_struct *mm)
 	clean_dcache_area(new_pgd, PTRS_PER_PGD * sizeof(pgd_t));
 
 	if (!vectors_high()) {
+#ifdef CONFIG_ARM_FCSE
+		/* FCSE does not work without high vectors. */
+		BUG();
+#endif /* CONFIG_ARM_FCSE */
+
 		/*
 		 * On ARM, first page must always be allocated since it
 		 * contains the machine vectors.
@@ -81,7 +86,7 @@ void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd)
 		return;
 
 	/* pgd is always present and good */
-	pmd = pmd_off(pgd, 0);
+	pmd = pmd_off(pgd + pgd_index(fcse_va_to_mva(mm, 0)), 0);
 	if (pmd_none(*pmd))
 		goto free;
 	if (pmd_bad(*pmd)) {
Signature · 1 line
1.5.6.5
#3 lore raw

PATCH 1/1 FCSE: Address-space limits.

Get the kernel macros to reflect address-space limits imposed by the FCSE.

---
 arch/arm/include/asm/memory.h    |    5 +++++
 arch/arm/include/asm/processor.h |    5 +++++
 arch/arm/mm/mmap.c               |   22 +++++++++++++++++++---
 3 files changed, 29 insertions(+), 3 deletions(-)
arch/arm/include/asm/memory.h +5 -0
index cefedf0..db9781a 100644
@@ -33,7 +33,12 @@
  */
 #define PAGE_OFFSET		UL(CONFIG_PAGE_OFFSET)
 #define TASK_SIZE		(UL(CONFIG_PAGE_OFFSET) - UL(0x01000000))
+#ifndef CONFIG_ARM_FCSE
 #define TASK_UNMAPPED_BASE	(UL(CONFIG_PAGE_OFFSET) / 3)
+#else /* CONFIG_ARM_FCSE */
+#define TASK_UNMAPPED_BASE	UL(0x01000000)
+#define FCSE_TASK_SIZE		UL(0x02000000)
+#endif /* CONFIG_ARM_FCSE */
 
 /*
  * The maximum size of a 26-bit user space task.
arch/arm/include/asm/processor.h +5 -0
index 6a89567..bc33df1 100644
@@ -23,9 +23,14 @@
 #include <asm/types.h>
 
 #ifdef __KERNEL__
+#ifndef CONFIG_ARM_FCSE
 #define STACK_TOP	((current->personality & ADDR_LIMIT_32BIT) ? \
 			 TASK_SIZE : TASK_SIZE_26)
 #define STACK_TOP_MAX	TASK_SIZE
+#else /* CONFIG_ARM_FCSE */
+#define STACK_TOP	FCSE_TASK_SIZE
+#define STACK_TOP_MAX	FCSE_TASK_SIZE
+#endif /* CONFIG_ARM_FCSE */
 #endif
 
 union debug_insn {
arch/arm/mm/mmap.c +19 -3
index f7457fe..e608bda 100644
@@ -50,13 +50,16 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr,
 #define aliasing 0
 #endif
 
+#ifdef CONFIG_ARM_FCSE
+	start_addr = addr;
+#endif /* CONFIG_ARM_FCSE */
 	/*
 	 * We enforce the MAP_FIXED case.
 	 */
 	if (flags & MAP_FIXED) {
 		if (aliasing && flags & MAP_SHARED && addr & (SHMLBA - 1))
 			return -EINVAL;
-		return addr;
+		goto found_addr;
 	}
 
 	if (len > TASK_SIZE)
@@ -71,7 +74,7 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr,
 		vma = find_vma(mm, addr);
 		if (TASK_SIZE - len >= addr &&
 		    (!vma || addr + len <= vma->vm_start))
-			return addr;
+			goto found_addr;
 	}
 	if (len > mm->cached_hole_size) {
 	        start_addr = addr = mm->free_area_cache;
@@ -105,7 +108,7 @@ full_search:
 			 * Remember the place where we stopped the search:
 			 */
 			mm->free_area_cache = addr + len;
-			return addr;
+			goto found_addr;
 		}
 		if (addr + mm->cached_hole_size < vma->vm_start)
 		        mm->cached_hole_size = vma->vm_start - addr;
@@ -113,6 +116,19 @@ full_search:
 		if (do_align)
 			addr = COLOUR_ALIGN(addr, pgoff);
 	}
+
+found_addr:
+#ifdef CONFIG_ARM_FCSE
+	if (addr + len > FCSE_TASK_SIZE) {
+		if (!(flags & MAP_FIXED) && start_addr != TASK_UNMAPPED_BASE) {
+			start_addr = addr = TASK_UNMAPPED_BASE;
+			mm->cached_hole_size = 0;
+			goto full_search;
+		}
+		return -ENOMEM;
+	}
+#endif /* CONFIG_ARM_FCSE */
+	return addr;
 }
 
 
Signature · 1 line
1.5.6.5
#4 lore raw

PATCH 1/1 FCSE: differentiate cpu_tlb_mask from cpu_vm_mask.

The ARM Linux kernel uses mm->cpu_vm_mask to avoid unnecessary TLB and cache flushes when the process about to be flushed has already been flushed because of a context switch.

When enabling FCSE, a context switch flushes the TLB, but no longer flushes the cache, so, for correctness, we need to distinguish whether a process is "present" in the TLB by using a second mask mm->context.cpu_tlb_mask, or whether it is "present" in the cache by using mm->cpu_vm_mask.

The current cpu bit is set and cleared from cpu_tlb_mask upon context switch, but only set in cpu_vm_mask.

---
 arch/arm/include/asm/fcse.h        |    5 +++++
 arch/arm/include/asm/mmu.h         |    1 +
 arch/arm/include/asm/mmu_context.h |    7 +++++--
 arch/arm/include/asm/tlbflush.h    |    4 ++--
 arch/arm/kernel/smp.c              |    7 ++++---
 5 files changed, 17 insertions(+), 7 deletions(-)
arch/arm/include/asm/fcse.h +5 -0
index e443278..4e9503d 100644
@@ -30,6 +30,9 @@
 /* Mask to get rid of PID from relocated address */
 #define FCSE_PID_MASK (FCSE_PID_TASK_SIZE - 1)
 
+#define fcse_tlb_mask(mm) (&(mm)->context.cpu_tlb_mask)
+#define fcse_cpu_set_vm_mask(cpu, mm) cpumask_set_cpu(cpu, mm_cpumask(mm))
+
 /* Sets the CPU's PID Register */
 static inline void fcse_pid_set(unsigned long pid)
 {
@@ -68,6 +71,8 @@ void fcse_pid_free(unsigned pid);
 #define fcse_pid_set(pid) do { } while (0)
 #define fcse_mva_to_va(x) (x)
 #define fcse_va_to_mva(mm, x) ({ (void)(mm); (x); })
+#define fcse_tlb_mask(mm) mm_cpumask(mm)
+#define fcse_cpu_set_vm_mask(cpu, mm) do { } while (0)
 #endif /* ! CONFIG_ARM_FCSE */
 
 #endif /* __ASM_ARM_FCSE_H */
arch/arm/include/asm/mmu.h +1 -0
index 4a5a8db..7472598 100644
@@ -9,6 +9,7 @@ typedef struct {
 #endif
 #ifdef CONFIG_ARM_FCSE
 	unsigned long pid;
+	cpumask_t cpu_tlb_mask;
 #endif /* CONFIG_ARM_FCSE */
 	unsigned int kvm_seq;
 } mm_context_t;
arch/arm/include/asm/mmu_context.h +5 -2
index ec7a84d..042a26d 100644
@@ -75,6 +75,8 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm)
 #ifdef CONFIG_ARM_FCSE
 	int pid;
 
+	cpus_clear(mm->context.cpu_tlb_mask);
+
 	pid = fcse_pid_alloc();
 	if (pid < 0)
 		return pid;
@@ -126,12 +128,13 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,
 	    !cpumask_test_cpu(cpu, mm_cpumask(next)))
 		__flush_icache_all();
 #endif
-	if (!cpumask_test_and_set_cpu(cpu, mm_cpumask(next)) || prev != next) {
+	if (!cpumask_test_and_set_cpu(cpu, fcse_tlb_mask(next)) || prev != next) {
+		fcse_cpu_set_vm_mask(cpu, next);
 		check_context(next);
 		fcse_pid_set(next->context.pid);
 		cpu_switch_mm(next->pgd, next);
 		if (cache_is_vivt())
-			cpumask_clear_cpu(cpu, mm_cpumask(prev));
+			cpumask_clear_cpu(cpu, fcse_tlb_mask(prev));
 	}
 #endif
 }
arch/arm/include/asm/tlbflush.h +2 -2
index 7e14a84..86355b3 100644
@@ -351,7 +351,7 @@ static inline void local_flush_tlb_mm(struct mm_struct *mm)
 	if (tlb_flag(TLB_WB))
 		dsb();
 
-	if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) {
+	if (cpumask_test_cpu(smp_processor_id(), fcse_tlb_mask(mm))) {
 		if (tlb_flag(TLB_V3_FULL))
 			asm("mcr p15, 0, %0, c6, c0, 0" : : "r" (zero) : "cc");
 		if (tlb_flag(TLB_V4_U_FULL))
@@ -390,7 +390,7 @@ local_flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr)
 	if (tlb_flag(TLB_WB))
 		dsb();
 
-	if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) {
+	if (cpumask_test_cpu(smp_processor_id(), fcse_tlb_mask(vma->vm_mm))) {
 		if (tlb_flag(TLB_V3_PAGE))
 			asm("mcr p15, 0, %0, c6, c0, 0" : : "r" (uaddr) : "cc");
 		if (tlb_flag(TLB_V4_U_PAGE))
arch/arm/kernel/smp.c +4 -3
index e0d3277..a0eb423 100644
@@ -36,6 +36,7 @@
 #include <asm/tlbflush.h>
 #include <asm/ptrace.h>
 #include <asm/localtimer.h>
+#include <asm/fcse.h>
 
 /*
  * as from 2.5, kernels no longer have an init_tasks structure
@@ -643,7 +644,7 @@ void flush_tlb_all(void)
 void flush_tlb_mm(struct mm_struct *mm)
 {
 	if (tlb_ops_need_broadcast())
-		on_each_cpu_mask(ipi_flush_tlb_mm, mm, 1, mm_cpumask(mm));
+		on_each_cpu_mask(ipi_flush_tlb_mm, mm, 1, fcse_tlb_mask(mm));
 	else
 		local_flush_tlb_mm(mm);
 }
@@ -654,7 +655,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr)
 		struct tlb_args ta;
 		ta.ta_vma = vma;
 		ta.ta_start = uaddr;
-		on_each_cpu_mask(ipi_flush_tlb_page, &ta, 1, mm_cpumask(vma->vm_mm));
+		on_each_cpu_mask(ipi_flush_tlb_page, &ta, 1, fcse_tlb_mask(vma->vm_mm));
 	} else
 		local_flush_tlb_page(vma, uaddr);
 }
@@ -677,7 +678,7 @@ void flush_tlb_range(struct vm_area_struct *vma,
 		ta.ta_vma = vma;
 		ta.ta_start = start;
 		ta.ta_end = end;
-		on_each_cpu_mask(ipi_flush_tlb_range, &ta, 1, mm_cpumask(vma->vm_mm));
+		on_each_cpu_mask(ipi_flush_tlb_range, &ta, 1, fcse_tlb_mask(vma->vm_mm));
 	} else
 		local_flush_tlb_range(vma, start, end);
 }
Signature · 1 line
1.5.6.5
#5 lore raw

PATCH 1/1 FCSE: Make the shared mappings uncacheable.

---
 arch/arm/mm/fault-armv.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)
arch/arm/mm/fault-armv.c +5 -0
index bc0099d..40bccb9 100644
@@ -94,6 +94,7 @@ no_pmd:
 static void
 make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigned long addr, unsigned long pfn)
 {
+#ifndef CONFIG_ARM_FCSE_GUARANTEED
 	struct mm_struct *mm = vma->vm_mm;
 	struct vm_area_struct *mpnt;
 	struct prio_tree_iter iter;
@@ -127,6 +128,10 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne
 		adjust_pte(vma, addr);
 	else
 		flush_cache_page(vma, addr, pfn);
+#else /* CONFIG_ARM_FCSE_GUARANTEED */
+	if (vma->vm_flags & VM_MAYSHARE)
+		adjust_pte(vma, addr);
+#endif /* CONFIG_ARM_FCSE_GUARANTEED */
 }
 
 /*
Signature · 1 line
1.5.6.5
#6 lore raw

PATCH 1/1 "Best-effort" FCSE: choose whether to flush cache at run-time.

The implementation of the FCSE "best-effort" mode requires to decide, when switching contexts, whether to flush the cache. So, we add a parameter to the mm switching function telling whether we want to flush the cache, and the switch_mm macro uses the fcse_needs_flush to get the value of this parameter.

---
 arch/arm/include/asm/cpu-multi32.h |    5 +++--
 arch/arm/include/asm/cpu-single.h  |    3 ++-
 arch/arm/include/asm/fcse.h        |    6 ++++++
 arch/arm/include/asm/mmu_context.h |    2 +-
 arch/arm/include/asm/proc-fns.h    |    3 ++-
 arch/arm/kernel/fcse.c             |   11 +++++++++++
 arch/arm/mm/proc-arm920.S          |    9 +++++++++
 arch/arm/mm/proc-arm926.S          |    9 +++++++++
 arch/arm/mm/proc-xscale.S          |    9 +++++++++
 9 files changed, 52 insertions(+), 5 deletions(-)
arch/arm/include/asm/cpu-multi32.h +3 -2
index e2b5b0b..18ca8a0 100644
@@ -52,7 +52,8 @@ extern struct processor {
 	/*
 	 * Set the page table
 	 */
-	void (*switch_mm)(unsigned long pgd_phys, struct mm_struct *mm);
+	void (*switch_mm)(unsigned long pgd_phys,
+			  struct mm_struct *mm, unsigned flush);
 	/*
 	 * Set a possibly extended PTE.  Non-extended PTEs should
 	 * ignore 'ext'.
@@ -66,4 +67,4 @@ extern struct processor {
 #define cpu_do_idle()			processor._do_idle()
 #define cpu_dcache_clean_area(addr,sz)	processor.dcache_clean_area(addr,sz)
 #define cpu_set_pte_ext(ptep,pte,ext)	processor.set_pte_ext(ptep,pte,ext)
-#define cpu_do_switch_mm(pgd,mm)	processor.switch_mm(pgd,mm)
+#define cpu_do_switch_mm(pgd,mm,flush)	processor.switch_mm(pgd,mm,flush)
arch/arm/include/asm/cpu-single.h +2 -1
index f073a6d..ec556e3 100644
@@ -39,6 +39,7 @@ extern void cpu_proc_init(void);
 extern void cpu_proc_fin(void);
 extern int cpu_do_idle(void);
 extern void cpu_dcache_clean_area(void *, int);
-extern void cpu_do_switch_mm(unsigned long pgd_phys, struct mm_struct *mm);
+extern void cpu_do_switch_mm(unsigned long pgd_phys,
+			     struct mm_struct *mm, unsigned flush);
 extern void cpu_set_pte_ext(pte_t *ptep, pte_t pte, unsigned int ext);
 extern void cpu_reset(unsigned long addr) __attribute__((noreturn));
arch/arm/include/asm/fcse.h +6 -0
index 4e9503d..d769a29 100644
@@ -66,6 +66,11 @@ fcse_va_to_mva(struct mm_struct *mm, unsigned long va)
 
 int fcse_pid_alloc(void);
 void fcse_pid_free(unsigned pid);
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next);
+#else /* CONFIG_ARM_FCSE_GUARANTEED */
+#define fcse_needs_flush(prev, next) (0)
+#endif /* CONFIG_ARM_FCSE_GUARANTEED */
 
 #else /* ! CONFIG_ARM_FCSE */
 #define fcse_pid_set(pid) do { } while (0)
@@ -73,6 +78,7 @@ void fcse_pid_free(unsigned pid);
 #define fcse_va_to_mva(mm, x) ({ (void)(mm); (x); })
 #define fcse_tlb_mask(mm) mm_cpumask(mm)
 #define fcse_cpu_set_vm_mask(cpu, mm) do { } while (0)
+#define fcse_needs_flush(prev, next) (1)
 #endif /* ! CONFIG_ARM_FCSE */
 
 #endif /* __ASM_ARM_FCSE_H */
arch/arm/include/asm/mmu_context.h +1 -1
index 042a26d..ad1bbb4 100644
@@ -132,7 +132,7 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,
 		fcse_cpu_set_vm_mask(cpu, next);
 		check_context(next);
 		fcse_pid_set(next->context.pid);
-		cpu_switch_mm(next->pgd, next);
+		cpu_switch_mm(next->pgd, next, fcse_needs_flush(prev, next));
 		if (cache_is_vivt())
 			cpumask_clear_cpu(cpu, fcse_tlb_mask(prev));
 	}
arch/arm/include/asm/proc-fns.h +2 -1
index 3976412..86f3131 100644
@@ -239,7 +239,8 @@
 
 #ifdef CONFIG_MMU
 
-#define cpu_switch_mm(pgd,mm) cpu_do_switch_mm(virt_to_phys(pgd),mm)
+#define cpu_switch_mm(pgd,mm,flush) \
+	cpu_do_switch_mm(virt_to_phys(pgd), (mm), (flush))
 
 #define cpu_get_pgd()	\
 	({						\
arch/arm/kernel/fcse.c +11 -0
index da074b8..7918be6 100644
@@ -55,3 +55,14 @@ void fcse_pid_free(unsigned pid)
 	fcse_pid_dereference(pid);
 	spin_unlock_irqrestore(&fcse_lock, flags);
 }
+
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next)
+{
+	unsigned res;
+
+	res = 0;
+
+	return res;
+}
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
arch/arm/mm/proc-arm920.S +9 -0
index 914d688..3eff19c 100644
@@ -321,6 +321,11 @@ ENTRY(cpu_arm920_dcache_clean_area)
 ENTRY(cpu_arm920_switch_mm)
 #ifdef CONFIG_MMU
 	mov	ip, #0
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	cmp	r2, #0
+	beq	3f
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
+#ifndef CONFIG_ARM_FCSE_GUARANTEED
 #ifdef CONFIG_CPU_DCACHE_WRITETHROUGH
 	mcr	p15, 0, ip, c7, c6, 0		@ invalidate D cache
 #else
@@ -338,6 +343,10 @@ ENTRY(cpu_arm920_switch_mm)
 #endif
 	mcr	p15, 0, ip, c7, c5, 0		@ invalidate I cache
 	mcr	p15, 0, ip, c7, c10, 4		@ drain WB
+#endif /* !CONFIG_ARM_FCSE_GUARANTEED */
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+3:
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 	mcr	p15, 0, r0, c2, c0, 0		@ load page table pointer
 	mcr	p15, 0, ip, c8, c7, 0		@ invalidate I & D TLBs
 #endif
arch/arm/mm/proc-arm926.S +9 -0
index 5446693..f7a6021 100644
@@ -337,6 +337,11 @@ ENTRY(cpu_arm926_dcache_clean_area)
 ENTRY(cpu_arm926_switch_mm)
 #ifdef CONFIG_MMU
 	mov	ip, #0
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	cmp	r2, #0
+	beq	2f
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
+#ifndef CONFIG_ARM_FCSE_GUARANTEED
 #ifdef CONFIG_CPU_DCACHE_WRITETHROUGH
 	mcr	p15, 0, ip, c7, c6, 0		@ invalidate D cache
 #else
@@ -346,6 +351,10 @@ ENTRY(cpu_arm926_switch_mm)
 #endif
 	mcr	p15, 0, ip, c7, c5, 0		@ invalidate I cache
 	mcr	p15, 0, ip, c7, c10, 4		@ drain WB
+#endif /* CONFIG_ARM_FCSE_GUARANTEED */
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+2:
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 	mcr	p15, 0, r0, c2, c0, 0		@ load page table pointer
 	mcr	p15, 0, ip, c8, c7, 0		@ invalidate I & D TLBs
 #endif
arch/arm/mm/proc-xscale.S +9 -0
index 4233942..f09d135 100644
@@ -415,9 +415,18 @@ ENTRY(cpu_xscale_dcache_clean_area)
  */
 	.align	5
 ENTRY(cpu_xscale_switch_mm)
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	cmp	r2, #0
+	beq	2f
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
+#ifndef CONFIG_ARM_FCSE_GUARANTEED
 	clean_d_cache r1, r2
 	mcr	p15, 0, ip, c7, c5, 0		@ Invalidate I cache & BTB
 	mcr	p15, 0, ip, c7, c10, 4		@ Drain Write (& Fill) Buffer
+#endif /* CONFIG_ARM_FCSE_GUARANTEED */
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+2:
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 	mcr	p15, 0, r0, c2, c0, 0		@ load page table pointer
 	mcr	p15, 0, ip, c8, c7, 0		@ invalidate I & D TLBs
 	cpwait_ret lr, ip
Signature · 1 line
1.5.6.5
#7 lore raw

PATCH 1/1 "Best-effort" FCSE: Allow PID re-use.

In addition to the bit telling whether or not a pid is allocated, maintain:
- a count of processes using a pid;
- the current process using a given pid;
- a bitfield telling whether or not the cache may contain entries for a given pid.

A function fcse_notify_flush_all() is called upon calls to flush_cache_mm() and clears all bits in this last bitfield, in order to avoid that fcse_needs_flush() believe that a pid is reused whereas the cache has been cleaned.

---
 arch/arm/include/asm/cacheflush.h |    4 ++-
 arch/arm/include/asm/fcse.h       |    3 ++
 arch/arm/kernel/fcse.c            |   73 +++++++++++++++++++++++++++++++++++--
 arch/arm/mm/flush.c               |    4 ++-
 4 files changed, 79 insertions(+), 5 deletions(-)
arch/arm/include/asm/cacheflush.h +3 -1
index db43631..7ced6d6 100644
@@ -335,8 +335,10 @@ static inline void outer_flush_range(unsigned long start, unsigned long end)
 #ifndef CONFIG_CPU_CACHE_VIPT
 static inline void flush_cache_mm(struct mm_struct *mm)
 {
-	if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm)))
+	if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) {
+		fcse_notify_flush_all();
 		__cpuc_flush_user_all();
+	}
 }
 
 static inline void
arch/arm/include/asm/fcse.h +3 -0
index d769a29..c8680c6 100644
@@ -68,8 +68,10 @@ int fcse_pid_alloc(void);
 void fcse_pid_free(unsigned pid);
 #ifdef CONFIG_ARM_FCSE_BEST_EFFORT
 int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next);
+void fcse_notify_flush_all(void);
 #else /* CONFIG_ARM_FCSE_GUARANTEED */
 #define fcse_needs_flush(prev, next) (0)
+#define fcse_notify_flush_all() do { } while (0)
 #endif /* CONFIG_ARM_FCSE_GUARANTEED */
 
 #else /* ! CONFIG_ARM_FCSE */
@@ -79,6 +81,7 @@ int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next);
 #define fcse_tlb_mask(mm) mm_cpumask(mm)
 #define fcse_cpu_set_vm_mask(cpu, mm) do { } while (0)
 #define fcse_needs_flush(prev, next) (1)
+#define fcse_notify_flush_all() do { } while (0)
 #endif /* ! CONFIG_ARM_FCSE */
 
 #endif /* __ASM_ARM_FCSE_H */
arch/arm/kernel/fcse.c +70 -3

104 lines not on this page · raw message

arch/arm/mm/flush.c +3 -1
index ec6fcda..fba8619 100644
@@ -50,8 +50,10 @@ static void flush_pfn_alias(unsigned long pfn, unsigned long vaddr)
 void flush_cache_mm(struct mm_struct *mm)
 {
 	if (cache_is_vivt()) {
-		if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm)))
+		if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) {
+			fcse_notify_flush_all();
 			__cpuc_flush_user_all();
+		}
 		return;
 	}
 
Signature · 1 line
1.5.6.5
#8 lore raw

PATCH 1/1 "Best-effort" FCSE: Handle shared mappings.

Account for writable shared mappings, so as to be able to flush cache when switching out a process with such a mapping.

This accounting is done by adding the L_PTE_SHARED flag to page belonging to such mappings in make_coherent(), then, set_pte_at() is intercepted to account for both addition and removal of pages with this flag set.

Note that using the L_PTE_SHARED flag makes the FCSE patch incompatible with ARMv6+ and Xscale3, but the FCSE feature is deprecated in ARMv6 anyway.

---
 arch/arm/include/asm/mmu.h         |    3 ++
 arch/arm/include/asm/mmu_context.h |    6 ++++
 arch/arm/include/asm/pgtable.h     |   42 ++++++++++++++++++++++++--
 arch/arm/kernel/fcse.c             |    3 +-
 arch/arm/mm/fault-armv.c           |   55 +++++++++++++++++++++++++++++++++++-
 5 files changed, 103 insertions(+), 6 deletions(-)
arch/arm/include/asm/mmu.h +3 -0
index 7472598..9a2fb51 100644
@@ -10,6 +10,9 @@ typedef struct {
 #ifdef CONFIG_ARM_FCSE
 	unsigned long pid;
 	cpumask_t cpu_tlb_mask;
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	unsigned shared_dirty_pages;
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 #endif /* CONFIG_ARM_FCSE */
 	unsigned int kvm_seq;
 } mm_context_t;
arch/arm/include/asm/mmu_context.h +6 -0
index ad1bbb4..035a9f2 100644
@@ -76,6 +76,9 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm)
 	int pid;
 
 	cpus_clear(mm->context.cpu_tlb_mask);
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	mm->context.shared_dirty_pages = 0;
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 
 	pid = fcse_pid_alloc();
 	if (pid < 0)
@@ -91,6 +94,9 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm)
 static inline void destroy_context(struct mm_struct *mm)
 {
 #ifdef CONFIG_ARM_FCSE
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	BUG_ON(mm->context.shared_dirty_pages);
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 	fcse_pid_free(mm->context.pid >> FCSE_PID_SHIFT);
 #endif /* CONFIG_ARM_FCSE */
 }
arch/arm/include/asm/pgtable.h +38 -4

60 lines not on this page · raw message

arch/arm/kernel/fcse.c +2 -1
index 774717c..5b257c9 100644
@@ -118,7 +118,8 @@ int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next)
 	__set_bit(pid, fcse_pids_cache_dirty);
 	spin_unlock_irqrestore(&fcse_lock, flags);
 
-	res = reused_pid;
+	res = reused_pid
+		|| prev->context.shared_dirty_pages;
 
 	if (res) {
 		cpu_clear(smp_processor_id(), prev->cpu_vm_mask);
arch/arm/mm/fault-armv.c +54 -1

68 lines not on this page · raw message

Signature · 1 line
1.5.6.5
#9 lore raw

PATCH 1/1 "Best-effort" FCSE: Handle mappings above 32 MB

In case an address larger than 32 MB is returned by arch_get_unmapped_area(), relocate the calling process to the null pid. When such a process forks, also give the child process a null pid. Account for pages above 32 MB, and when a process with such pages is switched in or out, flush the cache.

As in handling shared mappings, we use set_pte_at() to do the pages accounting.

---
 arch/arm/include/asm/fcse.h        |    2 +
 arch/arm/include/asm/mmu.h         |    2 +
 arch/arm/include/asm/mmu_context.h |   17 +++++++++++++-
 arch/arm/include/asm/pgtable.h     |    7 ++++++
 arch/arm/kernel/fcse.c             |   41 +++++++++++++++++++++++++++++++++++-
 arch/arm/mm/mmap.c                 |    7 ++++++
 6 files changed, 73 insertions(+), 3 deletions(-)
arch/arm/include/asm/fcse.h +2 -0
index c8680c6..66fa9de 100644
@@ -69,6 +69,8 @@ void fcse_pid_free(unsigned pid);
 #ifdef CONFIG_ARM_FCSE_BEST_EFFORT
 int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next);
 void fcse_notify_flush_all(void);
+void fcse_pid_reference(unsigned pid);
+void fcse_relocate_mm_to_null_pid(struct mm_struct *mm);
 #else /* CONFIG_ARM_FCSE_GUARANTEED */
 #define fcse_needs_flush(prev, next) (0)
 #define fcse_notify_flush_all() do { } while (0)
arch/arm/include/asm/mmu.h +2 -0
index 9a2fb51..13c844f 100644
@@ -12,6 +12,8 @@ typedef struct {
 	cpumask_t cpu_tlb_mask;
 #ifdef CONFIG_ARM_FCSE_BEST_EFFORT
 	unsigned shared_dirty_pages;
+	unsigned big;
+	unsigned high_pages;
 #endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 #endif /* CONFIG_ARM_FCSE */
 	unsigned int kvm_seq;
arch/arm/include/asm/mmu_context.h +15 -2
index 035a9f2..85eac0a 100644
@@ -77,13 +77,25 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm)
 
 	cpus_clear(mm->context.cpu_tlb_mask);
 #ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	if (!mm->context.big) {
+		pid = fcse_pid_alloc();
+		mm->context.pid = pid << FCSE_PID_SHIFT;
+	} else {
+		/* We are normally forking a process vith a virtual address
+		   space larger than 32 MB, so its pid should be 0. */
+		BUG_ON(mm->context.pid);
+		fcse_pid_reference(0);
+	}
+	/* If we are forking, set_pte_at will restore the correct high pages
+	   count, and shared writable pages are write-protected again. */
 	mm->context.shared_dirty_pages = 0;
-#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
-
+	mm->context.high_pages = 0;
+#else /* CONFIG_ARM_FCSE_GUARANTEED */
 	pid = fcse_pid_alloc();
 	if (pid < 0)
 		return pid;
 	mm->context.pid = pid << FCSE_PID_SHIFT;
+#endif /* CONFIG_ARM_FCSE_GUARANTEED */
 #endif /* CONFIG_ARM_FCSE */
 
 	return 0;
@@ -96,6 +108,7 @@ static inline void destroy_context(struct mm_struct *mm)
 #ifdef CONFIG_ARM_FCSE
 #ifdef CONFIG_ARM_FCSE_BEST_EFFORT
 	BUG_ON(mm->context.shared_dirty_pages);
+	BUG_ON(mm->context.high_pages);
 #endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 	fcse_pid_free(mm->context.pid >> FCSE_PID_SHIFT);
 #endif /* CONFIG_ARM_FCSE */
arch/arm/include/asm/pgtable.h +7 -0
index 7775c1b..6c79d71 100644
@@ -257,6 +257,10 @@ extern pgprot_t		pgprot_kernel;
 	unsigned long _val = (val);				\
 	if (pte_present(_val) && ((_val) & L_PTE_SHARED))	\
 		--_mm->context.shared_dirty_pages;		\
+	if (pte_present(_val) && _addr < TASK_SIZE) {		\
+		if (_addr >= FCSE_TASK_SIZE)			\
+			--_mm->context.high_pages;		\
+	}							\
 } while (0)
 
 #define fcse_account_page_addition(mm, addr, val) ({			\
@@ -270,6 +274,9 @@ extern pgprot_t		pgprot_kernel;
 		else							\
 			++_mm->context.shared_dirty_pages;		\
 	}								\
+	if (pte_present(_val)						\
+	    && _addr < TASK_SIZE && _addr >= FCSE_TASK_SIZE)		\
+		++_mm->context.high_pages;				\
 	_val;								\
 })
 #else /* CONFIG_ARM_FCSE_GUARANTEED || !CONFIG_ARM_FCSE */
arch/arm/kernel/fcse.c +40 -1

51 lines not on this page · raw message

arch/arm/mm/mmap.c +7 -0
index e608bda..ac81369 100644
@@ -125,7 +125,14 @@ found_addr:
 			mm->cached_hole_size = 0;
 			goto full_search;
 		}
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+		if (mm->context.pid) {
+			mm->context.big = 1;
+			fcse_relocate_mm_to_null_pid(mm);
+		}
+#else /* CONFIG_ARM_FCSE_GUARANTEED */
 		return -ENOMEM;
+#endif /* CONFIG_ARM_FCSE_GUARANTEED */
 	}
 #endif /* CONFIG_ARM_FCSE */
 	return addr;
Signature · 1 line
1.5.6.5
#10 lore raw

PATCH 1/1 FCSE: add Makefile and compilation option.

---
 arch/arm/kernel/Makefile |    1 +
 arch/arm/mm/Kconfig      |   45 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+), 0 deletions(-)
arch/arm/kernel/Makefile +1 -0
index 79087dd..574e77e 100644
@@ -52,5 +52,6 @@ endif
 
 head-y			:= head$(MMUEXT).o
 obj-$(CONFIG_DEBUG_LL)	+= debug.o
+obj-$(CONFIG_ARM_FCSE)	+= fcse.o
 
 extra-y := $(head-y) init_task.o vmlinux.lds
arch/arm/mm/Kconfig +45 -0

48 lines not on this page · raw message

Signature · 1 line
1.5.6.5
#11 lore raw

PATCH 1/1 FCSE: pid life cycle.

In reply to Gilles Chanteperdrix
Quoted 8 lines · Gilles Chanteperdrix

Add a pid member to the mm_context_t structure, allocate this pid in the init_new_context, free it in the destroy_context, switch it in switch_mm.

A bitfield is maintained to know what pids are currently in use.

...

arch/arm/include/asm/fcse.h line 8

Quoted 11 lines
arch/arm/include/asm/fcse.h new file +8 -0
@@ -0,0 +1,44 @@
+/*
+ * Filename:    arch/arm/include/asm/fcse.h
+ * Description: ARM Process ID (PID) includes for Fast Address Space Switching
+ *              (FASS) in ARM Linux.
+ * Created:     14/10/2001
+ * Changes:     19/02/2002 - Macros added.
+ *              03/08/2007 - Adapted to kernel 2.6.21 (ssm)
+ *              Feb 2008   - Simplified a bit (rco)

We normally avoid changelogs in sources.

arch/arm/include/asm/fcse.h line 35

Quoted 6 lines
arch/arm/include/asm/fcse.h new file +6 -0
+/* Sets the CPU's PID Register */
+static inline void fcse_pid_set(unsigned long pid)
+{
+	__asm__ __volatile__ ("mcr p15, 0, %0, c13, c0, 0"
+			      : /* */: "r" (pid) : "memory");
+}

Calling this 'pid' is extremely confusing. PID already means process id...

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#12 lore raw

PATCH 1/1 FCSE: pid life cycle.

Quoted 10 lines · Pavel Machek
Quoted 8 lines · Gilles Chanteperdrix

Add a pid member to the mm_context_t structure, allocate this pid in the init_new_context, free it in the destroy_context, switch it in switch_mm.

A bitfield is maintained to know what pids are currently in use.

...

Quoted 13 lines

arch/arm/include/asm/fcse.h line 8

Quoted 11 lines
arch/arm/include/asm/fcse.h new file +8 -0
@@ -0,0 +1,44 @@
+/*
+ * Filename:    arch/arm/include/asm/fcse.h
+ * Description: ARM Process ID (PID) includes for Fast Address Space Switching
+ *              (FASS) in ARM Linux.
+ * Created:     14/10/2001
+ * Changes:     19/02/2002 - Macros added.
+ *              03/08/2007 - Adapted to kernel 2.6.21 (ssm)
+ *              Feb 2008   - Simplified a bit (rco)

We normally avoid changelogs in sources.

Ok. Will remove that, it was inherited.

Quoted 10 lines

arch/arm/include/asm/fcse.h line 35

Quoted 7 lines
arch/arm/include/asm/fcse.h new file +6 -0
+/* Sets the CPU's PID Register */
+static inline void fcse_pid_set(unsigned long pid)
+{
+	__asm__ __volatile__ ("mcr p15, 0, %0, c13, c0, 0"
+			      : /* */: "r" (pid) : "memory");
+}

Calling this 'pid' is extremely confusing. PID already means process id...

Guess what, it also means "process id" in the FCSE case, and it is the name it has in the documentation, I always find a good idea to name the hardware registers as per documentation. I guess the original intent was to put the real process id into the hardware register, but that is not really possible with Linux. So, I guess we can call it fcse_pid.

In any case, I will keep all modifications for a second post once I had sufficient review (namely, I am expecting Russell's review, since he is the one who will accept or reject the patch in the end).

Signature · 1 line
					    Gilles.