some mm improvements + s390 tlb flush.
Subsystems (5)
cover some mm improvements + s390 tlb flush.
I have updated my mm patch set. The diff:
- The tlb gather restart patch has been changed as discussed with Hugh.
- The ptep_establish patch now really removes all traces of the call.
- The mm_struct / vm_area_struct move to mm_types.h has been test compiled on a number of architectures.
- The page_mkclean_one patch has been dropped because it doesn't seem to have any effect.
- There is a new patch to fix a theoretical architectural problem on s390. The patch is required for the 1K/2K page tables for KVM as well.
Signature · 10 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/5 avoid tlb gather restarts.
If need_resched() is false in the inner loop of unmap_vmas it is unnecessary to do a full blown tlb_finish_mmu / tlb_gather_mmu for each ZAP_BLOCK_SIZE ptes. Do a tlb_flush_mmu() instead. That gives architectures with a non-generic tlb flush implementation room for optimization. The tlb_flush_mmu primitive is a available with the generic tlb flush code, the ia64_tlb_flush_mm needs to be renamed and a dummy function is added to arm and arm26.
---
include/asm-arm/tlb.h | 5 +++++ include/asm-arm26/tlb.h | 5 +++++ include/asm-ia64/tlb.h | 6 +++--- mm/memory.c | 16 ++++++---------- 4 files changed, 19 insertions(+), 13 deletions(-)
include/asm-arm/tlb.h +5 -0
@@ -52,6 +52,11 @@ tlb_gather_mmu(struct mm_struct *mm, uns }static inline void +tlb_flush_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) +{ +} + +static inline void tlb_finish_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) {if (tlb->fullmm)
include/asm-arm26/tlb.h +5 -0
@@ -29,6 +29,11 @@ tlb_gather_mmu(struct mm_struct *mm, uns }static inline void +tlb_flush_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) +{ +} + +static inline void tlb_finish_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) {if (tlb->need_flush)
include/asm-ia64/tlb.h +3 -3
@@ -72,7 +72,7 @@ DECLARE_PER_CPU(struct mmu_gather, mmu_g* freed pages that where gathered up to this point. */ static inline void -ia64_tlb_flush_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) +tlb_flush_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) {unsigned int nr;
@@ -160,7 +160,7 @@ tlb_finish_mmu (struct mmu_gather *tlb,* Note: tlb->nr may be 0 at this point, so we can't rely on tlb->start_addr and * tlb->end_addr. */ -ia64_tlb_flush_mmu (tlb, start, end); +tlb_flush_mmu (tlb, start, end);/* keep the page table cache within bounds */ check_pgt_cache ();
@@ -184,7 +184,7 @@ tlb_remove_page (struct mmu_gather *tlb, } tlb->pages[tlb->nr++] = page;if (tlb->nr >=FREE_PTE_NR ) -ia64_tlb_flush_mmu (tlb, tlb->start_addr, tlb->end_addr); +tlb_flush_mmu (tlb, tlb->start_addr, tlb->end_addr); }/*
mm/memory.c +6 -10
@@ -853,18 +853,15 @@ unsigned long unmap_vmas(struct mmu_gathbreak ; } -tlb_finish_mmu (*tlbp, tlb_start, start); -if (need_resched () || (i_mmap_lock &&need_lockbreak (i_mmap_lock))) { -if (i_mmap_lock) { - *tlbp =NULL ; +if (i_mmap_lock) goto out; - } +tlb_finish_mmu (*tlbp, tlb_start, start);cond_resched (); - } - - *tlbp =tlb_gather_mmu (vma->vm_mm, fullmm); + *tlbp =tlb_gather_mmu (vma->vm_mm, fullmm); + }else +tlb_flush_mmu (*tlbp, tlb_start, start); tlb_start_valid =0 ; zap_work =ZAP_BLOCK_SIZE ; }
@@ -892,8 +889,7 @@ unsigned long zap_page_range(struct vm_a tlb =tlb_gather_mmu (mm,0 );update_hiwater_rss (mm); end =unmap_vmas (&tlb, vma, address, end, &nr_accounted, details); -if (tlb) -tlb_finish_mmu (tlb, address, end); +tlb_finish_mmu (tlb, address, end);return end; }
Signature · 10 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/5 remove ptep_establish.
In reply to Martin SchwidefskyThe last user of ptep_establish in mm/ is long gone. Remove the architecture primitive as well.
---
include/asm-arm/pgtable.h | 6 ++--- include/asm-generic/pgtable.h | 19 ------------------ include/asm-i386/pgtable.h | 11 ---------- include/asm-ia64/pgtable.h | 6 +++-- include/asm-s390/pgtable.h | 43 ++++++++++++++++++------------------------ 5 files changed, 26 insertions(+), 59 deletions(-)
include/asm-arm/pgtable.h +3 -3
@@ -83,14 +83,14 @@* means that a write to a clean page will cause a permission fault, and * the Linux MM layer will mark the page dirty via handle_pte_fault(). * For the hardware to notice the permission change, the TLB entry must -* be flushed, and ptep_establish() does that for us. +* be flushed, and ptep_set_access_flags() does that for us. * * The "accessed" or "young" bit is emulated by a similar method; we only * allow accesses to the page if the "young" bit is set. Accesses to the * page will cause a fault, and handle_pte_fault() will set the young bit * for us as long as the page is marked present in the corresponding Linux -* PTE entry. Again, ptep_establish() will ensure that the TLB is up to -* date. +* PTE entry. Again, ptep_set_access_flags() will ensure that the TLB is +* up to date. * * However, when the "young" bit is cleared, we deny access to the page * by clearing the hardware PTE. Currently Linux does not flush the TLB
include/asm-generic/pgtable.h +0 -19
@@ -3,25 +3,6 @@#ifndef __ASSEMBLY__ -#ifndef __HAVE_ARCH_PTEP_ESTABLISH -/* -* Establish a new mapping: -* - flush the old one -* - update the page tables -* - inform the TLB about the new one -* -* We hold the mm semaphore for reading, and the pte lock. -* -* Note: the old pte is known to not be writable, so we don't need to -* worry about dirty bits etc getting lost. -*/ -#define ptep_establish (__vma, __address, __ptep, __entry) \ -do { \ - set_pte_at((__vma)->vm_mm, (__address), __ptep, __entry); \ - flush_tlb_page(__vma, __address); \ -} while (0) -#endif -#ifndef __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS/* * Largely same as above, but only sets the access flags (dirty,
include/asm-i386/pgtable.h +0 -11
@@ -317,17 +317,6 @@ static inline pte_t native_local_ptep_ge __ret; \ }) -/* -* Rules for using ptep_establish: the pte MUST be a user pte, and -* must be a present->present transition. -*/ -#define __HAVE_ARCH_PTEP_ESTABLISH -#define ptep_establish (vma, address, ptep, pteval) \ -do { \ - set_pte_present((vma)->vm_mm, address, ptep, pteval); \ - flush_tlb_page(vma, address); \ -} while (0) -#define __HAVE_ARCH_PTEP_CLEAR_DIRTY_FLUSH#define ptep_clear_flush_dirty (vma, address, ptep) \ ({ \
include/asm-ia64/pgtable.h +4 -2
@@ -546,8 +546,10 @@ extern void lazy_mmu_prot_update (pte_t# define ptep_set_access_flags (__vma, __addr, __ptep, __entry, __safely_writable) \ ({ \ int __changed = !pte_same(*(__ptep), __entry); \ - if (__changed) \ - ptep_establish(__vma, __addr, __ptep, __entry); \ + if (__changed) { \ + set_pte_at((__vma)->vm_mm, (__addr), __ptep, __entry); \ + flush_tlb_page(__vma, __addr); \ + } \ __changed; \ })#endif
include/asm-s390/pgtable.h +19 -24
63 lines not on this page · raw message
Signature · 10 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/5 remove ptep_test_and_clear_dirty and ptep_clear_flush_dirty.
In reply to Martin SchwidefskyNobody is using ptep_test_and_clear_dirty and ptep_clear_flush_dirty. Remove the functions from all architectures.
---
include/asm-frv/pgtable.h | 8 -------- include/asm-generic/pgtable.h | 25 ------------------------- include/asm-i386/pgtable.h | 21 --------------------- include/asm-ia64/pgtable.h | 17 ----------------- include/asm-m32r/pgtable.h | 6 ------ include/asm-parisc/pgtable.h | 16 ---------------- include/asm-powerpc/pgtable-ppc32.h | 7 ------- include/asm-powerpc/pgtable-ppc64.h | 31 ------------------------------- include/asm-ppc/pgtable.h | 7 ------- include/asm-s390/pgtable.h | 15 --------------- include/asm-x86_64/pgtable.h | 8 -------- include/asm-xtensa/pgtable.h | 12 ------------ 12 files changed, 173 deletions(-)
include/asm-frv/pgtable.h +0 -8
@@ -394,13 +394,6 @@ static inline pte_t pte_mkdirty(pte_t ptstatic inline pte_t pte_mkyoung (pte_t pte) { (pte).pte |= _PAGE_ACCESSED;return pte; }static inline pte_t pte_mkwrite (pte_t pte) { (pte).pte &= ~_PAGE_WP;return pte; } -static inline int ptep_test_and_clear_dirty (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) -{ -int i =test_and_clear_bit (_PAGE_BIT_DIRTY, ptep); - asmvolatile ("dcf %M0" ::"U" (*ptep)); -return i; -} -static inline int ptep_test_and_clear_young (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) {int i =test_and_clear_bit (_PAGE_BIT_ACCESSED, ptep);
@@ -510,7 +503,6 @@ static inline int pte_file(pte_t pte) remap_pfn_range(vma, vaddr, pfn, size, prot)#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY#define __HAVE_ARCH_PTEP_GET_AND_CLEAR#define __HAVE_ARCH_PTEP_SET_WRPROTECT#define __HAVE_ARCH_PTE_SAME
include/asm-generic/pgtable.h +0 -25
@@ -49,31 +49,6 @@ })#endif -#ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY -#define ptep_test_and_clear_dirty (__vma, __address, __ptep) \ -({ \ - pte_t __pte = *__ptep; \ - int r = 1; \ - if (!pte_dirty(__pte)) \ - r = 0; \ - else \ - set_pte_at((__vma)->vm_mm, (__address), (__ptep), \ - pte_mkclean(__pte)); \ - r; \ -}) -#endif - -#ifndef __HAVE_ARCH_PTEP_CLEAR_DIRTY_FLUSH -#define ptep_clear_flush_dirty (__vma, __address, __ptep) \ -({ \ - int __dirty; \ - __dirty = ptep_test_and_clear_dirty(__vma, __address, __ptep); \ - if (__dirty) \ - flush_tlb_page(__vma, __address); \ - __dirty; \ -}) -#endif -#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR#define ptep_get_and_clear (__mm, __address, __ptep) \ ({ \
include/asm-i386/pgtable.h +0 -21
@@ -295,17 +295,6 @@ static inline pte_t native_local_ptep_ge __changed; \ }) -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY -#define ptep_test_and_clear_dirty (vma, addr, ptep) ({ \ - int __ret = 0; \ - if (pte_dirty(*(ptep))) \ - __ret = test_and_clear_bit(_PAGE_BIT_DIRTY, \ - &(ptep)->pte_low); \ - if (__ret) \ - pte_update((vma)->vm_mm, addr, ptep); \ - __ret; \ -}) -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG#define ptep_test_and_clear_young (vma, addr, ptep) ({ \ int __ret = 0; \
@@ -317,16 +306,6 @@ static inline pte_t native_local_ptep_ge __ret; \ }) -#define __HAVE_ARCH_PTEP_CLEAR_DIRTY_FLUSH -#define ptep_clear_flush_dirty (vma, address, ptep) \ -({ \ - int __dirty; \ - __dirty = ptep_test_and_clear_dirty((vma), (address), (ptep)); \ - if (__dirty) \ - flush_tlb_page(vma, address); \ - __dirty; \ -}) -#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH#define ptep_clear_flush_young (vma, address, ptep) \ ({ \
include/asm-ia64/pgtable.h +0 -17
@@ -398,22 +398,6 @@ ptep_test_and_clear_young (struct vm_are#endif } -static inline int -ptep_test_and_clear_dirty (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) -{ -#ifdef CONFIG_SMP -if (!pte_dirty (*ptep)) -return 0 ; -return test_and_clear_bit (_PAGE_D_BIT, ptep); -#else -pte_t pte = *ptep; -if (!pte_dirty (pte)) -return 0 ; -set_pte_at (vma->vm_mm, addr, ptep,pte_mkclean (pte)); -return 1 ; -#endif -} -static inline pte_t ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {
@@ -593,7 +577,6 @@ extern void lazy_mmu_prot_update (pte_t#endif #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY#define __HAVE_ARCH_PTEP_GET_AND_CLEAR#define __HAVE_ARCH_PTEP_SET_WRPROTECT#define __HAVE_ARCH_PTE_SAME
include/asm-m32r/pgtable.h +0 -6
@@ -284,11 +284,6 @@ static inline pte_t pte_mkwrite(pte_t ptreturn pte; } -static inline int ptep_test_and_clear_dirty (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) -{ -return test_and_clear_bit (_PAGE_BIT_DIRTY, ptep); -} -static inline int ptep_test_and_clear_young (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) {return test_and_clear_bit (_PAGE_BIT_ACCESSED, ptep);
@@ -382,7 +377,6 @@ static inline void pmd_set(pmd_t * pmdp, remap_pfn_range(vma, vaddr, pfn, size, prot)#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY#define __HAVE_ARCH_PTEP_GET_AND_CLEAR#define __HAVE_ARCH_PTEP_SET_WRPROTECT#define __HAVE_ARCH_PTE_SAME
include/asm-parisc/pgtable.h +0 -16
@@ -451,21 +451,6 @@ static inline int ptep_test_and_clear_yo#endif } -static inline int ptep_test_and_clear_dirty (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) -{ -#ifdef CONFIG_SMP -if (!pte_dirty (*ptep)) -return 0 ; -return test_and_clear_bit (xlate_pabit (_PAGE_DIRTY_BIT), &pte_val (*ptep)); -#else -pte_t pte = *ptep; -if (!pte_dirty (pte)) -return 0 ; -set_pte_at (vma->vm_mm, addr, ptep,pte_mkclean (pte)); -return 1 ; -#endif -} -extern spinlock_t pa_dbit_lock;struct mm_struct ;
@@ -533,7 +518,6 @@ static inline void ptep_set_wrprotect(st#define HAVE_ARCH_UNMAPPED_AREA #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY#define __HAVE_ARCH_PTEP_GET_AND_CLEAR#define __HAVE_ARCH_PTEP_SET_WRPROTECT#define __HAVE_ARCH_PTE_SAME
include/asm-powerpc/pgtable-ppc32.h +0 -7
@@ -643,13 +643,6 @@ static inline int __ptep_test_and_clear_#define ptep_test_and_clear_young (__vma, __addr, __ptep) \ __ptep_test_and_clear_young((__vma)->vm_mm->context.id, __addr, __ptep) -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY -static inline int ptep_test_and_clear_dirty (struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep) -{ -return (pte_update (ptep, (_PAGE_DIRTY | _PAGE_HWWRITE),0 ) & _PAGE_DIRTY) !=0 ; -} -#define __HAVE_ARCH_PTEP_GET_AND_CLEARstatic inline pte_t ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep)
include/asm-powerpc/pgtable-ppc64.h +0 -31
@@ -307,29 +307,6 @@ static inline int __ptep_test_and_clear_ __r; \ }) -/* -* On RW/DIRTY bit transitions we can avoid flushing the hpte. For the -* moment we always flush but we need to fix hpte_update and test if the -* optimisation is worth it. -*/ -static inline int __ptep_test_and_clear_dirty (struct mm_struct *mm, -unsigned long addr,pte_t *ptep) -{ -unsigned long old; - -if ((pte_val (*ptep) & _PAGE_DIRTY) ==0 ) -return 0 ; - old =pte_update (mm, addr, ptep, _PAGE_DIRTY,0 ); -return (old & _PAGE_DIRTY) !=0 ; -} -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY -#define ptep_test_and_clear_dirty (__vma, __addr, __ptep) \ -({ \ - int __r; \ - __r = __ptep_test_and_clear_dirty((__vma)->vm_mm, __addr, __ptep); \ - __r; \ -}) -#define __HAVE_ARCH_PTEP_SET_WRPROTECTstatic inline void ptep_set_wrprotect (struct mm_struct *mm,unsigned long addr,pte_t *ptep)
@@ -357,14 +334,6 @@ static inline void ptep_set_wrprotect(st __young; \ }) -#define __HAVE_ARCH_PTEP_CLEAR_DIRTY_FLUSH -#define ptep_clear_flush_dirty (__vma, __address, __ptep) \ -({ \ - int __dirty = __ptep_test_and_clear_dirty((__vma)->vm_mm, __address, \ - __ptep); \ - __dirty; \ -}) -#define __HAVE_ARCH_PTEP_GET_AND_CLEARstatic inline pte_t ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep)
include/asm-ppc/pgtable.h +0 -7
@@ -664,13 +664,6 @@ static inline int __ptep_test_and_clear_#define ptep_test_and_clear_young (__vma, __addr, __ptep) \ __ptep_test_and_clear_young((__vma)->vm_mm->context.id, __addr, __ptep) -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY -static inline int ptep_test_and_clear_dirty (struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep) -{ -return (pte_update (ptep, (_PAGE_DIRTY | _PAGE_HWWRITE),0 ) & _PAGE_DIRTY) !=0 ; -} -#define __HAVE_ARCH_PTEP_GET_AND_CLEARstatic inline pte_t ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep)
include/asm-s390/pgtable.h +0 -15
@@ -677,19 +677,6 @@ ptep_clear_flush_young(struct vm_area_streturn ptep_test_and_clear_young (vma, address, ptep); } -static inline int ptep_test_and_clear_dirty (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) -{ -return 0 ; -} - -static inline int -ptep_clear_flush_dirty (struct vm_area_struct *vma, -unsigned long address,pte_t *ptep) -{ -/* No need to flush TLB; bits are in storage key */ -return ptep_test_and_clear_dirty (vma, address, ptep); -} -static inline pte_t ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {pte_t pte = *ptep;
@@ -947,8 +934,6 @@ extern void memmap_init(unsigned long, i#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY -#define __HAVE_ARCH_PTEP_CLEAR_DIRTY_FLUSH#define __HAVE_ARCH_PTEP_GET_AND_CLEAR#define __HAVE_ARCH_PTEP_CLEAR_FLUSH#define __HAVE_ARCH_PTEP_SET_WRPROTECT
include/asm-x86_64/pgtable.h +0 -8
@@ -290,13 +290,6 @@ static inline pte_t pte_clrhuge(pte_t ptstruct vm_area_struct ; -static inline int ptep_test_and_clear_dirty (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) -{ -if (!pte_dirty (*ptep)) -return 0 ; -return test_and_clear_bit (_PAGE_BIT_DIRTY, &ptep->pte); -} -static inline int ptep_test_and_clear_young (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) {if (!pte_young (*ptep))
@@ -433,7 +426,6 @@ extern int kern_addr_valid(unsigned long (((o) & (1UL << (__VIRTUAL_MASK_SHIFT-1))) ? ((o) | (~__VIRTUAL_MASK)) : (o))#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY#define __HAVE_ARCH_PTEP_GET_AND_CLEAR#define __HAVE_ARCH_PTEP_GET_AND_CLEAR_FULL#define __HAVE_ARCH_PTEP_SET_WRPROTECT
include/asm-xtensa/pgtable.h +0 -12
@@ -270,17 +270,6 @@ ptep_test_and_clear_young(struct vm_areareturn 1 ; } -static inline int -ptep_test_and_clear_dirty (struct vm_area_struct *vma,unsigned long addr, -pte_t *ptep) -{ -pte_t pte = *ptep; -if (!pte_dirty (pte)) -return 0 ; -update_pte (ptep,pte_mkclean (pte)); -return 1 ; -} -static inline pte_t ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {
@@ -421,7 +410,6 @@ typedef pte_t *pte_addr_t;#endif /* !defined (__ASSEMBLY__) */ #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY#define __HAVE_ARCH_PTEP_GET_AND_CLEAR#define __HAVE_ARCH_PTEP_SET_WRPROTECT#define __HAVE_ARCH_PTEP_MKDIRTY
Signature · 10 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/5 move mm_struct and vm_area_struct.
In reply to Martin SchwidefskyMove the definitions of struct mm_struct and struct vma_area_struct to include/mm_types.h. This allows to define more function in asm/pgtable.h and friends with inline assemblies instead of macros. Compile tested on i386, ia64, powerpc, powerpc64, s390-32, s390-64 and x86_64.
---
include/linux/mm.h | 63 -------------------- include/linux/mm_types.h | 143 +++++++++++++++++++++++++++++++++++++++++++++++ include/linux/sched.h | 74 ------------------------ 3 files changed, 144 insertions(+), 136 deletions(-)
include/linux/mm.h +0 -63
@@ -51,69 +51,6 @@ extern int sysctl_legacy_va_layout;* mmap() functions). */ -/* -* This struct defines a memory VMM memory area. There is one of these -* per VM-area/task. A VM area is any part of the process virtual memory -* space that has a special rule for the page-fault handlers (ie a shared -* library, the executable area etc). -*/ -struct vm_area_struct { -struct mm_struct * vm_mm;/* The address space we belong to. */ -unsigned long vm_start;/* Our start address within vm_mm. */ -unsigned long vm_end;/* The first byte after our end address -within vm_mm. */ - -/* linked list of VM areas per task, sorted by address */ -struct vm_area_struct *vm_next; - -pgprot_t vm_page_prot;/* Access permissions of this VMA. */ -unsigned long vm_flags;/* Flags, listed below. */ - -struct rb_node vm_rb; - -/* -* For areas with an address space and backing store, -* linkage into the address_space->i_mmap prio tree, or -* linkage to the list of like vmas hanging off its node, or -* linkage of vma in the address_space->i_mmap_nonlinear list. -*/ -union { -struct { -struct list_head list; -void *parent;/* aligns with prio_tree_node parent */ -struct vm_area_struct *head; - } vm_set; - -struct raw_prio_tree_node prio_tree_node; - } shared; - -/* -* A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma -* list, after a COW of one of the file pages. A MAP_SHARED vma -* can only be in the i_mmap tree. An anonymous MAP_PRIVATE, stack -* or brk vma (with NULL file) can only be in an anon_vma list. -*/ -struct list_head anon_vma_node;/* Serialized by anon_vma->lock */ -struct anon_vma *anon_vma;/* Serialized by page_table_lock */ - -/* Function pointers to deal with this struct. */ -struct vm_operations_struct * vm_ops; - -/* Information about our backing store: */ -unsigned long vm_pgoff;/* Offset (within vm_file) in PAGE_SIZE -units, *not* PAGE_CACHE_SIZE */ -struct file * vm_file;/* File we map to (can be NULL). */ -void * vm_private_data;/* was vm_pte (shared mem) */ -unsigned long vm_truncate_count;/* truncate_count or restart_addr */ - -#ifndef CONFIG_MMU -atomic_t vm_usage;/* refcount (VMAs shared if !MMU) */ -#endif -#ifdef CONFIG_NUMA -struct mempolicy *vm_policy;/* NUMA policy for the VMA */ -#endif -}; -extern struct kmem_cache *vm_area_cachep;/*
include/linux/mm_types.h +143 -0
160 lines not on this page · raw message
include/linux/sched.h +1 -73
109 lines not on this page · raw message
Signature · 10 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/5 s390 tlb flush fix.
In reply to Martin SchwidefskyThe current tlb flushing code for page table entries violates the s390 architecture in a small detail. The relevant section from the principles of operation (SA22-7832-02 page 3-47):
"A valid table entry must not be changed while it is attached to any CPU and may be used for translation by that CPU except to
(1) invalidate the entry by using INVALIDATE PAGE TABLE ENTRY or INVALIDATE DAT TABLE ENTRY, (2) alter bits 56-63 of a page-table entry, or (3) make a change by means of a COMPARE AND SWAP AND PURGE instruction that purges the TLB."
That means if one thread of a multithreaded applciation uses a vma while another thread does an unmap on it, the page table entries of that vma needs to get removed with IPTE, IDTE or CSP. In some strange and rare situations a cpu could check-stop (die) because a entry has been pushed out of the TLB that is still needed to complete a (milli-coded) instruction. I've never seen it happen with the current code on any of the supported machines, so right now this is a theoretical problem. But I want to fix it nevertheless, to avoid headaches in the futures.
To get this implemented correctly without changing common code the primitives ptep_get_and_clear, ptep_get_and_clear_full and ptep_set_wrprotect need to use the IPTE instruction to invalidate the pte before the new pte value gets stored. If IPTE is always used for the three primitives three important operations will have a performace
hit: fork, mprotect and exit_mmap. Time for some workarounds:
* 1: ptep_get_and_clear_full is used in unmap_vmas to remove page tables entries in a batched tlb gather operation. If the mmu_gather context passed to unmap_vmas has been started with full_mm_flush==1 or if only one cpu is online or if the mm_struct doesn't has more than one user the fullmm indication in the mmu_gather context is set to one. All TLBs for mm_struct are flushed by the tlb_gather_mmu call. No new TLBs can be created while the unmap is in progress. In this case ptep_get_and_clear_full clears the ptes with a simple store.
* 2: ptep_get_and_clear is used in change_protection to clear the ptes from the page tables before they are reentered with the new access flags. At the end of the update flush_tlb_range clears the remaining TLBs. In general the ptep_get_and_clear has to issue IPTE for each pte and flush_tlb_range is a nop. But if there is only one user of the mm_struct then ptep_get_and_clear uses simple stores to do the update and flush_tlb_range will flush the TLBs.
* 3: Similar to 2, ptep_set_wrprotect is used in copy_page_range for a fork to make all ptes of a cow mapping read-only. At the end of of copy_page_range dup_mmap will flush the TLBs with a call to flush_tlb_mm. Check for mm->mm_users and if there is only one user avoid using IPTE in ptep_set_wrprotect and let flush_tlb_mm clear the TLBs.
Overall for single threaded programs the tlb flush code now performs better, for multi threaded programs it is slightly worse. In particular exit_mmap() now does a single IDTE for the mm and then just frees every page cache reference and every page table page directly without a delay over the mmu_gather structure.
---
arch/s390/kernel/smp.c | 2 include/asm-s390/pgalloc.h | 17 ---- include/asm-s390/pgtable.h | 90 +++++++++++++++++++----- include/asm-s390/tlb.h | 127 ++++++++++++++++++++++++++++++++--- include/asm-s390/tlbflush.h | 159 +++++++++++++++----------------------------- 5 files changed, 245 insertions(+), 150 deletions(-)
arch/s390/kernel/smp.c +1 -1
@@ -328,7 +328,7 @@ static void smp_ext_bitcall(int cpu, ec_*/ void smp_ptlb_callback (void *info) { -local_flush_tlb (); +__tlb_flush_local (); }void smp_ptlb_all (void )
include/asm-s390/pgalloc.h +0 -17
@@ -84,7 +84,6 @@ static inline void pgd_free(pgd_t *pgd)*/ #define pmd_alloc_one (mm,address) ({ BUG(); ((pmd_t *)2); })#define pmd_free (x) do { } while (0) -#define __pmd_free_tlb (tlb,x) do { } while (0)#define pgd_populate (mm, pmd, pte) BUG()#define pgd_populate_kernel (mm, pmd, pte) BUG()#else /* __s390x__ */
@@ -120,12 +119,6 @@ static inline void pmd_free (pmd_t *pmd)free_pages ((unsigned long ) pmd,PMD_ALLOC_ORDER ); } -#define __pmd_free_tlb (tlb,pmd) \ - do { \ - tlb_flush_mmu(tlb, 0, 0); \ - pmd_free(pmd); \ - } while (0) -static inline void pgd_populate_kernel (struct mm_struct *mm,pgd_t *pgd,pmd_t *pmd) {
@@ -226,14 +219,4 @@ static inline void pte_free(struct page __free_page(pte); } -#define __pte_free_tlb (tlb, pte) \ -({ \ - struct mmu_gather *__tlb = (tlb); \ - struct page *__pte = (pte); \ - struct page *shadow_page = get_shadow_page(__pte); \ - if (shadow_page) \ - tlb_remove_page(__tlb, shadow_page); \ - tlb_remove_page(__tlb, __pte); \ -}) -#endif /* _S390_PGALLOC_H */
include/asm-s390/pgtable.h +69 -21
158 lines not on this page · raw message
include/asm-s390/tlbflush.h +57 -102
198 lines not on this page · raw message
include/asm-s390/tlb.h +118 -9
137 lines not on this page · raw message
Signature · 10 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [patch 1/5] avoid tlb gather restarts.
In reply to Martin SchwidefskyQuoted 11 lines · Martin Schwidefsky
From: Martin Schwidefsky <[email protected]>
If
need_resched()is false in the inner loop ofunmap_vmasit is unnecessary to do a full blowntlb_finish_mmu/tlb_gather_mmufor eachZAP_BLOCK_SIZEptes. Do atlb_flush_mmu()instead. That gives architectures with a non-generic tlb flush implementation room for optimization. Thetlb_flush_mmuprimitive is a available with the generic tlb flush code, theia64_tlb_flush_mmneeds to be renamed and a dummy function is added to arm and arm26.Signed-off-by Martin Schwidefsky <[email protected]>
(Looking at it, I see that we could argue that there ought to be a need_resched() etc. check after your tlb_flush_mmu() in unmap_vmas, in case it's spent a long while in there on some arches; but I don't think we have the ZAP_BLOCK_SIZE tuned with any great precision, and you'd at worst be doubling the latency there, so let's not worry about it. I write this merely in order to reserve myself an "I told you so" if anyone ever notices increased latency ;)
Quoted 96 lines
---
include/asm-arm/tlb.h | 5 +++++ include/asm-arm26/tlb.h | 5 +++++ include/asm-ia64/tlb.h | 6 +++--- mm/memory.c | 16 ++++++---------- 4 files changed, 19 insertions(+), 13 deletions(-)diff -urpN
linux-2.6/include/asm-arm/tlb.hlinux-2.6-patched/include/asm-arm/tlb.hinclude/asm-arm/tlb.h +5
-0@@ -52,6 +52,11 @@ tlb_gather_mmu(struct mm_struct *mm, uns }static inline void +tlb_flush_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) +{ +} + +static inline void tlb_finish_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) {if (tlb->fullmm)diff -urpN
linux-2.6/include/asm-arm26/tlb.hlinux-2.6-patched/include/asm-arm26/tlb.hinclude/asm-arm26/tlb.h +5
-0@@ -29,6 +29,11 @@ tlb_gather_mmu(struct mm_struct *mm, uns }static inline void +tlb_flush_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) +{ +} + +static inline void tlb_finish_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) {if (tlb->need_flush)diff -urpN
linux-2.6/include/asm-ia64/tlb.hlinux-2.6-patched/include/asm-ia64/tlb.hinclude/asm-ia64/tlb.h +3
-3@@ -72,7 +72,7 @@ DECLARE_PER_CPU(struct mmu_gather, mmu_g* freed pages that where gathered up to this point. */ static inline void -ia64_tlb_flush_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) +tlb_flush_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) {unsigned int nr;@@ -160,7 +160,7 @@ tlb_finish_mmu (struct mmu_gather *tlb,* Note: tlb->nr may be 0 at this point, so we can't rely on tlb->start_addr and * tlb->end_addr. */ -ia64_tlb_flush_mmu (tlb, start, end); +tlb_flush_mmu (tlb, start, end);/* keep the page table cache within bounds */ check_pgt_cache ();@@ -184,7 +184,7 @@ tlb_remove_page (struct mmu_gather *tlb, } tlb->pages[tlb->nr++] = page;if (tlb->nr >=FREE_PTE_NR ) -ia64_tlb_flush_mmu (tlb, tlb->start_addr, tlb->end_addr); +tlb_flush_mmu (tlb, tlb->start_addr, tlb->end_addr); }/* diff -urpN
linux-2.6/mm/memory.clinux-2.6-patched/mm/memory.cmm/memory.c +5
-8@@ -853,18 +853,15 @@ unsigned long unmap_vmas(struct mmu_gathbreak ; } -tlb_finish_mmu (*tlbp, tlb_start, start); -if (need_resched () || (i_mmap_lock &&need_lockbreak (i_mmap_lock))) { -if (i_mmap_lock) { - *tlbp =NULL ; +if (i_mmap_lock) goto out; - } +tlb_finish_mmu (*tlbp, tlb_start, start);cond_resched (); - } - - *tlbp =tlb_gather_mmu (vma->vm_mm, fullmm); + *tlbp =tlb_gather_mmu (vma->vm_mm, fullmm); + }else +tlb_flush_mmu (*tlbp, tlb_start, start); tlb_start_valid =0 ; zap_work =ZAP_BLOCK_SIZE ; }
mm/memory.c unsigned long zap_page_range(struct vm_a · line 895
Quoted 9 lines
mm/memory.c +1
-2@@ -892,8 +889,7 @@ unsigned long zap_page_range(struct vm_a tlb =tlb_gather_mmu (mm,0 );update_hiwater_rss (mm); end =unmap_vmas (&tlb, vma, address, end, &nr_accounted, details); -if (tlb) -tlb_finish_mmu (tlb, address, end); +tlb_finish_mmu (tlb, address, end);return end; }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [patch 5/5] s390 tlb flush fix.
In reply to Martin SchwidefskyQuoted 5 lines · Martin Schwidefsky
include/asm-s390/tlb.h +5
-0+ +static inline struct mmu_gather *tlb_gather_mmu (struct mm_struct *mm, +unsigned int full_mm_flush) +{ +struct mmu_gather *tlb = &get_cpu_var (mmu_gathers);
include/asm-s390/tlb.h line 60
Quoted 10 lines
include/asm-s390/tlb.h +10
-0+ + tlb->mm = mm; + tlb->fullmm = full_mm_flush || (num_online_cpus () ==1 ) || + (atomic_read (&mm->mm_users) <=1 ); + tlb->nr_ptes =0 ; + tlb->nr_pmds =TLB_NR_PTRS ; +if (tlb->fullmm) +__tlb_flush_mm (mm); +return tlb; +}
I'm afraid that mm_users test (and probably some of your other mm_users tests) is not good: because this also gets called when a file is truncated while it is mapped - the active mm at that time is likely not to be one of the mm_users. (Do any other arches use mm_users in that way? No: that should be a warning.)
You might do better to make more use of cpu_vm_mask (though I didn't see where any bits get cleared from it on s390 at present).
Though it seems sensible to aim for one TLB flush at the beginning as you're doing, that's not what other arches do (some have to worry about speculative execution, but you don't?), and it worries me that you're taking s390 further away into its own
implementation: which you're surely entitled to do, but then we're more likely to screw you over by mistake in future.
Is there perhaps another architecture whose procedures you can copy? Changing a pte while another cpu is accessing it is not a problem unique to s390.
Patches 1-4 looked fine to me, but I believe this 5/5 is the rationale behind all of them.
Hugh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [patch 5/5] s390 tlb flush fix.
Quoted 14 lines · Hugh Dickins
Quoted 13 lines · Martin Schwidefsky
include/asm-s390/tlb.h +13
-0+ +static inline struct mmu_gather *tlb_gather_mmu (struct mm_struct *mm, +unsigned int full_mm_flush) +{ +struct mmu_gather *tlb = &get_cpu_var (mmu_gathers); + + tlb->mm = mm; + tlb->fullmm = full_mm_flush || (num_online_cpus () ==1 ) || + (atomic_read (&mm->mm_users) <=1 ); + tlb->nr_ptes =0 ; + tlb->nr_pmds =TLB_NR_PTRS ; +if (tlb->fullmm) +__tlb_flush_mm (mm);
Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ return tlb; +}I'm afraid that
mm_userstest (and probably some of your othermm_userstests) is not good: because this also gets called when a file is truncated while it is mapped - the active mm at that time is likely not to be one of themm_users. (Do any other arches usemm_usersin that way? No: that should be a warning.)
Good catch, that would have caused me some headache. So I need to add a current->active_mm==mm check if mm_users==1.
Quoted 2 lines
You might do better to make more use of
cpu_vm_mask(though I didn't see where any bits get cleared from it on s390 at present).
We don't clear any of the bits in cpu_vm_mask. I though about it for a while and got tangled in race conditions. The cpu_vm_mask optimization works for short-lived processes which always executed on the same cpu.
Quoted 6 lines
Though it seems sensible to aim for one TLB flush at the beginning as you're doing, that's not what other arches do (some have to worry about speculative execution, but you don't?), and it worries me that you're taking s390 further away into its own
implementation: which you're surely entitled to do, but then we're more likely to screw you over by mistake in future.
We do not have to worry about speculative execution because s390 uses special instruction to do user access (mvcs, mvcp and mvcos) and the kernel has its own address space. The compiler doesn't know about these instruction and cannot "accidentally" access a user space address over the user page table when it shouldn't.
Quoted 3 lines
Is there perhaps another architecture whose procedures you can copy? Changing a pte while another cpu is accessing it is not a problem unique to s390.
No, I don't think so. s390 is quite unique with the restriction that you may not do a set_pte_at .. flush_tlb_xxx while a pte might get accessed by a different cpu.
Quoted 2 lines
Patches 1-4 looked fine to me, but I believe this 5/5 is the rationale behind all of them.
Yes, indeed the tlb flush fix and the 1K/2K page tables are my reasons for all these patches.
Signature · 11 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [patch 1/5] avoid tlb gather restarts.
In reply to Hugh DickinsQuoted 12 lines · Hugh Dickins
Quoted 9 lines
If
need_resched()is false in the inner loop ofunmap_vmasit is unnecessary to do a full blowntlb_finish_mmu/tlb_gather_mmufor eachZAP_BLOCK_SIZEptes. Do atlb_flush_mmu()instead. That gives architectures with a non-generic tlb flush implementation room for optimization. Thetlb_flush_mmuprimitive is a available with the generic tlb flush code, theia64_tlb_flush_mmneeds to be renamed and a dummy function is added to arm and arm26.Signed-off-by Martin Schwidefsky <[email protected]>Acked-by Hugh Dickins <[email protected]>
Quoted 7 lines
(Looking at it, I see that we could argue that there ought to be a
need_resched()etc. check after yourtlb_flush_mmu()inunmap_vmas, in case it's spent a long while in there on some arches; but I don't think we have theZAP_BLOCK_SIZEtuned with any great precision, and you'd at worst be doubling the latency there, so let's not worry about it. I write this merely in order to reserve myself an "I told you so" if anyone ever notices increased latency ;)
Hmm, we'd have to repeat the longish if statement to make sure we don't miss a cond_resched after tlb_flush_mmu. I'd rather not do that.
Signature · 11 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerRe: [patch 1/5] avoid tlb gather restarts.
In reply to Martin SchwidefskyQuoted 10 lines · On Tue, 03 Jul 2007 13:18:23 +0200 Martin Schwidefsky <[email protected]>…
From: Martin Schwidefsky <[email protected]>
If
need_resched()is false in the inner loop ofunmap_vmasit is unnecessary to do a full blowntlb_finish_mmu/tlb_gather_mmufor eachZAP_BLOCK_SIZEptes. Do atlb_flush_mmu()instead. That gives architectures with a non-generic tlb flush implementation room for optimization. Thetlb_flush_mmuprimitive is a available with the generic tlb flush code, theia64_tlb_flush_mmneeds to be renamed and a dummy function is added to arm and arm26.
Quoted 8 lines
Signed-off-by Martin Schwidefsky <[email protected]>---
include/asm-arm/tlb.h | 5 +++++ include/asm-arm26/tlb.h | 5 +++++ include/asm-ia64/tlb.h | 6 +++--- mm/memory.c | 16 ++++++---------- 4 files changed, 19 insertions(+), 13 deletions(-)
sparc64 broke:
mm/memory.c: In function `unmap_vmas':mm/memory.c:862: error: too many arguments to function `tlb_flush_mmu'
grep, please.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>