mm: page fault scalability prep
Subsystems (16)
cover mm: page fault scalability prep
Here comes the preparatory batch for my page fault scalability patches. This batch makes a few fixes - I suggest 01 and 02 should go in 2.6.14 - and a lot of tidyups, clearing some undergrowth for the real patches.
Just occasionally there's a hint of where we shall be heading: as in the prototype posted to linux-mm a month ago, narrowing the scope of the mm page_table_lock, so we can descend pgd,pud,pmd without it; and then, on machines with more cpus, using a lock per page-table page for the ptes. Thanks to Christoph Lameter for his generous endorsement of that approach.
This first batch is one half to one third of the work. The next batch should follow in a few days' time. The long delay, mainly an odyssey through the architectures to satisfy myself of safety, should be over - though I still need to clarify some issues with maintainers, perhaps by posting a few dubious arch patches.
This batch is against 2.6.14-rc2-mm1 plus Hirofumi's msync patch; or against 2.6.14-rc2-git5 plus that and Nick's move_pte patch.
21/21 unfairly weights the deletions in the diffstat, without it we're
36 files changed, 404 insertions(+), 702 deletions(-)
Hugh
Documentation/kernel-parameters.txt | 2 Documentation/m68k/kernel-options.txt | 24 arch/ia64/kernel/perfmon.c | 3 arch/ia64/mm/fault.c | 2 arch/m68k/Kconfig | 24 arch/m68k/atari/stram.c | 918 ---------------------------------- arch/mips/kernel/irixelf.c | 1 arch/sh/mm/hugetlbpage.c | 2 arch/sh64/mm/hugetlbpage.c | 188 ------ arch/sparc64/kernel/binfmt_aout32.c | 1 arch/sparc64/mm/tlb.c | 4 arch/x86_64/ia32/ia32_aout.c | 1 fs/binfmt_aout.c | 1 fs/binfmt_elf.c | 1 fs/binfmt_elf_fdpic.c | 7 fs/binfmt_flat.c | 1 fs/binfmt_som.c | 1 fs/exec.c | 2 fs/proc/array.c | 2 fs/proc/task_mmu.c | 8 include/asm-arm/tlb.h | 23 include/asm-arm26/tlb.h | 47 - include/asm-generic/tlb.h | 23 include/asm-ia64/tlb.h | 19 include/asm-ppc64/pgtable.h | 4 include/asm-sparc64/tlb.h | 29 - include/linux/mm.h | 17 include/linux/sched.h | 4 kernel/acct.c | 2 kernel/fork.c | 29 - mm/fremap.c | 4 mm/hugetlb.c | 37 - mm/memory.c | 325 ++++++------ mm/mmap.c | 87 +-- mm/mprotect.c | 4 mm/mremap.c | 170 ++---- mm/msync.c | 38 - mm/nommu.c | 2 mm/rmap.c | 8 mm/swapfile.c | 9 40 files changed, 421 insertions(+), 1653 deletions(-)
PATCH 1/21 mm: hugetlb truncation fixes
hugetlbfs allows truncation of its files (should it?), but hugetlb.c often forgets that: crashes and misaccounting ensue.
copy_hugetlb_page_range better grab the src page_table_lock since we don't want to guess what happens if concurrently truncated. unmap_hugepage_range rss accounting must not assume the full range was mapped. follow_hugetlb_page must guard with page_table_lock and be prepared to exit early.
Restyle copy_hugetlb_page_range with a for loop like the others there.
---
mm/hugetlb.c | 35 +++++++++++++++++++++-------------- 1 files changed, 21 insertions(+), 14 deletions(-)
mm/hugetlb.c +21 -14
@@ -273,21 +273,22 @@ int copy_hugetlb_page_range(struct mm_st {pte_t *src_pte, *dst_pte, entry;struct page *ptepage; -unsigned long addr = vma->vm_start; -unsigned long end = vma->vm_end; +unsigned long addr; -while (addr < end) { +for (addr = vma->vm_start; addr < vma->vm_end; addr +=HPAGE_SIZE ) { dst_pte =huge_pte_alloc (dst, addr);if (!dst_pte) goto nomem; +spin_lock (&src->page_table_lock); src_pte =huge_pte_offset (src, addr); -BUG_ON (!src_pte ||pte_none (*src_pte));/* prefaulted */ - entry = *src_pte; - ptepage =pte_page (entry); -get_page (ptepage); -add_mm_counter (dst, rss,HPAGE_SIZE /PAGE_SIZE ); -set_huge_pte_at (dst, addr, dst_pte, entry); - addr +=HPAGE_SIZE ; +if (src_pte && !pte_none (*src_pte)) { + entry = *src_pte; + ptepage =pte_page (entry); +get_page (ptepage); +add_mm_counter (dst, rss,HPAGE_SIZE /PAGE_SIZE ); +set_huge_pte_at (dst, addr, dst_pte, entry); + } +spin_unlock (&src->page_table_lock); }return 0 ;
@@ -322,8 +323,8 @@ void unmap_hugepage_range(struct vm_area page =pte_page (pte);put_page (page); +add_mm_counter (mm, rss, - (HPAGE_SIZE /PAGE_SIZE )); } -add_mm_counter (mm, rss, -((end - start) >>PAGE_SHIFT ));flush_tlb_range (vma, start, end); }
@@ -402,6 +403,7 @@ int follow_hugetlb_page(struct mm_structBUG_ON (!is_vm_hugetlb_page (vma)); vpfn = vaddr/PAGE_SIZE ; +spin_lock (&mm->page_table_lock);while (vaddr < vma->vm_end && remainder) {if (pages) {
@@ -414,8 +416,13 @@ int follow_hugetlb_page(struct mm_struct* indexing below to work. */ pte =huge_pte_offset (mm, vaddr &HPAGE_MASK ); -/* hugetlb should be locked, and hence, prefaulted */ -WARN_ON (!pte ||pte_none (*pte)); +/* the hugetlb file might have been truncated */ +if (!pte ||pte_none (*pte)) { + remainder =0 ; +if (!i) + i = -EFAULT ; +break ; + } page = &pte_page (*pte)[vpfn % (HPAGE_SIZE /PAGE_SIZE )];
@@ -433,7 +440,7 @@ int follow_hugetlb_page(struct mm_struct --remainder; ++i; } - +spin_unlock (&mm->page_table_lock); *length = remainder; *position = vaddr;
PATCH 2/21 mm: copy_pte_range progress fix
In reply to Hugh Dickins[PATCH 02/21] mm: copy_pte_range progress fix
My latency breaking in copy_pte_range didn't work as intended: instead of checking at regularish intervals, after the first interval it checked every time around the loop, too impatient to be preempted. Fix that.
---
mm/memory.c | 14 ++++++++------ 1 files changed, 8 insertions(+), 6 deletions(-)
mm/memory.c +8 -6
@@ -410,7 +410,7 @@ static int copy_pte_range(struct mm_stru {pte_t *src_pte, *dst_pte;unsigned long vm_flags = vma->vm_flags; -int progress; +int progress =0 ; again: dst_pte =pte_alloc_map (dst_mm, dst_pmd, addr);
@@ -418,17 +418,19 @@ again:return -ENOMEM ; src_pte =pte_offset_map_nested (src_pmd, addr); - progress =0 ;spin_lock (&src_mm->page_table_lock); do { /* * We are holding two locks at this point - either of them * could generate latencies in another task on another CPU. */ - if (progress >= 32 && (need_resched() || - need_lockbreak(&src_mm->page_table_lock) || - need_lockbreak(&dst_mm->page_table_lock))) - break; + if (progress >= 32) { + progress = 0; + if (need_resched() || + need_lockbreak(&src_mm->page_table_lock) || + need_lockbreak(&dst_mm->page_table_lock)) + break; + } if (pte_none(*src_pte)) { progress++; continue;
PATCH 3/21 mm: msync_pte_range progress
In reply to Hugh DickinsUse latency breaking in msync_pte_range like that in copy_pte_range, instead of the ugly CONFIG_PREEMPT filemap_msync alternatives.
---
mm/msync.c | 38 ++++++++++++++------------------------ 1 files changed, 14 insertions(+), 24 deletions(-)
mm/msync.c +14 -24
@@ -26,12 +26,21 @@ static void msync_pte_range(struct vm_ar unsigned long addr, unsigned long end) {pte_t *pte; +int progress =0 ; +again: pte =pte_offset_map (pmd, addr); do { unsigned long pfn; struct page *page; + if (progress >= 64) { + progress = 0; + if (need_resched() || + need_lockbreak(&vma->vm_mm->page_table_lock)) + break; + } + progress++; if (!pte_present(*pte)) continue; if (!pte_maybe_dirty(*pte))
@@ -46,8 +55,12 @@ static void msync_pte_range(struct vm_arif (ptep_clear_flush_dirty (vma, addr, pte) ||page_test_and_clear_dirty (page))set_page_dirty (page); + progress +=3 ; }while (pte++, addr +=PAGE_SIZE , addr != end);pte_unmap (pte -1 ); +cond_resched_lock (&vma->vm_mm->page_table_lock); +if (addr != end) + goto again; } static inline void msync_pmd_range(struct vm_area_struct *vma, pud_t *pud,
@@ -106,29 +119,6 @@ static void msync_page_range(struct vm_aspin_unlock (&mm->page_table_lock); } -#ifdef CONFIG_PREEMPT -static inline void filemap_msync (struct vm_area_struct *vma, -unsigned long addr,unsigned long end) -{ -const size_t chunk =64 *1024 ;/* bytes */ -unsigned long next; - -do { - next = addr + chunk; -if (next > end || next < addr) - next = end; -msync_page_range (vma, addr, next); -cond_resched (); - }while (addr = next, addr != end); -} -#else -static inline void filemap_msync (struct vm_area_struct *vma, -unsigned long addr,unsigned long end) -{ -msync_page_range (vma, addr, end); -} -#endif -/* * MS_SYNC syncs the entire file - including mappings. *
@@ -150,7 +140,7 @@ static int msync_interval(struct vm_areareturn -EBUSY ;if (file && (vma->vm_flags &VM_SHARED )) { -filemap_msync (vma, addr, end); +msync_page_range (vma, addr, end);if (flags &MS_SYNC ) {struct address_space *mapping = file->f_mapping;
PATCH 4/21 mm: zap_pte_range dont dirty anon
In reply to Hugh Dickinszap_pte_range already avoids wasting time to mark_page_accessed on anon
pages: it can also skip anon set_page_dirty - the page only needs to be marked dirty if shared with another mm, but that will say pte_dirty too.
---
mm/memory.c | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-)
mm/memory.c +6 -4
@@ -574,12 +574,14 @@ static void zap_pte_range(struct mmu_gat addr) != page->index)set_pte_at (tlb->mm, addr, pte,pgoff_to_pte (page->index)); -if (pte_dirty (ptent)) -set_page_dirty (page);if (PageAnon (page))dec_mm_counter (tlb->mm, anon_rss); -else if (pte_young (ptent)) -mark_page_accessed (page); +else { +if (pte_dirty (ptent)) +set_page_dirty (page); +if (pte_young (ptent)) +mark_page_accessed (page); + } tlb->freed++;page_remove_rmap (page);tlb_remove_page (tlb, page);
Re: [PATCH 04/21] mm: zap_pte_range dont dirty anon
Quoted 5 lines · Hugh Dickins
zap_pte_rangealready avoids wasting time tomark_page_accessedon anon
pages: it can also skip anonset_page_dirty- the page only needs to be marked dirty if shared with another mm, but that will saypte_dirtytoo.
Are you sure about this?
Quoted 11 lines
mm/memory.c +0
-3@@ -574,12 +574,14 @@ static void zap_pte_range(struct mmu_gat addr) != page->index)set_pte_at (tlb->mm, addr, pte,pgoff_to_pte (page->index)); -if (pte_dirty (ptent)) -set_page_dirty (page);if (PageAnon (page))dec_mm_counter (tlb->mm, anon_rss); -else if (pte_young (ptent))
mm/memory.c static void zap_pte_range(struct mmu_gat · line 587
Quoted 10 lines
mm/memory.c +6
-1-mark_page_accessed (page); +else { +if (pte_dirty (ptent)) +set_page_dirty (page); +if (pte_young (ptent)) +mark_page_accessed (page); + } tlb->freed++;page_remove_rmap (page);tlb_remove_page (tlb, page);
What is the page is (for example) clean swapcache, having been recently faulted in. If this pte indicates that this process has modified the page and we don't run set_page_dirty(), the page could be reclaimed and the change is lost.
Or what is the page was an anon page resulting from (say) a swapoff, and it's shared by two mm's and one has modified it and we drop that dirty pte?
Or <other scenarios>.
Need more convincing.
Re: [PATCH 04/21] mm: zap_pte_range dont dirty anon
Quoted 7 lines · Andrew Morton
Quoted 4 lines · Hugh Dickins
zap_pte_rangealready avoids wasting time tomark_page_accessedon anon
pages: it can also skip anonset_page_dirty- the page only needs to be marked dirty if shared with another mm, but that will saypte_dirtytoo.Are you sure about this?
Yeerrrssss, well, I'm never _sure_ about anything, especially when faced by the question.
Quoted 4 lines
What is the page is (for example) clean swapcache, having been recently faulted in. If this pte indicates that this process has modified the page and we don't run
set_page_dirty(), the page could be reclaimed and the change is lost.
Absolutely. But either the page is unique to this mm, shared only with
swapcache: in which case we're about to do a free_swap_cache on it (that may be delayed in actually freeing the swap because of not getting page lock, presumably because vmscan just got to it, but no matter), and we don't care at all that the page no longer represents what's on swap disk.
Or, the page is shared with another mm. But it's an anonymous page (a private page), so it's shared via fork, and COW applies to it. copy_one_pte did ptep_set_wrprotect on it, and did not pte_mkclean [*].
So if it's dirty from before the fork, the sharing mm will also have it marked pte_dirty, which will get propagated through to the page via that mm, and everything's fine even though we're ignoring the pte_dirty in this unmapping mm. Or if it's dirty from after the fork, well, something's gone very wrong if the page is still shared - unless it's because there's been a further fork since it was dirtied, in which case that sibling carries the pte_dirty which guarantees its integrity.
The change would be very wrong for the !PageAnon pages:
but it's right for the PageAnon ones, don't you agree?
Quoted 2 lines
Or what is the page was an anon page resulting from (say) a swapoff, and it's shared by two mm's and one has modified it and we drop that dirty pte?
Again, the one which tried to modify it will have got a Copy-On-Write fault, been given a copy of the page, and the original won't be shared.
Quoted 3 lines
Or <other scenarios>.
Need more convincing.
Convinced? I hope it's just that you forgot something and now remember. But do demand more from me if not (and don't feel obliged to devise more cases to ask about, though they do help - thanks). Or just drop the patch, it was merely a passing observation - I'm not building any great edifice upon it in later patches!
Hugh
[*] This ignores the issue of the "Linus" pages, those anonymous pages which have got into a shared vma by ptrace writing while the vma was unwritable - copy_one_pte's tests are on VM_SHARED. They're in a limbo between private and shared, and may indeed behave oddly. For the moment we'll continue to ignore the oddities of that case.
Re: [PATCH 04/21] mm: zap_pte_range dont dirty anon
Quoted 6 lines · Hugh Dickins
Quoted 5 lines
What is the page is (for example) clean swapcache, having been recently faulted in. If this pte indicates that this process has modified the page and we don't run
set_page_dirty(), the page could be reclaimed and the change is lost.
Quoted 8 lines
Absolutely. But either the page is unique to this mm, shared only with
swapcache: in which case we're about to do afree_swap_cacheon it (that may be delayed in actually freeing the swap because of not getting page lock, presumably because vmscan just got to it, but no matter), and we don't care at all that the page no longer represents what's on swap disk.Or, the page is shared with another mm. But it's an anonymous page (a private page), so it's shared via fork, and COW applies to it.
mmap(MAP_ANONYMOUS|MAP_SHARED) fork() swapout swapin swapoff
Now we have two mm's sharing a clean, non-cowable, non-swapcache anonymous page, no?
Re: [PATCH 04/21] mm: zap_pte_range dont dirty anon
Quoted 9 lines · Andrew Morton
mmap(MAP_ANONYMOUS|MAP_SHARED) fork() swapout swapin swapoffNow we have two mm's sharing a clean, non-cowable, non-swapcache anonymous page, no?
No, MAP_ANONYMOUS|MAP_SHARED gives you a tmpfs object via shmem_zero_setup:
all those pages are shared file pages, not PageAnon at all.
Hugh
PATCH 5/21 mm: anon is already wrprotected
In reply to Hugh Dickinsdo_anonymous_page's pte_wrprotect causes some confusion: in such a case, vm_page_prot must already be forcing COW, so must omit write permission, and so the pte_wrprotect is redundant. Replace it by a comment to that effect, and reword the comment on unuse_pte which also caused confusion.
---
mm/memory.c | 7 ++++--- mm/swapfile.c | 7 +++---- 2 files changed, 7 insertions(+), 7 deletions(-)
mm/memory.c +4 -3
@@ -1768,13 +1768,14 @@ do_anonymous_page(struct mm_struct *mm, unsigned long addr) {pte_t entry; -struct page * page =ZERO_PAGE (addr); -/* Read-only mapping of ZERO_PAGE. */ - entry =pte_wrprotect (mk_pte (ZERO_PAGE (addr), vma->vm_page_prot)); +/* Mapping of ZERO_PAGE - vm_page_prot is readonly */ + entry =mk_pte (ZERO_PAGE (addr), vma->vm_page_prot);/* ..except if it's a write access */ if (write_access) { +struct page *page; +/* Allocate our own private page. */ pte_unmap (page_table);spin_unlock (&mm->page_table_lock);
mm/swapfile.c +3 -4
@@ -396,10 +396,9 @@ void free_swap_and_cache(swp_entry_t ent }/* -* Always set the resulting pte to be nowrite (the same as COW pages -* after one process has exited). We don't know just how many PTEs will -* share this swap entry, so be cautious and let do_wp_page work out -* what to do if a write is requested later. +* No need to decide whether this PTE shares the swap entry with others, +* just let do_wp_page work it out if a write is requested later - to +* force COW, vm_page_prot omits write permission from any private vma. * * vma->vm_mm->page_table_lock is held. */
PATCH 6/21 mm: vm_stat_account unshackled
In reply to Hugh DickinsThe original vm_stat_account has fallen into disuse, with only one user, and only one user of vm_stat_unaccount. It's easier to keep track if we convert them all to __vm_stat_account, then free it from its __shackles.
---
arch/ia64/kernel/perfmon.c | 3 ++- arch/ia64/mm/fault.c | 2 +- include/linux/mm.h | 16 ++-------------- kernel/fork.c | 2 +- mm/mmap.c | 20 ++++++++++---------- mm/mprotect.c | 4 ++-- mm/mremap.c | 4 ++-- 7 files changed, 20 insertions(+), 31 deletions(-)
arch/ia64/kernel/perfmon.c +2 -1
@@ -2352,7 +2352,8 @@ pfm_smpl_buffer_alloc(struct task_structinsert_vm_struct (mm, vma); mm->total_vm += size >>PAGE_SHIFT ; -vm_stat_account (vma); +vm_stat_account (vma->vm_mm, vma->vm_flags, vma->vm_file, +vma_pages (vma));up_write (&task->mm->mmap_sem);/*
arch/ia64/mm/fault.c +1 -1
@@ -41,7 +41,7 @@ expand_backing_store (struct vm_area_str vma->vm_mm->total_vm += grow;if (vma->vm_flags &VM_LOCKED ) vma->vm_mm->locked_vm += grow; -__vm_stat_account (vma->vm_mm, vma->vm_flags, vma->vm_file, grow); +vm_stat_account (vma->vm_mm, vma->vm_flags, vma->vm_file, grow);return 0 ; }
include/linux/mm.h +2 -14
@@ -936,26 +936,14 @@ int remap_pfn_range(struct vm_area_struc unsigned long, unsigned long, pgprot_t);#ifdef CONFIG_PROC_FS -void __vm_stat_account (struct mm_struct *,unsigned long ,struct file *,long ); +void vm_stat_account (struct mm_struct *,unsigned long ,struct file *,long );#else -static inline void __vm_stat_account (struct mm_struct *mm, +static inline void vm_stat_account (struct mm_struct *mm,unsigned long flags,struct file *file,long pages) { }#endif /* CONFIG_PROC_FS */ -static inline void vm_stat_account (struct vm_area_struct *vma) -{ -__vm_stat_account (vma->vm_mm, vma->vm_flags, vma->vm_file, -vma_pages (vma)); -} - -static inline void vm_stat_unaccount (struct vm_area_struct *vma) -{ -__vm_stat_account (vma->vm_mm, vma->vm_flags, vma->vm_file, - -vma_pages (vma)); -} -/* update per process rss and vm hiwater data */ extern void update_mem_hiwater (struct task_struct *tsk);
kernel/fork.c +1 -1
@@ -212,7 +212,7 @@ static inline int dup_mmap(struct mm_strif (mpnt->vm_flags &VM_DONTCOPY ) {long pages =vma_pages (mpnt); mm->total_vm -= pages; -__vm_stat_account (mm, mpnt->vm_flags, mpnt->vm_file, +vm_stat_account (mm, mpnt->vm_flags, mpnt->vm_file, -pages);continue ; }
mm/mmap.c +10 -10
@@ -828,7 +828,7 @@ none: }#ifdef CONFIG_PROC_FS -void __vm_stat_account (struct mm_struct *mm,unsigned long flags, +void vm_stat_account (struct mm_struct *mm,unsigned long flags,struct file *file,long pages) {const unsigned long stack_flags
@@ -1106,7 +1106,7 @@ munmap_back: } out: mm->total_vm += len >>PAGE_SHIFT ; -__vm_stat_account (mm, vm_flags, file, len >>PAGE_SHIFT ); +vm_stat_account (mm, vm_flags, file, len >>PAGE_SHIFT );if (vm_flags &VM_LOCKED ) { mm->locked_vm += len >>PAGE_SHIFT ;make_pages_present (addr, addr + len);
@@ -1471,7 +1471,7 @@ static int acct_stack_growth(struct vm_a mm->total_vm += grow;if (vma->vm_flags &VM_LOCKED ) mm->locked_vm += grow; -__vm_stat_account (mm, vma->vm_flags, vma->vm_file, grow); +vm_stat_account (mm, vma->vm_flags, vma->vm_file, grow);return 0 ; }
@@ -1606,15 +1606,15 @@ find_extend_vma(struct mm_struct * mm, u* By the time this function is called, the area struct has been * removed from the process mapping list. */ -static void unmap_vma (struct mm_struct *mm,struct vm_area_struct *area) +static void unmap_vma (struct mm_struct *mm,struct vm_area_struct *vma) { -size_t len = area->vm_end - area->vm_start; +long nrpages =vma_pages (vma); - area->vm_mm->total_vm -= len >>PAGE_SHIFT ; -if (area->vm_flags &VM_LOCKED ) - area->vm_mm->locked_vm -= len >>PAGE_SHIFT ; -vm_stat_unaccount (area); -remove_vm_struct (area); + mm->total_vm -= nrpages; +if (vma->vm_flags &VM_LOCKED ) + mm->locked_vm -= nrpages; +vm_stat_account (mm, vma->vm_flags, vma->vm_file, -nrpages); +remove_vm_struct (vma); }/*
mm/mprotect.c +2 -2
@@ -168,8 +168,8 @@ success: vma->vm_flags = newflags; vma->vm_page_prot = newprot;change_protection (vma, start, end, newprot); -__vm_stat_account (mm, oldflags, vma->vm_file, -nrpages); -__vm_stat_account (mm, newflags, vma->vm_file, nrpages); +vm_stat_account (mm, oldflags, vma->vm_file, -nrpages); +vm_stat_account (mm, newflags, vma->vm_file, nrpages);return 0 ; fail:
mm/mremap.c +2 -2
@@ -233,7 +233,7 @@ static unsigned long move_vma(struct vm_* since do_munmap() will decrement it by old_len == new_len */ mm->total_vm += new_len >>PAGE_SHIFT ; -__vm_stat_account (mm, vma->vm_flags, vma->vm_file, new_len>>PAGE_SHIFT ); +vm_stat_account (mm, vma->vm_flags, vma->vm_file, new_len>>PAGE_SHIFT );if (do_munmap (mm, old_addr, old_len) <0 ) {/* OOM: unable to split vma, just get accounts right */
@@ -384,7 +384,7 @@ unsigned long do_mremap(unsigned long ad addr + new_len, vma->vm_pgoff, NULL); current->mm->total_vm += pages; -__vm_stat_account (vma->vm_mm, vma->vm_flags, +vm_stat_account (vma->vm_mm, vma->vm_flags, vma->vm_file, pages);if (vma->vm_flags &VM_LOCKED ) { current->mm->locked_vm += pages;
PATCH 7/21 mm: remove_vma_list consolidation
In reply to Hugh Dickinsunmap_vma doesn't amount to much, let's put it inside unmap_vma_list. Except it doesn't unmap anything, unmap_region just did the unmapping:
rename it to remove_vma_list.
---
mm/mmap.c | 36 ++++++++++++------------------------ 1 files changed, 12 insertions(+), 24 deletions(-)
mm/mmap.c +12 -24
@@ -1599,35 +1599,23 @@ find_extend_vma(struct mm_struct * mm, u }#endif -/* Normal function to fix up a mapping -* This function is the default for when an area has no specific -* function. This may be used as part of a more specific routine. -* -* By the time this function is called, the area struct has been -* removed from the process mapping list. -*/ -static void unmap_vma (struct mm_struct *mm,struct vm_area_struct *vma) -{ -long nrpages =vma_pages (vma); - - mm->total_vm -= nrpages; -if (vma->vm_flags &VM_LOCKED ) - mm->locked_vm -= nrpages; -vm_stat_account (mm, vma->vm_flags, vma->vm_file, -nrpages); -remove_vm_struct (vma); -} -/* -* Update the VMA and inode share lists. -* -* Ok - we have the memory areas we should free on the 'free' list, +* Ok - we have the memory areas we should free on the vma list, * so release them, and do the vma updates. +* +* Called with the mm semaphore held. */ -static void unmap_vma_list (struct mm_struct *mm,struct vm_area_struct *vma) +static void remove_vma_list (struct mm_struct *mm,struct vm_area_struct *vma) {do {struct vm_area_struct *next = vma->vm_next; -unmap_vma (mm, vma); +long nrpages =vma_pages (vma); + + mm->total_vm -= nrpages; +if (vma->vm_flags &VM_LOCKED ) + mm->locked_vm -= nrpages; +vm_stat_account (mm, vma->vm_flags, vma->vm_file, -nrpages); +remove_vm_struct (vma); vma = next; }while (vma);validate_mm (mm);
@@ -1795,7 +1783,7 @@ int do_munmap(struct mm_struct *mm, unsiunmap_region (mm, vma, prev, start, end);/* Fix up all other VM information */ -unmap_vma_list (mm, vma); +remove_vma_list (mm, vma);return 0 ; }
PATCH 8/21 mm: unlink_file_vma, remove_vma
In reply to Hugh DickinsDivide remove_vm_struct into two parts: first anon_vma_unlink plus unlink_file_vma, to unlink the vma from the list and tree by which rmap or vmtruncate might find it; then remove_vma to close, fput and free.
The intention here is to do the anon_vma_unlink and unlink_file_vma earlier, in free_pgtables before freeing any page tables: so we can be sure that any page tables traversed by rmap and vmtruncate are stable (and other, ordinary cases are stabilized by holding mmap_sem).
This will be crucial to traversing pgd,pud,pmd without page_table_lock. But testing the split-out patch showed that lifting the page_table_lock is symbiotically necessary to make this change - the lock ordering is wrong to move those unlinks into free_pgtables while it's under ptlock.
---
include/linux/mm.h | 1 + mm/mmap.c | 41 +++++++++++++++++++++++++++-------------- 2 files changed, 28 insertions(+), 14 deletions(-)
include/linux/mm.h +1 -0
@@ -840,6 +840,7 @@ extern int split_vma(struct mm_struct *,extern int insert_vm_struct (struct mm_struct *,struct vm_area_struct *);extern void __vma_link_rb (struct mm_struct *,struct vm_area_struct *,struct rb_node **,struct rb_node *); +extern void unlink_file_vma (struct vm_area_struct *);extern struct vm_area_struct *copy_vma (struct vm_area_struct **,unsigned long addr,unsigned long len,pgoff_t pgoff);extern void exit_mmap (struct mm_struct *);
mm/mmap.c +27 -14
79 lines not on this page · raw message
PATCH 9/21 mm: exit_mmap need not reset
In reply to Hugh Dickinsexit_mmap resets various mm_struct fields, but the mm is well on its way out, and none of those fields matter by this point.
---
mm/mmap.c | 6 ------ 1 files changed, 6 deletions(-)
mm/mmap.c +0 -6
@@ -1944,12 +1944,6 @@ void exit_mmap(struct mm_struct *mm)free_pgtables (&tlb, vma,FIRST_USER_ADDRESS ,0 );tlb_finish_mmu (tlb,0 , end); - mm->mmap = mm->mmap_cache =NULL ; - mm->mm_rb =RB_ROOT ; -set_mm_counter (mm, rss,0 ); - mm->total_vm =0 ; - mm->locked_vm =0 ; -spin_unlock (&mm->page_table_lock);/*
PATCH 10/21 mm: page fault handlers tidyup
In reply to Hugh DickinsImpose a little more consistency on the page fault handlers do_wp_page, do_swap_page, do_anonymous_page, do_no_page, do_file_page: why not pass their arguments in the same order, called the same names?
break_cow is all very well, but what it did was inlined elsewhere:
easier to compare if it's brought back into do_wp_page.
do_file_page's fallback to do_no_page dates from a time when we were testing pte_file by using it wherever possible: currently it's peculiar to nonlinear vmas, so just check that. BUG_ON if not? Better not, it's probably page table corruption, so just show the pte: hmm, there's a pte_ERROR macro, let's use that for do_wp_page's invalid pfn too.
Hah! Someone in the ppc64 world noticed pte_ERROR was unused so removed
it: restored (and say "pud" not "pmd" in its pud_ERROR).
---
include/asm-ppc64/pgtable.h | 4 mm/memory.c | 220 +++++++++++++++++++------------------------- 2 files changed, 100 insertions(+), 124 deletions(-)
include/asm-ppc64/pgtable.h +3 -1
@@ -478,10 +478,12 @@ extern pgprot_t phys_mem_access_prot(str#define __HAVE_ARCH_PTE_SAME#define pte_same (A ,B ) (((pte_val(A) ^ pte_val(B)) & ~_PAGE_HPTEFLAGS) == 0) +#define pte_ERROR (e) \ + printk("%s:%d: bad pte %08lx.\n", __FILE__, __LINE__, pte_val(e))#define pmd_ERROR (e) \ printk("%s:%d: bad pmd %08lx.\n", __FILE__, __LINE__, pmd_val(e))#define pud_ERROR (e) \ - printk("%s:%d: bad pmd %08lx.\n", __FILE__, __LINE__, pud_val(e)) + printk("%s:%d: bad pud %08lx.\n", __FILE__, __LINE__, pud_val(e))#define pgd_ERROR (e) \ printk("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
mm/memory.c +97 -123
439 lines not on this page · raw message
PATCH 11/21 mm: move_page_tables by extents
In reply to Hugh DickinsSpeeding up mremap's moving of ptes has never been a priority, but the locking will get more complicated shortly, and is already too baroque.
Scrap the current one-by-one moving, do an extent at a time: curtailed by end of src and dst pmds (have to use PMD_SIZE: the way pmd_addr_end gets elided doesn't match this usage), and by latency considerations.
One nice property of the old method is lost: it never allocated a page table unless absolutely necessary, so you could free empty page tables by mremapping to and fro. Whereas this way, it allocates a dst table wherever there was a src table. I keep diving in to reinstate the old behaviour, then come out preferring not to clutter how it now is.
---
mm/mremap.c | 166 +++++++++++++++++++++++++----------------------------------- 1 files changed, 71 insertions(+), 95 deletions(-)
mm/mremap.c +71 -95
@@ -22,40 +22,15 @@#include <asm/cacheflush.h> #include <asm/tlbflush.h> -static pte_t *get_one_pte_map_nested (struct mm_struct *mm,unsigned long addr) -{ -pgd_t *pgd; -pud_t *pud; -pmd_t *pmd; -pte_t *pte =NULL ; - - pgd =pgd_offset (mm, addr); -if (pgd_none_or_clear_bad (pgd)) - goto end; - - pud =pud_offset (pgd, addr); -if (pud_none_or_clear_bad (pud)) - goto end; - - pmd =pmd_offset (pud, addr); -if (pmd_none_or_clear_bad (pmd)) - goto end; - - pte =pte_offset_map_nested (pmd, addr); -if (pte_none (*pte)) { -pte_unmap_nested (pte); - pte =NULL ; - } -end: -return pte; -} - -static pte_t *get_one_pte_map (struct mm_struct *mm,unsigned long addr) +static pmd_t *get_old_pmd (struct mm_struct *mm,unsigned long addr) {pgd_t *pgd;pud_t *pud;pmd_t *pmd; +/* +* We don't need page_table_lock: we have mmap_sem exclusively. +*/ pgd =pgd_offset (mm, addr);if (pgd_none_or_clear_bad (pgd))return NULL ;
@@ -68,35 +43,48 @@ static pte_t *get_one_pte_map(struct mm_if (pmd_none_or_clear_bad (pmd))return NULL ; -return pte_offset_map (pmd, addr); +return pmd; } -static inline pte_t *alloc_one_pte_map (struct mm_struct *mm,unsigned long addr) +static pmd_t *alloc_new_pmd (struct mm_struct *mm,unsigned long addr) {pgd_t *pgd;pud_t *pud; -pmd_t *pmd; -pte_t *pte =NULL ; +pmd_t *pmd =NULL ; +pte_t *pte; +/* +* We do need page_table_lock: because allocators expect that. +*/ +spin_lock (&mm->page_table_lock); pgd =pgd_offset (mm, addr); - pud =pud_alloc (mm, pgd, addr);if (!pud) -return NULL ; + goto out; + pmd =pmd_alloc (mm, pud, addr); -if (pmd) - pte =pte_alloc_map (mm, pmd, addr); -return pte; +if (!pmd) + goto out; + + pte =pte_alloc_map (mm, pmd, addr); +if (!pte) { + pmd =NULL ; + goto out; + } +pte_unmap (pte); +out: +spin_unlock (&mm->page_table_lock); +return pmd; } -static int -move_one_page (struct vm_area_struct *vma,unsigned long old_addr, -struct vm_area_struct *new_vma,unsigned long new_addr) +static void move_ptes (struct vm_area_struct *vma,pmd_t *old_pmd, +unsigned long old_addr,unsigned long old_end, +struct vm_area_struct *new_vma,pmd_t *new_pmd, +unsigned long new_addr) {struct address_space *mapping =NULL ;struct mm_struct *mm = vma->vm_mm; -int error =0 ; -pte_t *src, *dst; +pte_t *old_pte, *new_pte, pte;if (vma->vm_file) {/*
@@ -111,74 +99,62 @@ move_one_page(struct vm_area_struct *vma new_vma->vm_truncate_count != vma->vm_truncate_count) new_vma->vm_truncate_count = 0; } +spin_lock (&mm->page_table_lock); + old_pte =pte_offset_map (old_pmd, old_addr); + new_pte =pte_offset_map_nested (new_pmd, new_addr); - src =get_one_pte_map_nested (mm, old_addr); -if (src) { -/* -* Look to see whether alloc_one_pte_map needs to perform a -* memory allocation. If it does then we need to drop the -* atomic kmap -*/ - dst =get_one_pte_map (mm, new_addr); -if (unlikely (!dst)) { -pte_unmap_nested (src); -if (mapping) -spin_unlock (&mapping->i_mmap_lock); - dst =alloc_one_pte_map (mm, new_addr); -if (mapping && !spin_trylock (&mapping->i_mmap_lock)) { -spin_unlock (&mm->page_table_lock); -spin_lock (&mapping->i_mmap_lock); -spin_lock (&mm->page_table_lock); - } - src =get_one_pte_map_nested (mm, old_addr); - } -/* -* Since alloc_one_pte_map can drop and re-acquire -* page_table_lock, we should re-check the src entry... -*/ -if (src) { -if (dst) { -pte_t pte; - pte =ptep_clear_flush (vma, old_addr, src); - -/* ZERO_PAGE can be dependant on virtual addr */ - pte =move_pte (pte, new_vma->vm_page_prot, - old_addr, new_addr); -set_pte_at (mm, new_addr, dst, pte); - }else - error = -ENOMEM ; -pte_unmap_nested (src); - } -if (dst) -pte_unmap (dst); +for (; old_addr < old_end; old_pte++, old_addr +=PAGE_SIZE , + new_pte++, new_addr +=PAGE_SIZE ) { +if (pte_none (*old_pte)) +continue ; + pte =ptep_clear_flush (vma, old_addr, old_pte); +/* ZERO_PAGE can be dependant on virtual addr */ + pte =move_pte (pte, new_vma->vm_page_prot, old_addr, new_addr); +set_pte_at (mm, new_addr, new_pte, pte); } + +pte_unmap_nested (new_pte -1 ); +pte_unmap (old_pte -1 );spin_unlock (&mm->page_table_lock);if (mapping)spin_unlock (&mapping->i_mmap_lock); -return error; } +#define LATENCY_LIMIT (64 * PAGE_SIZE) +static unsigned long move_page_tables (struct vm_area_struct *vma,unsigned long old_addr,struct vm_area_struct *new_vma,unsigned long new_addr,unsigned long len) { -unsigned long offset; +unsigned long extent, next, old_end; +pmd_t *old_pmd, *new_pmd; -flush_cache_range (vma, old_addr, old_addr + len); + old_end = old_addr + len; +flush_cache_range (vma, old_addr, old_end); -/* -* This is not the clever way to do this, but we're taking the -* easy way out on the assumption that most remappings will be -* only a few pages.. This also makes error recovery easier. -*/ -for (offset =0 ; offset < len; offset +=PAGE_SIZE ) { -if (move_one_page (vma, old_addr + offset, - new_vma, new_addr + offset) <0 ) -break ; +for (; old_addr < old_end; old_addr += extent, new_addr += extent) {cond_resched (); + next = (old_addr +PMD_SIZE ) &PMD_MASK ; +if (next -1 > old_end) + next = old_end; + extent = next - old_addr; + old_pmd =get_old_pmd (vma->vm_mm, old_addr); +if (!old_pmd) +continue ; + new_pmd =alloc_new_pmd (vma->vm_mm, new_addr); +if (!new_pmd) +break ; + next = (new_addr +PMD_SIZE ) &PMD_MASK ; +if (extent > next - new_addr) + extent = next - new_addr; +if (extent >LATENCY_LIMIT ) + extent =LATENCY_LIMIT ; +move_ptes (vma, old_pmd, old_addr, old_addr + extent, + new_vma, new_pmd, new_addr); } -return offset; + +return len + old_addr - old_end;/* how much done */ } static unsigned long move_vma(struct vm_area_struct *vma,
PATCH 12/21 mm: tlb_gather_mmu get_cpu_var
In reply to Hugh Dickinstlb_gather_mmu dates from before kernel preemption was allowed, and uses smp_processor_id or __get_cpu_var to find its per-cpu mmu_gather. That works because it's currently only called after getting page_table_lock, which is not dropped until after the matching tlb_finish_mmu. But don't rely on that, it will soon change: now disable preemption internally by proper get_cpu_var in tlb_gather_mmu, put_cpu_var in tlb_finish_mmu.
---
include/asm-arm/tlb.h | 5 +++-- include/asm-arm26/tlb.h | 7 ++++--- include/asm-generic/tlb.h | 10 +++++----- include/asm-ia64/tlb.h | 6 ++++-- include/asm-sparc64/tlb.h | 4 +++- 5 files changed, 19 insertions(+), 13 deletions(-)
include/asm-arm/tlb.h +3 -2
@@ -39,8 +39,7 @@ DECLARE_PER_CPU(struct mmu_gather, mmu_gstatic inline struct mmu_gather *tlb_gather_mmu (struct mm_struct *mm,unsigned int full_mm_flush) { -int cpu =smp_processor_id (); -struct mmu_gather *tlb = &per_cpu (mmu_gathers, cpu); +struct mmu_gather *tlb = &get_cpu_var (mmu_gathers); tlb->mm = mm; tlb->freed =0 ;
@@ -65,6 +64,8 @@ tlb_finish_mmu(struct mmu_gather *tlb, u/* keep the page table cache within bounds */ check_pgt_cache (); + +put_cpu_var (mmu_gathers); }static inline unsigned int tlb_is_full_mm (struct mmu_gather *tlb)
include/asm-arm26/tlb.h +4 -3
@@ -17,13 +17,12 @@ struct mmu_gather {unsigned int avoided_flushes; }; -extern struct mmu_gather mmu_gathers[NR_CPUS ]; +DECLARE_PER_CPU(struct mmu_gather, mmu_gathers);static inline struct mmu_gather *tlb_gather_mmu (struct mm_struct *mm,unsigned int full_mm_flush) { -int cpu =smp_processor_id (); -struct mmu_gather *tlb = &mmu_gathers[cpu]; +struct mmu_gather *tlb = &get_cpu_var (mmu_gathers); tlb->mm = mm; tlb->freed =0 ;
@@ -52,6 +51,8 @@ tlb_finish_mmu(struct mmu_gather *tlb, u/* keep the page table cache within bounds */ check_pgt_cache (); + +put_cpu_var (mmu_gathers); }
include/asm-generic/tlb.h +5 -5
@@ -35,9 +35,7 @@#endif /* struct mmu_gather is an opaque type used by the mm code for passing around -* any data needed by arch specific code for tlb_remove_page. This structure -* can be per-CPU or per-MM as the page table lock is held for the duration of -* TLB shootdown. +* any data needed by arch specific code for tlb_remove_page. */ struct mmu_gather {struct mm_struct *mm;
@@ -57,7 +55,7 @@ DECLARE_PER_CPU(struct mmu_gather, mmu_gstatic inline struct mmu_gather *tlb_gather_mmu (struct mm_struct *mm,unsigned int full_mm_flush) { -struct mmu_gather *tlb = &per_cpu (mmu_gathers,smp_processor_id ()); +struct mmu_gather *tlb = &get_cpu_var (mmu_gathers); tlb->mm = mm;
@@ -85,7 +83,7 @@ tlb_flush_mmu(struct mmu_gather *tlb, un/* tlb_finish_mmu * Called at the end of the shootdown operation to free up any resources -* that were required. The page table lock is still held at this point. +* that were required. */ static inline void tlb_finish_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end)
@@ -101,6 +99,8 @@ tlb_finish_mmu(struct mmu_gather *tlb, u/* keep the page table cache within bounds */ check_pgt_cache (); + +put_cpu_var (mmu_gathers); } static inline unsigned int
include/asm-ia64/tlb.h +4 -2
@@ -129,7 +129,7 @@ ia64_tlb_flush_mmu (struct mmu_gather *tstatic inline struct mmu_gather *tlb_gather_mmu (struct mm_struct *mm,unsigned int full_mm_flush) { -struct mmu_gather *tlb = &__get_cpu_var (mmu_gathers); +struct mmu_gather *tlb = &get_cpu_var (mmu_gathers); tlb->mm = mm;/*
@@ -154,7 +154,7 @@ tlb_gather_mmu (struct mm_struct *mm, un/* * Called at the end of the shootdown operation to free up any resources that were -* collected. The page table lock is still held at this point. +* collected. */ static inline void tlb_finish_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end)
@@ -174,6 +174,8 @@ tlb_finish_mmu (struct mmu_gather *tlb,/* keep the page table cache within bounds */ check_pgt_cache (); + +put_cpu_var (mmu_gathers); } static inline unsigned int
include/asm-sparc64/tlb.h +3 -1
@@ -44,7 +44,7 @@ extern void flush_tlb_pending(void);static inline struct mmu_gather *tlb_gather_mmu (struct mm_struct *mm,unsigned int full_mm_flush) { -struct mmu_gather *mp = &__get_cpu_var (mmu_gathers); +struct mmu_gather *mp = &get_cpu_var (mmu_gathers);BUG_ON (mp->tlb_nr);
@@ -97,6 +97,8 @@ static inline void tlb_finish_mmu(struct/* keep the page table cache within bounds */ check_pgt_cache (); + +put_cpu_var (mmu_gathers); }static inline unsigned int tlb_is_full_mm (struct mmu_gather *mp)
PATCH 13/21 mm: tlb_is_full_mm was obscure
In reply to Hugh Dickinstlb_is_full_mm? What does that mean? The TLB is full? No, it means that the mm's last user has gone and the whole mm is being torn down. And it's an inline function because sparc64 uses a different (slightly better) "tlb_frozen" name for the flag others call "fullmm".
And now the ptep_get_and_clear_full macro used in zap_pte_range refers directly to tlb->fullmm, which would be wrong for sparc64. Rather than correct that, I'd prefer to scrap tlb_is_full_mm altogether, and change sparc64 to just use the same poor name as everyone else - is that okay?
---
arch/sparc64/mm/tlb.c | 4 ++-- include/asm-arm/tlb.h | 5 ----- include/asm-arm26/tlb.h | 7 ------- include/asm-generic/tlb.h | 6 ------ include/asm-ia64/tlb.h | 6 ------ include/asm-sparc64/tlb.h | 13 ++++--------- mm/memory.c | 4 ++-- 7 files changed, 8 insertions(+), 37 deletions(-)
arch/sparc64/mm/tlb.c +2 -2
@@ -72,7 +72,7 @@ void tlb_batch_add(struct mm_struct *mm, no_cache_flush: -if (mp->tlb_frozen) +if (mp->fullmm)return ; nr = mp->tlb_nr;
@@ -97,7 +97,7 @@ void flush_tlb_pgtables(struct mm_structunsigned long nr = mp->tlb_nr;long s = start, e = end, vpte_base; -if (mp->tlb_frozen) +if (mp->fullmm)return ;/* If start is greater than end, that is a real problem. */
include/asm-arm/tlb.h +0 -5
@@ -68,11 +68,6 @@ tlb_finish_mmu(struct mmu_gather *tlb, uput_cpu_var (mmu_gathers); } -static inline unsigned int tlb_is_full_mm (struct mmu_gather *tlb) -{ -return tlb->fullmm; -} -#define tlb_remove_tlb_entry (tlb,ptep,address) do { } while (0)/*
include/asm-arm26/tlb.h +0 -7
@@ -55,13 +55,6 @@ tlb_finish_mmu(struct mmu_gather *tlb, uput_cpu_var (mmu_gathers); } - -static inline unsigned int -tlb_is_full_mm (struct mmu_gather *tlb) -{ -return tlb->fullmm; -} -#define tlb_remove_tlb_entry (tlb,ptep,address) do { } while (0)//#define tlb_start_vma(tlb,vma) do { } while (0) //FIXME - ARM32 uses this now that things changed in the kernel. seems like it may be pointless on arm26, however to get things compiling...
include/asm-generic/tlb.h +0 -6
@@ -103,12 +103,6 @@ tlb_finish_mmu(struct mmu_gather *tlb, uput_cpu_var (mmu_gathers); } -static inline unsigned int -tlb_is_full_mm (struct mmu_gather *tlb) -{ -return tlb->fullmm; -} -/* tlb_remove_page * Must perform the equivalent to __free_pte(pte_get_and_clear(ptep)), while * handling the additional races in SMP caused by other CPUs caching valid
include/asm-ia64/tlb.h +0 -6
@@ -178,12 +178,6 @@ tlb_finish_mmu (struct mmu_gather *tlb,put_cpu_var (mmu_gathers); } -static inline unsigned int -tlb_is_full_mm (struct mmu_gather *tlb) -{ -return tlb->fullmm; -} -/* * Logically, this routine frees PAGE. On MP machines, the actual freeing of the page * must be delayed until after the TLB has been flushed (see comments at the beginning of
include/asm-sparc64/tlb.h +4 -9
@@ -25,7 +25,7 @@ struct mmu_gather {struct mm_struct *mm;unsigned int pages_nr;unsigned int need_flush; -unsigned int tlb_frozen; +unsigned int fullmm;unsigned int tlb_nr;unsigned long freed;unsigned long vaddrs[TLB_BATCH_NR ];
@@ -50,7 +50,7 @@ static inline struct mmu_gather *tlb_gat mp->mm = mm; mp->pages_nr =num_online_cpus () >1 ?0U : ~0U ; - mp->tlb_frozen = full_mm_flush; + mp->fullmm = full_mm_flush; mp->freed =0 ;return mp;
@@ -88,10 +88,10 @@ static inline void tlb_finish_mmu(structtlb_flush_mmu (mp); -if (mp->tlb_frozen) { +if (mp->fullmm) {if (CTX_VALID (mm->context))do_flush_tlb_mm (mm); - mp->tlb_frozen =0 ; + mp->fullmm =0 ; }else flush_tlb_pending ();
@@ -101,11 +101,6 @@ static inline void tlb_finish_mmu(structput_cpu_var (mmu_gathers); } -static inline unsigned int tlb_is_full_mm (struct mmu_gather *mp) -{ -return mp->tlb_frozen; -} -static inline void tlb_remove_page (struct mmu_gather *mp,struct page *page) { mp->need_flush =1 ;
mm/memory.c +2 -2
@@ -249,7 +249,7 @@ void free_pgd_range(struct mmu_gather **free_pud_range (*tlb, pgd, addr, next, floor, ceiling); }while (pgd++, addr = next, addr != end); -if (!tlb_is_full_mm (*tlb)) +if (!(*tlb)->fullmm)flush_tlb_pgtables ((*tlb)->mm, start, end); }
@@ -698,7 +698,7 @@ unsigned long unmap_vmas(struct mmu_gathint tlb_start_valid =0 ;unsigned long start = start_addr;spinlock_t *i_mmap_lock = details? details->i_mmap_lock:NULL ; -int fullmm =tlb_is_full_mm (*tlbp); +int fullmm = (*tlbp)->fullmm;for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) {unsigned long end;
PATCH 14/21 mm: tlb_finish_mmu forget rss
In reply to Hugh Dickinszap_pte_range has been counting the pages it frees in tlb->freed, then tlb_finish_mmu has used that to update the mm's rss. That got stranger when I added anon_rss, yet updated it by a different route; and stranger when rss and anon_rss became mm_counters with special access macros. And it would no longer be viable if we're relying on page_table_lock to stabilize the mm_counter, but calling tlb_finish_mmu outside that lock.
Remove the mmu_gather's freed field, let tlb_finish_mmu stick to its own business, just decrement the rss mm_counter in zap_pte_range (yes, there was some point to batching the update, and a subsequent patch restores that). And forget the anal paranoia of first reading the counter to avoid going negative - if rss does go negative, just fix that bug.
Remove the mmu_gather's flushes and avoided_flushes from arm and arm26:
no use was being made of them. But arm26 alone was actually using the freed, in the way some others use need_flush: give it a need_flush. arm26 seems to prefer spaces to tabs here: respect that.
---
include/asm-arm/tlb.h | 15 +-------------- include/asm-arm26/tlb.h | 35 +++++++++++++---------------------- include/asm-generic/tlb.h | 9 --------- include/asm-ia64/tlb.h | 9 --------- include/asm-sparc64/tlb.h | 14 ++------------ mm/memory.c | 2 +- 6 files changed, 17 insertions(+), 67 deletions(-)
include/asm-arm/tlb.h +1 -14
@@ -27,11 +27,7 @@*/ struct mmu_gather {struct mm_struct *mm; -unsigned int freed;unsigned int fullmm; - -unsigned int flushes; -unsigned int avoided_flushes; }; DECLARE_PER_CPU(struct mmu_gather, mmu_gathers);
@@ -42,7 +38,6 @@ tlb_gather_mmu(struct mm_struct *mm, unsstruct mmu_gather *tlb = &get_cpu_var (mmu_gathers); tlb->mm = mm; - tlb->freed =0 ; tlb->fullmm = full_mm_flush;return tlb;
@@ -51,16 +46,8 @@ tlb_gather_mmu(struct mm_struct *mm, unsstatic inline void tlb_finish_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) { -struct mm_struct *mm = tlb->mm; -unsigned long freed = tlb->freed; -int rss =get_mm_counter (mm, rss); - -if (rss < freed) - freed = rss; -add_mm_counter (mm, rss, -freed); -if (tlb->fullmm) -flush_tlb_mm (mm); +flush_tlb_mm (tlb->mm);/* keep the page table cache within bounds */ check_pgt_cache ();
include/asm-arm26/tlb.h +13 -22
@@ -10,11 +10,8 @@*/ struct mmu_gather {struct mm_struct *mm; -unsigned int freed; -unsigned int fullmm; - -unsigned int flushes; -unsigned int avoided_flushes; +unsigned int need_flush; +unsigned int fullmm; }; DECLARE_PER_CPU(struct mmu_gather, mmu_gathers);
@@ -25,8 +22,8 @@ tlb_gather_mmu(struct mm_struct *mm, unsstruct mmu_gather *tlb = &get_cpu_var (mmu_gathers); tlb->mm = mm; - tlb->freed =0 ; - tlb->fullmm = full_mm_flush; + tlb->need_flush =0 ; + tlb->fullmm = full_mm_flush;return tlb; }
@@ -34,20 +31,8 @@ tlb_gather_mmu(struct mm_struct *mm, unsstatic inline void tlb_finish_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) { -struct mm_struct *mm = tlb->mm; -unsigned long freed = tlb->freed; -int rss =get_mm_counter (mm, rss); - -if (rss < freed) - freed = rss; -add_mm_counter (mm, rss, -freed); - -if (freed) { -flush_tlb_mm (mm); - tlb->flushes++; - }else { - tlb->avoided_flushes++; - } +if (tlb->need_flush) +flush_tlb_mm (tlb->mm);/* keep the page table cache within bounds */ check_pgt_cache ();
@@ -65,7 +50,13 @@ tlb_finish_mmu(struct mmu_gather *tlb, u } while (0)#define tlb_end_vma (tlb,vma) do { } while (0) -#define tlb_remove_page (tlb,page) free_page_and_swap_cache(page) +static inline void +tlb_remove_page (struct mmu_gather *tlb,struct page *page) +{ + tlb->need_flush =1 ; +free_page_and_swap_cache (page); +} +#define pte_free_tlb (tlb,ptep) pte_free(ptep)#define pmd_free_tlb (tlb,pmdp) pmd_free(pmdp)
include/asm-generic/tlb.h +0 -9
@@ -42,7 +42,6 @@ struct mmu_gather {unsigned int nr;/* set to ~0U means fast mode */ unsigned int need_flush;/* Really unmapped some ptes? */ unsigned int fullmm;/* non-zero means full mm flush */ -unsigned long freed;struct page * pages[FREE_PTE_NR ]; };
@@ -63,7 +62,6 @@ tlb_gather_mmu(struct mm_struct *mm, uns tlb->nr =num_online_cpus () >1 ?0U : ~0U ; tlb->fullmm = full_mm_flush; - tlb->freed =0 ;return tlb; }
@@ -88,13 +86,6 @@ tlb_flush_mmu(struct mmu_gather *tlb, unstatic inline void tlb_finish_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) { -int freed = tlb->freed; -struct mm_struct *mm = tlb->mm; -int rss =get_mm_counter (mm, rss); - -if (rss < freed) - freed = rss; -add_mm_counter (mm, rss, -freed);tlb_flush_mmu (tlb, start, end);/* keep the page table cache within bounds */
include/asm-ia64/tlb.h +0 -9
@@ -60,7 +60,6 @@ struct mmu_gather {unsigned int nr;/* == ~0U => fast mode */ unsigned char fullmm;/* non-zero means full mm flush */ unsigned char need_flush;/* really unmapped some PTEs? */ -unsigned long freed;/* number of pages freed */ unsigned long start_addr;unsigned long end_addr;struct page *pages[FREE_PTE_NR ];
@@ -147,7 +146,6 @@ tlb_gather_mmu (struct mm_struct *mm, un*/ tlb->nr = (num_online_cpus () ==1 ) ? ~0U :0 ; tlb->fullmm = full_mm_flush; - tlb->freed =0 ; tlb->start_addr = ~0UL ;return tlb; }
@@ -159,13 +157,6 @@ tlb_gather_mmu (struct mm_struct *mm, unstatic inline void tlb_finish_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end) { -unsigned long freed = tlb->freed; -struct mm_struct *mm = tlb->mm; -unsigned long rss =get_mm_counter (mm, rss); - -if (rss < freed) - freed = rss; -add_mm_counter (mm, rss, -freed);/* * Note: tlb->nr may be 0 at this point, so we can't rely on tlb->start_addr and * tlb->end_addr.
include/asm-sparc64/tlb.h +2 -12
@@ -27,7 +27,6 @@ struct mmu_gather {unsigned int need_flush;unsigned int fullmm;unsigned int tlb_nr; -unsigned long freed;unsigned long vaddrs[TLB_BATCH_NR ];struct page *pages[FREE_PTE_NR ]; };
@@ -51,7 +50,6 @@ static inline struct mmu_gather *tlb_gat mp->mm = mm; mp->pages_nr =num_online_cpus () >1 ?0U : ~0U ; mp->fullmm = full_mm_flush; - mp->freed =0 ;return mp; }
@@ -78,19 +76,11 @@ extern void smp_flush_tlb_mm(struct mm_sstatic inline void tlb_finish_mmu (struct mmu_gather *mp,unsigned long start,unsigned long end) { -unsigned long freed = mp->freed; -struct mm_struct *mm = mp->mm; -unsigned long rss =get_mm_counter (mm, rss); - -if (rss < freed) - freed = rss; -add_mm_counter (mm, rss, -freed); -tlb_flush_mmu (mp);if (mp->fullmm) { -if (CTX_VALID (mm->context)) -do_flush_tlb_mm (mm); +if (CTX_VALID (mp->mm->context)) +do_flush_tlb_mm (mp->mm); mp->fullmm =0 ; }else flush_tlb_pending ();
mm/memory.c +1 -1
@@ -582,7 +582,7 @@ static void zap_pte_range(struct mmu_gatif (pte_young (ptent))mark_page_accessed (page); } - tlb->freed++; +dec_mm_counter (tlb->mm, rss);page_remove_rmap (page);tlb_remove_page (tlb, page);continue ;
PATCH 15/21 mm: mm_init set_mm_counters
In reply to Hugh DickinsHow is anon_rss initialized? In dup_mmap, and by mm_alloc's memset; but that's not so good if an mm_counter_t is a special type. And how is rss initialized? By set_mm_counter, all over the place. Come on, we just need to initialize them both at once by set_mm_counter in mm_init (which follows the memcpy when forking).
---
arch/mips/kernel/irixelf.c | 1 - arch/sparc64/kernel/binfmt_aout32.c | 1 - arch/x86_64/ia32/ia32_aout.c | 1 - fs/binfmt_aout.c | 1 - fs/binfmt_elf.c | 1 - fs/binfmt_elf_fdpic.c | 7 ------- fs/binfmt_flat.c | 1 - fs/binfmt_som.c | 1 - kernel/fork.c | 4 ++-- 9 files changed, 2 insertions(+), 16 deletions(-)
arch/mips/kernel/irixelf.c +0 -1
@@ -692,7 +692,6 @@ static int load_irix_binary(struct linux/* Do this so that we can load the interpreter, if need be. We will * change some of these later. */ -set_mm_counter (current->mm, rss,0 );setup_arg_pages (bprm,STACK_TOP ,EXSTACK_DEFAULT ); current->mm->start_stack = bprm->p;
arch/sparc64/kernel/binfmt_aout32.c +0 -1
@@ -241,7 +241,6 @@ static int load_aout32_binary(struct lin current->mm->brk = ex.a_bss + (current->mm->start_brk =N_BSSADDR (ex)); -set_mm_counter (current->mm, rss,0 ); current->mm->mmap =NULL ;compute_creds (bprm); current->flags &= ~PF_FORKNOEXEC ;
arch/x86_64/ia32/ia32_aout.c +0 -1
@@ -314,7 +314,6 @@ static int load_aout_binary(struct linux current->mm->free_area_cache =TASK_UNMAPPED_BASE ; current->mm->cached_hole_size =0 ; -set_mm_counter (current->mm, rss,0 ); current->mm->mmap =NULL ;compute_creds (bprm); current->flags &= ~PF_FORKNOEXEC ;
fs/binfmt_aout.c +0 -1
@@ -318,7 +318,6 @@ static int load_aout_binary(struct linux current->mm->free_area_cache = current->mm->mmap_base; current->mm->cached_hole_size =0 ; -set_mm_counter (current->mm, rss,0 ); current->mm->mmap =NULL ;compute_creds (bprm); current->flags &= ~PF_FORKNOEXEC ;
fs/binfmt_elf.c +0 -1
@@ -773,7 +773,6 @@ static int load_elf_binary(struct linux_/* Do this so that we can load the interpreter, if need be. We will change some of these later */ -set_mm_counter (current->mm, rss,0 ); current->mm->free_area_cache = current->mm->mmap_base; current->mm->cached_hole_size =0 ; retval = setup_arg_pages(bprm, randomize_stack_top(STACK_TOP),
fs/binfmt_elf_fdpic.c +0 -7
@@ -294,14 +294,7 @@ static int load_elf_fdpic_binary(struct &interp_params, ¤t->mm->start_stack, ¤t->mm->start_brk); -#endif - -/* do this so that we can load the interpreter, if need be -* - we will change some of these later -*/ -set_mm_counter (current->mm, rss,0 ); -#ifdef CONFIG_MMU retval =setup_arg_pages (bprm, current->mm->start_stack, executable_stack);if (retval <0 ) {send_sig (SIGKILL , current,0 );
fs/binfmt_flat.c +0 -1
@@ -650,7 +650,6 @@ static int load_flat_file(struct linux_b current->mm->start_brk = datapos + data_len + bss_len; current->mm->brk = (current->mm->start_brk +3 ) & ~3 ; current->mm->context.end_brk = memp +ksize ((void *) memp) - stack_len; -set_mm_counter (current->mm, rss,0 ); }if (flags &FLAT_FLAG_KTRACE )
fs/binfmt_som.c +0 -1
@@ -259,7 +259,6 @@ load_som_binary(struct linux_binprm * bpcreate_som_tables (bprm); current->mm->start_stack = bprm->p; -set_mm_counter (current->mm, rss,0 );#if 0 printk ("(start_brk) %08lx\n" , (unsigned long ) current->mm->start_brk);
kernel/fork.c +2 -2
@@ -198,8 +198,6 @@ static inline int dup_mmap(struct mm_str mm->free_area_cache = oldmm->mmap_base; mm->cached_hole_size = ~0UL ; mm->map_count =0 ; -set_mm_counter (mm, rss,0 ); -set_mm_counter (mm, anon_rss,0 );cpus_clear (mm->cpu_vm_mask); mm->mm_rb =RB_ROOT ; rb_link = &mm->mm_rb.rb_node;
@@ -323,6 +321,8 @@ static struct mm_struct * mm_init(structINIT_LIST_HEAD (&mm->mmlist); mm->core_waiters =0 ; mm->nr_ptes =0 ; +set_mm_counter (mm, rss,0 ); +set_mm_counter (mm, anon_rss,0 );spin_lock_init (&mm->page_table_lock);rwlock_init (&mm->ioctx_list_lock); mm->ioctx_list =NULL ;
PATCH 16/21 mm: rss = file_rss + anon_rss
In reply to Hugh DickinsI was lazy when we added anon_rss, and chose to change as few places as possible. So currently each anonymous page has to be counted twice, in rss and in anon_rss. Which won't be so good if those are atomic counts in some configurations.
Change that around: keep file_rss and anon_rss separately, and add them together (with get_mm_rss macro) when the total is needed - reading two atomics is much cheaper than updating two atomics. And update anon_rss upfront, typically in memory.c, not tucked away in page_add_anon_rmap.
---
fs/exec.c | 2 +- fs/proc/array.c | 2 +- fs/proc/task_mmu.c | 8 +++----- include/linux/sched.h | 4 +++- kernel/acct.c | 2 +- kernel/fork.c | 4 ++-- mm/fremap.c | 4 ++-- mm/hugetlb.c | 6 +++--- mm/memory.c | 31 +++++++++++++++++-------------- mm/nommu.c | 2 +- mm/rmap.c | 8 +++----- mm/swapfile.c | 2 +- 12 files changed, 38 insertions(+), 37 deletions(-)
fs/exec.c +1 -1
@@ -330,7 +330,7 @@ void install_arg_page(struct vm_area_strpte_unmap (pte); goto out; } -inc_mm_counter (mm, rss); +inc_mm_counter (mm, anon_rss);lru_cache_add_active (page);set_pte_at (mm, address, pte,pte_mkdirty (pte_mkwrite (mk_pte ( page, vma->vm_page_prot))));
fs/proc/array.c +1 -1
@@ -438,7 +438,7 @@ static int do_task_stat(struct task_strujiffies_to_clock_t (it_real_value), start_time, vsize, - mm ?get_mm_counter (mm, rss) :0 ,/* you might want to shift this left 3 */ + mm ?get_mm_rss (mm) :0 , rsslim, mm ? mm->start_code :0 , mm ? mm->end_code : 0,
fs/proc/task_mmu.c +3 -5
@@ -29,7 +29,7 @@ char *task_mem(struct mm_struct *mm, cha"VmPTE:\t%8lu kB\n" , (mm->total_vm - mm->reserved_vm) << (PAGE_SHIFT -10 ), mm->locked_vm << (PAGE_SHIFT -10 ), -get_mm_counter (mm, rss) << (PAGE_SHIFT -10 ), +get_mm_rss (mm) << (PAGE_SHIFT -10 ), data << (PAGE_SHIFT -10 ), mm->stack_vm << (PAGE_SHIFT -10 ), text, lib, (PTRS_PER_PTE*sizeof(pte_t)*mm->nr_ptes) >> 10);
@@ -44,13 +44,11 @@ unsigned long task_vsize(struct mm_strucint task_statm (struct mm_struct *mm,int *shared,int *text,int *data,int *resident) { -int rss =get_mm_counter (mm, rss); - - *shared = rss -get_mm_counter (mm, anon_rss); + *shared =get_mm_counter (mm, file_rss); *text = (PAGE_ALIGN (mm->end_code) - (mm->start_code &PAGE_MASK )) >>PAGE_SHIFT ; *data = mm->total_vm - mm->shared_vm; - *resident = rss; + *resident = *shared +get_mm_counter (mm, anon_rss);return mm->total_vm; }
include/linux/sched.h +3 -1
@@ -243,6 +243,8 @@ extern void arch_unmap_area_topdown(stru#define add_mm_counter (mm, member, value) (mm)->_##member += (value)#define inc_mm_counter (mm, member) (mm)->_##member++#define dec_mm_counter (mm, member) (mm)->_##member-- +#define get_mm_rss (mm) ((mm)->_file_rss + (mm)->_anon_rss) +typedef unsigned long mm_counter_t ;struct mm_struct {
@@ -275,7 +277,7 @@ struct mm_struct {unsigned long exec_vm, stack_vm, reserved_vm, def_flags, nr_ptes;/* Special counters protected by the page_table_lock */ -mm_counter_t _rss; +mm_counter_t _file_rss;mm_counter_t _anon_rss;unsigned long saved_auxv[AT_VECTOR_SIZE ];/* for /proc/PID/auxv */
kernel/acct.c +1 -1
@@ -553,7 +553,7 @@ void acct_update_integrals(struct task_sif (delta ==0 )return ; tsk->acct_stimexpd = tsk->stime; - tsk->acct_rss_mem1 += delta *get_mm_counter (tsk->mm, rss); + tsk->acct_rss_mem1 += delta *get_mm_rss (tsk->mm); tsk->acct_vm_mem1 += delta * tsk->mm->total_vm; } }
kernel/fork.c +2 -2
@@ -321,7 +321,7 @@ static struct mm_struct * mm_init(structINIT_LIST_HEAD (&mm->mmlist); mm->core_waiters =0 ; mm->nr_ptes =0 ; -set_mm_counter (mm, rss,0 ); +set_mm_counter (mm, file_rss,0 );set_mm_counter (mm, anon_rss,0 );spin_lock_init (&mm->page_table_lock);rwlock_init (&mm->ioctx_list_lock);
@@ -499,7 +499,7 @@ static int copy_mm(unsigned long clone_fif (retval) goto free_pt; - mm->hiwater_rss =get_mm_counter (mm,rss); + mm->hiwater_rss =get_mm_rss (mm); mm->hiwater_vm = mm->total_vm; good_mm:
mm/fremap.c +2 -2
@@ -39,7 +39,7 @@ static inline void zap_pte(struct mm_strset_page_dirty (page);page_remove_rmap (page);page_cache_release (page); -dec_mm_counter (mm, rss); +dec_mm_counter (mm, file_rss); } } }else {
@@ -92,7 +92,7 @@ int install_page(struct mm_struct *mm, szap_pte (mm, vma, addr, pte); -inc_mm_counter (mm,rss); +inc_mm_counter (mm, file_rss);flush_icache_page (vma, page);set_pte_at (mm, addr, pte,mk_pte (page, prot));page_add_file_rmap (page);
mm/hugetlb.c +3 -3
@@ -285,7 +285,7 @@ int copy_hugetlb_page_range(struct mm_st entry = *src_pte; ptepage =pte_page (entry);get_page (ptepage); -add_mm_counter (dst, rss,HPAGE_SIZE /PAGE_SIZE ); +add_mm_counter (dst, file_rss,HPAGE_SIZE /PAGE_SIZE );set_huge_pte_at (dst, addr, dst_pte, entry); }spin_unlock (&src->page_table_lock);
@@ -323,7 +323,7 @@ void unmap_hugepage_range(struct vm_area page =pte_page (pte);put_page (page); -add_mm_counter (mm, rss, - (HPAGE_SIZE /PAGE_SIZE )); +add_mm_counter (mm, file_rss, - (HPAGE_SIZE /PAGE_SIZE )); }flush_tlb_range (vma, start, end); }
@@ -385,7 +385,7 @@ int hugetlb_prefault(struct address_spac goto out; } } -add_mm_counter (mm, rss,HPAGE_SIZE /PAGE_SIZE ); +add_mm_counter (mm, file_rss,HPAGE_SIZE /PAGE_SIZE );set_huge_pte_at (mm, addr, pte,make_huge_pte (vma, page)); } out:
mm/memory.c +17 -14
@@ -397,9 +397,10 @@ copy_one_pte(struct mm_struct *dst_mm, s pte =pte_mkclean (pte); pte =pte_mkold (pte);get_page (page); -inc_mm_counter (dst_mm, rss);if (PageAnon (page))inc_mm_counter (dst_mm, anon_rss); +else +inc_mm_counter (dst_mm, file_rss);set_pte_at (dst_mm, addr, dst_pte, pte);page_dup_rmap (page); }
@@ -581,8 +582,8 @@ static void zap_pte_range(struct mmu_gatset_page_dirty (page);if (pte_young (ptent))mark_page_accessed (page); +dec_mm_counter (tlb->mm, file_rss); } -dec_mm_counter (tlb->mm, rss);page_remove_rmap (page);tlb_remove_page (tlb, page);continue ;
@@ -1290,13 +1291,15 @@ static int do_wp_page(struct mm_struct *spin_lock (&mm->page_table_lock); page_table =pte_offset_map (pmd, address);if (likely (pte_same (*page_table, orig_pte))) { -if (PageAnon (old_page)) -dec_mm_counter (mm, anon_rss);if (PageReserved (old_page)) -inc_mm_counter (mm, rss); -else +inc_mm_counter (mm, anon_rss); +else {page_remove_rmap (old_page); - +if (!PageAnon (old_page)) { +inc_mm_counter (mm, anon_rss); +dec_mm_counter (mm, file_rss); + } + }flush_cache_page (vma, address, pfn); entry =mk_pte (new_page, vma->vm_page_prot); entry =maybe_mkwrite (pte_mkdirty (entry), vma);
@@ -1701,7 +1704,7 @@ static int do_swap_page(struct mm_struct/* The page isn't present yet, go ahead with the fault. */ -inc_mm_counter (mm, rss); +inc_mm_counter (mm, anon_rss); pte =mk_pte (page, vma->vm_page_prot);if (write_access &&can_share_swap_page (page)) { pte =maybe_mkwrite (pte_mkdirty (pte), vma);
@@ -1774,7 +1777,7 @@ static int do_anonymous_page(struct mm_spage_cache_release (page); goto unlock; } -inc_mm_counter (mm, rss); +inc_mm_counter (mm, anon_rss); entry =mk_pte (page, vma->vm_page_prot); entry =maybe_mkwrite (pte_mkdirty (entry), vma);lru_cache_add_active (page);
@@ -1887,19 +1890,19 @@ retry:*/ /* Only go through if we didn't race with anybody else... */ if (pte_none (*page_table)) { -if (!PageReserved (new_page)) -inc_mm_counter (mm, rss); -flush_icache_page (vma, new_page); entry =mk_pte (new_page, vma->vm_page_prot);if (write_access) entry =maybe_mkwrite (pte_mkdirty (entry), vma);set_pte_at (mm, address, page_table, entry);if (anon) { +inc_mm_counter (mm, anon_rss);lru_cache_add_active (new_page);page_add_anon_rmap (new_page, vma, address); - }else + }else if (!PageReserved (new_page)) { +inc_mm_counter (mm, file_rss);page_add_file_rmap (new_page); + } }else {/* One of our sibling threads was faster, back out. */ page_cache_release (new_page);
@@ -2192,7 +2195,7 @@ EXPORT_SYMBOL(vmalloc_to_pfn);void update_mem_hiwater (struct task_struct *tsk) {if (tsk->mm) { -unsigned long rss =get_mm_counter (tsk->mm, rss); +unsigned long rss =get_mm_rss (tsk->mm);if (tsk->mm->hiwater_rss < rss) tsk->mm->hiwater_rss = rss;
mm/nommu.c +1 -1
@@ -1080,7 +1080,7 @@ void update_mem_hiwater(struct task_struunsigned long rss;if (likely (tsk->mm)) { - rss =get_mm_counter (tsk->mm, rss); + rss =get_mm_rss (tsk->mm);if (tsk->mm->hiwater_rss < rss) tsk->mm->hiwater_rss = rss;if (tsk->mm->hiwater_vm < tsk->mm->total_vm)
mm/rmap.c +3 -5
@@ -445,8 +445,6 @@ void page_add_anon_rmap(struct page *pag {BUG_ON (PageReserved (page)); -inc_mm_counter (vma->vm_mm, anon_rss); -if (atomic_inc_and_test (&page->_mapcount)) {struct anon_vma *anon_vma = vma->anon_vma;
@@ -561,9 +559,9 @@ static int try_to_unmap_one(struct pageset_pte_at (mm, address, pte,swp_entry_to_pte (entry));BUG_ON (pte_file (*pte));dec_mm_counter (mm, anon_rss); - } + }else +dec_mm_counter (mm, file_rss); -dec_mm_counter (mm, rss);page_remove_rmap (page);page_cache_release (page);
@@ -667,7 +665,7 @@ static void try_to_unmap_cluster(unsignepage_remove_rmap (page);page_cache_release (page); -dec_mm_counter (mm, rss); +dec_mm_counter (mm, file_rss); (*mapcount)--; }
mm/swapfile.c +1 -1
@@ -405,7 +405,7 @@ void free_swap_and_cache(swp_entry_t entstatic void unuse_pte (struct vm_area_struct *vma,pte_t *pte,unsigned long addr,swp_entry_t entry,struct page *page) { -inc_mm_counter (vma->vm_mm, rss); +inc_mm_counter (vma->vm_mm, anon_rss);get_page (page);set_pte_at (vma->vm_mm, addr, pte,pte_mkold (mk_pte (page, vma->vm_page_prot)));
PATCH 17/21 mm: batch updating mm_counters
In reply to Hugh Dickinstlb_finish_mmu used to batch zap_pte_range's update of mm rss, which may be worthwhile if the mm is contended, and would reduce atomic operations if the counts were atomic. Let zap_pte_range now batch its updates to file_rss and anon_rss, per page-table in case we drop the lock outside; and copy_pte_range batch them too.
---
mm/memory.c | 47 ++++++++++++++++++++++++++++++++--------------- 1 files changed, 32 insertions(+), 15 deletions(-)
mm/memory.c +32 -15
@@ -332,6 +332,16 @@ out:return pte_offset_kernel (pmd, address); } +static inline void add_mm_rss (struct mm_struct *mm,int file_rss,int anon_rss) +{ +if (file_rss) +add_mm_counter (mm, file_rss, file_rss); +if (anon_rss) +add_mm_counter (mm, anon_rss, anon_rss); +} + +#define NO_RSS 2/* Increment neither file_rss nor anon_rss */ +/* * copy one vm_area from one task to the other. Assumes the page tables * already present in the new task to be cleared in the whole range
@@ -341,7 +351,7 @@ out:* but may be dropped within p[mg]d_alloc() and pte_alloc_map(). */ -static inline void +static inline int copy_one_pte (struct mm_struct *dst_mm,struct mm_struct *src_mm,pte_t *dst_pte,pte_t *src_pte,unsigned long vm_flags,unsigned long addr)
@@ -349,6 +359,7 @@ copy_one_pte(struct mm_struct *dst_mm, spte_t pte = *src_pte;struct page *page;unsigned long pfn; +int anon =NO_RSS ;/* pte contains position in swap or file, so copy. */ if (unlikely (!pte_present (pte))) {
@@ -361,8 +372,7 @@ copy_one_pte(struct mm_struct *dst_mm, sspin_unlock (&mmlist_lock); } } -set_pte_at (dst_mm, addr, dst_pte, pte); -return ; + goto out_set_pte; } pfn =pte_pfn (pte);
@@ -375,10 +385,8 @@ copy_one_pte(struct mm_struct *dst_mm, sif (pfn_valid (pfn)) page =pfn_to_page (pfn); -if (!page ||PageReserved (page)) { -set_pte_at (dst_mm, addr, dst_pte, pte); -return ; - } +if (!page ||PageReserved (page)) + goto out_set_pte;/* * If it's a COW mapping, write protect it both
@@ -397,12 +405,12 @@ copy_one_pte(struct mm_struct *dst_mm, s pte =pte_mkclean (pte); pte =pte_mkold (pte);get_page (page); -if (PageAnon (page)) -inc_mm_counter (dst_mm, anon_rss); -else -inc_mm_counter (dst_mm, file_rss); -set_pte_at (dst_mm, addr, dst_pte, pte);page_dup_rmap (page); + anon = !!PageAnon (page); + +out_set_pte: +set_pte_at (dst_mm, addr, dst_pte, pte); +return anon; } static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
@@ -412,8 +420,10 @@ static int copy_pte_range(struct mm_strupte_t *src_pte, *dst_pte;unsigned long vm_flags = vma->vm_flags;int progress =0 ; +int rss[NO_RSS +1 ], anon; again: + rss[1 ] = rss[0 ] =0 ; dst_pte =pte_alloc_map (dst_mm, dst_pmd, addr);if (!dst_pte)return -ENOMEM ;
@@ -436,13 +446,16 @@ again: progress++;continue ; } -copy_one_pte (dst_mm, src_mm, dst_pte, src_pte, vm_flags, addr); + anon =copy_one_pte (dst_mm, src_mm, dst_pte, src_pte, + vm_flags, addr); + rss[anon]++; progress +=8 ; }while (dst_pte++, src_pte++, addr +=PAGE_SIZE , addr != end);spin_unlock (&src_mm->page_table_lock);pte_unmap_nested (src_pte -1 );pte_unmap (dst_pte -1 ); +add_mm_rss (dst_mm, rss[0 ], rss[1 ]);cond_resched_lock (&dst_mm->page_table_lock);if (addr != end) goto again;
@@ -533,6 +546,8 @@ static void zap_pte_range(struct mmu_gat struct zap_details *details) {pte_t *pte; +int file_rss =0 ; +int anon_rss =0 ; pte =pte_offset_map (pmd, addr); do {
@@ -576,13 +591,13 @@ static void zap_pte_range(struct mmu_gatset_pte_at (tlb->mm, addr, pte,pgoff_to_pte (page->index));if (PageAnon (page)) -dec_mm_counter (tlb->mm, anon_rss); + anon_rss++;else {if (pte_dirty (ptent))set_page_dirty (page);if (pte_young (ptent))mark_page_accessed (page); -dec_mm_counter (tlb->mm, file_rss); + file_rss++; }page_remove_rmap (page);tlb_remove_page (tlb, page);
@@ -598,6 +613,8 @@ static void zap_pte_range(struct mmu_gatfree_swap_and_cache (pte_to_swp_entry (ptent));pte_clear_full (tlb->mm, addr, pte, tlb->fullmm); }while (pte++, addr +=PAGE_SIZE , addr != end); + +add_mm_rss (tlb->mm, -file_rss, -anon_rss);pte_unmap (pte -1 ); }
Re: [PATCH 17/21] mm: batch updating mm_counters
Quoted 5 lines · Hugh Dickins
tlb_finish_mmuused to batch zap_pte_range's update of mm rss, which may be worthwhile if the mm is contended, and would reduce atomic operations if the counts were atomic. Letzap_pte_rangenow batch its updates tofile_rssandanon_rss, per page-table in case we drop the lock outside; andcopy_pte_rangebatch them too.
Good idea.
mm/memory.c again: · line 451
Quoted 7 lines
mm/memory.c +3
-1progress++;continue ; } -copy_one_pte (dst_mm, src_mm, dst_pte, src_pte, vm_flags, addr); + anon =copy_one_pte (dst_mm, src_mm, dst_pte, src_pte, + vm_flags, addr); + rss[anon]++;
How about passing rss[2] to copy_one_pte, and have that increment the correct rss value accordingly? Not that you may consider that any nicer than what you have here.
Nick
Signature · 5 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Re: [PATCH 17/21] mm: batch updating mm_counters
Quoted 9 lines · Nick Piggin
mm/memory.c again: · line 451
Quoted 4 lines · Hugh Dickins
mm/memory.c +3
-1-copy_one_pte (dst_mm, src_mm, dst_pte, src_pte, vm_flags, addr); + anon =copy_one_pte (dst_mm, src_mm, dst_pte, src_pte, + vm_flags, addr); + rss[anon]++;How about passing rss[2] to
copy_one_pte, and have that increment the correct rss value accordingly? Not that you may consider that any nicer than what you have here.
That does seem a more _normal_ way of doing it.
Though adding a seventh argument doesn't appeal (perhaps irrelevant since copy_one_pte is inlined).
I don't mind much either way: anyone have strong feelings?
Hugh
PATCH 18/21 mm: dup_mmap use oldmm more
In reply to Hugh DickinsUse the parent's oldmm throughout dup_mmap, instead of perversely going back to current->mm. (Can you hear the sigh of relief from those mpnts? Usually I squash them, but not today.)
---
kernel/fork.c | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-)
kernel/fork.c +6 -6
@@ -182,16 +182,16 @@ static struct task_struct *dup_task_stru }#ifdef CONFIG_MMU -static inline int dup_mmap (struct mm_struct * mm,struct mm_struct * oldmm) +static inline int dup_mmap (struct mm_struct *mm,struct mm_struct *oldmm) { -struct vm_area_struct * mpnt, *tmp, **pprev; +struct vm_area_struct *mpnt, *tmp, **pprev;struct rb_node **rb_link, *rb_parent;int retval;unsigned long charge;struct mempolicy *pol;down_write (&oldmm->mmap_sem); -flush_cache_mm (current->mm); +flush_cache_mm (oldmm); mm->locked_vm =0 ; mm->mmap =NULL ; mm->mmap_cache =NULL ;
@@ -204,7 +204,7 @@ static inline int dup_mmap(struct mm_str rb_parent =NULL ; pprev = &mm->mmap; -for (mpnt = current->mm->mmap ; mpnt ; mpnt = mpnt->vm_next) { +for (mpnt = oldmm->mmap; mpnt; mpnt = mpnt->vm_next) {struct file *file;if (mpnt->vm_flags &VM_DONTCOPY ) {
@@ -265,7 +265,7 @@ static inline int dup_mmap(struct mm_str rb_parent = &tmp->vm_rb; mm->map_count++; - retval =copy_page_range (mm, current->mm, tmp); + retval =copy_page_range (mm, oldmm, tmp);spin_unlock (&mm->page_table_lock);if (tmp->vm_ops && tmp->vm_ops->open)
@@ -277,7 +277,7 @@ static inline int dup_mmap(struct mm_str retval =0 ; out: -flush_tlb_mm (current->mm); +flush_tlb_mm (oldmm);up_write (&oldmm->mmap_sem);return retval; fail_nomem_policy:
PATCH 19/21 mm: dup_mmap down new mmap_sem
In reply to Hugh DickinsOne anomaly remains from when Andrea rationalized the responsibilities of mmap_sem and page_table_lock: in dup_mmap we add vmas to the child holding its page_table_lock, but not the mmap_sem which normally guards the vma list and rbtree. Which could be an issue for unuse_mm: though since it just walks down the list (today with page_table_lock, tomorrow not), it's probably okay. Will need a memory barrier? Oh, keep it simple, Nick and I agreed, no harm in taking child's mmap_sem here.
---
kernel/fork.c | 9 ++++----- 1 files changed, 4 insertions(+), 5 deletions(-)
kernel/fork.c +4 -5
@@ -192,6 +192,8 @@ static inline int dup_mmap(struct mm_strdown_write (&oldmm->mmap_sem);flush_cache_mm (oldmm); +down_write (&mm->mmap_sem); + mm->locked_vm =0 ; mm->mmap =NULL ; mm->mmap_cache =NULL ;
@@ -251,10 +253,7 @@ static inline int dup_mmap(struct mm_str }/* -* Link in the new vma and copy the page table entries: -* link in first so that swapoff can see swap entries. -* Note that, exceptionally, here the vma is inserted -* without holding mm->mmap_sem. +* Link in the new vma and copy the page table entries. */ spin_lock (&mm->page_table_lock); *pprev = tmp;
@@ -275,8 +274,8 @@ static inline int dup_mmap(struct mm_str goto out; } retval =0 ; - out: +up_write (&mm->mmap_sem);flush_tlb_mm (oldmm);up_write (&oldmm->mmap_sem);return retval;
PATCH 20/21 mm: sh64 hugetlbpage.c
In reply to Hugh DickinsThe sh64 hugetlbpage.c seems to be erroneous, left over from a bygone age, clashing with the common hugetlb.c. Replace it by a copy of the sh hugetlbpage.c. Except, delete that mk_pte_huge macro neither uses.
---
arch/sh/mm/hugetlbpage.c | 2 arch/sh64/mm/hugetlbpage.c | 188 ++------------------------------------------- 2 files changed, 12 insertions(+), 178 deletions(-)
arch/sh/mm/hugetlbpage.c +0 -2
@@ -54,8 +54,6 @@ pte_t *huge_pte_offset(struct mm_structreturn pte; } -#define mk_pte_huge (entry) do { pte_val(entry) |= _PAGE_SZHUGE; } while (0) -void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t entry) {
arch/sh64/mm/hugetlbpage.c +12 -176
216 lines not on this page · raw message
Re: [PATCH 20/21] mm: sh64 hugetlbpage.c
Quoted 11 lines · Hugh Dickins
The sh64
hugetlbpage.cseems to be erroneous, left over from a bygone age, clashing with the commonhugetlb.c. Replace it by a copy of the shhugetlbpage.c. Except, delete thatmk_pte_hugemacro neither uses.Signed-off-by Hugh Dickins <[email protected]>---
arch/sh/mm/hugetlbpage.c | 2 arch/sh64/mm/hugetlbpage.c | 188 ++------------------------------------------- 2 files changed, 12 insertions(+), 178 deletions(-)
Looks good, thanks Hugh.
PATCH 21/21 mm: m68k kill stram swap
In reply to Hugh DickinsPlease, please now delete the Atari CONFIG_STRAM_SWAP code. It may be excellent and ingenious code, but its reference to swap_vfsmnt betrays that it hasn't been built since 2.5.1 (four years old come December), it's delving deep into matters which are the preserve of core mm code, its only purpose is to give the more conscientious mm guys an anxiety attack from time to time; yet we keep on breaking it more and more.
If you want to use RAM for swap, then if the MTD driver does not already provide just what you need, I'm sure David could be persuaded to add the extra. But you'd also like to be able to allocate extents of that swap for other use: we can give you a core interface for that if you need. But unbuilt for four years suggests to me that there's no need at all.
I cannot swear the patch below won't break your build, but believe so.
---
Documentation/kernel-parameters.txt | 2 Documentation/m68k/kernel-options.txt | 24 arch/m68k/Kconfig | 24 arch/m68k/atari/stram.c | 918 ---------------------------------- 4 files changed, 17 insertions(+), 951 deletions(-)
Documentation/kernel-parameters.txt +0 -2
@@ -1443,8 +1443,6 @@ running once the system is up. stifb= [HW] Format: bpp:<bpp1>[:<bpp2>[:<bpp3>...]] - stram_swap= [HW,M68k] - swiotlb= [IA-64] Number of I/O TLB slabs switches= [HW,M68k]
Documentation/m68k/kernel-options.txt +1 -23
@@ -626,7 +626,7 @@ ignored (others aren't affected). can be performed in optimal order. Not all SCSI devices support tagged queuing (:-(). -4.6 switches= +4.5 switches= ------------- Syntax: switches=<list of switches>
@@ -661,28 +661,6 @@ correctly. earlier initialization ("ov_"-less) takes precedence. But the switching-off on reset still happens in this case. -4.5) stram_swap= ----------------- - -Syntax: stram_swap=<do_swap>[,<max_swap>] - - This option is available only if the kernel has been compiled with -CONFIG_STRAM_SWAP enabled. Normally, the kernel then determines -dynamically whether to actually use ST-RAM as swap space. (Currently, -the fraction of ST-RAM must be less or equal 1/3 of total memory to -enable this swapping.) You can override the kernel's decision by -specifying this option. 1 for <do_swap> means always enable the swap, -even if you have less alternate RAM. 0 stands for never swap to -ST-RAM, even if it's small enough compared to the rest of memory. - - If ST-RAM swapping is enabled, the kernel usually uses all free -ST-RAM as swap "device". If the kernel resides in ST-RAM, the region -allocated by it is obviously never used for swapping :-) You can also -limit this amount by specifying the second parameter, <max_swap>, if -you want to use parts of ST-RAM as normal system memory. <max_swap> is -in kBytes and the number should be a multiple of 4 (otherwise: rounded -down). - 5) Options for Amiga Only: ==========================
arch/m68k/Kconfig +1 -23
@@ -388,33 +388,11 @@ config AMIGA_PCMCIA Include support in the kernel for pcmcia on Amiga 1200 and Amiga 600. If you intend to use pcmcia cards say Y; otherwise say N. -config STRAM_SWAP - bool "Support for ST-RAM as swap space" - depends on ATARI && BROKEN - ---help--- - Some Atari 68k machines (including the 520STF and 1020STE) divide - their addressable memory into ST and TT sections. The TT section - (up to 512MB) is the main memory; the ST section (up to 4MB) is - accessible to the built-in graphics board, runs slower, and is - present mainly for backward compatibility with older machines. - - This enables support for using (parts of) ST-RAM as swap space, - instead of as normal system memory. This can first enhance system - performance if you have lots of alternate RAM (compared to the size - of ST-RAM), because executable code always will reside in faster - memory. ST-RAM will remain as ultra-fast swap space. On the other - hand, it allows much improved dynamic allocations of ST-RAM buffers - for device driver modules (e.g. floppy, ACSI, SLM printer, DMA - sound). The probability that such allocations at module load time - fail is drastically reduced. - config STRAM_PROC bool "ST-RAM statistics in /proc" depends on ATARI help - Say Y here to report ST-RAM usage statistics in /proc/stram. See - the help for CONFIG_STRAM_SWAP for discussion of ST-RAM and its - uses. + Say Y here to report ST-RAM usage statistics in /proc/stram. config HEARTBEAT bool "Use power LED as a heartbeat" if AMIGA || APOLLO || ATARI || MAC ||Q40
arch/m68k/atari/stram.c +15 -903
1,027 lines not on this page · raw message
Quoted 3 lines · Hugh Dickins
Here comes the preparatory batch for my page fault scalability patches. This batch makes a few fixes - I suggest 01 and 02 should go in 2.6.14 - and a lot of tidyups, clearing some undergrowth for the real patches.
Well. A mind-boogling patchset. Cannot say that I understand all of it but this does a lot of mine sweeping through the troublespots that I also have seen. Great work!