mmthprmap-lock_compound_mapcounts-on-thp-mapcounts.patch removed from -mm tree
The quilt patch titled
Subject: mm,thp,rmap: lock_compound_mapcounts() on THP mapcounts
has been removed from the -mm tree. Its filename was
mmthprmap-lock_compound_mapcounts-on-thp-mapcounts.patch
This patch was dropped because it was merged into the mm-stable branch of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
------------------------------------------------------
From: Hugh Dickins <[email protected]>
Subject: mm,thp,rmap: lock_compound_mapcounts() on THP mapcounts
Date: Wed, 2 Nov 2022 18:53:45 -0700 (PDT)
Fix the races in maintaining compound_mapcount, subpages_mapcount and subpage _mapcount by using PG_locked in the first tail of any compound page for a bit_spin_lock() on such modifications; skipping the usual atomic operations on those fields in this case.
Bring page_remove_file_rmap() and page_remove_anon_compound_rmap() back into page_remove_rmap() itself. Rearrange page_add_anon_rmap() and page_add_file_rmap() and page_remove_rmap() to follow the same "if (compound) {lock} else if (PageCompound) {lock} else {atomic}" pattern (with a PageTransHuge in the compound test, like before, to avoid BUG_ONs and optimize away that block when THP is not configured). Move all the stats updates outside, after the bit_spin_locked section, so that it is sure to be a leaf lock.
Add page_dup_compound_rmap() to manage compound locking versus atomics in sync with the rest. In particular, hugetlb pages are still using the
atomics: to avoid unnecessary interference there, and because they never have subpage mappings; but this exception can easily be changed. Conveniently, page_dup_compound_rmap() turns out to suit an anon THP's __split_huge_pmd_locked() too.
bit_spin_lock() is not popular with PREEMPT_RT folks: but PREEMPT_RT sensibly excludes TRANSPARENT_HUGEPAGE already, so its only exposure is to the non-hugetlb non-THP pte-mapped compound pages (with large folios being currently dependent on TRANSPARENT_HUGEPAGE). There is never any scan of subpages in this case; but we have chosen to use PageCompound tests rather than PageTransCompound tests to gate the use of lock_compound_mapcounts(), so that page_mapped() is correct on all compound pages, whether or not TRANSPARENT_HUGEPAGE is enabled: could that be a problem for PREEMPT_RT, when there is contention on the lock - under heavy concurrent forking for example? If so, then it can be turned into a sleeping lock (like folio_lock()) when PREEMPT_RT.
A simple 100 X munmap(mmap(2GB, MAP_SHARED|MAP_POPULATE, tmpfs), 2GB) took 18 seconds on small pages, and used to take 1 second on huge pages, but now takes 115 milliseconds on huge pages. Mapping by pmds a second time used to take 860ms and now takes 86ms; mapping by pmds after mapping by ptes (when the scan is needed) used to take 870ms and now takes 495ms. Mapping huge pages by ptes is largely unaffected but variable: between 5% faster and 5% slower in what I've recorded. Contention on the lock is likely to behave worse than contention on the atomics behaved.
---
Documentation/mm/transhuge.rst | 14 - include/linux/rmap.h | 14 - mm/huge_memory.c | 3 mm/rmap.c | 337 +++++++++++++++++-------------- 4 files changed, 205 insertions(+), 163 deletions(-)
Documentation/mm/transhuge.rst renamed from Documentation/mm/transhuge.rst~mmthprmap-lock_compound_mapcounts-on-thp-mapcounts +8 -6
@@ -117,13 +117,15 @@ pages: - ->_refcount in tail pages is always zero: get_page_unless_zero() never succeeds on tail pages. - - map/unmap of the pages with PTE entry increment/decrement ->_mapcount - on relevant sub-page of the compound page. + - map/unmap of PMD entry for the whole compound page increment/decrement + ->compound_mapcount, stored in the first tail page of the compound page. - - map/unmap of the whole compound page is accounted for in compound_mapcount - (stored in first tail page). For file huge pages, we also increment - ->_mapcount of all sub-pages in order to have race-free detection of - last unmap of subpages. + - map/unmap of sub-pages with PTE entry increment/decrement ->_mapcount + on relevant sub-page of the compound page, and also increment/decrement + ->subpages_mapcount, stored in first tail page of the compound page. + In order to have race-free accounting of sub-pages mapped, changes to + sub-page ->_mapcount, ->subpages_mapcount and ->compound_mapcount are + are all locked by bit_spin_lock of PG_locked in the first tail ->flags. split_huge_page internally has to distribute the refcounts in the head page to the tail pages before clearing all PG_head/tail bits from the page
include/linux/rmap.h renamed from include/linux/rmap.h~mmthprmap-lock_compound_mapcounts-on-thp-mapcounts +6 -8
@@ -204,16 +204,14 @@ void hugepage_add_anon_rmap(struct pagevoid hugepage_add_new_anon_rmap (struct page *,struct vm_area_struct *,unsigned long address); -static inline void __page_dup_rmap (struct page *page,bool compound) -{ -if (!compound &&PageCompound (page)) -atomic_inc (subpages_mapcount_ptr (compound_head (page))); -atomic_inc (compound ?compound_mapcount_ptr (page) : &page->_mapcount); -} +void page_dup_compound_rmap (struct page *page,bool compound);static inline void page_dup_file_rmap (struct page *page,bool compound) { -__page_dup_rmap (page, compound); +if (PageCompound (page)) +page_dup_compound_rmap (page, compound); +else +atomic_inc (&page->_mapcount); }/**
@@ -262,7 +260,7 @@ static inline int page_try_dup_anon_rmap* the page R/O into both processes. */ dup: -__page_dup_rmap (page, compound); +page_dup_file_rmap (page, compound);return 0 ; }
mm/huge_memory.c renamed from mm/huge_memory.c~mmthprmap-lock_compound_mapcounts-on-thp-mapcounts +1 -2
@@ -2142,7 +2142,6 @@ static void __split_huge_pmd_locked(struVM_BUG_ON_PAGE (!page_count (page), page);page_ref_add (page,HPAGE_PMD_NR -1 ); -atomic_add (HPAGE_PMD_NR ,subpages_mapcount_ptr (page));/* * Without "freeze", we'll simply split the PMD, propagating the
@@ -2222,7 +2221,7 @@ static void __split_huge_pmd_locked(struBUG_ON (!pte_none (*pte));set_pte_at (mm, addr, pte, entry);if (!pmd_migration) -atomic_inc (&page[i]._mapcount); +page_dup_compound_rmap (page + i, false);pte_unmap (pte); }
mm/rmap.c renamed from mm/rmap.c~mmthprmap-lock_compound_mapcounts-on-thp-mapcounts +190 -147
423 lines not on this page · raw message
_
Patches currently in -mm which might be from [email protected] are