mm: COW fixes part 2: reliable GUP pins of anonymous pages
Subsystems (14)
v3 cover mm: COW fixes part 2: reliable GUP pins of anonymous pages
This series is the result of the discussion on the previous approach [2]. More information on the general COW issues can be found there. It is based on latest linus/master (post v5.17, with relevant core-MM changes for v5.18-rc1).
v3 is located at:
https://github.com/davidhildenbrand/linux/tree/cow_fixes_part_2_v3
This series fixes memory corruptions when a GUP pin (FOLL_PIN) was taken on an anonymous page and COW logic fails to detect exclusivity of the page to then replacing the anonymous page by a copy in the page table: The GUP pin lost synchronicity with the pages mapped into the page tables.
This issue, including other related COW issues, has been summarized in [3] under 3):
"
3. Intra Process Memory Corruptions due to Wrong COW (FOLL_PIN)
page_maybe_dma_pinned() is used to check if a page may be pinned for DMA (using FOLL_PIN instead of FOLL_GET). While false positives are tolerable, false negatives are problematic: pages that are pinned for DMA must not be added to the swapcache. If it happens, the (now pinned) page could be faulted back from the swapcache into page tables read-only. Future write-access would detect the pinning and COW the page, losing synchronicity. For the interested reader, this is nicely documented in feb889fb40fa ("mm: don't put pinned pages into the swap cache").
Peter reports [8] that page_maybe_dma_pinned() as used is racy in some cases and can result in a violation of the documented semantics:
giving false negatives because of the race.
There are cases where we call it without properly taking a per-process sequence lock, turning the usage of page_maybe_dma_pinned() racy. While one case (clear_refs SOFTDIRTY tracking, see below) seems to be easy to handle, there is especially one rmap case (shrink_page_list) that's hard to fix: in the rmap world, we're not limited to a single process.
The shrink_page_list() issue is really subtle. If we race with someone pinning a page, we can trigger the same issue as in the FOLL_GET case. See the detail section at the end of this mail on a discussion how bad this can bite us with VFIO or other FOLL_PIN user.
It's harder to reproduce, but I managed to modify the O_DIRECT reproducer to use io_uring fixed buffers [15] instead, which ends up using FOLL_PIN | FOLL_WRITE | FOLL_LONGTERM to pin buffer pages and can similarly trigger a loss of synchronicity and consequently a memory corruption.
Again, the root issue is that a write-fault on a page that has additional references results in a COW and thereby a loss of synchronicity and consequently a memory corruption if two parties believe they are referencing the same page.
"
This series makes GUP pins (R/O and R/W) on anonymous pages fully reliable, especially also taking care of concurrent pinning via GUP-fast, for example, also fully fixing an issue reported regarding NUMA balancing [4] recently. While doing that, it further reduces "unnecessary COWs", especially when we don't fork()/KSM and don't swapout, and fixes the COW security for hugetlb for FOLL_PIN.
In summary, we track via a pageflag (PG_anon_exclusive) whether a mapped anonymous page is exclusive. Exclusive anonymous pages that are mapped R/O can directly be mapped R/W by the COW logic in the write fault handler. Exclusive anonymous pages that want to be shared (fork(), KSM) first have to be marked shared -- which will fail if there are GUP pins on the page. GUP is only allowed to take a pin on anonymous pages that are exclusive. The PT lock is the primary mechanism to synchronize modifications of PG_anon_exclusive. We synchronize against GUP-fast either via the src_mm->write_protect_seq (during fork()) or via clear/invalidate+flush of the relevant page table entry.
Special care has to be taken about swap, migration, and THPs (whereby a PMD-mapping can be converted to a PTE mapping and we have to track information for subpages). Besides these, we let the rmap code handle most magic. For reliable R/O pins of anonymous pages, we need FAULT_FLAG_UNSHARE logic as part of our previous approach [2], however, it's now 100% mapcount free and I further simplified it a bit.
#1 is a fix
#3-#10 are mostly rmap preparations for PG_anon_exclusive handling #11 introduces PG_anon_exclusive
#12 uses PG_anon_exclusive and make R/W pins of anonymous pages
reliable
#13 is a preparation for reliable R/O pins
#14 and #15 is reused/modified GUP-triggered unsharing for R/O GUP pins
make R/O pins of anonymous pages reliable
#16 adds sanity check when (un)pinning anonymous pages
[1] https://lkml.kernel.org/r/[email protected]
[2] https://lkml.kernel.org/r/[email protected]
[3] https://lore.kernel.org/r/[email protected]
[4] https://bugzilla.kernel.org/show_bug.cgi?id=215616
v2 -> v3:
* Note 1: Left the terminology "unshare" in place for now instead of switching to "make anon exclusive".
* Note 2: We might have to tackle undoing effects of arch_unmap_one() on sparc, to free some tag memory immediately instead of when tearing down the vma/mm; looks like this needs more care either way, so I'll ignore it for now.
* Rebased on top of core MM changes for v5.18-rc1 (most conflicts were due to folio and ZONE_DEVICE migration rework). No severe changes were necessary -- mostly folio conversion and code movement.
* Retested on aarch64, ppc64, s390x and x86_64
* "mm/rmap: convert RMAP flags to a proper distinct rmap_t type" -> Missed to convert one instance in restore_exclusive_pte()
* "mm/rmap: pass rmap flags to hugepage_add_anon_rmap()"
-> Use "!!(flags & RMAP_EXCLUSIVE)" to avoid sparse warnings
* "mm/huge_memory: remove outdated VM_WARN_ON_ONCE_PAGE from unmap_page()" -> Added, as we can trigger that now more frequently
* "mm: remember exclusively mapped anonymous pages with PG_anon_exclusive" -> Use subpage in VM_BUG_ON_PAGE() in try_to_migrate_one()
-> Move comment from folio_migrate_mapping() to folio_migrate_flags()
regarding PG_anon_exclusive/PG_mappedtodisk
-> s/int rmap_flags/rmap_t rmap_flags/ in remove_migration_pmd()
* "mm/gup: sanity-check with CONFIG_DEBUG_VM that anonymous pages are exclusive when (un)pinning"
-> Use IS_ENABLED(CONFIG_DEBUG_VM) instead of ifdef
v1 -> v2:
* Tested on aarch64, ppc64, s390x and x86_64
* "mm/page-flags: reuse PG_mappedtodisk as PG_anon_exclusive for PageAnon() pages"
-> Use PG_mappedtodisk instead of PG_slab (thanks Willy!), this simlifies
the patch and necessary handling a lot. Add safety BUG_ON's
-> Move most documentation to the patch description, to be placed in a
proper documentation doc in the future, once everything's in place
* ""mm: remember exclusively mapped anonymous pages with PG_anon_exclusive -> Skip check+clearing in page_try_dup_anon_rmap(), otherwise we might trigger a wrong VM_BUG_ON() for KSM pages in ClearPageAnonExclusive() -> In __split_huge_pmd_locked(), call page_try_share_anon_rmap() only for "anon_exclusive", otherwise we might trigger a wrong VM_BUG_ON() -> In __split_huge_page_tail(), drop any remaining PG_anon_exclusive on tail pages, and document why that is fine
RFC -> v1:
* Rephrased/extended some patch descriptions+comments
* Tested on aarch64, ppc64 and x86_64
* "mm/rmap: convert RMAP flags to a proper distinct rmap_t type" -> Added
* "mm/rmap: drop "compound" parameter from page_add_new_anon_rmap()" -> Added
* "mm: remember exclusively mapped anonymous pages with PG_anon_exclusive" -> Fixed __do_huge_pmd_anonymous_page() to recheck after temporarily dropping the PT lock.
-> Use "reuse" label in __do_huge_pmd_anonymous_page()
-> Slightly simplify logic in hugetlb_cow()
-> In remove_migration_pte(), remove unrelated changes around
page_remove_rmap()
* "mm: support GUP-triggered unsharing of anonymous pages"
-> In handle_pte_fault(), trigger pte_mkdirty() only with
FAULT_FLAG_WRITE
-> In __handle_mm_fault(), extend comment regarding anonymous PUDs
* "mm/gup: trigger FAULT_FLAG_UNSHARE when R/O-pinning a possibly shared anonymous page"
-> Added unsharing logic to gup_hugepte() and gup_huge_pud() -> Changed return logic in __follow_hugetlb_must_fault(), making sure
that "unshare" is always set
* "mm/gup: sanity-check with CONFIG_DEBUG_VM that anonymous pages are exclusive when (un)pinning"
-> Slightly simplified sanity_check_pinned_pages()
David Hildenbrand (16):
mm/rmap: fix missing swap_free() in try_to_unmap() after
arch_unmap_one() failed
mm/hugetlb: take src_mm->write_protect_seq in
copy_hugetlb_page_range()
mm/memory: slightly simplify copy_present_pte()
mm/rmap: split page_dup_rmap() into page_dup_file_rmap() and
page_try_dup_anon_rmap()
mm/rmap: convert RMAP flags to a proper distinct rmap_t type mm/rmap: remove do_page_add_anon_rmap()
mm/rmap: pass rmap flags to hugepage_add_anon_rmap() mm/rmap: drop "compound" parameter from page_add_new_anon_rmap() mm/rmap: use page_move_anon_rmap() when reusing a mapped PageAnon()
page exclusively
mm/huge_memory: remove outdated VM_WARN_ON_ONCE_PAGE from unmap_page() mm/page-flags: reuse PG_mappedtodisk as PG_anon_exclusive for
PageAnon() pages
mm: remember exclusively mapped anonymous pages with PG_anon_exclusive mm/gup: disallow follow_page(FOLL_PIN)
mm: support GUP-triggered unsharing of anonymous pages mm/gup: trigger FAULT_FLAG_UNSHARE when R/O-pinning a possibly shared
anonymous page
mm/gup: sanity-check with CONFIG_DEBUG_VM that anonymous pages are
exclusive when (un)pinning
include/linux/mm.h | 46 +++++++- include/linux/mm_types.h | 8 ++ include/linux/page-flags.h | 39 ++++++- include/linux/rmap.h | 118 +++++++++++++++++-- include/linux/swap.h | 15 ++- include/linux/swapops.h | 25 ++++ kernel/events/uprobes.c | 2 +- mm/gup.c | 106 ++++++++++++++++- mm/huge_memory.c | 127 +++++++++++++++----- mm/hugetlb.c | 135 ++++++++++++++------- mm/khugepaged.c | 2 +- mm/ksm.c | 15 ++- mm/memory.c | 234 +++++++++++++++++++++++-------------- mm/memremap.c | 9 ++ mm/migrate.c | 18 ++- mm/migrate_device.c | 23 +++- mm/mprotect.c | 8 +- mm/rmap.c | 97 +++++++++++---- mm/swapfile.c | 8 +- mm/userfaultfd.c | 2 +- tools/vm/page-types.c | 8 +- 21 files changed, 825 insertions(+), 220 deletions(-)
base-commit: 1930a6e739c4b4a654a69164dbe39e554d228915
Signature · 1 line
2.35.1
PATCH v3 1/16 mm/rmap: fix missing swap_free() in try_to_unmap() after arch_unmap_one() failed
In case arch_unmap_one() fails, we already did a swap_duplicate(). let's undo that properly via swap_free().
--- mm/rmap.c | 1 + 1 file changed, 1 insertion(+)
mm/rmap.c +1 -0
@@ -1637,6 +1637,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,break ; }if (arch_unmap_one (mm, vma, address, pteval) <0 ) { +swap_free (entry);set_pte_at (mm, address, pvmw.pte, pteval); ret = false;page_vma_mapped_walk_done (&pvmw);
Signature · 1 line
2.35.1
Re: [PATCH v3 01/16] mm/rmap: fix missing swap_free() in try_to_unmap() after arch_unmap_one() failed
Quoted 6 lines · David Hildenbrand
In case
arch_unmap_one()fails, we already did aswap_duplicate(). let's undo that properly viaswap_free().Fixes ca827d55ebaa ("mm, swap: Add infrastructure for saving page metadata on swap")Reviewed-by Khalid Aziz <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Quoted 16 lines
--- mm/rmap.c | 1 + 1 file changed, 1 insertion(+)mm/rmap.c +1
-0@@ -1637,6 +1637,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,break ; }if (arch_unmap_one (mm, vma, address, pteval) <0 ) { +swap_free (entry);set_pte_at (mm, address, pvmw.pte, pteval); ret = false;page_vma_mapped_walk_done (&pvmw);
PATCH v3 2/16 mm/hugetlb: take src_mm->write_protect_seq in copy_hugetlb_page_range()
In reply to David HildenbrandLet's do it just like copy_page_range(), taking the seqlock and making sure the mmap_lock is held in write mode.
This allows for add a VM_BUG_ON to page_needs_cow_for_dma() and properly synchronizes cocnurrent fork() with GUP-fast of hugetlb pages, which will be relevant for further changes.
--- include/linux/mm.h | 4 ++++ mm/hugetlb.c | 8 ++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)
include/linux/mm.h +4 -0
@@ -1576,6 +1576,8 @@ static inline bool page_maybe_dma_pinned(struct page *page)/* * This should most likely only be called during fork() to see whether we * should break the cow immediately for a page on the src mm. +* +* The caller has to hold the PT lock and the vma->vm_mm->->write_protect_seq. */ static inline bool page_needs_cow_for_dma (struct vm_area_struct *vma,struct page *page)
@@ -1583,6 +1585,8 @@ static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma,if (!is_cow_mapping (vma->vm_flags))return false; +VM_BUG_ON (!(raw_read_seqcount (&vma->vm_mm->write_protect_seq) &1 )); +if (!test_bit (MMF_HAS_PINNED , &vma->vm_mm->flags))return false;
mm/hugetlb.c +6 -2
@@ -4714,6 +4714,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, vma->vm_start, vma->vm_end);mmu_notifier_invalidate_range_start (&range); +mmap_assert_write_locked (src); +raw_write_seqcount_begin (&src->write_protect_seq); }else {/* * For shared mappings i_mmap_rwsem must be held to call
@@ -4846,10 +4848,12 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,spin_unlock (dst_ptl); } -if (cow) +if (cow) { +raw_write_seqcount_end (&src->write_protect_seq);mmu_notifier_invalidate_range_end (&range); -else + }else {i_mmap_unlock_read (mapping); + }return ret; }
Signature · 1 line
2.35.1
Re: [PATCH v3 02/16] mm/hugetlb: take src_mm->write_protect_seq in copy_hugetlb_page_range()
Quoted 5 lines · David Hildenbrand
Let's do it just like
copy_page_range(), taking the seqlock and making sure themmap_lockis held in write mode.This allows for add a
VM_BUG_ONtopage_needs_cow_for_dma()and properly synchronizes cocnurrentfork()with GUP-fast of hugetlb pages,
concurrent
Quoted 3 lines
which will be relevant for further changes.
Signed-off-by David Hildenbrand <[email protected]>
PATCH v3 3/16 mm/memory: slightly simplify copy_present_pte()
In reply to David HildenbrandLet's move the pinning check into the caller, to simplify return code logic and prepare for further changes: relocating the page_needs_cow_for_dma() into rmap handling code.
While at it, remove the unused pte parameter and simplify the comments a bit.
No functional change intended.
--- mm/memory.c | 53 ++++++++++++++++------------------------------------- 1 file changed, 16 insertions(+), 37 deletions(-)
mm/memory.c +16 -37
@@ -862,19 +862,11 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, }/* -* Copy a present and normal page if necessary. +* Copy a present and normal page. * -* NOTE! The usual case is that this doesn't need to do -* anything, and can just return a positive value. That -* will let the caller know that it can just increase -* the page refcount and re-use the pte the traditional -* way. -* -* But _if_ we need to copy it because it needs to be -* pinned in the parent (and the child should get its own -* copy rather than just a reference to the same page), -* we'll do that here and return zero to let the caller -* know we're done. +* NOTE! The usual case is that this isn't required; +* instead, the caller can just increase the page refcount +* and re-use the pte the traditional way. * * And if we need a pre-allocated page but don't yet have * one, return a negative error to let the preallocation
@@ -884,25 +876,10 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,static inline int copy_present_page (struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma,pte_t *dst_pte,pte_t *src_pte,unsigned long addr,int *rss, -struct page **prealloc,pte_t pte,struct page *page) +struct page **prealloc,struct page *page) {struct page *new_page; - -/* -* What we want to do is to check whether this page may -* have been pinned by the parent process. If so, -* instead of wrprotect the pte on both sides, we copy -* the page immediately so that we'll always guarantee -* the pinned page won't be randomly replaced in the -* future. -* -* The page pinning checks are just "has this mm ever -* seen pinning", along with the (inexact) check of -* the page count. That might give false positives for -* for pinning, but it will work correctly. -*/ -if (likely (!page_needs_cow_for_dma (src_vma, page))) -return 1 ; +pte_t pte; new_page = *prealloc;if (!new_page)
@@ -944,14 +921,16 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,struct page *page; page =vm_normal_page (src_vma, addr, pte); -if (page) { -int retval; - - retval =copy_present_page (dst_vma, src_vma, dst_pte, src_pte, - addr, rss, prealloc, pte, page); -if (retval <=0 ) -return retval; - +if (page &&unlikely (page_needs_cow_for_dma (src_vma, page))) { +/* +* If this page may have been pinned by the parent process, +* copy the page immediately for the child so that we'll always +* guarantee the pinned page won't be randomly replaced in the +* future. +*/ +return copy_present_page (dst_vma, src_vma, dst_pte, src_pte, + addr, rss, prealloc, page); + }else if (page) {get_page (page);page_dup_rmap (page, false); rss[mm_counter (page)]++;
Signature · 1 line
2.35.1
Re: [PATCH v3 03/16] mm/memory: slightly simplify copy_present_pte()
Quoted 10 lines · David Hildenbrand
Let's move the pinning check into the caller, to simplify return code logic and prepare for further changes: relocating the
page_needs_cow_for_dma()into rmap handling code.While at it, remove the unused pte parameter and simplify the comments a bit.
No functional change intended.
Signed-off-by David Hildenbrand <[email protected]>
Yeah, much better.
PATCH v3 4/16 mm/rmap: split page_dup_rmap() into page_dup_file_rmap() and page_try_dup_anon_rmap()
In reply to David Hildenbrand... and move the special check for pinned pages into page_try_dup_anon_rmap() to prepare for tracking exclusive anonymous pages via a new pageflag, clearing it only after making sure that there are no GUP pins on the anonymous page.
We really only care about pins on anonymous pages, because they are prone to getting replaced in the COW handler once mapped R/O. For !anon pages in cow-mappings (!VM_SHARED && VM_MAYWRITE) we shouldn't really care about that, at least not that I could come up with an example.
Let's drop the is_cow_mapping() check from page_needs_cow_for_dma(), as we know we're dealing with anonymous pages. Also, drop the handling of pinned pages from copy_huge_pud() and add a comment if ever supporting anonymous pages on the PUD level.
This is a preparation for tracking exclusivity of anonymous pages in the rmap code, and disallowing marking a page shared (-> failing to duplicate) if there are GUP pins on a page.
--- include/linux/mm.h | 5 +---- include/linux/rmap.h | 49 +++++++++++++++++++++++++++++++++++++++++++- mm/huge_memory.c | 27 ++++++++---------------- mm/hugetlb.c | 16 ++++++++------- mm/memory.c | 17 ++++++++++----- mm/migrate.c | 2 +- 6 files changed, 79 insertions(+), 37 deletions(-)
include/linux/mm.h +1 -4
@@ -1575,16 +1575,13 @@ static inline bool page_maybe_dma_pinned(struct page *page)/* * This should most likely only be called during fork() to see whether we -* should break the cow immediately for a page on the src mm. +* should break the cow immediately for an anon page on the src mm. * * The caller has to hold the PT lock and the vma->vm_mm->->write_protect_seq. */ static inline bool page_needs_cow_for_dma (struct vm_area_struct *vma,struct page *page) { -if (!is_cow_mapping (vma->vm_flags)) -return false; -VM_BUG_ON (!(raw_read_seqcount (&vma->vm_mm->write_protect_seq) &1 ));if (!test_bit (MMF_HAS_PINNED , &vma->vm_mm->flags))
include/linux/rmap.h +48 -1
65 lines not on this page · raw message
mm/huge_memory.c +8 -19
@@ -1097,23 +1097,16 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, src_page =pmd_page (pmd);VM_BUG_ON_PAGE (!PageHead (src_page), src_page); -/* -* If this page is a potentially pinned page, split and retry the fault -* with smaller page size. Normally this should not happen because the -* userspace should use MADV_DONTFORK upon pinned regions. This is a -* best effort that the pinned pages won't be replaced by another -* random page during the coming copy-on-write. -*/ -if (unlikely (page_needs_cow_for_dma (src_vma, src_page))) { +get_page (src_page); +if (unlikely (page_try_dup_anon_rmap (src_page, true, src_vma))) { +/* Page maybe pinned: split and retry the fault on PTEs. */ +put_page (src_page);pte_free (dst_mm, pgtable);spin_unlock (src_ptl);spin_unlock (dst_ptl);__split_huge_pmd (src_vma, src_pmd, addr, false,NULL );return -EAGAIN ; } - -get_page (src_page); -page_dup_rmap (src_page, true);add_mm_counter (dst_mm,MM_ANONPAGES ,HPAGE_PMD_NR ); out_zero_page:mm_inc_nr_ptes (dst_mm);
@@ -1217,14 +1210,10 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm,/* No huge zero pud yet */ } -/* Please refer to comments in copy_huge_pmd() */ -if (unlikely (page_needs_cow_for_dma (vma,pud_page (pud)))) { -spin_unlock (src_ptl); -spin_unlock (dst_ptl); -__split_huge_pud (vma, src_pud, addr); -return -EAGAIN ; - } - +/* +* TODO: once we support anonymous pages, use page_try_dup_anon_rmap() +* and split if duplicating fails. +*/ pudp_set_wrprotect (src_mm, addr, src_pud); pud =pud_mkold (pud_wrprotect (pud));set_pud_at (dst_mm, addr, dst_pud, pud);
mm/hugetlb.c +9 -7
@@ -4785,15 +4785,18 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,get_page (ptepage);/* -* This is a rare case where we see pinned hugetlb -* pages while they're prone to COW. We need to do the -* COW earlier during fork. +* Failing to duplicate the anon rmap is a rare case +* where we see pinned hugetlb pages while they're +* prone to COW. We need to do the COW earlier during +* fork. * * When pre-allocating the page or copying data, we * need to be without the pgtable locks since we could * sleep during the process. */ -if (unlikely (page_needs_cow_for_dma (vma, ptepage))) { +if (!PageAnon (ptepage)) { +page_dup_file_rmap (ptepage, true); + }else if (page_try_dup_anon_rmap (ptepage, true, vma)) {pte_t src_pte_old = entry;struct page *new;
@@ -4840,7 +4843,6 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, entry =huge_pte_wrprotect (entry); } -page_dup_rmap (ptepage, true);set_huge_pte_at (dst, addr, dst_pte, entry);hugetlb_count_add (npages, dst); }
@@ -5520,7 +5522,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,ClearHPageRestoreReserve (page);hugepage_add_new_anon_rmap (page, vma, haddr); }else -page_dup_rmap (page, true); +page_dup_file_rmap (page, true); new_pte =make_huge_pte (vma, page, ((vma->vm_flags &VM_WRITE ) && (vma->vm_flags &VM_SHARED )));set_huge_pte_at (mm, haddr, ptep, new_pte);
@@ -5881,7 +5883,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, goto out_release_unlock;if (vm_shared) { -page_dup_rmap (page, true); +page_dup_file_rmap (page, true); }else {ClearHPageRestoreReserve (page);hugepage_add_new_anon_rmap (page, dst_vma, dst_addr);
mm/memory.c +12 -5
@@ -825,7 +825,8 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,*/ get_page (page); rss[mm_counter (page)]++; -page_dup_rmap (page, false); +/* Cannot fail as these pages cannot get pinned. */ +BUG_ON (page_try_dup_anon_rmap (page, false, src_vma));/* * We do not preserve soft-dirty information, because so
@@ -921,18 +922,24 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,struct page *page; page =vm_normal_page (src_vma, addr, pte); -if (page &&unlikely (page_needs_cow_for_dma (src_vma, page))) { +if (page &&PageAnon (page)) {/* * If this page may have been pinned by the parent process, * copy the page immediately for the child so that we'll always * guarantee the pinned page won't be randomly replaced in the * future. */ -return copy_present_page (dst_vma, src_vma, dst_pte, src_pte, - addr, rss, prealloc, page); +get_page (page); +if (unlikely (page_try_dup_anon_rmap (page, false, src_vma))) { +/* Page maybe pinned, we have to copy. */ +put_page (page); +return copy_present_page (dst_vma, src_vma, dst_pte, src_pte, + addr, rss, prealloc, page); + } + rss[mm_counter (page)]++; }else if (page) {get_page (page); -page_dup_rmap (page, false); +page_dup_file_rmap (page, false); rss[mm_counter (page)]++; }
mm/migrate.c +1 -1
@@ -234,7 +234,7 @@ static bool remove_migration_pte(struct folio *folio,if (folio_test_anon (folio))hugepage_add_anon_rmap (new, vma, pvmw.address);else -page_dup_rmap (new, true); +page_dup_file_rmap (new, true);set_huge_pte_at (vma->vm_mm, pvmw.address, pvmw.pte, pte); } else#endif
Signature · 1 line
2.35.1
Re: [PATCH v3 04/16] mm/rmap: split page_dup_rmap() into page_dup_file_rmap() and page_try_dup_anon_rmap()
Quoted 15 lines · David Hildenbrand
... and move the special check for pinned pages into
page_try_dup_anon_rmap()to prepare for tracking exclusive anonymous pages via a new pageflag, clearing it only after making sure that there are no GUP pins on the anonymous page.We really only care about pins on anonymous pages, because they are prone to getting replaced in the COW handler once mapped R/O. For !anon pages in cow-mappings (!VM_SHARED &&
VM_MAYWRITE) we shouldn't really care about that, at least not that I could come up with an example.Let's drop the
is_cow_mapping()check frompage_needs_cow_for_dma(), as we know we're dealing with anonymous pages. Also, drop the handling of pinned pages fromcopy_huge_pud()and add a comment if ever supporting anonymous pages on the PUD level.
Quoted 5 lines
This is a preparation for tracking exclusivity of anonymous pages in the rmap code, and disallowing marking a page shared (-> failing to duplicate) if there are GUP pins on a page.
Signed-off-by David Hildenbrand <[email protected]>
Nit:
mm/memory.c copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, · line 829
Quoted 9 lines
mm/memory.c +2
-1@@ -825,7 +825,8 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,*/ get_page (page); rss[mm_counter (page)]++; -page_dup_rmap (page, false); +/* Cannot fail as these pages cannot get pinned. */ +BUG_ON (page_try_dup_anon_rmap (page, false, src_vma));
Should we just call __page_dup_rmap() here? This is block for the condition is_device_private_entry(), and page_try_dup_anon_rmap() can't return -EBUSY for is_device_private_page().
Quoted 45 lines
mm/memory.c +10
-4/* * We do not preserve soft-dirty information, because so @@ -921,18 +922,24 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,struct page *page; page =vm_normal_page (src_vma, addr, pte); -if (page &&unlikely (page_needs_cow_for_dma (src_vma, page))) { +if (page &&PageAnon (page)) {/* * If this page may have been pinned by the parent process, * copy the page immediately for the child so that we'll always * guarantee the pinned page won't be randomly replaced in the * future. */ -return copy_present_page (dst_vma, src_vma, dst_pte, src_pte, - addr, rss, prealloc, page); +get_page (page); +if (unlikely (page_try_dup_anon_rmap (page, false, src_vma))) { +/* Page maybe pinned, we have to copy. */ +put_page (page); +return copy_present_page (dst_vma, src_vma, dst_pte, src_pte, + addr, rss, prealloc, page); + } + rss[mm_counter (page)]++; }else if (page) {get_page (page); -page_dup_rmap (page, false); +page_dup_file_rmap (page, false); rss[mm_counter (page)]++; }mm/migrate.c +1
-1@@ -234,7 +234,7 @@ static bool remove_migration_pte(struct folio *folio,if (folio_test_anon (folio))hugepage_add_anon_rmap (new, vma, pvmw.address);else -page_dup_rmap (new, true); +page_dup_file_rmap (new, true);set_huge_pte_at (vma->vm_mm, pvmw.address, pvmw.pte, pte); } else#endif
Re: [PATCH v3 04/16] mm/rmap: split page_dup_rmap() into page_dup_file_rmap() and page_try_dup_anon_rmap()
Quoted 26 lines · Vlastimil Babka
Quoted 15 lines · David Hildenbrand
... and move the special check for pinned pages into
page_try_dup_anon_rmap()to prepare for tracking exclusive anonymous pages via a new pageflag, clearing it only after making sure that there are no GUP pins on the anonymous page.We really only care about pins on anonymous pages, because they are prone to getting replaced in the COW handler once mapped R/O. For !anon pages in cow-mappings (!VM_SHARED &&
VM_MAYWRITE) we shouldn't really care about that, at least not that I could come up with an example.Let's drop the
is_cow_mapping()check frompage_needs_cow_for_dma(), as we know we're dealing with anonymous pages. Also, drop the handling of pinned pages fromcopy_huge_pud()and add a comment if ever supporting anonymous pages on the PUD level.Quoted 5 lines
This is a preparation for tracking exclusivity of anonymous pages in the rmap code, and disallowing marking a page shared (-> failing to duplicate) if there are GUP pins on a page.
Signed-off-by David Hildenbrand <[email protected]>Acked-by Vlastimil Babka <[email protected]>Nit:
Quoted 13 lines
mm/memory.c copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, · line 829
Quoted 9 lines
mm/memory.c +2
-1@@ -825,7 +825,8 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,*/ get_page (page); rss[mm_counter (page)]++; -page_dup_rmap (page, false); +/* Cannot fail as these pages cannot get pinned. */ +BUG_ON (page_try_dup_anon_rmap (page, false, src_vma));Should we just call
__page_dup_rmap()here? This is block for the conditionis_device_private_entry(), andpage_try_dup_anon_rmap()can't return-EBUSYforis_device_private_page().
Hi Vlastimil,
thanks for your review!
We want to keep page_try_dup_anon_rmap() here, because we extend page_try_dup_anon_rmap() in patch #12 to properly clear PageAnonExclusive() of there are no GUP pins. Just like with current page_try_dup_anon_rmap(), that can't fail for device private pages.
Signature · 3 lines
Thanks, David / dhildenb
PATCH v3 5/16 mm/rmap: convert RMAP flags to a proper distinct rmap_t type
In reply to David HildenbrandWe want to pass the flags to more than one anon rmap function, getting rid of special "do_page_add_anon_rmap()". So let's pass around a distinct __bitwise type and refine documentation.
--- include/linux/rmap.h | 22 ++++++++++++++++++---- mm/memory.c | 6 +++--- mm/rmap.c | 7 ++++--- 3 files changed, 25 insertions(+), 10 deletions(-)
include/linux/rmap.h +18 -4
@@ -160,9 +160,23 @@ static inline void anon_vma_merge(struct vm_area_struct *vma,struct anon_vma *page_get_anon_vma (struct page *page); -/* bitflags for do_page_add_anon_rmap() */ -#define RMAP_EXCLUSIVE 0x01 -#define RMAP_COMPOUND 0x02 +/* RMAP flags, currently only relevant for some anon rmap operations. */ +typedef int __bitwise rmap_t ; + +/* +* No special request: if the page is a subpage of a compound page, it is +* mapped via a PTE. The mapped (sub)page is possibly shared between processes. +*/ +#define RMAP_NONE ((__force rmap_t)0) + +/* The (sub)page is exclusive to a single process. */ +#define RMAP_EXCLUSIVE ((__force rmap_t)BIT(0)) + +/* +* The compound page is not mapped via PTEs, but instead via a single PMD and +* should be accounted accordingly. +*/ +#define RMAP_COMPOUND ((__force rmap_t)BIT(1))/* * rmap interfaces called when adding or removing pte of page
@@ -171,7 +185,7 @@ void page_move_anon_rmap(struct page *, struct vm_area_struct *);void page_add_anon_rmap (struct page *,struct vm_area_struct *,unsigned long address,bool compound);void do_page_add_anon_rmap (struct page *,struct vm_area_struct *, -unsigned long address,int flags); +unsigned long address,rmap_t flags);void page_add_new_anon_rmap (struct page *,struct vm_area_struct *,unsigned long address,bool compound); void page_add_file_rmap(struct page *, struct vm_area_struct *,
mm/memory.c +3 -3
@@ -3507,10 +3507,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)struct vm_area_struct *vma = vmf->vma;struct page *page =NULL , *swapcache;struct swap_info_struct *si =NULL ; +rmap_t rmap_flags =RMAP_NONE ;swp_entry_t entry;pte_t pte;int locked; -int exclusive =0 ;vm_fault_t ret =0 ;void *shadow =NULL ;
@@ -3685,7 +3685,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte =maybe_mkwrite (pte_mkdirty (pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE ; ret |=VM_FAULT_WRITE ; - exclusive =RMAP_EXCLUSIVE ; + rmap_flags |=RMAP_EXCLUSIVE ; }flush_icache_page (vma, page);if (pte_swp_soft_dirty (vmf->orig_pte))
@@ -3701,7 +3701,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)page_add_new_anon_rmap (page, vma, vmf->address, false);lru_cache_add_inactive_or_unevictable (page, vma); }else { -do_page_add_anon_rmap (page, vma, vmf->address, exclusive); +do_page_add_anon_rmap (page, vma, vmf->address, rmap_flags); }set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte);
mm/rmap.c +4 -3
@@ -1137,7 +1137,8 @@ static void __page_check_anon_rmap(struct page *page,void page_add_anon_rmap (struct page *page,struct vm_area_struct *vma,unsigned long address,bool compound) { -do_page_add_anon_rmap (page, vma, address, compound ?RMAP_COMPOUND :0 ); +do_page_add_anon_rmap (page, vma, address, + compound ?RMAP_COMPOUND :RMAP_NONE ); }/*
@@ -1146,7 +1147,7 @@ void page_add_anon_rmap(struct page *page,* Everybody else should continue to use page_add_anon_rmap above. */ void do_page_add_anon_rmap (struct page *page, -struct vm_area_struct *vma,unsigned long address,int flags) +struct vm_area_struct *vma,unsigned long address,rmap_t flags) {bool compound = flags &RMAP_COMPOUND ;bool first;
@@ -1185,7 +1186,7 @@ void do_page_add_anon_rmap(struct page *page,/* address might be in next vma when migration races vma_adjust */ else if (first) __page_set_anon_rmap(page, vma, address, - flags & RMAP_EXCLUSIVE); + !!(flags & RMAP_EXCLUSIVE)); else__page_check_anon_rmap (page ,vma ,address );
Signature · 1 line
2.35.1
Re: [PATCH v3 05/16] mm/rmap: convert RMAP flags to a proper distinct rmap_t type
Quoted 5 lines · David Hildenbrand
We want to pass the flags to more than one anon rmap function, getting rid of special "
do_page_add_anon_rmap()". So let's pass around a distinct__bitwisetype and refine documentation.Signed-off-by David Hildenbrand <[email protected]>
PATCH v3 6/16 mm/rmap: remove do_page_add_anon_rmap()
In reply to David Hildenbrand... and instead convert page_add_anon_rmap() to accept flags.
Passing flags instead of bools is usually nicer either way, and we want to more often also pass RMAP_EXCLUSIVE in follow up patches when detecting that an anonymous page is exclusive: for example, when restoring an anonymous page from a writable migration entry.
This is a preparation for marking an anonymous page inside page_add_anon_rmap() as exclusive when RMAP_EXCLUSIVE is passed.
--- include/linux/rmap.h | 2 -- mm/huge_memory.c | 2 +- mm/ksm.c | 2 +- mm/memory.c | 4 ++-- mm/migrate.c | 3 ++- mm/rmap.c | 14 +------------- mm/swapfile.c | 2 +- 7 files changed, 8 insertions(+), 21 deletions(-)
include/linux/rmap.h +0 -2
@@ -183,8 +183,6 @@ typedef int __bitwise rmap_t;*/ void page_move_anon_rmap (struct page *,struct vm_area_struct *);void page_add_anon_rmap (struct page *,struct vm_area_struct *, -unsigned long address,bool compound); -void do_page_add_anon_rmap (struct page *,struct vm_area_struct *,unsigned long address,rmap_t flags);void page_add_new_anon_rmap (struct page *,struct vm_area_struct *,unsigned long address,bool compound);
mm/huge_memory.c +1 -1
@@ -3073,7 +3073,7 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) pmde =pmd_wrprotect (pmd_mkuffd_wp (pmde));if (PageAnon (new)) -page_add_anon_rmap (new, vma, mmun_start, true); +page_add_anon_rmap (new, vma, mmun_start,RMAP_COMPOUND );else page_add_file_rmap (new, vma, true);set_pmd_at (mm, mmun_start, pvmw->pmd, pmde);
mm/ksm.c +1 -1
@@ -1150,7 +1150,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,*/ if (!is_zero_pfn (page_to_pfn (kpage))) {get_page (kpage); -page_add_anon_rmap (kpage, vma, addr, false); +page_add_anon_rmap (kpage, vma, addr,RMAP_NONE ); newpte =mk_pte (kpage, vma->vm_page_prot); }else { newpte = pte_mkspecial(pfn_pte(page_to_pfn(kpage),
mm/memory.c +2 -2
@@ -725,7 +725,7 @@ static void restore_exclusive_pte(struct vm_area_struct *vma,* created when the swap entry was made. */ if (PageAnon (page)) -page_add_anon_rmap (page, vma, address, false); +page_add_anon_rmap (page, vma, address,RMAP_NONE );else /* * Currently device exclusive access only supports anonymous
@@ -3701,7 +3701,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)page_add_new_anon_rmap (page, vma, vmf->address, false);lru_cache_add_inactive_or_unevictable (page, vma); }else { -do_page_add_anon_rmap (page, vma, vmf->address, rmap_flags); +page_add_anon_rmap (page, vma, vmf->address, rmap_flags); }set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte);
mm/migrate.c +2 -1
@@ -240,7 +240,8 @@ static bool remove_migration_pte(struct folio *folio,#endif {if (folio_test_anon (folio)) -page_add_anon_rmap (new, vma, pvmw.address, false); +page_add_anon_rmap (new, vma, pvmw.address, +RMAP_NONE );else page_add_file_rmap (new, vma, false);set_pte_at (vma->vm_mm, pvmw.address, pvmw.pte, pte);
mm/rmap.c +1 -13
@@ -1127,7 +1127,7 @@ static void __page_check_anon_rmap(struct page *page,* @page: the page to add the mapping to * @vma: the vm area in which the mapping is added * @address: the user virtual address mapped -* @compound: charge the page as compound or small page +* @flags: the rmap flags * * The caller needs to hold the pte lock, and the page must be locked in * the anon_vma case: to serialize mapping,index checking after setting,
@@ -1135,18 +1135,6 @@ static void __page_check_anon_rmap(struct page *page,* (but PageKsm is never downgraded to PageAnon). */ void page_add_anon_rmap (struct page *page, -struct vm_area_struct *vma,unsigned long address,bool compound) -{ -do_page_add_anon_rmap (page, vma, address, - compound ?RMAP_COMPOUND :RMAP_NONE ); -} - -/* -* Special version of the above for do_swap_page, which often runs -* into pages that are exclusively owned by the current process. -* Everybody else should continue to use page_add_anon_rmap above. -*/ -void do_page_add_anon_rmap (struct page *page,struct vm_area_struct *vma,unsigned long address,rmap_t flags) {bool compound = flags &RMAP_COMPOUND ;
mm/swapfile.c +1 -1
@@ -1800,7 +1800,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,inc_mm_counter (vma->vm_mm,MM_ANONPAGES );get_page (page);if (page == swapcache) { -page_add_anon_rmap (page, vma, addr, false); +page_add_anon_rmap (page, vma, addr,RMAP_NONE ); }else {/* ksm created a completely new copy */ page_add_new_anon_rmap (page, vma, addr, false);lru_cache_add_inactive_or_unevictable (page, vma);
Signature · 1 line
2.35.1
Re: [PATCH v3 06/16] mm/rmap: remove do_page_add_anon_rmap()
Quoted 11 lines · David Hildenbrand
... and instead convert
page_add_anon_rmap()to accept flags.Passing flags instead of bools is usually nicer either way, and we want to more often also pass
RMAP_EXCLUSIVEin follow up patches when detecting that an anonymous page is exclusive: for example, when restoring an anonymous page from a writable migration entry.This is a preparation for marking an anonymous page inside
page_add_anon_rmap()as exclusive whenRMAP_EXCLUSIVEis passed.Signed-off-by David Hildenbrand <[email protected]>
PATCH v3 7/16 mm/rmap: pass rmap flags to hugepage_add_anon_rmap()
In reply to David HildenbrandLet's prepare for passing RMAP_EXCLUSIVE, similarly as we do for page_add_anon_rmap() now. RMAP_COMPOUND is implicit for hugetlb pages and ignored.
--- include/linux/rmap.h | 2 +- mm/migrate.c | 3 ++- mm/rmap.c | 9 ++++++--- 3 files changed, 9 insertions(+), 5 deletions(-)
include/linux/rmap.h +1 -1
@@ -191,7 +191,7 @@ void page_add_file_rmap(struct page *, struct vm_area_struct *,void page_remove_rmap (struct page *,struct vm_area_struct *,bool compound);void hugepage_add_anon_rmap (struct page *,struct vm_area_struct *, -unsigned long address); +unsigned long address,rmap_t flags);void hugepage_add_new_anon_rmap (struct page *,struct vm_area_struct *,unsigned long address);
mm/migrate.c +2 -1
@@ -232,7 +232,8 @@ static bool remove_migration_pte(struct folio *folio, pte =pte_mkhuge (pte); pte =arch_make_huge_pte (pte, shift, vma->vm_flags);if (folio_test_anon (folio)) -hugepage_add_anon_rmap (new, vma, pvmw.address); +hugepage_add_anon_rmap (new, vma, pvmw.address, +RMAP_NONE );else page_dup_file_rmap (new, true);set_huge_pte_at (vma->vm_mm, pvmw.address, pvmw.pte, pte);
mm/rmap.c +6 -3
@@ -2347,9 +2347,11 @@ void rmap_walk_locked(struct folio *folio, const struct rmap_walk_control *rwc)* The following two functions are for anonymous (private mapped) hugepages. * Unlike common anonymous pages, anonymous hugepages have no accounting code * and no lru code, because we handle hugepages differently from common pages. +* +* RMAP_COMPOUND is ignored. */ -void hugepage_add_anon_rmap (struct page *page, -struct vm_area_struct *vma,unsigned long address) +void hugepage_add_anon_rmap (struct page *page,struct vm_area_struct *vma, +unsigned long address,rmap_t flags) {struct anon_vma *anon_vma = vma->anon_vma;int first;
@@ -2359,7 +2361,8 @@ void hugepage_add_anon_rmap(struct page *page,/* address might be in next vma when migration races vma_adjust */ first =atomic_inc_and_test (compound_mapcount_ptr (page));if (first) -__page_set_anon_rmap (page, vma, address,0 ); +__page_set_anon_rmap (page, vma, address, + !!(flags &RMAP_EXCLUSIVE )); } void hugepage_add_new_anon_rmap(struct page *page,
Signature · 1 line
2.35.1
Re: [PATCH v3 07/16] mm/rmap: pass rmap flags to hugepage_add_anon_rmap()
Quoted 5 lines · David Hildenbrand
Let's prepare for passing
RMAP_EXCLUSIVE, similarly as we do forpage_add_anon_rmap()now.RMAP_COMPOUNDis implicit for hugetlb pages and ignored.Signed-off-by David Hildenbrand <[email protected]>
Quoted 61 lines
--- include/linux/rmap.h | 2 +- mm/migrate.c | 3 ++- mm/rmap.c | 9 ++++++--- 3 files changed, 9 insertions(+), 5 deletions(-)include/linux/rmap.h +1
-1@@ -191,7 +191,7 @@ void page_add_file_rmap(struct page *, struct vm_area_struct *,void page_remove_rmap (struct page *,struct vm_area_struct *,bool compound);void hugepage_add_anon_rmap (struct page *,struct vm_area_struct *, -unsigned long address); +unsigned long address,rmap_t flags);void hugepage_add_new_anon_rmap (struct page *,struct vm_area_struct *,unsigned long address);mm/migrate.c +2
-1@@ -232,7 +232,8 @@ static bool remove_migration_pte(struct folio *folio, pte =pte_mkhuge (pte); pte =arch_make_huge_pte (pte, shift, vma->vm_flags);if (folio_test_anon (folio)) -hugepage_add_anon_rmap (new, vma, pvmw.address); +hugepage_add_anon_rmap (new, vma, pvmw.address, +RMAP_NONE );else page_dup_file_rmap (new, true);set_huge_pte_at (vma->vm_mm, pvmw.address, pvmw.pte, pte);mm/rmap.c +6
-3@@ -2347,9 +2347,11 @@ void rmap_walk_locked(struct folio *folio, const struct rmap_walk_control *rwc)* The following two functions are for anonymous (private mapped) hugepages. * Unlike common anonymous pages, anonymous hugepages have no accounting code * and no lru code, because we handle hugepages differently from common pages. +* +* RMAP_COMPOUND is ignored. */ -void hugepage_add_anon_rmap (struct page *page, -struct vm_area_struct *vma,unsigned long address) +void hugepage_add_anon_rmap (struct page *page,struct vm_area_struct *vma, +unsigned long address,rmap_t flags) {struct anon_vma *anon_vma = vma->anon_vma;int first;@@ -2359,7 +2361,8 @@ void hugepage_add_anon_rmap(struct page *page,/* address might be in next vma when migration races vma_adjust */ first =atomic_inc_and_test (compound_mapcount_ptr (page));if (first) -__page_set_anon_rmap (page, vma, address,0 ); +__page_set_anon_rmap (page, vma, address, + !!(flags &RMAP_EXCLUSIVE )); } void hugepage_add_new_anon_rmap(struct page *page,
PATCH v3 8/16 mm/rmap: drop "compound" parameter from page_add_new_anon_rmap()
In reply to David HildenbrandNew anonymous pages are always mapped natively: only THP/khugepagd code maps a new compound anonymous page and passes "true". Otherwise, we're just dealing with simple, non-compound pages.
Let's give the interface clearer semantics and document these.
--- include/linux/rmap.h | 3 ++- kernel/events/uprobes.c | 2 +- mm/huge_memory.c | 2 +- mm/khugepaged.c | 2 +- mm/memory.c | 10 +++++----- mm/migrate_device.c | 2 +- mm/rmap.c | 9 ++++++--- mm/swapfile.c | 2 +- mm/userfaultfd.c | 2 +- 9 files changed, 19 insertions(+), 15 deletions(-)
include/linux/rmap.h +2 -1
@@ -185,11 +185,12 @@ void page_move_anon_rmap(struct page *, struct vm_area_struct *);void page_add_anon_rmap (struct page *,struct vm_area_struct *,unsigned long address,rmap_t flags);void page_add_new_anon_rmap (struct page *,struct vm_area_struct *, -unsigned long address,bool compound); +unsigned long address);void page_add_file_rmap (struct page *,struct vm_area_struct *,bool compound);void page_remove_rmap (struct page *,struct vm_area_struct *,bool compound); +void hugepage_add_anon_rmap (struct page *,struct vm_area_struct *,unsigned long address,rmap_t flags); void hugepage_add_new_anon_rmap(struct page *, struct vm_area_struct *,
kernel/events/uprobes.c +1 -1
@@ -180,7 +180,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr,if (new_page) {get_page (new_page); -page_add_new_anon_rmap (new_page, vma, addr, false); +page_add_new_anon_rmap (new_page, vma, addr);lru_cache_add_inactive_or_unevictable (new_page, vma); }else /* no new page, just dec_mm_counter for old_page */
mm/huge_memory.c +1 -1
@@ -647,7 +647,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, entry =mk_huge_pmd (page, vma->vm_page_prot); entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); -page_add_new_anon_rmap (page, vma, haddr, true); +page_add_new_anon_rmap (page, vma, haddr);lru_cache_add_inactive_or_unevictable (page, vma);pgtable_trans_huge_deposit (vma->vm_mm, vmf->pmd, pgtable);set_pmd_at (vma->vm_mm, haddr, vmf->pmd, entry);
mm/khugepaged.c +1 -1
@@ -1183,7 +1183,7 @@ static void collapse_huge_page(struct mm_struct *mm,spin_lock (pmd_ptl);BUG_ON (!pmd_none (*pmd)); -page_add_new_anon_rmap (new_page, vma, address, true); +page_add_new_anon_rmap (new_page, vma, address);lru_cache_add_inactive_or_unevictable (new_page, vma);pgtable_trans_huge_deposit (mm, pmd, pgtable);set_pmd_at (mm, address, pmd, _pmd);
mm/memory.c +5 -5
@@ -893,7 +893,7 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma *prealloc =NULL ;copy_user_highpage (new_page, page, addr, src_vma);__SetPageUptodate (new_page); -page_add_new_anon_rmap (new_page, dst_vma, addr, false); +page_add_new_anon_rmap (new_page, dst_vma, addr);lru_cache_add_inactive_or_unevictable (new_page, dst_vma); rss[mm_counter (new_page)]++;
@@ -3058,7 +3058,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)* some TLBs while the old PTE remains in others. */ ptep_clear_flush_notify (vma, vmf->address, vmf->pte); -page_add_new_anon_rmap (new_page, vma, vmf->address, false); +page_add_new_anon_rmap (new_page, vma, vmf->address);lru_cache_add_inactive_or_unevictable (new_page, vma);/* * We call the notify macro here because, when using secondary
@@ -3698,7 +3698,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)/* ksm created a completely new copy */ if (unlikely (page != swapcache && swapcache)) { -page_add_new_anon_rmap (page, vma, vmf->address, false); +page_add_new_anon_rmap (page, vma, vmf->address);lru_cache_add_inactive_or_unevictable (page, vma); }else {page_add_anon_rmap (page, vma, vmf->address, rmap_flags);
@@ -3848,7 +3848,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) }inc_mm_counter_fast (vma->vm_mm,MM_ANONPAGES ); -page_add_new_anon_rmap (page, vma, vmf->address, false); +page_add_new_anon_rmap (page, vma, vmf->address);lru_cache_add_inactive_or_unevictable (page, vma); setpte:set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry);
@@ -4031,7 +4031,7 @@ void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr)/* copy-on-write page */ if (write && !(vma->vm_flags &VM_SHARED )) {inc_mm_counter_fast (vma->vm_mm,MM_ANONPAGES ); -page_add_new_anon_rmap (page, vma, addr, false); +page_add_new_anon_rmap (page, vma, addr);lru_cache_add_inactive_or_unevictable (page, vma); }else {inc_mm_counter_fast (vma->vm_mm,mm_counter_file (page));
mm/migrate_device.c +1 -1
@@ -610,7 +610,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, goto unlock_abort;inc_mm_counter (mm,MM_ANONPAGES ); -page_add_new_anon_rmap (page, vma, addr, false); +page_add_new_anon_rmap (page, vma, addr);if (!is_zone_device_page (page))lru_cache_add_inactive_or_unevictable (page, vma);get_page (page);
mm/rmap.c +6 -3
@@ -1182,19 +1182,22 @@ void page_add_anon_rmap(struct page *page, }/** -* page_add_new_anon_rmap - add pte mapping to a new anonymous page +* page_add_new_anon_rmap - add mapping to a new anonymous page * @page: the page to add the mapping to * @vma: the vm area in which the mapping is added * @address: the user virtual address mapped -* @compound: charge the page as compound or small page +* +* If it's a compound page, it is accounted as a compound page. As the page +* is new, it's assume to get mapped exclusively by a single process. * * Same as page_add_anon_rmap but must only be called on *new* pages. * This means the inc-and-test can be bypassed. * Page does not have to be locked. */ void page_add_new_anon_rmap (struct page *page, -struct vm_area_struct *vma,unsigned long address,bool compound) +struct vm_area_struct *vma,unsigned long address) { +const bool compound =PageCompound (page);int nr = compound ?thp_nr_pages (page) :1 ;VM_BUG_ON_VMA (address < vma->vm_start || address >= vma->vm_end, vma);
mm/swapfile.c +1 -1
@@ -1802,7 +1802,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,if (page == swapcache) {page_add_anon_rmap (page, vma, addr,RMAP_NONE ); }else {/* ksm created a completely new copy */ -page_add_new_anon_rmap (page, vma, addr, false); +page_add_new_anon_rmap (page, vma, addr);lru_cache_add_inactive_or_unevictable (page, vma); } set_pte_at(vma->vm_mm, addr, pte,
mm/userfaultfd.c +1 -1
@@ -101,7 +101,7 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd,lru_cache_add (page);page_add_file_rmap (page, dst_vma, false); }else { -page_add_new_anon_rmap (page, dst_vma, dst_addr, false); +page_add_new_anon_rmap (page, dst_vma, dst_addr);lru_cache_add_inactive_or_unevictable (page, dst_vma); }
Signature · 1 line
2.35.1
Re: [PATCH v3 08/16] mm/rmap: drop "compound" parameter from page_add_new_anon_rmap()
Quoted 1 line · David Hildenbrand
New anonymous pages are always mapped natively: only THP/khugepagd code
khugepaged ^
Quoted 6 lines
maps a new compound anonymous page and passes "true". Otherwise, we're just dealing with simple, non-compound pages.
Let's give the interface clearer semantics and document these.
Signed-off-by David Hildenbrand <[email protected]>
Nit:
Quoted 18 lines
mm/rmap.c +4
-2@@ -1182,19 +1182,22 @@ void page_add_anon_rmap(struct page *page, }/** -* page_add_new_anon_rmap - add pte mapping to a new anonymous page +* page_add_new_anon_rmap - add mapping to a new anonymous page * @page: the page to add the mapping to * @vma: the vm area in which the mapping is added * @address: the user virtual address mapped -* @compound: charge the page as compound or small page +* +* If it's a compound page, it is accounted as a compound page. As the page +* is new, it's assume to get mapped exclusively by a single process. * * Same as page_add_anon_rmap but must only be called on *new* pages. * This means the inc-and-test can be bypassed.
mm/rmap.c void page_add_anon_rmap(struct page *page, · line 1203
Quoted 10 lines
mm/rmap.c +2
-1* Page does not have to be locked. */ void page_add_new_anon_rmap (struct page *page, -struct vm_area_struct *vma,unsigned long address,bool compound) +struct vm_area_struct *vma,unsigned long address) { +const bool compound =PageCompound (page);int nr = compound ?thp_nr_pages (page) :1 ;VM_BUG_ON_VMA (address < vma->vm_start || address >= vma->vm_end, vma);
There's a VM_BUG_ON_PAGE(PageTransCompound(page), page); later in a !compound branch. Since compound is now determined by the same check, could be deleted.
Re: [PATCH v3 08/16] mm/rmap: drop "compound" parameter from page_add_new_anon_rmap()
Quoted 43 lines · Vlastimil Babka
Quoted 1 line · David Hildenbrand
New anonymous pages are always mapped natively: only THP/khugepagd code
khugepaged ^
Quoted 6 lines
maps a new compound anonymous page and passes "true". Otherwise, we're just dealing with simple, non-compound pages.
Let's give the interface clearer semantics and document these.
Signed-off-by David Hildenbrand <[email protected]>Acked-by Vlastimil Babka <[email protected]>Nit:
Quoted 27 lines
mm/rmap.c +6
-3@@ -1182,19 +1182,22 @@ void page_add_anon_rmap(struct page *page, }/** -* page_add_new_anon_rmap - add pte mapping to a new anonymous page +* page_add_new_anon_rmap - add mapping to a new anonymous page * @page: the page to add the mapping to * @vma: the vm area in which the mapping is added * @address: the user virtual address mapped -* @compound: charge the page as compound or small page +* +* If it's a compound page, it is accounted as a compound page. As the page +* is new, it's assume to get mapped exclusively by a single process. * * Same as page_add_anon_rmap but must only be called on *new* pages. * This means the inc-and-test can be bypassed. * Page does not have to be locked. */ void page_add_new_anon_rmap (struct page *page, -struct vm_area_struct *vma,unsigned long address,bool compound) +struct vm_area_struct *vma,unsigned long address) { +const bool compound =PageCompound (page);int nr = compound ?thp_nr_pages (page) :1 ;
Quoted 6 lines
mm/rmap.c void page_add_anon_rmap(struct page *page, · line 1203
Quoted 1 line
mm/rmap.c +0
-0VM_BUG_ON_VMA (address < vma->vm_start || address >= vma->vm_end, vma);There's a
VM_BUG_ON_PAGE(PageTransCompound(page), page); later in a !compound branch. Since compound is now determined by the same check, could be deleted.
Yes, eventually we could get rid of both VM_BUG_ON_PAGE() on both branches and add a single VM_BUG_ON_PAGE(PageTail(page), page) check on the compound branch. (we could also make sure that we're not given a hugetlb page)
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v3 08/16] mm/rmap: drop "compound" parameter from page_add_new_anon_rmap()
Quoted 9 lines · David Hildenbrand
Quoted 3 lines · Vlastimil Babka
There's a
VM_BUG_ON_PAGE(PageTransCompound(page), page); later in a !compound branch. Since compound is now determined by the same check, could be deleted.Yes, eventually we could get rid of both
VM_BUG_ON_PAGE()on both branches and add a singleVM_BUG_ON_PAGE(PageTail(page), page)check on the compound branch. (we could also make sure that we're not given a hugetlb page)
As a rule of thumb, if you find yourself wanting to add VM_BUG_ON_PAGE(PageTail(page), page), you probably want to change the interface to take a folio.
Re: [PATCH v3 08/16] mm/rmap: drop "compound" parameter from page_add_new_anon_rmap()
Quoted 6 lines · Matthew Wilcox
Quoted 5 lines · David Hildenbrand
Quoted 3 lines · Vlastimil Babka
There's a
VM_BUG_ON_PAGE(PageTransCompound(page), page); later in a !compound branch. Since compound is now determined by the same check, could be deleted.
Quoted 8 lines
Quoted 4 lines
Yes, eventually we could get rid of both
VM_BUG_ON_PAGE()on both branches and add a singleVM_BUG_ON_PAGE(PageTail(page), page)check on the compound branch. (we could also make sure that we're not given a hugetlb page)As a rule of thumb, if you find yourself wanting to add
VM_BUG_ON_PAGE(PageTail(page), page), you probably want to change the interface to take a folio.
Yeah, I had the same in mind. Might be a reasonable addon on top -- although it would stick out in the rmap code a bit because most functions deal with both, folios and subpages.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v3 08/16] mm/rmap: drop "compound" parameter from page_add_new_anon_rmap()
Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Matthew Wilcox
Quoted 9 lines · David Hildenbrand
Quoted 3 lines · Vlastimil Babka
There's a
VM_BUG_ON_PAGE(PageTransCompound(page), page); later in a !compound branch. Since compound is now determined by the same check, could be deleted.Yes, eventually we could get rid of both
VM_BUG_ON_PAGE()on both branches and add a singleVM_BUG_ON_PAGE(PageTail(page), page)check on the compound branch. (we could also make sure that we're not given a hugetlb page)
Quoted 7 lines
Quoted 3 lines
As a rule of thumb, if you find yourself wanting to add
VM_BUG_ON_PAGE(PageTail(page), page), you probably want to change the interface to take a folio.Yeah, I had the same in mind. Might be a reasonable addon on top -- although it would stick out in the rmap code a bit because most functions deal with both, folios and subpages.
I have the start of a series which starts looking at the fault path to see where it makes sense to use folios and where it makes sense to use pages.
We're (generally) faulting on a PTE, so we need the precise page to be returned in vmf->page. However vmf->cow_page can/should be a folio (because it's definitely not a tail page). That trickles down into copy_present_page() (new_page and prealloc both become folios) and so page_add_new_anon_rmap() then looks like a good target to take a folio.
The finish_fault() -> do_set_pte() -> page_add_new_anon_rmap() looks like the only kind of strange place where we don't necessarily have a folio (all the others we just allocated it).
Re: [PATCH v3 08/16] mm/rmap: drop "compound" parameter from page_add_new_anon_rmap()
Quoted 32 lines · Matthew Wilcox
Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Matthew Wilcox
Quoted 9 lines · David Hildenbrand
Quoted 3 lines · Vlastimil Babka
There's a
VM_BUG_ON_PAGE(PageTransCompound(page), page); later in a !compound branch. Since compound is now determined by the same check, could be deleted.Yes, eventually we could get rid of both
VM_BUG_ON_PAGE()on both branches and add a singleVM_BUG_ON_PAGE(PageTail(page), page)check on the compound branch. (we could also make sure that we're not given a hugetlb page)Quoted 7 lines
Quoted 3 lines
As a rule of thumb, if you find yourself wanting to add
VM_BUG_ON_PAGE(PageTail(page), page), you probably want to change the interface to take a folio.Yeah, I had the same in mind. Might be a reasonable addon on top -- although it would stick out in the rmap code a bit because most functions deal with both, folios and subpages.
I have the start of a series which starts looking at the fault path to see where it makes sense to use folios and where it makes sense to use pages.
We're (generally) faulting on a PTE, so we need the precise page to be returned in
vmf->page. Howevervmf->cow_pagecan/should be a folio (because it's definitely not a tail page). That trickles down intocopy_present_page()(new_pageand prealloc both become folios) and sopage_add_new_anon_rmap()then looks like a good target to take a folio.
Quoted 4 lines
The
finish_fault()->do_set_pte()->page_add_new_anon_rmap()looks like the only kind of strange place where we don't necessarily have a folio (all the others we just allocated it).
That's an interesting point. In this patch I'm assuming that we don't have a compound page here (see below).
Which makes sense, because as the interface states "Same as page_add_anon_rmap but must only be called on *new* pages.".
At least to me it would be weird to allocate a new compound page to then pass a subpage to do_set_pte() page_add_new_anon_rmap().
And in fact, inside page_add_new_anon_rmap(compound=false) we have
/* Anon THP always mapped first with PMD */VM_BUG_ON_PAGE(PageTransCompound(page), page);
which makes sure that we cannot have a compound page here, but in fact a folio.
So unless I am missing something, do_set_pte() should in fact have a folio here unless BUG?
Signature · 3 lines
Thanks, David / dhildenb
PATCH v3 9/16 mm/rmap: use page_move_anon_rmap() when reusing a mapped PageAnon() page exclusively
In reply to David HildenbrandWe want to mark anonymous pages exclusive, and when using page_move_anon_rmap() we know that we are the exclusive user, as properly documented. This is a preparation for marking anonymous pages exclusive in page_move_anon_rmap().
In both instances, we're holding page lock and are sure that we're the exclusive owner (page_count() == 1). hugetlb already properly uses page_move_anon_rmap() in the write fault handler.
Note that in case of a PTE-mapped THP, we'll only end up calling this function if the whole THP is only referenced by the single PTE mapping a single subpage (page_count() == 1); consequently, it's fine to modify the compound page mapping inside page_move_anon_rmap().
--- mm/huge_memory.c | 2 ++ mm/memory.c | 1 + 2 files changed, 3 insertions(+)
mm/huge_memory.c +2 -0
@@ -1317,6 +1317,8 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf)try_to_free_swap (page);if (page_count (page) ==1 ) {pmd_t entry; + +page_move_anon_rmap (page, vma); entry =pmd_mkyoung (orig_pmd); entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma);if (pmdp_set_access_flags (vma, haddr, vmf->pmd, entry,1 ))
mm/memory.c +1 -0
@@ -3303,6 +3303,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)* and the page is locked, it's dark out, and we're wearing * sunglasses. Hit it. */ +page_move_anon_rmap (page, vma);unlock_page (page);wp_page_reuse (vmf);return VM_FAULT_WRITE ;
Signature · 1 line
2.35.1
Re: [PATCH v3 09/16] mm/rmap: use page_move_anon_rmap() when reusing a mapped PageAnon() page exclusively
Quoted 8 lines · David Hildenbrand
We want to mark anonymous pages exclusive, and when using
page_move_anon_rmap()we know that we are the exclusive user, as properly documented. This is a preparation for marking anonymous pages exclusive inpage_move_anon_rmap().In both instances, we're holding page lock and are sure that we're the exclusive owner (
page_count()== 1). hugetlb already properly usespage_move_anon_rmap()in the write fault handler.
Yeah, note that do_wp_page() used to call page_move_anon_rmap() always since the latter was introduced, until commit 09854ba94c6a ("mm: do_wp_page() simplification"). Probably not intended.
Quoted 6 lines
Note that in case of a PTE-mapped THP, we'll only end up calling this function if the whole THP is only referenced by the single PTE mapping a single subpage (
page_count()== 1); consequently, it's fine to modify the compound page mapping insidepage_move_anon_rmap().Signed-off-by David Hildenbrand <[email protected]>
Quoted 30 lines
--- mm/huge_memory.c | 2 ++ mm/memory.c | 1 + 2 files changed, 3 insertions(+)mm/huge_memory.c +2
-0@@ -1317,6 +1317,8 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf)try_to_free_swap (page);if (page_count (page) ==1 ) {pmd_t entry; + +page_move_anon_rmap (page, vma); entry =pmd_mkyoung (orig_pmd); entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma);if (pmdp_set_access_flags (vma, haddr, vmf->pmd, entry,1 ))mm/memory.c +1
-0@@ -3303,6 +3303,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)* and the page is locked, it's dark out, and we're wearing * sunglasses. Hit it. */ +page_move_anon_rmap (page, vma);unlock_page (page);wp_page_reuse (vmf);return VM_FAULT_WRITE ;
Re: [PATCH v3 09/16] mm/rmap: use page_move_anon_rmap() when reusing a mapped PageAnon() page exclusively
Quoted 6 lines · Vlastimil Babka
Quoted 5 lines · David Hildenbrand
We want to mark anonymous pages exclusive, and when using
page_move_anon_rmap()we know that we are the exclusive user, as properly documented. This is a preparation for marking anonymous pages exclusive inpage_move_anon_rmap().
Quoted 7 lines
Quoted 3 lines
In both instances, we're holding page lock and are sure that we're the exclusive owner (
page_count()== 1). hugetlb already properly usespage_move_anon_rmap()in the write fault handler.Yeah, note that
do_wp_page()used to callpage_move_anon_rmap()always since the latter was introduced, until commit09854ba94c6a("mm:do_wp_page()simplification"). Probably not intended.
Yeah, it was buried underneath all that reuse_swap_page() complexity.
Signature · 3 lines
Thanks, David / dhildenb
PATCH v3 10/16 mm/huge_memory: remove outdated VM_WARN_ON_ONCE_PAGE from unmap_page()
In reply to David HildenbrandWe can already theoretically fail to unmap (still having page_mapped()) in case arch_unmap_one() fails, which can happen on sparc. Failures to unmap are handled gracefully, just as if there are other references on the target page: freezing the refcount in split_huge_page_to_list() will fail if still mapped and we'll simply remap.
In commit 504e070dc08f ("mm: thp: replace DEBUG_VM BUG with VM_WARN when unmap fails for split") we already converted to VM_WARN_ON_ONCE_PAGE, let's get rid of it completely now.
This is a preparation for making try_to_migrate() fail on anonymous pages with GUP pins, which will make this VM_WARN_ON_ONCE_PAGE trigger more frequently.
--- mm/huge_memory.c | 2 -- 1 file changed, 2 deletions(-)
mm/huge_memory.c +0 -2
@@ -2241,8 +2241,6 @@ static void unmap_page(struct page *page)try_to_migrate (folio, ttu_flags); else try_to_unmap(folio, ttu_flags | TTU_IGNORE_MLOCK); - -VM_WARN_ON_ONCE_PAGE (page_mapped (page), page); }static void remap_page (struct folio *folio,unsigned long nr)
Signature · 1 line
2.35.1
Re: [PATCH v3 10/16] mm/huge_memory: remove outdated VM_WARN_ON_ONCE_PAGE from unmap_page()
Quoted 10 lines · David Hildenbrand
We can already theoretically fail to unmap (still having
page_mapped()) in casearch_unmap_one()fails, which can happen on sparc. Failures to unmap are handled gracefully, just as if there are other references on the target page: freezing the refcount insplit_huge_page_to_list()will fail if still mapped and we'll simply remap.In commit
504e070dc08f("mm: thp: replaceDEBUG_VMBUG withVM_WARNwhen unmap fails for split") we already converted toVM_WARN_ON_ONCE_PAGE, let's get rid of it completely now.
Quoted 7 lines
This is a preparation for making
try_to_migrate()fail on anonymous pages with GUP pins, which will make thisVM_WARN_ON_ONCE_PAGEtrigger more frequently.Reported-by Yang Shi <[email protected]>Reviewed-by Yang Shi <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Quoted 17 lines
--- mm/huge_memory.c | 2 -- 1 file changed, 2 deletions(-)mm/huge_memory.c +0
-2@@ -2241,8 +2241,6 @@ static void unmap_page(struct page *page)try_to_migrate (folio, ttu_flags); else try_to_unmap(folio, ttu_flags | TTU_IGNORE_MLOCK); - -VM_WARN_ON_ONCE_PAGE (page_mapped (page), page); }static void remap_page (struct folio *folio,unsigned long nr)
PATCH v3 11/16 mm/page-flags: reuse PG_mappedtodisk as PG_anon_exclusive for PageAnon() pages
In reply to David HildenbrandThe basic question we would like to have a reliable and efficient answer to is: is this anonymous page exclusive to a single process or might it be shared? We need that information for ordinary/single pages, hugetlb pages, and possibly each subpage of a THP.
Introduce a way to mark an anonymous page as exclusive, with the ultimate goal of teaching our COW logic to not do "wrong COWs", whereby GUP pins lose consistency with the pages mapped into the page table, resulting in reported memory corruptions.
Most pageflags already have semantics for anonymous pages, however, PG_mappedtodisk should never apply to pages in the swapcache, so let's reuse that flag.
As PG_has_hwpoisoned also uses that flag on the second tail page of a compound page, convert it to PG_error instead, which is marked as PF_NO_TAIL, so never used for tail pages.
Use custom page flag modification functions such that we can do additional sanity checks. The semantics we'll put into some kernel doc in the future are:
"
PG_anon_exclusive is *usually* only expressive in combination with a page table entry. Depending on the page table entry type it might store the following information:
Is what's mapped via this page table entry exclusive to the single process and can be mapped writable without further checks? If not, it might be shared and we might have to COW.
For now, we only expect PTE-mapped THPs to make use of PG_anon_exclusive in subpages. For other anonymous compound folios (i.e., hugetlb), only the head page is logically mapped and holds this information.
For example, an exclusive, PMD-mapped THP only has PG_anon_exclusive set on the head page. When replacing the PMD by a page table full of PTEs, PG_anon_exclusive, if set on the head page, will be set on all tail pages accordingly. Note that converting from a PTE-mapping to a PMD mapping using the same compound page is currently not possible and consequently doesn't require care.
If GUP wants to take a reliable pin (FOLL_PIN) on an anonymous page, it should only pin if the relevant PG_anon_bit is set. In that case, the pin will be fully reliable and stay consistent with the pages mapped into the page table, as the bit cannot get cleared (e.g., by fork(), KSM) while the page is pinned. For anonymous pages that are mapped R/W, PG_anon_exclusive can be assumed to always be set because such pages cannot possibly be shared.
The page table lock protecting the page table entry is the primary synchronization mechanism for PG_anon_exclusive; GUP-fast that does not take the PT lock needs special care when trying to clear the flag.
Page table entry types and PG_anon_exclusive:
* Present: PG_anon_exclusive applies.
* Swap: the information is lost. PG_anon_exclusive was cleared.
* Migration: the entry holds this information instead. PG_anon_exclusive was cleared.
* Device private: PG_anon_exclusive applies.
* Device exclusive: PG_anon_exclusive applies.
* HW Poison: PG_anon_exclusive is stale and not changed.
If the page may be pinned (FOLL_PIN), clearing PG_anon_exclusive is not allowed and the flag will stick around until the page is freed and folio->mapping is cleared.
"
We won't be clearing PG_anon_exclusive on destructive unmapping (i.e., zapping) of page table entries, page freeing code will handle that when also invalidate page->mapping to not indicate PageAnon() anymore. Letting information about exclusivity stick around will be an important property when adding sanity checks to unpinning code.
Note that we properly clear the flag in free_pages_prepare() via PAGE_FLAGS_CHECK_AT_PREP for each individual subpage of a compound page, so there is no need to manually clear the flag.
--- include/linux/page-flags.h | 39 +++++++++++++++++++++++++++++++++++++- mm/hugetlb.c | 2 ++ mm/memory.c | 11 +++++++++++ mm/memremap.c | 9 +++++++++ mm/swapfile.c | 4 ++++ tools/vm/page-types.c | 8 +++++++- 6 files changed, 71 insertions(+), 2 deletions(-)
include/linux/page-flags.h +38 -1
@@ -142,6 +142,15 @@ enum pageflags { PG_readahead = PG_reclaim, +/* +* Depending on the way an anonymous folio can be mapped into a page +* table (e.g., single PMD/PUD/CONT of the head page vs. PTE-mapped +* THP), PG_anon_exclusive may be set only for the head page or for +* tail pages of an anonymous folio. For now, we only expect it to be +* set on tail pages for PTE-mapped THP. +*/ + PG_anon_exclusive = PG_mappedtodisk, +/* Filesystems */ PG_checked = PG_owner_priv_1,
@@ -176,7 +185,7 @@ enum pageflags {* Indicates that at least one subpage is hwpoisoned in the * THP. */ - PG_has_hwpoisoned = PG_mappedtodisk, + PG_has_hwpoisoned = PG_error,#endif /* non-lru isolated movable page */
@@ -1002,6 +1011,34 @@ extern bool is_free_buddy_page(struct page *page);PAGEFLAG (Isolated, isolated,PF_ANY ); +static __always_inline int PageAnonExclusive(struct page *page) +{ +VM_BUG_ON_PGFLAGS (!PageAnon (page), page); +VM_BUG_ON_PGFLAGS (PageHuge (page) && !PageHead (page), page); +return test_bit (PG_anon_exclusive, &PF_ANY (page,1 )->flags); +} + +static __always_inline void SetPageAnonExclusive(struct page *page) +{ +VM_BUG_ON_PGFLAGS (!PageAnon (page) ||PageKsm (page), page); +VM_BUG_ON_PGFLAGS (PageHuge (page) && !PageHead (page), page); +set_bit (PG_anon_exclusive, &PF_ANY (page,1 )->flags); +} + +static __always_inline void ClearPageAnonExclusive(struct page *page) +{ +VM_BUG_ON_PGFLAGS (!PageAnon (page) ||PageKsm (page), page); +VM_BUG_ON_PGFLAGS (PageHuge (page) && !PageHead (page), page); +clear_bit (PG_anon_exclusive, &PF_ANY (page,1 )->flags); +} + +static __always_inline void __ClearPageAnonExclusive(struct page *page) +{ +VM_BUG_ON_PGFLAGS (!PageAnon (page), page); +VM_BUG_ON_PGFLAGS (PageHuge (page) && !PageHead (page), page); +__clear_bit (PG_anon_exclusive, &PF_ANY (page,1 )->flags); +} +#ifdef CONFIG_MMU #define __PG_MLOCKED (1UL << PG_mlocked)#else
mm/hugetlb.c +2 -0
@@ -1672,6 +1672,8 @@ void free_huge_page(struct page *page)VM_BUG_ON_PAGE (page_mapcount (page), page);hugetlb_set_page_subpool (page,NULL ); +if (PageAnon (page)) +__ClearPageAnonExclusive (page); page->mapping =NULL ; restore_reserve =HPageRestoreReserve (page);ClearHPageRestoreReserve (page);
mm/memory.c +11 -0
@@ -3663,6 +3663,17 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } +/* +* PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte +* must never point at an anonymous page in the swapcache that is +* PG_anon_exclusive. Sanity check that this holds and especially, that +* no filesystem set PG_mappedtodisk on a page in the swapcache. Sanity +* check after taking the PT lock and making sure that nobody +* concurrently faulted in this page and set PG_anon_exclusive. +*/ +BUG_ON (!PageAnon (page) &&PageMappedToDisk (page)); +BUG_ON (PageAnon (page) &&PageAnonExclusive (page)); +/* * Remove the swap entry and conditionally try to free up the swapcache. * We're already holding a reference on the page but haven't mapped it
mm/memremap.c +9 -0
@@ -458,6 +458,15 @@ void free_zone_device_page(struct page *page)mem_cgroup_uncharge (page_folio (page)); +/* +* Note: we don't expect anonymous compound pages yet. Once supported +* and we could PTE-map them similar to THP, we'd have to clear +* PG_anon_exclusive on all tail pages. +*/ +VM_BUG_ON_PAGE (PageAnon (page) &&PageCompound (page), page); +if (PageAnon (page)) +__ClearPageAnonExclusive (page); +/* * When a device managed page is freed, the page->mapping field * may still contain a (stale) mapping value. For example, the
mm/swapfile.c +4 -0
@@ -1796,6 +1796,10 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, goto out; } +/* See do_swap_page() */ +BUG_ON (!PageAnon (page) &&PageMappedToDisk (page)); +BUG_ON (PageAnon (page) &&PageAnonExclusive (page)); +dec_mm_counter (vma->vm_mm,MM_SWAPENTS );inc_mm_counter (vma->vm_mm,MM_ANONPAGES );get_page (page);
tools/vm/page-types.c +7 -1
@@ -80,9 +80,10 @@#define KPF_SOFTDIRTY 40#define KPF_ARCH_2 41 -/* [48-] take some arbitrary free slots for expanding overloaded flags +/* [47-] take some arbitrary free slots for expanding overloaded flags * not part of kernel API */ +#define KPF_ANON_EXCLUSIVE 47#define KPF_READAHEAD 48#define KPF_SLOB_FREE 49#define KPF_SLUB_FROZEN 50
@@ -138,6 +139,7 @@ static const char * const page_flag_names[] = { [KPF_SOFTDIRTY] = "f:softdirty", [KPF_ARCH_2] = "H:arch_2", + [KPF_ANON_EXCLUSIVE] = "d:anon_exclusive", [KPF_READAHEAD] = "I:readahead", [KPF_SLOB_FREE] = "P:slob_free", [KPF_SLUB_FROZEN] = "A:slub_frozen",
@@ -472,6 +474,10 @@ static int bit_mask_ok(uint64_t flags)static uint64_t expand_overloaded_flags (uint64_t flags,uint64_t pme) { +/* Anonymous pages overload PG_mappedtodisk */ +if ((flags &BIT (ANON )) && (flags &BIT (MAPPEDTODISK ))) + flags ^=BIT (MAPPEDTODISK ) |BIT (ANON_EXCLUSIVE ); +/* SLOB/SLUB overload several page flags */ if (flags &BIT (SLAB )) {if (flags &BIT (PRIVATE ))
Signature · 1 line
2.35.1
Re: [PATCH v3 11/16] mm/page-flags: reuse PG_mappedtodisk as PG_anon_exclusive for PageAnon() pages
Quoted 43 lines · David Hildenbrand
The basic question we would like to have a reliable and efficient answer to is: is this anonymous page exclusive to a single process or might it be shared? We need that information for ordinary/single pages, hugetlb pages, and possibly each subpage of a THP.
Introduce a way to mark an anonymous page as exclusive, with the ultimate goal of teaching our COW logic to not do "wrong COWs", whereby GUP pins lose consistency with the pages mapped into the page table, resulting in reported memory corruptions.
Most pageflags already have semantics for anonymous pages, however,
PG_mappedtodiskshould never apply to pages in the swapcache, so let's reuse that flag.As
PG_has_hwpoisonedalso uses that flag on the second tail page of a compound page, convert it toPG_errorinstead, which is marked asPF_NO_TAIL, so never used for tail pages.Use custom page flag modification functions such that we can do additional sanity checks. The semantics we'll put into some kernel doc in the future are:
"
PG_anon_exclusiveis *usually* only expressive in combination with a page table entry. Depending on the page table entry type it might store the following information:Is what's mapped via this page table entry exclusive to the single process and can be mapped writable without further checks? If not, it might be shared and we might have to COW.
For now, we only expect PTE-mapped THPs to make use of
PG_anon_exclusivein subpages. For other anonymous compound folios (i.e., hugetlb), only the head page is logically mapped and holds this information.For example, an exclusive, PMD-mapped THP only has
PG_anon_exclusiveset on the head page. When replacing the PMD by a page table full of PTEs,PG_anon_exclusive, if set on the head page, will be set on all tail pages accordingly. Note that converting from a PTE-mapping to a PMD mapping using the same compound page is currently not possible and consequently doesn't require care.
Quoted 2 lines
If GUP wants to take a reliable pin (
FOLL_PIN) on an anonymous page, it should only pin if the relevantPG_anon_bitis set. In that case,
^ PG_anon_exclusive bit ?
Quoted 20 lines
the pin will be fully reliable and stay consistent with the pages mapped into the page table, as the bit cannot get cleared (e.g., by
fork(), KSM) while the page is pinned. For anonymous pages that are mapped R/W,PG_anon_exclusivecan be assumed to always be set because such pages cannot possibly be shared.The page table lock protecting the page table entry is the primary synchronization mechanism for
PG_anon_exclusive; GUP-fast that does not take the PT lock needs special care when trying to clear the flag.Page table entry types and
PG_anon_exclusive:
* Present:PG_anon_exclusiveapplies.
* Swap: the information is lost.PG_anon_exclusivewas cleared.
* Migration: the entry holds this information instead.PG_anon_exclusivewas cleared.
* Device private:PG_anon_exclusiveapplies.
* Device exclusive:PG_anon_exclusiveapplies.
* HW Poison:PG_anon_exclusiveis stale and not changed.
Quoted 3 lines
If the page may be pinned (
FOLL_PIN), clearingPG_anon_exclusiveis not allowed and the flag will stick around until the page is freed andfolio->mappingis cleared.
Or also if it's unpinned?
Quoted 8 lines
"
We won't be clearing
PG_anon_exclusiveon destructive unmapping (i.e., zapping) of page table entries, page freeing code will handle that when also invalidatepage->mappingto not indicatePageAnon()anymore. Letting information about exclusivity stick around will be an important property when adding sanity checks to unpinning code.
Quoted 5 lines
Note that we properly clear the flag in
free_pages_prepare()viaPAGE_FLAGS_CHECK_AT_PREPfor each individual subpage of a compound page, so there is no need to manually clear the flag.Signed-off-by David Hildenbrand <[email protected]>
Quoted 7 lines
mm/memory.c +1
-0@@ -3663,6 +3663,17 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } +/*
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3676
Quoted 10 lines
mm/memory.c +10
-0+* PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte +* must never point at an anonymous page in the swapcache that is +* PG_anon_exclusive. Sanity check that this holds and especially, that +* no filesystem set PG_mappedtodisk on a page in the swapcache. Sanity +* check after taking the PT lock and making sure that nobody +* concurrently faulted in this page and set PG_anon_exclusive. +*/ +BUG_ON (!PageAnon (page) &&PageMappedToDisk (page)); +BUG_ON (PageAnon (page) &&PageAnonExclusive (page)); +
Hmm, dunno why not VM_BUG_ON?
Quoted 5 lines
mm/memory.c +0
-0/* * Remove the swap entry and conditionally try to free up the swapcache. * We're already holding a reference on the page but haven't mapped it mm/memremap.c +0
-0
Re: [PATCH v3 11/16] mm/page-flags: reuse PG_mappedtodisk as PG_anon_exclusive for PageAnon() pages
Quoted 69 lines · Vlastimil Babka
Quoted 43 lines · David Hildenbrand
The basic question we would like to have a reliable and efficient answer to is: is this anonymous page exclusive to a single process or might it be shared? We need that information for ordinary/single pages, hugetlb pages, and possibly each subpage of a THP.
Introduce a way to mark an anonymous page as exclusive, with the ultimate goal of teaching our COW logic to not do "wrong COWs", whereby GUP pins lose consistency with the pages mapped into the page table, resulting in reported memory corruptions.
Most pageflags already have semantics for anonymous pages, however,
PG_mappedtodiskshould never apply to pages in the swapcache, so let's reuse that flag.As
PG_has_hwpoisonedalso uses that flag on the second tail page of a compound page, convert it toPG_errorinstead, which is marked asPF_NO_TAIL, so never used for tail pages.Use custom page flag modification functions such that we can do additional sanity checks. The semantics we'll put into some kernel doc in the future are:
"
PG_anon_exclusiveis *usually* only expressive in combination with a page table entry. Depending on the page table entry type it might store the following information:Is what's mapped via this page table entry exclusive to the single process and can be mapped writable without further checks? If not, it might be shared and we might have to COW.
For now, we only expect PTE-mapped THPs to make use of
PG_anon_exclusivein subpages. For other anonymous compound folios (i.e., hugetlb), only the head page is logically mapped and holds this information.For example, an exclusive, PMD-mapped THP only has
PG_anon_exclusiveset on the head page. When replacing the PMD by a page table full of PTEs,PG_anon_exclusive, if set on the head page, will be set on all tail pages accordingly. Note that converting from a PTE-mapping to a PMD mapping using the same compound page is currently not possible and consequently doesn't require care.Quoted 2 lines
If GUP wants to take a reliable pin (
FOLL_PIN) on an anonymous page, it should only pin if the relevantPG_anon_bitis set. In that case,^
PG_anon_exclusivebit ?Quoted 20 lines
the pin will be fully reliable and stay consistent with the pages mapped into the page table, as the bit cannot get cleared (e.g., by
fork(), KSM) while the page is pinned. For anonymous pages that are mapped R/W,PG_anon_exclusivecan be assumed to always be set because such pages cannot possibly be shared.The page table lock protecting the page table entry is the primary synchronization mechanism for
PG_anon_exclusive; GUP-fast that does not take the PT lock needs special care when trying to clear the flag.Page table entry types and
PG_anon_exclusive:
* Present:PG_anon_exclusiveapplies.
* Swap: the information is lost.PG_anon_exclusivewas cleared.
* Migration: the entry holds this information instead.PG_anon_exclusivewas cleared.
* Device private:PG_anon_exclusiveapplies.
* Device exclusive:PG_anon_exclusiveapplies.
* HW Poison:PG_anon_exclusiveis stale and not changed.
Quoted 5 lines
Quoted 3 lines
If the page may be pinned (
FOLL_PIN), clearingPG_anon_exclusiveis not allowed and the flag will stick around until the page is freed andfolio->mappingis cleared.Or also if it's unpinned?
I'm afraid I didn't get your question. Once the page is no longer pinned, we can succeed in clearing PG_anon_exclusive (just like pinning never happened). Does that answer your question?
Quoted 9 lines
Quoted 9 lines
"
We won't be clearing
PG_anon_exclusiveon destructive unmapping (i.e., zapping) of page table entries, page freeing code will handle that when also invalidatepage->mappingto not indicatePageAnon()anymore. Letting information about exclusivity stick around will be an important property when adding sanity checks to unpinning code.
Quoted 7 lines
Quoted 5 lines
Note that we properly clear the flag in
free_pages_prepare()viaPAGE_FLAGS_CHECK_AT_PREPfor each individual subpage of a compound page, so there is no need to manually clear the flag.Signed-off-by David Hildenbrand <[email protected]>Acked-by Vlastimil Babka <[email protected]>
Thanks!
Quoted 7 lines
Quoted 7 lines
mm/memory.c +0
-0@@ -3663,6 +3663,17 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; }
Quoted 13 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3676
Quoted 11 lines
mm/memory.c +11
-0+/* +* PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte +* must never point at an anonymous page in the swapcache that is +* PG_anon_exclusive. Sanity check that this holds and especially, that +* no filesystem set PG_mappedtodisk on a page in the swapcache. Sanity +* check after taking the PT lock and making sure that nobody +* concurrently faulted in this page and set PG_anon_exclusive. +*/ +BUG_ON (!PageAnon (page) &&PageMappedToDisk (page)); +BUG_ON (PageAnon (page) &&PageAnonExclusive (page)); +Hmm, dunno why not
VM_BUG_ON?
Getting PageAnonExclusive accidentally set by a file system would result in an extremely unpleasant security issue. I most surely want to catch something like that in any case, especially in the foreseeable future.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v3 11/16] mm/page-flags: reuse PG_mappedtodisk as PG_anon_exclusive for PageAnon() pages
Quoted 26 lines · David Hildenbrand
Quoted 25 lines · Vlastimil Babka
Quoted 23 lines · David Hildenbrand
the pin will be fully reliable and stay consistent with the pages mapped into the page table, as the bit cannot get cleared (e.g., by
fork(), KSM) while the page is pinned. For anonymous pages that are mapped R/W,PG_anon_exclusivecan be assumed to always be set because such pages cannot possibly be shared.The page table lock protecting the page table entry is the primary synchronization mechanism for
PG_anon_exclusive; GUP-fast that does not take the PT lock needs special care when trying to clear the flag.Page table entry types and
PG_anon_exclusive:
* Present:PG_anon_exclusiveapplies.
* Swap: the information is lost.PG_anon_exclusivewas cleared.
* Migration: the entry holds this information instead.PG_anon_exclusivewas cleared.
* Device private:PG_anon_exclusiveapplies.
* Device exclusive:PG_anon_exclusiveapplies.
* HW Poison:PG_anon_exclusiveis stale and not changed.If the page may be pinned (
FOLL_PIN), clearingPG_anon_exclusiveis not allowed and the flag will stick around until the page is freed andfolio->mappingis cleared.
Quoted 5 lines
Quoted 1 line
Or also if it's unpinned?
I'm afraid I didn't get your question. Once the page is no longer pinned, we can succeed in clearing
PG_anon_exclusive(just like pinning never happened). Does that answer your question?
Yeah it looked like a scenario that's oddly missing in that description, yet probably obvious. Now I feel it's indeed obvious, so nevermind :)
Quoted 37 lines
Quoted 6 lines
Quoted 6 lines
We won't be clearing
PG_anon_exclusiveon destructive unmapping (i.e., zapping) of page table entries, page freeing code will handle that when also invalidatepage->mappingto not indicatePageAnon()anymore. Letting information about exclusivity stick around will be an important property when adding sanity checks to unpinning code.Quoted 7 lines
Quoted 5 lines
Note that we properly clear the flag in
free_pages_prepare()viaPAGE_FLAGS_CHECK_AT_PREPfor each individual subpage of a compound page, so there is no need to manually clear the flag.Signed-off-by David Hildenbrand <[email protected]>Acked-by Vlastimil Babka <[email protected]>Thanks!
Quoted 7 lines
Quoted 7 lines
mm/memory.c +0
-0@@ -3663,6 +3663,17 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; }Quoted 13 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3676
Quoted 11 lines
mm/memory.c +11
-0+/* +* PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte +* must never point at an anonymous page in the swapcache that is +* PG_anon_exclusive. Sanity check that this holds and especially, that +* no filesystem set PG_mappedtodisk on a page in the swapcache. Sanity +* check after taking the PT lock and making sure that nobody +* concurrently faulted in this page and set PG_anon_exclusive. +*/ +BUG_ON (!PageAnon (page) &&PageMappedToDisk (page)); +BUG_ON (PageAnon (page) &&PageAnonExclusive (page)); +Hmm, dunno why not
VM_BUG_ON?
Quoted 3 lines
Getting PageAnonExclusive accidentally set by a file system would result in an extremely unpleasant security issue. I most surely want to catch something like that in any case, especially in the foreseeable future.
OK then.
PATCH v3 12/16 mm: remember exclusively mapped anonymous pages with PG_anon_exclusive
In reply to David HildenbrandLet's mark exclusively mapped anonymous pages with PG_anon_exclusive as exclusive, and use that information to make GUP pins reliable and stay consistent with the page mapped into the page table even if the page table entry gets write-protected.
With that information at hand, we can extend our COW logic to always reuse anonymous pages that are exclusive. For anonymous pages that might be shared, the existing logic applies.
As already documented, PG_anon_exclusive is usually only expressive in combination with a page table entry. Especially PTE vs. PMD-mapped anonymous pages require more thought, some examples: due to mremap() we can easily have a single compound page PTE-mapped into multiple page tables exclusively in a single process -- multiple page table locks apply. Further, due to MADV_WIPEONFORK we might not necessarily write-protect all PTEs, and only some subpages might be pinned. Long story short: once PTE-mapped, we have to track information about exclusivity per sub-page, but until then, we can just track it for the compound page in the head page and not having to update a whole bunch of subpages all of the time for a simple PMD mapping of a THP.
For simplicity, this commit mostly talks about "anonymous pages", while it's for THP actually "the part of an anonymous folio referenced via a page table entry".
To not spill PG_anon_exclusive code all over the mm code-base, we let the anon rmap code to handle all PG_anon_exclusive logic it can easily handle.
If a writable, present page table entry points at an anonymous (sub)page, that (sub)page must be PG_anon_exclusive. If GUP wants to take a reliably pin (FOLL_PIN) on an anonymous page references via a present page table entry, it must only pin if PG_anon_exclusive is set for the mapped (sub)page.
This commit doesn't adjust GUP, so this is only implicitly handled for FOLL_WRITE, follow-up commits will teach GUP to also respect it for FOLL_PIN without !FOLL_WRITE, to make all GUP pins of anonymous pages fully reliable.
Whenever an anonymous page is to be shared (fork(), KSM), or when temporarily unmapping an anonymous page (swap, migration), the relevant PG_anon_exclusive bit has to be cleared to mark the anonymous page possibly shared. Clearing will fail if there are GUP pins on the page:
* For fork(), this means having to copy the page and not being able to share it. fork() protects against concurrent GUP using the PT lock and the src_mm->write_protect_seq.
* For KSM, this means sharing will fail. For swap this means, unmapping will fail, For migration this means, migration will fail early. All three cases protect against concurrent GUP using the PT lock and a proper clear/invalidate+flush of the relevant page table entry.
This fixes memory corruptions reported for FOLL_PIN | FOLL_WRITE, when a pinned page gets mapped R/O and the successive write fault ends up replacing the page instead of reusing it. It improves the situation for O_DIRECT/vmsplice/... that still use FOLL_GET instead of FOLL_PIN, if fork() is *not* involved, however swapout and fork() are still problematic. Properly using FOLL_PIN instead of FOLL_GET for these GUP users will fix the issue for them.
I. Details about basic handling
I.1. Fresh anonymous pages
page_add_new_anon_rmap() and hugepage_add_new_anon_rmap() will mark the given page exclusive via __page_set_anon_rmap(exclusive=1). As that is the mechanism fresh anonymous pages come into life (besides migration code where we copy the page->mapping), all fresh anonymous pages will start out as exclusive.
I.2. COW reuse handling of anonymous pages
When a COW handler stumbles over a (sub)page that's marked exclusive, it simply reuses it. Otherwise, the handler tries harder under page lock to detect if the (sub)page is exclusive and can be reused. If exclusive, page_move_anon_rmap() will mark the given (sub)page exclusive.
Note that hugetlb code does not yet check for PageAnonExclusive(), as it still uses the old COW logic that is prone to the COW security issue because hugetlb code cannot really tolerate unnecessary/wrong COW as huge pages are a scarce resource.
I.3. Migration handling
try_to_migrate() has to try marking an exclusive anonymous page shared via page_try_share_anon_rmap(). If it fails because there are GUP pins on the page, unmap fails. migrate_vma_collect_pmd() and __split_huge_pmd_locked() are handled similarly.
Writable migration entries implicitly point at shared anonymous pages. For readable migration entries that information is stored via a new "readable-exclusive" migration entry, specific to anonymous pages.
When restoring a migration entry in remove_migration_pte(), information about exlusivity is detected via the migration entry type, and RMAP_EXCLUSIVE is set accordingly for page_add_anon_rmap()/hugepage_add_anon_rmap() to restore that information.
I.4. Swapout handling
try_to_unmap() has to try marking the mapped page possibly shared via page_try_share_anon_rmap(). If it fails because there are GUP pins on the page, unmap fails. For now, information about exclusivity is lost. In the future, we might want to remember that information in the swap entry in some cases, however, it requires more thought, care, and a way to store that information in swap entries.
I.5. Swapin handling
do_swap_page() will never stumble over exclusive anonymous pages in the swap cache, as try_to_migrate() prohibits that. do_swap_page() always has to detect manually if an anonymous page is exclusive and has to set RMAP_EXCLUSIVE for page_add_anon_rmap() accordingly.
I.6. THP handling
__split_huge_pmd_locked() has to move the information about exclusivity from the PMD to the PTEs.
a) In case we have a readable-exclusive PMD migration entry, simply insert readable-exclusive PTE migration entries.
b) In case we have a present PMD entry and we don't want to freeze ("convert to migration entries"), simply forward PG_anon_exclusive to all sub-pages, no need to temporarily clear the bit.
c) In case we have a present PMD entry and want to freeze, handle it similar to try_to_migrate(): try marking the page shared first. In case we fail, we ignore the "freeze" instruction and simply split ordinarily. try_to_migrate() will properly fail because the THP is still mapped via PTEs.
When splitting a compound anonymous folio (THP), the information about exclusivity is implicitly handled via the migration entries: no need to replicate PG_anon_exclusive manually.
I.7. fork() handling
fork() handling is relatively easy, because PG_anon_exclusive is only expressive for some page table entry types.
a) Present anonymous pages
page_try_dup_anon_rmap() will mark the given subpage shared -- which will fail if the page is pinned. If it failed, we have to copy (or PTE-map a PMD to handle it on the PTE level).
Note that device exclusive entries are just a pointer at a PageAnon() page. fork() will first convert a device exclusive entry to a present page table and handle it just like present anonymous pages.
b) Device private entry
Device private entries point at PageAnon() pages that cannot be mapped directly and, therefore, cannot get pinned.
page_try_dup_anon_rmap() will mark the given subpage shared, which cannot fail because they cannot get pinned.
c) HW poison entries
PG_anon_exclusive will remain untouched and is stale -- the page table entry is just a placeholder after all.
d) Migration entries
Writable and readable-exclusive entries are converted to readable
entries: possibly shared.
I.8. mprotect() handling
mprotect() only has to properly handle the new readable-exclusive migration entry:
When write-protecting a migration entry that points at an anonymous page, remember the information about exclusivity via the "readable-exclusive" migration entry type.
II. Migration and GUP-fast
Whenever replacing a present page table entry that maps an exclusive anonymous page by a migration entry, we have to mark the page possibly shared and synchronize against GUP-fast by a proper clear/invalidate+flush to make the following scenario impossible:
1. try_to_migrate() places a migration entry after checking for GUP pins and marks the page possibly shared.
2. GUP-fast pins the page due to lack of synchronization
3. fork() converts the "writable/readable-exclusive" migration entry into a readable migration entry
4. Migration fails due to the GUP pin (failing to freeze the refcount)
5. Migration entries are restored. PG_anon_exclusive is lost
-> We have a pinned page that is not marked exclusive anymore.
Note that we move information about exclusivity from the page to the migration entry as it otherwise highly overcomplicates fork() and PTE-mapping a THP.
III. Swapout and GUP-fast
Whenever replacing a present page table entry that maps an exclusive anonymous page by a swap entry, we have to mark the page possibly shared and synchronize against GUP-fast by a proper clear/invalidate+flush to make the following scenario impossible:
1. try_to_unmap() places a swap entry after checking for GUP pins and clears exclusivity information on the page.
2. GUP-fast pins the page due to lack of synchronization.
-> We have a pinned page that is not marked exclusive anymore.
If we'd ever store information about exclusivity in the swap entry, similar to migration handling, the same considerations as in II would apply. This is future work.
--- include/linux/rmap.h | 40 +++++++++++++++++++++ include/linux/swap.h | 15 +++++--- include/linux/swapops.h | 25 +++++++++++++ mm/huge_memory.c | 78 +++++++++++++++++++++++++++++++++++++---- mm/hugetlb.c | 15 +++++--- mm/ksm.c | 13 ++++++- mm/memory.c | 33 ++++++++++++----- mm/migrate.c | 14 ++++++-- mm/migrate_device.c | 21 ++++++++++- mm/mprotect.c | 8 +++-- mm/rmap.c | 61 +++++++++++++++++++++++++++++--- 11 files changed, 289 insertions(+), 34 deletions(-)
include/linux/rmap.h +40 -0
@@ -228,6 +228,13 @@ static inline int page_try_dup_anon_rmap(struct page *page, bool compound, {VM_BUG_ON_PAGE (!PageAnon (page), page); +/* +* No need to check+clear for already shared pages, including KSM +* pages. +*/ +if (!PageAnonExclusive (page)) + goto dup; +/* * If this page may have been pinned by the parent process, * don't allow to duplicate the mapping but instead require to e.g.,
@@ -239,14 +246,47 @@ static inline int page_try_dup_anon_rmap(struct page *page, bool compound, unlikely(page_needs_cow_for_dma(vma, page)))) return -EBUSY; +ClearPageAnonExclusive (page);/* * It's okay to share the anon page between both processes, mapping * the page R/O into both processes. */ +dup:__page_dup_rmap (page, compound);return 0 ; } +/** +* page_try_share_anon_rmap - try marking an exclusive anonymous page possibly +* shared to prepare for KSM or temporary unmapping +* @page: the exclusive anonymous page to try marking possibly shared +* +* The caller needs to hold the PT lock and has to have the page table entry +* cleared/invalidated+flushed, to properly sync against GUP-fast. +* +* This is similar to page_try_dup_anon_rmap(), however, not used during fork() +* to duplicate a mapping, but instead to prepare for KSM or temporarily +* unmapping a page (swap, migration) via page_remove_rmap(). +* +* Marking the page shared can only fail if the page may be pinned; device +* private pages cannot get pinned and consequently this function cannot fail. +* +* Returns 0 if marking the page possibly shared succeeded. Returns -EBUSY +* otherwise. +*/ +static inline int page_try_share_anon_rmap (struct page *page) +{ +VM_BUG_ON_PAGE (!PageAnon (page) || !PageAnonExclusive (page), page); + +/* See page_try_dup_anon_rmap(). */ +if (likely (!is_device_private_page (page) && +unlikely (page_maybe_dma_pinned (page)))) +return -EBUSY ; + +ClearPageAnonExclusive (page); +return 0 ; +} +/* * Called from mm/vmscan.c to handle paging out */
include/linux/swap.h +11 -4
@@ -78,12 +78,19 @@ static inline int current_is_kswapd(void)#endif /* -* NUMA node memory migration support +* Page migration support. +* +* SWP_MIGRATION_READ_EXCLUSIVE is only applicable to anonymous pages and +* indicates that the referenced (part of) an anonymous page is exclusive to +* a single process. For SWP_MIGRATION_WRITE, that information is implicit: +* (part of) an anonymous page that are mapped writable are exclusive to a +* single process. */ #ifdef CONFIG_MIGRATION -#define SWP_MIGRATION_NUM 2 -#define SWP_MIGRATION_READ (MAX_SWAPFILES + SWP_HWPOISON_NUM) -#define SWP_MIGRATION_WRITE (MAX_SWAPFILES + SWP_HWPOISON_NUM + 1) +#define SWP_MIGRATION_NUM 3 +#define SWP_MIGRATION_READ (MAX_SWAPFILES + SWP_HWPOISON_NUM) +#define SWP_MIGRATION_READ_EXCLUSIVE (MAX_SWAPFILES + SWP_HWPOISON_NUM + 1) +#define SWP_MIGRATION_WRITE (MAX_SWAPFILES + SWP_HWPOISON_NUM + 2)#else #define SWP_MIGRATION_NUM 0#endif
include/linux/swapops.h +25 -0
@@ -194,6 +194,7 @@ static inline bool is_writable_device_exclusive_entry(swp_entry_t entry)static inline int is_migration_entry (swp_entry_t entry) {return unlikely (swp_type (entry) ==SWP_MIGRATION_READ || +swp_type (entry) ==SWP_MIGRATION_READ_EXCLUSIVE ||swp_type (entry) ==SWP_MIGRATION_WRITE ); }
@@ -202,11 +203,26 @@ static inline int is_writable_migration_entry(swp_entry_t entry)return unlikely (swp_type (entry) ==SWP_MIGRATION_WRITE ); } +static inline int is_readable_migration_entry (swp_entry_t entry) +{ +return unlikely (swp_type (entry) ==SWP_MIGRATION_READ ); +} + +static inline int is_readable_exclusive_migration_entry (swp_entry_t entry) +{ +return unlikely (swp_type (entry) ==SWP_MIGRATION_READ_EXCLUSIVE ); +} +static inline swp_entry_t make_readable_migration_entry (pgoff_t offset) {return swp_entry (SWP_MIGRATION_READ , offset); } +static inline swp_entry_t make_readable_exclusive_migration_entry (pgoff_t offset) +{ +return swp_entry (SWP_MIGRATION_READ_EXCLUSIVE , offset); +} +static inline swp_entry_t make_writable_migration_entry (pgoff_t offset) {return swp_entry (SWP_MIGRATION_WRITE , offset);
@@ -224,6 +240,11 @@ static inline swp_entry_t make_readable_migration_entry(pgoff_t offset)return swp_entry (0 ,0 ); } +static inline swp_entry_t make_readable_exclusive_migration_entry (pgoff_t offset) +{ +return swp_entry (0 ,0 ); +} +static inline swp_entry_t make_writable_migration_entry (pgoff_t offset) {return swp_entry (0 ,0 );
@@ -244,6 +265,10 @@ static inline int is_writable_migration_entry(swp_entry_t entry) {return 0 ; } +static inline int is_readable_migration_entry (swp_entry_t entry) +{ +return 0 ; +}#endif
mm/huge_memory.c +71 -7
179 lines not on this page · raw message
mm/hugetlb.c +11 -4
@@ -4769,7 +4769,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, is_hugetlb_entry_hwpoisoned(entry))) {swp_entry_t swp_entry =pte_to_swp_entry (entry); -if (is_writable_migration_entry (swp_entry) && cow) { +if (!is_readable_migration_entry (swp_entry) && cow) {/* * COW mappings require pages in both * parent and child to be set to read.
@@ -5169,6 +5169,8 @@ static vm_fault_t hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma,set_huge_ptep_writable (vma, haddr, ptep);return 0 ; } +VM_BUG_ON_PAGE (PageAnon (old_page) &&PageAnonExclusive (old_page), + old_page);/* * If the process that created a MAP_PRIVATE mapping is about to
@@ -6166,12 +6168,17 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, }if (unlikely (is_hugetlb_entry_migration (pte))) {swp_entry_t entry =pte_to_swp_entry (pte); +struct page *page =pfn_swap_entry_to_page (entry); -if (is_writable_migration_entry (entry)) { +if (!is_readable_migration_entry (entry)) {pte_t newpte; - entry =make_readable_migration_entry ( -swp_offset (entry)); +if (PageAnon (page)) + entry =make_readable_exclusive_migration_entry ( +swp_offset (entry)); +else + entry =make_readable_migration_entry ( +swp_offset (entry)); newpte =swp_entry_to_pte (entry);set_huge_swap_pte_at (mm, address, ptep, newpte,huge_page_size (h));
mm/ksm.c +12 -1
@@ -866,6 +866,7 @@ static inline struct stable_node *page_stable_node(struct page *page)static inline void set_page_stable_node (struct page *page,struct stable_node *stable_node) { +VM_BUG_ON_PAGE (PageAnon (page) &&PageAnonExclusive (page), page); page->mapping = (void *)((unsigned long )stable_node |PAGE_MAPPING_KSM ); }
@@ -1038,6 +1039,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page,int swapped;int err = -EFAULT ;struct mmu_notifier_range range; +bool anon_exclusive; pvmw.address =page_address_in_vma (page, vma);if (pvmw.address == -EFAULT )
@@ -1055,9 +1057,10 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page,if (WARN_ONCE (!pvmw.pte,"Unexpected PMD mapping?" )) goto out_unlock; + anon_exclusive =PageAnonExclusive (page);if (pte_write (*pvmw.pte) ||pte_dirty (*pvmw.pte) || (pte_protnone (*pvmw.pte) &&pte_savedwrite (*pvmw.pte)) || -mm_tlb_flush_pending (mm)) { + anon_exclusive ||mm_tlb_flush_pending (mm)) {pte_t entry; swapped =PageSwapCache (page);
@@ -1085,6 +1088,12 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page,set_pte_at (mm, pvmw.address, pvmw.pte, entry); goto out_unlock; } + +if (anon_exclusive &&page_try_share_anon_rmap (page)) { +set_pte_at (mm, pvmw.address, pvmw.pte, entry); + goto out_unlock; + } +if (pte_dirty (entry))set_page_dirty (page);
@@ -1143,6 +1152,8 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,pte_unmap_unlock (ptep, ptl); goto out_mn; } +VM_BUG_ON_PAGE (PageAnonExclusive (page), page); +VM_BUG_ON_PAGE (PageAnon (kpage) &&PageAnonExclusive (kpage), kpage);/* * No need to check ksm_use_zero_pages here: we can only have a
mm/memory.c +25 -8
@@ -720,6 +720,8 @@ static void restore_exclusive_pte(struct vm_area_struct *vma,else if (is_writable_device_exclusive_entry (entry)) pte =maybe_mkwrite (pte_mkdirty (pte), vma); +VM_BUG_ON (pte_write (pte) && !(PageAnon (page) &&PageAnonExclusive (page))); +/* * No need to take a page reference as one was already * created when the swap entry was made.
@@ -796,11 +798,12 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, rss[mm_counter (page)]++; -if (is_writable_migration_entry (entry) && +if (!is_readable_migration_entry (entry) &&is_cow_mapping (vm_flags)) {/* -* COW mappings require pages in both -* parent and child to be set to read. +* COW mappings require pages in both parent and child +* to be set to read. A previously exclusive entry is +* now shared. */ entry =make_readable_migration_entry (swp_offset (entry));
@@ -951,6 +954,7 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,ptep_set_wrprotect (src_mm, addr, src_pte); pte =pte_wrprotect (pte); } +VM_BUG_ON (page &&PageAnon (page) &&PageAnonExclusive (page));/* * If it's a shared mapping, mark it clean in
@@ -2949,6 +2953,9 @@ static inline void wp_page_reuse(struct vm_fault *vmf)struct vm_area_struct *vma = vmf->vma;struct page *page = vmf->page;pte_t entry; + +VM_BUG_ON (PageAnon (page) && !PageAnonExclusive (page)); +/* * Clear the pages cpupid information as the existing * information potentially belongs to a now completely
@@ -3273,6 +3280,13 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)if (PageAnon (vmf->page)) {struct page *page = vmf->page; +/* +* If the page is exclusive to this process we must reuse the +* page without further checks. +*/ +if (PageAnonExclusive (page)) + goto reuse; +/* * We have to verify under page lock: these early checks are * just an optimization to avoid locking the page and freeing
@@ -3305,6 +3319,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)*/ page_move_anon_rmap (page, vma);unlock_page (page); +reuse:wp_page_reuse (vmf);return VM_FAULT_WRITE ; } else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
@@ -3692,11 +3707,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* that are certainly not shared because we just allocated them without * exposing them to the swapcache. */ -if ((vmf->flags &FAULT_FLAG_WRITE ) && !PageKsm (page) && - (page != swapcache ||page_count (page) ==1 )) { - pte =maybe_mkwrite (pte_mkdirty (pte), vma); - vmf->flags &= ~FAULT_FLAG_WRITE ; - ret |=VM_FAULT_WRITE ; +if (!PageKsm (page) && (page != swapcache ||page_count (page) ==1 )) { +if (vmf->flags &FAULT_FLAG_WRITE ) { + pte =maybe_mkwrite (pte_mkdirty (pte), vma); + vmf->flags &= ~FAULT_FLAG_WRITE ; + ret |=VM_FAULT_WRITE ; + } rmap_flags |=RMAP_EXCLUSIVE ; }flush_icache_page (vma, page);
@@ -3716,6 +3732,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)page_add_anon_rmap (page, vma, vmf->address, rmap_flags); } +VM_BUG_ON (!PageAnon (page) || (pte_write (pte) && !PageAnonExclusive (page)));set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte);arch_do_swap_page (vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte);
mm/migrate.c +12 -2
@@ -177,6 +177,7 @@ static bool remove_migration_pte(struct folio *folio,DEFINE_FOLIO_VMA_WALK (pvmw, old, vma, addr,PVMW_SYNC |PVMW_MIGRATION );while (page_vma_mapped_walk (&pvmw)) { +rmap_t rmap_flags =RMAP_NONE ;pte_t pte;swp_entry_t entry;struct page *new;
@@ -211,6 +212,9 @@ static bool remove_migration_pte(struct folio *folio,else if (pte_swp_uffd_wp (*pvmw.pte)) pte =pte_mkuffd_wp (pte); +if (folio_test_anon (folio) && !is_readable_migration_entry (entry)) + rmap_flags |=RMAP_EXCLUSIVE ; +if (unlikely (is_device_private_page (new))) {if (pte_write (pte)) entry = make_writable_device_private_entry(
@@ -233,7 +237,7 @@ static bool remove_migration_pte(struct folio *folio, pte =arch_make_huge_pte (pte, shift, vma->vm_flags);if (folio_test_anon (folio))hugepage_add_anon_rmap (new, vma, pvmw.address, -RMAP_NONE ); + rmap_flags);else page_dup_file_rmap (new, true);set_huge_pte_at (vma->vm_mm, pvmw.address, pvmw.pte, pte);
@@ -242,7 +246,7 @@ static bool remove_migration_pte(struct folio *folio, {if (folio_test_anon (folio))page_add_anon_rmap (new, vma, pvmw.address, -RMAP_NONE ); + rmap_flags);else page_add_file_rmap (new, vma, false);set_pte_at (vma->vm_mm, pvmw.address, pvmw.pte, pte);
@@ -519,6 +523,12 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)folio_set_workingset (newfolio);if (folio_test_checked (folio))folio_set_checked (newfolio); +/* +* PG_anon_exclusive (-> PG_mappedtodisk) is always migrated via +* migration entries. We can still have PG_anon_exclusive set on an +* effectively unmapped and unreferenced first sub-pages of an +* anonymous THP: we can simply copy it here via PG_mappedtodisk. +*/ if (folio_test_mappedtodisk (folio))folio_set_mappedtodisk (newfolio);
mm/migrate_device.c +20 -1
@@ -184,15 +184,34 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,* set up a special migration page table entry now. */ if (trylock_page (page)) { +bool anon_exclusive;pte_t swp_pte; + anon_exclusive =PageAnon (page) &&PageAnonExclusive (page); +if (anon_exclusive) { +flush_cache_page (vma, addr,pte_pfn (*ptep)); +ptep_clear_flush (vma, addr, ptep); + +if (page_try_share_anon_rmap (page)) { +set_pte_at (mm, addr, ptep, pte); +unlock_page (page); +put_page (page); + mpfn =0 ; + goto next; + } + }else { +ptep_get_and_clear (mm, addr, ptep); + } + migrate->cpages++; -ptep_get_and_clear (mm, addr, ptep);/* Setup special migration page table entry */ if (mpfn &MIGRATE_PFN_WRITE ) entry =make_writable_migration_entry (page_to_pfn (page)); +else if (anon_exclusive) + entry =make_readable_exclusive_migration_entry ( +page_to_pfn (page));else entry =make_readable_migration_entry (page_to_pfn (page));
mm/mprotect.c +6 -2
@@ -152,6 +152,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, pages++; }else if (is_swap_pte (oldpte)) {swp_entry_t entry =pte_to_swp_entry (oldpte); +struct page *page =pfn_swap_entry_to_page (entry);pte_t newpte;if (is_writable_migration_entry (entry)) {
@@ -159,8 +160,11 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,* A protection check is difficult so * just be safe and disable write */ - entry =make_readable_migration_entry ( -swp_offset (entry)); +if (PageAnon (page)) + entry =make_readable_exclusive_migration_entry ( +swp_offset (entry)); +else + entry =make_readable_migration_entry (swp_offset (entry)); newpte =swp_entry_to_pte (entry);if (pte_swp_soft_dirty (oldpte)) newpte =pte_swp_mksoft_dirty (newpte);
mm/rmap.c +56 -5
158 lines not on this page · raw message
Signature · 1 line
2.35.1
Re: [PATCH v3 12/16] mm: remember exclusively mapped anonymous pages with PG_anon_exclusive
Quoted 35 lines · David Hildenbrand
Let's mark exclusively mapped anonymous pages with
PG_anon_exclusiveas exclusive, and use that information to make GUP pins reliable and stay consistent with the page mapped into the page table even if the page table entry gets write-protected.With that information at hand, we can extend our COW logic to always reuse anonymous pages that are exclusive. For anonymous pages that might be shared, the existing logic applies.
As already documented,
PG_anon_exclusiveis usually only expressive in combination with a page table entry. Especially PTE vs. PMD-mapped anonymous pages require more thought, some examples: due tomremap()we can easily have a single compound page PTE-mapped into multiple page tables exclusively in a single process -- multiple page table locks apply. Further, due toMADV_WIPEONFORKwe might not necessarily write-protect all PTEs, and only some subpages might be pinned. Long story short: once PTE-mapped, we have to track information about exclusivity per sub-page, but until then, we can just track it for the compound page in the head page and not having to update a whole bunch of subpages all of the time for a simple PMD mapping of a THP.For simplicity, this commit mostly talks about "anonymous pages", while it's for THP actually "the part of an anonymous folio referenced via a page table entry".
To not spill
PG_anon_exclusivecode all over the mm code-base, we let the anon rmap code to handle allPG_anon_exclusivelogic it can easily handle.If a writable, present page table entry points at an anonymous (sub)page, that (sub)page must be
PG_anon_exclusive. If GUP wants to take a reliably pin (FOLL_PIN) on an anonymous page references via a present page table entry, it must only pin ifPG_anon_exclusiveis set for the mapped (sub)page.
Quoted 3 lines
This commit doesn't adjust GUP, so this is only implicitly handled for
FOLL_WRITE, follow-up commits will teach GUP to also respect it forFOLL_PINwithout !FOLL_WRITE, to make all GUP pins of anonymous pages
without FOLL_WRITE ?
Quoted 1 line
fully reliable.
<snip>
include/linux/swapops.h static inline int is_writable_migration_entry(swp_entry_t entry) · line 214
Quoted 13 lines
include/linux/swapops.h +9
-0@@ -202,11 +203,26 @@ static inline int is_writable_migration_entry(swp_entry_t entry)return unlikely (swp_type (entry) ==SWP_MIGRATION_WRITE ); } +static inline int is_readable_migration_entry (swp_entry_t entry) +{ +return unlikely (swp_type (entry) ==SWP_MIGRATION_READ ); +} + +static inline int is_readable_exclusive_migration_entry (swp_entry_t entry) +{ +return unlikely (swp_type (entry) ==SWP_MIGRATION_READ_EXCLUSIVE ); +}
This one seems to be missing a !CONFIG_MIGRATION counterpart. Although the only caller __split_huge_pmd_locked() probably indirectly only exists with CONFIG_MIGRATION so it's not an immediate issue. (THP selects COMPACTION selects MIGRATION)
<snip>
mm/huge_memory.c void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, · line 3090
Quoted 9 lines
mm/huge_memory.c +5
-0@@ -3035,10 +3083,19 @@ void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,flush_cache_range (vma, address, address +HPAGE_PMD_SIZE ); pmdval =pmdp_invalidate (vma, address, pvmw->pmd); + + anon_exclusive =PageAnon (page) &&PageAnonExclusive (page); +if (anon_exclusive &&page_try_share_anon_rmap (page)) { +set_pmd_at (mm, address, pvmw->pmd, pmdval); +return ;
I am admittedly not too familiar with this code, but looks like this means we fail to migrate the THP, right? But we don't seem to be telling the caller, which is try_to_migrate_one(), so it will continue and not terminate the walk and return false?
Quoted 11 lines
mm/huge_memory.c +4
-0+ } +if (pmd_dirty (pmdval))set_page_dirty (page);if (pmd_write (pmdval)) entry =make_writable_migration_entry (page_to_pfn (page)); +else if (anon_exclusive) + entry =make_readable_exclusive_migration_entry (page_to_pfn (page));else entry =make_readable_migration_entry (page_to_pfn (page)); pmdswp =swp_entry_to_pmd (entry);
<snip>
mm/rmap.c static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, · line 1966
Quoted 13 lines
mm/rmap.c +9
-0@@ -1918,6 +1955,15 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,page_vma_mapped_walk_done (&pvmw);break ; } +VM_BUG_ON_PAGE (pte_write (pteval) &&folio_test_anon (folio) && + !anon_exclusive, subpage); +if (anon_exclusive && +page_try_share_anon_rmap (subpage)) { +set_pte_at (mm, address, pvmw.pte, pteval); + ret = false; +page_vma_mapped_walk_done (&pvmw); +break ; + }
Yeah for the PTE version it seems to do what I'd expect.
Quoted 2 lines
(quoted diff) +0
-0/* * Store the pfn of the page in a special migration
Re: [PATCH v3 12/16] mm: remember exclusively mapped anonymous pages with PG_anon_exclusive
Quoted 36 lines · Vlastimil Babka
Quoted 35 lines · David Hildenbrand
Let's mark exclusively mapped anonymous pages with
PG_anon_exclusiveas exclusive, and use that information to make GUP pins reliable and stay consistent with the page mapped into the page table even if the page table entry gets write-protected.With that information at hand, we can extend our COW logic to always reuse anonymous pages that are exclusive. For anonymous pages that might be shared, the existing logic applies.
As already documented,
PG_anon_exclusiveis usually only expressive in combination with a page table entry. Especially PTE vs. PMD-mapped anonymous pages require more thought, some examples: due tomremap()we can easily have a single compound page PTE-mapped into multiple page tables exclusively in a single process -- multiple page table locks apply. Further, due toMADV_WIPEONFORKwe might not necessarily write-protect all PTEs, and only some subpages might be pinned. Long story short: once PTE-mapped, we have to track information about exclusivity per sub-page, but until then, we can just track it for the compound page in the head page and not having to update a whole bunch of subpages all of the time for a simple PMD mapping of a THP.For simplicity, this commit mostly talks about "anonymous pages", while it's for THP actually "the part of an anonymous folio referenced via a page table entry".
To not spill
PG_anon_exclusivecode all over the mm code-base, we let the anon rmap code to handle allPG_anon_exclusivelogic it can easily handle.If a writable, present page table entry points at an anonymous (sub)page, that (sub)page must be
PG_anon_exclusive. If GUP wants to take a reliably pin (FOLL_PIN) on an anonymous page references via a present page table entry, it must only pin ifPG_anon_exclusiveis set for the mapped (sub)page.
Quoted 5 lines
Quoted 3 lines
This commit doesn't adjust GUP, so this is only implicitly handled for
FOLL_WRITE, follow-up commits will teach GUP to also respect it forFOLL_PINwithout !FOLL_WRITE, to make all GUP pins of anonymous pageswithout
FOLL_WRITE?
Indeed, thanks.
Quoted 9 lines
Quoted 2 lines
fully reliable.
<snip>
Quoted 4 lines
include/linux/swapops.h +0
-0@@ -202,11 +203,26 @@ static inline int is_writable_migration_entry(swp_entry_t entry)return unlikely (swp_type (entry) ==SWP_MIGRATION_WRITE ); }
Quoted 14 lines
include/linux/swapops.h static inline int is_writable_migration_entry(swp_entry_t entry) · line 214
Quoted 9 lines
include/linux/swapops.h +9
-0+static inline int is_readable_migration_entry (swp_entry_t entry) +{ +return unlikely (swp_type (entry) ==SWP_MIGRATION_READ ); +} + +static inline int is_readable_exclusive_migration_entry (swp_entry_t entry) +{ +return unlikely (swp_type (entry) ==SWP_MIGRATION_READ_EXCLUSIVE ); +}This one seems to be missing a !CONFIG_MIGRATION counterpart. Although the only caller
__split_huge_pmd_locked()probably indirectly only exists withCONFIG_MIGRATIONso it's not an immediate issue. (THP selects COMPACTION selects MIGRATION)
So far no builds bailed out. And yes, I think it's for the reason stated. THP without compaction would be a lost bet.
Quoted 5 lines
<snip>
Quoted 2 lines
(quoted diff) +0
-0@@ -3035,10 +3083,19 @@ void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,
Quoted 12 lines
mm/huge_memory.c void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, · line 3090
Quoted 7 lines
mm/huge_memory.c +5
-0flush_cache_range (vma, address, address +HPAGE_PMD_SIZE ); pmdval =pmdp_invalidate (vma, address, pvmw->pmd); + + anon_exclusive =PageAnon (page) &&PageAnonExclusive (page); +if (anon_exclusive &&page_try_share_anon_rmap (page)) { +set_pmd_at (mm, address, pvmw->pmd, pmdval); +return ;I am admittedly not too familiar with this code, but looks like this means we fail to migrate the THP, right? But we don't seem to be telling the caller, which is
try_to_migrate_one(), so it will continue and not terminate the walk and return false?
Right, we're not returning "false". Returning "false" would be an optimization to make rmap_walk_anon() fail faster.
But, after all, the THP is exclusive (-> single mapping), so anon_vma_interval_tree_foreach() would most probably not have a lot work to do either way I'd assume?
In any case, once we return from try_to_migrate(), the page will still be mapped.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v3 12/16] mm: remember exclusively mapped anonymous pages with PG_anon_exclusive
Quoted 15 lines · David Hildenbrand
Quoted 14 lines
mm/huge_memory.c void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, · line 3090
Quoted 9 lines
mm/huge_memory.c +5
-0@@ -3035,10 +3083,19 @@ void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,flush_cache_range (vma, address, address +HPAGE_PMD_SIZE ); pmdval =pmdp_invalidate (vma, address, pvmw->pmd); + + anon_exclusive =PageAnon (page) &&PageAnonExclusive (page); +if (anon_exclusive &&page_try_share_anon_rmap (page)) { +set_pmd_at (mm, address, pvmw->pmd, pmdval); +return ;I am admittedly not too familiar with this code, but looks like this means we fail to migrate the THP, right? But we don't seem to be telling the caller, which is
try_to_migrate_one(), so it will continue and not terminate the walk and return false?
Quoted 2 lines
Right, we're not returning "false". Returning "false" would be an optimization to make
rmap_walk_anon()fail faster.
Ah right, that's what I missed, it's an optimization and we will realize elsewhere afterwards that the page has still mappings and we can't migrate...
Quoted 7 lines
But, after all, the THP is exclusive (-> single mapping), so
anon_vma_interval_tree_foreach()would most probably not have a lot work to do either way I'd assume?In any case, once we return from
try_to_migrate(), the page will still be mapped.
Re: [PATCH v3 12/16] mm: remember exclusively mapped anonymous pages with PG_anon_exclusive
Quoted 16 lines · Vlastimil Babka
Quoted 15 lines · David Hildenbrand
Quoted 14 lines
Review void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, · line 3090
Quoted 9 lines
(quoted diff) +5
-0@@ -3035,10 +3083,19 @@ void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, flush_cache_range(vma, address, address + HPAGE_PMD_SIZE); pmdval = pmdp_invalidate(vma, address, pvmw->pmd); + + anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + if (anon_exclusive && page_try_share_anon_rmap(page)) { + set_pmd_at(mm, address, pvmw->pmd, pmdval); + return;I am admittedly not too familiar with this code, but looks like this means we fail to migrate the THP, right? But we don't seem to be telling the caller, which is
try_to_migrate_one(), so it will continue and not terminate the walk and return false?
Quoted 5 lines
Quoted 2 lines
Right, we're not returning "false". Returning "false" would be an optimization to make
rmap_walk_anon()fail faster.Ah right, that's what I missed, it's an optimization and we will realize elsewhere afterwards that the page has still mappings and we can't migrate...
I'll include that patch in v4 (to be tested):
From 08fb0e45404e3d0f85c2ad23a473e95053396376 Mon Sep 17 00:00:00 2001
From: David Hildenbrand <[email protected]>
Date: Tue, 19 Apr 2022 18:39:23 +0200
Subject: [PATCH] mm/rmap: fail try_to_migrate() early when setting a PMD
migration entry fails
Let's fail right away in case we cannot clear PG_anon_exclusive because the anon THP may be pinned. Right now, we continue trying to install migration entries and the caller of try_to_migrate() will realize that the page is still mapped and has to restore the migration entries. Let's just fail fast just like for PTE migration entries.
--- include/linux/swapops.h | 4 ++-- mm/huge_memory.c | 8 +++++--- mm/rmap.c | 6 +++++- 3 files changed, 12 insertions(+), 6 deletions(-)
include/linux/swapops.h +2 -2
@@ -299,7 +299,7 @@ static inline bool is_pfn_swap_entry(swp_entry_t entry)struct page_vma_mapped_walk ;#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION -extern void set_pmd_migration_entry (struct page_vma_mapped_walk *pvmw, +extern int set_pmd_migration_entry (struct page_vma_mapped_walk *pvmw,struct page *page); extern void remove_migration_pmd(struct page_vma_mapped_walk *pvmw,
@@ -332,7 +332,7 @@ static inline int is_pmd_migration_entry(pmd_t pmd) return !pmd_present(pmd) && is_migration_entry(pmd_to_swp_entry(pmd)); }#else -static inline void set_pmd_migration_entry (struct page_vma_mapped_walk *pvmw, +static inline int set_pmd_migration_entry (struct page_vma_mapped_walk *pvmw,struct page *page) {BUILD_BUG ();
mm/huge_memory.c +5 -3
@@ -3080,7 +3080,7 @@ late_initcall(split_huge_pages_debugfs);#endif #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION -void set_pmd_migration_entry (struct page_vma_mapped_walk *pvmw, +int set_pmd_migration_entry (struct page_vma_mapped_walk *pvmw,struct page *page) {struct vm_area_struct *vma = pvmw->vma;
@@ -3092,7 +3092,7 @@ void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,pmd_t pmdswp;if (!(pvmw->pmd && !pvmw->pte)) -return ; +return 0 ;flush_cache_range (vma, address, address +HPAGE_PMD_SIZE ); pmdval =pmdp_invalidate (vma, address, pvmw->pmd);
@@ -3100,7 +3100,7 @@ void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, anon_exclusive =PageAnon (page) &&PageAnonExclusive (page);if (anon_exclusive &&page_try_share_anon_rmap (page)) {set_pmd_at (mm, address, pvmw->pmd, pmdval); -return ; +return -EBUSY ; }if (pmd_dirty (pmdval))
@@ -3118,6 +3118,8 @@ void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,page_remove_rmap (page, vma, true);put_page (page);trace_set_migration_pmd (address,pmd_val (pmdswp)); + +return 0 ; }void remove_migration_pmd (struct page_vma_mapped_walk *pvmw,struct page *new)
mm/rmap.c +5 -1
@@ -1814,7 +1814,11 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,VM_BUG_ON_FOLIO (folio_test_hugetlb (folio) || !folio_test_pmd_mappable (folio), folio); -set_pmd_migration_entry (&pvmw, subpage); +if (set_pmd_migration_entry (&pvmw, subpage)) { + ret = false; +page_vma_mapped_walk_done (&pvmw); +break ; + } continue; }#endif
Signature · 8 lines
2.35.1 -- Thanks, David / dhildenb
Re: [PATCH v3 12/16] mm: remember exclusively mapped anonymous pages with PG_anon_exclusive
In reply to David HildenbrandQuoted 211 lines · David Hildenbrand
Let's mark exclusively mapped anonymous pages with
PG_anon_exclusiveas exclusive, and use that information to make GUP pins reliable and stay consistent with the page mapped into the page table even if the page table entry gets write-protected.With that information at hand, we can extend our COW logic to always reuse anonymous pages that are exclusive. For anonymous pages that might be shared, the existing logic applies.
As already documented,
PG_anon_exclusiveis usually only expressive in combination with a page table entry. Especially PTE vs. PMD-mapped anonymous pages require more thought, some examples: due tomremap()we can easily have a single compound page PTE-mapped into multiple page tables exclusively in a single process -- multiple page table locks apply. Further, due toMADV_WIPEONFORKwe might not necessarily write-protect all PTEs, and only some subpages might be pinned. Long story short: once PTE-mapped, we have to track information about exclusivity per sub-page, but until then, we can just track it for the compound page in the head page and not having to update a whole bunch of subpages all of the time for a simple PMD mapping of a THP.For simplicity, this commit mostly talks about "anonymous pages", while it's for THP actually "the part of an anonymous folio referenced via a page table entry".
To not spill
PG_anon_exclusivecode all over the mm code-base, we let the anon rmap code to handle allPG_anon_exclusivelogic it can easily handle.If a writable, present page table entry points at an anonymous (sub)page, that (sub)page must be
PG_anon_exclusive. If GUP wants to take a reliably pin (FOLL_PIN) on an anonymous page references via a present page table entry, it must only pin ifPG_anon_exclusiveis set for the mapped (sub)page.This commit doesn't adjust GUP, so this is only implicitly handled for
FOLL_WRITE, follow-up commits will teach GUP to also respect it forFOLL_PINwithout !FOLL_WRITE, to make all GUP pins of anonymous pages fully reliable.Whenever an anonymous page is to be shared (
fork(), KSM), or when temporarily unmapping an anonymous page (swap, migration), the relevantPG_anon_exclusivebit has to be cleared to mark the anonymous page possibly shared. Clearing will fail if there are GUP pins on the page:
* Forfork(), this means having to copy the page and not being able to share it.fork()protects against concurrent GUP using the PT lock and thesrc_mm->write_protect_seq.
* For KSM, this means sharing will fail. For swap this means, unmapping will fail, For migration this means, migration will fail early. All three cases protect against concurrent GUP using the PT lock and a proper clear/invalidate+flush of the relevant page table entry.This fixes memory corruptions reported for
FOLL_PIN|FOLL_WRITE, when a pinned page gets mapped R/O and the successive write fault ends up replacing the page instead of reusing it. It improves the situation for O_DIRECT/vmsplice/... that still useFOLL_GETinstead ofFOLL_PIN, iffork()is *not* involved, however swapout andfork()are still problematic. Properly usingFOLL_PINinstead ofFOLL_GETfor these GUP users will fix the issue for them.I. Details about basic handling
I.1. Fresh anonymous pages
page_add_new_anon_rmap()andhugepage_add_new_anon_rmap()will mark the given page exclusive via__page_set_anon_rmap(exclusive=1). As that is the mechanism fresh anonymous pages come into life (besides migration code where we copy thepage->mapping), all fresh anonymous pages will start out as exclusive.I.2. COW reuse handling of anonymous pages
When a COW handler stumbles over a (sub)page that's marked exclusive, it simply reuses it. Otherwise, the handler tries harder under page lock to detect if the (sub)page is exclusive and can be reused. If exclusive,
page_move_anon_rmap()will mark the given (sub)page exclusive.Note that hugetlb code does not yet check for
PageAnonExclusive(), as it still uses the old COW logic that is prone to the COW security issue because hugetlb code cannot really tolerate unnecessary/wrong COW as huge pages are a scarce resource.I.3. Migration handling
try_to_migrate()has to try marking an exclusive anonymous page shared viapage_try_share_anon_rmap(). If it fails because there are GUP pins on the page, unmap fails.migrate_vma_collect_pmd()and__split_huge_pmd_locked()are handled similarly.Writable migration entries implicitly point at shared anonymous pages. For readable migration entries that information is stored via a new "readable-exclusive" migration entry, specific to anonymous pages.
When restoring a migration entry in
remove_migration_pte(), information about exlusivity is detected via the migration entry type, andRMAP_EXCLUSIVEis set accordingly for page_add_anon_rmap()/hugepage_add_anon_rmap() to restore that information.I.4. Swapout handling
try_to_unmap()has to try marking the mapped page possibly shared viapage_try_share_anon_rmap(). If it fails because there are GUP pins on the page, unmap fails. For now, information about exclusivity is lost. In the future, we might want to remember that information in the swap entry in some cases, however, it requires more thought, care, and a way to store that information in swap entries.I.5. Swapin handling
do_swap_page()will never stumble over exclusive anonymous pages in the swap cache, astry_to_migrate()prohibits that.do_swap_page()always has to detect manually if an anonymous page is exclusive and has to setRMAP_EXCLUSIVEforpage_add_anon_rmap()accordingly.I.6. THP handling
__split_huge_pmd_locked()has to move the information about exclusivity from the PMD to the PTEs.a) In case we have a readable-exclusive PMD migration entry, simply insert readable-exclusive PTE migration entries.
b) In case we have a present PMD entry and we don't want to freeze ("convert to migration entries"), simply forward
PG_anon_exclusiveto all sub-pages, no need to temporarily clear the bit.c) In case we have a present PMD entry and want to freeze, handle it similar to
try_to_migrate(): try marking the page shared first. In case we fail, we ignore the "freeze" instruction and simply split ordinarily.try_to_migrate()will properly fail because the THP is still mapped via PTEs.When splitting a compound anonymous folio (THP), the information about exclusivity is implicitly handled via the migration entries: no need to replicate
PG_anon_exclusivemanually.I.7.
fork()handling
fork()handling is relatively easy, becausePG_anon_exclusiveis only expressive for some page table entry types.a) Present anonymous pages
page_try_dup_anon_rmap()will mark the given subpage shared -- which will fail if the page is pinned. If it failed, we have to copy (or PTE-map a PMD to handle it on the PTE level).Note that device exclusive entries are just a pointer at a
PageAnon()page.fork()will first convert a device exclusive entry to a present page table and handle it just like present anonymous pages.b) Device private entry
Device private entries point at
PageAnon()pages that cannot be mapped directly and, therefore, cannot get pinned.
page_try_dup_anon_rmap()will mark the given subpage shared, which cannot fail because they cannot get pinned.c) HW poison entries
PG_anon_exclusivewill remain untouched and is stale -- the page table entry is just a placeholder after all.d) Migration entries
Writable and readable-exclusive entries are converted to readable
entries: possibly shared.I.8.
mprotect()handling
mprotect()only has to properly handle the new readable-exclusive migration entry:When write-protecting a migration entry that points at an anonymous page, remember the information about exclusivity via the "readable-exclusive" migration entry type.
II. Migration and GUP-fast
Whenever replacing a present page table entry that maps an exclusive anonymous page by a migration entry, we have to mark the page possibly shared and synchronize against GUP-fast by a proper clear/invalidate+flush to make the following scenario impossible:
1.
try_to_migrate()places a migration entry after checking for GUP pins and marks the page possibly shared.
2. GUP-fast pins the page due to lack of synchronization
3.fork()converts the "writable/readable-exclusive" migration entry into a readable migration entry
4. Migration fails due to the GUP pin (failing to freeze the refcount)
5. Migration entries are restored.PG_anon_exclusiveis lost-> We have a pinned page that is not marked exclusive anymore.
Note that we move information about exclusivity from the page to the migration entry as it otherwise highly overcomplicates
fork()and PTE-mapping a THP.III. Swapout and GUP-fast
Whenever replacing a present page table entry that maps an exclusive anonymous page by a swap entry, we have to mark the page possibly shared and synchronize against GUP-fast by a proper clear/invalidate+flush to make the following scenario impossible:
1.
try_to_unmap()places a swap entry after checking for GUP pins and clears exclusivity information on the page.
2. GUP-fast pins the page due to lack of synchronization.
Quoted 7 lines
-> We have a pinned page that is not marked exclusive anymore.
If we'd ever store information about exclusivity in the swap entry, similar to migration handling, the same considerations as in II would apply. This is future work.
Signed-off-by David Hildenbrand <[email protected]>
PATCH v3 13/16 mm/gup: disallow follow_page(FOLL_PIN)
In reply to David HildenbrandWe want to change the way we handle R/O pins on anonymous pages that might be shared: if we detect a possibly shared anonymous page -- mapped R/O and not !PageAnonExclusive() -- we want to trigger unsharing via a page fault, resulting in an exclusive anonymous page that can be pinned reliably without getting replaced via COW on the next write fault.
However, the required page fault will be problematic for follow_page():
in contrast to ordinary GUP, follow_page() doesn't trigger faults internally. So we would have to end up failing a R/O pin via follow_page(), although there is something mapped R/O into the page table, which might be rather surprising.
We don't seem to have follow_page(FOLL_PIN) users, and it's a purely internal MM function. Let's just make our life easier and the semantics of follow_page() clearer by just disallowing FOLL_PIN for follow_page() completely.
--- mm/gup.c | 3 +++ mm/hugetlb.c | 8 +++++--- 2 files changed, 8 insertions(+), 3 deletions(-)
mm/gup.c +3 -0
@@ -787,6 +787,9 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,if (vma_is_secretmem (vma))return NULL ; +if (foll_flags &FOLL_PIN ) +return NULL ; + page =follow_page_mask (vma, address, foll_flags, &ctx);if (ctx.pgmap)put_dev_pagemap (ctx.pgmap);
mm/hugetlb.c +5 -3
@@ -6698,9 +6698,11 @@ follow_huge_pmd(struct mm_struct *mm, unsigned long address,spinlock_t *ptl;pte_t pte; -/* FOLL_GET and FOLL_PIN are mutually exclusive. */ -if (WARN_ON_ONCE ((flags & (FOLL_PIN |FOLL_GET )) == - (FOLL_PIN |FOLL_GET ))) +/* +* FOLL_PIN is not supported for follow_page(). Ordinary GUP goes via +* follow_hugetlb_page(). +*/ +if (WARN_ON_ONCE (flags &FOLL_PIN ))return NULL ; retry:
Signature · 1 line
2.35.1
Re: [PATCH v3 13/16] mm/gup: disallow follow_page(FOLL_PIN)
Quoted 13 lines · David Hildenbrand
We want to change the way we handle R/O pins on anonymous pages that might be shared: if we detect a possibly shared anonymous page -- mapped R/O and not !PageAnonExclusive() -- we want to trigger unsharing via a page fault, resulting in an exclusive anonymous page that can be pinned reliably without getting replaced via COW on the next write fault.
However, the required page fault will be problematic for
follow_page():
in contrast to ordinary GUP,follow_page()doesn't trigger faults internally. So we would have to end up failing a R/O pin viafollow_page(), although there is something mapped R/O into the page table, which might be rather surprising.
Quoted 6 lines
We don't seem to have
follow_page(FOLL_PIN)users, and it's a purely internal MM function. Let's just make our life easier and the semantics offollow_page()clearer by just disallowingFOLL_PINforfollow_page()completely.Signed-off-by David Hildenbrand <[email protected]>
PATCH v3 14/16 mm: support GUP-triggered unsharing of anonymous pages
In reply to David HildenbrandWhenever GUP currently ends up taking a R/O pin on an anonymous page that might be shared -- mapped R/O and !PageAnonExclusive() -- any write fault on the page table entry will end up replacing the mapped anonymous page due to COW, resulting in the GUP pin no longer being consistent with the page actually mapped into the page table.
The possible ways to deal with this situation are:
(1) Ignore and pin -- what we do right now.
(2) Fail to pin -- which would be rather surprising to callers and could break user space.
(3) Trigger unsharing and pin the now exclusive page -- reliable R/O pins.
We want to implement 3) because it provides the clearest semantics and allows for checking in unpin_user_pages() and friends for possible BUGs:
when trying to unpin a page that's no longer exclusive, clearly something went very wrong and might result in memory corruptions that might be hard to debug. So we better have a nice way to spot such issues.
To implement 3), we need a way for GUP to trigger unsharing:FAULT_FLAG_UNSHARE. FAULT_FLAG_UNSHARE is only applicable to R/O mapped anonymous pages and resembles COW logic during a write fault. However, in contrast to a write fault, GUP-triggered unsharing will, for example, still maintain the write protection.
Let's implement FAULT_FLAG_UNSHARE by hooking into the existing write fault handlers for all applicable anonymous page types: ordinary pages, THP and hugetlb.
* If FAULT_FLAG_UNSHARE finds a R/O-mapped anonymous page that has been marked exclusive in the meantime by someone else, there is nothing to do.
* If FAULT_FLAG_UNSHARE finds a R/O-mapped anonymous page that's not marked exclusive, it will try detecting if the process is the exclusive owner. If exclusive, it can be set exclusive similar to reuse logic during write faults via page_move_anon_rmap() and there is nothing else to do; otherwise, we either have to copy and map a fresh, anonymous exclusive page R/O (ordinary pages, hugetlb), or split the THP.
This commit is heavily based on patches by Andrea.
--- include/linux/mm_types.h | 8 +++ mm/huge_memory.c | 10 +++- mm/hugetlb.c | 56 ++++++++++++-------- mm/memory.c | 107 +++++++++++++++++++++++++++------------ 4 files changed, 126 insertions(+), 55 deletions(-)
include/linux/mm_types.h +8 -0
@@ -812,6 +812,9 @@ typedef struct {* @FAULT_FLAG_REMOTE: The fault is not for current task/mm. * @FAULT_FLAG_INSTRUCTION: The fault was during an instruction fetch. * @FAULT_FLAG_INTERRUPTIBLE: The fault can be interrupted by non-fatal signals. +* @FAULT_FLAG_UNSHARE: The fault is an unsharing request to unshare (and mark +* exclusive) a possibly shared anonymous page that is +* mapped R/O. * * About @FAULT_FLAG_ALLOW_RETRY and @FAULT_FLAG_TRIED: we can specify * whether we would allow page faults to retry by specifying these two
@@ -831,6 +834,10 @@ typedef struct {* continuous faults with flags (b). We should always try to detect pending * signals before a retry to make sure the continuous page faults can still be * interrupted if necessary. +* +* The combination FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE is illegal. +* FAULT_FLAG_UNSHARE is ignored and treated like an ordinary read fault when +* no existing R/O-mapped anonymous page is encountered. */ enum fault_flag {FAULT_FLAG_WRITE =1 <<0 ,
@@ -843,6 +850,7 @@ enum fault_flag {FAULT_FLAG_REMOTE =1 <<7 ,FAULT_FLAG_INSTRUCTION =1 <<8 ,FAULT_FLAG_INTERRUPTIBLE =1 <<9 , + FAULT_FLAG_UNSHARE = 1 << 10, };#endif /* _LINUX_MM_TYPES_H */
mm/huge_memory.c +9 -1
@@ -1271,6 +1271,7 @@ void huge_pmd_set_accessed(struct vm_fault *vmf)vm_fault_t do_huge_pmd_wp_page (struct vm_fault *vmf) { +const bool unshare = vmf->flags &FAULT_FLAG_UNSHARE ;struct vm_area_struct *vma = vmf->vma;struct page *page;unsigned long haddr = vmf->address &HPAGE_PMD_MASK ;
@@ -1279,6 +1280,9 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) vmf->ptl =pmd_lockptr (vma->vm_mm, vmf->pmd);VM_BUG_ON_VMA (!vma->anon_vma, vma); +VM_BUG_ON (unshare && (vmf->flags &FAULT_FLAG_WRITE )); +VM_BUG_ON (!unshare && !(vmf->flags &FAULT_FLAG_WRITE )); +if (is_huge_zero_pmd (orig_pmd)) goto fallback;
@@ -1317,7 +1321,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) }/* -* See do_wp_page(): we can only map the page writable if there are +* See do_wp_page(): we can only reuse the page exclusively if there are * no additional references. Note that we always drain the LRU * pagevecs immediately after adding a THP. */
@@ -1331,6 +1335,10 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf)page_move_anon_rmap (page, vma);unlock_page (page); reuse: +if (unlikely (unshare)) { +spin_unlock (vmf->ptl); +return 0 ; + } entry =pmd_mkyoung (orig_pmd); entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma);if (pmdp_set_access_flags (vma, haddr, vmf->pmd, entry,1 ))
mm/hugetlb.c +35 -21
138 lines not on this page · raw message
mm/memory.c +74 -33
218 lines not on this page · raw message
Signature · 1 line
2.35.1
Re: [PATCH v3 14/16] mm: support GUP-triggered unsharing of anonymous pages
Quoted 40 lines · David Hildenbrand
Whenever GUP currently ends up taking a R/O pin on an anonymous page that might be shared -- mapped R/O and !PageAnonExclusive() -- any write fault on the page table entry will end up replacing the mapped anonymous page due to COW, resulting in the GUP pin no longer being consistent with the page actually mapped into the page table.
The possible ways to deal with this situation are:
(1) Ignore and pin -- what we do right now.
(2) Fail to pin -- which would be rather surprising to callers and could break user space.
(3) Trigger unsharing and pin the now exclusive page -- reliable R/O pins.We want to implement 3) because it provides the clearest semantics and allows for checking in
unpin_user_pages()and friends for possible BUGs:
when trying to unpin a page that's no longer exclusive, clearly something went very wrong and might result in memory corruptions that might be hard to debug. So we better have a nice way to spot such issues.To implement 3), we need a way for GUP to trigger unsharing:
FAULT_FLAG_UNSHARE.FAULT_FLAG_UNSHAREis only applicable to R/O mapped anonymous pages and resembles COW logic during a write fault. However, in contrast to a write fault, GUP-triggered unsharing will, for example, still maintain the write protection.Let's implement
FAULT_FLAG_UNSHAREby hooking into the existing write fault handlers for all applicable anonymous page types: ordinary pages, THP and hugetlb.* If
FAULT_FLAG_UNSHAREfinds a R/O-mapped anonymous page that has been marked exclusive in the meantime by someone else, there is nothing to do.
* IfFAULT_FLAG_UNSHAREfinds a R/O-mapped anonymous page that's not marked exclusive, it will try detecting if the process is the exclusive owner. If exclusive, it can be set exclusive similar to reuse logic during write faults viapage_move_anon_rmap()and there is nothing else to do; otherwise, we either have to copy and map a fresh, anonymous exclusive page R/O (ordinary pages, hugetlb), or split the THP.
Quoted 5 lines
This commit is heavily based on patches by Andrea.
Co-developed-by Andrea Arcangeli <[email protected]>Signed-off-by Andrea Arcangeli <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Modulo a nit and suspected logical bug below.
<snip>
Quoted 9 lines
mm/memory.c +1
-0@@ -3072,6 +3082,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)* mmu page tables (such as kvm shadow page tables), we want the * new page to be mapped directly into the secondary page table. */ +BUG_ON (unshare &&pte_write (entry));set_pte_at_notify (mm, vmf->address, vmf->pte, entry);update_mmu_cache (vma, vmf->address, vmf->pte);if (old_page) {
mm/memory.c static vm_fault_t wp_page_copy(struct vm_fault *vmf) · line 3135
Quoted 5 lines
mm/memory.c +1
-1@@ -3121,7 +3132,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)free_swap_cache (old_page);put_page (old_page); } -return page_copied ?VM_FAULT_WRITE :0 ; +return page_copied && !unshare ?VM_FAULT_WRITE :0 ;
Could be just me but I would prefer (page_copied && !unshare) as I rarely see these operators together like this to remember their relative priority very well.
Quoted 3 lines
mm/memory.c +0
-0oom_free_new:put_page (new_page); oom:
<snip>
mm/memory.c static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf) · line 4556
Quoted 9 lines
mm/memory.c +3
-1@@ -4515,8 +4550,11 @@ static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf)/* `inline' is required to avoid gcc 4.1.2 build error */ static inline vm_fault_t wp_huge_pmd (struct vm_fault *vmf) { +const bool unshare = vmf->flags &FAULT_FLAG_UNSHARE ; +if (vma_is_anonymous (vmf->vma)) { -if (userfaultfd_huge_pmd_wp (vmf->vma, vmf->orig_pmd)) + if (unlikely(unshare) &&
Is this condition flipped, should it be "likely(!unshare)"? As the similar code in do_wp_page() does.
Quoted 18 lines
mm/memory.c +4
-2+ userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))return handle_userfault (vmf,VM_UFFD_WP );return do_huge_pmd_wp_page (vmf); }@@ -4651,10 +4689,11 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)update_mmu_tlb (vmf->vma, vmf->address, vmf->pte); goto unlock; } -if (vmf->flags &FAULT_FLAG_WRITE ) { +if (vmf->flags & (FAULT_FLAG_WRITE |FAULT_FLAG_UNSHARE )) {if (!pte_write (entry))return do_wp_page (vmf); - entry =pte_mkdirty (entry); +else if (likely (vmf->flags &FAULT_FLAG_WRITE )) + entry =pte_mkdirty (entry); } entry =pte_mkyoung (entry); if (ptep_set_access_flags(vmf->vma, vmf->address, vmf->pte, entry,
Re: [PATCH v3 14/16] mm: support GUP-triggered unsharing of anonymous pages
Quoted 43 lines · Vlastimil Babka
Quoted 42 lines · David Hildenbrand
Whenever GUP currently ends up taking a R/O pin on an anonymous page that might be shared -- mapped R/O and !PageAnonExclusive() -- any write fault on the page table entry will end up replacing the mapped anonymous page due to COW, resulting in the GUP pin no longer being consistent with the page actually mapped into the page table.
The possible ways to deal with this situation are:
(1) Ignore and pin -- what we do right now.
(2) Fail to pin -- which would be rather surprising to callers and could break user space.
(3) Trigger unsharing and pin the now exclusive page -- reliable R/O pins.We want to implement 3) because it provides the clearest semantics and allows for checking in
unpin_user_pages()and friends for possible BUGs:
when trying to unpin a page that's no longer exclusive, clearly something went very wrong and might result in memory corruptions that might be hard to debug. So we better have a nice way to spot such issues.To implement 3), we need a way for GUP to trigger unsharing:
FAULT_FLAG_UNSHARE.FAULT_FLAG_UNSHAREis only applicable to R/O mapped anonymous pages and resembles COW logic during a write fault. However, in contrast to a write fault, GUP-triggered unsharing will, for example, still maintain the write protection.Let's implement
FAULT_FLAG_UNSHAREby hooking into the existing write fault handlers for all applicable anonymous page types: ordinary pages, THP and hugetlb.* If
FAULT_FLAG_UNSHAREfinds a R/O-mapped anonymous page that has been marked exclusive in the meantime by someone else, there is nothing to do.
* IfFAULT_FLAG_UNSHAREfinds a R/O-mapped anonymous page that's not marked exclusive, it will try detecting if the process is the exclusive owner. If exclusive, it can be set exclusive similar to reuse logic during write faults viapage_move_anon_rmap()and there is nothing else to do; otherwise, we either have to copy and map a fresh, anonymous exclusive page R/O (ordinary pages, hugetlb), or split the THP.This commit is heavily based on patches by Andrea.
Quoted 7 lines
Quoted 3 lines
Co-developed-by Andrea Arcangeli <[email protected]>Signed-off-by Andrea Arcangeli <[email protected]>Signed-off-by David Hildenbrand <[email protected]>Acked-by Vlastimil Babka <[email protected]>Modulo a nit and suspected logical bug below.
Thanks!
Quoted 4 lines
Quoted 4 lines
(quoted diff) +0
-0@@ -4515,8 +4550,11 @@ static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf) /* `inline' is required to avoid gcc 4.1.2 build error */ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) {
Quoted 8 lines
Review static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf) · line 4556
Quoted 5 lines
(quoted diff) +3
-1+ const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; + if (vma_is_anonymous(vmf->vma)) { - if (userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) + if (unlikely(unshare) &&Is this condition flipped, should it be "likely(!unshare)"? As the similar code in
do_wp_page()does.
Good catch, this should affect uffd-wp on THP -- it wouldn't trigger as expected. Thanks a lot for finding that!
Quoted 11 lines
Quoted 11 lines
(quoted diff) +2
-1+ userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) return handle_userfault(vmf, VM_UFFD_WP); return do_huge_pmd_wp_page(vmf); }@@ -4651,10 +4689,11 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) update_mmu_tlb(vmf->vma, vmf->address, vmf->pte); goto unlock; } - if (vmf->flags & FAULT_FLAG_WRITE) { + if (vmf->flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) {
Quoted 9 lines
Review static vm_fault_t handle_pte_fault(struct vm_fault *vmf) · line 4699
Quoted 8 lines
(quoted diff) +2
-1if (!pte_write(entry)) return do_wp_page(vmf); - entry = pte_mkdirty(entry); + else if (likely(vmf->flags & FAULT_FLAG_WRITE)) + entry = pte_mkdirty(entry); } entry = pte_mkyoung(entry); if (ptep_set_access_flags(vmf->vma, vmf->address, vmf->pte, entry,
So the following on top, right?
mm/memory.c +2 -2
@@ -3137,7 +3137,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)free_swap_cache (old_page);put_page (old_page); } -return page_copied && !unshare ?VM_FAULT_WRITE :0 ; +return (page_copied && !unshare) ?VM_FAULT_WRITE :0 ; oom_free_new:put_page (new_page); oom:
@@ -4604,7 +4604,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)const bool unshare = vmf->flags &FAULT_FLAG_UNSHARE ;if (vma_is_anonymous (vmf->vma)) { -if (unlikely (unshare) && +if (likely (!unshare) &&userfaultfd_huge_pmd_wp (vmf->vma, vmf->orig_pmd))return handle_userfault (vmf,VM_UFFD_WP );return do_huge_pmd_wp_page (vmf);
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v3 14/16] mm: support GUP-triggered unsharing of anonymous pages
Quoted 13 lines · David Hildenbrand
Quoted 4 lines
Quoted 4 lines
(quoted diff) +0
-0@@ -4515,8 +4550,11 @@ static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf) /* `inline' is required to avoid gcc 4.1.2 build error */ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) {Quoted 8 lines
Review static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf) · line 4556
Quoted 5 lines
(quoted diff) +3
-1+ const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; + if (vma_is_anonymous(vmf->vma)) { - if (userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) + if (unlikely(unshare) &&Is this condition flipped, should it be "likely(!unshare)"? As the similar code in
do_wp_page()does.
Quoted 1 line
Good catch, this should affect uffd-wp on THP -- it wouldn't trigger as expected. Thanks a lot for finding that!
Yay, glad I was right this time.
Quoted 22 lines
Quoted 11 lines
Quoted 11 lines
mm/memory.c +2
-1+ userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))return handle_userfault (vmf,VM_UFFD_WP );return do_huge_pmd_wp_page (vmf); }@@ -4651,10 +4689,11 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)update_mmu_tlb (vmf->vma, vmf->address, vmf->pte); goto unlock; } -if (vmf->flags &FAULT_FLAG_WRITE ) { +if (vmf->flags & (FAULT_FLAG_WRITE |FAULT_FLAG_UNSHARE )) {Quoted 9 lines
Review static vm_fault_t handle_pte_fault(struct vm_fault *vmf) · line 4699
Quoted 8 lines
(quoted diff) +2
-1if (!pte_write(entry)) return do_wp_page(vmf); - entry = pte_mkdirty(entry); + else if (likely(vmf->flags & FAULT_FLAG_WRITE)) + entry = pte_mkdirty(entry); } entry = pte_mkyoung(entry); if (ptep_set_access_flags(vmf->vma, vmf->address, vmf->pte, entry,
Quoted 1 line
So the following on top, right?
Looks good!
Quoted 24 lines
mm/memory.c +2
-2@@ -3137,7 +3137,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)free_swap_cache (old_page);put_page (old_page); } -return page_copied && !unshare ?VM_FAULT_WRITE :0 ; +return (page_copied && !unshare) ?VM_FAULT_WRITE :0 ; oom_free_new:put_page (new_page); oom:@@ -4604,7 +4604,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)const bool unshare = vmf->flags &FAULT_FLAG_UNSHARE ;if (vma_is_anonymous (vmf->vma)) { -if (unlikely (unshare) && +if (likely (!unshare) &&userfaultfd_huge_pmd_wp (vmf->vma, vmf->orig_pmd))return handle_userfault (vmf,VM_UFFD_WP );return do_huge_pmd_wp_page (vmf);
PATCH v3 15/16 mm/gup: trigger FAULT_FLAG_UNSHARE when R/O-pinning a possibly shared anonymous page
In reply to David HildenbrandWhenever GUP currently ends up taking a R/O pin on an anonymous page that might be shared -- mapped R/O and !PageAnonExclusive() -- any write fault on the page table entry will end up replacing the mapped anonymous page due to COW, resulting in the GUP pin no longer being consistent with the page actually mapped into the page table.
The possible ways to deal with this situation are:
(1) Ignore and pin -- what we do right now.
(2) Fail to pin -- which would be rather surprising to callers and could break user space.
(3) Trigger unsharing and pin the now exclusive page -- reliable R/O pins.
Let's implement 3) because it provides the clearest semantics and allows for checking in unpin_user_pages() and friends for possible BUGs:
when trying to unpin a page that's no longer exclusive, clearly something went very wrong and might result in memory corruptions that might be hard to debug. So we better have a nice way to spot such issues.
This change implies that whenever user space *wrote* to a private mapping (IOW, we have an anonymous page mapped), that GUP pins will always remain consistent: reliable R/O GUP pins of anonymous pages.
As a side note, this commit fixes the COW security issue for hugetlb with FOLL_PIN as documented in:
https://lore.kernel.org/r/[email protected]
The vmsplice reproducer still applies, because vmsplice uses FOLL_GET instead of FOLL_PIN.
Note that follow_huge_pmd() doesn't apply because we cannot end up in there with FOLL_PIN.
This commit is heavily based on prototype patches by Andrea.
--- include/linux/mm.h | 39 +++++++++++++++++++++++++++++++++++++++ mm/gup.c | 42 +++++++++++++++++++++++++++++++++++++++--- mm/huge_memory.c | 3 +++ mm/hugetlb.c | 27 ++++++++++++++++++++++++--- 4 files changed, 105 insertions(+), 6 deletions(-)
include/linux/mm.h +39 -0
@@ -3005,6 +3005,45 @@ static inline int vm_fault_to_errno(vm_fault_t vm_fault, int foll_flags)return 0 ; } +/* +* Indicates for which pages that are write-protected in the page table, +* whether GUP has to trigger unsharing via FAULT_FLAG_UNSHARE such that the +* GUP pin will remain consistent with the pages mapped into the page tables +* of the MM. +* +* Temporary unmapping of PageAnonExclusive() pages or clearing of +* PageAnonExclusive() has to protect against concurrent GUP: +* * Ordinary GUP: Using the PT lock +* * GUP-fast and fork(): mm->write_protect_seq +* * GUP-fast and KSM or temporary unmapping (swap, migration): +* clear/invalidate+flush of the page table entry +* +* Must be called with the (sub)page that's actually referenced via the +* page table entry, which might not necessarily be the head page for a +* PTE-mapped THP. +*/ +static inline bool gup_must_unshare (unsigned int flags,struct page *page) +{ +/* +* FOLL_WRITE is implicitly handled correctly as the page table entry +* has to be writable -- and if it references (part of) an anonymous +* folio, that part is required to be marked exclusive. +*/ +if ((flags & (FOLL_WRITE |FOLL_PIN )) !=FOLL_PIN ) +return false; +/* +* Note: PageAnon(page) is stable until the page is actually getting +* freed. +*/ +if (!PageAnon (page)) +return false; +/* +* Note that PageKsm() pages cannot be exclusive, and consequently, +* cannot get pinned. +*/ +return !PageAnonExclusive (page); +} +typedef int (*pte_fn_t )(pte_t *pte,unsigned long addr,void *data);extern int apply_to_page_range (struct mm_struct *mm,unsigned long address,unsigned long size,pte_fn_t fn,void *data);
mm/gup.c +39 -3
96 lines not on this page · raw message
mm/huge_memory.c +3 -0
@@ -1389,6 +1389,9 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, page =pmd_page (*pmd);VM_BUG_ON_PAGE (!PageHead (page) && !is_zone_device_page (page), page); +if (!pmd_write (*pmd) &&gup_must_unshare (flags, page)) +return ERR_PTR (-EMLINK ); +if (!try_grab_page (page, flags))return ERR_PTR (-ENOMEM );
mm/hugetlb.c +24 -3
@@ -5961,6 +5961,25 @@ static void record_subpages_vmas(struct page *page, struct vm_area_struct *vma, } } +static inline bool __follow_hugetlb_must_fault (unsigned int flags,pte_t *pte, +bool *unshare) +{ +pte_t pteval =huge_ptep_get (pte); + + *unshare = false; +if (is_swap_pte (pteval)) +return true; +if (huge_pte_write (pteval)) +return false; +if (flags &FOLL_WRITE ) +return true; +if (gup_must_unshare (flags,pte_page (pteval))) { + *unshare = true; +return true; + } +return false; +} + long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, struct page **pages, struct vm_area_struct **vmas, unsigned long *position, unsigned long *nr_pages,
@@ -5975,6 +5994,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,while (vaddr < vma->vm_end && remainder) {pte_t *pte;spinlock_t *ptl =NULL ; +bool unshare = false;int absent;struct page *page;
@@ -6025,9 +6045,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,* both cases, and because we can't follow correct pages * directly from any kind of swap entries. */ -if (absent ||is_swap_pte (huge_ptep_get (pte)) || - ((flags &FOLL_WRITE ) && - !huge_pte_write (huge_ptep_get (pte)))) { +if (absent || +__follow_hugetlb_must_fault (flags, pte, &unshare)) {vm_fault_t ret;unsigned int fault_flags =0 ;
@@ -6035,6 +6054,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,spin_unlock (ptl);if (flags &FOLL_WRITE ) fault_flags |=FAULT_FLAG_WRITE ; +else if (unshare) + fault_flags |=FAULT_FLAG_UNSHARE ;if (locked) fault_flags |=FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ;
Signature · 1 line
2.35.1
Re: [PATCH v3 15/16] mm/gup: trigger FAULT_FLAG_UNSHARE when R/O-pinning a possibly shared anonymous page
Quoted 30 lines · David Hildenbrand
Whenever GUP currently ends up taking a R/O pin on an anonymous page that might be shared -- mapped R/O and !PageAnonExclusive() -- any write fault on the page table entry will end up replacing the mapped anonymous page due to COW, resulting in the GUP pin no longer being consistent with the page actually mapped into the page table.
The possible ways to deal with this situation are:
(1) Ignore and pin -- what we do right now.
(2) Fail to pin -- which would be rather surprising to callers and could break user space.
(3) Trigger unsharing and pin the now exclusive page -- reliable R/O pins.Let's implement 3) because it provides the clearest semantics and allows for checking in
unpin_user_pages()and friends for possible BUGs:
when trying to unpin a page that's no longer exclusive, clearly something went very wrong and might result in memory corruptions that might be hard to debug. So we better have a nice way to spot such issues.This change implies that whenever user space *wrote* to a private mapping (IOW, we have an anonymous page mapped), that GUP pins will always remain consistent: reliable R/O GUP pins of anonymous pages.
As a side note, this commit fixes the COW security issue for hugetlb with
FOLL_PINas documented in:
https://lore.kernel.org/r/[email protected]
The vmsplice reproducer still applies, because vmsplice usesFOLL_GETinstead ofFOLL_PIN.
Quoted 8 lines
Note that
follow_huge_pmd()doesn't apply because we cannot end up in there withFOLL_PIN.This commit is heavily based on prototype patches by Andrea.
Co-developed-by Andrea Arcangeli <[email protected]>Signed-off-by Andrea Arcangeli <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
PATCH v3 16/16 mm/gup: sanity-check with CONFIG_DEBUG_VM that anonymous pages are exclusive when (un)pinning
In reply to David HildenbrandLet's verify when (un)pinning anonymous pages that we always deal with exclusive anonymous pages, which guarantees that we'll have a reliable PIN, meaning that we cannot end up with the GUP pin being inconsistent with he pages mapped into the page tables due to a COW triggered by a write fault.
When pinning pages, after conditionally triggering GUP unsharing of possibly shared anonymous pages, we should always only see exclusive anonymous pages. Note that anonymous pages that are mapped writable must be marked exclusive, otherwise we'd have a BUG.
When pinning during ordinary GUP, simply add a check after our conditional GUP-triggered unsharing checks. As we know exactly how the page is mapped, we know exactly in which page we have to check for PageAnonExclusive().
When pinning via GUP-fast we have to be careful, because we can race with fork(): verify only after we made sure via the seqcount that we didn't race with concurrent fork() that we didn't end up pinning a possibly shared anonymous page.
Similarly, when unpinning, verify that the pages are still marked as
exclusive: otherwise something turned the pages possibly shared, which can result in random memory corruptions, which we really want to catch.
With only the pinned pages at hand and not the actual page table entries we have to be a bit careful: hugetlb pages are always mapped via a single logical page table entry referencing the head page and PG_anon_exclusive of the head page applies. Anon THP are a bit more complicated, because we might have obtained the page reference either via a PMD or a PTE -- depending on the mapping type we either have to check PageAnonExclusive of the head page (PMD-mapped THP) or the tail page (PTE-mapped THP) applies: as we don't know and to make our life easier, check that either is set.
Take care to not verify in case we're unpinning during GUP-fast because we detected concurrent fork(): we might stumble over an anonymous page that is now shared.
--- mm/gup.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++- mm/huge_memory.c | 3 +++ mm/hugetlb.c | 3 +++ 3 files changed, 66 insertions(+), 1 deletion(-)
mm/gup.c +60 -1
@@ -29,6 +29,39 @@ struct follow_page_context {unsigned int page_mask; }; +static inline void sanity_check_pinned_pages (struct page **pages, +unsigned long npages) +{ +if (!IS_ENABLED (CONFIG_DEBUG_VM )) +return ; + +/* +* We only pin anonymous pages if they are exclusive. Once pinned, we +* can no longer turn them possibly shared and PageAnonExclusive() will +* stick around until the page is freed. +* +* We'd like to verify that our pinned anonymous pages are still mapped +* exclusively. The issue with anon THP is that we don't know how +* they are/were mapped when pinning them. However, for anon +* THP we can assume that either the given page (PTE-mapped THP) or +* the head page (PMD-mapped THP) should be PageAnonExclusive(). If +* neither is the case, there is certainly something wrong. +*/ +for (; npages; npages--, pages++) { +struct page *page = *pages; +struct folio *folio =page_folio (page); + +if (!folio_test_anon (folio)) +continue ; +if (!folio_test_large (folio) ||folio_test_hugetlb (folio)) +VM_BUG_ON_PAGE (!PageAnonExclusive (&folio->page), page); +else +/* Either a PTE-mapped or a PMD-mapped THP. */ +VM_BUG_ON_PAGE (!PageAnonExclusive (&folio->page) && + !PageAnonExclusive (page), page); + } +} +/* * Return the folio with ref appropriately incremented, * or NULL if that failed.
@@ -204,6 +237,7 @@ bool __must_check try_grab_page(struct page *page, unsigned int flags)*/ void unpin_user_page (struct page *page) { +sanity_check_pinned_pages (&page,1 );gup_put_folio (page_folio (page),1 ,FOLL_PIN ); }EXPORT_SYMBOL (unpin_user_page);
@@ -272,6 +306,7 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages,return ; } +sanity_check_pinned_pages (pages, npages);for (i =0 ; i < npages; i += nr) { folio =gup_folio_next (pages, npages, i, &nr);/*
@@ -344,6 +379,23 @@ void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, }EXPORT_SYMBOL (unpin_user_page_range_dirty_lock); +static void unpin_user_pages_lockless (struct page **pages,unsigned long npages) +{ +unsigned long i; +struct folio *folio; +unsigned int nr; + +/* +* Don't perform any sanity checks because we might have raced with +* fork() and some anonymous pages might now actually be shared -- +* which is why we're unpinning after all. +*/ +for (i =0 ; i < npages; i += nr) { + folio =gup_folio_next (pages, npages, i, &nr); +gup_put_folio (folio, nr,FOLL_PIN ); + } +} +/** * unpin_user_pages() - release an array of gup-pinned pages. * @pages: array of pages to be marked dirty and released.
@@ -367,6 +419,7 @@ void unpin_user_pages(struct page **pages, unsigned long npages)if (WARN_ON (IS_ERR_VALUE (npages)))return ; +sanity_check_pinned_pages (pages, npages);for (i =0 ; i < npages; i += nr) { folio =gup_folio_next (pages, npages, i, &nr);gup_put_folio (folio, nr,FOLL_PIN );
@@ -510,6 +563,10 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, page =ERR_PTR (-EMLINK ); goto out; } + +VM_BUG_ON ((flags &FOLL_PIN ) &&PageAnon (page) && + !PageAnonExclusive (page)); +/* try_grab_page() does nothing unless FOLL_GET or FOLL_PIN is set. */ if (unlikely (!try_grab_page (page, flags))) { page =ERR_PTR (-ENOMEM );
@@ -2744,8 +2801,10 @@ static unsigned long lockless_pages_from_mm(unsigned long start,*/ if (gup_flags &FOLL_PIN ) {if (read_seqcount_retry (¤t->mm->write_protect_seq, seq)) { -unpin_user_pages (pages, nr_pinned); +unpin_user_pages_lockless (pages, nr_pinned);return 0 ; + }else { +sanity_check_pinned_pages (pages, nr_pinned); } }return nr_pinned;
mm/huge_memory.c +3 -0
@@ -1392,6 +1392,9 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,if (!pmd_write (*pmd) &&gup_must_unshare (flags, page))return ERR_PTR (-EMLINK ); +VM_BUG_ON ((flags &FOLL_PIN ) &&PageAnon (page) && + !PageAnonExclusive (page)); +if (!try_grab_page (page, flags))return ERR_PTR (-ENOMEM );
mm/hugetlb.c +3 -0
@@ -6097,6 +6097,9 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, pfn_offset = (vaddr & ~huge_page_mask (h)) >>PAGE_SHIFT ; page =pte_page (huge_ptep_get (pte)); +VM_BUG_ON ((flags &FOLL_PIN ) &&PageAnon (page) && + !PageAnonExclusive (page)); +/* * If subpage information not requested, update counters * and skip the same_page loop below.
Signature · 1 line
2.35.1
Re: [PATCH v3 16/16] mm/gup: sanity-check with CONFIG_DEBUG_VM that anonymous pages are exclusive when (un)pinning
Quoted 35 lines · David Hildenbrand
Let's verify when (un)pinning anonymous pages that we always deal with exclusive anonymous pages, which guarantees that we'll have a reliable PIN, meaning that we cannot end up with the GUP pin being inconsistent with he pages mapped into the page tables due to a COW triggered by a write fault.
When pinning pages, after conditionally triggering GUP unsharing of possibly shared anonymous pages, we should always only see exclusive anonymous pages. Note that anonymous pages that are mapped writable must be marked exclusive, otherwise we'd have a BUG.
When pinning during ordinary GUP, simply add a check after our conditional GUP-triggered unsharing checks. As we know exactly how the page is mapped, we know exactly in which page we have to check for
PageAnonExclusive().When pinning via GUP-fast we have to be careful, because we can race with
fork(): verify only after we made sure via the seqcount that we didn't race with concurrentfork()that we didn't end up pinning a possibly shared anonymous page.Similarly, when unpinning, verify that the pages are still marked as
exclusive: otherwise something turned the pages possibly shared, which can result in random memory corruptions, which we really want to catch.With only the pinned pages at hand and not the actual page table entries we have to be a bit careful: hugetlb pages are always mapped via a single logical page table entry referencing the head page and
PG_anon_exclusiveof the head page applies. Anon THP are a bit more complicated, because we might have obtained the page reference either via a PMD or a PTE -- depending on the mapping type we either have to check PageAnonExclusive of the head page (PMD-mapped THP) or the tail page (PTE-mapped THP) applies: as we don't know and to make our life easier, check that either is set.
Quoted 5 lines
Take care to not verify in case we're unpinning during GUP-fast because we detected concurrent
fork(): we might stumble over an anonymous page that is now shared.Signed-off-by David Hildenbrand <[email protected]>
Nits:
mm/gup.c static struct page *follow_page_pte(struct vm_area_struct *vma, · line 568
Quoted 7 lines
mm/gup.c +3
-0@@ -510,6 +563,10 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, page =ERR_PTR (-EMLINK ); goto out; } + +VM_BUG_ON ((flags &FOLL_PIN ) &&PageAnon (page) && + !PageAnonExclusive (page));
Do we rather want VM_BUG_ON_PAGE? Also for the same tests in mm/huge*.c below.
Quoted 44 lines
mm/gup.c +4
-1+/* try_grab_page() does nothing unless FOLL_GET or FOLL_PIN is set. */ if (unlikely (!try_grab_page (page, flags))) { page =ERR_PTR (-ENOMEM );@@ -2744,8 +2801,10 @@ static unsigned long lockless_pages_from_mm(unsigned long start,*/ if (gup_flags &FOLL_PIN ) {if (read_seqcount_retry (¤t->mm->write_protect_seq, seq)) { -unpin_user_pages (pages, nr_pinned); +unpin_user_pages_lockless (pages, nr_pinned);return 0 ; + }else { +sanity_check_pinned_pages (pages, nr_pinned); } }return nr_pinned;mm/huge_memory.c +3
-0@@ -1392,6 +1392,9 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,if (!pmd_write (*pmd) &&gup_must_unshare (flags, page))return ERR_PTR (-EMLINK ); +VM_BUG_ON ((flags &FOLL_PIN ) &&PageAnon (page) && + !PageAnonExclusive (page)); +if (!try_grab_page (page, flags))return ERR_PTR (-ENOMEM );mm/hugetlb.c +3
-0@@ -6097,6 +6097,9 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, pfn_offset = (vaddr & ~huge_page_mask (h)) >>PAGE_SHIFT ; page =pte_page (huge_ptep_get (pte)); +VM_BUG_ON ((flags &FOLL_PIN ) &&PageAnon (page) && + !PageAnonExclusive (page)); +/* * If subpage information not requested, update counters * and skip the same_page loop below.
Re: [PATCH v3 16/16] mm/gup: sanity-check with CONFIG_DEBUG_VM that anonymous pages are exclusive when (un)pinning
Quoted 46 lines · Vlastimil Babka
Quoted 35 lines · David Hildenbrand
Let's verify when (un)pinning anonymous pages that we always deal with exclusive anonymous pages, which guarantees that we'll have a reliable PIN, meaning that we cannot end up with the GUP pin being inconsistent with he pages mapped into the page tables due to a COW triggered by a write fault.
When pinning pages, after conditionally triggering GUP unsharing of possibly shared anonymous pages, we should always only see exclusive anonymous pages. Note that anonymous pages that are mapped writable must be marked exclusive, otherwise we'd have a BUG.
When pinning during ordinary GUP, simply add a check after our conditional GUP-triggered unsharing checks. As we know exactly how the page is mapped, we know exactly in which page we have to check for
PageAnonExclusive().When pinning via GUP-fast we have to be careful, because we can race with
fork(): verify only after we made sure via the seqcount that we didn't race with concurrentfork()that we didn't end up pinning a possibly shared anonymous page.Similarly, when unpinning, verify that the pages are still marked as
exclusive: otherwise something turned the pages possibly shared, which can result in random memory corruptions, which we really want to catch.With only the pinned pages at hand and not the actual page table entries we have to be a bit careful: hugetlb pages are always mapped via a single logical page table entry referencing the head page and
PG_anon_exclusiveof the head page applies. Anon THP are a bit more complicated, because we might have obtained the page reference either via a PMD or a PTE -- depending on the mapping type we either have to check PageAnonExclusive of the head page (PMD-mapped THP) or the tail page (PTE-mapped THP) applies: as we don't know and to make our life easier, check that either is set.Quoted 5 lines
Take care to not verify in case we're unpinning during GUP-fast because we detected concurrent
fork(): we might stumble over an anonymous page that is now shared.Signed-off-by David Hildenbrand <[email protected]>Acked-by Vlastimil Babka <[email protected]>Nits:
Quoted 9 lines
Review static struct page *follow_page_pte(struct vm_area_struct *vma, · line 568
Quoted 7 lines
(quoted diff) +3
-0@@ -510,6 +563,10 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, page = ERR_PTR(-EMLINK); goto out; } + + VM_BUG_ON((flags & FOLL_PIN) && PageAnon(page) && + !PageAnonExclusive(page));Do we rather want
VM_BUG_ON_PAGE? Also for the same tests inmm/huge*.cbelow.
Make sense, thanks:
mm/gup.c +2 -2
@@ -564,8 +564,8 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, goto out; } -VM_BUG_ON ((flags &FOLL_PIN ) &&PageAnon (page) && - !PageAnonExclusive (page)); +VM_BUG_ON_PAGE ((flags &FOLL_PIN ) &&PageAnon (page) && + !PageAnonExclusive (page), page);/* try_grab_page() does nothing unless FOLL_GET or FOLL_PIN is set. */ if (unlikely (!try_grab_page (page, flags))) {
mm/huge_memory.c +2 -2
@@ -1392,8 +1392,8 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,if (!pmd_write (*pmd) &&gup_must_unshare (flags, page))return ERR_PTR (-EMLINK ); -VM_BUG_ON ((flags &FOLL_PIN ) &&PageAnon (page) && - !PageAnonExclusive (page)); +VM_BUG_ON_PAGE ((flags &FOLL_PIN ) &&PageAnon (page) && + !PageAnonExclusive (page), page);if (!try_grab_page (page, flags))return ERR_PTR (-ENOMEM );
mm/hugetlb.c +2 -2
@@ -6100,8 +6100,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, pfn_offset = (vaddr & ~huge_page_mask (h)) >>PAGE_SHIFT ; page =pte_page (huge_ptep_get (pte)); -VM_BUG_ON ((flags &FOLL_PIN ) &&PageAnon (page) && - !PageAnonExclusive (page)); +VM_BUG_ON_PAGE ((flags &FOLL_PIN ) &&PageAnon (page) && + !PageAnonExclusive (page), page);/* * If subpage information not requested, update counters
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v3 16/16] mm/gup: sanity-check with CONFIG_DEBUG_VM that anonymous pages are exclusive when (un)pinning
Quoted 57 lines · David Hildenbrand
Quoted 46 lines · Vlastimil Babka
Quoted 35 lines · David Hildenbrand
Let's verify when (un)pinning anonymous pages that we always deal with exclusive anonymous pages, which guarantees that we'll have a reliable PIN, meaning that we cannot end up with the GUP pin being inconsistent with he pages mapped into the page tables due to a COW triggered by a write fault.
When pinning pages, after conditionally triggering GUP unsharing of possibly shared anonymous pages, we should always only see exclusive anonymous pages. Note that anonymous pages that are mapped writable must be marked exclusive, otherwise we'd have a BUG.
When pinning during ordinary GUP, simply add a check after our conditional GUP-triggered unsharing checks. As we know exactly how the page is mapped, we know exactly in which page we have to check for
PageAnonExclusive().When pinning via GUP-fast we have to be careful, because we can race with
fork(): verify only after we made sure via the seqcount that we didn't race with concurrentfork()that we didn't end up pinning a possibly shared anonymous page.Similarly, when unpinning, verify that the pages are still marked as
exclusive: otherwise something turned the pages possibly shared, which can result in random memory corruptions, which we really want to catch.With only the pinned pages at hand and not the actual page table entries we have to be a bit careful: hugetlb pages are always mapped via a single logical page table entry referencing the head page and
PG_anon_exclusiveof the head page applies. Anon THP are a bit more complicated, because we might have obtained the page reference either via a PMD or a PTE -- depending on the mapping type we either have to check PageAnonExclusive of the head page (PMD-mapped THP) or the tail page (PTE-mapped THP) applies: as we don't know and to make our life easier, check that either is set.Quoted 5 lines
Take care to not verify in case we're unpinning during GUP-fast because we detected concurrent
fork(): we might stumble over an anonymous page that is now shared.Signed-off-by David Hildenbrand <[email protected]>Acked-by Vlastimil Babka <[email protected]>Nits:
Quoted 9 lines
Review static struct page *follow_page_pte(struct vm_area_struct *vma, · line 568
Quoted 7 lines
(quoted diff) +3
-0@@ -510,6 +563,10 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, page = ERR_PTR(-EMLINK); goto out; } + + VM_BUG_ON((flags & FOLL_PIN) && PageAnon(page) && + !PageAnonExclusive(page));Do we rather want
VM_BUG_ON_PAGE? Also for the same tests inmm/huge*.cbelow.
Quoted 1 line
Make sense, thanks:
LGTM
Quoted 48 lines
mm/gup.c +2
-2@@ -564,8 +564,8 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, goto out; } -VM_BUG_ON ((flags &FOLL_PIN ) &&PageAnon (page) && - !PageAnonExclusive (page)); +VM_BUG_ON_PAGE ((flags &FOLL_PIN ) &&PageAnon (page) && + !PageAnonExclusive (page), page);/* try_grab_page() does nothing unless FOLL_GET or FOLL_PIN is set. */ if (unlikely (!try_grab_page (page, flags))) {mm/huge_memory.c +2
-2@@ -1392,8 +1392,8 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,if (!pmd_write (*pmd) &&gup_must_unshare (flags, page))return ERR_PTR (-EMLINK ); -VM_BUG_ON ((flags &FOLL_PIN ) &&PageAnon (page) && - !PageAnonExclusive (page)); +VM_BUG_ON_PAGE ((flags &FOLL_PIN ) &&PageAnon (page) && + !PageAnonExclusive (page), page);if (!try_grab_page (page, flags))return ERR_PTR (-ENOMEM );mm/hugetlb.c +2
-2@@ -6100,8 +6100,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, pfn_offset = (vaddr & ~huge_page_mask (h)) >>PAGE_SHIFT ; page =pte_page (huge_ptep_get (pte)); -VM_BUG_ON ((flags &FOLL_PIN ) &&PageAnon (page) && - !PageAnonExclusive (page)); +VM_BUG_ON_PAGE ((flags &FOLL_PIN ) &&PageAnon (page) && + !PageAnonExclusive (page), page);/* * If subpage information not requested, update counters
Quoted 216 lines · David Hildenbrand
This series is the result of the discussion on the previous approach [2]. More information on the general COW issues can be found there. It is based on latest linus/master (post v5.17, with relevant core-MM changes for v5.18-rc1).
v3 is located at:
https://github.com/davidhildenbrand/linux/tree/cow_fixes_part_2_v3This series fixes memory corruptions when a GUP pin (
FOLL_PIN) was taken on an anonymous page and COW logic fails to detect exclusivity of the page to then replacing the anonymous page by a copy in the page table: The GUP pin lost synchronicity with the pages mapped into the page tables.This issue, including other related COW issues, has been summarized in [3] under 3):
"
3. Intra Process Memory Corruptions due to Wrong COW (FOLL_PIN)
page_maybe_dma_pinned()is used to check if a page may be pinned for DMA (usingFOLL_PINinstead ofFOLL_GET). While false positives are tolerable, false negatives are problematic: pages that are pinned for DMA must not be added to the swapcache. If it happens, the (now pinned) page could be faulted back from the swapcache into page tables read-only. Future write-access would detect the pinning and COW the page, losing synchronicity. For the interested reader, this is nicely documented infeb889fb40fa("mm: don't put pinned pages into the swap cache").Peter reports [8] that
page_maybe_dma_pinned()as used is racy in some cases and can result in a violation of the documented semantics:
giving false negatives because of the race.There are cases where we call it without properly taking a per-process sequence lock, turning the usage of
page_maybe_dma_pinned()racy. While one case (clear_refsSOFTDIRTY tracking, see below) seems to be easy to handle, there is especially one rmap case (shrink_page_list) that's hard to fix: in the rmap world, we're not limited to a single process.The
shrink_page_list()issue is really subtle. If we race with someone pinning a page, we can trigger the same issue as in theFOLL_GETcase. See the detail section at the end of this mail on a discussion how bad this can bite us with VFIO or otherFOLL_PINuser.It's harder to reproduce, but I managed to modify the
O_DIRECTreproducer to useio_uringfixed buffers [15] instead, which ends up usingFOLL_PIN|FOLL_WRITE|FOLL_LONGTERMto pin buffer pages and can similarly trigger a loss of synchronicity and consequently a memory corruption.Again, the root issue is that a write-fault on a page that has additional references results in a COW and thereby a loss of synchronicity and consequently a memory corruption if two parties believe they are referencing the same page.
"This series makes GUP pins (R/O and R/W) on anonymous pages fully reliable, especially also taking care of concurrent pinning via GUP-fast, for example, also fully fixing an issue reported regarding NUMA balancing [4] recently. While doing that, it further reduces "unnecessary COWs", especially when we don't fork()/KSM and don't swapout, and fixes the COW security for hugetlb for
FOLL_PIN.In summary, we track via a pageflag (
PG_anon_exclusive) whether a mapped anonymous page is exclusive. Exclusive anonymous pages that are mapped R/O can directly be mapped R/W by the COW logic in the write fault handler. Exclusive anonymous pages that want to be shared (fork(), KSM) first have to be marked shared -- which will fail if there are GUP pins on the page. GUP is only allowed to take a pin on anonymous pages that are exclusive. The PT lock is the primary mechanism to synchronize modifications ofPG_anon_exclusive. We synchronize against GUP-fast either via thesrc_mm->write_protect_seq(duringfork()) or via clear/invalidate+flush of the relevant page table entry.Special care has to be taken about swap, migration, and THPs (whereby a PMD-mapping can be converted to a PTE mapping and we have to track information for subpages). Besides these, we let the rmap code handle most magic. For reliable R/O pins of anonymous pages, we need
FAULT_FLAG_UNSHARElogic as part of our previous approach [2], however, it's now 100% mapcount free and I further simplified it a bit.#1 is a fix
#3-#10 are mostly rmap preparations forPG_anon_exclusivehandling #11 introducesPG_anon_exclusive
#12 usesPG_anon_exclusiveand make R/W pins of anonymous pages
reliable
#13 is a preparation for reliable R/O pins
#14 and #15 is reused/modified GUP-triggered unsharing for R/O GUP pins
make R/O pins of anonymous pages reliable
#16 adds sanity check when (un)pinning anonymous pages[1] https://lkml.kernel.org/r/[email protected]
[2] https://lkml.kernel.org/r/[email protected]
[3] https://lore.kernel.org/r/[email protected]
[4] https://bugzilla.kernel.org/show_bug.cgi?id=215616v2 -> v3:
* Note 1: Left the terminology "unshare" in place for now instead of switching to "make anon exclusive".
* Note 2: We might have to tackle undoing effects ofarch_unmap_one()on sparc, to free some tag memory immediately instead of when tearing down the vma/mm; looks like this needs more care either way, so I'll ignore it for now.
* Rebased on top of core MM changes for v5.18-rc1 (most conflicts were due to folio andZONE_DEVICEmigration rework). No severe changes were necessary -- mostly folio conversion and code movement.
* Retested on aarch64, ppc64, s390x andx86_64
* "mm/rmap: convert RMAP flags to a proper distinctrmap_ttype" -> Missed to convert one instance inrestore_exclusive_pte()
* "mm/rmap: pass rmap flags tohugepage_add_anon_rmap()"
-> Use "!!(flags &RMAP_EXCLUSIVE)" to avoid sparse warnings
* "mm/huge_memory: remove outdatedVM_WARN_ON_ONCE_PAGEfromunmap_page()" -> Added, as we can trigger that now more frequently
* "mm: remember exclusively mapped anonymous pages withPG_anon_exclusive" -> Use subpage inVM_BUG_ON_PAGE()intry_to_migrate_one()
-> Move comment fromfolio_migrate_mapping()tofolio_migrate_flags()
regarding PG_anon_exclusive/PG_mappedtodisk
-> s/int rmap_flags/rmap_t rmap_flags/ inremove_migration_pmd()
* "mm/gup: sanity-check withCONFIG_DEBUG_VMthat anonymous pages are exclusive when (un)pinning"
-> UseIS_ENABLED(CONFIG_DEBUG_VM)instead of ifdefv1 -> v2:
* Tested on aarch64, ppc64, s390x andx86_64
* "mm/page-flags: reusePG_mappedtodiskasPG_anon_exclusiveforPageAnon()pages"
-> UsePG_mappedtodiskinstead ofPG_slab(thanks Willy!), this simlifies
the patch and necessary handling a lot. Add safety BUG_ON's
-> Move most documentation to the patch description, to be placed in a
proper documentation doc in the future, once everything's in place
* ""mm: remember exclusively mapped anonymous pages withPG_anon_exclusive-> Skip check+clearing inpage_try_dup_anon_rmap(), otherwise we might trigger a wrongVM_BUG_ON()for KSM pages inClearPageAnonExclusive()-> In__split_huge_pmd_locked(), callpage_try_share_anon_rmap()only for "anon_exclusive", otherwise we might trigger a wrongVM_BUG_ON()-> In__split_huge_page_tail(), drop any remainingPG_anon_exclusiveon tail pages, and document why that is fineRFC -> v1:
* Rephrased/extended some patch descriptions+comments
* Tested on aarch64, ppc64 andx86_64
* "mm/rmap: convert RMAP flags to a proper distinctrmap_ttype" -> Added
* "mm/rmap: drop "compound" parameter frompage_add_new_anon_rmap()" -> Added
* "mm: remember exclusively mapped anonymous pages withPG_anon_exclusive" -> Fixed__do_huge_pmd_anonymous_page()to recheck after temporarily dropping the PT lock.
-> Use "reuse" label in__do_huge_pmd_anonymous_page()
-> Slightly simplify logic inhugetlb_cow()
-> Inremove_migration_pte(), remove unrelated changes around
page_remove_rmap()
* "mm: support GUP-triggered unsharing of anonymous pages"
-> Inhandle_pte_fault(), triggerpte_mkdirty()only with
FAULT_FLAG_WRITE
-> In__handle_mm_fault(), extend comment regarding anonymous PUDs
* "mm/gup: triggerFAULT_FLAG_UNSHAREwhen R/O-pinning a possibly shared anonymous page"
-> Added unsharing logic togup_hugepte()andgup_huge_pud()-> Changed return logic in__follow_hugetlb_must_fault(), making sure
that "unshare" is always set
* "mm/gup: sanity-check withCONFIG_DEBUG_VMthat anonymous pages are exclusive when (un)pinning"
-> Slightly simplifiedsanity_check_pinned_pages()David Hildenbrand (16):
mm/rmap: fix missingswap_free()intry_to_unmap()after
arch_unmap_one()failed
mm/hugetlb: takesrc_mm->write_protect_seqin
copy_hugetlb_page_range()
mm/memory: slightly simplifycopy_present_pte()
mm/rmap: splitpage_dup_rmap()intopage_dup_file_rmap()and
page_try_dup_anon_rmap()
mm/rmap: convert RMAP flags to a proper distinctrmap_ttype mm/rmap: removedo_page_add_anon_rmap()
mm/rmap: pass rmap flags tohugepage_add_anon_rmap()mm/rmap: drop "compound" parameter frompage_add_new_anon_rmap()mm/rmap: usepage_move_anon_rmap()when reusing a mappedPageAnon()
page exclusively
mm/huge_memory: remove outdatedVM_WARN_ON_ONCE_PAGEfromunmap_page()mm/page-flags: reusePG_mappedtodiskasPG_anon_exclusivefor
PageAnon()pages
mm: remember exclusively mapped anonymous pages withPG_anon_exclusivemm/gup: disallowfollow_page(FOLL_PIN)
mm: support GUP-triggered unsharing of anonymous pages mm/gup: triggerFAULT_FLAG_UNSHAREwhen R/O-pinning a possibly shared
anonymous page
mm/gup: sanity-check withCONFIG_DEBUG_VMthat anonymous pages are
exclusive when (un)pinninginclude/linux/mm.h | 46 +++++++- include/linux/mm_types.h | 8 ++ include/linux/page-flags.h | 39 ++++++- include/linux/rmap.h | 118 +++++++++++++++++-- include/linux/swap.h | 15 ++- include/linux/swapops.h | 25 ++++ kernel/events/uprobes.c | 2 +- mm/gup.c | 106 ++++++++++++++++- mm/huge_memory.c | 127 +++++++++++++++----- mm/hugetlb.c | 135 ++++++++++++++------- mm/khugepaged.c | 2 +- mm/ksm.c | 15 ++- mm/memory.c | 234 +++++++++++++++++++++++-------------- mm/memremap.c | 9 ++ mm/migrate.c | 18 ++- mm/migrate_device.c | 23 +++- mm/mprotect.c | 8 +- mm/rmap.c | 97 +++++++++++---- mm/swapfile.c | 8 +- mm/userfaultfd.c | 2 +- tools/vm/page-types.c | 8 +- 21 files changed, 825 insertions(+), 220 deletions(-)
Quoted 1 line
base-commit:
1930a6e739c4b4a654a69164dbe39e554d228915
The effective change on v5.17 *before rebasing* compared to v2 is:
mm/gup.c +3 -2
@@ -48,7 +48,9 @@ static void hpage_pincount_sub(struct page *page, int refs)static inline void sanity_check_pinned_pages (struct page **pages,unsigned long npages) { -#ifdef CONFIG_DEBUG_VM +if (!IS_ENABLED (CONFIG_DEBUG_VM )) +return ; +/* * We only pin anonymous pages if they are exclusive. Once pinned, we * can no longer turn them possibly shared and PageAnonExclusive() will
@@ -74,7 +76,6 @@ static inline void sanity_check_pinned_pages(struct page **pages,VM_BUG_ON_PAGE (!PageAnonExclusive (head) && !PageAnonExclusive (page), page); } -#endif /* CONFIG_DEBUG_VM */ }/* Equivalent to calling put_page() @refs times. */
mm/huge_memory.c +1 -3
@@ -2316,8 +2316,6 @@ static void unmap_page(struct page *page)try_to_migrate (page, ttu_flags); else try_to_unmap(page, ttu_flags | TTU_IGNORE_MLOCK); - -VM_WARN_ON_ONCE_PAGE (page_mapped (page), page); }static void remap_page (struct page *page,unsigned int nr)
@@ -3187,7 +3185,7 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new)flush_cache_range (vma, mmun_start, mmun_start +HPAGE_PMD_SIZE );if (PageAnon (new)) { -int rmap_flags =RMAP_COMPOUND ; +rmap_t rmap_flags =RMAP_COMPOUND ;if (!is_readable_migration_entry (entry)) rmap_flags |=RMAP_EXCLUSIVE ;
mm/memory.c +1 -1
@@ -727,7 +727,7 @@ static void restore_exclusive_pte(struct vm_area_struct *vma,* created when the swap entry was made. */ if (PageAnon (page)) -page_add_anon_rmap (page, vma, address, false); +page_add_anon_rmap (page, vma, address,RMAP_NONE );else /* * Currently device exclusive access only supports anonymous
mm/migrate.c +6 -4
@@ -384,10 +384,6 @@ int folio_migrate_mapping(struct address_space *mapping,/* No turning back from here */ newfolio->index = folio->index; newfolio->mapping = folio->mapping; -/* -* Note: PG_anon_exclusive is always migrated via migration -* entries. -*/ if (folio_test_swapbacked (folio))__folio_set_swapbacked (newfolio);
@@ -540,6 +536,12 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)folio_set_workingset (newfolio);if (folio_test_checked (folio))folio_set_checked (newfolio); +/* +* PG_anon_exclusive (-> PG_mappedtodisk) is always migrated via +* migration entries. We can still have PG_anon_exclusive set on an +* effectively unmapped and unreferenced first sub-pages of an +* anonymous THP: we can simply copy it here via PG_mappedtodisk. +*/ if (folio_test_mappedtodisk (folio))folio_set_mappedtodisk (newfolio);
mm/rmap.c +2 -2
@@ -1939,7 +1939,7 @@ static bool try_to_migrate_one(struct page *page, struct vm_area_struct *vma,break ; }VM_BUG_ON_PAGE (pte_write (pteval) &&PageAnon (page) && - !anon_exclusive, page); + !anon_exclusive, subpage);if (anon_exclusive &&page_try_share_anon_rmap (subpage)) {set_pte_at (mm, address, pvmw.pte, pteval);
@@ -2476,7 +2476,7 @@ void hugepage_add_anon_rmap(struct page *page, struct vm_area_struct *vma,VM_BUG_ON_PAGE (!first &&PageAnonExclusive (page), page);if (first)__page_set_anon_rmap (page, vma, address, - flags &RMAP_EXCLUSIVE ); + !!(flags &RMAP_EXCLUSIVE )); } void hugepage_add_new_anon_rmap(struct page *page,
Signature · 3 lines
Thanks, David / dhildenb