Skip to contentSkip to conversation

mm-gup-trigger-fault_flag_unshare-when-r-o-pinning-a-possibly-shared-anonymous-page.patch removed from -mm tree

merged mm-stable Andrew Morton 1 participant 1 message May 10, 2022

lore
Share & tools
lore

Andrew Morton

author tree notice Applied
#0 lore raw

The quilt patch titled
Subject: mm/gup: trigger FAULT_FLAG_UNSHARE when R/O-pinning a possibly shared anonymous page
has been removed from the -mm tree. Its filename was
mm-gup-trigger-fault_flag_unshare-when-r-o-pinning-a-possibly-shared-anonymous-page.patch

This patch was dropped because it was merged into mm-stable

------------------------------------------------------
From: David Hildenbrand <[email protected]>
Subject: mm/gup: trigger FAULT_FLAG_UNSHARE when R/O-pinning a possibly shared anonymous page

Whenever GUP currently ends up taking a R/O pin on an anonymous page that might be shared -- mapped R/O and !PageAnonExclusive() -- any write fault on the page table entry will end up replacing the mapped anonymous page due to COW, resulting in the GUP pin no longer being consistent with the page actually mapped into the page table.

The possible ways to deal with this situation are:
(1) Ignore and pin -- what we do right now.
(2) Fail to pin -- which would be rather surprising to callers and could break user space.
(3) Trigger unsharing and pin the now exclusive page -- reliable R/O pins.

Let's implement 3) because it provides the clearest semantics and allows for checking in unpin_user_pages() and friends for possible BUGs: when trying to unpin a page that's no longer exclusive, clearly something went very wrong and might result in memory corruptions that might be hard to debug. So we better have a nice way to spot such issues.

This change implies that whenever user space *wrote* to a private mapping (IOW, we have an anonymous page mapped), that GUP pins will always remain
consistent: reliable R/O GUP pins of anonymous pages.

As a side note, this commit fixes the COW security issue for hugetlb with FOLL_PIN as documented in:
https://lore.kernel.org/r/[email protected]
The vmsplice reproducer still applies, because vmsplice uses FOLL_GET instead of FOLL_PIN.

Note that follow_huge_pmd() doesn't apply because we cannot end up in there with FOLL_PIN.

This commit is heavily based on prototype patches by Andrea.

Cc Christoph Hellwig <[email protected]>
Cc David Rientjes <[email protected]>
Cc Don Dutile <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Jan Kara <[email protected]>
Cc Jann Horn <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc John Hubbard <[email protected]>
Cc Khalid Aziz <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Liang Zhang <[email protected]>
Cc "Matthew Wilcox (Oracle)" <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Mike Kravetz <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Nadav Amit <[email protected]>
Cc Oded Gabbay <[email protected]>
Cc Oleg Nesterov <[email protected]>
Cc Pedro Demarchi Gomes <[email protected]>
Cc Peter Xu <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Roman Gushchin <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Yang Shi <[email protected]>

---

 include/linux/mm.h |   39 +++++++++++++++++++++++++++++++++++++++
 mm/gup.c           |   42 +++++++++++++++++++++++++++++++++++++++---
 mm/huge_memory.c   |    3 +++
 mm/hugetlb.c       |   27 ++++++++++++++++++++++++---
 4 files changed, 105 insertions(+), 6 deletions(-)
include/linux/mm.h renamed from include/linux/mm.h~mm-gup-trigger-fault_flag_unshare-when-r-o-pinning-a-possibly-shared-anonymous-page +39 -0
@@ -3002,6 +3002,45 @@ static inline int vm_fault_to_errno(vm_f
 	return 0;
 }
 
+/*
+ * Indicates for which pages that are write-protected in the page table,
+ * whether GUP has to trigger unsharing via FAULT_FLAG_UNSHARE such that the
+ * GUP pin will remain consistent with the pages mapped into the page tables
+ * of the MM.
+ *
+ * Temporary unmapping of PageAnonExclusive() pages or clearing of
+ * PageAnonExclusive() has to protect against concurrent GUP:
+ * * Ordinary GUP: Using the PT lock
+ * * GUP-fast and fork(): mm->write_protect_seq
+ * * GUP-fast and KSM or temporary unmapping (swap, migration):
+ *   clear/invalidate+flush of the page table entry
+ *
+ * Must be called with the (sub)page that's actually referenced via the
+ * page table entry, which might not necessarily be the head page for a
+ * PTE-mapped THP.
+ */
+static inline bool gup_must_unshare(unsigned int flags, struct page *page)
+{
+	/*
+	 * FOLL_WRITE is implicitly handled correctly as the page table entry
+	 * has to be writable -- and if it references (part of) an anonymous
+	 * folio, that part is required to be marked exclusive.
+	 */
+	if ((flags & (FOLL_WRITE | FOLL_PIN)) != FOLL_PIN)
+		return false;
+	/*
+	 * Note: PageAnon(page) is stable until the page is actually getting
+	 * freed.
+	 */
+	if (!PageAnon(page))
+		return false;
+	/*
+	 * Note that PageKsm() pages cannot be exclusive, and consequently,
+	 * cannot get pinned.
+	 */
+	return !PageAnonExclusive(page);
+}
+
 typedef int (*pte_fn_t)(pte_t *pte, unsigned long addr, void *data);
 extern int apply_to_page_range(struct mm_struct *mm, unsigned long address,
 			       unsigned long size, pte_fn_t fn, void *data);
mm/gup.c renamed from mm/gup.c~mm-gup-trigger-fault_flag_unshare-when-r-o-pinning-a-possibly-shared-anonymous-page +39 -3

96 lines not on this page · raw message

mm/huge_memory.c renamed from mm/huge_memory.c~mm-gup-trigger-fault_flag_unshare-when-r-o-pinning-a-possibly-shared-anonymous-page +3 -0
@@ -1389,6 +1389,9 @@ struct page *follow_trans_huge_pmd(struc
 	page = pmd_page(*pmd);
 	VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page);
 
+	if (!pmd_write(*pmd) && gup_must_unshare(flags, page))
+		return ERR_PTR(-EMLINK);
+
 	if (!try_grab_page(page, flags))
 		return ERR_PTR(-ENOMEM);
 
mm/hugetlb.c renamed from mm/hugetlb.c~mm-gup-trigger-fault_flag_unshare-when-r-o-pinning-a-possibly-shared-anonymous-page +24 -3
@@ -5982,6 +5982,25 @@ static void record_subpages_vmas(struct
 	}
 }
 
+static inline bool __follow_hugetlb_must_fault(unsigned int flags, pte_t *pte,
+					       bool *unshare)
+{
+	pte_t pteval = huge_ptep_get(pte);
+
+	*unshare = false;
+	if (is_swap_pte(pteval))
+		return true;
+	if (huge_pte_write(pteval))
+		return false;
+	if (flags & FOLL_WRITE)
+		return true;
+	if (gup_must_unshare(flags, pte_page(pteval))) {
+		*unshare = true;
+		return true;
+	}
+	return false;
+}
+
 long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
 			 struct page **pages, struct vm_area_struct **vmas,
 			 unsigned long *position, unsigned long *nr_pages,
@@ -5996,6 +6015,7 @@ long follow_hugetlb_page(struct mm_struc
 	while (vaddr < vma->vm_end && remainder) {
 		pte_t *pte;
 		spinlock_t *ptl = NULL;
+		bool unshare = false;
 		int absent;
 		struct page *page;
 
@@ -6046,9 +6066,8 @@ long follow_hugetlb_page(struct mm_struc
 		 * both cases, and because we can't follow correct pages
 		 * directly from any kind of swap entries.
 		 */
-		if (absent || is_swap_pte(huge_ptep_get(pte)) ||
-		    ((flags & FOLL_WRITE) &&
-		      !huge_pte_write(huge_ptep_get(pte)))) {
+		if (absent ||
+		    __follow_hugetlb_must_fault(flags, pte, &unshare)) {
 			vm_fault_t ret;
 			unsigned int fault_flags = 0;
 
@@ -6056,6 +6075,8 @@ long follow_hugetlb_page(struct mm_struc
 				spin_unlock(ptl);
 			if (flags & FOLL_WRITE)
 				fault_flags |= FAULT_FLAG_WRITE;
+			else if (unshare)
+				fault_flags |= FAULT_FLAG_UNSHARE;
 			if (locked)
 				fault_flags |= FAULT_FLAG_ALLOW_RETRY |
 					FAULT_FLAG_KILLABLE;

_

Patches currently in -mm which might be from [email protected] are