Add support for SVM atomics in Nouveau

include/linux/swapops.h +10 -0

+
+/*
+ * A pfn swap entry is a special type of swap entry that always has a pfn stored
+ * in the swap offset. They are used to represent unaddressable device memory
+ * and to restrict access to a page undergoing migration.
+ */
+static inline bool is_pfn_swap_entry(swp_entry_t entry)
+{
+	return is_migration_entry(entry) || is_device_private_entry(entry);
+}

include/linux/swapops.h static inline int is_write_migration_entry(swp_entry_t entry) · line 209

Quoted 12 lines · Alistair Popple

include/linux/swapops.h +12 -0

+static inline struct page *pfn_swap_entry_to_page(swp_entry_t entry)
+{
+     struct page *p = pfn_to_page(swp_offset(entry));
+
+     /*
+      * Any use of migration entries may only occur while the
+      * corresponding page is locked
+      */
+     BUG_ON(is_migration_entry(entry) && !PageLocked(p));
+
+     return p;
+}

Would swap_pfn_entry_to_page() be slightly better?

The thing is it's very easy to read pfn_*() as a function to take a pfn as parameter...

Since I'm also recently working on some swap-related new ptes [1], I'm thinking whether we could name these swap entries as "swap XXX entries". Say, "swap hwpoison entry", "swap pfn entry" (which is a superset of "swap migration entry", "swap device exclusive entry", ...). That's where I came with the above swap_pfn_entry_to_page(), then below will be naturally is_swap_pfn_entry().

No strong opinion as this is already a v8 series (and sorry to chim in this late), just to raise this up.

[1] https://lore.kernel.org/lkml/[email protected]/

Thanks,

include/linux/swapops.h static inline int is_write_migration_entry(swp_entry_t entry) · line 219

Quoted 10 lines

include/linux/swapops.h +3 -0

+
+/*
+ * A pfn swap entry is a special type of swap entry that always has a pfn

stored + * in the swap offset. They are used to represent unaddressable device memory + * and to restrict access to a page undergoing migration.
+ */

include/linux/swapops.h +4 -0

+static inline bool is_pfn_swap_entry(swp_entry_t entry)
+{
+     return is_migration_entry(entry) || is_device_private_entry(entry);
+}

--
Peter Xu

Quoted 19 lines · AEST Peter Xu

include/linux/swapops.h static inline int is_write_migration_entry(swp_entry_t entry) · line 209

Quoted 12 lines · Alistair Popple

include/linux/swapops.h +12 -0

+static inline struct page *pfn_swap_entry_to_page(swp_entry_t entry)
+{
+     struct page *p = pfn_to_page(swp_offset(entry));
+
+     /*
+      * Any use of migration entries may only occur while the
+      * corresponding page is locked
+      */
+     BUG_ON(is_migration_entry(entry) && !PageLocked(p));
+
+     return p;
+}

Would swap_pfn_entry_to_page() be slightly better?

The thing is it's very easy to read pfn_*() as a function to take a pfn as parameter...

Quoted 6 lines

Since I'm also recently working on some swap-related new ptes [1], I'm thinking whether we could name these swap entries as "swap XXX entries". Say, "swap hwpoison entry", "swap pfn entry" (which is a superset of "swap migration entry", "swap device exclusive entry", ...). That's where I came with the above swap_pfn_entry_to_page(), then below will be naturally is_swap_pfn_entry().

Equally though "hwpoison swap entry", "pfn swap entry", "migration swap entry", etc. also makes sense (at least to me), but does that not fit in as well with your series? I haven't looked too deeply at your series but have been meaning to so thanks for the pointer.

Quoted 3 lines

No strong opinion as this is already a v8 series (and sorry to chim in this late), just to raise this up.

No worries, it's good timing as I was about to send a v9 which was just a rebase anyway. I am hoping to try and get this accepted for the next merge window but I will wait before sending v9 to see if anyone else has thoughts on the naming here.

I don't have a particularly strong opinion either, and your justification is more thought than I gave to naming these originally so am happy to rename if it's more readable or fits better with your series.

The behaviour of try_to_unmap_one() is difficult to follow because it performs different operations based on a fairly large set of flags used in different combinations.

TTU_MUNLOCK is one such flag. However it is exclusively used by try_to_munlock() which specifies no other flags. Therefore rather than overload try_to_unmap_one() with unrelated behaviour split this out into it's own function and remove the flag.

Signed-off-by Alistair Popple <[email protected]>

Reviewed-by Ralph Campbell <[email protected]>

Reviewed-by Christoph Hellwig <[email protected]>

---

v8:
* Renamed try_to_munlock to page_mlock to better reflect what the function actually does.
* Removed the TODO from the documentation that this patch addresses.

v7:
* Added Christoph's Reviewed-by

v4:
* Removed redundant check for VM_LOCKED

---
 Documentation/vm/unevictable-lru.rst | 33 ++++++++-----------
 include/linux/rmap.h                 |  3 +-
 mm/mlock.c                           | 10 +++---
 mm/rmap.c                            | 48 +++++++++++++++++++++-------
 4 files changed, 55 insertions(+), 39 deletions(-)

Documentation/vm/unevictable-lru.rst +13 -20

index 0e1490524f53..eae3af17f2d9 100644
@@ -389,14 +389,14 @@ mlocked, munlock_vma_page() updates that zone statistics for the number of
 mlocked pages.  Note, however, that at this point we haven't checked whether
 the page is mapped by other VM_LOCKED VMAs.
 
-We can't call try_to_munlock(), the function that walks the reverse map to
+We can't call page_mlock(), the function that walks the reverse map to
 check for other VM_LOCKED VMAs, without first isolating the page from the LRU.
-try_to_munlock() is a variant of try_to_unmap() and thus requires that the page
+page_mlock() is a variant of try_to_unmap() and thus requires that the page
 not be on an LRU list [more on these below].  However, the call to
-isolate_lru_page() could fail, in which case we couldn't try_to_munlock().  So,
+isolate_lru_page() could fail, in which case we can't call page_mlock().  So,
 we go ahead and clear PG_mlocked up front, as this might be the only chance we
-have.  If we can successfully isolate the page, we go ahead and
-try_to_munlock(), which will restore the PG_mlocked flag and update the zone
+have.  If we can successfully isolate the page, we go ahead and call
+page_mlock(), which will restore the PG_mlocked flag and update the zone
 page statistics if it finds another VMA holding the page mlocked.  If we fail
 to isolate the page, we'll have left a potentially mlocked page on the LRU.
 This is fine, because we'll catch it later if and if vmscan tries to reclaim

@@ -545,31 +545,24 @@ munlock or munmap system calls, mm teardown (munlock_vma_pages_all), reclaim,
 holepunching, and truncation of file pages and their anonymous COWed pages.
 
 
-try_to_munlock() Reverse Map Scan
+page_mlock() Reverse Map Scan
 ---------------------------------
 
-.. warning::
-   [!] TODO/FIXME: a better name might be page_mlocked() - analogous to the
-   page_referenced() reverse map walker.
-
 When munlock_vma_page() [see section :ref:`munlock()/munlockall() System Call
 Handling <munlock_munlockall_handling>` above] tries to munlock a
 page, it needs to determine whether or not the page is mapped by any
 VM_LOCKED VMA without actually attempting to unmap all PTEs from the
 page.  For this purpose, the unevictable/mlock infrastructure
-introduced a variant of try_to_unmap() called try_to_munlock().
+introduced a variant of try_to_unmap() called page_mlock().
 
-try_to_munlock() calls the same functions as try_to_unmap() for anonymous and
-mapped file and KSM pages with a flag argument specifying unlock versus unmap
-processing.  Again, these functions walk the respective reverse maps looking
-for VM_LOCKED VMAs.  When such a VMA is found, as in the try_to_unmap() case,
-the functions mlock the page via mlock_vma_page() and return SWAP_MLOCK.  This
-undoes the pre-clearing of the page's PG_mlocked done by munlock_vma_page.
+page_mlock() walks the respective reverse maps looking for VM_LOCKED VMAs. When
+such a VMA is found the page is mlocked via mlock_vma_page(). This undoes the
+pre-clearing of the page's PG_mlocked done by munlock_vma_page.
 
-Note that try_to_munlock()'s reverse map walk must visit every VMA in a page's
+Note that page_mlock()'s reverse map walk must visit every VMA in a page's
 reverse map to determine that a page is NOT mapped into any VM_LOCKED VMA.
 However, the scan can terminate when it encounters a VM_LOCKED VMA.
-Although try_to_munlock() might be called a great many times when munlocking a
+Although page_mlock() might be called a great many times when munlocking a
 large region or tearing down a large address space that has been mlocked via
 mlockall(), overall this is a fairly rare event.
 

@@ -602,7 +595,7 @@ inactive lists to the appropriate node's unevictable list.
 shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd
 after shrink_active_list() had moved them to the inactive list, or pages mapped
 into VM_LOCKED VMAs that munlock_vma_page() couldn't isolate from the LRU to
-recheck via try_to_munlock().  shrink_inactive_list() won't notice the latter,
+recheck via page_mlock().  shrink_inactive_list() won't notice the latter,
 but will pass on to shrink_page_list().
 
 shrink_page_list() again culls obviously unevictable pages that it could

include/linux/rmap.h +1 -2

index def5c62c93b3..38a746787c2f 100644
@@ -87,7 +87,6 @@ struct anon_vma_chain {
 
 enum ttu_flags {
 	TTU_MIGRATION		= 0x1,	/* migration mode */
-	TTU_MUNLOCK		= 0x2,	/* munlock mode */
 
 	TTU_SPLIT_HUGE_PMD	= 0x4,	/* split huge PMD if any */
 	TTU_IGNORE_MLOCK	= 0x8,	/* ignore mlock */

@@ -239,7 +238,7 @@ int page_mkclean(struct page *);
  * called in munlock()/munmap() path to check for other vmas holding
  * the page mlocked.
  */
-void try_to_munlock(struct page *);
+void page_mlock(struct page *page);
 
 void remove_migration_ptes(struct page *old, struct page *new, bool locked);
 

mm/mlock.c +5 -5

index f8f8cc32d03d..9b8b82cfbbff 100644
@@ -108,7 +108,7 @@ void mlock_vma_page(struct page *page)
 /*
  * Finish munlock after successful page isolation
  *
- * Page must be locked. This is a wrapper for try_to_munlock()
+ * Page must be locked. This is a wrapper for page_mlock()
  * and putback_lru_page() with munlock accounting.
  */
 static void __munlock_isolated_page(struct page *page)

@@ -118,7 +118,7 @@ static void __munlock_isolated_page(struct page *page)
 	 * and we don't need to check all the other vmas.
 	 */
 	if (page_mapcount(page) > 1)
-		try_to_munlock(page);
+		page_mlock(page);
 
 	/* Did try_to_unlock() succeed or punt? */
 	if (!PageMlocked(page))

@@ -158,7 +158,7 @@ static void __munlock_isolation_failed(struct page *page)
  * munlock()ed or munmap()ed, we want to check whether other vmas hold the
  * page locked so that we can leave it on the unevictable lru list and not
  * bother vmscan with it.  However, to walk the page's rmap list in
- * try_to_munlock() we must isolate the page from the LRU.  If some other
+ * page_mlock() we must isolate the page from the LRU.  If some other
  * task has removed the page from the LRU, we won't be able to do that.
  * So we clear the PageMlocked as we might not get another chance.  If we
  * can't isolate the page, we leave it for putback_lru_page() and vmscan

@@ -168,7 +168,7 @@ unsigned int munlock_vma_page(struct page *page)
 {
 	int nr_pages;
 
-	/* For try_to_munlock() and to serialize with page migration */
+	/* For page_mlock() and to serialize with page migration */
 	BUG_ON(!PageLocked(page));
 	VM_BUG_ON_PAGE(PageTail(page), page);
 

@@ -205,7 +205,7 @@ static int __mlock_posix_error_return(long retval)
  *
  * The fast path is available only for evictable pages with single mapping.
  * Then we can bypass the per-cpu pvec and get better performance.
- * when mapcount > 1 we need try_to_munlock() which can fail.
+ * when mapcount > 1 we need page_mlock() which can fail.
  * when !page_evictable(), we need the full redo logic of putback_lru_page to
  * avoid leaving evictable page in unevictable list.
  *

mm/rmap.c +0 -6

index 977e70803ed8..f09d522725b9 100644
@@ -1405,10 +1405,6 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 	struct mmu_notifier_range range;
 	enum ttu_flags flags = (enum ttu_flags)(long)arg;
 
-	/* munlock has nothing to gain from examining un-locked vmas */
-	if ((flags & TTU_MUNLOCK) && !(vma->vm_flags & VM_LOCKED))
-		return true;
-
 	if (IS_ENABLED(CONFIG_MIGRATION) && (flags & TTU_MIGRATION) &&
 	    is_zone_device_page(page) && !is_device_private_page(page))
 		return true;

@@ -1469,8 +1465,6 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 				page_vma_mapped_walk_done(&pvmw);
 				break;
 			}
-			if (flags & TTU_MUNLOCK)
-				continue;
 		}
 
 		/* Unexpected PMD-mapped THP? */

mm/rmap.c +3 -4

  * @page: the page to be munlocked
  *
  * Called from munlock code.  Checks all of the VMAs mapping the page

@@ -1793,11 +1818,10 @@ bool try_to_unmap(struct page *page, enum ttu_flags flags)
  * returned with PG_mlocked cleared if no other vmas have it mlocked.
  */
 
-void try_to_munlock(struct page *page)
+void page_mlock(struct page *page)
 {
 	struct rmap_walk_control rwc = {
-		.rmap_one = try_to_unmap_one,
-		.arg = (void *)TTU_MUNLOCK,
+		.rmap_one = page_mlock_one,
 		.done = page_not_mapped,
 		.anon_lock = page_lock_anon_vma_read,
 

@@ -1849,7 +1873,7 @@ static struct anon_vma *rmap_walk_anon_lock(struct page *page,
  * Find all the mappings of a page using the mapping pointer and the vma chains
  * contained in the anon_vma struct it points to.
  *
- * When called from try_to_munlock(), the mmap_lock of the mm containing the vma
+ * When called from page_mlock(), the mmap_lock of the mm containing the vma
  * where the page was found will be held for write.  So, we won't recheck
  * vm_flags for that VMA.  That should be OK, because that vma shouldn't be
  * LOCKED.

mm/rmap.c +0 -0

  * vm_flags for that VMA.  That should be OK, because that vma shouldn't be
  * LOCKED.

Signature · 1 line

2.20.1

External email: Use caution opening links or attachments

Quoted 266 lines · Alistair Popple

The behaviour of try_to_unmap_one() is difficult to follow because it performs different operations based on a fairly large set of flags used in different combinations.

TTU_MUNLOCK is one such flag. However it is exclusively used by try_to_munlock() which specifies no other flags. Therefore rather than overload try_to_unmap_one() with unrelated behaviour split this out into it's own function and remove the flag.

Signed-off-by Alistair Popple <[email protected]>

Reviewed-by Ralph Campbell <[email protected]>

Reviewed-by Christoph Hellwig <[email protected]>

---

v8:
* Renamed try_to_munlock to page_mlock to better reflect what the

function actually does.

* Removed the TODO from the documentation that this patch addresses.

v7:
* Added Christoph's Reviewed-by

v4:
* Removed redundant check for VM_LOCKED
---

 Documentation/vm/unevictable-lru.rst | 33 ++++++++-----------
 include/linux/rmap.h                 |  3 +-
 mm/mlock.c                           | 10 +++---
 mm/rmap.c                            | 48 +++++++++++++++++++++-------
 4 files changed, 55 insertions(+), 39 deletions(-)

(quoted diff) +0 -0

b/Documentation/vm/unevictable-lru.rst index 0e1490524f53..eae3af17f2d9 100644

Documentation/vm/unevictable-lru.rst +0 -0

@@ -389,14 +389,14 @@ mlocked, munlock_vma_page() updates that zone

statistics for the number of>
mlocked pages. Note, however, that at this point we haven't checked whether the page is mapped by other VM_LOCKED VMAs.

-We can't call try_to_munlock(), the function that walks the reverse map to

Documentation/vm/unevictable-lru.rst +1 -1

+We can't call page_mlock(), the function that walks the reverse map to
 
 check for other VM_LOCKED VMAs, without first isolating the page from the
 LRU.> 
-try_to_munlock() is a variant of try_to_unmap() and thus requires that

the page +page_mlock() is a variant of try_to_unmap() and thus requires that the page>
not be on an LRU list [more on these below]. However, the call to

-isolate_lru_page() could fail, in which case we couldn't try_to_munlock(). So, +isolate_lru_page() could fail, in which case we can't call page_mlock(). So,>
we go ahead and clear PG_mlocked up front, as this might be the only chance we>

Documentation/vm/unevictable-lru.rst +0 -2

-have.  If we can successfully isolate the page, we go ahead and
-try_to_munlock(), which will restore the PG_mlocked flag and update the

zone +have. If we can successfully isolate the page, we go ahead and call +page_mlock(), which will restore the PG_mlocked flag and update the zone>
page statistics if it finds another VMA holding the page mlocked. If we fail to isolate the page, we'll have left a potentially mlocked page on the LRU. This is fine, because we'll catch it later if and if vmscan tries to reclaim>

Documentation/vm/unevictable-lru.rst +0 -0

@@ -545,31 +545,24 @@ munlock or munmap system calls, mm teardown

(munlock_vma_pages_all), reclaim,>
holepunching, and truncation of file pages and their anonymous COWed pages.

Documentation/vm/unevictable-lru.rst +1 -3

-try_to_munlock() Reverse Map Scan
+page_mlock() Reverse Map Scan
 
 ---------------------------------
 
-.. warning::
-   [!] TODO/FIXME: a better name might be page_mlocked() - analogous to

the - page_referenced() reverse map walker.

Documentation/vm/unevictable-lru.rst +1 -3

-
 
 When munlock_vma_page() [see section :ref:`munlock()/munlockall() System
 Call Handling <munlock_munlockall_handling>` above] tries to munlock a
 page, it needs to determine whether or not the page is mapped by any
 VM_LOCKED VMA without actually attempting to unmap all PTEs from the
 page.  For this purpose, the unevictable/mlock infrastructure
 
-introduced a variant of try_to_unmap() called try_to_munlock().
+introduced a variant of try_to_unmap() called page_mlock().
 
-try_to_munlock() calls the same functions as try_to_unmap() for anonymous

and -mapped file and KSM pages with a flag argument specifying unlock versus unmap -processing. Again, these functions walk the respective reverse maps looking -for VM_LOCKED VMAs. When such a VMA is found, as in the try_to_unmap() case, -the functions mlock the page via mlock_vma_page() and return SWAP_MLOCK. This -undoes the pre-clearing of the page's PG_mlocked done by munlock_vma_page. +page_mlock() walks the respective reverse maps looking for VM_LOCKED VMAs. When +such a VMA is found the page is mlocked via mlock_vma_page(). This undoes the

Documentation/vm/unevictable-lru.rst +1 -1

+pre-clearing of the page's PG_mlocked done by munlock_vma_page.
 
-Note that try_to_munlock()'s reverse map walk must visit every VMA in a

page's +Note that page_mlock()'s reverse map walk must visit every VMA in a page's>
reverse map to determine that a page is NOT mapped into any VM_LOCKED VMA.
However, the scan can terminate when it encounters a VM_LOCKED VMA.

-Although try_to_munlock() might be called a great many times when munlocking a +Although page_mlock() might be called a great many times when munlocking a>
large region or tearing down a large address space that has been mlocked via mlockall(), overall this is a fairly rare event.

Documentation/vm/unevictable-lru.rst +0 -0

@@ -602,7 +595,7 @@ inactive lists to the appropriate node's unevictable

list.>
shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd after shrink_active_list() had moved them to the inactive list, or pages mapped into VM_LOCKED VMAs that munlock_vma_page() couldn't isolate from the LRU to>
-recheck via try_to_munlock(). shrink_inactive_list() won't notice the latter, +recheck via page_mlock(). shrink_inactive_list() won't notice the latter,>
but will pass on to shrink_page_list().

shrink_page_list() again culls obviously unevictable pages that it could

include/linux/rmap.h +0 -1

index def5c62c93b3..38a746787c2f 100644
@@ -87,7 +87,6 @@ struct anon_vma_chain {
 
 enum ttu_flags {
 
      TTU_MIGRATION           = 0x1,  /* migration mode */
 
-     TTU_MUNLOCK             = 0x2,  /* munlock mode */
 

      TTU_SPLIT_HUGE_PMD      = 0x4,  /* split huge PMD if any */
      TTU_IGNORE_MLOCK        = 0x8,  /* ignore mlock */

include/linux/rmap.h +1 -1

@@ -239,7 +238,7 @@ int page_mkclean(struct page *);
 
  * called in munlock()/munmap() path to check for other vmas holding
  * the page mlocked.
  */
 
-void try_to_munlock(struct page *);
+void page_mlock(struct page *page);
 

void remove_migration_ptes(struct page *old, struct page *new, bool locked);>

mm/mlock.c +1 -1

index f8f8cc32d03d..9b8b82cfbbff 100644
@@ -108,7 +108,7 @@ void mlock_vma_page(struct page *page)
 
 /*
 
  * Finish munlock after successful page isolation
  *
 
- * Page must be locked. This is a wrapper for try_to_munlock()
+ * Page must be locked. This is a wrapper for page_mlock()

* and putback_lru_page() with munlock accounting. */

static void __munlock_isolated_page(struct page *page)

mm/mlock.c +1 -1

@@ -118,7 +118,7 @@ static void __munlock_isolated_page(struct page *page)
 
       * and we don't need to check all the other vmas.
       */
 
      if (page_mapcount(page) > 1)
 
-             try_to_munlock(page);
+             page_mlock(page);

      /* Did try_to_unlock() succeed or punt? */
      if (!PageMlocked(page))

mm/mlock.c +2 -2

@@ -158,7 +158,7 @@ static void __munlock_isolation_failed(struct page
 *page)> 
  * munlock()ed or munmap()ed, we want to check whether other vmas hold
  the
  * page locked so that we can leave it on the unevictable lru list and
  not
  * bother vmscan with it.  However, to walk the page's rmap list in
 
- * try_to_munlock() we must isolate the page from the LRU.  If some other
+ * page_mlock() we must isolate the page from the LRU.  If some other
 
  * task has removed the page from the LRU, we won't be able to do that.
  * So we clear the PageMlocked as we might not get another chance.  If we
  * can't isolate the page, we leave it for putback_lru_page() and vmscan

@@ -168,7 +168,7 @@ unsigned int munlock_vma_page(struct page *page)
 
 {
 
      int nr_pages;
 
-     /* For try_to_munlock() and to serialize with page migration */
+     /* For page_mlock() and to serialize with page migration */
 

      BUG_ON(!PageLocked(page));
      VM_BUG_ON_PAGE(PageTail(page), page);

mm/mlock.c +1 -1

@@ -205,7 +205,7 @@ static int __mlock_posix_error_return(long retval)
 
  *
  * The fast path is available only for evictable pages with single
  mapping.
  * Then we can bypass the per-cpu pvec and get better performance.
 
- * when mapcount > 1 we need try_to_munlock() which can fail.
+ * when mapcount > 1 we need page_mlock() which can fail.

* when !page_evictable(), we need the full redo logic of putback_lru_page to * avoid leaving evictable page in unevictable list. *

mm/rmap.c +0 -0

index 977e70803ed8..f09d522725b9 100644
@@ -1405,10 +1405,6 @@ static bool try_to_unmap_one(struct page *page,

struct vm_area_struct *vma,>

      struct mmu_notifier_range range;
      enum ttu_flags flags = (enum ttu_flags)(long)arg;

mm/rmap.c +0 -4

-     /* munlock has nothing to gain from examining un-locked vmas */
-     if ((flags & TTU_MUNLOCK) && !(vma->vm_flags & VM_LOCKED))
-             return true;
-
 
      if (IS_ENABLED(CONFIG_MIGRATION) && (flags & TTU_MIGRATION) &&
 
          is_zone_device_page(page) && !is_device_private_page(page))
 
              return true;

@@ -1469,8 +1465,6 @@ static bool try_to_unmap_one(struct page *page,

struct vm_area_struct *vma,>

                              page_vma_mapped_walk_done(&pvmw);
                              break;
                      
                      }

mm/rmap.c +0 -2

-                     if (flags & TTU_MUNLOCK)
-                             continue;
 
              }
 
              /* Unexpected PMD-mapped THP? */

@@ -1784,8 +1778,39 @@ bool try_to_unmap(struct page *page, enum ttu_flags

flags)>

Quoted 3 lines

      return !page_mapcount(page) ? true : false;

}

Please add a comment here, especially around locking.

mm/rmap.c bool try_to_unmap(struct page *page, enum ttu_flags flags) · line 1792

Quoted 12 lines

mm/rmap.c +12 -0

+static bool page_mlock_one(struct page *page, struct vm_area_struct *vma,
+                              unsigned long address, void *arg)
+{
+     struct page_vma_mapped_walk pvmw = {
+             .page = page,
+             .vma = vma,
+             .address = address,
+     };
+
+     /* munlock has nothing to gain from examining un-locked vmas */
+     if (!(vma->vm_flags & VM_LOCKED))
+             return true;

The logic here doesn't make sense. You called page_mlock_one() on a VMA that isn't locked and it returns true? Is this a check to see if the VMA has zero mlock'ed pages?

mm/rmap.c bool try_to_unmap(struct page *page, enum ttu_flags flags) · line 1800

Quoted 8 lines

mm/rmap.c +8 -0

+
+     while (page_vma_mapped_walk(&pvmw)) {
+             /* PTE-mapped THP are never mlocked */
+             if (!PageTransCompound(page)) {
+                     /*
+                      * Holding pte lock, we do *not* need
+                      * mmap_lock here
+                      */

Are you sure? I think you at least need to hold the mmap lock for reading to ensure there's no race here? mlock_vma_page() eludes to such a scenario when lazy mlocking.

The mmap_lock is held for writing in the scenarios I have checked.

Quoted 6 lines

mm/rmap.c +5 -0

+                     mlock_vma_page(page);
+             }
+             page_vma_mapped_walk_done(&pvmw);
+
+             /* found a mlocked page, no point continuing munlock check

*/

I think you need to check_pte() to be sure it is mapped?

mm/rmap.c bool try_to_unmap(struct page *page, enum ttu_flags flags) · line 1809

Quoted 4 lines

mm/rmap.c +4 -0

+             return false;
+     }
+
+     return true;

Again, I don't get the return values. If page_mlock_one() returns true, I'd expect for my page to now be locked. This isn't the case here, page_mlock_one() returns true if there are no pages present for a locked VMA, correct?

mm/rmap.c bool try_to_unmap(struct page *page, enum ttu_flags flags) · line 1813

Quoted 7 lines

mm/rmap.c +3 -1

+}
+
 
 /**
 
- * try_to_munlock - try to munlock a page
+ * page_mlock - try to munlock a page

Is this an mlock or an munlock? I'm not confident it's either, but more of a check to see if there are pages mapped in a locked VMA?

Quoted 43 lines

* @page: the page to be munlocked
*
* Called from munlock code. Checks all of the VMAs mapping the page

mm/rmap.c +0 -0

@@ -1793,11 +1818,10 @@ bool try_to_unmap(struct page *page, enum

ttu_flags flags)>
* returned with PG_mlocked cleared if no other vmas have it mlocked. */

mm/rmap.c +2 -3

-void try_to_munlock(struct page *page)
+void page_mlock(struct page *page)
 
 {
 
      struct rmap_walk_control rwc = {
 
-             .rmap_one = try_to_unmap_one,
-             .arg = (void *)TTU_MUNLOCK,
+             .rmap_one = page_mlock_one,
 
              .done = page_not_mapped,
              .anon_lock = page_lock_anon_vma_read,

@@ -1849,7 +1873,7 @@ static struct anon_vma *rmap_walk_anon_lock(struct

page *page,>
* Find all the mappings of a page using the mapping pointer and the vma chains * contained in the anon_vma struct it points to.
*

- * When called from try_to_munlock(), the mmap_lock of the mm containing the vma + * When called from page_mlock(), the mmap_lock of the mm containing the vma>
* where the page was found will be held for write. So, we won't recheck
* vm_flags for that VMA. That should be OK, because that vma shouldn't be
* LOCKED.

mm/rmap.c +0 -0

@@ -1901,7 +1925,7 @@ static void rmap_walk_anon(struct page *page, struct

rmap_walk_control *rwc,>
* Find all the mappings of a page using the mapping pointer and the vma chains * contained in the address_space struct it points to. *

Quoted 4 lines

- * When called from try_to_munlock(), the mmap_lock of the mm containing the vma + * When called from page_mlock(), the mmap_lock of the mm containing the vma>
* where the page was found will be held for write. So, we won't recheck

Above it is stated that the lock does not need to be held, but this comment says it is already held for writing - which is it?

Quoted 6 lines

* vm_flags for that VMA. That should be OK, because that vma shouldn't be
* LOCKED.

--
2.20.1

munlock_vma_pages_range() comments references try_to_{munlock|unmap}

Quoted 267 lines · AEST Liam Howlett

External email: Use caution opening links or attachments

Quoted 264 lines · Alistair Popple

The behaviour of try_to_unmap_one() is difficult to follow because it performs different operations based on a fairly large set of flags used in different combinations.

TTU_MUNLOCK is one such flag. However it is exclusively used by try_to_munlock() which specifies no other flags. Therefore rather than overload try_to_unmap_one() with unrelated behaviour split this out into it's own function and remove the flag.

Signed-off-by Alistair Popple <[email protected]>

Reviewed-by Ralph Campbell <[email protected]>

Reviewed-by Christoph Hellwig <[email protected]>

---

v8:
* Renamed try_to_munlock to page_mlock to better reflect what the

function actually does.

* Removed the TODO from the documentation that this patch addresses.

v7:
* Added Christoph's Reviewed-by

v4:
* Removed redundant check for VM_LOCKED
---

 Documentation/vm/unevictable-lru.rst | 33 ++++++++-----------
 include/linux/rmap.h                 |  3 +-
 mm/mlock.c                           | 10 +++---
 mm/rmap.c                            | 48 +++++++++++++++++++++-------
 4 files changed, 55 insertions(+), 39 deletions(-)

(quoted diff) +0 -0

b/Documentation/vm/unevictable-lru.rst index 0e1490524f53..eae3af17f2d9 100644

Documentation/vm/unevictable-lru.rst +0 -0

@@ -389,14 +389,14 @@ mlocked, munlock_vma_page() updates that zone

statistics for the number of>
mlocked pages. Note, however, that at this point we haven't checked whether the page is mapped by other VM_LOCKED VMAs.

-We can't call try_to_munlock(), the function that walks the reverse map to

Documentation/vm/unevictable-lru.rst +1 -1

+We can't call page_mlock(), the function that walks the reverse map to
 
 check for other VM_LOCKED VMAs, without first isolating the page from the
 LRU.> 
-try_to_munlock() is a variant of try_to_unmap() and thus requires that

the page +page_mlock() is a variant of try_to_unmap() and thus requires that the page>
not be on an LRU list [more on these below]. However, the call to

-isolate_lru_page() could fail, in which case we couldn't try_to_munlock(). So, +isolate_lru_page() could fail, in which case we can't call page_mlock(). So,>
we go ahead and clear PG_mlocked up front, as this might be the only chance we>

Documentation/vm/unevictable-lru.rst +0 -2

-have.  If we can successfully isolate the page, we go ahead and
-try_to_munlock(), which will restore the PG_mlocked flag and update the

zone +have. If we can successfully isolate the page, we go ahead and call +page_mlock(), which will restore the PG_mlocked flag and update the zone>
page statistics if it finds another VMA holding the page mlocked. If we fail to isolate the page, we'll have left a potentially mlocked page on the LRU. This is fine, because we'll catch it later if and if vmscan tries to reclaim>

Documentation/vm/unevictable-lru.rst +0 -0

@@ -545,31 +545,24 @@ munlock or munmap system calls, mm teardown

(munlock_vma_pages_all), reclaim,>
holepunching, and truncation of file pages and their anonymous COWed pages.

Documentation/vm/unevictable-lru.rst +1 -3

-try_to_munlock() Reverse Map Scan
+page_mlock() Reverse Map Scan
 
 ---------------------------------
 
-.. warning::
-   [!] TODO/FIXME: a better name might be page_mlocked() - analogous to

the - page_referenced() reverse map walker.

Documentation/vm/unevictable-lru.rst +1 -3

-
 
 When munlock_vma_page() [see section :ref:`munlock()/munlockall() System
 Call Handling <munlock_munlockall_handling>` above] tries to munlock a
 page, it needs to determine whether or not the page is mapped by any
 VM_LOCKED VMA without actually attempting to unmap all PTEs from the
 page.  For this purpose, the unevictable/mlock infrastructure
 
-introduced a variant of try_to_unmap() called try_to_munlock().
+introduced a variant of try_to_unmap() called page_mlock().
 
-try_to_munlock() calls the same functions as try_to_unmap() for anonymous

and -mapped file and KSM pages with a flag argument specifying unlock versus unmap -processing. Again, these functions walk the respective reverse maps looking -for VM_LOCKED VMAs. When such a VMA is found, as in the try_to_unmap() case, -the functions mlock the page via mlock_vma_page() and return SWAP_MLOCK. This -undoes the pre-clearing of the page's PG_mlocked done by munlock_vma_page. +page_mlock() walks the respective reverse maps looking for VM_LOCKED VMAs. When +such a VMA is found the page is mlocked via mlock_vma_page(). This undoes the

Documentation/vm/unevictable-lru.rst +1 -1

+pre-clearing of the page's PG_mlocked done by munlock_vma_page.
 
-Note that try_to_munlock()'s reverse map walk must visit every VMA in a

page's +Note that page_mlock()'s reverse map walk must visit every VMA in a page's>
reverse map to determine that a page is NOT mapped into any VM_LOCKED VMA.
However, the scan can terminate when it encounters a VM_LOCKED VMA.

-Although try_to_munlock() might be called a great many times when munlocking a +Although page_mlock() might be called a great many times when munlocking a>
large region or tearing down a large address space that has been mlocked via mlockall(), overall this is a fairly rare event.

Documentation/vm/unevictable-lru.rst +0 -0

@@ -602,7 +595,7 @@ inactive lists to the appropriate node's unevictable

list.>
shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd after shrink_active_list() had moved them to the inactive list, or pages mapped into VM_LOCKED VMAs that munlock_vma_page() couldn't isolate from the LRU to>
-recheck via try_to_munlock(). shrink_inactive_list() won't notice the latter, +recheck via page_mlock(). shrink_inactive_list() won't notice the latter,>
but will pass on to shrink_page_list().

shrink_page_list() again culls obviously unevictable pages that it could

include/linux/rmap.h +0 -1

index def5c62c93b3..38a746787c2f 100644
@@ -87,7 +87,6 @@ struct anon_vma_chain {
 
 enum ttu_flags {
 
      TTU_MIGRATION           = 0x1,  /* migration mode */
 
-     TTU_MUNLOCK             = 0x2,  /* munlock mode */
 

      TTU_SPLIT_HUGE_PMD      = 0x4,  /* split huge PMD if any */
      TTU_IGNORE_MLOCK        = 0x8,  /* ignore mlock */

include/linux/rmap.h +1 -1

@@ -239,7 +238,7 @@ int page_mkclean(struct page *);
 
  * called in munlock()/munmap() path to check for other vmas holding
  * the page mlocked.
  */
 
-void try_to_munlock(struct page *);
+void page_mlock(struct page *page);
 

void remove_migration_ptes(struct page *old, struct page *new, bool locked);>

mm/mlock.c +1 -1

index f8f8cc32d03d..9b8b82cfbbff 100644
@@ -108,7 +108,7 @@ void mlock_vma_page(struct page *page)
 
 /*
 
  * Finish munlock after successful page isolation
  *
 
- * Page must be locked. This is a wrapper for try_to_munlock()
+ * Page must be locked. This is a wrapper for page_mlock()

* and putback_lru_page() with munlock accounting. */

static void __munlock_isolated_page(struct page *page)

mm/mlock.c +1 -1

@@ -118,7 +118,7 @@ static void __munlock_isolated_page(struct page *page)
 
       * and we don't need to check all the other vmas.
       */
 
      if (page_mapcount(page) > 1)
 
-             try_to_munlock(page);
+             page_mlock(page);

      /* Did try_to_unlock() succeed or punt? */
      if (!PageMlocked(page))

mm/mlock.c +2 -2

@@ -158,7 +158,7 @@ static void __munlock_isolation_failed(struct page
 *page)> 
  * munlock()ed or munmap()ed, we want to check whether other vmas hold
  the
  * page locked so that we can leave it on the unevictable lru list and
  not
  * bother vmscan with it.  However, to walk the page's rmap list in
 
- * try_to_munlock() we must isolate the page from the LRU.  If some other
+ * page_mlock() we must isolate the page from the LRU.  If some other
 
  * task has removed the page from the LRU, we won't be able to do that.
  * So we clear the PageMlocked as we might not get another chance.  If we
  * can't isolate the page, we leave it for putback_lru_page() and vmscan

@@ -168,7 +168,7 @@ unsigned int munlock_vma_page(struct page *page)
 
 {
 
      int nr_pages;
 
-     /* For try_to_munlock() and to serialize with page migration */
+     /* For page_mlock() and to serialize with page migration */
 

      BUG_ON(!PageLocked(page));
      VM_BUG_ON_PAGE(PageTail(page), page);

mm/mlock.c +1 -1

@@ -205,7 +205,7 @@ static int __mlock_posix_error_return(long retval)
 
  *
  * The fast path is available only for evictable pages with single
  mapping.
  * Then we can bypass the per-cpu pvec and get better performance.
 
- * when mapcount > 1 we need try_to_munlock() which can fail.
+ * when mapcount > 1 we need page_mlock() which can fail.

* when !page_evictable(), we need the full redo logic of putback_lru_page to * avoid leaving evictable page in unevictable list. *

mm/rmap.c +0 -0

index 977e70803ed8..f09d522725b9 100644
@@ -1405,10 +1405,6 @@ static bool try_to_unmap_one(struct page *page,

struct vm_area_struct *vma,>

      struct mmu_notifier_range range;
      enum ttu_flags flags = (enum ttu_flags)(long)arg;

mm/rmap.c +0 -4

-     /* munlock has nothing to gain from examining un-locked vmas */
-     if ((flags & TTU_MUNLOCK) && !(vma->vm_flags & VM_LOCKED))
-             return true;
-
 
      if (IS_ENABLED(CONFIG_MIGRATION) && (flags & TTU_MIGRATION) &&
 
          is_zone_device_page(page) && !is_device_private_page(page))
 
              return true;

@@ -1469,8 +1465,6 @@ static bool try_to_unmap_one(struct page *page,

struct vm_area_struct *vma,>

                              page_vma_mapped_walk_done(&pvmw);
                              break;
                      
                      }

mm/rmap.c +0 -2

-                     if (flags & TTU_MUNLOCK)
-                             continue;
 
              }
 
              /* Unexpected PMD-mapped THP? */

Quoted 7 lines

Quoted 5 lines

mm/rmap.c +0 -0

@@ -1784,8 +1778,39 @@ bool try_to_unmap(struct page *page, enum ttu_flags

flags)>

      return !page_mapcount(page) ? true : false;

}

Please add a comment here, especially around locking.

Quoted 17 lines

mm/rmap.c bool try_to_unmap(struct page *page, enum ttu_flags flags) · line 1792

Quoted 13 lines

mm/rmap.c +12 -0

+static bool page_mlock_one(struct page *page, struct vm_area_struct *vma,
+                              unsigned long address, void *arg)
+{
+     struct page_vma_mapped_walk pvmw = {
+             .page = page,
+             .vma = vma,
+             .address = address,
+     };
+
+     /* munlock has nothing to gain from examining un-locked vmas */
+     if (!(vma->vm_flags & VM_LOCKED))
+             return true;

The logic here doesn't make sense. You called page_mlock_one() on a VMA that isn't locked and it returns true? Is this a check to see if the VMA has zero mlock'ed pages?

I'm pretty sure the logic is correct. This is used for an rmap_walk, so we return true to continue the page table scan to see if other VMAs have the page locked.

Quoted 10 lines

mm/rmap.c bool try_to_unmap(struct page *page, enum ttu_flags flags) · line 1800

Quoted 9 lines

mm/rmap.c +8 -0

+
+     while (page_vma_mapped_walk(&pvmw)) {
+             /* PTE-mapped THP are never mlocked */
+             if (!PageTransCompound(page)) {
+                     /*
+                      * Holding pte lock, we do *not* need
+                      * mmap_lock here
+                      */

Quoted 3 lines

Are you sure? I think you at least need to hold the mmap lock for reading to ensure there's no race here? mlock_vma_page() eludes to such a scenario when lazy mlocking.

Not really. I don't claim to be an mlock expert but as this is a clean-up for try_to_unmap() the intent was to not change existing behaviour.

However presenting the function in this simplified form did raise this and some other questions during previous reviews - see https://lore.kernel.org/
dri-devel/[email protected]/ for the previous discussion.

To answer the questions around locking though I did do some git sha1 mining. The best explanation seemed to be contained in https://git.kernel.org/pub/scm/
linux/kernel/git/torvalds/linux.git/commit/? id=b87537d9e2feb30f6a962f27eb32768682698d3b from Hugh (whom I've added again here in case he can help answer some of these).

Quoted 11 lines

The mmap_lock is held for writing in the scenarios I have checked.

Quoted 6 lines

mm/rmap.c +5 -0

+                     mlock_vma_page(page);
+             }
+             page_vma_mapped_walk_done(&pvmw);
+
+             /* found a mlocked page, no point continuing munlock check

*/

I think you need to check_pte() to be sure it is mapped?

Quoted 6 lines

Quoted 5 lines

mm/rmap.c +4 -0

+             return false;
+     }
+
+     return true;

Quoted 4 lines

Again, I don't get the return values. If page_mlock_one() returns true, I'd expect for my page to now be locked. This isn't the case here, page_mlock_one() returns true if there are no pages present for a locked VMA, correct?

This is used for an rmap walk, if we were able to mlock the page we return false to stop the walk as there is no need to examine other VMAs if the page has already been mlocked.

Quoted 6 lines

Quoted 6 lines

mm/rmap.c +2 -0

+}
+
 
 /**

Quoted 5 lines

Quoted 2 lines

- * try_to_munlock - try to munlock a page
+ * page_mlock - try to munlock a page

Is this an mlock or an munlock? I'm not confident it's either, but more of a check to see if there are pages mapped in a locked VMA?

It is called (only) from the munlock code to check a page does not need to be mlocked, or to mlock it if it does.

Quoted 44 lines

Quoted 44 lines

* @page: the page to be munlocked
*
* Called from munlock code. Checks all of the VMAs mapping the page

mm/rmap.c +0 -0

@@ -1793,11 +1818,10 @@ bool try_to_unmap(struct page *page, enum

ttu_flags flags)>
* returned with PG_mlocked cleared if no other vmas have it mlocked. */

mm/rmap.c +2 -3

-void try_to_munlock(struct page *page)
+void page_mlock(struct page *page)
 
 {
 
      struct rmap_walk_control rwc = {
 
-             .rmap_one = try_to_unmap_one,
-             .arg = (void *)TTU_MUNLOCK,
+             .rmap_one = page_mlock_one,
 
              .done = page_not_mapped,
              .anon_lock = page_lock_anon_vma_read,

@@ -1849,7 +1873,7 @@ static struct anon_vma *rmap_walk_anon_lock(struct

page *page,>
* Find all the mappings of a page using the mapping pointer and the vma chains * contained in the anon_vma struct it points to.
*

- * When called from try_to_munlock(), the mmap_lock of the mm containing the vma + * When called from page_mlock(), the mmap_lock of the mm containing the vma>
* where the page was found will be held for write. So, we won't recheck
* vm_flags for that VMA. That should be OK, because that vma shouldn't be
* LOCKED.

mm/rmap.c +0 -0

@@ -1901,7 +1925,7 @@ static void rmap_walk_anon(struct page *page, struct

rmap_walk_control *rwc,>
* Find all the mappings of a page using the mapping pointer and the vma chains * contained in the address_space struct it points to. *

Quoted 7 lines

Quoted 4 lines

- * When called from try_to_munlock(), the mmap_lock of the mm containing the vma + * When called from page_mlock(), the mmap_lock of the mm containing the vma>
* where the page was found will be held for write. So, we won't recheck

Above it is stated that the lock does not need to be held, but this comment says it is already held for writing - which is it?

I will have to check.

Quoted 8 lines

Quoted 6 lines

* vm_flags for that VMA. That should be OK, because that vma shouldn't be
* LOCKED.

--
2.20.1

munlock_vma_pages_range() comments references try_to_{munlock|unmap}

Thanks, I noticed that too when I was rereading it just now. Will fix that up.

- Alistair

Quoted 40 lines

Quoted 40 lines

Quoted 40 lines

mm/rmap.c +0 -0

index 977e70803ed8..f09d522725b9 100644
@@ -1405,10 +1405,6 @@ static bool try_to_unmap_one(struct page *page,

struct vm_area_struct *vma,>

      struct mmu_notifier_range range;
      enum ttu_flags flags = (enum ttu_flags)(long)arg;

mm/rmap.c +0 -4

-     /* munlock has nothing to gain from examining un-locked vmas */
-     if ((flags & TTU_MUNLOCK) && !(vma->vm_flags & VM_LOCKED))
-             return true;
-
 
      if (IS_ENABLED(CONFIG_MIGRATION) && (flags & TTU_MIGRATION) &&
 
          is_zone_device_page(page) && !is_device_private_page(page))
 
              return true;

@@ -1469,8 +1465,6 @@ static bool try_to_unmap_one(struct page *page,

struct vm_area_struct *vma,>

                              page_vma_mapped_walk_done(&pvmw);
                              break;
                      
                      }

mm/rmap.c +0 -2

-                     if (flags & TTU_MUNLOCK)
-                             continue;
 
              }
 
              /* Unexpected PMD-mapped THP? */

@@ -1784,8 +1778,39 @@ bool try_to_unmap(struct page *page, enum

ttu_flags
flags)>

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

      return !page_mapcount(page) ? true : false;

}

Please add a comment here, especially around locking.

Did you miss this comment? I think the name confusion alone means this needs some documentation. It's also worth mentioning arg is unused.

Quoted 18 lines

Quoted 17 lines

mm/rmap.c bool try_to_unmap(struct page *page, enum ttu_flags flags) · line 1792

Quoted 13 lines

+static bool page_mlock_one(struct page *page, struct vm_area_struct *vma,

mm/rmap.c +11 -0

+                              unsigned long address, void *arg)
+{
+     struct page_vma_mapped_walk pvmw = {
+             .page = page,
+             .vma = vma,
+             .address = address,
+     };
+
+     /* munlock has nothing to gain from examining un-locked vmas */
+     if (!(vma->vm_flags & VM_LOCKED))
+             return true;

The logic here doesn't make sense. You called page_mlock_one() on a VMA that isn't locked and it returns true? Is this a check to see if the VMA has zero mlock'ed pages?

Quoted 3 lines

I'm pretty sure the logic is correct. This is used for an rmap_walk, so we return true to continue the page table scan to see if other VMAs have the page locked.

yes, sorry. The logic is correct but doesn't read as though it does. I cannot see what is going on easily and there are no comments stating what is happening.

Quoted 13 lines

Quoted 9 lines

mm/rmap.c bool try_to_unmap(struct page *page, enum ttu_flags flags) · line 1800

Quoted 8 lines

mm/rmap.c +8 -0

+
+     while (page_vma_mapped_walk(&pvmw)) {
+             /* PTE-mapped THP are never mlocked */
+             if (!PageTransCompound(page)) {
+                     /*
+                      * Holding pte lock, we do *not* need
+                      * mmap_lock here
+                      */

Quoted 3 lines

Are you sure? I think you at least need to hold the mmap lock for reading to ensure there's no race here? mlock_vma_page() eludes to such a scenario when lazy mlocking.

Quoted 8 lines

Not really. I don't claim to be an mlock expert but as this is a clean-up for try_to_unmap() the intent was to not change existing behaviour.

However presenting the function in this simplified form did raise this and some other questions during previous reviews - see
https://lore.kernel.org/
dri-devel/[email protected]/ for the previous discussion.

From what I can see, at least the following paths have mmap_lock held for writing:

munlock_vma_pages_range() from __do_munmap()
munlokc_vma_pages_range() from remap_file_pages()

Quoted 6 lines

To answer the questions around locking though I did do some git sha1 mining. The best explanation seemed to be contained in
https://git.kernel.org/pub/scm/
linux/kernel/git/torvalds/linux.git/commit/? id=b87537d9e2feb30f6a962f27eb32768682698d3b from Hugh (whom I've added again here in case he can help answer some of these).

Thanks for the pointer. That race doesn't make the lock unnecessary. It is the exception to the rule because the exit of the process will correct what has happened - and in all other cases, the mmap_lock is held already (or ought to be) at a higher level. I think this comment is just wrong and should be removed as it is confusing. I see that this is a copy from elsewhere, but it's at least wrong in this context.

Quoted 10 lines

Quoted 10 lines

The mmap_lock is held for writing in the scenarios I have checked.

Quoted 7 lines

mm/rmap.c +5 -0

+                     mlock_vma_page(page);
+             }
+             page_vma_mapped_walk_done(&pvmw);
+
+             /* found a mlocked page, no point continuing munlock

check
*/

Quoted 1 line

Quoted 1 line

I think you need to check_pte() to be sure it is mapped?

On second thought, my comment here is wrong.

Quoted 10 lines

Quoted 9 lines

mm/rmap.c bool try_to_unmap(struct page *page, enum ttu_flags flags) · line 1809

Quoted 4 lines

mm/rmap.c +4 -0

+             return false;
+     }
+
+     return true;

Again, I don't get the return values. If page_mlock_one() returns true, I'd expect for my page to now be locked. This isn't the case here, page_mlock_one() returns true if there are no pages present for a locked VMA, correct?

Quoted 3 lines

This is used for an rmap walk, if we were able to mlock the page we return false to stop the walk as there is no need to examine other VMAs if the page has already been mlocked.

My confusion is about the function name. The code is not self-documenting as it is written and there are no comments. Maybe add a comment block saying what it returns and why?

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

mm/rmap.c +2 -0

+}
+
 
 /**

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

- * try_to_munlock - try to munlock a page
+ * page_mlock - try to munlock a page

Is this an mlock or an munlock? I'm not confident it's either, but more of a check to see if there are pages mapped in a locked VMA?

It is called (only) from the munlock code to check a page does not need to be mlocked, or to mlock it if it does.

I don't want to re-open the discussion on the name, but I just might with this comment: Do you think it is correct to have a comment for page_mlock that says "try to munlock a page"? Those seem like opposite functions to me. The @page also references the munlock'ing. This is ensuring a locked page that is also locked elsewhere will not be removed, right?

Quoted 80 lines

Quoted 59 lines

Quoted 59 lines

* @page: the page to be munlocked
*
* Called from munlock code. Checks all of the VMAs mapping the page

mm/rmap.c +0 -0

@@ -1793,11 +1818,10 @@ bool try_to_unmap(struct page *page, enum

ttu_flags flags)>

* returned with PG_mlocked cleared if no other vmas have it mlocked. */

mm/rmap.c +2 -3

-void try_to_munlock(struct page *page)
+void page_mlock(struct page *page)
 
 {
 
      struct rmap_walk_control rwc = {
 
-             .rmap_one = try_to_unmap_one,
-             .arg = (void *)TTU_MUNLOCK,
+             .rmap_one = page_mlock_one,
 
              .done = page_not_mapped,
              .anon_lock = page_lock_anon_vma_read,

@@ -1849,7 +1873,7 @@ static struct anon_vma
 *rmap_walk_anon_lock(struct

page *page,>

* Find all the mappings of a page using the mapping pointer and the vma
chains * contained in the anon_vma struct it points to. *

- * When called from try_to_munlock(), the mmap_lock of the mm containing
the vma + * When called from page_mlock(), the mmap_lock of the mm containing the vma>

* where the page was found will be held for write. So, we won't recheck
* vm_flags for that VMA. That should be OK, because that vma shouldn't
be
* LOCKED.

mm/rmap.c +0 -0

@@ -1901,7 +1925,7 @@ static void rmap_walk_anon(struct page *page,

struct
rmap_walk_control *rwc,>

* Find all the mappings of a page using the mapping pointer and the vma
chains * contained in the address_space struct it points to. *

- * When called from try_to_munlock(), the mmap_lock of the mm containing
the vma + * When called from page_mlock(), the mmap_lock of the mm containing the vma>

Quoted 5 lines

Quoted 2 lines

* where the page was found will be held for write. So, we won't recheck

Above it is stated that the lock does not need to be held, but this comment says it is already held for writing - which is it?

I will have to check.

Quoted 9 lines

Quoted 7 lines

* vm_flags for that VMA. That should be OK, because that vma shouldn't
be
* LOCKED.

--
2.20.1

munlock_vma_pages_range() comments references try_to_{munlock|unmap}

Thanks, I noticed that too when I was rereading it just now. Will fix that up.>
- Alistair

Some devices require exclusive write access to shared virtual memory (SVM) ranges to perform atomic operations on that memory. This requires CPU page tables to be updated to deny access whilst atomic operations are occurring.

In order to do this introduce a new swap entry
type (SWP_DEVICE_EXCLUSIVE). When a SVM range needs to be marked for exclusive access by a device all page table mappings for the particular range are replaced with device exclusive swap entries. This causes any CPU access to the page to result in a fault.

Faults are resovled by replacing the faulting entry with the original mapping. This results in MMU notifiers being called which a driver uses to update access permissions such as revoking atomic access. After notifiers have been called the device will no longer have exclusive access to the region.

Signed-off-by Alistair Popple <[email protected]>

Reviewed-by Christoph Hellwig <[email protected]>

---

v8:
* Remove device exclusive entries on fork rather than copy them.

v7:
* Added Christoph's Reviewed-by.
* Minor cosmetic cleanups suggested by Christoph.
* Replace mmu_notifier_range_init_migrate/exclusive with mmu_notifier_range_init_owner as suggested by Christoph.
* Replaced lock_page() with lock_page_retry() when handling faults.
* Restrict to anonymous pages for now.

v6:
* Fixed a bisectablity issue due to incorrectly applying the rename of migrate_pgmap_owner to the wrong patches for Nouveau and hmm_test.

v5:
* Renamed range->migrate_pgmap_owner to range->owner.

* Added MMU_NOTIFY_EXCLUSIVE to allow passing of a driver cookie which allows notifiers called as a result of make_device_exclusive_range() to be ignored.
* Added a check to try_to_protect_one() to detect if the pages originally returned from get_user_pages() have been unmapped or not.
* Removed check_device_exclusive_range() as it is no longer required with the other changes.
* Documentation update.

v4:
* Add function to check that mappings are still valid and exclusive.
* s/long/unsigned long/ in make_device_exclusive_entry().

---
 Documentation/vm/hmm.rst              |  19 ++-
 drivers/gpu/drm/nouveau/nouveau_svm.c |   2 +-
 include/linux/mmu_notifier.h          |  26 ++--
 include/linux/rmap.h                  |   4 +
 include/linux/swap.h                  |   4 +-
 include/linux/swapops.h               |  44 +++++-
 lib/test_hmm.c                        |   2 +-
 mm/hmm.c                              |   5 +
 mm/memory.c                           | 176 +++++++++++++++++++--
 mm/migrate.c                          |  10 +-
 mm/mprotect.c                         |   8 +
 mm/page_vma_mapped.c                  |   9 +-
 mm/rmap.c                             | 210 ++++++++++++++++++++++++++
 13 files changed, 487 insertions(+), 32 deletions(-)

Documentation/vm/hmm.rst +18 -1

index 09e28507f5b2..a14c2938e7af 100644
@@ -332,7 +332,7 @@ between device driver specific code and shared common code:
    walks to fill in the ``args->src`` array with PFNs to be migrated.
    The ``invalidate_range_start()`` callback is passed a
    ``struct mmu_notifier_range`` with the ``event`` field set to
-   ``MMU_NOTIFY_MIGRATE`` and the ``migrate_pgmap_owner`` field set to
+   ``MMU_NOTIFY_MIGRATE`` and the ``owner`` field set to
    the ``args->pgmap_owner`` field passed to migrate_vma_setup(). This is
    allows the device driver to skip the invalidation callback and only
    invalidate device private MMU mappings that are actually migrating.

@@ -405,6 +405,23 @@ between device driver specific code and shared common code:
 
    The lock can now be released.
 
+Exclusive access memory
+=======================
+
+Some devices have features such as atomic PTE bits that can be used to implement
+atomic access to system memory. To support atomic operations to a shared virtual
+memory page such a device needs access to that page which is exclusive of any
+userspace access from the CPU. The ``make_device_exclusive_range()`` function
+can be used to make a memory range inaccessible from userspace.
+
+This replaces all mappings for pages in the given range with special swap
+entries. Any attempt to access the swap entry results in a fault which is
+resovled by replacing the entry with the original mapping. A driver gets
+notified that the mapping has been changed by MMU notifiers, after which point
+it will no longer have exclusive access to the page. Exclusive access is
+guranteed to last until the driver drops the page lock and page reference, at
+which point any CPU faults on the page may proceed as described.
+
 Memory cgroup (memcg) and rss accounting
 ========================================
 

drivers/gpu/drm/nouveau/nouveau_svm.c +1 -1

index f18bd53da052..94f841026c3b 100644
@@ -265,7 +265,7 @@ nouveau_svmm_invalidate_range_start(struct mmu_notifier *mn,
 	 * the invalidation is handled as part of the migration process.
 	 */
 	if (update->event == MMU_NOTIFY_MIGRATE &&
-	    update->migrate_pgmap_owner == svmm->vmm->cli->drm->dev)
+	    update->owner == svmm->vmm->cli->drm->dev)
 		goto out;
 
 	if (limit > svmm->unmanaged.start && start < svmm->unmanaged.limit) {

include/linux/mmu_notifier.h +16 -10

index b8200782dede..2e6068d3fb9f 100644
@@ -41,7 +41,12 @@ struct mmu_interval_notifier;
  *
  * @MMU_NOTIFY_MIGRATE: used during migrate_vma_collect() invalidate to signal
  * a device driver to possibly ignore the invalidation if the
- * migrate_pgmap_owner field matches the driver's device private pgmap owner.
+ * owner field matches the driver's device private pgmap owner.
+ *
+ * @MMU_NOTIFY_EXCLUSIVE: to signal a device driver that the device will no
+ * longer have exclusive access to the page. May ignore the invalidation that's
+ * part of make_device_exclusive_range() if the owner field
+ * matches the value passed to make_device_exclusive_range().
  */
 enum mmu_notifier_event {
 	MMU_NOTIFY_UNMAP = 0,

@@ -51,6 +56,7 @@ enum mmu_notifier_event {
 	MMU_NOTIFY_SOFT_DIRTY,
 	MMU_NOTIFY_RELEASE,
 	MMU_NOTIFY_MIGRATE,
+	MMU_NOTIFY_EXCLUSIVE,
 };
 
 #define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0)

@@ -269,7 +275,7 @@ struct mmu_notifier_range {
 	unsigned long end;
 	unsigned flags;
 	enum mmu_notifier_event event;
-	void *migrate_pgmap_owner;
+	void *owner;
 };
 
 static inline int mm_has_notifiers(struct mm_struct *mm)

@@ -521,14 +527,14 @@ static inline void mmu_notifier_range_init(struct mmu_notifier_range *range,
 	range->flags = flags;
 }
 
-static inline void mmu_notifier_range_init_migrate(
-			struct mmu_notifier_range *range, unsigned int flags,
+static inline void mmu_notifier_range_init_owner(
+			struct mmu_notifier_range *range,
+			enum mmu_notifier_event event, unsigned int flags,
 			struct vm_area_struct *vma, struct mm_struct *mm,
-			unsigned long start, unsigned long end, void *pgmap)
+			unsigned long start, unsigned long end, void *owner)
 {
-	mmu_notifier_range_init(range, MMU_NOTIFY_MIGRATE, flags, vma, mm,
-				start, end);
-	range->migrate_pgmap_owner = pgmap;
+	mmu_notifier_range_init(range, event, flags, vma, mm, start, end);
+	range->owner = owner;
 }
 
 #define ptep_clear_flush_young_notify(__vma, __address, __ptep)		\

@@ -655,8 +661,8 @@ static inline void _mmu_notifier_range_init(struct mmu_notifier_range *range,
 
 #define mmu_notifier_range_init(range,event,flags,vma,mm,start,end)  \
 	_mmu_notifier_range_init(range, start, end)
-#define mmu_notifier_range_init_migrate(range, flags, vma, mm, start, end, \
-					pgmap) \
+#define mmu_notifier_range_init_owner(range, event, flags, vma, mm, start, \
+					end, owner) \
 	_mmu_notifier_range_init(range, start, end)
 
 static inline bool

include/linux/rmap.h +4 -0

index 0e25d829f742..3a1ce4ef9276 100644
@@ -193,6 +193,10 @@ int page_referenced(struct page *, int is_locked,
 bool try_to_migrate(struct page *page, enum ttu_flags flags);
 bool try_to_unmap(struct page *, enum ttu_flags flags);
 
+int make_device_exclusive_range(struct mm_struct *mm, unsigned long start,
+				unsigned long end, struct page **pages,
+				void *arg);
+
 /* Avoid racy checks */
 #define PVMW_SYNC		(1 << 0)
 /* Look for migarion entries rather than present PTEs */

include/linux/swap.h +0 -0

index 516104b9334b..7a3c260146df 100644

include/linux/swap.h +3 -1

 #ifdef CONFIG_DEVICE_PRIVATE
-#define SWP_DEVICE_NUM 2
+#define SWP_DEVICE_NUM 4
 #define SWP_DEVICE_WRITE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM)
 #define SWP_DEVICE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+1)
+#define SWP_DEVICE_EXCLUSIVE_WRITE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+2)
+#define SWP_DEVICE_EXCLUSIVE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+3)
 #else
 #define SWP_DEVICE_NUM 0
 #endif

include/linux/swapops.h +43 -1

index 4dfd807ae52a..4129bd2ff9d6 100644
@@ -120,6 +120,27 @@ static inline bool is_writable_device_private_entry(swp_entry_t entry)
 {
 	return unlikely(swp_type(entry) == SWP_DEVICE_WRITE);
 }
+
+static inline swp_entry_t make_readable_device_exclusive_entry(pgoff_t offset)
+{
+	return swp_entry(SWP_DEVICE_EXCLUSIVE_READ, offset);
+}
+
+static inline swp_entry_t make_writable_device_exclusive_entry(pgoff_t offset)
+{
+	return swp_entry(SWP_DEVICE_EXCLUSIVE_WRITE, offset);
+}
+
+static inline bool is_device_exclusive_entry(swp_entry_t entry)
+{
+	return swp_type(entry) == SWP_DEVICE_EXCLUSIVE_READ ||
+		swp_type(entry) == SWP_DEVICE_EXCLUSIVE_WRITE;
+}
+
+static inline bool is_writable_device_exclusive_entry(swp_entry_t entry)
+{
+	return unlikely(swp_type(entry) == SWP_DEVICE_EXCLUSIVE_WRITE);
+}
 #else /* CONFIG_DEVICE_PRIVATE */
 static inline swp_entry_t make_readable_device_private_entry(pgoff_t offset)
 {

@@ -140,6 +161,26 @@ static inline bool is_writable_device_private_entry(swp_entry_t entry)
 {
 	return false;
 }
+
+static inline swp_entry_t make_readable_device_exclusive_entry(pgoff_t offset)
+{
+	return swp_entry(0, 0);
+}
+
+static inline swp_entry_t make_writable_device_exclusive_entry(pgoff_t offset)
+{
+	return swp_entry(0, 0);
+}
+
+static inline bool is_device_exclusive_entry(swp_entry_t entry)
+{
+	return false;
+}
+
+static inline bool is_writable_device_exclusive_entry(swp_entry_t entry)
+{
+	return false;
+}
 #endif /* CONFIG_DEVICE_PRIVATE */
 
 #ifdef CONFIG_MIGRATION

@@ -219,7 +260,8 @@ static inline struct page *pfn_swap_entry_to_page(swp_entry_t entry)
  */
 static inline bool is_pfn_swap_entry(swp_entry_t entry)
 {
-	return is_migration_entry(entry) || is_device_private_entry(entry);
+	return is_migration_entry(entry) || is_device_private_entry(entry) ||
+	       is_device_exclusive_entry(entry);
 }
 
 struct page_vma_mapped_walk;

lib/test_hmm.c +1 -1

index 80a78877bd93..5c9f5a020c1d 100644
@@ -218,7 +218,7 @@ static bool dmirror_interval_invalidate(struct mmu_interval_notifier *mni,
 	 * the invalidation is handled as part of the migration process.
 	 */
 	if (range->event == MMU_NOTIFY_MIGRATE &&
-	    range->migrate_pgmap_owner == dmirror->mdevice)
+	    range->owner == dmirror->mdevice)
 		return true;
 
 	if (mmu_notifier_range_blockable(range))

mm/hmm.c +5 -0

index 11df3ca30b82..fad6be2bf072 100644
@@ -26,6 +26,8 @@
 #include <linux/mmu_notifier.h>
 #include <linux/memory_hotplug.h>
 
+#include "internal.h"
+
 struct hmm_vma_walk {
 	struct hmm_range	*range;
 	unsigned long		last;

@@ -271,6 +273,9 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
 		if (!non_swap_entry(entry))
 			goto fault;
 
+		if (is_device_exclusive_entry(entry))
+			goto fault;
+
 		if (is_migration_entry(entry)) {
 			pte_unmap(ptep);
 			hmm_vma_walk->last = addr;

mm/memory.c +138 -9

index 3a5705cfc891..556ff396f2e9 100644
@@ -700,6 +700,84 @@ struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
 }
 #endif
 
+static void restore_exclusive_pte(struct vm_area_struct *vma,
+				  struct page *page, unsigned long address,
+				  pte_t *ptep)
+{
+	pte_t pte;
+	swp_entry_t entry;
+
+	pte = pte_mkold(mk_pte(page, READ_ONCE(vma->vm_page_prot)));
+	if (pte_swp_soft_dirty(*ptep))
+		pte = pte_mksoft_dirty(pte);
+
+	entry = pte_to_swp_entry(*ptep);
+	if (pte_swp_uffd_wp(*ptep))
+		pte = pte_mkuffd_wp(pte);
+	else if (is_writable_device_exclusive_entry(entry))
+		pte = maybe_mkwrite(pte_mkdirty(pte), vma);
+
+	set_pte_at(vma->vm_mm, address, ptep, pte);
+
+	/*
+	 * No need to take a page reference as one was already
+	 * created when the swap entry was made.
+	 */
+	if (PageAnon(page))
+		page_add_anon_rmap(page, vma, address, false);
+	else
+		page_add_file_rmap(page, false);
+
+	if (vma->vm_flags & VM_LOCKED)
+		mlock_vma_page(page);
+
+	/*
+	 * No need to invalidate - it was non-present before. However
+	 * secondary CPUs may have mappings that need invalidating.
+	 */
+	update_mmu_cache(vma, address, ptep);
+}
+
+/*
+ * Tries to restore an exclusive pte if the page lock can be acquired without
+ * sleeping. Returns 0 on success or -EBUSY if the page could not be locked or
+ * the entry no longer points at locked_page in which case locked_page should be
+ * locked before retrying the call.
+ */
+static unsigned long
+try_restore_exclusive_pte(struct mm_struct *src_mm, pte_t *src_pte,
+			  struct vm_area_struct *vma, unsigned long addr,
+			  struct page **locked_page)
+{
+	swp_entry_t entry = pte_to_swp_entry(*src_pte);
+	struct page *page = pfn_swap_entry_to_page(entry);
+
+	if (*locked_page) {
+		/* The entry changed, retry */
+		if (unlikely(*locked_page != page)) {
+			unlock_page(*locked_page);
+			put_page(*locked_page);
+			*locked_page = page;
+			return -EBUSY;
+		}
+		restore_exclusive_pte(vma, page, addr, src_pte);
+		unlock_page(page);
+		put_page(page);
+		*locked_page = NULL;
+		return 0;
+	}
+
+	if (trylock_page(page)) {
+		restore_exclusive_pte(vma, page, addr, src_pte);
+		unlock_page(page);
+		return 0;
+	}
+
+	/* The page couldn't be locked so drop the locks and retry. */
+	*locked_page = page;
+	return -EBUSY;
+}
+
 /*
  * copy one vm_area from one task to the other. Assumes the page tables
  * already present in the new task to be cleared in the whole range

@@ -781,6 +859,12 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 				pte = pte_swp_mkuffd_wp(pte);
 			set_pte_at(src_mm, addr, src_pte, pte);
 		}
+	} else if (is_device_exclusive_entry(entry)) {
+		/* COW mappings should be dealt with by removing the entry */
+		VM_BUG_ON(is_cow_mapping(vm_flags));
+		page = pfn_swap_entry_to_page(entry);
+		get_page(page);
+		rss[mm_counter(page)]++;
 	}
 	set_pte_at(dst_mm, addr, dst_pte, pte);
 	return 0;

@@ -947,6 +1031,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
 	int rss[NR_MM_COUNTERS];
 	swp_entry_t entry = (swp_entry_t){0};
 	struct page *prealloc = NULL;
+	struct page *locked_page = NULL;
 
 again:
 	progress = 0;

@@ -980,13 +1065,36 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
 			continue;
 		}
 		if (unlikely(!pte_present(*src_pte))) {
-			entry.val = copy_nonpresent_pte(dst_mm, src_mm,
-							dst_pte, src_pte,
-							src_vma, addr, rss);
-			if (entry.val)
-				break;
-			progress += 8;
-			continue;
+			swp_entry_t swp_entry = pte_to_swp_entry(*src_pte);
+
+			if (unlikely(is_cow_mapping(src_vma->vm_flags) &&
+			    is_device_exclusive_entry(swp_entry))) {
+				/*
+				 * Normally this would require sending mmu
+				 * notifiers, but copy_page_range() has already
+				 * done that for COW mappings.
+				 */
+				ret = try_restore_exclusive_pte(src_mm, src_pte,
+								src_vma, addr,
+								&locked_page);
+				if (ret == -EBUSY)
+					break;
+			} else {
+				entry.val = copy_nonpresent_pte(dst_mm, src_mm,
+								dst_pte, src_pte,
+								src_vma, addr,
+								rss);
+				if (entry.val)
+					break;
+				progress += 8;
+				continue;
+			}
+		}
+		/* a non-present pte became present after dropping the ptl */
+		if (unlikely(locked_page)) {
+			unlock_page(locked_page);
+			put_page(locked_page);
+			locked_page = NULL;
 		}
 		/* copy_present_pte() will clear `*prealloc' if consumed */
 		ret = copy_present_pte(dst_vma, src_vma, dst_pte, src_pte,

@@ -1023,6 +1131,11 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
 			goto out;
 		}
 		entry.val = 0;
+	} else if (ret == -EBUSY) {
+		if (get_page_unless_zero(locked_page))
+			lock_page(locked_page);
+		else
+			locked_page = NULL;
 	} else if (ret) {
 		WARN_ON_ONCE(ret != -EAGAIN);
 		prealloc = page_copy_prealloc(src_mm, src_vma, addr);

@@ -1287,7 +1400,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 		}
 
 		entry = pte_to_swp_entry(ptent);
-		if (is_device_private_entry(entry)) {
+		if (is_device_private_entry(entry) ||
+		    is_device_exclusive_entry(entry)) {
 			struct page *page = pfn_swap_entry_to_page(entry);
 
 			if (unlikely(details && details->check_mapping)) {

@@ -1303,7 +1417,10 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 
 			pte_clear_not_present_full(mm, addr, pte, tlb->fullmm);
 			rss[mm_counter(page)]--;
-			page_remove_rmap(page, false);
+
+			if (is_device_private_entry(entry))
+				page_remove_rmap(page, false);
+
 			put_page(page);
 			continue;
 		}

@@ -3256,6 +3373,44 @@ void unmap_mapping_range(struct address_space *mapping,
 }
 EXPORT_SYMBOL(unmap_mapping_range);
 
+/*
+ * Restore a potential device exclusive pte to a working pte entry
+ */
+static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf)
+{
+	struct page *page = vmf->page;
+	struct vm_area_struct *vma = vmf->vma;
+	struct page_vma_mapped_walk pvmw = {
+		.page = page,
+		.vma = vma,
+		.address = vmf->address,
+		.flags = PVMW_SYNC,

mm/memory.c +11 -0

+	}
+
+	unlock_page(page);
+
+	mmu_notifier_invalidate_range_end(&range);
+	return ret;
+}
+
 /*
  * We enter with non-exclusive mmap_lock (to exclude vma changes,
  * but allow concurrent faults), and pte mapped but not yet locked.

@@ -3283,6 +3438,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		if (is_migration_entry(entry)) {
 			migration_entry_wait(vma->vm_mm, vmf->pmd,
 					     vmf->address);
+		} else if (is_device_exclusive_entry(entry)) {
+			vmf->page = pfn_swap_entry_to_page(entry);
+			ret = remove_device_exclusive_entry(vmf);
 		} else if (is_device_private_entry(entry)) {
 			vmf->page = pfn_swap_entry_to_page(entry);
 			ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);

mm/migrate.c +2 -2

index cc4612e2a246..9cc9251d4802 100644
@@ -2570,8 +2570,8 @@ static void migrate_vma_collect(struct migrate_vma *migrate)
 	 * that the registered device driver can skip invalidating device
 	 * private page mappings that won't be migrated.
 	 */
-	mmu_notifier_range_init_migrate(&range, 0, migrate->vma,
-		migrate->vma->vm_mm, migrate->start, migrate->end,
+	mmu_notifier_range_init_owner(&range, MMU_NOTIFY_MIGRATE, 0,
+		migrate->vma, migrate->vma->vm_mm, migrate->start, migrate->end,
 		migrate->pgmap_owner);
 	mmu_notifier_invalidate_range_start(&range);
 

mm/migrate.c +0 -0

 				mmu_notifier_invalidate_range_start(&range);
 			}

mm/mprotect.c +8 -0

index f21b760ec809..c6018541ea3d 100644
@@ -165,6 +165,14 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 				newpte = swp_entry_to_pte(entry);
 				if (pte_swp_uffd_wp(oldpte))
 					newpte = pte_swp_mkuffd_wp(newpte);
+			} else if (is_writable_device_exclusive_entry(entry)) {
+				entry = make_readable_device_exclusive_entry(
+							swp_offset(entry));
+				newpte = swp_entry_to_pte(entry);
+				if (pte_swp_soft_dirty(oldpte))
+					newpte = pte_swp_mksoft_dirty(newpte);
+				if (pte_swp_uffd_wp(oldpte))
+					newpte = pte_swp_mkuffd_wp(newpte);
 			} else {
 				newpte = oldpte;
 			}

mm/page_vma_mapped.c +6 -3

index eed988ab2e81..29842f169219 100644
@@ -41,7 +41,8 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw)
 
 				/* Handle un-addressable ZONE_DEVICE memory */
 				entry = pte_to_swp_entry(*pvmw->pte);
-				if (!is_device_private_entry(entry))
+				if (!is_device_private_entry(entry) &&
+				    !is_device_exclusive_entry(entry))
 					return false;
 			} else if (!pte_present(*pvmw->pte))
 				return false;

@@ -93,7 +94,8 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw)
 			return false;
 		entry = pte_to_swp_entry(*pvmw->pte);
 
-		if (!is_migration_entry(entry))
+		if (!is_migration_entry(entry) &&
+		    !is_device_exclusive_entry(entry))
 			return false;
 
 		pfn = swp_offset(entry);

@@ -102,7 +104,8 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw)
 
 		/* Handle un-addressable ZONE_DEVICE memory */
 		entry = pte_to_swp_entry(*pvmw->pte);
-		if (!is_device_private_entry(entry))
+		if (!is_device_private_entry(entry) &&
+		    !is_device_exclusive_entry(entry))
 			return false;
 
 		pfn = swp_offset(entry);

mm/rmap.c +18 -0

index 7f91f058f1f5..32b99a7bb358 100644
@@ -2005,6 +2005,216 @@ void page_mlock(struct page *page)
 	rmap_walk(page, &rwc);
 }
 
+struct ttp_args {
+	struct mm_struct *mm;
+	unsigned long address;
+	void *arg;
+	bool valid;
+};
+
+static bool try_to_protect_one(struct page *page, struct vm_area_struct *vma,
+			unsigned long address, void *arg)
+{
+	struct mm_struct *mm = vma->vm_mm;
+	struct page_vma_mapped_walk pvmw = {
+		.page = page,
+		.vma = vma,
+		.address = address,
+	};
+	struct ttp_args *ttp = arg;
+	pte_t pteval;

mm/rmap.c +5 -0

+
+		if (arch_unmap_one(mm, vma, address, pteval) < 0) {
+			set_pte_at(mm, address, pvmw.pte, pteval);
+			ret = false;
+			page_vma_mapped_walk_done(&pvmw);

mm/rmap.c +11 -0

+		swp_pte = swp_entry_to_pte(entry);
+		if (pte_soft_dirty(pteval))
+			swp_pte = pte_swp_mksoft_dirty(swp_pte);
+		if (pte_uffd_wp(pteval))
+			swp_pte = pte_swp_mkuffd_wp(swp_pte);
+
+		/* Take a reference for the swap entry */
+		get_page(page);
+		set_pte_at(mm, address, pvmw.pte, swp_pte);
+
+		page_remove_rmap(subpage, PageHuge(page));

mm/rmap.c +33 -0

+ * @mm: the mm_struct where the page is expected to be mapped
+ * @address: address where the page is expected to be mapped
+ * @arg: passed to MMU_NOTIFY_EXCLUSIVE range notifier callbacks
+ *
+ * Tries to remove all the page table entries which are mapping this page and
+ * replace them with special swap entries to grant a device exclusive access to
+ * the page. Caller must hold the page lock.
+ *
+ * Returns false if the page is still mapped, or if it could not be unmapped
+ * from the expected address. Otherwise returns true (success).
+ */
+static bool try_to_protect(struct page *page, struct mm_struct *mm,
+			   unsigned long address, void *arg)
+{
+	struct ttp_args ttp = {
+		.mm = mm,
+		.address = address,
+		.arg = arg,
+		.valid = false,
+	};
+	struct rmap_walk_control rwc = {
+		.rmap_one = try_to_protect_one,
+		.done = page_not_mapped,
+		.anon_lock = page_lock_anon_vma_read,
+		.arg = &ttp,
+	};
+
+	/*
+	 * Restrict to anonymous pages for now to avoid potential writeback
+	 * issues.
+	 */
+	if (!PageAnon(page))
+		return false;

mm/rmap.c +4 -0

+		rwc.invalid_vma = invalid_migration_vma;
+
+	rmap_walk(page, &rwc);
+